交集、并集、差集

交集会产生shuffle操作

差集

  2.   /**
  3.     * 交集
  4.     */
  5.   @Test
  6.   def intersection(): Unit ={
  7.     val rdd1 = sc.parallelize(List(1,2,3,4,5))
  8.     val rdd2 = sc.parallelize(List(4,5,6,7,8,9))
  10.     val rdd3 = rdd1.intersection(rdd2)
  12.     println(rdd3.collect().toList)
  13.   }
  15.   /**
  16.     * 差集
  17.     */
  18.   @Test
  19.   def subtract(): Unit ={
  20.     val rdd1 = sc.parallelize(List(1,2,3,4,5))
  21.     val rdd2 = sc.parallelize(List(4,5,6,7,8,9))
  23.     val rdd3 = rdd1.subtract(rdd2)
  25.     println(rdd3.collect().toList)
  26.   }
  28.   @Test
  29.   def union(): Unit ={
  30.     val rdd1 = sc.parallelize(List(1,2,3,4,5))
  31.     val rdd2 = sc.parallelize(List(4,5,6,7,8,9))
  33.     val rdd3 = rdd1.union(rdd2)
  35.     println(rdd3.partitions.length)
  36.     println(rdd3.collect().toList)
  37.   }

zip拉链

  1.   /**
  2.     * spark的拉链必须要求两个RDD的分区数以及元素个数一致
  3.     */
  4.   @Test
  5.   def zip(): Unit ={
  6.     val rdd1 = sc.parallelize(List("zhangsan","lisi","wangwu"),5)
  7.     val rdd2 = sc.parallelize(List(20,30,40))
  9.     val rdd3 = rdd1.zip(rdd2)
  11.     println(rdd3.collect().toList)
  12.   }

join

一个join会产生两个shuffler，用广播变量减少shuffler

union

union并集没有shuffle，RDD分区数=union的两个rdd分区数总和