判断一个算子是否会导致shuffle的方法

微电子学与固体电子学-俞驰

于 2020-08-01 18:27:48 发布

阅读量431

点赞数

CC 4.0 BY-SA版权

分类专栏： Scala与Spark

原文链接：https://stackoverflow.com/questions/34597255/how-to-use-aggregatebykey-on-javapairrdd-in-java

Scala与Spark 专栏收录该内容

119 篇文章

订阅专栏

本文详细解析了Apache Spark中RDD的distinct操作过程，通过对比parallelize创建的RDD与应用distinct后的RDD，展示了数据去重的实现机制。文章通过具体的代码示例，深入探讨了MapPartitionsRDD和ShuffledRDD在distinct操作中的角色，为读者提供了理解Spark内部工作原理的视角。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scala> val a = sc.parallelize(Array(1,2,3)).distinct
a: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[3] at distinct at <console>:24

scala> a.toDebugString
res0: String =
(16) MapPartitionsRDD[3] at distinct at <console>:24 []
| ShuffledRDD[2] at distinct at <console>:24 []
+-(16) MapPartitionsRDD[1] at distinct at <console>:24 []
| ParallelCollectionRDD[0] at parallelize at <console>:24 []

scala>

scala> val b = sc.parallelize(Array(1,2,3))
b: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[4] at parallelize at <console>:24

scala> b.toDebugString
res1: String = (16) ParallelCollectionRDD[4] at parallelize at <console>:24 []