Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

初级问题,就是文章里是有很多RDD的依赖关系图,但是我找不到这些plan怎么在worker执行的相关代码?多谢! #37

Open
guotong1988 opened this issue Aug 19, 2015 · 1 comment

Comments

@guotong1988
Copy link

比如这个testcase

test("sortByKey") {

val pairs = sc.parallelize(Array((1, 0), (2, 0), (0, 0), (3, 0)), 2)

assert(pairs.sortByKey().collect() === Array((0, 0), (1, 0), (2, 0), (3, 0)))

}

collect()之前都是一个一个new新的RDD,好像没有实际计算,之后runJob里面又debug不到,

全部的归并结果的代码应该在RDD.scala的Array.concat(results: _*)

def collect(): Array[T] = withScope {

val results = sc.runJob(this, (iter: Iterator[T]) => iter.toArray)

Array.concat(results: _*)

}

您的shuffle那章有sortByKey的转化图,可是这些计算过程在哪呢?

@JerryLead
Copy link
Owner

计算逻辑在RDD.compute(),计算过程是pipeline的,你可以通过finalRDD.debug()看到RDD的依赖图,建议你仔细看下LogicalPlan和PhysicalPlan那两章就明白了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants