Use Shims v2 for ShuffledBatchRDD #3459
Conversation
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
revans2
requested review from
GaryShen2008,
jlowe and
NvTimLiu
as code owners
|
build |
| val n = parent.numPartitions | ||
| val result = new Array[Int](n) | ||
| for (i <- partitionStartIndices.indices) { | ||
| for (i <- 0 until partitionStartIndices.length) { |
There was a problem hiding this comment.
Just a note. There are several changes here, like this one, that make it look like the code is going backwards based off of the IDEs warnings. This is because I made the code match as closely as possible to the Spark 3.2.0-RC2 code. This was so that differences between it and other versions of the code are much more obvious.
| numReducers, | ||
| context, | ||
| sqlMetricsReporter) | ||
| val blocksByAddress = shim.getMapSizesByExecutorId( |
There was a problem hiding this comment.
This is the only place that I have really added new code from scratch. The rest has been refactoring existing code and pulling in things either from Spark or from existing code in our plugin. I am not really sure how to trigger this part of the code base to test it. If someone really wants me to get tests for this I'll try to find a way to make sure that I cover all of the existing code paths.
|
Converted to draft because I need to work out some issues with datbricks build. |
|
build |
1 similar comment
|
build |
|
Build failed trying to get the code coverage dependency, so I kicked it again. |
|
build |
|
build |
1 similar comment
|
build |
This fixes #3372