[SPARK-29296][BUILD][CORE] Remove use of .par to make 2.13 support easier; add scala-2.13 profile to enable pulling in par collections library separately, for the future #25980
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e to enable pulling in par collections library separately, for the future
|
Test build #111617 has finished for PR 25980 at commit
|
MaxGekk
reviewed
Oct 1, 2019
| @@ -113,7 +115,7 @@ class CastSuite extends SparkFunSuite with ExpressionEvalHelper { | |||
| } | |||
|
|
|||
| test("cast string to timestamp") { | |||
| ALL_TIMEZONES.par.foreach { tz => | |||
| new ParVector(ALL_TIMEZONES.toVector).foreach { tz => | |||
There was a problem hiding this comment.
ALL_TIMEZONES is Seq here. Maybe, ParSeq instead of ParVector?
There was a problem hiding this comment.
ParSeq is a trait unfortunately. I think this is the right impl for this case.
| @@ -46,7 +46,7 @@ class SQLExecutionSuite extends SparkFunSuite { | |||
| import spark.implicits._ | |||
| try { | |||
| // Should not throw IllegalArgumentException | |||
| (1 to 100).par.foreach { _ => | |||
| new ParVector((1 to 100).toVector).foreach { _ => | |||
srowen
changed the title
|
Test build #111645 has finished for PR 25980 at commit
|
|
Merged to master |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Scala 2.13 removes the parallel collections classes to a separate library, so first, this establishes a
scala-2.13profile to bring it back, for future use.However the library enables use of
.parimplicit conversions via a new class that is not in 2.12, which makes cross-building hard. This implements a suggested workaround from scala/scala-parallel-collections#22 to avoid.parentirely.Why are the changes needed?
To compile for 2.13 and later to work with 2.13.
Does this PR introduce any user-facing change?
Should not, no.
How was this patch tested?
Existing tests.