Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Commonize shim module join and shuffle files #3559

Merged
merged 18 commits into from
Sep 21, 2021
Merged

Commonize shim module join and shuffle files #3559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4aa0209
rework databricks shuffle to match others
tgravescs Sep 17, 2021
351c986
Commonize GpuShuffledHashJoinExec
tgravescs Sep 17, 2021
f29f2ee
commonize GpuBroadcastNestedLoopJoinExec
tgravescs Sep 17, 2021
5d7f5a7
301db has different ShuffledHashJoinExec
tgravescs Sep 17, 2021
97b5478
Fix 301db
tgravescs Sep 17, 2021
8f96327
Merge branch 'branch-21.10' of github.com:NVIDIA/spark-rapids into co…
tgravescs Sep 17, 2021
f81a8a2
Merge branch 'commonJoin' of github.com:tgravescs/spark-rapids into c…
tgravescs Sep 17, 2021
7e8bf65
update databricks 311 api to match
tgravescs Sep 17, 2021
5cea4cc
Move a bunch more join and exchange files to common
tgravescs Sep 17, 2021
989a982
fixes
tgravescs Sep 17, 2021
5a325e7
Fix 320
tgravescs Sep 17, 2021
a9d8f63
updates
tgravescs Sep 20, 2021
e499644
cleanup extra files
tgravescs Sep 20, 2021
d73cce2
fix include
tgravescs Sep 20, 2021
095edcd
Merge remote-tracking branch 'origin/branch-21.10' into commonJoin
tgravescs Sep 20, 2021
2885609
Fix imports
tgravescs Sep 20, 2021
41669d3
Merge remote-tracking branch 'origin/branch-21.10' into commonJoin
tgravescs Sep 20, 2021
e8fa559
Rename files to Base
tgravescs Sep 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -140,6 +141,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -184,6 +186,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -223,6 +226,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -263,7 +267,9 @@
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: here and elsewhere now that we touch this section often let us keep the lines sorted for easier reviews. In IDEA "Edit->Sort Lines" produces the order

                                        <source>${project.basedir}/src/main/301+-nondb/scala</source>
                                        <source>${project.basedir}/src/main/301until320-all/scala</source>
                                        <source>${project.basedir}/src/main/301until320-nondb/scala</source>
                                        <source>${project.basedir}/src/main/311+-all/scala</source>
                                        <source>${project.basedir}/src/main/311+-nondb/scala</source>
                                        <source>${project.basedir}/src/main/311until320-all/scala</source>
                                        <source>${project.basedir}/src/main/311until320-apache/scala</source>

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry missed this one, I'll fix it in the next pr

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also note I have older ide which doesn't have that option. I'll look at upgrading later, but I'm not sure I agree with its sorting going by the other ones, for instance:

<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
 <source>${project.basedir}/src/main/301until310-all/scala</source>
 <source>${project.basedir}/src/main/301+-nondb/scala</source>

Why is it putting nondb before all, doesn't seem like by alphabetical? As long as we are consistent I'm fine with it but don't want to require certain version of IDE either though. Or perhaps those weren't sorted?

</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -410,7 +416,9 @@
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -451,7 +459,9 @@
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -490,6 +500,8 @@
<sources>
<source>${project.basedir}/src/main/320/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -530,6 +542,8 @@
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
</execution>
Expand Down
40 changes: 0 additions & 40 deletions ...rc/main/scala/com/nvidia/spark/rapids/shims/spark301/GpuBroadcastNestedLoopJoinExec.scala
View file
Open in desktop

This file was deleted.

49 changes: 0 additions & 49 deletions ...ark301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/GpuShuffleExchangeExec.scala
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion shims/spark301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/SparkBaseShims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import java.nio.ByteBuffer
import com.esotericsoftware.kryo.Kryo
import com.esotericsoftware.kryo.serializers.{JavaSerializer => KryoJavaSerializer}
import com.nvidia.spark.rapids._
import com.nvidia.spark.rapids.shims.v2.Spark30XShims
import com.nvidia.spark.rapids.shims.v2._
import org.apache.arrow.memory.ReferenceManager
import org.apache.arrow.vector.ValueVector
import org.apache.hadoop.fs.{FileStatus, Path}
Expand Down
40 changes: 0 additions & 40 deletions .../main/scala/com/nvidia/spark/rapids/shims/spark301db/GpuBroadcastNestedLoopJoinExec.scala
View file
Open in desktop

This file was deleted.

89 changes: 0 additions & 89 deletions ...1db/src/main/scala/com/nvidia/spark/rapids/shims/spark301db/GpuShuffledHashJoinExec.scala
View file
Open in desktop

This file was deleted.

Loading