Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Commonize more shim module files [databricks] #3577

Merged
merged 54 commits into from
Sep 22, 2021
Merged

Commonize more shim module files [databricks] #3577

Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
4aa0209
rework databricks shuffle to match others
tgravescs Sep 17, 2021
351c986
Commonize GpuShuffledHashJoinExec
tgravescs Sep 17, 2021
f29f2ee
commonize GpuBroadcastNestedLoopJoinExec
tgravescs Sep 17, 2021
5d7f5a7
301db has different ShuffledHashJoinExec
tgravescs Sep 17, 2021
97b5478
Fix 301db
tgravescs Sep 17, 2021
8f96327
Merge branch 'branch-21.10' of github.com:NVIDIA/spark-rapids into co…
tgravescs Sep 17, 2021
f81a8a2
Merge branch 'commonJoin' of github.com:tgravescs/spark-rapids into c…
tgravescs Sep 17, 2021
7e8bf65
update databricks 311 api to match
tgravescs Sep 17, 2021
5cea4cc
Move a bunch more join and exchange files to common
tgravescs Sep 17, 2021
989a982
fixes
tgravescs Sep 17, 2021
5a325e7
Fix 320
tgravescs Sep 17, 2021
a9d8f63
updates
tgravescs Sep 20, 2021
e499644
cleanup extra files
tgravescs Sep 20, 2021
d73cce2
fix include
tgravescs Sep 20, 2021
095edcd
Merge remote-tracking branch 'origin/branch-21.10' into commonJoin
tgravescs Sep 20, 2021
2885609
Fix imports
tgravescs Sep 20, 2021
4582003
Commonize python files from shim module
tgravescs Sep 20, 2021
b485424
commonize more
tgravescs Sep 20, 2021
e2acfc0
fixes
tgravescs Sep 20, 2021
9df4983
320 working
tgravescs Sep 20, 2021
9705480
fix 311db
tgravescs Sep 20, 2021
0e45394
Merge branch 'commonpands' of https://github.com/tgravescs/spark-rapi…
tgravescs Sep 20, 2021
0cf1a86
Fix 301db
tgravescs Sep 20, 2021
41669d3
Merge remote-tracking branch 'origin/branch-21.10' into commonJoin
tgravescs Sep 20, 2021
0f6d63c
move GpuOrcScan
tgravescs Sep 20, 2021
40ab13a
commonize GpuParquetScan
tgravescs Sep 20, 2021
daf67a0
remove unneeded files
tgravescs Sep 20, 2021
9750eaf
fixes
tgravescs Sep 21, 2021
0d4c465
Merge remote-tracking branch 'origin/branch-21.10' into commonpands
tgravescs Sep 21, 2021
420f3c9
remove unneeded
tgravescs Sep 21, 2021
582b304
Commonize Base shims for 30x
tgravescs Sep 21, 2021
6463d47
changes
tgravescs Sep 21, 2021
9e7b31d
move 31x and db base shims
tgravescs Sep 21, 2021
4421070
fixes
tgravescs Sep 21, 2021
1d2d225
Merge remote-tracking branch 'tgravescs/commonJoin' into commonpandsW…
tgravescs Sep 21, 2021
32fa617
updates
tgravescs Sep 21, 2021
01de6e7
move the rest of databricks files
tgravescs Sep 21, 2021
483b54f
301db fixes
tgravescs Sep 21, 2021
2a8112e
Merge remote-tracking branch 'origin/branch-21.10' into commonpandsWi…
tgravescs Sep 21, 2021
fb6b37b
fix copyrights
tgravescs Sep 21, 2021
c1a8156
cleanup
tgravescs Sep 21, 2021
a885cb1
Merge remote-tracking branch 'origin/branch-21.10' into commonpandsWi…
tgravescs Sep 21, 2021
6ac913f
remove lang3 workaround
tgravescs Sep 21, 2021
dbf29c9
common order
tgravescs Sep 21, 2021
aaba484
move
tgravescs Sep 21, 2021
e6e7084
Move OffsetwindowFunctionMeta into sql plugin and make it version
tgravescs Sep 21, 2021
0360a07
Merge remote-tracking branch 'origin/branch-21.10' into commonpandsWi…
tgravescs Sep 21, 2021
a96096a
Add missing file
tgravescs Sep 21, 2021
b410ab3
move 320 offset window
tgravescs Sep 21, 2021
b27d1d8
Sort lines in pom
tgravescs Sep 21, 2021
4609147
cleanup
tgravescs Sep 21, 2021
662b0ef
Merge remote-tracking branch 'origin/branch-21.10' into commonpandsWi…
tgravescs Sep 22, 2021
6c38344
Ensure the correct ParquetCachedBatchSerializer picked up
tgravescs Sep 22, 2021
8f4f85a
spacing
tgravescs Sep 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 21 additions & 12 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@
<sources>
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until310-all/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
Expand Down Expand Up @@ -141,6 +142,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301until310-all/scala</source>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: There are getting to be enough directories now, that it would be good to have a consistent ordering to them, so it is simpler to tell if something is missing for example.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I missed a few I tried moved some of them but didn't get them all. I need to update my IntelliJ as well as Gera mentioned it has a sort lines in new version, but we should decide on order

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorted them

<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
Expand Down Expand Up @@ -186,6 +188,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301until310-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
Expand Down Expand Up @@ -226,6 +229,7 @@
<source>${project.basedir}/src/main/301until310-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/301until310-all/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
</sources>
</configuration>
Expand Down Expand Up @@ -264,12 +268,13 @@
<configuration>
<sources>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -322,6 +327,7 @@
<configuration>
<sources>
<source>${project.basedir}/src/main/301db/scala</source>
<source>${project.basedir}/src/main/301until310-all/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
</sources>
</configuration>
Expand Down Expand Up @@ -375,8 +381,8 @@
<source>${project.basedir}/src/main/311db/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -413,12 +419,13 @@
<configuration>
<sources>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -456,12 +463,13 @@
<configuration>
<sources>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-apache/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -499,9 +507,9 @@
<configuration>
<sources>
<source>${project.basedir}/src/main/320/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down Expand Up @@ -537,13 +545,14 @@
<phase>generate-sources</phase>
<configuration>
<sources>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/311cdh/scala</source>
<source>${project.basedir}/src/main/301until320-nondb/scala</source>
<source>${project.basedir}/src/main/311until320-nondb/scala</source>
<source>${project.basedir}/src/main/301until320-all/scala</source>
<source>${project.basedir}/src/main/311until320-all/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/301+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-nondb/scala</source>
<source>${project.basedir}/src/main/311+-all/scala</source>
</sources>
</configuration>
</execution>
Expand Down
42 changes: 0 additions & 42 deletions ...park301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/GpuWindowInPandasExec.scala
View file
Open in desktop

This file was deleted.

18 changes: 18 additions & 0 deletions shims/spark301/src/main/scala/com/nvidia/spark/rapids/shims/spark301/Spark301Shims.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,12 @@
package com.nvidia.spark.rapids.shims.spark301

import com.nvidia.spark.rapids._
import com.nvidia.spark.rapids.shims.v2.SparkBaseShims
import com.nvidia.spark.rapids.spark301.RapidsShuffleManager
import org.apache.parquet.schema.MessageType

import org.apache.spark.sql.execution.datasources.parquet.ParquetFilters
import org.apache.spark.sql.internal.SQLConf

class Spark301Shims extends SparkBaseShims {

Expand All @@ -26,4 +31,17 @@ class Spark301Shims extends SparkBaseShims {
override def getRapidsShuffleManagerClass: String = {
classOf[RapidsShuffleManager].getCanonicalName
}

override def getParquetFilters(
schema: MessageType,
pushDownDate: Boolean,
pushDownTimestamp: Boolean,
pushDownDecimal: Boolean,
pushDownStartWith: Boolean,
pushDownInFilterThreshold: Int,
caseSensitive: Boolean,
datetimeRebaseMode: SQLConf.LegacyBehaviorPolicy.Value): ParquetFilters = {
new ParquetFilters(schema, pushDownDate, pushDownTimestamp, pushDownDecimal, pushDownStartWith,
pushDownInFilterThreshold, caseSensitive)
}
}
Loading