Commonize more shim module files [databricks] #3577
Conversation
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
… into commonpands
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
pom.xml
Outdated
| @@ -141,6 +142,7 @@ | |||
| <source>${project.basedir}/src/main/301until310-nondb/scala</source> | |||
| <source>${project.basedir}/src/main/301until320-nondb/scala</source> | |||
| <source>${project.basedir}/src/main/301until320-all/scala</source> | |||
| <source>${project.basedir}/src/main/301until310-all/scala</source> | |||
There was a problem hiding this comment.
nit: There are getting to be enough directories now, that it would be good to have a consistent ordering to them, so it is simpler to tell if something is missing for example.
There was a problem hiding this comment.
ah, I missed a few I tried moved some of them but didn't get them all. I need to update my IntelliJ as well as Gera mentioned it has a sort lines in new version, but we should decide on order
|
fyi - seeing some test failures locally so investigating them. |
|
I am happy to help resolve the conflicts. Those are related to a patch I did, that really just wants us to fall back to the CPU when ANSI is enabled for some timestamp operations. The fallback is for Spark 3.1.1+ |
|
wait to build until #3580 fixed |
| @@ -14,62 +14,47 @@ | |||
| * limitations under the License. | |||
| */ | |||
|
|
|||
| package com.nvidia.spark.rapids.shims.spark313 | |||
| package com.nvidia.spark.rapids.shims.v2 | |||
There was a problem hiding this comment.
this whole file is a very odd diff, it did not get moved from 313 ithis got copied from GpuWindowExpression.scala
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
This commonizes the majority of the rest of the shim module files excluding shuffle and the main Spark3XXShim file.
I tried to keep this easy to review and mostly moved files, a few functions in SparkBaseShims had to move to the respective Spark3XX file to make it common.