-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add shim for Databricks 9.1 [databricks] #3767
Conversation
Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Note that the code in the sql-plugin v2 shims is mostly identical with either 311db or 310until320 with the exception of |
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we need to get ci builds setup before checking this in, otherwise builds that try to pull this jar will fail.
shims/spark312db/src/main/scala/com/nvidia/spark/rapids/spark312db/RapidsShuffleManager.scala
Show resolved
Hide resolved
build |
@tgravescs @NvTimLiu please take another look. If this looks correct then I'll revert the changes to the dist pom (so we don't try to pull in the 312db classifier jar when we don't have a Jenkins build setup yet to produce it) and get that through CI. Then we can setup the Jenkins build to produce the spark312db jar and followup with a PR to restore the dist pom changes to pull it into the monolithic dist jar. |
...312db/src/main/scala/com/nvidia/spark/rapids/shims/spark312db/SparkShimServiceProvider.scala
Outdated
Show resolved
Hide resolved
...311db/src/main/scala/com/nvidia/spark/rapids/shims/spark311db/SparkShimServiceProvider.scala
Outdated
Show resolved
Hide resolved
build |
I've manually verified all tests pass, including the cudf_udf tests once the environment is setup (which is a bit different since conda is no longer provided on the node). I left out the changes to dist/pom.xml so this shouldn't be trying to pull in the spark312db shim in the nightly build until we have a chance to setup the nightly CI pipeline to build and publish the artifacts. |
Fixes #3370, #3352.
Adds shim code to support the Databricks 9.1 environment.