Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shim for Databricks 9.1 [databricks] #3767

Merged
merged 13 commits into from
Nov 2, 2021
Merged

Conversation

jlowe
Copy link
Member

@jlowe jlowe commented Oct 7, 2021

Fixes #3370, #3352.

Adds shim code to support the Databricks 9.1 environment.

Signed-off-by: Jason Lowe <jlowe@nvidia.com>
@jlowe jlowe added this to the Oct 4 - Oct 15 milestone Oct 7, 2021
@jlowe jlowe self-assigned this Oct 7, 2021
@jlowe
Copy link
Member Author

jlowe commented Oct 7, 2021

Note that the code in the sql-plugin v2 shims is mostly identical with either 311db or 310until320 with the exception of TreeNode since Databricks decided to pull back that change from 3.2. Otherwise I think it mostly adheres to the same style the 311db shim was doing. Open to suggestions on what names/refactoring to use to reduce the code duplication there.

@jlowe
Copy link
Member Author

jlowe commented Oct 7, 2021

build

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to get ci builds setup before checking this in, otherwise builds that try to pull this jar will fail.

pom.xml Outdated Show resolved Hide resolved
@sameerz sameerz added the feature request New feature or request label Oct 8, 2021
@jlowe jlowe linked an issue Oct 8, 2021 that may be closed by this pull request
@jlowe
Copy link
Member Author

jlowe commented Oct 8, 2021

build

dist/pom.xml Outdated Show resolved Hide resolved
@jlowe
Copy link
Member Author

jlowe commented Oct 25, 2021

@tgravescs @NvTimLiu please take another look. If this looks correct then I'll revert the changes to the dist pom (so we don't try to pull in the 312db classifier jar when we don't have a Jenkins build setup yet to produce it) and get that through CI. Then we can setup the Jenkins build to produce the spark312db jar and followup with a PR to restore the dist pom changes to pull it into the monolithic dist jar.

tgravescs
tgravescs previously approved these changes Oct 27, 2021
@jlowe
Copy link
Member Author

jlowe commented Nov 2, 2021

build

@jlowe
Copy link
Member Author

jlowe commented Nov 2, 2021

I've manually verified all tests pass, including the cudf_udf tests once the environment is setup (which is a bit different since conda is no longer provided on the node). I left out the changes to dist/pom.xml so this shouldn't be trying to pull in the spark312db shim in the nightly build until we have a chance to setup the nightly CI pipeline to build and publish the artifacts.

@jlowe jlowe merged commit bbf1590 into NVIDIA:branch-21.12 Nov 2, 2021
@jlowe jlowe deleted the db91-shim branch November 2, 2021 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] DatabricksShimVersion must carry runtime version info [FEA] Add support for Databricks 9.1 runtime
5 participants