Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADD 321cdh shim to rapids and remove 311cdh shim #5346

Merged
merged 7 commits into from
May 2, 2022

Conversation

sririshindra
Copy link
Contributor

@sririshindra sririshindra commented Apr 27, 2022

This PR adds cdh321 shim and removes 311cdh shim.

contributes to #5281 and fixes #5404

@sameerz sameerz added the feature request New feature or request label Apr 28, 2022
@tgravescs
Copy link
Collaborator

pom.xml Show resolved Hide resolved
pom.xml Outdated Show resolved Hide resolved
pom.xml Outdated Show resolved Hide resolved
dist/pom.xml Show resolved Hide resolved
@tgravescs
Copy link
Collaborator

Also please upmerge to the latest branch-22.06

@tgravescs
Copy link
Collaborator

Build

@tgravescs
Copy link
Collaborator

it looks like we missed the premerge script being updated. @sririshindra can you update https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/jenkins/spark-premerge-build.sh#L43 to change 311cdh to 321cdh

Could you also update our docs to have 321cdh instead of 311cdh:
docs/additional-functionality/rapids-shuffle.md
CONTRIBUTING.md

dist/pom.xml Outdated Show resolved Hide resolved
@tgravescs
Copy link
Collaborator

tgravescs commented May 2, 2022

Trying to run locally the unit tests still don't run , it looks like the arrow-vector is pulling in an older version of jackso, so we need to exclude it:

<groupId>org.apache.arrow</groupId>
                    <artifactId>arrow-vector</artifactId>
                    <version>2.0.0</version>
                     <exclusions>
                        <exclusion>
                            <groupId>com.fasterxml.jackson.core</groupId>
                            <artifactId>jackson-core</artifactId>
                        </exclusion>
                        <exclusion>
                            <groupId>com.fasterxml.jackson.core</groupId>
                            <artifactId>jackson-annotations</artifactId>
                        </exclusion>
                       <exclusion>
                          <groupId>io.netty</groupId>
                          <artifactId>netty-common</artifactId>
                      </exclusion>
                    </exclusions>

It would also be nice to make a arrow.version variable in top level pom for the 2.0.0

Signed-off-by: rishi <sririshindra@gmail.com>
@tgravescs
Copy link
Collaborator

build

@tgravescs tgravescs merged commit c2adace into NVIDIA:branch-22.06 May 2, 2022
@tgravescs
Copy link
Collaborator

thanks @sririshindra

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Add Shim for the Spark version shipped with Cloudera CDH 7.1.7
3 participants