Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add test for encrypted ORC write [databricks] #5764

Merged
merged 6 commits into from
Jun 7, 2022
Merged

Add test for encrypted ORC write [databricks] #5764

merged 6 commits into from
Jun 7, 2022

Conversation

razajafri
Copy link
Collaborator

@razajafri razajafri commented Jun 7, 2022 •
edited
Loading

This PR uses the InMemoryKeystore to write ORC and make sure we fallback to the CPU

  • Added a test method to SparkQueryCompareTestSuite to capture the DataWritingCommandExec
  • Added OrcEncryptionSuite to test the encrypted ORC write
  • Added the OrcEncryptionSuite to different Spark 3.2+ profiles

fixes #5722

razajafri added 4 commits June 6, 2022 21:48
@razajafri
Write Scala test for ORC encrypted write
1d39dc7 
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri
add a defensive piece of code to make sure we don't run this code for…
3414fc7 
… 3.1.x

Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri
park
72f60d9 
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri
added test for ORC write fallback
0b407c3 
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri razajafri changed the title Add test for encrypted ORC write Add test for encrypted ORC write [databricks] Jun 7, 2022
@razajafri
Copy link
Collaborator Author

build

@razajafri
add comment
f5f9be7 
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri
Copy link
Collaborator Author

build

@razajafri
addressed revied comments
568a5c0 
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
@razajafri
Copy link
Collaborator Author

@tgravescs PTAL

@tgravescs
Copy link
Collaborator

build

@sameerz sameerz added the test Only impacts tests label Jun 7, 2022
@sameerz sameerz added this to the Jun 6 - Jun 17 milestone Jun 7, 2022
@sameerz sameerz linked an issue Jun 7, 2022 that may be closed by this pull request
Need tests for writing encrypted ORC file #5722
Closed
@tgravescs tgravescs merged commit c047f58 into NVIDIA:branch-22.06 Jun 7, 2022
@jlowe jlowe mentioned this pull request Jun 7, 2022
Merged
@GaryShen2008 GaryShen2008 mentioned this pull request Jun 8, 2022
Closed
@NvTimLiu NvTimLiu mentioned this pull request Jun 14, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tgravescs tgravescs tgravescs approved these changes

@jlowe jlowe Awaiting requested review from jlowe jlowe is a code owner

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@zhanga5 zhanga5 Awaiting requested review from zhanga5

Assignees

@razajafri razajafri

Labels
test Only impacts tests
Projects
None yet
Milestone
Jun 6 - Jun 17
Development

Successfully merging this pull request may close these issues.

Need tests for writing encrypted ORC file
3 participants
@razajafri @tgravescs @sameerz