Re-enable the struct support for the ORC reader. #3262
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also add tests for the nested predicate pushdown, and the support for nested column pruning. Relevant PRs: NVIDIA#3079 NVIDIA#2887 Signed-off-by: Firestarman <firestarmanllc@gmail.com>
|
build |
firestarman
added
the
cudf_dependency
|
There is still an issue in CUDF for a coner case when reading a struct column, and here is the fix rapidsai/cudf#9060. |
jlowe
reviewed
Aug 20, 2021
|
Move to draft because of the open blocking issue rapidsai/cudf#9060. |
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
|
build |
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
|
build |
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
signed-off-by: Firestarman <firestarmanllc@gmail.com>
jlowe
reviewed
Aug 23, 2021
|
build |
|
The fix rapidsai/cudf#9060 has been merged. |
jlowe
previously approved these changes
Aug 25, 2021
revans2
previously approved these changes
Aug 25, 2021
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala
Outdated
Show resolved
Hide resolved
|
All the code looks fine, just a few minor nits that I can live without. |
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
|
build |
Thanks for reivew. I decided to address the nits in this PR because there is one for the code change. |
revans2
approved these changes
Aug 26, 2021
gerashegalov
approved these changes
Aug 26, 2021
Comment on lines
+1228
to
+1234
| fileSchema.getCategory == readSchema.getCategory && { | ||
| if (readSchema.getChildren != null) { | ||
| readSchema.getChildren.asScala.forall(rc => | ||
| fileSchema.getChildren.asScala.exists(fc => isSchemaCompatible(fc, rc))) | ||
| } else { | ||
| false | ||
| } |
There was a problem hiding this comment.
nit:
Suggested change
| fileSchema.getCategory == readSchema.getCategory && { | |
| if (readSchema.getChildren != null) { | |
| readSchema.getChildren.asScala.forall(rc => | |
| fileSchema.getChildren.asScala.exists(fc => isSchemaCompatible(fc, rc))) | |
| } else { | |
| false | |
| } | |
| fileSchema.getCategory == readSchema.getCategory && | |
| readSchema.getChildren != null) && | |
| readSchema.getChildren.asScala.forall(rc => | |
| fileSchema.getChildren.asScala.exists(fc => isSchemaCompatible(fc, rc))) |
There was a problem hiding this comment.
Thanks for reivew. I will update this in antoher PR.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It adds the struct support in ORC reader, along with the tests for the nested predicate pushdown, and the support for nested column pruning.
fixes #2879
fixes #1481
fixes #463
The old relevant PRs:
#3079
#2887
Signed-off-by: Firestarman firestarmanllc@gmail.com