Update GpuFileFormatWriter to stay in sync with recent Spark changes, but still not support writing Hive bucketed table on GPU. #4484
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
throw an exception when trying to do hive hash partition on GPU Signed-off-by: remzi <13716567376yh@gmail.com>
Signed-off-by: remzi <13716567376yh@gmail.com>
Signed-off-by: remzi <13716567376yh@gmail.com>
Signed-off-by: remzi <13716567376yh@gmail.com>
Signed-off-by: remzi <13716567376yh@gmail.com>
HaoYang670
changed the title
|
build |
jlowe
reviewed
Jan 10, 2022
There was a problem hiding this comment.
Headline implies that this adds support for writing Hive bucketed tables, but it is not supported even after this PR is merged. We weren't accidentally trying to support this before the PR either, so maybe headline should just state we're updating GpuWriteJobDescription to stay in sync with recent Spark changes.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileFormatWriter.scala
Outdated
Show resolved
Hide resolved
HaoYang670
changed the title
Signed-off-by: remzi <13716567376yh@gmail.com>
|
build |
Because Spark330 and Spark301 behave differently on insertInto Signed-off-by: remzi <13716567376yh@gmail.com>
|
build |
jlowe
reviewed
Jan 11, 2022
There was a problem hiding this comment.
Looks better, but tests are failing with:
The format of the existing table default.tmp_table_574308_0 is `HiveFileFormat`. It doesn't match the specified format `ParquetDataSourceV2`.
We may need a .format("hive") when writing the dataframe, but I'm not an expert on Spark's Hive support.
|
Should we update the title of this PR as we are not adding in support for hived bucketed table writes. |
Signed-off-by: remzi <13716567376yh@gmail.com>
|
build |
HaoYang670
changed the title
|
The title has been updated to tell users that we do not support writing hive bucketed table on GPU so far. |
revans2
approved these changes
Jan 12, 2022
jlowe
approved these changes
Jan 12, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: remzi 13716567376yh@gmail.com
close #3949
In this PR, we: