Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet read unsigned int: uint8, uin16, uint32 #3438

Merged
merged 24 commits into from
Sep 23, 2021
Merged

Parquet read unsigned int: uint8, uin16, uint32 #3438

merged 24 commits into from
Sep 23, 2021

Conversation

res-life
Copy link
Collaborator

parquet read unsigned int: uint8, uin16, uint32
This fixes #2741

Signed-off-by: Chong Gao res_life@163.com

parquet read unsigned int: uint8, uin16, uint32
b7ff86f 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life res-life added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Sep 10, 2021
@res-life res-life marked this pull request as draft September 10, 2021 04:08
parquet read unsigned int: uint8, uin16, uint32
3ead7cb 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life res-life removed the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Sep 10, 2021
@res-life
Copy link
Collaborator Author

build

1 similar comment
@res-life
Copy link
Collaborator Author

build

firestarman
firestarman reviewed Sep 10, 2021
View reviewed changes
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
@sameerz sameerz added this to the Sep 13 - Sep 24 milestone Sep 11, 2021
Chong Gao added 2 commits September 13, 2021 19:17
refactor
c991312 
Signed-off-by: Chong Gao <res_life@163.com>
skip test case when version is not ok
1e0ecb1
@res-life
Copy link
Collaborator Author

build

1 similar comment
@pxLi
Copy link
Collaborator

pxLi commented Sep 14, 2021

build

@pxLi
Copy link
Collaborator

pxLi commented Sep 14, 2021

UT failed


[2021-09-14T01:28:49.909Z] com.nvidia.spark.rapids.ParquetScanSuite *** ABORTED ***

[2021-09-14T01:28:49.909Z]   java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: com.nvidia.spark.rapids.ParquetScanSuite

[2021-09-14T01:28:49.909Z]   at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)

[2021-09-14T01:28:49.909Z]   at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)

[2021-09-14T01:28:49.909Z]   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)

[2021-09-14T01:28:49.909Z]   at scala.collection.Iterator.foreach(Iterator.scala:941)

[2021-09-14T01:28:49.909Z]   at scala.collection.Iterator.foreach$(Iterator.scala:941)

[2021-09-14T01:28:49.909Z]   at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)

[2021-09-14T01:28:49.909Z]   at scala.collection.IterableLike.foreach(IterableLike.scala:74)

[2021-09-14T01:28:49.909Z]   at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

[2021-09-14T01:28:49.909Z]   at scala.collection.AbstractIterable.foreach(Iterable.scala:56)

[2021-09-14T01:28:49.909Z]   at scala.collection.TraversableLike.map(TraversableLike.scala:237)

@res-life
Copy link
Collaborator Author

build

@firestarman firestarman requested a review from wbo4958 September 14, 2021 07:54
refactor code
48e824a 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

refactor code
416a945 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

Chong Gao added 2 commits September 15, 2021 09:24
add comment
46213ea 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

@res-life res-life marked this pull request as ready for review September 17, 2021 05:56
fix read map type error
5857972 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

fix line length exceeds 100
46ae63b 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

add map(uint,struct(uint,uint,uint)) test
10da87d 
Signed-off-by: Chong Gao <res_life@163.com>
@firestarman
Copy link
Collaborator

LGTM, But still need other experts to review it too.

@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

@jlowe @revans2 help to review

jlowe
jlowe reviewed Sep 20, 2021
View reviewed changes
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small wording and comment nits but otherwise lgtm.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/java/com/nvidia/spark/rapids/ColumnCastUtil.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala Outdated Show resolved Hide resolved
tests/src/test/scala/com/nvidia/spark/rapids/ParquetScanSuite.scala Outdated Show resolved Hide resolved
update comments
6357c04 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

update comments
37322e1 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

build

Chong Gao added 2 commits September 22, 2021 13:20
@res-life
Copy link
Collaborator Author

build

@jlowe jlowe changed the title parquet read unsigned int: uint8, uin16, uint32 Parquet read unsigned int: uint8, uin16, uint32 Sep 22, 2021
@jlowe
Copy link
Member

jlowe commented Sep 22, 2021

Need to resolve merge conflict but otherwise lgtm.

@res-life
Copy link
Collaborator Author

build

@res-life res-life merged commit 2bf1d4d into NVIDIA:branch-21.10 Sep 23, 2021
@res-life res-life deleted the parquet-read-unsigned-int branch September 23, 2021 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@firestarman firestarman firestarman left review comments

@jlowe jlowe jlowe approved these changes

@wbo4958 wbo4958 wbo4958 left review comments

@revans2 revans2 Awaiting requested review from revans2

@tgravescs tgravescs Awaiting requested review from tgravescs

Assignees

@res-life res-life

Labels
Spark 3.2+
Projects
None yet
Milestone
Sep 13 - Sep 24
Development

Successfully merging this pull request may close these issues.

[FEA] Audit 3.2.0 - Read parquet unsigned types that stored as int32 physical type in parquet
6 participants
@res-life @pxLi @wbo4958 @firestarman @jlowe @sameerz