Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support running CPU based UDF efficiently [databricks] #3897

Merged
merged 18 commits into from
Oct 29, 2021
Merged

Support running CPU based UDF efficiently [databricks] #3897

merged 18 commits into from
Oct 29, 2021

Conversation

firestarman
Copy link
Collaborator

@firestarman firestarman commented Oct 22, 2021 •
edited by jlowe
Loading

This PR is to support running CPU based UDF efficiently by pulling back only the columns the UDF needs to host and do the processing on CPU, instead of falling back a whole plan to CPU.

TODOs

  • More unit tests for decimal and nested types (list, struct, map).
  • More investigation on the errors when using input encoders. (Tracked by #3924)
  • Update hive UDFs to support this CPU based UDF. (Tracked by #3904)

This partially addresses #3855

Signed-off-by: Firestarman firestarmanllc@gmail.com

@firestarman
Support running CPU based UDF.
a547390 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

Filed this draft PR for early review.

@firestarman
Copy link
Collaborator Author

build

1 similar comment
@firestarman
Copy link
Collaborator Author

build

@firestarman
Address the comments
d34df5a 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
This was referenced Oct 25, 2021
Closed
Closed
@firestarman
Add more tests
b244b3f 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@sameerz sameerz added the feature request New feature or request label Oct 26, 2021
@firestarman
Add log for the time of running the UDF.
f4ee421 
along with some small refactors.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman mentioned this pull request Oct 26, 2021
Closed
@firestarman
Add comments for disabling the input encoder
ef62aa1 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman self-assigned this Oct 26, 2021
@firestarman firestarman marked this pull request as ready for review October 26, 2021 09:39
@firestarman firestarman requested review from jlowe and revans2 October 26, 2021 09:39
@firestarman
Copy link
Collaborator Author

build

@firestarman
Comment update
f6fcdba 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Fix a build error on DB
1b09508 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Copy link
Collaborator Author

build

@pxLi pxLi changed the title Support running CPU based UDF efficiently Support running CPU based UDF efficiently [databricks] Oct 26, 2021
@pxLi
Copy link
Collaborator

pxLi commented Oct 26, 2021

added [databricks] to make sure we pass db runtime tests

jlowe
jlowe reviewed Oct 26, 2021
View reviewed changes
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuUserDefinedFunction.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuUserDefinedFunction.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuUserDefinedFunction.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuUserDefinedFunction.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuScalaUDF.scala Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuScalaUDF.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuScalaUDF.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuScalaUDF.scala Outdated Show resolved Hide resolved
@firestarman
Copy link
Collaborator Author

firestarman commented Oct 27, 2021 •
edited
Loading

added [databricks] to make sure we pass db runtime tests

Any action is required for this ? I did a verification on DB by a dev job and it passed.

@firestarman
Address the new comments
e2cc107 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Update comments
b83a8c3 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman firestarman requested a review from jlowe October 27, 2021 06:05
@firestarman @jlowe
Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.s…
78d34b4 
…cala


Update the config doc

Co-authored-by: Jason Lowe <jlowe@nvidia.com>
@firestarman firestarman mentioned this pull request Oct 28, 2021
Closed
@firestarman
Use the code of 3.0 for input type conversion
3f6202e 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman requested a review from jlowe October 28, 2021 06:10
@firestarman
Copy link
Collaborator Author

build

@firestarman
comment update
1e2cddd 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Doc update
5740b64 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Address the new comments
2b6d66d 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman requested a review from revans2 October 29, 2021 03:02
@firestarman
Add shims for GpuRowBasedScalaUDF
132751e 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Copy link
Collaborator Author

build

@firestarman
Update the test file name
52e800a 
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman
Merge branch 'branch-21.12' into scala-udf
a0cb70c
@firestarman
Copy link
Collaborator Author

build

@jlowe jlowe merged commit 96ca730 into NVIDIA:branch-21.12 Oct 29, 2021
@jlowe jlowe mentioned this pull request Oct 29, 2021
Closed
@firestarman firestarman mentioned this pull request Nov 1, 2021
Closed
3 tasks
@jlowe jlowe mentioned this pull request Jan 14, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

@jlowe jlowe jlowe approved these changes

Assignees

@firestarman firestarman

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@firestarman @pxLi @jlowe @revans2 @sameerz