Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement support for ArrayExists expression #4973

Merged
merged 13 commits into from
Mar 21, 2022
Merged

Implement support for ArrayExists expression #4973

merged 13 commits into from
Mar 21, 2022

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Mar 17, 2022 •
edited
Loading

This PR implements ArrayExists, it has two major phases

  1. first apply function to produce array of Booleans
  2. run segmented reduce ANY to if any of the values are true

Spark 3.x default is the 3VL logic:

  • if any element is true the array maps to true
  • if no element is true and there is at least one null, the array maps to null
  • if no element is true and none is null, the array maps to false

Legacy mode 2VL:

  • if any element is true the array maps to true
  • if no element is true , the array maps to false

Closes #4815

Signed-off-by: Gera Shegalov gera@apache.org

@gerashegalov
exploring
380d81b
@gerashegalov
wip
b4f7108
@gerashegalov
Merge remote-tracking branch 'origin/branch-22.04' into gerashegalov/…
a07f99a 
…issue4815
@gerashegalov
wip
cec991b 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
wip
19fea67 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov self-assigned this Mar 17, 2022
@gerashegalov
Add post processing for empty arrays
2355174 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this pull request Mar 18, 2022
Closed
@sameerz sameerz added the feature request New feature or request label Mar 18, 2022
@sameerz sameerz added this to the Mar 21 - Apr 1 milestone Mar 18, 2022
@gerashegalov
3VL done
1744934 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
Merge remote-tracking branch 'origin/branch-22.04' into gerashegalov/…
0d92e58 
…issue4815
@gerashegalov
refactor
1654535 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
2VL done
fdf626b 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
comments
fa64b0c 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov changed the title [WIP] Implement GpuArrayExists Implement support of ArrayExists expression Mar 19, 2022
@gerashegalov gerashegalov changed the title Implement support of ArrayExists expression Implement support for ArrayExists expression Mar 19, 2022
@gerashegalov gerashegalov marked this pull request as ready for review March 19, 2022 17:06
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov gerashegalov requested a review from revans2 March 21, 2022 13:27
revans2
revans2 previously approved these changes Mar 21, 2022
View reviewed changes
Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks really good. Just some nits and docs to clean up.

integration_tests/src/main/python/array_test.py Outdated Show resolved Hide resolved
integration_tests/src/main/python/array_test.py Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala Outdated Show resolved Hide resolved
@gerashegalov
Bobby's review 1
3ee7389 
Signed-off-by: Gera Shegalov <gera@apache.org>
revans2
revans2 previously approved these changes Mar 21, 2022
View reviewed changes
@gerashegalov
doc update
21ea15f 
Signed-off-by: Gera Shegalov <gera@apache.org>
@revans2
Copy link
Collaborator

revans2 commented Mar 21, 2022

build

@gerashegalov gerashegalov merged commit 5ed86dd into NVIDIA:branch-22.04 Mar 21, 2022
@gerashegalov gerashegalov deleted the gerashegalov/issue4815 branch March 21, 2022 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees

@gerashegalov gerashegalov

Labels
feature request New feature or request
Projects
None yet
Milestone
Mar 21 - Apr 1
Development

Successfully merging this pull request may close these issues.

[FEA] Support org.apache.spark.sql.catalyst.expressions.ArrayExists
3 participants
@gerashegalov @revans2 @sameerz