Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Conditional hash join for left semi and left anti joins #9695

Closed
jlowe opened this issue Nov 16, 2021 · 2 comments · Fixed by #10037
Closed

[FEA] Conditional hash join for left semi and left anti joins #9695

jlowe opened this issue Nov 16, 2021 · 2 comments · Fixed by #10037
Assignees
Labels
feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS

Comments

@jlowe
Copy link
Member

jlowe commented Nov 16, 2021

Is your feature request related to a problem? Please describe.
Currently the RAPIDS Accelerator for Apache Spark does not accelerate left semi or anti joins that have both an equality and an inequality condition. Currently the only way to accelerate this is by using a nested loop join with an AST condition, but this often performs far worse than the CPU implementation which first implements a hash-based lookup on the equality condition then evaluates the inequality condition before processing it as a "join hit."

Describe the solution you'd like
libcudf supports accepting a supplemental condition represented as an AST expression to the existing left semi and left anti hash-based join APIs. Internally the join kernel performs the hash lookup based on the equality keys then if there's a hit in the hash lookup it evaluates the AST expression to produce a boolean result indicating whether the row should be considered a join match or not. The API can take two sets of table_view pairs, one pair for the left and right equality keys to use and one pair to use for the AST expression evaulation. The result is a gather map for the left table as it is for the hash-based semi/anti join today.

Describe alternatives you've considered
The interface could specify a just two table_views for the left/right table and separately two vectors of ints to specify which columns are the equality keys, but it seems simpler to pass separate table_views for equality vs. AST expression, especially when the application needs to generate the result of an expression for the equality portion of the join (e.g.: t1_col2 + 1 == t2_col3 * 2)

Additional context
This is part of #5401

@jlowe jlowe added feature request New feature or request Needs Triage Need team to review and classify libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS labels Nov 16, 2021
@nvliyuan
Copy link

Maybe TPC-H Q21 is a demo for LeftAnti join, I get the following unsupported log while running the benchmark:
!Exec <SortMergeJoinExec> cannot run on GPU because LeftAnti joins currently do not support conditions

@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

rapids-bot bot pushed a commit that referenced this issue Jan 24, 2022
This PR is a follow-up to #9917 and should be merged after that PR. This resolves #9695 and resolves #5401. The implementation here is only a first pass, but in the interest of prioritizing a working feature for the upcoming release I'm postponing making various additional changes (including some breaking ones).

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Jason Lowe (https://github.com/jlowe)
  - David Wendt (https://github.com/davidwendt)
  - Robert Maynard (https://github.com/robertmaynard)

URL: #10037
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants