You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
currently we do not support inequality joins (hash join and sort-merge join). it is hard to implement such feature because datafusion has no direct supports to row-based evaluation.
Describe the solution you'd like Describe alternatives you've considered
simulate row-based evaluation with one-row columnar evaluation, which has super low performance in practice. in cases where the equality pred has filtered away most records, this method may work. but if the equality pred takes no effects (like tpcds q72). the query will hang.
supports limited row-based filter in datafusion. currently datafusion already has some supports like make_comparator to build a row-based evaluator. we can extend it to support more row-based evaluations, like make_binary_op etc.
fallback the post-filter evaluation to spark, and use codegen to speedup the evaluation. but we also have to consider the fallback overheads.
Additional context
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
currently we do not support inequality joins (hash join and sort-merge join). it is hard to implement such feature because datafusion has no direct supports to row-based evaluation.
Describe the solution you'd like
Describe alternatives you've considered
make_comparator
to build a row-based evaluator. we can extend it to support more row-based evaluations, likemake_binary_op
etc.Additional context
The text was updated successfully, but these errors were encountered: