Support build right with HashJoin in DataFusion

[viirya]

Is your feature request related to a problem or challenge?

See the discussion at Comet: apache/datafusion-comet#194 (comment)

Yea, in DataFusion, only left side could be the build side. But in Spark, the HashJoin operator has a build side parameter to indicate which side is build side. The operator will do right thing accordingly internally. So currently we cannot just create a DataFusion HashJoin operator with right side as build side.

It can be swapped between left and right side, only if we also swap outputs and also column binding in joining keys and joining filter. I'd like to relax the build side constraint in DataFusion instead of doing the swap in Comet.

Describe the solution you'd like

HashJoin supports build right option.

Describe alternatives you've considered

No response

Additional context

No response

[metesynnada]

Is there a technical problem with swapping join sides in Comet?

[viirya]

No, I think technically we can do the swapping on joining keys, join filter, join output, etc. in Comet. I think it would be good if we can make DataFusion HashJoin more flexible to remove this constraint.

[metesynnada]

Got it, I was figuring out if this could be a blocker.

[comphead]

No, I think technically we can do the swapping on joining keys, join filter, join output, etc. in Comet. I think it would be good if we can make DataFusion HashJoin more flexible to remove this constraint.

That sounds cool, and such initiative will help DF to support its own join order hints instead of hardcoded join order

[edmondop]

@comphead how would this work? do you think we should extend the SQL parser to support hints?

[viirya]

I think this is not related to SQL parser. The SQL query looks the same. But in Spark, it can decide hash side based on relation statistics or the hint given by users.

[viirya]

I will work on this.