-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[OPPRO-233] Add SingularOrList support #45
Conversation
@rui-mo @zhejiangxiaomai Can you help review? Thanks! |
Thanks, I'm working on fixing the issue you met when running a query like below, and will refactor a bit based on your change for
|
SubstraitVeloxExprConverter::toVeloxExpr( | ||
const ::substrait::Expression::SingularOrList& singularOrList, | ||
const RowTypePtr& inputType) { | ||
std::vector<std::shared_ptr<const core::ITypedExpr>> params; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr
@@ -54,6 +54,10 @@ class SubstraitVeloxExprConverter { | |||
const ::substrait::Expression::ScalarFunction& sFunc, | |||
const RowTypePtr& inputType); | |||
|
|||
std::shared_ptr<const core::ITypedExpr> toVeloxExpr( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr
selection = singularOrList.value().selection(); | ||
} | ||
return subParser_->parseReferenceSegment(selection.direct_reference()); | ||
; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove ;
@@ -480,9 +495,13 @@ std::shared_ptr<const core::PlanNode> SubstraitVeloxPlanConverter::toVeloxPlan( | |||
subfieldFilters.clear(); | |||
remainingFilter = | |||
connectWithAnd(colNameList, veloxTypeList, scalarFunctions); | |||
remainingFilter = connectWithAnd( | |||
remainingFilter, colNameList, veloxTypeList, singularOrLists); | |||
} else { | |||
remainingFilter = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable “remainingFilter” will not be used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
bool inExists = false; | ||
for (const auto& arg : scalarFunction.arguments()) { | ||
if (!getExprFromFunctionArgument(arg).has_scalar_function()) { | ||
if (getExprFromFunctionArgument(arg).has_scalar_function()) { | ||
// Or relation betweeen literals is not supported to be pushded down | ||
// currently. | ||
return false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems to be not correct to just return false here. Will fix this locally.
const ::substrait::Expression_SingularOrList& singularList) { | ||
if (singularList.value().has_selection()) { | ||
return true; | ||
} else if (singularList.value().has_scalar_function()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When value
is not field, the IN filter cannot be pushed down. Will fix this in a following PR.
7e62f59
to
4032e69
Compare
…nction lookup (oap-project#45) * logical scalar function support & code refactor * method 'loadExtension' should be load once at runtime * revert setup-ubuntu.sh * remove unused methods * code style fix * code style fix * license header fix * merge logical op test into same method
…nction lookup (oap-project#45) * logical scalar function support & code refactor * method 'loadExtension' should be load once at runtime * revert setup-ubuntu.sh * remove unused methods * code style fix * code style fix * license header fix * merge logical op test into same method
…nction lookup (oap-project#45) * logical scalar function support & code refactor * method 'loadExtension' should be load once at runtime * revert setup-ubuntu.sh * remove unused methods * code style fix * code style fix * license header fix * merge logical op test into same method
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
relative pr: Check a fallback case in validation: using literal partition key in window function oap-project#148 Fix might_contain validate fallback and support struct literal oap-project#137 Implement datetime functions in velox/sparksql. oap-project#81 Parse options in SingularOrList correctly oap-project#48 Add SingularOrList support oap-project#45 Support if then in filter oap-project#74 Fix semi join output type and support existence join oap-project#67 Support decimal as partition column oap-project#167 Add the window support oap-project#61 Add expand operator oap-project#65 Support more cases of filter and its pushdown #14
No description provided.