Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OPPRO-233] Add SingularOrList support #45

Merged
merged 4 commits into from
Sep 7, 2022

Conversation

jinchengchenghh
Copy link
Collaborator

No description provided.

@jinchengchenghh
Copy link
Collaborator Author

@rui-mo @zhejiangxiaomai Can you help review? Thanks!

@rui-mo
Copy link
Collaborator

rui-mo commented Sep 2, 2022

Thanks, I'm working on fixing the issue you met when running a query like below, and will refactor a bit based on your change for in.

select l_orderkey from lineitem where l_orderkey > 2  and l_orderkey = 1

SubstraitVeloxExprConverter::toVeloxExpr(
const ::substrait::Expression::SingularOrList& singularOrList,
const RowTypePtr& inputType) {
std::vector<std::shared_ptr<const core::ITypedExpr>> params;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr

@@ -54,6 +54,10 @@ class SubstraitVeloxExprConverter {
const ::substrait::Expression::ScalarFunction& sFunc,
const RowTypePtr& inputType);

std::shared_ptr<const core::ITypedExpr> toVeloxExpr(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"std::shared_ptr<const` core::ITypedExpr>" -> TypedExprPtr

selection = singularOrList.value().selection();
}
return subParser_->parseReferenceSegment(selection.direct_reference());
;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove ;

@@ -480,9 +495,13 @@ std::shared_ptr<const core::PlanNode> SubstraitVeloxPlanConverter::toVeloxPlan(
subfieldFilters.clear();
remainingFilter =
connectWithAnd(colNameList, veloxTypeList, scalarFunctions);
remainingFilter = connectWithAnd(
remainingFilter, colNameList, veloxTypeList, singularOrLists);
} else {
remainingFilter =
Copy link
Collaborator

@zhejiangxiaomai zhejiangxiaomai Sep 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable “remainingFilter” will not be used.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


bool inExists = false;
for (const auto& arg : scalarFunction.arguments()) {
if (!getExprFromFunctionArgument(arg).has_scalar_function()) {
if (getExprFromFunctionArgument(arg).has_scalar_function()) {
// Or relation betweeen literals is not supported to be pushded down
// currently.
return false;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be not correct to just return false here. Will fix this locally.

const ::substrait::Expression_SingularOrList& singularList) {
if (singularList.value().has_selection()) {
return true;
} else if (singularList.value().has_scalar_function()) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When value is not field, the IN filter cannot be pushed down. Will fix this in a following PR.

@rui-mo rui-mo merged commit 1f7758b into oap-project:main Sep 7, 2022
xieqi pushed a commit to Intel-bigdata/velox that referenced this pull request Sep 16, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Sep 16, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Sep 22, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Oct 20, 2022
ZJie1 pushed a commit to Intel-bigdata/velox that referenced this pull request Oct 20, 2022
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Oct 20, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Nov 18, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 12, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 13, 2022
ZJie1 pushed a commit to Intel-bigdata/velox that referenced this pull request Dec 14, 2022
…nction lookup (oap-project#45)

* logical scalar function support & code refactor

* method 'loadExtension' should be load once at runtime

* revert setup-ubuntu.sh

* remove unused methods

* code style fix

* code style fix

* license header fix

* merge logical op test into same method
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 14, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 14, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 15, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 15, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Dec 21, 2022
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jan 3, 2023
@jinchengchenghh jinchengchenghh deleted the orlist branch February 16, 2023 02:00
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Feb 27, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 6, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 27, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Mar 29, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 14, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 17, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 19, 2023
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Apr 20, 2023
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 3, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 4, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 11, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 17, 2023
relative pr:

Check a fallback case in validation: using literal partition key in window function oap-project#148
Fix might_contain validate fallback and support struct literal oap-project#137
Implement datetime functions in velox/sparksql. oap-project#81
Parse options in SingularOrList correctly oap-project#48
Add SingularOrList support oap-project#45
Support if then in filter oap-project#74
Fix semi join output type and support existence join oap-project#67
Support decimal as partition column oap-project#167
Add the window support oap-project#61
Add expand operator oap-project#65
Support more cases of filter and its pushdown #14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants