Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix First/Last aggregate functions intermediate type and support decimal #245

Merged
merged 4 commits into from
May 16, 2023
Merged

Fix First/Last aggregate functions intermediate type and support decimal #245

merged 4 commits into from
May 16, 2023

Conversation

Yohahaha
Copy link

@Yohahaha Yohahaha commented May 9, 2023 •
edited
Loading

Align First/Last aggregate function's intermediate result with Spark.

Support decimal.

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023

CI log shows the UT related with this PR passed.

2023-05-09T04:05:24.8387588Z 190/214 Test #200: velox_functions_spark_aggregates_test ..........................   Passed   14.95 sec

@zhejiangxiaomai Could you check these UT?

2023-05-09T04:42:46.8637254Z The following tests FAILED:
2023-05-09T04:42:46.8637913Z 	205 - velox_hdfs_file_test (Failed)
2023-05-09T04:42:46.8638371Z 	209 - velox_exec_test (Failed)

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023 •
edited
Loading

2023-05-09T05:43:07.9152977Z [ RUN ] MultiFragmentTest.customPlanNodeWithExchangeClient
2023-05-09T05:43:07.9153318Z /__w/velox/velox/velox/exec/tests/MultiFragmentTest.cpp:1120: Failure
2023-05-09T05:43:07.9153850Z Expected: (toPlanStats(task->taskStats()) .at(testNodeId) .customStats.count("testCustomExchangeStat")) != (0), actual: 0 vs 0
2023-05-09T05:43:07.9154281Z [ FAILED ] MultiFragmentTest.customPlanNodeWithExchangeClient (161 ms)

@zhejiangxiaomai
Copy link
Collaborator

Can you run velox_exec_test --gtest_filter=MultiFragmentTest.customPlanNodeWithExchangeClient locally ?

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023

Failed UT can pass in my local machine

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023

image

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023

Hi @rui-mo could you help take a look?

@rui-mo
Copy link
Collaborator

rui-mo commented May 9, 2023

@Yohahaha Thank you. Can we open a PR in Gluten to test this one? In that Gluten PR, the fetched Velox branch should be changed to your branch. Here is an example: apache/incubator-gluten#1496.
We can also consider adding some unit tests in VeloxAggregateFunctionsSuite to test the newly supported aggregate functions.

@Yohahaha
Copy link
Author

Yohahaha commented May 9, 2023 •
edited
Loading

Hi @rui-mo

Related PR in Gluten is apache/incubator-gluten#1581, UT also added.

@Yohahaha
Copy link
Author

Hi @zhejiangxiaomai

Could we skip velox_hdfs_file_test?

@Yohahaha Yohahaha changed the title Refine First/Last aggregate functions. Fix First/Last aggregate functions intermediate type and support decimal May 11, 2023
@Yohahaha Yohahaha force-pushed the first_last branch 3 times, most recently from 594ef92 to 8cde6d1 Compare May 12, 2023 03:40
@zuochunwei zuochunwei mentioned this pull request May 15, 2023
Closed
rui-mo
rui-mo reviewed May 16, 2023
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM generally.

velox/functions/sparksql/aggregates/FirstLastAggregate.cpp Show resolved Hide resolved
@rui-mo rui-mo merged commit e26f9ef into oap-project:main May 16, 2023
@Yohahaha Yohahaha deleted the first_last branch May 16, 2023 06:29
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request May 18, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
14d8c95 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
96fea3e 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@zhejiangxiaomai
Folder: function
1bbd42b 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request May 31, 2023
@zhejiangxiaomai
Folder: function
731b4be 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jun 25, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
0071e4f 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jun 25, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
a33270e 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jun 26, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
0c94165 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
zhejiangxiaomai pushed a commit to zhejiangxiaomai/velox that referenced this pull request Jun 27, 2023
@Yohahaha @zhejiangxiaomai
Fix the intermediate type of First/Last, and support decimal (oap-pro…
9af1ee5 
…ject#245)

Align First/Last aggregate function's intermediate result with Spark. Support decimal.
@Yohahaha Yohahaha mentioned this pull request Jun 29, 2023
Closed
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 3, 2023
@zhejiangxiaomai
Folder: function
44b2919 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 4, 2023
@zhejiangxiaomai
Folder: function
eaf2b52 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
facebook-github-bot pushed a commit to facebookincubator/velox that referenced this pull request Jul 6, 2023
@Yohahaha @facebook-github-bot
Fix Spark first/last aggregate functions intermediate type from T to …
9fdf36e 
…row(T, boolean) (#4873)

Summary:
Fix first/last Spark aggregate functions' intermediate type from T to row(T, boolean) to align with Spark, the second column in signature is just a place holder.

oap-project#245
apache/incubator-gluten#1581

Pull Request resolved: #4873

Reviewed By: pedroerp

Differential Revision: D47250246

Pulled By: mbasmanova

fbshipit-source-id: dd6ccff394f820b40f59696adeee244172b27a68
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 11, 2023
@zhejiangxiaomai
Folder: function
4de772f 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
@zhejiangxiaomai
Folder: function
e8d7b27 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 12, 2023
@zhejiangxiaomai
Folder: function
20e47f5 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
zhejiangxiaomai added a commit to zhejiangxiaomai/velox that referenced this pull request Jul 17, 2023
@zhejiangxiaomai
Folder: function
e7fa7f5 
relative pr:

Fix replace SparkSQL function oap-project#277
Support kPreceeding & kFollowing for window range frame type oap-project#287
support timestamp hash oap-project#269
Spark sum can overflow oap-project#101
Support float & double types in pmod function oap-project#157
Implement datetime functions in velox/sparksql. oap-project#81
Fix type check in MapFunction oap-project#273
Let function validation fail for lookaround pattern in RE2-based implementation oap-project#124
Register lpad/rpad functions for Spark SQL. oap-project#63
Support substring_index sql function oap-project#189
Fix First/Last aggregate functions intermediate type and support decimal oap-project#245
Support date_add spark sql function oap-project#144
marin-ma pushed a commit to marin-ma/velox-oap that referenced this pull request Dec 15, 2023
@zzcclp
[Gluten-244] Remove 'testFailureIgnore' = true in pom.xml and fix som…
9dbf7b0 
…e memory leak for ClickHouse Backend (oap-project#245)

* [Gluten-244] Remove 'testFailureIgnore' = true in pom.xml and fix some memory leak for ClickHouse Backend

* Revert the pom.xml for velox backend
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rui-mo rui-mo rui-mo left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Yohahaha @zhejiangxiaomai @rui-mo