Unexpectedly, Distinct cannot be pushed across Projection when read partition table

[SunRunAway]

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

CREATE TABLE `tp` (
  `timestamp` timestamp NULL DEFAULT NULL,
  KEY `idx_timestamp` (`timestamp`)
) PARTITION BY RANGE ( UNIX_TIMESTAMP(`timestamp`) ) (
  PARTITION `p2020072312` VALUES LESS THAN (1595480400),
  PARTITION `p2020072313` VALUES LESS THAN (1595484000));

2. What did you expect to see? (Required)

mysql> desc select  DATE_FORMAT(timestamp, '%Y-%m-%d %H') as tt from tp group by tt;
+------------------------------------+----------+-----------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| id                                 | estRows  | task      | access object                   | operator info                                                                                                                     |
+------------------------------------+----------+-----------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Projection_10                      | 16000.00 | root      |                                 | date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#3                                                                             |
| └─HashAgg_13                       | 16000.00 | root      |                                 | group by:Column#5, funcs:firstrow(Column#6)->test.tp.timestamp                                                                    |
|   └─PartitionUnion_14              | 16000.00 | root      |                                 |                                                                                                                                   |
|     ├─HashAgg_17                   | 8000.00  | root      |                                 | group by:Column#15, funcs:firstrow(Column#13)->Column#6, funcs:firstrow(Column#14)->Column#5                                      |
|     │ └─Projection_37              | 10000.00 | root      |                                 | test.tp.timestamp, date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#14, date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#15 |
|     │   └─TableReader_23           | 10000.00 | root      |                                 | data:TableFullScan_22                                                                                                             |
|     │     └─TableFullScan_22       | 10000.00 | cop[tikv] | table:tp, partition:p2020072312 | keep order:false, stats:pseudo                                                                                                    |
|     └─HashAgg_28                   | 8000.00  | root      |                                 | group by:Column#18, funcs:firstrow(Column#16)->Column#6, funcs:firstrow(Column#17)->Column#5                                      |
|       └─Projection_38              | 10000.00 | root      |                                 | test.tp.timestamp, date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#17, date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#18 |
|         └─TableReader_34           | 10000.00 | root      |                                 | data:TableFullScan_33                                                                                                             |
|           └─TableFullScan_33       | 10000.00 | cop[tikv] | table:tp, partition:p2020072313 | keep order:false, stats:pseudo                                                                                                    |
+------------------------------------+----------+-----------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
11 rows in set (0.01 sec)

3. What did you see instead (Required)

mysql> desc select distinct DATE_FORMAT(timestamp, '%Y-%m-%d %H') as tt from tp ;
+--------------------------------+----------+-----------+---------------------------------+-------------------------------------------------------+
| id                             | estRows  | task      | access object                   | operator info                                         |
+--------------------------------+----------+-----------+---------------------------------+-------------------------------------------------------+
| HashAgg_9                      | 16000.00 | root      |                                 | group by:Column#3, funcs:firstrow(Column#3)->Column#3 |
| └─Projection_10                | 20000.00 | root      |                                 | date_format(test.tp.timestamp, %Y-%m-%d %H)->Column#3 |
|   └─PartitionUnion_11          | 20000.00 | root      |                                 |                                                       |
|     ├─TableReader_13           | 10000.00 | root      |                                 | data:TableFullScan_12                                 |
|     │ └─TableFullScan_12       | 10000.00 | cop[tikv] | table:tp, partition:p2020072312 | keep order:false, stats:pseudo                        |
|     └─TableReader_17           | 10000.00 | root      |                                 | data:TableFullScan_16                                 |
|       └─TableFullScan_16       | 10000.00 | cop[tikv] | table:tp, partition:p2020072313 | keep order:false, stats:pseudo                        |
+--------------------------------+----------+-----------+---------------------------------+-------------------------------------------------------+
7 rows in set (0.00 sec)

4. Affected version (Required)

5. Root Cause Analysis

SIG slack channel

#sig-planner

Score

• 900

Mentor

• @lzmhhh123

[Reminiscent]

/assign

[Reminiscent]

We need to set @@session.tidb_opt_agg_push_down = 1; for test.

[newcworld]

hi guys, Is there anyone who is working for this? I want to have a try

[Reminiscent]

hi guys, Is there anyone who is working for this? I want to have a try

@pengdaqian2020 Sorry, I'm working on this issue. Do you want to try other issues on #20804 ?

[ti-challenge-bot]

The issue has been removed from the challenge program.

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions