planner, type: fix AggFieldType error when encouter unsigned and sign type #21062

rogeryk · 2020-11-15T11:37:36Z

What problem does this PR solve?

Issue Number:
close #9869
close #17652

Problem Summary: not process UnsignedFlag in AggFieldType

What is changed and how it works?

What's Changed:

types/field_type.go

How it Works:

add UnsignedFlag check in mergeFlag
add integral promotion when encounter signed and unsigned

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

fix Bigint unsigned column value in CASE WHEN operator is converted to bigint signed Bigint unsigned column value in CASE WHEN operator is converted to bigint signed #9869

sre-bot · 2020-11-15T11:37:45Z

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

rogeryk · 2020-11-15T11:40:25Z

/run-all-test

rogeryk · 2020-11-15T11:44:05Z

/rebuild

…cap#9869) add ingegral promotion in AggFieldType

XuHuaiyu · 2020-11-16T03:27:39Z

expression/constant.go

@@ -28,17 +28,21 @@ import (

 // NewOne stands for a number 1.
 func NewOne() *Constant {
+	retT := types.NewFieldType(mysql.TypeTiny)


I do not understand why we need to change these 2 funcs

because i make test fail when i modify AggFieldType. The test is https://github.com/pingcap/tidb/blob/master/planner/cascades/transformation_rules_test.go#L389. "SQL": "select count(b), sum(b), avg(b), b, max(b), min(b), bit_and(b), bit_or(b), bit_xor(b) from t group by a having sum(b) >= 0 and count(b) >= 0 order by b",

The reason for fail is that the bit_or will add a cast wrap (cast(b as bigint unsigned binary), the arg is unsigned.
the TransformAggToProj will add ifnull wrap ifnull(cast(b as bigint unsigned binary), expression.NewZero()).
ifnull use AggFieldType infer type to tigger integral promotion.
https://github.com/pingcap/tidb/blob/master/planner/core/rule_aggregation_elimination.go#L124

I think this is a easy way to process. what do you think?

XuHuaiyu · 2020-11-16T03:33:40Z

types/field_type.go

+	if isMixedSign && IsTypeInteger(currType.Tp) {
+		bumpRange := false // indicate one of tps bump currType range
+		for _, t := range tps {
+			bumpRange = bumpRange || (mysql.HasUnsignedFlag(t.Flag) && (t.Tp == currType.Tp || t.Tp == mysql.TypeBit))


Why check mysql.TypeBit specially

because the TypeBit can't be mergeType and assume it max range bit(64). so it must bump.
I find the all IntegerType merege mysql.TypeBit is mysql.TypeVarchar.
It is inconsistent with mysql 8.0 https://github.com/mysql/mysql-server/blob/8.0/sql/field.cc#L248.

Whether it will be consistent with mysql in the future. if not, No need to consider mysql.TypeBit here

XuHuaiyu · 2020-11-16T03:33:53Z

types/field_type.go

 		currType.Tp = mtp
 		currType.Flag = mergeTypeFlag(currType.Flag, t.Flag)
 	}
+	// integral promotion when tps contains signed and unsigned
+	if isMixedSign && IsTypeInteger(currType.Tp) {
+		bumpRange := false // indicate one of tps bump currType range


Does MySQL also bumpRange ?

yes, I refer from here https://github.com/mysql/mysql-server/blob/8.0/sql/item.cc#L557

XuHuaiyu

LGTM

XuHuaiyu · 2020-11-16T07:04:40Z

PTAL @lzmhhh123

XuHuaiyu · 2020-11-16T08:01:53Z

@rogeryk Please add the test case in #17652 to this commit.
It seems that this PR can also fix that issue.

rogeryk · 2020-11-16T08:11:51Z

@rogeryk Please add the test case in #17652 to this commit.
It seems that this PR can also fix that issue.

Ok i will add

…cap#17652)

rogeryk · 2020-11-16T13:47:50Z

@rogeryk Please add the test case in #17652 to this commit.
It seems that this PR can also fix that issue.

done

XuHuaiyu · 2020-11-17T08:44:44Z

PTAL @ichn-hu

ichn-hu

LGTM

…-signed

rogeryk · 2020-11-20T11:02:01Z

/run-all-tests

rogeryk · 2020-11-20T17:40:17Z

/run-all-tests

rogeryk · 2020-11-21T07:05:31Z

/run-unit-test

rogeryk · 2020-11-21T07:16:52Z

/run-all-tests

rogeryk · 2020-11-22T03:22:17Z

@lzmhhh123 The tests look not stable. How can I avoid it?

lzmhhh123 · 2020-11-23T02:56:59Z

@lzmhhh123 The tests look not stable. How can I avoid it?

Which test?

rogeryk · 2020-11-23T06:15:54Z

@lzmhhh123 The tests look not stable. How can I avoid it?

Which test?

https://github.com/pingcap/tidb/blob/master/planner/core/physical_plan_test.go#L1647
https://github.com/pingcap/tidb/blob/master/planner/core/physical_plan_test.go#L1691
the test offen fail. but i replay test, it pass.

lzmhhh123 · 2020-11-23T06:17:58Z

What's the error of the fail.

rogeryk · 2020-11-23T06:50:59Z

What's the error of the fail.

The error is the estRows is not equal. now I have no infomation.

lzmhhh123 · 2020-11-23T09:48:07Z

Is this test failed?


[2020-11-23T09:45:42.107Z] physical_plan_test.go:1731:

[2020-11-23T09:45:42.107Z]     tk.MustQuery("explain " + ts).Check(testkit.Rows(output[i].Plan...))

[2020-11-23T09:45:42.107Z] /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:63:

[2020-11-23T09:45:42.107Z]     res.c.Assert(resBuff.String(), check.Equals, needBuff.String(), res.comment)

[2020-11-23T09:45:42.107Z] ... obtained string = "" +

[2020-11-23T09:45:42.107Z] ...     "[IndexLookUp_7 1441.12 root  ]\n" +

[2020-11-23T09:45:42.107Z] ...     "[├─IndexRangeScan_5(Build) 1441.12 cop[tikv] table:test, index:idx_age(age) range:[5,5], keep order:false]\n" +

[2020-11-23T09:45:42.108Z] ...     "[└─TableRowIDScan_6(Probe) 1441.12 cop[tikv] table:test keep order:false]\n"

[2020-11-23T09:45:42.108Z] ... expected string = "" +

[2020-11-23T09:45:42.108Z] ...     "[IndexLookUp_7 2048.00 root  ]\n" +

[2020-11-23T09:45:42.108Z] ...     "[├─IndexRangeScan_5(Build) 2048.00 cop[tikv] table:test, index:idx_age(age) range:[5,5], keep order:false]\n" +

[2020-11-23T09:45:42.108Z] ...     "[└─TableRowIDScan_6(Probe) 2048.00 cop[tikv] table:test keep order:false]\n"

rogeryk · 2020-11-23T10:02:30Z

Is this test failed?


[2020-11-23T09:45:42.107Z] physical_plan_test.go:1731:

[2020-11-23T09:45:42.107Z]     tk.MustQuery("explain " + ts).Check(testkit.Rows(output[i].Plan...))

[2020-11-23T09:45:42.107Z] /home/jenkins/agent/workspace/tidb_ghpr_check_2/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:63:

[2020-11-23T09:45:42.107Z]     res.c.Assert(resBuff.String(), check.Equals, needBuff.String(), res.comment)

[2020-11-23T09:45:42.107Z] ... obtained string = "" +

[2020-11-23T09:45:42.107Z] ...     "[IndexLookUp_7 1441.12 root  ]\n" +

[2020-11-23T09:45:42.107Z] ...     "[├─IndexRangeScan_5(Build) 1441.12 cop[tikv] table:test, index:idx_age(age) range:[5,5], keep order:false]\n" +

[2020-11-23T09:45:42.108Z] ...     "[└─TableRowIDScan_6(Probe) 1441.12 cop[tikv] table:test keep order:false]\n"

[2020-11-23T09:45:42.108Z] ... expected string = "" +

[2020-11-23T09:45:42.108Z] ...     "[IndexLookUp_7 2048.00 root  ]\n" +

[2020-11-23T09:45:42.108Z] ...     "[├─IndexRangeScan_5(Build) 2048.00 cop[tikv] table:test, index:idx_age(age) range:[5,5], keep order:false]\n" +

[2020-11-23T09:45:42.108Z] ...     "[└─TableRowIDScan_6(Probe) 2048.00 cop[tikv] table:test keep order:false]\n"

yes, sometime it is right.

XuHuaiyu · 2020-11-24T03:06:41Z

#21092

It has nothing to do with this PR. I'll trigger merge again.

XuHuaiyu · 2020-11-24T03:06:47Z

/merge

ti-srebot · 2020-11-24T03:06:49Z

Your auto merge job has been accepted, waiting for:

21124

…-signed

ti-srebot · 2020-11-24T03:21:21Z

/run-all-tests

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-11-24T03:34:07Z

cherry pick to release-4.0 in PR #21236

… type (#21062) (#21236)

rogeryk requested review from a team as code owners November 15, 2020 11:37

rogeryk requested review from SunRunAway and removed request for a team November 15, 2020 11:37

ti-srebot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Nov 15, 2020

github-actions bot added the component/expression label Nov 15, 2020

ichn-hu mentioned this pull request Nov 15, 2020

Welcome to contribute #20804

Closed

planner, type: fix AggFieldType error when encouter unsign type (ping…

efc9e9f

…cap#9869) add ingegral promotion in AggFieldType

rogeryk force-pushed the 9869-bigint-unsigned-in-casefunc-covert-to-signed branch from be2fcc3 to efc9e9f Compare November 15, 2020 15:33

XuHuaiyu added the type/bugfix This PR fixes a bug. label Nov 16, 2020

XuHuaiyu changed the title ~~planner, type: fix AggFieldType error when encouter unsigned and sign type (#9869)~~ planner, type: fix AggFieldType error when encouter unsigned and sign type Nov 16, 2020

XuHuaiyu reviewed Nov 16, 2020

View reviewed changes

XuHuaiyu mentioned this pull request Nov 16, 2020

IFNULL produce different result with MySQL on bigint column with unsigned flag #17652

Closed

XuHuaiyu reviewed Nov 16, 2020

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 16, 2020

XuHuaiyu requested review from lzmhhh123 and removed request for SunRunAway November 16, 2020 07:04

XuHuaiyu added the needs-cherry-pick-4.0 label Nov 16, 2020

palnner: add unit test for ifnull args with signed and unsigned (ping…

ab1fafe

…cap#17652)

XuHuaiyu requested a review from ichn-hu November 17, 2020 08:02

ichn-hu previously approved these changes Nov 19, 2020

View reviewed changes

ti-srebot removed the status/LGT1 Indicates that a PR has LGTM 1. label Nov 19, 2020

Merge branch 'master' into 9869-bigint-unsigned-in-casefunc-covert-to…

4e5c29e

…-signed

rogeryk dismissed stale reviews from ti-srebot and ichn-hu via 4e5c29e November 20, 2020 10:50

rogeryk force-pushed the 9869-bigint-unsigned-in-casefunc-covert-to-signed branch from 9fbdb62 to 4e5c29e Compare November 20, 2020 10:50

rogeryk force-pushed the 9869-bigint-unsigned-in-casefunc-covert-to-signed branch from 64c679b to 3306832 Compare November 20, 2020 17:39

rogeryk force-pushed the 9869-bigint-unsigned-in-casefunc-covert-to-signed branch from 3306832 to 4e5c29e Compare November 20, 2020 17:52

Merge branch 'master' into 9869-bigint-unsigned-in-casefunc-covert-to…

ee78e69

…-signed

XuHuaiyu approved these changes Nov 24, 2020

View reviewed changes

XuHuaiyu merged commit 4897ecd into pingcap:master Nov 24, 2020

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 24, 2020

cherry pick pingcap#21062 to release-4.0

5da6e48

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Nov 24, 2020

planner, type: fix AggFieldType error when encouter unsigned and sign type (#21062) #21236

Merged

XuHuaiyu pushed a commit that referenced this pull request Nov 26, 2020

planner, type: fix AggFieldType error when encouter unsigned and sign…

1e73c51

… type (#21062) (#21236)

rebelice mentioned this pull request Nov 27, 2020

The common type of bigint and bigint unsigned is not stable in TiDB #15771

Closed

rogeryk deleted the 9869-bigint-unsigned-in-casefunc-covert-to-signed branch January 2, 2021 08:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner, type: fix AggFieldType error when encouter unsigned and sign type #21062

planner, type: fix AggFieldType error when encouter unsigned and sign type #21062

rogeryk commented Nov 15, 2020 •

edited by XuHuaiyu

Loading

sre-bot commented Nov 15, 2020

rogeryk commented Nov 15, 2020

rogeryk commented Nov 15, 2020

XuHuaiyu Nov 16, 2020

rogeryk Nov 16, 2020 •

edited

Loading

XuHuaiyu Nov 16, 2020

rogeryk Nov 16, 2020

XuHuaiyu Nov 16, 2020

rogeryk Nov 16, 2020 •

edited

Loading

XuHuaiyu left a comment

XuHuaiyu commented Nov 16, 2020

XuHuaiyu commented Nov 16, 2020

rogeryk commented Nov 16, 2020

rogeryk commented Nov 16, 2020

XuHuaiyu commented Nov 17, 2020

ichn-hu left a comment

rogeryk commented Nov 20, 2020

rogeryk commented Nov 20, 2020

rogeryk commented Nov 21, 2020

rogeryk commented Nov 21, 2020

rogeryk commented Nov 22, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

XuHuaiyu commented Nov 24, 2020

XuHuaiyu commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

planner, type: fix AggFieldType error when encouter unsigned and sign type #21062

planner, type: fix AggFieldType error when encouter unsigned and sign type #21062

Conversation

rogeryk commented Nov 15, 2020 • edited by XuHuaiyu Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

sre-bot commented Nov 15, 2020

rogeryk commented Nov 15, 2020

rogeryk commented Nov 15, 2020

XuHuaiyu Nov 16, 2020

Choose a reason for hiding this comment

rogeryk Nov 16, 2020 • edited Loading

Choose a reason for hiding this comment

XuHuaiyu Nov 16, 2020

Choose a reason for hiding this comment

rogeryk Nov 16, 2020

Choose a reason for hiding this comment

XuHuaiyu Nov 16, 2020

Choose a reason for hiding this comment

rogeryk Nov 16, 2020 • edited Loading

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Nov 16, 2020

XuHuaiyu commented Nov 16, 2020

rogeryk commented Nov 16, 2020

rogeryk commented Nov 16, 2020

XuHuaiyu commented Nov 17, 2020

ichn-hu left a comment

Choose a reason for hiding this comment

rogeryk commented Nov 20, 2020

rogeryk commented Nov 20, 2020

rogeryk commented Nov 21, 2020

rogeryk commented Nov 21, 2020

rogeryk commented Nov 22, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

lzmhhh123 commented Nov 23, 2020

rogeryk commented Nov 23, 2020

XuHuaiyu commented Nov 24, 2020

XuHuaiyu commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

ti-srebot commented Nov 24, 2020

rogeryk commented Nov 15, 2020 •

edited by XuHuaiyu

Loading

rogeryk Nov 16, 2020 •

edited

Loading

rogeryk Nov 16, 2020 •

edited

Loading