tidb panic while query a table which was set collation

[aytrack]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. set new_collations_enabled_on_first_bootstrap = true
2. execute the sqls.

create table t3(a char(10)  CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, primary key (a));
create table t4(a char(10), primary key (a)) CHARACTER SET utf8 COLLATE utf8_general_ci;
select * from t3 where a > 0x80;
select * from t4 where a > 0x80;

2. What did you expect to see? (Required)

execute successfully and TIDB not panic

3. What did you see instead (Required)

mysql > select * from t4 where a > 0x80;
Reconnecting...
(1105, 'runtime error: index out of range [1] with length 1')
mysql > select * from t3 where a > 0x80;
Reconnecting...
(1105, 'runtime error: index out of range [1] with length 1')

tidb log:

[2021/03/24 16:07:05.791 +08:00] [WARN] [collate.go:145] ["Unable to get collator by name, use binCollator instead."] [name=] [stack="github.com/pingcap/tidb/util/collate.GetCollator\n\t/Users/aytrack/gitproject/pingcap/tidb/util/collate/collate.go:148\ngithub.com/pingcap/tidb/types.CompareString\n\t/Users/aytrack/gitproject/pingcap/tidb/types/compare.go:118\ngithub.com/pingcap/tidb/types.(*Datum).compareString\n\t/Users/aytrack/gitproject/pingcap/tidb/types/datum.go:681\ngithub.com/pingcap/tidb/types.(*Datum).CompareDatum\n\t/Users/aytrack/gitproject/pingcap/tidb/types/datum.go:565\ngithub.com/pingcap/tidb/util/ranger.convertPoint\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:104\ngithub.com/pingcap/tidb/util/ranger.points2Ranges\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:58\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).buildCNFIndexRange\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:361\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).detachCNFCondAndBuildRangeForIndex\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:334\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).detachCondAndBuildRangeForCols\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:660\ngithub.com/pingcap/tidb/util/ranger.DetachCondAndBuildRangeForIndex\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:626\ngithub.com/pingcap/tidb/planner/core.(*DataSource).fillIndexPath\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/logical_plans.go:846\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:272\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:125\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:118\ngithub.com/pingcap/tidb/planner/core.physicalOptimize\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/optimizer.go:212\ngithub.com/pingcap/tidb/planner/core.DoOptimize\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/optimizer.go:143\ngithub.com/pingcap/tidb/planner.optimize\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/optimize.go:289\ngithub.com/pingcap/tidb/planner.Optimize\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/optimize.go:125\ngithub.com/pingcap/tidb/executor.(*Compiler).Compile\n\t/Users/aytrack/gitproject/pingcap/tidb/executor/compiler.go:62\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/Users/aytrack/gitproject/pingcap/tidb/session/session.go:1411\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/Users/aytrack/gitproject/pingcap/tidb/server/driver_tidb.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/Users/aytrack/gitproject/pingcap/tidb/server/conn.go:1623\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/Users/aytrack/gitproject/pingcap/tidb/server/conn.go:1496\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/Users/aytrack/gitproject/pingcap/tidb/server/conn.go:1030\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/Users/aytrack/gitproject/pingcap/tidb/server/conn.go:795\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/Users/aytrack/gitproject/pingcap/tidb/server/server.go:477"]
[2021/03/24 16:07:05.810 +08:00] [ERROR] [conn.go:736] ["connection running loop panic"] [conn=7] [lastSQL="select * from t4 where a > 0x80"] [err="runtime error: index out of range [1] with length 1"] [stack="goroutine 1025 [running]:\ngithub.com/pingcap/tidb/server.(*clientConn).Run.func1(0x6a019e0, 0xc012c47d10, 0xc012a9ba00)\n\t/Users/aytrack/gitproject/pingcap/tidb/server/conn.go:734 +0xf5\npanic(0x63be320, 0xc01444ba00)\n\t/usr/local/go/src/runtime/panic.go:969 +0x175\ngithub.com/pingcap/tidb/util/collate.decodeRune(0x808a880, 0x1, 0x0, 0xc0145b616a, 0x0)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/collate/collate.go:258 +0x135\ngithub.com/pingcap/tidb/util/collate.(*generalCICollator).Key(0x830ec70, 0x808a880, 0x1, 0x830ec70, 0x10, 0x60f64a0)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/collate/general_ci.go:48 +0xd3\ngithub.com/pingcap/tidb/util/codec.encodeString(0xc0145b6160, 0x0, 0xa, 0x5, 0x0, 0xc011949d60, 0xf, 0x808a880, 0x1, 0x1, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/codec/codec.go:200 +0x175\ngithub.com/pingcap/tidb/util/codec.encode(0xc014530f00, 0xc0145b6160, 0x0, 0xa, 0xc012daba48, 0x1, 0x1, 0x4010401, 0xc01475b7c0, 0x50, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/codec/codec.go:94 +0x4b0\ngithub.com/pingcap/tidb/util/codec.EncodeKey(...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/codec/codec.go:287\ngithub.com/pingcap/tidb/util/ranger.validInterval(0xc014530f00, 0xc01475b7c0, 0xc01475b770, 0xc01475b770, 0x0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:36 +0x189\ngithub.com/pingcap/tidb/util/ranger.points2Ranges(0xc014530f00, 0xc014864700, 0x2, 0x4, 0xc01484cae0, 0x2, 0x2, 0x0, 0xc014864700, 0x2)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:66 +0x1ae\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).buildCNFIndexRange(0xc012dabfa0, 0xc014346e00, 0x2, 0x2, 0x0, 0xc0142b5eb0, 0x1, 0x1, 0x1, 0x0, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/ranger.go:361 +0x759\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).detachCNFCondAndBuildRangeForIndex(0xc012dabfa0, 0xc0142b5e50, 0x1, 0x1, 0xc014346e00, 0x1, 0x1, 0x808a701, 0x1, 0x1, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:334 +0x10fb\ngithub.com/pingcap/tidb/util/ranger.(*rangeDetacher).detachCondAndBuildRangeForCols(0xc012dabfa0, 0x0, 0x1, 0x1)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:660 +0x212\ngithub.com/pingcap/tidb/util/ranger.DetachCondAndBuildRangeForIndex(0x6a44a80, 0xc012a9bb00, 0xc0142b5e50, 0x1, 0x1, 0xc014346df0, 0x1, 0x1, 0xc0145b6128, 0x1, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/util/ranger/detacher.go:626 +0xf5\ngithub.com/pingcap/tidb/planner/core.(*DataSource).fillIndexPath(0xc013be6f00, 0xc014520b40, 0xc0142b5e50, 0x1, 0x1, 0x6a705e0, 0xc01475b360)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/logical_plans.go:846 +0x55f\ngithub.com/pingcap/tidb/planner/core.(*DataSource).DeriveStats(0xc013be6f00, 0x830ec70, 0x0, 0x0, 0xc01475b220, 0x830ec70, 0x0, 0x0, 0x0, 0x0, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:272 +0x145\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats(0xc013be6f20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:125 +0x30f\ngithub.com/pingcap/tidb/planner/core.(*baseLogicalPlan).recursiveDeriveStats(0xc0145288e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/stats.go:118 +0x1b0\ngithub.com/pingcap/tidb/planner/core.physicalOptimize(0x6a434c0, 0xc0145288c0, 0xc0145b6118, 0x6a434c0, 0xc0145288c0, 0x6a434c0, 0xc0145288c0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/optimizer.go:212 +0x5d\ngithub.com/pingcap/tidb/planner/core.DoOptimize(0x6a019e0, 0xc01484ff50, 0x6a44a80, 0xc012a9bb00, 0xa2, 0x6a434c0, 0xc0145288c0, 0x0, 0x0, 0x0, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/core/optimizer.go:143 +0x145\ngithub.com/pingcap/tidb/planner.optimize(0x6a019e0, 0xc01484ff50, 0x6a44a80, 0xc012a9bb00, 0x6a18ac0, 0xc0140d1500, 0x6a3c7a0, 0xc01271d590, 0x0, 0x0, ...)\n\t/Users/aytrack/gitproject/pingcap/tidb/planner/optimize.go:289 +0x77f\ngithub.com/pingcap/tidb/planner.Optimize(0x6a019e0, 0xc01484ff50, 0x6a44a80, 0xc012a9bb00, 0x6a18ac0, 0xc0140d1500, 0x6a3c7a0, 0xc01271d590, 0x0"]

4. What is your TiDB version? (Required)

master: 1cebae2
release-5.0: 7ce49da
release-4.0: 11a9254

[Reminiscent]

/assign

[Reminiscent]

It's related to the collation decode. Assign to @xiongjiwei

[Reminiscent]

/assign @xiongjiwei

[xiongjiwei]

it happens because we assume all the string to compare is valid utf8, but it is not true, it may be a binary charset. MySQL does binary compare if it is an incorrect string. https://github.com/mysql/mysql-server/blob/7ed30a748964c009d4909cb8b4b22036ebdef239/strings/ctype-utf8.cc#L7541-L7544
PTAL @bb7133 @wjhuang2016

fix this issue will cause performance regression

[eurekaka]

@wjhuang2016 PTAL

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.