ddl: fix cast date as timestamp will write invalid value #26362

AilinKid · 2021-07-20T03:32:25Z

Signed-off-by: ailinkid 314806019@qq.com

What problem does this PR solve?

Issue Number: close #26292

What is changed and how it works?

What's Changed:

Reorg Process include 2 stage:
1: cast the relative column to changing column at write stage.
2: cast the old record to new record at the background reorg stage.

Reorg process have a special control of cast function.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

ddl: fix cast date as timestamp will write invalid value

Signed-off-by: ailinkid <314806019@qq.com>

sessionctx/stmtctx/stmtctx.go

ddl/column_type_change_test.go

zimulala · 2021-07-20T03:54:35Z

ddl/column_type_change_test.go

+		switch job.SchemaState {
+		case model.StateWriteOnly:
+			_, checkErr1 = internalTK.Exec("insert into `t` set  `a` = '3977-02-22'") // this(string) will be cast to a as date, then cast a(date) as timestamp to changing column.
+			_, checkErr2 = internalTK.Exec("update t set `a` = '3977-02-22'")


Do we need to add a delete statement here?

only WRITE will cause failure

Signed-off-by: ailinkid <314806019@qq.com>

AilinKid · 2021-07-20T08:56:03Z

/run-all-tests

types/datum.go

Signed-off-by: ailinkid <314806019@qq.com>

AilinKid · 2021-07-20T10:21:27Z

Running in native mode.
ok      github.com/pingcap/tidb/bindinfo        4.713s  coverage: 86.2% of statements
ok      github.com/pingcap/tidb/config  0.045s  coverage: 67.6% of statements
ok      github.com/pingcap/tidb/ddl     172.363s        coverage: 85.6% of statements
ok      github.com/pingcap/tidb/ddl/failtest    133.268s        coverage: [no statements]
ok      github.com/pingcap/tidb/ddl/placement   0.025s  coverage: 100.0% of statements
?       github.com/pingcap/tidb/ddl/testutil    [no test files]
ok      github.com/pingcap/tidb/ddl/util        0.337s  coverage: 46.5% of statements
ok      github.com/pingcap/tidb/distsql 0.102s  coverage: 59.8% of statements
ok      github.com/pingcap/tidb/domain  0.899s  coverage: 74.3% of statements
ok      github.com/pingcap/tidb/domain/infosync 0.148s  coverage: 18.3% of statements
ok      github.com/pingcap/tidb/errno   0.007s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/executor        262.851s        coverage: 77.5% of statements
ok      github.com/pingcap/tidb/executor/aggfuncs       2.035s  coverage: 77.3% of statements
ok      github.com/pingcap/tidb/executor/oomtest        0.114s  coverage: [no statements] [no tests to run]
ok      github.com/pingcap/tidb/executor/seqtest        6.408s  coverage: [no statements]
ok      github.com/pingcap/tidb/expression      23.092s coverage: 81.8% of statements
ok      github.com/pingcap/tidb/expression/aggregation  0.101s  coverage: 42.6% of statements
?       github.com/pingcap/tidb/expression/generator/helper     [no test files]
ok      github.com/pingcap/tidb/infoschema      4.133s  coverage: 67.2% of statements
ok      github.com/pingcap/tidb/infoschema/perfschema   0.436s  coverage: 70.1% of statements
ok      github.com/pingcap/tidb/kv      0.114s  coverage: 60.8% of statements
?       github.com/pingcap/tidb/lock    [no test files]
ok      github.com/pingcap/tidb/meta    0.150s  coverage: 73.3% of statements
ok      github.com/pingcap/tidb/meta/autoid     0.622s  coverage: 66.4% of statements
ok      github.com/pingcap/tidb/metrics 0.018s  coverage: 90.2% of statements
ok      github.com/pingcap/tidb/owner   15.861s coverage: 68.4% of statements
?       github.com/pingcap/tidb/planner [no test files]
ok      github.com/pingcap/tidb/planner/cascades        0.973s  coverage: 94.7% of statements
ok      github.com/pingcap/tidb/planner/core    80.572s coverage: 78.3% of statements
ok      github.com/pingcap/tidb/planner/implementation  0.032s  coverage: 3.7% of statements
ok      github.com/pingcap/tidb/planner/memo    0.063s  coverage: 86.9% of statements
?       github.com/pingcap/tidb/planner/property        [no test files]
?       github.com/pingcap/tidb/planner/util    [no test files]
ok      github.com/pingcap/tidb/plugin  0.035s  coverage: 58.7% of statements
ok      github.com/pingcap/tidb/plugin/conn_ip_example  0.067s  coverage: 81.8% of statements
?       github.com/pingcap/tidb/privilege       [no test files]
ok      github.com/pingcap/tidb/privilege/privileges    90.135s coverage: 82.7% of statements
ok      github.com/pingcap/tidb/server  58.948s coverage: 68.5% of statements
ok      github.com/pingcap/tidb/session 69.350s coverage: 79.5% of statements
?       github.com/pingcap/tidb/session/txninfo [no test files]
ok      github.com/pingcap/tidb/sessionctx      0.011s  coverage: 20.8% of statements
ok      github.com/pingcap/tidb/sessionctx/binloginfo   1.437s  coverage: 74.8% of statements
ok      github.com/pingcap/tidb/sessionctx/stmtctx      0.011s  coverage: 29.5% of statements
ok      github.com/pingcap/tidb/sessionctx/variable     0.094s  coverage: 47.2% of statements
ok      github.com/pingcap/tidb/statistics      6.598s  coverage: 74.9% of statements
ok      github.com/pingcap/tidb/statistics/handle       37.833s coverage: 82.5% of statements
ok      github.com/pingcap/tidb/store   85.238s coverage: 95.5% of statements
ok      github.com/pingcap/tidb/store/copr      0.064s  coverage: 13.3% of statements
ok      github.com/pingcap/tidb/store/driver    0.649s  coverage: 26.8% of statements
?       github.com/pingcap/tidb/store/driver/backoff    [no test files]
?       github.com/pingcap/tidb/store/driver/error      [no test files]
?       github.com/pingcap/tidb/store/driver/options    [no test files]
ok      github.com/pingcap/tidb/store/driver/txn        0.013s  coverage: 16.5% of statements
ok      github.com/pingcap/tidb/store/gcworker  29.006s coverage: 76.2% of statements
ok      github.com/pingcap/tidb/store/helper    0.126s  coverage: 42.3% of statements
ok      github.com/pingcap/tidb/store/mockstore 0.060s  coverage: 43.2% of statements
ok      github.com/pingcap/tidb/store/mockstore/mockcopr        0.237s  coverage: 34.8% of statements
?       github.com/pingcap/tidb/store/mockstore/mockstorage     [no test files]
ok      github.com/pingcap/tidb/store/mockstore/unistore        0.020s  coverage: 14.8% of statements
?       github.com/pingcap/tidb/store/mockstore/unistore/client [no test files]
?       github.com/pingcap/tidb/store/mockstore/unistore/config [no test files]
ok      github.com/pingcap/tidb/store/mockstore/unistore/cophandler     0.264s  coverage: 12.4% of statements
ok      github.com/pingcap/tidb/store/mockstore/unistore/lockstore      11.048s coverage: 60.8% of statements
?       github.com/pingcap/tidb/store/mockstore/unistore/metrics        [no test files]
?       github.com/pingcap/tidb/store/mockstore/unistore/pd     [no test files]
?       github.com/pingcap/tidb/store/mockstore/unistore/server [no test files]
ok      github.com/pingcap/tidb/store/mockstore/unistore/tikv   4.019s  coverage: 33.9% of statements
?       github.com/pingcap/tidb/store/mockstore/unistore/tikv/dbreader  [no test files]
?       github.com/pingcap/tidb/store/mockstore/unistore/tikv/mvcc      [no test files]
?       github.com/pingcap/tidb/store/mockstore/unistore/tikv/pberror   [no test files]
ok      github.com/pingcap/tidb/store/mockstore/unistore/util/lockwaiter        0.111s  coverage: 84.0% of statements
ok      github.com/pingcap/tidb/structure       0.040s  coverage: 79.6% of statements
ok      github.com/pingcap/tidb/table   0.014s  coverage: 69.1% of statements
ok      github.com/pingcap/tidb/table/tables    1.549s  coverage: 55.0% of statements
ok      github.com/pingcap/tidb/tablecodec      0.012s  coverage: 32.7% of statements
ok      github.com/pingcap/tidb/telemetry       5.658s  coverage: 54.9% of statements
?       github.com/pingcap/tidb/testkit [no test files]
ok      github.com/pingcap/tidb/tidb-server     0.029s  coverage: 22.2% of statements
ok      github.com/pingcap/tidb/types   0.556s  coverage: 82.6% of statements
ok      github.com/pingcap/tidb/types/json      0.010s  coverage: 76.7% of statements
ok      github.com/pingcap/tidb/types/parser_driver     0.008s  coverage: 29.5% of statements
ok      github.com/pingcap/tidb/util    0.032s  coverage: 53.2% of statements
ok      github.com/pingcap/tidb/util/admin      0.811s  coverage: 55.9% of statements
ok      github.com/pingcap/tidb/util/arena      0.003s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/util/bitmap     0.162s  coverage: 85.7% of statements
ok      github.com/pingcap/tidb/util/checksum   0.015s  coverage: 84.3% of statements
ok      github.com/pingcap/tidb/util/chunk      2.869s  coverage: 82.3% of statements
ok      github.com/pingcap/tidb/util/codec      0.020s  coverage: 82.5% of statements
ok      github.com/pingcap/tidb/util/collate    0.007s  coverage: 69.4% of statements
ok      github.com/pingcap/tidb/util/cteutil    0.016s  coverage: 69.7% of statements
ok      github.com/pingcap/tidb/util/dbterror   0.005s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/util/deadlockhistory    0.008s  coverage: 95.7% of statements
ok      github.com/pingcap/tidb/util/disjointset        0.004s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/util/disk       0.008s  coverage: 50.0% of statements
?       github.com/pingcap/tidb/util/domainutil [no test files]
ok      github.com/pingcap/tidb/util/encrypt    0.006s  coverage: 91.9% of statements
ok      github.com/pingcap/tidb/util/execdetails        0.005s  coverage: 70.8% of statements
ok      github.com/pingcap/tidb/util/expensivequery     0.009s  coverage: 13.5% of statements
ok      github.com/pingcap/tidb/util/fastrand   0.004s  coverage: 86.7% of statements
ok      github.com/pingcap/tidb/util/filesort   0.010s  coverage: 85.5% of statements
ok      github.com/pingcap/tidb/util/format     0.003s  coverage: 83.6% of statements
?       github.com/pingcap/tidb/util/gcutil     [no test files]
ok      github.com/pingcap/tidb/util/generatedexpr      0.011s  coverage: 0.0% of statements [no tests to run]
ok      github.com/pingcap/tidb/util/hack       0.002s  coverage: 92.3% of statements
?       github.com/pingcap/tidb/util/hint       [no test files]
?       github.com/pingcap/tidb/util/israce     [no test files]
ok      github.com/pingcap/tidb/util/kvcache    0.010s  coverage: 63.5% of statements
ok      github.com/pingcap/tidb/util/localpool  0.014s  coverage: 96.8% of statements
ok      github.com/pingcap/tidb/util/logutil    0.019s  coverage: 60.8% of statements
ok      github.com/pingcap/tidb/util/math       0.311s  coverage: 30.0% of statements
ok      github.com/pingcap/tidb/util/memory     0.012s  coverage: 64.2% of statements
ok      github.com/pingcap/tidb/util/mock       0.012s  coverage: 14.7% of statements
ok      github.com/pingcap/tidb/util/mvmap      0.005s  coverage: 86.6% of statements
ok      github.com/pingcap/tidb/util/parser     0.018s  coverage: 60.0% of statements
?       github.com/pingcap/tidb/util/pdapi      [no test files]
ok      github.com/pingcap/tidb/util/plancodec  0.009s  coverage: 8.2% of statements
ok      github.com/pingcap/tidb/util/printer    0.009s  coverage: 88.3% of statements
ok      github.com/pingcap/tidb/util/profile    0.885s  coverage: 55.4% of statements
ok      github.com/pingcap/tidb/util/ranger     3.568s  coverage: 81.4% of statements
ok      github.com/pingcap/tidb/util/resourcegrouptag   0.007s  coverage: 77.8% of statements
ok      github.com/pingcap/tidb/util/rowDecoder 0.020s  coverage: 59.8% of statements
ok      github.com/pingcap/tidb/util/rowcodec   0.029s  coverage: 88.0% of statements
ok      github.com/pingcap/tidb/util/selection  0.772s  coverage: 86.8% of statements
ok      github.com/pingcap/tidb/util/sem        0.014s  coverage: 65.4% of statements
ok      github.com/pingcap/tidb/util/set        0.038s  coverage: 48.6% of statements
?       github.com/pingcap/tidb/util/signal     [no test files]
?       github.com/pingcap/tidb/util/sli        [no test files]
ok      github.com/pingcap/tidb/util/slice      0.004s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/util/sqlexec    0.011s  coverage: 89.7% of statements
ok      github.com/pingcap/tidb/util/stmtsummary        0.032s  coverage: 94.4% of statements
ok      github.com/pingcap/tidb/util/stringutil 0.006s  coverage: 87.7% of statements
ok      github.com/pingcap/tidb/util/sys/linux  0.004s  coverage: 68.4% of statements
ok      github.com/pingcap/tidb/util/sys/storage        0.004s  coverage: 83.3% of statements
ok      github.com/pingcap/tidb/util/systimemon 1.007s  coverage: 100.0% of statements
?       github.com/pingcap/tidb/util/tableutil  [no test files]
?       github.com/pingcap/tidb/util/testbridge [no test files]
ok      github.com/pingcap/tidb/util/testkit    0.029s  coverage: 5.6% of statements
?       github.com/pingcap/tidb/util/testleak   [no test files]
ok      github.com/pingcap/tidb/util/testutil   0.015s  coverage: 19.2% of statements
ok      github.com/pingcap/tidb/util/texttree   0.004s  coverage: 100.0% of statements
ok      github.com/pingcap/tidb/util/timeutil   0.008s  coverage: 48.7% of statements
ok      github.com/pingcap/tidb/util/topsql     2.751s  coverage: 51.9% of statements
ok      github.com/pingcap/tidb/util/topsql/reporter    4.586s  coverage: 84.1% of statements
?       github.com/pingcap/tidb/util/topsql/reporter/mock       [no test files]
?       github.com/pingcap/tidb/util/topsql/tracecpu    [no test files]
?       github.com/pingcap/tidb/util/topsql/tracecpu/mock       [no test files]
ok      github.com/pingcap/tidb/util/tracing    0.005s  coverage: 93.3% of statements
?       github.com/pingcap/tidb/util/versioninfo        [no test files]
ok      github.com/pingcap/tidb/util/vitess     0.004s  coverage: 88.9% of statements
go generate ./...
./tools/check/check-gogenerate.sh
Great, all tests passed.

I'v made the test locally.

AilinKid · 2021-07-20T10:21:53Z

/build

zimulala

LGTM

ti-chi-bot · 2021-07-20T10:33:28Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

tangenta
zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

AilinKid · 2021-07-20T15:11:23Z

/run-check_dev_2

AilinKid · 2021-07-20T15:11:59Z

/merge

ti-chi-bot · 2021-07-20T15:12:03Z

This pull request has been accepted and is ready to merge.

Commit hash: 111eefd

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-07-20T15:20:08Z

cherry pick to release-5.1 in PR #26395

…6395)

fix cast date as timestamp will write invalid value

f75e15c

Signed-off-by: ailinkid <314806019@qq.com>

ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 20, 2021

Merge branch 'master' into fix-issue-cast-date-timestamp

2ac2b84

zimulala reviewed Jul 20, 2021

View reviewed changes

github-actions bot added component/expression sig/sql-infra SIG: SQL Infra labels Jul 20, 2021

AilinKid added 3 commits July 20, 2021 13:18

address comment

b349e5e

Signed-off-by: ailinkid <314806019@qq.com>

address comment

0b43263

Signed-off-by: ailinkid <314806019@qq.com>

Merge branch 'master' into fix-issue-cast-date-timestamp

267da9e

AilinKid added the needs-cherry-pick-release-5.1 label Jul 20, 2021

AilinKid added 2 commits July 20, 2021 15:50

address comment

06511b0

Signed-off-by: ailinkid <314806019@qq.com>

Merge branch 'master' into fix-issue-cast-date-timestamp

3016ad2

zimulala reviewed Jul 20, 2021

View reviewed changes

types/datum.go Outdated Show resolved Hide resolved

AilinKid added 2 commits July 20, 2021 17:16

address comment

c2600ee

Signed-off-by: ailinkid <314806019@qq.com>

Merge branch 'master' into fix-issue-cast-date-timestamp

48f6e1b

zimulala approved these changes Jul 20, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 20, 2021

tangenta approved these changes Jul 20, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 20, 2021

AilinKid added 2 commits July 20, 2021 18:51

Merge branch 'master' into fix-issue-cast-date-timestamp

3c79c05

Merge branch 'master' into fix-issue-cast-date-timestamp

111eefd

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 20, 2021

ti-chi-bot merged commit 6329c86 into pingcap:master Jul 20, 2021

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 20, 2021

cherry pick pingcap#26362 to release-5.1

9d1ab78

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jul 20, 2021

ddl: fix cast date as timestamp will write invalid value (#26362) #26395

Merged

11 tasks

ti-chi-bot pushed a commit that referenced this pull request Jul 21, 2021

ddl: fix cast date as timestamp will write invalid value (#26362) (#2…

849b206

…6395)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: fix cast date as timestamp will write invalid value #26362

ddl: fix cast date as timestamp will write invalid value #26362

AilinKid commented Jul 20, 2021

zimulala Jul 20, 2021

AilinKid Jul 20, 2021

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

zimulala left a comment

ti-chi-bot commented Jul 20, 2021

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

ti-chi-bot commented Jul 20, 2021

ti-srebot commented Jul 20, 2021

ddl: fix cast date as timestamp will write invalid value #26362

ddl: fix cast date as timestamp will write invalid value #26362

Conversation

AilinKid commented Jul 20, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

zimulala Jul 20, 2021

Choose a reason for hiding this comment

AilinKid Jul 20, 2021

Choose a reason for hiding this comment

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

zimulala left a comment

Choose a reason for hiding this comment

ti-chi-bot commented Jul 20, 2021

AilinKid commented Jul 20, 2021

AilinKid commented Jul 20, 2021

ti-chi-bot commented Jul 20, 2021

ti-srebot commented Jul 20, 2021