v4.0.15 upgrade to v4.0.16, cdc panic for interface conversion: interface {} is nil, not types.Time

[seiya-annie]

What did you do?

1. ticdc stoped for https://github.com/pingcap/ticdc/issues/3810
2. then set global sql_mode="" on downstream tidb server
3. ticdc begin to sync data, but soon panic for interface conversion: interface {} is nil, not types.Time

[2021/12/16 15:02:12.080 +08:00] [WARN] [client_changefeed.go:175] ["query changefeed info failed"] [error="owner not found"]
[
{
"id": "simple-cdc-task",
"summary": null
}
]
[root@cdc-0 bin]# ./cdc cli changefeed list --pd=http://pd-peer:12379
[
{
"id": "simple-cdc-task",
"summary": {
"state": "normal",
"tso": 429819315919519752,
"checkpoint": "2021-12-16 12:56:09.151",
"error": {
"addr": "cdc-peer:8300",
"code": "CDC:ErrProcessorUnknown",
"message": "[CDC:ErrReachMaxTry]reach maximum try: 8"
}
}
}
]

maybe failed for
2021/12/16 04:56:06 [ddl/all.jsonnet#ddl/add_column/addcolumn_datetime_notnull] passed
2021/12/16 04:56:09 [ddl/all.jsonnet#ddl/add_column/addcolumn_datetime_notnull_after] passed

What did you expect to see?

ticdc sync data successfully

What did you see instead?

goroutine 1020 [running]:
github.com/pingcap/tidb/types.(*Datum).GetMysqlTime(...)
github.com/pingcap/tidb@v1.1.0-beta.0.20210512055339-e25d1d0b7354/types/datum.go:347
github.com/pingcap/ticdc/cdc/entry.formatColVal(0x5, 0x29c5ce6, 0xb, 0x0, 0x0, 0xc0058ebc60, 0x13, 0x13, 0x0, 0x0, ...)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:411 +0x6f9
github.com/pingcap/ticdc/cdc/entry.getDefaultOrZeroValue(0xc00232bb80, 0xc00b0ab170, 0x7, 0x4c9f560, 0x0, 0x0, 0x0)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:477 +0x122
github.com/pingcap/ticdc/cdc/entry.datum2Column(0xc0057869a0, 0xc00b0ab170, 0x1, 0x0, 0x100c008669ac8, 0x1e8483, 0x0, 0xc00b0ab170)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:321 +0x79c
github.com/pingcap/ticdc/cdc/entry.(*mounterImpl).mountRowKVEntry(0xc0015b6bc0, 0xc0057869a0, 0xc0072e9dc0, 0x45, 0x17, 0xc0000beb80, 0x32)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:370 +0x32a
github.com/pingcap/ticdc/cdc/entry.(*mounterImpl).unmarshalAndMountRowChanged.func1(0xc002d20370, 0x250, 0xc005845e30, 0xc00673e4a9, 0xa, 0x17, 0xc0015b6bc0, 0x5f7066406b4000c, 0x5f7066428800007, 0x250, ...)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:210 +0x422
github.com/pingcap/ticdc/cdc/entry.(*mounterImpl).unmarshalAndMountRowChanged(0xc0015b6bc0, 0x305df00, 0xc000d91000, 0xc005845e30, 0xc00b102101, 0x0, 0x0)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:213 +0x1ed
github.com/pingcap/ticdc/cdc/entry.(*mounterImpl).codecWorker(0xc0015b6bc0, 0x305df00, 0xc000d91000, 0x8, 0x0, 0x0)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:135 +0x41b
github.com/pingcap/ticdc/cdc/entry.(*mounterImpl).Run.func2(0xc001630f68, 0x0)
github.com/pingcap/ticdc@/cdc/entry/mounter.go:107 +0x45
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0033b7680, 0xc0033b7830)
golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
golang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:54 +0x66
panic: interface conversion: interface {} is nil, not types.Time

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

v4.0.16
TiCDC version (execute cdc version):

[release-version=v4.0.16] [git-hash=238d38dcf82c927ce303d77ccc998275cb8a5dba] [git-branch=heads/refs/tags/v4.0.16] [utc-build-time="2021-12-14 01:54:44"] [go-version="go version go1.13 linux/amd64"]

[seiya-annie]

cdc_stderr.log

[overvenus]

https://github.com/pingcap/ticdc/pull/3921 reverts #3855, as temporally workaround.

[maxshuang]

After test, this panic can reproduce on TIDB 4.0.15 by "amend + add column datetime not null default '2020-10-10 10:10:10'", but will not occur on TIDB 5.3.0.

The root cause is TiDB default value type is not consistent with column type, and TiCDC not deal it properly.
I will make a pr to fix it.

[VelocityLight]

update