Default value may cause more data inconsistency

[maxshuang]

What did you do?

Default value may cause more data inconsistency. We create this issue to track related issue and progress.

First default value error is from #3793. In this case, I find TiDB "insert + amend + add column datetime default 'xxxx'" will trigger mounter getDefaultOrZeroValue logic, which means the new column is miss in the underlaying TiKV.

"getDefaultOrZeroValue" logic will return raw TiDB time type, which is not supported by go sql. So we format all default value or zero value.

• upgrade from v4.0.15 to v4.0.16, ticdc stoped with CDC:ErrProcessorUnknown #3793

After Pr for #3793, we find another panic issue raised by default value #3918. In this case, I find TiDB will return default value with string or int like type which is inconsistent with the original column type, and cdc get panic therefore.

Issue #3929 is found when testing issue 3918. It seems that when column data is miss in the underlaying TiKV, TiDB would use default value to fill the placeholder before checking not null flag.

• v4.0.15 upgrade to v4.0.16, cdc panic for interface conversion: interface {} is nil, not types.Time #3918
• Data Inconsistency when meet TiDB "amend + add column default 'xxx'" #3929

After using default value to fill the placeholder before checking not null flag, I find the integration test multi_source always detect data inconsistency.

So, after discussing with wenjun, I find we use the wrong default value to fill the missing column data. We should use OriginalDefaultValue instead.

1. OriginalDefaultValue is set by "add column default xxx".
2. Only DefaultValue will be set When we define column default xxx in the create table or call Alter/Change column.

When the DefaultValue is not null, TiDB will always insert the column data with default value to the underlying storge even we don't specify the column explicitly.

TiDB will miss column data in the following 3 scenes:

1. column define default null at creating+ insert without explicit column
2. alter table add column default xxx + old existing data
3. amend + insert without explicit column + alter table add column default xxx

At all these 3 scenes, OriginalDefaultValue is always consistent with the missing column data.

Please note that following scene will not cause missing column data:

create table t(id int);
alter table t add column c1 int default 11;
alter table t modify column c1 int default null;
insert into table(id) values(1); ---> TiDB insert <1, nilFlag>, not <1>

So we can expect "getDefaultOrZeroValue" being called at following 4 scenes:

1. column define default null at creating+ insert without explicit column
2. alter table add column default xxx + old existing data
3. amend + insert without explicit column + alter table add column default xxx
4. online DDL drop column + data insert at state delete-only

Currently, default value error (using wrong value to fill missing column data) will affect following scenes:

1. ineligible table + forceupdate flag
2. amend + insert without explicit column + alter table add column default xxx

ToDoList:

• Add more online DDL debug log for testing #4066
• Check some dynamic column type support, including auto_increment, current_timestamp, or other expression
• Check the support of different timezone for default time value，ref to：TiDB default vaule deal
• Add related integration test and test-infra for DefaultValue #4223

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

v5.3.0

TiCDC version (execute cdc version):

v5.4.0 and before

[maxshuang]

/remove-severity major

[maxshuang]

/severity moderate

[3AceShowHand]

• mounter(ticdc): default value convert to the correct data type #10804
• mounter(ticdc): timezone fill default value should also consider tz. #10932

After these 2 PRs gets merged, known default value related issue fixed, so close this .