row delete event may lose field value when new collation is enabled.

[glorv]

What did you do?

Update tidb dep to newest master version (with new_collation is enabled by default), unit test TestMounterDisableOldValue fails consistently. Seems delete event (with old value disabled and clustered_index enabled) is not compatible with new collation.

Root cause analysis:

When old value is disabled, cdc will parse corresponding pk fields from row handle by function tablecodec.DecodeHandleToDatumMap. But if new_collation is enabled, this function will skip string fields if the collation is not binary. Thus, the returned event will miss some fields.
See: https://github.com/pingcap/tidb/blob/aa24b9a7b40ee51671e43ee1ebea63fdbd0bc57f/tablecodec/tablecodec.go#L482

What did you expect to see?

No response

What did you see instead?

unit test TestMounterDisableOldValue failed. See: https://ci.pingcap.net/blue/organizations/jenkins/ut-check/detail/ut-check/1290/pipeline

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

[glorv]

/cc @liuzix @overvenus