[ticdc 5.3.0] kafka changefeed error: kafka server: Message was too large

[Tammyxia]

What did you do?

-here's kafka topic server.log:

• upgrade tidb cluster from 5.2.1 to 5.3.0

• remove all changefeed, than create 2 new changefeeds: tiup ctl:v5.3.0 cdc changefeed create --pd=http://172.16.4.158:2379 --sink-uri="mysql://root:123456@172.16.6.35:35002/" --changefeed-id="mysql-task-1" ;tiup ctl:v5.3.0 cdc changefeed create --pd=http://172.16.4.158:2379 --sink-uri='kafka://172.16.6.24:9092/cdc-events?' --changefeed-id="kafka-task-1"

• Run workload:go-tpc tpcc --db workload --warehouses 20 -T 20 --host 172.16.4.158 --port 4000 --parts 1 prepare

• kafka topic cdc-events has 6 partitions, max.message.bytes -> 1000012

What did you expect to see?

No response

What did you see instead?

"id": "kafka-task-1",
"summary": {
"state": "error",
"tso": 428963091419234305,
"checkpoint": "2021-11-08 17:38:51.891",
"error": {
"addr": "172.16.4.158:8300",
"code": "CDC:ErrProcessorUnknown",
"message": "[CDC:ErrKafkaAsyncSendMessage]kafka: Failed to produce message to topic cdc-events: kafka server: Message was too large, server rejected it to avoid allocation error.: kafka: Failed to produce message to topic cdc-events: kafka server: Message was too large, server rejected it to avoid allocation error."
}
}
}
]

• kafka changefeed error and won't sync data.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

elease Version: v5.3.0
Edition: Community
Git Commit Hash: e401a0606bf84214bd8e765295ec6cfadd9ba24e
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-08 04:40:23
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false |```

TiCDC version (execute `cdc version`):

./cdc version
Release Version: v5.3.0
Git Commit Hash: 7dc044e
Git Branch: heads/refs/tags/v5.3.0
UTC Build Time: 2021-11-08 04:37:11
Go Version: go version go1.16.4 linux/amd64
Failpoint Build: false```

[3AceShowHand]

the problem cannot be reproduced.

[Tammyxia]

Event cdc new created topic: tiup ctl:v5.3.0 cdc changefeed create --pd=http://172.16.4.158:2379 --sink-uri='kafka://172.16.6.24:9092/cdc-test-a?' --changefeed-id="kafka-task-1"
Still this changefeed cannot work:
{
"id": "kafka-task-1",
"summary": {
"state": "stopped",
"tso": 429001921801224213,
"checkpoint": "2021-11-10 10:47:38.043",
"error": {
"addr": "172.16.4.158:8300",
"code": "CDC:ErrProcessorUnknown",
"message": "[CDC:ErrKafkaAsyncSendMessage]kafka: Failed to produce message to topic cdc-test-a: kafka server: Message was too large, server rejected it to avoid allocation error.: kafka: Failed to produce message to topic cdc-test-a: kafka server: Message was too large, server rejected it to avoid allocation error."
}