Owner OOM when upstream TiDB has too many tables

[liuzix]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

• Prepare a TiDB cluster with 40k+ tables, create 8 changefeeds, each of which only replicates a few tables (< 10).

2. What did you expect to see?

• That replication goes smoothly, and that memory consumption is not affected by the number of unreplicated tables.

3. What did you see instead?

• Owner runs out of memory.

4. Versions of the cluster

   • TiCDC version (execute cdc version):

     5.0.0-rc
     

[amyangfei]

Design document (internal only): Design: TiCDC Optimization for Large Number of Unreplicated Tables

[lonng]

I think this issue should not be marked as severity/critical because support a large number of tables is not our design goal.

This issue should be marked as a feature request.

@IANTHEREAL @amyangfei @overvenus How do you think?

[IANTHEREAL]

How many tables do we define as a large number of tables?

As many tables as tidb can support, we should support as many tables as we can. This can refer to the definition of TiDB; in addition, we need to support incremental data backup in Q2/Q3. By then, this goal will definitely be accomplished.

[overvenus]

For now, OOM only occurs when upstream TiDB 1) has many tables and DDL frequently, it's not the case for most users, change to severity/moderate.

[overvenus]

In #4068, we found that truncating tables also triggers the issue.

Reproduce steps:

mysql -h $HOST -P $PORT -u root -E -e 'create database if not exists truncate500;'

for i in $(seq 1 500); do {
    mysql -h $HOST -P $PORT -u root -E -e "
CREATE TABLE if not exists truncate500.orders_$i (
  o_id int(11) NOT NULL,
  o_d_id int(11) NOT NULL,
  o_w_id int(11) NOT NULL,
  o_c_id int(11) DEFAULT NULL,
  o_entry_d datetime DEFAULT NULL,
  o_carrier_id int(11) DEFAULT NULL,
  o_ol_cnt int(11) DEFAULT NULL,
  o_all_local int(11) DEFAULT NULL,
  PRIMARY KEY (o_w_id,o_d_id,o_id),
  KEY idx_order (o_w_id,o_d_id,o_c_id,o_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;";
} done

for i in $(seq 0 25000); do {
    t=$(( i % 500 ))
    mysql -h $HOST -P $PORT -u root -E -e "TRUNCATE TABLE truncate500.orders_$t;"
} done

[nongfushanquan]

/label affects-5.1

[nongfushanquan]

/unassign @overvenus
/assign @hicqu