ddl: introduce a new system variable to control the store-write-bwlimit when ingesting (#57145)

[ti-chi-bot]

This is an automated cherry-pick of #57145

What problem does this PR solve?

Issue Number: close #57156

Problem Summary:

See the issue description

What changed and how does it work?

Introduce a new system variable to control the store-write-bwlimit when ingesting

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manual test:

1. bootstrap a cluster
2. prepare 1M rows data
3. Turn on tidb_ddl_enable_fast_reorg, and with tidb_enable_dist_task on or off, do the following test:
   1. set the tidb_ddl_reorg_max_write_speed to 0
   2. add an index for a table, the speed is fast
   3. after setting the tidb_ddl_reorg_max_write_speed to a lower value, the speed of adding index decreases

TiDB root@127.0.0.1:test> select @@global.tidb_enable_dist_task
+--------------------------------+
| @@global.tidb_enable_dist_task |
+--------------------------------+
| 0                              |
+--------------------------------+
1 row in set
Time: 0.004s
TiDB root@127.0.0.1:test> select @@global.tidb_ddl_reorg_max_write_speed
+-----------------------------------------+
| @@global.tidb_ddl_reorg_max_write_speed |
+-----------------------------------------+
| 0                                       |
+-----------------------------------------+
1 row in set
Time: 0.006s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.285s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 3.171s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.031s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.370s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 9.628s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.016s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.286s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 18.394s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.011s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
^[[AQuery OK, 0 rows affected
Time: 0.294s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 35.796s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.011s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.272s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 71.730s
TiDB root@127.0.0.1:test> set global tidb_enable_dist_task = 1;
Query OK, 0 rows affected
Time: 0.016s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '0';
Query OK, 0 rows affected
Time: 0.016s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.281s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 8.926s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.016s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.286s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 20.346s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.017s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.277s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 38.910s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.012s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.292s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 75.169s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.007s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.320s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 142.759s

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add a system variable tidb_ddl_reorg_max_write_speed to limit the speed of ingesting when adding index
增加系统变量 tidb_ddl_reorg_max_write_speed 来限制加索引时 ingesting 的速度上限

[CbcWestwolf]

/retest

[codecov]

Codecov Report

Attention: Patch coverage is 66.66667% with 8 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-7.5@9d5e282). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-7.5     #57376   +/-   ##
================================================
  Coverage               ?   73.0823%           
================================================
  Files                  ?       1415           
  Lines                  ?     420100           
  Branches               ?          0           
================================================
  Hits                   ?     307019           
  Misses                 ?      93342           
  Partials               ?      19739           

Flag	Coverage Δ
unit	73.0823% <66.6666%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9400% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	54.5877% <0.0000%> (?)

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-12-10 06:17:38.767165866 +0000 UTC m=+332848.855968412: ☑️ agreed by CbcWestwolf.
• 2024-12-10 07:48:40.249023271 +0000 UTC m=+338310.337825814: ☑️ agreed by tangenta.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CbcWestwolf, tangenta, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CbcWestwolf,tangenta]
• pkg/ddl/OWNERS [tangenta]
• pkg/sessionctx/variable/OWNERS [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[CbcWestwolf]

/retest

[CbcWestwolf]

/retest

[CbcWestwolf]

/retest