TTL table may not have change to schedule when the whole TTL schedule is heavy

[lcwangchao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. create 1000+ TTL tables and each table's ttl_job_schedule_interval is set to 1m
2. wait about 30 minutes
3. check whether any TTL table's schedule is delayed for a long time by

select now(),watermark, TIMESTAMPDIFF(minute, watermark, now()) from mysql.tidb_timers where TIMESTAMPDIFF(minute, watermark, now()) > 20 limit 1;

2. What did you expect to see? (Required)

No rows found

3. What did you see instead (Required)

TiDB root@172.16.4.180:(none)> select now(),watermark, TIMESTAMPDIFF(minute, watermark, now()) from mysql.tidb_timers where TIMESTAMPDIFF(minute, watermark, now()) > 20 limit 1;
+---------------------+---------------------+-----------------------------------------+
| now()               | watermark           | TIMESTAMPDIFF(minute, watermark, now()) |
+---------------------+---------------------+-----------------------------------------+
| 2024-11-05 09:56:12 | 2024-11-05 09:33:41 | 22                                      |
+---------------------+---------------------+-----------------------------------------+

You can see, some table's schedule is delayed by about 20+ minutes...

4. What is your TiDB version? (Required)