Lightning:The time to import data into Lightning significantly extends with the increase in the number of databases

[shaoxiqian]

lightning version: nightly
cluster version:nightly
cluster topo: 1 tidb (16c48g) + 6 tikv (24c48g + 1500G)

lightning:toml

[lightning]
level = "info"
check-requirements = false
status-addr = ':8289'
index-concurrency = 64
table-concurrency = 64
io-concurrency = 32
region-concurrency = 64

[tikv-importer]
backend = "local"
incremental-import = true
sorted-kv-dir = "/tiup/sort"
range-concurrency = 64

[tidb]
# Information of the target cluster
port = 4000
user = "root"
password = ""
host = "tidb-1-peer"
status-port = 10080
pd-addr = "pd-peer:2379"
build-stats-concurrency = 20
distsql-scan-concurrency = 15
index-serial-scan-concurrency = 20
checksum-table-concurrency = 2

[mydumper]
no-schema = true
data-source-dir = 's3://xxx/xxx/50k-195000?access-key=xxx&secret-access-key=xxx&endpoint=http://xxx.com&force-path-style=false&region=Beijing&provider=ks'
[mydumper.csv]
header = false

[checkpoint]
# Whether to enable checkpoints.
enable = true
driver = "file"

[post-restore]
checksum = false
analyze = false

[conflict]
strategy = "replace"

time for Importing data into cluster:

2000 databases take 8-9 minutes.

5000 databases take 1-2.5 hours. ["the whole procedure completed"] [takeTime=2h16m33.498780422s] []
10000 databases take 7-8 hours. ["the whole procedure failed"] [takeTime=7h56m44.007541687s]

Every database has 5 tables and 3 rows peer table.
The time to import data into Lightning significantly extends with the increase in the number of libraries, ranging from just a few minutes for smaller datasets to several hours for larger ones, showcasing a pronounced time amplification effect.

[D3Hunter]

after #42136, table info is part of the checkpoint, in the 5k db case in the issue, the checkpoint file is about 71M, it become the bottleneck as data size of each table is quite small.

we can disable checkpoint or use mysql driver(we only save changed content in this case) as a workaround

put checkpoint file in a faster disk would help a little