fix: add parallelisation to migrate-lid and improve logging

[LexLuthr]

Tested on filcollins:

./migrate-lid --boost-repo ~/boost-test-migrate --vv leveldb --parallel 10 dagstore
Setting up connection and creating data structures... this might take a minute...
Migrating to leveldb Local Index Directory. See detailed logs of the migration at
migrate-leveldb.log
Migrating indices...   0% [                                                                                                                                                                                                                                                                                                                               ] (0/468) [0s:0s]2023-09-11T07:26:32.598-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "2.184337ms"}
2023-09-11T07:26:32.598-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "2.12958ms"}
2023-09-11T07:26:32.599-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "3.414967ms"}
2023-09-11T07:26:32.599-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "3.626186ms"}
2023-09-11T07:26:32.599-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "3.767945ms"}
2023-09-11T07:26:32.600-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "2.302942ms"}
2023-09-11T07:26:32.600-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "2.362373ms"}
2023-09-11T07:26:32.601-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "2.80938ms"}
2023-09-11T07:26:32.604-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "8.288777ms"}
2023-09-11T07:26:32.605-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:385	ReadIndex	{"took": "7.588953ms"}
2023-09-11T07:26:32.610-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:398	ConvertIndex	{"took": "10.152786ms"}
2023-09-11T07:26:32.621-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:398	ConvertIndex	{"took": "20.614302ms"}
2023-09-11T07:26:32.622-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:398	ConvertIndex	{"took": "22.467042ms"}
2023-09-11T07:26:32.623-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:398	ConvertIndex	{"took": "25.195107ms"}
2023-09-11T07:26:32.624-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:398	ConvertIndex	{"took": "23.350412ms"}

Migrating indices...   0% [                                                                                                                                                                                                                                                                                                                            ] (1/468) [0s:1m44s]2023-09-11T07:26:33.446-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "824.429865ms"}
Migrating indices...   0% [                                                                                                                                                                                                                                                                                                                            ] (2/468) [0s:3m18s]2023-09-11T07:26:34.058-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "1.43635134s"}
Migrating indices...   0% [                                                                                                                                                                                                                                                                                                                            ] (3/468) [1s:3m53s]2023-09-11T07:26:34.663-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "2.03988041s"}
Migrating indices...   0% [                                                                                                                                                                                                                                                                                                                             ] (4/468) [2s:4m7s]2023-09-11T07:26:35.394-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "2.769649578s"}
Migrating indices...   1% [==>                                                                                                                                                                                                                                                                                                                         ] (5/468) [2s:4m24s]2023-09-11T07:26:36.075-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "3.450320056s"}
Migrating indices...   1% [==>                                                                                                                                                                                                                                                                                                                         ] (6/468) [3s:4m32s]2023-09-11T07:26:36.773-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "4.148041261s"}
Migrating indices...   1% [==>                                                                                                                                                                                                                                                                                                                         ] (7/468) [4s:4m39s]2023-09-11T07:26:37.618-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "4.993020253s"}
Migrating indices...   1% [==>                                                                                                                                                                                                                                                                                                                         ] (8/468) [5s:4m50s]2023-09-11T07:26:38.385-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "5.759071721s"}
Migrating indices...   1% [==>                                                                                                                                                                                                                                                                                                                         ] (9/468) [5s:4m56s]2023-09-11T07:26:39.408-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:408	AddIndex	{"took": "6.782559442s"}
Migrating indices...   2% [=====>

Resumption works with a higher number of parallelisation:

./migrate-lid --boost-repo ~/boost-test-migrate --vv leveldb --parallel 25 dagstore
Setting up connection and creating data structures... this might take a minute...
Migrating to leveldb Local Index Directory. See detailed logs of the migration at
migrate-leveldb.log
2023-09-11T07:31:19.503-0400    DEBUG   migrate migrate-lid/migrate_lid.go:385  ReadIndex       {"took": "1.024511ms"}                                                                                                                                                                                                                                                     
Migrating indices...  18% [========================================================>

[LexLuthr]

We had some timeouts in 60 seconds as expected. Rest processed.

2023-09-11T07:45:23.436-0400 INFO migrate-lid/migrate_lid.go:325 migrated indices {"total": 468, "took": "14m3.951975083s"}

[dirkmc]

LGTM 👍

[LexLuthr]

Tested after refactor:

./migrate-lid --boost-repo ~/boost-test-migrate --vv leveldb --parallel 25 dagstore
Setting up connection and creating data structures... this might take a minute...
Migrating to leveldb Local Index Directory. See detailed logs of the migration at
migrate-leveldb.log
2023-09-11T09:41:32.357-0400    DEBUG   migrate migrate-lid/migrate_lid.go:374  ReadIndex       {"took": "969.026µs"}

Migrating indices...  99% [=====================================================================================================================================================================================================================================================================================================================>    ] (466/468) [2m46s:7s]2023-09-11T09:44:20.213-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:397	AddIndex	{"took": "1m20.809231537s"}
Migrating indices...  99% [=====================================================================================================================================================================================================================================================================================================================>    ] (467/468) [2m48s:3s]2023-09-11T09:44:23.219-0400	DEBUG	migrate	migrate-lid/migrate_lid.go:397	AddIndex	{"took": "1m23.815433702s"}
Migrating indices... 100% [==========================================================================================================================================================================================================================================================================================================================] (468/468)           
Warning: there were errors migrating 48 indices. See the log for details:
migrate-leveldb.log