restore: add importing progress and optimize the accuracy of restore progress #506

glorv · 2020-12-03T04:04:13Z

What problem does this PR solve?

Better progress calculation and log
resolves partially #440

What is changed and how it works?

Remove the progress of file and add progress of engine(count for import). After all files restore finished, the state will be 'importing' instead of 'post-progress'. And after all engines import finished, the state will be 'post-progress'.
Add engine count metrics at estimating chunks count
Use the total pending chunks count instead of estimated chunks if pending chunk count > estimated chunk count to gain better accuracy. This can avoid the chunks progress exceeding 100%
Add total progress percentage. The value of total progress is estimatedWriteProgress * 0.8 + estimatedImportProgress * 0.2
Calculate chunk count based on chunk start offset to be consistent with the chunk count estimation logic.
Flush data and index engine and save all chunk checkpoints so that if restore engine meets context cancel all chunk checkpoints are saved.

New log output style:

[2020/12/07 00:59:44.725 +08:00] [INFO] [restore.go:694] [progress] [total=57.2%] [tables="7/9 (77.8%)"] [chunks="2642/4247 (62.2%)"] [engines="17/27 (63.0%)"] [speed(MiB/s)=131.06370869464234] [state=writing] [remaining=52m21s]
[2020/12/07 01:09:44.725 +08:00] [INFO] [restore.go:694] [progress] [total=65.0%] [tables="7/9 (77.8%)"] [chunks="3020/4247 (71.1%)"] [engines="18/27 (66.7%)"] [speed(MiB/s)=134.3286817015894] [state=writing] [remaining=43m10s]
[2020/12/07 01:59:44.725 +08:00] [INFO] [restore.go:694] [progress] [total=97.4%] [tables="7/9 (77.8%)"] [chunks="4601/4601 (100.0%)"] [engines="25/27 (92.6%)"] [speed(MiB/s)=132.1332473121803] [state=importing] [remaining=3m30s]

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Related changes

july2993 · 2020-12-11T15:38:36Z

count you also add the estimated progress in /progress/task and /progress/table

glorv · 2020-12-14T09:39:59Z

count you also add the estimated progress in /progress/task and /progress/table

Lightning doesn't support return the progress for any individual table. Please submit a feature request issue.

lance6716 · 2021-01-04T08:34:31Z

lightning/backend/local.go

@@ -486,6 +487,11 @@ func (local *local) getImportClient(ctx context.Context, peer *metapb.Peer) (sst
 	return sst.NewImportSSTClient(conn), nil
 }

+type rangeStats struct {
+	count      int64


seems this member is not used

yes, currently only uses the totalBytes

lightning/restore/restore.go

lance6716 · 2021-01-04T08:58:20Z

lightning/restore/restore.go

 		}
-		metric.ChunkCounter.WithLabelValues(metric.ChunkStatePending).Inc()
+		remainChunkCnt := float64(chunk.Chunk.EndOffset-chunk.Chunk.Offset) / float64(chunk.Chunk.EndOffset-chunk.Key.Offset)


What's the meaning of chunk.Key.Offset, when does it not equal chunk.Chunk.Offset. 🤔

And hope there is no division by zero error

Key.Offset is the start offset, chunk.offset is the current read position

lightning/restore/restore.go

…htning into import-progress

lance6716 · 2021-01-06T08:11:09Z

LGTM

CI fail since TiDB keep the format d: 18446744073709551616.0 back

…to import-progress

kennytm

rest LGTM

lightning/restore/restore.go

kennytm · 2021-01-27T07:34:03Z

/lgtm

glorv · 2021-01-27T07:37:51Z

/merge

ti-srebot · 2021-01-27T07:37:53Z

@glorv Oops! This PR requires at least 2 LGTMs to merge. The current number of LGTM is 1

lance6716 · 2021-01-27T07:39:33Z

/merge

ti-srebot · 2021-01-27T07:39:35Z

@lance6716 Oops! This PR requires at least 2 LGTMs to merge. The current number of LGTM is 1

lance6716 · 2021-01-27T07:39:43Z

/lgtm

lance6716 · 2021-01-27T07:39:58Z

/merge

ti-srebot · 2021-01-27T07:40:26Z

/run-all-tests

…progress (#506)

add importing progress and more accurate restore progress

b4c2e65

glorv force-pushed the import-progress branch from c140cc7 to b4c2e65 Compare December 3, 2020 04:04

fix

a937eb7

glorv requested review from kennytm, lance6716 and Little-Wallace December 3, 2020 06:20

add total progress

c2f60f6

glorv force-pushed the import-progress branch from f5a8086 to c2f60f6 Compare December 7, 2020 02:14

Merge branch 'master' into import-progress

6403c98

lance6716 reviewed Jan 4, 2021

View reviewed changes

glorv added 2 commits January 6, 2021 13:58

resolve comments

65e0ac2

Merge branch 'import-progress' of https://github.com/pingcap/tidb-lig…

ab68717

…htning into import-progress

Merge branch 'master' of https://github.com/pingcap/tidb-lightning in…

078d4ea

…to import-progress

lance6716 added the status/LGT1 One reviewer already commented LGTM (LGTM1) label Jan 11, 2021

kennytm reviewed Jan 26, 2021

View reviewed changes

lightning/restore/restore.go Outdated Show resolved Hide resolved

replace TODO with Background context

aa6e750

glorv added status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) and removed status/LGT1 One reviewer already commented LGTM (LGTM1) labels Jan 27, 2021

Merge branch 'master' into import-progress

82b5b3c

ti-srebot approved these changes Jan 27, 2021

View reviewed changes

ti-srebot added the CanMerge label Jan 27, 2021

ti-srebot merged commit 6c382e2 into master Jan 27, 2021

glorv deleted the import-progress branch January 27, 2021 08:30

glorv added a commit that referenced this pull request Jan 29, 2021

restore: add importing progress and optimize the accuracy of restore …

682ddac

…progress (#506)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restore: add importing progress and optimize the accuracy of restore progress #506

restore: add importing progress and optimize the accuracy of restore progress #506

glorv commented Dec 3, 2020 •

edited

Loading

july2993 commented Dec 11, 2020

glorv commented Dec 14, 2020

lance6716 Jan 4, 2021

glorv Jan 6, 2021

lance6716 Jan 4, 2021 •

edited

Loading

glorv Jan 6, 2021

lance6716 commented Jan 6, 2021

kennytm left a comment

kennytm commented Jan 27, 2021

glorv commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

lance6716 commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

lance6716 commented Jan 27, 2021

lance6716 commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

restore: add importing progress and optimize the accuracy of restore progress #506

restore: add importing progress and optimize the accuracy of restore progress #506

Conversation

glorv commented Dec 3, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

july2993 commented Dec 11, 2020

glorv commented Dec 14, 2020

lance6716 Jan 4, 2021

Choose a reason for hiding this comment

glorv Jan 6, 2021

Choose a reason for hiding this comment

lance6716 Jan 4, 2021 • edited Loading

Choose a reason for hiding this comment

glorv Jan 6, 2021

Choose a reason for hiding this comment

lance6716 commented Jan 6, 2021

kennytm left a comment

Choose a reason for hiding this comment

kennytm commented Jan 27, 2021

glorv commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

lance6716 commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

lance6716 commented Jan 27, 2021

lance6716 commented Jan 27, 2021

ti-srebot commented Jan 27, 2021

glorv commented Dec 3, 2020 •

edited

Loading

lance6716 Jan 4, 2021 •

edited

Loading