restore: make split and restore pipelined. #427

YuJuncen · 2020-07-16T11:24:28Z

Signed-off-by: Hillium maruruku@stu.csust.edu.cn

What problem does this PR solve?

What is changed and how it works?

We changed the interface of BatchSender, so it can run asynchronously:

type TableSink interface {
	EmitTables(tables ...CreatedTable)
	EmitError(error)
	Close()
}

// BatchSender is the abstract of how the batcher send a batch.
type BatchSender interface {
	// PutSink sets the sink of this sender, user to this interface promise
	// call this function at least once before first call to `RestoreBatch`.
	PutSink(sink TableSink)
	// RestoreBatch will send the restore request.
	RestoreBatch(ranges DrainResult)
	Close()
}

Then, we must change the call site of Batcher, and make another goroutine(known as contextCleaner) to accept tables that restored asynchronously, do the clean job, and send tables to next.

(TODO?: since the sender can be internally asynchronous, we can remove sendWorker and send requests directly at the call site. However the risk of this is that the channel size of BatchSender might block the caller, and we must pass a ctx to methods like Add since ContextManager need to be in the tree of structured concurrent too.)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Internal test: real time 12m28s -> 9m5s (45000+ ranges)

Release Note

Speed up restore by pipelining (split & scatter) and import.

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

YuJuncen · 2020-07-16T13:00:15Z

/build

YuJuncen · 2020-07-16T14:00:45Z

/run-integration-test

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

remove reject stores sometime always grow too huge and provides little useful information, move it to DEBUG. add a log when any file is fully restored, which is useful troubleshooting when some restore goes slow. Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

pkg/restore/pipeline_items.go

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

pkg/restore/batcher.go

…hen exit Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

YuJuncen · 2020-07-20T13:27:48Z

/run-integration-test

"transport: Error while dialing dial tcp 127.0.0.1:20163: connect: connection refused"

YuJuncen · 2020-07-21T03:24:00Z

/run-integration-test

YuJuncen · 2020-07-21T03:53:25Z

Seems it's buggy, will it reproduce again?

[2020-07-21T03:31:23.703Z] [2020/07/21 11:31:23.613 +08:00] [INFO] [client.go:178] ["load backupmeta"] [databases=0] [jobs=3]

/run-integration-test

kennytm · 2020-07-22T10:24:54Z

It's green now.

pkg/restore/util.go

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

YuJuncen · 2020-09-18T06:11:44Z

/run-integration-test

…-split

… pipelined-split

YuJuncen · 2020-09-21T07:36:12Z

[2020-09-21T05:41:08.800Z] [2020/09/21 13:41:08.664 +08:00] [INFO] [collector.go:187] ["Full backup Failed summary : total backup ranges: 0, total success: 0, total failed: 0"] [BackupTS=419605212977168385]

/run-integration-test 🤔

YuJuncen · 2020-09-21T08:11:41Z

/run-integration-test

kennytm

LGTM

3pointer · 2020-10-09T09:02:56Z

pkg/restore/import.go

@@ -338,9 +338,9 @@ func (importer *FileImporter) Import(
 					zap.Error(errIngest))
 				return errIngest
 			}
+			summary.CollectSuccessUnit(summary.TotalKV, 1, file.TotalKvs)
+			summary.CollectSuccessUnit(summary.TotalBytes, 1, file.TotalBytes)


How strange... 🤔

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

3pointer · 2020-10-09T09:55:17Z

LGTM

YuJuncen · 2020-10-09T13:52:59Z

/merge

ti-srebot · 2020-10-09T13:53:08Z

/run-all-tests

ti-srebot · 2020-10-09T14:03:25Z

cherry pick to release-4.0 failed

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

*: remove in-struct context (#452) * restore: make split and restore pipelined. (#427) *: when cleaning up, use isolated context (#559) * pd: disable location replacement (#555)

Hillium added 2 commits July 16, 2020 18:59

restore: make split and restore pipelined.

bc6224c

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: adapt the unit test.

fb6880c

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

YuJuncen added status/WIP needs-cherry-pick-release-3.1 needs-cherry-pick-release-4.0 labels Jul 16, 2020

Hillium and others added 8 commits July 16, 2020 22:55

restore: disable concurrent split for now

781e634

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: remove concurrency limit, fixed checksum time summary

d63e437

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: handle a error has been omitted past

5f1449a

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: abstract sink type

32197cd

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

Merge branch 'master' into pipelined-split

e350877

restore: add log when file backed up

683dbdf

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: unexport some private fields

9946b3e

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

3pointer reviewed Jul 20, 2020

View reviewed changes

pkg/restore/pipeline_items.go Outdated Show resolved Hide resolved

Hillium added 2 commits July 20, 2020 17:26

restore: grow the restore channel size

7b34f2b

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

restore: grow all channel size

b457a96

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

3pointer reviewed Jul 20, 2020

View reviewed changes

pkg/restore/batcher.go Outdated Show resolved Hide resolved

restore: move some synchronous logic to async, call leave on tables w…

c0fe56c

…hen exit Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

Merge branch 'master' into pipelined-split

e4c9073

YuJuncen removed the status/WIP label Jul 22, 2020

YuJuncen and others added 4 commits July 22, 2020 21:36

Merge branch 'master' into pipelined-split

55fc7f4

restore: fix conflicted merge

6df57bd

Merge branch 'master' into pipelined-split

41095e7

Merge branch 'master' into pipelined-split

2353785

kennytm reviewed Sep 18, 2020

View reviewed changes

pkg/restore/util.go Outdated Show resolved Hide resolved

Hillium and others added 2 commits September 18, 2020 12:31

restore: fix a malformed printf

461f40c

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

Merge branch 'master' into pipelined-split

0f7af92

YuJuncen and others added 4 commits September 18, 2020 18:01

Merge branch 'master' into pipelined-split

a8f0f26

Merge branch 'master' of https://github.com/pingcap/br into pipelined…

7308cba

…-split

Merge branch 'pipelined-split' of https://github.com/YuJuncen/br into…

cd40878

… pipelined-split

restore: pass context to RestoreFiles

038e37d

YuJuncen added 2 commits September 27, 2020 17:46

Merge branch 'master' into pipelined-split

b2c01fc

Merge branch 'master' into pipelined-split

273fca1

kennytm reviewed Sep 27, 2020

View reviewed changes

ti-srebot added the status/LGT1 LGTM1 label Sep 27, 2020

Merge branch 'master' into pipelined-split

c5fe188

3pointer reviewed Oct 9, 2020

View reviewed changes

Hillium and others added 2 commits October 9, 2020 17:42

restore: reset unexpectedly moved to outside

1cb094a

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

Merge branch 'master' into pipelined-split

baa2c7e

ti-srebot removed the status/LGT1 LGTM1 label Oct 9, 2020

ti-srebot approved these changes Oct 9, 2020

View reviewed changes

ti-srebot added the status/LGT2 LGTM2 label Oct 9, 2020

ti-srebot added the CanMerge label Oct 9, 2020

ti-srebot merged commit ed2b143 into pingcap:master Oct 9, 2020

YuJuncen added a commit to YuJuncen/br that referenced this pull request Oct 21, 2020

restore: make split and restore pipelined. (pingcap#427)

fbcefe1

Signed-off-by: Hillium <maruruku@stu.csust.edu.cn>

YuJuncen mentioned this pull request Oct 21, 2020

*: cherry-picking some PRs for v4.0.8 #562

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restore: make split and restore pipelined. #427

restore: make split and restore pipelined. #427

YuJuncen commented Jul 16, 2020 •

edited

Loading

YuJuncen commented Jul 16, 2020

YuJuncen commented Jul 16, 2020

YuJuncen commented Jul 20, 2020 •

edited

Loading

YuJuncen commented Jul 21, 2020

YuJuncen commented Jul 21, 2020

kennytm commented Jul 22, 2020

YuJuncen commented Sep 18, 2020

YuJuncen commented Sep 21, 2020

YuJuncen commented Sep 21, 2020

kennytm left a comment

3pointer Oct 9, 2020

YuJuncen Oct 9, 2020

3pointer commented Oct 9, 2020

YuJuncen commented Oct 9, 2020

ti-srebot commented Oct 9, 2020

ti-srebot commented Oct 9, 2020

restore: make split and restore pipelined. #427

restore: make split and restore pipelined. #427

Conversation

YuJuncen commented Jul 16, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release Note

YuJuncen commented Jul 16, 2020

YuJuncen commented Jul 16, 2020

YuJuncen commented Jul 20, 2020 • edited Loading

YuJuncen commented Jul 21, 2020

YuJuncen commented Jul 21, 2020

kennytm commented Jul 22, 2020

YuJuncen commented Sep 18, 2020

YuJuncen commented Sep 21, 2020

YuJuncen commented Sep 21, 2020

kennytm left a comment

Choose a reason for hiding this comment

3pointer Oct 9, 2020

Choose a reason for hiding this comment

YuJuncen Oct 9, 2020

Choose a reason for hiding this comment

3pointer commented Oct 9, 2020

YuJuncen commented Oct 9, 2020

ti-srebot commented Oct 9, 2020

ti-srebot commented Oct 9, 2020

YuJuncen commented Jul 16, 2020 •

edited

Loading

YuJuncen commented Jul 20, 2020 •

edited

Loading