Commits on Jun 16, 2020

1. Pipelined restore. (pingcap#266) …

   * restore: add pipelined CreateTable.
   
   * restore: add pipelined ValidateFileRanges.
   
   * restore: pipelining restore process.
   
   * restore, task: use batching when pipelining.
   
   * restore: batcher split by range(instead of table).
   
   * restore,task: new way to for polling errCh.
   
   We use select instead of for range, so we can send error when context cancelled.
   
   * restore, task: pipelining checksum.
   
   * restore, task: cancel parallel DDL request.
   
   * restore: restore will now send batch periodly.
   
   * restore: refactor batcher.
   
   * restore: add tests on batcher.
   
   * restore, task: make linter happy.
   
   * *: add dep to multierr.
   
   * task: adjust to new function sig.
   
   * task, restore: close updateCh until all task finish.
   
   * task, restore: pipelined restore supports parition.
   
   * backup: always wait worker to finish.
   
   * backup, task: skip checksum when needed.
   
   * *: make linter happy.
   
   * restore: move batcher test to restore_test package.
   
   * Apply suggestions from code review
   
   Co-authored-by: kennytm <kennytm@gmail.com>
   
   * restore, task: remove context on struct types.
   
   * restore: batcher auto commit can be disabled now.
   
   * restore, task: fix typos.
   
   * recover: fix a bug about removing tiflash.
   
   * restore: MapTableToFiles issues Error log when key range not match.
   
   * *: merge master.
   
   * restore: fix test to match new change of master.
   
   * Apply suggestions from code review
   
   * restore: merge two progresses.
   
   * restore: fix a bug.
   
   that is, when table is too big or batch size is too low,
   we will fail to restore the head part of this table.
   
   * restore: extract batcher to another file
   
   * task: don't return imediately when files is empty.
   
   * restore,task: do some refactor
   
   We move `splitPrepareWork` into a struct named `ContextManager`,
   so that we can batchly set placement rules on online restore.
   
   * restore: fix a shaming bug... :|
   
   * task,restore: panic on file broken
   
   * restore: record tiflash count to disk when removed
   
   * restore,task: simplify some code,
   
   * task,restore: fix a bug.
   
   The bug causes,
   when a singal table is splt into multi part of batches,
   it sometimes fail to checksum.
   
   * restore: some factory and fix
   
   1. make the batcher worker has two send style
   2. make functions for debuging tables and ranges
   3. rewrite a test case to adapt the new batcher
   
   * tests: try to fix CI
   
   * tests: try to fix CI, again
   
   * Apply suggestions from code review
   
   Co-authored-by: 3pointer <qdlc2010@gmail.com>
   
   * restore: change some log levels
   
   * restore: merge joiner of sendWorker into messagebox
   
   ... and, some small changes:
   - don't send sending request if here is one.
   - the method of how a batcher is send move to log level debug
   
   * restore,task: run RemoveRestoreLabels at restore post work
   
   * task: adapt the remove-tiflash flag
   
   * restore,task: fetch new placement rules each time
   
   * Apply suggestions from code review
   
   Co-authored-by: kennytm <kennytm@gmail.com>
   
   * restore,task: run Leave always, and modify some log level
   
   * restore: fix a bug that may cause checksum time incorrect
   
   * restore: don't Leave if never Enter
   
   Co-authored-by: kennytm <kennytm@gmail.com>
   Co-authored-by: 3pointer <qdlc2010@gmail.com>
   Co-authored-by: 3pointer <luancheng@pingcap.com>

   

   4 people authored and Hillium committed Jun 16, 2020
   Configuration menu

   • View commit details
   • Copy full SHA for cfa7527
   • Browse repository at this point

   Copy the full SHA

   cfa7527 View commit details

   Browse the repository at this point in the history

2. Merge branch 'release-4.0' into release-4.0

   

   3pointer authored Jun 16, 2020
   Configuration menu

   • View commit details
   • Copy full SHA for a9be1fd
   • Browse repository at this point

   Copy the full SHA

   a9be1fd View commit details

   Browse the repository at this point in the history