Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Pipelined restore. #266

Merged
merged 69 commits into from
Jun 15, 2020
Merged

Pipelined restore. #266

merged 69 commits into from
Jun 15, 2020

Commits on Apr 28, 2020

  1. restore: add pipelined CreateTable.

    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    0b07b64 View commit details
    Browse the repository at this point in the history

  2. restore: add pipelined ValidateFileRanges.

    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    061b669 View commit details
    Browse the repository at this point in the history

  3. restore: pipelining restore process.

    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    d711ded View commit details
    Browse the repository at this point in the history

  4. restore, task: use batching when pipelining.

    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    f258d99 View commit details
    Browse the repository at this point in the history

  5. restore: batcher split by range(instead of table).

    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    cefb696 View commit details
    Browse the repository at this point in the history

  6. restore,task: new way to for polling errCh. 

    We use select instead of for range, so we can send error when context cancelled.
    Hillium committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    7846324 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2020

  1. restore, task: pipelining checksum.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    c78a24b View commit details
    Browse the repository at this point in the history

  2. restore, task: cancel parallel DDL request.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    93b5942 View commit details
    Browse the repository at this point in the history

  3. restore: restore will now send batch periodly.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    f3ec5ee View commit details
    Browse the repository at this point in the history

  4. restore: refactor batcher.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    bb31ec0 View commit details
    Browse the repository at this point in the history

  5. restore: add tests on batcher.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    d54b99c View commit details
    Browse the repository at this point in the history

  6. restore, task: make linter happy.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    4c4a3d8 View commit details
    Browse the repository at this point in the history

  7. *: add dep to multierr.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    3cffa5d View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' of https://github.com/pingcap/br into pipelined… 

    …-restore
    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    965779a View commit details
    Browse the repository at this point in the history

  9. task: adjust to new function sig.

    Hillium committed May 8, 2020
    Configuration menu
    Copy the full SHA
    59e33b0 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2020

  1. task, restore: close updateCh until all task finish.

    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    44d52be View commit details
    Browse the repository at this point in the history

  2. task, restore: pipelined restore supports parition.

    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    a49e1f0 View commit details
    Browse the repository at this point in the history

  3. backup: always wait worker to finish.

    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    84e267c View commit details
    Browse the repository at this point in the history

  4. backup, task: skip checksum when needed.

    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    7fab3c3 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' of https://github.com/pingcap/br into pipelined… 

    …-restore
    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    39d2312 View commit details
    Browse the repository at this point in the history

  6. *: make linter happy.

    Hillium committed May 9, 2020
    Configuration menu
    Copy the full SHA
    ac6f5be View commit details
    Browse the repository at this point in the history

Commits on May 11, 2020

  1. restore: move batcher test to restore_test package.

    Hillium committed May 11, 2020
    Configuration menu
    Copy the full SHA
    25a9a3a View commit details
    Browse the repository at this point in the history

Commits on May 12, 2020

  1. Apply suggestions from code review 

    Co-authored-by: kennytm <kennytm@gmail.com>
    @YuJuncen @kennytm
    YuJuncen and kennytm authored May 12, 2020
    Configuration menu
    Copy the full SHA
    7f8251e View commit details
    Browse the repository at this point in the history

  2. restore, task: remove context on struct types.

    Hillium committed May 12, 2020
    Configuration menu
    Copy the full SHA
    4bf507e View commit details
    Browse the repository at this point in the history

  3. restore: batcher auto commit can be disabled now.

    Hillium committed May 12, 2020
    Configuration menu
    Copy the full SHA
    c45c772 View commit details
    Browse the repository at this point in the history

  4. restore, task: fix typos.

    Hillium committed May 12, 2020
    Configuration menu
    Copy the full SHA
    afea125 View commit details
    Browse the repository at this point in the history

  5. recover: fix a bug about removing tiflash.

    Hillium committed May 12, 2020
    Configuration menu
    Copy the full SHA
    3c7530c View commit details
    Browse the repository at this point in the history

Commits on May 13, 2020

  1. restore: MapTableToFiles issues Error log when key range not match.

    Hillium committed May 13, 2020
    Configuration menu
    Copy the full SHA
    e164c4f View commit details
    Browse the repository at this point in the history

  2. *: merge master.

    Hillium committed May 13, 2020
    Configuration menu
    Copy the full SHA
    2af022a View commit details
    Browse the repository at this point in the history

  3. restore: fix test to match new change of master.

    Hillium committed May 13, 2020
    Configuration menu
    Copy the full SHA
    2bc34d9 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored May 13, 2020
    Configuration menu
    Copy the full SHA
    cee0fc3 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2020

  1. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored May 14, 2020
    Configuration menu
    Copy the full SHA
    acbb2fa View commit details
    Browse the repository at this point in the history

Commits on May 17, 2020

  1. Apply suggestions from code review

    @YuJuncen
    YuJuncen authored May 17, 2020
    Configuration menu
    Copy the full SHA
    b96de5b View commit details
    Browse the repository at this point in the history

Commits on May 18, 2020

  1. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored May 18, 2020
    Configuration menu
    Copy the full SHA
    8c9bf99 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2020

  1. restore: merge two progresses.

    Hillium committed May 22, 2020
    Configuration menu
    Copy the full SHA
    deb8848 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'pipelined-restore' of https://github.com/YuJuncen/br in… 

    …to pipelined-restore
    Hillium committed May 22, 2020
    Configuration menu
    Copy the full SHA
    879b553 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored May 22, 2020
    Configuration menu
    Copy the full SHA
    30d22b4 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2020

  1. restore: fix a bug. 

    that is, when table is too big or batch size is too low,
we will fail to restore the head part of this table.
    Hillium committed May 25, 2020
    Configuration menu
    Copy the full SHA
    4060eec View commit details
    Browse the repository at this point in the history

  2. Merge branch 'pipelined-restore' of https://github.com/YuJuncen/br in… 

    …to pipelined-restore
    Hillium committed May 25, 2020
    Configuration menu
    Copy the full SHA
    1027d28 View commit details
    Browse the repository at this point in the history

  3. restore: extract batcher to another file

    Hillium committed May 25, 2020
    Configuration menu
    Copy the full SHA
    c1d2064 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2020

  1. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored May 29, 2020
    Configuration menu
    Copy the full SHA
    557e4e9 View commit details
    Browse the repository at this point in the history

  2. task: don't return imediately when files is empty.

    yujuncen committed May 29, 2020
    Configuration menu
    Copy the full SHA
    b0dd355 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2020

  1. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored Jun 2, 2020
    Configuration menu
    Copy the full SHA
    7bca7ad View commit details
    Browse the repository at this point in the history

  2. restore,task: do some refactor 

    We move `splitPrepareWork` into a struct named `ContextManager`,
so that we can batchly set placement rules on online restore.
    yujuncen committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    b53526e View commit details
    Browse the repository at this point in the history

  3. restore: fix a shaming bug... :|

    yujuncen committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    dd6af60 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2020

  1. Merge branch 'master' of https://github.com/pingcap/br into pipelined… 

    …-restore
    Hillium committed Jun 5, 2020
    Configuration menu
    Copy the full SHA
    f758994 View commit details
    Browse the repository at this point in the history

  2. task,restore: panic on file broken

    yujuncen committed Jun 5, 2020
    Configuration menu
    Copy the full SHA
    3303cef View commit details
    Browse the repository at this point in the history

  3. restore: record tiflash count to disk when removed

    yujuncen committed Jun 5, 2020
    Configuration menu
    Copy the full SHA
    a31e44c View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2020

  1. restore,task: simplify some code,

    yujuncen committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    88c8117 View commit details
    Browse the repository at this point in the history

  2. task,restore: fix a bug. 

    The bug causes,
when a singal table is splt into multi part of batches,
it sometimes fail to checksum.
    yujuncen committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    cc43d9d View commit details
    Browse the repository at this point in the history

  3. restore: some factory and fix 

    1. make the batcher worker has two send style
2. make functions for debuging tables and ranges
3. rewrite a test case to adapt the new batcher
    yujuncen committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    312039d View commit details
    Browse the repository at this point in the history

  4. tests: try to fix CI

    yujuncen committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    bbdbecd View commit details
    Browse the repository at this point in the history

  5. tests: try to fix CI, again

    yujuncen committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    2c269dc View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored Jun 9, 2020
    Configuration menu
    Copy the full SHA
    22cf64c View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2020

  1. Merge branch 'master' of https://github.com/pingcap/br into pipelined… 

    …-restore
    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    4705a55 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: 3pointer <qdlc2010@gmail.com>
    @YuJuncen @3pointer
    YuJuncen and 3pointer authored Jun 12, 2020
    Configuration menu
    Copy the full SHA
    55d22e4 View commit details
    Browse the repository at this point in the history

  3. restore: change some log levels

    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    c94435d View commit details
    Browse the repository at this point in the history

  4. restore: merge joiner of sendWorker into messagebox 

    ... and, some small changes:
- don't send sending request if here is one.
- the method of how a batcher is send move to log level debug
    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    fe59bc1 View commit details
    Browse the repository at this point in the history

  5. restore,task: run RemoveRestoreLabels at restore post work

    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    4d67e54 View commit details
    Browse the repository at this point in the history

  6. task: adapt the remove-tiflash flag

    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    dc1d293 View commit details
    Browse the repository at this point in the history

  7. restore,task: fetch new placement rules each time

    yujuncen committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    5ac8cfe View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2020

  1. Apply suggestions from code review 

    Co-authored-by: kennytm <kennytm@gmail.com>
    @YuJuncen @kennytm
    YuJuncen and kennytm authored Jun 15, 2020
    Configuration menu
    Copy the full SHA
    4cbbff0 View commit details
    Browse the repository at this point in the history

  2. restore,task: run Leave always, and modify some log level

    yujuncen committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    460331f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into pipelined-restore

    @YuJuncen
    YuJuncen authored Jun 15, 2020
    Configuration menu
    Copy the full SHA
    b450532 View commit details
    Browse the repository at this point in the history

  4. restore: fix a bug that may cause checksum time incorrect

    yujuncen committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    0ee5223 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'pipelined-restore' of https://github.com/Yujuncen/br in… 

    …to pipelined-restore
    yujuncen committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    7437950 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into pipelined-restore

    @3pointer
    3pointer authored Jun 15, 2020
    Configuration menu
    Copy the full SHA
    8f96c30 View commit details
    Browse the repository at this point in the history

  7. restore: don't Leave if never Enter

    yujuncen committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    77ab77f View commit details
    Browse the repository at this point in the history

  8. Merge branch 'pipelined-restore' of https://github.com/Yujuncen/br in… 

    …to pipelined-restore
    yujuncen committed Jun 15, 2020
    Configuration menu
    Copy the full SHA
    0ecbcd6 View commit details
    Browse the repository at this point in the history