This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
Don't log secret of s3. (#292) #293
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Merged
Codecov Report
@@ Coverage Diff @@
## release-4.0 #293 +/- ##
===============================================
+ Coverage 71.32% 71.55% +0.22%
===============================================
Files 48 48
Lines 4997 5009 +12
===============================================
+ Hits 3564 3584 +20
+ Misses 973 965 -8
Partials 460 460
Continue to review full report at Codecov.
|
YuJuncen
added a commit
to YuJuncen/br
that referenced
this pull request
May 22, 2020
* restore: fix alter auto increment id for no-primary-key table (pingcap#139) * restore: fix alter auto increment id for no-primary-key table Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add integration test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: use oracle package to manipulate ts and test gc safe point (pingcap#121) Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: convert version command to flags (pingcap#144) * cmd: convert version command to flags Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add changelog and github templates (pingcap#143) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge tidb-tools/pkg/restore-util (pingcap#146) * restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (pingcap#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (pingcap#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (pingcap#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (pingcap#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (pingcap#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (pingcap#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (pingcap#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (pingcap#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (pingcap#101) * backup: add raw backup command * restore: speed up retry on not leader (pingcap#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (pingcap#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (pingcap#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (pingcap#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (pingcap#176) * *: add license header (pingcap#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (pingcap#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (pingcap#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (pingcap#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (pingcap#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (pingcap#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (pingcap#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (pingcap#174) * Fix summary log (pingcap#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (pingcap#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * update dep * restore: remove tiflash replica before restore (pingcap#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (pingcap#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (pingcap#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <ArGregoryIan@gmail.com> * *: refline logs (pingcap#189) * adapt pd/tidb related 3.1 changes * tests: disable TLS test (pingcap#204) Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add S3 quick start and few enhancement of log (pingcap#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <overvenus@gmail.com> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <overvenus@gmail.com> * Update pkg/task/common.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: cat log if br fails Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * restore: add error field to `DownloadResponse` (pingcap#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore: populate restore cancel error (pingcap#207) Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * enhance usability of br (pingcap#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <kennytm@gmail.com> * task: do not run checksum if restore failed (pingcap#209) * fix incremental bug in llroad test (pingcap#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (pingcap#211) * Revert "tests: disable TLS test (pingcap#204)" (pingcap#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (pingcap#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <overvenus@gmail.com> * move waiting reject stores in import file (pingcap#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix ci Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Max index length (pingcap#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (pingcap#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <luancheng@pingcap.com> * update deps Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 庄天翼 <zty0826@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
kennytm
added a commit
to kennytm/br
that referenced
this pull request
May 27, 2020
* restore: fix alter auto increment id for no-primary-key table (pingcap#139) * restore: fix alter auto increment id for no-primary-key table Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add integration test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: use oracle package to manipulate ts and test gc safe point (pingcap#121) Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: convert version command to flags (pingcap#144) * cmd: convert version command to flags Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add changelog and github templates (pingcap#143) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge tidb-tools/pkg/restore-util (pingcap#146) * restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (pingcap#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (pingcap#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (pingcap#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@kolbekegel.com>
YuJuncen
added a commit
to YuJuncen/br
that referenced
this pull request
Jun 16, 2020
* restore: merge tidb-tools/pkg/restore-util (pingcap#146) * restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (pingcap#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (pingcap#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (pingcap#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (pingcap#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (pingcap#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (pingcap#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (pingcap#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (pingcap#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (pingcap#101) * backup: add raw backup command * restore: speed up retry on not leader (pingcap#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (pingcap#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (pingcap#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (pingcap#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (pingcap#176) * *: add license header (pingcap#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (pingcap#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (pingcap#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (pingcap#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (pingcap#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (pingcap#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (pingcap#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (pingcap#174) * Fix summary log (pingcap#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (pingcap#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * restore: remove tiflash replica before restore (pingcap#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (pingcap#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (pingcap#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <ArGregoryIan@gmail.com> * *: refline logs (pingcap#189) * tests: disable TLS test (pingcap#204) Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add S3 quick start and few enhancement of log (pingcap#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <overvenus@gmail.com> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <overvenus@gmail.com> * Update pkg/task/common.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: cat log if br fails Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * restore: add error field to `DownloadResponse` (pingcap#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore: populate restore cancel error (pingcap#207) Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * enhance usability of br (pingcap#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <kennytm@gmail.com> * task: do not run checksum if restore failed (pingcap#209) * fix incremental bug in llroad test (pingcap#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (pingcap#211) * Revert "tests: disable TLS test (pingcap#204)" (pingcap#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (pingcap#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <overvenus@gmail.com> * move waiting reject stores in import file (pingcap#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix ci Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Max index length (pingcap#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (pingcap#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> Co-authored-by: 庄天翼 <zty0826@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
3pointer
added a commit
to 3pointer/br
that referenced
this pull request
Jul 15, 2020
* restore: merge tidb-tools/pkg/restore-util (pingcap#146) * restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> * Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (pingcap#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (pingcap#150) Co-authored-by: kennytm <kennytm@gmail.com> * restore: enhance error handling (pingcap#152) * restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * Incremental BR: support DDL (pingcap#155) * support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Reduce TiDB dependencies (pingcap#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com> * go.mod: update tidb (pingcap#168) Signed-off-by: Neil Shen <overvenus@gmail.com> * BR support TLS (pingcap#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (pingcap#171) Signed-off-by: Neil Shen <overvenus@gmail.com> * backup: add raw backup command (pingcap#101) * backup: add raw backup command * restore: speed up retry on not leader (pingcap#179) * tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * conn, restore: paginate scan regions (pingcap#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com> * Batch restore (pingcap#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com> * README, docker: add quick start (pingcap#181) * README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com> * *: update tidb dependency build with go1.14 (pingcap#176) * *: add license header (pingcap#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: support not shutting down the storage when closing the connection (pingcap#185) Co-authored-by: 3pointer <luancheng@pingcap.com> * conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * fix check safepoint & unhide experimental features (pingcap#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> * support backupts (pingcap#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (pingcap#184) Co-authored-by: 3pointer <luancheng@pingcap.com> * restore: support online restore (pingcap#114) Signed-off-by: disksing <i@disksing.com> * metrics: add grafana scripts (pingcap#140) * add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com> * filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (pingcap#174) * Fix summary log (pingcap#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com> * Implement Raw Restore (pingcap#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com> * restore: remove tiflash replica before restore (pingcap#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com> * summary: put summary log at last (pingcap#197) * summary: put summary log at last * fix switch sql Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Kolbe Kegel <kolbe@pingcap.com> Co-authored-by: WangXiangUSTC <wx347249478@gmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
overvenus
added a commit
to overvenus/br-1
that referenced
this pull request
Feb 1, 2021
* restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cherry-pick #292 to release-4.0
What problem does this PR solve?
fix bug hunting #71
What is changed and how it works?
I change the line that logging the secret information:
br/pkg/backup/client.go
Line 763 in 00fe08c
Check List
Tests
Code changes
LogArgumentsfromutilstotask).Release Note