-
Notifications
You must be signed in to change notification settings - Fork 102
Commits on Jan 22, 2020
-
restore: merge tidb-tools/pkg/restore-util (pingcap#146)
* restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * redesign output/input of the lib Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add commments and more tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add ScanRegions interface to Client Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix potential data race Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <kennytm@gmail.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update dependency Signed-off-by: 5kbpers <tangminghua@pingcap.com> * resolve conflicts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix prefix rewrite Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check if region has peer Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix go.sum Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <overvenus@gmail.com> * Nil check onSplit Co-Authored-By: kennytm <kennytm@gmail.com> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <tangminghua@pingcap.com> * more logs Signed-off-by: 5kbpers <tangminghua@pingcap.com> * *: gofmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <overvenus@gmail.com> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log message Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce error trace Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <i@disksing.com> * minor fix Signed-off-by: disksing <i@disksing.com> * address comment Signed-off-by: disksing <i@disksing.com> * add GetPlacementRules Signed-off-by: disksing <i@disksing.com> * fix test Signed-off-by: disksing <i@disksing.com> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <tangminghua@pingcap.com> * go fmt Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * address commits Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <kennytm@gmail.com> * add onSplit callback Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <i@disksing.com> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * warn table id does not match Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: Neil Shen <overvenus@gmail.com> * fix compile error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ian <ArGregoryIan@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> * *: prune tidb-tools Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: address linters suggestions Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <overvenus@gmail.com> * address comment Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: disksing <i@disksing.com> Co-authored-by: Ian <ArGregoryIan@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 0a43d2a - Browse repository at this point
Copy the full SHA 0a43d2aView commit details
Commits on Feb 1, 2020
-
Fixed handling for a dbName that do not exist in the backup being res…
…tored (pingcap#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored
Kolbe Kegel authoredFeb 1, 2020 Configuration menu - View commit details
-
Copy full SHA for 3863a3a - Browse repository at this point
Copy the full SHA 3863a3aView commit details
Commits on Feb 5, 2020
-
validate: fix debug meta test ci (pingcap#153)
* validate: fix debug meta test ci
Configuration menu - View commit details
-
Copy full SHA for 6b65080 - Browse repository at this point
Copy the full SHA 6b65080View commit details
Commits on Feb 10, 2020
-
*: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156)
* *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta
Configuration menu - View commit details
-
Copy full SHA for 0dfe82d - Browse repository at this point
Copy the full SHA 0dfe82dView commit details
Commits on Feb 13, 2020
-
restore: fix restore summary log (pingcap#150)
Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 8c97452 - Browse repository at this point
Copy the full SHA 8c97452View commit details -
restore: enhance error handling (pingcap#152)
* restore: enhance error handling Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix region epoch error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove `Restore*` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add debug log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Co-Authored-By: kennytm <kennytm@gmail.com> * Update pkg/restore/import.go Co-Authored-By: kennytm <kennytm@gmail.com> * fix retry error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * handle RegionNotFound error Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 008ec45 - Browse repository at this point
Copy the full SHA 008ec45View commit details
Commits on Feb 20, 2020
-
Incremental BR: support DDL (pingcap#155)
* support backup&restore ddl Signed-off-by: 5kbpers <tangminghua@pingcap.com> * integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * update kvproto Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix integration tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add unit test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable fast checksum in incremental br Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix no valid key error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com>
Configuration menu - View commit details
-
Copy full SHA for 1e1fc97 - Browse repository at this point
Copy the full SHA 1e1fc97View commit details
Commits on Feb 22, 2020
-
Reduce TiDB dependencies (pingcap#158)
* utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <qdlc2010@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 4492b8f - Browse repository at this point
Copy the full SHA 4492b8fView commit details
Commits on Feb 23, 2020
-
go.mod: update tidb (pingcap#168)
Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3c9d42f - Browse repository at this point
Copy the full SHA 3c9d42fView commit details
Commits on Feb 27, 2020
-
* *: support tls * move tikv.driver to glue * fix comments
Configuration menu - View commit details
-
Copy full SHA for 028963d - Browse repository at this point
Copy the full SHA 028963dView commit details
Commits on Feb 28, 2020
-
upgrade golangci and prepare for go 1.14 (pingcap#171)
Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for e462f80 - Browse repository at this point
Copy the full SHA e462f80View commit details
Commits on Mar 4, 2020
-
backup: add raw backup command (pingcap#101)
* backup: add raw backup command
Configuration menu - View commit details
-
Copy full SHA for 4657932 - Browse repository at this point
Copy the full SHA 4657932View commit details
Commits on Mar 5, 2020
-
restore: speed up retry on not leader (pingcap#179)
* tests: stable cluster start up Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: fix unbound var Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: speed retry on not leader Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: add --cacert flag Signed-off-by: Neil Shen <overvenus@gmail.com> * make codecov green Signed-off-by: Neil Shen <overvenus@gmail.com> * address comments Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 0a1a044 - Browse repository at this point
Copy the full SHA 0a1a044View commit details -
conn, restore: paginate scan regions (pingcap#165)
* conn, restore: paginate scan regions Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: large timeout Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 09fb715 - Browse repository at this point
Copy the full SHA 09fb715View commit details
Commits on Mar 6, 2020
-
* *: unify Range and RangeTree Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: split restore files into small batch Signed-off-by: Neil Shen <overvenus@gmail.com> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: unused table worker pool Signed-off-by: Neil Shen <overvenus@gmail.com> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <overvenus@gmail.com> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 82b5016 - Browse repository at this point
Copy the full SHA 82b5016View commit details
Commits on Mar 7, 2020
-
README, docker: add quick start (pingcap#181)
* README, docker: add quick start Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: disable some TiDB log Signed-off-by: Neil Shen <overvenus@gmail.com> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <overvenus@gmail.com> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <overvenus@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3fc9ee2 - Browse repository at this point
Copy the full SHA 3fc9ee2View commit details
Commits on Mar 10, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 237fe5b - Browse repository at this point
Copy the full SHA 237fe5bView commit details
Commits on Mar 11, 2020
-
*: add license header (pingcap#182)
* rtree: move checkFile into backup Signed-off-by: Neil Shen <overvenus@gmail.com> * *: add license header Signed-off-by: Neil Shen <overvenus@gmail.com> * Update LICENSE.md Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com> Co-authored-by: 3pointer <luancheng@pingcap.com>
Configuration menu - View commit details
-
Copy full SHA for 0672ab3 - Browse repository at this point
Copy the full SHA 0672ab3View commit details -
conn: support not shutting down the storage when closing the connecti…
…on (pingcap#185) Co-authored-by: 3pointer <luancheng@pingcap.com>
Configuration menu - View commit details
-
Copy full SHA for 9caa6de - Browse repository at this point
Copy the full SHA 9caa6deView commit details -
conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186)
* conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <overvenus@gmail.com> * minor usability improvement Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 7d59284 - Browse repository at this point
Copy the full SHA 7d59284View commit details -
fix check safepoint & unhide experimental features (pingcap#175)
* backup: check safepoint for last backup ts Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lastbackupts > 0 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * unhide experimental features Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <kennytm@gmail.com> Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 0e25496 - Browse repository at this point
Copy the full SHA 0e25496View commit details
Commits on Mar 12, 2020
-
support backupts (pingcap#172)
* support backupts * address comment * address comment * fix space
Configuration menu - View commit details
-
Copy full SHA for 3419d8a - Browse repository at this point
Copy the full SHA 3419d8aView commit details -
*: update pd deps to v4 (pingcap#184)
Co-authored-by: 3pointer <luancheng@pingcap.com>
Configuration menu - View commit details
-
Copy full SHA for e476c82 - Browse repository at this point
Copy the full SHA e476c82View commit details
Commits on Mar 13, 2020
-
restore: support online restore (pingcap#114)
Signed-off-by: disksing <i@disksing.com>
Configuration menu - View commit details
-
Copy full SHA for 9c2bf9d - Browse repository at this point
Copy the full SHA 9c2bf9dView commit details
Commits on Mar 15, 2020
-
metrics: add grafana scripts (pingcap#140)
* add grafana scripts * fix Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 512855d - Browse repository at this point
Copy the full SHA 512855dView commit details
Commits on Mar 16, 2020
-
filter out all TiFlash nodes when retrieving lists of stores from PD (p…
…ingcap#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup
Configuration menu - View commit details
-
Copy full SHA for 6b88e51 - Browse repository at this point
Copy the full SHA 6b88e51View commit details -
Configuration menu - View commit details
-
Copy full SHA for 156f003 - Browse repository at this point
Copy the full SHA 156f003View commit details -
* *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <kennytm@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for bbedfc8 - Browse repository at this point
Copy the full SHA bbedfc8View commit details
Commits on Mar 17, 2020
-
Implement Raw Restore (pingcap#104)
* Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Remove unnecessary comments Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <overvenus@gmail.com> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
Configuration menu - View commit details
-
Copy full SHA for c3d26d9 - Browse repository at this point
Copy the full SHA c3d26d9View commit details
Commits on Mar 18, 2020
-
restore: remove tiflash replica before restore (pingcap#194)
* restore: remove tiflash replica before restore Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename errSplit variable Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check replica count by region info Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup Signed-off-by: 5kbpers <tangminghua@pingcap.com> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix save crcxor Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix decode the key of placement rule Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address lint Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Update pkg/task/restore.go Co-Authored-By: 3pointer <qdlc2010@gmail.com> Co-authored-by: 3pointer <qdlc2010@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 6268cde - Browse repository at this point
Copy the full SHA 6268cdeView commit details
Commits on Mar 19, 2020
-
summary: put summary log at last (pingcap#197)
* summary: put summary log at last * fix switch sql
Configuration menu - View commit details
-
Copy full SHA for 4ea6c1c - Browse repository at this point
Copy the full SHA 4ea6c1cView commit details