*: Reduce data inconsistencies #31547

ekexium · 2022-01-11T07:35:58Z

What problem does this PR solve?

Issue Number: ref #26833

Problem Summary:

Defend against data inconsistency and improve its troubleshooting.
It's developed in a separate feature branch ft-data-inconsistency.
PRs can be found in the tracking issue #26833.

What is changed and how it works?

Two features: the mutation checker and the assertion, and error message improvements.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Changed error messages for data-index inconsistency errors.
Add new error types for data-index inconsistency errors

* feat: check index key in AddRecord Signed-off-by: ekexium <ekexium@gmail.com> * feat: check for AddRecord/UpdateRecord Signed-off-by: ekexium <ekexium@gmail.com> * ignore deletion of indices if NeedRestoredData Signed-off-by: ekexium <ekexium@gmail.com> * check values of row mutations Signed-off-by: ekexium <ekexium@gmail.com> * style: fix naming Signed-off-by: ekexium <ekexium@gmail.com> * skip when sh == 0 Signed-off-by: ekexium <ekexium@gmail.com> * check in RemoveRecord Signed-off-by: ekexium <ekexium@gmail.com> * modify license header Signed-off-by: ekexium <ekexium@gmail.com> * test: add unit tests Signed-off-by: ekexium <ekexium@gmail.com> * add lisence header Signed-off-by: ekexium <ekexium@gmail.com> * also truncate decoded mutation Signed-off-by: ekexium <ekexium@gmail.com> * tidy up Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com> Auto stash before rebase of "ft-data-inconsistency" * skip partitioned table Signed-off-by: ekexium <ekexium@gmail.com> * save columnMap in stmtctx * skip when sh == 0 Some implementations of MemBuffer doesn't support staging. We don't care about them for now * reuse a slice in a loop * save reusable maps in stmtctx * save reusable maps in txn option * add unit test for checkIndexKeys * address comments in test

* check handle consistency Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com>

* add a sysvar to enable mutation checker Signed-off-by: ekexium <ekexium@gmail.com> * add an HTTP API to modify tidb_enable_mutation_checker Signed-off-by: ekexium <ekexium@gmail.com> * hide tidb_enable_mutation_checker Signed-off-by: ekexium <ekexium@gmail.com> * change the default behavior: only enable mutation checker for new clusters Signed-off-by: ekexium <ekexium@gmail.com> * fix name Signed-off-by: ekexium <ekexium@gmail.com>

* close #29469; fix: mistakenly shadowed name idx Signed-off-by: ekexium <ekexium@gmail.com> * test admin check table when uniqueness is violated Signed-off-by: ekexium <ekexium@gmail.com>

Signed-off-by: ekexium <ekexium@gmail.com>

Signed-off-by: ekexium <ekexium@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Signed-off-by: ekexium <ekexium@gmail.com>

* fix admin test Signed-off-by: ekexium <ekexium@gmail.com> * fix gosec Signed-off-by: ekexium <ekexium@gmail.com> * satisfy gosec Signed-off-by: ekexium <ekexium@gmail.com>

…ency

* Pick unistore changes from #28313, ref #26833 * fix build Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* fix admin test Signed-off-by: ekexium <ekexium@gmail.com> * fix gosec Signed-off-by: ekexium <ekexium@gmail.com> * satisfy gosec Signed-off-by: ekexium <ekexium@gmail.com> * add error codes for mutation checker; redact log if needed Signed-off-by: ekexium <ekexium@gmail.com> * add comments Signed-off-by: ekexium <ekexium@gmail.com> * fix errors.toml Signed-off-by: ekexium <ekexium@gmail.com> * better redact log Signed-off-by: ekexium <ekexium@gmail.com> * address comments Signed-off-by: ekexium <ekexium@gmail.com> * fix argument order Signed-off-by: ekexium <ekexium@gmail.com> * log error using redacted error object Signed-off-by: ekexium <ekexium@gmail.com> * modify log message Signed-off-by: ekexium <ekexium@gmail.com>

* *: set consistent assertion for DML * table: check duplicate for row_id * update client-go * fix fail in br * update client-go, kvproto and add failpoints * update kvproto and client-go * update client-go for metrics * Support assertion with existance state fetched during pessimistic lock phase * Fix incorrect CheckExistence implementation in unistore * support assertion level * Fix the bug that untouched index has wrong assertion * Remove change to explaintest script * Implement assertion in unistore * Add a basic test case for assertion * Add tests to SetAssertion interface * Add test case for previsously found bug * update client-go * Fix tests * Fix store not closed in TestSetAssertion * Address comments * Fix ignore assertion in CreateIndex overwritten * Fix build * Test clustered index * fix fmt * Enable assertion only for new clusters Co-authored-by: lysu <sulifx@gmail.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

…ogs (#30897) Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…ency

Signed-off-by: ekexium <ekexium@gmail.com>

ti-chi-bot · 2022-01-11T07:36:00Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

MyonKeminta
cfzjywxk

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: ekexium <ekexium@gmail.com>

* fix build Signed-off-by: ekexium <ekexium@gmail.com> * fix test Signed-off-by: ekexium <ekexium@gmail.com>

ekexium · 2022-01-11T14:42:38Z

/run-unit-test
/run-check_dev_2

ekexium · 2022-01-12T03:25:40Z

/run-unit-test

ekexium · 2022-01-12T06:13:32Z

/run-unit-test

* add a error code for assertion Signed-off-by: ekexium <ekexium@gmail.com> * take care of log redaction Signed-off-by: ekexium <ekexium@gmail.com> * generate errors.toml Signed-off-by: ekexium <ekexium@gmail.com> * get mvcc history Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com> * use ctx logger Signed-off-by: ekexium <ekexium@gmail.com>

sre-bot · 2022-01-18T13:53:47Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/b706ea398868c26d725fed58be2b1c2ea51c5934

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix unstable test TestCheckFailReport * Update executor/distsql.go Co-authored-by: Ziqian Qin <ekexium@gmail.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ziqian Qin <ekexium@gmail.com>

ekexium · 2022-01-19T15:17:21Z

/run-unit-test

store/driver/txn/txn_driver.go

table/tables/tables.go

table/tables/tables_test.go

Signed-off-by: ekexium <ekexium@gmail.com>

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…ency

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

…ency

Signed-off-by: ekexium <ekexium@gmail.com>

ekexium · 2022-02-11T05:38:34Z

/merge

ti-chi-bot · 2022-02-11T05:38:37Z

@ekexium: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

cfzjywxk · 2022-02-11T05:39:11Z

/merge

ti-chi-bot · 2022-02-11T05:39:14Z

This pull request has been accepted and is ready to merge.

Commit hash: b706ea3

lysu and others added 16 commits September 13, 2021 18:13

executor: enhance data inconsistency report (#27388)

32c8c9e

tables: check handle consistency (#28465)

a0e18f1

* check handle consistency Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com> * refactor according to comments Signed-off-by: ekexium <ekexium@gmail.com>

util: fix mistakenly shadowed name idx (#29470)

89d537c

* close #29469; fix: mistakenly shadowed name idx Signed-off-by: ekexium <ekexium@gmail.com> * test admin check table when uniqueness is violated Signed-off-by: ekexium <ekexium@gmail.com>

close #29520 (#29600)

7849fd9

Signed-off-by: ekexium <ekexium@gmail.com>

enable mutation checker for any store type (#29654)

c95a06d

Signed-off-by: ekexium <ekexium@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Merge branch 'master' into ft-data-inconsistency

e59565e

Signed-off-by: ekexium <ekexium@gmail.com>

executor: fix admin test (#30310)

a065bc6

* fix admin test Signed-off-by: ekexium <ekexium@gmail.com> * fix gosec Signed-off-by: ekexium <ekexium@gmail.com> * satisfy gosec Signed-off-by: ekexium <ekexium@gmail.com>

Merge remote-tracking branch 'upstream/master' into ft-data-inconsist…

2f2df86

…ency

store: Update unistore for assertion (#29902)

583ff25

* Pick unistore changes from #28313, ref #26833 * fix build Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Print readable log when TestCheckFailReport; Add stack trace to the l…

44479df

…ogs (#30897) Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Merge remote-tracking branch 'upstream/master' into ft-data-inconsist…

a2c535e

…ency

fix build

1557caa

Signed-off-by: ekexium <ekexium@gmail.com>

ekexium added 2 commits January 11, 2022 16:07

fix build

5c07595

Signed-off-by: ekexium <ekexium@gmail.com>

executor: fix test on dev branch (#31556)

ffafb71

* fix build Signed-off-by: ekexium <ekexium@gmail.com> * fix test Signed-off-by: ekexium <ekexium@gmail.com>

MyonKeminta and others added 2 commits January 19, 2022 13:34

Fix an incomplete comment (#31800)

6acfe3a

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

executor: Fix unstable test TestCheckFailReport (#31701)

fdf1296

* Fix unstable test TestCheckFailReport * Update executor/distsql.go Co-authored-by: Ziqian Qin <ekexium@gmail.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ziqian Qin <ekexium@gmail.com>

cfzjywxk reviewed Jan 22, 2022

View reviewed changes

store/driver/txn/txn_driver.go Outdated Show resolved Hide resolved

table/tables/tables.go Show resolved Hide resolved

table/tables/tables_test.go Outdated Show resolved Hide resolved

add grafana panels for assertion (#31145)

f1c52ca

Signed-off-by: ekexium <ekexium@gmail.com>

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2022

MyonKeminta mentioned this pull request Jan 25, 2022

* : Fix mistakes in comments about assertion; remove unnecessary deep copy to key when setting assertion #31945

Merged

4 tasks

Fix comments; remove unnecessary deep copy (#31945)

f95b022

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ekexium force-pushed the ft-data-inconsistency branch from c966685 to f95b022 Compare January 26, 2022 05:14

Merge remote-tracking branch 'upstream/master' into ft-data-inconsist…

e7bd1fa

…ency

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 26, 2022

MyonKeminta mentioned this pull request Feb 9, 2022

unistore: Update unistore for assertion #32189

Merged

12 tasks

Add a simple test case for assertion with lazy constraint check (#32130)

13a57fb

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 9, 2022

Merge remote-tracking branch 'upstream/master' into ft-data-inconsist…

18ae8d2

…ency

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 10, 2022

remove replace client-go (#32230)

b706ea3

Signed-off-by: ekexium <ekexium@gmail.com>

MyonKeminta approved these changes Feb 11, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 11, 2022

cfzjywxk approved these changes Feb 11, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 11, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 11, 2022

Merge branch 'master' into ft-data-inconsistency

bf00a57

ti-chi-bot merged commit 9c64224 into master Feb 11, 2022

ti-chi-bot deleted the ft-data-inconsistency branch February 11, 2022 05:51

ekexium mentioned this pull request Feb 11, 2022

Delete using outer join can mistakenly delete the row whose handle=0 #31321

Closed

morgo mentioned this pull request Apr 14, 2022

docs for instance scoped system variables pingcap/docs#7897

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Reduce data inconsistencies #31547

*: Reduce data inconsistencies #31547

ekexium commented Jan 11, 2022 •

edited

Loading

ti-chi-bot commented Jan 11, 2022 •

edited

Loading

ekexium commented Jan 11, 2022

ekexium commented Jan 12, 2022

ekexium commented Jan 12, 2022

sre-bot commented Jan 18, 2022 •

edited

Loading

ekexium commented Jan 19, 2022

ekexium commented Feb 11, 2022

ti-chi-bot commented Feb 11, 2022

cfzjywxk commented Feb 11, 2022

ti-chi-bot commented Feb 11, 2022

*: Reduce data inconsistencies #31547

*: Reduce data inconsistencies #31547

Conversation

ekexium commented Jan 11, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jan 11, 2022 • edited Loading

ekexium commented Jan 11, 2022

ekexium commented Jan 12, 2022

ekexium commented Jan 12, 2022

sre-bot commented Jan 18, 2022 • edited Loading

ekexium commented Jan 19, 2022

ekexium commented Feb 11, 2022

ti-chi-bot commented Feb 11, 2022

cfzjywxk commented Feb 11, 2022

ti-chi-bot commented Feb 11, 2022

ekexium commented Jan 11, 2022 •

edited

Loading

ti-chi-bot commented Jan 11, 2022 •

edited

Loading

sre-bot commented Jan 18, 2022 •

edited

Loading