infoschema, executor, txn: implement DATA_LOCK_WAITS table #24750

longfangsong · 2021-05-19T07:55:49Z

What problem does this PR solve?

We are going to implement the feature Lock View.

A part of it is to support display information about current lock waiting.

What is changed and how it works?

What's Changed:
Just implement the DATA_LOCK_WAITS table.

How it Works:

Add an API in Storage interface for querying lock waits info and do the query when requested.

Related changes

(the following pr need to be merged first)

PR on TiKV to export get_lock_wait_info API.

Check List

Tests

Unit test
Integration test(WIP)

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM

Release note

Add DATA_LOCK_WAITS table

store/driver/tikv_driver.go

MyonKeminta · 2021-05-24T05:52:19Z

store/driver/tikv_driver.go

+// GetLockWaits get return lock waits info
+func (s *tikvStore) GetLockWaits() ([]*deadlockPB.WaitForEntry, error) {
+	stores := s.GetRegionCache().GetStoresByType(tikvrpc.TiKV)
+	var resp *tikvrpc.Response
+	var err error
+	for _, store := range stores {
+		resp, err = s.GetTiKVClient().SendRequest(context.TODO(), store.Addr, tikvrpc.NewRequest(tikvrpc.CmdLockWaitInfo, &kvrpcpb.GetLockWaitInfoRequest{}), time.Second*30)
+		if err == nil {
+			break
+		}
+	}
+	if resp == nil {
+		return nil, err
+	}
+	return resp.Resp.(*kvrpcpb.GetLockWaitInfoResponse).Entries, nil
+}


I'm just noticing that there are several problems:

If the cluster is large and the transactions conflict very frequently, is it possible that the amount of lock waiting will be too large, and consumes too much memory in TiDB? Perhaps the problem is not that bad since the deadlock detector works well.

We didn't set a limit of entries returned by the GetLockWaitInfo request. But is it possible that there are too many lock wait entries in a single TiKV node, and scanning so many entries occupies the lock manager's thread for too long and increases the node's latency?

PTAL @youjiali1995

I think it's ok for now, but we should file issues for potential risks of the feature.

store/tikv/region_cache.go

ti-chi-bot · 2021-05-27T07:04:40Z

@youjiali1995: Thanks for your review. The bot only counts /lgtm from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

MyonKeminta · 2021-05-27T07:17:59Z

Please also fix the test caused by removing the HASH column. Rest LGTM.

ti-chi-bot · 2021-05-27T07:38:28Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

MyonKeminta
youjiali1995

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

MyonKeminta · 2021-05-27T07:38:34Z

/merge

ti-chi-bot · 2021-05-27T07:38:38Z

This pull request has been accepted and is ready to merge.

Commit hash: ee154c1

MyonKeminta · 2021-05-27T07:53:13Z

/run-mybatis-test

MyonKeminta · 2021-05-27T08:03:53Z

😕 I don't understand what's wrong with mybatis-test
@cfzjywxk Do you have any idea?

cfzjywxk · 2021-05-27T08:14:45Z

😕 I don't understand what's wrong with mybatis-test
@cfzjywxk Do you have any idea?

That's strange, it seems not related to this PR from the error messages .

Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ibtest.numerics' doesn't exist

cfzjywxk · 2021-05-27T08:15:40Z

/merge

ti-chi-bot · 2021-05-27T08:57:48Z

@longfangsong: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2021-05-27T09:49:14Z

/run-all-tests

This bot automatically retries jobs that failed on can merge PRs (send feedback to hi-rustin).

Silence the bot with the /merge cancel comment for consistent failures.

ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 19, 2021

github-actions bot added sig/execution SIG execution sig/sql-infra SIG: SQL Infra labels May 19, 2021

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 20, 2021

longfangsong force-pushed the lock-wait-table branch from a1611e3 to 25706c7 Compare May 20, 2021 05:59

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 20, 2021

longfangsong changed the title ~~txn: implement DATA_LOCK_WAITS table~~ infoschema, executor, txn: implement DATA_LOCK_WAITS table May 20, 2021

MyonKeminta mentioned this pull request May 20, 2021

Implement Lock View (Sprint 3) #24199

Closed

20 tasks

longfangsong marked this pull request as ready for review May 20, 2021 08:51

longfangsong requested a review from a team as a code owner May 20, 2021 08:51

longfangsong requested review from wshwsh12 and removed request for a team May 20, 2021 08:51

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 20, 2021

longfangsong commented May 21, 2021

View reviewed changes

store/driver/tikv_driver.go Outdated Show resolved Hide resolved

longfangsong force-pushed the lock-wait-table branch from 5b75241 to f80f6ee Compare May 21, 2021 02:37

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 21, 2021

longfangsong force-pushed the lock-wait-table branch from f80f6ee to 9e58797 Compare May 21, 2021 02:41

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 21, 2021

longfangsong force-pushed the lock-wait-table branch 5 times, most recently from 0cb2fa3 to 2d156a3 Compare May 21, 2021 06:12

MyonKeminta reviewed May 24, 2021

View reviewed changes

longfangsong added 5 commits May 26, 2021 15:33

Basic implementation

0967709

Make lint happy

f8fc3ae

check privilege

bb368cf

Add unit test

85a2e28

Add test for privilege

afdb321

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 27, 2021

youjiali1995 reviewed May 27, 2021

View reviewed changes

store/tikv/region_cache.go Outdated Show resolved Hide resolved

longfangsong added 3 commits May 27, 2021 14:54

remove hash field

eb7c354

address comments

5d60603

Merge branch 'master' into lock-wait-table

1f6f44d

youjiali1995 approved these changes May 27, 2021

View reviewed changes

MyonKeminta added the sig/transaction SIG:Transaction label May 27, 2021

youjiali1995 approved these changes May 27, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 27, 2021

fix test

ee154c1

MyonKeminta approved these changes May 27, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 27, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 27, 2021

ti-chi-bot added 2 commits May 27, 2021 15:46

Merge branch 'master' into lock-wait-table

e72acea

Merge branch 'master' into lock-wait-table

32adcb9

Merge branch 'master' into lock-wait-table

ce1c7bd

ti-chi-bot merged commit e87d035 into pingcap:master May 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infoschema, executor, txn: implement DATA_LOCK_WAITS table #24750

infoschema, executor, txn: implement DATA_LOCK_WAITS table #24750

longfangsong commented May 19, 2021 •

edited

Loading

MyonKeminta May 24, 2021

youjiali1995 May 25, 2021

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

MyonKeminta commented May 27, 2021 •

edited

Loading

cfzjywxk commented May 27, 2021 •

edited

Loading

cfzjywxk commented May 27, 2021

ti-chi-bot commented May 27, 2021

ti-chi-bot commented May 27, 2021

infoschema, executor, txn: implement DATA_LOCK_WAITS table #24750

infoschema, executor, txn: implement DATA_LOCK_WAITS table #24750

Conversation

longfangsong commented May 19, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

MyonKeminta May 24, 2021

Choose a reason for hiding this comment

youjiali1995 May 25, 2021

Choose a reason for hiding this comment

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

ti-chi-bot commented May 27, 2021

MyonKeminta commented May 27, 2021

MyonKeminta commented May 27, 2021 • edited Loading

cfzjywxk commented May 27, 2021 • edited Loading

cfzjywxk commented May 27, 2021

ti-chi-bot commented May 27, 2021

ti-chi-bot commented May 27, 2021

longfangsong commented May 19, 2021 •

edited

Loading

MyonKeminta commented May 27, 2021 •

edited

Loading

cfzjywxk commented May 27, 2021 •

edited

Loading