Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: add expensive_txn log and metric for long-running transaction #44248

Merged
merged 12 commits into from
Jun 8, 2023
Merged

*: add expensive_txn log and metric for long-running transaction #44248

merged 12 commits into from
Jun 8, 2023

Conversation

crazycs520
Copy link
Contributor

@crazycs520 crazycs520 commented May 29, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: close #41471

Problem Summary: add expensive_txn log and metric for long-running transaction

What is changed and how it works?

  • Add expensive_txn log for long-running transactions.
    • Add instant scope variable tidb_expensive_txn_time_threshold and config tidb_expensive_txn_time_threshold to control the log threshold.
[2023/06/05 14:16:08.671 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=600.011440938s] [conn=2199023255955] [user=root] [database=test] [txn_start_ts=441960410294517760] [mem_max="0 Bytes (0 Bytes)"] [sql=]
[2023/06/05 14:16:41.670 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=600.027316001s] [conn=2199023255965] [user=root] [database=test] [txn_start_ts=441960418941075456] [mem_max="0 Bytes (0 Bytes)"] [sql="select sleep(100000)"]
[2023/06/05 14:26:08.765 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=1200.111538202s] [conn=2199023255955] [user=root] [database=test] [txn_start_ts=441960410294517760] [mem_max="0 Bytes (0 Bytes)"] [sql=]
[2023/06/05 14:26:41.764 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=1200.127301835s] [conn=2199023255965] [user=root] [database=test] [txn_start_ts=441960418941075456] [mem_max="0 Bytes (0 Bytes)"] [sql="select sleep(100000)"]
[2023/06/05 14:36:08.759 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=1800.111627455s] [conn=2199023255955] [user=root] [database=test] [txn_start_ts=441960410294517760] [mem_max="0 Bytes (0 Bytes)"] [sql=]
[2023/06/05 14:36:41.759 +08:00] [WARN] [expensivequery.go:145] [expensive_txn] [cost_time=1800.128159439s] [conn=2199023255965] [user=root] [database=test] [txn_start_ts=441960418941075456] [mem_max="0 Bytes (0 Bytes)"] [sql="select sleep(100000)"]
  • Add metric for long-running transactions.
image

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@crazycs520
*: add expensive_txn log and metric for long-running transaction
3f5e424 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
add metric into grafana
6afd36b 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@ti-chi-bot
Copy link

ti-chi-bot bot commented May 29, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • zyguan

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 29, 2023
@crazycs520 crazycs520 added component/metrics sig/transaction SIG:Transaction and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 29, 2023
@crazycs520
make bazel_prepare
3828210 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@ti-chi-bot ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 29, 2023
@crazycs520
Copy link
Contributor Author

/retest

@crazycs520
fix fieldalignment ci lint
9aac28a 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@pingcap pingcap deleted a comment from ti-chi-bot bot May 30, 2023
zyguan
zyguan reviewed May 30, 2023
View reviewed changes
util/expensivequery/expensivequery.go Outdated Show resolved Hide resolved
metrics/grafana/tidb.json Outdated Show resolved Hide resolved
@crazycs520
fix grafana
e6896d1 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
use heatmap
7785019 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
address comment
d6f190f 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520 crazycs520 requested a review from a team as a code owner May 30, 2023 07:41
@crazycs520
add test
9167416 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
refine logic
a463a03 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
zyguan
zyguan reviewed May 30, 2023
View reviewed changes
Copy link
Contributor

@zyguan zyguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

util/expensivequery/expensivequery.go Show resolved Hide resolved
@crazycs520
address comment
6de8c46 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@ti-chi-bot ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 31, 2023
@hawkingrei hawkingrei requested a review from qw4990 May 31, 2023 07:20
cfzjywxk
cfzjywxk approved these changes Jun 7, 2023
View reviewed changes
Copy link
Contributor

@cfzjywxk cfzjywxk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
As a new system variable is introduced we may need to change the document.

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jun 7, 2023
@cfzjywxk
Copy link
Contributor

cfzjywxk commented Jun 7, 2023

/merge

@crazycs520 crazycs520 added the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2023
@crazycs520
Copy link
Contributor Author

/merge

@crazycs520 crazycs520 removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 7, 2023
@zyguan
Copy link
Contributor

zyguan commented Jun 8, 2023

/lgtm

@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 8, 2023
@zyguan
Copy link
Contributor

zyguan commented Jun 8, 2023

/merge

@pingcap pingcap deleted a comment from ti-chi-bot bot Jun 8, 2023
@pingcap pingcap deleted a comment from ti-chi-bot bot Jun 8, 2023
@crazycs520 crazycs520 added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 8, 2023
@crazycs520
Copy link
Contributor Author

/merge

@crazycs520 crazycs520 removed the lgtm label Jun 8, 2023
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 8, 2023
zyguan
zyguan approved these changes Jun 8, 2023
View reviewed changes
Copy link
Contributor

@zyguan zyguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@zyguan zyguan removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 8, 2023
@cfzjywxk
Copy link
Contributor

cfzjywxk commented Jun 8, 2023

/merge

@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 8, 2023
@ti-chi-bot
Copy link

ti-chi-bot bot commented Jun 8, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-06-07 04:15:26.156773056 +0000 UTC m=+64418.001407640: ☑️ agreed by cfzjywxk.
  • 2023-06-08 05:22:56.2375394 +0000 UTC m=+154868.082174001: ☑️ agreed by zyguan.
  • 2023-06-08 05:32:00.117764416 +0000 UTC m=+155411.962399016: ☑️ agreed by hawkingrei.
  • 2023-06-08 07:11:21.217087546 +0000 UTC m=+161373.061722144: ☑️ agreed by lance6716.

@hawkingrei
Copy link
Member

/lgtm

@ti-chi-bot
Copy link

ti-chi-bot bot commented Jun 8, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, hawkingrei, lance6716, zyguan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [cfzjywxk,hawkingrei,lance6716,zyguan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hawkingrei hawkingrei added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 8, 2023
@ti-chi-bot ti-chi-bot bot added lgtm and removed lgtm labels Jun 8, 2023
@ti-chi-bot ti-chi-bot bot merged commit 1e98f86 into pingcap:master Jun 8, 2023
@crazycs520 crazycs520 mentioned this pull request Jun 12, 2023
Merged
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zyguan zyguan zyguan approved these changes

@cfzjywxk cfzjywxk cfzjywxk approved these changes

@hawkingrei hawkingrei hawkingrei approved these changes

@lance6716 lance6716 lance6716 approved these changes

@qw4990 qw4990 Awaiting requested review from qw4990

Assignees
No one assigned
Labels
approved component/metrics lgtm release-note-none Denotes a PR that doesn't merit a release note. sig/transaction SIG:Transaction size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

txn: record long running transaction and session information for diagnosis
5 participants
@crazycs520 @cfzjywxk @zyguan @hawkingrei @lance6716