Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

util: optimize the performance of restore with db #22910

Merged
merged 12 commits into from
Mar 12, 2021
Merged

util: optimize the performance of restore with db #22910

merged 12 commits into from
Mar 12, 2021

Conversation

rebelice
Copy link
Contributor

@rebelice rebelice commented Feb 24, 2021 •
edited
Loading

What problem does this PR solve?

Issue Number: close #22398

Problem Summary:

We use restore to generate SQL with default DB, it will consume more CPU. This problem will be more prominent in batch insert, because insert is usually faster. Also, the column list and the value list can be very long.

In order to deal with this situation, for a simple insert, we do not use restore but string replacement.

Related changes

  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
  • Breaking backward compatibility

Release note

  • optimize the performance of restore with DB

@rebelice rebelice requested a review from a team as a code owner February 24, 2021 06:32
@rebelice rebelice requested review from lzmhhh123 and removed request for a team February 24, 2021 06:32
@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 24, 2021
@github-actions github-actions bot added sig/planner SIG: Planner sig/sql-infra SIG: SQL Infra labels Feb 24, 2021
@ichn-hu ichn-hu mentioned this pull request Feb 24, 2021
Closed
qw4990
qw4990 reviewed Mar 3, 2021
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@rebelice
Copy link
Contributor Author

rebelice commented Mar 4, 2021

/cc @eurekaka

@ti-chi-bot ti-chi-bot requested a review from eurekaka March 4, 2021 10:40
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 11, 2021
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 11, 2021
Reminiscent
Reminiscent approved these changes Mar 11, 2021
View reviewed changes
Copy link
Contributor

@Reminiscent Reminiscent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 11, 2021
eurekaka
eurekaka reviewed Mar 12, 2021
View reviewed changes
util/parser/ast.go
if tn.Schema.String() != "" {
builder.WriteString(tn.Schema.String())
} else {
builder.WriteString(defaultDB)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we consider the case sensitivity of the built string here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, Normalize() would convert the string to lower case then.

eurekaka
eurekaka approved these changes Mar 12, 2021
View reviewed changes
Copy link
Contributor

@eurekaka eurekaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

util/parser/ast.go
if tn.Schema.String() != "" {
builder.WriteString(tn.Schema.String())
} else {
builder.WriteString(defaultDB)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, Normalize() would convert the string to lower case then.

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Reminiscent
  • eurekaka

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 12, 2021
@eurekaka
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 0a2f723

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 12, 2021
@eurekaka eurekaka added priority/release-blocker This issue blocks a release. Please solve it ASAP. needs-cherry-pick-4.0 labels Mar 12, 2021
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 12, 2021
@rebelice
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@rebelice: /merge is only allowed for the committers in list.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@eurekaka
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 6ec703a

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 12, 2021
@ti-chi-bot ti-chi-bot merged commit 276dd0e into pingcap:master Mar 12, 2021
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Mar 12, 2021
@rebelice @ti-srebot
cherry pick pingcap#22910 to release-4.0
b4f03cb 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Mar 12, 2021
Closed
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #23283

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qw4990 qw4990 qw4990 left review comments

@eurekaka eurekaka eurekaka approved these changes

@Reminiscent Reminiscent Reminiscent approved these changes

@lzmhhh123 lzmhhh123 Awaiting requested review from lzmhhh123

Assignees
No one assigned
Labels
priority/release-blocker This issue blocks a release. Please solve it ASAP. sig/planner SIG: Planner sig/sql-infra SIG: SQL Infra size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

01-06 performance regression on batch_insert
6 participants
@rebelice @ti-chi-bot @eurekaka @ti-srebot @qw4990 @Reminiscent