Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: fix data race of parallel apply operator #24257

Merged
merged 2 commits into from
Apr 28, 2021
Merged

executor: fix data race of parallel apply operator #24257

merged 2 commits into from
Apr 28, 2021

Conversation

guo-shaoge
Copy link
Collaborator

@guo-shaoge guo-shaoge commented Apr 25, 2021 •
edited
Loading

What problem does this PR solve?

Issue Number: close #23280

Problem Summary: TestApplyGoroutinePanic test the situation of inner work panics. Sometime we got data race error. That's because:

  1. Apply operator initializes, outer worker(TableReaderExecutor) starts.
  2. Apply's Close() method is called by this case.
  3. outExec.Close() is called.
  4. Notify outer worker to finish.

Feedback info will be read in step 3. But before step 4 really works, outer worker still runs normally, it will write Feedbackinfo.

So we got data race. (Only happens on feedback, it's disabled by default.)

What is changed and how it works?

What's Changed: In Apply's Close() method, we first notify outer worker to finish and wait untill it's really finished. Then we call outer exec's Close() method.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests: difficult to reproduce.

Side effects: no

Release note

  • executor: fix data race of parallel apply operator

@guo-shaoge
executor: fix data race of parallel apply operator
bd9141d 
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge requested a review from a team as a code owner April 25, 2021 03:16
@guo-shaoge guo-shaoge requested review from wshwsh12 and removed request for a team April 25, 2021 03:16
@ti-chi-bot ti-chi-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Apr 25, 2021
@ti-chi-bot ti-chi-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 25, 2021
@guo-shaoge
Copy link
Collaborator Author

@wshwsh12 PTAL

@github-actions github-actions bot added the sig/execution SIG execution label Apr 25, 2021
XuHuaiyu
XuHuaiyu reviewed Apr 25, 2021
View reviewed changes
executor/parallel_apply.go
if atomic.LoadUint32(&e.started) == 1 {
close(e.exit)
e.notifyWg.Wait()
e.started = 0
}
// Wait all workers to finish before Close() is called.
// Otherwise we may got data race.
err := e.outerExec.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a failpoint test for this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bug is found by an existing case(TestApplyGoroutinePanic).
I think the case can cover the code.

@ichn-hu ichn-hu mentioned this pull request Apr 25, 2021
Closed
@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Apr 27, 2021
@ti-chi-bot ti-chi-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 28, 2021
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 28, 2021
@XuHuaiyu
Copy link
Contributor

/lgtm

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • XuHuaiyu
  • lzmhhh123

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 28, 2021
@djshow832 djshow832 added the sig/sql-infra SIG: SQL Infra label Apr 28, 2021
@djshow832
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: bd9141d

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 28, 2021
@guo-shaoge
Copy link
Collaborator Author

/run-unit-test

1 similar comment
@guo-shaoge
Copy link
Collaborator Author

/run-unit-test

@ti-chi-bot ti-chi-bot merged commit f07b62f into pingcap:master Apr 28, 2021
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Apr 28, 2021
@guo-shaoge @ti-srebot
cherry pick pingcap#24257 to release-5.0
cc80524 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Apr 28, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #24345

ti-chi-bot pushed a commit that referenced this pull request Jun 18, 2021
@ti-srebot
executor: fix data race of parallel apply operator (#24257) (#24345)
2c4be62
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XuHuaiyu XuHuaiyu XuHuaiyu left review comments

@lzmhhh123 lzmhhh123 lzmhhh123 approved these changes

@wshwsh12 wshwsh12 Awaiting requested review from wshwsh12

Assignees
No one assigned
Labels
needs-cherry-pick-release-5.0 sig/execution SIG execution sig/sql-infra SIG: SQL Infra size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

TiDB CI data race (once)
6 participants
@guo-shaoge @XuHuaiyu @ti-chi-bot @djshow832 @ti-srebot @lzmhhh123