Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiDB CI data race (once) #23280

Closed
LittleFall opened this issue Mar 12, 2021 · 3 comments · Fixed by #24257
Closed

TiDB CI data race (once) #23280

LittleFall opened this issue Mar 12, 2021 · 3 comments · Fixed by #24257
Assignees
@guo-shaoge
Labels
severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@LittleFall
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

see:
https://internal.pingcap.net/idc-jenkins/blue/rest/organizations/jenkins/pipelines/tidb_ghpr_unit_test/runs/67419/nodes/82/steps/323/log/?start=0

2. What did you expect to see? (Required)

test success

3. What did you see instead (Required)

(*ParallelNestedLoopApplyExec).innerWorker\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:238"]
[2021-03-12T07:05:37.307Z] ==================
[2021-03-12T07:05:37.307Z] WARNING: DATA RACE
[2021-03-12T07:05:37.308Z] Write at 0x00c08f9bf300 by goroutine 332:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/statistics.(*QueryFeedback).Update()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/statistics/feedback.go:256 +0x8e4
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/distsql.(*selectResult).fetchResp()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/distsql/select_result.go:153 +0xaae
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/distsql.(*selectResult).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/distsql/select_result.go:180 +0x2b3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*tableResultHandler).nextChunk()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:329 +0xaa
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*TableReaderExecutor).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:206 +0x21e
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:277 +0x27d
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).outerWorker()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:202 +0x5a9
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Previous read at 0x00c08f9bf300 by goroutine 407:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/session.(*session).StoreQueryFeedback()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:372 +0x92e
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*TableReaderExecutor).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:226 +0x121
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:162 +0xa8
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*baseExecutor).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:172 +0xad
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:320 +0x255
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*recordSet).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:166 +0x59
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/session.(*execStmtResult).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1565 +0x7b
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).QueryToErr()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:306 +0x56a
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor_test.(*testSuite).TestApplyGoroutinePanic()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply_test.go:587 +0x546
[2021-03-12T07:05:37.308Z]   runtime.call32()
[2021-03-12T07:05:37.308Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2021-03-12T07:05:37.308Z]   reflect.Value.Call()
[2021-03-12T07:05:37.308Z]       /usr/local/go/src/reflect/value.go:321 +0xd3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Goroutine 332 (running) created at:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:137 +0xdb
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:277 +0x27d
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:373 +0x48f
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Goroutine 407 (running) created at:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkCall()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x4a3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkTest()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:832 +0x1b9
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).doRun()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:666 +0x13a
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).asyncRun.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:650 +0xf7
[2021-03-12T07:05:37.308Z] ==================

4. What is your TiDB version? (Required)

2021.3.12 master
@LittleFall LittleFall added the type/bug The issue is confirmed as a bug. label Mar 12, 2021
@ichn-hu ichn-hu mentioned this issue Mar 12, 2021
Closed
@sticnarf sticnarf mentioned this issue Mar 19, 2021
Merged
@jebter jebter added this to the v5.0.0 ga milestone Mar 21, 2021
@zimulala zimulala removed this from the v5.0.0 ga milestone Mar 22, 2021
@guo-shaoge
Copy link
Collaborator

/assign

@guo-shaoge
Copy link
Collaborator

guo-shaoge commented Apr 18, 2021 •
edited
Loading

As @eurekaka suggested, this data race happens when Apply operator panic and its' Close() is called, which will read feedback. At the same time, its outer worker(TableReaderExecutor) runs normally, which will write feedback.

So we may need to wait Apply operator's outer worker to finish, then read feedback.

@guo-shaoge guo-shaoge mentioned this issue Apr 25, 2021
Merged
@ti-srebot
Copy link
Contributor

Please edit this comment or add a new comment to complete the following information

Not a bug

  1. Remove the 'type/bug' label
  2. Add notes to indicate why it is not a bug

Duplicate bug

  1. Add the 'type/duplicate' label
  2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions

@ti-srebot ti-srebot mentioned this issue Apr 28, 2021
Merged
@bb7133 bb7133 mentioned this issue Jun 4, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guo-shaoge guo-shaoge

Labels
severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

executor: fix data race of parallel apply operator guo-shaoge/tidb
6 participants
@zimulala @jebter @ChenPeng2013 @guo-shaoge @LittleFall @ti-srebot