Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

potential DATA RACE in update operation #25496

Closed
tiancaiamao opened this issue Jun 16, 2021 · 3 comments
Closed

potential DATA RACE in update operation #25496

tiancaiamao opened this issue Jun 16, 2021 · 3 comments
Labels
component/test severity/minor sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

It is found in CI of #21148 but not necessarily caused by that change.

Update executor will read data from its children executor and write the data to the unistore, the children executor maybe
running concurrently, so there will be goroutine reading from the unistore, and at the same time some goroutine writing the unistore. This is potential risky, especially when the selectLock executor exists: selectLock is a read executor but it also write locks to the unistore.

The data race affect the unistore engine, the tikv engine is not affected, because lockKeys on TiKV will marthal the data and send the distsql request.

[2021-06-16T07:44:45.335Z] WARNING: DATA RACE
[2021-06-16T07:44:45.335Z] Write at 0x00c07fa89830 by goroutine 151:
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/unionstore.(*MemDB).allocNode()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/unionstore/memdb.go:741 +0x6e
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/unionstore.(*MemDB).traverse()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/unionstore/memdb.go:364 +0x604
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/unionstore.(*MemDB).set()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/unionstore/memdb.go:296 +0x13a
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/unionstore.(*MemDB).UpdateFlags()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/unionstore/memdb.go:211 +0x5f6
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/tikv.(*KVTxn).LockKeys()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/tikv/txn.go:635 +0x576
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/store/driver/txn.(*tikvTxn).LockKeys()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/store/driver/txn/txn_driver.go:61 +0x11e
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/session.(*LazyTxn).LockKeys()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:384 +0x1d8
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.doLockKeys()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:1016 +0x2ef
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*SelectLockExec).Next()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:966 +0xe6f
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.Next()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2a8
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:380 +0x498
[2021-06-16T07:44:45.335Z] 
[2021-06-16T07:44:45.335Z] Previous read at 0x00c07fa89830 by goroutine 378:
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/unionstore.(*MemDB).Size()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/unionstore/memdb.go:267 +0x7c
[2021-06-16T07:44:45.335Z]   github.com/tikv/client-go/v2/tikv.(*KVTxn).Size()
[2021-06-16T07:44:45.335Z]       /nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-20210616060203-94f269a0f96a/tikv/txn.go:726 +0x2f
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/store/driver/txn.(*tikvTxn).Size()
[2021-06-16T07:44:45.335Z]       <autogenerated>:1 +0x59
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/session.(*LazyTxn).Size()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/txn.go:159 +0x6b
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.updateRecord()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/write.go:62 +0x126
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*UpdateExec).exec()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/update.go:188 +0x765
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*UpdateExec).updateRows()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/update.go:301 +0x7dd
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*UpdateExec).Next()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/update.go:219 +0xa9
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.Next()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2a8
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:586 +0x3bc
[2021-06-16T07:44:45.335Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML()
[2021-06-16T07:44:45.335Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:605 +0x1d2
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:465 +0x1ef
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).Exec()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:416 +0x63a
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/session.runStmt()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1653 +0x323
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/session.(*session).ExecuteStmt()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1548 +0xc37
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).Exec()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:170 +0x2fc
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).MustExec()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:216 +0x96
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor_test.(*testSuite).TestIssue20975UpdateNoChangeWithPartitionTable()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor_test.go:7593 +0xfb
[2021-06-16T07:44:45.336Z]   runtime.call16()
[2021-06-16T07:44:45.336Z]       /usr/local/go/src/runtime/asm_amd64.s:550 +0x3d
[2021-06-16T07:44:45.336Z]   reflect.Value.Call()
[2021-06-16T07:44:45.336Z]       /usr/local/go/src/reflect/value.go:337 +0xd8
[2021-06-16T07:44:45.336Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-06-16T07:44:45.336Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0xb3b
[2021-06-16T07:44:45.336Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-06-16T07:44:45.336Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x11d
[2021-06-16T07:44:45.336Z] 
[2021-06-16T07:44:45.336Z] Goroutine 151 (running) created at:
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).prepare()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:271 +0xd04
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).parallelExecute()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:208 +0x5cb
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).Next()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:181 +0x10d
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.Next()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2a8
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*UpdateExec).updateRows()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/update.go:251 +0xcb3
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*UpdateExec).Next()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/update.go:219 +0xa9
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.Next()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2a8
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:586 +0x3bc
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:605 +0x1d2
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:465 +0x1ef
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/executor.(*ExecStmt).Exec()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:416 +0x63a
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/session.runStmt()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1653 +0x323
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/session.(*session).ExecuteStmt()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1548 +0xc37
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).Exec()
[2021-06-16T07:44:45.336Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:170 +0x2fc
[2021-06-16T07:44:45.336Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).MustExec()
[2021-06-16T07:44:45.337Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:216 +0x96
[2021-06-16T07:44:45.337Z]   github.com/pingcap/tidb/executor_test.(*testSuite).TestIssue20975UpdateNoChangeWithPartitionTable()
[2021-06-16T07:44:45.337Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor_test.go:7593 +0xfb
[2021-06-16T07:44:45.337Z]   runtime.call16()
[2021-06-16T07:44:45.337Z]       /usr/local/go/src/runtime/asm_amd64.s:550 +0x3d
[2021-06-16T07:44:45.337Z]   reflect.Value.Call()
[2021-06-16T07:44:45.337Z]       /usr/local/go/src/reflect/value.go:337 +0xd8
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0xb3b
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x11d
[2021-06-16T07:44:45.337Z] 
[2021-06-16T07:44:45.337Z] Goroutine 378 (running) created at:
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).forkCall()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x4cc
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).forkTest()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:832 +0x1c4
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).doRun()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:666 +0x144
[2021-06-16T07:44:45.337Z]   github.com/pingcap/check.(*suiteRunner).asyncRun.func1()
[2021-06-16T07:44:45.337Z]       /nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:650 +0x105
[2021-06-16T07:44:45.337Z] ==================

2. What did you expect to see? (Required)

No data race

3. What did you see instead (Required)

data race

4. What is your TiDB version? (Required)

#21148
@tiancaiamao tiancaiamao added type/bug The issue is confirmed as a bug. severity/minor labels Jun 16, 2021
@lysu
Copy link
Contributor

lysu commented Jun 17, 2021

maybe it also affects on tikv-engine, unionstore.*MemDB is an implemention for previous mem buffer

parallel inside executor is devil 😭

@jebter jebter added the sig/transaction SIG:Transaction label Dec 7, 2021
@tiancaiamao
Copy link
Contributor Author

This should have been fixed by #30290
It's the same problem with #26832

@github-actions
Copy link

github-actions bot commented Dec 7, 2021

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/test severity/minor sig/transaction SIG:Transaction type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lysu @tiancaiamao @jebter