Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tsan: data race on ComputeServerRunner.cancelJoinTasks #7432

Closed
SeaRise opened this issue May 7, 2023 · 1 comment · Fixed by #7434
Closed

Tsan: data race on ComputeServerRunner.cancelJoinTasks #7432

SeaRise opened this issue May 7, 2023 · 1 comment · Fixed by #7434
Labels
affects-6.5 affects-7.1 component/compute report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.

Comments

@SeaRise
Copy link
Contributor

SeaRise commented May 7, 2023

[2023-05-06T18:09:32.507Z] ==================
[2023-05-06T18:09:32.507Z] WARNING: ThreadSanitizer: data race (pid=37941)
[2023-05-06T18:09:32.507Z]   Read of size 8 at 0x7b0c00167ac0 by thread T176:
[2023-05-06T18:09:32.507Z]     #0 memcpy /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x27bb5e0)
[2023-05-06T18:09:32.507Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__assign_external(char const*, unsigned long) string.cpp (libc++.so.1+0x69254)
[2023-05-06T18:09:32.507Z]     #2 DB::MPPTask::runImpl() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:484:25 (gtests_dbms+0xdb9d53f)
[2023-05-06T18:09:32.507Z]     #3 DB::MPPTask::run()::$_17::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:154:97 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #4 decltype(static_cast<DB::MPPTask::run()::$_17&>(fp)()) std::__1::__invoke<DB::MPPTask::run()::$_17&>(DB::MPPTask::run()::$_17&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #5 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::MPPTask::run()::$_17&>(DB::MPPTask::run()::$_17&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #6 std::__1::__function::__alloc_func<DB::MPPTask::run()::$_17, std::__1::allocator<DB::MPPTask::run()::$_17>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #7 std::__1::__function::__func<DB::MPPTask::run()::$_17, std::__1::allocator<DB::MPPTask::run()::$_17>, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xdb9f0a3)
[2023-05-06T18:09:32.507Z]     #8 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #9 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #10 decltype(static_cast<std::__1::function<void ()> const&>(fp)()) std::__1::__invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #11 std::__1::invoke_result<std::__1::function<void ()> const&>::type std::__1::invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #12 auto std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:46:20 (gtests_dbms+0xe319f1d)
[2023-05-06T18:09:32.507Z]     #13 decltype(static_cast<std::__1::function<void ()>>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::function<void ()>&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z]     #14 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::tuple<std::__1::function<void ()>, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z]     #15 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xe319ba7)
[2023-05-06T18:09:32.507Z] 
[2023-05-06T18:09:32.507Z]   Previous write of size 8 at 0x7b0c00167ac0 by main thread:
[2023-05-06T18:09:32.507Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2826598)
[2023-05-06T18:09:32.507Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (libc++.so.1+0x6a328)
[2023-05-06T18:09:32.507Z]     #2 DB::MPPTaskManager::abortMPPQuery(DB::MPPQueryId const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::AbortType) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTaskManager.cpp:201:20 (gtests_dbms+0xdba8d77)
[2023-05-06T18:09:32.507Z]     #3 DB::FlashService::cancelMPPTaskForTest(mpp::CancelTaskRequest const*, mpp::CancelTaskResponse*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:518:19 (gtests_dbms+0xd9c9a7d)
[2023-05-06T18:09:32.507Z]     #4 DB::tests::MockComputeServerManager::cancelQuery(DB::MPPQueryId const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:133:40 (gtests_dbms+0xcad03be)
[2023-05-06T18:09:32.507Z]     #5 DB::tests::ComputeServerRunner_cancelJoinTasks_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:545:46 (gtests_dbms+0x2c9693c)
[2023-05-06T18:09:32.507Z]     #6 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb63e885)
[2023-05-06T18:09:32.508Z]     #7 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb63e885)
[2023-05-06T18:09:32.508Z]     #8 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xb618425)
[2023-05-06T18:09:32.508Z]     #9 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xb619c54)
[2023-05-06T18:09:32.508Z]     #10 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xb61a91d)
[2023-05-06T18:09:32.508Z]     #11 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xb62aad0)
[2023-05-06T18:09:32.508Z]     #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb63fd25)
[2023-05-06T18:09:32.508Z]     #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb63fd25)
[2023-05-06T18:09:32.508Z]     #14 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xb629db8)
[2023-05-06T18:09:32.508Z]     #15 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x59356e6)
[2023-05-06T18:09:32.508Z]     #16 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:115:16 (gtests_dbms+0x59356e6)
[2023-05-06T18:09:32.508Z] 
[2023-05-06T18:09:32.508Z]   Location is heap block of size 48 at 0x7b0c00167ac0 allocated by main thread:
[2023-05-06T18:09:32.508Z]     #0 operator new(unsigned long) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (gtests_dbms+0x2826598)
[2023-05-06T18:09:32.508Z]     #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator=(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (libc++.so.1+0x6a328)
[2023-05-06T18:09:32.508Z]     #2 DB::MPPTaskManager::abortMPPQuery(DB::MPPQueryId const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::AbortType) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTaskManager.cpp:201:20 (gtests_dbms+0xdba8d77)
[2023-05-06T18:09:32.508Z]     #3 DB::FlashService::cancelMPPTaskForTest(mpp::CancelTaskRequest const*, mpp::CancelTaskResponse*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/FlashService.cpp:518:19 (gtests_dbms+0xd9c9a7d)
[2023-05-06T18:09:32.508Z]     #4 DB::tests::MockComputeServerManager::cancelQuery(DB::MPPQueryId const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Debug/MockComputeServerManager.cpp:133:40 (gtests_dbms+0xcad03be)
[2023-05-06T18:09:32.508Z]     #5 DB::tests::ComputeServerRunner_cancelJoinTasks_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:545:46 (gtests_dbms+0x2c9693c)
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 affects-7.1 component/compute report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants