Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tcp tunneling integration flake #12253

Closed
mattklein123 opened this issue Jul 23, 2020 · 4 comments · Fixed by #12267
Closed

tcp tunneling integration flake #12253

mattklein123 opened this issue Jul 23, 2020 · 4 comments · Fixed by #12267
Assignees
@alyssawilk
Labels
area/test flakes help wanted Needs help!

Comments

@mattklein123
Copy link
Member 

[ RUN      ] IpVersions/TcpTunnelingIntegrationTest.TcpProxyUpstreamFlush/IPv6
[2020-07-23 13:00:06.534][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x61616161616179
[2020-07-23 13:00:06.534][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-07-23 13:00:06.534][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.16.0-dev/test/RELEASE/BoringSSL
[2020-07-23 13:00:06.534][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __kernel_rt_sigreturn [0xffffbd4cb5b8]
[2020-07-23 13:00:06.539][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: Envoy::Event::DispatcherImpl::runPostCallbacks() [0xe60c5c]
[2020-07-23 13:00:06.544][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: event_process_active_single_queue [0x1570b00]
[2020-07-23 13:00:06.549][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: event_base_loop [0x156f624]
[2020-07-23 13:00:06.553][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::FakeUpstream::threadRoutine() [0x872914]
[2020-07-23 13:00:06.558][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x156a8e0]
[2020-07-23 13:00:06.558][10423][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: start_thread [0xffffbd396088]
@mattklein123
Copy link
Member Author

cc @alyssawilk

mattklein123 added a commit that referenced this issue Jul 24, 2020
@mattklein123
tcp tunneling: fix integration test flake
5ad5ab3 
We need to wait for all listeners to be up.

Fixes #12253

Signed-off-by: Matt Klein <mklein@lyft.com>
@mattklein123 mattklein123 mentioned this issue Jul 24, 2020
Merged
@mattklein123
Copy link
Member Author

There are still more issues here. I saw the same crash on master after my fix.

@mattklein123 mattklein123 reopened this Jul 27, 2020
KBaichoo pushed a commit to KBaichoo/envoy that referenced this issue Jul 30, 2020
@mattklein123 @KBaichoo
tcp tunneling: fix integration test flake (envoyproxy#12267)
0005526 
We need to wait for all listeners to be up.

Fixes envoyproxy#12253 (and maybe other flakes)

Risk Level: None
Testing: Existing tests
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Kevin Baichoo <kbaichoo@google.com>
@mattklein123 mattklein123 mentioned this issue Jul 31, 2020
Closed
@antoniovicente
Copy link
Contributor

antoniovicente commented Aug 3, 2020 •
edited
Loading

Trying to repo under ASAN. I suspect it has something to do with FakeUpstream post callbacks which may result in use-after-free.

[ RUN ] IpVersions/ConnectTerminationIntegrationTest.BasicMaxStreamDuration/IPv6
==12==ERROR: AddressSanitizer: heap-use-after-free on address 0x6080019a2d48 at pc 0x0000171145db bp 0x7f7161de14e0 sp 0x7f7161de14d8
READ of size 8 at 0x6080019a2d48 thread T131
#0 0x171145da in Envoy::Thread::LockGuard::LockGuard(Envoy::Thread::BasicLockable&) ??:?
#1 0x1751cebf in Envoy::FakeRawConnection::ReadFilter::onData(Envoy::Buffer::Instance&, bool) ??:?
#2 0x1ed88933 in Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) ??:?
#3 0x1ed89222 in Envoy::Network::FilterManagerImpl::onRead() ??:?
#4 0x1ed16a8c in Envoy::Network::ConnectionImpl::onRead(unsigned long) ??:?
#5 0x1ed2698b in Envoy::Network::ConnectionImpl::onReadReady() ??:?
#6 0x1ed20833 in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) ??:?
#7 0x1ed5fbfd in Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6::operator()(unsigned int) const connection_impl.cc:?
#8 0x1ed5fb26 in _ZNSt3__18__invokeIRZN5Envoy7Network14ConnectionImplC1ERNS1_5Event10DispatcherEONS_10unique_ptrINS2_16ConnectionSocketENS_14default_deleteIS8_EEEEONS7_INS2_15TransportSocketENS9_ISD_EEEERNS1_10StreamInfo10StreamInfoEbE3$6JjEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSM_DpOSN connection_impl.cc:?
#9 0x1ed5f8a6 in void std::__1::__invoke_void_return_wrapper::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6&, unsigned int&&) connection_impl.cc:?
#10 0x1ed5f76c in std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:?
#11 0x1ed5a2f7 in std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:?
#12 0x1b0f1070 in std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const mocks.cc:?
#13 0x1b0f0d32 in std::__1::function<void (unsigned int)>::operator()(unsigned int) const ??:?
#14 0x1ecd8f8c in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) ??:?
#15 0x1ecd9586 in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::operator()(int, short, void*) const file_event_impl.cc:?
#16 0x1ecd8fc5 in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*) file_event_impl.cc:?
#17 0x25085ead in event_persist_closure event.c:?
#18 0x2507ff34 in event_process_active_single_queue event.c:?
#19 0x25057e30 in event_process_active event.c:?
#20 0x2504f709 in event_base_loop ??:?
#21 0x1f511e4b in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:?
#22 0x1ec9d779 in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:?
#23 0x17513a9f in Envoy::FakeUpstream::threadRoutine() ??:?
#24 0x1757eb27 in Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12::operator()() const fake_upstream.cc:?
#25 0x1757ea59 in _ZNSt3__18__invokeIRZN5Envoy12FakeUpstreamC1EONS_10unique_ptrINS1_7Network22TransportSocketFactoryENS_14default_deleteIS5_EEEEONS3_INS4_6SocketENS6_ISA_EEEENS1_18FakeHttpConnection4TypeERNS1_5Event14TestTimeSystemEbE4$12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSL_DpOSM fake_upstream.cc:?
#26 0x1757e8a9 in void std::__1::__invoke_void_return_wrapper::__call<Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12&>(Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12&) fake_upstream.cc:?
#27 0x1757e7ff in std::__1::__function::__alloc_func<Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12, std::__1::allocator<Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12>, void ()>::operator()() fake_upstream.cc:?
#28 0x1757940a in std::__1::__function::__func<Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12, std::__1::allocator<Envoy::FakeUpstream::FakeUpstream(std::__1::unique_ptr<Envoy::Network::TransportSocketFactory, std::__1::default_deleteEnvoy::Network::TransportSocketFactory >&&, std::__1::unique_ptr<Envoy::Network::Socket, std::__1::default_deleteEnvoy::Network::Socket >&&, Envoy::FakeHttpConnection::Type, Envoy::Event::TestTimeSystem&, bool)::$_12>, void ()>::operator()() fake_upstream.cc:?
#29 0x174ba8e3 in std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:?
#30 0x174b9fea in std::__1::function<void ()>::operator()() const ??:?
#31 0x24ff195e in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optionalEnvoy::Thread::Options const&)::{lambda(void*)#1}::operator()(void*) const ??:?
#32 0x24ff18b4 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optionalEnvoy::Thread::Options const&)::{lambda(void*)#1}::__invoke(void*) ??:?
#33 0x7f716d601f26 in start_thread /build/glibc-M65Gwz/glibc-2.30/nptl/pthread_create.c:479
#34 0x7f716d5172ee in __GI___clone /build/glibc-M65Gwz/glibc-2.30/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x6080019a2d48 is located 40 bytes inside of 96-byte region [0x6080019a2d20,0x6080019a2d80)
freed by thread T0 here:
#0 0x16e5e3ed in __interceptor_free ??:?
#1 0x1765f850 in Envoy::FakeRawConnection::~FakeRawConnection() ??:?
#2 0x170fee4a in std::__1::default_deleteEnvoy::FakeRawConnection::operator()(Envoy::FakeRawConnection*) const tcp_tunneling_integration_test.cc:?
#3 0x170feb7c in std::__1::unique_ptr<Envoy::FakeRawConnection, std::__1::default_deleteEnvoy::FakeRawConnection >::reset(Envoy::FakeRawConnection*) tcp_tunneling_integration_test.cc:?
#4 0x170fe368 in std::__1::unique_ptr<Envoy::FakeRawConnection, std::_1::default_deleteEnvoy::FakeRawConnection >::~unique_ptr() tcp_tunneling_integration_test.cc:?
#5 0x16ea2a96 in Envoy::(anonymous namespace)::ConnectTerminationIntegrationTest::~ConnectTerminationIntegrationTest() tcp_tunneling_integration_test.cc:?
#6 0x16eb38a7 in Envoy::(anonymous namespace)::ConnectTerminationIntegrationTest_BasicMaxStreamDuration_Test::~ConnectTerminationIntegrationTest_BasicMaxStreamDuration_Test() tcp_tunneling_integration_test.cc:?
#7 0x16eb3904 in Envoy::(anonymous namespace)::ConnectTerminationIntegrationTest_BasicMaxStreamDuration_Test::~ConnectTerminationIntegrationTest_BasicMaxStreamDuration_Test() tcp_tunneling_integration_test.cc:?
#8 0x254de08d in testing::Test::DeleteSelf() ??:?
#9 0x2552759b in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ??:?
#10 0x254dc320 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ??:?
#11 0x2549a267 in testing::TestInfo::Run() ??:?
#12 0x2549ba74 in testing::TestSuite::Run() ??:?
#13 0x254c347a in testing::internal::UnitTestImpl::RunAllTests() ??:?
#14 0x2553cb9b in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ??:?
#15 0x254e5085 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ??:?
#16 0x254c2012 in testing::UnitTest::Run() ??:?
#17 0x1ea414a1 in RUN_ALL_TESTS() ??:?
#18 0x1ea3f243 in Envoy::TestRunner::RunTests(int, char**) ??:?
#19 0x1ea3a7b6 in main ??:?
#20 0x7f716d440e0a in __libc_start_main /build/glibc-M65Gwz/glibc-2.30/csu/../csu/libc-start.c:308

previously allocated by thread T0 here:
#0 0x16e5e66d in __interceptor_malloc ??:?
#1 0x7f716d6d8a37 in operator new(unsigned long) ??:?
#2 0x175181b0 in Envoy::FakeUpstream::waitForRawConnection(std::__1::unique_ptr<Envoy::FakeRawConnection, std::__1::default_deleteEnvoy::FakeRawConnection >&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >) ??:?
#3 0x16ea384d in Envoy::(anonymous namespace)::ConnectTerminationIntegrationTest::setUpConnection() tcp_tunneling_integration_test.cc:?
#4 0x16eb3e19 in Envoy::(anonymous namespace)::ConnectTerminationIntegrationTest_BasicMaxStreamDuration_Test::TestBody() tcp_tunneling_integration_test.cc:?
#5 0x2552759b in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ??:?
#6 0x254dc320 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) ??:?
#7 0x25497d2f in testing::Test::Run() ??:?
#8 0x2549a09f in testing::TestInfo::Run() ??:?
#9 0x2549ba74 in testing::TestSuite::Run() ??:?
#10 0x254c347a in testing::internal::UnitTestImpl::RunAllTests() ??:?
#11 0x2553cb9b in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ??:?
#12 0x254e5085 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) ??:?
#13 0x254c2012 in testing::UnitTest::Run() ??:?
#14 0x1ea414a1 in RUN_ALL_TESTS() ??:?
#15 0x1ea3f243 in Envoy::TestRunner::RunTests(int, char**) ??:?
#16 0x1ea3a7b6 in main ??:?
#17 0x7f716d440e0a in __libc_start_main /build/glibc-M65Gwz/glibc-2.30/csu/../csu/libc-start.c:308

@mattklein123
Copy link
Member Author

@antoniovicente I was able to repro on an ARM machine with opt build (???). I have a fix soon.

chaoqin-li1123 pushed a commit to chaoqin-li1123/envoy that referenced this issue Aug 7, 2020
@mattklein123
tcp tunneling: fix integration test flake (envoyproxy#12267)
76584cd 
We need to wait for all listeners to be up.

Fixes envoyproxy#12253 (and maybe other flakes)

Risk Level: None
Testing: Existing tests
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: chaoqinli <chaoqinli@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alyssawilk alyssawilk

Labels
area/test flakes help wanted Needs help!
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

tcp tunneling: fix integration test flake envoyproxy/envoy
3 participants
@mattklein123 @alyssawilk @antoniovicente