Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vhds: tsan failure in test/integration/vhds_integration_test #9784

Closed
jmarantz opened this issue Jan 22, 2020 · 22 comments · Fixed by #12134
Closed

vhds: tsan failure in test/integration/vhds_integration_test #9784

jmarantz opened this issue Jan 22, 2020 · 22 comments · Fixed by #12134
Assignees
@dmitri-d
Labels
area/test flakes bug no stalebot Disables stalebot from closing an issue
Milestone
1.16.0

Comments

@jmarantz
Copy link
Contributor

I am not too familiar with VHDS, but i testing #9774 locally I ran into this. It's possible the PR I was testing was responsible but it seems likely to be unrelated, given the stack trace.

[ RUN      ] IpVersionsClientType/VhdsIntegrationTest.VhdsVirtualHostAddUpdateRemove/0
==================
WARNING: ThreadSanitizer: data race (pid=14)
  Write of size 8 at 0x7b140000f0f0 by thread T7:
    #0 free ??:? (vhds_integration_test+0x21dd4f8)
    #1 std::__1::_DeallocateCaller::__do_call(void*) ??:? (vhds_integration_test+0x2273fb8)
    #2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) ??:? (vhds_integration_test+0x2273f84)
    #3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) vhds_integration_test.cc:? (vhds_integration_test+0x2273f24)
    #4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) vhds_integration_test.cc:? (vhds_integration_test+0x2273eb8)
    #5 std::__1::allocator<std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > > >::deallocate(std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >*, unsigned long) on_demand_update.cc:? (vhds_integration_test+0x3041fa1)
    #6 std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >::__on_zero_shared_weak() ??:? (vhds_integration_test+0x30419b5)
    #7 std::__1::weak_ptr<std::__1::function<void (bool)> >::~weak_ptr() ??:? (vhds_integration_test+0x30d3867)
    #8 Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate() ??:? (vhds_integration_test+0x317c55a)
    #9 Envoy::Router::VhdsSubscription::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x31ab4f8)
    #10 Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x4397540)
    #11 non-virtual thunk to Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x439768c)
    #12 Envoy::Config::WatchMap::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x43cdfe1)
    #13 Envoy::Config::DeltaSubscriptionState::handleGoodResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1f1d)
    #14 Envoy::Config::DeltaSubscriptionState::handleResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1751)
    #15 Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439977a)
    #16 non-virtual thunk to Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439996f)
    #17 Envoy::Config::GrpcStream<envoy::service::discovery::v3::DeltaDiscoveryRequest, envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessage(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x294ffca)
    #18 Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&) ??:? (vhds_integration_test+0x294fca6)
    #19 Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420c3f)
    #20 non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420dcf)
    #21 Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4435301)
    #22 non-virtual thunk to Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x443540f)
    #23 Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::Filter::UpstreamRequest&, bool) ??:? (vhds_integration_test+0x445fc73)
    #24 Envoy::Router::Filter::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4463c0f)
    #25 Envoy::Http::StreamDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x2e561d9)
    #26 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) ??:? (vhds_integration_test+0x47ffa2a)
    #27 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::operator()(nghttp2_session*, nghttp2_frame const*, void*) const codec_impl.cc:? (vhds_integration_test+0x4808d20)
    #28 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::__invoke(nghttp2_session*, nghttp2_frame const*, void*) codec_impl.cc:? (vhds_integration_test+0x4808c98)
    #29 session_call_on_frame_received nghttp2_session.c:? (vhds_integration_test+0x4848541)
    #30 nghttp2_session_on_data_received ??:? (vhds_integration_test+0x484bec4)
    #31 session_process_data_frame nghttp2_session.c:? (vhds_integration_test+0x485859b)
    #32 nghttp2_session_mem_recv ??:? (vhds_integration_test+0x4853ae2)
    #33 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fd794)
    #34 virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fdce0)
    #35 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x434cb2c)
    #36 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4351d3c)
    #37 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) ??:? (vhds_integration_test+0x499eaa5)
    #38 Envoy::Network::FilterManagerImpl::onRead() ??:? (vhds_integration_test+0x499ed14)
    #39 Envoy::Network::ConnectionImpl::onRead(unsigned long) ??:? (vhds_integration_test+0x4985bf7)
    #40 Envoy::Network::ConnectionImpl::onReadReady() ??:? (vhds_integration_test+0x4989e30)
    #41 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) ??:? (vhds_integration_test+0x49899e6)
    #42 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2::operator()(unsigned int) const connection_impl.cc:? (vhds_integration_test+0x4992967)
    #43 _ZNSt3__18__invokeIRZN5Envoy7Network14ConnectionImplC1ERNS1_5Event10DispatcherEONS_10unique_ptrINS2_16ConnectionSocketENS_14default_deleteIS8_EEEEONS7_INS2_15TransportSocketENS9_ISD_EEEEbE3$_2JjEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSJ_DpOSK_ connection_impl.cc:? (vhds_integration_test+0x49928dc)
    #44 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499280d)
    #45 std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499278d)
    #46 std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x4990dcc)
    #47 std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const mocks.cc:? (vhds_integration_test+0x39e5d24)
    #48 std::__1::function<void (unsigned int)>::operator()(unsigned int) const ??:? (vhds_integration_test+0x39e5c18)
    #49 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::operator()(int, short, void*) const file_event_impl.cc:? (vhds_integration_test+0x4975beb)
    #50 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) file_event_impl.cc:? (vhds_integration_test+0x49759f6)
    #51 event_persist_closure event.c:? (vhds_integration_test+0x500c840)
    #52 event_process_active_single_queue event.c:? (vhds_integration_test+0x500b268)
    #53 event_process_active event.c:? (vhds_integration_test+0x5000bb8)
    #54 event_base_loop ??:? (vhds_integration_test+0x4ffe883)
    #55 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4acbd8f)
    #56 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4963f11)
    #57 Envoy::Server::InstanceImpl::run() ??:? (vhds_integration_test+0x265517b)
    #58 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >) ??:? (vhds_integration_test+0x24745e0)
    #59 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2474032)
    #60 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0::operator()() const server.cc:? (vhds_integration_test+0x2477690)
    #61 _ZNSt3__18__invokeIRZN5Envoy21IntegrationTestServer5startENS1_7Network7Address9IpVersionENS_8functionIFvvEEEbbN4absl8optionalINS_17reference_wrapperINS1_13ProcessObjectEEEEEbbjE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ server.cc:? (vhds_integration_test+0x2477530)
    #62 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&) server.cc:? (vhds_integration_test+0x2477490)
    #63 std::__1::__function::__alloc_func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2477430)
    #64 std::__1::__function::__func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2475ddf)
    #65 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #66 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #67 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const thread_impl.cc:? (vhds_integration_test+0x4fea0fb)
    #68 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) thread_impl.cc:? (vhds_integration_test+0x4fea088)

  Previous read of size 8 at 0x7b140000f0f0 by thread T6:
    #0 std::__1::__shared_count::__release_shared() vhds_integration_test.cc:? (vhds_integration_test+0x22803ef)
    #1 std::__1::__shared_weak_count::__release_shared() vhds_integration_test.cc:? (vhds_integration_test+0x2280372)
    #2 std::__1::shared_ptr<std::__1::function<void (bool)> >::~shared_ptr() ??:? (vhds_integration_test+0x301fd27)
    #3 Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::~OnDemandRouteUpdate() ??:? (vhds_integration_test+0x301fca5)
    #4 std::__1::__shared_ptr_emplace<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate, std::__1::allocator<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate> >::__on_zero_shared() ??:? (vhds_integration_test+0x301f92d)
    #5 std::__1::__shared_count::__release_shared() vhds_integration_test.cc:? (vhds_integration_test+0x22803fc)
    #6 std::__1::__shared_weak_count::__release_shared() vhds_integration_test.cc:? (vhds_integration_test+0x2280372)
    #7 std::__1::shared_ptr<Envoy::Http::StreamDecoderFilter>::~shared_ptr() ??:? (vhds_integration_test+0x2f1a0f7)
    #8 Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::~ActiveStreamDecoderFilter() ??:? (vhds_integration_test+0x30e4f00)
    #9 Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::~ActiveStreamDecoderFilter() ??:? (vhds_integration_test+0x30da896)
    #10 Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::~ActiveStreamDecoderFilter() ??:? (vhds_integration_test+0x30da8df)
    #11 std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter>::operator()(Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter*) const conn_manager_impl.cc:? (vhds_integration_test+0x30ec546)
    #12 std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >::reset(Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter*) conn_manager_impl.cc:? (vhds_integration_test+0x30ec460)
    #13 std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >::~unique_ptr() conn_manager_impl.cc:? (vhds_integration_test+0x30d590c)
    #14 void std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, void*> > >::__destroy<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> > >(std::__1::integral_constant<bool, false>, std::__1::allocator<std::__1::__list_node<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, void*> >&, std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >*) ??:? (vhds_integration_test+0x30e23b4)
    #15 void std::__1::allocator_traits<std::__1::allocator<std::__1::__list_node<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, void*> > >::destroy<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> > >(std::__1::allocator<std::__1::__list_node<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, void*> >&, std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >*) ??:? (vhds_integration_test+0x30e2128)
    #16 std::__1::__list_imp<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, std::__1::allocator<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> > > >::clear() ??:? (vhds_integration_test+0x30e1dfb)
    #17 std::__1::__list_imp<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, std::__1::allocator<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> > > >::~__list_imp() ??:? (vhds_integration_test+0x30e1cb8)
    #18 std::__1::list<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> >, std::__1::allocator<std::__1::unique_ptr<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter, std::__1::default_delete<Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter> > > >::~list() ??:? (vhds_integration_test+0x30d4bdb)
    #19 Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() ??:? (vhds_integration_test+0x3099312)
    #20 Envoy::Http::ConnectionManagerImpl::ActiveStream::~ActiveStream() ??:? (vhds_integration_test+0x309985f)
    #21 std::__1::default_delete<Envoy::Event::DeferredDeletable>::operator()(Envoy::Event::DeferredDeletable*) const connection_handler_impl.cc:? (vhds_integration_test+0x284fcc6)
    #22 std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> >::reset(Envoy::Event::DeferredDeletable*) connection_handler_impl.cc:? (vhds_integration_test+0x284fbe0)
    #23 Envoy::Event::DispatcherImpl::clearDeferredDeleteList() ??:? (vhds_integration_test+0x496218c)
    #24 Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0::operator()() const dispatcher_impl.cc:? (vhds_integration_test+0x4966492)
    #25 _ZNSt3__18__invokeIRZN5Envoy5Event14DispatcherImplC1EONS_10unique_ptrINS1_6Buffer16WatermarkFactoryENS_14default_deleteIS6_EEEERNS1_3Api3ApiERNS2_10TimeSystemEE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ dispatcher_impl.cc:? (vhds_integration_test+0x4966400)
    #26 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0&) dispatcher_impl.cc:? (vhds_integration_test+0x4966360)
    #27 std::__1::__function::__alloc_func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0>, void ()>::operator()() dispatcher_impl.cc:? (vhds_integration_test+0x4966300)
    #28 std::__1::__function::__func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_0>, void ()>::operator()() dispatcher_impl.cc:? (vhds_integration_test+0x496495f)
    #29 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #30 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #31 Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &event_base_free>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::operator()(int, short, void*) const timer_impl.cc:? (vhds_integration_test+0x4acdeb5)
    #32 Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &event_base_free>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::__invoke(int, short, void*) timer_impl.cc:? (vhds_integration_test+0x4acde16)
    #33 event_process_active_single_queue event.c:? (vhds_integration_test+0x500b45a)
    #34 event_process_active event.c:? (vhds_integration_test+0x5000bb8)
    #35 event_base_loop ??:? (vhds_integration_test+0x4ffe883)
    #36 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4acbd8f)
    #37 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4963f11)
    #38 Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&) ??:? (vhds_integration_test+0x2826970)
    #39 Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3::operator()() const worker_impl.cc:? (vhds_integration_test+0x28306a7)
    #40 _ZNSt3__18__invokeIRZN5Envoy6Server10WorkerImpl5startERNS2_8GuardDogEE3$_3JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS8_DpOS9_ worker_impl.cc:? (vhds_integration_test+0x2830610)
    #41 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3&>(Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3&) worker_impl.cc:? (vhds_integration_test+0x2830570)
    #42 std::__1::__function::__alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3>, void ()>::operator()() worker_impl.cc:? (vhds_integration_test+0x2830510)
    #43 std::__1::__function::__func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_3>, void ()>::operator()() worker_impl.cc:? (vhds_integration_test+0x282eb6f)
    #44 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #45 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #46 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const thread_impl.cc:? (vhds_integration_test+0x4fea0fb)
    #47 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) thread_impl.cc:? (vhds_integration_test+0x4fea088)

  Thread T7 (tid=106, running) created by main thread at:
    #0 pthread_create ??:? (vhds_integration_test+0x21de78b)
    #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9bfe)
    #2 std::__1::__unique_if<Envoy::Thread::ThreadImplPosix>::__unique_single std::__1::make_unique<Envoy::Thread::ThreadImplPosix, std::__1::function<void ()>&>(std::__1::function<void ()>&) thread_impl.cc:? (vhds_integration_test+0x4fea3ee)
    #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9f93)
    #4 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2472c9d)
    #5 Envoy::IntegrationTestServer::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Network::Address::IpVersion, std::__1::function<void (Envoy::IntegrationTestServer&)>, std::__1::function<void ()>, bool, Envoy::Event::TestTimeSystem&, Envoy::Api::Api&, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2472849)
    #6 Envoy::BaseIntegrationTest::createGeneratedApiTestServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, bool, bool) ??:? (vhds_integration_test+0x23d50e3)
    #7 Envoy::BaseIntegrationTest::createEnvoy() ??:? (vhds_integration_test+0x23d4ced)
    #8 Envoy::BaseIntegrationTest::initialize() ??:? (vhds_integration_test+0x23d36d4)
    #9 Envoy::(anonymous namespace)::VhdsIntegrationTest::initialize() vhds_integration_test.cc:? (vhds_integration_test+0x2258771)
    #10 Envoy::HttpIntegrationTest::makeHeaderOnlyRequest(std::__1::function<std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> > ()>*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x22db056)
    #11 Envoy::HttpIntegrationTest::testRouterHeaderOnlyRequestAndResponse(std::__1::function<std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> > ()>*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x22db580)
    #12 Envoy::(anonymous namespace)::VhdsIntegrationTest_VhdsVirtualHostAddUpdateRemove_Test::TestBody() vhds_integration_test.cc:? (vhds_integration_test+0x225bb24)
    #13 non-virtual thunk to Envoy::(anonymous namespace)::VhdsIntegrationTest_VhdsVirtualHostAddUpdateRemove_Test::TestBody() vhds_integration_test.cc:? (vhds_integration_test+0x225e142)
    #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (vhds_integration_test+0x67ad5dc)
    #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (vhds_integration_test+0x679341e)
    #16 testing::Test::Run() ??:? (vhds_integration_test+0x6778b31)
    #17 testing::TestInfo::Run() ??:? (vhds_integration_test+0x67799d3)
    #18 testing::TestSuite::Run() ??:? (vhds_integration_test+0x677a3be)
    #19 testing::internal::UnitTestImpl::RunAllTests() ??:? (vhds_integration_test+0x678966d)
    #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (vhds_integration_test+0x67b3cfc)
    #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (vhds_integration_test+0x679713e)
    #22 testing::UnitTest::Run() ??:? (vhds_integration_test+0x6788f2b)
    #23 RUN_ALL_TESTS() ??:? (vhds_integration_test+0x47f5847)
    #24 Envoy::TestRunner::RunTests(int, char**) ??:? (vhds_integration_test+0x47f4ba6)
    #25 main ??:? (vhds_integration_test+0x47f201a)

  Thread T6 (tid=112, running) created by thread T7 at:
    #0 pthread_create ??:? (vhds_integration_test+0x21de78b)
    #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9bfe)
    #2 std::__1::__unique_if<Envoy::Thread::ThreadImplPosix>::__unique_single std::__1::make_unique<Envoy::Thread::ThreadImplPosix, std::__1::function<void ()>&>(std::__1::function<void ()>&) thread_impl.cc:? (vhds_integration_test+0x4fea3ee)
    #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9f93)
    #4 Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&) ??:? (vhds_integration_test+0x2826142)
    #5 Envoy::Server::ListenerManagerImpl::startWorkers(Envoy::Server::GuardDog&) ??:? (vhds_integration_test+0x2714e47)
    #6 Envoy::Server::InstanceImpl::startWorkers() ??:? (vhds_integration_test+0x2653dd1)
    #7 Envoy::Server::InstanceImpl::run()::$_11::operator()() const server.cc:? (vhds_integration_test+0x2671941)
    #8 _ZNSt3__18__invokeIRZN5Envoy6Server12InstanceImpl3runEvE4$_11JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ server.cc:? (vhds_integration_test+0x2671880)
    #9 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::InstanceImpl::run()::$_11&>(Envoy::Server::InstanceImpl::run()::$_11&) server.cc:? (vhds_integration_test+0x26717e0)
    #10 std::__1::__function::__alloc_func<Envoy::Server::InstanceImpl::run()::$_11, std::__1::allocator<Envoy::Server::InstanceImpl::run()::$_11>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2671780)
    #11 std::__1::__function::__func<Envoy::Server::InstanceImpl::run()::$_11, std::__1::allocator<Envoy::Server::InstanceImpl::run()::$_11>, void ()>::operator()() server.cc:? (vhds_integration_test+0x266fddf)
    #12 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #13 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #14 Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5::operator()() const server.cc:? (vhds_integration_test+0x2663f9d)
    #15 _ZNSt3__18__invokeIRZN5Envoy6Server9RunHelperC1ERNS2_8InstanceERKNS2_7OptionsERNS1_5Event10DispatcherERNS1_8Upstream14ClusterManagerERNS1_9AccessLog16AccessLogManagerERNS1_4Init7ManagerERNS2_15OverloadManagerENS_8functionIFvvEEEE3$_5JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSS_DpOST_ server.cc:? (vhds_integration_test+0x2663ee0)
    #16 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5&>(Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5&) server.cc:? (vhds_integration_test+0x2663e40)
    #17 std::__1::__function::__alloc_func<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5, std::__1::allocator<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2663de0)
    #18 std::__1::__function::__func<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5, std::__1::allocator<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5>, void ()>::operator()() server.cc:? (vhds_integration_test+0x26624af)
    #19 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #20 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #21 Envoy::Init::WatcherHandleImpl::ready() const ??:? (vhds_integration_test+0x4734f05)
    #22 Envoy::Init::ManagerImpl::ready() ??:? (vhds_integration_test+0x472fd79)
    #23 Envoy::Init::ManagerImpl::onTargetReady() ??:? (vhds_integration_test+0x472ff14)
    #24 Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0::operator()() const manager_impl.cc:? (vhds_integration_test+0x473220b)
    #25 _ZNSt3__18__invokeIRZN5Envoy4Init11ManagerImplC1EN4absl11string_viewEE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS8_DpOS9_ manager_impl.cc:? (vhds_integration_test+0x4732190)
    #26 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0&>(Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0&) manager_impl.cc:? (vhds_integration_test+0x47320f0)
    #27 std::__1::__function::__alloc_func<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0, std::__1::allocator<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0>, void ()>::operator()() manager_impl.cc:? (vhds_integration_test+0x4732090)
    #28 std::__1::__function::__func<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0, std::__1::allocator<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0>, void ()>::operator()() manager_impl.cc:? (vhds_integration_test+0x47306ef)
    #29 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #30 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #31 Envoy::Init::WatcherHandleImpl::ready() const ??:? (vhds_integration_test+0x4734f05)
    #32 Envoy::Init::TargetImpl::ready() ??:? (vhds_integration_test+0x4c946c2)
    #33 Envoy::Router::VhdsSubscription::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x31ab525)
    #34 Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x4397540)
    #35 non-virtual thunk to Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x439768c)
    #36 Envoy::Config::WatchMap::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x43cdfe1)
    #37 Envoy::Config::DeltaSubscriptionState::handleGoodResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1f1d)
    #38 Envoy::Config::DeltaSubscriptionState::handleResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1751)
    #39 Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439977a)
    #40 non-virtual thunk to Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439996f)
    #41 Envoy::Config::GrpcStream<envoy::service::discovery::v3::DeltaDiscoveryRequest, envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessage(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x294ffca)
    #42 Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&) ??:? (vhds_integration_test+0x294fca6)
    #43 Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420c3f)
    #44 non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420dcf)
    #45 Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4435301)
    #46 non-virtual thunk to Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x443540f)
    #47 Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::Filter::UpstreamRequest&, bool) ??:? (vhds_integration_test+0x445fc73)
    #48 Envoy::Router::Filter::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4463c0f)
    #49 Envoy::Http::StreamDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x2e561d9)
    #50 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) ??:? (vhds_integration_test+0x47ffa2a)
    #51 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::operator()(nghttp2_session*, nghttp2_frame const*, void*) const codec_impl.cc:? (vhds_integration_test+0x4808d20)
    #52 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::__invoke(nghttp2_session*, nghttp2_frame const*, void*) codec_impl.cc:? (vhds_integration_test+0x4808c98)
    #53 session_call_on_frame_received nghttp2_session.c:? (vhds_integration_test+0x4848541)
    #54 nghttp2_session_on_data_received ??:? (vhds_integration_test+0x484bec4)
    #55 session_process_data_frame nghttp2_session.c:? (vhds_integration_test+0x485859b)
    #56 nghttp2_session_mem_recv ??:? (vhds_integration_test+0x4853ae2)
    #57 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fd794)
    #58 virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fdce0)
    #59 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x434cb2c)
    #60 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4351d3c)
    #61 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) ??:? (vhds_integration_test+0x499eaa5)
    #62 Envoy::Network::FilterManagerImpl::onRead() ??:? (vhds_integration_test+0x499ed14)
    #63 Envoy::Network::ConnectionImpl::onRead(unsigned long) ??:? (vhds_integration_test+0x4985bf7)
    #64 Envoy::Network::ConnectionImpl::onReadReady() ??:? (vhds_integration_test+0x4989e30)
    #65 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) ??:? (vhds_integration_test+0x49899e6)
    #66 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2::operator()(unsigned int) const connection_impl.cc:? (vhds_integration_test+0x4992967)
    #67 _ZNSt3__18__invokeIRZN5Envoy7Network14ConnectionImplC1ERNS1_5Event10DispatcherEONS_10unique_ptrINS2_16ConnectionSocketENS_14default_deleteIS8_EEEEONS7_INS2_15TransportSocketENS9_ISD_EEEEbE3$_2JjEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSJ_DpOSK_ connection_impl.cc:? (vhds_integration_test+0x49928dc)
    #68 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499280d)
    #69 std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499278d)
    #70 std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x4990dcc)
    #71 std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const mocks.cc:? (vhds_integration_test+0x39e5d24)
    #72 std::__1::function<void (unsigned int)>::operator()(unsigned int) const ??:? (vhds_integration_test+0x39e5c18)
    #73 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::operator()(int, short, void*) const file_event_impl.cc:? (vhds_integration_test+0x4975beb)
    #74 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) file_event_impl.cc:? (vhds_integration_test+0x49759f6)
    #75 event_persist_closure event.c:? (vhds_integration_test+0x500c840)
    #76 event_process_active_single_queue event.c:? (vhds_integration_test+0x500b268)
    #77 event_process_active event.c:? (vhds_integration_test+0x5000bb8)
    #78 event_base_loop ??:? (vhds_integration_test+0x4ffe883)
    #79 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4acbd8f)
    #80 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4963f11)
    #81 Envoy::Server::InstanceImpl::run() ??:? (vhds_integration_test+0x265517b)
    #82 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >) ??:? (vhds_integration_test+0x24745e0)
    #83 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2474032)
    #84 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0::operator()() const server.cc:? (vhds_integration_test+0x2477690)
    #85 _ZNSt3__18__invokeIRZN5Envoy21IntegrationTestServer5startENS1_7Network7Address9IpVersionENS_8functionIFvvEEEbbN4absl8optionalINS_17reference_wrapperINS1_13ProcessObjectEEEEEbbjE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ server.cc:? (vhds_integration_test+0x2477530)
    #86 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&) server.cc:? (vhds_integration_test+0x2477490)
    #87 std::__1::__function::__alloc_func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2477430)
    #88 std::__1::__function::__func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2475ddf)
    #89 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #90 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #91 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const thread_impl.cc:? (vhds_integration_test+0x4fea0fb)
    #92 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) thread_impl.cc:? (vhds_integration_test+0x4fea088)

SUMMARY: ThreadSanitizer: data race ??:? in __interceptor_free
==================
==================
WARNING: ThreadSanitizer: data race (pid=14)
  Write of size 8 at 0x7b140000f0f8 by thread T7:
    #0 free ??:? (vhds_integration_test+0x21dd4f8)
    #1 std::__1::_DeallocateCaller::__do_call(void*) ??:? (vhds_integration_test+0x2273fb8)
    #2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) ??:? (vhds_integration_test+0x2273f84)
    #3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) vhds_integration_test.cc:? (vhds_integration_test+0x2273f24)
    #4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) vhds_integration_test.cc:? (vhds_integration_test+0x2273eb8)
    #5 std::__1::allocator<std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > > >::deallocate(std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >*, unsigned long) on_demand_update.cc:? (vhds_integration_test+0x3041fa1)
    #6 std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >::__on_zero_shared_weak() ??:? (vhds_integration_test+0x30419b5)
    #7 std::__1::weak_ptr<std::__1::function<void (bool)> >::~weak_ptr() ??:? (vhds_integration_test+0x30d3867)
    #8 Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate() ??:? (vhds_integration_test+0x317c55a)
    #9 Envoy::Router::VhdsSubscription::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x31ab4f8)
    #10 Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x4397540)
    #11 non-virtual thunk to Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x439768c)
    #12 Envoy::Config::WatchMap::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x43cdfe1)
    #13 Envoy::Config::DeltaSubscriptionState::handleGoodResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1f1d)
    #14 Envoy::Config::DeltaSubscriptionState::handleResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1751)
    #15 Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439977a)
    #16 non-virtual thunk to Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439996f)
    #17 Envoy::Config::GrpcStream<envoy::service::discovery::v3::DeltaDiscoveryRequest, envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessage(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x294ffca)
    #18 Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&) ??:? (vhds_integration_test+0x294fca6)
    #19 Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420c3f)
    #20 non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420dcf)
    #21 Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4435301)
    #22 non-virtual thunk to Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x443540f)
    #23 Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::Filter::UpstreamRequest&, bool) ??:? (vhds_integration_test+0x445fc73)
    #24 Envoy::Router::Filter::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4463c0f)
    #25 Envoy::Http::StreamDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x2e561d9)
    #26 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) ??:? (vhds_integration_test+0x47ffa2a)
    #27 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::operator()(nghttp2_session*, nghttp2_frame const*, void*) const codec_impl.cc:? (vhds_integration_test+0x4808d20)
    #28 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::__invoke(nghttp2_session*, nghttp2_frame const*, void*) codec_impl.cc:? (vhds_integration_test+0x4808c98)
    #29 session_call_on_frame_received nghttp2_session.c:? (vhds_integration_test+0x4848541)
    #30 nghttp2_session_on_data_received ??:? (vhds_integration_test+0x484bec4)
    #31 session_process_data_frame nghttp2_session.c:? (vhds_integration_test+0x485859b)
    #32 nghttp2_session_mem_recv ??:? (vhds_integration_test+0x4853ae2)
    #33 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fd794)
    #34 virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fdce0)
    #35 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x434cb2c)
    #36 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4351d3c)
    #37 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) ??:? (vhds_integration_test+0x499eaa5)
    #38 Envoy::Network::FilterManagerImpl::onRead() ??:? (vhds_integration_test+0x499ed14)
    #39 Envoy::Network::ConnectionImpl::onRead(unsigned long) ??:? (vhds_integration_test+0x4985bf7)
    #40 Envoy::Network::ConnectionImpl::onReadReady() ??:? (vhds_integration_test+0x4989e30)
    #41 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) ??:? (vhds_integration_test+0x49899e6)
    #42 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2::operator()(unsigned int) const connection_impl.cc:? (vhds_integration_test+0x4992967)
    #43 _ZNSt3__18__invokeIRZN5Envoy7Network14ConnectionImplC1ERNS1_5Event10DispatcherEONS_10unique_ptrINS2_16ConnectionSocketENS_14default_deleteIS8_EEEEONS7_INS2_15TransportSocketENS9_ISD_EEEEbE3$_2JjEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSJ_DpOSK_ connection_impl.cc:? (vhds_integration_test+0x49928dc)
    #44 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499280d)
    #45 std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499278d)
    #46 std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x4990dcc)
    #47 std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const mocks.cc:? (vhds_integration_test+0x39e5d24)
    #48 std::__1::function<void (unsigned int)>::operator()(unsigned int) const ??:? (vhds_integration_test+0x39e5c18)
    #49 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::operator()(int, short, void*) const file_event_impl.cc:? (vhds_integration_test+0x4975beb)
    #50 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) file_event_impl.cc:? (vhds_integration_test+0x49759f6)
    #51 event_persist_closure event.c:? (vhds_integration_test+0x500c840)
    #52 event_process_active_single_queue event.c:? (vhds_integration_test+0x500b268)
    #53 event_process_active event.c:? (vhds_integration_test+0x5000bb8)
    #54 event_base_loop ??:? (vhds_integration_test+0x4ffe883)
    #55 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4acbd8f)
    #56 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4963f11)
    #57 Envoy::Server::InstanceImpl::run() ??:? (vhds_integration_test+0x265517b)
    #58 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >) ??:? (vhds_integration_test+0x24745e0)
    #59 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2474032)
    #60 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0::operator()() const server.cc:? (vhds_integration_test+0x2477690)
    #61 _ZNSt3__18__invokeIRZN5Envoy21IntegrationTestServer5startENS1_7Network7Address9IpVersionENS_8functionIFvvEEEbbN4absl8optionalINS_17reference_wrapperINS1_13ProcessObjectEEEEEbbjE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ server.cc:? (vhds_integration_test+0x2477530)
    #62 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&) server.cc:? (vhds_integration_test+0x2477490)
    #63 std::__1::__function::__alloc_func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2477430)
    #64 std::__1::__function::__func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2475ddf)
    #65 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #66 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #67 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const thread_impl.cc:? (vhds_integration_test+0x4fea0fb)
    #68 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) thread_impl.cc:? (vhds_integration_test+0x4fea088)

  Previous atomic write of size 8 at 0x7b140000f0f8 by thread T6:
    [failed to restore the stack]

  Thread T7 (tid=106, running) created by main thread at:
    #0 pthread_create ??:? (vhds_integration_test+0x21de78b)
    #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9bfe)
    #2 std::__1::__unique_if<Envoy::Thread::ThreadImplPosix>::__unique_single std::__1::make_unique<Envoy::Thread::ThreadImplPosix, std::__1::function<void ()>&>(std::__1::function<void ()>&) thread_impl.cc:? (vhds_integration_test+0x4fea3ee)
    #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9f93)
    #4 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2472c9d)
    #5 Envoy::IntegrationTestServer::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, Envoy::Network::Address::IpVersion, std::__1::function<void (Envoy::IntegrationTestServer&)>, std::__1::function<void ()>, bool, Envoy::Event::TestTimeSystem&, Envoy::Api::Api&, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2472849)
    #6 Envoy::BaseIntegrationTest::createGeneratedApiTestServer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, bool, bool) ??:? (vhds_integration_test+0x23d50e3)
    #7 Envoy::BaseIntegrationTest::createEnvoy() ??:? (vhds_integration_test+0x23d4ced)
    #8 Envoy::BaseIntegrationTest::initialize() ??:? (vhds_integration_test+0x23d36d4)
    #9 Envoy::(anonymous namespace)::VhdsIntegrationTest::initialize() vhds_integration_test.cc:? (vhds_integration_test+0x2258771)
    #10 Envoy::HttpIntegrationTest::makeHeaderOnlyRequest(std::__1::function<std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> > ()>*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x22db056)
    #11 Envoy::HttpIntegrationTest::testRouterHeaderOnlyRequestAndResponse(std::__1::function<std::__1::unique_ptr<Envoy::Network::ClientConnection, std::__1::default_delete<Envoy::Network::ClientConnection> > ()>*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x22db580)
    #12 Envoy::(anonymous namespace)::VhdsIntegrationTest_VhdsVirtualHostAddUpdateRemove_Test::TestBody() vhds_integration_test.cc:? (vhds_integration_test+0x225bb24)
    #13 non-virtual thunk to Envoy::(anonymous namespace)::VhdsIntegrationTest_VhdsVirtualHostAddUpdateRemove_Test::TestBody() vhds_integration_test.cc:? (vhds_integration_test+0x225e142)
    #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (vhds_integration_test+0x67ad5dc)
    #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ??:? (vhds_integration_test+0x679341e)
    #16 testing::Test::Run() ??:? (vhds_integration_test+0x6778b31)
    #17 testing::TestInfo::Run() ??:? (vhds_integration_test+0x67799d3)
    #18 testing::TestSuite::Run() ??:? (vhds_integration_test+0x677a3be)
    #19 testing::internal::UnitTestImpl::RunAllTests() ??:? (vhds_integration_test+0x678966d)
    #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (vhds_integration_test+0x67b3cfc)
    #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ??:? (vhds_integration_test+0x679713e)
    #22 testing::UnitTest::Run() ??:? (vhds_integration_test+0x6788f2b)
    #23 RUN_ALL_TESTS() ??:? (vhds_integration_test+0x47f5847)
    #24 Envoy::TestRunner::RunTests(int, char**) ??:? (vhds_integration_test+0x47f4ba6)
    #25 main ??:? (vhds_integration_test+0x47f201a)

  Thread T6 (tid=112, running) created by thread T7 at:
    #0 pthread_create ??:? (vhds_integration_test+0x21de78b)
    #1 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9bfe)
    #2 std::__1::__unique_if<Envoy::Thread::ThreadImplPosix>::__unique_single std::__1::make_unique<Envoy::Thread::ThreadImplPosix, std::__1::function<void ()>&>(std::__1::function<void ()>&) thread_impl.cc:? (vhds_integration_test+0x4fea3ee)
    #3 Envoy::Thread::ThreadFactoryImplPosix::createThread(std::__1::function<void ()>) ??:? (vhds_integration_test+0x4fe9f93)
    #4 Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&) ??:? (vhds_integration_test+0x2826142)
    #5 Envoy::Server::ListenerManagerImpl::startWorkers(Envoy::Server::GuardDog&) ??:? (vhds_integration_test+0x2714e47)
    #6 Envoy::Server::InstanceImpl::startWorkers() ??:? (vhds_integration_test+0x2653dd1)
    #7 Envoy::Server::InstanceImpl::run()::$_11::operator()() const server.cc:? (vhds_integration_test+0x2671941)
    #8 _ZNSt3__18__invokeIRZN5Envoy6Server12InstanceImpl3runEvE4$_11JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ server.cc:? (vhds_integration_test+0x2671880)
    #9 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::InstanceImpl::run()::$_11&>(Envoy::Server::InstanceImpl::run()::$_11&) server.cc:? (vhds_integration_test+0x26717e0)
    #10 std::__1::__function::__alloc_func<Envoy::Server::InstanceImpl::run()::$_11, std::__1::allocator<Envoy::Server::InstanceImpl::run()::$_11>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2671780)
    #11 std::__1::__function::__func<Envoy::Server::InstanceImpl::run()::$_11, std::__1::allocator<Envoy::Server::InstanceImpl::run()::$_11>, void ()>::operator()() server.cc:? (vhds_integration_test+0x266fddf)
    #12 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #13 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #14 Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5::operator()() const server.cc:? (vhds_integration_test+0x2663f9d)
    #15 _ZNSt3__18__invokeIRZN5Envoy6Server9RunHelperC1ERNS2_8InstanceERKNS2_7OptionsERNS1_5Event10DispatcherERNS1_8Upstream14ClusterManagerERNS1_9AccessLog16AccessLogManagerERNS1_4Init7ManagerERNS2_15OverloadManagerENS_8functionIFvvEEEE3$_5JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSS_DpOST_ server.cc:? (vhds_integration_test+0x2663ee0)
    #16 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5&>(Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5&) server.cc:? (vhds_integration_test+0x2663e40)
    #17 std::__1::__function::__alloc_func<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5, std::__1::allocator<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2663de0)
    #18 std::__1::__function::__func<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5, std::__1::allocator<Envoy::Server::RunHelper::RunHelper(Envoy::Server::Instance&, Envoy::Server::Options const&, Envoy::Event::Dispatcher&, Envoy::Upstream::ClusterManager&, Envoy::AccessLog::AccessLogManager&, Envoy::Init::Manager&, Envoy::Server::OverloadManager&, std::__1::function<void ()>)::$_5>, void ()>::operator()() server.cc:? (vhds_integration_test+0x26624af)
    #19 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #20 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #21 Envoy::Init::WatcherHandleImpl::ready() const ??:? (vhds_integration_test+0x4734f05)
    #22 Envoy::Init::ManagerImpl::ready() ??:? (vhds_integration_test+0x472fd79)
    #23 Envoy::Init::ManagerImpl::onTargetReady() ??:? (vhds_integration_test+0x472ff14)
    #24 Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0::operator()() const manager_impl.cc:? (vhds_integration_test+0x473220b)
    #25 _ZNSt3__18__invokeIRZN5Envoy4Init11ManagerImplC1EN4absl11string_viewEE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS8_DpOS9_ manager_impl.cc:? (vhds_integration_test+0x4732190)
    #26 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0&>(Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0&) manager_impl.cc:? (vhds_integration_test+0x47320f0)
    #27 std::__1::__function::__alloc_func<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0, std::__1::allocator<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0>, void ()>::operator()() manager_impl.cc:? (vhds_integration_test+0x4732090)
    #28 std::__1::__function::__func<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0, std::__1::allocator<Envoy::Init::ManagerImpl::ManagerImpl(absl::string_view)::$_0>, void ()>::operator()() manager_impl.cc:? (vhds_integration_test+0x47306ef)
    #29 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #30 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #31 Envoy::Init::WatcherHandleImpl::ready() const ??:? (vhds_integration_test+0x4734f05)
    #32 Envoy::Init::TargetImpl::ready() ??:? (vhds_integration_test+0x4c946c2)
    #33 Envoy::Router::VhdsSubscription::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x31ab525)
    #34 Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x4397540)
    #35 non-virtual thunk to Envoy::Config::DeltaSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x439768c)
    #36 Envoy::Config::WatchMap::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ??:? (vhds_integration_test+0x43cdfe1)
    #37 Envoy::Config::DeltaSubscriptionState::handleGoodResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1f1d)
    #38 Envoy::Config::DeltaSubscriptionState::handleResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) ??:? (vhds_integration_test+0x43b1751)
    #39 Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439977a)
    #40 non-virtual thunk to Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x439996f)
    #41 Envoy::Config::GrpcStream<envoy::service::discovery::v3::DeltaDiscoveryRequest, envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessage(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) ??:? (vhds_integration_test+0x294ffca)
    #42 Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&) ??:? (vhds_integration_test+0x294fca6)
    #43 Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420c3f)
    #44 non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4420dcf)
    #45 Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4435301)
    #46 non-virtual thunk to Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x443540f)
    #47 Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::Filter::UpstreamRequest&, bool) ??:? (vhds_integration_test+0x445fc73)
    #48 Envoy::Router::Filter::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4463c0f)
    #49 Envoy::Http::StreamDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x2e561d9)
    #50 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) ??:? (vhds_integration_test+0x47ffa2a)
    #51 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::operator()(nghttp2_session*, nghttp2_frame const*, void*) const codec_impl.cc:? (vhds_integration_test+0x4808d20)
    #52 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_11::__invoke(nghttp2_session*, nghttp2_frame const*, void*) codec_impl.cc:? (vhds_integration_test+0x4808c98)
    #53 session_call_on_frame_received nghttp2_session.c:? (vhds_integration_test+0x4848541)
    #54 nghttp2_session_on_data_received ??:? (vhds_integration_test+0x484bec4)
    #55 session_process_data_frame nghttp2_session.c:? (vhds_integration_test+0x485859b)
    #56 nghttp2_session_mem_recv ??:? (vhds_integration_test+0x4853ae2)
    #57 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fd794)
    #58 virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x47fdce0)
    #59 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) ??:? (vhds_integration_test+0x434cb2c)
    #60 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) ??:? (vhds_integration_test+0x4351d3c)
    #61 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) ??:? (vhds_integration_test+0x499eaa5)
    #62 Envoy::Network::FilterManagerImpl::onRead() ??:? (vhds_integration_test+0x499ed14)
    #63 Envoy::Network::ConnectionImpl::onRead(unsigned long) ??:? (vhds_integration_test+0x4985bf7)
    #64 Envoy::Network::ConnectionImpl::onReadReady() ??:? (vhds_integration_test+0x4989e30)
    #65 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) ??:? (vhds_integration_test+0x49899e6)
    #66 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2::operator()(unsigned int) const connection_impl.cc:? (vhds_integration_test+0x4992967)
    #67 _ZNSt3__18__invokeIRZN5Envoy7Network14ConnectionImplC1ERNS1_5Event10DispatcherEONS_10unique_ptrINS2_16ConnectionSocketENS_14default_deleteIS8_EEEEONS7_INS2_15TransportSocketENS9_ISD_EEEEbE3$_2JjEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSJ_DpOSK_ connection_impl.cc:? (vhds_integration_test+0x49928dc)
    #68 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2&, unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499280d)
    #69 std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x499278d)
    #70 std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) connection_impl.cc:? (vhds_integration_test+0x4990dcc)
    #71 std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const mocks.cc:? (vhds_integration_test+0x39e5d24)
    #72 std::__1::function<void (unsigned int)>::operator()(unsigned int) const ??:? (vhds_integration_test+0x39e5c18)
    #73 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::operator()(int, short, void*) const file_event_impl.cc:? (vhds_integration_test+0x4975beb)
    #74 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) file_event_impl.cc:? (vhds_integration_test+0x49759f6)
    #75 event_persist_closure event.c:? (vhds_integration_test+0x500c840)
    #76 event_process_active_single_queue event.c:? (vhds_integration_test+0x500b268)
    #77 event_process_active event.c:? (vhds_integration_test+0x5000bb8)
    #78 event_base_loop ??:? (vhds_integration_test+0x4ffe883)
    #79 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4acbd8f)
    #80 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) ??:? (vhds_integration_test+0x4963f11)
    #81 Envoy::Server::InstanceImpl::run() ??:? (vhds_integration_test+0x265517b)
    #82 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >) ??:? (vhds_integration_test+0x24745e0)
    #83 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int) ??:? (vhds_integration_test+0x2474032)
    #84 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0::operator()() const server.cc:? (vhds_integration_test+0x2477690)
    #85 _ZNSt3__18__invokeIRZN5Envoy21IntegrationTestServer5startENS1_7Network7Address9IpVersionENS_8functionIFvvEEEbbN4absl8optionalINS_17reference_wrapperINS1_13ProcessObjectEEEEEbbjE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ server.cc:? (vhds_integration_test+0x2477530)
    #86 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0&) server.cc:? (vhds_integration_test+0x2477490)
    #87 std::__1::__function::__alloc_func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2477430)
    #88 std::__1::__function::__func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, bool, bool, unsigned int)::$_0>, void ()>::operator()() server.cc:? (vhds_integration_test+0x2475ddf)
    #89 std::__1::__function::__value_func<void ()>::operator()() const autonomous_upstream.cc:? (vhds_integration_test+0x235e766)
    #90 std::__1::function<void ()>::operator()() const ??:? (vhds_integration_test+0x235e428)
    #91 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const thread_impl.cc:? (vhds_integration_test+0x4fea0fb)
    #92 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) thread_impl.cc:? (vhds_integration_test+0x4fea088)

SUMMARY: ThreadSanitizer: data race ??:? in __interceptor_free
==================
@mattklein123
Copy link
Member

@dmitri-d PTAL

@dmitri-d
Copy link
Contributor

looking into it...

@dmitri-d dmitri-d mentioned this issue Jan 24, 2020
Closed
@stale
Copy link

stale bot commented Feb 23, 2020

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

@stale stale bot added the stale stalebot believes this issue/PR has not been touched recently label Feb 23, 2020
@stale
Copy link

stale bot commented Mar 1, 2020

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

@stale stale bot closed this as completed Mar 1, 2020
@stale
Copy link

stale bot commented Mar 8, 2020

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

@jmarantz
Copy link
Contributor Author

jmarantz commented Apr 5, 2020 •
edited
Loading

ran into this again with a local tsan test.

@jmarantz jmarantz reopened this Apr 5, 2020
@stale stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Apr 5, 2020
@jmarantz jmarantz added the no stalebot Disables stalebot from closing an issue label Apr 5, 2020
@dmitri-d
Copy link
Contributor

dmitri-d commented Apr 6, 2020

Please see my comment here: #9828 (comment)

@alphashr alphashr mentioned this issue Apr 23, 2020
Merged
3 tasks
@dmitri-d dmitri-d mentioned this issue May 21, 2020
Closed
@alyssawilk
Copy link
Contributor

I've yet to see tsan fail when there wasn't a race. Looks like there might be a teardown issue? I'd suggest adding a test where we kill the request and tear down the filter when there's a route request in progress. I suspect that'll make the failure mode more clear - if not I'll agree to take a whack at it.

@mattklein123
Copy link
Member

So, apologies, but I think I actually noticed this issue months ago and meant to open an issue but never did. AFAICT the on demand filter does not handle onDestroy(). It has to have some facility to cancel an inflight route update request if the stream gets reset. This is definitely a real bug. Please add an integration test for this case and then we can fix. Thanks!

@mattklein123 mattklein123 added this to the 1.15.0 milestone May 27, 2020
@dmitri-d dmitri-d mentioned this issue May 28, 2020
Merged
@dmitri-d
Copy link
Contributor

Opened #11341 which adds a test to verify vhds updates when client closes its connection. In an attempt to make things a bit more deterministic when client connection is closed midflight, also added an implementation for onDestroy().

However, vhds as currently implemented already handles client connections that disappear before/during vhds update propagation: the filter-chain of the active stream is paused in the on-demand update filter for the duration of the on-demand update request-response cycle. The filter-chain is resumed once the on-demand vhds update has been propagated. This is done via a callback to the on-demand filter, which is stored in RdsRouteConfigProviderImpl in a weak_ptr that is evaluated before invocation (here: https://github.com/envoyproxy/envoy/blob/master/source/common/router/rds_impl.cc#L288).

If I'm reading the thread sanitizer report correctly, it's detecting a conflict between a weak_ptr and its shared_ptr: the shared_ptr (callback stored in the on-demand filter) reading its counter in the destructor, and the weak_ptr is updating the counter in its destructor.

@mattklein123, @alyssawilk.

@dmitri-d
Copy link
Contributor

Slightly different race reported here (and with line-numbers). This time between the original shared_ptr and the weak_ptr destructor.

WARNING: ThreadSanitizer: data race (pid=14)
  Write of size 8 at 0x7b1400031268 by thread T24:
    #0 free /local/mnt/workspace/bcain_0721/llvm/utils/release/final/llvm.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706:3 (vhds_integration_test+0x2863578)
    #1 std::__1::_DeallocateCaller::__do_call(void*) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/new:334:12 (vhds_integration_test+0x291e148)
    #2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/new:292:12 (vhds_integration_test+0x291e114)
    #3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/new:262:12 (vhds_integration_test+0x291e0b4)
    #4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/new:340:3 (vhds_integration_test+0x291e048)
    #5 std::__1::allocator<std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > > >::deallocate(std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >*, unsigned long) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/memory:1816:10 (vhds_integration_test+0x39b1c41)
    #6 std::__1::__shared_ptr_emplace<std::__1::function<void (bool)>, std::__1::allocator<std::__1::function<void (bool)> > >::__on_zero_shared_weak() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/memory:3593:9 (vhds_integration_test+0x39b1535)
    #7 std::__1::weak_ptr<std::__1::function<void (bool)> >::~weak_ptr() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/memory:5037:19 (vhds_integration_test+0x3a67597)
    #8 Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate() /proc/self/cwd/source/common/router/rds_impl.cc:293:5 (vhds_integration_test+0x3b251ba)
    #9 Envoy::Router::VhdsSubscription::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /proc/self/cwd/source/common/router/vhds.cc:72:17 (vhds_integration_test+0x3b5ddf2)
    #10 Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /proc/self/cwd/source/common/config/grpc_subscription_impl.cc:78:14 (vhds_integration_test+0x4f6f257)
    #11 non-virtual thunk to Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /proc/self/cwd/source/common/config/grpc_subscription_impl.cc (vhds_integration_test+0x4f6f4bc)
    #12 Envoy::Config::WatchMap::onConfigUpdate(google::protobuf::RepeatedPtrField<envoy::service::discovery::v3::Resource> const&, google::protobuf::RepeatedPtrField<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /proc/self/cwd/source/common/config/watch_map.cc:153:29 (vhds_integration_test+0x5275151)
    #13 Envoy::Config::DeltaSubscriptionState::handleGoodResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) /proc/self/cwd/source/common/config/delta_subscription_state.cc:84:14 (vhds_integration_test+0x4fb68d0)
    #14 Envoy::Config::DeltaSubscriptionState::handleResponse(envoy::service::discovery::v3::DeltaDiscoveryResponse const&) /proc/self/cwd/source/common/config/delta_subscription_state.cc:52:5 (vhds_integration_test+0x4fb6111)
    #15 Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) /proc/self/cwd/source/common/config/new_grpc_mux_impl.cc:62:38 (vhds_integration_test+0x4f9c0d5)
    #16 non-virtual thunk to Envoy::Config::NewGrpcMuxImpl::onDiscoveryResponse(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) /proc/self/cwd/source/common/config/new_grpc_mux_impl.cc (vhds_integration_test+0x4f9c2df)
    #17 Envoy::Config::GrpcStream<envoy::service::discovery::v3::DeltaDiscoveryRequest, envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessage(std::__1::unique_ptr<envoy::service::discovery::v3::DeltaDiscoveryResponse, std::__1::default_delete<envoy::service::discovery::v3::DeltaDiscoveryResponse> >&&) /proc/self/cwd/bazel-out/k8-dbg/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:83:17 (vhds_integration_test+0x321f37a)
    #18 Envoy::Grpc::AsyncStreamCallbacks<envoy::service::discovery::v3::DeltaDiscoveryResponse>::onReceiveMessageRaw(std::__1::unique_ptr<Envoy::Buffer::Instance, std::__1::default_delete<Envoy::Buffer::Instance> >&&) /proc/self/cwd/bazel-out/k8-dbg/bin/source/common/grpc/_virtual_includes/typed_async_client_lib/common/grpc/typed_async_client.h:93:5 (vhds_integration_test+0x321ef86)
    #19 Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/grpc/async_client_impl.cc:140:21 (vhds_integration_test+0x4fc9baf)
    #20 non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/grpc/async_client_impl.cc (vhds_integration_test+0x4fc9d3f)
    #21 Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/http/async_client_impl.cc:117:21 (vhds_integration_test+0x4fe0c29)
    #22 non-virtual thunk to Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/http/async_client_impl.cc (vhds_integration_test+0x4fe0d3f)
    #23 Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) /proc/self/cwd/source/common/router/router.cc:1338:15 (vhds_integration_test+0x500f348)
    #24 non-virtual thunk to Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) /proc/self/cwd/source/common/router/router.cc (vhds_integration_test+0x500f414)
    #25 Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/source/common/router/upstream_request.cc:148:11 (vhds_integration_test+0x505b8b8)
    #26 Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/bazel-out/k8-dbg/bin/source/common/http/_virtual_includes/codec_wrappers_lib/common/http/codec_wrappers.h:35:12 (vhds_integration_test+0x376351b)
    #27 Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) /proc/self/cwd/source/common/http/http2/codec_impl.cc:657:25 (vhds_integration_test+0x53393ea)
    #28 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_14::operator()(nghttp2_session*, nghttp2_frame const*, void*) const /proc/self/cwd/source/common/http/http2/codec_impl.cc:1067:57 (vhds_integration_test+0x5342da0)
    #29 Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_14::__invoke(nghttp2_session*, nghttp2_frame const*, void*) /proc/self/cwd/source/common/http/http2/codec_impl.cc:1066:19 (vhds_integration_test+0x5342d18)
    #30 session_call_on_frame_received /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/3279/execroot/envoy/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:3300:10 (vhds_integration_test+0x5a3df91)
    #31 nghttp2_session_on_data_received /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/3279/execroot/envoy/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:4976:8 (vhds_integration_test+0x5a41914)
    #32 session_process_data_frame /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/3279/execroot/envoy/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:4995:8 (vhds_integration_test+0x5a4dfeb)
    #33 nghttp2_session_mem_recv /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/3279/execroot/envoy/external/com_github_nghttp2_nghttp2/lib/nghttp2_session.c:6616:12 (vhds_integration_test+0x5a49532)
    #34 Envoy::Http::Http2::ConnectionImpl::innerDispatch(Envoy::Buffer::Instance&) /proc/self/cwd/source/common/http/http2/codec_impl.cc:495:9 (vhds_integration_test+0x533714f)
    #35 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5::operator()(Envoy::Buffer::Instance&) const /proc/self/cwd/source/common/http/http2/codec_impl.cc:483:60 (vhds_integration_test+0x534e0a1)
    #36 decltype(std::__1::forward<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5&>(fp)(std::__1::forward<Envoy::Buffer::Instance&>(fp0))) std::__1::__invoke<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5&, Envoy::Buffer::Instance&>(Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5&, Envoy::Buffer::Instance&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x534dfb4)
    #37 absl::Status std::__1::__invoke_void_return_wrapper<absl::Status>::__call<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5&, Envoy::Buffer::Instance&>(Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5&, Envoy::Buffer::Instance&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:317:16 (vhds_integration_test+0x534dee4)
    #38 std::__1::__function::__alloc_func<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5, std::__1::allocator<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5>, absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x534de54)
    #39 std::__1::__function::__func<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5, std::__1::allocator<Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_5>, absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x534c0f3)
    #40 std::__1::__function::__value_func<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x44dff5b)
    #41 std::__1::function<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x44dfe6f)
    #42 Envoy::Http::Utility::exceptionToStatus(std::__1::function<absl::Status (Envoy::Buffer::Instance&)>, Envoy::Buffer::Instance&) /proc/self/cwd/source/common/http/utility.cc:41:14 (vhds_integration_test+0x5933ab0)
    #43 Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) /proc/self/cwd/source/common/http/http2/codec_impl.cc:482:10 (vhds_integration_test+0x5336b62)
    #44 virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) /proc/self/cwd/source/common/http/http2/codec_impl.cc (vhds_integration_test+0x5336d1a)
    #45 Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) /proc/self/cwd/source/common/http/codec_client.cc:125:33 (vhds_integration_test+0x4f3f459)
    #46 Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) /proc/self/cwd/bazel-out/k8-dbg/bin/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:174:15 (vhds_integration_test+0x4f450dc)
    #47 Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) /proc/self/cwd/source/common/network/filter_manager_impl.cc:66:48 (vhds_integration_test+0x553fce8)
    #48 Envoy::Network::FilterManagerImpl::onRead() /proc/self/cwd/source/common/network/filter_manager_impl.cc:76:3 (vhds_integration_test+0x553ffe1)
    #49 Envoy::Network::ConnectionImpl::onRead(unsigned long) /proc/self/cwd/source/common/network/connection_impl.cc:305:19 (vhds_integration_test+0x551c9c2)
    #50 Envoy::Network::ConnectionImpl::onReadReady() /proc/self/cwd/source/common/network/connection_impl.cc:587:5 (vhds_integration_test+0x5521bb0)
    #51 Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) /proc/self/cwd/source/common/network/connection_impl.cc:547:5 (vhds_integration_test+0x5520120)
    #52 Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4::operator()(unsigned int) const /proc/self/cwd/source/common/network/connection_impl.cc:76:74 (vhds_integration_test+0x5531077)
    #53 decltype(std::__1::forward<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4&>(fp)(std::__1::forward<unsigned int>(fp0))) std::__1::__invoke<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4&, unsigned int&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x5530fbc)
    #54 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4&, unsigned int>(Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4&, unsigned int&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x5530eed)
    #55 std::__1::__function::__alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4>, void (unsigned int)>::operator()(unsigned int&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x5530e6d)
    #56 std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_delete<Envoy::Network::ConnectionSocket> >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_delete<Envoy::Network::TransportSocket> >&&, Envoy::StreamInfo::StreamInfo&, bool)::$_4>, void (unsigned int)>::operator()(unsigned int&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x552f11c)
    #57 std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x4514764)
    #58 std::__1::function<void (unsigned int)>::operator()(unsigned int) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x4514658)
    #59 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::operator()(int, short, void*) const /proc/self/cwd/source/common/event/file_event_impl.cc:69:11 (vhds_integration_test+0x550ab2c)
    #60 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) /proc/self/cwd/source/common/event/file_event_impl.cc:50:7 (vhds_integration_test+0x550aa46)
    #61 event_persist_closure /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:1639:9 (vhds_integration_test+0x7748ab0)
    #62 event_process_active_single_queue /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:1698:4 (vhds_integration_test+0x77474d8)
    #63 event_process_active /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:1799:9 (vhds_integration_test+0x773cd18)
    #64 event_base_loop /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:2041:12 (vhds_integration_test+0x773a9c3)
    #65 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/libevent_scheduler.cc:47:3 (vhds_integration_test+0x57f634f)
    #66 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/dispatcher_impl.cc:203:19 (vhds_integration_test+0x54f69a1)
    #67 Envoy::Server::InstanceImpl::run() /proc/self/cwd/source/server/server.cc:627:16 (vhds_integration_test+0x2e5541c)
    #68 Envoy::IntegrationTestServerImpl::createAndRunEnvoyServer(Envoy::OptionsImpl&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::ListenerHooks&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >) /proc/self/cwd/test/integration/server.cc:208:12 (vhds_integration_test+0x2c6ee50)
    #69 Envoy::IntegrationTestServer::threadRoutine(Envoy::Network::Address::IpVersion, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int) /proc/self/cwd/test/integration/server.cc:177:3 (vhds_integration_test+0x2c6e8a6)
    #70 Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0::operator()() const /proc/self/cwd/test/integration/server.cc:90:9 (vhds_integration_test+0x2c72736)
    #71 decltype(std::__1::forward<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0&>(fp)()) std::__1::__invoke<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x2c72590)
    #72 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0&>(Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x2c724f0)
    #73 std::__1::__function::__alloc_func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x2c72490)
    #74 std::__1::__function::__func<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0, std::__1::allocator<Envoy::IntegrationTestServer::start(Envoy::Network::Address::IpVersion, std::__1::function<void ()>, bool, bool, absl::optional<std::__1::reference_wrapper<Envoy::ProcessObject> >, Envoy::Server::FieldValidationConfig, unsigned int)::$_0>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x2c70b5f)
    #75 std::__1::__function::__value_func<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x2af5c66)
    #76 std::__1::function<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x2af58f8)
    #77 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const /proc/self/cwd/source/common/common/posix/thread_impl.cc:33:9 (vhds_integration_test+0x77206cb)
    #78 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) /proc/self/cwd/source/common/common/posix/thread_impl.cc:32:7 (vhds_integration_test+0x7720658)

  Previous read of size 8 at 0x7b1400031268 by thread T20:
    #0 Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0::operator()(bool) const /proc/self/cwd/source/extensions/filters/http/on_demand/on_demand_update.cc:20 (vhds_integration_test+0x39aff2c)
    #1 decltype(std::__1::forward<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0&>(fp)(std::__1::forward<bool>(fp0))) std::__1::__invoke<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0&, bool>(Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0&, bool&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x39afe82)
    #2 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0&, bool>(Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0&, bool&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x39afdad)
    #3 std::__1::__function::__alloc_func<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0, std::__1::allocator<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0>, void (bool)>::operator()(bool&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x39afd2d)
    #4 std::__1::__function::__func<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0, std::__1::allocator<Envoy::Extensions::HttpFilters::OnDemand::OnDemandRouteUpdate::decodeHeaders(Envoy::Http::RequestHeaderMap&, bool)::$_0>, void (bool)>::operator()(bool&&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x39adfdc)
    #5 std::__1::__function::__value_func<void (bool)>::operator()(bool&&) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x30d36d4)
    #6 std::__1::function<void (bool)>::operator()(bool) const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x30d3602)
    #7 Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6::operator()() const /proc/self/cwd/source/common/router/rds_impl.cc:289:11 (vhds_integration_test+0x3b3c8fc)
    #8 decltype(std::__1::forward<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6&>(fp)()) std::__1::__invoke<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6&>(Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x3b3c810)
    #9 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6&>(Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x3b3c770)
    #10 std::__1::__function::__alloc_func<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6, std::__1::allocator<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x3b3c710)
    #11 std::__1::__function::__func<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6, std::__1::allocator<Envoy::Router::RdsRouteConfigProviderImpl::onConfigUpdate()::$_6>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x3b3a78f)
    #12 std::__1::__function::__value_func<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x2af5c66)
    #13 std::__1::function<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x2af58f8)
    #14 Envoy::Event::DispatcherImpl::runPostCallbacks() /proc/self/cwd/source/common/event/dispatcher_impl.cc:233:5 (vhds_integration_test+0x54f6acf)
    #15 Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1::operator()() const /proc/self/cwd/source/common/event/dispatcher_impl.cc:44:58 (vhds_integration_test+0x54fbcbb)
    #16 decltype(std::__1::forward<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&>(fp)()) std::__1::__invoke<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x54fbc10)
    #17 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&>(Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x54fbb70)
    #18 std::__1::__function::__alloc_func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x54fbb10)
    #19 std::__1::__function::__func<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1, std::__1::allocator<Envoy::Event::DispatcherImpl::DispatcherImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::unique_ptr<Envoy::Buffer::WatermarkFactory, std::__1::default_delete<Envoy::Buffer::WatermarkFactory> >&&, Envoy::Api::Api&, Envoy::Event::TimeSystem&)::$_1>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x54f9ddf)
    #20 std::__1::__function::__value_func<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x2af5c66)
    #21 std::__1::function<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x2af58f8)
    #22 Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::operator()(int, short, void*) const /proc/self/cwd/source/common/event/timer_impl.cc:15:3 (vhds_integration_test+0x57f8935)
    #23 Envoy::Event::TimerImpl::TimerImpl(Envoy::CSmartPtr<event_base, &(event_base_free)>&, std::__1::function<void ()>, Envoy::Event::Dispatcher&)::$_0::__invoke(int, short, void*) /proc/self/cwd/source/common/event/timer_impl.cc:15:3 (vhds_integration_test+0x57f8896)
    #24 event_process_active_single_queue /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:1707:4 (vhds_integration_test+0x77476c5)
    #25 event_process_active /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:1799:9 (vhds_integration_test+0x773cd18)
    #26 event_base_loop /home/wb/.cache/bazel/_bazel_wb/363df6af2a92e025e191d91dc704db31/sandbox/linux-sandbox/4006/execroot/envoy/external/com_github_libevent_libevent/event.c:2041:12 (vhds_integration_test+0x773a9c3)
    #27 Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/libevent_scheduler.cc:47:3 (vhds_integration_test+0x57f634f)
    #28 Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) /proc/self/cwd/source/common/event/dispatcher_impl.cc:203:19 (vhds_integration_test+0x54f69a1)
    #29 Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&) /proc/self/cwd/source/server/worker_impl.cc:119:16 (vhds_integration_test+0x30be523)
    #30 Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4::operator()() const /proc/self/cwd/source/server/worker_impl.cc:85:72 (vhds_integration_test+0x30cc777)
    #31 decltype(std::__1::forward<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4&>(fp)()) std::__1::__invoke<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4&>(Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/type_traits:3530:1 (vhds_integration_test+0x30cc6b0)
    #32 void std::__1::__invoke_void_return_wrapper<void>::__call<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4&>(Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4&) /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/__functional_base:348:9 (vhds_integration_test+0x30cc610)
    #33 std::__1::__function::__alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1533:16 (vhds_integration_test+0x30cc5b0)
    #34 std::__1::__function::__func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4, std::__1::allocator<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&)::$_4>, void ()>::operator()() /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1707:12 (vhds_integration_test+0x30ca87f)
    #35 std::__1::__function::__value_func<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:1860:16 (vhds_integration_test+0x2af5c66)
    #36 std::__1::function<void ()>::operator()() const /home/wb/opt/clang+llvm-9.0.0-x86_64-linux-sles11.3/bin/../include/c++/v1/functional:2419:12 (vhds_integration_test+0x2af58f8)
    #37 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::operator()(void*) const /proc/self/cwd/source/common/common/posix/thread_impl.cc:33:9 (vhds_integration_test+0x77206cb)
    #38 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>)::$_0::__invoke(void*) /proc/self/cwd/source/common/common/posix/thread_impl.cc:32:7 (vhds_integration_test+0x7720658)

@mattklein123
Copy link
Member

I would need to look in more detail but I strongly suspect what you are doing is not thread safe somehow. I highly doubt this is a false positive. I can look tomorrow.

alyssawilk pushed a commit that referenced this issue Jun 10, 2020
vhds: fixing a filter teardown bug (#11341)
9d702c1 
This is to help with #9784

Risk Level: Low (added a single test)
Testing: a new unit test and integration test
Docs Changes: n/a
Release Notes: n/a

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
@alyssawilk
Copy link
Contributor

This is still failing, and is pretty easy to reproduce with
bazel test --config=clang-tsan //test/integration:vhds_integration_test --runs_per_test=200
@dmitri-d can you please take another look?

yashwant121 pushed a commit to yashwant121/envoy that referenced this issue Jun 24, 2020
@yashwant121
vhds: fixing a filter teardown bug (envoyproxy#11341)
9426e3f 
This is to help with envoyproxy#9784

Risk Level: Low (added a single test)
Testing: a new unit test and integration test
Docs Changes: n/a
Release Notes: n/a

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
Signed-off-by: yashwant121 <yadavyashwant36@gmail.com>
songhu pushed a commit to songhu/envoy that referenced this issue Jun 25, 2020
@songhu
vhds: fixing a filter teardown bug (envoyproxy#11341)
51a9396 
This is to help with envoyproxy#9784

Risk Level: Low (added a single test)
Testing: a new unit test and integration test
Docs Changes: n/a
Release Notes: n/a

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
@alyssawilk alyssawilk modified the milestones: 1.15.0, 1.16.0 Jul 6, 2020
@dmitri-d
Copy link
Contributor

dmitri-d commented Jul 10, 2020 •
edited
Loading

This issue is due to use of libc++ not instrumented for use with thread sanitiser. I couldn't find much information about it, but there's this though: https://reviews.llvm.org/D21609?id=61547. I followed the suggestion and built an instrumented version of libc++ (see here: https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo, replace -DLLVM_USE_SANITIZER=MemoryWithOrigins with -DLLVM_USE_SANITIZER=Thread). Running tsan tests with this instrumented version produced no errors.

@dmitri-d
Copy link
Contributor

Ping @alyssawilk

@zuercher zuercher mentioned this issue Jul 10, 2020
Merged
@alyssawilk
Copy link
Contributor

I'd think we could just add those options to the tsan build? cc @lizan and @htuch

@htuch htuch mentioned this issue Jul 13, 2020
Merged
@htuch
Copy link
Member

htuch commented Jul 13, 2020 •
edited
Loading

We'll need a dedicated TSAN base image for this in https://github.com/envoyproxy/envoy-build-tools. This is basically the same issue that was blocking us having MSAN working in OSS Envoy (#918).

@dmitri-d
Copy link
Contributor

I can look into building a dedicated image.

@akonradi akonradi mentioned this issue Jul 13, 2020
Merged
@dmitri-d
Copy link
Contributor

Had a chat with @lizan, I'll add a tsan-specific build image

@lizan
Copy link
Member

lizan commented Jul 13, 2020

We built MSAN instrumented libc++ in the same image: https://github.com/envoyproxy/envoy-build-tools/blob/master/build_container/build_container_common.sh#L88-L102
and using it with .bazelrc https://github.com/envoyproxy/envoy/blob/master/.bazelrc#L156-L158

TSAN instumented libc++ can be done in a similar way.

@htuch
Copy link
Member

htuch commented Jul 13, 2020

@lizan should we close out #918? Do we have MSAN yet in CI?

@lizan
Copy link
Member

lizan commented Jul 13, 2020

@htuch we didn't enable it full to reduce CI pressure. Though we can enable them if someone is taking the work (fixing MSAN failures in master)

@KBaichoo KBaichoo mentioned this issue Jul 14, 2020
Merged
@dmitri-d dmitri-d mentioned this issue Jul 16, 2020
Merged
lizan pushed a commit that referenced this issue Jul 17, 2020
Switch to a tsan-instrumented libc++ for tsan tests (#12134)
06fd1d1 
This fixes #9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
@jwendell jwendell mentioned this issue Jul 22, 2020
Closed
yashwant121 pushed a commit to yashwant121/envoy that referenced this issue Jul 24, 2020
@yashwant121
vhds: fixing a filter teardown bug (envoyproxy#11341)
106819e 
This is to help with envoyproxy#9784

Risk Level: Low (added a single test)
Testing: a new unit test and integration test
Docs Changes: n/a
Release Notes: n/a

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>
Signed-off-by: yashwant121 <yadavyashwant36@gmail.com>
KBaichoo pushed a commit to KBaichoo/envoy that referenced this issue Jul 30, 2020
@KBaichoo
Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134)
30a0356 
This fixes envoyproxy#9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>

Signed-off-by: Kevin Baichoo <kbaichoo@google.com>
scheler pushed a commit to scheler/envoy that referenced this issue Aug 4, 2020
Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134)
42453e8 
This fixes envoyproxy#9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>

Signed-off-by: scheler <santosh.cheler@appdynamics.com>
antoniovicente pushed a commit to antoniovicente/envoy that referenced this issue Sep 30, 2020
@antoniovicente
Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134)
6d30545 
This fixes envoyproxy#9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>

Signed-off-by: Antonio Vicente <avd@google.com>
lambdai pushed a commit that referenced this issue Oct 16, 2020
@antoniovicente @mattklein123 @lizan
Backport flaky test and tsan fixes to releases/v1.15 branch (#13337)
4c143df 
* hds: fix integration test flakes (#12214)

Part of #12184

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* Switch to a tsan-instrumented libc++ for tsan tests (#12134)

This fixes #9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>

Signed-off-by: Antonio Vicente <avd@google.com>

* test: shard hds_integration_test (#12482)

This should avoid TSAN timeout flakes.

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* test: shard http2_integration_test (#11939)

This should mitigate TSAN timeout.

Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Signed-off-by: Antonio Vicente <avd@google.com>

* test: fix http2_integration_test flake (#12450)

Fixes #12442

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* Kick CI

Signed-off-by: Antonio Vicente <avd@google.com>

Co-authored-by: Matt Klein <mklein@lyft.com>
Co-authored-by: Dmitri Dolguikh <ddolguik@redhat.com>
Co-authored-by: Lizan Zhou <lizan@tetrate.io>
istio-testing pushed a commit to istio/envoy that referenced this issue Jan 8, 2021
@bianpengyuan @PiotrSikora
@antoniovicente @mattklein123 @lizan @cpakulski @asraa @jmarantz @Shikugawa @alyssawilk
[release-1.7] Backport Envoy 1.15 fixes (#290)
a0eff54 
* docs: kick-off 1.15.1 release. (envoyproxy#12166)

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

* tls: update BoringSSL-FIPS to 20190808. (envoyproxy#12170)

Signed-off-by: Piotr Sikora <piotrsikora@google.com>

* test: Exclude wasm_vm_test from CI by making it a "manual" test. (#207)

The wee v8 build times out in CI under --config=asan because the machine the job is scheduled on is too small.

Signed-off-by: Antonio Vicente <avd@google.com>

* [v1.15] http: header map security fixes for duplicate headers (#197) (#200)

Previously header matching did not match on all headers for
non-inline headers. This patch changes the default behavior to
always logically match on all headers. Multiple individual
headers will be logically concatenated with ',' similar to what
is done with inline headers. This makes the behavior effectively
consistent. This behavior can be temporary reverted by setting
the runtime value "envoy.reloadable_features.header_match_on_all_headers"
to "false".

Targeted fixes have been additionally performed on the following
extensions which make them consider all duplicate headers by default as
a comma concatenated list:
1) Any extension using CEL matching on headers.
2) The header to metadata filter.
3) The JWT filter.
4) The Lua filter.
Like primary header matching used in routing, RBAC, etc. this behavior
can be disabled by setting the runtime value
"envoy.reloadable_features.header_match_on_all_headers" to false.

Finally, the setCopy() header map API previously only set the first
header in the case of duplicate non-inline headers. setCopy() now
behaves similiarly to the other set*() APIs and replaces all found
headers with a single value. This may have had security implications
in the extauth filter which uses this API. This behavior can be disabled
by setting the runtime value
"envoy.reloadable_features.http_set_copy_replace_all_headers" to false.

Fixes https://github.com/envoyproxy/envoy-setec/issues/188

Signed-off-by: Matt Klein <mklein@lyft.com>

* backport to v1.15: Fix Kafka Repository Location (#223)

Update mirror used to fetch kafka dependency to a valid, working mirror.

Based on envoyproxy#13025
Resolves envoyproxy#13011

Signed-off-by: Antonio Vicente <avd@google.com>

* release: cutting 1.15.1 (#217)

Signed-off-by: Antonio Vicente <avd@google.com>

* docs: Fix release notes for v1.15.1 release. (envoyproxy#13318)

Signed-off-by: Antonio Vicente <avd@google.com>

* Backport flaky test and tsan fixes to releases/v1.15 branch (envoyproxy#13337)

* hds: fix integration test flakes (envoyproxy#12214)

Part of envoyproxy#12184

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* Switch to a tsan-instrumented libc++ for tsan tests (envoyproxy#12134)

This fixes envoyproxy#9784 and re-enables vhds_integration_test

Risk Level: Low, but will most likely increase memory usage

Signed-off-by: Dmitri Dolguikh <ddolguik@redhat.com>

Signed-off-by: Antonio Vicente <avd@google.com>

* test: shard hds_integration_test (envoyproxy#12482)

This should avoid TSAN timeout flakes.

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* test: shard http2_integration_test (envoyproxy#11939)

This should mitigate TSAN timeout.

Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Signed-off-by: Antonio Vicente <avd@google.com>

* test: fix http2_integration_test flake (envoyproxy#12450)

Fixes envoyproxy#12442

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Antonio Vicente <avd@google.com>

* Kick CI

Signed-off-by: Antonio Vicente <avd@google.com>

Co-authored-by: Matt Klein <mklein@lyft.com>
Co-authored-by: Dmitri Dolguikh <ddolguik@redhat.com>
Co-authored-by: Lizan Zhou <lizan@tetrate.io>

* docs: kick off v1.15.3-dev (envoyproxy#13695)

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* 1.15: CI fixes backport (envoyproxy#13697)

Backport following commits to 1.15:
748b2ab (mac ci: try ignoring update failure (envoyproxy#13658), 2020-10-20)
f95f539 (ci: various improvements (envoyproxy#13660), 2020-10-20)
73d78f8 (ci: use multiple stage (envoyproxy#13557), 2020-10-15)
b7a4756 (ci: use azp for api and go-control-plane sync (envoyproxy#13550), 2020-10-14)
876a6bb (ci use azp to sync filter example (envoyproxy#13501), 2020-10-12)
a0f31ee (ci: use azp to generate docs (envoyproxy#13481), 2020-10-12)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Co-authored-by: asraa <asraa@google.com>

* 1.15: fix CI script (envoyproxy#13724)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* Prevent SEGFAULT when disabling listener (envoyproxy#13515) (envoyproxy#13903)

This prevents the stop_listening overload action from causing
segmentation faults that can occur if the action is enabled after the
listener has already shut down.

Signed-off-by: Alex Konradi <akonradi@google.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* proxy protocol: set downstreamRemoteAddress on StreamInfo (envoyproxy#14131) (envoyproxy#14169)

This fixes a regression which resulted in the downstreamRemoteAddress
on the StreamInfo for a connection not having the address supplied by
the proxy protocol filter, but instead having the address of the
directly connected peer.

This issue does not affect HTTP filters.

Fixes envoyproxy#14087

Signed-off-by: Greg Greenway <ggreenway@apple.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* ci: temproray disable vhds_integration_test in TSAN (envoyproxy#12067) (envoyproxy#14217)

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") (envoyproxy#14165)

Commit Message: tcmalloc changed and the data coming out of tcmalloc::MallocExtension::GetNumericProperty("generic.current_allocated_bytes") no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out
Additional Description:
Risk Level: low
Testing: just thread_local_store_test
Docs Changes: n/a
Release Notes: n/a

no longer appears to be deterministic, even in unthreaded tests. So disable exact mem checks till we sort that out

Signed-off-by: Joshua Marantz <jmarantz@google.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Co-authored-by: Joshua Marantz <jmarantz@google.com>

* backport to v1.15: connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772) (envoyproxy#14173)

* connection: Remember transport socket read resumption requests and replay them when re-enabling read. (envoyproxy#13772)

Fixes SslSocket read resumption after readDisable when processing the SSL record that contains the last bytes of the HTTP message

Signed-off-by: Antonio Vicente <avd@google.com>

* backport to 1.15: udp: properly handle truncated/dropped datagrams (envoyproxy#14122) (envoyproxy#14166)

Signed-off-by: Matt Klein <mklein@lyft.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
Co-authored-by: Matt Klein <mklein@lyft.com>
Co-authored-by: Christoph Pakulski <christoph@tetrate.io>

* backport to 1.15: vrp: allow supervisord to open its log file (envoyproxy#14066) (envoyproxy#14280)

Commit Message: Allow supervisord to open its log file
Additional Description:
Change the default location of the log file and give supervisord
permissions to write to it.

Risk Level: low
Testing: built image locally
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

Signed-off-by: Alex Konradi <akonradi@google.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* rel 1.15: close release 1.15.3 (envoyproxy#14303)

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* Kick off rel 1.15.4. (envoyproxy#14323)

Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

* backport to 1.15: http: fix datadog and squash handling of responses without body (envoyproxy#13328) (envoyproxy#14458)

Commit Message: Fixing bugs in datadog and sqaush where unexpected bodyless responses would crash Envoy
Risk Level: low
Testing: new unit tests, updated certs
Docs Changes: n/a
Release Notes: inline
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
Co-authored-by: alyssawilk <alyssar@chromium.org>

* backport 1.15: http: fixing a bug with IPv6 hosts (envoyproxy#14273)

Fixing a bug where HTTP parser offsets for IPv6 hosts did not include [] and Envoy assumed it did.
This results in mis-parsing addresses for IPv6 CONNECT requests and IPv6 hosts in fully URLs over HTTP/1.1

Risk Level: low
Testing: new unit, integration tests
Docs Changes: n/a
Release Notes: inline

Signed-off-by: Shikugawa <rei@tetrate.io>
Co-authored-by: alyssawilk <alyssar@chromium.org>

* backport to 1.15: tls: fix detection of the upstream connection close event. (envoyproxy#13858) (envoyproxy#14568)

Fixes envoyproxy#13856.

Signed-off-by: Piotr Sikora <piotrsikora@google.com>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>

Co-authored-by: Piotr Sikora <piotrsikora@google.com>
Co-authored-by: antonio <avd@google.com>
Co-authored-by: Matt Klein <mklein@lyft.com>
Co-authored-by: Dmitri Dolguikh <ddolguik@redhat.com>
Co-authored-by: Lizan Zhou <lizan@tetrate.io>
Co-authored-by: Christoph Pakulski <christoph@tetrate.io>
Co-authored-by: asraa <asraa@google.com>
Co-authored-by: Joshua Marantz <jmarantz@google.com>
Co-authored-by: Rei Shimizu <Shikugawa@gmail.com>
Co-authored-by: alyssawilk <alyssar@chromium.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitri-d dmitri-d

Labels
area/test flakes bug no stalebot Disables stalebot from closing an issue
Projects
None yet
Milestone
1.16.0
6 participants
@dmitri-d @lizan @jmarantz @mattklein123 @htuch @alyssawilk