Segfault from ECDS update #14930

bianpengyuan · 2021-02-03T20:49:57Z

Got a crash when updating filter config with ECDS.

2021-02-03T19:51:58.400705Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x55fa1e573d00
2021-02-03T19:51:58.400743Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2021-02-03T19:51:58.400748Z     critical        envoy backtrace Envoy version: 7ac437fc43bad86eb5e3bbdb0ebb948c096e00d4/1.18.0-dev/Clean/RELEASE/BoringSSL
2021-02-03T19:51:58.401112Z     critical        envoy backtrace #0: __restore_rt [0x7f9f4c64e980]
2021-02-03T19:51:58.413825Z     critical        envoy backtrace #1: Envoy::Filter::Http::FilterConfigSubscription::onConfigUpdate() [0x55fa1b8d24ff]
2021-02-03T19:51:58.426169Z     critical        envoy backtrace #2: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x55fa1b950aac]
2021-02-03T19:51:58.437933Z     critical        envoy backtrace #3: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x55fa1b9559d7]
2021-02-03T19:51:58.449607Z     critical        envoy backtrace #4: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x55fa1b6270a1]
2021-02-03T19:51:58.461479Z     critical        envoy backtrace #5: Envoy::Grpc::AsyncStreamImpl::onData() [0x55fa1b96a127]
2021-02-03T19:51:58.473328Z     critical        envoy backtrace #6: Envoy::Http::AsyncStreamImpl::encodeData() [0x55fa1b96f772]
2021-02-03T19:51:58.485140Z     critical        envoy backtrace #7: Envoy::Router::UpstreamRequest::decodeData() [0x55fa1b9a55f8]
2021-02-03T19:51:58.496932Z     critical        envoy backtrace #8: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55fa1b7b359b]
2021-02-03T19:51:58.508667Z     critical        envoy backtrace #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x55fa1b923c3e]
2021-02-03T19:51:58.520481Z     critical        envoy backtrace #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_18::__invoke() [0x55fa1b92ab18]
2021-02-03T19:51:58.532145Z     critical        envoy backtrace #11: nghttp2_session_on_data_received [0x55fa1ba920fc]
2021-02-03T19:51:58.543929Z     critical        envoy backtrace #12: nghttp2_session_mem_recv [0x55fa1ba9406c]
2021-02-03T19:51:58.555646Z     critical        envoy backtrace #13: Envoy::Http::Http2::ConnectionImpl::innerDispatch() [0x55fa1b922659]
2021-02-03T19:51:58.567405Z     critical        envoy backtrace #14: std::__1::__function::__func<>::operator()() [0x55fa1b92bd53]
2021-02-03T19:51:58.579239Z     critical        envoy backtrace #15: Envoy::Http::Utility::exceptionToStatus() [0x55fa1ba5b6e7]
2021-02-03T19:51:58.591168Z     critical        envoy backtrace #16: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55fa1b9222f1]
2021-02-03T19:51:58.602924Z     critical        envoy backtrace #17: Envoy::Http::CodecClient::onData() [0x55fa1b810bf0]
2021-02-03T19:51:58.614566Z     critical        envoy backtrace #18: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x55fa1b811ea5]
2021-02-03T19:51:58.626408Z     critical        envoy backtrace #19: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55fa1b5e978f]
2021-02-03T19:51:58.637954Z     critical        envoy backtrace #20: Envoy::Network::ConnectionImpl::onReadReady() [0x55fa1b5e3bc9]
2021-02-03T19:51:58.649832Z     critical        envoy backtrace #21: Envoy::Network::ConnectionImpl::onFileEvent() [0x55fa1b5e16d9]
2021-02-03T19:51:58.661516Z     critical        envoy backtrace #22: std::__1::__function::__func<>::operator()() [0x55fa1b5d6431]
2021-02-03T19:51:58.673221Z     critical        envoy backtrace #23: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55fa1b5d73cc]
2021-02-03T19:51:58.684889Z     critical        envoy backtrace #24: event_process_active_single_queue [0x55fa1ba8492b]
2021-02-03T19:51:58.696687Z     critical        envoy backtrace #25: event_base_loop [0x55fa1ba8328e]
2021-02-03T19:51:58.708510Z     critical        envoy backtrace #26: Envoy::Server::InstanceImpl::run() [0x55fa1b5b57d8]
2021-02-03T19:51:58.720199Z     critical        envoy backtrace #27: Envoy::MainCommonBase::run() [0x55fa19d74c08]
2021-02-03T19:51:58.731800Z     critical        envoy backtrace #28: Envoy::MainCommon::main() [0x55fa19d75437]
2021-02-03T19:51:58.743435Z     critical        envoy backtrace #29: main [0x55fa19d7377c]
2021-02-03T19:51:58.743633Z     critical        envoy backtrace #30: __libc_start_main [0x7f9f4c26cbf7]

The text was updated successfully, but these errors were encountered:

snowp · 2021-02-03T21:24:25Z

Is there any more information you can provide? A repro would be ideal, but config used for Envoy, config pushed via ECDS, envoy version, etc. would be helpful in debugging this

bianpengyuan · 2021-02-03T21:45:27Z

This is from a running istio cluster. I am not yet able to reproduce it. Trying to get a core dump if this happens again.

github-actions · 2021-03-14T08:03:30Z

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions · 2021-03-21T08:03:40Z

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

bianpengyuan · 2021-12-21T23:55:44Z

I can easily reproduce this now by deleting and recreating a filter which uses ECDS. I added some log instrument and find that crash is from this line at factory_context.messageValidationContext().dynamicValidationVisitor().

envoy/source/common/filter/config_discovery_impl.cc

Line 251 in 8c2da05

    
           factory_context.messageValidationContext().dynamicValidationVisitor(), factory);

I enabled debug log and found there is a simultaneous listener update when ecds update happens. @lambdai has a theory that the crash is because factory context here is from the old listener.

cc @kyessenov

kyessenov · 2021-12-22T00:23:43Z

The factory context looks like server factory context, right? So it's the same across listeners.

bianpengyuan · 2021-12-22T03:20:55Z

The factory context looks like server factory context, right? So it's the same across listeners.

@kyessenov I am not sure.. you and @lambdai know this better than me : )

Could someone please reopen this issue? Maybe @snowp ?

kyessenov · 2021-12-22T18:37:33Z

Ok, feel free to assign to me or @lambdai.

kyessenov · 2022-01-14T23:52:15Z

Asan logs:

external/envoy/source/common/filter/config_discovery_impl.h:101:17: runtime error: member access within address 0x612000123940 which does not point to an object of type 'Envoy::Filter::DynamicFilterConfigProviderImpl<std::function<void (Envoy::Http::FilterChainFactoryCallbacks &)>>'
0x612000123940: note: object has invalid vptr
 00 00 00 00  46 02 80 75 00 00 00 00  10 be 35 00 50 61 00 00  00 be 35 00 50 61 00 00  40 c7 3b e3
              ^~~~~~~~~~~~~~~~~~~~~~~
              invalid vptr
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior external/envoy/source/common/filter/config_discovery_impl.h:101:17 in
=================================================================
==4977==ERROR: AddressSanitizer: heap-use-after-free on address 0x6120001239f8 at pc 0x5583fb99d614 bp 0x7ffea8221ba0 sp 0x7ffea8221b98
READ of size 1 at 0x6120001239f8 thread T0
    #0 0x5583fb99d613 in void absl::optional_internal::optional_data_base<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> >::assign<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> const&>(std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> const&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x3a750613)
    #1 0x5583fb99d244 in absl::optional<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> >& absl::optional<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> >::operator=<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> const&, void>(std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> const&) (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x3a750244)
    #2 0x5583fb99de39 in Envoy::Filter::DynamicFilterConfigProviderImpl<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> >::onConfigUpdate(google::protobuf::Message const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>)::'lambda'()::operator()() const (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x3a750e39)
2

Deleted here:
    #0 0x5583f25c1b82 in free (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x31374b82)
    #1 0x5583fb995e9d in Envoy::Filter::DynamicFilterConfigProviderImpl<std::function<void (Envoy::Http::FilterChainFactoryCallbacks&)> >::~DynamicFilterConfigProviderImpl() (/home/kuat/.cache/bazel/_bazel_kuat/c9ce937492879f803e7d5d4bb6e36e5c/execroot/io_istio_proxy/bazel-out/k8-fastbuild/bin/src/envoy/envoy+0x3a748e9d)

Seems like the raw pointer from subscription to provider is used because of async cross thread post and deleted from main as owned by a listener. Will need to restructure using shared and weak pointers.

kyessenov · 2022-01-14T23:52:23Z

/reopen

rahulanand16nov · 2022-03-14T11:12:40Z

The problem is still producible with the 1.21.2-dev release.

kyessenov · 2022-03-14T16:21:58Z

@rahulanand16nov do you have a stack trace or reproducer?

rahulanand16nov · 2022-03-14T17:55:42Z

@kyessenov Sorry for the noise but for some reason it's working now even though it was reproducible when I recreated the whole cluster previously. Very weird.

huanghuangzym · 2022-06-30T12:43:31Z

2022-06-29T08:40:44.324552Z critical envoy assert assert failure: Thread::MainThread::isMainThread().
2022-06-29T08:40:44.324633Z critical envoy backtrace Caught Aborted, suspect faulting address 0x53900000022
2022-06-29T08:40:44.324644Z critical envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2022-06-29T08:40:44.324653Z critical envoy backtrace Envoy version: edfec5636eb3494b1e5728891f8c0a6eba3e7d9f/1.20.3-dev/Clean/DEBUG/BoringSSL
2022-06-29T08:40:44.361192Z critical envoy backtrace #0: Envoy::SignalAction::sigHandler() [0x563ae96b5b59]
2022-06-29T08:40:44.361292Z critical envoy backtrace #1: __restore_rt [0x7f42a282a980]
2022-06-29T08:40:44.429488Z critical envoy backtrace #2: Envoy::ThreadLocal::InstanceImpl::SlotImpl::~SlotImpl() [0x563ae8208aee]
2022-06-29T08:40:44.515387Z critical envoy backtrace #3: Envoy::ThreadLocal::InstanceImpl::SlotImpl::~SlotImpl() [0x563ae8208b59]
2022-06-29T08:40:44.600766Z critical envoy backtrace #4: std::default_delete<>::operator()() [0x563ae558203c]
2022-06-29T08:40:44.636288Z critical envoy backtrace #5: std::unique_ptr<>::~unique_ptr() [0x563ae557f720]
2022-06-29T08:40:44.721708Z critical envoy backtrace #6: Envoy::ThreadLocal::TypedSlot<>::~TypedSlot() [0x563ae565a055]
2022-06-29T08:40:44.809034Z critical envoy backtrace #7: std::default_delete<>::operator()() [0x563ae565a01b]
2022-06-29T08:40:44.844375Z critical envoy backtrace #8: std::unique_ptr<>::~unique_ptr() [0x563ae56584e0]
2022-06-29T08:40:44.929817Z critical envoy backtrace #9: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::~FilterConfig() [0x563ae5cc9bc6]
2022-06-29T08:40:45.015266Z critical envoy backtrace #10: __gnu_cxx::new_allocator<>::destroy<>() [0x563ae5cc9b99]
2022-06-29T08:40:45.100660Z critical envoy backtrace #11: std::allocator_traits<>::destroy<>() [0x563ae5cc9b2d]
2022-06-29T08:40:45.136206Z critical envoy backtrace #12: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x563ae5cc9976]
2022-06-29T08:40:45.223556Z critical envoy backtrace #13: std::_Sp_counted_base<>::_M_release() [0x563ae51e9125]

2022-06-29T08:40:45.308881Z critical 2022-06-29T08:40:45.400249Z critical 2022-06-29T08:40:45.435501Z critical 2022-06-29T08:40:45.521008Z critical 2022-06-29T08:40:45.606335Z critical 2022-06-29T08:40:45.641617Z critical 2022-06-29T08:40:45.727030Z critical 2022-06-29T08:40:45.811359Z critical 2022-06-29T08:40:45.897092Z critical 2022-06-29T08:40:45.932664Z critical 2022-06-29T08:40:46.019211Z critical 2022-06-29T08:40:46.104726Z critical 2022-06-29T08:40:46.140487Z critical 2022-06-29T08:40:46.226030Z critical 2022-06-29T08:40:46.311554Z critical 2022-06-29T08:40:46.396042Z critical 2022-06-29T08:40:46.431645Z critical 2022-06-29T08:40:46.431674Z critical 2022-06-29T08:40:46.431680Z critical 2022-06-29T08:40:46.521658Z critical envoy backtrace #14: std::__shared_count<>::__shared_count() [0x563ae51e90da]
envoy backtrace #15: std::__shared_ptr<>::__shared_ptr() [0x563ae5cc9169]
envoy backtrace #16: std::shared_ptr<>::shared_ptr() [0x563ae5cc8765]
envoy backtrace #17: Envoy::Extensions::HttpFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped()::$FactoryFromProtoTyped() [0x563ae5cc7ee5]
envoy backtrace #18: std::_Function_base::_Base_manager<>::_M_destroy() [0x563ae5cc857f]
envoy backtrace #19: std::_Function_base::_Base_manager<>::_M_manager() [0x563ae5cc829d]
envoy backtrace #20: std::_Function_base::_Function_base() [0x563ae51e9281]
envoy backtrace #21: std::function<>::~function() [0x563ae590f755]
envoy backtrace #22: absl::optional_internal::optional_data_dtor_base<>::destruct() [0x563ae8b4357a]
envoy backtrace #23: absl::optional_internal::optional_data_dtor_base<>::~optional_data_dtor_base() [0x563ae8b43545
envoy backtrace #24: absl::optional_internal::optional_data_base<>::~optional_data_base() [0x563ae8b43525]
envoy backtrace #25: absl::optional_internal::optional_data<>::~optional_data() [0x563ae8b43505]
envoy backtrace #26: absl::optional<>::optional() [0x563ae8b3a895]
envoy backtrace #27: Envoy::Filter::DynamicFilterConfigProviderImpl<>::ThreadLocalConfig::ThreadLocalConfig() [0x5
envoy backtrace #28: __gnu_cxx::new_allocator<>::destroy<>() [0x563ae8b4ca39]
envoy backtrace #29: std::allocator_traits<>::destroy<>() [0x563ae8b4c9cd]
envoy backtrace #30: std::_Sp_counted_ptr_inplace<>::_M_dispose() [0x563ae8b4c796]
envoy backtrace #31: std::_Sp_counted_base<>::_M_release() [0x563ae51e9125]
envoy backtrace #32: std::__shared_count<>::__shared_count() [0x563ae51e90da]
envoy backtrace #33: std::__shared_ptr<>::__shared_ptr() [0x563ae5584599]

2022-06-29T08:40:46.607164Z critical envoy backtrace #34: std::__shared_ptr<>::operator=() [0x563ae820b953]
2022-06-29T08:40:46.642655Z critical envoy backtrace #35: std::shared_ptr<>::operator=() [0x563ae820b901]
2022-06-29T08:40:46.730345Z critical envoy backtrace #36: Envoy::ThreadLocal::InstanceImpl::removeSlot()::$_4::operator()() [0x563ae820703d]
2022-06-29T08:40:46.818964Z critical envoy backtrace #37: std::_Function_handler<>::_M_invoke() [0x563ae8206ead]
2022-06-29T08:40:46.905273Z critical envoy backtrace #38: std::function<>::operator()() [0x563ae538bfa5]
2022-06-29T08:40:46.940887Z critical envoy backtrace #39: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x563ae934eb3d]
2022-06-29T08:40:47.026670Z critical envoy backtrace #40: Envoy::Event::DispatcherImpl::DispatcherImpl()::$_3::operator()() [0x563ae9352a48]
2022-06-29T08:40:47.112979Z critical envoy backtrace #41: std::_Function_handler<>::_M_invoke() [0x563ae935290d]
2022-06-29T08:40:47.113012Z critical envoy backtrace #42: std::function<>::operator()() [0x563ae538bfa5]
2022-06-29T08:40:47.199023Z critical envoy backtrace #43: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::operator()() [0x563ae96
2022-06-29T08:40:47.310752Z critical envoy backtrace #44: Envoy::Event::SchedulableCallbackImpl::SchedulableCallbackImpl()::$_0::__invoke() [0x563ae963d
2022-06-29T08:40:47.497682Z critical envoy backtrace #45: event_process_active_single_queue [0x563ae969222c]
2022-06-29T08:40:47.613175Z critical envoy backtrace #46: event_process_active [0x563ae968caa8]
2022-06-29T08:40:47.806289Z critical envoy backtrace #47: event_base_loop [0x563ae968b9ac]
2022-06-29T08:40:47.917951Z critical envoy backtrace #48: Envoy::Event::LibeventScheduler::run() [0x563ae963ae3f]
2022-06-29T08:40:48.104769Z critical envoy backtrace #49: Envoy::Event::DispatcherImpl::run() [0x563ae934e662]
2022-06-29T08:40:48.218057Z critical envoy backtrace #50: Envoy::Server::WorkerImpl::threadRoutine() [0x563ae82660cb]
2022-06-29T08:40:48.418346Z critical envoy backtrace #51: Envoy::Server::WorkerImpl::start()::$_6::operator()() [0x563ae8267f00]
2022-06-29T08:40:48.606595Z critical envoy backtrace #52: std::_Function_handler<>::_M_invoke() [0x563ae8267d2d]
2022-06-29T08:40:48.606635Z critical envoy backtrace #53: std::function<>::operator()() [0x563ae538bfa5]
2022-06-29T08:40:48.797299Z critical envoy backtrace #54: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x563ae9aa83ed]
2022-06-29T08:40:48.911041Z critical envoy backtrace #55: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x563ae9aa83c5]
2022-06-29T08:40:48.911146Z critical envoy backtrace #56: start_thread [0x7f42a281f6db]

it seems it happend again, and stack trace ,it seems the lambada update cause ecds segmentfault ?
@kyessenov

kyessenov · 2022-06-30T16:14:04Z

Can you confirm this is on latest Envoy? The error is about removal of a filter from a non-main worker, which I think we handled.

huanghuangzym · 2022-07-01T01:27:20Z

Can you confirm this is on latest Envoy? The error is about removal of a filter from a non-main worker, which I think we handled.

we used envoy version is
version: edfec5636eb3494b1e5728891f8c0a6eba3e7d9f/1.20.3-dev/Clean/DEBUG/BoringSSL

and we cherry-pick this pr
#19630 build it self
can help to show which commit remove of a filter from a non-main worker?

kyessenov · 2022-07-08T18:16:26Z

I don't seem to have SHA edfec563.
I have trouble understanding how this happens. ECDS should delete Wasm filter config last on main thread due to runOnAllThreads. Is there any way to reproduce this issue?

fatedier · 2022-09-08T07:38:17Z

@kyessenov I use istio1.13.7 and envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL

Segmentation fault occured when I modify a wasm filter configruation.

2022-09-08T07:28:01.889994Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x0
2022-09-08T07:28:01.890037Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2022-09-08T07:28:01.890042Z     critical        envoy backtrace Envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL
2022-09-08T07:28:01.890303Z     critical        envoy backtrace #0: __restore_rt [0x7f0fa5ca1420]
2022-09-08T07:28:01.899460Z     critical        envoy backtrace #1: Envoy::Filter::FilterConfigSubscription::onConfigUpdate() [0x55e1d1ce7ca3]
2022-09-08T07:28:01.908382Z     critical        envoy backtrace #2: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x55e1d1d8e129]
2022-09-08T07:28:01.917527Z     critical        envoy backtrace #3: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x55e1d1d93eb7]
2022-09-08T07:28:01.926208Z     critical        envoy backtrace #4: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x55e1d1d95ba6]
2022-09-08T07:28:01.934716Z     critical        envoy backtrace #5: Envoy::Grpc::AsyncStreamImpl::onData() [0x55e1d1dc82c5]
2022-09-08T07:28:01.943020Z     critical        envoy backtrace #6: Envoy::Http::AsyncStreamImpl::encodeData() [0x55e1d1dcd6c2]
2022-09-08T07:28:01.951295Z     critical        envoy backtrace #7: Envoy::Router::UpstreamRequest::decodeData() [0x55e1d1e00bc9]
2022-09-08T07:28:01.959942Z     critical        envoy backtrace #8: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55e1d1b8350b]
2022-09-08T07:28:01.968511Z     critical        envoy backtrace #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x55e1d1d54708]
2022-09-08T07:28:01.977042Z     critical        envoy backtrace #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_25::__invoke() [0x55e1d1d5e660]
2022-09-08T07:28:01.985656Z     critical        envoy backtrace #11: nghttp2_session_on_data_received [0x55e1d20e31ec]
2022-09-08T07:28:01.994141Z     critical        envoy backtrace #12: nghttp2_session_mem_recv [0x55e1d20e525f]
2022-09-08T07:28:02.002280Z     critical        envoy backtrace #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d52e90]
2022-09-08T07:28:02.010711Z     critical        envoy backtrace #14: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d53bd5]
2022-09-08T07:28:02.018989Z     critical        envoy backtrace #15: Envoy::Http::CodecClient::onData() [0x55e1d1c20910]
2022-09-08T07:28:02.026995Z     critical        envoy backtrace #16: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x55e1d1c21ab5]
2022-09-08T07:28:02.036027Z     critical        envoy backtrace #17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55e1d1fe824f]
2022-09-08T07:28:02.046409Z     critical        envoy backtrace #18: Envoy::Network::ConnectionImpl::onReadReady() [0x55e1d1fe09e6]
2022-09-08T07:28:02.055543Z     critical        envoy backtrace #19: Envoy::Network::ConnectionImpl::onFileEvent() [0x55e1d1fde3df]
2022-09-08T07:28:02.064443Z     critical        envoy backtrace #20: std::__1::__function::__func<>::operator()() [0x55e1d1fd3c51]
2022-09-08T07:28:02.074518Z     critical        envoy backtrace #21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55e1d1fd515c]
2022-09-08T07:28:02.082977Z     critical        envoy backtrace #22: event_process_active_single_queue [0x55e1d20fa220]
2022-09-08T07:28:02.091519Z     critical        envoy backtrace #23: event_base_loop [0x55e1d20f8c31]
2022-09-08T07:28:02.100503Z     critical        envoy backtrace #24: Envoy::Server::InstanceImpl::run() [0x55e1d194a19b]
2022-09-08T07:28:02.109335Z     critical        envoy backtrace #25: Envoy::MainCommonBase::run() [0x55e1d0565264]
2022-09-08T07:28:02.117919Z     critical        envoy backtrace #26: Envoy::MainCommon::main() [0x55e1d05659c6]
2022-09-08T07:28:02.126530Z     critical        envoy backtrace #27: main [0x55e1d0561c3c]
2022-09-08T07:28:02.126699Z     critical        envoy backtrace #28: __libc_start_main [0x7f0fa5abf083]
AsyncClient 0x55e1d43a6d00, stream_id_: 16569796223946643133
&stream_info_:
  StreamInfoImpl 0x55e1d43a6ed0, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_:   upstream_info_:
    UpstreamInfoImpl 0x55e1d4730378, upstream_connection_id_: 18774
Http2::ConnectionImpl 0x55e1d42ed4b0, max_headers_kb_: 60, max_headers_count_: 100, per_stream_buffer_limit_: 268435456, allow_metadata_: 0, stream_error_on_invalid_http_messaging_: 0, is_out
bound_flood_monitored_control_frame_: 0, dispatching_: 1, raised_goaway_: 0, pending_deferred_reset_streams_.size(): 0
&protocol_constraints_:
  ProtocolConstraints 0x55e1d42ed530, outbound_frames_: 0, max_outbound_frames_: 10000, outbound_control_frames_: 0, max_outbound_control_frames_: 1000, consecutive_inbound_frames_with_empty_
payload_: 0, max_consecutive_inbound_frames_with_empty_payload_: 1, opened_streams_: 1, inbound_priority_frames_: 0, max_inbound_priority_frames_per_stream_: 100, inbound_window_update_frames
_: 39, outbound_data_frames_: 57, max_inbound_window_update_frames_per_data_frame_sent_: 10
Number of active streams: 1, current_stream_id_: 1 Dumping current stream:
stream:
  ConnectionImpl::StreamImpl 0x55e1d4178400, stream_id_: 1, unconsumed_bytes_: 0, read_disable_count_: 0, local_end_stream_: 0, local_end_stream_sent_: 0, remote_end_stream_: 0, data_deferred
_: 0, received_noninformational_headers_: 1, pending_receive_buffer_high_watermark_called_: 0, pending_send_buffer_high_watermark_called_: 0, reset_due_to_messaging_error_: 0, cookies_:   pen
ding_trailers_to_encode_:   null
  absl::get<ResponseHeaderMapPtr>(headers_or_trailers_):   null
Dumping corresponding downstream request for upstream stream 1:
  UpstreamRequest 0x55e1d472cc00
2022-09-08T07:28:02.246843Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-2 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.246851Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-1 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.402659Z     error   Epoch 0 exited with error: signal: segmentation fault (core dumped)
2022-09-08T07:28:02.402694Z     info    No more active epochs, terminating

kyessenov · 2022-10-25T19:03:52Z

Istio 1.13 doesn't have this fix.

bianpengyuan added bug triage Issue requires triage labels Feb 3, 2021

mattklein123 added area/configuration and removed triage Issue requires triage labels Feb 12, 2021

github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Mar 14, 2021

github-actions bot closed this as completed Mar 21, 2021

bianpengyuan mentioned this issue Dec 21, 2021

ECDS based Wasm filter configuration causes proxy segmentation fault istio/istio#35267

Closed

15 tasks

bianpengyuan mentioned this issue Jan 20, 2022

Bug of Extension Config Discovery Service #19628

Closed

kyessenov mentioned this issue Jan 20, 2022

ecds: rationalize ownership #19630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfault from ECDS update #14930

Segfault from ECDS update #14930

bianpengyuan commented Feb 3, 2021

snowp commented Feb 3, 2021

bianpengyuan commented Feb 3, 2021

github-actions bot commented Mar 14, 2021

github-actions bot commented Mar 21, 2021

bianpengyuan commented Dec 21, 2021

kyessenov commented Dec 22, 2021

bianpengyuan commented Dec 22, 2021

kyessenov commented Dec 22, 2021

kyessenov commented Jan 14, 2022

kyessenov commented Jan 14, 2022

rahulanand16nov commented Mar 14, 2022

kyessenov commented Mar 14, 2022

rahulanand16nov commented Mar 14, 2022

huanghuangzym commented Jun 30, 2022

kyessenov commented Jun 30, 2022

huanghuangzym commented Jul 1, 2022

kyessenov commented Jul 8, 2022

fatedier commented Sep 8, 2022

kyessenov commented Oct 25, 2022

Segfault from ECDS update #14930

Segfault from ECDS update #14930

Comments

bianpengyuan commented Feb 3, 2021

snowp commented Feb 3, 2021

bianpengyuan commented Feb 3, 2021

github-actions bot commented Mar 14, 2021

github-actions bot commented Mar 21, 2021

bianpengyuan commented Dec 21, 2021

kyessenov commented Dec 22, 2021

bianpengyuan commented Dec 22, 2021

kyessenov commented Dec 22, 2021

kyessenov commented Jan 14, 2022

kyessenov commented Jan 14, 2022

rahulanand16nov commented Mar 14, 2022

kyessenov commented Mar 14, 2022

rahulanand16nov commented Mar 14, 2022

huanghuangzym commented Jun 30, 2022

kyessenov commented Jun 30, 2022

huanghuangzym commented Jul 1, 2022

kyessenov commented Jul 8, 2022

fatedier commented Sep 8, 2022

kyessenov commented Oct 25, 2022