crash in ext::http::client::curl::HttpOperation::Send

[ZDzhao]

Enviroment: CentOS Linux release 7.3.1611 (Core)

#0 0x00007ff6ee439a37 in abort () from /lib64/libc.so.6
#1 0x00007ff6ee477f47 in __libc_message () from /lib64/libc.so.6
#2 0x00007ff6ee512d87 in __fortify_fail () from /lib64/libc.so.6
#3 0x00007ff6ee510f40 in __chk_fail () from /lib64/libc.so.6
#4 0x00007ff6ee512cf7 in __fdelt_warn () from /lib64/libc.so.6
#5 0x0000000000d55ed8 in WaitOnSocket (timeout_ms=, for_recv=0, sockfd=3175)
at bazel-out/k8-opt/bin/external/io_opentelemetry_cpp/ext/_virtual_includes/headers/opentelemetry/ext/http/client/curl/http_operation_curl.h:460
#6 opentelemetry::v1::ext::http::client::curl::HttpOperation::Send (this=0x7ff6b4c02d00)
at bazel-out/k8-opt/bin/external/io_opentelemetry_cpp/ext/_virtual_includes/headers/opentelemetry/ext/http/client/curl/http_operation_curl.h:214
#7 0x0000000000d5a3f9 in operator() (__closure=0x7ff3e0f4fc88)
at bazel-out/k8-opt/bin/external/io_opentelemetry_cpp/ext/_virtual_includes/headers/opentelemetry/ext/http/client/curl/http_operation_curl.h:298
#8 __invoke_impl<long int, opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void(opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::<lambda()> > (__f=)
at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#9 __invoke<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void(opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::<lambda()> > (__fn=) at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:96
#10 _M_invoke<0> (this=0x7ff3e0f4fc88) at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#11 operator() (this=0x7ff3e0f4fc88) at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#12 operator() (this=0x7ff3971fb720) at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/future:1339
#13 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >, long> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:302
#14 0x00000000006146bf in operator() (this=)
at /usr/local/gcc-7.5/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706

same as this question https://stackoverflow.com/questions/22827510/how-to-avoid-bad-fd-set-buffer-overflow-crash

[lalitb]

@ZDzhao Can you share a bit more context on this :

• Is this OTLP HTTP export?
• What commit id/release of otlp-cpp ?
• How often the crash is happening, and when is it happening - Is it after processing a few ( hundreds/ thousands) spans?

Ideally, fd/socket buffer-overflow shouldn't happen, as the Export is sequential i.e, next export will be initiated only after the previous export is completed. So there is shouldn't be more than one socket connection open at a given time. Would be interesting to understand the scenario here.

[github-actions]

This issue was marked as stale due to lack of activity. It will be closed in 7 days if no furthur activity occurs.

[github-actions]

Closed as inactive. Feel free to reopen if this is still an issue.

[taozhijiang]

Same here.
Tested with SDK v1.2.0 and pre v1.4.0 both happens.

[owent]

Could you please test async-changes?
We will not call select in async-changes and I will start to work on merge async-changes back to main soon.

[Zjie]

Same here. Test with SDK v1.2.0 and using OTLP HTTP export.

Env: gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)

Code:
auto span = tracer->StartSpan(trace_info->span_name_, op);
for (const auto& tag : trace_info->attributes_) {
span->SetAttribute(tag.first, tag.second);
}
span->End(end);

Stack:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f7113f27b25 in __GI_abort () at abort.c:79
#2 0x00007f7113f80897 in __libc_message (action=,
fmt=fmt@entry=0x7f711408ce45 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007f7114013905 in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=false,
msg=msg@entry=0x7f711408ce23 "stack smashing detected") at fortify_fail.c:28
#4 0x00007f71140138b8 in __stack_chk_fail () at stack_chk_fail.c:29
#5 0x000000000081c0a1 in opentelemetry::v1::ext::http::client::curl::HttpOperation::Send (this=0x7f614c07b570)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/basic_string.h:2290
#6 0x0000000000822879 in opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void
(opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}::operator()() const (
__closure=0x7f614c002578)
at bazel-out/k8-fastbuild/bin/external/io_opentelemetry_cpp/ext/_virtual_includes/headers/opentelemetry/ext/h
ttp/client/curl/http_operation_curl.h:299
#7 std::__invoke_impl<long, opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<v
oid (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}>(std::__invoke_other, openteleme
try::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::clie
nt::curl::HttpOperation&)>)::{lambda()#1}&&) (
__f=<unknown type in /usr/local/trpc/bin/xsearch_data_import, CU 0x52572af, DIE 0x52974e5>)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#8 std::__invoke<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opente
lemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}>(std::__invoke_result&&, (opentelemetry::v1
::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::cur
l::HttpOperation&)>)::{lambda()#1}&&)...) (
__fn=<unknown type in /usr/local/trpc/bin/xsearch_data_import, CU 0x52572af, DIE 0x52974bd>)
---Type to continue, or q to quit---
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#9 std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::fu
nction<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >::_M_invoke<0ul>(std::
_Index_tuple<0ul>) (this=0x7f614c002578)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#10 std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::fu
nction<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >::operator()() (
this=0x7f614c002578) at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#11 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_base::_Resu
lt_base::_Deleter>, std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::S
endAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >, long

::operator()() const (this=0x7f5f0afb0fb0)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/future:1339
#12 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_D
eleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result, std::__future_bas
e::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOper
ation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}>
, long> >::_M_invoke(std::_Any_data const&) (__functor=...)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:283
#13 0x000000000081f85f in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Re
sult_base::_Deleter> ()>::operator()() const (this=)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/std_function.h:682
#14 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base,
std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x7f614c002540, __f=,
__did_set=0x7f5f0afb0f57) at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/future:561
---Type to continue, or q to quit---
#15 0x00007f7114d06e67 in __pthread_once_slow (once_control=0x7f614c002558,
init_routine=0x7f711636cbf0 <__once_proxy>) at pthread_once.c:116
#16 0x0000000000822bd3 in __gthread_once (__func=, __once=0x7f614c002558)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/x86_64-redhat-linux/bits/gthr-default.h:699
#17 std::call_once<void (std::__future_base::_State_baseV2::)(std::function<std::unique_ptr<std::__future_base::
_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool*), std::__future_base::_State_baseV2*, std::
function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool
>(std::once_flag&, void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::__future_bas
e::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool*), std::__future_base::_State_baseV2*&&,
std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&
&, bool&&) (__f=<unknown type in /usr/local/trpc/bin/xsearch_data_import, CU 0x52572af, DIE 0x52913cd>,
__once=...) at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/mutex:684
#18 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_ba
se, std::__future_base::_Result_base::_Deleter> ()>, bool) (__ignore_failure=false, __res=...,
this=0x7f614c002540) at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/future:401
#19 std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::http::client::
curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>):
:{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::
SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}>&&)::{l
ambda()#1}::operator()() const (this=)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/future:1667
#20 std::__invoke_impl<void, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opentelemetry
::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client:
:curl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::http::clie
nt::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&
---Type to continue, or q to quit---
)>)::{lambda()#1}>&&)::{lambda()#1}>(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Inv
oker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentele
metry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opent
elemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http:
:client::curl::HttpOperation&)>)::{lambda()#1}>&&)::{lambda()#1}&&) (
__f=<unknown type in /usr/local/trpc/bin/xsearch_data_import, CU 0x52572af, DIE 0x52912f8>)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:60
#21 std::__invoke<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opentelemetry::v1::ext::
http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::Http
Operation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::http::client::curl::H
ttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambd
a()#1}>&&)::{lambda()#1}>(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opentelemetry::v
1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::cu
rl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::http::client:
:curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOperation&)>)
::{lambda()#1}>&&)::{lambda()#1}&&, (std::__invoke_result&&)...) (
__fn=<unknown type in /usr/local/trpc/bin/xsearch_data_import, CU 0x52572af, DIE 0x52912d0>)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/bits/invoke.h:95
#22 std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opent
elemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http:
:client::curl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::ht
tp::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOp
eration&)>)::{lambda()#1}>&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7f614c079bf8)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:244
#23 std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<opent
---Type to continue, or q to quit---
elemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http:
:client::curl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<opentelemetry::v1::ext::ht
tp::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::http::client::curl::HttpOp
eration&)>)::{lambda()#1}>&&)::{lambda()#1}> >::operator()() (this=0x7f614c079bf8)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:253
#24 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::
_Invoker<std::tuple<opentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (open
telemetry::v1::ext::http::client::curl::HttpOperation&)>)::{lambda()#1}> >, long>::_Async_state_impl(std::tuple<o
pentelemetry::v1::ext::http::client::curl::HttpOperation::SendAsync(std::function<void (opentelemetry::v1::ext::h
ttp::client::curl::HttpOperation&)>)::{lambda()#1}>&&)::{lambda()#1}> > >::_M_run() (this=0x7f614c079bf0)
at /usr/lib/gcc/x86_64-redhat-linux/8/../../../../include/c++/8/thread:196
#25 0x00007f711636db73 in ?? () from /lib64/libstdc++.so.6
#26 0x00007f7114cff2de in start_thread (arg=) at pthread_create.c:486
#27 0x00007f7114001e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[lalitb]

@Zjie - Can you try with latest commit, as the issue is fixed with #1410.

[Zjie]

@Zjie - Can you try with latest commit, as the issue is fixed with #1410.

Thanks, I will try this commit later.

[Zjie]

#1410 works for me, as least it doesn't crash down anymore(so far). Anyway when will #1410 be released? @lalitb

[owent]

Close as #1413 is merged