[EXAMPLE] The otlp_http client crashes on exit #1867
Comments
|
Fix below, I will send a PR to fix all clients. |
|
This problem seems relate to #1718 and #1767 .Maybe it's a better way to use a guard variable to tell whether the global handle is still available when calling |
|
This problem is not limited to examples, lots of other apps where people already use opentelemetry-cpp library or will use it in the future are affected too. This problem requires proper fix in opentelemetry-cpp library itself. Looks that library needs some global cleanup handler which would deinit everything in proper order. One approach is to have priority queue, where every part of opentelemetry-cpp could register its own deinit routine. This queue will be then processed in exit handler. I also would like to see new global function to explicitly initiate shutdown when app exits. This can be useful for apps which register their own log handler using |
|
Hi @sirzooro I agree with you that the opentelemetry-cpp library needs a more robust way to initialize and shutdown. This will be needed in particular to support features like the because currently there is no obvious places to put initialization code to implement these features. Would the following pattern be better ? One issue here is that the global tracer provider is held with a Agreed, CleanupTracer() is a work around, I will be happy to have more robust fix as well. |
|
It looks that for me it would not change things too much. You can go with it if it is better for you. I have helper library for OpenTelemetry, which performs initialization and shutdown. Main in my app looks like below. It would be fine as long as everything can be put into single main()
{
AppLogger::init();
OtelHelper::init();
// app init also creates tracer instance for the app, and inits other libs which may create their tracers too
App::init();
App::run();
App::shutdown();
OtelHelper::shutdown();
AppLogger::shutdown();
} |
|
I can't reproduce the crash, which platform is it ? labhas@DESKTOP-D0BLHPQ:~/obs/ot/opentelemetry-cpp/build/examples/otlp$ ./example_otlp_http
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:213 [OTLP HTTP Client] Session state: connection failed.Couldn't connect to server
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_exporter.cc:107 [OTLP HTTP Client] ERROR: Export 1 trace span(s) error: 1
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:213 [OTLP HTTP Client] Session state: connection failed.Couldn't connect to server
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_exporter.cc:107 [OTLP HTTP Client] ERROR: Export 1 trace span(s) error: 1
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:213 [OTLP HTTP Client] Session state: connection failed.Couldn't connect to server
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_exporter.cc:107 [OTLP HTTP Client] ERROR: Export 1 trace span(s) error: 1
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:213 [OTLP HTTP Client] Session state: connection failed.Couldn't connect to server
[Error] File: /home/labhas/obs/ot/opentelemetry-cpp/exporters/otlp/src/otlp_http_exporter.cc:107 [OTLP HTTP Client] ERROR: Export 1 trace span(s) error: 1
labhas@DESKTOP-D0BLHPQ:~/obs/ot/opentelemetry-cpp/build/examples/otlp$ .I thought #1767 should have fixed the destruction order for GlobalLogHandler and *Provider, with GlobalLogHandler being destroyed in the end. |
Reproduced with opentelemetry-cpp 9197bf3 (current code as of Jan 3rd, 2023), |
Do not start any opentelemetry-collector process.
Execute:
./examples/otlp/example_otlp_http
It crashes in:
The line that crashes is:
From the call stack, after main() is done, C++ exit handlers now cleanup global variables.
In particular, the global trace provider shared pointer is destroyed, causing a shutdown on the provider.
This causes a shutdown on HttpClient, and the pending sessions.
The session cancel event causes a message to be written to the global log handler,
which is most likely long gone by now.
The opentelemetry-cpp shutdown needs to be more orderly.
The text was updated successfully, but these errors were encountered: