envoy1.7 can't work with native jaeger, envoy depend on opentracing 1.4.2, but jaeger-client-cpp depend on 1.4.0

[xmm1989218]

Issue Template

Title: One line description

Description:

Describe the issue. Please be detailed. If a feature request, please
describe the desired behaviour, what scenario it enables and how it
would be used.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

Bug Template

Title: One line description

Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: Envoy should not crash, the expected value isn't
returned, etc.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

Note: The Envoy_collect tool
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

Include the config used to configure Envoy.

Logs:

Include the access logs and the Envoy logs.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required.
Please refer to the Bazel Stack trace documentation.

[mattklein123]

cc @objectiser

[objectiser]

I think this is best answered by @rnburn and @isaachier.

@isaachier Will a release including jaegertracing/jaeger-client-cpp#120 fix this?

[isaachier]

Ya I'll try to get one out today.

[isaachier]

Done. See https://github.com/jaegertracing/jaeger-client-cpp/releases/v0.4.2.

[BenPope]

I was building this earlier, envoy currently (v1.7.1) uses opentracing v1.4.2. Jaeger now uses opentracing v1.5.0. Envoy failed to load the so.

I'll investigate further.

[BenPope]

I had previously built jaeger-client-cpp v0.4.1 against opentracing-cpp v1.4.2 and had been seeing segfaults.

I updated envoy to use opentracing v1.5.0, libjaegertracing_plugin.so v0.4.2 is successfully loaded, but I'm still getting segfaults.

I'm trying to get a sensible core dump.

[isaachier]

Any luck @BenPope?

[BenPope]

Not yet. Previous core dump was full of ??, I suspect the stack was trashed. I'm trying to build with asan, but struggling to get envoy to run, seems to be an issue with ld. Strangely it will run after I've done apk add --no-cache install bash, but if I build that into the image envoy still won't run.

[isaachier]

I have had issues in the past with missing symbols/dynamic libs not found. You can try ldd libjaegertracing_plugin.linux_amd64.so and post the output.

[xmm1989218]

yesterday I try to fix it, use envoy master, then change opentracing to 1.5.0 and use jaeger-client-cpp master(depend on opentracing 1.5.0), will no segfault, but I met a new problem, "param: 1" which in jaeger-native-tracing/front-envoy-jaeger.yaml will parsed as string, finally in envoy log will see "invalid configuration .... at line". it's so strange. I run tests use jaeger-client-cpp, it can parse successful.

[isaachier]

Please don't use a custom build. Use the release I posted above.

[xmm1989218]

@isaachier use envoy master, then change opentracing to 1.5.0 and use https://github.com/jaegertracing/jaeger-client-cpp/releases/download/v0.4.2/libjaegertracing_plugin.linux_amd64.so. it works well. thanks.

[isaachier]

@xmm1989218 sorry for the problems you experience and glad they are gone. Would you mind closing this now?

[BenPope]

@isaachier I haven't been able to spend a great deal of time on this, but the crashing doesn't appear to be due to tracing, so I'm also happy for this to be closed.

[isaachier]

@BenPope as I explained to @xmm1989218, custom builds of Jaeger are likely to crash the host application. A specific build process is required for the dynamic loading to work. This is the reason we release the shared object as part of the release. It seems you had been using a custom build as well, in which case, the same fix should work.

[BenPope]

@isaachier The custom build was done as I couldn't find a version of the plugin that envoy would load without segfault (which I believed to be due to mismatched opentracing version).

Prior to building the plugin and envoy inside the envoy-build container, I was using something like this and tweaking version numbers:

FROM envoyproxy/envoy-alpine:v1.7.0
ADD https://github.com/jaegertracing/jaeger-client-cpp/releases/download/v0.4.2/libjaegertracing_plugin.linux_amd64.so /usr/local/lib/libjaegertracing_plugin.so
ENTRYPOINT ["envoy"]

[isaachier]

OK at least you have not built Jaeger from sources. That's what I thought you had done.