Discusses trace propagation concerns with trace reporting #42
Conversation
This formalizes a growing problem where tracing systems are used in a mesh environment, instrumented with B3, and proxying the trace data endpoint. This was noticed before in Zipkin, but the problem was rare. Now that it is a routine concern, it is worth informing others using B3 format. Thanks @basvanbeek for the analysis and @anuraaga for input leading to this.
| We recommend having your trace reporting library send `b3: 0`. This cements a | ||
| cost to everyone, only to help those proxying their trace data. However, it is | ||
| becoming common in mesh environments to proxy everything, and in reality there is | ||
| limited ability for end users to change the sampling policy of proxies. |
There was a problem hiding this comment.
Is https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-go-configuration.html#xray-sdk-go-configuration-sampling is a sample of "sampling policy" in this context?
There was a problem hiding this comment.
exactly.
spring-cloud-sleuth has similar except the policy can merge from different sources, ex a deny list for well known paths that shouldn't be traced, and user-supplied policy.
however you compose the list, it would look like amazon's format in abstract, especially subject to a request rate (per second) is better than a percentage (0.1). amazon I think mixes these words a little but in general the format is good.
There was a problem hiding this comment.
an interesting more comprehensive format for trace config is this, mentioning for research, not necessarily direct input to how proxy config works https://github.com/kamon-io/Kamon/blob/master/core/kamon-core/src/main/resources/reference.conf#L218-L226
| data sent to a tracing system, there are generally 2 ways to address this proxy | ||
| setup: | ||
|
|
||
| 1. Configure the proxy to not trace your trace reporting endpont (Ex in Zipkin, `POST /api/v2/spans`) |
There was a problem hiding this comment.
Aside to anyone who can control the code of proxies:
It would be a much better solution for them to add the endpoint they have configured for their tracer to a "deny list" for HTTP sampling policy. In other words, the sampling policy dictated by users would collaborate with the system concern like this. Users don't need to do any config maintenance this way.
Such an approach would be safe as intentionally tracing the tracing system is edge case, and tracing the tracing system through a proxy, is even more so.
Understanding some changes to proxies aren't accepted or can take many months to merge, then even longer to roll out, this is more mentioned for long term vs an expectation of this changing anything today.
This formalizes a growing problem where tracing systems are used in a mesh
environment, instrumented with B3, and proxying the trace data endpoint.
This was noticed before in Zipkin, but the problem was rare. Now that it is
a routine concern, it is worth informing others using B3 format.
Thanks @basvanbeek for the analysis and @anuraaga for input leading to
this.