Establishing HTTP connection spec

[mateuszrzeszutek]

What are you trying to achieve?

I started working on implementing the http.resend_count spec in the Java instrumentations, and I ran into an interesting issue with the current spec.

Currently (that is, without the resends), in pretty much all Java HTTP client instrumentations, we are creating HTTP client spans for the outermost operation, usually the top-level interface exposed by a particular HTTP client library. This means that there is only one HTTP span created for the entire operation; if any resends occur during the call they'll go unnoticed, and the child SERVER spans will all receive the same parent span id (which is why we need to modify/rewrite all HTTP client instrumentations we have, they're all too "shallow"). The HTTP span also includes the "establishing the connection" phase (in most cases; I'll mention the exceptions to that later); in case a connection error occurs (closed port, server not responding at all, etc) the HTTP span will end with an error, and the telemetry about the thrown exception will be recorded.

This changes with the resend counter spec. As I understand it, we must now create a span for every attempt to actually send an HTTP request over the wire (and bump the counter if there's a retry). Most HTTP clients that I've read through separate the "establishing the connection" phase from the "sending the request" phase pretty cleanly; which means that whenever a DNS resolution/connection/TLS handshake error occurs, the actual HTTP instrumentation won't catch that - because it runs later than that.

I find this to be somewhat problematic, since usually you'd want to know if this kind of situations occur with your application.

There are several low-level HTTP clients that completely separate these two phases - where any details about the HTTP request are simply not accessible at the connection phase (mostly Netty and Netty-based HTTP clients). For these, we already had to introduce something that'll capture the connection level details, which we've called a CONNECT span. Note that this already exists in some of our instrumentations, and predates the resend spec.

Now, to solve the problem with the missing connection-level telemetry we could do one of the following things:

• In the top-level HTTP client interface: if there was an exception thrown, and no attempts to send an HTTP request have been made, create a "fake" HTTP span just for the connection error. (Note that this is impossible to implement in the Netty-based clients that I've briefly mentioned above)
• Introduce the CONNECT span to the HTTP client spec: create a CONNECT span whenever a connection is established (actual connection; taking a connection from a pool should not emit a span), regardless of whether it was successful or not. (We've had customers ask us to implement spans that capture low-level connection details; this kind of information would be generally useful, at least for some people).

Additional context.

• HTTP resend PR: Implement HTTP resend spec for OkHttp 3 opentelemetry-java-instrumentation#7652
• Some examples of the existing CONNECT spans in the javaagent (tests): https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/netty/netty-4.1/javaagent/src/test/groovy/Netty41ConnectionSpanTest.groovy#L116-L126, https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/reactor/reactor-netty/reactor-netty-1.0/javaagent/src/test/java/io/opentelemetry/javaagent/instrumentation/reactornetty/v1_0/ReactorNettyConnectionSpanTest.java#L99-L107

[mateuszrzeszutek]

CC @trask this is a part of the HTTP semconv effort

[lmolkova]

@mateuszrzeszutek great find!

I'd like to entertain the idea of having top-level logical HTTP span to capture connect time, DNS time, circuit-breaking, the duration for tries with backoff, and overall operation success.

We might need to give instrumentations some freedom to instrument only client call, HTTP attempts, or both:

• I heard from @tedsuo that AWS includes full HTTP request with tracing headers when creating request signature and it can't change between tries. (on Azure, we only include x-ms-* headers or a closed set of headers into the signature)
• Client library instrumentations instrument public API surface anyway and don't necessarily need top-level span to account for other things

[mateuszrzeszutek]

We might need to give instrumentations some freedom to instrument only client call, HTTP attempts, or both:

• I heard from @tedsuo that AWS includes full HTTP request with tracing headers when creating request signature and it can't change between tries. (on Azure, we only include x-ms-* headers or a closed set of headers into the signature)
• Client library instrumentations instrument public API surface anyway and don't necessarily need top-level span to account for other things

💯

Had the same thoughts exactly -- I'm pretty sure there are HTTP clients that may do retries but it's impossible to instrument each send attempt because of how code is structured (I did not have a good example of that before; the AWS SDK is a great one), and the spec should also somehow fit these cases too.

[trask]

@mateuszrzeszutek do you think we still need this for Java http instrumentation stability? (or now with open-telemetry/opentelemetry-specification#3290, is the Java http instrumentation default state addressed, and we just need this for opt-in behaviors)

[mateuszrzeszutek]

@mateuszrzeszutek do you think we still need this for Java http instrumentation stability? (or now with open-telemetry/opentelemetry-specification#3290, is the Java http instrumentation default state addressed, and we just need this for opt-in behaviors)

OOF, I totally missed this issue 🙈
We've talked about this offline, I don't think it's required; at this point this is an addition that might be introduced later.