The need for more granularity/clarity in CLIENT span conventions

[jkwatson]

What are you trying to achieve?

Currently, the span specification reads, when describing span kinds:

CLIENT Indicates that the span describes a synchronous request to some remote service. This span is the parent of a remote SERVER span and waits for its response.

This seems fine on the surface, but when you get into the details of actually writing instrumentation (and auto-instrumentation in particular), things get murky very quickly. Multiple layers of an application-framework stack can (and do) consider themselves logically as CLIENTs, even if none of them is actually opening up TCP connections and writing bytes to the wire.

For example, let's take a typical java HTTP call. Here are some possible spans that might want to be created by auto-instrumentation:

• JAX-RS HTTP client library makes a GET call
  • Apache HTTP client library makes a GET call
    • Java TCP stack opens up a socket
      • Java TCP stack requests that the OS makes a DNS lookup
      • Java TLS implementation negotiates TLS with the server

(details here might be off, but I think the idea is clear).

So, which of these operations should create a CLIENT span? If library instrumentation for JAX-RS decides it should be a CLIENT, how can the Apache HTTP client library instrumentation create spans at all, given that a CLIENT has already been created (although it can't tell that, since the parent span is write-only, and the kind is immutable after the span has been created!).

At the moment, instrumentation does not have a way to deal with this situation, with the spec as-written. We should solve this problem and provide clarity on how instrumentation should be written in order to handle scenarios like this one.

I will also note, that it becomes even more confusing and difficult when dealing with databases that implement their protocol over HTTP (elastic, for example). Now we have database CLIENT spans that really need HTTP CLIENT spans below them, with all the additional complexity of observing the underlying network stack, as decsribed above.

[jkwatson]

Note: see this discussion in the java-instrumentation repository: open-telemetry/opentelemetry-java-instrumentation#1822

[Oberon00]

Probably we have to live with that. It is impossible for the parent to know if there will be a child, as maybe the child is not instrumented.

This span is the parent of a remote SERVER span and waits for its response.

This is not true anyway, since the remote end might not be instrumented at all. Probably it should relaxed to say that it can also be an indirect parent.

[reyang]

Related to #110

[blumamir]

I find this attribute useful only when there is a possibility for ambiguity - the same looking span can be generated from the same instrumentation library for both incoming and outgoing operations.

For databases, I never need to look at the kind, as I know that the operation can only be outgoing request.

At least for my usage, it is most comfortable and makes the most sense to tag everything that is by nature "logical output" of the application as CLIENT/PRODUCER, and everything that is "logical input" to the application as SERVER/CONSUMER.

[jkwatson]

Unless there is strong objection, I will write up a spec change that allows nested logical CLIENT/SERVER/etc spans to unstick this issue.