Tracing

[christiankjaer]

Very cool project.

Any thoughts on having a natchez module exported from this project? There might be some common parts.

Prior art: typelevel/natchez#539

[rossabaker]

@ChristopherDavenport is looking at adding a Tracing API. That's absolutely in scope; I just started small.

From there, this could certainly be a target for Natchez and be a good solution for libraries already instrumented with Natchez. But libraries might eventually wish to instrument against this core directly, to avoid that second wrapper and to get access to the full Otel spec.

[christiankjaer]

Sounds freaking great.

[rossabaker]

I'll leave it open so someone eventually connects the natchez dots when we are capable of tracing.

[iRevive]

Not sure how suitable this issue is, but I would like to share my pain points from natchez experience:

1) No access to spanId from Trace[F]

It's a matter of the API. spanId is accessible with a few workarounds, but it would be nice to have it out of the box:

trait Trace[F[_]] {
  def traceId: F[Option[String]]
  def spanId: F[Option[String]]
} 

for {
  traceId <- Trace[F].traceId
  spanId  <- Trace[F].spanId
  _       <- logger.info("A log correlated with the trace", ctx = Map("traceId" -> traceId, "spanId" -> spanId))
} yield ()

2) No option to create a new root span from Trace[F]

It's possible to do this from EntryPoint, but the entry point cannot be passed to the inner components when traces are managed via Kleisli[F, Span[F], *].

3) No option to run an effect without tracing

Trace[F].span("create-db") {
  for {
    pool <- createDBPool
    _    <- reportLeakedResource(pool).runForever.background
  } yield ()
}

Even though reportLeakedResource is running in the background, it captures the outside span at the start. If reportLeakedResource has a step that creates a child span, that leads to an infinite create-db span.

It would be nice to have something like detached or similar:

Trace[F].span("create-db") {
  for {
    pool <- createDBPool
    _    <- Trace[F].detached(reportLeakedResource(pool)).runForever.start
  } yield ()
}

4) Proper tracing of a Resource

It would be nice to have a correct structure of traced resources. Details - typelevel/natchez#514.

5) Ability to trace fs2.Stream

A rare case for me, but it would be nice to have.

6) Memory pressure when Trace is not used

If the service uses Trace.Implicits.noop some API methods still allocate some objects:

for {
  _ <- trace.put("route" -> "route", "hash" -> calculateHash(value))
} yield () 

In the example above, calculateHash is still being evaluated, even though the trace is running in no-op mode.
Mostly related to this issue #3

7) Richer API

This one does not belong to otel4s-core but can live in an extras module perhaps.

trait Trace[F[_]] {
  def span[A](name: String)(fa: F[A]): F[A]
  def span[A](name: String, tags: TraceParam*)(fa: F[A]): F[A] = span(name)(put(tags) >> fa)
}

[rossabaker]

I have personally felt the pain of 2, 4, and 5 and have speculated on the pain of 6. And the rest look believable, too. 😄

[janstenpickle]

@iRevive bit of shameless self promotion on 1: Trace4Cats solves this here and here, plus provides interop with Natchez.

Regarding 2: I think it might partially addressed in t4c here

3: We have a noop span instance for just this problem

5: Is really hard, but I had a go here, hopefully it becomes clear in the code why it's hard, but basically each evalMap in an FS2 stream is its own program, so you have to exchange a span context between each closure. I do this by using a WriterT to pass the span along.

6: All t4c span attributes are lazily evaluated. Objects are still allocated for Eval, but processing like calculateHash isn't done unless the span is sampled

7: See here again

As I mentioned in #29 I think there is an opportunity to share a common core here