Support for Akka HTTP connection source

[zackman0010]

Currently, the Akka HTTP instrumentation only supports using .bindAndHandleSync and .bindAndHandleAsync methods (.bind and .bindAsync in Akka HTTP 10.2's new ServerBuilder methods). I'd like to add support for the connection source method (Http().bind in 10.0 and 10.1, builder.connectionSource() in 10.2).

The connection source method works by providing the user with a Source[Http.IncomingConnection]. We can instrument this class's .handleWith function to wrap the handler in instrumentation.

The instrumentation required to wrap .handleWith can also be used to complete #5137, as they require wrapping a handler of type Flow.

We're currently internally testing an implementation of this instrumentation, but decided to create this issue first for feedback as recommended in the contributing guidelines.

[laurit]

@zackman0010 feel free to submit a PR

[zackman0010]

We're still testing our implantation internally, but we'll be submitting a PR as soon as we're done!

We're actually running into a strange issue right now. We're calling instrumenter().start(..) and context.setCurrent() on each request, saving both the Context and Scope values to be used after the response returns. When we get the response, we make sure to call instrumenter().end(..) with the same Context provided by the call to start and calling scope.close() on the Scope provided by the setCurrent call.
Everything appears to be working correctly. We can look in Grafana and see the parent span generated by our instrumentation and all the children spans generated by the application code underneath our HTTP server. However, later requests are reusing the same trace IDs. When we investigated, it appears to happen because currentContext() is still returning the Context object from the previous request.

Do you have any suggestions on why this might be happening? We can go ahead and push our PR if you'd need to see code.

[trask]

hi @zackman0010, it sounds most likely that you are leaking a Scope, meaning you are setting the context as current (which binds it to thread local) and then not closing the associated scope in the same thread.

it's best to always call makeCurrent in a try-with-resources block, or if you are inside of bytebuddy advice it's a little tricker, but make sure you are starting the scope in OnMethodEnter, passing the Scope object to OnMethodExit and closing it there

[zackman0010]

That sounds very plausible, I know Akka doesn't play well with ThreadLocals. The comment on the existing Akka instrumentation about there being no clean way to handle closing scopes is making more sense now as well.

We'll keep looking into this, see if we can find a way around it.

[zackman0010]

We successfully found a workaround to fix the scope leakage. Akka has the concept of a PinnedDispatcher, which is used to enforce a "one thread per actor" rule. Internally, it assigns each actor with a thread pool of size 1, which causes that actor to exclusively use that one thread as well as to have exclusive use of that thread.

We were able to use instrumentation to inject a new PinnedDispatcher into the configuration, then have have the wrapper we made utilize our dispatcher instead of Akka's default dispatcher. This fixed the Scope leakage issue that we were previously seeing.

We created PR #6109 for both this issue and the issue described in #5137, as both could be fixed with the same dispatcher/wrapper combination.