Tempo Query Frontend Refactor

[joe-elliott]

The query frontend code has been pushed to its absolute limits. We need a refactor for improved code clarity, performance, enable streaming on all endpoints and pave the way for more advanced features demanded by TraceQL metrics.

The first PR has been opened and will address the first two tasks when merged: #3400

Tasks

• Build tools to construct asynchronous pipelines
• Migrate search to the new pipelines
• Migrate tag endpoints to the new pipelines
• Migrate trace id endpoints to the new pipelines. (remain synchronous but use new patterns)
• Migrate metrics summary api to the new pipelines. (remain synchronous but use new patterns)
• Build and use structures to support deterministic search
• Build experimental metrics streaming API

Stretch

• Measure/Improve job throughput. At least don't slow things down.
• Issue multiple ingester search queries instead of one.

Existing/Relevant Issues

• Query limits don't work or work unexpectedly  #1392
• Add streaming support to all read path endpoints #3366
• Deterministic search
  • TraceQL: Sorting traces when searching from Grafana #2659
  • Grafana Cloud Tempo: Default Start Time view for traces in Tempo does not order by most recent #3109

[joe-elliott]

We discovered some goroutine leaks in the existing implementation. Fix here:

#3495

[joe-elliott]

Tags streaming:

#3460

[joe-elliott]

Multiple ingester queries are being handled here:

#3574