Response headers aren't merged across responding threads

[dnhatn]

This is related to #99926.

It seems that our infrastructure does not merge response headers across multiple asynchronous calls. Response headers are not merged properly in this scenario:

1. The caller initiates two asynchronous calls, c1 and c2, which can involve network requests.

2. c1 responded with a warning in the header responses. We merge these response headers with the original ThreadContext of the calling thread and update the ThreadContext of the current thread while leaving the ThreadContext of the calling thread untouched.

3. c2 responded with no warning in the header responses. Since the original ThreadContext of the calling thread did not get updated after c1, as it's immutable, we won't be able to merge response headers between c1 and c2.

4. The caller receives a response from the responding thread of c2 without any warning

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[dnhatn]

I've thought about this issue a bit more. I think the task framework can provide a solution without requiring changes in other places. The parent task should collect and merge response headers from its child tasks. While it may not be a complete solution, it's an appealing option due to its contained nature. I'm tagging the @elastic/es-distributed team for their awareness.

[ywangd]

It seems to me that threadContext management is an explicit choice of the caller. The ThreadContext class provides an array of methods so that request and response headers can be preserved or dropped. Therefore baking response header merging into the task framework seems to be too coarse grained. Would either GroupedActionListener or RefCountingListener be a better place for having this behaviour? My thinking is that these classes can be instantiated with an extra parameter to indicate how response headers should be handled across its multiple threads.

[DaveCTurner]

I was wondering if it should be even lower-level: anywhere that we fan-in responses could potentially need to combine response headers from different thread contexts. There's various fundamental fan-in mechanisms, including CountDown and AbstractRefCounted.