store-gateway: merged series from different blocks concurrently

[dimitarvdimitrov]

Before series streaming was introduced the work for fetching series was done concurrently for all the blocks in the requests. After the changes in #3355 the series of each block are merged serially in mergedSeriesChunkRefsSet. The merged set checks each of its two items one after the other by calling ensureItemAvailableToRead for each (code). In reality ensureItemAvailableToRead may involve fetching the series from the bucket and/or the cache and can block for considerable amount of time.

This may become a problem for requests that query more than 2-3 blocks. Below is a tempo trace to visualize the delay that serially merging series can introduce. In this particular request the initial 16 preloadSeries could have happened concurrently.

[wilfriedroset]

We might have encountered something similar with mimir 2.11.
We can see something similar in our traces where mimir's store-gateway lookup in the indexes from the object storage sequentially while querying the ckunks in parallel. In fact the store-gateway could lookup in parallel within the same index but the index list is processed in parallel. After discussing with @dimitarvdimitrov this could be related to this issue.

See: https://grafana.slack.com/archives/C039863E8P7/p1707316840756999