Partition HTTP requests to loki by stream for improved throughput

[jszwedko]

Broken off from #5973 (comment)

Right now users are limited to only sending one HTTP request at a time to Loki due to the ordering requirements Loki has around streams which result in possible failures when requests are issued in parallel for a given stream. This constrains the throughput vector is able to achieve.

#5973 improves this situation by letting users modify the behavior on failure by, for example, rewriting the timestamps but I think it would be advantageous to modify the loki sink to batch partition HTTP requests by stream to increase throughput by allowing multiple HTTP requests for different streams to be issued simultaneously.

[jszwedko]

Discussion in discord with a user that would benefit from this: https://discord.com/channels/742820443487993987/746070591097798688/799285661449322537

[spencergilbert]

This would be great :)

[wgb1990]

@jszwedko Hello, what can this feature plan to support, the throughput of vector cannot be improved at present

[jszwedko]

Hi @wgb1990. This feature hasn't been scheduled yet.

[jszwedko]

Noting this came up in discord again: https://discord.com/channels/742820443487993987/746070591097798688/821864991270633512

[ctorrisi]

@jszwedko As part of #5973, the request.concurrency configuration was disabled as a workaround to eliminate out-of-order issues. Although this eliminates out-of-order issues in Loki, it also reduces overall throughput. You made a review comment stating:

Instead of hard-coding to 1, maybe we could still let users set it if they really know what they are doing, but output a warning?

I would preference this to be a warning instead of a hard-coded value. Perhaps this should be included as part of the documentation rather than limiting the performance of vector?

In addition, there is progress in Loki to provide some leniency around out-of-order log submissions.

Is there a chance we could get request.concurrency re-enabled with a warning and/or documentation for users who may not understand the implications of increasing this value?

[jszwedko]

@ctorrisi agreed, I opened #6932 to track that work. Thanks for suggesting!