prometheus-agent on alba / peu01 fails to send its data to the MC because it reached mimir's max ingestion rate

[QuantumEnigmaa]

The prometheus-agent was logging these errors :

server returned HTTP status 429 Too Many Requests: the request has been rejected because the tenant exceeded the ingestion rate limit, set to 1e+06 items/s with a maximum allowed burst of 10000000. This limit is applied on the total number of samples, exemplars and metadata received across all distributors (err-mimir-tenant-max-ingestion-rate). To adjust the related per-tenant limits, configure -distributor.ingestion-rate-limit and -distributor.ingestion-burst-size, or contact your service administrator.

As a quick fix I increased the max ingestion rate and ingestion burst limit in the shared configs but it's not a long-lasting solution as we can't increase these numbers endlessly.

We should thus investigate if there is another way to work around this issue.

[QuantumEnigmaa]

After some investigations on alba, I didn't found any way to work around increasing mimir's max ingestion rate and ingestion burst limit.