Mimir testing

[Rotfuks]

Motivation

n order to raise our confidence in the stability of our observability platform and be sure that our ongoing work and releases won't negatively impact our observability platform operations we need to create extensive tests giving us early feedback loops. As mimir is one of our core components, we should make sure it's thoroughly tested. 

Todo

• Investigate some good initial test cases to give us feedback on the stability of a release on mimir on CI. This can be:

  • Validate Helm chart templating
  • Deploy chart on Kind (single instance)

• Implement those CI test cases

• Investigate some good initial test cases to give us feedback on the stability of a release before releasing. This can be:

  • Deploy chart on AWS, Azure, CAPI (later)
  • Integration test; Verify all component work together
  • Using canary to generate traffic

• Implement those test cases

Make sure to stay with a set of minimal but valuable test cases, nothing to detailed and fancy.

Outcome

• We have early, automatic feedback about the impact of a release of Mimir before releasing.

[QuentinBisson]

I think it would be nice to run mimir continous testing (similar to loki canary) for e2e tests

[QuantumEnigmaa]

Yeah that's a nice idea :)

[QuentinBisson]

I am running mimir continuous_test on grizzly with the following config:

mimir:
  continuous_test:
    enabled: true
    auth:
      tenant: anonymous

and this renders the following metrics:

and we could just use those alerts https://github.com/grafana/mimir/blob/f52911d917c8c52e0da6a59348a64dd7f7622072/operations/mimir-mixin-compiled/alerts.yaml#L1097

The only downside is that we need to wait for the next minor helm chart release or use a weekly version because this grafana/mimir#8654 is not yet released

[QuantumEnigmaa]

So what's the best plan of action IMO is to wait for the continuous testing to be a default config for our mimir before doing anything else.
In the meantime, I'll create a dashboard using the metrics from the rules' mixins and if it's good enough, I'll think about pushing it upstream as a mixins dashboard.

[QuentinBisson]

I think so yes, maybe we can have a pr ready with the alerts? The mixins contains some that could be useful

[QuentinBisson]

@QuantumEnigmaa we decided in retro to use the chart version rc0 for now but keep the old image of mimir 2.13

[QuantumEnigmaa]

All good with me 👍

[QuentinBisson]

We can start this again once we're done with multi-tenancy :)

[QuentinBisson]

Taken over the dashboard PR giantswarm/dashboards#624 to close the epic

[QuentinBisson]

Blocked waiting for reviews

[QuentinBisson]

Continuous test is enabled on all MCs

Added chart testing:

• https://github.com/giantswarm/mimir-app/pull/133

Dashboard PR has been merged:

• add mimir continous test dashboard dashboards#624

Alert based on failures under review:

• add MimirContinuousTestFailingOnWrites and MimirContinuousTestFailing… prometheus-rules#1355

Test procedure tbd:

• https://github.com/giantswarm/mimir-app/pull/138

[QuentinBisson]

All is done. Thanks @hervenicol for the reviews