Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alert rules don't use correct selectors #46

Open
simu opened this issue Aug 23, 2022 · 0 comments
Open

Alert rules don't use correct selectors #46

simu opened this issue Aug 23, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@simu
Copy link
Member

simu commented Aug 23, 2022

We currently use the Thanos mixing Jsonnet library to define alerts per instance of the component. However, because we don't customize the alert selectors in component/alerts.libsonnet, each instance's alerts will pick up metrics from all instances of the deployed Thanos component (e.g. receive).

This makes it currently impossible to have different sets of alerts for different instances of the same Thanos component in a single cluster. Since we advertise multi-instance support, we should make sure that alert rules are configured to only match their corresponding component instances (e.g. by adding 'namespace=%s'%instanceNs to the alert selector(s).

Steps to Reproduce the Problem

  1. Create two instances of the component which deploy the same Thanos component (e.g. receive)
  2. Disable alert rules for one of the instances
  3. Observe that remaining alert rules will pick up both instances

Actual Behavior

Alert rules pick up all Thanos component instances on the cluster instead of only the ones associated with the Commodore component instance.

Expected Behavior

Alert rules are constrained to only cover Thanos instances associated with the Component instance

@simu simu added the bug Something isn't working label Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant