Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Mixin] MimirGossipMembersMismatch always fires in GEM deployments #5640

Closed
eamonryan opened this issue Aug 1, 2023 · 0 comments · Fixed by #5641
Closed

[Mixin] MimirGossipMembersMismatch always fires in GEM deployments #5640

eamonryan opened this issue Aug 1, 2023 · 0 comments · Fixed by #5641
Assignees
Labels
bug Something isn't working mixin

Comments

@eamonryan
Copy link
Member

Describe the bug

The alert MimirGossipMembersMismatch always fires in GEM deployments because the admin-api is not included in the list of job names, so there is always a mismatch of exactly the number of admin-api pods.

To Reproduce

Steps to reproduce the behavior:

  1. Deploy GEM via the mimir-distributed helm chart
  2. Set up metamonitoring via the Grafana Cloud self-hosted Mimir integration
  3. See the MimirGossipMembersMismatch alert starts firing at the soonest opportunity

Expected behavior

The alert should not fire under normal circumstances since the count of memberlist_client_cluster_members_count should match the number of pods that are up.

Environment

  • Infrastructure: Kubernetes
  • Deployment tool: Helm
  • GEM: 2.9.x

Additional Context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working mixin
Projects
None yet
1 participant