-
Notifications
You must be signed in to change notification settings - Fork 458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix target allocator readiness check #2883
Fix target allocator readiness check #2883
Conversation
2b2a8f2
to
45bb65d
Compare
45bb65d
to
661d079
Compare
setupLog.Error(err, "Can't load initial Prometheus configuration from Prometheus CRs") | ||
os.Exit(1) | ||
} | ||
loadErr = targetDiscoverer.ApplyConfig(allocatorWatcher.EventSourcePrometheusCR, promConfig.ScrapeConfigs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is done below in the rungroup which should be run once when it's first loaded. Do we need to also do it out here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i.e. it's fine to do the load config here, but couldn't we just set it as cfg.PromConfig and let the rungroup handle the call?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't in general, because cfg.PromConfig
might not be empty. I could move it into the rungroup which established the Prometheus CR Watcher, if you think that'd be cleaner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The calls look similar, but they actually refer to different event sources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh i see that now, yes. I wonder if we should just send a message to the events channel to force an initial refresh rather than making new calls here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@swiatekm-sumo and I discussed a different approach where we just send the event for the channel which for now is going to make the config loading fail silently to a user which seems not ideal.
* Load initial Prometheus CR config at startup * Fix target allocator readiness check
* Load initial Prometheus CR config at startup * Fix target allocator readiness check
* Load initial Prometheus CR config at startup * Fix target allocator readiness check
Description:
The target allocator readiness check is broken in two ways:
This PR fixes both of these issues by:
Link to tracking Issue(s): #2903
Testing:
I changed an integration test with only Prometheus CR enabled to run without any actual CRs created initially.