-
Notifications
You must be signed in to change notification settings - Fork 543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add sanity check at startup to ensure the configured filesystem directories don't overlap for different components #2828
Conversation
aa10858
to
7f3d94e
Compare
I need to improve the checks on the filesystem bucket, because ruler and alertmanager use a different prefix so they don't conflict (and blocks storage could have an optional prefix too). |
7f3d94e
to
484f90d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM.
I've got just one small nitpick: now we have two completely different paths to validate the bucket config:
- When config is just parsed, we call
.Validate()
and it checks bucket configs: that is basically ensuring that if two buckets are the same, they should have different prefixes. - Then your new logic is triggered as part of the SanityCheck module.
Can we move the former call to validateBucketConfigs
somewhere near the call to checkFilesystemPathsOvelapping
? Maybe checkFilesystemPathsOvelapping
should be called from validateBucketConfigs
?
Otherwise we have two completely paths for validating two aspects of the same config.
…tories don't overlap for different components Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
720ef95
to
7fe2550
Compare
Signed-off-by: Marco Pracucci <marco@pracucci.com>
This is a very good feedback. This new check doesn't need to be a sanity check (it doesn't run anything against the real setup) but it's just a static config check. I moved it to be a function of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks.
What this PR does
While answering the question #2698 (reply in thread) I've realized that if you set overlapping local directories for different purposes you could end up with weird behaviour (e.g. alertmanager deleting the alerts store content). These errors are quite difficult to detect, so in this PR I'm proposing to introduce a sanity check to ensure the configured filesystem directories don't overlap for different components (checking only the running components).
Thought: the logic encoded in this new check is non trivial, but that's one more reason why we should encode it instead of letting our users figuring out through a try-and-error process.
Which issue(s) this PR fixes or relates to
N/A
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]