Configurable shard-level concurrent reconciliation #1124
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 6, 2023
|
Here's an example operator config change, CRD definition, and logs showing the flow of reconciliation using this new mode: Operator file config change: CRD example (2x4x2) Reconcile wall time via get pods view (simple): Full event log via kubectl describe chi |
|
and: fyi @sunsingerus . |
|
Thank you, @zcross , nice work! We are busy with 0.21.0 release that should be complete in a couple of weeks, it will go to the next one. |
|
Thanks for that update! I'll avoid adding any chatter to the PRs then, unless I discover any new open questions or run into any problems while doing some local manual testing / e2e testing. |
|
Another example: one cluster (for brevity), but this time 8 shards x 2 replicas. Still using 100% concurrency percent and 4 goroutines: |
|
Update: I'm working on a change to support overriding the k8s rate limit QPS and burst parameters, basically necessary to get around the above. Currently pondering how to do it if not through CHOP YAML config (due to the ... edit: see newest commit for simple passage of this via env vars, which avoided refactoring of the initialization of |
|
However, some celebration worthy early results: I was just able to test this out in a real environment instead of minikube and reconciled a 32-replica cluster in about 20 minutes instead of the several hours that we typically observe! Needless to say, my team is pretty excited about the potential value this will have for us (especially in incident response) |
|
Just resolved (trivial, import lines) merge conflicts after doing the same in #1119 |
Signed-off-by: Zach Cross <zcross@chronosphere.io>
|
Just rebased off latest |
|
Apologies for not offering up a bunch of new e2e test coverage. I'd like to, but:
For example, I'd like to at least have one test case do something like "bring up a n>2 shards cluster results in a healthy cluster" with the new By the way: since the other 2 PRs landed – what are the chances this makes it into Thanks! |
|
Thanks @sunsingerus ! |
This was referenced Jul 5, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background: #1109
Related PR(s):
This PR adds a configurable level of concurrency to the operator for the purpose of CHI reconciliation, with concurrency being applied at the shards level.
Key design decisions:
Important items to consider before making a Pull Request
Please check items PR complies to:
next-releasebranch, not intomasterbranch1. More info