Make sk-tracer able to horizontally scale #80
drmorr0
started this conversation in
Feature Requests and Future Work
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Right now there's just a single sk-tracer instance that sets a watch on the apiserver. For very large/production clusters this could potentially not scale well. We should do some benchmarking here first (#73), but at some point in the future we may need to make sk-tracer able to horizontally scale.
I think in theory this is straightforward: just have a queue with the apiserver watch, and then multiple instances of sk-tracer that pull things off the queue and record them in their local trace instance (probably keyed/sharded by hash, so that one instance doesn't get the "object creation" event and another instance the "object deletion" event). They could either then all merge to some external/persistent datastore asynchronously, or possibly just do a merge "on demand" when a user initiates an export API call.
This is all a bit hypothetical, since I don't think we're there yet, but wanted to record it for posterity.
Related issue: #79
Beta Was this translation helpful? Give feedback.
All reactions