-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimisation for Neighbourhood discovery on scale #55
Comments
Hi @myaser The first point (make the interval configurable) is something that is definitely worth to implement and is not a big deal. Your second point is very interesting and I need to have a deeper look at the SWIM paper. From what I've seen now, it should be a good mechanism to detect communication loss to other instances of the kubenurse while reducing the network traffic. Our initial reason why we developed the kubenurse the way it is now was to understand where network issues happen. Therefore we wanted to do a real HTTP request and record the latencies between each instance to narrow down issues on routing, virtualisation, switches, firewalls, kubelet etc. This is easy to process when the check interval is static and low. With the SWIM protocol I'm not sure if we can easily detect which network path is causing troubles because of the non uniform check interval and distribution. But it would be a good solution to find if something is fishy in the network at larger scales. Therefore I see that the first point (make the interval configurable) can easily be implemented and is a quick win. For larger clusters we would probably have to implement a SWIM (or similar) based check additionally. This is my impression after scratching the paper. But probably there are also other solutions to this problem. I definitely need to dig deeper into it. It looks interesting. What about you @zbindenren ? |
btw, I started with #45 before my holidays and need to finish it when I got time. This could probably be impacted but is no big deal |
Maybe we can use SWIM to track membership and not rely on k8s API server discovery and still call the However, this leaves us with the problem of the number of TCP/IP handshakes that remain unsolved |
I am colleague of @myaser and suggested internally the use of SWIM and hashicorp memberlist. Maybe we need not to use a full Mesh of TCP/IP handshakes or http requests, but just a subset that you can define via option. For example test 10% of your neighbors and find via memberlist who is your neighbor peer, so you can have the full ring tested without having "wholes". I agree testing http would be great. |
Maybe using a cached client could reduce the load on the API server? |
If you use informers you have a cache and a maintained WATCH, which in case you get a new node with kubenurse it will send to all kubernurse. Then to get unstuck in most cases you should also use refresh interval and then the client automatically does a LIST (get all endpoints) and this from every node all time.Duration. |
Yes that is the plan. |
@clementnuss thanks for doing this. Here some of my thoughts: When using a cached client we could get rid of this nodeCache struct and the load on the api server for getting all pods here would hopefully decrease too. I propose to switch to controller-runtime's client, then it would be pretty easy to use their cached client like in this simplified example: // TODO: handle errors
config, _ := rest.InClusterConfig()
if err != nil {
return nil, fmt.Errorf("kubernetes config: %w", err)
}
c, _ := cache.New(config, cache.Options{
ByObject: map[client.Object]cache.ByObject{
&corev1.Pod{}: {},
},
})
_ = c.Start(ctx) You can even use labels for the cache. Here you can find the documentation: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/cache |
permits to get rid of kubediscovery package and to simplify the codebase linked to: #55 Signed-off-by: Clément Nussbaumer <clement.nussbaumer@postfinance.ch>
you can test caching with release |
instead of querying all (valid) neighbouring pods, we filter and only query KUBENURSE_NEIGHBOUR_LIMIT other neighbours. To make sure that all nodes get the right number of queries, we hash each node name, put that in a sorted list (l), and each pod will query the e.g. 10 next nodes in the list, starting from the position given by the node name' hash. i.e. for node named n, we query the 10 next nodes in the list, starting from position l[hash(n)] + 1 cf #55 Signed-off-by: Clément Nussbaumer <clement.nussbaumer@postfinance.ch>
https://github.com/postfinance/kubenurse/blob/master/CHANGELOG.md#1110---2024-03-15 v1.11.0 is out! most notable change is using hashing to distribute the neighbouring nodes checks. for details see the PR #120 and the commit description. thanks to that, each pod only queries 10 other neighbours. please try it out and let me know how this goes. this basically should get us from |
Hello
Neighborhood check is expensive when running on the scale. For instance:
My proposal would be:
While the first point is straightforward and does not change the behavior much, the latter requires discussion
The text was updated successfully, but these errors were encountered: