Avoid infinite loop in Kubernetes watcher #6353
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In case of unmarshalling errors the Watcher was ending up in a infinite loop, this code ignores those errors and only restarts the watcher in case of EOF.
exekias
force-pushed
the
6.2-fix-k8s-watcher-loop
branch
from
3b44e64 to
32debe1
Compare
vjsamuel
suggested changes
Feb 12, 2018
libbeat/common/kubernetes/watcher.go
Outdated
| // In case of EOF, stop watching and restart the process | ||
| if err == io.EOF || err == io.ErrUnexpectedEOF { | ||
| watcher.Close() | ||
| time.Sleep(time.Second) |
There was a problem hiding this comment.
can you please make this binary back off?
|
failing tests are unrelated to this PR |
exekias
added
review
and removed
in progress
vjsamuel
approved these changes
Feb 13, 2018
|
We updated to |
|
Uhm, could you paste the log output? Some things should have, at least, changed |
|
Sure: Thanks for looking into it! Edit: I see this addition in this pull request: However, there doesn't appear to be any wait between failures. |
|
This is weird, it looks like the change is in, but watcher connection backoff doesn't seem in place. Do you know by a chance what's the event originating this? I haven't reproduced it myself |
|
So far I haven't been able to find a specific suspicious event right before the loop occurs, do you have any suggestions for what I could look for? |
|
Hello, We are experiencing this very same issue on Filebeat 6.2.2 when the 'add_kubernetes_metadata' processor is added on our Openshift Kubernetes nodes. Grtz |
|
@jimsheldon @willemdh Any chance both of you could enable debug logging and share the log output here (or link to a gist)? |
|
I once met t this before, that's what I thought client-go would make help here with better compatibility.
…________________________________
From: Nicolas Ruflin <notifications@github.com>
Sent: Tuesday, February 27, 2018 8:49:22 AM
To: elastic/beats
Cc: Subscribed
Subject: Re: [elastic/beats] Avoid infinite loop in Kubernetes watcher (#6353)
@jimsheldon<https://github.com/jimsheldon> @willemdh<https://github.com/willemdh> Any chance both of you could enable debug logging and share the log output here (or link to a gist)?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#6353 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AK0p_rs_j-Bz7vwSGZ70Ff53Bgp29YTDks5tY1ESgaJpZM4SCCFd>.
|
|
I once make a issue for this ericchiang/k8s#68 before while using beat, but didn't get response for this, while when I test #6117 with client-go, it didn't happen again. |
|
@ruflin Well it seems after restarting the FIlebeat agents on our 10 Openshift nodes, the problem did not reappear.. I'm monitoring closely to find a pattern in the hope to reproduce. |
|
@willemdh Keep us posted if it happens again. I think a point that I only realised now is that you are running OpenShift. There were some subtle differences in the past, so this is probably also a good data point to add to the equation when we see this again. |
|
We are facing the same Issues with Filebeat 6.2.2 using the add_kubernetes_metadata processor. |
|
@exekias no. So maybe the updated client library make help here? |
|
That was my expectation, would anyone suffering from this issue be up to give a try to |
|
@exekias I would give it a try |
|
I'm experiencing the same issue in custom build image from master(5fa9be4). Also, I'm not sure if it caused crash, but the last logs of the previous instance of restarted pods end with this error. |
|
I might not have entire context about the issue. I am guessing that this is because of an incompatible kube client as the issue seems to be coming from proto unmarshalling. @wflyer what is the version of Kubernetes that you're running on? |
|
@vjsamuel I'm using kubernetes version v1.9.2. |
|
For the crash part that I mentioned (#6353 (comment)) it looks like that this issue significantly increases the memory usage of the pod and makes the pod to be killed due to OOM. |
|
Thank you for the patience everyone, I've built a 6.3 snapshot: |
|
@exekias |
|
Seeing the same issue with es stack 6.2 on k8s 1.8.5 multiple lines of: and causing the pods to run OOM and crash |
|
Seeing the same issue with filebeat 6.2.2 on k8s 1.7.2. It also appears that when the issue occurs the k8s api server is flooded with WATCH requests which causes the CPU on the k8s api servers to increase to almost 100% when the load is normally < 10%. Would it be better to create a new issue rather than comment on a merged pull request? |
|
Maybe I can build a image with latest beat master but using kubernetes/client-go instead of the library currently in used? I am pretty sure this is compatibility issue in k8s library. |
|
@vjsamuel I've not been able to reproduce it. Generally after restarting Filebeat everything works as expected. It takes 3-5 days before this issue appears. |
|
never mind @willemdh. I was able to reproduce. I have a PR with the fix. |
|
Just an update on this, a fix for #6503 has been merged and should go out in both |
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
In case of unmarshalling errors the Watcher was ending up in a infinite loop, this code ignores those errors and only restarts the watcher in case of EOF.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In case of unmarshalling errors the Watcher was ending up in a infinite loop,
this code ignores those errors and only restarts the watcher in case of
EOF.
This was reproduced by @cdahlqvist:
Opening directly to 6.2 as master is not affected, after its refactoring and client update