Ensure kubernetes caches don't expire if they are being read #10946
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsoriano
added
module
review
Metricbeat
jsoriano
force-pushed
the
kubernetes-metrics-cache-cleanup
branch
2 times, most recently
from
1902871 to
c36d589
Compare
jsoriano
commented
Feb 26, 2019
jsoriano
force-pushed
the
kubernetes-metrics-cache-cleanup
branch
from
c36d589 to
1f26a33
Compare
sayden
reviewed
Feb 27, 2019
jsoriano
added
needs_backport
|
I have been testing this with master and 6.6.1 and it seems to work. As caches are now, #10658 can still happen if period of container metricset is greater than 120 seconds, fixing this would require to further refactor the global cache. I leave this for future improvements (#10965) as I don't think many people will have these periods. A quick "fix" would be to increase the default timeout. |
|
It only quickly peaked at the cache, but it looks similar to libbeat/common/cache.go (godoc) which expires elements based on their last access time, has a goroutine to periodically do cleanup, and allows injecting a clock for testing (but injecting the clock isn't exported so you'd need to expose this). |
|
@andrewkroh oh, this actually looks like what we'd need here, I will give a try to this implementation, thanks! |
houndci-bot
reviewed
Feb 28, 2019
jsoriano
force-pushed
the
kubernetes-metrics-cache-cleanup
branch
from
8481def to
304bec5
Compare
jsoriano
commented
Feb 28, 2019
|
|
||
| // Check it expired | ||
| assert.Equal(t, 0.0, test.Get("foo")) | ||
| } |
There was a problem hiding this comment.
Test removed because libbeat cache implementation is used now and it has its own tests for this.
sayden
approved these changes
Mar 4, 2019
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Mar 4, 2019
…#10946) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit 106df3d)
jsoriano
removed
the
needs_backport
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Mar 4, 2019
…#10946) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Mar 4, 2019
…#10946) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
to jsoriano/beats
that referenced
this pull request
Mar 4, 2019
…#10946) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
that referenced
this pull request
Mar 5, 2019
…#11057) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix #10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
that referenced
this pull request
Mar 5, 2019
…#11058) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix #10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
that referenced
this pull request
Mar 5, 2019
…#11059) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix #10658 (cherry picked from commit 106df3d)
jsoriano
added a commit
that referenced
this pull request
Mar 14, 2019
…#11060) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix #10658 (cherry picked from commit 106df3d)
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…#10946) (elastic#11059) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit db4b4c2)
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…#10946) (elastic#11060) Some metrics in metricbeat kubernetes module are cached during a time, if they are not updated they are removed. But it is usual to have pods or containers that are not updated during more time that the expiration cache. Current implementation was not renovating expiration times for cache entries so all were eventually removed if updates for them are not received. Replace it with the cache implementation available in libbeat, but keeping the existing interface. Also, use slashes instead of dashes to generate unique container uids. Dashes can be used by kubernetes names, what could lead to ambiguous keys for the caches. Fix elastic#10658 (cherry picked from commit db4b4c2)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some metrics in metricbeat kubernetes module are cached during a time,
if they are not updated they are removed. But it is usual to have pods or
containers that are not updated during more time that the expiration cache.
Current implementation was not renovating expiration times for cache
entries so all were eventually removed if updates for them are not received.
Replace it with the cache implementation available in libbeat, but keeping
the existing interface.
Also, use slashes instead of dashes to generate unique container uids.
Dashes can be used by kubernetes names, what could lead to ambiguous
keys for the caches.
Fix #10658