add_kubernetes_metadata processor: add support for "/var/log/containers/" log path (version 3) #5011
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
The add_kubernetes_metadata processor's LogPathMatcher could extract a Docker container ID - and hence enrich a log document with Kubernetes metadata - only if the log path was '/var/lib/docker/containers/'. With this commit, the LogPathMatcher can extract the container ID also from a '/var/log/containers/' log path (Kubernetes symlinks).
SvenWoltmann
force-pushed
the
add_kubernetes_metadata_generic_log_path_improvement_2
branch
from
76cee4b to
e260550
Compare
|
As discussed, closed in favor of #4981 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following up on this topic:
https://discuss.elastic.co/t/add-kubernetes-metadata-should-work-in-var-log-containers/97945
and on my previous pull requests #4981 and #4995.
This is the third implementation, more details below in "About the Pull Request".
The Issue
The “add_kubernetes_metadata” processor should also work on the "/var/log/containers/" log path for the following reasons:
You may want to exclude log files from certain pods, e.g. the filebeat pod itself with the
exclude_files: ['filebeat-*.log']option. That would work only in/var/log/containers, as only the symlinks there contain the pod name.You may want to read only the log files of docker containers used by active Kubernetes pods, not any other docker containers running on the system now or in the past. That also works only by following the symlinks in
/var/log/containers.The “source” field in the log documents would be much more informative if it contained a value like
/var/log/containers/kube-proxy-4d7nt_kube-system_kube-proxy-1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266.loginstead of
/var/lib/docker/containers/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266/1bddb0001161285462528b7170a53d13dfe4e17b541319485b9020eef5433266-json.logAbout the Pull Request
This solution
logs_pathconfiguration in the processor as in add_kubernetes_metadata processor: add support for "/var/log/containers/" log path #4981;sourcewas too short.The unit tests are exactly the same as in #4995, except that the generic test case was removed as this solution doesn't have a generic fallback anymore.