Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address issue #10505 (add_kubernetes_metadata processor matcher not working) #10506

Merged
merged 5 commits into from
Feb 4, 2019
Merged

Address issue #10505 (add_kubernetes_metadata processor matcher not working) #10506

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b35e38
Address issue #10505 - add_kubernetes_metadata processor is broken be…
Feb 2, 2019
dd762d6
Address issue 10505 - update documentation
Feb 2, 2019
58a2df0
Add pull # to CHANGELOG.asciidoc
Feb 2, 2019
f27b50e
Fix unit test case
Feb 4, 2019
88fadcc
Move update to CHANGELOG.next.asciidoc
Feb 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- The `elasticsearch/deprecation` fileset now indexes the `component` field under `elasticsearch` instead of `elasticsearch.server`. {pull}10445[10445]
- Remove field `kafka.log.trace.full` from kafka.log fielset. {pull}10398[10398]
- Change field `kafka.log.class` for kafka.log fileset from text to keyword. {pull}10398[10398]
- Address add_kubernetes_metadata processor issue where old source field is
still used for matcher. {issue}10505[10505] {pull}10506[10506]
alakahakai marked this conversation as resolved.
Show resolved Hide resolved

*Heartbeat*

Expand Down
4 changes: 2 additions & 2 deletions filebeat/processor/add_kubernetes_metadata/matchers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@ const containerIdLen = 64
const podUIDPos = 5

func (f *LogPathMatcher) MetadataIndex(event common.MapStr) string {
if value, ok := event["source"]; ok {
if value, ok := event["log"].(common.MapStr)["file"].(common.MapStr)["path"]; ok {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line probably breaks with the redis input as it does not have the fields in here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alakahakai I think using event.Get("log.file.path") should be safe w.r.t. #11543 (comment).

source := value.(string)
logp.Debug("kubernetes", "Incoming source value: %s", source)
logp.Debug("kubernetes", "Incoming log.file.path value: %s", source)

if !strings.Contains(source, f.LogsPath) {
logp.Debug("kubernetes", "Error extracting container id - source value does not contain matcher's logs_path '%s'.", f.LogsPath)
Expand Down
6 changes: 5 additions & 1 deletion filebeat/processor/add_kubernetes_metadata/matchers_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,11 @@ func executeTestWithResourceType(t *testing.T, cfgLogsPath string, cfgResourceTy
assert.Nil(t, err)

input := common.MapStr{
"source": source,
"log": common.MapStr{
"file": common.MapStr{
"path": source,
},
},
}
output := logMatcher.MetadataIndex(input)
assert.Equal(t, expectedResult, output)
Expand Down
3 changes: 2 additions & 1 deletion libbeat/docs/processors-using.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -755,7 +755,8 @@ construct a lookup key with the value of the field `metricset.host`.
Each Beat can define its own default indexers and matchers which are enabled by
default. For example, FileBeat enables the `container` indexer, which indexes
pod metadata based on all container IDs, and a `logs_path` matcher, which takes
the `source` field, extracts the container ID, and uses it to retrieve metadata.
the `log.file.path` field, extracts the container ID, and uses it to retrieve
metadata.

The configuration below enables the processor when {beatname_lc} is run as a pod in
Kubernetes.
Expand Down