Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix autodiscover for docker in filebeat #10836

Closed
wants to merge 6 commits into from
Closed

Fix autodiscover for docker in filebeat #10836

wants to merge 6 commits into from

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Feb 20, 2019

Results from autodiscover for docker with filebea have docker.container.id fields instead of the ECS fields container.id. This PR is to figure out why this is happening.

With this change, now in ES, filebeat output looks like:
screen shot 2019-02-20 at 8 25 25 am

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner February 20, 2019 05:40
@kaiyan-sheng
Copy link
Contributor Author

kaiyan-sheng commented Feb 20, 2019

Now the output contains both container.id and docker.container.id and same for container.image.name, container.name and container.labels. Is this expected? I need to spend more time to figure this out.

@kaiyan-sheng kaiyan-sheng self-assigned this Feb 20, 2019
@kaiyan-sheng kaiyan-sheng added the Team:Integrations Label for the Integrations team label Feb 20, 2019
@kaiyan-sheng
Copy link
Contributor Author

Interesting "discover":

  1. container.* and docker.container.* are both from docker autodiscover meta field.
  2. If comment out "container": metaNew, in docker autodiscover, then the filebeat metrics will only have docker.container.* fields but not container.*.
		event := bus.Event{
			"provider":  d.uuid,
			"id":        container.ID,
			flag:        true,
			"host":      host,
			"docker":    metaOld,
			"container": metaNew,
			"meta": common.MapStr{
				"docker":    metaOld,
				// "container": metaNew,
			},
		}

With this event meta, enable add_docker_metadata processor in filebeat.yml with labels.dedot: true, docker.container.labels are not dedotted.

  1. If comment out "docker": metaOld, in docker autodiscover, then the filebeat metrics will only have container.* fields but not docker.container.*.
		event := bus.Event{
			"provider":  d.uuid,
			"id":        container.ID,
			flag:        true,
			"host":      host,
			"docker":    metaOld,
			"container": metaNew,
			"meta": common.MapStr{
				// "docker":    metaOld,
				"container": metaNew,
			},
		}

With this event meta, enable add_docker_metadata processor in filebeat.yml with labels.dedot: true, container.labels are successfully dedotted.

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner February 20, 2019 18:10
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. v7.0.0 review labels Feb 20, 2019
@kaiyan-sheng
Copy link
Contributor Author

jenkins, test this please

@kaiyan-sheng
Copy link
Contributor Author

Because this PR is originally started with Nicolas's work. I will close this PR and recreate a new one #10862 based on his PR.

@kaiyan-sheng kaiyan-sheng deleted the docker_metadata_ecs branch February 21, 2019 05:41
@kaiyan-sheng kaiyan-sheng removed needs_backport PR is waiting to be backported to other branches. v7.0.0 labels Feb 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant