Migrate add_docker_metadata to ECS #9412

ruflin · 2018-12-06T08:44:56Z

Migrate the docker fields to ECS container fields.

docker.container.id -> container.id
docker.container.image -> container.image.name
docker.container.name -> container.name
docker.container.labels -> container.labels

ruflin · 2018-12-06T08:45:17Z

dev-tools/ecs-migration.yml

+  alias6: true
+  alias: true
+
+- from: docker.container.labels  # TODO: How to map these?


@graphaelli Have you found a good way to migrate such fields?

I don't think we can. I'm assuming this is user-populated, including key names? If that's the case, then the list of fields to migrate is unknowable.

Perhaps if some fields are often present by default, we can create aliases for those, though.

I have not found a good way. Can the migration assistant create those aliases dynamically? That is, iterate over the source labels and create aliases on the old indices for each one? Another option is to inform users to include both fields with an or clause in searches and aggregations.

APM has the same issue where context.tags are moving to labels. It may be possible there to introduce an APM UI-only concept of ~~"tags"~~ "labels" and make the context.tags.$key=$value or labels.$key=value query on behalf of the user - we have not discussed this possibility but I'd be curious what you think.

@graphaelli Indeed that is something that the migration assistant could potentially do. I think in this case it's more important for apm-server as you probably need it for the 6.x compatiblity layer and in our case it's nice to have as we can also break it in 7 and inform the user about this one. If we can use then the same feature to, that is great.

Yeah I think the migration assistant will be our best bet.

@graphaelli Can you take this up with the Kibana team working on the migration assistant to see what they think?

They've added it to their use cases for the migration assistant, there will be hooks for us to handle it there.

Great news. Thanks for following up.

ruflin · 2018-12-06T08:45:48Z

libbeat/processors/add_docker_metadata/add_docker_metadata.go

@@ -88,7 +88,7 @@ func buildDockerMetadataProcessor(cfg *common.Config, watcherConstructor docker.
 			"field":     "source",
 			"separator": string(os.PathSeparator),
 			"index":     config.SourceIndex,
-			"target":    "docker.container.id",
+			"target":    "container.id",


Was not sure why here the constant was not used. @exekias ok to change this one or could this have other side effects?

uhm, probably they were introduced at different points in time, I don't see a reason for it to not use the constant

ok, replaced it by the constant then.

ruflin · 2018-12-06T08:46:18Z

libbeat/processors/add_docker_metadata/add_docker_metadata.go

@@ -166,10 +166,6 @@ func (d *addDockerMetadata) Run(event *beat.Event) (*beat.Event, error) {
 	container := d.watcher.Container(cid)
 	if container != nil {
 		meta := common.MapStr{}
-		metaIface, ok := event.Fields["docker"]


Remove this one as I felt it's not needed anymore when using Put with the DeepUpdate. Please double check if that is correct.

while that's true, I think that will create more unneeded allocations? this is the kind of code that runs lots of times

Not sure I can fully follow here. You are mostly worried about the DeepUpdate part?

Going to merge for now. @exekias Let's revisit this next year to make sure we have the correct optimisations in place.

libbeat/processors/add_docker_metadata/_meta/fields.yml

elasticmachine · 2018-12-10T09:16:20Z

Pinging @elastic/infrastructure

ruflin · 2018-12-13T08:46:00Z

I suggest we move forward with this PR even though alias for docker.container.labels do not exist as it's not straight forward. I added a note about it to our ECS issue so we can track it there.

webmat

One small issue in ecs-migration.yml, and two comments that will help inform next steps.

dev-tools/ecs-migration.yml

webmat · 2018-12-13T19:40:03Z

dev-tools/ecs-migration.yml

+  alias6: true
+  alias: true
+
+- from: docker.container.labels  # TODO: How to map these?


I don't think we can. I'm assuming this is user-populated, including key names? If that's the case, then the list of fields to migrate is unknowable.

Perhaps if some fields are often present by default, we can create aliases for those, though.

libbeat/processors/add_docker_metadata/_meta/fields.yml

ruflin · 2018-12-17T08:56:14Z

@webmat Could you take a look again to get this in?

webmat

LGTM

ruflin · 2018-12-18T08:07:21Z

auditbeat/docs/modules_list.asciidoc

@@ -8,5 +8,6 @@ This file is generated! See scripts/docs_collector.py

 --

+include::./modules/.DS_Store.asciidoc[]


@adriansr @cwurm This probably needs some fixing in the collection script. Will fix it in this PR manually.

Migrate the docker fields to ECS container fields. * docker.container.id -> container.id * docker.container.image -> container.image.name * docker.container.name -> container.name * docker.container.labels -> container.labels make image fix update generator script

Fields injected by docker autodiscover provider were being placed in alias fields introduced for ECS, change them to the new location and add selectors accordingly. This PR includes #10862 and #10758 As a summary: * Autodiscover selectors using ECS structure are added to autodiscover events, old selectors are kept for backwards compatibility * Autodiscover generated metadata follows ECS * Dedotting of labels is added, enabled by default, will be backported for 6.7, but disabled `docker.containers.labels` is not migrated, as it wasn't for `add_docker_metadata` (see #9412) Fixes #10757 Co-Authored-By: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

Fields injected by docker autodiscover provider were being placed in alias fields introduced for ECS, change them to the new location and add selectors accordingly. This PR includes elastic#10862 and elastic#10758 As a summary: * Autodiscover selectors using ECS structure are added to autodiscover events, old selectors are kept for backwards compatibility * Autodiscover generated metadata follows ECS * Dedotting of labels is added, enabled by default, will be backported for 6.7, but disabled `docker.containers.labels` is not migrated, as it wasn't for `add_docker_metadata` (see elastic#9412) Fixes elastic#10757 Co-Authored-By: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-Authored-By: Nicolas Ruflin <spam@ruflin.com> (cherry picked from commit 1bf8087)

Fields injected by docker autodiscover provider were being placed in alias fields introduced for ECS, change them to the new location and add selectors accordingly. This PR includes #10862 and #10758 As a summary: * Autodiscover selectors using ECS structure are added to autodiscover events, old selectors are kept for backwards compatibility * Autodiscover generated metadata follows ECS * Dedotting of labels is added, enabled by default, will be backported for 6.7, but disabled `docker.containers.labels` is not migrated, as it wasn't for `add_docker_metadata` (see #9412) Fixes #10757 (cherry picked from commit 1bf8087) Co-Authored-By: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-Authored-By: Nicolas Ruflin <spam@ruflin.com>

Migrate the docker fields to ECS container fields. * docker.container.id -> container.id * docker.container.image -> container.image.name * docker.container.name -> container.name * docker.container.labels -> container.labels

ruflin added in progress Pull request is currently in progress. libbeat ecs labels Dec 6, 2018

ruflin commented Dec 6, 2018

View reviewed changes

ruflin mentioned this pull request Dec 6, 2018

Transition Beats to ECS #8655

Closed

ruflin added the Team:Integrations Label for the Integrations team label Dec 10, 2018

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from d02919b to 309df10 Compare December 10, 2018 09:21

alvarolobato assigned ruflin Dec 10, 2018

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch 2 times, most recently from 0e4e3d3 to d93cca7 Compare December 12, 2018 13:43

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from b671461 to 275de21 Compare December 13, 2018 08:50

ruflin added review and removed in progress Pull request is currently in progress. labels Dec 13, 2018

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch 2 times, most recently from 0dbc2f5 to 90746ae Compare December 13, 2018 16:00

webmat reviewed Dec 13, 2018

View reviewed changes

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch 2 times, most recently from ce274f3 to 763e16b Compare December 14, 2018 14:34

ruflin requested a review from a team as a code owner December 14, 2018 14:34

graphaelli mentioned this pull request Dec 15, 2018

model: ECS for 6.x elastic/apm-server#1609

Merged

5 tasks

webmat approved these changes Dec 17, 2018

View reviewed changes

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from 763e16b to c66ed45 Compare December 18, 2018 08:06

ruflin commented Dec 18, 2018

View reviewed changes

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch 2 times, most recently from d32ddf1 to cb8f556 Compare December 18, 2018 12:46

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from cb8f556 to d2e260d Compare December 27, 2018 08:11

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from d2e260d to 3578b0f Compare December 27, 2018 09:16

ruflin force-pushed the migrate-add-docker-metadata-to-ecs branch from 3578b0f to 0a3caab Compare December 27, 2018 12:02

ruflin merged commit f43683f into elastic:master Dec 27, 2018

ruflin deleted the migrate-add-docker-metadata-to-ecs branch December 27, 2018 15:12

This was referenced Feb 19, 2019

docker autodiscover not ECS ready #10757

Closed

Migrate docker autodiscovery to ECS #10898

Merged

Metricbeat docker module is not ECS ready #10900

Closed

This was referenced Feb 25, 2019

Cherry-pick #10898 to 7.0: Migrate docker autodiscovery to ECS #10929

Merged

Migrate metricbeat docker module to ECS #10927

Merged

tetianakravchenko mentioned this pull request Mar 28, 2023

Migrate docker provider to ECS fields: container.image -> container.image.name elastic/elastic-agent#2411

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate add_docker_metadata to ECS #9412

Migrate add_docker_metadata to ECS #9412

ruflin commented Dec 6, 2018

ruflin Dec 6, 2018

webmat Dec 13, 2018

graphaelli Dec 13, 2018 •

edited

Loading

ruflin Dec 14, 2018

webmat Dec 14, 2018

ruflin Dec 17, 2018

graphaelli Dec 18, 2018

ruflin Dec 19, 2018

ruflin Dec 6, 2018

exekias Dec 7, 2018

ruflin Dec 10, 2018

ruflin Dec 6, 2018

exekias Dec 7, 2018

ruflin Dec 10, 2018

ruflin Dec 27, 2018

elasticmachine commented Dec 10, 2018

ruflin commented Dec 13, 2018

webmat left a comment

webmat Dec 13, 2018

ruflin commented Dec 17, 2018

webmat left a comment

ruflin Dec 18, 2018

		@@ -8,5 +8,6 @@ This file is generated! See scripts/docs_collector.py

		--

		include::./modules/.DS_Store.asciidoc[]

Migrate add_docker_metadata to ECS #9412

Migrate add_docker_metadata to ECS #9412

Conversation

ruflin commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graphaelli Dec 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Dec 10, 2018

ruflin commented Dec 13, 2018

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Dec 17, 2018

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graphaelli Dec 13, 2018 •

edited

Loading