Script to generate breaking field changes list

[ruflin]

This script generates a list with breaking field changes based on ecs-migration.yml. For each Beat it can generate the list containing libbeat fields + the fields from the Beat itself. For now it's included in the Breaking changes for 7.0 and one table exists for each Beat. Few questions:

• Should we have this in libbeat or as part of each Beat breaking changes?
• How will this conflict with the breaking changes listed in auditbeat?
• Should we add more details on how the fields changed like in some cases there is a scaling factor or the content changed. Most of this info is already in ecs-migration.yml
• Who do we do about the changes where we changed the type. It's not yet in the ecs-migration.yml files.

An example output looks as following:

In version 7.0 the following fields were renamed.

.Renamed Fields in 7.0
[frame="topbot",options="header"]
|======================
|Old Field|New Field
|`apache2.access.body_sent.bytes`            |`http.response.body.bytes`
|`beat.hostname`            |`agent.hostname`
|`beat.name`            |`host.name`
|`beat.timezone`            |`event.timezone`
|`beat.version`            |`agent.version`
|`docker.container.id`            |`container.id`
|`docker.container.image`            |`container.image.name`
|`docker.container.labels`            |`container.labels`
|`docker.container.name`            |`container.name`
|`elasticsearch.slowlog.took_millis`            |`event.duration`
|`haproxy.http.request.time_active_ms`            |`event.duration`
|`haproxy.total_waiting_time_ms`            |`event.duration`
|`iis.access.body_received.bytes`            |`http.request.body.bytes`
|`iis.access.body_sent.bytes`            |`http.response.body.bytes`
|`iis.access.request_time_ms`            |`event.duration`
|`meta.cloud.availability_zone`            |`cloud.availability_zone`
|`meta.cloud.instance_id`            |`cloud.instance.id`
|`meta.cloud.instance_name`            |`cloud.instance.name`
|`meta.cloud.machine_type`            |`cloud.machine.type`
|`meta.cloud.project_id`            |`cloud.project.id`
|`meta.cloud.provider`            |`cloud.provider`
|`meta.cloud.region`            |`cloud.region`
|`mysql.slowlog.query_time.sec`            |`event.duration`
|`nginx.access.body_sent.bytes`            |`http.response.body.bytes`
|`postgresql.log.duration`            |`event.duration`
|`process.exe`            |`process.executable`
|`traefik.access.body_sent.bytes`            |`http.response.body.bytes`
|`traefik.access.duration`            |`event.duration`
|======================

[ruflin]

@dedemorton Where should we place these fields? I assume either inside breaking changes or linked from there?

@andrewkroh I saw in your breaking changes you also listed the type changes. I wonder if we should now add the type changes we did also to ecs-migration.yml file to also generate this or if it would be overkill.

[dedemorton]

Where should we place these fields? I assume either inside breaking changes or linked from there?

@ruflin I would definitely put the list under breaking changes. Right now, we have all the BC content for each release in a single html topic. If we change this (break the topic down into multiple topics in the published docs), we will need to modify the book structure to accommodate another level of nesting in the TOC (this is a more complicated change than you'd expect it to be).

For now, it might be easier to generate a file and include it where relevant. If necessary, we can break the existing source file into multiple files to make it easier to automate adding the content.

If you generate a file with the following heading and add an include to the existing BC file in the right place, you'll get a heading at the same level as the other headings, and all the BC content will be in a single topic.

[[breaking-changes-fields]]
==== Field name changes

[ruflin]

I updated the PR to generate a table for each Beat. @andrewkroh At the moment this also contains the fields which you already have put into the Breaking Change doc for auditbeat.

My suggestion for now is to move forward with this and get it into master. We should adjust it as soon as we start tackling the breaking changes docs because there is more needed then just these fields.

[adriansr]

LGTM

[ruflin]

I rebased again on master and run the script again to get last minute changes in.

[ruflin]

Merged for now to at least have a list of the breaking fields in. The command to rerun the script is:

python renamed_fields.py > ../libbeat/docs/field-name-changes.asciidoc