[Change Proposal] Introduce an "Agent Common Schema"

[jsoriano]

This issue is a follow up of the discussions in elastic/integrations#4236 and #199

We need a source of truth for the mappings of the fields added by data providers, mainly for Elastic Agent and the processors it includes. These mappings will be used by any feature that gets these fields populated. It could be mappings in package definitions, or Fleet when installing index templates for specific policies.

The proposal would be to create a new repository, called elastic-agent-common-schema, that includes the mappings in paths like <collector>/<feature>/fields.yml. So for example the mappings for the add_host_metadata processor of beats can be defined in beats/add_host_metadata/fields.yml.

Each one of these fields definitions would use the format of ECS flat fields (such as these files), and could be generated from other smaller files if needed, using ECS tooling. Alternatively, the format for fields in the package spec could be used, that is similar.

Its versioning should be aligned with Elastic Agent versioning.

Use cases

• Allow elastic-package to import definitions in packages in build time, similar to the external fields, but for groups of packages.
• Allow Fleet to add mappings for fields of custom processors.

[ebeahan]

Instead of introducing an entirely new repo, what about introducing the idea of extensions into ECS and introduce the mappings into the existing ECS repo as extensions?

ECS already releases aligned with Elastic Agent versioning, and the extension field definitions would exist beside the existing definitions and tooling that manages them.

[P1llus]

Instead of introducing an entirely new repo, what about introducing the idea of extensions into ECS and introduce the mappings into the existing ECS repo as extensions?

ECS already releases aligned with Elastic Agent versioning, and the extension field definitions would exist beside the existing definitions and tooling that manages them.

I think the idea here is to still use the ECS schema, the "common schema" is simply a list of all fields included in all packages.

[jsoriano]

Instead of introducing an entirely new repo, what about introducing the idea of extensions into ECS and introduce the mappings into the existing ECS repo as extensions?

Take into account that this new schema is not an extension to ECS, but something like a collection of packages of mappings, or lists of fields as Marius says. They can contain fields already available in ECS, or other different fields specific to a given processor or collector.

Said that, it could be actually an option to include them in a new directory in ECS repository if the ECS team is fine with it. This would make it easier to reuse tooling and processes, but it may be a bit out of scope of ECS purpouses.

[efd6]

I like the idea of field definition modules (I think this is essentially what Eric is suggesting). Then if we had an agent module, we could write in a fields file something like (subject to bike-shedding):

- module: agent
- module: base
- name: my_local_package_field_group
  type: group
  fields:
    - name: …
  …

which would then expand out the agent and base modules to the fields that are agreed to be needed for those uses.

This would reduce churn on field definition files and reduce the possibility of missed fields and field definition disagreement.