[Fleet] Support dynamic_template mappings from object field

[nchaulet]

Summary

Related to #129344

Fleet generate mappings from fields yaml provided by integrations, we were not handling field with object and object_type correctly as these fields should not result in properties mappings but in dynamic templates.

That PR fix that, if a field is of type object and contains an object_type we generate a dynamic template, if the path do not contains a * we add .* at the end of the path.

For example this fields:

- name: cockroachdb.status.*.histogram
  type: object
  object_type: histogram
  object_type_mapping_type: '*'
  description: >-
    Prometheus histogram metric

Will result in the following mappings

"dynamic_templates": [{
      "cockroachdb.status.*.histogram": {
        "mapping": {
          "type": "histogram"
        },
        "match_mapping_type": "*",
        "path_match": "cockroachdb.status.*.histogram"
      }
    }
  ]

How this was implemented in beats https://github.com/elastic/beats/blob/ea207346d651448b8917b0791b2b117b9f9b9212/libbeat/template/processor.go#L444

In a second PR, we will introduce a way to generate dynamic template just by adding a * in the property name.

@jsoriano Does it make sense to you?

I tested this by installing all the integrations (to verify all the integrations still install) and looked a few mappings to see if they seemed correct to me.

Should we target 8.4 with that change?

[nchaulet]

I created that file to start moving mapping generation to his own module.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jsoriano]

That PR fix that, if a field is of type object and contains an object_type we generate a dynamic template, if the path do not contains a * we add .* at the end of the path.

How this was implemented in beats https://github.com/elastic/beats/blob/ea207346d651448b8917b0791b2b117b9f9b9212/libbeat/template/processor.go#L444

In a second PR, we will introduce a way to generate dynamic template just by adding a * in the property name.

@jsoriano Does it make sense to you?

Makes sense, yes, keep this separated as this is something new. Thanks!

[jsoriano]

This static field shouldn't be here, right? This creates a mapping for a field called literally cockroachdb.status.labels.*.

[nchaulet]

Yes you are correct, I just pushed a fix for that we do not create mappings is there only dynamic template for that path.

[kpollich]

Not an area of the Fleet codebase I'm too familiar with, but I think I understand everything happening here for the most part. 🚀

[nchaulet]

@kpollich Do you think we should backport this one to 8.4?

[kpollich]

@nchaulet - Yes let's backport to 8.4

[nchaulet]

@elasticmachine merge upstream

[nchaulet]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 3be2c5e

Metrics [docs]

✅ unchanged

History

• 💚 Build #62613 succeeded 4589dd9
• 💔 Build #62597 failed 2a22f69
• 💚 Build #62567 succeeded 54a6c4e
• 💔 Build #62553 failed 6c8cb71
• 💔 Build #62520 failed d988ac5
• 💚 Build #62449 succeeded 7e6084c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[jsoriano]

Thanks!

[andrewkroh]

This is adding features and behavior that is not defined in the package-spec which undermines the value of working from a specification as our source of truth. Please go through the process to update package-spec so that everyone knows how to benefit from these useful changes.

[jsoriano]

Thanks @andrewkroh for raising this up, I have created an issue to follow on this elastic/package-spec#393.