[Fleet] Move mappings from index template to component template

[hop-dev]

Summary

Closes #121184.

A few changes to the component templates we create as part of package install and fleet setup:

• Move integration mappings to the @mapping component template. This means that every data stream will have the @mapping component template going forward.
• split .fleet_component_template-1 into 2 component templates (more detail below):
  • .fleet_globals-1 - contains fleet global settings previously applied to the index template
  • .fleet_agent_id_verification-1 - optionally added when agent id verification is enabled (basically just sets .fleet_final_pipeline-1)
• when a change is detected in the fleet component templates, we now force reinstall bundled packages (previously we only reinstalled registry packages)

These changes will be applied to all existing datastreams on startup by force installing each package, triggering a rollover.

Technical Details / Decisions

What to do with .fleet_component_template-1 ?

I had considered deleting the old component template to "clean up", however I have taken the stance that this introduces more risk than benefit. There is the chance that a data stream is still using the component template. We could investigate a way of finding out if it is being used but I thought there is little harm in leaving the template.

Package Custom Mappings

We do allow integrations to specify custom elastic mappings directly (on top of the field mappings that are calculated from the package .yml files in the package spec) for examples APM enables dynamic mapping.

Previously these custom mappings were the ONLY mappings specified in the @mapping component template, and therefore the @mapping component template was only created if a package specified these mappings. This PR moves ALL mappings into the @mapping component template, so we manually merge the field mappings with the packages custom mappings if they are present (previously this was handled by template inheritence).

Mapping settings

One complication has been that if an integration exceeds an elasticsearch default (e.g the max number of nested fields allowed) in the mapping for a given datastream, (e.g logs-endpoint.alerts
has more than 50 nested fields, so they have to add a settings override to increase that limit) then the @mapping component template becomes invalid and is rejected by elasticsearch. This means we have to add any custom index settings which are related to the mapping to ensure the component template is valid. This means the @mapping component template now must contain settings.index.mapping template settings.

New component template hierarchy and content

Here is a summary of the index and component template content as they now stand (highest precedence to lowest precedence):

• index template - empty except for _meta
• .fleet_agent_id_verification-1 - optionally added when agent id verification is enabled, sets the .fleet_final_pipeline-1 and agent ID mappings.
• .fleet_globals-1 - contains fleet global settings and mappings, applied to every data stream
• @custom component template - empty, available for user to apply custom settings
• @settings component template - fleet default settings plus any custom settings specified by the integration (EXCEPT settings.index.mapping in both cases)
• @mappings component template - fleet default settings for settings.index.mapping (e.g field_limit=10000) and any settings.index.mapping custom settings specified by the integration. Plus also all field mappings which previously were specified on the index template.

Test Steps

In order to test this we need to verify that the computed template remains the same now that we have moved the mapping to the component template.

We also need to verify that any custom mappings set by the package are applied, for example APM sets custom mappings.

the broad test steps are as follows:

Part 1 (on main branch)
If you want to skip step 1 here is the computed template for metrics-apm.app from the APM integration:

metrics-apm.app template

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "metrics-apm.app_metrics-default_policy"
        },
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "final_pipeline": ".fleet_final_pipeline-1",
        "query": {
          "default_field": [
            "agent.ephemeral_id",
            "agent.name",
            "agent.version",
            "client.domain",
            "client.geo.city_name",
            "client.geo.continent_name",
            "client.geo.country_iso_code",
            "client.geo.country_name",
            "client.geo.region_iso_code",
            "client.geo.region_name",
            "cloud.account.id",
            "cloud.account.name",
            "cloud.availability_zone",
            "cloud.instance.id",
            "cloud.instance.name",
            "cloud.machine.type",
            "cloud.project.id",
            "cloud.project.name",
            "cloud.provider",
            "cloud.region",
            "cloud.service.name",
            "container.id",
            "destination.address",
            "ecs.version",
            "event.outcome",
            "host.architecture",
            "host.hostname",
            "host.name",
            "host.os.platform",
            "observer.hostname",
            "observer.name",
            "observer.type",
            "observer.version",
            "process.args",
            "process.title",
            "service.environment",
            "service.name",
            "service.node.name",
            "service.version",
            "source.domain",
            "user.domain",
            "user.email",
            "user.id",
            "user.name",
            "user_agent.device.name",
            "user_agent.name",
            "user_agent.original",
            "user_agent.os.family",
            "user_agent.os.full",
            "user_agent.os.kernel",
            "user_agent.os.name",
            "user_agent.os.platform",
            "user_agent.os.version",
            "user_agent.version",
            "kubernetes.namespace",
            "kubernetes.node.name",
            "kubernetes.pod.name",
            "kubernetes.pod.uid",
            "metricset.name",
            "network.connection.type",
            "observer.ephemeral_id",
            "observer.id",
            "service.framework.name",
            "service.framework.version",
            "service.language.name",
            "service.language.version",
            "service.runtime.name",
            "service.runtime.version",
            "system.process.cgroup.cpu.id",
            "system.process.cgroup.cpuacct.id"
          ]
        },
        "default_pipeline": "metrics-apm.app-8.2.0-dev3"
      }
    },
    "mappings": {
      "dynamic": "true",
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      },
      "dynamic_templates": [
        {
          "histogram": {
            "mapping": {
              "type": "histogram"
            }
          }
        },
        {
          "numeric_labels": {
            "path_match": "numeric_labels.*",
            "mapping": {
              "scaling_factor": 1000000,
              "type": "scaled_float"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "client": {
          "properties": {
            "domain": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "geo": {
              "properties": {
                "city_name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "continent_name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "country_iso_code": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "country_name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "location": {
                  "type": "geo_point"
                },
                "region_iso_code": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "region_name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "ip": {
              "type": "ip"
            },
            "port": {
              "type": "long"
            }
          }
        },
        "cloud": {
          "properties": {
            "account": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "availability_zone": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "instance": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "machine": {
              "properties": {
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "project": {
              "properties": {
                "id": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "provider": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "region": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "service": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "container": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            },
            "type": {
              "type": "constant_keyword"
            }
          }
        },
        "destination": {
          "properties": {
            "address": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            },
            "port": {
              "type": "long"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "event": {
          "properties": {
            "agent_id_status": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ingested": {
              "type": "date",
              "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
            },
            "outcome": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "properties": {
            "architecture": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "os": {
              "properties": {
                "platform": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "kubernetes": {
          "properties": {
            "namespace": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "node": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "pod": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "uid": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "labels": {
          "type": "object",
          "dynamic": "true"
        },
        "metricset": {
          "properties": {
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "network": {
          "properties": {
            "connection": {
              "properties": {
                "type": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "numeric_labels": {
          "type": "object",
          "dynamic": "true"
        },
        "observer": {
          "properties": {
            "ephemeral_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "hostname": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version_major": {
              "type": "byte"
            }
          }
        },
        "process": {
          "properties": {
            "args": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "pid": {
              "type": "long"
            },
            "ppid": {
              "type": "long"
            },
            "title": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "processor": {
          "properties": {
            "event": {
              "type": "constant_keyword"
            },
            "name": {
              "type": "constant_keyword"
            }
          }
        },
        "service": {
          "properties": {
            "environment": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "framework": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "language": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "node": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "runtime": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "source": {
          "properties": {
            "domain": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ip": {
              "type": "ip"
            },
            "port": {
              "type": "long"
            }
          }
        },
        "system": {
          "properties": {
            "cpu": {
              "properties": {
                "total": {
                  "properties": {
                    "norm": {
                      "properties": {
                        "pct": {
                          "type": "scaled_float",
                          "meta": {
                            "metric_type": "gauge",
                            "unit": "percent"
                          },
                          "scaling_factor": 1000,
                          "time_series_metric": "gauge"
                        }
                      }
                    }
                  }
                }
              }
            },
            "memory": {
              "properties": {
                "actual": {
                  "properties": {
                    "free": {
                      "type": "long",
                      "meta": {
                        "metric_type": "gauge",
                        "unit": "byte"
                      }
                    }
                  }
                },
                "total": {
                  "type": "long",
                  "meta": {
                    "metric_type": "gauge",
                    "unit": "byte"
                  }
                }
              }
            },
            "process": {
              "properties": {
                "cgroup": {
                  "properties": {
                    "cpu": {
                      "properties": {
                        "cfs": {
                          "properties": {
                            "period": {
                              "properties": {
                                "us": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "gauge",
                                    "unit": "micros"
                                  }
                                }
                              }
                            },
                            "quota": {
                              "properties": {
                                "us": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "gauge",
                                    "unit": "micros"
                                  }
                                }
                              }
                            }
                          }
                        },
                        "id": {
                          "type": "keyword",
                          "ignore_above": 1024
                        },
                        "stats": {
                          "properties": {
                            "periods": {
                              "type": "long",
                              "meta": {
                                "metric_type": "counter"
                              }
                            },
                            "throttled": {
                              "properties": {
                                "ns": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "counter",
                                    "unit": "nanos"
                                  }
                                },
                                "periods": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "counter"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    },
                    "cpuacct": {
                      "properties": {
                        "id": {
                          "type": "keyword",
                          "ignore_above": 1024
                        },
                        "total": {
                          "properties": {
                            "ns": {
                              "type": "long",
                              "meta": {
                                "metric_type": "counter",
                                "unit": "nanos"
                              }
                            }
                          }
                        }
                      }
                    },
                    "memory": {
                      "properties": {
                        "mem": {
                          "properties": {
                            "limit": {
                              "properties": {
                                "bytes": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "gauge",
                                    "unit": "byte"
                                  }
                                }
                              }
                            },
                            "usage": {
                              "properties": {
                                "bytes": {
                                  "type": "long",
                                  "meta": {
                                    "metric_type": "gauge",
                                    "unit": "byte"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                },
                "cpu": {
                  "properties": {
                    "total": {
                      "properties": {
                        "norm": {
                          "properties": {
                            "pct": {
                              "type": "scaled_float",
                              "meta": {
                                "metric_type": "gauge",
                                "unit": "percent"
                              },
                              "scaling_factor": 1000,
                              "time_series_metric": "gauge"
                            }
                          }
                        }
                      }
                    }
                  }
                },
                "memory": {
                  "properties": {
                    "rss": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "meta": {
                            "metric_type": "gauge",
                            "unit": "byte"
                          }
                        }
                      }
                    },
                    "size": {
                      "type": "long",
                      "meta": {
                        "metric_type": "gauge",
                        "unit": "byte"
                      }
                    }
                  }
                }
              }
            }
          }
        },
        "user": {
          "properties": {
            "domain": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "email": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "user_agent": {
          "properties": {
            "device": {
              "properties": {
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "original": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "os": {
              "properties": {
                "family": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "full": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "kernel": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "name": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "platform": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "version": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

• choose an integration and data stream (e.g APM, metrics-apm.app), install the integration
• view the computed template (Stack Management > Index Management > Index Templates > (your stream) > Preview)
• save this template for reference later

Part 2 (on this feature branch, clean install)

• Install the same integration
• view the computed template, it should be the same as the template from part 1 (bar array ordering, you can use a tool such as json-diff for the comparison)

Upgrade testing

To test that the new component templates are created and applied on upgrade, we can take a 8.1.0 system, set up some integrations and then switch branch and restart. It is important to test that both registry and bundled packages are force reinstalled on first startup.

Detailed instructions:

• create vanilla 8.1.0 with bundled packages (yarn build) locally
• install a few integrations including some bundled ones
• switch to this branch
• start kibana and observe packages being installed (with no errors 🤞 )
• check template content (see part 1)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

We do allow integrations to specify custom mappings, such as APM adds it's own dynamic mapping, so I merge those settings with our properties mapping (previously this was done by template inheritance)

Can you elaborate on this? What is exactly merged with what?

settings.index.mapping custom settings/templates specified by the integration

I assume you refer here to settings made into the package and not by the users so these are constant for a certain version of a package.

More general question, it would be good to have docs / overview of all the templates that exist for each data stream and what the priority of each is. You have started this above but I think at least @custom is missing.

How is the migration going to work to this new setup? As the full change is non breaking I assume no rollover of the data streams is triggered. So the new structure will only be visible after the rollover of a data stream? Could this have any side effects?

[joshdover]

We do allow integrations to specify custom mappings, such as APM adds it's own dynamic mapping, so I merge those settings with our properties mapping (previously this was done by template inheritance)

Can you elaborate on this? What is exactly merged with what?

+1 on this, I thought that the package-specific mappings would be in @mappings and our properties would be in .fleet_component_template without needing to do any merging?

[joshdover]

I have kept mappings._meta on the index template as I think that is fundamental and we wouldn't want it to be ever overwritten.

What's the difference between putting this in the index template and putting this in the .fleet_component_template? Wouldn't that always take precedence anyways since it's the last component template. I'd prefer to go for consistency if at all possible.

[hop-dev]

@elasticmachine merge upstream

[hop-dev]

@joshdover @ruflin I have rewritten the PR description to hopefully be clearer about the changes I am making. Here are some quick summaries to address your comments above:

• the migration will happen on startup and a rollover will be triggered for each datastream, this logic already exists to check for changes in the fleet global template and was used in 8.0.0 I believe.
• there are 3 places a data stream's mapping is specified:
  1. fleet global mappings (now in .fleet_globals-1)
  2. the mapping generated from the fields in the package spec (this is the bulk of the mapping, was previously specified directly on the index template and is now always specified in @mapping component template)
  3. the package can also specify elasticsearch mappings to be applied directly (e.g APM enables dynamic mapping.) these were previously the only content of the @mapping component template, but are now merged with ii. above
• I'll be creating a dev doc to summarise all of this!

[ruflin]

Looked at the list of component templates again above and I think the @custom on is still missing?

when a change is detected in the fleet component templates, we now force reinstall bundled packages (previously we only reinstalled registry packages)

Do we really have to reinstall? I don't know exactly what happens on reinstall but I assume it will push all assets to the stack again. If someone has quite a few packages installed, this could take quite a bit of time. I understand it is to cleanup.

My assumption is that the summary of all mappings before and after this change is identical. The ingest pipeline names are different. If we do these changes, do we have to force it on startup? Lets assume we got something wrong, now Kibana can't start anymore?

Is there any asset that exists now but is not around anymore afterwards? Could it happen that we get stuck in the middle of the migration?

My concerns are not around the change itself but I want to make sure, migration (if needed) for our users is seamless and we don't block any users from upgrading.

[joshdover]

Do we really have to reinstall? I don't know exactly what happens on reinstall but I assume it will push all assets to the stack again. If someone has quite a few packages installed, this could take quite a bit of time. I understand it is to cleanup.

We're planning to post documentation about how to use @custom which won't work if the packages are in an inconsistent state. I think we should reinstall. We have broader plans to determine how to handle this type of change most gracefully and reliably in #121099.

For context, the change to reinstall bundled packages in this PR is more of a fix than a new change. We already reinstalled registry packages when these global component templates changed (eg. in the 8.0.0 release for #119380), but it wasn't being done for bundled packages because those are marked as being installed by upload.

My assumption is that the summary of all mappings before and after this change is identical. The ingest pipeline names are different. If we do these changes, do we have to force it on startup? Lets assume we got something wrong, now Kibana can't start anymore?

We don't block or fail Kibana startup today and don't plan to in the near future. So if something failed, Kibana would still be functional, but Fleet may not be able to setup correctly. We should always be sure that Fleet can setup correctly though :)

Is there any asset that exists now but is not around anymore afterwards? Could it happen that we get stuck in the middle of the migration?

Good question, I think we discussed not removing the old component templates for now, just in case something like this happens. @hop-dev could you confirm this is the case?

Nevermind, I see it was addressed in the description above, this template is not removed for now.

[joshdover]

@hop-dev is there a reason we need to retain the _meta key under the mappings in the index template instead of in the @mappings one? My assumption is that leaving it in the index template helps ensure it doesn't get overridden by a @custom component template.

As an aside, do you know what this is actually used for? Maybe the Managed label in the Data streams or Indices tabs of Index Management?

[joshdover]

Looking good! This is my first(-ish) pass, still need to do more in-depth upgrade testing and review of test code, but would like some of these questions answer first.

Questions:

• Is there any reason we don't want the content in .fleet_globals-1 to be override-able by @custom templates? It currently contains the dynamic template for forcing keyword mappings and the date_detection: false setting.

[hop-dev]

Yes that is the case, I've added a section in the PR description.

I've been trying to run some performance tests on the reinstalls but I've been severely hampered by #126611 causing too many install failures at the moment.

Anecdotally for 20 packages I was seeing individual install times of approx 20 - 50s (but with an error rate of ~20%) but the whole process taking ~75s as they are run concurrently.

[hop-dev]

@joshdover @nchaulet I've created an issue for agentIdVerificationEnabled deprecation #127041

[joshdover]

Last few comments on the implementation here. Working on some final manual testing right now.

[joshdover]

I wonder if there's ever an issue in a case where a pipeline was removed from an integration package and we don't unset the pipeline from the current write index.

Maybe we should track a separate issue for this?

[joshdover]

Ran some upgrade testing today and found no issues. Here's a summary of what I did:

1. Start ES 8.2.0-SNAPSHOT (for simplicity's sake I didn't test upgrading ES)
2. Start Kibana 8.1.0 and install several packages
3. Make some changes to some @custom templates, for example I added settings and mappings to the metrics-system.cpu@custom component template
4. Shutdown Kibana 8.1.0
5. Start Kibana from this branch against the same ES instance
6. Wait for Fleet setup to complete
7. Verify that @Custom settings and mappings were preserved and new @mappings template was added to all index templates.

I'm comfortable merging this PR after my last few suggestions are fixed. I do wonder if we're introducing a subtle breaking change by allowing @custom to override mappings when they previously couldn't. However, I think being able to override the mappings was the intended behavior and we could consider the old behavior a bug.

[joshdover]

I do wonder if we're introducing a subtle breaking change by allowing @custom to override mappings when they previously couldn't. However, I think being able to override the mappings was the intended behavior and we could consider the old behavior a bug.

Confirmed with @ruflin that this is the desired behavior. LGTM ✅

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 67aafbb
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1241	1249	+8

Unknown metric groups

API count

id	before	after	diff
fleet	1358	1366	+8

History

• 💔 Build #29905 failed 204a4c71d45aadeb65768bac6d86e4e486ecb347
• 💔 Build #29488 failed cca119e
• 💛 Build #28177 was flaky 8ddebf0
• 💔 Build #28064 failed 74ccc14
• 💔 Build #28043 failed c067e5e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @hop-dev

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 124013 or prevent reminders by adding the backport:skip label.

[hop-dev]

Thanks Jen!

[amolnater-qasource]

Hi @hop-dev
Thanks for the detailed steps, we have validated this PR with the provided steps on latest 8.2 Snapshot Kibana cloud environment.
Steps followed:

1. Setup 8.2 Snapshot environment.
2. Install the APM integration and add to a policy
3. Stack management > index management > index templates > find "metrics-apm.app" > preview (see screenshot)
4. Copy the JSON document shown
5. Using a tool such as JSON diff, compare the JSON document with the attached JSON document "metrics-apm.app-7x.json"
   metrics-apm.app-7x.zip

Observations:

As per results there are 02 missing properties and 36 unequal results.

• Missing version-major field on the fleet side data.
• Missing nat property on old json file.

Build details:
BUILD: 52001
COMMIT: cf743d3

Json file copied from fleet:
Copied from Fleet.txt

Screenshots:

Please let us know if we are missing anything.
cc: @joshdover
Thanks

[hop-dev]

Hi @amolnater-qasource, thanks for this, I have reviewed the differences and they are all acceptable ✅ . The issue is that I generated metrics-apm.app-7x.json from a slightly older dev version of the apm integration, there have since been changes to the integration which have added or changed fields.

[amolnater-qasource]

Hi @hop-dev
Thanks for confirming the expected differences in configuration.
As the testing is done on this, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks