[Fleet] Move mappings from index template to component template #123383

hop-dev · 2022-01-19T16:49:24Z

Summary

Moves all mappings to the @mapping component template. This means that ever data stream will have the @mapping component template going forward.

We do allow integrations to specify custom mappings, such as APM adds it's own dynamic mapping, so I've had to merge the integration mapping with our base mapping (previously this was done by template inheritance.

Test Steps

In order to test this we need to verify that the computed template remains the same now that we have moved the mapping to the component template.

We also need to verify that any custom mappings set by the package are applied, for example APM enables dynamic mapping

the broad test steps are as follows:

Part 1 (on main branch)
If you want to skip step 1 here is the computed template for metrics-apm.app from the APM integration:

metrics-apm.app template

{
    "template": {
      "settings": {
        "index": {
          "lifecycle": {
            "name": "metrics-apm.app_metrics-default_policy"
          },
          "codec": "best_compression",
          "default_pipeline": "metrics-apm.app-8.0.0-dev4",
          "mapping": {
            "total_fields": {
              "limit": "10000"
            }
          },
          "final_pipeline": ".fleet_final_pipeline-1",
          "query": {
            "default_field": [
              "agent.ephemeral_id",
              "agent.name",
              "agent.version",
              "client.domain",
              "client.geo.city_name",
              "client.geo.continent_name",
              "client.geo.country_iso_code",
              "client.geo.country_name",
              "client.geo.region_iso_code",
              "client.geo.region_name",
              "cloud.account.id",
              "cloud.account.name",
              "cloud.availability_zone",
              "cloud.instance.id",
              "cloud.instance.name",
              "cloud.machine.type",
              "cloud.project.id",
              "cloud.project.name",
              "cloud.provider",
              "cloud.region",
              "cloud.service.name",
              "container.id",
              "destination.address",
              "ecs.version",
              "event.outcome",
              "host.architecture",
              "host.hostname",
              "host.name",
              "host.os.platform",
              "observer.hostname",
              "observer.name",
              "observer.type",
              "observer.version",
              "process.args",
              "process.title",
              "service.environment",
              "service.name",
              "service.node.name",
              "service.version",
              "source.domain",
              "user.domain",
              "user.email",
              "user.id",
              "user.name",
              "user_agent.device.name",
              "user_agent.name",
              "user_agent.original",
              "user_agent.os.family",
              "user_agent.os.full",
              "user_agent.os.kernel",
              "user_agent.os.name",
              "user_agent.os.platform",
              "user_agent.os.version",
              "user_agent.version",
              "kubernetes.namespace",
              "kubernetes.node.name",
              "kubernetes.pod.name",
              "kubernetes.pod.uid",
              "metricset.name",
              "network.connection.type",
              "observer.ephemeral_id",
              "observer.id",
              "service.framework.name",
              "service.framework.version",
              "service.language.name",
              "service.language.version",
              "service.runtime.name",
              "service.runtime.version",
              "system.process.cgroup.cpu.id",
              "system.process.cgroup.cpuacct.id"
            ]
          }
        }
      },
      "mappings": {
        "dynamic": "true",
        "_meta": {
          "package": {
            "name": "apm"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "dynamic_templates": [
          {
            "histogram": {
              "mapping": {
                "type": "histogram"
              }
            }
          },
          {
            "numeric_labels": {
              "path_match": "numeric_labels.*",
              "mapping": {
                "scaling_factor": 1000000,
                "type": "scaled_float"
              }
            }
          },
          {
            "strings_as_keyword": {
              "match_mapping_type": "string",
              "mapping": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          }
        ],
        "date_detection": false,
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "agent": {
            "properties": {
              "ephemeral_id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "client": {
            "properties": {
              "domain": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "geo": {
                "properties": {
                  "city_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "continent_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "country_iso_code": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "country_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "location": {
                    "type": "geo_point"
                  },
                  "region_iso_code": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "region_name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "ip": {
                "type": "ip"
              },
              "port": {
                "type": "long"
              }
            }
          },
          "cloud": {
            "properties": {
              "account": {
                "properties": {
                  "id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "availability_zone": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "instance": {
                "properties": {
                  "id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "machine": {
                "properties": {
                  "type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "project": {
                "properties": {
                  "id": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "provider": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "region": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "service": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "container": {
            "properties": {
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "data_stream": {
            "properties": {
              "dataset": {
                "type": "constant_keyword"
              },
              "namespace": {
                "type": "constant_keyword"
              },
              "type": {
                "type": "constant_keyword"
              }
            }
          },
          "destination": {
            "properties": {
              "address": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "ip": {
                "type": "ip"
              },
              "port": {
                "type": "long"
              }
            }
          },
          "ecs": {
            "properties": {
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "event": {
            "properties": {
              "agent_id_status": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "ingested": {
                "type": "date",
                "format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
              },
              "outcome": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "host": {
            "properties": {
              "architecture": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "hostname": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "ip": {
                "type": "ip"
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "os": {
                "properties": {
                  "platform": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "kubernetes": {
            "properties": {
              "namespace": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "node": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "pod": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "uid": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "labels": {
            "type": "object",
            "dynamic": "true"
          },
          "metricset": {
            "properties": {
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "network": {
            "properties": {
              "connection": {
                "properties": {
                  "type": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              }
            }
          },
          "numeric_labels": {
            "type": "object",
            "dynamic": "true"
          },
          "observer": {
            "properties": {
              "ephemeral_id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "hostname": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "type": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "version_major": {
                "type": "byte"
              }
            }
          },
          "process": {
            "properties": {
              "args": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "pid": {
                "type": "long"
              },
              "ppid": {
                "type": "long"
              },
              "title": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "processor": {
            "properties": {
              "event": {
                "type": "constant_keyword"
              },
              "name": {
                "type": "constant_keyword"
              }
            }
          },
          "service": {
            "properties": {
              "environment": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "framework": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "language": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "node": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "runtime": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "source": {
            "properties": {
              "domain": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "ip": {
                "type": "ip"
              },
              "port": {
                "type": "long"
              }
            }
          },
          "system": {
            "properties": {
              "cpu": {
                "properties": {
                  "total": {
                    "properties": {
                      "norm": {
                        "properties": {
                          "pct": {
                            "type": "scaled_float",
                            "meta": {
                              "metric_type": "gauge",
                              "unit": "percent"
                            },
                            "scaling_factor": 1000
                          }
                        }
                      }
                    }
                  }
                }
              },
              "memory": {
                "properties": {
                  "actual": {
                    "properties": {
                      "free": {
                        "type": "long",
                        "meta": {
                          "metric_type": "gauge",
                          "unit": "byte"
                        }
                      }
                    }
                  },
                  "total": {
                    "type": "long",
                    "meta": {
                      "metric_type": "gauge",
                      "unit": "byte"
                    }
                  }
                }
              },
              "process": {
                "properties": {
                  "cgroup": {
                    "properties": {
                      "cpu": {
                        "properties": {
                          "cfs": {
                            "properties": {
                              "period": {
                                "properties": {
                                  "us": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "gauge",
                                      "unit": "micros"
                                    }
                                  }
                                }
                              },
                              "quota": {
                                "properties": {
                                  "us": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "gauge",
                                      "unit": "micros"
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "id": {
                            "type": "keyword",
                            "ignore_above": 1024
                          },
                          "stats": {
                            "properties": {
                              "periods": {
                                "type": "long",
                                "meta": {
                                  "metric_type": "counter"
                                }
                              },
                              "throttled": {
                                "properties": {
                                  "ns": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "counter",
                                      "unit": "nanos"
                                    }
                                  },
                                  "periods": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "counter"
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      },
                      "cpuacct": {
                        "properties": {
                          "id": {
                            "type": "keyword",
                            "ignore_above": 1024
                          },
                          "total": {
                            "properties": {
                              "ns": {
                                "type": "long",
                                "meta": {
                                  "metric_type": "counter",
                                  "unit": "nanos"
                                }
                              }
                            }
                          }
                        }
                      },
                      "memory": {
                        "properties": {
                          "mem": {
                            "properties": {
                              "limit": {
                                "properties": {
                                  "bytes": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "gauge",
                                      "unit": "byte"
                                    }
                                  }
                                }
                              },
                              "usage": {
                                "properties": {
                                  "bytes": {
                                    "type": "long",
                                    "meta": {
                                      "metric_type": "gauge",
                                      "unit": "byte"
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "cpu": {
                    "properties": {
                      "total": {
                        "properties": {
                          "norm": {
                            "properties": {
                              "pct": {
                                "type": "scaled_float",
                                "meta": {
                                  "metric_type": "gauge",
                                  "unit": "percent"
                                },
                                "scaling_factor": 1000
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "memory": {
                    "properties": {
                      "rss": {
                        "properties": {
                          "bytes": {
                            "type": "long",
                            "meta": {
                              "metric_type": "gauge",
                              "unit": "byte"
                            }
                          }
                        }
                      },
                      "size": {
                        "type": "long",
                        "meta": {
                          "metric_type": "gauge",
                          "unit": "byte"
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "user": {
            "properties": {
              "domain": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "email": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "id": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "user_agent": {
            "properties": {
              "device": {
                "properties": {
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "name": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "original": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "os": {
                "properties": {
                  "family": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "full": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "kernel": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "name": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "platform": {
                    "type": "keyword",
                    "ignore_above": 1024
                  },
                  "version": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "version": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          }
        }
      },
      "aliases": {}
    }
  }

choose an integration and data stream (e.g APM, metrics-apm.app), install the integration
view the computed template (Stack Management > Index Management > Index Templates > (your stream) > Preview)
save this template for reference later

Part 2 (on this feature branch, clean install)

Install the same integration
view the computed template, it should be the same as the template from part 1 (bar array ordering, you can use a tool such as json-diff for the comparison)

hop-dev · 2022-01-20T11:53:33Z

@elasticmachine merge upstream

elasticmachine · 2022-01-20T11:53:42Z

Pinging @elastic/fleet (Team:Fleet)

hop-dev · 2022-01-20T16:01:46Z

@elasticmachine merge upstream

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts

juliaElastic · 2022-01-24T08:07:11Z

how can this change be tested?

…into 121184-mapping-component

…onent

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…#123554) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

hop-dev · 2022-01-24T13:43:43Z

how can this change be tested?

Hi @juliaElastic I've added test steps to the PR now 👍

…tic#123534) * Timelion rescaling vertical bounds incorrectly on window resize * Update test axis labels data as chart has another scale due to updated padding Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

mattkime · 2022-01-27T20:06:15Z

I'm attempting to view the changes and my browser is having a really hard time. Is this something that could be completed across multiple PRs? 1.3k files changed in a single PR is a lot!

cjcenizal · 2022-01-27T20:15:15Z

@hop-dev Looks like something went wrong with a merge and a ton of teams got erroneously pinged for code review. If that's so, would you mind closing this and reopening it in a new PR to avoid hitting a lot of unrelated folks with unnecessary pings?

hop-dev · 2022-01-28T10:06:31Z

No idea what happened sorry about this!

hop-dev added 10 commits January 19, 2022 14:47

component template working

ab21926

move component template construction to function

0aa2468

remove unused mapping args

c04e210

fix template tests

e25ca80

fix types

f1a652c

component template working

a65ff6e

move component template construction to function

f3eef91

remove unused mapping args

ad2d483

fix template tests

0241696

fix types

2a76a6c

hop-dev marked this pull request as ready for review January 20, 2022 11:53

hop-dev requested a review from a team as a code owner January 20, 2022 11:53

Merge branch 'main' into 121184-mapping-component

bb14cdd

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 20, 2022

Merge branch 'main' into 121184-mapping-component

d2f896c

juliaElastic reviewed Jan 24, 2022

View reviewed changes

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts Show resolved Hide resolved

hop-dev and others added 5 commits January 24, 2022 11:50

Merge branch '121184-mapping-component' of github.com:hop-dev/kibana …

ce1909a

…into 121184-mapping-component

Merge remote-tracking branch 'upstream/main' into 121184-mapping-comp…

208206d

…onent

[APM] Simplify browser router types (elastic#123549)

429b252

tooltip added to OS column (elastic#123377)

e76cedb

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

[APM] Include error documents in fallback query for services (elastic…

650919c

…#123554) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

hop-dev added v8.1.0 release_note:skip Skip the PR/issue when compiling release notes labels Jan 24, 2022

DianaDerevyankina and others added 2 commits January 24, 2022 16:46

Add comment to mergeWith function

a6a2082

hop-dev requested review from a team as code owners January 27, 2022 15:59

hop-dev requested review from joeypoon and parkiino January 27, 2022 15:59

hop-dev closed this Jan 28, 2022

hop-dev deleted the 121184-mapping-component branch January 28, 2022 10:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Move mappings from index template to component template #123383

[Fleet] Move mappings from index template to component template #123383

hop-dev commented Jan 19, 2022 •

edited

Loading

hop-dev commented Jan 20, 2022

elasticmachine commented Jan 20, 2022

hop-dev commented Jan 20, 2022

juliaElastic commented Jan 24, 2022

hop-dev commented Jan 24, 2022

mattkime commented Jan 27, 2022

cjcenizal commented Jan 27, 2022

hop-dev commented Jan 28, 2022

[Fleet] Move mappings from index template to component template #123383

[Fleet] Move mappings from index template to component template #123383

Conversation

hop-dev commented Jan 19, 2022 • edited Loading

Summary

Test Steps

hop-dev commented Jan 20, 2022

elasticmachine commented Jan 20, 2022

hop-dev commented Jan 20, 2022

juliaElastic commented Jan 24, 2022

hop-dev commented Jan 24, 2022

mattkime commented Jan 27, 2022

cjcenizal commented Jan 27, 2022

hop-dev commented Jan 28, 2022

hop-dev commented Jan 19, 2022 •

edited

Loading