Improve and extend template and dashboard configurations

[ruflin]

In 6.0 the template configuration changes as it is dynamically loaded. Also recently dashboards configs to load dashboards were introduced. In addition logic to load certain data on startup like pipelines were added. This issue propose a potential way forward with and extensible config format for the above cases.

Current configuration

The current configuration for dashboard and template looks as following:

dashboard:
  enabled: false
  url:
  directory:
  file:
  snapshot: false
  snapshot_url
  beat: metricbeat
  kibana_index: .kibana
  index:
elasticsearch:
  template:
    enabled: true
    name: "metricbeat"
    fields: "${path.config}/fields.yml"
    overwrite: false

The template part is inside elasticsearch output as it currently only applies to the ES output.

Proposal

The proposal is to place all setup related configuration options under one namespace called setup and move all config options there. This would look as following:

setup:
  fields: "${path.config}/fields.yml"
  dashboard:
    ...
  template:
    ...

The fields path is on the top level as it can not only apply to templates as it does now, but also can be used to dynamically generate the index pattern for kibana dashboards or potential pipelines in the future.

This approach would have a few advantages:

• Setup configs are decoupled from all the other options. Potentially an extra tool (beats-tool) could make use of it.
• Template is not directly related to the elasticsearch output. Potentially in the future the template can also be loaded through LS. It would also decouple the code.
• An elasticserach output could be added (similar to monitoring) which inherits from the main es output, but could for example have different credentials with different access rights (load template).
• More outputs like kibana could be added for example for the dashboards
• More options can be added under setup like pipeline to define a list of pipelines which should be loaded

The above is a breaking change and we would need to have a migration script if possible.

Further ideas

The above change will allow to solve some more problem in #3654. These changes are not tied to the above but it would be good to do them at the same time.

• Create a local template.json file
• Add settings to the template

For this the config could be extended as following:

setup:
  fields: "${path.config}/fields.yml"
  beat.version: 6.2.3
  dashboard:
  template:
    enabled: true
    name: "metricbeat"
    overwrite: false
    # Defines if template should be loaded to elasticsearch
    load: true
    # Creates the json format of the template for manual loading
    output: metricbeat.template.json
    # Settings is a dynamic dict. These fields will be directly added to the template -> no logic check
    settings:
      number_of_shards: 1
      number_or_replicas: 1

In the above the follwing changed:

• load config option was added to be able to only generate the local file and not load it into elaticsearch
• output config option was added to define the file the template should be written to. Probably other name needed to not confuse with output config.
• beat.version config option was added. This would be need if someone wants to load a template for a different version. Thinking of beats-tool which is not updated that often.
• settings config option was added. This could be a dict that is added to the template to provide the possibility to configure shards / replicas and others directly when generating the template.

Related to #3654 and #3691

[tsg]

SGTM.

load config option was added to be able to only generate the local file and not load it into elaticsearch
output config option was added to define the file the template should be written to. Probably other name needed to not confuse with output config.

Perhaps we could reduce these to a single option (output or output_to_file) which implies load: false. Otherwise, people might get confused between enabled and load.

settings config option was added. This could be a dict that is added to the template to provide the possibility to configure shards / replicas and others directly when generating the template.

The question here is if we want to allow overwriting any mapping setting or we explicitly add things? I'd say we start with explicit settings for the things we know are useful (number of shards, number of replicas, index numerics).

[tsg]

Just to have mentioned it, this will make setting up the dahsboards URL from the CLI a bit more annoying: -E setup.dashboards.url=... vs the current -E dashboards.url=.... I think it's not a big deal, but just so that we're aware about this.

I also noticed that it's dashboard (singular) in your examples. Is that intentional? Because there generally are multiple dashboards per Beat, right?

[exekias]

I like the idea, kudos!

[ruflin]

I will start the migration step by step.

For the longer -E option: In case this becomes an issue, we could introduce something like -dashboard.url=... as a flag but I think specifying the url is more an expert or testing use case.

For dashboards: Agree that it should be plural as it is now.

[tsg]

Closing as old and no longer relevant.