Dynamically generate and load index templates #3654

ruflin · 2017-02-23T07:20:16Z

Currently index templates are generated with python and then packaged with each beat. This has the disadvantage that with more ES versions more files have to be generated and packaged. In addition it does not allow to only provide a template for one metricbeat module for example. Generating the templates as part of the beat will allow to generate the needed templates on request and does not required anymore prepackage templates.

Implementation steps:

The text was updated successfully, but these errors were encountered:

Each beat has its own way of generating the fields.yml file. Some have `_meta/fields.yml`, others `_meta/fields.generated.yml`. All scripts that used the fields.yml file had to do checks for these files and include the `libbeat/_meta/fields.generate.yml` file to get all data in. Now all beats will create `_meta/fields.generated.yml`. In case it does not exist, it is just a copy of `_meta/fields.yml` and will be automatically generated. In addition the file `_meta/fields.full.generated.yml` was introduced, which also contains the content from the `libbeat/_meta/fields.generate.yml`. This file can be used in most scripts to get all fields. The scripts were changed / simplified accordingly. This simplifies the implementation of dynamic template loading as only 1 file has to be taken into account. Part of elastic#3654

Each beat has its own way of generating the fields.yml file. Some have `_meta/fields.yml`, others `_meta/fields.generated.yml`. All scripts that used the fields.yml file had to do checks for these files and include the `libbeat/_meta/fields.generate.yml` file to get all data in. Now all beats will create `_meta/fields.generated.yml`. In case it does not exist, it is just a copy of `_meta/fields.yml` and will be automatically generated. In addition the file `_meta/fields.full.generated.yml` was introduced, which also contains the content from the `libbeat/_meta/fields.generate.yml`. This file can be used in most scripts to get all fields. The scripts were changed / simplified accordingly. This simplifies the implementation of dynamic template loading as only 1 file has to be taken into account. Part of #3654

* Only one file is passed instead of an array as now only file is needed. Introduce `-file` param. * Local loading of beats version was removed as it can lean to cyclic imports * Move `_meta/fields.full.generated.yml` to `fields.yml` in preparation for packaging `fields.yml` and ignore it in git * Cleanup Template generation package * Rename -beat.name param to -index as this describes it better Part of elastic#3654

* Only one file is passed instead of an array as now only file is needed. Introduce `-file` param. * Local loading of beats version was removed as it can lean to cyclic imports * Move `_meta/fields.full.generated.yml` to `fields.yml` in preparation for packaging `fields.yml` and ignore it in git * Cleanup Template generation package * Rename -beat.name param to -index as this describes it better Part of #3654

ruflin · 2017-03-22T13:19:38Z

Elasticsearch will potentially remove the support for template inheritance in 6.0 and will always pick the one if the highest order. This will impact the last point "template generation per module".

So far the template was loaded from the `beatname.template.json` files. This now uses the `fields.yml` file to dynamically generate the template based on the ES version connected to. The name of the template is set dynamically based on `template.name`. By default it has the value {beatname}. * [x] Remove generated templates * [x] Package fields.yml * [x] Clean up config options as 2.x option is not needed anymore * [x] Update docs * [x] Update and test packaging Further changes: * Remove tests in libbeat to load all templates. This requires global knowledge of the other beats in libbeat which is not good. * Create new makefile target to generate template for testing purpose. Questions: * Config option is called `template.path` which points to `fields.yml`. Is the name still ok? * What is our recommendation on loading the template manually? Generate first with `index_template` binary and then load manually? * Do we ship in index_template binary similar to import_dashboards? Can we integrate it into beats directly? -> setup * Should template config be moved to the top level (like `dashboard`) Part of elastic#3654

* Dynamically generate template on startup So far the template was loaded from the `beatname.template.json` files. This now uses the `fields.yml` file to dynamically generate the template based on the ES version connected to. The name of the template is set dynamically based on `template.name`. By default it has the value {beatname}. * [x] Remove generated templates * [x] Package fields.yml * [x] Clean up config options as 2.x option is not needed anymore * [x] Update docs * [x] Update and test packaging Further changes: * Remove tests in libbeat to load all templates. This requires global knowledge of the other beats in libbeat which is not good. * Create new makefile target to generate template for testing purpose. Questions: * Config option is called `template.path` which points to `fields.yml`. Is the name still ok? * What is our recommendation on loading the template manually? Generate first with `index_template` binary and then load manually? * Do we ship in index_template binary similar to import_dashboards? Can we integrate it into beats directly? -> setup * Should template config be moved to the top level (like `dashboard`) Part of #3654 * rename template.path to template.fields

* Change config option to setup.template.* from outputs.elasticsearch.template.* * Move loading logic into template package * Remove template loading logic from elasticsearch output * Changelog updated * Template tests were moved from output to template package * Documentation was updated. Will need some more work for which a follow up Github issue will be created. * Add `GetVersion()` to elasticsearch client. * Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended. This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR. Part of elastic#3654 and elastic#3921

* Change config option to setup.template.* from outputs.elasticsearch.template.* * Move loading logic into template package * Remove template loading logic from elasticsearch output * Changelog updated * Template tests were moved from output to template package * Documentation was updated. Will need some more work for which a follow up Github issue will be created. * Add `GetVersion()` to elasticsearch client. * Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended. This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR. Part of #3654 and #3921

This allows adding and overwriting index/template settings via the configuration file. Example config: setup.template.settings: index.number_of_shards: 1 index.number_of_replicas: 1 Needed for elastic#4112, part of elastic#3654.

This allows adding and overwriting index/template settings via the configuration file. Example config: setup.template.settings: index.number_of_shards: 1 index.number_of_replicas: 1 Needed for #4112, part of #3654.

This adds/fixes: * Ability to disable _source, or set other _source related options * Moves template index settings under `settings.index` * Fixes the overwrite logic (was using the wrong template name on the check) * Fixes error handling * Integration tests for overwritting Part of elastic#3654 and elastic#4112.

This adds/fixes: * Ability to disable _source, or set other _source related options * Moves template index settings under `settings.index` * Fixes the overwrite logic (was using the wrong template name on the check) * Fixes error handling * Integration tests for overwritting Part of #3654 and #4112.

Part of elastic#3654.

tsg · 2017-05-17T10:40:51Z

@dedemorton Pinging you here for awareness, for 6.0 we'll be significantly changing the way templates and dashboards are loaded. This will likely require some updates in the existing guides, but lets wait for things to stabilize a bit before updating them.

Part of #3654. This adds two settings: `setup.template.output_to_file.path` and `setup.template.output_to_file.version`. The version refers to the ES version and is optional, we'll use the Beats version if not specified. I put it under `output_to_file` to make it clear that it only applies when outputting to a file: To generate a config, one can do: ``` ./metricbeat -e -E setup.template.output_to_file.path=template.json ``` In the current version, the Beat automatically stops after generating the template, but the output might be slightly confusing: ``` 2017/05/16 09:55:02.043671 load.go:116: INFO Template for Elasticsearch 6.0.0-alpha2 written to: template.json 2017/05/16 09:55:02.043717 beat.go:538: CRIT Exiting: Stopping after successfully writing the template to the file. Exiting: Stopping after successfully writing the template to the file. ``` IMO this is better than the alternative of leaving it running. To generate the template for the 2.x version, one can do: ``` ./metricbeat -e -E setup.template.output_to_file.path=template.json -E setup.template.output_to_file.version=2.4.0 ``` Remaining TODOs: * [x] System test * [x] Docs * [x] Changelog

tsg · 2019-12-06T18:35:55Z

Closing, looks mostly done.

ruflin added the meta label Feb 23, 2017

ruflin mentioned this issue Feb 23, 2017

Rename object-type to object_type #3655

Merged

ruflin mentioned this issue Feb 27, 2017

Unify fields generated file across all beats #3677

Merged

ruflin mentioned this issue Feb 27, 2017

Consider defaulting index.codec in metricbeat ES template to best_compression? #3141

Closed

ruflin mentioned this issue Feb 27, 2017

Simplify template generation and handling #3683

Merged

ruflin mentioned this issue Mar 29, 2017

Dynamically generate template on startup #3681

Merged

5 tasks

ruflin mentioned this issue Apr 5, 2017

Improve and extend template and dashboard configurations #3921

Closed

ruflin mentioned this issue Apr 24, 2017

Change template loading configuration to be part of setup #4080

Merged

ruflin mentioned this issue May 10, 2017

Template setting in the configuration file #4284

Merged

4 tasks

tsg mentioned this issue May 15, 2017

Add option to adjust _source options + improvements #4317

Merged

2 tasks

tsg pushed a commit to tsg/beats that referenced this issue May 16, 2017

Add the option to generate the template into a file

dabc10c

Part of elastic#3654.

tsg mentioned this issue May 16, 2017

Add the option to generate the template into a file #4323

Merged

3 tasks

ruflin added the libbeat label Feb 26, 2018

tsg closed this as completed Dec 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamically generate and load index templates #3654

Dynamically generate and load index templates #3654

ruflin commented Feb 23, 2017 •

edited by tsg

Loading

ruflin commented Mar 22, 2017

tsg commented May 17, 2017

tsg commented Dec 6, 2019

Dynamically generate and load index templates #3654

Dynamically generate and load index templates #3654

Comments

ruflin commented Feb 23, 2017 • edited by tsg Loading

ruflin commented Mar 22, 2017

tsg commented May 17, 2017

tsg commented Dec 6, 2019

ruflin commented Feb 23, 2017 •

edited by tsg

Loading