index patterns, for example metricbeat-* do not always get created when running setup

[DanRoscigno]

• 7.6.2
• Debian Linux to ESS

Sometimes the index pattern is created via setup, and sometimes it is not. @ycombinator suggested running setup with --dashboards, which seems to be a default. In very limited testing adding --dashboards seems to get the index pattern created.

I used Metricbeat in the example, but Filebeat is the same.

cc: @EthanStrider

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[andresrc]

This is currently the intended behaviour, so we will document it.

[DanRoscigno]

I am probably misunderstanding. What is the intended behavior?

[DanRoscigno]

@andresrc , can you please give me some more information. Are you saying that the intended behavior is that the dashboards will be created by default, but to get the index patterns we have to specify --dashboards? If so, could --dashboards be changed to --index-pattern?

[kvch]

@DanRoscigno The intended behaviour is that index patterns are loaded with dashboards. That is what we wanted to document as an intended behaviour. In the long run, we would like to get rid of the subcommand setup, so we rather not put much effort into maintaining it.

Also, after searching a bit, I have found an undocumented setting for setup: only_index which lets you load only index patterns without loading dashboards: #7285

You can set it under setup.dashboards section of the configuration:

setup.dashboards.only_index: true

[DanRoscigno]

@DanRoscigno The intended behaviour is that index patterns are loaded with dashboards. That is what we wanted to document as an intended behaviour.

Thanks for the info @kvch . The reason I opened the issue is that when the dashboards are loaded the index patterns do not always get created. Sometimes they do, and sometimes they do not. The dashboards always get loaded when setup is run with the default beatname.yml file. setup.dashboards.only_index does not help, as users want dashboards and index patterns.

Since loading the dashboards when running setup is the default, and creating the index pattern is supposed to happen when the dashboards get loaded, the index pattern should be created also.

[EamonnTP]

Hi @kvch @DanRoscigno Would recommending that a user should include the setup.dashboards.only_index: true and setup.dashboards.enabled: true settings in the config file, help clarify this issue? I can make the necessary doc updates and add you both as reviewers.

[DanRoscigno]

Thanks @EamonnTP . I don't think "fixing the bug in the docs" is a good habit. If we can modify the default .yml file without a side effect of causing setup to run every time the Beat is run, then I would be good with fixing it in the docs and config files instead of in the code.

BUT

Have you tested combining those two settings? From their names they should be mutually exclusive.

[kvch]

@DanRoscigno You are right. It is indeed not a good habit to fix the bug in the docs. However, we are retiring the setup subcommand of Beats in favour of Elastic Integrations. In that solution, dashboards, templates, etc. are installed by Kibana, not Beats. At the moment we are focusing on creating better integrations, rather than trying to fix a feature which is going to be discontinued. :)

[EamonnTP]

@DanRoscigno Confirmed with @kvch that it's unfortunate both setting names sound mutually exclusive, however, both can be used in the config file.

[DanRoscigno]

Cool, I am glad to review the PR and update the best practices. Thanks.

[dedemorton]

@EamonnTP When you work on this issue, please check the troubleshooting topic about this problem to see if it can be improved, too: https://www.elastic.co/guide/en/beats/metricbeat/current/could-not-locate-index-pattern.html

[EamonnTP]

Hi @kvch @DanRoscigno I need a little guidance with this. I enabled both settings setup.dashboards.enabled: true
setup.dashboards.only_index: true in the metricbeat.yml, but it returns an error Error importing Kibana dashboards: fail to import the dashboards in Kibana.

[kvch]

@DanRoscigno I know it has been a while since you opened this issue. But is there any chance you could share what privileges the user you used to setup has, what modules did you enabled, etc.? I have spent the last few hours trying to reproduce the issue with multiple versions of Beats, Elasticsearch and Kibana. Even if I enabled all modules or none of them, the index template was loaded successfully.

Or if someone else is facing this issue, I would like to know about every little detail there is, because so far I cannot reproduce the problem.

[DanRoscigno]

I don't remember @kvch , way too long ago and I was told it was the intended behavior.