Metricbeat reinstalls dashboards on every restart. Overwrites user changes to dashboards

[LeeDr]

• Version: 7.17.0 (but issue started in 7.16)
• Operating System: CentOS 7

I'm labeling this a bug because I don't think it was an intentional change back in 7.16 to always load the dashboards when metricbeat starts. I haven't checked this yet on a tar.gz install, only the rpm package install. I'm filing this on metricbeat but I think all beats are doing this.

• Steps to Reproduce:

1. Install metricbeat 7.17.0-SNAPSHOT from rpm
2. see config pasted below;
3. start metricbeat service
4. see in logs that metricbeat is loading all the saved objects with overwrite
5. modify a metricbeat dashboard
6. restart metricbeat service
7. watch metricbeat log shows its loading dashboards again
8. reopen the dashboard and see your changes are gone

I verified metricbeat 7.15 doesn't load saved objects unless you call the setup step. But 7.16 and 7.17 do.

metricbeat.yml:

logging.level: debug
logging.to_files: true

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 10s
  reload.enabled: false
metricbeat.max_start_delay: 10s
metricbeat.modules:
- module: system
  enabled: true
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
  processes: ['.*']
- module: system
  enabled: true
  period: 1m
  metricsets:
    - filesystem
    - fsstat
- module: kibana
  metricsets: ["status"]
  enabled: false
  period: 10s
  hosts: ["localhost:5601"]
- module: elasticsearch
  metricsets: ["node", "node_stats"]
  enabled: false
  period: 10s
  hosts: ["localhost:9200"]


output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: https
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

setup.dashboards.snapshot: true

setup.kibana:
  protocol: https
  host: localhost:5601
  username: "beats_internal"
  password: "changeme"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/ca.crt"]
  ssl.certificate: /etc/metricbeat/elasticsearch.crt
  ssl.key: /etc/metricbeat/elasticsearch.key

xpack.monitoring:
  enabled: false
  elasticsearch:
    url: "https://localhost:9200"
    username: elastic
    password: changeit
http.enabled: true
http.port: 5068

I'll attached the debug log which shows the dashboards loading.

[LeeDr]

metricbeat.log

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[kvch]

It is not a change/bug in Beats behaviour. If you configure any options under setup.dashboards.*, Beats will automatically enable dashboard loading without the user explicitly setting setup.dashboards.enabled. This option group behaves similarly to option groups that have an enabled setting, to e.g. ssl.* or kerberos.*. Given that you added setup.dashboard.snapshot to the configuration, Metricbeat loads the dashboards on every run.

This might not be straightforward, but this is how Beats work. The problem is rather that we missing configuration for this option. For SSL settings we have a note that clearly lets users know about this behaviour: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-ssl.html#enabled We should add the same clarifications instead of breaking existing behaviour.

[joshdover]

I verified metricbeat 7.15 doesn't load saved objects unless you call the setup step. But 7.16 and 7.17 do.

I'm curious about this, is this true that this behavior wasn't present prior to 7.16?

[kvch]

I verified metricbeat 7.15 doesn't load saved objects unless you call the setup step. But 7.16 and 7.17 do.

I'm curious about this, is this true that this behavior wasn't present prior to 7.16?

No, it is not true. We haven't changed anything in enabling/disabling dashboard loading since 2017. Also, Beats adopted the Saved Objects API in 7.15, so it has nothing to do with the migration. There were no changes to dashboard loading between 7.15 and 7.16/7.17 in Beats.

To see it for myself, I have tested the attached configuration with Metricbeat 7.15 and it loads the dashboard when I run ./metricbeat. Exactly as Metricbeat 7.17. When I removed setup.dashboards.snapshot from the configuration the two versions behaved the same, and the dashboards were not loaded.

[kvch]

Also, what is setup.dashboards.snapshot is supposed to do? I cannot see it anywhere in our codebase or documentation.

[LeeDr]

my understanding (which could have been wrong from the beginning) was that when we run the setup command, and we are using beats from a snapshot build, that parameter was necessary. We've been setting it only when we test snapshot builds, not BCs.

[kvch]

I have found it! The setting was removed in #4675. Setting the option is no longer needed/possible.

[LeeDr]

Thanks @kvch. I also have an update/correction. The reason I thought 7.15 was different from 7.16+ is that we're not building snapshots of 7.15 any longer so I didn't install a snapshot build, which also means our scripts didn't set setup.dashboards.snapshot.

We've been using setup.dashboards.snapshot since 6.0 release. All this time we were thinking it had this impact;

# If this option is enabled, the snapshot URL is used instead of the default URL.
#setup.dashboards.snapshot: false

But since that setting was removed, it seems like maybe setup.dashboards.snapshot: true is being interpreted as setup.dashboards: true ? Because if I take it out, I don't get the dashboards automatically installed.

NOTE: Beats doesn't seem to WARN or ERROR on having this invalid config parameter present. This is different from Elasticsearch and Kibana which would both treat it as FATAL.