Fix default_field for metricbeat

[ruflin]

In Elasticsearch 7.0 there is a limit of 1024 fields (see #5275). As Metricbeat exceeds the limit of 1024 fields per index it has to be defined which the default fields are as otherwise Elasticsearch returns an error.

This change set the index config option index.query.default_field. To fields chosen to be searchable are all string and keyword values as these are the ones that I would expect users to put directly in the search without specifying a field. For all numbers I expect that the exact field is specified.

This PR also removes all presets from dashboards which did set default_fields as they were *. In Kibana the default * was removed also in master: elastic/kibana#16232

[ruflin]

@Bargs Few questions to default_field in Kibana related to elastic/kibana#16232

• What are the priorites of default_field in Kibana. If a visualization has it, will it overwrite the global setting?
• And will the global setting overwrite the Elasticsearch setting?
• Is there a way to overwrite the global default_field setting per index pattern?

[Bargs]

What are the priorites of default_field in Kibana. If a visualization has it, will it overwrite the global setting?

I believe the global option will take precedence for the query from the query bar. However, in 7.0 query_string filters will no longer get the default options, so one option might be to add a query_string filter to the vis instead of using the query bar.

And will the global setting overwrite the Elasticsearch setting?

Yes, it is sent as a param on the query_string query in the search request body, so it takes precedence over the index settings in ES.

Is there a way to overwrite the global default_field setting per index pattern?

Nope

[ruflin]

@Bargs Thanks for the details. Do you have a time frame when elastic/kibana#16232 could happen? Because I think it is also prerequisit for this PR to fully fix the problem without having to overwrite it in each visualisation.

[Bargs]

It's easy to do, I've just been putting it off. If it's blocking you guys I'll try to get it done in the next few days.

[Bargs]

Here we go elastic/kibana#18966

[ruflin]

@Bargs Thanks a lot for this. This will make it much easier for us / me to test if my above solution to the problem actually works.

[ruflin]

I tested this again with the most recent snapshot builds from Kibana and for the system dashboards this seems to fix the problem.

An other thing I stumbled over partially related is that the searchSourceJSON is always a bit different for the visualisations. I don't think it depends on the order how the visualisations were built. The outcome is normally the same. A few example:

beats/metricbeat/module/system/_meta/kibana/6/dashboard/Metricbeat-system-overview.json

Line 7 in 726fb00

"searchSourceJSON": "{\"query\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"language\":\"lucene\"},\"filter\":[]}"

beats/metricbeat/module/system/_meta/kibana/6/dashboard/Metricbeat-system-overview.json

Line 22 in 726fb00

"searchSourceJSON": "{\"index\":\"metricbeat-*\",\"query\":{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true,\"default_field\":\"*\"}},\"language\":\"lucene\"},\"filter\":[]}"

beats/metricbeat/module/system/_meta/kibana/6/dashboard/Metricbeat-system-overview.json

Line 52 in 726fb00

"searchSourceJSON": "{\"query\":{\"query\":{\"query_string\":{\"query\":\"*\"}},\"language\":\"lucene\"},\"filter\":[]}"

All similar but not 100% identical. Should we standardise these after the export to make sure they are all the same?

[exekias]

💯 we had issues in the past with wrong values here, AFAIK index must be set to metricbeat-*, for instance

[ruflin]

I changed the PR that for now default_fields only apply when running against ES 7. This makes sure it's not a breaking change in case it has side effects. It also gives us time to test it.

@exekias Lets sync up separately on how we should "clean" our dashboards. Ready for review.

[exekias]

LGTM, should this have a changelog entry?

[ruflin]

@exekias Yes, added one and rebase on master.

[exekias]

WFG