Indices stats: field "_all" is problematic

[lukas-vlcek]

The data returned from Indices stats API contains _all field in the root level (the top most one). This is causing troubles when indexing such data to ES. Such as the following exceptions:

MapperParsingException[Field [_all] is a metadata field and cannot be added inside a document. Use the index API request parameters.]

or

MapperParsingException[Cannot generate dynamic mappings of type [_all] for [_all]]

The _all field cannot be created in index mapping because _all is metafield name. Historically, ES was creating the _all field for every indexed document automatically. There does not seem to be a way how to get output from the Indices stats API without the _all field in the root.

Moreover, starting with ES 6.x the _all field is no longer supported (including related include_in_all mapping option).

Other relevant resources:

• Explore disabling the _all field by default: Explore disabling the _all field by default elastic/elasticsearch#19784
• _all and include_in_all mapping in ElasticSearch 6.0: https://discuss.elastic.co/t/-all-and-include-in-all-mapping-in-elasticsearch-6-0/80431
• Disable _all by default, disallow configuring _all on 6.0+ indices: Disable _all by default, disallow configuring _all on 6.0+ indices elastic/elasticsearch#22144

Possible solutions:

• May be we could use copy_to to copy content from _all field to other field before indexing and set "index": "no" for _all field. However, this still keeps _all field in the game and I am not sure if this is supported.

• Rename the field before it is sent to Elasticsearch for indexing.

[lukas-vlcek]

For now the watches-cli has workaround, see ViaQ/watches-cli@77dd829

[lukas-vlcek]

Closed by b349155