[Filebeat] user_agent field errors with Filebeat 7 and Elasticsearch 6

[andrewkroh]

The Suricata modules's ingest pipeline fails due to changes (elastic/elasticsearch#37329) in the user_agent processor in ES ingest node. I suspect other modules are affected by this problem.

• {"type":"mapper_parsing_exception","reason":"object mapping for [user_agent.os] tried to parse field [os] as object, but found a concrete value"}
• {"type":"mapper_parsing_exception","reason":"object mapping for [user_agent.device] tried to parse field [device] as object, but found a concrete value"}

In order to make the ES 6.x user_agent processor's output compatible with the Filebeat template we need to add some renames/sets after the user_agent in the pipeline.

[ruflin]

Elasticsearch 6.6 / 6.7 has a config option ecs: enabled for the user_agent. Adding these would create the same data structure. The problem here is that this flag is deprecated in 7.x, meaning if we add it we keep deprecated log messages. It will also mean Filebeat 7.x is only compatible with Elasticsearch 6.6 / 6.7.

This is also related to: #10650

[andrewkroh]

In order to allow Filebeat 7.x remain compatible with Elasticsearch 6.x, Filebeat would need to dynamically inject ecs: true into the pipelines that it creates (for ES 6.x only). Otherwise the data produced by the user_agent processor will not be compatible with the mapping used in Filebeat 7.

[jsoriano]

In order to allow Filebeat 7.x remain compatible with Elasticsearch 6.x, Filebeat would need to dynamically inject ecs: true into the pipelines that it creates (for ES 6.x only). Otherwise the data produced by the user_agent processor will not be compatible with the mapping used in Filebeat 7.

@andrewkroh thanks for this suggestion, I am giving a try to this idea in #10875. I have seen that ecs option is not available in in Elasticsearch 6.6, so Filebeat 7 pipelines with user_agent will only be compatible with 6.7. As you mentioned this issue affects all modules using user_agent processor.