Unable to send monitoring data due to 400 error from Elasticsearch bulk API

[ycombinator]

Version: Filebeat 7.8.0, 8.0.0

Minimal Filebeat configuration:

filebeat.inputs:
- type: stdin

output.elasticsearch:
  hosts: [ "localhost:9200" ]

monitoring.enabled: true

When running this against any Elasticsearch 7.x or 8.0.0, Filebeat logs show the following errors:

2020-05-27T13:48:19.198-0400    ERROR   [publisher_pipeline_output]     pipeline/output.go:181  failed to publish events: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/.monitoring-beats-7-2020.05.27/_bulk] contains unrecognized parameters: [system_api_version], [system_id]"}],"type":"illegal_argument_exception","reason":"request [/.monitoring-beats-7-2020.05.27/_bulk] contains unrecognized parameters: [system_api_version], [system_id]"},"status":400}

Note that this error is NOT observed with Filebeat 7.7.0.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[ycombinator]

I've narrowed down the commit that exposed this bug: 0575e15. That commit itself fixed another bug so we have to keep it. We just need to fix forward now.