Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

Closed
andrewkroh opened this issue Jun 30, 2021 · 13 comments
Closed

[Filebeat] Module incompatibility with older ES/Kibana versions #26629

andrewkroh opened this issue Jun 30, 2021 · 13 comments
Labels
discuss Issue needs further discussion. Filebeat Filebeat Stalled Team:Elastic-Agent Label for the Agent team

Comments

@andrewkroh
Copy link
Member

andrewkroh commented Jun 30, 2021

The Elastic support matrix indicates that the latest Filebeat 7.x version works with all 7.x versions of Elasticsearch. This is an assumption I'm making based on the table pictured below. There is a "Compatibility with Beats" table but it does not include Elasticsearch or Kibana columns.

Screen Shot 2021-08-10 at 8 50 28 AM

This is not true for all Filebeat modules because there are specific Elasticsearch features utilized in module ingest node pipelines that require newer ES versions. For example:

  • The set processors copy_from feature. Used to copy objects.
  • The convert processor's type: ip. Used to ensure strings are valid IPs for ip mapping fields.
  • registered_domain is new in 7.13.
  • network_direction is new in 7.12.

There are several other features where Filebeat automatically rewrites the ingest node pipeline to create a compatible pipeline for the current ES version (possibly with a reduced feature set).

  • The append processors allow_duplicates.
  • community_id is new in 7.12
  • The set processor's ignore_empty.
  • uri_parts is new in 7.12
  • user_agent requires ecs: true in ES [6.7, 7.0).

How do we want to handle modules that require specific Elasticsearch versions and how do we indicate this to users in the support matrix?

  • It might just be to say a particular module requires ES 7.N, documented it, add a runtime check, and then put a footnote in the support matrix that says certain Filebeat modules have additional version constraints.
  • Or indicate that Filebeat requires a specific ES version like Filebeat 7.13 requires at least ES 7.12 (FB works with previous ES minor). This would mean that modules can be developed using more recent ES features and it's easy to test with one previous ES version.
  • Manually fix every module pipeline that uses new features like copy_from or convert with type: ip. With this we never get to take advantage of new features in ES.
  • Try to automatically rewrite pipelines for compatibility. This solution is hard to maintain and test for every edge case. And some configs cannot be automatically rewritten. I attempted this in [Filebeat] Make set processor with copy_from compatible with ES < 7.13 #26593.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 30, 2021
@andrewkroh andrewkroh added Filebeat Filebeat discuss Issue needs further discussion. labels Jun 30, 2021
andrewkroh added a commit to andrewkroh/beats that referenced this issue Jun 30, 2021
To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates elastic#26629
@andrewkroh
Copy link
Member Author

While we wait to make a decision I opened #26631 to get a quick fix for copy_from. I didn't find any usages of convert with type: ip.

@andrewkroh
Copy link
Member Author

@adriansr added automatic rewriting of pipelines for these cases in #26676.

  • Replaces usages of convert processor using type: ip with an equivalent grok expression.
  • Removes the network_direction processor.

andrewkroh added a commit that referenced this issue Jul 6, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
mergify bot pushed a commit that referenced this issue Jul 6, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
(cherry picked from commit a7b0110)

# Conflicts:
#	x-pack/filebeat/module/panw/panos/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/abuseurl/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomali/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/anomalithreatstream/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/misp/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/otx/ingest/pipeline.yml
#	x-pack/filebeat/module/threatintel/recordedfuture/ingest/pipeline.yml
#	x-pack/filebeat/module/zoom/webhook/ingest/meeting.yml
mergify bot pushed a commit that referenced this issue Jul 6, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
(cherry picked from commit a7b0110)
@andrewkroh
Copy link
Member Author

@adriansr added module testing using ES 7.11 so that we can catch problems with future module changes. See #26737.

andrewkroh added a commit that referenced this issue Jul 7, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
(cherry picked from commit a7b0110)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
mergify bot pushed a commit that referenced this issue Jul 19, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
(cherry picked from commit a7b0110)
andrewkroh added a commit that referenced this issue Jul 20, 2021
* Replace copy_from with templated value

To ensure compatibility with Elasticsearch versions <7.13 this removes usage of `copy_from` in `set` processors.

Relates #26629

* panw: replace copy_from usage with script

Replaces the usage of a set processor with copy_from (ES 7.13+)
with a painless script that performs the same operation and it's
backwards compatible.

* cyberarkpas: Replace usage of copy_from with script

This updates the ID-mapping script to set fields instead of constructing
and op-list that is latter processed with foreach/set.

* Update CHANGELOG.next.asciidoc

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
(cherry picked from commit a7b0110)

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
@P1llus
Copy link
Member

P1llus commented Aug 9, 2021

Adding the fact that when a module (or package/integration) has a dashboard, this dashboard can only be installed on a Kibana that runs minimum the same version as the Kibana used to export the included dashboard.

That means if a dashboard has a hotfix, then suddenly that dashboard will no longer be available to any older versions of kibana (even though it might have worked before).

Personally I feel we should be more strict with the versions supported, we should not be more than one minor version behind the rest of the stack, and should commonly not be newer than the rest of the stack either.

There should also be some sort of Asterisk next to the versions, maybe mentioning that beat modules might require a higher minimum level for all functionality to work?

@andrewkroh andrewkroh changed the title [Filebeat] Module incompatibility with older ES versions [Filebeat] Module incompatibility with older ES/Kibana versions Aug 10, 2021
@andrewkroh andrewkroh added the Team:Elastic-Agent Label for the Agent team label Aug 10, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@kvch
Copy link
Contributor

kvch commented Aug 10, 2021

There should also be some sort of Asterisk next to the versions, maybe mentioning that beat modules might require a higher minimum level for all functionality to work?

The support matrix has an asterisk that has a following clarification:

We recommend running the latest version of Beats, Logstash, Elastic Agent and ES-Hadoop; earlier versions will work with reduced functionality.

@andrewkroh
Copy link
Member Author

@nimarezainia Could you help by confirming that my interpretation of the support matrix is correct? Is the current requirement that Beats 7.14 be compatible with ES/Kibana 7.0? Or have I misinterpreted it.

@ruflin
Copy link
Contributor

ruflin commented Aug 16, 2021

@nimarezainia @mostlyjason @sorantis I know you start to think about this also in the context of packages.

My personal take: There are 2 layers of compatibility:

  • Elasticsearch output
  • Modules

Elasticsearch output stays compatible because Elasticsearch is not expected to make breaking changes in minor. For modules we should require the same version of the stack or newer.

@andrewkroh
Copy link
Member Author

andrewkroh commented Aug 16, 2021

The merging of #27398 will mean that Beats 7.15+ require at least Kibana 7.15 to import dashboards and Kibana index patterns. The release notes will say "Beats can only import and export dasbhboards using at least Kibana 7.15."

The support matrix needs to be updating to reflect this when 7.15 is released.

@nimarezainia
Copy link
Contributor

@andrewkroh this is complex as you have outlined above with many components that have their own support considerations.

to begin with, these matrices need to be pruned and show only the beats versions that are currently under support (i.e > 6.8.x)

then I propose that we only support two versions (current and one previous) as also mentioned above. If you look at the Endgame table that is a good example.

For beats we also have this footnote: "It's highly recommended to run the same minor versions across Elasticsearch, Kibana, and Beats for best monitoring functionality."

Allow me to work with @andresrc on these tables and update them and clarify accordingly.

@andrewkroh
Copy link
Member Author

then I propose that we only support two versions (current and one previous) as also mentioned above.

+1

Another issue the broad compatibility affects is the ability to upgrade to ECS 1.11 which uses the flattened data type added in 7.3. #27107 (comment)

adriansr added a commit to adriansr/beats that referenced this issue Oct 6, 2021
This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates elastic#26629
adriansr added a commit that referenced this issue Oct 7, 2021
…nor instead of 7.11 (#28274)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates #26629
mergify bot pushed a commit that referenced this issue Oct 7, 2021
…nor instead of 7.11 (#28274)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates #26629

(cherry picked from commit 9806f91)
mergify bot pushed a commit that referenced this issue Oct 7, 2021
…nor instead of 7.11 (#28274)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates #26629

(cherry picked from commit 9806f91)
mergify bot pushed a commit that referenced this issue Oct 7, 2021
…nor instead of 7.11 (#28274)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates #26629

(cherry picked from commit 9806f91)

# Conflicts:
#	filebeat/Jenkinsfile.yml
#	testing/environments/prev-minor.yml
#	x-pack/filebeat/Jenkinsfile.yml
adriansr added a commit that referenced this issue Oct 8, 2021
…nor instead of 7.11 (#28274) (#28307)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates #26629

(cherry picked from commit 9806f91)

Co-authored-by: Adrian Serrano <adrisr83@gmail.com>
Icedroid pushed a commit to Icedroid/beats that referenced this issue Nov 1, 2021
…nor instead of 7.11 (elastic#28274)

This changes the compatibility test for Filebeat modules to test with
the previous released minor.

Relates elastic#26629
ruflin added a commit to ruflin/beats that referenced this issue Mar 21, 2022
In elastic#26629 the issue around running Filebeat against older version of Elasticsearch was discussed and in elastic#28274 testing against the previous minor was introduced. But since 8.0, Filebeat can only ship data to equal or newer versions of Elasticsearch. Because of this, in the tests `TESTING_FILEBEAT_ALLOW_OLDER=1` had to be introduced.

I'm opening this PR to remove this testing as it is not something we support anymore.
ruflin added a commit that referenced this issue Mar 24, 2022
In #26629 the issue around running Filebeat against older version of Elasticsearch was discussed and in #28274 testing against the previous minor was introduced. But since 8.0, Filebeat can only ship data to equal or newer versions of Elasticsearch. Because of this, in the tests `TESTING_FILEBEAT_ALLOW_OLDER=1` had to be introduced.

I'm opening this PR to remove this testing as it is not something we support anymore.
kush-elastic pushed a commit to kush-elastic/beats that referenced this issue May 2, 2022
In elastic#26629 the issue around running Filebeat against older version of Elasticsearch was discussed and in elastic#28274 testing against the previous minor was introduced. But since 8.0, Filebeat can only ship data to equal or newer versions of Elasticsearch. Because of this, in the tests `TESTING_FILEBEAT_ALLOW_OLDER=1` had to be introduced.

I'm opening this PR to remove this testing as it is not something we support anymore.
@botelastic
Copy link

botelastic bot commented Sep 2, 2022

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Sep 2, 2022
@botelastic botelastic bot closed this as completed Mar 1, 2023
chrisberkhout pushed a commit that referenced this issue Jun 1, 2023
In #26629 the issue around running Filebeat against older version of Elasticsearch was discussed and in #28274 testing against the previous minor was introduced. But since 8.0, Filebeat can only ship data to equal or newer versions of Elasticsearch. Because of this, in the tests `TESTING_FILEBEAT_ALLOW_OLDER=1` had to be introduced.

I'm opening this PR to remove this testing as it is not something we support anymore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Issue needs further discussion. Filebeat Filebeat Stalled Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

No branches or pull requests

6 participants