Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Change Proposal] Add support for ingest pipelines in transforms #833

Open
jsoriano opened this issue Nov 19, 2024 · 0 comments
Open

[Change Proposal] Add support for ingest pipelines in transforms #833

jsoriano opened this issue Nov 19, 2024 · 0 comments
Labels
discuss Issue needs discussion

Comments

@jsoriano
Copy link
Member

jsoriano commented Nov 19, 2024

Transforms allow to define ingest pipelines, so documents are processed before ingestion. This allows for example to remove fields that are not relevant in the transform, or to adjust fields intended to have different values, as in elastic/elastic-package#2218 (comment).

This is partly supported now, a destination index can have a pipeline, but currently it needs to be hard-coded:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: "1.23.0-latest_ioc"

We should allow to use templates there as we allow in other places, so configuration can be something like this:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: '{{ IngestPipeline "latest_ioc" }}'

This may not need changes in the spec but will need changes in Fleet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss Issue needs discussion
Projects
None yet
Development

No branches or pull requests

1 participant