Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow_duplicates option for append processor #61916

Merged
merged 1 commit into from
Sep 11, 2020

Conversation

danhermann
Copy link
Contributor

Adds an allow_duplicates option to the append processor. When set to false, any of the values to be appended that are already present in the target field will be ignored. Defaults to true to preserve existing behavior.

Closes #57543

@danhermann danhermann added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v8.0.0 v7.10.0 labels Sep 3, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Sep 3, 2020
@danhermann
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/packaging-sample-windows

1 similar comment
@danhermann
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/packaging-sample-windows

@probakowski probakowski self-requested a review September 3, 2020 15:29
Copy link
Contributor

@probakowski probakowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @danhermann!
I left one super minor nit, you can skip it if you want

Comment on lines +206 to +207
List.of(new UppercaseProcessor("_tag_upper", null, "_ingest._value", false, "_ingest._value")),
List.of(new AppendProcessor("_tag", null, template, (model) -> (Collections.singletonList("added")), true))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd stick with Collections.singletonList just for the sake of backporting

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Java8-compatible implementations of List, Set, and Map in the collect package make these backports pretty easy.

@danhermann
Copy link
Contributor Author

Thanks for the review, @probakowski!

@danhermann danhermann merged commit 2d8b7a9 into elastic:master Sep 11, 2020
@danhermann danhermann deleted the 57543_append_skip_duplicates branch September 11, 2020 13:18
danhermann added a commit to danhermann/elasticsearch that referenced this pull request Sep 14, 2020
danhermann added a commit to danhermann/elasticsearch that referenced this pull request Oct 5, 2020
stevejgordon added a commit to elastic/elasticsearch-net that referenced this pull request Dec 22, 2020
stevejgordon added a commit to elastic/elasticsearch-net that referenced this pull request Jan 7, 2021
* Support allow_duplicates on append processor

Related to elastic/elasticsearch#61916
github-actions bot pushed a commit to elastic/elasticsearch-net that referenced this pull request Jan 7, 2021
* Support allow_duplicates on append processor

Related to elastic/elasticsearch#61916
stevejgordon added a commit to elastic/elasticsearch-net that referenced this pull request Jan 7, 2021
* Support allow_duplicates on append processor

Related to elastic/elasticsearch#61916

(cherry picked from commit 2b75c5e)
stevejgordon added a commit to elastic/elasticsearch-net that referenced this pull request Jan 7, 2021
* Support allow_duplicates on append processor

Related to elastic/elasticsearch#61916

Co-authored-by: Steve Gordon <sgordon@hotmail.co.uk>
stevejgordon added a commit to elastic/elasticsearch-net that referenced this pull request Jan 7, 2021
* Support allow_duplicates on append processor

Related to elastic/elasticsearch#61916

(cherry picked from commit 2b75c5e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team v7.10.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Ingest Pipelines][Append Processor] Adding a "unique" argument
4 participants