LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #17342

premendrasingh · 2020-03-30T21:06:20Z

What does this PR do?

This PR is to add a replace processor. This processor takes in a field name, search string and replacement string. Searches field value for pattern and replaces it with replacement string.

Why is it important?

This PR will help remove extra strings or add additional string to values

How to test this PR locally

Added unit test cases.

Use cases

While using auditbeat we get full path to file inside the pod on Kubernetes
"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/${data.kubernetes.container.id}/rootfs/etc/runit/runsvdir/default/mcelog/supervise/pid.new"
This PR helps trim the beginning part of the string to get
/etc/runit/runsvdir/default/mcelog/supervise/pid.new"

Using config below

      processors:
        - replace:
            fields:
            - field: "file.path"
              pattern: "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/${data.kubernetes.container.id}/rootfs/"
              replacement: "/"

elasticmachine · 2020-03-30T21:06:22Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-03-30T21:06:22Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-03-31T10:10:13Z

Pinging @elastic/integrations-services (Team:Services)

exekias · 2020-04-01T08:20:34Z

This seems very similar to https://www.elastic.co/guide/en/elasticsearch/reference/master/gsub-processor.html, could you please use the same signature? I think having field, pattern and replacement would be enough

…eplace_string

vjsamuel · 2020-04-02T05:34:54Z

libbeat/processors/actions/replace.go

-	ReplaceWith string `config:"replace_with"`
+type replaceConfig struct {
+	Field   string `config:"field"`
+	Pattern string `config:"pattern"`


@urso @exekias should https://github.com/elastic/beats/blob/master/libbeat/common/match/matchers.go be enhanced to do replacing strings and just use matcher.Match here?

using plain old regex could be slower as compared to the optimized one in libbeat.

for Pattern in the config use Patter *regexp.Regexp. Config unpacking will automatically try to compile the regex and fail with error + setting name if this has failed.

matcher.Match is an optimization for some custom cases only, but falls back to regexp if the case becomes more 'complicated'. Given the optimizations we have in matcher, I only see the case for a constant string match being helpful (which would become a sub-string search, or in some cases string-prefix/suffix comparison).

The matcher package also replaces capturing-group-matches with non-capturing-groups (greatly reduces allocations). Having patterns and replacement like gsub, do we want to allow users to use capturing group in the replacement in the future? E.g.

pattern: 'some (?P<important>[a-zA-Z]) string' replace: 'found: {{important}}'

For now I would not enhance the matcher package. Only if we figure this is indeed a common problem. When doing so we might have to remove some of the optimizations. In case we find we really need to optimize another type (e.g. matcher.Replacer) might give us better flexibility in applying the kind of optimizations we need for the use-case, while not un-optimizing matcher.Matcher.

@urso Changed to pattern, as suggested.

…eplace_string

urso · 2020-04-16T13:33:39Z

The implementation itself LGTM.

Please fix the code format. The intake CI job fails already. See: https://travis-ci.org/github/elastic/beats/jobs/670699207

Please add some reference documentation as well. See docs sub-directory for examples.

Thank you!

…eplace_string

…beats into processor_replace_string

…eplace_string

urso · 2020-04-21T11:46:55Z

libbeat/processors/actions/docs/replace.asciidoc

+    fields:
+     - field: "file.path"
+       pattern: "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/${data.kubernetes.container.id}/rootfs/"
+       replacement: "/"


In the implementation patterns can not reference events contents. A more correct regexp would be /run/containerd/io.containerd.runtime.v1.linux/k8s.io/.+/rootfs/.

Maybe we can have a simpler example?

@urso I have updated the documentation to a simpler example. Can you please help fix the CI issue? It keeps failing with this error.

Error: copy failed: cannot stat source file ../../vendor/github.com/elastic/beats/libbeat/common/file: stat ../../vendor/github.com/elastic/beats/libbeat/common/file: no such file or directory

Thanks

…eplace_string

urso · 2020-04-22T12:14:05Z

Do not import github.com/elastic/beats/libbeat directly. Beats has switched to use go modules. When using the import like this the go build tools try to fetch libbeat from github, which is not the version/branch you are developing against. Replace the imports to say github.com/elastic/beats/v7/libbeat. Also check the go.mod and go.sum files. If they mention github.com/elastic/beats you will need to clean this up as well.

…eplace_string

urso · 2020-04-23T10:16:18Z

jenkins run the tests please

urso · 2020-04-24T13:10:38Z

jenkins run the tests please

…eplace_string

urso · 2020-04-27T12:52:22Z

jenkins run the tests please

urso · 2020-04-27T12:52:50Z

Appropriate tests have been run through on Travis. I triggered a (hopefully) final test run on Jenkins.

…alues of fields. (elastic#17342) This PR is to add a replace processor. This processor takes in a field name, search string and replacement string. Searches field value for pattern and replaces it with replacement string. (cherry picked from commit 09fd4df)

…unbld * upstream/master: ci: comment PRs with the build status (elastic#17971) Add domain state metricset to kvm module (elastic#17673) [Agent] Allow CLI paths override (elastic#17781) Fix generated metricbeat so create-metricset works. (elastic#18020) LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. (elastic#17342) Update stale references to _xpack to refer to _license instead (elastic#18030) Review dependency patterns collection in Jenkins (elastic#18004)

premendrasingh · 2020-04-28T18:11:17Z

Thanks a lot @urso

…sor for replacing strings values of fields. (#18047)

urso · 2020-04-29T12:24:26Z

Changes have been backported and will be released in 7.8.0. Thank you for contributing.

Add processor for replacing strings values

0eaa435

Update CHANGELOG.next.asciidoc

121b558

ChrsMark added review Team:Services (Deprecated) Label for the former Integrations-Services team labels Mar 31, 2020

exekias added the Team:Integrations Label for the Integrations team label Apr 1, 2020

exekias added :Processors enhancement labels Apr 1, 2020

premendrasingh added 2 commits April 1, 2020 11:19

Changed signature according to review comments.

0ccce99

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

52b4d6f

…eplace_string

vjsamuel reviewed Apr 2, 2020

View reviewed changes

premendrasingh added 3 commits April 2, 2020 14:20

Changed pattern to *regexp.Regexp as suggested.

e2ba62a

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

cf2608a

…eplace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

f80ec20

…eplace_string

urso self-assigned this Apr 7, 2020

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Apr 14, 2020

premendrasingh added 8 commits April 16, 2020 16:18

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

9ed18dd

…eplace_string

Fix formatting errors. Add doc for replace processor

1256e55

Fix documentation for replace processor

a396379

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

147c67c

…eplace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

0c0c44f

…eplace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

4d9f433

…eplace_string

Merge branch 'processor_replace_string' of github.com:premendrasingh/…

277978a

…beats into processor_replace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

851f33a

…eplace_string

premendrasingh added 2 commits April 20, 2020 11:20

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

2bb9cbc

…eplace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

dcd2244

…eplace_string

urso reviewed Apr 21, 2020

View reviewed changes

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

b4f7d2c

…eplace_string

dedemorton mentioned this pull request Apr 21, 2020

Backlog of content that needs an edit #17163

Closed

5 tasks

premendrasingh added 2 commits April 21, 2020 13:37

Changed to a simpler example. Thanks for the suggestion

2561c12

Fix sentence

6bc80cb

premendrasingh added 2 commits April 22, 2020 06:08

Update imports to use github.com/elastic/beats/v7/libbeat

889c0cf

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

2acc815

…eplace_string

Merge remote-tracking branch 'beats_upstream/master' into processor_r…

4a3427d

…eplace_string

urso added the needs_backport PR is waiting to be backported to other branches. label Apr 28, 2020

urso merged commit 09fd4df into elastic:master Apr 28, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 28, 2020

urso mentioned this pull request Apr 28, 2020

Cherry-pick #17342 to 7.x: LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #18047

Merged

urso added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels Apr 28, 2020

urso pushed a commit that referenced this pull request Apr 29, 2020

Cherry-pick #17342 to 7.x: LIBBEAT: Enhancement replace_string proces…

b669b5d

…sor for replacing strings values of fields. (#18047)

urso added the test-plan Add this PR to be manual test plan label Apr 29, 2020

andresrc added test-plan-added This PR has been added to the test plan and removed [zube]: Done labels May 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #17342

LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #17342

premendrasingh commented Mar 30, 2020 •

edited

Loading

elasticmachine commented Mar 30, 2020

elasticmachine commented Mar 30, 2020

elasticmachine commented Mar 31, 2020

exekias commented Apr 1, 2020

vjsamuel Apr 2, 2020 •

edited

Loading

urso Apr 2, 2020

premendrasingh Apr 7, 2020

urso commented Apr 16, 2020

urso Apr 21, 2020

premendrasingh Apr 22, 2020

urso commented Apr 22, 2020

urso commented Apr 23, 2020

urso commented Apr 24, 2020

urso commented Apr 27, 2020

urso commented Apr 27, 2020

premendrasingh commented Apr 28, 2020

urso commented Apr 29, 2020

LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #17342

LIBBEAT: Enhancement replace_string processor for replacing strings values of fields. #17342

Conversation

premendrasingh commented Mar 30, 2020 • edited Loading

What does this PR do?

Why is it important?

How to test this PR locally

Use cases

elasticmachine commented Mar 30, 2020

elasticmachine commented Mar 30, 2020

elasticmachine commented Mar 31, 2020

exekias commented Apr 1, 2020

vjsamuel Apr 2, 2020 • edited Loading

Choose a reason for hiding this comment

urso Apr 2, 2020

Choose a reason for hiding this comment

premendrasingh Apr 7, 2020

Choose a reason for hiding this comment

urso commented Apr 16, 2020

urso Apr 21, 2020

Choose a reason for hiding this comment

premendrasingh Apr 22, 2020

Choose a reason for hiding this comment

urso commented Apr 22, 2020

urso commented Apr 23, 2020

urso commented Apr 24, 2020

urso commented Apr 27, 2020

urso commented Apr 27, 2020

premendrasingh commented Apr 28, 2020

urso commented Apr 29, 2020

premendrasingh commented Mar 30, 2020 •

edited

Loading

vjsamuel Apr 2, 2020 •

edited

Loading