Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Automatic Import] Handle the fields generated by grok processor before handling rename processor #194205

Closed
bhapas opened this issue Sep 26, 2024 · 1 comment · Fixed by #194727
Closed

[Automatic Import] Handle the fields generated by grok processor before handling rename processor #194205

bhapas opened this issue Sep 26, 2024 · 1 comment · Fixed by #194727
Assignees
@bhapas
Labels
bug Fixes for quality problems that affect the customer experience Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team

Comments

@bhapas
Copy link
Contributor

bhapas commented Sep 26, 2024 •
edited
Loading

Background

#192817 and #191749 have introduced support for unstructured and structured syslog samples in Automatic Import.

Current Behaviour

While fixing #194203 , it has been identified that the field names produced by grok processor does not have packageName.datastreamName as a prefix to them.

Like

  - grok:
      tag: grok_header_pattern
      field: message
      patterns:
        - >-
          <%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp}
          %{HOSTNAME:hostname} %{WORD:program}: %{GREEDYDATA:message}

And then it will be followed by few rename processors that rename the field to an ecs field and it looks like

- rename:
      ignore_missing: true
      field: integ.ds.hostname
      target_field: host.hostname
  - rename:
      ignore_missing: true
      field: integ.ds.program
      target_field: process.name

Issue

Since grok processor does not produce the fields with integ.ds prefix rename processor does not rename them to host.hostname or process.name , thereby leaving the final document with non ecs compatible field mapping

Image

Workaround

Currently the pipeline generated by Automatic Import can be edited to map to right ecs fields and that works , along with #194203.
@bhapas bhapas added bug Fixes for quality problems that affect the customer experience Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team labels Sep 26, 2024
@bhapas bhapas self-assigned this Sep 26, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-scalability (Team:Security-Scalability)

@bhapas bhapas mentioned this issue Oct 2, 2024
Merged
1 task
@bhapas bhapas closed this as completed in b38941b Oct 3, 2024
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 3, 2024
@bhapas
[Automatic Import] Fix Non-ecs compatible fields in grok processor (e…
aafe771 
…lastic#194727)

## Release Note

Fixes a bug to resolve non-ecs compatible fields in Structured /
Unstructured syslog processing in Automatic Import.

## Summary

elastic#194205 explains the issue.

This PR fixes `packageName.dataStreamName` for handling header values
from grok processor for KV graph so that ecs mapping gets the header
values in the converted json Samples too..

### Before this PR

![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)

### After this PR

<img width="706" alt="image"
src="https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901">

- Closes elastic#194205

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit b38941b)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 3, 2024
@bhapas
[Automatic Import] Fix Non-ecs compatible fields in grok processor (e…
8caa836 
…lastic#194727)

## Release Note

Fixes a bug to resolve non-ecs compatible fields in Structured /
Unstructured syslog processing in Automatic Import.

## Summary

elastic#194205 explains the issue.

This PR fixes `packageName.dataStreamName` for handling header values
from grok processor for KV graph so that ecs mapping gets the header
values in the converted json Samples too..

### Before this PR

![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)

### After this PR

<img width="706" alt="image"
src="https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901">

- Closes elastic#194205

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit b38941b)
kibanamachine added a commit that referenced this issue Oct 3, 2024
@kibanamachine @bhapas
[8.15] [Automatic Import] Fix Non-ecs compatible fields in grok proce…
b659922 
…ssor (#194727) (#194792)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Automatic Import] Fix Non-ecs compatible fields in grok processor
(#194727)](#194727)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-10-03T09:14:39Z","message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Fix Non-ecs compatible fields in grok
processor","number":194727,"url":"https://github.com/elastic/kibana/pull/194727","mergeCommit":{"message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194727","number":194727,"mergeCommit":{"message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
kibanamachine added a commit that referenced this issue Oct 3, 2024
@kibanamachine @bhapas
[8.x] [Automatic Import] Fix Non-ecs compatible fields in grok proces…
ad7c3eb 
…sor (#194727) (#194793)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Automatic Import] Fix Non-ecs compatible fields in grok processor
(#194727)](#194727)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Bharat
Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-10-03T09:14:39Z","message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic
Import] Fix Non-ecs compatible fields in grok
processor","number":194727,"url":"https://github.com/elastic/kibana/pull/194727","mergeCommit":{"message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194727","number":194727,"mergeCommit":{"message":"[Automatic
Import] Fix Non-ecs compatible fields in grok processor (#194727)\n\n##
Release Note\r\n\r\nFixes a bug to resolve non-ecs compatible fields in
Structured /\r\nUnstructured syslog processing in Automatic
Import.\r\n\r\n##
Summary\r\n\r\nhttps://github.com//issues/194205 explains
the issue. \r\n\r\nThis PR fixes `packageName.dataStreamName` for
handling header values\r\nfrom grok processor for KV graph so that ecs
mapping gets the header\r\nvalues in the converted json Samples
too..\r\n\r\n### Before this
PR\r\n\r\n![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)\r\n\r\n\r\n###
After this PR\r\n\r\n<img width=\"706\"
alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901\">\r\n\r\n\r\n-
Closes https://github.com/elastic/kibana/issues/194205\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b38941be7a253c80d426a49af806575ba15652a5"}}]}]
BACKPORT-->

Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
tiansivive pushed a commit to tiansivive/kibana that referenced this issue Oct 7, 2024
@bhapas @tiansivive
[Automatic Import] Fix Non-ecs compatible fields in grok processor (e…
ed599ab 
…lastic#194727)

## Release Note

Fixes a bug to resolve non-ecs compatible fields in Structured /
Unstructured syslog processing in Automatic Import.

## Summary

elastic#194205 explains the issue. 

This PR fixes `packageName.dataStreamName` for handling header values
from grok processor for KV graph so that ecs mapping gets the header
values in the converted json Samples too..

### Before this PR

![image](https://github.com/user-attachments/assets/d2660f7d-2cca-413c-ab90-1a0f3e1b4a03)


### After this PR

<img width="706" alt="image"
src="https://github.com/user-attachments/assets/954b5a91-2123-46f9-b822-1709c3247901">


- Closes elastic#194205

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bhapas bhapas

Labels
bug Fixes for quality problems that affect the customer experience Feature:AutomaticImport Team:Security-Scalability Team label for Security Integrations Scalability Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Automatic Import] Fix Non-ecs compatible fields in grok processor bhapas/kibana
2 participants
@elasticmachine @bhapas