[Automatic Import] Add support for handling unstructured syslog samples #192817
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bhapas
added
release_note:enhancement
enhancement
|
Pinging @elastic/security-scalability (Team:Security-Scalability) |
bhapas
added
backport:prev-major
haetamoudi
reviewed
Sep 16, 2024
x-pack/plugins/integration_assistant/server/graphs/unstructured/unstructured.ts
Show resolved
Hide resolved
haetamoudi
reviewed
Sep 16, 2024
x-pack/plugins/integration_assistant/server/graphs/unstructured/validate.ts
Show resolved
Hide resolved
haetamoudi
approved these changes
Sep 16, 2024
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @bhapas |
kgeller
approved these changes
Sep 17, 2024
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 17, 2024
…es (elastic#192817) ## Summary This PR handles the `unstructured` syslog samples in Automatic Import. Examples of unstructured samples would be: ``` <34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8 <34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5 <34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3 ``` https://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 77fe423)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Sep 17, 2024
…es (elastic#192817) ## Summary This PR handles the `unstructured` syslog samples in Automatic Import. Examples of unstructured samples would be: ``` <34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8 <34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5 <34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3 ``` https://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 77fe423)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Sep 17, 2024
… samples (#192817) (#193159) # Backport This will backport the following commits from `main` to `8.x`: - [[Automatic Import] Add support for handling unstructured syslog samples (#192817)](#192817) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-17T12:28:01Z","message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","enhancement","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic Import] Add support for handling unstructured syslog samples","number":192817,"url":"https://github.com/elastic/kibana/pull/192817","mergeCommit":{"message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192817","number":192817,"mergeCommit":{"message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
kibanamachine
added a commit
that referenced
this pull request
Sep 17, 2024
…g samples (#192817) (#193158) # Backport This will backport the following commits from `main` to `8.15`: - [[Automatic Import] Add support for handling unstructured syslog samples (#192817)](#192817) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"123897612+bhapas@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-09-17T12:28:01Z","message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","enhancement","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Automatic Import] Add support for handling unstructured syslog samples","number":192817,"url":"https://github.com/elastic/kibana/pull/192817","mergeCommit":{"message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192817","number":192817,"mergeCommit":{"message":"[Automatic Import] Add support for handling unstructured syslog samples (#192817)\n\n## Summary\r\n\r\nThis PR handles the `unstructured` syslog samples in Automatic Import.\r\n\r\nExamples of unstructured samples would be:\r\n\r\n```\r\n<34>Oct 11 00:14:05 mymachine su: 'su root' failed for user on /dev/pts/8\r\n<34>Dec 11 00:14:43 yourmachine su: 'su root' failed for someone on /dev/pts/5\r\n<34>Apr 11 00:14:05 mymachine su: 'su root' failed for otheruser on /dev/pts/3\r\n```\r\n\r\n\r\nhttps://github.com/user-attachments/assets/d1381ac9-4889-42cf-b3c1-d1b7a88def02\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n### For maintainers\r\n\r\n- [ ] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"77fe423f7b621b2ece51ca44544c430256437802"}}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <123897612+bhapas@users.noreply.github.com>
|
Hey @bhapas Are aware if there is any plan to support beat processors as alternative to elasticsearch processors? |
@zez3 Integrations support beat processors in the integration configuration. But Automatic Import currently generates Elastic Agent based integrations which always go through elasticsearch ingest pipelines for writing docs into ES. Do you have any usecase that you are looking to solve? |
Yes dns beats processor are not available on the elasticsearch processors. We ingest syslogs and the log.source.ip is reverse resolved to get the hostname |
@zez3 You can use https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html in your integration to achieve this. |
This is exactly what I do. |
|
@zez3 This is not something in the near product roadmap. But I shall take this in to check the possibility of having this in roadmap. Thanks for raising. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release note
Adds a feature to add support for handling
syslogs with unsupported message body.Summary
This PR handles the
unstructuredsyslog samples in Automatic Import.Examples of unstructured samples would be:
video1656541084.mp4
Checklist
For maintainers