Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Analyzer with sysmon via filebeat #152418

Conversation

kqualters-elastic
Copy link
Contributor

@kqualters-elastic kqualters-elastic commented Mar 1, 2023

Summary

Related issue: #148043

With the switch from beats to elastic agent, analyzer broke for sysmon data ingested via elastic agent, as some fields the code expects to exist became slightly different, this pr updates the frontend and server side api of analyzer to work with both old winlogbeat style sysmom ingestion and new elastic agent + filebeat shipping sysmon generated data.

Screen.Recording.2023-03-01.at.1.35.41.AM.mov

Video with 8.6 agent/fleet + latest main kibana/es + windows 10 w/sysmon vm

Checklist

@@ -86,6 +86,7 @@ export class EventsQuery extends BaseResolverQuery {
return {
body: this.query(filters),
index: this.indexPatterns,
allow_partial_search_results: true,
Copy link
Contributor Author

@kqualters-elastic kqualters-elastic Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

during testing it seemed that some integrations did not have a mapping for event.id, and so had to add this for queries to still return results. since we are searching by an id this can't really hurt right? ha

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, it shouldn't as we're searching on the doc id.

@@ -31,7 +31,7 @@ export type SortFields = [
{
'@timestamp': string;
},
{ [x: string]: string }
{ [x: string]: { order: string; unmapped_type?: string } }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@kqualters-elastic kqualters-elastic Mar 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had basically every windows related integration installed when trying to collect sysmon data in my local setup, and after it eventually started flowing, I noticed calls to /api/endpoint/resolver/events would sometimes fail due to the event.id not being mapped in a few esoteric logs-* windows system indices, adding this made the error not happen. Don't think it'll ever have any unintended consequences, as the indices events come from should have that field mapped already, and even if not, this is just for a tiebreak if timestamps are equal.

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 15.7MB 15.7MB +272.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 428 430 +2

Total ESLint disabled count

id before after diff
securitySolution 506 508 +2

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Contributor

@michaelolo24 michaelolo24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for providing the video demo with real data! Tested it locally with mock data and it showed the analyzer cube properly rendered for sysmon data!

@kqualters-elastic kqualters-elastic merged commit a9313ee into elastic:main Mar 1, 2023
@kqualters-elastic kqualters-elastic deleted the analyzer-with-sysmon-via-filebeat branch March 1, 2023 18:07
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Mar 1, 2023
## Summary

Related issue: elastic#148043

With the switch from beats to elastic agent, analyzer broke for sysmon
data ingested via elastic agent, as some fields the code expects to
exist became slightly different, this pr updates the frontend and server
side api of analyzer to work with both old winlogbeat style sysmom
ingestion and new elastic agent + filebeat shipping sysmon generated
data.

https://user-images.githubusercontent.com/56408403/222063497-64b2853e-5d09-4178-b336-1007886c396b.mov

Video with 8.6 agent/fleet + latest main kibana/es + windows 10 w/sysmon
vm

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit a9313ee)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.7

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Mar 1, 2023
…#152492)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[Security Solution] Analyzer with sysmon via filebeat
(#152418)](#152418)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kevin
Qualters","email":"56408403+kqualters-elastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-03-01T18:07:05Z","message":"[Security
Solution] Analyzer with sysmon via filebeat (#152418)\n\n##
Summary\r\n\r\nRelated issue:
https://github.com/elastic/kibana/issues/148043\r\n\r\nWith the switch
from beats to elastic agent, analyzer broke for sysmon\r\ndata ingested
via elastic agent, as some fields the code expects to\r\nexist became
slightly different, this pr updates the frontend and server\r\nside api
of analyzer to work with both old winlogbeat style sysmom\r\ningestion
and new elastic agent + filebeat shipping sysmon
generated\r\ndata.\r\n\r\n\r\n\r\nhttps://user-images.githubusercontent.com/56408403/222063497-64b2853e-5d09-4178-b336-1007886c396b.mov\r\n\r\nVideo
with 8.6 agent/fleet + latest main kibana/es + windows 10
w/sysmon\r\nvm\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a9313eefe0799d032e5176b1de8d3f1608e4e914","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Threat
Hunting:Investigations","v8.7.0","v8.8.0"],"number":152418,"url":"https://github.com/elastic/kibana/pull/152418","mergeCommit":{"message":"[Security
Solution] Analyzer with sysmon via filebeat (#152418)\n\n##
Summary\r\n\r\nRelated issue:
https://github.com/elastic/kibana/issues/148043\r\n\r\nWith the switch
from beats to elastic agent, analyzer broke for sysmon\r\ndata ingested
via elastic agent, as some fields the code expects to\r\nexist became
slightly different, this pr updates the frontend and server\r\nside api
of analyzer to work with both old winlogbeat style sysmom\r\ningestion
and new elastic agent + filebeat shipping sysmon
generated\r\ndata.\r\n\r\n\r\n\r\nhttps://user-images.githubusercontent.com/56408403/222063497-64b2853e-5d09-4178-b336-1007886c396b.mov\r\n\r\nVideo
with 8.6 agent/fleet + latest main kibana/es + windows 10
w/sysmon\r\nvm\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a9313eefe0799d032e5176b1de8d3f1608e4e914"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/152418","number":152418,"mergeCommit":{"message":"[Security
Solution] Analyzer with sysmon via filebeat (#152418)\n\n##
Summary\r\n\r\nRelated issue:
https://github.com/elastic/kibana/issues/148043\r\n\r\nWith the switch
from beats to elastic agent, analyzer broke for sysmon\r\ndata ingested
via elastic agent, as some fields the code expects to\r\nexist became
slightly different, this pr updates the frontend and server\r\nside api
of analyzer to work with both old winlogbeat style sysmom\r\ningestion
and new elastic agent + filebeat shipping sysmon
generated\r\ndata.\r\n\r\n\r\n\r\nhttps://user-images.githubusercontent.com/56408403/222063497-64b2853e-5d09-4178-b336-1007886c396b.mov\r\n\r\nVideo
with 8.6 agent/fleet + latest main kibana/es + windows 10
w/sysmon\r\nvm\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a9313eefe0799d032e5176b1de8d3f1608e4e914"}}]}]
BACKPORT-->

Co-authored-by: Kevin Qualters <56408403+kqualters-elastic@users.noreply.github.com>
bmorelli25 pushed a commit to bmorelli25/kibana that referenced this pull request Mar 10, 2023
## Summary

Related issue: elastic#148043

With the switch from beats to elastic agent, analyzer broke for sysmon
data ingested via elastic agent, as some fields the code expects to
exist became slightly different, this pr updates the frontend and server
side api of analyzer to work with both old winlogbeat style sysmom
ingestion and new elastic agent + filebeat shipping sysmon generated
data.



https://user-images.githubusercontent.com/56408403/222063497-64b2853e-5d09-4178-b336-1007886c396b.mov

Video with 8.6 agent/fleet + latest main kibana/es + windows 10 w/sysmon
vm

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants