[Security Solution][Detections] Add alert source to detection rule action context #85488

marshallmain · 2020-12-09T22:55:05Z

Summary

This PR adds the source of the detection alert documents to the rule action context so notification actions can access fields within the detection alerts. The source alerts are available as an array through Mustache templating at {{context.alerts}}, e.g. {{context.alerts.0.process.name}} would get the process.name field of the first alert.

Docs update: Alert data is now available in detection rule actions at {{context.alerts}} as an array. This array contains each alert generated since the last time the action executed. Mustache templating can be used to iterate over all alerts in the array and capture information from each one. For example, {{#context.alerts}}Detection alert for user: {{user.name}}{{/context.alerts}} would create the string Detection alert for user: <user.name> for every alert in the array. Any alerts that don't have user.name will still generate the string but leave <user.name> blank.

Test process:

Go through the rule creation process for a rule type up to step 4 (Rule Actions).
Create a webhook connector that POSTs to http://localhost:<some port> with no auth and no http header, ex
Choose either "On rule execution" or one of the throttled options for when the action should fire. If using the throttled options, change the throttle time in /kibana/x-pack/plugins/security_solution/public/detections/components/rules/throttle_select_field/index.tsx down to 5m or less so you don't have to wait an hour for the action to trigger
Use netcat or equivalent to listen on the chosen port. e.g. sudo nc -l 392 on ubuntu listens on port 392.
Create a document that triggers the rule. When the action fires, netcat should print out the request it received from the webhook. You may need to restart netcat for it to print further requests even though the process continues to run.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

marshallmain · 2020-12-10T06:47:25Z

@elasticmachine merge upstream

marshallmain · 2020-12-10T15:31:23Z

jenkins test this

marshallmain · 2020-12-14T19:45:02Z

@elasticmachine merge upstream

madirey · 2020-12-14T21:46:36Z

Reviewed, pulled, and tested that notifications fire. Looks good!

dhurley14

LGTM

…tion context (elastic#85488) * Adds context.alerts as available parameter for detection rule actions Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (66 commits) [Alerting] fixes broken Alerting Example plugin (elastic#85774) [APM] Service overview instances table (elastic#85770) [Security Solution] Unskip timeline creation Cypress test (elastic#85871) properly recognize enterprise licenses (elastic#85849) [SecuritySolution][Detections] Adds SavedObject persistence to Signals Migrations (elastic#85690) [TSVB] Fix functional tests flakiness and unskip them (elastic#85388) [Fleet] Change permissions for Fleet enroll role (elastic#85802) Gauge visualization can no longer be clicked to filter on values since Kibana 7.10.0 (elastic#84768) [Security Solution][Detections] Add alert source to detection rule action context (elastic#85488) [Discover] Don't display hide/show button for histogram when there's no time filter (elastic#85424) skip flaky suite (elastic#78553) License checks for alerts plugin (elastic#85649) skip flaky suite (elastic#84992) skip 'query return results valid for scripted field' elastic#78553 Allow action types to perform their own mustache variable escaping in parameter templates (elastic#83919) [ML] More machine learning links in doc_links_service.ts (elastic#85365) Removed Alerting & Event Log deprecated fields that should not be using (elastic#85652) Closes elastic#79995 by adding new tab in transaction details to show related trace logs. (elastic#85859) Fix outdated jest snapshot [Maps] Surface on prem EMS (elastic#85729) ...

…tion context (#85488) (#85910) * Adds context.alerts as available parameter for detection rule actions Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

austinsonger · 2020-12-31T16:59:41Z

So this pull request going to give the ability to add following for context fields?

Source IP
Source Hostname
Destination IP
Destination Hostname

jamesspi · 2021-01-04T09:34:38Z

@austinsonger Yes, any fields in the document that are present will be able to be used as context for external actions.

kibanamachine · 2021-01-06T09:03:10Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 6f4171e
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Test Failures

Jest Integration Tests.src/core/server/ui_settings/integration_tests.uiSettings/routes doc exists get route returns a 200 and includes userValues

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 4 times on tracked branches: https://github.com/elastic/kibana/issues/51584

Stack Trace

ConnectionError: socket hang up
    at ClientRequest.onError (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:114:16)
    at ClientRequest.emit (events.js:315:20)
    at Socket.socketOnEnd (_http_client.js:493:9)
    at Socket.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

Jest Integration Tests.src/core/server/ui_settings/integration_tests.uiSettings/routes doc exists set route returns a 200 and all values including update

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 4 times on tracked branches: https://github.com/elastic/kibana/issues/51585

Stack Trace

Error: unknown error
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:351:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at ClientRequest.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
    at ClientRequest.emit (events.js:315:20)
    at Socket.socketCloseListener (_http_client.js:443:11)
    at Socket.emit (events.js:327:22)
    at TCP.<anonymous> (net.js:673:12)

Jest Integration Tests.src/core/server/ui_settings/integration_tests.uiSettings/routes doc exists set route returns a 400 if trying to set overridden value

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 4 times on tracked branches: https://github.com/elastic/kibana/issues/51586

Stack Trace

Error: unknown error
    at respond (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:351:15)
    at checkRespForFailure (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/dev/shm/workspace/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at ClientRequest.wrapper (/dev/shm/workspace/kibana/node_modules/lodash/lodash.js:4949:19)
    at ClientRequest.emit (events.js:315:20)
    at Socket.socketErrorListener (_http_client.js:469:9)
    at Socket.emit (events.js:315:20)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

and 4 more failures, only showing the first 3.

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	8.3MB	8.3MB	+45.0B

Distributable file count

id	before	after	diff
`default`	47138	47899	+761

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	215.7KB	215.7KB	+47.0B

History

💛 Build #94167 was flaky 6f4171e
💛 Build #93525 was flaky b155080
💔 Build #93367 failed b155080
💔 Build #93362 failed aea4201

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

marshallmain added 2 commits December 9, 2020 17:32

Adds context.alerts as available parameter for detection rule actions

ad6c7e9

Update tests

7845567

marshallmain added release_note:enhancement Team:Detections and Resp Security Detection Response Team v7.11.0 v8.0.0 labels Dec 9, 2020

marshallmain added 2 commits December 10, 2020 00:10

Fix tests

346b95d

Linting

aea4201

Merge branch 'master' into improve-rule-actions

b155080

marshallmain marked this pull request as ready for review December 10, 2020 20:58

marshallmain requested review from a team as code owners December 10, 2020 20:58

Merge branch 'master' into improve-rule-actions

6f4171e

madirey approved these changes Dec 14, 2020

View reviewed changes

dhurley14 approved these changes Dec 15, 2020

View reviewed changes

marshallmain merged commit 818246e into elastic:master Dec 15, 2020

marshallmain deleted the improve-rule-actions branch December 15, 2020 06:43

marshallmain mentioned this pull request Dec 15, 2020

[7.x] [Security Solution][Detections] Add alert source to detection rule action context (#85488) #85910

Merged

peluja1012 mentioned this pull request Dec 15, 2020

Adding event data to SIEM alerting actions #66587

Closed

peluja1012 added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Dec 15, 2020

marshallmain mentioned this pull request Jan 13, 2021

Add context.alerts to rule creation UI and API docs elastic/security-docs#448

Merged

jmikell821 mentioned this pull request Jan 14, 2021

[DOCS] Add alert source to detection rule action context elastic/security-docs#449

Closed

spong mentioned this pull request Feb 15, 2021

Detections Hostname Variable #91338

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detections] Add alert source to detection rule action context #85488

[Security Solution][Detections] Add alert source to detection rule action context #85488

marshallmain commented Dec 9, 2020 •

edited by peluja1012

Loading

marshallmain commented Dec 10, 2020

marshallmain commented Dec 10, 2020

marshallmain commented Dec 14, 2020

madirey commented Dec 14, 2020

dhurley14 left a comment

austinsonger commented Dec 31, 2020

jamesspi commented Jan 4, 2021

kibanamachine commented Jan 6, 2021 •

edited

Loading

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

[Security Solution][Detections] Add alert source to detection rule action context #85488

[Security Solution][Detections] Add alert source to detection rule action context #85488

Conversation

marshallmain commented Dec 9, 2020 • edited by peluja1012 Loading

Summary

Checklist

For maintainers

marshallmain commented Dec 10, 2020

marshallmain commented Dec 10, 2020

marshallmain commented Dec 14, 2020

madirey commented Dec 14, 2020

dhurley14 left a comment

Choose a reason for hiding this comment

austinsonger commented Dec 31, 2020

So this pull request going to give the ability to add following for context fields?

jamesspi commented Jan 4, 2021

kibanamachine commented Jan 6, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Standard Out

Stack Trace

Metrics [docs]

Async chunks

Distributable file count

Page load bundle

History

marshallmain commented Dec 9, 2020 •

edited by peluja1012

Loading

kibanamachine commented Jan 6, 2021 •

edited

Loading