[Security] Alert Telemetry for the Security app #77200

tsg · 2020-09-10T15:45:48Z

Summary

This adds a TelemetryEventsSender component that can be used to publish Endpoint alerts to our Telemetry service. The alerts are filtered by a set of allowed fields (for PII) and batched in a queue to be sent once per minute. There is a cap of 100 alerts per minute to be sent. The component respects the telemetry opt-in status and enriches the alerts with the cluster ID and name.

The Detection Engine is slightly modified to send endpoint telemetry events via the TelemetryEventsSender. Only the the "custom query" rule type is modified because that's the only one that can create Endpoint Alerts.

Remaining TODOs:

actually send the alerts once the URL is finalized
filter-out non-endpoint alerts (currently sends them all for testing)
retrieve the cluster-ID and license info and include it in the telemetry
set sending time to 1m

Checklist

Delete any items that are not applicable to this PR.

~~[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
~~[ ] This was checked for keyboard-only and screenreader accessibility~~
~~[ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser~~
~~[ ] This was checked for cross-browser compatibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately

tsg · 2020-09-18T16:04:51Z

@elasticmachine merge upstream

This is using recursion now. Also, based on Xavier's review, moved up the try and the isSending check to avoid building up queries.

stevewritescode · 2020-09-28T14:33:31Z

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

+// Allow list for the data we include in the events. True means that it is deep-cloned
+// blindly. Object contents means that we only copy the fields that appear explicitly in
+// the sub-object.
+const allowlistEventFields: AllowlistFields = {


This looks good to me. FYI @bfilar @pjhampton @jeska

x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts

spong

Detections changes LGTM! 👍

…ignals/send_telemetry_events.ts Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

elasticmachine · 2020-09-30T11:56:08Z

Pinging @elastic/siem (Team:SIEM)

tsg · 2020-09-30T12:35:41Z

@elasticmachine merge upstream

XavierM

Well done!

Tudor 1 - Typescript 0

XavierM · 2020-09-30T14:23:30Z

@elasticmachine merge upstream

tsg · 2020-09-30T18:37:33Z

@elasticmachine merge upstream

kibanamachine · 2020-09-30T20:10:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e8b364d

Metrics [docs]

distributable file count

id	value	diff	baseline
`default`	45828	+2	45826

History

💛 Build #78304 was flaky d0d52f4
💔 Build #78260 failed 52390cf
💚 Build #78241 succeeded 0dda88c
💔 Build #78233 failed 1a29cbd
💔 Build #77837 failed 5f25435

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…aly-detection-partition-field * 'master' of github.com:elastic/kibana: (37 commits) Fixes for the Ticket 78375 (elastic#79004) [Security] Alert Telemetry for the Security app (elastic#77200) [Search bar] Remove duplicate `popoverProps` (elastic#79025) [Security Solution][Detections] Add rule overrides for single event EQL rules (elastic#78876) [SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest (elastic#74305) remove file accidentally checked in (elastic#79005) [ML] DF Analytics creation wizard: replace select input with job type cards with icons (elastic#78872) [Design] A couple fixes for 7.10 (elastic#78801) Fix KQL autocomplete value suggestions (elastic#78676) [Security Solution][Resolver] New mock with cursor (elastic#78863) Embeddables: basic documentation (elastic#78900) [security solution] only import beat_schema when needed (elastic#78708) [Reporting] API Integration tests: fix flaky tests for Spaces CSV formatting (elastic#78849) [Actions] Adds a "Test Connector" button on the Connectors List to make discovery of the Test tab easier (elastic#78746) [Discover] Fix functional time picker test permissions (elastic#78564) [ML] Fixing module datafeed overrides (elastic#78925) Adds some missing licenses to the CSV export (elastic#78719) [dev/cli] ensure plugins/ and all watch source dirs exist (elastic#78973) [Lens] Stop using scripted metric to collect telemetry (elastic#78687) [Lens] fix wrong message in fields accordion (elastic#78924) ...

This adds a `TelemetryEventsSender` component that can be used to publish Endpoint alerts to our Telemetry service. The alerts are filtered by a set of allowed fields (for PII) and batched in a queue to be sent once per minute. There is a cap of 100 alerts per minute to be sent. The component respects the telemetry opt-in status and enriches the alerts with the cluster ID and name. The Detection Engine is slightly modified to send endpoint telemetry events via the `TelemetryEventsSender`. Only the "custom query" rule type is modified because that's the only one that can create Endpoint Alerts. Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

elasticmachine · 2021-09-23T14:25:54Z

Pinging @elastic/security-solution (Team: SecuritySolution)

tsg added 6 commits September 7, 2020 22:24

Empty recurring task

e2250a9

Added processEvents with tests

da753fe

SendIfDue + tests

6335857

Connect telemetry in the detection engine

c302fdd

Respect opt-in status

5e14df8

test fixes + test for telemetry disabled

7bcf141

tsg added Team:SIEM v7.10.0 v8.0.0 release_note:enhancement labels Sep 10, 2020

Various type fixes

483e3cf

elasticmachine and others added 3 commits September 18, 2020 12:04

Merge branch 'master' into telemetry_events

0cd31af

Add cluster_uuid and cluster_name

65a5078

Filter by endpoint alerts

fb6b3ea

pjhampton mentioned this pull request Sep 23, 2020

[7.10][Telemetry] Update data collection panel to show example security event. #78262

Closed

6 tasks

tsg added 5 commits September 23, 2020 12:44

type fixes + tests

2fdc70c

mege master

725fc54

fix types

069147f

merge master

896ca96

Refactor processEvents

0ae4046

This is using recursion now. Also, based on Xavier's review, moved up the try and the isSending check to avoid building up queries.

stevewritescode reviewed Sep 28, 2020

View reviewed changes

spong reviewed Sep 29, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/detection_engine/signals/send_telemetry_events.ts Outdated Show resolved Hide resolved

spong approved these changes Sep 29, 2020

View reviewed changes

tsg and others added 6 commits September 29, 2020 12:55

Send events to the telemetry server

2f31f49

Small refactoring

f8309c9

Add license fields

b2df3a8

Update x-pack/plugins/security_solution/server/lib/detection_engine/s…

e6a2d71

…ignals/send_telemetry_events.ts Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

Move undefined check in the function to simplify top level code

722e68d

Correct datastream to data_stream

5f25435

XavierM reviewed Sep 29, 2020

View reviewed changes

tsg added 2 commits September 30, 2020 13:06

Incorporated Xavier's feedback + add license header

1a29cbd

Test fix + minor changes

0dda88c

tsg marked this pull request as ready for review September 30, 2020 11:56

tsg requested review from a team as code owners September 30, 2020 11:56

tsg requested a review from XavierM September 30, 2020 11:56

Commented out verbose debug logs

23fbe32

Merge branch 'master' into telemetry_events

52390cf

XavierM approved these changes Sep 30, 2020

View reviewed changes

Merge branch 'master' into telemetry_events

d0d52f4

pjhampton mentioned this pull request Sep 30, 2020

[7.10][Telemetry] Display collected security event sample #78963

Merged

6 tasks

Merge branch 'master' into telemetry_events

e8b364d

tsg merged commit 49c8ff3 into elastic:master Sep 30, 2020

tsg mentioned this pull request Sep 30, 2020

[7.x] [Security] Alert Telemetry for the Security app (#77200) #79031

Merged

pjhampton mentioned this pull request Oct 14, 2020

[7.10][Telemetry] Add in cluster version number for sec telemetry sender #80545

Merged

2 tasks

pjhampton added the Feature:Telemetry label Oct 14, 2020

tsg mentioned this pull request Nov 24, 2020

[WIP] Diagnostic telemetry sender #84268

Closed

14 tasks

pjhampton mentioned this pull request Nov 26, 2020

[7.11][Telemetry] Diagnostic Alert Telemetry #84422

Merged

9 tasks

This was referenced Feb 18, 2021

[7.12][Telemetry] Add missing fields for security telemetry #91920

Merged

[7.12][Telemetry] Security telemetry allowlist fix. #92850

Merged

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security] Alert Telemetry for the Security app #77200

[Security] Alert Telemetry for the Security app #77200

tsg commented Sep 10, 2020 •

edited

Loading

tsg commented Sep 18, 2020

stevewritescode Sep 28, 2020

bfilar Sep 28, 2020

spong left a comment

elasticmachine commented Sep 30, 2020

tsg commented Sep 30, 2020

XavierM left a comment •

edited

Loading

XavierM commented Sep 30, 2020

tsg commented Sep 30, 2020

kibanamachine commented Sep 30, 2020

elasticmachine commented Sep 23, 2021

[Security] Alert Telemetry for the Security app #77200

[Security] Alert Telemetry for the Security app #77200

Conversation

tsg commented Sep 10, 2020 • edited Loading

Summary

Checklist

For maintainers

tsg commented Sep 18, 2020

stevewritescode Sep 28, 2020

Choose a reason for hiding this comment

bfilar Sep 28, 2020

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 30, 2020

tsg commented Sep 30, 2020

XavierM left a comment • edited Loading

Choose a reason for hiding this comment

XavierM commented Sep 30, 2020

tsg commented Sep 30, 2020

kibanamachine commented Sep 30, 2020

💚 Build Succeeded

Metrics [docs]

distributable file count

History

elasticmachine commented Sep 23, 2021

tsg commented Sep 10, 2020 •

edited

Loading

XavierM left a comment •

edited

Loading