[7.11][Telemetry] Diagnostic Alert Telemetry #84422

pjhampton · 2020-11-26T15:20:52Z

Summary

This PR extends the existing security telemetry collection by transmitting diagnostic alerts via a Kibana task manager.

Related PRs:

Implementation

@tsg - create a task manager task that is executed every ~5 minutes
@pjhampton - each execution:
- Confirm telemetry is enabled. If it's disabled, don't query for diag alerts.
- sorting by the event.ingested field.
- Limit the result set to 100 events.
@pjhampton - Call the queueTelemetryEvents function from the EventsTelemetry (See: [Security] Alert Telemetry for the Security app #77200)
~~Query the index we decide to for the time since last execution to present. Record the last execution time~~

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

pjhampton · 2020-12-03T19:22:46Z

Just dropping an update on this work item - there turned out to be a couple of background data plumbing pieces that needed to be put in place before this could be tested e2e. I hope to get them all in within the next day or 2.

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

Remove 2nd var to track telemetry opt in. Add ES client to start querying index. Use query to get docs from a dummy index. Change how index is queried. Get diagnostic alerts to send to staging cluster. Record last timestamp. PoC on telemetry opt in via 2 processes. Revert to original solution

pjhampton · 2020-12-04T17:19:52Z

@elasticmachine merge upstream

…stic/kibana into pjhampton/diagnostic-alert-telemetry

elasticmachine · 2020-12-08T16:34:46Z

Pinging @elastic/kibana-security (Team:Security)

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

pjhampton · 2020-12-08T20:11:06Z

@elasticmachine merge upstream

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

tsg · 2020-12-09T11:08:38Z

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

+        sort: [
+          {
+            'event.ingested': {
+              order: 'asc',


Do you want asc here so we get the most recent events?

I got it wrong. Updated here 5638da9
desc will order by most recent I believe from my testing.

tsg · 2020-12-09T11:23:49Z

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts

+  public async fetchDiagnosticAlerts() {
+    const query = {
+      expand_wildcards: 'open,hidden',
+      index: 'logs-endpoint.diagnostic.collection-default*',


I think we need logs-endpoint.diagnostic.collection-* here, because I think @ferullo was saying that the diagnostic alerts will respect the namespace setting, so they might come with something else than default.

Thanks! Updated here: 2018132

pjhampton · 2020-12-09T13:53:26Z

@elasticmachine merge upstream

jeska

Doc in analytics-staging looks good!

kibanamachine · 2020-12-09T19:26:38Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: b707729

Metrics [docs]

Distributable file count

id	before	after	diff
`default`	46981	47743	+762

History

💚 Build #93142 succeeded 4c17199
💔 Build #93090 failed 58672b3
💔 Build #93076 failed 07cdf34
💔 Build #93065 failed 43558f7
💔 Build #93053 failed 2018132

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

stevewritescode

LGTM. I reviewed the new allowlist fields and the content is consistent with other data that we're already collecting, so good to go on that front.

One minor question about the task scheduler.

stevewritescode · 2020-12-09T21:41:03Z

x-pack/plugins/security_solution/server/lib/telemetry/task.ts

+    return `${TelemetryDiagTaskConstants.TYPE}:${TelemetryDiagTaskConstants.VERSION}`;
+  };
+
+  public runTask = async (taskId: string, searchFrom: string, searchTo: string) => {


This isn't an objection in the code, just a question: If there are multiple Kibana instances, it is possible for this task to be running simultaneously in multiple instances? Or does the task manager ensure that only one execution can happen at any given time?

@stevewritescode Good question. The taskManager uses a distributed model which ensures that only a single Kibana instance will run the task. Each Kibana instance polls for new tasks on an interval and attempts to "claim" tasks that are ready to fire. If another Kibana instance tries to claim the same task, only one will succeed and the others will get a 409 Conflict as per OCC. If I'm remembering all this correctly. :)

tsg

LGTM, nice work!

@tsg

* Port @tsg's work on task manager. Remove 2nd var to track telemetry opt in. Add ES client to start querying index. Use query to get docs from a dummy index. Change how index is queried. Get diagnostic alerts to send to staging cluster. Record last timestamp. PoC on telemetry opt in via 2 processes. Revert to original solution * Update on agreed method. Fixes race condition. * Expand wildcards. * stage. * Add rule.ruleset collection. * Update telemetry sender with correct query for loading diag alerts. * Add similar task tests to endpont artifact work. * Fix broken import statement. * Create sender mocks. * Update test to check for func call. * Update unused reference. * record last run. * Update index. * fix import * Fix test. * test fix. * Pass unit to time diff calc. * Tests should pass now hopefully. * Add additional process fields to allowlist. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

pjhampton added Team:Security Team focused on: Auth, Users, Roles, Spaces, Audit Logging, and more! v7.11.0 labels Nov 26, 2020

pjhampton self-assigned this Nov 26, 2020

pjhampton mentioned this pull request Nov 26, 2020

[WIP] Diagnostic telemetry sender #84268

Closed

14 tasks

tsg reviewed Dec 2, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Outdated Show resolved Hide resolved

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Outdated Show resolved Hide resolved

pjhampton changed the title ~~Diagnostic Alert Telemetry~~ [7.10][Telemetry] Diagnostic Alert Telemetry Dec 4, 2020

pjhampton added the release_note:enhancement label Dec 4, 2020

pjhampton changed the title ~~[7.10][Telemetry] Diagnostic Alert Telemetry~~ [7.11][Telemetry] Diagnostic Alert Telemetry Dec 4, 2020

tsg reviewed Dec 4, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Show resolved Hide resolved

pjhampton force-pushed the pjhampton/diagnostic-alert-telemetry branch from bace39b to e091be3 Compare December 4, 2020 16:56

Update on agreed method. Fixes race condition.

e5c72c1

kibanamachine and others added 11 commits December 4, 2020 12:22

Merge branch 'master' into pjhampton/diagnostic-alert-telemetry

f0f5438

Expand wildcards.

ae8e2f4

Merge branch 'pjhampton/diagnostic-alert-telemetry' of github.com:ela…

49a480d

…stic/kibana into pjhampton/diagnostic-alert-telemetry

stage.

76cb626

Merge branch 'master' into pjhampton/diagnostic-alert-telemetry

49da9f4

Add rule.ruleset collection.

2d685a4

Update telemetry sender with correct query for loading diag alerts.

7becfb7

Add similar task tests to endpont artifact work.

1974c94

Fix broken import statement.

0a13f63

Create sender mocks.

c709fc6

Update test to check for func call.

b8d9ddb

pjhampton marked this pull request as ready for review December 8, 2020 16:34

pjhampton requested review from a team as code owners December 8, 2020 16:34

pjhampton requested a review from jeska December 8, 2020 16:35

legrego added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Dec 8, 2020

jeska reviewed Dec 8, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Show resolved Hide resolved

kibanamachine and others added 2 commits December 8, 2020 15:11

Merge branch 'master' into pjhampton/diagnostic-alert-telemetry

535098e

Update unused reference.

f0d226e

madirey reviewed Dec 8, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/telemetry/sender.ts Outdated Show resolved Hide resolved

tsg mentioned this pull request Dec 9, 2020

Add read permissions to the Kibana system user for the diagnostic telemetry #85391

Closed

tsg reviewed Dec 9, 2020

View reviewed changes

pjhampton added 6 commits December 9, 2020 11:57

record last run.

5638da9

Update index.

2018132

fix import

43558f7

Fix test.

07cdf34

test fix.

1ddd2ef

Pass unit to time diff calc.

d14f8f9

Merge branch 'master' into pjhampton/diagnostic-alert-telemetry

58672b3

madirey approved these changes Dec 9, 2020

View reviewed changes

pjhampton added 2 commits December 9, 2020 15:43

Tests should pass now hopefully.

4c17199

Add additional process fields to allowlist.

b707729

jeska approved these changes Dec 9, 2020

View reviewed changes

stevewritescode approved these changes Dec 9, 2020

View reviewed changes

tsg approved these changes Dec 10, 2020

View reviewed changes

pjhampton merged commit 6e7fb4a into master Dec 10, 2020

pjhampton mentioned this pull request Dec 10, 2020

[7.x] [Telemetry] Diagnostic Alert Telemetry (#84422) #85631

Merged

This was referenced Dec 16, 2020

[7.11][Telemetry] Fix diagnostic index name #86134

Closed

[7.11][Telemetry] Update index name for diagnostic telemetry. #86468

Merged

[7.11][Telemetry] Add run and alert count to task state. #86776

Merged

pjhampton deleted the pjhampton/diagnostic-alert-telemetry branch February 3, 2021 10:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.11][Telemetry] Diagnostic Alert Telemetry #84422

[7.11][Telemetry] Diagnostic Alert Telemetry #84422

pjhampton commented Nov 26, 2020 •

edited

Loading

pjhampton commented Dec 3, 2020

pjhampton commented Dec 4, 2020

elasticmachine commented Dec 8, 2020

pjhampton commented Dec 8, 2020

tsg Dec 9, 2020

pjhampton Dec 9, 2020 •

edited

Loading

tsg Dec 9, 2020

pjhampton Dec 9, 2020

pjhampton commented Dec 9, 2020

jeska left a comment

kibanamachine commented Dec 9, 2020

stevewritescode left a comment

stevewritescode Dec 9, 2020

madirey Dec 10, 2020

tsg left a comment

[7.11][Telemetry] Diagnostic Alert Telemetry #84422

[7.11][Telemetry] Diagnostic Alert Telemetry #84422

Conversation

pjhampton commented Nov 26, 2020 • edited Loading

Summary

Implementation

Checklist

For maintainers

pjhampton commented Dec 3, 2020

pjhampton commented Dec 4, 2020

elasticmachine commented Dec 8, 2020

pjhampton commented Dec 8, 2020

tsg Dec 9, 2020

Choose a reason for hiding this comment

pjhampton Dec 9, 2020 • edited Loading

Choose a reason for hiding this comment

tsg Dec 9, 2020

Choose a reason for hiding this comment

pjhampton Dec 9, 2020

Choose a reason for hiding this comment

pjhampton commented Dec 9, 2020

jeska left a comment

Choose a reason for hiding this comment

kibanamachine commented Dec 9, 2020

💚 Build Succeeded

Metrics [docs]

Distributable file count

History

stevewritescode left a comment

Choose a reason for hiding this comment

stevewritescode Dec 9, 2020

Choose a reason for hiding this comment

madirey Dec 10, 2020

Choose a reason for hiding this comment

tsg left a comment

Choose a reason for hiding this comment

pjhampton commented Nov 26, 2020 •

edited

Loading

pjhampton Dec 9, 2020 •

edited

Loading