[Security Solution] Cleanup endpoint telemetry #71950

michaelolo24 · 2020-07-15T20:25:47Z

Summary

This PR is to improve the logic for endpoint usage telemetry.

It removes an unnecessary reduce and confines all updating activities to a single loop
It cleans up our savedObjects requests to only require the necessary details for our telemetry
Improves policy usage logic and provides error handling in the case of missing data and request failures
Added tests

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

michaelolo24 · 2020-07-16T17:37:05Z

x-pack/plugins/security_solution/server/usage/endpoints/index.ts

@@ -69,13 +71,14 @@ export const getDefaultEndpointTelemetry = (): EndpointUsage => ({
 });

 /**
- * @description this fun
+ * @description this function updates the os telemetry. We use the fullName field as the key as it contains the name and version details.


@oatkiller fixed 😆

michaelolo24 · 2020-07-16T17:38:09Z

x-pack/plugins/security_solution/server/usage/endpoints/fleet_saved_objects.ts

    type: AGENT_EVENT_SAVED_OBJECT_TYPE,
-    filter: `${AGENT_EVENT_SAVED_OBJECT_TYPE}.attributes.agent_id: ${agentId} and ${AGENT_EVENT_SAVED_OBJECT_TYPE}.attributes.message: "${FLEET_ENDPOINT_PACKAGE_CONSTANT}"`,


removed that initial agent.id check as it's redundant giving the search and searchFields parameters below

michaelolo24 · 2020-07-16T17:41:21Z

x-pack/plugins/security_solution/server/usage/endpoints/fleet_saved_objects.ts

@@ -29,9 +37,11 @@ export const getLatestFleetEndpointEvent = async (
  agentId: string
 ) =>
  savedObjectsClient.find<AgentEventSOAttributes>({
+    // Get the most recent endpoint event.


@oatkiller, after speaking with @rudolf yesterday, there doesn't seem to be a way we can really batch this. I.e. we cannot say "given a list of agent id's, give me the most recent event for each id".

Thinking about it more, we may be able to say, "give me every event for each of these agent id's using an or in the filter, but that would be a massive response to then filter". This will definitely need perf testing

michaelolo24 · 2020-07-16T17:43:07Z

x-pack/plugins/security_solution/server/usage/endpoints/index.ts

-  endpointTelemetry: EndpointUsage
-): Promise<EndpointUsage> => {
-  const updatedEndpointTelemetry = { ...endpointTelemetry };
+export const updateEndpointDailyActiveCount = (


@oatkiller, moved the daily count logic here

michaelolo24 · 2020-07-16T17:44:14Z

x-pack/plugins/security_solution/server/usage/endpoints/index.ts

+      if (agentId) {
+        let agentEvents;
+        try {
+          const response = await getLatestFleetEndpointEvent(soClient, agentId);


Will gladly take any suggestions people may have on optimizing this, as currently there is the potential for us to make 10000 of these requests if there are that many agents.

michaelolo24 · 2020-07-16T17:46:46Z

x-pack/plugins/security_solution/server/usage/endpoints/index.ts

+  } else if (isAnActiveMalwareState && failedToEnable) {
+    updatedPoliciesTracker.malware.failure += 1;
+  }
+


Not currently tracking the situation where an endpoint failed to turn the policy off as it's unlikely / not a high priority. Also, since we're in FF, I'm not sure if I can add an additional field to telemetry, @afharo?

since we're in FF, I'm not sure if I can add an additional field to telemetry, @afharo?

If it's a bug, I guess you can add it. If it isn't, you'll have to wait :)

elasticmachine · 2020-07-16T17:55:53Z

Pinging @elastic/endpoint-data-visibility-team (Team:Endpoint Data Visibility)

oatkiller

thanks!

michaelolo24 · 2020-07-20T14:32:55Z

@elasticmachine merge upstream

kibanamachine · 2020-07-20T16:12:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 462d2d0

Build metrics

✅ unchanged

History

💛 Build #62368 was flaky 9175e9e
💔 Build #62085 failed bba51b44c3e670dcd9a77f7cabe29523ff190c52
💔 Build #62078 failed b502aa4d6f7b7f1609365f291ec5176e3ef54189

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: (60 commits) [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) [ML] improve annotation flyout performance (elastic#72299) [APM] Testing error rate API and restructuring folders (elastic#72257) ...

* master: (26 commits) [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) [ML] improve annotation flyout performance (elastic#72299) [APM] Testing error rate API and restructuring folders (elastic#72257) ...

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…feature-privileges * alerting/consumer-based-rbac: (45 commits) fixed alerts test [SIEM][Detection Engine][Lists] Adds list permissions (elastic#72335) [SIEM][Detection Engine][Lists] Adds conflict versioning and io-ts improvements to lists (elastic#72337) [Resolver] no longer pass related event stats to process node component (elastic#72435) Revert "skip flaky suite (elastic#72146)" [Security Solution] Cleanup endpoint telemetry (elastic#71950) Unskip dashboard embeddable rendering tests (elastic#71824) [ENDPOINT] Added unerolling status for host. (elastic#72303) [Alerting][Connectors] Increase the size of the logos (elastic#72419) [SECURITY] [Timeline] Raw events not displayed (elastic#72387) [ML] Fixes display of regression stop stats if one is NaN (elastic#72412) [Ingest Pipelines] Processor Editor Move Tooltip (elastic#72239) Fix match phrase and not match phrase comparators (elastic#71850) [Plugin Generator] Generate tsconfig and useDefaultBehaviors (elastic#72040) [Security Solution][Timeline] Fix timeline styling and createFrom beh… (elastic#72152) allow user to disable alert even if they dont have privileges to the underlying action [Resolver] Selector performance (elastic#72380) [Ingest Manager] Set `_meta` in the index.mappings (elastic#72026) [Ingest Manager] Do not bumb config revision during config creation (elastic#72270) [ML] Adding missing index pattern name to new job wizards (elastic#72400) ...

michaelolo24 force-pushed the optimize-endpoint-telemetry branch from b502aa4 to bba51b4 Compare July 15, 2020 20:47

michaelolo24 added 2 commits July 16, 2020 13:35

cleanup endpoint telemetry

373760c

more cleanup & add tests

9175e9e

michaelolo24 force-pushed the optimize-endpoint-telemetry branch from bba51b4 to 9175e9e Compare July 16, 2020 17:35

michaelolo24 commented Jul 16, 2020

View reviewed changes

michaelolo24 marked this pull request as ready for review July 16, 2020 17:50

michaelolo24 requested review from a team as code owners July 16, 2020 17:50

michaelolo24 changed the title ~~cleanup endpoint telemetry~~ [Security Solution] Cleanup endpoint telemetry Jul 16, 2020

michaelolo24 added release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Data Visibility Team managing the endpoint resolver v7.9.0 v8.0.0 labels Jul 16, 2020

michaelolo24 added the Feature:Telemetry label Jul 16, 2020

michaelolo24 requested a review from oatkiller July 16, 2020 17:56

james-elastic approved these changes Jul 17, 2020

View reviewed changes

oatkiller approved these changes Jul 20, 2020

View reviewed changes

Merge branch 'master' into optimize-endpoint-telemetry

462d2d0

michaelolo24 merged commit 2094f33 into elastic:master Jul 20, 2020

michaelolo24 deleted the optimize-endpoint-telemetry branch July 20, 2020 16:28

michaelolo24 mentioned this pull request Jul 20, 2020

[7.9] [Security Solution] Cleanup endpoint telemetry (#71950) #72476

Closed

michaelolo24 added a commit to michaelolo24/kibana that referenced this pull request Jul 20, 2020

[Security Solution] Cleanup endpoint telemetry (elastic#71950)

3c7d7f6

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

michaelolo24 mentioned this pull request Jul 20, 2020

[7.x] [Security Solution] Cleanup endpoint telemetry (#71950) #72477

Merged

michaelolo24 added a commit to michaelolo24/kibana that referenced this pull request Jul 20, 2020

[Security Solution] Cleanup endpoint telemetry (elastic#71950)

6a2d69c

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

michaelolo24 added a commit to michaelolo24/kibana that referenced this pull request Jul 20, 2020

[Security Solution] Cleanup endpoint telemetry (elastic#71950)

8348054

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

michaelolo24 mentioned this pull request Jul 20, 2020

[7.9] [Security Solution] Cleanup endpoint telemetry (#71950) #72478

Merged

michaelolo24 added a commit that referenced this pull request Jul 20, 2020

[7.9] [Security Solution] Cleanup endpoint telemetry (#71950) (#72478)

8399dc5

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

michaelolo24 added a commit that referenced this pull request Jul 20, 2020

[7.x] [Security Solution] Cleanup endpoint telemetry (#71950) (#72477)

b2cb216

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Cleanup endpoint telemetry #71950

[Security Solution] Cleanup endpoint telemetry #71950

michaelolo24 commented Jul 15, 2020 •

edited

Loading

michaelolo24 Jul 16, 2020

michaelolo24 Jul 16, 2020

michaelolo24 Jul 16, 2020

michaelolo24 Jul 16, 2020

michaelolo24 Jul 16, 2020

michaelolo24 Jul 16, 2020

afharo Jul 16, 2020

elasticmachine commented Jul 16, 2020

oatkiller left a comment

michaelolo24 commented Jul 20, 2020

kibanamachine commented Jul 20, 2020

		type: AGENT_EVENT_SAVED_OBJECT_TYPE,
		filter: `${AGENT_EVENT_SAVED_OBJECT_TYPE}.attributes.agent_id: ${agentId} and ${AGENT_EVENT_SAVED_OBJECT_TYPE}.attributes.message: "${FLEET_ENDPOINT_PACKAGE_CONSTANT}"`,

[Security Solution] Cleanup endpoint telemetry #71950

[Security Solution] Cleanup endpoint telemetry #71950

Conversation

michaelolo24 commented Jul 15, 2020 • edited Loading

Summary

Checklist

For maintainers

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

michaelolo24 Jul 16, 2020

Choose a reason for hiding this comment

afharo Jul 16, 2020

Choose a reason for hiding this comment

elasticmachine commented Jul 16, 2020

oatkiller left a comment

Choose a reason for hiding this comment

michaelolo24 commented Jul 20, 2020

kibanamachine commented Jul 20, 2020

💚 Build Succeeded

Build metrics

History

michaelolo24 commented Jul 15, 2020 •

edited

Loading