[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

paul-tavares · 2020-08-04T21:46:19Z

Summary

When the Endpoint generator is run, in addtion to indexing endpoint host metadata, it should also create the integrations needed for the endpoints in Ingest, which are:

Create an Endpoint Package Configuration (and associated Agent Configuration)
Enroll an agent in Fleet for the endpoint Host being indexed
Check the agent in with Fleet (to remove it from enrolling)
Have the Agent send an ack for a configuration change (tells fleet that the agent configuration was applied and lists the Agent in Fleet as a user of endpoint package)

TODO

Add CLI option to "skip ingest integration"
Find better way to call Fleet APIs that require ApiKey
(?) move Ingest/Fleet interaction methods to a separate lib/service for reusability
Can the --delete option be supported for the Ingest/Fleet created docs?
Caching of created agent private API keys (so that we don't keep creating them if we need subsquent updates to the agent)

Fleet showing an Agent that was created via the Endpoint data generator:

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

kevinlog · 2020-08-21T21:09:51Z

@elasticmachine merge upstream

…t-data-generator-create-policy

elasticmachine · 2020-09-21T17:09:47Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

elasticmachine · 2020-09-21T17:09:47Z

Pinging @elastic/endpoint-management (Team:Endpoint Management)

kevinlog · 2020-09-21T17:44:55Z

x-pack/plugins/security_solution/common/endpoint/types/index.ts

@@ -880,6 +883,14 @@ export interface PolicyConfig {
  };
 }

+export interface AdvancedFields {


is this in for the changes that @jmiller263 is making for Advanced policy?

she's using this updated generator code for her changes, but its not explicitly in her pr

I'd prefer that we make changes to the Policy types when we add those new features. @jmiller263 could use the type changes locally until it's time to merge those features.

…t-data-generator-create-policy

spalger

KbnClient changes LGTM

…t-data-generator-create-policy

paul-tavares

LGTM.
I added a few more comments, but I think they are ok to be worked on after this goes in. :)

Thanks for picking this up.

paul-tavares · 2020-09-28T20:31:11Z

x-pack/plugins/security_solution/common/endpoint/index_data.ts

+  // Enroll an agent for the Host
+  const body: PostAgentEnrollRequest['body'] = {
+    type: 'PERMANENT',
+    metadata: {


Here is a better metadata message, which will no longer cause the Fleet agent list to show - for host:

{ //... metadata: { local: { "elastic": { "agent": { "version": versionNumber } }, "host": { "architecture": "x86_64", "hostname": `artifact-downloader.${Date.now()}.elastic.co`, "name": "artifact-downloader", "id": "1c032ec0-3a94-4d54-9ad2-c5610c0eaba4", "ip": [ "fe80::703b:b9e6:887d:7f5/64", "10.0.2.15/24", "::1/128", "127.0.0.1/8" ], "mac": [ "08:00:27:d8:c5:c0" ] }, "os": { "family": "windows", "kernel": "10.0.19041.388 (WinBuild.160101.0800)", "platform": "windows", "version": "10.0", "name": "Windows 10 Pro", "full": "Windows 10 Pro(10.0)" } }, }

change local.elastic.agent.version to 8.0.0 (although, we should address that as well by using the kbnClient to query Kibana to get its version number, but we can do that another time (can you open an issue to track? :) )
Also - you will want to change the host.hostname ++ host.name to be endpointHost.host

paul-tavares · 2020-09-28T20:35:15Z

x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts

 import { Client, ClientOptions } from '@elastic/elasticsearch';
 import { ResponseError } from '@elastic/elasticsearch/lib/errors';
+import { KbnClient, ToolingLog } from '@kbn/dev-utils';
+import { AxiosResponse } from 'axios';
+import { KibanaConfig } from '@kbn/dev-utils/target/kbn_client/kbn_client_requester';


Can you check if this can be imported directly from @kbn/dev-utils? if so, then just add it to the import above.

i don't think it can be

kevinlog

LGTM

nnamdifrankie · 2020-09-29T14:13:10Z

x-pack/plugins/security_solution/common/endpoint/index_data.ts

@@ -98,3 +198,262 @@ async function indexAlerts(
    await client.bulk({ body, refresh: true });
  }
 }
+


@paul-tavares this is a significant code change to the core class. I think it may be good to separate this code out. And from a cursory look it may be difficult to maintain and change. Perhaps a follow up PR with that view in mind. Because if there is any changes in the system we have to change this code also.

@nnamdifrankie Agreed - some of this needs to be broken out into smaller set of classes/namespaces. but: could you clarify what class you are referring to? (or did you mean the entire module index_data.ts?)

Also agree that if the system changes, this will have to be updated, but I think it valuable to have the ability to create a fake set of data across the entire system (minus the actual endpoint). Lets disucss if perhaps you feel strongly that we should not be creating this type of "full feature" tooling 😄

also - just FYI: my idea is to ultimately create a FakeAgent class that will allows us to enroll and control agents against fleet without actually having an agent running (just sending messages to fleet - some of that will land in this external tool). The ability to do this will significantly improve our ability to do CI end-to-end testing within Kibana (Functional testing) - especially for our Endpoint (hosts) List page which is driven by a combination of both endpoint metadata and Fleet/Agent data (maybe even create a FakeEndpoint class to mimic Endpoints sending data to elastic).

marshallmain

Looks great overall, just a few questions and type nits

marshallmain · 2020-09-29T14:00:52Z

x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts

-      );
-      // eslint-disable-next-line no-process-exit
-      process.exit(1);
+    const setupResponse = (await kbnClient.request({


👍 kbnClient looks like a much cleaner way to do this

x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts

marshallmain · 2020-09-29T14:08:05Z

x-pack/plugins/security_solution/common/endpoint/index_data.ts

+  agentPolicyId: string
+): Promise<undefined | PostAgentEnrollResponse['item']> => {
+  // Get Enrollement key for host's applied policy
+  const enrollmentApiKey = await kbnClient


Could consider making this request.then.then.catch its own function called getEnrollmentApiKey or similar

x-pack/plugins/security_solution/common/endpoint/index_data.ts

…t-data-generator-create-policy

kevinlog · 2020-09-30T14:53:17Z

x-pack/plugins/security_solution/common/endpoint/index_data.ts

@@ -32,12 +32,11 @@ import {
 } from '../../../ingest_manager/common';
 import { factory as policyConfigFactory } from './models/policy_config';
 import { HostMetadata } from './types';
+import { KbnClientWithApiKeySupport } from '../../scripts/endpoint/resolver_generator_script';


This import from resolver_generator_script.ts is causing a cyclic dependency since that file also imports from here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts#L13

kevinlog · 2020-09-30T14:57:59Z

x-pack/plugins/security_solution/scripts/endpoint/resolver_generator_script.ts

+  PostIngestSetupResponse,
+} from '../../../ingest_manager/common/types/rest_spec';
+
+export class KbnClientWithApiKeySupport extends KbnClient {


if you move this to another file and import it from the files you need it, you should get rid of the cyclic dependency

kibanamachine · 2020-09-30T20:01:29Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 369318c

Metrics [docs]

distributable file count

id	value	diff	baseline
`default`	45807	+1	45806

History

💔 Build #78308 failed 575282e
💔 Build #78000 failed 56d2c99
💚 Build #77620 succeeded 321546a
💔 Build #77460 failed 02d122f
💛 Build #76728 was flaky 2f4d043

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…aly-detection-partition-field * 'master' of github.com:elastic/kibana: (37 commits) Fixes for the Ticket 78375 (elastic#79004) [Security] Alert Telemetry for the Security app (elastic#77200) [Search bar] Remove duplicate `popoverProps` (elastic#79025) [Security Solution][Detections] Add rule overrides for single event EQL rules (elastic#78876) [SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest (elastic#74305) remove file accidentally checked in (elastic#79005) [ML] DF Analytics creation wizard: replace select input with job type cards with icons (elastic#78872) [Design] A couple fixes for 7.10 (elastic#78801) Fix KQL autocomplete value suggestions (elastic#78676) [Security Solution][Resolver] New mock with cursor (elastic#78863) Embeddables: basic documentation (elastic#78900) [security solution] only import beat_schema when needed (elastic#78708) [Reporting] API Integration tests: fix flaky tests for Spaces CSV formatting (elastic#78849) [Actions] Adds a "Test Connector" button on the Connectors List to make discovery of the Test tab easier (elastic#78746) [Discover] Fix functional time picker test permissions (elastic#78564) [ML] Fixing module datafeed overrides (elastic#78925) Adds some missing licenses to the CSV export (elastic#78719) [dev/cli] ensure plugins/ and all watch source dirs exist (elastic#78973) [Lens] Stop using scripted metric to collect telemetry (elastic#78687) [Lens] fix wrong message in fields accordion (elastic#78924) ...

… also integrate with Ingest (#74305) (#79028) * Endpoint generator connects host with a real policy and enrolls agent Co-authored-by: Paul Tavares <paul.tavares@elastic.co> Co-authored-by: kevinlog <kevin.logan@elastic.co> Co-authored-by: Candace Park <candace.park@elastic.co>

paul-tavares self-assigned this Aug 4, 2020

paul-tavares mentioned this pull request Sep 3, 2020

Task/endpointlist actions #76555

Merged

7 tasks

paul-tavares and others added 11 commits September 17, 2020 16:54

Create Policies for each generated host

293eeaf

Refactor Ingest setup to also setup Fleet

2b5ab65

Rename prop name

ad39d54

Add generic response type to KbnClient.request + support for headers

c2aea57

first attempt at adding fleet agent registration

b71e438

a little closer with fleet integration

73a0754

SUCCESS. Able to enroll agent and set it to online

a8221bb

update names to be policy

e3ef509

policy generator has advanced types in endpoint confit

4deebbf

use KbnClientWithRequestAPI

0e61d00

fix typecheck errors

dfb9627

parkiino force-pushed the task/endpoint-host-data-generator-create-policy branch from ead9c3a to dfb9627 Compare September 17, 2020 20:55

parkiino added 3 commits September 17, 2020 18:13

add flag to skip fleet

76db23b

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

a6b6d3f

…t-data-generator-create-policy

fix tests

848eec5

parkiino marked this pull request as ready for review September 21, 2020 17:01

parkiino requested review from a team as code owners September 21, 2020 17:01

parkiino added Feature:Endpoint Elastic Endpoint feature release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Management v7.10.0 v8.0.0 labels Sep 21, 2020

kevinlog reviewed Sep 21, 2020

View reviewed changes

parkiino added 9 commits September 22, 2020 15:53

remove advanced fields

02cd8fa

another advanced removal

58ef6d5

revert experimental change

7b82e05

shorten urls that use requestWithApiKey

e8ead7f

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

51730fa

…t-data-generator-create-policy

remove excess quaotes

65596ec

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

047925b

…t-data-generator-create-policy

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

c6ee44e

…t-data-generator-create-policy

fix ci errors

2f4d043

spalger approved these changes Sep 23, 2020

View reviewed changes

parkiino added 3 commits September 28, 2020 11:00

agent enrollment fix

c3d9282

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

02d122f

…t-data-generator-create-policy

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

321546a

…t-data-generator-create-policy

paul-tavares commented Sep 28, 2020

View reviewed changes

kevinlog approved these changes Sep 29, 2020

View reviewed changes

nnamdifrankie reviewed Sep 29, 2020

View reviewed changes

marshallmain reviewed Sep 29, 2020

View reviewed changes

marshall's changes + edits to hostname

56d2c99

marshallmain approved these changes Sep 29, 2020

View reviewed changes

parkiino added 2 commits September 30, 2020 10:36

fetch kibana version

edd7bde

Merge remote-tracking branch 'upstream/master' into task/endpoint-hos…

575282e

…t-data-generator-create-policy

kevinlog reviewed Sep 30, 2020

View reviewed changes

remove circular dependency

369318c

parkiino merged commit 15e7623 into elastic:master Sep 30, 2020

parkiino deleted the task/endpoint-host-data-generator-create-policy branch September 30, 2020 20:08

parkiino mentioned this pull request Sep 30, 2020

[7.x] [SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest (#74305) #79028

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

paul-tavares commented Aug 4, 2020 •

edited by parkiino

Loading

kevinlog commented Aug 21, 2020

elasticmachine commented Sep 21, 2020

elasticmachine commented Sep 21, 2020

kevinlog Sep 21, 2020

parkiino Sep 21, 2020

kevinlog Sep 21, 2020

spalger left a comment

paul-tavares left a comment

paul-tavares Sep 28, 2020

paul-tavares Sep 28, 2020

parkiino Sep 28, 2020

kevinlog left a comment

nnamdifrankie Sep 29, 2020

paul-tavares Sep 30, 2020

marshallmain left a comment

marshallmain Sep 29, 2020

marshallmain Sep 29, 2020

kevinlog Sep 30, 2020 •

edited

Loading

kevinlog Sep 30, 2020

kibanamachine commented Sep 30, 2020

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

[SECURITY_SOLUTION][ENDPOINT] Improve Endpoint Host data generator to also integrate with Ingest #74305

Conversation

paul-tavares commented Aug 4, 2020 • edited by parkiino Loading

Summary

TODO

Fleet showing an Agent that was created via the Endpoint data generator:

Checklist

For maintainers

kevinlog commented Aug 21, 2020

elasticmachine commented Sep 21, 2020

elasticmachine commented Sep 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spalger left a comment

Choose a reason for hiding this comment

paul-tavares left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinlog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marshallmain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinlog Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Sep 30, 2020

💚 Build Succeeded

Metrics [docs]

distributable file count

History

paul-tavares commented Aug 4, 2020 •

edited by parkiino

Loading

kevinlog Sep 30, 2020 •

edited

Loading