[Ingest Manager] Shared Fleet agent policy action #76013

nchaulet · 2020-08-26T18:14:40Z

Summary

Currently during a config change (or during the first checkin) we create one saved object for each agent, this PR introduce a new concept of PolicyAction that can be shared between agents.

Done in this PR

introduce AgentPolicyAction an action that is shared between all agent assigned to a policy
add some cache to fetch action in the acknowledge handler
introduce ack_data to avoid fetching the whole config in the acknowledge handler.

Details

We now have two kind of action:
AgentAction that target individual agent
AgentPolicyAction that target all the agent assigned to a policy (support only the CONFIG_CHANGE action for now)

Also I introduce ack_data that are data useful to acknowledge an action, this allow to avoid decrypting the whole config every-time an agent acknowledge an action.

This PR introduce a lot of new types, let me know if you have better naming suggestions.
Also I did some cleaning by removing the action type we do not support.

Performance impact

While enrolling 200 agents, than performing a checkin and a ack you can notice the first checkin checkin_first_time_duration is a lot faster

Before

ack_duration..................: avg=2.11s    min=2.05s med=2.09s max=2.28s  p(90)=2.23s  p(95)=2.28s 
checkin_first_time_duration...: avg=2.24s    min=2.06s med=2.09s max=3.09s  p(90)=3.06s  p(95)=3.09s

After

ack_duration..................: avg=2.02s   min=2.01s med=2.02s max=2.04s  p(90)=1.04s p(95)=2.04s
checkin_first_time_duration...: avg=1.14s    min=1.01s    med=1.04s max=2.14s  p(90)=1.17s p(95)=2.12s

…-policy-action

nchaulet · 2020-08-31T13:34:18Z

x-pack/plugins/ingest_manager/common/types/models/agent.ts

@@ -21,28 +21,53 @@ export type AgentStatus =
  | 'unenrolling'
  | 'degraded';

-export type AgentActionType = 'CONFIG_CHANGE' | 'DATA_DUMP' | 'RESUME' | 'PAUSE' | 'UNENROLL';
+export type AgentActionType = 'CONFIG_CHANGE' | 'UNENROLL';


I did some cleanup as currently we only support this two actions

…-policy-action

elasticmachine · 2020-08-31T20:26:23Z

Pinging @elastic/ingest-management (Team:Ingest Management)

nchaulet · 2020-09-01T13:09:52Z

@elasticmachine merge upstream

ph · 2020-09-01T15:05:47Z

Not sure why I was assigned to this. I blame zube.

…-policy-action

nchaulet · 2020-09-09T18:58:59Z

@EricDavisX This one introduce a lot of change and it's probably good candidate for some manual QA :)

EricDavisX · 2020-09-09T20:31:31Z

@rahulgupta-qasource - can you pull Nicolas' fork/branch above and run Kibana in source with latest 8.0 / master ES and Agent SNAPSHOT builds and run some tests please?

Nicolas and I discussed changes and want to target the tests as such:

Set up 2 persistent Windows, and 1 persistent Linux Agent & Endpoint and see them call in to the default config.

Then create a new config and change them all to that, without Endpoint.
Then create 3 new configs and change them each to a new one, with Endpoint as fast as you can in the UI (use 2 browsers or enlist a friend to login to Kibana to help!)
Then re-start all the hosts at the same time and see them all come back on line.
Then unenroll all 3 and see them go off-line
then re-enroll them all again to different Agent policies.

Note that viewing the Agent in Fleet and seeing it on-line and config changes is enough validation, it doesn't require a deep validation for now.

Then we can do some interesting tests, to try harder to break it:

config-changes when the agent is unreachable / off-line (do you have a set way to easily simulate that?). It should pick up the change when its comes on line
test when multiple config changes come in near the same time (maybe 2 different Admins make changes to the same policy, which is a bad idea, but it can mimic real world potential.
any other creative nuance you can think of, anything based on our test suites and how config changes are validated.

Let us know how it goes, thank you in advance.

ghost · 2020-09-10T08:35:16Z

Hi @EricDavisX

Thank you for the update.

Could you please share the environment w.r.t above Nicolas fork/branch to validate this ticket.

Moreover, Error 'Unable to initialize Ingest Manager: Bad Request' is displayed on clicking 'Ingest Manager' link on 8.0.0-SNAPSHOT Kibana. We have reported bug #77133 for the same.

Please let us know if anything is missing from our end.

EricDavisX · 2020-09-10T13:15:33Z

I'll discuss testing with Rahul offline and we'll confirm back.

neptunian · 2020-09-10T15:56:25Z

x-pack/plugins/ingest_manager/server/services/agents/actions.ts

@@ -15,12 +22,37 @@ export async function createAgentAction(
  soClient: SavedObjectsClientContract,
  newAgentAction: Omit<AgentAction, 'id'>
 ): Promise<AgentAction> {
-  const so = await soClient.create<AgentActionSOAttributes>(AGENT_ACTION_SAVED_OBJECT_TYPE, {
+  return createAction(soClient, newAgentAction);


is it possible to overload createAgentAction or have it accept different types instead of having to create the new functions createAgentPolicyAction and createAction?

jfsiii

Nice overview in the description and the code is clear. I left some comments and questions but nothing to prevent a 🚀

x-pack/plugins/ingest_manager/common/types/models/agent.ts

x-pack/plugins/ingest_manager/server/routes/agent/actions_handlers.ts

x-pack/plugins/ingest_manager/server/services/agent_policy.ts

x-pack/plugins/ingest_manager/server/services/agents/actions.ts

EricDavisX · 2020-09-10T23:43:14Z

I'll discuss testing with Rahul offline and we'll confirm back.
Rahul is not able to easily (quickly) stand up Kibana from source to test this, and I don't have bandwidth to easily help cover it. So we can do some testing prior and some testing after merge. Please do cite what you can cover and we'll manage the rest after check-in I guess?

…-policy-action

kibanamachine · 2020-09-14T00:32:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ad6a42c

Build metrics

Saved Objects .kibana field count

id	value	diff	baseline
fleet-agent-actions	9	+3	6

History

💔 Build #73902 failed 664dc85
💚 Build #73252 succeeded 664dc85
💔 Build #71400 failed 82009b04e988338735914ff6d377314acdfdd63e
💔 Build #71377 failed 1b7f3ef133c47ec6124a8166210e38fd8324cbc0
💔 Build #71251 failed 7bc681e35f6a5f21eea79cb4c500361f7d6bb8f3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…s-for-710 * 'master' of github.com:elastic/kibana: (65 commits) Separate url forwarding logic and legacy services (elastic#76892) Bump yargs-parser to v13.1.2+ (elastic#77009) [Ingest Manager] Shared Fleet agent policy action (elastic#76013) [Search] Re-add support for aborting when a connection is closed (elastic#76470) [Search] Remove long-running query pop-up (elastic#75385) [Monitoring] Fix UI error when alerting is not available (elastic#77179) do not log plugin id format warning in dist mode (elastic#77134) [ML] Improving client side error handling (elastic#76743) [Alerting][Connectors] Refactor IBM Resilient: Generic Implementation (phase one) (elastic#74357) [Docs] some basic searchsource api docs (elastic#77038) add cGroupOverrides to the legacy config (elastic#77180) Change saved object bulkUpdate to work across multiple namespaces (elastic#75478) [Security Solution][Resolver] Replace Selectable popover with badges (elastic#76997) Removing ml-state index from archive (elastic#77143) [Security Solution] Add unit tests for histograms (elastic#77081) [Lens] Filters aggregation (elastic#75635) [Enterprise Search] Update WS Overview logic to use new config data (elastic#77122) Cleanup type output before building new types (elastic#77211) [Security Solution] Use safe type in resolver backend (elastic#76969) Use proper lodash syntax (elastic#77105) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/node_allocation.tsx

EricDavisX · 2020-09-14T19:32:21Z

we're running the testing now in a separate test ticket, just fyi.

nchaulet added 3 commits August 26, 2020 13:40

[Ingest Manager] Shared policy action

f4a5fca

Fix ack

cd188cb

Merge branch 'master' of github.com:elastic/kibana into feature-share…

22aa70b

…-policy-action

nchaulet self-assigned this Aug 31, 2020

nchaulet added Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Aug 31, 2020

nchaulet commented Aug 31, 2020

View reviewed changes

ph self-assigned this Aug 31, 2020

nchaulet force-pushed the feature-share-policy-action branch from 7bc681e to 1b7f3ef Compare August 31, 2020 18:05

nchaulet added 2 commits August 31, 2020 15:28

Refacto types

2d552ac

Merge branch 'master' of github.com:elastic/kibana into feature-share…

ec3cd82

…-policy-action

nchaulet changed the title ~~[Ingest Manager] Shared policy action~~ [Ingest Manager] Shared Fleet agebt policy action Aug 31, 2020

nchaulet force-pushed the feature-share-policy-action branch 2 times, most recently from b8a3bed to 82009b0 Compare August 31, 2020 19:41

nchaulet changed the title ~~[Ingest Manager] Shared Fleet agebt policy action~~ [Ingest Manager] Shared Fleet agent policy action Aug 31, 2020

Create config change action during setup

cd86268

nchaulet force-pushed the feature-share-policy-action branch from 82009b0 to cd86268 Compare August 31, 2020 20:11

nchaulet marked this pull request as ready for review August 31, 2020 20:26

nchaulet requested a review from a team August 31, 2020 20:26

Merge branch 'master' into feature-share-policy-action

2f6121e

ph removed their assignment Sep 1, 2020

nchaulet added 4 commits September 2, 2020 20:48

Merge branch 'master' of github.com:elastic/kibana into feature-share…

a827690

…-policy-action

Fix tests

218832e

Merge branch 'master' of github.com:elastic/kibana into feature-share…

e191112

…-policy-action

Fix tests

4f73397

nchaulet added 3 commits September 8, 2020 13:21

Merge branch 'master' of github.com:elastic/kibana into feature-share…

032ebf3

…-policy-action

Fix tests

fe9d42a

Fix tests

664dc85

neptunian reviewed Sep 10, 2020

View reviewed changes

jfsiii approved these changes Sep 10, 2020

View reviewed changes

nchaulet added 2 commits September 13, 2020 18:01

Merge branch 'master' of github.com:elastic/kibana into feature-share…

c8bcabb

…-policy-action

Fix after review

ad6a42c

nchaulet merged commit 61951a5 into elastic:master Sep 14, 2020

nchaulet deleted the feature-share-policy-action branch September 14, 2020 01:09

nchaulet mentioned this pull request Sep 14, 2020

[7.x] [Ingest Manager] Shared Fleet agent policy action (#76013) #77297

Merged

nchaulet added a commit to nchaulet/kibana that referenced this pull request Sep 14, 2020

[Ingest Manager] Shared Fleet agent policy action (elastic#76013)

a9c7f39

nchaulet added a commit that referenced this pull request Sep 14, 2020

[Ingest Manager] Shared Fleet agent policy action (#76013) (#77297)

23eaf58

nchaulet mentioned this pull request Sep 21, 2020

[Ingest Manager] Fix agent action acknowledgement #78089

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Shared Fleet agent policy action #76013

[Ingest Manager] Shared Fleet agent policy action #76013

nchaulet commented Aug 26, 2020 •

edited

Loading

nchaulet Aug 31, 2020

elasticmachine commented Aug 31, 2020

nchaulet commented Sep 1, 2020

ph commented Sep 1, 2020

nchaulet commented Sep 9, 2020

EricDavisX commented Sep 9, 2020

ghost commented Sep 10, 2020

EricDavisX commented Sep 10, 2020

neptunian Sep 10, 2020

jfsiii left a comment

EricDavisX commented Sep 10, 2020

kibanamachine commented Sep 14, 2020

EricDavisX commented Sep 14, 2020

[Ingest Manager] Shared Fleet agent policy action #76013

[Ingest Manager] Shared Fleet agent policy action #76013

Conversation

nchaulet commented Aug 26, 2020 • edited Loading

Summary

Details

Performance impact

Before

After

nchaulet Aug 31, 2020

Choose a reason for hiding this comment

elasticmachine commented Aug 31, 2020

nchaulet commented Sep 1, 2020

ph commented Sep 1, 2020

nchaulet commented Sep 9, 2020

EricDavisX commented Sep 9, 2020

ghost commented Sep 10, 2020

EricDavisX commented Sep 10, 2020

neptunian Sep 10, 2020

Choose a reason for hiding this comment

jfsiii left a comment

Choose a reason for hiding this comment

EricDavisX commented Sep 10, 2020

kibanamachine commented Sep 14, 2020

💚 Build Succeeded

Build metrics

Saved Objects .kibana field count

History

EricDavisX commented Sep 14, 2020

nchaulet commented Aug 26, 2020 •

edited

Loading