[APM] Syncs agent config settings to APM Fleet policies #100744

ogupte · 2021-05-26T21:25:49Z

Synchronizes all agent configuration settings with existing APM fleet policies whenever any user creates/updates any agent configuration via API endpoint PUT /api/apm/settings/agent-configuration.

Can be tested with apm-integration-testing python3 ./scripts/compose.py start master --with-elastic-agent --apm-server-managed --all. Update an agent confuration in APM UI, then inspect the response from http://localhost:5601/api/fleet/package_policies to verify that the agent configs are updated in the apm package policy object.

elasticmachine · 2021-05-26T21:25:54Z

Pinging @elastic/apm-ui (Team:apm)

axw · 2021-05-27T02:11:01Z

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts

+    const configurations = await listConfigurations({ setup });
+    const agentConfigValue = configurations.map((configuration) => ({
+      service: configuration.service,
+      settings: configuration.settings,


Suggested change

settings: configuration.settings,

config: configuration.settings,

Here's how things look in apm-server currentlly: https://github.com/elastic/apm-server/blob/91645bde2d96a79b943c90253d62da3861ab2997/apm-server.yml#L275-L297

#---------------------------- APM Server - Agent Configuration ---------------------------- # Directly specify agent configuration. If `agent_config` is set, agent # configuration under `kibana` will be ignored. # An agent's incoming configuration request will be matched to an # agent_config with the following precedence: # - service.name and service.environment match an agent_config # - service.name matches an agent_config, service.environment == "" # - service.environment matches an agent_config, service.name == "" # - an agent_config without a name or environment set # An empty result is is returned if no matching result is found. # agent_config: # - service.name: ten_percent # service.environment: production # config: # capture_body: off # capture_body: true # log_level: info # recording: true # transaction_sample_rate: 0.1 # - service.name: frontend # agent.name: rum-js # config: # transaction_sample_rate: 0.1

We could alternatively change the apm-server implementation if you/anyone feels strongly about calling this "settings".

I'm ok changing it to conform with apm-server's implementation. thanks!

axw · 2021-05-27T02:16:18Z

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts

+    const agentConfigValue = configurations.map((configuration) => ({
+      service: configuration.service,
+      settings: configuration.settings,
+    }));


Suggested change

}));

etag: configuration.etag,

'agent.name': configuration.agent_name,

}));

By passing the etag value down, apm-server will (later) be able to use it when recording which config has been applied by agents. The UI can then match this up to the config blocks it knows about.

We also need agent.name for restricting config for untrusted agents (RUM and Jaeger)

…icy objects - update package policy agent_configs to include etag, agent.name, and change settings -> config

cauemarcondes · 2021-05-27T13:58:30Z

@ogupte could you give some instructions about how to test it, please?

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts

stuartnelson3 · 2021-05-27T15:28:31Z

I was running this branch locally, and I have a couple questions. Forgive my ignorance if this has been answered elsewhere before (:grimacing:):

I noticed my agent_config being rendered like this:

The config options are being rendered as strings, but they should be an int and a float. Is this expected?
The agent_config and apm-server blocks are separate. Does agent_config get merged into apm-server when it's pushed? And, should it be nested under apm-server?

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts

cauemarcondes · 2021-05-27T15:47:44Z

x-pack/plugins/apm/server/routes/settings/agent_configuration.ts

+        setup,
+      });
+      logger.info(
+        `Saved latest agent settings to Fleet integration policy for APM.`


Since you're logging something, maybe would be good to have a specific message when it is deleting?

cauemarcondes

LGTM - Nice work @ogupte

ogupte · 2021-05-27T18:28:48Z

I was running this branch locally, and I have a couple questions. Forgive my ignorance if this has been answered elsewhere before (😬):

1. I noticed my `agent_config` being rendered like this:
   ![image](https://user-images.githubusercontent.com/1398104/119853174-3e44a480-bf10-11eb-8424-3385917df860.png)
   The `config` options are being rendered as strings, but they should be an int and a float. Is this expected?

2. The `agent_config` and `apm-server` blocks are separate. Does `agent_config` get merged into `apm-server` when it's pushed? And, should it be nested under `apm-server`?
   ![image](https://user-images.githubusercontent.com/1398104/119853715-bad78300-bf10-11eb-8469-9ea26a048c5d.png)

@stuartnelson3, thanks for checking this out and testing it.

config options are intentionally strings, sort of like how ENV variables are always strings. This was something we decided when we implemented agent configurations in the first place.
While it's technically possible to combine Agent Configs and APM server configs into the same object, they configurations for different purposes, and it the APM server's responsibility to deliver agent configurations to the appropriate agent. If folks working on apm-server preferred them combined, i'll be happy to accommodate. @axw

axw · 2021-05-28T00:56:11Z

config options are intentionally strings, sort of like how ENV variables are always strings. This was something we decided when we implemented agent configurations in the first place.

👍 sorry @stuartnelson3, I missed that we had numbers in our config in the apm-server.yml example. I just checked the code and it's a map[string]string so I think we just need to update the example.

While it's technically possible to combine Agent Configs and APM server configs into the same object, they configurations for different purposes, and it the APM server's responsibility to deliver agent configurations to the appropriate agent. If folks working on apm-server preferred them combined, i'll be happy to accommodate. @axw

I would prefer that everything specific to APM is nested under the apm-server key, so apm-server.agent_configs in this case.

stuartnelson3 · 2021-06-01T09:37:15Z

this ended up being correctly sent to apm-server, but I suppose it'd be nice to have it rendered within the apm-server config block:

diff --git a/x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts b/x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts
index bfc9d360..8ff34d96 100644
--- a/x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts
+++ b/x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts
@@ -106,7 +106,7 @@ export function getPackagePolicyWithAgentConfigurations(
       {
         ...firstInput,
         config: {
-          agent_config: {
+          'apm-server.agent_config': {
             value: agentConfigurations.map((configuration) => ({
               service: configuration.service,
               config: configuration.settings,

in the UI:

EDIT: I'm not sure what's going on with the rendering of span_frames_min_duration, though

ogupte · 2021-06-01T15:52:38Z

@elasticmachine merge upstream

ogupte · 2021-06-02T07:19:09Z

this ended up being correctly sent to apm-server, but I suppose it'd be nice to have it rendered within the apm-server config block:
...

@stuartnelson3 can you confirm that this works: https://github.com/elastic/kibana/pull/100744/files#diff-ab7e62a4c711e2133e13004dec4ecfedd8bb1bf46da87eed88f61bd743b2ffd4R118-R128

ogupte · 2021-06-02T13:48:44Z

@elasticmachine merge upstream

stuartnelson3 · 2021-06-02T13:53:14Z

I started from a fresh ES + new build of kibana off this branch, and when I tried to add the APM server integration I got this error:

cause: SchemaTypeError: definition for this key is missing

looks like static typing is at it again

…-server'] in order to pass validation checks

ogupte · 2021-06-02T20:47:25Z

@stuartnelson3 in order to fix the validation error I had to define a property value for config['apm-server'] within the package policy. Overall the structure looks like this: packagePolicy.inputs[0].config['apm-server'].value.agent_config = Array<{service, etrag, config: {...}}>. I also see there's a compiled_input object in the input, that contains apm-server (packagePolicy.inputs[0].compiled_input['apm-server'] = { api_key, secret_token, host, ... }), but i assume we want to continue to use config. Can you confirm if this schema works for apm-server?

- parallelizes operations for improved performance

stuartnelson3 · 2021-06-03T07:11:03Z

config is correct, cool. I pulled in the changes and no longer get the validation error when creating the integration, but after adding an agent config, I'm not seeing it rendered in the UI:

ogupte · 2021-06-07T07:14:27Z

config is correct, cool. I pulled in the changes and no longer get the validation error when creating the integration, but after adding an agent config, I'm not seeing it rendered in the UI:

@stuartnelson3 It looks like the fleet kibana code which generates that yaml spreads in the contents of config followed by the contents of compiled_input, and since they both have properties named apm-server the latter overwrites the former: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/common/services/package_policies_to_agent_inputs.ts#L34-L38

is apm-server consuming these agent configs with the raw policy object, or only through this output yaml?

stuartnelson3 · 2021-06-07T10:05:47Z

@ogupte I'm not sure, I've been doing manual testing by copying+pasting the output yaml. @axw had surmised this to possibly be the case -- maybe he also knows if the raw policy object is being pushed or not?

cauemarcondes · 2021-06-07T18:11:00Z

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts

+        config: {
+          [APM_SERVER]: {


@ogupte Don't you need to add the others config here to avoid overwriting the entire config object?

Suggested change

config: {

[APM_SERVER]: {

config: {

...firstInput.config,

[APM_SERVER]: {

axw · 2021-06-08T01:38:44Z

is apm-server consuming these agent configs with the raw policy object, or only through this output yaml?

@ogupte apm-server will receive the combined output yaml. Maybe flatten the config string, so it's added like apm-server.agent_config rather than as a nested object structure.

ogupte · 2021-06-08T17:24:43Z

@elasticmachine merge upstream

kibanamachine · 2021-06-08T19:25:12Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #128815 failed 6793d62
💚 Build #128767 succeeded 2d522ce
💚 Build #128639 succeeded 62379ad
💔 Build #128571 failed 6129f4f
💚 Build #128405 succeeded a25497e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [APM] Syncs agent config settings to APM Fleet policies (elastic#95501) * fixes eslint issues * fixes malformed line comment * - consolidated logic that applies agent configurations to package policy objects - update package policy agent_configs to include etag, agent.name, and change settings -> config * Synchronizes agent configs whenever configuration is deleted. * PR feedback * nest agent_config within `apm-server` in the package policy input * nests agent_config under the requried 'value' property of config['apm-server'] in order to pass validation checks * - externalizes getApmPackagePolicies for reusability - parallelizes operations for improved performance Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…01685) * [APM] Syncs agent config settings to APM Fleet policies (#95501) * fixes eslint issues * fixes malformed line comment * - consolidated logic that applies agent configurations to package policy objects - update package policy agent_configs to include etag, agent.name, and change settings -> config * Synchronizes agent configs whenever configuration is deleted. * PR feedback * nest agent_config within `apm-server` in the package policy input * nests agent_config under the requried 'value' property of config['apm-server'] in order to pass validation checks * - externalizes getApmPackagePolicies for reusability - parallelizes operations for improved performance Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (54 commits) Implement "select all" rules feature (elastic#100554) [ML] Remove script fields from the Anomaly detection alerting rule executor (elastic#101607) [Security solutions][Endpoint] Update event filtering texts (elastic#101563) [Enterprise Search] Mocks/tests tech debt - avoid hungry mocking (elastic#101107) [FTR] Updates esArchive paths [FTR] Updates esArchive paths [Security Solution][Detection Engine] Adds runtime field tests (elastic#101664) Added APM PHP agent to the list of agent names (elastic#101062) [CI] Restore old version_info behavior when .git directory is present (elastic#101642) [Fleet] Add fleet server telemetry (elastic#101400) [APM] Syncs agent config settings to APM Fleet policies (elastic#100744) [esArchiver] drop support for --dir, use repo-relative paths instead (elastic#101345) Revert "[xpack/test] restore incremental: false in ts project" [Security Solution] Remove Host Isolation feature flag (elastic#101655) [xpack/test] restore incremental: false in ts project [DOCS] Adds link to video landing page (elastic#101413) [ML] Move Index Data Visualizer into separate plugin (Part 1) (elastic#100922) Improve security plugin return types (elastic#101492) [ts] migrate `x-pack/test` to composite ts project (elastic#101441) [App Search] Updated Search UI to new URL (elastic#101320) ...

[APM] Syncs agent config settings to APM Fleet policies (elastic#95501)

c82917e

ogupte requested a review from a team as a code owner May 26, 2021 21:25

botelastic bot added the Team:APM All issues that need APM UI Team support label May 26, 2021

ogupte added release_note:feature Makes this part of the condensed release notes v7.14.0 labels May 26, 2021

ogupte added 2 commits May 26, 2021 21:02

fixes eslint issues

b2bb676

fixes malformed line comment

9ff99d1

axw reviewed May 27, 2021

View reviewed changes

- consolidated logic that applies agent configurations to package pol…

c95249f

…icy objects - update package policy agent_configs to include etag, agent.name, and change settings -> config

Synchronizes agent configs whenever configuration is deleted.

200b23f

cauemarcondes reviewed May 27, 2021

View reviewed changes

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts Show resolved Hide resolved

cauemarcondes reviewed May 27, 2021

View reviewed changes

x-pack/plugins/apm/server/lib/fleet/register_fleet_policy_callbacks.ts Show resolved Hide resolved

cauemarcondes reviewed May 27, 2021

View reviewed changes

cauemarcondes approved these changes May 27, 2021

View reviewed changes

ogupte added 2 commits May 27, 2021 16:10

PR feedback

007cefc

Merge branch 'master' into apm-95501-agent-config-fleet-sync

49d660a

This was referenced May 28, 2021

update agent_config examples elastic/apm-server#5364

Closed

fix direct agent configuration elastic/apm-server#5380

Merged

kibanamachine and others added 2 commits June 1, 2021 11:52

Merge branch 'master' into apm-95501-agent-config-fleet-sync

a25497e

nest agent_config within apm-server in the package policy input

6129f4f

Merge branch 'master' into apm-95501-agent-config-fleet-sync

62379ad

nests agent_config under the requried 'value' property of config['apm…

2d522ce

…-server'] in order to pass validation checks

- externalizes getApmPackagePolicies for reusability

6793d62

- parallelizes operations for improved performance

cauemarcondes reviewed Jun 7, 2021

View reviewed changes

Merge branch 'master' into apm-95501-agent-config-fleet-sync

aaf5263

ogupte merged commit ff0349d into elastic:master Jun 8, 2021

ogupte mentioned this pull request Jun 8, 2021

[7.x] [APM] Syncs agent config settings to APM Fleet policies (#100744) #101685

Merged

simitt mentioned this pull request Jul 5, 2021

[APM] Inject agent config directly into APM Fleet policies #99193

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Syncs agent config settings to APM Fleet policies #100744

[APM] Syncs agent config settings to APM Fleet policies #100744

ogupte commented May 26, 2021 •

edited

Loading

elasticmachine commented May 26, 2021

axw May 27, 2021

ogupte May 27, 2021

axw May 27, 2021

cauemarcondes commented May 27, 2021

stuartnelson3 commented May 27, 2021

cauemarcondes May 27, 2021

cauemarcondes left a comment

ogupte commented May 27, 2021

axw commented May 28, 2021

stuartnelson3 commented Jun 1, 2021 •

edited

Loading

ogupte commented Jun 1, 2021

ogupte commented Jun 2, 2021

ogupte commented Jun 2, 2021

stuartnelson3 commented Jun 2, 2021 •

edited

Loading

ogupte commented Jun 2, 2021

stuartnelson3 commented Jun 3, 2021

ogupte commented Jun 7, 2021 •

edited

Loading

stuartnelson3 commented Jun 7, 2021

cauemarcondes Jun 7, 2021

axw commented Jun 8, 2021 •

edited

Loading

ogupte commented Jun 8, 2021

kibanamachine commented Jun 8, 2021

	settings: configuration.settings,
	config: configuration.settings,

-    }));
+      etag: configuration.etag,
+      'agent.name': configuration.agent_name,
+    }));

[APM] Syncs agent config settings to APM Fleet policies #100744

[APM] Syncs agent config settings to APM Fleet policies #100744

Conversation

ogupte commented May 26, 2021 • edited Loading

elasticmachine commented May 26, 2021

axw May 27, 2021

Choose a reason for hiding this comment

ogupte May 27, 2021

Choose a reason for hiding this comment

axw May 27, 2021

Choose a reason for hiding this comment

cauemarcondes commented May 27, 2021

stuartnelson3 commented May 27, 2021

cauemarcondes May 27, 2021

Choose a reason for hiding this comment

cauemarcondes left a comment

Choose a reason for hiding this comment

ogupte commented May 27, 2021

axw commented May 28, 2021

stuartnelson3 commented Jun 1, 2021 • edited Loading

ogupte commented Jun 1, 2021

ogupte commented Jun 2, 2021

ogupte commented Jun 2, 2021

stuartnelson3 commented Jun 2, 2021 • edited Loading

ogupte commented Jun 2, 2021

stuartnelson3 commented Jun 3, 2021

ogupte commented Jun 7, 2021 • edited Loading

stuartnelson3 commented Jun 7, 2021

cauemarcondes Jun 7, 2021

Choose a reason for hiding this comment

axw commented Jun 8, 2021 • edited Loading

ogupte commented Jun 8, 2021

kibanamachine commented Jun 8, 2021

💚 Build Succeeded

Metrics [docs]

History

ogupte commented May 26, 2021 •

edited

Loading

stuartnelson3 commented Jun 1, 2021 •

edited

Loading

stuartnelson3 commented Jun 2, 2021 •

edited

Loading

ogupte commented Jun 7, 2021 •

edited

Loading

axw commented Jun 8, 2021 •

edited

Loading