[APM] agent config applied indicator does not work with Fleet integration

[axw]

Kibana version

8.0.0

APM Server version (if applicable)

8.0.0

Elasticsearch version (if applicable)

No response

Steps to Reproduce

1. Install APM Fleet integration
2. Run an instrumented service
3. Define agent config for that service
4. Wait until agent config has been received (tail agent logs)

Expected Behavior

UI indicates agent config has been applied.

Actual Behavior

UI indicates agent config is not yet applied.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[axw]

When running as a Fleet integration, agent config is delivered to APM Server rather than APM Server actively querying Kibana for the config. Thus, we cannot actively mark agent config as having been applied, as we did in the past. Instead, APM Server produces agent_config metric documents which the UI should search for.

The docs have the following fields that may be used:

• metricset.name: agent_config
• labels.etag: <agent_config_etag>

The value of labels.etag corresponds to the etag value that is passed by the APM UI down through the Fleet policy.

[nicpenning]

Thanks for posting this issue. This single bug has halted our POV with APM and has caused much frustration with testing.

Even though the icon doesn't change, does this mean the config changes are still not applying? From our experience, the changes made in Kibana do not get sent to the agent in the application to enable, for example, request body data.

[axw]

@nicpenning sorry about the understandable frustration.

Even though the icon doesn't change, does this mean the config changes are still not applying?

Yes. We have tests that show agent config works when using Fleet - it's just that this doesn't reflect in the indicator.

From our experience, the changes made in Kibana do not get sent to the agent in the application to enable, for example, request body data.

Would you mind opening a post at https://discuss.elastic.co/c/observability/apm/58 with the steps you took? Maybe there is some other bug that we're not aware of.

[nicpenning]

Yes, I can do that. Thanks!

[gbamparop]

@axw how the configs applied by Jaeger agent will be handled? (see elastic/apm-server#3579).

Also, do you think we should add a timerange filter as mentioned here?

[gbamparop]

@axw how the configs applied by Jaeger agent will be handled? (see elastic/apm-server#3579).

Also, do you think we should add a timerange filter as mentioned here?

cc @elastic/apm-server

[axw]

@gbamparop

how the configs applied by Jaeger agent will be handled? (see elastic/apm-server#3579).

We immediately mark config as applied when it is queried by a Jaeger client. This is slightly different to what we do for Elastic APM agents, where we mark the config as applied when the agent queries the config a second time (as identified by its Etag), at which point we know it has already been successfully applied by the agent.

Either way though, there should be no difference to the UI: once there's an agent_config metricset, it's applied.

Also, do you think we should add a timerange filter as mentioned here?

We'll definitely want a time range filter, as this is an append-only data stream which will be updated every 30 seconds. So filter on the last minute? If there's no active APM Server(s), then there would be no agent_config metrics. In that case I think we can either just show the config as not applied, or have some ? indicator.

[dgieselaar]

@axw is the etag value available in agent_config metrics for Jaeger agents? We use that value to join agent_config metrics w/ the agent configuration documents on the Kibana side.

[axw]

@dgieselaar yes, see #123109 (comment)

EDIT: sorry just saw you specifically asked "for Jaeger agents". Yes, it'll be available for them as well.

[dgieselaar]

@axw great, we should be set then, correct @gbamparop?

[gbamparop]

@axw great, we should be set then, correct @gbamparop?

Yes that's correct as long as it's being set.

[gbamparop]

We'll definitely want a time range filter, as this is an append-only data stream which will be updated every 30 seconds. So filter on the last minute? If there's no active APM Server(s), then there would be no agent_config metrics.

Thanks, I'll update the time range to the last minute then.

[nicpenning]

Yes, I can do that. Thanks!

Since upgrading to 8.2.0 everything with APM broke so we are working on fixing this before we can talk about the agent configuration issues.