[APM] Additional data telemetry changes #71112

smith · 2020-07-08T15:59:20Z

Add a date range of now-1d to the cloud query
Add a timeout of 5m to all queries (we'll investigate using async queries to improve this in the future.)
Factor out the date range filter into a variable
Fix a bug with the indices_stats tasks when it doesn't return data
Delete the merge mapping script and update instructions
Add a schema property to the usage collector
Add a complete JSON snapshot of our mapping
Add cardinality for client.geo.country_iso_code

elasticmachine · 2020-07-08T15:59:24Z

Pinging @elastic/apm-ui (Team:apm)

dgieselaar · 2020-07-08T16:32:00Z

@smith I'm a bit worried about 10s. I expect some of the tasks to take significantly longer for our bigger users. We are also storing took for the tasks if I remember correctly. You could have a look at the telemetry data to see whether 10s is a good default.

smith · 2020-07-08T19:54:46Z

@smith I'm a bit worried about 10s. I expect some of the tasks to take significantly longer for our bigger users. We are also storing took for the tasks if I remember correctly. You could have a look at the telemetry data to see whether 10s is a good default.

There were a few that were over 350s, and you were right, many in the 30s range. I'm going to set it to 5m.

* Add a date range of `now-1d` to the cloud query * Add a timeout of 5m to all queries (we'll investigate using async queries to improve this in the future.) * Factor out the date range filter into a variable * Fix a bug with the `indices_stats` tasks when it doesn't return data * Update the merge mapping script to create a migration file

x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts

sorenlouv

looks good!

dgieselaar · 2020-07-09T15:44:15Z

@smith can you add telemetry for aggregated transactions? I'd like to record the number of metric documents we can expect with the current implementation. We can simulate this by requesting data for 1m, adding a composite aggregation on a number of fields and counting the number of buckets. The resulting count will be an approximation of the amount of metric documents that will be created. We should record both the expected metric document count plus the transaction count for that time range.

I'd like to record several combinations:

current_implementation: observer.name, transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name, user_agent.name
no_observer_name: transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name, user_agent.name
no_rum: observer.name, transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name (add a filter to exclude RUM data)
no_rum_no_observer_name: observer.name, transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name (add a filter to exclude RUM data)
only_rum: observer.name, transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name, user_agent.name (add a filter to exclude all non-RUM data)
only_rum_no_observer_name: transaction.name, transaction.result, transaction.type, agent.name, service.environment, service.version, host.hostname, container.id, kubernetes.pod.name, user_agent.name (add a filter to exclude all non-RUM data)

I'd also like to record the cardinality of client.geo.country_iso_code separately.

For an example of how to create the composite aggregation and paginate through the buckets, see scripts/aggregate-latency-metrics/index.ts.

smith · 2020-07-09T22:27:58Z

@dgieselaar I should be able to add all of that stuff on Monday but I'll do it in another PR.

TinaHeiligers

I pulled the code and ran the telemetry checker on the collector schema. Unfortunately, I'm getting type errors from the typescript parser that I can't seem to work around. I've asked the team to pitch in and help get this over the finish line.

afharo

LGTM with some NITs re the schema property 👍

afharo · 2020-07-13T11:54:50Z

x-pack/plugins/apm/dev_docs/telemetry.md

@@ -55,15 +55,9 @@ The mapping for the telemetry data is here under `stack_stats.kibana.plugins.apm

 The mapping used there can be generated with the output of the [`getTelemetryMapping`](../common/apm_telemetry.ts) function.

-To make a change to the mapping, edit this function, run the tests to update the snapshots, then use the `merge_telemetry_mapping` script to merge the data into the telemetry repository.
+The `schema` property of the `makeUsageCollector` call in the [`createApmTelemetry` function](../server/lib/apm_telemetry/index.ts) contains the output of `getTelemetryMapping`. Pull requests with changes to the schema should automatically notify the Telemetry team so they can update the mapping in the telemetry clusters.


NIT: In order for the Kibana Telemetry team to be notified, devs need to run node scripts/telemetry_check.js --fix. It will update the file containing all the schemas (and owned by the Telemetry team).

At the moment, APM is excluded so it doesn't fail (

kibana/x-pack/.telemetryrc.json

Line 7 in 637a0d9

"plugins/apm/server/lib/apm_telemetry/index.ts",

) and listed in this issue as pending to be fixed: #70180

I tried with removing the exclusion but it fails.
For progress, I'd say we merge this PR and tackle this issue separately 👍

It looks like the reason it's failing is because file with the usage collector call is parsed directly so the whole schema has to be there as a literal, whereas we're calling a function schema: getApmTelemetryMapping(), which it fails to parse as a valid schema.

@smith there are two other issues I've found thus far while trying to resolve the telemetry-check failure:

The schema needs to be what's displayed in the doc (as types, not actual values) and that the usage collector needs to be explicitly typed.

Here's the schema I got from your mapping

export const apmTelemetrySchema = { agents: { dotnet: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, go: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, java: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, 'js-base': { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, nodejs: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, python: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, ruby: { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, 'rum-js': { agent: { version: { type: 'keyword' }, }, service: { framework: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, language: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, runtime: { composite: { type: 'keyword' }, name: { type: 'keyword' }, version: { type: 'keyword' }, }, }, }, }, cloud: { availability_zone: { type: 'keyword' }, provider: { type: 'keyword' }, region: { type: 'keyword' }, }, counts: { agent_configuration: { all: { type: 'long' }, }, error: { '1d': { type: 'long' }, all: { type: 'long' }, }, max_error_groups_per_service: { '1d': { type: 'long' }, }, max_transaction_groups_per_service: { '1d': { type: 'long' }, }, metric: { '1d': { type: 'long' }, all: { type: 'long' }, }, onboarding: { '1d': { type: 'long' }, all: { type: 'long' }, }, services: { '1d': { type: 'long' }, }, sourcemap: { '1d': { type: 'long' }, all: { type: 'long' }, }, span: { '1d': { type: 'long' }, all: { type: 'long' }, }, traces: { '1d': { type: 'long' }, }, transaction: { '1d': { type: 'long' }, all: { type: 'long' }, }, }, cardinality: { user_agent: { original: { all_agents: { '1d': { type: 'long' }, }, rum: { '1d': { type: 'long' }, }, }, }, transaction: { name: { all_agents: { '1d': { type: 'long' }, }, rum: { '1d': { type: 'long' }, }, }, }, }, has_any_services: { type: 'boolean', }, indices: { all: { total: { docs: { count: { type: 'long' } }, store: { size_in_bytes: { type: 'long' } }, }, }, shards: { total: { type: 'long' } }, }, integrations: { ml: { all_jobs_count: { type: 'long' } }, }, retainment: { error: { ms: { type: 'long' } }, metric: { ms: { type: 'long' } }, onboarding: { ms: { type: 'long' } }, span: { ms: { type: 'long' } }, transaction: { ms: { type: 'long' } }, }, services_per_agent: { dotnet: { type: 'long' }, go: { type: 'long' }, java: { type: 'long' }, 'js-base': { type: 'long' }, nodejs: { type: 'long' }, python: { type: 'long' }, ruby: { type: 'long' }, 'rum-js': { type: 'long' }, }, tasks: { agent_configuration: { took: { ms: { type: 'long' } } }, agents: { took: { ms: { type: 'long' } } }, cardinality: { took: { ms: { type: 'long' } } }, cloud: { took: { ms: { type: 'long' } } }, groupings: { took: { ms: { type: 'long' } } }, indices_stats: { took: { ms: { type: 'long' } } }, integrations: { took: { ms: { type: 'long' } } }, processor_events: { took: { ms: { type: 'long' } } }, services: { took: { ms: { type: 'long' } } }, versions: { took: { ms: { type: 'long' } } }, }, version: { apm_server: { major: { type: 'long' }, minor: { type: 'long' }, patch: { type: 'long' }, }, }, };

(note: there are a few improvements we also need to make to the allowed schema types, including allowing ignore_above for keyword fields)
2. The usageCollector is complaining of a missing type, even though type: "apm" is there. I think it's because the collector isn't typed explicitly but we'll figure all these issues out in #70180

smith · 2020-07-14T05:20:25Z

@smith can you add telemetry for aggregated transactions?

@dgieselaar yes, but opened #71593 to do it in a separate PR.

kibanamachine · 2020-07-14T06:49:38Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 331ea8a

Build metrics

✅ unchanged

History

💔 Build #61127 failed 6477c06
💔 Build #61111 failed 8fde7b6
💚 Build #60494 succeeded 81bce44
💚 Build #60457 succeeded 6982972
💚 Build #60428 succeeded 374d6a3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Add a date range of `now-1d` to the cloud query * Add a timeout of 5m to all queries (we'll investigate using async queries to improve this in the future.) * Factor out the date range filter into a variable * Fix a bug with the `indices_stats` tasks when it doesn't return data * Update the merge mapping script to create a migration file

* master: (21 commits) [Maps] 7.9 design improvements (elastic#71563) [ML] Changing all calls to ML endpoints to use internal user (elastic#70487) [eventLog] prevent log writing when initialization fails (elastic#71339) [Observability] landing page always being displayed (elastic#71494) [IM] Address data stream copy feedback (elastic#71615) [Logs UI] Anomalies page dataset filtering (elastic#71110) [data.search.aggs] Remove `use_field_mapping` from top hits agg (elastic#71168) [ML] Anomaly swim lane embeddable navigation and filter actions (elastic#71082) Fixes typo in siem_cloudtrail job description (elastic#71569) Require granted API Keys to have a name (elastic#71623) Update getUsageForCollection (elastic#71609) Only fetch saved elements once (elastic#71310) [SecuritySolution][Resolver] Adding siem index and guarding process ancestry (elastic#71570) [APM] Additional data telemetry changes (elastic#71112) [Visualize] Fix export table for table export links (elastic#71249) [Search] Server side search API (elastic#70446) use inclusive language (elastic#71607) [Security Solution] Hide timeline footer when Resolver is open (elastic#71516) [Index template wizard] Remove shadow and use border for components panels (elastic#71606) [ML] Kibana API endpoint for histogram chart data (elastic#70976) ...

smith requested a review from a team as a code owner July 8, 2020 15:59

botelastic bot added the Team:APM All issues that need APM UI Team support label Jul 8, 2020

smith added v7.9.0 release_note:skip Skip the PR/issue when compiling release notes labels Jul 8, 2020

smith force-pushed the nls/bounded-tasks branch from 4c6fae0 to 9de1027 Compare July 8, 2020 20:46

smith force-pushed the nls/bounded-tasks branch from 9de1027 to 043c722 Compare July 8, 2020 20:47

Merge remote-tracking branch 'upstream/master' into nls/bounded-tasks

1ad64e3

sorenlouv approved these changes Jul 9, 2020

View reviewed changes

sorenlouv reviewed Jul 9, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts Show resolved Hide resolved

sorenlouv reviewed Jul 9, 2020

View reviewed changes

x-pack/plugins/apm/server/lib/apm_telemetry/collect_data_telemetry/tasks.ts Outdated Show resolved Hide resolved

sorenlouv approved these changes Jul 9, 2020

View reviewed changes

smith changed the title ~~[APM] Add more bounds to telemetry tasks~~ [APM] Additional data telemetry changes Jul 9, 2020

smith added 7 commits July 9, 2020 13:08

Merge remote-tracking branch 'upstream/master' into nls/bounded-tasks

924d913

update serializer

99c8691

Use full path for mapping snapshot

5e403b9

Cloud properties

374d6a3

remove script; update docs; add schema

0786190

Merge remote-tracking branch 'upstream/master' into nls/bounded-tasks

6982972

Updates based on Søren's feedback

81bce44

smith requested a review from a team July 9, 2020 22:28

TinaHeiligers reviewed Jul 9, 2020

View reviewed changes

afharo approved these changes Jul 13, 2020

View reviewed changes

smith added 2 commits July 13, 2020 13:46

Fix test name; add country code cardinality

a092d45

Merge remote-tracking branch 'upstream/master' into nls/bounded-tasks

1246d52

smith added 5 commits July 13, 2020 16:18

Docs update

8fde7b6

More stuff

ec33c3b

More stuff

6477c06

Merge remote-tracking branch 'upstream/master' into nls/bounded-tasks

0ff1adb

snapshot update

331ea8a

smith mentioned this pull request Jul 14, 2020

APM telemetry for aggregated transactions #71593

Closed

smith mentioned this pull request Jul 14, 2020

[APM] Aggregated transaction telemetry #71594

Merged

smith merged commit c6705e8 into elastic:master Jul 14, 2020

smith deleted the nls/bounded-tasks branch July 14, 2020 13:16

smith mentioned this pull request Jul 14, 2020

[7.x] [APM] Additional data telemetry changes (#71112) #71634

Merged

smith mentioned this pull request Jul 21, 2020

Add schema to APM data telemetry #72736

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Additional data telemetry changes #71112

[APM] Additional data telemetry changes #71112

smith commented Jul 8, 2020 •

edited

Loading

elasticmachine commented Jul 8, 2020

dgieselaar commented Jul 8, 2020

smith commented Jul 8, 2020

sorenlouv left a comment

dgieselaar commented Jul 9, 2020 •

edited

Loading

smith commented Jul 9, 2020

TinaHeiligers left a comment

afharo left a comment

afharo Jul 13, 2020

smith Jul 13, 2020

TinaHeiligers Jul 13, 2020 •

edited

Loading

smith commented Jul 14, 2020

kibanamachine commented Jul 14, 2020

[APM] Additional data telemetry changes #71112

[APM] Additional data telemetry changes #71112

Conversation

smith commented Jul 8, 2020 • edited Loading

elasticmachine commented Jul 8, 2020

dgieselaar commented Jul 8, 2020

smith commented Jul 8, 2020

sorenlouv left a comment

Choose a reason for hiding this comment

dgieselaar commented Jul 9, 2020 • edited Loading

smith commented Jul 9, 2020

TinaHeiligers left a comment

Choose a reason for hiding this comment

afharo left a comment

Choose a reason for hiding this comment

afharo Jul 13, 2020

Choose a reason for hiding this comment

smith Jul 13, 2020

Choose a reason for hiding this comment

TinaHeiligers Jul 13, 2020 • edited Loading

Choose a reason for hiding this comment

smith commented Jul 14, 2020

kibanamachine commented Jul 14, 2020

💚 Build Succeeded

Build metrics

History

smith commented Jul 8, 2020 •

edited

Loading

dgieselaar commented Jul 9, 2020 •

edited

Loading

TinaHeiligers Jul 13, 2020 •

edited

Loading