[Usage Collection] [schema] `actions` #78832

afharo · 2020-09-29T17:54:06Z

Summary

Add schema definition to the collector actions.

Using DYNAMIC_KEY as a way to tell when the key can be dynamic, and we can't know the possible values.

Related to #70180.

For maintainers

This was checked for breaking API changes and was labeled appropriately

afharo · 2020-09-29T17:58:24Z

x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

+            "DYNAMIC_KEY": {
+              "type": "long"
+            }


@mindbat for now, when there is a dynamic key with unknown values, schema will be reported with DYNAMIC_KEY. Would that work for you?

I'm afraid I don't understand the question. This is the schema internal to Kibana? So there's no direct correspondence to what gets sent in the object to be indexed?

@mindbat that's not quite true. The schema is a way of indicating what is sent in the object and the field type we can expect to map that as (long vs float, date, keyword etc). The schema won't necessarily correspond directly to the index mapping but it will indicate the value type for that field.

That being said, the point of introducing schemas was to provide known fields and their expected types. We (from the Kibana side) shouldn't provide anything that we, um, don't know!

@afharo we'll still need to update mappings for fields we do know and are certain about and I'm we're highly unlikely to be maintaining mappings for dynamic fields now anyway (how could we if we don't know what they are 🤷‍♀️ ).

@mindbat Would you be ok with perhaps ignoring the mappings for these types of "DYNAMIC_KEY" fields? i.e. we maintain and update diffs in the fields we know but don't concern ourselves (i.e. don't map them at all) with those that we don't?

@mindbat everything reported in the schema is directly correlated to the data that gets shipped under stack_stats.kibana.plugins.

Unfortunately, we've still got a few objects whose keys are dynamic and we need the schema to describe them in some way. Then it's up to the Remote Telemetry Service to map them via Dynamic Mapping Templates, change their structure in the Indexer, or don't index/map them at all. Our take up until now has been: index them but don't map them (so they are in the _source for advanced visualisations like Vega, but can't really be searched/aggregated).

The question is: is DYNAMIC_KEY a valid indicator for you to identify these type of keys?

FYI: Based on the discussion in here (#78832 (comment)), we can provide some of the already-known keys. But the list is likely to grow dynamically (and we'll try to keep it up to date). Should, in those cases, still provide the known keys and the extra DYNAMIC_KEY for you to be aware of that case?

elasticmachine · 2020-09-29T19:38:19Z

Pinging @elastic/kibana-app-arch (Team:AppArch)

elasticmachine · 2020-09-29T19:38:20Z

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

mikecote · 2020-09-30T12:51:50Z

x-pack/plugins/actions/server/usage/actions_usage_collector.ts

+      // TODO: Find out all the possible values for DYNAMIC_KEY or reformat into an array
+      count_by_type: { DYNAMIC_KEY: { type: 'long' } },
+      count_active_by_type: { DYNAMIC_KEY: { type: 'long' } },


It looks like we can get a list of types here since the collector is created after setup phase (https://github.com/elastic/kibana/blob/master/x-pack/plugins/actions/server/plugin.ts#L228). Did you need some help to access the list of types?

Also not sure how this would then correlate to x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json.

Also not sure how this would then correlate to x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json.

We run the command node scripts/telemetry_extract.js to find all the schema declarations and merge them into that file.

It looks like we can get a list of types here since the collector is created after setup phase (master/x-pack/plugins/actions/server/plugin.ts#L228). Did you need some help to access the list of types?

Unfortunately, the command mentioned above doesn't run Kibana. It scans the code instead. But if you can point me to way other plugins register actions, I can search them and populate the current list. We'll likely need to refresh the list on every FF, until we find a way to automate it. Is it just looking for registerType(ACTION_TYPE)?

Is it just looking for registerType(ACTION_TYPE)?

I've looked for those and I couldn't find any use of those (only in test plugins) 🤔

Ah, it would be because only the actions plugin registers action types at this time. They are all defined here but call register(...) which is a lower level function.

https://github.com/elastic/kibana/blob/master/x-pack/plugins/actions/server/builtin_action_types/index.ts#L30-L38

Interesting about scanning the code. You will find plugins registering alert types but we don't have any yet registering action types (in 1-2 release from now maybe).

@mikecote thank you for the pointer! I'll update the list with the known keys.

Similar to the question in alerts, do you think it's still a good idea to maintain the DYNAMIC_KEY to let the Remote Telemetry cluster know that it might grow (just in case they want to set up a dynamic mapping template of some sort)?

@mikecote done! I just pushed the updated schema with the list of known built-in actions. Thank you for pointing me to where to find them

Bamieh

can we map them to dynamic and ask the actions/alerting team to map those since they'd be able to get the list of keys?

…ion/schema/actions

kibanamachine · 2020-10-01T14:36:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c0b5c05

Metrics [docs]

✅ unchanged

History

💚 Build #78317 succeeded a55a9e5
💚 Build #78004 succeeded d3a92cc2d651fb057fbab617909a26923de99b71

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

TinaHeiligers

LGTM

mikecote

Alerting changes LGTM!

mikecote · 2020-10-02T14:59:08Z

x-pack/plugins/actions/server/usage/actions_usage_collector.ts

 import { get } from 'lodash';
 import { TaskManagerStartContract } from '../../../task_manager/server';
 import { ActionsUsage } from './types';

+const byTypeSchema: MakeSchemaFrom<ActionsUsage>['count_by_type'] = {


@mindbat did you want the alerting team to keep updating this list on each release when new actions exist? or will it be possible with the DYNAMIC_KEY to automatically handle new types of actions per release on your end?

I'm a little fuzzy on the code here (not a ts dev 😅 ), but it's my understanding that @afharo's DYNAMIC_KEY in the schema that gets handed off to my team will be enough for us to know "this key's value will have this type, but its name might change"

elasticmachine · 2021-10-01T21:30:07Z

Pinging @elastic/kibana-core (Team:Core)

afharo added review Feature:Telemetry v8.0.0 Team:AppArch release_note:skip Skip the PR/issue when compiling release notes Feature:Actions v7.10.0 labels Sep 29, 2020

afharo mentioned this pull request Sep 29, 2020

[Telemetry] [Collectors] Required schema fixes #70180

Closed

14 tasks

afharo commented Sep 29, 2020

View reviewed changes

afharo marked this pull request as ready for review September 29, 2020 19:38

afharo requested review from a team as code owners September 29, 2020 19:38

afharo requested a review from mindbat September 30, 2020 11:49

afharo mentioned this pull request Sep 30, 2020

[Usage Collection] [schema] alerts #78933

Merged

1 task

mikecote reviewed Sep 30, 2020

View reviewed changes

afharo mentioned this pull request Sep 30, 2020

[Usage Collection] [schema] maps #78952

Merged

1 task

[Usage Collection] [schema] actions

a55a9e5

afharo force-pushed the usage_collection/schema/actions branch from d3a92cc to a55a9e5 Compare September 30, 2020 14:54

Bamieh approved these changes Oct 1, 2020

View reviewed changes

afharo added 2 commits October 1, 2020 13:47

Merge branch 'master' of github.com:elastic/kibana into usage_collect…

7d74f6c

…ion/schema/actions

List known actions

c0b5c05

TinaHeiligers approved these changes Oct 1, 2020

View reviewed changes

mikecote approved these changes Oct 2, 2020

View reviewed changes

afharo merged commit f398b49 into elastic:master Oct 2, 2020

afharo deleted the usage_collection/schema/actions branch October 2, 2020 15:03

afharo mentioned this pull request Oct 2, 2020

[7.x] [Usage Collection] [schema] actions (#78832) #79304

Merged

afharo added a commit that referenced this pull request Oct 2, 2020

[7.x] [Usage Collection] [schema] actions (#78832) (#79304)

6f57cc1

lukeelmers added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Oct 1, 2021

lukeelmers removed the Team:KibanaTelemetry label Oct 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Usage Collection] [schema] `actions` #78832

[Usage Collection] [schema] `actions` #78832

afharo commented Sep 29, 2020 •

edited

Loading

afharo Sep 29, 2020

mindbat Sep 30, 2020

TinaHeiligers Sep 30, 2020

afharo Oct 1, 2020

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

mikecote Sep 30, 2020 •

edited

Loading

afharo Sep 30, 2020

afharo Sep 30, 2020

mikecote Sep 30, 2020

afharo Oct 1, 2020

afharo Oct 1, 2020

Bamieh left a comment

kibanamachine commented Oct 1, 2020

TinaHeiligers left a comment

mikecote left a comment

mikecote Oct 2, 2020

mindbat Oct 7, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2021

[Usage Collection] [schema] actions #78832

[Usage Collection] [schema] actions #78832

Conversation

afharo commented Sep 29, 2020 • edited Loading

Summary

For maintainers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Sep 29, 2020

elasticmachine commented Sep 29, 2020

mikecote Sep 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bamieh left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 1, 2020

💚 Build Succeeded

Metrics [docs]

History

TinaHeiligers left a comment

Choose a reason for hiding this comment

mikecote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mindbat Oct 7, 2020 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Oct 1, 2021

[Usage Collection] [schema] `actions` #78832

[Usage Collection] [schema] `actions` #78832

afharo commented Sep 29, 2020 •

edited

Loading

mikecote Sep 30, 2020 •

edited

Loading

mindbat Oct 7, 2020 •

edited

Loading