Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Application Usage] Functional test to validate the full list of appIds in the schema #88080

Merged
merged 10 commits into from
Jan 19, 2021
Merged

[Application Usage] Functional test to validate the full list of appIds in the schema #88080

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3365808
[Application Usage] Functional test to validate the full list of appI…
afharo Jan 12, 2021
5f41295
Enable test in the CI
afharo Jan 13, 2021
28902a7
No need to filter out 'test' appIds
afharo Jan 13, 2021
1f3ddd8
Merge branch 'master' into application_usage/functiona-test-appIds-in…
kibanamachine Jan 15, 2021
7e0df18
Test the apps in the test suite instead of the plugin
afharo Jan 15, 2021
3d4915f
TO BE REVERTED: Extra key in schema to make CI fail
afharo Jan 15, 2021
4ccc488
Remove extra key for all green
afharo Jan 15, 2021
c44c628
Merge branch 'master' into application_usage/functiona-test-appIds-in…
kibanamachine Jan 18, 2021
280e880
Merge branch 'master' of github.com:elastic/kibana into application_u…
afharo Jan 19, 2021
4406dee
Error message pointing to the location of the schema to fix
afharo Jan 19, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,7 @@ const commonSchema: MakeSchemaFrom<ApplicationUsageTelemetryReport[string]> = {
},
};

// These keys obtained by searching for `/application\w*\.register\(/` and checking the value of the attr `id`.
// TODO: Find a way to update these keys automatically.
// There is a test in x-pack/test/usage_collection that validates that the keys in here match all the registered apps
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As adding a new app will cause the test to fail, maybe there should be a comment on the test pointing to this file to help developers fix it when they'll add new apps?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pgayvallet makes sense! I've updated the test to point at the file in the error message.

Here's an example of the output:

0 passing (10.4s)
     │1 failing
     │
     │1)    Application Usage
     │       keys in the schema match the registered application IDs:
     │
     │
     │       Error: Application Usage's schema is not up-to-date with the actual registered apps. Please update it at src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts.
     │ expected [ 'apm',
     │   'appSearch',
     │   'canvas',
     │   'dashboard_mode',
     │   'dashboards',
     │   'dev_tools',
     │   'discover',
     │   'enterpriseSearch',
     │   'error',
     │   'fleet',
     │   'graph',
     │   'home',
     │   'infra',
     │   'ingestManager',
     │   'kibana',
     │   'kibanaOverview',
     │   'lens',
     │   'logs',
     │   'management',
     │   'maps',
     │   'metrics',
     │   'ml',
     │   'monitoring',
     │   'observability-overview',
     │   'securitySolution',
     │   'securitySolution:administration',
     │   'securitySolution:case',
     │   'securitySolution:detections',
     │   'securitySolution:hosts',
     │   'securitySolution:network',
     │   'securitySolution:overview',
     │   'securitySolution:timelines',
     │   'security_access_agreement',
     │   'security_account',
     │   'security_capture_url',
     │   'security_logged_out',
     │   'security_login',
     │   'security_logout',
     │   'security_overwritten_session',
     │   'short_url_redirect',
     │   'siem',
     │   'space_selector',
     │   'status',
     │   'test',
     │   'timelion',
     │   'uptime',
     │   'ux',
     │   'visualize',
     │   'workplaceSearch' ] to sort of equal [ 'apm',
     │   'appSearch',
     │   'canvas',
     │   'dashboard_mode',
     │   'dashboards',
     │   'dev_tools',
     │   'discover',
     │   'enterpriseSearch',
     │   'error',
     │   'fleet',
     │   'graph',
     │   'home',
     │   'infra',
     │   'ingestManager',
     │   'kibana',
     │   'kibanaOverview',
     │   'lens',
     │   'logs',
     │   'management',
     │   'maps',
     │   'metrics',
     │   'ml',
     │   'monitoring',
     │   'observability-overview',
     │   'securitySolution',
     │   'securitySolution:administration',
     │   'securitySolution:case',
     │   'securitySolution:detections',
     │   'securitySolution:hosts',
     │   'securitySolution:network',
     │   'securitySolution:overview',
     │   'securitySolution:timelines',
     │   'security_access_agreement',
     │   'security_account',
     │   'security_capture_url',
     │   'security_logged_out',
     │   'security_login',
     │   'security_logout',
     │   'security_overwritten_session',
     │   'short_url_redirect',
     │   'siem',
     │   'space_selector',
     │   'status',
     │   'timelion',
     │   'uptime',
     │   'ux',
     │   'visualize',
     │   'workplaceSearch' ]
     │       + expected - actual
     │
     │          "short_url_redirect"
     │          "siem"
     │          "space_selector"
     │          "status"
     │       -  "test"
     │          "timelion"
     │          "uptime"
     │          "ux"
     │          "visualize"
     │
     │       at Assertion.assert (/Users/afharo/Developer/elastic/kibana/packages/kbn-expect/expect.js:100:11)
     │       at Assertion.eql (/Users/afharo/Developer/elastic/kibana/packages/kbn-expect/expect.js:244:8)
     │       at Context.<anonymous> (test/usage_collection/test_suites/application_usage/index.ts:21:63)
     │       at runMicrotasks (<anonymous>)
     │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
     │       at Object.apply (/Users/afharo/Developer/elastic/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
     │
     │

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect! thanks.

export const applicationUsageSchema = {
// OSS
dashboards: commonSchema,
Expand All @@ -61,10 +60,12 @@ export const applicationUsageSchema = {
short_url_redirect: commonSchema, // It's a forward app so we'll likely never report it
timelion: commonSchema,
visualize: commonSchema,
error: commonSchema,
status: commonSchema,
kibanaOverview: commonSchema,

// X-Pack
apm: commonSchema,
csm: commonSchema,
canvas: commonSchema,
dashboard_mode: commonSchema, // It's a forward app so we'll likely never report it
enterpriseSearch: commonSchema,
Expand All @@ -75,6 +76,7 @@ export const applicationUsageSchema = {
metrics: commonSchema,
infra: commonSchema, // It's a forward app so we'll likely never report it
fleet: commonSchema,
ingestManager: commonSchema,
lens: commonSchema,
maps: commonSchema,
ml: commonSchema,
Expand All @@ -98,4 +100,5 @@ export const applicationUsageSchema = {
siem: commonSchema,
space_selector: commonSchema,
uptime: commonSchema,
ux: commonSchema,
};
288 changes: 286 additions & 2 deletions src/plugins/telemetry/schema/oss_plugins.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -719,7 +719,7 @@
}
}
},
"apm": {
"error": {
"properties": {
"appId": {
"type": "keyword"
Expand Down Expand Up @@ -790,7 +790,149 @@
}
}
},
"csm": {
"status": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
},
"views": {
"type": "array",
"items": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
}
}
}
}
}
},
"kibanaOverview": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
},
"views": {
"type": "array",
"items": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
}
}
}
}
}
},
"apm": {
"properties": {
"appId": {
"type": "keyword"
Expand Down Expand Up @@ -1571,6 +1713,77 @@
}
}
},
"ingestManager": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
},
"views": {
"type": "array",
"items": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
}
}
}
}
}
},
"lens": {
"properties": {
"appId": {
Expand Down Expand Up @@ -3203,6 +3416,77 @@
}
}
}
},
"ux": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
},
"views": {
"type": "array",
"items": {
"properties": {
"appId": {
"type": "keyword"
},
"viewId": {
"type": "keyword"
},
"clicks_total": {
"type": "long"
},
"clicks_7_days": {
"type": "long"
},
"clicks_30_days": {
"type": "long"
},
"clicks_90_days": {
"type": "long"
},
"minutes_on_screen_total": {
"type": "float"
},
"minutes_on_screen_7_days": {
"type": "float"
},
"minutes_on_screen_30_days": {
"type": "float"
},
"minutes_on_screen_90_days": {
"type": "float"
}
}
}
}
}
}
}
},
Expand Down
1 change: 1 addition & 0 deletions test/scripts/jenkins_xpack_build_plugins.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@ node scripts/build_kibana_platform_plugins \
--scan-dir "$XPACK_DIR/test/plugin_api_integration/plugins" \
--scan-dir "$XPACK_DIR/test/plugin_api_perf/plugins" \
--scan-dir "$XPACK_DIR/test/licensing_plugin/plugins" \
--scan-dir "$XPACK_DIR/test/usage_collection/plugins" \
--workers 12 \
--verbose
1 change: 1 addition & 0 deletions x-pack/scripts/functional_tests.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ const onlyNotInCoverageTests = [
require.resolve('../test/saved_object_tagging/functional/config.ts'),
require.resolve('../test/saved_object_tagging/api_integration/security_and_spaces/config.ts'),
require.resolve('../test/saved_object_tagging/api_integration/tagging_api/config.ts'),
require.resolve('../test/usage_collection/config.ts'),
];

require('../../src/setup_node_env');
Expand Down
Loading