Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Application Usage] Functional test to validate the full list of appIds in the schema #88080

Merged

Conversation

afharo
Copy link
Member

@afharo afharo commented Jan 12, 2021

Summary

This PR adds functional tests to the Application Usage schema definition: it runs Kibana and checks that all the full list of registered apps is detailed in the schema (all appIds should be set as keys in the applicationUsage schema).

Resolves #75329

Checklist

Delete any items that are not applicable to this PR.

For maintainers

Sorry, something went wrong.

@afharo afharo added release_note:skip Skip the PR/issue when compiling release notes v7.12.0 labels Jan 12, 2021
@@ -0,0 +1,44 @@
/*
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this new folder of functional tests can be used as well when implementing #83704

@afharo afharo force-pushed the application_usage/functiona-test-appIds-in-schema branch from 04f3052 to 5f41295 Compare January 13, 2021 09:32
@afharo afharo marked this pull request as ready for review January 13, 2021 13:41
@afharo afharo requested a review from a team as a code owner January 13, 2021 13:41
@afharo
Copy link
Member Author

afharo commented Jan 15, 2021

@elasticmachine merge upstream

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…-schema
Copy link
Contributor

@joshdover joshdover left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just be sure the test suite is completely set up for CI correctly, have we tested making pushing a failing version of the test to CI? Sometimes it's easy to miss a detail about adding a new functional suite and I like to use that as a gut-check.

Comment on lines 22 to 28
try {
// When the lists don't match, the entire page load will fail.
expect(Object.keys(applicationUsageSchema).sort()).to.eql(appIds);
} catch (err) {
err.message = `Application Usage's schema is not up-to-date with the actual registered apps. Please update it at src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts.\n${err.message}`;
throw err;
}
Copy link
Contributor

@joshdover joshdover Jan 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of throwing an error on startup, it may be nicer to have this plugin set a property on window.__usageCollectionTest__ or similar and have the test read that property. That way, other tests that we add to this suite can still run even if this test is currently failing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes perfect sense! In fact, I had a thought about it, but couldn't find a way to read the window object in the tests and thought it was not possible. Now that you mentioned, I ran a second search and found the right way to do it. Thanks! I'll push the changes in a bit.

Re the CI, I've checked Jenkins' logs to make sure the test runs. And I tried locally the fail scenario in the test. I'll push a failed scenario anyway to ensure we are setting things right by seeing the CI fail 👍

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failed as expected: https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/99129/testReport/ 🎉

Running a last build with the correct schema.

@afharo afharo requested a review from joshdover January 15, 2021 18:39
@afharo
Copy link
Member Author

afharo commented Jan 18, 2021

@elasticmachine merge upstream

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…-schema
Copy link
Contributor

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

// These keys obtained by searching for `/application\w*\.register\(/` and checking the value of the attr `id`.
// TODO: Find a way to update these keys automatically.
// There is a test in x-pack/test/usage_collection that validates that the keys in here match all the registered apps
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As adding a new app will cause the test to fail, maybe there should be a comment on the test pointing to this file to help developers fix it when they'll add new apps?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pgayvallet makes sense! I've updated the test to point at the file in the error message.

Here's an example of the output:

0 passing (10.4s)
     │1 failing
     │
     │1)    Application Usage
     │       keys in the schema match the registered application IDs:
     │
     │
     │       Error: Application Usage's schema is not up-to-date with the actual registered apps. Please update it at src/plugins/kibana_usage_collection/server/collectors/application_usage/schema.ts.
     │ expected [ 'apm',
     │   'appSearch',
     │   'canvas',
     │   'dashboard_mode',
     │   'dashboards',
     │   'dev_tools',
     │   'discover',
     │   'enterpriseSearch',
     │   'error',
     │   'fleet',
     │   'graph',
     │   'home',
     │   'infra',
     │   'ingestManager',
     │   'kibana',
     │   'kibanaOverview',
     │   'lens',
     │   'logs',
     │   'management',
     │   'maps',
     │   'metrics',
     │   'ml',
     │   'monitoring',
     │   'observability-overview',
     │   'securitySolution',
     │   'securitySolution:administration',
     │   'securitySolution:case',
     │   'securitySolution:detections',
     │   'securitySolution:hosts',
     │   'securitySolution:network',
     │   'securitySolution:overview',
     │   'securitySolution:timelines',
     │   'security_access_agreement',
     │   'security_account',
     │   'security_capture_url',
     │   'security_logged_out',
     │   'security_login',
     │   'security_logout',
     │   'security_overwritten_session',
     │   'short_url_redirect',
     │   'siem',
     │   'space_selector',
     │   'status',
     │   'test',
     │   'timelion',
     │   'uptime',
     │   'ux',
     │   'visualize',
     │   'workplaceSearch' ] to sort of equal [ 'apm',
     │   'appSearch',
     │   'canvas',
     │   'dashboard_mode',
     │   'dashboards',
     │   'dev_tools',
     │   'discover',
     │   'enterpriseSearch',
     │   'error',
     │   'fleet',
     │   'graph',
     │   'home',
     │   'infra',
     │   'ingestManager',
     │   'kibana',
     │   'kibanaOverview',
     │   'lens',
     │   'logs',
     │   'management',
     │   'maps',
     │   'metrics',
     │   'ml',
     │   'monitoring',
     │   'observability-overview',
     │   'securitySolution',
     │   'securitySolution:administration',
     │   'securitySolution:case',
     │   'securitySolution:detections',
     │   'securitySolution:hosts',
     │   'securitySolution:network',
     │   'securitySolution:overview',
     │   'securitySolution:timelines',
     │   'security_access_agreement',
     │   'security_account',
     │   'security_capture_url',
     │   'security_logged_out',
     │   'security_login',
     │   'security_logout',
     │   'security_overwritten_session',
     │   'short_url_redirect',
     │   'siem',
     │   'space_selector',
     │   'status',
     │   'timelion',
     │   'uptime',
     │   'ux',
     │   'visualize',
     │   'workplaceSearch' ]
     │       + expected - actual
     │
     │          "short_url_redirect"
     │          "siem"
     │          "space_selector"
     │          "status"
     │       -  "test"
     │          "timelion"
     │          "uptime"
     │          "ux"
     │          "visualize"
     │
     │       at Assertion.assert (/Users/afharo/Developer/elastic/kibana/packages/kbn-expect/expect.js:100:11)
     │       at Assertion.eql (/Users/afharo/Developer/elastic/kibana/packages/kbn-expect/expect.js:244:8)
     │       at Context.<anonymous> (test/usage_collection/test_suites/application_usage/index.ts:21:63)
     │       at runMicrotasks (<anonymous>)
     │       at processTicksAndRejections (internal/process/task_queues.js:93:5)
     │       at Object.apply (/Users/afharo/Developer/elastic/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
     │
     │

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect! thanks.

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@afharo afharo merged commit 7c9b114 into elastic:master Jan 19, 2021
@afharo afharo deleted the application_usage/functiona-test-appIds-in-schema branch January 19, 2021 14:44
afharo added a commit that referenced this pull request Jan 19, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…f appIds in the schema (#88080) (#88681)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Telemetry release_note:skip Skip the PR/issue when compiling release notes v7.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Usage Collection] Application Usage automatic schema
4 participants