[AO] - Add functional tests for the new Rules page #129349

fkanout · 2022-04-04T13:38:29Z

Summary

If fixes #129605 by adding functional tests for the newly created Rules pages.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

fkanout · 2022-04-12T10:40:38Z

@elasticmachine merge upstream

fkanout · 2022-05-12T09:01:58Z

@elasticmachine merge upstream

fkanout · 2022-05-16T06:58:46Z

@elasticmachine merge upstream

LeeDr · 2022-05-16T17:48:14Z

I was going to run these tests locally as part of my review but I'm not sure how to run them. There's usually a config file for the tests but I don't see one in kibana/x-pack/test/observability_functional/apps/observability. Can you explain how to run the new tests?

UPDATE: I think I found it kibana/x-pack/test/observability_functional/with_rac_write.config.ts

LeeDr · 2022-05-16T18:29:23Z

When I ran the tests locally Kibana logged a bunch of 403's like;

[WARN ][plugins.security.api-authorization] User not authorized for "/internal/uptime/index_status": responding with 403
[WARN ][plugins.security.api-authorization] User not authorized for "/internal/apm/observability_overview/has_data": responding with 403

And then crashed;

proc [kibana] [2022-05-16T13:10:13.023-05:00][WARN ][plugins.security.api-authorization] User not authorized for "/internal/uptime/index_status": responding with 403
 proc [kibana] [2022-05-16T13:10:13.034-05:00][WARN ][plugins.security.api-authorization] User not authorized for "/internal/apm/observability_overview/has_data": responding with 403
 info [o.e.x.s.a.u.TransportPutUserAction] [ftr] updated user [test_user]
 proc [kibana] Unhandled Promise rejection detected:
 proc [kibana] 
 proc [kibana] Error: Saved object [config/8.3.0] not found
 proc [kibana]     at Function.createGenericNotFoundError (/Users/leedr/git/main/kibana/src/core/server/saved_objects/service/lib/errors.ts:133:28)
 proc [kibana]     at SavedObjectsRepository.get (/Users/leedr/git/main/kibana/src/core/server/saved_objects/service/lib/repository.ts:1255:38)
 proc [kibana]     at runMicrotasks (<anonymous>)
 proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
 proc [kibana]     at SavedObjectsClient.get (/Users/leedr/git/main/kibana/src/core/server/saved_objects/service/saved_objects_client.ts:531:12)
 proc [kibana]     at EncryptedSavedObjectsClientWrapper.get (/Users/leedr/git/main/kibana/x-pack/plugins/encrypted_saved_objects/server/saved_objects/encrypted_saved_objects_client_wrapper.ts:188:7)
 proc [kibana]     at SecureSavedObjectsClientWrapper.get (/Users/leedr/git/main/kibana/x-pack/plugins/security/server/saved_objects/secure_saved_objects_client_wrapper.ts:343:25)
 proc [kibana]     at SpacesSavedObjectsClient.get (/Users/leedr/git/main/kibana/x-pack/plugins/spaces/server/saved_objects/spaces_saved_objects_client.ts:233:12)
 proc [kibana]     at UiSettingsClient.read (/Users/leedr/git/main/kibana/src/core/server/ui_settings/ui_settings_client.ts:167:20)
 proc [kibana]     at UiSettingsClient.read (/Users/leedr/git/main/kibana/src/core/server/ui_settings/ui_settings_client.ts:180:18)
 proc [kibana]     at UiSettingsClient.getUserProvided (/Users/leedr/git/main/kibana/src/core/server/ui_settings/ui_settings_client.ts:63:59)
 proc [kibana]     at UiSettingsClient.getAll (/Users/leedr/git/main/kibana/src/core/server/ui_settings/base_ui_settings_client.ts:61:26)
 proc [kibana]     at Object.fieldFormatServiceFactory (/Users/leedr/git/main/kibana/src/plugins/field_formats/server/plugin.ts:44:31)
 proc [kibana]     at Object.<anonymous> (/Users/leedr/git/main/kibana/src/plugins/data_views/server/data_views_service_factory.ts:41:21) {
 proc [kibana]   data: null,
 proc [kibana]   isBoom: true,
 proc [kibana]   isServer: false,
 proc [kibana]   output: {
 proc [kibana]     statusCode: 404,
 proc [kibana]     payload: {
 proc [kibana]       statusCode: 404,
 proc [kibana]       error: 'Not Found',
 proc [kibana]       message: 'Saved object [config/8.3.0] not found'
 proc [kibana]     },
 proc [kibana]     headers: {}
 proc [kibana]   },
 proc [kibana]   [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/notFound'
 proc [kibana] }
 proc [kibana] 
 proc [kibana] Terminating process...
 proc [kibana]  server crashed  with status code 1

Seems to be from this point of the tests;

       └-> "after all" hook for "Shows the no data page on load"
         │ info [x-pack/test/functional/es_archives/cases/default] Unloading indices from "mappings.json"
         │ warn delete by query deleted 31 of 32 total documents, trying again
         │ warn delete by query deleted 0 of 32 total documents, trying again
         │ warn delete by query deleted 0 of 32 total documents, trying again
         │ warn since spaces are enabled, all objects other than the default space were deleted from .kibana rather than deleting the whole index
         │ info [x-pack/test/functional/es_archives/cases/default] Deleted existing index ".kibana"
         │ info [x-pack/test/functional/es_archives/cases/default] Unloading indices from "data.json.gz"
       └-> "after all" hook: afterTestSuite.trigger for "Shows the no data page on load"
     └-> "after all" hook: afterTestSuite.trigger in "feature controls"
   └-: Observability Rules page
     └-> "before all" hook: beforeTestSuite.trigger in "Observability Rules page"
     └-> "before all" hook in "Observability Rules page"
       │ERROR [GET http://elastic:changeme@localhost:5620/api/status] request failed (attempt=1/5): connect ECONNREFUSED 127.0.0.1:5620
       │ERROR [GET http://elastic:changeme@localhost:5620/api/status] request failed (attempt=2/5): connect ECONNREFUSED 127.0.0.1:5620

LeeDr · 2022-05-16T19:00:03Z

I ran again but this time ran all together with;
node scripts/functional_tests.js --config x-pack/test/observability_functional/with_rac_write.config.ts
(instead of separate tests_servers and test_runner)

This test failed, but the screenshot shows 2 rules both enabled.

     └-: Rules table
       └-> "before all" hook: beforeTestSuite.trigger for "shows the rules table "
       └-> "before all" hook for "shows the rules table "
         │ info [o.e.c.m.MetadataMappingService] [ftr] [.kibana_8.3.0_001/SH0LEk2TRI2GrfFIv0Bh7Q] update_mapping [_doc]
         │ info [o.e.c.m.MetadataMappingService] [ftr] [.kibana_task_manager_8.3.0_001/jt4HzHfdTGuiH-5du7786A] update_mapping [_doc]
       └-> shows the rules table 
         └-> "before each" hook: global before each for "shows the rules table "
         │ info Taking screenshot "/Users/leedr/git/main/kibana/x-pack/test/functional/screenshots/failure/ObservabilityApp Observability Rules page Rules table shows the rules table .png"
         │ info Current URL is: http://localhost:5620/app/observability/alerts/rules?rangeFrom=now-15m&rangeTo=now&_a=(lastResponse%3A!()%2Cstatus%3A!())
         │ info Saving page source to: /Users/leedr/git/main/kibana/x-pack/test/functional/failure_debug/html/ObservabilityApp Observability Rules page Rules table shows the rules table .html
         └- ✖ fail: ObservabilityApp Observability Rules page Rules table shows the rules table 
         │      Error: expected undefined to equal 'Enabled'
         │       at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
         │       at Assertion.be.Assertion.equal (node_modules/@kbn/expect/expect.js:227:8)
         │       at Assertion.be (node_modules/@kbn/expect/expect.js:69:22)
         │       at Context.<anonymous> (x-pack/test/observability_functional/apps/observability/alerts/rules_page.ts:135:36)
         │       at runMicrotasks (<anonymous>)
         │       at processTicksAndRejections (node:internal/process/task_queues:96:5)
         │       at Object.apply (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
         │

I ran the test again and it passed.

I ran again with TEST_THROTTLE_NETWORK=1 and had the same failure.

1)    ObservabilityApp
       Observability Rules page
         Rules table
           shows the rules table :

      Error: expected undefined to equal 'Enabled'
       at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
       at Assertion.be.Assertion.equal (node_modules/@kbn/expect/expect.js:227:8)
       at Assertion.be (node_modules/@kbn/expect/expect.js:69:22)
       at Context.<anonymous> (x-pack/test/observability_functional/apps/observability/alerts/rules_page.ts:135:36)
       at runMicrotasks (<anonymous>)
       at processTicksAndRejections (node:internal/process/task_queues:96:5)
       at Object.apply (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

LeeDr · 2022-05-16T20:06:12Z

I went ahead and started a flaky test runner job here; https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/632
Run 19 was the same error as I hit locally Error: expected undefined to equal 'Enabled' but most or all of the other failures were on a different test.

LeeDr · 2022-05-17T02:05:56Z

I did a bit more debugging on the failure. in rules_page.ts I added this console.log line to output the euiTableRow innerHTML;

  const getRulesList = async (tableRows: any[]) => {
    const rows = [];
    for (const euiTableRow of tableRows) {
      const $ = await euiTableRow.parseDomContent();
      console.log(`---------- ${await euiTableRow.getAttribute('innerHTML')}`);
      rows.push({
        name: $.findTestSubjects('rulesTableCell-name').find('a').text(),
        enabled: $.findTestSubjects('rulesTableCell-ContextStatus').find('button').attr('title'),
      });
    }
    return rows;
  };

I captured one row of that output. Then I added a 5s sleep in the test before finding all the table rows;

     it('shows the rules table ', async () => {
        await testSubjects.existOrFail('rulesList');
        await PageObjects.common.sleep(5000);
        const tableRows = await find.allByCssSelector('.euiTableRow');
        const rows = await getRulesList(tableRows);
        expect(rows.length).to.be(2);
        expect(rows[0].name).to.be('error-log');
        expect(rows[0].enabled).to.be('Enabled');
        expect(rows[1].name).to.be('uptime');
        expect(rows[1].enabled).to.be('Enabled');
      });

I captured that first row again and diff'd them. The one on the right is the original without the sleep and when the test fails. In that html I can see centerJustifiedSpinner that I assume means the table is still loading.
<div class="euiFlexGroup euiFlexGroup--gutterLarge euiFlexGroup--justifyContentCenter euiFlexGroup--directionRow euiFlexGroup--responsive" data-test-subj="centerJustifiedSpinner">

Also an euiLoadingSpinner
<span class="euiLoadingSpinner euiLoadingSpinner--medium">

Also euiIcon-isLoading;

In the left html after the sleep, we see euiIcon-isLoaded in a few places instead of euiIcon-isLoading.
I don't know if there's a method that already exists to wait for loading that could be used. In the header_page page object there's this method awaitGlobalLoadingIndicatorHidden() but I don't think it works for this table.

x-pack/test/observability_functional/apps/observability/alerts/rules_page.ts

fkanout · 2022-05-17T09:05:47Z

@LeeDr Thanks for the effort and the time you dedicate to this! I had to put the steps on how to run the tests locally.

First run the server with the config you already found
node scripts/functional_tests_server --config x-pack/test/observability_functional/with_rac_write.config.ts
Once the server is running, then the test runner
node scripts/functional_test_runner --config x-pack/test/observability_functional/with_rac_write.config.ts

LeeDr

LGTM - Passes locally for me now and I also ran 50 iterations on the flaky test runner and they all passed

fkanout · 2022-05-18T15:18:59Z

@elasticmachine merge upstream

kibana-ci · 2022-05-18T16:20:01Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observability`	451.9KB	452.0KB	+101.0B
`triggersActionsUi`	782.0KB	782.2KB	+182.0B
total			+283.0B

History

💚 Build #45235 succeeded 89ad376
💚 Build #44770 succeeded 6532a14
💚 Build #44340 succeeded bbb4198
💚 Build #44061 succeeded dee22e1
💔 Build #44005 failed 94992f5
💔 Build #43977 failed c4db139

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @fkanout

kibanamachine · 2022-05-18T16:23:17Z

💔 All backports failed

Status	Branch	Result
❌	8.2	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 129349

Questions ?

Please refer to the Backport tool documentation

mgiota · 2022-05-19T07:45:52Z

@fkanout I would remove 8.2 in this case and skip the backport. I remember I had similar issue when I wanted to backport the unit tests PR. If I recall correctly the branches had diverged a bit, so the best option was to not backport the tests in 8.2.

kibanamachine · 2022-05-20T16:26:19Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 129349 or prevent reminders by adding the backport:skip label.

kibanamachine · 2022-05-23T16:36:17Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 129349 or prevent reminders by adding the backport:skip label.

WIP

ba4a1d6

fkanout added auto-backport Deprecated - use backport:version if exact versions are needed Theme: rac label obsolete Team: Actionable Observability - DEPRECATED For Observability Alerting and SLOs use "Team:obs-ux-management", for AIops "Team:obs-knowledge" v8.2.0 v8.3.0 labels Apr 4, 2022

fkanout self-assigned this Apr 4, 2022

fkanout added 7 commits April 4, 2022 17:15

Add permissions tests

01170a1

Clean up

7544337

Clean up

b53c055

Add create rule flyout test

261bb50

Add rule creating and check rules table

edfbfba

Update wording

b86f24a

Enable tests

8da061a

fkanout linked an issue Apr 6, 2022 that may be closed by this pull request

Add unit tests for new o11y Rule page #127440

Closed

fkanout removed a link to an issue Apr 6, 2022

Add unit tests for new o11y Rule page #127440

Closed

fkanout changed the title ~~[RAC][AO] - Add functional tests for the new Rules page~~ [AO] - Add functional tests for the new Rules page Apr 6, 2022

fkanout removed the Theme: rac label obsolete label Apr 6, 2022

fkanout added 4 commits April 11, 2022 16:58

Add rules table tests

f584667

Merge branch 'main' into new-o11y-rules-page-functional-tests

7cd098e

disable "only"

d967595

Merge branch 'main' into new-o11y-rules-page-functional-tests

35baf49

kibanamachine and others added 3 commits April 12, 2022 05:40

Merge branch 'main' into new-o11y-rules-page-functional-tests

cb5d3f1

Add enabled/disabled test case

86de56b

fix style

d2b3bd0

fkanout marked this pull request as ready for review April 12, 2022 12:15

fkanout requested review from a team as code owners April 12, 2022 12:15

Merge branch 'main' into new-o11y-rules-page-functional-tests

33cc1d6

fkanout enabled auto-merge (squash) May 11, 2022 15:23

fkanout requested a review from LeeDr May 12, 2022 09:01

Merge branch 'main' into new-o11y-rules-page-functional-tests

bbb4198

Merge branch 'main' into new-o11y-rules-page-functional-tests

6532a14

LeeDr reviewed May 17, 2022

View reviewed changes

x-pack/test/observability_functional/apps/observability/alerts/rules_page.ts Show resolved Hide resolved

Code review

89ad376

fkanout requested a review from LeeDr May 17, 2022 09:08

LeeDr approved these changes May 17, 2022

View reviewed changes

Merge branch 'main' into new-o11y-rules-page-functional-tests

9fb6aa5

chrisronline approved these changes May 18, 2022

View reviewed changes

fkanout merged commit e4a365a into elastic:main May 18, 2022

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label May 20, 2022

emma-raffenne removed the v8.2.0 label May 24, 2022

kibanamachine added backport:skip This commit does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels May 24, 2022

elasticmachine unassigned fkanout Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AO] - Add functional tests for the new Rules page #129349

[AO] - Add functional tests for the new Rules page #129349

fkanout commented Apr 4, 2022 •

edited

Loading

fkanout commented Apr 12, 2022

fkanout commented May 12, 2022

fkanout commented May 16, 2022

LeeDr commented May 16, 2022 •

edited

Loading

LeeDr commented May 16, 2022

LeeDr commented May 16, 2022

LeeDr commented May 16, 2022

LeeDr commented May 17, 2022

fkanout commented May 17, 2022

LeeDr left a comment

fkanout commented May 18, 2022

kibana-ci commented May 18, 2022

kibanamachine commented May 18, 2022

mgiota commented May 19, 2022

kibanamachine commented May 20, 2022

kibanamachine commented May 23, 2022

[AO] - Add functional tests for the new Rules page #129349

[AO] - Add functional tests for the new Rules page #129349

Conversation

fkanout commented Apr 4, 2022 • edited Loading

Summary

Checklist

fkanout commented Apr 12, 2022

fkanout commented May 12, 2022

fkanout commented May 16, 2022

LeeDr commented May 16, 2022 • edited Loading

LeeDr commented May 16, 2022

LeeDr commented May 16, 2022

LeeDr commented May 16, 2022

LeeDr commented May 17, 2022

fkanout commented May 17, 2022

LeeDr left a comment

Choose a reason for hiding this comment

fkanout commented May 18, 2022

kibana-ci commented May 18, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented May 18, 2022

💔 All backports failed

Manual backport

Questions ?

mgiota commented May 19, 2022

kibanamachine commented May 20, 2022

kibanamachine commented May 23, 2022

fkanout commented Apr 4, 2022 •

edited

Loading

LeeDr commented May 16, 2022 •

edited

Loading