Restore webdriver-based functional test runner driver #31388

spalger · 2019-02-18T03:30:08Z

Reopening #26477, we'll take another shot and this time run the tests even more often to make sure we address even more sources of flakiness before merging.

Note: we're running all of the functional tests 6 times per build. Hopefully this will help us see flakiness before we merge into master.

spalger · 2019-02-18T03:41:59Z

test/functional/services/find.js

+      log.debug(`Find.existsByDisplayedByCssSelector('${selector}') with timeout=${timeout}`);
+      return await this.exists(async (driver) => {
+        const elements = wrapAll(await driver.findElements(By.css(selector)));
+        return await this.filterElementIsDisplayed(elements);


Filtering after the findElements() call like this means that the timeout isn't used if the findElements() call returns elements but none of them are displayed, I'm pretty sure this is the reason we're seeing flakiness like https://kibana-ci.elastic.co/job/elastic+kibana+master/66/JOB=x-pack-ciGroup6,node=immutable/testReport/junit/X-Pack%20Chromium%20Functional%20Reporting%20Tests/x-pack_test_reporting_functional_reporting%C2%B7js/reporting_app_Reporting_Dashboard_Print_PDF_button_becomes_available_when_saved/

I tried to fix this like so https://github.com/elastic/kibana/pull/31368/files#diff-6238cdd3f41d65e4fddb017713976292R209, but it caused other timeout issues, since calls to exist() were waiting for longer, and subsequent timeout calls were missing temporary elements like the success toast here:

[00:20:37] │ debg Waiting up to 20000ms for flyout closed... [00:20:37] │ debg TestSubjects.exists(dashboardAddPanel) [00:20:37] │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="dashboardAddPanel"]') with timeout=2500 [00:20:48] │ debg Find.findByCssSelector('.euiToast') with timeout=10000 ... [00:20:58] └- ✖ fail: "dashboard app using current data dashboard snapshots compare area chart snapshot" [00:20:58] │ TimeoutError: Waiting for element to be located By(css selector, .euiToast) [00:20:58] │ Wait timed out after 10033ms [00:20:58] │ at node_modules/selenium-webdriver/lib/webdriver.js:834:17 [00:20:58] │ at process._tickCallback (internal/process/next_tick.js:68:7)

https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/1118/JOB=kibana-ciGroup3,node=immutable/testReport/junit/UI%20Functional%20Tests/test_functional_apps_dashboard__dashboard_snapshots%C2%B7js/dashboard_app_using_current_data_dashboard_snapshots_compare_area_chart_snapshot/

elasticmachine · 2019-02-18T08:00:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

[00:04:06]                   │ debg --- retry.try error: unknown error: Element <button class="kuiLocalMenuItem" ng-repeat="menuItem in kbnTopNav.menuItems" aria-label="... Visualization" aria-haspopup="false" aria-expanded="false" aria-disabled="false" ng-class="{'kuiLocalMenuItem-isSelected': kbnTopNav.isCurrent(menuItem.key), 'kuiLocalMenuItem-isDisabled': menuItem.disableButton()}" ng-click="kbnTopNav.handleClick(menuItem, $event)" ng-bind="menuItem.label" tooltip="" tooltip-placement="bottom" tooltip-popup-delay="400" tooltip-append-to-body="1" data-test-subj="visualizeSaveButton">Save</button> is not clickable at point (73, 67). Other element would receive the click: <div class="euiOverlayMask">...</div>

https://kibana-ci.elastic.co/job/elastic+kibana+pull-request/1149/JOB=x-pack-ciGroup6-2,node=immutable/testReport/junit/X-Pack%20Chromium%20Functional%20Reporting%20Tests/x-pack_test_reporting_functional_reporting%C2%B7js/reporting_app_Reporting_Visualize_Print_PDF_button_becomes_available_when_saved/

spalger · 2019-02-18T08:10:22Z

retest

elasticmachine · 2019-02-18T08:52:16Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…driver

dmlemeshko · 2019-02-18T21:59:42Z

X-Pack API Integration Tests failed, merging latest master to see if that helps.

elasticmachine · 2019-02-18T22:39:53Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

1 failure

[00:09:41]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="globalLoadingIndicator"]') with timeout=2500
[00:09:44]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:44]                   │ debg Find.existsByCssSelector('[data-test-subj~="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:44]                   │ debg TestSubjects.click(mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time)
[00:09:44]                   │ debg Find.clickByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
[00:09:44]                   │ debg Find.findByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
[00:09:54]                   │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:09:54]                   │      Wait timed out after 10026ms
[00:09:54]                   │ debg Find.findByCssSelector('[data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"]') with timeout=10000
...
[00:11:50]                   │ debg --- retry.try error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:11:50]                   │      Wait timed out after 10043ms
[00:11:50]                   │          at /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/node_modules/selenium-webdriver/lib/webdriver.js:834:17
[00:11:50]                   │          at process._tickCallback (internal/process/next_tick.js:68:7)
...
[00:11:51]                   └- ✖ fail: "maps app  maps loaded from sample data flights "before all" hook"
[00:11:51]                   │      Error: retry.try timeout: Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj~="mapListingTitleLink-[Flights]-Origin-and-Destination-Flight-Time"])
[00:11:51]                   │ Wait timed out after 10043ms
[00:11:51]                   │     at /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/node_modules/selenium-webdriver/lib/webdriver.js:834:17
[00:11:51]                   │     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:11:51]                   │     at lastError (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:29:9)
[00:11:51]                   │     at retryForSuccess (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:63:13)
[00:11:51]                   │       at lastError (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:29:9)
[00:11:51]                   │       at retryForSuccess (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7-3/node/immutable/kibana/test/common/services/retry/retry_for_success.js:63:13)

spalger · 2019-02-18T23:03:04Z

Jenkins, please test this so we can track down more flaky tests

elasticmachine · 2019-02-18T23:45:10Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

1 Failure

This one appears to be related to the issue described in #31388 (comment)

# note times, the timeout was basically ignored, it failed "instantly"
[00:01:36]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="saveDashboardSuccess"]') with timeout=120000
...
[00:01:37]                   └- ✖ fail: "reporting app Reporting Dashboard Print PDF button becomes available when saved"
[00:01:37]                   │      Error: expected testSubject(saveDashboardSuccess) to exist
[00:01:37]                   │       at TestSubjects.existOrFail (/var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup6-2/node/immutable/kibana/test/functional/services/test_subjects.js:53:15)
[00:01:37]                   │       at process._tickCallback (internal/process/next_tick.js:68:7)

spalger · 2019-02-19T00:14:42Z

Jenkins, please test this so we can track down more flaky tests

elasticmachine · 2019-02-19T00:58:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger · 2019-02-19T01:44:14Z

Jenkins, please test this so we can track down more flaky tests

spalger · 2019-02-19T01:49:09Z

Reopening as #31446 so that the root branch is on elastic/kibana and we can open PRs to it to focus on different ciGroups.

elasticmachine · 2019-02-19T02:27:41Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

spalger changed the title ~~Implement/ftr webdriver~~ [ ](/spalger) [Restore webdriver-based functional test runner driver](/elastic/kibana/pull/31388/commits/65fed1b3529a2e88dc8e10304f44662251b580a4) Feb 18, 2019

spalger changed the title ~~[ ](/spalger) [Restore webdriver-based functional test runner driver](/elastic/kibana/pull/31388/commits/65fed1b3529a2e88dc8e10304f44662251b580a4)~~ Restore webdriver-based functional test runner driver Feb 18, 2019

elastic deleted a comment from elasticmachine Feb 18, 2019

spalger commented Feb 18, 2019

View reviewed changes

spalger force-pushed the implement/ftr-webdriver branch 2 times, most recently from 4779b00 to 7f1ea2f Compare February 18, 2019 07:16

elastic deleted a comment from elasticmachine Feb 18, 2019

spalger added 2 commits February 17, 2019 23:18

Restore webdriver-based functional test runner driver

7127c40

run all function test suites 6 times per build, to help spot flakiness

4770993

spalger force-pushed the implement/ftr-webdriver branch from 532945b to 4770993 Compare February 18, 2019 07:18

[tests/reporting] fix flaky click on report generation

dbdbab2

This comment has been minimized.

Sign in to view

Merge remote-tracking branch 'upstream/master' into implement/ftr-web…

e92784a

…driver

spalger mentioned this pull request Feb 19, 2019

Restore webdriver-based functional test runner driver #31446

Merged

spalger closed this Feb 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore webdriver-based functional test runner driver #31388

Restore webdriver-based functional test runner driver #31388

spalger commented Feb 18, 2019 •

edited

Loading

spalger Feb 18, 2019 •

edited

Loading

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

spalger commented Feb 18, 2019

elasticmachine commented Feb 18, 2019

This comment has been minimized.

dmlemeshko commented Feb 18, 2019

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

spalger commented Feb 18, 2019

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

spalger commented Feb 19, 2019 •

edited

Loading

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

Restore webdriver-based functional test runner driver #31388

Restore webdriver-based functional test runner driver #31388

Conversation

spalger commented Feb 18, 2019 • edited Loading

spalger Feb 18, 2019 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Feb 18, 2019 • edited by spalger Loading

💔 Build Failed

spalger commented Feb 18, 2019

elasticmachine commented Feb 18, 2019

💚 Build Succeeded

This comment has been minimized.

dmlemeshko commented Feb 18, 2019

elasticmachine commented Feb 18, 2019 • edited by spalger Loading

💔 Build Failed

spalger commented Feb 18, 2019

elasticmachine commented Feb 18, 2019 • edited by spalger Loading

💔 Build Failed

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

💚 Build Succeeded

spalger commented Feb 19, 2019 • edited Loading

spalger commented Feb 19, 2019

elasticmachine commented Feb 19, 2019

💔 Build Failed

spalger commented Feb 18, 2019 •

edited

Loading

spalger Feb 18, 2019 •

edited

Loading

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

elasticmachine commented Feb 18, 2019 •

edited by spalger

Loading

spalger commented Feb 19, 2019 •

edited

Loading