Split Kibana ci job into much smaller chunks, more Jenkins workers, shorter elapsed time #22359

LeeDr · 2018-08-24T14:54:39Z

This is the first crack at splitting the UI Functional tests into 2 jobs. One for dashboard tests and the other non_dashboard (everything else).

No changes to x-pack tests yet.

Baseline from a very recent passing master build;

	elastic+kibana+master+multijob-intake	        ( 24 min )
	elastic+kibana+master+multijob-selenium     	( 1 hr 39 min )
	elastic+kibana+master+multijob-x-pack		( 1 hr 8 min )

LeeDr · 2018-08-24T17:49:03Z

Based on these results I'm going to split the visualize tests out into their own job. Dashboard about 25 minutes, Visualize about 26 minutes, and all the rest should be about 20 minutes)

Dashboard tests setup starting:

00:18:14.220 Running "run:functionalTestsReleaseDashboard" (run) task
00:18:17.206  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config_dashboard.js"
00:18:18.826  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"

Dashboard tests actually starting:

00:22:23.761    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config_dashboard.js"
00:22:23.766    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"

Dashboard tests finishing (26 minutes not counting setup):

00:48:09.979    │121 passing (25.0m)
00:48:09.979    │3 pending

Non-Dashboard tests setup starting:

00:48:40.581 Running "run:functionalTestsReleaseNonDashboard" (run) task
00:48:43.173  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config_non_dashboard.js"
00:48:43.508  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"

Non-Dashboard tests actually starting:

00:48:59.233    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config_non_dashboard.js"
00:48:59.233    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"
...test/functional/config_visualize.js
00:49:04.781      └-: console app (9 seconds)
00:49:13.235      └-: Getting Started  (1 minute)
00:50:54.701      └-: context app (16 seconds)
00:51:50.460      └-: discover app (3 minutes)
00:54:53.444      └-: homepage app (1 minute)
00:55:36.898      └-: management (12 minutes)
01:07:02.644      └-: status page (seconds)
01:07:05.049      └-: timelion app (seconds)
01:07:49.430      └-: visualize app (26 minutes)

Non-Dashboard tests finishing (45 minutes not counting setup):

01:33:54.066    │266 passing (44.0m)
01:33:54.067    │49 pending

Plugin functional tests setup starting:

01:34:24.694 Running "run:pluginFunctionalTestsRelease" (run) task
01:34:27.183  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/plugin_functional/config.js"
01:34:27.277  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config.js"
01:34:27.553  debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"

Plugin function tests actually starting:

01:36:04.500    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/plugin_functional/config.js"
01:36:04.500    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/functional/config.js"
01:36:04.502    │ debg Loading config file from "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/kibana/test/common/config.js"

Plugin functional tests finishing:

01:37:03.363    │5 passing (55.2s)

LeeDr · 2018-08-24T21:01:07Z

Starting to dig into the x-pack tests;

00:04:28.432 $ gulp test
00:04:49.391   1383 passing (5s)
00:04:49.392 
00:04:49.467 [15:26:41] Finished 'testserver' after 15 s
00:04:49.467 [15:26:41] Starting 'testbrowser'...

00:07:49.847 Done.
00:07:49.956 [15:29:41] Finished 'testbrowser' after 3 min
00:07:49.957 [15:29:41] Finished 'test' after 3.27 min
00:07:50.888 Done in 202.57s.
00:07:50.900 
00:07:50.900 
00:07:50.900  -> Running jest tests
00:08:54.338 Test Suites: 139 passed, 139 total
00:08:54.338 Tests:       703 passed, 703 total
00:08:54.340 Snapshots:   279 passed, 279 total
00:08:54.340 Time:        60.314s
00:08:54.342 Ran all test suites.

00:08:55.017  -> building and extracting default Kibana distributable for use in functional tests

00:28:04.152  -> Running functional and api tests
00:31:31.803    │ debg BUILD SUCCESSFUL in 3m 19s

  require.resolve('../test/reporting/configs/phantom_api.js'),
00:34:24.175      └-: phantom
00:36:32.391    │11 passing (2.0m)

  require.resolve('../test/reporting/configs/phantom_functional.js'),
00:37:14.190      └-: reporting app
00:43:02.764    │12 passing (5.0m)

  require.resolve('../test/functional/config.js'),
00:43:44.638    │ info Starting tests
00:43:44.639    │
00:43:44.639      └-: graph app (these are all skipped right now)
00:43:44.747      └-: Monitoring app
00:50:31.139      └-: watcher app
00:50:43.643      └-: dashboard mode
00:52:56.018      └-: security app
01:00:26.62600:04:28.432 $ gulp test
      └-: logstash
01:01:54.751        └-: grok debugger app
01:02:04.075    │133 passing (18.0m)
01:02:04.075    │12 pending
01:02:04.076    │
01:02:04.291    │ info [kibana] exited with null after 19 minutes

  require.resolve('../test/api_integration/config.js'),
01:02:42.324      └-: apis


  require.resolve('../test/saml_api_integration/config.js'),
01:04:06.123      └-: apis SAML

  require.resolve('../test/rbac_api_integration/config.js'),
01:05:51.290      └-: apis RBAC
01:06:22.421    │108 passing (31.1s)

Total for this jenkins job ( 1 hr 8 min )

LeeDr · 2018-08-24T21:10:41Z

With the latest commit to split Visualize out to it's own config, the total Selenium job was ( 1 hr 41 min ).

00:13:05.042    │ debg Deleted 370 files/directories
00:13:05.043    │ succ ✓ 4 min 48 sec  < almost 5 minutes to delete files/dirs

00:19:19.111 Running "run:functionalTestsReleaseDashboard" (run) task
00:22:00.043    │ debg BUILD SUCCESSFUL in 2m 34s
00:23:57.059    │ info Starting tests
00:23:57.061    │
00:23:57.062      └-: dashboard app
00:49:06.214    │121 passing (25.0m)
00:49:06.214    │3 pending

00:49:36.833 Running "run:functionalTestsReleaseVisualize" (run) task
00:50:00.042      └-: visualize app
01:16:09.988    │134 passing (26.0m)
01:16:09.989    │43 pending

01:16:40.566 Running "run:functionalTestsReleaseNonDashboard" (run) task
01:17:04.162      └-: console app
01:35:43.807    │132 passing (18.0m)
01:35:43.808    │6 pending

01:36:14.445 Running "run:pluginFunctionalTestsRelease" (run) task
01:38:49.463    │5 passing (54.0s)

What I'm seeing from this, is that the first set of functional tests in the Selenium job (currently the dashboard tests) don't start until 00:19:19 (in this case, to 00:22:23 in the first case further above). And those dashboard tests finish 48 to 49 minutes after the job started. @spalger has some ideas to speed this up.
At the moment, if we split this Selenium job into 3 jobs which each have to do all the same setup, they would each take 40 to 50 minutes. This is about an hour shorter than the current 1 hr 39 minutes.

liza-mae · 2018-08-27T20:04:56Z

@LeeDr I would still like to be able to run everything together locally, will that be possible? Also, will I need to make the same changes in current cloud CI and upcoming windows support? If so, I will take a look at what changes we need to do and we can discuss after this PR is finalized. Thanks.

LeeDr · 2018-09-01T00:14:00Z

OK with the new ciGroups implementation (on UI functional Selenium only, not X-Pack) we got this;

00:23:56.966      └-: console app @ciGroup1 ( 28 seconds)
00:24:24.115        └-: @ciGroup10 (need to use 01, 02, etc.)
00:33:41.821        └-: @ciGroup11 (need to use 01, 02, etc.)

00:51:07.304      └-: context app @ciGroup2 (3 minutes)
00:54:02.853          └-: @ciGroup3  (and we should name them all I think) (8 minutes)
01:01:46.292          └-: @ciGroup4   (2 minutes)
01:03:57.782        └-: using legacy data @ciGroup5 (14 minutes)
01:17:05.727      └-: discover app @ciGroup6 (4 minutes)
01:20:59.809      └-: Getting Started @ciGroup7 (2 minutes)
01:22:43.136      └-: homepage app @ciGroup7 (2 minutes)
01:24:19.173      └-: management @ciGroup8 (11 minutes)
01:35:41.824      └-: status page @ciGroup8 (2 minutes for both)
01:35:44.226      └-: timelion app @ciGroup8
01:37:41.654        └-: @ciGroup9 (9 minutes)
01:46:12.653        └-: @ciGroup10 (10 minutes)
01:56:35.883        └-: @ciGroup11 (17 minutes)
02:13:02.309    │92 passing (16.0m)
02:13:02.310    │21 pending

$ grep -rn ciGroup test/functional/apps/*
test/functional/apps/console/index.js:23:  describe('console app @ciGroup1', function () {
test/functional/apps/context/index.js:25:  describe('context app @ciGroup2', function () {
test/functional/apps/dashboard/index.js:45:      describe('@ciGroup3', function () {
test/functional/apps/dashboard/index.js:56:      describe('@ciGroup4', function () {
test/functional/apps/dashboard/index.js:72:    describe('using legacy data @ciGroup5', function () {
test/functional/apps/discover/index.js:24:  describe('discover app @ciGroup6', function () {
test/functional/apps/getting_started/index.js:23:  describe('Getting Started @ciGroup7', function () {
test/functional/apps/home/index.js:23:  describe('homepage app @ciGroup7', function () {
test/functional/apps/management/index.js:23:  describe('management @ciGroup8', function () {
test/functional/apps/status_page/index.js:27:  describe('status page @ciGroup8', function () {
test/functional/apps/timelion/index.js:26:  describe('timelion app @ciGroup8', function () {
test/functional/apps/visualize/index.js:35:    describe('@ciGroup9', function () {
test/functional/apps/visualize/index.js:46:    describe('@ciGroup10', function () {
test/functional/apps/visualize/index.js:59:    describe('@ciGroup11', function () {

I'm going to change a few things up and try again.

LeeDr · 2018-09-01T03:39:54Z

Latest test;

00:22:58.238      └-: console app @ciGroup01 (2.5 minutes)
00:23:06.446      └-: context app @ciGroup01
00:25:28.104          └-: dashboard @ciGroup02 (8 minutes)
00:33:09.986          └-: dashboard @ciGroup03 (7 minutes)
00:40:03.408        └-: using legacy data @ciGroup04 (8 minutes)
00:47:47.511        └-: using legacy data @ciGroup05 (7 minutes)
00:54:16.288      └-: Getting Started @ciGroup06 (6 minutes) 
00:55:59.056      └-: discover app @ciGroup06 
00:59:02.940      └-: homepage app @ciGroup06
01:00:38.927      └-: management @ciGroup07  (12 minutes)
01:12:59.153      └-: status page @ciGroup08 (3 minutes) (-> group01)
01:13:03.127      └-: timelion app @ciGroup08         (-> group01)
01:15:36.928        └-: visualize @ciGroup09 (10 minutes)
01:25:09.631        └-: visualize @ciGroup10 (10 minutes)
01:35:48.652        └-: visualize @ciGroup11 (17 minutes) (I'll split half of these out to group 12)
01:52:14.669        └-> "after all" hook

LeeDr · 2018-09-02T14:44:07Z

jenkins test this

LeeDr · 2018-09-02T20:05:49Z

Last run failed on this selenium test.

01:34:00.123              └- ✖ fail: "visualize app visualize @ciGroup10 metric chart should show Count"
01:34:00.123              │      retry.try timeout: Error: retry.try timeout: [POST http://localhost:9515/session/c8c1bb858140aa9b404ba3f13129d73c/element / {"using":"css selector","value":"div[ng-controller=\"KbnMetricVisController\"]"}] no such element: Unable to locate element: {"method":"css selector","selector":"div[ng-controller="KbnMetricVisController"]"}

Jenkins test this

…alJobs

elasticmachine · 2018-11-16T07:47:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-11-16T09:37:42Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-11-16T17:29:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-11-16T18:39:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-11-16T20:09:25Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

## Summary This pr adds a `--test-stats` flag to the functional test runner that will allow extracting information from a specific config/tags combo about how many tests will run and how many are excluded. This is necessary as part of elastic#22359 so we can effectively ignore specific configs (avoiding starting es and kibana) when there aren't any tests that will be run. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

## Summary This pr adds a `--test-stats` flag to the functional test runner that will allow extracting information from a specific config/tags combo about how many tests will run and how many are excluded. This is necessary as part of #22359 so we can effectively ignore specific configs (avoiding starting es and kibana) when there aren't any tests that will be run. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

spalger · 2018-11-17T00:25:57Z

This PR has gotten a little out of control, so I'm resubmitting as #25838

## Summary This pr adds a `--test-stats` flag to the functional test runner that will allow extracting information from a specific config/tags combo about how many tests will run and how many are excluded. This is necessary as part of #22359 so we can effectively ignore specific configs (avoiding starting es and kibana) when there aren't any tests that will be run. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

Split out dashboard tests to separate test_runner run

034f693

LeeDr added test WIP Work in progress test_ui_functional labels Aug 24, 2018

LeeDr requested a review from spalger August 24, 2018 14:57

Add missing config files

a188f8b

This comment has been minimized.

Sign in to view

Split out visualize tests

9dbaef8

This comment has been minimized.

Sign in to view

Switch to using grep parameters to create ciGroups

1e0457b

This comment has been minimized.

Sign in to view

refactor ciGroups, use 2-digit index numbers

a338bdd

This comment has been minimized.

Sign in to view

refactor groups

a4cddb9

This comment has been minimized.

Sign in to view

sorted visualize tests into groups

717cfda

This comment has been minimized.

Sign in to view

remove metric chart

2b1c3e3

This comment has been minimized.

Sign in to view

Merge branch 'master' of github.com:elastic/kibana into splitFunction…

ab829dd

…alJobs

[ftr] fix description of --test-stats flag

9a0a42c

spalger mentioned this pull request Nov 15, 2018

[ftr] add ability to get included/excluded test counts #25760

Merged

spalger added 3 commits November 15, 2018 15:05

[kbn/test] fix returns in a for loop

e934540

[xpack/ftr] use unique report names

e9f516a

[ftr] cache config values

4dfee2f

spalger force-pushed the splitFunctionalJobs branch from 685a673 to 4dfee2f Compare November 16, 2018 01:10

elastic deleted a comment from elasticmachine Nov 16, 2018

[x-pack] create a 5th ciGroup

8c863eb

elastic deleted a comment from elasticmachine Nov 16, 2018

[ci] split xpack group 1

2c54862

spalger force-pushed the splitFunctionalJobs branch from a6b7de4 to 2c54862 Compare November 16, 2018 08:11

spalger removed their request for review November 16, 2018 08:39

spalger removed the review label Nov 16, 2018

[ci] distinguish between kibana and xpack jobs of same type

b565d96

This comment has been minimized.

Sign in to view

elastic deleted a comment from elasticmachine Nov 16, 2018

spalger mentioned this pull request Nov 17, 2018

[ci][ftr][kbn/test] split up CI jobs #25838

Merged

spalger closed this Nov 17, 2018

LeeDr deleted the splitFunctionalJobs branch August 20, 2020 17:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split Kibana ci job into much smaller chunks, more Jenkins workers, shorter elapsed time #22359

Split Kibana ci job into much smaller chunks, more Jenkins workers, shorter elapsed time #22359

LeeDr commented Aug 24, 2018 •

edited

Loading

This comment has been minimized.

LeeDr commented Aug 24, 2018

This comment has been minimized.

LeeDr commented Aug 24, 2018

LeeDr commented Aug 24, 2018 •

edited

Loading

liza-mae commented Aug 27, 2018

This comment has been minimized.

LeeDr commented Sep 1, 2018

This comment has been minimized.

LeeDr commented Sep 1, 2018

This comment has been minimized.

This comment has been minimized.

LeeDr commented Sep 2, 2018

This comment has been minimized.

LeeDr commented Sep 2, 2018

This comment has been minimized.

This comment has been minimized.

elasticmachine commented Nov 16, 2018

This comment has been minimized.

elasticmachine commented Nov 16, 2018

elasticmachine commented Nov 16, 2018

elasticmachine commented Nov 16, 2018

elasticmachine commented Nov 16, 2018

spalger commented Nov 17, 2018

Split Kibana ci job into much smaller chunks, more Jenkins workers, shorter elapsed time #22359

Split Kibana ci job into much smaller chunks, more Jenkins workers, shorter elapsed time #22359

Conversation

LeeDr commented Aug 24, 2018 • edited Loading

This comment has been minimized.

LeeDr commented Aug 24, 2018

This comment has been minimized.

LeeDr commented Aug 24, 2018

LeeDr commented Aug 24, 2018 • edited Loading

liza-mae commented Aug 27, 2018

This comment has been minimized.

LeeDr commented Sep 1, 2018

This comment has been minimized.

LeeDr commented Sep 1, 2018

This comment has been minimized.

This comment has been minimized.

LeeDr commented Sep 2, 2018

This comment has been minimized.

LeeDr commented Sep 2, 2018

This comment has been minimized.

This comment has been minimized.

elasticmachine commented Nov 16, 2018

💚 Build Succeeded

This comment has been minimized.

elasticmachine commented Nov 16, 2018

💚 Build Succeeded

elasticmachine commented Nov 16, 2018

💚 Build Succeeded

elasticmachine commented Nov 16, 2018

💚 Build Succeeded

elasticmachine commented Nov 16, 2018

💚 Build Succeeded

spalger commented Nov 17, 2018

LeeDr commented Aug 24, 2018 •

edited

Loading

LeeDr commented Aug 24, 2018 •

edited

Loading