Refactor test entry by runner (elastic#44679)

Over the last few years we've increased the number of test runners. Entry points by test type have become too lage of a category for unique names, so this moves top level test scripts under yarn and replaces test types with the runner name. e.g. `yarn test:browser` -> `yarn test:karma` Closes elastic#41133 Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # packages/kbn-plugin-generator/sao_template/template/README.md # packages/kbn-plugin-helpers/lib/tasks.js # test/scripts/jenkins_xpack.sh
spalger · May 15, 2020 · f455d6a · f455d6a
1 parent 97a6cec
commit f455d6a
Show file tree

Hide file tree

Showing 30 changed files with 88 additions and 136 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -459,11 +459,11 @@ The following table outlines possible test file locations and how to invoke them
 
 | Test runner        | Test location                                                                                                                                           | Runner command (working directory is kibana root)                                       |
 | -----------------  | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
-| Jest               | `src/**/*.test.js`<br>`src/**/*.test.ts`                                                                                                                | `node scripts/jest -t regexp [test path]`                                               |
-| Jest (integration) | `**/integration_tests/**/*.test.js`                                                                                                                     | `node scripts/jest_integration -t regexp [test path]`                                   |
+| Jest               | `src/**/*.test.js`<br>`src/**/*.test.ts`                                                                                                                | `yarn test:jest -t regexp [test path]`                                               |
+| Jest (integration) | `**/integration_tests/**/*.test.js`                                                                                                                     | `yarn test:jest_integration -t regexp [test path]`                                   |
 | Mocha              | `src/**/__tests__/**/*.js`<br>`!src/**/public/__tests__/*.js`<br>`packages/kbn-datemath/test/**/*.js`<br>`packages/kbn-dev-utils/src/**/__tests__/**/*.js`<br>`tasks/**/__tests__/**/*.js` | `node scripts/mocha --grep=regexp [test path]`       |
-| Functional         | `test/*integration/**/config.js`<br>`test/*functional/**/config.js`<br>`test/accessibility/config.js`                                                                                    | `node scripts/functional_tests_server --config test/[directory]/config.js`<br>`node scripts/functional_test_runner --config test/[directory]/config.js --grep=regexp`       |
-| Karma              | `src/**/public/__tests__/*.js`                                                                                                                          | `npm run test:dev`                                                                      |
+| Functional         | `test/*integration/**/config.js`<br>`test/*functional/**/config.js`<br>`test/accessibility/config.js`                                                                                    | `yarn test:ftr:server --config test/[directory]/config.js`<br>`yarn test:ftr:runner --config test/[directory]/config.js --grep=regexp`       |
+| Karma              | `src/**/public/__tests__/*.js`                                                                                                                          | `yarn test:karma:debug`                                                                      |
 
 For X-Pack tests located in `x-pack/` see [X-Pack Testing](x-pack/README.md#testing)
 
@@ -474,56 +474,38 @@ Test runner arguments:
  Examples:
   - Run the entire elasticsearch_service test suite:
     ```
-    node scripts/jest src/core/server/elasticsearch/elasticsearch_service.test.ts
+    yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts
     ```
   - Run the jest test case whose description matches `stops both admin and data clients`:
     ```
-    node scripts/jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts
+    yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts
     ```
   - Run the api integration test case whose description matches the given string:
     ```
-    node scripts/functional_tests_server --config test/api_integration/config.js
-    node scripts/functional_test_runner --config test/api_integration/config.js --grep='should return 404 if id does not match any sample data sets'
+    yarn test:ftr:server --config test/api_integration/config.js
+    yarn test:ftr:runner --config test/api_integration/config.js --grep='should return 404 if id does not match any sample data sets'
     ```
 
 ### Debugging Unit Tests
 
 The standard `yarn test` task runs several sub tasks and can take several minutes to complete, making debugging failures pretty painful. In order to ease the pain specialized tasks provide alternate methods for running the tests.
 
-To execute both server and browser tests, but skip linting, use `yarn test:quick`.
-
-```bash
-yarn test:quick
-```
-
-Use `yarn test:mocha` when you want to run the mocha tests.
-
-```bash
-yarn test:mocha
-```
-
-When you'd like to execute individual server-side test files, you can use the command below. Note that this command takes care of configuring Mocha with Babel compilation for you, and you'll be better off avoiding a globally installed `mocha` package. This command is great for development and for quickly identifying bugs.
-
-```bash
-node scripts/mocha <file>
-```
-
 You could also add the `--debug` option so that `node` is run using the `--debug-brk` flag. You'll need to connect a remote debugger such as [`node-inspector`](https://github.com/node-inspector/node-inspector) to proceed in this mode.
 
 ```bash
 node scripts/mocha --debug <file>
 ```
 
-With `yarn test:browser`, you can run only the browser tests. Coverage reports are available for browser tests by running `yarn test:coverage`. You can find the results under the `coverage/` directory that will be created upon completion.
+With `yarn test:karma`, you can run only the browser tests. Coverage reports are available for browser tests by running `yarn test:coverage`. You can find the results under the `coverage/` directory that will be created upon completion.
 
 ```bash
-yarn test:browser
+yarn test:karma
 ```
 
-Using `yarn test:dev` initializes an environment for debugging the browser tests. Includes an dedicated instance of the kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests.
+Using `yarn test:karma:debug` initializes an environment for debugging the browser tests. Includes an dedicated instance of the kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests.
 
 ```bash
-yarn test:dev
+yarn test:karma:debug
 ```
 
 In the screenshot below, you'll notice the URL is `localhost:9876/debug.html`. You can append a `grep` query parameter to this URL and set it to a string value which will be used to exclude tests which don't match. For example, if you changed the URL to `localhost:9876/debug.html?query=my test` and then refreshed the browser, you'd only see tests run which contain "my test" in the test description.
@@ -539,7 +521,7 @@ To run the tests for just your particular plugin run the following command from
 
 ```bash
 yarn test:mocha
-yarn test:browser --dev # remove the --dev flag to run them once and close
+yarn test:karma:debug # remove the debug flag to run them once and close
 ```
 
 ### Cross-browser Compatibility

diff --git a/package.json b/package.json
@@ -40,23 +40,21 @@
     "kbn": "node scripts/kbn",
     "es": "node scripts/es",
     "test": "grunt test",
-    "test:dev": "grunt test:dev",
-    "test:quick": "grunt test:quick",
-    "test:browser": "grunt test:browser",
+    "test:karma": "grunt test:karma",
+    "test:karma:debug": "grunt test:karmaDebug",
     "test:jest": "node scripts/jest",
     "test:mocha": "node scripts/mocha",
     "test:mocha:coverage": "grunt test:mochaCoverage",
-    "test:ui": "node scripts/functional_tests",
-    "test:ui:server": "node scripts/functional_tests_server",
-    "test:ui:runner": "node scripts/functional_test_runner",
+    "test:ftr": "node scripts/functional_tests",
+    "test:ftr:server": "node scripts/functional_tests_server",
+    "test:ftr:runner": "node scripts/functional_test_runner",
     "test:coverage": "grunt test:coverage",
     "typespec": "typings-tester --config x-pack/legacy/plugins/canvas/public/lib/aeroelastic/tsconfig.json x-pack/legacy/plugins/canvas/public/lib/aeroelastic/__fixtures__/typescript/typespec_tests.ts",
     "checkLicenses": "node scripts/check_licenses --dev",
     "build": "node scripts/build --all-platforms",
     "start": "node --trace-warnings --throw-deprecation scripts/kibana --dev",
     "debug": "node --nolazy --inspect scripts/kibana --dev",
     "debug-break": "node --nolazy --inspect-brk scripts/kibana --dev",
-    "karma": "karma start",
     "lint": "yarn run lint:es && yarn run lint:sass",
     "lint:es": "node scripts/eslint",
     "lint:sass": "node scripts/sasslint",

diff --git a/packages/kbn-plugin-generator/README.md b/packages/kbn-plugin-generator/README.md
@@ -71,7 +71,7 @@ Generated plugins receive a handful of scripts that can be used during developme
 
     Build a distributable archive of your plugin.
 
-  - `yarn test:browser`
+  - `yarn test:karma`
 
     Run the browser tests in a real web browser.
 

diff --git a/packages/kbn-plugin-generator/integration_tests/generate_plugin.test.js b/packages/kbn-plugin-generator/integration_tests/generate_plugin.test.js
@@ -69,8 +69,8 @@ describe(`running the plugin-generator via 'node scripts/generate_plugin.js plug
   });
 
   describe(`then running`, () => {
-    it(`'yarn test:browser' should exit 0`, async () => {
-      await execa('yarn', ['test:browser'], {
+    it(`'yarn test:karma' should exit 0`, async () => {
+      await execa('yarn', ['test:karma'], {
         cwd: generatedPath,
         env: {
           DISABLE_JUNIT_REPORTER: '1',

diff --git a/packages/kbn-plugin-generator/sao_template/template/README.md b/packages/kbn-plugin-generator/sao_template/template/README.md
@@ -8,32 +8,4 @@
 
 ## development
 
-See the [kibana contributing guide](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md) for instructions setting up your development environment. Once you have completed that, use the following yarn scripts.
-
-  - `yarn kbn bootstrap`
-
-    Install dependencies and crosslink Kibana and all projects/plugins.
-
-    > ***IMPORTANT:*** Use this script instead of `yarn` to install dependencies when switching branches, and re-run it whenever your dependencies change.
-
-  - `yarn start`
-
-    Start kibana and have it include this plugin. You can pass any arguments that you would normally send to `bin/kibana`
-
-      ```
-      yarn start --elasticsearch.hosts http://localhost:9220
-      ```
-
-  - `yarn build`
-
-    Build a distributable archive of your plugin.
-
-  - `yarn test:browser`
-
-    Run the browser tests in a real web browser.
-
-  - `yarn test:mocha`
-
-    Run the server tests using mocha.
-
-For more information about any of these commands run `yarn ${task} --help`. For a full list of tasks checkout the `package.json` file, or run `yarn run`.
+See the [kibana contributing guide](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md) for instructions setting up your development environment.
diff --git a/packages/kbn-plugin-helpers/README.md b/packages/kbn-plugin-helpers/README.md
@@ -30,8 +30,8 @@ $ plugin-helpers help
     start                       Start kibana and have it include this plugin
     build [options] [files...]  Build a distributable archive
     test                        Run the server and browser tests
-    test:browser [options]      Run the browser tests in a real web browser
-    test:server [files...]      Run the server tests using mocha
+    test:karma [options]      Run the browser tests in a real web browser
+    test:mocha [files...]      Run the server tests using mocha
 
   Options:
 

diff --git a/packages/kbn-plugin-helpers/cli.js b/packages/kbn-plugin-helpers/cli.js
@@ -66,24 +66,24 @@ program
   .action(createCommanderAction('testAll'));
 
 program
-  .command('test:browser')
+  .command('test:karma')
   .description('Run the browser tests in a real web browser')
   .option('--dev', 'Enable dev mode, keeps the test server running')
   .option('-p, --plugins <plugin-ids>', "Manually specify which plugins' test bundles to run")
-  .on('--help', docs('test/browser'))
+  .on('--help', docs('test/karma'))
   .action(
-    createCommanderAction('testBrowser', command => ({
+    createCommanderAction('testKarma', command => ({
       dev: Boolean(command.dev),
       plugins: command.plugins,
     }))
   );
 
 program
-  .command('test:server [files...]')
+  .command('test:mocha [files...]')
   .description('Run the server tests using mocha')
-  .on('--help', docs('test/server'))
+  .on('--help', docs('test/mocha'))
   .action(
-    createCommanderAction('testServer', (command, files) => ({
+    createCommanderAction('testMocha', (command, files) => ({
       files: files,
     }))
   );

diff --git a/packages/kbn-plugin-helpers/lib/index.d.ts b/packages/kbn-plugin-helpers/lib/index.d.ts
@@ -21,6 +21,6 @@ export function babelRegister(): void;
 export function resolveKibanaPath(path: string): string;
 export function readFtrConfigFile(path: string): any;
 export function run(
-  task: 'build' | 'start' | 'testAll' | 'testBrowser' | 'testServer' | 'postinstall',
+  task: 'build' | 'start' | 'testAll' | 'testKarma' | 'testMocha' | 'postinstall',
   options: any
 ): Promise<void>;
diff --git a/packages/kbn-plugin-helpers/lib/tasks.js b/packages/kbn-plugin-helpers/lib/tasks.js
@@ -20,13 +20,13 @@
 const buildTask = require('../tasks/build');
 const startTask = require('../tasks/start');
 const testAllTask = require('../tasks/test/all');
-const testBrowserTask = require('../tasks/test/browser');
-const testServerTask = require('../tasks/test/server');
+const testKarmaTask = require('../tasks/test/karma');
+const testMochaTask = require('../tasks/test/mocha');
 
 module.exports = {
   build: buildTask,
   start: startTask,
   testAll: testAllTask,
-  testBrowser: testBrowserTask,
-  testServer: testServerTask,
+  testKarma: testKarmaTask,
+  testMocha: testMochaTask,
 };
diff --git a/packages/kbn-plugin-helpers/tasks/test/all/README.md b/packages/kbn-plugin-helpers/tasks/test/all/README.md
@@ -1,3 +1,3 @@
-Runs both the server and browser tests, in that order. 
+Runs both the mocha and karma tests, in that order.
 
-This is just a simple caller to both `test/server` and `test/browser`
+This is just a simple caller to both `test/mocha` and `test/karma`
diff --git a/packages/kbn-plugin-helpers/tasks/test/all/test_all_action.js b/packages/kbn-plugin-helpers/tasks/test/all/test_all_action.js
@@ -18,6 +18,6 @@
  */
 
 module.exports = function testAllAction(plugin, run) {
-  run('testServer');
-  run('testBrowser');
+  run('testMocha');
+  run('testKarma');
 };
diff --git a/...ugin-helpers/tasks/test/browser/README.md → ...plugin-helpers/tasks/test/karma/README.md b/...ugin-helpers/tasks/test/browser/README.md → ...plugin-helpers/tasks/test/karma/README.md
@@ -21,12 +21,12 @@ Browser tests are written just like server tests, they are just executed differe
 starting the test runner
 ========================
 
-Under the covers this command uses the `test:browser` task from kibana. This will execute
+Under the covers this command uses the `test:karma` task from kibana. This will execute
 your tasks once and exit when complete.
 
-When run with the `--dev` option, the command uses the `test:dev` task from kibana. 
-This task sets-up a test runner that will watch your code for changes and rebuild your 
-tests when necessary. You access the test runner through a browser that it starts itself 
+When run with the `--dev` option, the command uses the `test:karma:debug` task from kibana.
+This task sets-up a test runner that will watch your code for changes and rebuild your
+tests when necessary. You access the test runner through a browser that it starts itself
 (via Karma).
 
 If your plugin consists of a number of internal plugins, you may wish to keep the tests

diff --git a/...plugin-helpers/tasks/test/server/index.js → ...-plugin-helpers/tasks/test/karma/index.js b/...plugin-helpers/tasks/test/server/index.js → ...-plugin-helpers/tasks/test/karma/index.js
@@ -17,4 +17,4 @@
  * under the License.
  */
 
-module.exports = require('./test_server_action');
+module.exports = require('./test_karma_action');
diff --git a/...tasks/test/browser/test_browser_action.js → ...ers/tasks/test/karma/test_karma_action.js b/...tasks/test/browser/test_browser_action.js → ...ers/tasks/test/karma/test_karma_action.js
@@ -20,7 +20,7 @@
 const execFileSync = require('child_process').execFileSync;
 const winCmd = require('../../../lib/win_cmd');
 
-module.exports = function testBrowserAction(plugin, run, options) {
+module.exports = function testKarmaAction(plugin, run, options) {
   options = options || {};
 
   const kbnServerArgs = ['--kbnServer.plugin-path=' + plugin.root];
@@ -31,7 +31,7 @@ module.exports = function testBrowserAction(plugin, run, options) {
     kbnServerArgs.push('--kbnServer.tests_bundle.pluginId=' + plugin.id);
   }
 
-  const task = options.dev ? 'test:dev' : 'test:browser';
+  const task = options.dev ? 'test:karma:debug' : 'test:karma';
   const args = [task].concat(kbnServerArgs);
   execFileSync(winCmd('yarn'), args, {
     cwd: plugin.kibanaRoot,

diff --git a/...lugin-helpers/tasks/test/server/README.md → ...plugin-helpers/tasks/test/mocha/README.md b/...lugin-helpers/tasks/test/server/README.md → ...plugin-helpers/tasks/test/mocha/README.md
@@ -20,14 +20,14 @@ Server tests are written just like browser tests, they are just executed differe
 running the tests
 =================
 
-Running the server tests is simple, just execute `yarn test:server` in your terminal
+Running the server tests is simple, just execute `yarn test:mocha` in your terminal
 and all of the tests in your server will be run.
 
 By default, the runner will look for tests in `server/**/__tests__/**/*.js`. If you'd prefer to
-use a different collection of globs and files, you can specify them after the `yarn test:server`
+use a different collection of globs and files, you can specify them after the `yarn test:mocha`
 task, like so:
 
-`yarn test:server 'plugins/myplugins/server/__tests__/**/*.js'`
+`yarn test:mocha 'plugins/myplugins/server/__tests__/**/*.js'`
 
 NOTE: quoting the glob pattern is not required, but helps to avoid issues with globbing expansion
 in your shell.

diff --git a/...lugin-helpers/tasks/test/browser/index.js → ...-plugin-helpers/tasks/test/mocha/index.js b/...lugin-helpers/tasks/test/browser/index.js → ...-plugin-helpers/tasks/test/mocha/index.js
@@ -17,4 +17,4 @@
  * under the License.
  */
 
-module.exports = require('./test_browser_action');
+module.exports = require('./test_mocha_action');
diff --git a/...s/tasks/test/server/test_server_action.js → ...ers/tasks/test/mocha/test_mocha_action.js b/...s/tasks/test/server/test_server_action.js → ...ers/tasks/test/mocha/test_mocha_action.js
diff --git a/tasks/config/karma.js b/tasks/config/karma.js
@@ -176,11 +176,11 @@ module.exports = function(grunt) {
    *  (&shard_num=Y), are added to the testing bundle url and read by the
    *  test_harness/setup_test_sharding[1] module. This allows us to use a
    *  different number of shards in different scenarios (ie. running
-   *  `yarn test:browser` runs the tests in a single shard, effectively
+   *  `yarn test:karma` runs the tests in a single shard, effectively
    *  disabling sharding)
    *
    *  These same parameters can also be defined in the URL/query string of the
-   *  karma debug page (started when you run `yarn test:dev`).
+   *  karma debug page (started when you run `yarn test:karma:debug`).
    *
    *  ## debugging
    *

diff --git a/tasks/config/run.js b/tasks/config/run.js
@@ -54,7 +54,7 @@ module.exports = function(grunt) {
     };
   }
 
-  const browserTestServerFlags = [
+  const karmaTestServerFlags = [
     '--env.name=development',
     '--plugins.initialize=false',
     '--optimize.bundleFilter=tests',
@@ -158,27 +158,27 @@ module.exports = function(grunt) {
       ],
     }),
 
-    // used by the test:browser task
+    // used by the test:karma task
     //    runs the kibana server to serve the browser test bundle
-    browserTestServer: createKbnServerTask({
-      flags: [...browserTestServerFlags],
+    karmaTestServer: createKbnServerTask({
+      flags: [...karmaTestServerFlags],
     }),
     browserSCSS: createKbnServerTask({
-      flags: [...browserTestServerFlags, '--optimize', '--optimize.enabled=false'],
+      flags: [...karmaTestServerFlags, '--optimize', '--optimize.enabled=false'],
     }),
 
     // used by the test:coverage task
     //    runs the kibana server to serve the instrumented version of the browser test bundle
-    browserTestCoverageServer: createKbnServerTask({
-      flags: [...browserTestServerFlags, '--tests_bundle.instrument=true'],
+    karmaTestCoverageServer: createKbnServerTask({
+      flags: [...karmaTestServerFlags, '--tests_bundle.instrument=true'],
     }),
 
-    // used by the test:dev task
+    // used by the test:karma:debug task
     //    runs the kibana server to serve the browser test bundle, but listens for changes
     //    to the public/browser code and rebuilds the test bundle on changes
-    devBrowserTestServer: createKbnServerTask({
+    karmaTestDebugServer: createKbnServerTask({
       flags: [
-        ...browserTestServerFlags,
+        ...karmaTestServerFlags,
         '--dev',
         '--no-dev-config',
         '--no-watch',
@@ -292,7 +292,7 @@ module.exports = function(grunt) {
       'test:jest_integration'
     ),
     test_projects: gruntTaskWithGithubChecks('Project tests', 'test:projects'),
-    test_browser_ci: gruntTaskWithGithubChecks('Browser tests', 'test:browser-ci'),
+    test_karma_ci: gruntTaskWithGithubChecks('Browser tests', 'test:karma-ci'),
 
     ...getFunctionalTestGroupRunConfigs({
       kibanaInstallDir: KIBANA_INSTALL_DIR,