Unit testing and E2E Testing

The ionic-quickstarter project contains two types of tests:

Unit tests implemented using Karma and Jasmine. The unit tests are in the test/unit folder.
End-to-end (e2e) tests implemented using Protractor. The e2e tests are in the test/e2e folder.

Unit tests should mainly test the logic of individual files/modules, and should run fast.

This is achieved by running with 'mock' implementations rather than 'production' implementations of services, for instance: using the "UserServiceMockImpl" implementation rather than the "UserServiceFirebaseImpl" implementation of the "UserService" service.

Protractor (e2e) tests run against the "real" app, emulating an end user interacting with the app through a browser. The app would be using 'mock' services (by default, if the app is running in 'dev' mode), or 'production' services (if the app is running in 'production' mode).

For both types of tests (unit and e2e) we created only a very limited number of tests. These represent in no way a complete test suite for the app, the goal is just to show how to set up the tooling (Protractor, Karma etc) and to demonstrate some best practices for structuring test code.

Unit Testing

Unit tests are under the test/unit folder. They are organized by AngularJS module, so there is a separate subdirectory per module. For instance, there is a config subdirectory containing the tests for the app.config module, and a mainPage subdirectory containing the tests for the app.mainPage module.

To run the unit tests, simply execute the following command from the project's main directory:

gulp test

This will start the Karma test runner and open a browser window (using Google Chrome by default).

Don't close the browser, if you do you stop the test runner (Karma). By default, Karma will keep running in the background, monitoring continuously for changed files under the test/unit folder and automatically re-executing the changed tests.

Configuration of the Karma test runner

Karma is configure through a karma.conf.js file in the project's base directory.

If you look at the karma.conf.js file supplied with the ionic-quickstarter project you can see how this works. For instance, you specify which app files to load for testing through the files array in the config file.

End to End (E2E) Testing

The project is set up for basic end to end (E2E) testing via Protractor. You can read more about Protractor here.

Machine Setup

Make sure your machine has python installed. Type python in the command line you use for building, and if it is not found, please install python from python.org.

Note that you may also need to add python to your path, especially if you are installing on Windows and running from PowerShell.

You will also need the Java Runtime Environment. Web driver will prompt you to install it, if you don't have it already installed.

If you're on a Mac, make sure you install the JRE from Apple, not Oracle. See here: https://support.apple.com/kb/DL1572

The tests use Google Chrome by default. You can change what browser you use in the protractor.conf.js file, or just make sure you have the Chrome browser installed on the machine you are using to test.

Finally, you don't need to install Protractor globally. The provided Protractor configuration will install a local copy of Protractor, along with any necessary web driver plugins.

Running tests

To run the e2e tests, you need a running instance of your app.

By default, the e2e tests currently run against a locally hosted instance of your app i.e. at http://localhost:8100. This can be changed in the protractor.conf.js file, or (more flexibly) by changing the --baseUrl command line argument of Protractor (see section below, "Alternative ways of executing Protractor tests").

The test workflow now consists of two steps:

Starting the app
Running Protractor

These steps are explained below.

Starting the app

Start the app by calling:

ionic serve

or variants thereof (for instance, ionic serve -c --browser google-chrome).

By default, this runs the app in 'dev' mode, with "mock" implementations of the services.

For e2e testing it will in many cases be desirable to run with the "real" production implementation of the services (e.g. the "UserServiceFirebaseImpl" implementation of the UserService), so that we are testing against the app's realistic production behavior.

To configure the app (when started through ionic serve) to use production services, follow the instructions on these pages:

https://github.com/leob/ionic-quickstarter/wiki/Dev-mode-and-production-mode-configuration

https://github.com/leob/ionic-quickstarter/wiki/Firebase-configuration

https://github.com/leob/ionic-quickstarter/wiki/Setting-up-Twitter-authentication-with-Firebase

Running Protractor

Next, you might want to create a new shell instance (since ionic serve will take over the first shell that you used to serve your app). From the second shell, call:

npm run protractor

Alternatively, run the shell script run_protractor.sh located in the bin directory which (by default) executes npm run protractor:

bin\run_protractor.sh

Protractor will then run all the tests as referenced in protractor.conf.js and will print the results to the console. You will see the browser launch (chrome by default), and all your test scenarios played out just like a user would by interacting with the browser.

Alternative ways to run Protractor

The above command (npm run protractor) is one of the two ways to run Protractor tests.

There is another, faster way to run them which is to use the directConnect = true option in protractor.conf.js, together with the node_modules/protractor/bin/protractor command instead of the npm run protractor command.

This way of running Protractor (with directConnect = true) is faster than npm run protractor for two reasons:

npm run protractor (with directConnect = false) starts up slower because it checks for updated versions of Protractor and Selenium EVERY time you run the command; this takes quite a lot of time, making startup a lot slower.
node_modules/protractor/bin/protractor (with directConnect = true) bypasses a large part of the Selenium infrastructure (the 'Selenium stand-alone server' and the 'Webdriver JSON wire protocol'), instead it has the Chrome Driver connect directly to the browser (Chrome).

To use this alternative faster way, follow these steps:

Edit the protractor.conf.js file, search for directConnect: false and change it to directConnect: true.
Edit the shell script run_protractor.sh located in the bin directory, comment out the line starting with npm run protractor, and uncomment the line starting with node_modules/protractor/bin/protractor.

If you now run Protractor by entering the command bin\run_protractor.sh you will see that at least the startup of Protractor is way faster.

In fact, this method (no npm, and directConnect = true) is so much faster that I have made this the default. The only reason to switch back to the npm/Selenium method is if you want to run your e2e tests with another browser than Chrome (e.g. Safari).

Issues with Selenium/Chromedriver

If, when running your e2e tests (in any of the ways described above), you get an error similar to this:

E/launcher - Error: Server terminated early with status 11

then chances are that there is a problem with your Chrome driver. To verify this, run the 'chromedriver' script located in the projects bin folder, as follows:

bin/protractor-chromedriver.sh

If the Chrome driver is missing then you will an error like this:

bin/protractor-chromedriver.sh: ERROR: Cannot find chromedriver: bin/../node_modules/protractor/selenium/chromedriver

The cause of this problem is that, as a result of a Protractor/Selenium upgrade, the chromedriver binary has moved to a different path. You then need to locate this new path (find node_modules/protractor --iname '*chromedriver* will do the trick) and edit the bin/protractor-chromedriver.sh script to correct the path.

Optional parameters when running Protractor

If you look at the bin\run_protractor.sh script, you will see that you can pass a number of command line arguments when running Protractor (this applies equally to both ways of running Protractor: npm run protractor and node_modules/protractor/bin/protractor).

The parameters are: PORT, USER, and PASSWORD.

If you don't pass any parameters to the bin\run_protractor.sh script, the following default values will be used:

PORT: 8100
USER: tester@test.com
PASSWORD: password

These default values will normally be what you want when running the app in 'dev' mode with "mock" services.

If you want to override these values then call the script with additional arguments, for instance:

bin\run_protractor.sh 8101 admin@test.com otherpassword

This would be useful if the app runs on another port (8101 in this example) and needs another user and/or password to log in.

Protractor configuration: Jasmine reporters 'jasmine-spec-reporter' and 'protractor-fail-fast'

The protractor.conf.js file shipped with the ionic-quickstarter project configures two so-called Jasmine reporters:

jasmine-spec-reporter: this produces a more readable and better looking test output (with color coding) in the console than the "default" reporter (which produces test output that looks rather awful).
protractor-fail-fast: this "reporter" causes Protractor to stop running any further tests whenever a test fails; the default (when not using protractor-fail-fast) is that Protractor runs ALL tests, even when a test fails - this is often undesirable because you may want to fix the failing test immediately first and may not be interested in the rest of the tests.

In case you do not want the "fail fast" behavior, locate the following 3 lines in the protractor.conf.js file and comment them out:

var failFast = require('protractor-fail-fast');

jasmine.getEnv().addReporter(failFast.init());

failFast.clean();   // cleans up the "fail file"

Test Structure

The e2e tests are in the test/e2e folder. The recommended approach to write e2e tests is to organize them by user scenario, rather than modules in your code. The e2e tests should emulate user behavior for your most critical scenarios.

Under the test/e2e folder, there are 3 subfolders:

specs: these are the actual Protractor tests ("specs")
pages: these are "Page Objects" according to the "Page Object Pattern" (for an explanation see https://www.thoughtworks.com/insights/blog/using-page-objects-overcome-protractors-shortcomings and http://moduscreate.com/protractor-and-page-objects/)
common: these are reusable utility/library modules to simplify the specs and keep them DRY (see http://pavelbogomolenko.github.io/dry-principles-with-protractor.html)

Using "Page Objects" and "Shared modules" are examples of best practices that help to simplify the actual specs and promote code reuse (avoiding copy-and-paste).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unit testing and E2E Testing

Unit Testing

Configuration of the Karma test runner

End to End (E2E) Testing

Machine Setup

Running tests

Starting the app

Running Protractor

Alternative ways to run Protractor

Issues with Selenium/Chromedriver

Optional parameters when running Protractor

Protractor configuration: Jasmine reporters 'jasmine-spec-reporter' and 'protractor-fail-fast'

Test Structure

References

Unit Testing

Protractor (e2e) testing

Clone this wiki locally