Skip to content

Latest commit

 

History

History

code-coverage-with-monocart-reporter

README.md

Playwright code coverage with monocart-reporter

Description

The demo at /demos/code-coverage-with-monocart-reporter shows how to get code coverage with Playwright by using the monocart-reporter npm package.

Note

As of writing this, there's an open GitHub issue about improving the built-in code coverage support for Playwright. Also check out this comment on that issue.

Thank you @mxschmitt !

How to build, run the app and run tests

Important

Required dependencies:

  • Node. Tested working with v20.10.0. If you need to have different versions of node installed it's recommended that you use Node Version Manager to install and swap between node versions.
  • npm@latest: package manager used on the demos. Tested working on 10.2.5.
  • VS Code is recommended as a code editor but you can use whatever you prefer.
  1. Clone the repo.
  2. Using your favorite shell go to /demos/code-coverage-with-monocart-reporter.
  3. Install the required npm packages with: 
    npm install
  4. Install the playwright browsers with: 
    npx playwright install
  5. Run the tests with: 
    npm test
    This will start the app and run the playwright tests against it.
  6. If you just want to run the app execute the command: 
    npm start
    Once the command finishes the app should open in your default browser at http://127.0.0.1:4200/.

How to view the test results and code coverage

After running the tests with npm test you can view test results with:

npm run test:show-report

After opening the test results report you can view the code coverage by opening the menu on the top right and then find the Coverage Report - playwright code coverage demo with monocart reporter option under artifacts.

code-coverage-monocart-reporter.mp4

Alternatively, you can view the code coverage report with:

npm run coverage:show-report

Tests and code coverage

Once you've installed the monocart-reporter NPM package with npm i -D monocart-reporter, there are two main building blocks to configure code coverage:

  • on the playwright.config.ts, use the monocart-reporter as one of the reporters in the reporter array.
  • use playwright's code coverage API to collect code coverage during the tests' execution.

Playwright configuration

The majority of the content of the playwright.config.ts file is what you get by default after adding Playwright to your project with npm init playwright@latest.

The main changes are:

  1. Declared a few variables at the start that are reused throughout the playwright configuration.
  2. Updated the reporter array. Instead of using the default html reporter, use the built-in list reporter and the third-party monocart-reporter.
  3. Defined a baseURL so that we can use relative URLs when doing page navigations on the tests.
  4. Configured the webServer block to run the Angular app locally so that the tests can be executed against it. If you're not testing an Angular app that's fine, you just need to adjust the webServer.command so that it launches your app and set the webServer.url to the url your app will be running at. For more information see the webServer docs.

Note

The _isRunningOnCI variable used on the playwright.config.ts changes the value of some options when tests running on CI. To set the _isRunningOnCI variable to true you must set the environment variable CI to true before running the tests. For more information regarding using Playwright on a CI environment see Playwright docs on Continuous Integration.

Furthermore, we have created:

Note

You don't have to create the playwright.monocart-reporter.ts, the playwright.cli-options.ts or the playwright.env-vars.ts file. You can have all of this on the playwright.config.ts. Code structure is up to you.

Note

Depending on your playwright.config.ts, make sure you update your .gitignore to exclude any directory used by test results, report results, etc. Scroll to the end of this demo's .gitignore to see an example.

monocart-reporter configuration

The monocart-reporter configuration is done at playwright.monocart-reporter.ts. The monocart-reporter has a coverage option which let's you configure the options for the code coverage reports. The configuration at playwright.monocart-reporter.ts will create:

  • an html report with monocart style: this is my prefered html report.
  • a console summary report.
  • a lcov report: which is useful to upload to some tools like SonarQube, etc.
  • a cobertura report: which is useful to upload to some tools like Azure DevOps, CodeCov, etc.
  • an html report with the istanbul html-spa style: this is not really necessary. We already have the monocart html report but it's here just to demo that you can have multiple html reports if you want.

The produced folder structure after running the tests looks like this:

tests/
└── test-results/
    ├── monocart-report.html
    ├── monocart-report.json
    └── code-coverage/
        ├── cobertura/
        │   └── code-coverage.cobertura.xml
        ├── html-spa/
        │   ├── index.html
        │   └── ...(several other files required for the html report)
        ├── lcov/
        │   └── code-coverage.lcov.info
        └── v8/
            └── index.html

Collect code coverage

Important

Playwright's Coverage APIs are only supported on Chromium-based browsers.

To collect the code coverage we use playwright's code coverage API. You could add these code coverage API calls to all your tests or you can use Playwright fixtures to code them once and reuse across tests.

The approach taken in this demo was to create an automatic fixture, named codeCoverageAutoTestFixture, so that the code coverage calls are automatically added to any test that is created without having to do anything extra as long as the test import comes from this fixture. See the example.spec.ts and note the import statment at the top:

import { test, expect } from "tests/_shared/app-fixtures";

To use the codeCoverageAutoTestFixture automatic fixture all your tests should import the test from the fixture instead of doing the usual:

import { test, expect } from "@playwright/test";

To learn more about why this is done study how fixtures work on Playwright.

Note that the code coverage logic that the codeCoverageAutoTestFixture fixture uses is encapsulated in the v8-code-coverage.ts file. The fixture is capturing both JS and CSS code coverage but what to cover is your choice by setting the values of enableJsCoverage and enableCssCoverage.

Once the code coverage is collected, it's being added to the monocart-reporter by invoking the addCoverageReport function.

Learn more

To learn more about configuring the monocart-reporter and code coverage see:

JS, CSS and HTML code coverage

The V8 code coverage is capable of collecting code coverage not only fo JS but also for CSS and HTML. In the code coverage report for this demo you can see coverage shown for the the TS and HTML files.

You should also see code coverage for the src/app/app.component.css file but it isn't showing up for this Angular app due to this GitHub issue microsoft/playwright [BUG] Missing CSS code coverage #28510.