Skip to content

Commit

Permalink
docs: document the new report options
Browse files Browse the repository at this point in the history
  • Loading branch information
ctron committed Aug 26, 2024
1 parent 5f51ea4 commit 3168b25
Show file tree
Hide file tree
Showing 7 changed files with 570 additions and 363 deletions.
872 changes: 526 additions & 346 deletions CHANGELOG.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ pub struct GooseConfiguration {
/// Doesn't display an error summary
#[options(no_short)]
pub no_error_summary: bool,
/// Create a report file
/// Create reports, can be used multiple times (supports .html, .htm, .md, .json)
#[options(no_short, meta = "NAME")]
pub report_file: Vec<String>,
/// Disable granular graphs in report file
Expand Down
4 changes: 2 additions & 2 deletions src/docs/goose-book/src/config/defaults.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The following defaults can be configured with a `&str`:
- host: `GooseDefault::Host`
- set a per-request timeout: `GooseDefault::Timeout`
- users to start per second: `GooseDefault::HatchRate`
- html-formatted report file name: `GooseDefault::ReportFile`
- report file names: `GooseDefault::ReportFile`
- goose log file name: `GooseDefault::GooseLog`
- request log file name: `GooseDefault::RequestLog`
- transaction log file name: `GooseDefault::TransactionLog`
Expand Down Expand Up @@ -62,7 +62,7 @@ The following defaults can be configured with a `bool`:
- enable Manager mode: `GooseDefault::Manager`
- enable Worker mode: `GooseDefault::Worker`
- ignore load test checksum: `GooseDefault::NoHashCheck`
- do not collect granular data in the HTML report: `GooseDefault::NoGranularData`
- do not collect granular data in the reports: `GooseDefault::NoGranularData`

The following defaults can be configured with a `GooseLogFormat`:
- request log file format: `GooseDefault::RequestFormat`
Expand Down
15 changes: 13 additions & 2 deletions src/docs/goose-book/src/getting-started/common.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,13 @@ cargo run --release -- --iterations 5

## Writing An HTML-formatted Report

By default, Goose displays [text-formatted metrics](metrics.md) when a load test finishes. It can also optionally write an HTML-formatted report if you enable the `--report-file <NAME>` run-time option, where `<NAME>` is an absolute or relative path to the report file to generate. Any file that already exists at the specified path will be overwritten.
By default, Goose displays [text-formatted metrics](metrics.md) when a load test finishes.

The HTML report includes some graphs that rely on the [eCharts JavaScript library](https://echarts.apache.org). The HTML report loads the library via CDN, which means that the graphs won't be loaded correctly if the CDN is not accessible.
It can also optionally write one or more reports in HTML, Markdown, or JSON format. For that, you need to provide one or more `--report-file <FILE>` run-time options. All requested reports will be written.

The value of `<FILE>` is an absolute or relative path to the report file to generate. The file extension will evaluate the type of report to write. Any file that already exists at the specified path will be overwritten.

For more information, see [Metrics Reports](metrics.md#metrics-reports).

![Requests per second graph](rps.png)

Expand All @@ -94,3 +98,10 @@ _Write an HTML-formatted report to `report.html` when the load test finishes._
```bash
cargo run --release -- --report-file report.html
```

### HTML & Markdown report example
_Write a Markdown and an HTML-formatted report when the load test finishes._

```bash
cargo run --release -- --report-file report.md --report-file report.html
```
34 changes: 25 additions & 9 deletions src/docs/goose-book/src/getting-started/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,38 +289,46 @@ All 9 users hatched.
------------------------------------------------------------------------------
```

## HTML metrics
In addition to the above metrics displayed on the CLI, we've also told Goose to create an HTML report.
## Metrics reports
In addition to the above metrics displayed on the CLI, we've also told Goose to create reports on other formats, like Markdown, JSON, or HTML.

### Overview
It is possible to create one or more reports at the same time, using one or more `--report-file` arguments. The type of report is chosen by the file extension. An unsupported file extension will lead to an error.

The following subsections describe the reports on more detail.

### HTML report

#### Overview
The HTML report starts with a brief overview table, offering the same information found in the [ASCII overview](#ascii-metrics) above:
![Metrics overview](metrics-overview.jpg)

### Requests
**NOTE:** The HTML report includes some graphs that rely on the [eCharts JavaScript library](https://echarts.apache.org). The HTML report loads the library via CDN, which means that the graphs won't be loaded correctly if the CDN is not accessible.

#### Requests
Next the report includes a graph of all requests made during the duration of the load test. By default, the graph includes an aggregated average, as well as per-request details. It's possible to click on the request names at the top of the graph to hide/show specific requests on the graphs. In this case, the graph shows that most requests made by the load test were for static assets.

Below the graph is a table that shows per-request details, only partially included in this screenshot:
![Request metrics](metrics-requests.jpg)

### Response times
#### Response times
The next graph shows the response times measured for each request made. In the following graph, it's apparent that POST requests had the slowest responses, which is logical as they are not cached. As before, it's possible to click on the request names at the top of the graph to hide/show details about specific requests.

Below the graph is a table that shows per-request details:
![Response time metrics](metrics-response-time.jpg)

### Status codes
#### Status codes
All status codes returned by the server are displayed in a table, per-request and in aggregate. In our simple test, we received only `200 OK` responses.
![Status code metrics](metrics-status-codes.jpg)

### Transactions
#### Transactions
The next graph summarizes all Transactions run during the load test. One or more requests are grouped logically inside Transactions. For example, the Transaction named `0.0 anon /` includes an anonymous (not-logged-in) request for the front page, as well as requests for all static assets found on the front page.

Whereas a Request automatically fails based on the web server response code, the code that defines a Transaction must manually return an error for a Task to be considered failed. For example, the logic may be written to fail the Transaction of the html request fails, but not if one or more static asset requests fail.

This graph is also followed by a table showing details on all Transactions, partially shown here:
![Transaction metrics](metrics-transactions.jpg)

### Scenarios
#### Scenarios
The next graph summarizes all Scenarios run during the load test. One or more Transactions are grouped logically inside Scenarios.

For example, the Scenario named `Anonymous English user` includes the above `anon /` Transaction, the `anon /en/basicpage`, and all the rest of the Transactions requesting pages in English.
Expand All @@ -330,9 +338,17 @@ It is followed by a table, shown in entirety here because this load test only ha
As our example only ran for 60 seconds, and the `Admin user` Scenario took >30 seconds to run once, the load test only ran completely through this scenario one time, also reflected in the following table:
![Scenario metrics](metrics-scenarios.jpg)

### Users
#### Users
The final graph shows how many users were running at the various stages of the load test. As configured, Goose quickly ramped up to 9 users, then sustained that level of traffic for a minute before shutting down:
![User metrics](metrics-users.jpg)

### Markdown report

The Markdown report follows the structure of the [HTML report](#html-report). However, it does not include the chart elements.

### JSON report

The JSON report is a dump of the internal metrics collection. It is a JSON serialization of the `ReportData` structure. Mainly having a field named `raw_metrics`, carrying the content of [`GooseMetrics`](https://docs.rs/goose/latest/goose/metrics/struct.GooseMetrics.html).

### Developer documentation
Additional details about how metrics are collected, stored, and displayed can be found [in the developer documentation](https://docs.rs/goose/*/goose/metrics/index.html).
2 changes: 1 addition & 1 deletion src/docs/goose-book/src/getting-started/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Error: InvalidOption { option: "--host", value: "", detail: "A host must be defi
The load test fails with an error as it hasn't been told the host you want to load test.
So, let's try again, this time passing in the `--host` flag. We will also add the `--report-file` flag, [which will generate a HTML report](common.html#writing-an-html-formatted-report), and `--no-reset-metrics` to preserve all information including the load test startup. The same information will also [be printed to the command line](metrics.md) (without graphs). After running for a few seconds, press `ctrl-c` one time to gracefully stop the load test:
So, let's try again, this time passing in the `--host` flag. We will also add the `--report-file` flag with a `.html` file extension, [which will generate an HTML report](common.html#writing-an-html-formatted-report), and `--no-reset-metrics` to preserve all information including the load test startup. The same information will also [be printed to the command line](metrics.md) (without graphs). After running for a few seconds, press `ctrl-c` one time to gracefully stop the load test:
```bash
% cargo run --release -- --host http://umami.ddev.site --report-file=report.html --no-reset-metrics
Expand Down
4 changes: 2 additions & 2 deletions src/docs/goose-book/src/getting-started/runtime-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Metrics:
--no-scenario-metrics Doesn't track scenario metrics
--no-print-metrics Doesn't display metrics at end of load test
--no-error-summary Doesn't display an error summary
--report-file NAME Create an html-formatted report
--report-file NAME Create reports, can be used multiple times (supports .html, .htm, .md, .json)
--no-granular-report Disable granular graphs in report file
-R, --request-log NAME Sets request log file name
--request-format FORMAT Sets request log format (csv, json, raw, pretty)
Expand Down Expand Up @@ -69,4 +69,4 @@ Advanced:
--accept-invalid-certs Disables validation of https certificates
```

All of the above configuration options are [defined in the developer documentation](https://docs.rs/goose/*/goose/config/struct.GooseConfiguration.html).
All of the above configuration options are [defined in the developer documentation](https://docs.rs/goose/*/goose/config/struct.GooseConfiguration.html).

0 comments on commit 3168b25

Please sign in to comment.