Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner: add snapshot testing #53169

Merged
merged 3 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -991,6 +991,16 @@ added: REPLACEME

Enable module mocking in the test runner.

### `--experimental-test-snapshots`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

Enable [snapshot testing][] in the test runner.

### `--experimental-vm-modules`

<!-- YAML
Expand Down Expand Up @@ -2129,6 +2139,18 @@ added:
A number of milliseconds the test execution will fail after. If unspecified,
subtests inherit this value from their parent. The default value is `Infinity`.

### `--test-update-snapshots`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

Regenerates the snapshot file used by the test runner for [snapshot testing][].
Node.js must be started with the `--experimental-test-snapshots` flag in order
to use this functionality.

### `--throw-deprecation`

<!-- YAML
Expand Down Expand Up @@ -3269,6 +3291,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
[security warning]: #warning-binding-inspector-to-a-public-ipport-combination-is-insecure
[semi-space]: https://www.memorymanagement.org/glossary/s.html#semi.space
[single executable application]: single-executable-applications.md
[snapshot testing]: test.md#snapshot-testing
[test reporters]: test.md#test-reporters
[timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
[tracking issue for user-land snapshots]: https://github.com/nodejs/node/issues/44014
Expand Down
166 changes: 166 additions & 0 deletions doc/api/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -920,6 +920,61 @@ test('runs timers as setTime passes ticks', (context) => {
});
```

## Snapshot testing

> Stability: 1.0 - Early development

Snapshot tests allow arbitrary values to be serialized into string values and
compared against a set of known good values. The known good values are known as
snapshots, and are stored in a snapshot file. Snapshot files are managed by the
test runner, but are designed to be human readable to aid in debugging. Best
practice is for snapshot files to be checked into source control along with your
test files. In order to enable snapshot testing, Node.js must be started with
the [`--experimental-test-snapshots`][] command-line flag.
Comment on lines +930 to +933
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
test runner, but are designed to be human readable to aid in debugging. Best
practice is for snapshot files to be checked into source control along with your
test files. In order to enable snapshot testing, Node.js must be started with
the [`--experimental-test-snapshots`][] command-line flag.
test runner, but are designed to be human readable to aid in debugging. We
recommend that snapshot files be checked into source control along with your
test files. In order to enable snapshot testing, Node.js must be started with
the [`--experimental-test-snapshots`][] command-line flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This exact wording came from this review comment: #53169 (comment)


Snapshot files are generated by starting Node.js with the
[`--test-update-snapshots`][] command-line flag. A separate snapshot file is
generated for each test file. By default, the snapshot file has the same name
as `process.argv[1]` with a `.snapshot` file extension. This behavior can be
cjihrig marked this conversation as resolved.
Show resolved Hide resolved
configured using the `snapshot.setResolveSnapshotPath()` function. Each
snapshot assertion corresponds to an export in the snapshot file.

An example snapshot test is shown below. The first time this test is executed,
it will fail because the corresponding snapshot file does not exist.

```js
// test.js
suite('suite of snapshot tests', () => {
test('snapshot test', (t) => {
t.assert.snapshot({ value1: 1, value2: 2 });
t.assert.snapshot(5);
});
});
```

Generate the snapshot file by running the test file with
`--test-update-snapshots`. The test should pass, and a file named
MoLow marked this conversation as resolved.
Show resolved Hide resolved
`test.js.snapshot` is created in the same directory as the test file. The
contents of the snapshot file are shown below. Each snapshot is identified by
the full name of test and a counter to differentiate between snapshots in the
same test.

```js
exports[`suite of snapshot tests > snapshot test 1`] = `
{
"value1": 1,
"value2": 2
Comment on lines +963 to +966

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cjihrig It seems like this is missing backtick consistency for string values in snapshots.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean.

}
`;

exports[`suite of snapshot tests > snapshot test 2`] = `
5
`;
```

Once the snapshot file is created, run the tests again without the
`--test-update-snapshots` flag. The tests should pass now.

## Test reporters

<!-- YAML
Expand Down Expand Up @@ -1622,6 +1677,54 @@ describe('tests', async () => {
});
```

## `snapshot`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

An object whose methods are used to configure default snapshot settings in the
current process. It is possible to apply the same configuration to all files by
placing common configuration code in a module preloaded with `--require` or
`--import`.

### `snapshot.setDefaultSnapshotSerializers(serializers)`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

* `serializers` {Array} An array of synchronous functions used as the default
serializers for snapshot tests.

This function is used to customize the default serialization mechanism used by
the test runner. By default, the test runner performs serialization by calling
`JSON.stringify(value, null, 2)` on the provided value. `JSON.stringify()` does
have limitations regarding circular structures and supported data types. If a
more robust serialization mechanism is required, this function should be used.

### `snapshot.setResolveSnapshotPath(fn)`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

* `fn` {Function} A function used to compute the location of the snapshot file.
The function receives the path of the test file as its only argument. If the
`process.argv[1]` is not associated with a file (for example in the REPL),
the input is undefined. `fn()` must return a string specifying the location of
the snapshot file.

This function is used to customize the location of the snapshot file used for
snapshot testing. By default, the snapshot filename is the same as the entry
point filename with a `.snapshot` file extension.

## Class: `MockFunctionContext`

<!-- YAML
Expand Down Expand Up @@ -3024,6 +3127,59 @@ test('top level test', async (t) => {
});
```

### `context.assert`

<!-- YAML
added:
- v22.2.0
-->

An object containing assertion methods bound to `context`. The top-level
functions from the `node:assert` module are exposed here for the purpose of
creating test plans.

```js
test('test', (t) => {
t.plan(1);
t.assert.strictEqual(true, true);
});
```

#### `context.assert.snapshot(value[, options])`

<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

* `value` {any} A value to serialize to a string. If Node.js was started with
the [`--test-update-snapshots`][] flag, the serialized value is written to
the snapshot file. Otherwise, the serialized value is compared to the
corresponding value in the existing snapshot file.
* `options` {Object} Optional configuration options. The following properties
are supported:
* `serializers` {Array} An array of synchronous functions used to serialize
cjihrig marked this conversation as resolved.
Show resolved Hide resolved
`value` into a string. `value` is passed as the only argument to the first
serializer function. The return value of each serializer is passed as input
to the next serializer. Once all serializers have run, the resulting value
is coerced to a string. **Default:** If no serializers are provided, the
test runner's default serializers are used.

This function implements assertions for snapshot testing.

```js
test('snapshot test with default serialization', (t) => {
t.assert.snapshot({ value1: 1, value2: 2 });
});

test('snapshot test with custom serialization', (t) => {
t.assert.snapshot({ value3: 3, value4: 4 }, {
serializers: [(value) => JSON.stringify(value)]
});
});
```

### `context.diagnostic(message)`

<!-- YAML
Expand All @@ -3044,6 +3200,14 @@ test('top level test', (t) => {
});
```

### `context.fullName`

<!-- YAML
added: REPLACEME
-->

The name of the test and each of its ancestors, separated by `>`.

### `context.name`

<!-- YAML
Expand Down Expand Up @@ -3294,13 +3458,15 @@ Can be used to abort test subtasks when the test has been aborted.
[TAP]: https://testanything.org/
[TTY]: tty.md
[`--experimental-test-coverage`]: cli.md#--experimental-test-coverage
[`--experimental-test-snapshots`]: cli.md#--experimental-test-snapshots
[`--import`]: cli.md#--importmodule
[`--test-concurrency`]: cli.md#--test-concurrency
[`--test-name-pattern`]: cli.md#--test-name-pattern
[`--test-only`]: cli.md#--test-only
[`--test-reporter-destination`]: cli.md#--test-reporter-destination
[`--test-reporter`]: cli.md#--test-reporter
[`--test-skip-pattern`]: cli.md#--test-skip-pattern
[`--test-update-snapshots`]: cli.md#--test-update-snapshots
[`--test`]: cli.md#--test
[`MockFunctionContext`]: #class-mockfunctioncontext
[`MockTimers`]: #class-mocktimers
Expand Down
6 changes: 6 additions & 0 deletions doc/node.1
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,9 @@ Enable code coverage in the test runner.
.It Fl -experimental-test-module-mocks
Enable module mocking in the test runner.
.
.It Fl -experimental-test-snapshots
Enable snapshot testing in the test runner.
.
.It Fl -experimental-eventsource
Enable experimental support for the EventSource Web API.
.
Expand Down Expand Up @@ -451,6 +454,9 @@ whose name matches the provided pattern.
.It Fl -test-timeout
A number of milliseconds the test execution will fail after.
.
.It Fl -test-update-snapshots
Regenerates the snapshot file used by the test runner for snapshot testing.
.
.It Fl -throw-deprecation
Throw errors for deprecations.
.
Expand Down
1 change: 1 addition & 0 deletions lib/internal/test_runner/harness.js
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ function setup(root) {
counters: null,
shouldColorizeTestFiles: false,
teardown: exitHandler,
snapshotManager: null,
};
root.harness.resetCounters();
root.startTime = hrtime();
Expand Down
Loading
Loading