Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(test runner): improve sharding algorithm to better spread similar tests among shards #30962

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
54bbba8
sharding algorithm to better spread similar tests among shards
muhqu May 22, 2024
ee64b15
improve shard algorithm by sorting test groups by number of tests
muhqu May 22, 2024
6543baf
adjust shard.spec.ts to new algorithm
muhqu May 22, 2024
59c625e
fix reporter-json.spec.ts due to sharding
muhqu May 22, 2024
d3b4fad
fix reporter-junit.spec.ts due to sharding
muhqu May 22, 2024
4bcfc78
empty commit to trigger ci
muhqu May 22, 2024
061f559
add test durations to last run info
muhqu May 27, 2024
14ca30a
allow .last-run.json to be generated via merge-reports
muhqu May 27, 2024
ce09f88
Add shardingMode configuration
muhqu May 27, 2024
d42c499
revert json/junit test changes
muhqu May 27, 2024
b5b8174
empty commit to trigger ci
muhqu May 27, 2024
389e571
fix(merge-reports) only change test ids when needed
muhqu May 29, 2024
6884fd7
remove special handling of merged test ids
muhqu May 29, 2024
ef2d35f
Merge branch 'main' into sharding-algorithm
muhqu May 29, 2024
d33e8da
fix(merge-reports) only change test ids when needed
muhqu May 29, 2024
6b051cb
fix(runner) don't write last run info when listing tests
muhqu May 29, 2024
32169b7
optimize: use single global set
muhqu May 30, 2024
fd90424
Feedback use outputLines
muhqu May 30, 2024
c7b58d7
Revert "fix(merge-reports) only change test ids when needed"
muhqu May 30, 2024
6cf1217
Merge branch 'fix-merge-reports-test-ids' into sharding-algorithm
muhqu May 30, 2024
7dd7115
empty commit to trigger ci
muhqu May 30, 2024
cabfa74
lint fix
muhqu May 30, 2024
899c068
Merge branch 'fix-merge-reports-test-ids' into sharding-algorithm
muhqu May 30, 2024
8aee7df
Merge branch 'main' into sharding-algorithm
muhqu May 30, 2024
843d629
Merge branch 'main' into sharding-algorithm
muhqu Jun 24, 2024
2536da7
fix test.d.ts
muhqu Jun 24, 2024
087a57f
add --last-run-file CLI parameter
muhqu Jun 25, 2024
7aa2d95
Adjust since to v1.46
muhqu Jul 2, 2024
29f67f3
Merge branch 'main' into sharding-algorithm
muhqu Jul 2, 2024
7b5f7c6
Add documentation on sharding modes
muhqu Jul 2, 2024
65a2b55
linting
muhqu Jul 2, 2024
df6d05d
Add into text to more complex round-robin scenario
muhqu Jul 2, 2024
3fed470
no ts highlight for sharding illustration code blocks
muhqu Jul 2, 2024
050e837
use yaml highlight for sharding illustration code blocks as NO highli…
muhqu Jul 2, 2024
f483ac6
revert / reorder import changes
muhqu Jul 2, 2024
7ea1506
Merge branch 'main' into sharding-algorithm
muhqu Sep 9, 2024
8d485f5
feedback
muhqu Sep 9, 2024
e7273de
feedback - remove redundant tests
muhqu Sep 9, 2024
1eb11ff
always use lastrun reporter and keep em internal
muhqu Sep 9, 2024
0503672
validate --sharding-mode CLI parameter
muhqu Sep 9, 2024
4b3d754
export type ShardingMode
muhqu Sep 9, 2024
89458db
fix .last-run.json location
muhqu Sep 9, 2024
7dab70c
fix .last-run.json location in reporter-lastrun spec
muhqu Sep 9, 2024
b0e5745
use lastRunFile from config when defined
muhqu Sep 9, 2024
5e3fa62
Merge branch 'main' into sharding-algorithm
muhqu Sep 9, 2024
4f758a1
Merge branch 'main' into sharding-algorithm
muhqu Sep 10, 2024
eb25f3c
validate config.shardingMode
muhqu Sep 11, 2024
16b39b8
fix --last-run-file parameter
muhqu Sep 11, 2024
ea72517
Merge branch 'main' into sharding-algorithm
muhqu Sep 11, 2024
b78b88f
adapt LastRunReporter
muhqu Sep 11, 2024
f62651c
use separate LastRunReporter instances
muhqu Sep 11, 2024
07914f7
allow lastRun.ts to be imported by merge.ts
muhqu Sep 11, 2024
e7d07bc
Merge branch 'main' into sharding-algorithm
muhqu Sep 12, 2024
412e05b
Add test.skip for Node.js bug in 'should not transform external' babe…
muhqu Sep 12, 2024
171c5e1
Merge branch 'main' into sharding-algorithm
muhqu Sep 13, 2024
ad6af69
Revert "Add test.skip for Node.js bug in 'should not transform extern…
muhqu Sep 13, 2024
b3b568b
Merge branch 'main' into sharding-algorithm
muhqu Sep 16, 2024
36433d0
Merge branch 'main' into sharding-algorithm
muhqu Sep 17, 2024
ae34689
Merge branch 'main' into sharding-algorithm
muhqu Sep 17, 2024
4dd2842
empty commit to trigger CI
muhqu Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion docs/src/test-api/class-testconfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@ export default defineConfig({

Shard tests and execute only the selected shard. Specify in the one-based form like `{ total: 5, current: 2 }`.

Learn more about [parallelism and sharding](../test-parallel.md) with Playwright Test.
Learn more about [parallelism](../test-parallel.md) and [sharding](../test-sharding.md) with Playwright Test.

**Usage**

Expand All @@ -482,6 +482,27 @@ export default defineConfig({
```


## property: TestConfig.shardingMode

* since: v1.48
- type: ?<[ShardingMode]<"partition"|"round-robin"|"duration-round-robin">>

Defines the algorithm to be used for sharding. Defaults to `'partition'`.
* `'partition'` - divide the set of test groups by number of shards. e.g. first
half goes to shard 1/2 and seconds half to shard 2/2.
* `'round-robin'` - spread test groups to shards in a round-robin way. e.g. loop
muhqu marked this conversation as resolved.
Show resolved Hide resolved
over test groups and always assign to the shard that has the lowest number of
tests.
* `'duration-round-robin'` - use duration info from `.last-run.json` to spread
test groups to shards in a round-robin way. e.g. loop over test groups and
always assign to the shard that has the lowest duration of tests. new tests
which were not present in the last run will use an average duration time. When
no `.last-run.json` could be found the behavior is identical to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the comment above about the round-robin strategy, I think we should use partition when no .last-run.json is present, and also use partition for any tests that were not found in the .last-run.json.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree.

Also please note that duration-round-robin and 'round-robin' are using the same implementation. The only difference is that in case of 'duration-round-robin' the last run info is passed to the implementation. However, in case the last run info is passed but it is empty, the behaviour is identical to 'round-robin' where no last run info is passed.

I don't think it would be a good to choose a different algorithm based on whether last run info is available or not.

`'round-robin'`.

Learn more about [sharding](../test-sharding.md) with Playwright Test.


## property: TestConfig.testDir
* since: v1.10
- type: ?<[string]>
Expand Down
70 changes: 70 additions & 0 deletions docs/src/test-sharding-js.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,3 +180,73 @@ Supported options:
reporter: [['html', { open: 'never' }]],
};
```

## Sharding Modes

Playwright offers different sharding modes to alter the behavior how test groups are assigned to shards, which can have an impact on overall execution time and resource utilization.

### `shardingMode: 'partition'`

This is the _default_. Test groups are ordered in the way they are discovered. Test groups are assigned the current shard until it has equal or more than 1/Nth of the overall number of tests. Then the next shard is filled, etc.

This has the effect that tests which share a common prefix are likely to execute on the same shard.

```yaml
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
Shard 1: ^---------^ : [ 1, 2, 3 ]
Shard 2: ^---------^ : [ 4, 5, 6 ]
Shard 3: ^---------^ : [ 7, 8, 9 ]
Shard 4: ^---------^ : [ 10,11,12 ]
```

### `shardingMode: 'round-robin'`

Spreads test groups evenly across shards. It sorts test groups by number of tests in descending order, then loops through the test groups and assigns them to the shard with the lowest number of tests.

Below is an example where every test group represents a single test (e.g. `--fully-parallel`).

```yaml
[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
Shard 1: ^ ^ ^ : [ 1, 5, 9 ]
Shard 2: ^ ^ ^ : [ 2, 6,10 ]
Shard 3: ^ ^ ^ : [ 3, 7,11 ]
Shard 4: ^ ^ ^ : [ 4, 8,12 ]
```

<details>
<summary>More complex scenario</summary>

Below is a scenario where tests [ 2 and 3 ], [ 4, 5 and 6 ] and [ 9 and 10 ] are executed in a test group which affects how tests are spread across the shards.

```yaml
Original Order: [ [1], [2, 3], [4, 5, 6], [7], [8], [9, 10], [11], [12] ]
Sorted Order: [ [4, 5, 6], [2, 3], [9, 10], [1], [7], [8], [11], [12] ]
Shard 1: ^-----^ : [ [ 4, 5, 6] ]
Shard 2: ^--^ ^ : [ [ 2, 3], [8] ]
Shard 3: ^---^ ^ : [ [ 9, 10], [11] ]
Shard 4: ^ ^ ^ : [ [1], [7], [12] ]
```

</details>

## `shardingMode: 'duration-round-robin'`

Very similar to `round-robin`, but uses the duration of a tests previous run as cost factor. The duration will be read from `.last-run.json` when available. When a test can not be found in `.last-run.json` it will use the average duration of available tests. When no last run info is available, the behavior is identical to `round-robin`.

As an example, consider we have 12 tests and test 7 and 8 take 5 seconds, test 10 and 11 takes 3 seconds and all other tests take 1 second to execute.

```yaml
Original Order: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
Sorted Order: [ 7, 8, 10, 11, 1, 2, 3, 4, 5, 6, 9, 12]
Shard 1: ^ ^ : [ 7, 5 ]
Shard 2: ^ ^ : [ 8, 6 ]
Shard 3: ^ ^ ^ ^ : [ 10, 1, 3, 9 ]
Shard 4: ^ ^ ^ ^ : [ 11, 2, 4, 12 ]
```

All shards would have an execution time of around 6 seconds...
* Shard 1 would execute tests 5 <sup>(1s)</sup> and 7 <sup>(5s)</sup> in 6 seconds.
* Shard 2 would execute tests 6 <sup>(1s)</sup> and 8 <sup>(5s)</sup> in 6 seconds.
* Shard 3 would execute tests 1 <sup>(1s)</sup>, 3 <sup>(1s)</sup>, 9 <sup>(1s)</sup> and 10 <sup>(3s)</sup> in 6 seconds.
* Shard 4 would execute tests 2 <sup>(1s)</sup>, 4 <sup>(1s)</sup>, 11 <sup>(3s)</sup> and 12 <sup>(1s)</sup> in 6 seconds.

8 changes: 6 additions & 2 deletions packages/playwright/src/common/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import fs from 'fs';
import path from 'path';
import os from 'os';
import type { Config, Fixtures, Project, ReporterDescription } from '../../types/test';
import type { Config, Fixtures, ShardingMode, Project, ReporterDescription } from '../../types/test';
import type { Location } from '../../types/testReporter';
import type { TestRunnerPluginRegistration } from '../plugins';
import { getPackageJsonPath, mergeObjects } from '../util';
Expand Down Expand Up @@ -56,6 +56,8 @@ export class FullConfigInternal {
cliLastFailed?: boolean;
testIdMatcher?: Matcher;
defineConfigWasUsed = false;
shardingMode: ShardingMode;
lastRunFile: string | undefined;

constructor(location: ConfigLocation, userConfig: Config, configCLIOverrides: ConfigCLIOverrides) {
if (configCLIOverrides.projects && userConfig.projects)
Expand Down Expand Up @@ -93,6 +95,8 @@ export class FullConfigInternal {
workers: 0,
webServer: null,
};
this.shardingMode = takeFirst(configCLIOverrides.shardingMode, userConfig.shardingMode, 'partition');
this.lastRunFile = configCLIOverrides.lastRunFile;
for (const key in userConfig) {
if (key.startsWith('@'))
(this.config as any)[key] = (userConfig as any)[key];
Expand Down Expand Up @@ -289,4 +293,4 @@ const configInternalSymbol = Symbol('configInternalSymbol');

export function getProjectId(project: FullProject): string {
return (project as any).__projectId!;
}
}
9 changes: 7 additions & 2 deletions packages/playwright/src/common/configLoader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,11 @@ function validateConfig(file: string, config: Config) {
throw errorWithFile(file, `config.shard.current must be a positive number, not greater than config.shard.total`);
}

if ('shardingMode' in config && config.shardingMode !== undefined) {
if (typeof config.shardingMode !== 'string' || !['partition', 'round-robin', 'duration-round-robin'].includes(config.shardingMode))
throw errorWithFile(file, `config.shardingMode must be one of "partition", "round-robin" or "duration-round-robin"`);
}

if ('updateSnapshots' in config && config.updateSnapshots !== undefined) {
if (typeof config.updateSnapshots !== 'string' || !['all', 'none', 'missing'].includes(config.updateSnapshots))
throw errorWithFile(file, `config.updateSnapshots must be one of "all", "none" or "missing"`);
Expand Down Expand Up @@ -341,9 +346,9 @@ export async function loadConfigFromFileRestartIfNeeded(configFile: string | und
return await loadConfig(location, overrides, ignoreDeps);
}

export async function loadEmptyConfigForMergeReports() {
export async function loadEmptyConfigForMergeReports(overrides?: ConfigCLIOverrides) {
// Merge reports is "different" for no good reason. It should not pick up local config from the cwd.
return await loadConfig({ configDir: process.cwd() });
return await loadConfig({ configDir: process.cwd() }, overrides);
}

export function restartWithExperimentalTsEsm(configFile: string | undefined, force: boolean = false): boolean {
Expand Down
4 changes: 3 additions & 1 deletion packages/playwright/src/common/ipc.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import util from 'util';
import { type SerializedCompilationCache, serializeCompilationCache } from '../transform/compilationCache';
import type { ConfigLocation, FullConfigInternal } from './config';
import type { ReporterDescription, TestInfoError, TestStatus } from '../../types/test';
import type { PlaywrightTestConfig, ReporterDescription, TestInfoError, TestStatus } from '../../types/test';

export type ConfigCLIOverrides = {
debug?: boolean;
Expand All @@ -33,6 +33,8 @@ export type ConfigCLIOverrides = {
reporter?: ReporterDescription[];
additionalReporters?: ReporterDescription[];
shard?: { current: number, total: number };
shardingMode?: PlaywrightTestConfig['shardingMode'];
lastRunFile?: string;
timeout?: number;
tsconfig?: string;
ignoreSnapshots?: boolean;
Expand Down
20 changes: 18 additions & 2 deletions packages/playwright/src/program.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ import { createMergedReport } from './reporters/merge';
import { loadConfigFromFileRestartIfNeeded, loadEmptyConfigForMergeReports, resolveConfigLocation } from './common/configLoader';
import type { ConfigCLIOverrides } from './common/ipc';
import type { TestError } from '../types/testReporter';
import type { TraceMode } from '../types/test';
import type { TraceMode, ShardingMode } from '../types/test';
import { builtInReporters, defaultReporter, defaultTimeout } from './common/config';
import { program } from 'playwright-core/lib/cli/program';
export { program } from 'playwright-core/lib/cli/program';
Expand Down Expand Up @@ -145,6 +145,7 @@ function addMergeReportsCommand(program: Command) {
});
command.option('-c, --config <file>', `Configuration file. Can be used to specify additional configuration for the output report.`);
command.option('--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${defaultReporter}")`);
command.option('--last-run-file <file>', `Path to a json file where the last run information is written to (default: test-results/.last-run.json)`);
command.addHelpText('afterAll', `
Arguments [dir]:
Directory containing blob reports.
Expand Down Expand Up @@ -264,7 +265,8 @@ async function listTestFiles(opts: { [key: string]: any }) {

async function mergeReports(reportDir: string | undefined, opts: { [key: string]: any }) {
const configFile = opts.config;
const config = configFile ? await loadConfigFromFileRestartIfNeeded(configFile) : await loadEmptyConfigForMergeReports();
const cliOverrides = overridesFromOptions(opts);
const config = configFile ? await loadConfigFromFileRestartIfNeeded(configFile, cliOverrides) : await loadEmptyConfigForMergeReports(cliOverrides);
if (!config)
return;

Expand Down Expand Up @@ -297,6 +299,8 @@ function overridesFromOptions(options: { [key: string]: any }): ConfigCLIOverrid
retries: options.retries ? parseInt(options.retries, 10) : undefined,
reporter: resolveReporterOption(options.reporter),
shard: shardPair ? { current: shardPair[0], total: shardPair[1] } : undefined,
shardingMode: options.shardingMode ? resolveShardingModeOption(options.shardingMode) : undefined,
lastRunFile: options.lastRunFile ? path.resolve(process.cwd(), options.lastRunFile) : undefined,
timeout: options.timeout ? parseInt(options.timeout, 10) : undefined,
tsconfig: options.tsconfig ? path.resolve(process.cwd(), options.tsconfig) : undefined,
ignoreSnapshots: options.ignoreSnapshots ? !!options.ignoreSnapshots : undefined,
Expand Down Expand Up @@ -332,6 +336,16 @@ function overridesFromOptions(options: { [key: string]: any }): ConfigCLIOverrid
return overrides;
}

const shardingModes: ShardingMode[] = ['partition', 'round-robin', 'duration-round-robin'];

function resolveShardingModeOption(shardingMode?: string): ShardingMode | undefined {
if (!shardingMode)
return undefined;
if (!shardingModes.includes(shardingMode as ShardingMode))
throw new Error(`Unsupported sharding mode "${shardingMode}", must be one of: ${shardingModes.map(mode => `"${mode}"`).join(', ')}`);
return shardingMode as ShardingMode;
}

function resolveReporterOption(reporter?: string): ReporterDescription[] | undefined {
if (!reporter || !reporter.length)
return undefined;
Expand Down Expand Up @@ -362,6 +376,7 @@ const testOptions: [string, string][] = [
['--headed', `Run tests in headed browsers (default: headless)`],
['--ignore-snapshots', `Ignore screenshot and snapshot expectations`],
['--last-failed', `Only re-run the failures`],
['--last-run-file <file>', `Path to a json file where the last run information is read from and written to (default: test-results/.last-run.json)`],
['--list', `Collect all the tests and report them, but do not run`],
['--max-failures <N>', `Stop after the first N failures`],
['--no-deps', 'Do not run project dependencies'],
Expand All @@ -374,6 +389,7 @@ const testOptions: [string, string][] = [
['--reporter <reporter>', `Reporter to use, comma-separated, can be ${builtInReporters.map(name => `"${name}"`).join(', ')} (default: "${defaultReporter}")`],
['--retries <retries>', `Maximum retry count for flaky tests, zero for no retries (default: no retries)`],
['--shard <shard>', `Shard tests and execute only the selected shard, specify in the form "current/all", 1-based, for example "3/5"`],
['--sharding-mode <mode>', `Sharding algorithm to use; "partition", "round-robin" or "duration-round-robin". Defaults to "partition".`],
['--timeout <timeout>', `Specify test timeout threshold in milliseconds, zero for unlimited (default: ${defaultTimeout})`],
['--trace <mode>', `Force tracing mode, can be ${kTraceModes.map(mode => `"${mode}"`).join(', ')}`],
['--tsconfig <path>', `Path to a single tsconfig applicable to all imported files (default: look up tsconfig for each imported file separately)`],
Expand Down
1 change: 1 addition & 0 deletions packages/playwright/src/reporters/DEPS.list
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

[merge.ts]
../runner/reporters.ts
../runner/lastRun.ts

[internalReporter.ts]
../transform/babelBundle.ts
Expand Down
4 changes: 3 additions & 1 deletion packages/playwright/src/reporters/merge.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import { currentBlobReportVersion, type BlobReportMetadata } from './blob';
import { relativeFilePath } from '../util';
import type { TestError } from '../../types/testReporter';
import type * as blobV1 from './versions/blobV1';
import { LastRunReporter } from '../runner/lastRun';

type StatusCallback = (message: string) => void;

Expand All @@ -39,7 +40,8 @@ type ReportData = {

export async function createMergedReport(config: FullConfigInternal, dir: string, reporterDescriptions: ReporterDescription[], rootDirOverride: string | undefined) {
const reporters = await createReporters(config, 'merge', false, reporterDescriptions);
const multiplexer = new Multiplexer(reporters);
const lastRun = new LastRunReporter(config);
const multiplexer = new Multiplexer([...reporters, lastRun]);
const stringPool = new StringInternPool();

let printStatus: StatusCallback = () => {};
Expand Down
32 changes: 24 additions & 8 deletions packages/playwright/src/runner/lastRun.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@ import { filterProjects } from './projectUtils';
import type { FullConfigInternal } from '../common/config';
import type { ReporterV2 } from '../reporters/reporterV2';

type LastRunInfo = {
export type LastRunInfo = {
status: FullResult['status'];
failedTests: string[];
testDurations?: { [testId: string]: number };
};

export class LastRunReporter implements ReporterV2 {
Expand All @@ -33,21 +34,32 @@ export class LastRunReporter implements ReporterV2 {

constructor(config: FullConfigInternal) {
this._config = config;
const [project] = filterProjects(config.projects, config.cliProjectFilter);
if (project)
this._lastRunFile = path.join(project.project.outputDir, '.last-run.json');
if (config.lastRunFile) {
// specified via command line argument
this._lastRunFile = config.lastRunFile;
} else {
const [project] = filterProjects(config.projects, config.cliProjectFilter);
if (project)
this._lastRunFile = path.join(project.project.outputDir, '.last-run.json');
}
}

async filterLastFailed() {
async lastRunInfo(): Promise<LastRunInfo | undefined> {
if (!this._lastRunFile)
return;
try {
const lastRunInfo = JSON.parse(await fs.promises.readFile(this._lastRunFile, 'utf8')) as LastRunInfo;
this._config.testIdMatcher = id => lastRunInfo.failedTests.includes(id);
return JSON.parse(await fs.promises.readFile(this._lastRunFile, 'utf8'));
} catch {
}
}

async filterLastFailed() {
const lastRunInfo = await this.lastRunInfo();
if (!lastRunInfo)
return;
this._config.testIdMatcher = id => lastRunInfo.failedTests.includes(id);
}

version(): 'v2' {
return 'v2';
}
Expand All @@ -65,7 +77,11 @@ export class LastRunReporter implements ReporterV2 {
return;
await fs.promises.mkdir(path.dirname(this._lastRunFile), { recursive: true });
const failedTests = this._suite?.allTests().filter(t => !t.ok()).map(t => t.id);
const lastRunReport = JSON.stringify({ status: result.status, failedTests }, undefined, 2);
const testDurations = this._suite?.allTests().reduce((map, t) => {
map[t.id] = t.results.map(r => r.duration).reduce((a, b) => a + b, 0);
return map;
}, {} as { [key: string]: number });
const lastRunReport = JSON.stringify({ status: result.status, failedTests, testDurations }, undefined, 2);
await fs.promises.writeFile(this._lastRunFile, lastRunReport);
}
}
2 changes: 1 addition & 1 deletion packages/playwright/src/runner/loadUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ export async function createRootSuite(testRun: TestRun, errors: TestError[], sho
testGroups.push(...createTestGroups(projectSuite, config.config.workers));

// Shard test groups.
const testGroupsInThisShard = filterForShard(config.config.shard, testGroups);
const testGroupsInThisShard = await filterForShard(config, testGroups);
const testsInThisShard = new Set<TestCase>();
for (const group of testGroupsInThisShard) {
for (const test of group.tests)
Expand Down
Loading
Loading