proposal: cmd/go: add an option to sequentially run tests #61318
Comments
ianlancetaylor
changed the title
|
To clarify, you are looking for a way to run each package test one at a time? |
Yes, only one package should execute at a time during
I think keeping inner-package parallelism is fine and could be controlled with the existing |
Compare #33974, which would add a |
|
At work, we have a mix of NodeJS and Golang services. Those services communicated with a database, and as such have tests that exercise queries against a Dockerized database. As that is a shared resource, tests that hit the database must be serialized, while tests that operate purely in memory are allowed to be parallelized. The NodeJS services have a pretty easy way of solving this using a custom Jest runner, which I'll link below. Our way of doing this in Golang is by putting all of the tests that use an external resource (like a database) in a singular package so that they run sequentially. This causes the test files to be located not alongside the source files though, which is a shame. In the NodeJS way, the stream of tests that must be sequential is defined by filename. That isn't very robust and is error prone. I think there should be a way to do this in Go using a import { Config } from "@jest/types";
import TestRunner, { Test, TestEvents, TestRunnerContext, TestWatcher, UnsubscribeFn } from "jest-runner";
export default class CustomTestRunner {
public supportsEventEmitters = true;
private parallelRunner: ParallelTestRunner;
private sequentialRunner: SequentialTestRunner;
constructor(globalConfig: Config.GlobalConfig, context: TestRunnerContext) {
this.parallelRunner = new ParallelTestRunner(globalConfig, context);
this.sequentialRunner = new SequentialTestRunner({ ...globalConfig, maxWorkers: 1 }, context);
}
on<Name extends keyof TestEvents>(
eventName: Name,
listener: (eventData: TestEvents[Name]) => void | Promise<void>,
): UnsubscribeFn {
const parallelUnsubscribe = this.parallelRunner.on(eventName, listener);
const sequentialUnsubscribe = this.sequentialRunner.on(eventName, listener);
return () => {
parallelUnsubscribe();
sequentialUnsubscribe();
};
}
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
await Promise.all([
this.sequentialRunner.runTests(tests, watcher),
this.parallelRunner.runTests(tests, watcher),
]);
}
}
class ParallelTestRunner extends TestRunner {
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
const isParallelizeable = (test: Test) => !test.path.includes(".sequential.");
const parallelizeableTests = tests.filter(isParallelizeable);
await super.runTests(parallelizeableTests, watcher, { serial: false });
}
}
class SequentialTestRunner extends TestRunner {
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
const isSequentialTest = (test: Test) => test.path.includes(".sequential.");
const sequential = tests.filter(isSequentialTest);
// { serial: false } triggers the parallel test execution leveraging workers,
// but along with { maxWorkers: 1, workerIdleMemoryLimit: ... } configuration
// this sole worker would be killed and restarted if it exceeds the memory threshold.
await super.runTests(sequential, watcher, { serial: false });
}
} |
|
I think we should think harder about if there's a way to better support having the individual tests synchronize access to the external resource. I wonder if there's a better way to act on tests being blocked. |
It is a common pattern to want to run Go tests in sequence rather than in parallel. Within a package, this is controlled by
t.Parallel()and-test.parallel. However, across packages there is a different control:-p N(typically-p 1for this use case).This is commonly used for integration tests which are depending on exclusive access to some resource outside of the test.
However, this comes at a huge cost:
-p 1does not control just test parallelism, but all compiler actions. This means all compiling and linking is done sequentially. In our project, this has caused a 10x increase in test execution times - with-p 1, it takes 8m25s; without, 50s.I would like a way to run test packages sequentially, while still benefiting from the parallelism in building.
Ecosystem usage
I did a rough analysis of usage of
go test -p 1across Github, and found a number of large projects doing the same. I use stars just as a rough measure of the project scope.If you search "go run tests sequentually", basically all sources will tell you to use
-p 1. This includes AI chatbots, popular stack overflow answers, blogs, and even some Go core maintainer recommendations.Alternatives
go test ./a && go test ./b). This is not great; we lose parallelism in building.go test -c -o tests/ ./...; go test -p 1 ./...). This is demonstrated as effective here. However, it is highly limited unless proposal: cmd/go: support naming compiled tests with full qualified name #61199 is resolved as well.The text was updated successfully, but these errors were encountered: