cmd/go: support compiling all tests without running

[bradfitz]

I want this to work:

$ go test -c std cmd
cannot use -c flag with multiple packages

But I don't actually care about the binaries. I just want to test that all the tests can compile, and how fast. @randall77 also wants this for SSA coverage reasons.

[cespare]

Is go test -run xxxxx std cmd a reasonable workaround for at least some of those use cases?

[josharian]

And I want this, but I do want the test binary. I'd use it to compile tests, change the compiler, compile tests again, and then run before/after benchmarks. (All in a script, of course.) Like toolstash for std tests. I currently do something similar but with go list std and a lot of bookkeeping and go test -c invocations.

It seems to me we could generate a testmain that invokes all the tests in all the included packages. There are some questions: How do you know from the -v output which package this particular instance of TestReader is from? How does the -run filter work? And so on. My hand-waving answer is: Mimic the UI for subtests and sub benchmarks, which already have this nesting problem; the nesting is just up a level rather than down.

[minux]

we can't generate one binary for tests in multiple packages because there is no guarantee that package A's test doesn't interfere with package B's use of package A.

[josharian]

What if we leave that up to the user to diagnose? If package A's test interferes with package B's use of package A, then don't test them together. This sort of issue can already arise in a single package.

[minux]

how could the user know that one can't test package A and package B together? consider "go test -c ./...", which could easily contain hundreds of packages.

[josharian]

They'll know when the test fails. If the tests pass when run independently (i.e. go test ./..., no -c), then there's clearly an interaction problem.

[minux]

Even if they could know the bad interaction between A & B (it's not that easy because given that B's tests failed, you still don't know which of B's dependent [both directly and indirectly] might be the cause. What's more, the behavior also depends on the order tests for different packages are run.) because cmd/go doesn't support negative patterns, there is no way to use ./... and exclude B. Given a large enough GOPATH, it's very likely there are bad interactions between the package tests, and then basically the new feature is useless (in the worse case, the user has to name all the packages on the command line, and it's no better than status quo.) We might be able to do this if multiple packages are compiled into a single package from the beginning, but I don't think we can add this feature now.

[josharian]

I can't imagine someone wanting to run go test -c ./... on their entire GOPATH unless they had a very good reason. I can, however, see this for a particular project or set of packages, in which case this becomes very useful. Think e.g. of cross-compiling all the tests for your project and having to only scp a single binary.

cmd/go doesn't support negative patterns

Yes, this is unfortunate.

I apologize, but I don't quite understand the resistance here. It seems like the resistance is "something could go wrong". But this doesn't really seem like a big footgun to me. And when this is helpful, like cross-compiling all tests for a big project or keeping around an old version for benchmarking or testing compilation speed, it is very helpful, and users will be motivated to make it work (or not use it). And it is entirely backwards compatible.

[minux]

I think this issue gives one reason for do go test -c ./... for the
whole GOPATH: test the SSA compiler code generation.

The key point is most tests are not prepared for this.
How many packages support go test -count=2?
Even some of the std packages don't, until they're
fixed not long ago.

And how do you propose to handle conflicting flags added
by tests and multiple TestMains? Again, these features
are just not designed to combine tests for multiple packages
into a single executable.

Also note that tests for a package needs to run at the
directory for the package, how to handle that when
tests for multiple packages are combined into one binary?

(FTR, I have been thinking about this feature a long time
ago, but my conclusion was and still is that the whole thing
is not designed for this.)

[josharian]

How many packages support go test -count=2? Even some of the std packages don't, until they're
fixed not long ago.

Exactly. And they got fixed because people cared. Same situation here, I'd say.

conflicting flags added by tests

Sorry, what do you mean by this?

multiple TestMains

Refuse to compile, with a lucid error message. Rare.

tests for a package needs to run at the directory for the package

os.Chdir? Note that the go command has the directory available at compilation time. This is a bit ugly for the cross-compilation case, but we could print a warning instead of failing when os.Chdir fails.

I have been thinking about this feature a long time
ago

Yeah, I recall there being another issue about this but couldn't find it.

I agree that there are limitations, I just see them as limitations rather than showstoppers.

[minux]

By conflicting flags added by tests, consider two
packages that both add -update flags to update
golden output of their test data.

The code will compile, but will fail at runtime with
a flag redefined panic.

There are just too many global states, hence I
said the whole system is not designed for this.

[mattklein123]

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz

[davecheney]

If you want to compile but run not tests,

 go test -run=nope ./...

Replace nope with another pattern if you have test cases that match.

On Fri, 27 May 2016, 03:45 Matt Klein notifications@github.com wrote:

I'm new to Go, but FWIW, the lack of this feature is one of the most
frustrating things that I have found so far about the tooling. I tend to
write production code and test code at the same time, and during my normal
dev cycle I want to see if things compile many more times than I want to
run the tests. So right now I'm doing roughly the same thing as stated
above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and
have it be up to the user to deal with conflicts as suggested by @bradfitz
https://github.com/bradfitz

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#15513 (comment)

[mattklein123]

@davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.

[davecheney]

Please try -run=nope

On Fri, 27 May 2016, 09:56 Matt Klein notifications@github.com wrote:

@davecheney https://github.com/davecheney I'm probably doing something
stupid but running that command still causes all the tests to execute. I
tried:

go test -run nope ./...

Also still runs tests.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15513 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8xNrwQtBDQEF3AeZEF-377nOLMUks5qFjMwgaJpZM4IV2e-
.

[mattklein123]

I tried both:

mklein@vm:~/Source/go/src/github.com/<redacted>$ go test -run=nope ./...
ok      github.com/<redacted>   0.025s

[cespare]

-run=nope and -run nope are equivalent.

@mattklein123 no tests are being run (probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

[davecheney]

That is correct. The test binary is built and executed, but none of the
test cases will be executed because they don't match the -run filter

On Fri, May 27, 2016 at 10:11 AM, Caleb Spare notifications@github.com
wrote:

-run=nope and -run nope are equivalent.

@mattklein123 https://github.com/mattklein123 no tests are being run
(probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15513 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8u95PM2MDjAGCytHBIVmp0py0RIks5qFjaxgaJpZM4IV2e-
.

[jurij]

i prefer -run ^$

[bradfitz]

My motivation is for the builders, and test sharding.

The problem with go test -run=^$ std cmd is that it deletes the *.test binaries afterwards. I want the test binaries, to ship around the network and shard out isolated test execution. But I want to burn CPU as hard as possible where it's not latency sensitive. (That is, I can run compilation jobs at 100% CPU, but tests are often flaky at 100% if they sleep or depend on time.)

[minux]

@bradfitz, I think for your purpose, having multiple go test -c processes is better than figuring out where to store the compiled binaries when building multiple tests (the only safe way is to put the compiled test binary in their own package directories, but then you need to collect them. It's probably easier to just go test -c with explicitly output filename). It won't do much more work either (assuming go install std cmd has passed.)

[bradfitz]

I can do it myself, but my concern is that I can't keep the CPU busy as well as the cmd/go binary could.

cmd/go can load the dependency graph once, come up with a plan, and keep a certain number of worker processes running at all times.

Any coordination I do trying to run multiple cmd/go binaries wouldn't be as good.

Definitely low priority, though.

[or-shachar]

Regarding this:

We also have go build -o /dev/null already special casing throwing the test binary away. And apparently that works for multiple binaries today already (see #37378 (comment)).

It seems like we should make go test -c -o /dev/null work for multiple tests, as well as go test -c -o .

What if we are interested to take the test execution binaries and execute them in another process?

This can be beneficial if you have long running tests and very different resources footprint between compilation and test runtime. For instance - e2e tests.

It will allow us to compile all tests on a strong expensive machine and then run the tests on cheaper ones.

[bcmills]

@mknyszek and @prattmic pointed out that on Unix platforms go test -exec /bin/true does seem to support compiling the tests without running them today. 😅

[howardjohn]

You can speed things up a lot faster with https://gist.github.com/howardjohn/c0f5d0bc293ef7d7fada533a2c9ffaf4 by avoiding linking the binaries. A warm run on even huge repos like k8s can do this in 1-2s on my machine.

[bcmills]

@howardjohn, it may be faster but it's not the same. Link-time errors are possible in pure Go programs and even fairly common in cgo programs.

[gopherbot]

Change https://go.dev/cl/466397 mentions this issue: cmd/go: make go test build multiple executables

[samsalisbury]

At last! Excited to see this supported by the toolchain, will avoid nasty hacks like this in the future 😂

[bcmills]

So, one minor detail: for a single package, go test -c $pkg would be equivalent to go test -c -o $(pwd)/ $pkg.

For consistency, should go test -c ./... also write test binaries in the current working directory for all matching packages?
(I think it probably should.)

[samsalisbury]

should go test -c ./... also write test binaries in the current working directory for all matching packages?

How would conflicts be avoided here, with multiple packages of the same name but different paths?

Just thinking about a general solution for different use-cases... If -o supported a template you could use e.g. -o '{{.pkgPathRel}}.test" (maybe that should be the default to avoid the possibility of conflicts). It could then be set to {{.pkgName}}.test to write them all to the same dir. This would also remove the need for a special-case for /dev/null, as /dev/null would be a valid template...

[bcmills]

How would conflicts be avoided here, with multiple packages of the same name but different paths?

Conflicts would cause the go invocation to fail. (But go test -c ./... already fails today if ./... matches more than one package at all, so no invocation that works today would start failing as a result.)

If -o supported a template you could use e.g. -o '{{.pkgPathRel}}.test" (maybe that should be the default to avoid the possibility of conflicts).

I think that's probably overkill. I expect that the vast majority of uses will be -o /dev/null, and consistency with go build -o is important to provide a predictable UX.

[samsalisbury]

Thanks for the explanation @bcmills

Conflicts would cause the go invocation to fail.

Just saw the test for this in the CL, apologies I missed it the first time. However, being able to robustly build all test binaries would seem necessary to satisfy e.g. @bradfitz's comment about sharding them over the network (it's an old comment but I can still see reasons we'd want to be able to do this).

For my own use cases, I've written scripts using go list and go test -c to achieve this sort of thing, but having it supported by the tool would be very useful... Having said that it's probably the 1% case, where -o /dev/null is the 99%...

[bcmills]

Yeah, I agree that those cases exist but I think the solution for them is, as you mentioned, to have those cases run go test -c with the specific packages they want to build. It's a bit more coding work but it's also a more complicated and nuanced use-case (precisely because of those kinds of collisions).

[gopherbot]

Change https://go.dev/cl/488275 mentions this issue: cmd/dist: actually only compile tests with -compile-only

[gopherbot]

Change https://go.dev/cl/500956 mentions this issue: doc/go1.21: document forward and backward compatibility

[gopherbot]

Change https://go.dev/cl/500955 mentions this issue: cmd/go: document changes to go test -c and -o

[howardjohn]

This is super useful, thanks everyone involved. I opened a followup issue here: #61199 that I think can help make this more useful.

[PapaCharlie]

For posterity, and anyone who wants to include this in their build, the following command finds all the packages in the current directory:

go list -f '{{if .TestGoFiles}}{{.ImportPath}}{{end}}' ./...

So checking if all the tests compile can be accomplished like this:

go test -c -o /dev/null $(go list -f '{{if .TestGoFiles}}{{.ImportPath}}{{end}}' ./...)