x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

bcmills · 2021-11-04T13:40:02Z

(Note that the below regexp is not specific to freebsd or to x/tools. It is just looking for logs that contain only non-error lines.)

greplogs --dashboard -md -l -e '(?m)\A[^ ]+ at.*\n\n(?:(?:[:][:] Running .+\n\n)|(?:go: downloading [^ ]+ v[^ ]+\n)|(?:ok\s+[^ ]+\s+[^ ]+s\n)|(?:[?]\s+[^ ]+\s+\[no test files\]\n))+\z'

CC @golang/release

The text was updated successfully, but these errors were encountered:

bcmills · 2021-11-11T17:48:31Z

I think the process may be being killed due to memory exhaustion.

The freebsd-amd64-race builder appears to be running on an e2-highcpu-16 builder, while the normal linux-amd64-race builder is running on an e2-standard-4. (https://play.golang.org/p/5LXUgwHEZPn)

According to https://cloud.google.com/compute/vm-instance-pricing, e2-standard-4 has 4 CPUs and 16GB memory, while e2-highcpu-16 has 16 CPUs and 16GB.

Since the default GOMAXPROCS is based on the CPU count, and the default test parallelism is based on GOMAXPROCS, I suspect that the increased CPU-to-RAM ratio on the highcpu host is causing it to run too many gopls tests in parallel and run out of RAM.

@findleyr, @golang/release: should we address this by adjusting the host shape, or by adding some kind of additional limiting to the test, or explicitly setting GOMAXPROCS on this builder (which would possibly reduce its testing throughput), or something else entirely?

findleyr · 2021-12-03T15:04:35Z

Comparing these two recent builds:
Failure: https://build.golang.org/log/4d807f4a6b0659eea741ece539bd6a39a4fc0c1a
Success: https://build.golang.org/log/2d9892c7aaf14942405fb6b82ae24234b1807bec

We can see that the failure is not finishing all x/tools tests -- the last test reported is go/loader.
The next test in the success is go/packages, so I suspect this is failing in go/packages (@bcmills is that a viable conclusion?)

findleyr · 2021-12-03T15:49:28Z

If this is OOMing in go/packages (which we suspect may lead to this type of silent failure), then I can think of two possible sources:

Increase memory footprint or allocation rate in go/types.
Parallelization of those tests in https://golang.org/cl/332351

Looking specifically at some of the go/packages tests, I am also curious about why the tests are so slow and memory intensive, based on what they are doing. e.g. TestLoadDifferentPatterns takes 3s, but that seems extremely slow based on my understanding of our type-checking speed and the scope of that test.

bcmills · 2021-12-03T16:14:54Z

The next test in the success is go/packages, so I suspect this is failing in go/packages (@bcmills is that a viable conclusion?)

I think so — well spotted!

I can think of two possible sources:

Increase memory footprint or allocation rate in go/types.

Parallelization of those tests in https://golang.org/cl/332351

That is an interesting theory! But that change was merged in July, and the first of these failures started in October.

My leading theory is still machine shape (this builder has a very high CPU-to-RAM ratio), but either or both of those could be aggravating factors as well.

bcmills · 2022-01-31T22:19:39Z

This builder is still failing multiple times per day on x/tools in multiple different packages. We could theoretically fix the tests for each of those packages to scale by RAM, but at this point I'd really just rather make a pointwise change and be done with it.

@golang/release: could we try scaling this builder back to a more balanced configuration?

greplogs --dashboard -md -l -e \\Afreebsd-amd64-race -E . --since=2022-01-28

2022-01-31T20:30:41-492cf3a-41f485b/freebsd-amd64-race
2022-01-31T13:43:52-939c2c0-41f485b/freebsd-amd64-race
2022-01-29T19:58:17-939c2c0-1226547/freebsd-amd64-race
2022-01-29T03:53:30-939c2c0-5fd8c9b/freebsd-amd64-race
2022-01-28T22:17:23-939c2c0-654d5f4/freebsd-amd64-race
2022-01-28T20:07:54-939c2c0-8314544/freebsd-amd64-race

gopherbot · 2022-01-31T22:28:02Z

Change https://golang.org/cl/382058 mentions this issue: dashboard: switch host-freebsd-13_0-big to e2-standard-4 machines

This reduces the freebsd-amd64-race builder to 4 vCPUs instead of 16, which should also reduce the default GOMAXPROCS and default -test.parallel settings on that builder. The theory is that by switching to a standard CPU-to-RAM ratio, we will avoid OOMing in memory-bound x/tools tests, which scale their parallelism by GOMAXPROCS rather than the machine's total RAM size. For golang/go#49347 Change-Id: I6960efc5636fef7d2e6197aee33212a7823f0970 Reviewed-on: https://go-review.googlesource.com/c/build/+/382058 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>

bcmills · 2022-05-16T15:12:48Z

Lots of recent failures on this builder, but — crucially! — none missing error messages. I'm calling this fixed. (We can always reopen — or open a new issue — if that turns out to be wrong.)

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 4, 2021

gopherbot added this to the Unreleased milestone Nov 4, 2021

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD labels Nov 4, 2021

bcmills mentioned this issue Nov 8, 2021

x/tools/internal/lsp/source: test failures with diffs on "//@complete" lines #49463

Closed

bcmills mentioned this issue Nov 19, 2021

runtime: failures with "cannot map pages in arena address space" on freebsd-amd64-race since 2021-11-09 #49687

Closed

bcmills self-assigned this Apr 27, 2022

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 27, 2022

bcmills closed this as completed May 16, 2022

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 16, 2022

rsc unassigned bcmills Jun 23, 2022

golang locked and limited conversation to collaborators Jun 23, 2023

gopherbot added the FrozenDueToAge label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

bcmills commented Nov 4, 2021

bcmills commented Nov 11, 2021

findleyr commented Dec 3, 2021 •

edited

Loading

findleyr commented Dec 3, 2021

bcmills commented Dec 3, 2021 •

edited

Loading

bcmills commented Jan 31, 2022

gopherbot commented Jan 31, 2022

bcmills commented May 16, 2022

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

Comments

bcmills commented Nov 4, 2021

bcmills commented Nov 11, 2021

findleyr commented Dec 3, 2021 • edited Loading

findleyr commented Dec 3, 2021

bcmills commented Dec 3, 2021 • edited Loading

bcmills commented Jan 31, 2022

gopherbot commented Jan 31, 2022

bcmills commented May 16, 2022

findleyr commented Dec 3, 2021 •

edited

Loading

bcmills commented Dec 3, 2021 •

edited

Loading