x/telemetry/internal/counter: persistent test failures with "fatal error: fault" on windows/386 using Go 1.20

[dmitshur]

The windows-386-2008 builder is persistently failing when running tests in this package on Go 1.20 release branch. Example failure:

unexpected fault address 0x3f00c0
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x3f00c0 pc=0x4022f4]

goroutine 7 [running]:
runtime.throw({0x55e0c7, 0x5})
	C:/workdir/go/src/runtime/panic.go:1047 +0x4d fp=0xa83dc5c sp=0xa83dc48 pc=0x438a5d
runtime.sigpanic()
	C:/workdir/go/src/runtime/signal_windows.go:270 +0x105 fp=0xa83dc80 sp=0xa83dc5c pc=0x44c965
runtime/internal/atomic.Load(0x3f00c0)
	C:/workdir/go/src/runtime/internal/atomic/atomic_386.go:19 +0x4 fp=0xa83dc84 sp=0xa83dc80 pc=0x4022f4
sync/atomic.(*Uint32).Load(...)
	C:/workdir/go/src/sync/atomic/type.go:121
golang.org/x/telemetry/internal/counter.(*mappedFile).load32(0xa8ba060, 0xc0)
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/file.go:462 +0x5c fp=0xa83dc90 sp=0xa83dc84 pc=0x52b1cc
golang.org/x/telemetry/internal/counter.(*mappedFile).newCounter(0xa8ba060, {0xa886ce0, 0x9})
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/file.go:561 +0x3a5 fp=0xa83dd28 sp=0xa83dc90 pc=0x52bb05
golang.org/x/telemetry/internal/counter.(*file).newCounter1(0xa8ac000, {0xa886ce0, 0x9})
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/file.go:287 +0x198 fp=0xa83dd84 sp=0xa83dd28 pc=0x52a4e8
golang.org/x/telemetry/internal/counter.(*file).newCounter(0xa8ac000, {0xa886ce0, 0x9})
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/file.go:270 +0x31 fp=0xa83dda0 sp=0xa83dd84 pc=0x52a321
golang.org/x/telemetry/internal/counter.(*file).lookup(0xa8ac000, {0xa886ce0, 0x9})
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/file.go:108 +0x52 fp=0xa83ddc4 sp=0xa83dda0 pc=0x529042
golang.org/x/telemetry/internal/counter.(*Counter).releaseLock(0xa892740, 0x620ffffffff)
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/counter.go:225 +0x215 fp=0xa83de44 sp=0xa83ddc4 pc=0x5282e5
golang.org/x/telemetry/internal/counter.(*Counter).Add(0xa892740, 0xc41)
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/counter.go:179 +0x540 fp=0xa83decc sp=0xa83de44 pc=0x527b90
golang.org/x/telemetry/internal/counter.TestLarge(0xa830c30)
	C:/workdir/gopath/src/golang.org/x/telemetry/internal/counter/counter_test.go:89 +0x16d fp=0xa83df9c sp=0xa83decc pc=0x52e1ed
testing.tRunner(0xa830c30, 0x56d750)
	C:/workdir/go/src/testing/testing.go:1576 +0x113 fp=0xa83dfe4 sp=0xa83df9c pc=0x4dabe3
testing.(*T).Run.func1()
	C:/workdir/go/src/testing/testing.go:1629 +0x2e fp=0xa83dff0 sp=0xa83dfe4 pc=0x4dbbce
runtime.goexit()
	C:/workdir/go/src/runtime/asm_386.s:1326 +0x1 fp=0xa83dff4 sp=0xa83dff0 pc=0x468331
created by testing.(*T).Run
	C:/workdir/go/src/testing/testing.go:1629 +0x427

[...]

goroutine 5 [finalizer wait]:
runtime.gopark(0x56d850, 0x695ed4, 0x10, 0x14, 0x1)
	C:/workdir/go/src/runtime/proc.go:381 +0xff fp=0xa836f94 sp=0xa836f80 pc=0x43b57f
runtime.runfinq()
	C:/workdir/go/src/runtime/mfinal.go:193 +0xf4 fp=0xa836ff0 sp=0xa836f94 pc=0x414ed4
runtime.goexit()
	C:/workdir/go/src/runtime/asm_386.s:1326 +0x1 fp=0xa836ff4 sp=0xa836ff0 pc=0x468331
created by runtime.createfing
	C:/workdir/go/src/runtime/mfinal.go:163 +0x60
FAIL	golang.org/x/telemetry/internal/counter	0.041s

See https://build.golang.org/log/c94132ef0694b636b4eb3038ebbfe8341319f390 for an example log. More are visible at https://build.golang.org/?repo=golang.org%2fx%2ftelemetry.

I've put this issue in an unreleased milestone, since none of the x/telemetry packages are vendored into the main Go repo and released with it, but with a release-blocker label since this is a persistent failure on a first-class port, and by definition those need to block the release. (That is, if packages here ended up being included in Go releases, then the Go 1.21 release would be blocked on the resolution of this issue.)

A prompt resolution would be appreciated, as it helps keep the dashboard green and without persistent failures (see #11811). If investigating and finding the root issue will take some time or is a lower priority for this repo, it is okay to add a temporary workaround and keep this issue open for a long term fix. Thanks.

CC @golang/tools-team.

[gopherbot]

Change https://go.dev/cl/505580 mentions this issue: internal/counter: skip tests on solaris, android, and GOARCH=386

[hyangah]

Thanks for the report.
The tests are skipped - windows/386 is now showing ok. #60615 is the issue to follow up.