runtime: failures with "cannot map pages in arena address space" on freebsd-amd64-race since 2021-11-09

[bcmills]

fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw({0x9f747c?, 0x0?})
	/tmp/workdir/go/src/runtime/panic.go:992 +0x71
runtime.sysMap(0x0, 0x40ccc9?, 0xc000611358?)
	/tmp/workdir/go/src/runtime/mem_bsd.go:76 +0xa5
runtime.(*mheap).grow(0xdd4320, 0xc000611200?)
	/tmp/workdir/go/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0xdd4320, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
	/tmp/workdir/go/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
	/tmp/workdir/go/src/runtime/asm_amd64.s:469 +0x49

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-01-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race
2021-11-09T00:08:42-b76863e-5344dca/freebsd-amd64-race

[bcmills]

I suspect that this may be due to memory exhaustion; as noted in #49347 (comment), the freebsd-amd64-race builder has a higher CPU-to-RAM ratio than most of the other -race builders, which may cause its GOMAXPROCS to be fairly aggressive for the amount of available memory.

[mknyszek]

Have there been any new instances of this? I can't find any on the main dashboard (I haven't downloaded the subrepo logs).

[bcmills]

None since Nov. 19, including the subrepos.

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-11-20

[bcmills]

Actually, hang on. I think there's something wrong with my regexp.

[bcmills]

Ah, it was a time-zone difference. There are indeed no occurrences since Nov. 19 UTC.

greplogs --dashboard -md -l -e 'cannot map pages in arena address space' --since=2021-11-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race

[mknyszek]

Since this was due to memory exhaustion (it looks like that to me too -- the OS won't let the runtime commit any more pages) and hasn't happened since Nov 19th, @bcmills what do you think about putting this in WaitingForInfo?

We can also try increasing decreasing the CPU-to-RAM ratio as you suggest, but I get the impression that this doesn't even get into any -race code. That is, unless I'm misunderstanding and the go command is actually built with -race. If that's the case, this may also be totally mitigated by https://go.devl/cl/333529 which imports a new TSAN runtime that uses a lot less memory (though I'm gonna guess that's coming in 1.19 at this point).

[cherrymui]

Any update on this? Does it happen again? (pinging as this is a release blocker)

the go command is actually built with -race

I think you're right, the go command is not built with -race flag.

Perhaps other processes that are built with -race use more memory and cause the go command fail to map the address?

[aclements]

As far as I can tell this hasn't happened again (though I only have a partial image of the subrepo logs). This seems like just random memory exhaustion, especially given the builder config. I think we should just close this.

[aclements]

The error reporting could definitely be better here (like, what was errno?). I think we should just improve the error reporting and close this. If it recurs, then we can reopen with more information.

[gopherbot]

Change https://golang.org/cl/379354 mentions this issue: runtime: print error if mmap fails