runtime: TestGdbCoreSignalBacktrace fails when run outside of all.bash

[siebenmann]

What version of Go are you using (go version)?

This is the current git tip:

$ go version
go version devel go1.22-57e2eb64eb Thu Jul 20 20:54:38 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

No. This doesn't happen with a 'git checkout go1.20.6', building that Go, and then running go test -short std cmd.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/cks/.cache/go-build'
GOENV='/home/cks/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/cks/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/cks/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/w/620/cks/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/w/620/cks/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.22-57e2eb64eb Thu Jul 20 20:54:38 2023 +0000'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/w/620/cks/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1776540564=/tmp/go-build -gno-record-gcc-switches'

This is on an up to date Ubuntu 22.04 LTS Linux server, on amd64.

What did you do?

In a git tip clone, run ./all.bash, which will succeed. Then run go test -short std cmd, which will report various things and eventually a FAIL for TestGdbCoreSignalBacktrace, although it will print a bunch of ?s and other things that may be the sign of additional problems. I'm attaching the full 'go test' output rather than trying to cut it down but omit something that turns out to matter.

go-test-out.txt

This may be somehow related to #61484.

[cherrymui]

The ?s are fine. They are just packages that don't have tests.

The TestGdbCoreSignalBacktrace failure is probably sporadic. Perhaps when you ran all.bash it succeeded. You may try running go test -count=1 -run=TestGdbCoreSignalBacktrace runtime and see if it always fails or fails sometimes. Thanks.

[siebenmann]

Running go test -count=1 -run=TestGdbCoreSignalBacktrace runtime appears to always fail for me. I ran it in a 'while ! go test ...' loop to be sure, and it's still failing away now as I write this. I've also never seen all.bash fail, although I haven't monitored that as extensively.

[heschi]

cc @golang/runtime

[cherrymui]

Perhaps all.bash (cmd/dist) changes some environment and that particular test is skipped (e.g. due to failing to create a core file or something)?

Are you running all.bash and the go test command under the same user and same environment?

[siebenmann]

I'm running both all.bash and go test in the same user and environment. I first noticed this in the context of #61484, when I was asked to run go test directly to see if the Go build cache growth occurred there or only when I ran all.bash. I can't tell if all.bash is running that test; in an all.bash run the report I get is:

[...]
ok      reflect 0.609s
ok      regexp  0.535s
ok      regexp/syntax   0.736s
ok      runtime 25.048s
ok      runtime/cgo     0.014s
ok      runtime/coverage        0.003s
ok      runtime/debug   0.122s
ok      runtime/internal/atomic 0.062s
ok      runtime/internal/math   0.019s
ok      runtime/internal/sys    0.014s
ok      runtime/internal/syscall        0.005s
ok      runtime/internal/wasitest       0.007s
ok      runtime/metrics 0.012s
ok      runtime/pprof   7.964s
ok      runtime/trace   1.999s
[...]

[ianlancetaylor]

The test is failing because it expects a core file to exits, but it doesn't. The test does pass for me on my linux-amd64 system.

There is a check in the test for whether a core file will be generated:

	// Ensure there is enough RLIMIT_CORE available to generate a full core.
	var lim syscall.Rlimit
	err := syscall.Getrlimit(syscall.RLIMIT_CORE, &lim)
	if err != nil {
		t.Fatalf("error getting rlimit: %v", err)
	}
	// Minimum RLIMIT_CORE max to allow. This is a conservative estimate.
	// Most systems allow infinity.
	const minRlimitCore = 100 << 20 // 100 MB
	if lim.Max < minRlimitCore {
		t.Skipf("RLIMIT_CORE max too low: %#+v", lim)
	}

In run.bash we disable core files:

# no core files, please
ulimit -c 0

Therefore, the reason this test passes when run via all.bash is that core files are disabled and the test is skipped.

When run directly, the test assumes that the core file will be named core. On modern Linux systems the name of a core file is controlled by the contents of /proc/sys/kernel/core_pattern. On my system that file contains just core. @siebenmann what does it look like on your system? Thanks.

[siebenmann]

On this system, /proc/sys/kernel/core_pattern contains core, but /proc/sys/kernel/core_uses_pid is set to 1, which creates core files named 'core.PID-VALUE'. The latter setting appears to be the Ubuntu 22.04 default, and perhaps the upstream kernel default.

[ianlancetaylor]

Thanks, that is what is causing the problem on your system. On my system /proc/sys/kernel/core_uses_pid is 0.

Can you see if https://go.dev/cl/511659 fixes the problem for you? Thanks.

[gopherbot]

Change https://go.dev/cl/511659 mentions this issue: runtime: consider core PID in gdb test

[siebenmann]

This CL fixes the problem for me; go test -count=1 -run=TestGdbCoreSignalBacktrace runtime now succeeds, as does go test -short std cmd.