runtime: large stack makes the program hang

[kjk]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

kjkmacpro:tst kjk$ go version
go version go1.10.3 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kjk/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kjk/src/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10.3/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v_/ksw1dqvd59v790zk2wqf_t_80000gn/T/go-build146241323=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

https://play.golang.org/p/bEhVv3kwiNr

package main

import (
	"fmt"
)

func stackOverflow(x *byte) {
	var buf [1024 * 1024 * 11]byte
	stackOverflow(&buf[0])
}

func main() {
	fmt.Printf("Before stack overflow\n")
	stackOverflow(nil)
	fmt.Printf("After stack overflow\n")
}

This program never finishes.

Change the buf size to [1024 * 1024 * 10] (https://play.golang.org/p/s5sf7m0qWcu) and it'll crash with stack overflow error.

It looks like runtime is bouncing system threads:

(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:56
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
* 8    Thread 10720.0xdd0 0x00007ffd1f3649bf in ?? ()
(gdb) c
Continuing.
[Thread 10720.0xdd0 exited with code 0]
[New Thread 10720.0x1538]
[New Thread 10720.0x2ba8]
[New Thread 10720.0x6a4]

Thread 11 received signal SIGINT, Interrupt.
[Switching to Thread 10720.0x6a4]
0x00007ffd1f3649bf in ?? ()
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc 0x00007ffd22d29f34 in ?? ()
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
  9    Thread 10720.0x1538 0x00007ffd22d29f34 in ?? ()
  10   Thread 10720.0x2ba8 runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:51
* 11   Thread 10720.0x6a4 0x00007ffd1f3649bf in ?? ()
(gdb) c
Continuing.
[Thread 10720.0x6a4 exited with code 0]
[New Thread 10720.0x2bf0]

Thread 12 received signal SIGINT, Interrupt.
[Switching to Thread 10720.0x2bf0]
0x00007ffd1f3649bf in ?? ()
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 10720.0x11f8 0x00007ffd22d29f34 in ?? ()
  2    Thread 10720.0x24e4 0x00007ffd22d2d804 in ?? ()
  3    Thread 10720.0x204c 0x00007ffd22d2d804 in ?? ()
  4    Thread 10720.0x1374 0x00007ffd22d29f34 in ?? ()
  5    Thread 10720.0xd50 0x00007ffd22d29f34 in ?? ()
  6    Thread 10720.0x24dc 0x00007ffd22d29f34 in ?? ()
  7    Thread 10720.0x19c 0x00007ffd22d29f34 in ?? ()
  9    Thread 10720.0x1538 runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
  10   Thread 10720.0x2ba8 0x00007ffd22d29f34 in ?? ()
* 12   Thread 10720.0x2bf0 0x00007ffd1f3649bf in ?? ()

(gdb) thread 9
[Switching to thread 9 (Thread 10720.0x1538)]
#0  runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
47              MOVOU   X0, 16(DI)
(gdb) bt
#0  runtime.memclrNoHeapPointers () at C:/Users/kjk/src/go/src/runtime/memclr_amd64.s:47
#1  0x0000000000421351 in runtime.(*mheap).alloc (h=<optimized out>, large=<optimized out>, needzero=true, npage=<optimized out>, spanclass=<optimized out>,
    ~r4=<optimized out>) at C:/Users/kjk/src/go/src/runtime/mheap.go:340
#2  0x000000000040bafe in runtime.largeAlloc (needzero=true, noscan=true, size=11534336, ~r3=<optimized out>) at C:/Users/kjk/src/go/src/runtime/malloc.go:1011
#3  0x000000000044c8ed in runtime.mallocgc.func1 () at C:/Users/kjk/src/go/src/runtime/malloc.go:906
#4  0x000000000044f1bb in runtime.systemstack () at C:/Users/kjk/src/go/src/runtime/asm_amd64.s:351
#5  0x000000000042e720 in runtime.startTheWorldWithSema () at C:/Users/kjk/src/go/src/runtime/proc.go:1146
#6  0x0000000000000000 in ?? ()

This came as part of investigating #21382

[ianlancetaylor]

Go 1.6 crashes with a stack overflow error, as does gccgo. Go versions after 1.6 appear to just burn CPU without allocating memory.

[randall77]

It looks like you're tripping across the boundary between stack-allocating and heap-allocating buf. We only allow objects to be stack allocated below a certain size, which happens to be:

cmd/compile/internal/gc/go.go:	maxStackVarSize = 10 * 1024 * 1024

When buf is stack allocated (when it is 1024*1024*10 bytes), we blow through stack really quickly, and you get a stack overflow error.
When buf is heap allocated (when it is 1024*1024*11 bytes), the stack frames are really small and it takes lots of them to overflow the stack. For each frame, we need to allocate 11MB in the heap, which is promptly garbage and needs to be collected. That's a lot of work per frame, and thus it takes a really long time to get to an overflow (but it will: 32 bytes/frame, ~1ms/frame, 1GB max stack, should take ~9 hours).

I'm going to close as not-a-bug. Definitely odd, but I'd argue that with the heap-allocated buf we're actually trying harder, and mostly succeeding, in doing what the programmer asked. Stack overflow errors are best effort to help the programmer avoid obvious bugs; they aren't required by the spec or anything like that.

@aclements This is kind of a weird case, where the GC is collecting a single 11MB object each iteration. Probably a larger heap goal size would be useful in situations like these (although this particular situation is contrived enough to be irrelevant).

gc 10415 @12.031s 7%: 0.002+0.72+0.048 ms clock, 0.025+0.69/0.042/0.69+0.58 ms cpu, 11->11->0 MB, 12 MB goal, 12 P

[aclements]

This is kind of a weird case, where the GC is collecting a single 11MB object each iteration.

Perhaps the proper solution is to solve #23044 (comment). That would happen to space out these GCs, and would mean that when they did run, the cost of marking (0.72 ms in that example) would be amortized against the less frequent GC scheduling.