cmd/go: mod tidy with ~1M subdirectories takes a very long time and then crashes computer

[anacrolix]

This probably seems a bit bizarre, and comes out of #56494 (comment) and #55955 (comment). I'm not sure how it's possible, I think it's a MacOS specific issue, possibly to do with ulimit -n, which I understand may have been changed around go1.19.

Testing it is painful as my computer actually crashes as a result, and the project is private, but I'm happy to what I can to expose the issue or make a reproduction.

What version of Go are you using (go version)?

go version devel go1.21-e007c515f3 Mon Apr 24 15:49:38 2023 +0000 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/anacrolix/Library/Caches/go-build"
GOENV="/Users/anacrolix/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/anacrolix/go/pkg/mod"
GONOPROXY="github.com/anacrolix,github.com/some-company"
GONOSUMDB="github.com/anacrolix,github.com/some-company"
GOOS="darwin"
GOPATH="/Users/anacrolix/go"
GOPRIVATE="github.com/anacrolix,github.com/some-company"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/anacrolix/src/go.master"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/anacrolix/src/go.master/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="devel go1.21-e007c515f3 Mon Apr 24 15:49:38 2023 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/abs/path/to/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/m4/f67w9zfx1pl386_2yjs7xtkm0000gn/T/go-build2760210553=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

go mod tidy

What did you expect to see?

Nothing. Maybe a tidier go.mod and go.sum.

What did you see instead?

CPU is at 300-400% and memory climbs to ~400-500MiB over a few minutes then my computer switches off.

[seankhliao]

pass in -v -x flags?

[anacrolix]

It only happens when the system has been running for a few hours.

After it restarts it shows

panic(cpu 2 caller 0xfffffe00236407b0): watchdog timeout: no checkins from watchdogd in 92 seconds (8454 total checkins since monitoring last enabled)

While the issue occurs, I can't start new applications, and I have one application that shows "xcrun too many open files".

There was no output from go mod tidy, even with -x -v.

[anacrolix]

It has something to do with having massive numbers of subdirectories and no hard limit on file descriptors. If I run with ulimit -nH 1024, I actually get output from go mod tidy -x -v. My go-build cache has 1.6m entries, and my project has 500k files in a lot of sub directories, and go mod tidy -x -v is outputting lots of go: finding module for package project-import-path/some/sub/directory.

[bcmills]

my project has 500k files in a lot of sub directories

Do each of those subdirectories contain at least one .go file, or are some or most of them non-Go?

If you set GODEBUG=goindex=0 in your environment, does that change the behavior? (I wonder if this is related to #54226.)

[bcmills]

CPU is at 300-400%

Perhaps send SIGQUIT to the go command once it has been running for a while and send me the stack dump?

[anacrolix]

I have a smaller reproduction:

cd `mktemp -d`
go mod init shit
for a in `seq 1000000`; do mkdir "$a"; done
go mod tidy

I don't know if this causes issues on platforms other than MacOS.

[bcmills]

I can reproduce high CPU usage on Linux, but it doesn't cause a crash — and a module with that many directory entries seems well outside of the limits of what Go can reasonably be expected to support anyway. (I see ~3.9GiB of directory entries alone!)

$ go version
go version devel go1.21-71ad46cd Fri Apr 28 01:33:31 2023 +0000 linux/amd64

$ go mod init example
go: creating new go.mod: module example

$ for a in `seq 1000000`; do mkdir "$a"; done

$ go mod tidy
go: warning: "all" matched no packages

$ du -hs .
3.9G    .

So, my suggestion is to either escalate this to Apple as a macOS kernel bug, or find a way to structure your project so that either you have fewer directory entries overall or they are excluded from go mod tidy (for example, by placing them in a parent directory beginning with _ or . or placing a go.mod file in a parent directory to prune it out of the module).

[anacrolix]

I think having ulimit -nH prevents the crash, so I think there's value in having a reasonable limit in place to prevent the worst case, even if the conditions for it to occur are fairly extreme.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[anacrolix]

This is a mistake.

[bcmills]

The crash seems to be platform-specific, and we have no reason to believe that the platform-specific crash is a Go bug. Again, I suggest that you escalate it to Apple as a potential macOS kernel bug.

The solution to the performance issue seems straightforward: find a different way to structure the project.
(Also note that #42965 would provide a mechanism for ignoring specific root directories, which could perhaps give you a way to keep go mod tidy from scanning subtrees if these aren't all in the module root.)

[anacrolix]

Thanks @bcmills, I think #42965 might work as a solution.