cmd/go: in go 1.21 module, go mod tidy and 'go list -m -u all' disagree over which go.sum lines are needed

[dmitshur]

What version of Go are you using (go version)?

$ go version
go version go1.21rc3 darwin/arm64

Does this issue reproduce with the latest release?

No(-ish).

I first ran into this in a module whose 'go' line is 1.21, so the latest release today (1.20.6) isn't applicable.

If I change the 'go' line to 1.20, then 'go mod tidy' no longer deletes as many go.sum lines, and the issue doesn't reproduce.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/gopher/Library/Caches/go-build'
GOENV='/Users/gopher/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='loopvar'
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/gopher/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/gopher/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21rc3'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/go-build1549452710=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I ran go mod tidy in a medium sized local module whose go.mod has 'go 1.21' (primarily to help test this language version and its semantics, not because I need it; this go.mod file isn't public).

Then I ran go list -m -u all.

Edit: It turned out trivial to trigger the issue with a very minified top-level go.mod file. Here's a complete reproducer:

$ cd $(mktemp -d)
$ go mod init issue61605.test
$ echo 'package p; import _ "golang.org/x/build/maintner/reclog"' > p.go
$ go get golang.org/x/build@v0.0.0-20230726111318-84c18c56e0b0
$ go mod tidy
$ go list -m -u all

What did you expect to see?

The usual list of modules and available updates, and exit code 0.

What did you see instead?

$ go list -m -u all
go: updates to go.sum needed, disabled by -mod=readonly
$ echo $?
1

Running the go list command with -mod=mod re-adds the go.sum lines and makes the command work.

Or setting go to 1.20 instead of 1.21 and re-running go mod tidy also works around the issue.

[dmitshur]

CC @bcmills, @matloob.

[rsc]

Here's a test case version. I have not managed to whittle it down but I think I see a way to make it stop breaking (not the right way, but a way).

[!net:golang.org] skip

env GOPROXY=https://proxy.golang.org/
env GOSUMDB=

! go list
stderr 'missing go.sum entry'
go mod tidy
go list all
cat go.sum

go list -m -u
cat go.sum

go list -m -u all
cat go.sum

go list -mod=mod -m -u
cat go.sum

go list -mod=mod -m -u all
cat go.sum
stdout xxx

-- go.mod --
module m
require golang.org/x/build v0.0.0-20230726111318-84c18c56e0b0

-- x.go --
package p; import _ "golang.org/x/build/maintner/reclog"

[gopherbot]

Change https://go.dev/cl/513778 mentions this issue: cmd/go: make go list -m -u all not complain about missing checksums

[rsc]

After CL 513778 is submitted to the Go 1.21 release branch, we can move this to the Go 1.22 milestone.

[gopherbot]

Change https://go.dev/cl/514899 mentions this issue: [release-branch.go1.21] cmd/go: make go list -m -u all not complain about missing checksums

[rsc]

Submitted the band-aid to the Go 1.21 release branch. Now a Go 1.22 bug.

[bcmills]

I think a proper fix is in the general direction of #40775. The set of checksums to retain should be associated with the modload.Requirements struct in some way rather than tracked as a global variable.

[bcmills]

@matloob, @samthanawalla: this is a code cleanup needed in cmd/go. I expect that a proper solution to it will be fairly involved; there is an unfortunate amount of communication through global variables in the code that updates the go.sum file.