cmd/go: cannot resolve subpackages of unfetched modules

[bcmills]

In the fix for #26238, @rsc introduced a new +incompatible convention for projects using legacy build tags.

However, that mechanism does not seem to work for resolving packages that are not at the root of the module tree.

$ go version
go version devel +a9e638380f Tue Jul 24 16:59:54 2018 -0400 linux/amd64
$ cd $(mktemp -d)
$ export GOPATH=$(pwd)
$ mkdir -p src/proxy
$ cd src/proxy

src/proxy$ cat > proxy.go
package proxy
import _ "github.com/google/martian/httpspec"

src/proxy$ GO111MODULE=on go mod -init -module example.com/proxy
go: creating new go.mod: module example.com/proxy

src/proxy$ GO111MODULE=on go get github.com/google/martian/httpspec@v2.0.0-beta.2
go: finding github.com/google/martian/httpspec v2.0.0-beta.2
go: finding github.com/google/martian/httpspec latest
go: finding github.com/google/martian v2.0.0-beta.2+incompatible
go: downloading github.com/google/martian v2.0.0-beta.2+incompatible
go get github.com/google/martian/httpspec@v2.0.0-beta.2: cannot find module providing package github.com/google/martian/httpspec

src/proxy$ GO111MODULE=on go list github.com/google/martian/httpspec
go: finding github.com/google/martian/httpspec latest
go: import "github.com/google/martian/httpspec": cannot find module providing package github.com/google/martian/httpspec

src/proxy$ GO111MODULE=off go get github.com/google/martian/httpspec

src/proxy$ GO111MODULE=off go list github.com/google/martian/httpspec
github.com/google/martian/httpspec

[bcmills]

This was found while attempting to produce a module definition for golang.org/x/build.

[bcmills]

I can reproduce this by moving a small block of mod_get_commit.txt to the beginning of the test (before the attempts using the module root path):

go/src/cmd/go/testdata/script/mod_get_commit.txt

Lines 11 to 18 in db810b6

	# golang.org/x/text/language@commit should not resolve with -m,
	# because that's not a module path.
	! go get -m golang.org/x/text/language@14c0d48
	# ... but it should work without -m.
	# because of -d, the compiler should not run
	go get -d -x golang.org/x/text/language@14c0d48
	! stderr 'compile|cp|gccgo .*language\.a$'

(CC @rsc, but I'm going to see if I can fix it tomorrow.)

[bcmills]

This turned out to be a complex of compensating bugs and surprises.

• modload.Import happily imports module roots as if they were packages, even if they aren't.
• go get uses the output of modload.Import to decide which module to fetch.
• search.CleanImportPaths returns "." if len(args) == 0.

As a result, a bunch of our existing tests end up compiling and installing empty packages in response to go get -u without arguments. There needs to be some step where we skip installation entirely if we invoked get without args in a module root that is not a package, and I haven't yet figured out how to do that cleanly. (I have a pretty big subset of the change done, but without that the tests fail right and left.)

[gopherbot]

Change https://golang.org/cl/127936 mentions this issue: cmd/go: do not rely on Import to find the module for a new package

[gopherbot]

Change https://golang.org/cl/127935 mentions this issue: cmd/go: reproduce #26602

[gopherbot]

Change https://golang.org/cl/128136 mentions this issue: cmd/go/internal: factor out modload.QueryPackage and use in in modget