cmd/go: cannot Resolve K8s Package from Dependency

[swtch1]

What version of Go are you using (go version)?

go version go1.11 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Windows 10 Enterprise x64
Reproduced with go1.11 on Windows Subsystem for Linux (Ubuntu) as well.

What did you do?

Tried to run go test/go build with code that uses methods from the prometheus monitoring project.

Reproduce with:

package main

import (
        "github.com/prometheus/prometheus/config"
)

func main() {}

var c config.Config

Ultimately related to issues I've opened before #26843 and dep issue 1976.

At this point I'm a bit perplexed as to how to deal with this.

What did you expect to see?

The standard behavior of a test or build.

What did you see instead?

$ go mod verify
all modules verified
$ go build main.go
go: finding github.com/Azure/azure-sdk-for-go/arm/compute latest
go: finding github.com/Azure/azure-sdk-for-go/arm/network latest
go: finding github.com/Azure/azure-sdk-for-go/arm latest
go: finding k8s.io/client-go/pkg/api latest
go: finding k8s.io/client-go/pkg/apis/extensions/v1beta1 latest
go: finding k8s.io/client-go/pkg latest
go: finding k8s.io/client-go/pkg/apis/extensions latest
go: finding k8s.io/client-go/pkg/apis latest
build promtest: cannot find module for path github.com/Azure/azure-sdk-for-go/arm/compute

Fix azure-sdk-for-go issue by adding to go.mod:
require github.com/Azure/azure-sdk-for-go v5.0.0-beta.0.20161028183111-bd73d950fa44+incompatible // indirect

$ go build main.go
go: finding github.com/Azure/azure-sdk-for-go v5.0.0-beta.0.20161028183111-bd73d950fa44+incompatible
go: finding k8s.io/client-go/pkg/apis/extensions/v1beta1 latest
go: finding k8s.io/client-go/pkg/api latest
go: finding k8s.io/client-go/pkg/apis/extensions latest
go: finding k8s.io/client-go/pkg latest
go: finding k8s.io/client-go/pkg/apis latest
../../go/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/kubernetes.go:34:2: unknown import path
 "k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api
../../go/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/endpoints.go:26:2: unknown import path
"k8s.io/client-go/pkg/api/v1": cannot find module providing package k8s.io/client-go/pkg/api/v1
../../go/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/ingress.go:25:2: unknown import path "k
8s.io/client-go/pkg/apis/extensions/v1beta1": cannot find module providing package k8s.io/client-go/pkg/apis/extensions/v1beta1

Oddly enough, k8s.io/client-go/pkg/api does not exist in the prometheus vendor directory and I cannot find it remapped in their vendor.json. Could k8s.io/client-go/pkg/api be using k8s.io/api?

[myitcv]

@swtch1 this situation comes about because of breaking changes in the github.com/Azure/azure-sdk-for-go packages. Because the import path of the github.com/Azure/azure-sdk-for-go/... packages did not change after the breaking change, it's impossible to work out which version is "correct". Hence the power of semantic import versioning.

There is a way to "fix" your case however, and that is to bootstrap your dependencies from the github.com/prometheus/prometheus module:

# use a clean GOPATH for testing
export GOPATH=$(mktemp -d)

cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
cat <<EOD > main.go
package main

import (
        "github.com/prometheus/prometheus/config"
)

func main() {}

var c config.Config
EOD

# create a temporary directory for a prometheus clone
pd=$(mktemp -d)
git clone https://github.com/prometheus/prometheus/ $pd
pushd $pd
go mod init
popd

# now back to our example
go mod edit -require=github.com/prometheus/prometheus@latest -replace=github.com/prometheus/prometheus=$pd

go build main.go

succeeds.

cc @rsc @bcmills. Perhaps when resolving github.com/prometheus/prometheus, we see that it's a non-module dependency, we also see that it could be converted from its existing dependency management configuration (govendor), should we use the dependencies implied by the result of go mod init?

EDIT: to merge the two go mod edit lines.

[bcmills]

should we use the dependencies implied by the result of go mod init?

In an earlier draft of module support we did that to synthesize a converted go.mod for every dependency, but @rsc realized that if we got anything wrong in the converted go.mod there would be no way for the user to fix it — and it wouldn't necessarily be stable over time as we fix bugs in the converters, either.

I suppose we could add the dependencies implied by go mod init for the non-module dependency directly to the parent go.mod, but that would add a lot of complexity (not only to the code, but also to the user's ability to reason about what the code does), and it's not obvious that that would leave us in a much better state.

[myitcv]

Thanks for the background @bcmills.

@swtch1 sounds like this is working as intended. Hopefully my explanation and workaround in #27457 (comment) are clear.

Indeed that workaround (or a variation on it) could be absorbed into a special gohack subcommand.

That said, this does seem like a fairly uncommon case (I'll wait to be disproved on that statement 😄)

[swtch1]

Hey @myitcv. Thank you very much for providing the script here, but this does not succeed for me.

The first error I get is:
build example.com/hello: cannot find module for path github.com/Azure/azure-sdk-for-go/arm/compute

Then after adding github.com/Azure/azure-sdk-for-go v5.0.0-beta.0.20161028183111-bd73d950fa44+incompatible // indirect to go mod I get:

go: finding k8s.io/client-go/pkg/apis latest
/tmp/tmp.O4RyoujYYn/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/kubernetes.go:34:2: unknown import path "k8s.io/client-go/pkg/api": cannot find module providing package k8s.io/client-go/pkg/api
/tmp/tmp.O4RyoujYYn/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/endpoints.go:26:2: unknown import path "k8s.io/client-go/pkg/api/v1": cannot find module providing package k8s.io/client-go/pkg/api/v1
/tmp/tmp.O4RyoujYYn/pkg/mod/github.com/prometheus/prometheus@v2.3.2+incompatible/discovery/kubernetes/ingress.go:25:2: unknown import path "k8s.io/client-go/pkg/apis/extensions/v1beta1": cannot find module providing package k8s.io/client-go/pkg/apis/extensions/v1beta1

I have tried running this script on git bash on Windows, Windows subsystem for linux (Ubuntu) and CentOS, all with the same result. All are running the latest Go build.

I do see this error in the middle of the scripted commands:

[root@x hello]# go mod edit -require=github.com/prometheus/prometheus@latest
[root@x hello]# go mod edit -replace=github.com/prometheus/prometheus=$pd
go: errors parsing go.mod:
/tmp/tmp.uBjO0NuXgH/hello/go.mod:3: invalid module version "latest": version must be of the form v1.2.3

I am very appreciative of the work you've done but at this point I still have no way to build. Do you have any further tips? Am I supposed to be entering a specific version there instead of @latest? and if so how do I acquire that? Thank you for the work thus far.

[myitcv]

@swtch1 just to be clear, does a copy-paste of the commands in #27457 (comment) work for you?

Please note I've just made a slight tweak to the comment to fix one of the commands apologies for that, so a page refresh might be required

If not, please can you include the full output?

I ask because those steps don't say anything about adding github.com/Azure/azure-sdk-for-go v5.0.0-beta.0.20161028183111-bd73d950fa44+incompatible // indirect to your go.mod.

[swtch1]

@myitcv your script modifications worked for me. I really can't thank you enough.. I haven't been able to build this project since I added those deps. This is seriously huge, thank you! Also not something I think I would have figured out on my own as I played with the replacements and other methods for hours.

[bcmills]

I'm a bit perplexed as to why you need a local clone and a replace directive. It should be possible to achieve the same results by using go get with the appropriate versions and possibly applying replace directives to remap newer packages to older ones.

I tried digging into it a bit, but ran into trouble because I had to clear my module cache (pre-1.11 weirdness) and then couldn't re-resolve gopkg.in dependencies due to a GitHub outage (niemeyer/gopkg#63).

[myitcv]

Not quite sure what you mean @bcmills? Doing a go get will just fail, because it starts to pull down modules at the "wrong" versions. Hence the clone approach, to sidestep all that wasted effort, and instead start with the go mod init which will get the "right" versions.

[bcmills]

I'll post more detail once I can get a trace of commands that work. 🙂

[bcmills]

Ok, there is one outstanding bug here, but the rest seems to work fine without any replace directives.

The bug is that, for some reason, go get github.com/Azure/azure-sdk-for-go@v5.0.0-beta adds v20.1.0+incompatible instead of the requested version.

If I fix that with a go mod edit, it seems to remain stable. I end up needing to fix up two other versions to point to non-semver revisions (one for k8s.io/apimachinery, which uses a different versioning scheme for its tags, and one for github.com/hashicorp/serf, which some other (tagged) hashicorp package requires at a version newer than the latest tag), but beyond that everything builds.

Using the source file from the original report:

$ go mod init github.com/golang/go/issues/27457
go: creating new go.mod: module github.com/golang/go/issues/27457

$ go get k8s.io/apimachinery@kubernetes-1.11.3
go: finding k8s.io/apimachinery kubernetes-1.11.3
go: finding k8s.io/apimachinery latest

$ go get github.com/hashicorp/serf@48d5794
go: finding github.com/hashicorp/serf 48d5794

$ go get github.com/Azure/azure-sdk-for-go@v5.0.0-beta
go: finding github.com/Azure/azure-sdk-for-go v5.0.0-beta

$ go build all
[…]
can't load package: package github.com/Azure/azure-sdk-for-go/arm/compute: unknown import path "github.com/Azure/azure-sdk-for-go/arm/compute": cannot find module providing package github.com/Azure/azure-sdk-for-go/arm/compute
can't load package: package github.com/Azure/azure-sdk-for-go/arm/network: unknown import path "github.com/Azure/azure-sdk-for-go/arm/network": cannot find module providing package github.com/Azure/azure-sdk-for-go/arm/network

$ go mod edit -require github.com/Azure/azure-sdk-for-go@v5.0.0-beta.0.20161028183111-bd73d950fa44+incompatible

$ go build all

$ go mod tidy

$ go build all

$

[myitcv]

Ah I see what you mean now @bcmills

Ok, there is one outstanding bug here, but the rest seems to work fine without any replace directives.

Indeed, but in its place you have to know to perform the various go get's listed in your solution, no?

Or was this more a case of chasing down why the approach originally described in this issue didn't work?

[bcmills]

Indeed, but in its place you have to know to perform the various go get's listed in your solution, no?

Yes. In general, if you have multiple pre-module dependencies you may need to use go get to make adjustments during the initial module definition, and as you noted, examining the output of go mod init for particularly problematic or subtle dependencies can help with that.

The number of such cases should drop as more repos get module definitions.

Or was this more a case of chasing down why the approach originally described in this issue didn't work?

The fact that you suggested replace instead of get seemed off to me, since that only fixes the dependencies locally (rather than for everyone downstream). I wanted to see whether there was some other issue preventing the get approach from working in this case — and there was, so, a win all around!

[myitcv]

The number of such cases should drop as more repos get module definitions.

Absolutely.

and there was, so, a win all around!

👍

[jharshman]

Having a similar issue as described here: https://groups.google.com/forum/#!topic/golang-nuts/BYV0IH_809Q. Thanks @myitcv for pointing me to this issue.

The "fix" seems to be some of what @bcmills described. ApiMachinery wasn't the version I wanted even after a go get. I had to set this manually with go mod edit -replace. This kind of thing doesn't really make me feel too confident about the system as even after I fixed that one package, it exposed others that I now have to also fix manually... 😕

# k8s.io/kubernetes/pkg/util/parsers
/go/pkg/mod/k8s.io/kubernetes@v1.11.3/pkg/util/parsers/parsers.go:36:16: undefined: reference.ParseNormalizedNamed
# github.com/hashicorp/consul/lib
/go/pkg/mod/github.com/hashicorp/consul@v1.2.3/lib/serf.go:18:6: base.MinQueueDepth undefined (type *serf.Config has no field or method MinQueueDepth)
/go/pkg/mod/github.com/hashicorp/consul@v1.2.3/lib/serf.go:24:6: base.LeavePropagateDelay undefined (type *serf.Config has no field or method LeavePropagateDelay)

[bcmills]

I decided to look into the k8s.io dependency web once and for all, and ended up making https://github.com/bcmills/k8s-mods.

For your convenience, try:

go get -m github.com/bcmills/k8s-mods && go mod tidy

That should at least rough in a consistent starting point for your own module.

[jharshman]

@bcmills thanks!