Down repository prevents vendoring of other packages

[calmh]

What version of Go (go version) and dep (git describe --tags) are you using?

Go 1.9beta1, v0.1.0-172-gdbd910b (latest as of right now)

What dep command did you run?

dep ensure golang.org/x/net@master

What did you expect to see?

The package should have been ensured.

What did you see instead?

solve error: No versions of (some private repo) met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"
ensure Solve(): No versions of (some private repo) met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"

The (some private repo) package depends on stvn.cc/broadcaster. That vanity domain is currently down.

But stvn.cc/broadcaster is already vendored, is in the lockfile, is not a dependency of golang.org/x/net and should not need to be touched at all.

[sdboyer]

hi, thanks for the feedback! 😄

there's a few things going on here.

is not a dependency of golang.org/x/net and should not need to be touched at all.

unfortunately, this is fundamental to the problem here: we can't know a priori that stvn.cc/broadcaster isn't a dependency of golang.org/x/net, or that they don't have some mutual shared dependency. the only way for us to guarantee the overall consistency of any ensure operation is to recompute the entire solution. that necessarily entails visiting the entire depgraph.

That vanity domain is currently down.

dep's behavior turns on our ability to infer where the root of a project is, just by looking at the import path. for standard, e.g. github/bitbucket imports, we have a general rule; for custom domains, we must rely on the vanity domain info to tell us where the repo root is. from there, we make the assumption that repo root == project root.

when a vanity domain goes down, it becomes impossible for us to know how to deal with vanity import paths at all, and because we have to revisit the whole depgraph (per the above), it necessarily causes the entire process to abort.

i would say that this issue is actually more broadly about supporting an "offline mode" - e.g., using dep on a plane, and falling back strictly to locally-available information. for vanity domains, that entails having a cache of the vanity domain's HTTP metadata - one of the things that the cache system in #431 will eventually need to support (though probably not in the first iteration).

[calmh]

Hi!

we can't know a priori that stvn.cc/broadcaster isn't a dependency of golang.org/x/net, or that they don't have some mutual shared dependency

Yeah, I can see that. But the golang.org/x/net part of it is available, and the other dependencies are already vendored. I get the part about needing to talk to the vanity domain to figure out the root etc, but my complaint is that these packages are already vendored and the information it needs is cached in the constraints and lock files (I think? at least I expect it should be.).

Perhaps I'm misunderstanding dep, but I would expect it to not have to talk to all the remotes of all existing, locked, dependencies in this case.

[sdboyer]

Perhaps I'm misunderstanding dep, but I would expect it to not have to talk to all the remotes of all existing, locked, dependencies in this case.

For the most part, talking to other upstreams can be avoided, given some cleverness. Maybe half of that cleverness is currently in place.

But, in general, the problem that dep solves is deceptively difficult. In the complex world of dependency management, our primary concern is safety.

[calmh]

Gotcha. Early days and so on. The offline story is something to consider though. As it is, I've switched over to manual cp -r managing of the vendor folder as dep can no longer operate on this project.