cmd/go: decide replace, exclude wildcard behavior

[rsc]

In CL 122400 @bcmills pointed out that we haven't defined what

replace (
	example.com/foo v1.0.0 => elucidation.com/foo v1.1.0
	example.com/foo => elucidation.com/foo v1.2.0
)

means for example v1.0.0. Obviously it must mean you get elucidation v1.1.0 not v1.2.0. But that's not documented nor implemented.

Bryan also raised the question of whether to allow

exclude example.com/foo

to make it completely unavailable. Probably we should.

[rsc]

Will have to solve in Go 1.12. Until then, "don't do that."

[bcmills]

Part of the problem here is that there are two different possible meanings for a “wildcard replace”.

One is, “replace every single version of this module with this specific source code”. That destroys all of the transitive requirements that appear anywhere in the module graph, which are important in general (see #31248), so it should be used only as a last resort. (For example, it could be used if the upstream origin is completely gone and most versions are not cached in any proxy, or if the origin is completely unusable due to a licensing or security issue and needs to be expunged completely from the module graph.)

The other — from what I can tell, by far the more common case — is “once you have selected a version of this module, pretend that this replacement is an even higher version and use it instead”. That's the replace you want, for example, when preparing upstream patches (#27542).

CC @jayconrod @thepudds @Helcaraxan

[Helcaraxan]

Adding my own brain-dump to the topic, mixed with some thoughts on adjacent issues (who said "multi-module repos"?).

• On the exclude mydomain.com/foo/bar to prevent any use of that module origin.

Supporting this would definitely make sense for me. It would provide native support for (security) blacklisting of specific hosts and I would even extend it to not necessarily be a module path but more a pattern: exclude github.com (or exclude github.com/* TBD) would prevent any use of sources provided by GitHub repos. Counter-point to my position here would be that such blacklisting should be managed via a custom module-proxy... but small OSS projects will not have one and the exclude could thus be a "poor man's alternative".

• With respect to the original example:

replace (
	example.com/foo v1.0.0 => elucidation.com/foo v1.1.0
	example.com/foo => elucidation.com/foo v1.2.0
)

It does seems sensible to make this result in replacing v1.0.0 with v1.1.0 instead of v1.2.0. However we could also argue that finding this in a go.mod should simply result in an error. If we were to turn it into an error I would also argue we add an error when a given replace statement is not used to construct the final build list. For cases where people do want to keep such unused lines around we can implement the // keep comment approach also used in tools such as gazelle to disable the new errors.

• On the general semantics and use-cases of the "wildcard" replace statement:

For example, it could be used if the upstream origin is completely gone and most versions are not cached in any proxy, or if the origin is completely unusable due to a licensing or security issue and needs to be expunged completely from the module graph.

This ties somewhat back to my note about the exclude above. If you are expunging an entire module I would very much suspect that you will also very likely not be using any forks or other replacements for it: you would start using an entirely different dependency with its own module path. The generalised exclude would be enough to handle such a scenario. For the case where you are in the process of expunging the module and you need to use a temporary fork users should fall back on a version-specific replace statement in addition to the exclude. This would also have the benefit of encouraging people to consider moving away from the fork as they would otherwise need to continuously update the replace statement at each change to keep builds working.

The other — from what I can tell, by far the more common case — is “once you have selected a version of this module, pretend that this replacement is an even higher version and use it instead”. That's the replace you want, for example, when preparing upstream patches (#27542).

The related use-case, although definitely valid, is not one I have personally encountered much yet. In practice this particular replace semantic is used in a much wider fashion. More in line with "pretend that this replacement is this specific version and use it instead".

One example is the way Kubernetes uses it to achieve the continuous pinning of versions for their dependencies. They rely on the wildcard version but they could work as well with the version-specific statement as well, but this would complicate their already complex automation around module-maintenance even further.

Another similar and frequent use is when migrating (old) projects to modules, whether they have been using third-party dependency management or simply relying on GOPATH. Projects frequently encounter the issue that the dependency graph they rely on for building is not compatible with MVS. Migration thus generally means adding a replace statement for each (in)direct dependency to use the exact versions from the existing dependency graph. Then in a second stage slowly iterate over dependencies and one-by-one update them to conform to MVS.

---

NB: I am not advocating that the way Kubernetes uses replace and their overall philosophy is close to being canonical or even a good example... merely that it is a big community project and thus is shaping the image many Go developers will have, or get, of the use of modules. Even today I have already heard from many individual developers converting to modules that they want, or need in their own words, to learn how to develop multi-module repositories just because Kubernetes is doing so. It requires a lot explaining each time as these developers have no valid reason for actually needing it themselves... but that's a different (albeit related) problem.

[Helcaraxan]

Instead of polluting this thread with my thoughts on multi-module repos I've posted them into the linked, and more appropriate, issue #27542.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping. If it no longer needs to be done early in cycle, that label can be removed.

[dmitshur]

This issue is currently labeled as early-in-cycle for Go 1.16.
That time is now, so this is a friendly ping so the issue is looked at again.

[rsc]

Not for this cycle.

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.17.
That time is now, so a friendly reminder to look at it again.

[ianlancetaylor]

Any update on this issue?

[bcmills]

exclude is pretty solidly defined as of Go 1.16. I'm going to make a sweep through the various replace semantics once lazy loading (#36460) is done, and hopefully simplify and clarify things at that point.

[cagedmantis]

This issue is currently labeled as early-in-cycle for Go 1.18.
That time is now, so this is a friendly ping so the issue is looked at again.

[gopherbot]

This issue is currently labeled as early-in-cycle for Go 1.19.
That time is now, so a friendly reminder to look at it again.

[ianlancetaylor]

@bcmills @matloob This issue is marked for 1.19. It's just been rolling forward in milestones. Should it move to Backlog?

[bcmills]

At this point the behavior for these directives is well established; it is, in effect, already decided. Closing as obsolete.