proposal: cmd/go: introduce 'module GOPATH' to allow transition from GOPATH to modules

[bcmills]

In versions of Go before modules, users were able to set up a single GOPATH containing a group of interdependent packages. With the advent of modules, a single module may contain multiple interdependent packages. However, all of those packages must share the same import-path prefix, which is the module path declared in the main module's go.mod file.

For some users — especially hobbyists who did not publish their code for external use — the process of migrating from this “global” namespace to the “local” namespaces of modules can be burdensome. (See, for example, the various discussions in #37755 and #44347.)

When we started using modules to manage the vendored dependencies of the Go standard library itself (#30241), we created a special std module which, unlike all other modules so far, does not add an import-path prefix to the packages it contains.

It recently occurred to me that we could use this same approach to allow “tinkering” users to more easily convert their GOPATH workspace to work in module mode.

For such a module:

• The go.mod file would explicitly indicate that the user wants a module with no import-path prefix, perhaps by declaring a distinguished module path (module GOPATH or module src?) or using a special directive (noprefix?).

• Within an unprefixed module, the usual module-mode commands, including go mod tidy and go get, would be supported.

  • go get would add module dependencies as in module mode, not clone repos into the local module as in GOPATH mode.

• No other module may itself require an unprefixed module. (This is to preserve the invariant that a user can identify which modules may contain a given package by comparing the package path to the module path.)

• As in an ordinary module, collisions between packages defined in the unprefixed module and packages defined in the module dependencies of that module would not be allowed.

  • However, because we know that no other module may require an unprefixed module, we could relax this restriction in the future. Packages definitions in the main module would always take precedence over package definitions in module dependencies.

CC @jayconrod @matloob

[robpike]

This is clever and would have helped me a number of times.

[robpike]

See #44660

[thepudds]

Hi Bryan, I like this idea, and I very much like the general concept of a gentler on-ramp to modules and import paths and so on. One thing to consider is a graduation mechamism. Ideally, cmd/go would rewrite the import paths and module path in a consistent manner. In other words, something like start via go mod intro, and later go mod intro -graduate <module path> (though not trying to suggest a specific spelling or trigger bikeshedding).

It would be nice if cmd/go could cross the import path editing rubicon (which might also justify a cmd/go version of #32014), but if not, some gopher from the broader community would almost certainly create some flavor of a go-mod-graduate or similar.

If some version of what you outlined was supported, I would likely recommend it as a starting point to some of my colleagues who have written plenty of Go, but who still fumble around when starting a module from scratch. An automated graduation path in turn would help them transition if needed to a working, consistent full-blown module, which is then much easier to expand upon.

[bcmills]

@thepudds, I think we already have a graduation mechanism: namely, carving out “nested” modules, where in this case a “nested” module is any module with a non-empty import prefix.

That is: at any point, you can carve out any subpackage into a module by... making it its own module, and adding a require directive to point the prefixless module to that nested module. (The only constraint on that refactoring is that the new (prefixed) module itself may only require other modules that have prefixes, so the process of converting individual paths to modules has to begin with modules containing the “leaf” packages.)

[mvdan]

I used to do the same kind of GOPATH tinkering and I've got used to module test pretty easily (#37641). Paths like test/foobar are not too long compared to foobar, and they're still clearly not for external use or publishable. Do we need to add a special case to avoid a test/ prefix?

[icholy]

This almost seems like a re-introduction of the previous vendor behaviour.

[bcmills]

@icholy, the crucial difference between this and GOPATH-mode vendor is that under this proposal each package still has only one definition (used by the whole module), rather than potentially one per importer.

(In module mode, there is only one vendor directory for the whole module, not one per package subtree as in GOPATH mode.)

[icholy]

The go.mod file would explicitly indicate that the user wants a module with no import-path prefix, perhaps by declaring a distinguished module path (module GOPATH or module src?) or using a special directive (noprefix?).

Why not just omit the module name?

module

go 1.16

[bcmills]

Maybe! But is that too difficult to distinguish from an accidental typo? (I'm honestly not sure.)

[icholy]

Actually, that won't work. Older versions of Go will choke before getting to the go 1.N directive.

go: errors parsing go.mod:
/home/icholy/src/go.mod:1: usage: module module/path

edit: but I guess that wouldn't matter because it can't be required.
edit: re: typo, I can see it being a footgun.

[fzipp]

Maybe! But is that too difficult to distinguish from an accidental typo? (I'm honestly not sure.)

module _

go 1.16

maybe?

[bcmills]

Older versions of Go will choke before getting to the go 1.N directive.

That doesn't particularly matter, because older versions of Go won't know how to interpret a prefixless user module anyway. 😅

[icholy]

warning, heavy speculation:

If this gets accepted/implemented, prefix-less mode will become the preferred choice for non-library modules. Once/if that becomes the case, people will also want to write prefix-less library code. This will become a point of contention.

[rsc]

If we do this I think it should probably be the suggested way to migrate from GOPATH to modules incrementally, at which point module GOPATH seems like the right name. But then it should also apply the GOPATH-mode-specific "vendor import rewrites" (https://golang.org/s/go15vendor). And then GO111MODULE=off mode would maybe transform to an imagined immutable module GOPATH go.mod in GOPATH/src. That would provide a way to keep GOPATH mode around for improved compatibility without holding modules back.

One detail I don't quite understand is what to do about imports of code from GOPATH that is itself covered by other go.mod files. For example if we have

GOPATH/src/go.mod: module GOPATH, no mention of example.com/b
GOPATH/src/a/a.go: imports "example.com/b"

then it seems clear that the import should resolve to GOPATH/src/example.com/b/b.go.
But what if GOPATH/src/example.com/b/go.mod exists?
It seems like we just ignore it?

[bcmills]

@ohir, this proposal is not intended to address simultaneous editing of multiple modules. That will need to be addressed separately.

This proposal is specifically intended to address module migration for existing projects (large or small) that rely on packages that, for whatever reason, already cannot be fetched using go get, and therefore cannot be published or consumed as modules using go get.

That may be because they are hobby programs, or programs that generate additional code using make or bazel or similar third-party build tools, or just large private codebases that never needed to be accessible to go get.

[ohir]

@ohir, this proposal is not intended to address simultaneous editing of multiple modules. That will need to be addressed separately.

Ah, indeed. I humbly apologize for my off-topic comment above.

[bcmills]

what if GOPATH/src/example.com/b/go.mod exists?
It seems like we just ignore it?

Hmm, good question. Normally we would prune out the nested module, but if we're applying the legacy GOPATH-mode vendor behavior, then I think we would also need to ignore go.mod files found within the GOPATH module (and pull those packages into the GOPATH module itself).

I'm tempted to suggest that we wire in nested modules using replace directives instead, but I think that would make the vendor behavior too complicated.

[rsc]

Hmm, good question. Normally we would prune out the nested module, but if we're applying the legacy GOPATH-mode vendor behavior, then I think we would also need to ignore go.mod files found within the GOPATH module (and pull those packages into the GOPATH module itself).

I think I agree.

If GOPATH/src/go.mod says 'module GOPATH' and GOPATH/src/example.com/b/go.mod says 'module example.com/b' and I cd into 'example.com/b' and run a go command, then I assume it uses example.com/b/go.mod and not GOPATH/src/go.mod? That is, the command runs in the example.com/b module and not the GOPATH module?

[bcmills]

Yes, I think that's correct. If you want to run a command in the GOPATH module, you need to be outside of any other (nested) module.

Changes to the source code found in of any of those nested modules would be reflected in the GOPATH module itself, but changes to their module requirements would not be. That's maybe a bit confusing, but not really any worse than switching in and out of GOPATH mode today.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[complyue]

@robpike But I think only if the content in syntax & semantics as proposed here goes into some venue other than go.mod, e.g. go.project, it will become the desirable Go "project" designator.

@bcmills I was not sure I'm on the same page until I see #27542 shortly earlier, I understand the situation exactly as you stated it there.

I had been and still am missing Go ever since switched to Haskell a couple of years ago, Go tooling is always the most pragmatic one to my experience, (but I need more powerful abstraction tools there plus the M:N scheduler of GHC RTS as well as Go offers). But w.r.t. the "project" thing, I realize that both Haskell build tools, namely Cabal and Stack, got it right beyond modules and packages.

Situation wrt building tools are even messier and more confusing to outsiders in the Haskell ecosystem, but this stackoverflow answer may explain why "project" is important (as well as to Gophers): https://stackoverflow.com/a/49776281/6394508

The *.cabal file is the package-level configuration. ... This configuration provides essential information about the package: dependencies, exported components (libraries, executables, test suites), and settings for the build process (preprocessors, custom Setup.hs). ...

The stack.yaml file is the project-level configuration, which specifies a particular environment to make a build reproducible, pinning versions of compiler and dependencies. This is usually specified by a resolver (such as lts-11.4).

stack.yaml is not redundant with package.yaml. package.yaml specifies what dependencies are needed. stack.yaml indicates one way to consistently resolve dependencies (specific package version, and/or where to get it from, for example: on Hackage, a remote repository, or a local directory).

stack.yaml is most useful to developers: we don't want our build to suddenly break because a dependency upgraded while working on another project, hence we pin everything with stack.yaml. This file is less useful to users, who may have external constraints (typically, versions are already fixed by some distribution).

Update: I would suggest you replace "package" as heard from a Haskeller, with "import-root" in your Gopher's mind for clearer understanding. I realize that Java might have started use "package" this way even before Go, that's apparently not wrong. But when you talk about "package" and "module" to Haskellers, or Python guys, the terminology can be a source of confusion.

[complyue]

@meling I have nothing of objection except for the nesting of Go modules, I'd think that's a bad idea if feasible. What's the identity of a sub-module nested within a parent-module, if it can be separately published as well as part of its parent module? Or if it can only get published with its parent-module, you'd allow a separate version from its parent-module's version be specified as a dependency? If yes, it can be confusing for a dependent app to require multiple different versions of a parent module at the same time, for different parts of it; if no, then this sub-module thing seems to have no purpose of existence.

[ohir]

@robpike

[...] creating multiple interdependent modules without uploading them to github. [...] I do think there is something to be done here.

This — working with many interdependent modules — is specifically addressed by the #44347 proposal.

This proposal (treat GOPATH tree as a modules tree) is in my opinion better regarded as a migration tool that can help to move big internal code trees to the modules ecosystem. Both proposals are complementary, IMO.

[benitogf]

without introducing a separate file for this info

agree on this point, does #44347 expects .mod files?

would be nice to have a way of dealing with this error that is not turning off modules or adding a .mod file:

$ go build
go: cannot find main module, but found .git/config in /root/go/src/bitbucket.org/org/project 

[ohir]

@benitogf

does 44347 expects .mod files?

It does recognize go.mod files and uses them for resolving and using dependencies that are not present under GOTINKER tree. It need not to mandate go.mod files to be present though.

Note: Please keep 44347 related discussion there, not here.

[bcmills]

This proposal (treat GOPATH tree as a modules tree) is in my opinion better regarded as a migration tool that can help to move big internal code trees to the modules ecosystem. Both proposals are complementary, IMO.

I agree. This proposal (module GOPATH) is intended to provide a migration path for self-contained projects that do not currently use an import-path prefix as recommended. It is not — and is not intended to be — an adequate solution for the broader problem of editing interdependent and/or unpublished modules. It would provide only one “main module” in the workspace — module GOPATH — whereas when editing interdependent modules you still want to treat those modules as separate, distinguishable entities with their own distinct requirements.

To reiterate: the problem of editing interdependent modules is important, and we (particularly @matloob) are actively thinking about ways to address it, but I think that problem is off-topic for this particular proposal, which explicitly does not address it. #27542 is the right place for brainstorming about that more general problem.

[eliasnaur]

This proposal (treat GOPATH tree as a modules tree) is in my opinion better regarded as a migration tool that can help to move big internal code trees to the modules ecosystem. Both proposals are complementary, IMO.

I agree. This proposal (module GOPATH) is intended to provide a migration path for self-contained projects that do not currently use an import-path prefix as recommended. It is not — and is not intended to be — an adequate solution for the broader problem of editing interdependent and/or unpublished modules. It would provide only one “main module” in the workspace — module GOPATH — whereas when editing interdependent modules you still want to treat those modules as separate, distinguishable entities with their own distinct requirements.

To reiterate: the problem of editing interdependent modules is important, and we (particularly @matloob) are actively thinking about ways to address it, but I think that problem is off-topic for this particular proposal, which explicitly does not address it. #27542 is the right place for brainstorming about that more general problem.

Like Russ, I think the value of this proposal is questionable when considered just a migration path.

However, this proposal's use cases and multi-module use cases seem similar enough that I don't understand why you don't want this proposal to cover the multi-module cases as well. Or, start with the multi-module case and let it cover "GOPATH-mode" described in this issue as well.

I'm asking because a draft proposal for multi-modules involves introducing yet another configuration file with its own semantics - which is much less appealing to me than extending go.mod to deal with multiple simultaneous modules.

I don't care much for migrating prefix-less import paths, but I care a lot for the simplicity of this proposal.

[complyue]

Out of curious, why that draft proposal points to a non-existing https://golang.org/issue/NNNNN issue for purpose of discussion?

I actually want to vote for that proposal, since an independent layer for dependency resolution is not unusual wrt software engineering, and I would suggest Haskell ecosystem/tooling has a success showcase with this regards.

[bcmills]

@eliasnaur, I have read Michael's multi-module draft and I believe that this proposal and that draft are complementary, not alternatives to each other.

This proposal allows for a single module with a prefixless import path and a single set of dependency requirements, but allows that module to span multiple disjoint import paths. In contrast, that draft allows for combining and manipulating the requirement sets of multiple otherwise-independent modules, but does not allow any of those modules to individually span disjoint import paths.

[matloob]

@eliasnaur What would extending go.mod to supporting multiple simultaneous modules would look like?

@complyue I'm planning to make a formal proposal later this week, but I don't think that the draft is completely ready to start that process yet. If you have feedback though, I'd definitely appreciate it. Feel free to send me an email or comment on the golang-tools thread!

[eliasnaur]

@matloob good question, and I don't have a complete answer. I'll describe what I had hoped this proposal could do for me.

I have two modules I regularly work on, gioui.org/example and its primary dependency, gioui.org. To facilitate work that crosses the module boundary, I have a go.local.mod in my checkout of gioui.org/example that replaces the gioui.org module with a local checkout. I imagine that's a common setup, and one of the cases @matloob's go.work proposal aims to improve.

Now, inspired by this proposal, I propose something like be made possible:

$ mkdir workspace
$ cd workspace
$ git clone https://git.sr.ht/~eliasnaur/gio gioui.org/
$ git clone https://git.sr.ht/~eliasnaur/gio-example gioui.org/example
$ cat > go.mod
module WORKSPACE # or GOPATH, or ...
$ go run gioui.org/example/hello

The hello program will run, with the gioui.org module automatically replaced because it exists in the matching directory, gioui.org.

Comparing the go.work proposal, by placing modules in directories matching their names, the directory directive can be omitted. And without the directive, go.work looks pretty much like go.mod to me.

[matloob]

@eliasnaur I think we need to add a new type of file to support the multiple module workflow. The scope of a go.mod file is already defined to carve out other modules that exist inside it, which would make it difficult to add support for the file to apply to other modules contained within that module's directory tree. And I think changing those semantics for a module GOPATH would be confusing to users. Alternatively, a user can set the -modfile flag to point to a go.mod file for a module GOPATH or module WORKSPACE that has replaces for all the modules contained inside it, but passing the flag or adding it to GOFLAGS is clunky. If we tried to add some "sugar" for the clunkiness, we'd have to come up with some alternate way of finding a special go.mod file other than the normal go.mod resolution to make it clear to users that the scope of the file works differently than a standard module go.mod file... and that would mean a different file name at the least.

About your proposed workflow: I agree that there's an extra clunkiness to having to specify each module in the file, but I think there could be a solution to that in the framework of go.work files: For instance, we might have a convenience option to provide a pattern to go mod intiwork for example go mod initwork ..., that would create a go.work file referencing all modules contained in subdirectories of the current directory. Would that be a reasonable replacement to the creation of the co.mod in your example? (And that could be combined with a module GOPATH for code not already in modules)

But I'm already getting off topic on this proposal...

[eliasnaur]

@eliasnaur I think we need to add a new type of file to support the multiple module workflow. The scope of a go.mod file is already defined to carve out other modules that exist inside it, which would make it difficult to add support for the file to apply to other modules contained within that module's directory tree. And I think changing those semantics for a module GOPATH would be confusing to users. Alternatively, a user can set the -modfile flag to point to a go.mod file for a module GOPATH or module WORKSPACE that has replaces for all the modules contained inside it, but passing the flag or adding it to GOFLAGS is clunky. If we tried to add some "sugar" for the clunkiness, we'd have to come up with some alternate way of finding a special go.mod file other than the normal go.mod resolution to make it clear to users that the scope of the file works differently than a standard module go.mod file... and that would mean a different file name at the least.

I'm not sure I understand your point about scope. If I cd into, say, gioui.org/example the module WORKSPACE go.mod would no longer affect the go command; the go.mod file from module gioui.org/example would. In that sense, the WORKSPACE go.mod scope is not unusual.

About your proposed workflow: I agree that there's an extra clunkiness to having to specify each module in the file, but I think there could be a solution to that in the framework of go.work files: For instance, we might have a convenience option to provide a pattern to go mod intiwork for example go mod initwork ..., that would create a go.work file referencing all modules contained in subdirectories of the current directory. Would that be a reasonable replacement to the creation of the co.mod in your example? (And that could be combined with a module GOPATH for code not already in modules)

go mod initwork isn't a great replacement for automatic replacements. I imagine it typical to check out more modules after initializing your go.work, which would then have to be added to go.work.

[rsc]

I spoke to @bcmills, @jayconrod, and @matloob about this yesterday. I think this would have been a great idea four years ago, but today it probably carries too much complexity for too little benefit. It sounds like we should probably just leave things as is and not adopt this proposal. Will wait a week to see if anyone wants to make the argument to the contrary.

Note also the design draft (as yet not formally proposed) at http://golang.org/design/draft-workspace.

[benitogf]

leave things as is

does this mean keep GOPATH and GO111MODULE until we have a solution that deals with cannot find main module, but found .git/config in /root/go/src/bitbucket.org/org/project ?

#44347 (comment)

GO111MODULE=off" does not care whether the code in $GOPATH is from a git repository, or a zip-file - AFAIK it even completely ignores the VCS after go get`

We should consider that not every team and project needs or wants a package manager. We should have a way to opt out, actually we do already, just not removing it would suffice.

#45611 (comment)

Also, the "keep GOPATH for private repositories" attitude will better be served by #‍44649 proposal.

I don't understand why wanting to keep compatibility with a build system that works for many people is considered an attitude I miss golang the most while dealing with graddle, npm or makefiles issues. Lets reconsider keeping a way for people to simply build and run code without having to deal with the complexity inherent in package managers.

[ohir]

@benitogf I think that your comment above belongs mostly to #‍44347, not here.

wanting to keep compatibility with a build system that works for many people is considered an attitude [Related to #‍45611: "keep GOPATH for private repositories" attitude ]

"Attitude" because GOPATH based builds are deprecated except for remnants in the form of private/company internal code. This (#‍44649) proposal tries to keep it buildable as-is.

Note, that most of public (open source) code is now attuned to the Go modules versioning. It means also that authors now need not to adher to the "HEAD/tip is the release" principle — most still do, but the mental barrier against pushing experimental code at the tip is wanishing.

Lets reconsider keeping a way for people to simply build and run code without having to deal with the complexity inherent in package managers.

Thinking about this is in progress, as seen in both cited proposals. There is also @‍matloob's #45713 "Add a workspace mode" proposal — bringing to us yet another way to write and build Go software.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group