cmd/go: Add a command to zip the current project and push it to a hosted repository

[j-s-3]

Problem

With go.mod and GOPROXY the go tooling now has a good story for proxying open source packages but doesn't have a clear story for how best to manage and consume private packages. This is especially problematic for Repository Managers (such as Nexus Repository Manager) and makes it difficult for Go developers to share their private packages within their organizations.

Goal

Add a new command the Go CLI that creates a zip in a format that can be consumed via GOPROXY and push/POST that zip to a HTTP(s) endpoint.

This would allow Repository Managers to implement this endpoint and consume Go packages, making those packages available to Go developers via the download APIs. Go developers could then use this for sharing their private packages within their organizations.

[triztian]

I think this can be achieved this by this zipping up $GOPATH/pkg/mod/cache/download and then extracting it elsewhere, after extracting the zip archive you can use the GOPROXY env var with an absolute path pointing to the extracted location:

Something like:

zip -r gomods.zip $GOPATH/pkg/cache/download
curl -X POST -H "Content-Type: application/zip" --data @gomods.zip
https://example.org/gomods/download

And then elswhere:

GOPROXY=file:///path/to/extracted/contents go mod verify

Unless I'm missing something?

[j-s-3]

@triztian I’m referring to zipping your current project rather than its dependencies and then publishing that to a remote server such as Nexus Repository Manager. This would make for a better user experience for companies that are developing their own private libraries for use by other developers or teams.

[bcmills]

This is closely related to #28835.

[bcmills]

@triztian, there are some constraints on the file paths within the zip files: they must be slash-separated and have a “<module>@<version>/” prefix.

They also need to omit any directory subtrees containing a go.mod file.

[bcmills]

CC @jayconrod

[bcmills]

Perhaps we should start with a function in golang.org/x/mod or standalone binary in golang.org/x/tools (CC @ianthehat).

[triztian]

@jlstephens89 Got it that makes sense, then yes, it's definitively missing such feature specially with the limitations that @bcmills mentioned. After looking at the linked issue #28835 it seems that the same could be achieved with a combination of go mod edit and downloading the remote source from the repository manager (i.e. Nexus Repository Manager).

For example one could still zip up the private source and then download it as a zip file and extract it somewhere on disk.

After extracting one could use go mod edit to replace the imports to point to the version on disk, for example, suppose that the package is provider.com/some/library and that we are able to download and extract the source of such package to /path/to/extracted/sources in our dev machine, then one could do something like this (assuming that the sources have been zipped as-is and pushed to repsmanager.org):

wget https://repsmanager.org/provider.com/some/library.zip # downloads library.zip
mkdir -p /path/to/extracted/sources
unzip library.zip -d /path/to/extracted/sources

cd myproject # the root of the project that depends on provider.com/some/library package
go mod edit -replace=provider.com/some/library=/path/to/extracted/sources
go build -mod=readonly .

Doing this avoids having to pre-populate the $GOPATH/pkg/mod/cache/download with the deps of the project, on the publishers side all that she/he has to do is zip the sources and make them available for download. The Repository Managers would also have access to the sources because they arere plain zip files so as long as the zip file is pushed to the RM no special command is needed. Also by using replace statements one no longer has to deal with the setup of GOPROXY:

• Replace Docs: https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive

I just did a sample test of the go mod edit, before the edit:

λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

Performing the edit:

 λ ui ~> go mod edit -replace=golang.org/x/image=/opt/thirdparty/go/sources/x/image

After the edit:

 λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

replace golang.org/x/image => /opt/thirdparty/go/sources/x/image

Based on the replace docs I think one could script to replace all deps to point to the Repository Manager.

[j-s-3]

@triztian. GOPROXY and the download API make for a very natural user experience when working with a repository manager.

If we went the route of adding a command that created a zip with the correct file paths then any developer that has set GOPROXY to point to a “Group” repository, a repository type that aggregates public and private sources (or alternatively list the URLs comma separated), would have access to both internal and external dependencies with no additional effort. This would be a one time configuration for all developers in an organisation and from then on would allow centralised tooling teams to manage remotes/sources in a single place.

To publish a private dependency developers could do something like “go push http://repourl” and that dependency would then be available to an entire organisation.

[jayconrod]

At the moment, it would be hard to implement this in an external tool or in x/mod: most of the logic for this is in cmd/go/internal/modfetch and cmd/go/internal/modfetch/codehost. Those packages should eventually be moved to x/mod, but at the moment, they are tightly integrated with many other packages in the go command.

[bcmills]

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

[triztian]

So basically it'd be a tool that performs the following HTTP (or similar) requests for the given module path?:

POST $GOPROXY/<module>/@v/list 
<version>

Append the version of this module to the ist of all known versions of the given module, one per line.

PUT $GOPROXY/<module>/@v/<version>.info

Set JSON-formatted metadata about that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.mod 

Set the go.mod file for that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.zip

Set the zip archive for that version of the given module.

[j-s-3]

@triztian that would work and some ecosystems work that way. Alternatively it could just POST the zip and leave the rest of the work to the repository manager. The repo manager could extract the mod from the zip and could update .info and list based of the version and module name provided in the upload.

[bcmills]

CC @dmitshur

[thepudds]

I think the k8s team might have already put together something in terms of a utility that can create valid Go module .zip files. I have not looked at it carefully myself, but could be something to examine for anyone here that is interested.

kubernetes/publishing-bot#185

From https://github.com/kubernetes/publishing-bot/tree/master/cmd/gomod-zip/zip.go:

Creates a zip file at $GOPATH/pkg/mod/cache/download/<package-name>/@v/<pseudo-version>.zip.
The zip file has the same hash as if it were created by go mod download.
This tool can be used to package modules which haven't been uploaded anywhere
yet and are only available locally.
This tool assumes that the package is already checked out at the commit
pointed by the pseudo-version.

There is also https://pagure.io/modist/, which I think might include the ability to create a proper Go modules zip starting with files on disk, but I haven’t looked at that recently, so not 100% sure.

I don't believe either of those include pushing the result somewhere else, but one or both of them might solve the piece of the problem discussed here about how to create a canonical zip file with the same resulting hash as a zip file created by the go command itself.

CC @nikhita

[nikhita]

The k8s author here 👋 Thanks for surfacing that, @thepudds!

https://github.com/kubernetes/publishing-bot/blob/master/cmd/gomod-zip/zip.go is indeed the tool used to create .zip files with the same hash as that created by go itself. As a side note, we had some trouble using the zip command directly since that wasn't creating the correct zip files, more details here: https://github.com/kubernetes/publishing-bot/blob/b05cc3ce103eb1c4a70e3aa84ec7ac11a461c634/cmd/gomod-zip/zip.go#L115-L122.

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

The tool essentially duplicates the internal go code to achieve this. It would be really helpful if this logic were extracted out though.

I don't believe either of those include pushing the result somewhere else

We "fake" publish the repo to the local go mod cache. The code for it can be found here:

• go: https://github.com/kubernetes/publishing-bot/blob/b05cc3ce103eb1c4a70e3aa84ec7ac11a461c634/cmd/sync-tags/gomod.go#L126.
• bash: https://github.com/kubernetes/publishing-bot/blob/b05cc3ce103eb1c4a70e3aa84ec7ac11a461c634/artifacts/scripts/util.sh#L930-L937

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

[j-s-3]

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

@nikhita My purpose with this issue is for publishing to a Repository Manager such as Nexus Repository Manager rather than the module cache. The RM can then take care of generating/extracting the additional metadata files provided the information is available within the zip. For those purposes a simple zip and POST command would be the easiest route.

[edganiukov]

We are using jfrog artifactory as a GOPROXY and jfrog-cli to publish go modules into the jfrog repository. For example, we use it for generated swagger API clients - we don't store generated code in git, but in Go repository. Also, we are publishing releases of libraries. I found this very convenient and I think it makes sense to have a RW Go proxy (or Go repository) and be able to publish/download modules from it.

[gopherbot]

Change https://golang.org/cl/193557 mentions this issue: cmd/go: package modules into mod format for upload

[bcmills]

CL 193557 implements this as a go subcommand, but I don't see any fundamental reason that it needs to be in the go repo at all, given that the algorithm for extracting a repository is (intentionally) simple and stable. Most of the existing complexity in the go command is due to interfacing with VCS tools, but the tool proposed here operates directly on the filesystem.

Perhaps it would be a better fit as a standalone tool within x/tools (CC @ianthehat) or x/mod.

[gopherbot]

Change https://golang.org/cl/202042 mentions this issue: zip: add package for creating and extracting module zip files

[shaunco]

While the new zip tool creates a mechanism for packaging modules, it doesn't answer the original request about managing and consuming the module zips...

It would be amazing if a module's vanity URL could redirect to an https hosted zip file. Something like setting the VCS to zip:

<meta name="go-import" content="go.vanity.dev/package zip https://somesite.com/go.vanity.dev-{version}.zip">
<meta name="go-tags" content="go.vanity.dev/package https://somesite.com/go.vanity.dev-tags.json">

Where go.vanity.dev-tags.json would be something like:

["1.2.3", "1.2.4", "1.2.5"]

This would allow for quite a bit of flexibility in hosting module zips.

[jayconrod]

@shaunco That should already work. In Finding a repository for a module path:

The mod scheme instructs the go command to download the module from the given URL using the GOPROXY protocol. This allows developers to distribute modules without exposing source repositories.

So the server needs to return a tag like the one below, and the URL needs to point to a server that implements the proxy protocol and provides the module.

<meta name="go-import" content="go.vanity.dev/package mod https://somesite.com/go.vanity.dev">

This only works in module mode; GOPATH users will not be able to depend on a module like this.

[shaunco]

@jayconrod

point to a server that implements the proxy protocol

Good to know, but the ask in my was specifically to avoid needing a full blown GOPROXY server. Posting a zip file an manifest would be substantially easier.

[bcmills]

Note that x/mod/zip now exists and contains functions that can be used to construct a zip file. That should be fairly straightforward to stitch into a standalone executable if desired.

I don't think we need to add that to cmd/go proper, but feel free to file a separate proposal to make the case for it.