Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: Add a command to zip the current project and push it to a hosted repository #33312

Closed
j-s-3 opened this issue Jul 27, 2019 · 24 comments
Closed
Labels
FeatureRequest FrozenDueToAge modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@j-s-3
Copy link

j-s-3 commented Jul 27, 2019

Problem

With go.mod and GOPROXY the go tooling now has a good story for proxying open source packages but doesn't have a clear story for how best to manage and consume private packages. This is especially problematic for Repository Managers (such as Nexus Repository Manager) and makes it difficult for Go developers to share their private packages within their organizations.

Goal

Add a new command the Go CLI that creates a zip in a format that can be consumed via GOPROXY and push/POST that zip to a HTTP(s) endpoint.

This would allow Repository Managers to implement this endpoint and consume Go packages, making those packages available to Go developers via the download APIs. Go developers could then use this for sharing their private packages within their organizations.

@triztian
Copy link

triztian commented Jul 27, 2019

I think this can be achieved this by this zipping up $GOPATH/pkg/mod/cache/download and then extracting it elsewhere, after extracting the zip archive you can use the GOPROXY env var with an absolute path pointing to the extracted location:

Something like:

zip -r gomods.zip $GOPATH/pkg/cache/download
curl -X POST -H "Content-Type: application/zip" --data @gomods.zip
https://example.org/gomods/download

And then elswhere:

GOPROXY=file:///path/to/extracted/contents go mod verify

Unless I'm missing something?

@j-s-3
Copy link
Author

j-s-3 commented Jul 27, 2019

@triztian I’m referring to zipping your current project rather than its dependencies and then publishing that to a remote server such as Nexus Repository Manager. This would make for a better user experience for companies that are developing their own private libraries for use by other developers or teams.

@bcmills bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 28, 2019
@bcmills
Copy link
Contributor

bcmills commented Jul 28, 2019

This is closely related to #28835.

@bcmills
Copy link
Contributor

bcmills commented Jul 28, 2019

@triztian, there are some constraints on the file paths within the zip files: they must be slash-separated and have a “<module>@<version>/” prefix.

They also need to omit any directory subtrees containing a go.mod file.

@bcmills
Copy link
Contributor

bcmills commented Jul 28, 2019

CC @jayconrod

@bcmills bcmills added this to the Go1.14 milestone Jul 28, 2019
@bcmills
Copy link
Contributor

bcmills commented Jul 28, 2019

Perhaps we should start with a function in golang.org/x/mod or standalone binary in golang.org/x/tools (CC @ianthehat).

@triztian
Copy link

@jlstephens89 Got it that makes sense, then yes, it's definitively missing such feature specially with the limitations that @bcmills mentioned. After looking at the linked issue #28835 it seems that the same could be achieved with a combination of go mod edit and downloading the remote source from the repository manager (i.e. Nexus Repository Manager).

For example one could still zip up the private source and then download it as a zip file and extract it somewhere on disk.

After extracting one could use go mod edit to replace the imports to point to the version on disk, for example, suppose that the package is provider.com/some/library and that we are able to download and extract the source of such package to /path/to/extracted/sources in our dev machine, then one could do something like this (assuming that the sources have been zipped as-is and pushed to repsmanager.org):

wget https://repsmanager.org/provider.com/some/library.zip # downloads library.zip
mkdir -p /path/to/extracted/sources
unzip library.zip -d /path/to/extracted/sources

cd myproject # the root of the project that depends on provider.com/some/library package
go mod edit -replace=provider.com/some/library=/path/to/extracted/sources
go build -mod=readonly .

Doing this avoids having to pre-populate the $GOPATH/pkg/mod/cache/download with the deps of the project, on the publishers side all that she/he has to do is zip the sources and make them available for download. The Repository Managers would also have access to the sources because they arere plain zip files so as long as the zip file is pushed to the RM no special command is needed. Also by using replace statements one no longer has to deal with the setup of GOPROXY:

I just did a sample test of the go mod edit, before the edit:

λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

Performing the edit:

 λ ui ~> go mod edit -replace=golang.org/x/image=/opt/thirdparty/go/sources/x/image

After the edit:

 λ ui ~> cat go.mod
module gioui.org/ui

go 1.12

require (
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9
	golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
)

replace golang.org/x/image => /opt/thirdparty/go/sources/x/image

Based on the replace docs I think one could script to replace all deps to point to the Repository Manager.

@j-s-3
Copy link
Author

j-s-3 commented Jul 28, 2019

@triztian. GOPROXY and the download API make for a very natural user experience when working with a repository manager.

If we went the route of adding a command that created a zip with the correct file paths then any developer that has set GOPROXY to point to a “Group” repository, a repository type that aggregates public and private sources (or alternatively list the URLs comma separated), would have access to both internal and external dependencies with no additional effort. This would be a one time configuration for all developers in an organisation and from then on would allow centralised tooling teams to manage remotes/sources in a single place.

To publish a private dependency developers could do something like “go push http://repourl” and that dependency would then be available to an entire organisation.

@jayconrod
Copy link
Contributor

At the moment, it would be hard to implement this in an external tool or in x/mod: most of the logic for this is in cmd/go/internal/modfetch and cmd/go/internal/modfetch/codehost. Those packages should eventually be moved to x/mod, but at the moment, they are tightly integrated with many other packages in the go command.

@bcmills
Copy link
Contributor

bcmills commented Jul 29, 2019

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

@triztian
Copy link

triztian commented Jul 29, 2019

So basically it'd be a tool that performs the following HTTP (or similar) requests for the given module path?:

POST $GOPROXY/<module>/@v/list 
<version>

Append the version of this module to the ist of all known versions of the given module, one per line.

PUT $GOPROXY/<module>/@v/<version>.info

Set JSON-formatted metadata about that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.mod 

Set the go.mod file for that version of the given module.

PUT $GOPROXY/<module>/@v/<version>.zip

Set the zip archive for that version of the given module.

@j-s-3
Copy link
Author

j-s-3 commented Jul 29, 2019

@triztian that would work and some ecosystems work that way. Alternatively it could just POST the zip and leave the rest of the work to the repository manager. The repo manager could extract the mod from the zip and could update .info and list based of the version and module name provided in the upload.

@bcmills
Copy link
Contributor

bcmills commented Aug 1, 2019

CC @dmitshur

@thepudds
Copy link
Contributor

thepudds commented Aug 7, 2019

I think the k8s team might have already put together something in terms of a utility that can create valid Go module .zip files. I have not looked at it carefully myself, but could be something to examine for anyone here that is interested.

kubernetes/publishing-bot#185

From https://github.com/kubernetes/publishing-bot/tree/master/cmd/gomod-zip/zip.go:

Creates a zip file at $GOPATH/pkg/mod/cache/download/<package-name>/@v/<pseudo-version>.zip.
The zip file has the same hash as if it were created by go mod download.
This tool can be used to package modules which haven't been uploaded anywhere
yet and are only available locally.
This tool assumes that the package is already checked out at the commit
pointed by the pseudo-version.

There is also https://pagure.io/modist/, which I think might include the ability to create a proper Go modules zip starting with files on disk, but I haven’t looked at that recently, so not 100% sure.

I don't believe either of those include pushing the result somewhere else, but one or both of them might solve the piece of the problem discussed here about how to create a canonical zip file with the same resulting hash as a zip file created by the go command itself.

CC @nikhita

@nikhita
Copy link

nikhita commented Aug 8, 2019

The k8s author here 👋 Thanks for surfacing that, @thepudds!

https://github.com/kubernetes/publishing-bot/blob/master/cmd/gomod-zip/zip.go is indeed the tool used to create .zip files with the same hash as that created by go itself. As a side note, we had some trouble using the zip command directly since that wasn't creating the correct zip files, more details here: https://github.com/kubernetes/publishing-bot/blob/b05cc3ce103eb1c4a70e3aa84ec7ac11a461c634/cmd/gomod-zip/zip.go#L115-L122.

Actually, I think all of the logic for actually constructing the zip file is in one location, at the end of (*modfetch.codeRepo).Zip.

I think it would be pretty straightforward to extract that body out to a function.

The tool essentially duplicates the internal go code to achieve this. It would be really helpful if this logic were extracted out though.

I don't believe either of those include pushing the result somewhere else

We "fake" publish the repo to the local go mod cache. The code for it can be found here:

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

@j-s-3
Copy link
Author

j-s-3 commented Aug 8, 2019

While having a zip command can be helpful, there would still be some more steps needed to actually publish the repo (as mentioned in #33312 (comment)), so IMHO a command like #28835 would be much more helpful than a simple zip command.

@nikhita My purpose with this issue is for publishing to a Repository Manager such as Nexus Repository Manager rather than the module cache. The RM can then take care of generating/extracting the additional metadata files provided the information is available within the zip. For those purposes a simple zip and POST command would be the easiest route.

@edganiukov
Copy link

We are using jfrog artifactory as a GOPROXY and jfrog-cli to publish go modules into the jfrog repository. For example, we use it for generated swagger API clients - we don't store generated code in git, but in Go repository. Also, we are publishing releases of libraries. I found this very convenient and I think it makes sense to have a RW Go proxy (or Go repository) and be able to publish/download modules from it.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/193557 mentions this issue: cmd/go: package modules into mod format for upload

@bcmills
Copy link
Contributor

bcmills commented Sep 11, 2019

CL 193557 implements this as a go subcommand, but I don't see any fundamental reason that it needs to be in the go repo at all, given that the algorithm for extracting a repository is (intentionally) simple and stable. Most of the existing complexity in the go command is due to interfacing with VCS tools, but the tool proposed here operates directly on the filesystem.

Perhaps it would be a better fit as a standalone tool within x/tools (CC @ianthehat) or x/mod.

@bcmills bcmills added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 11, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/202042 mentions this issue: zip: add package for creating and extracting module zip files

gopherbot pushed a commit to golang/mod that referenced this issue Nov 1, 2019
zip provides three new functions:

* Create - build a zip from an abstract list of files, filtering out
  files in submodules and vendor directories. This is useful for
  filtering a zip produced by a VCS tool (as the go command does).
* CreateFromDir - build a zip from a directory. This is a convenience
  wrapper for Create.
* Unzip - extract a zip file, checking various restrictions.

A list of restrictions on module paths, versions, files within zips,
and size limits is included in the package documentation. Both Create
and Unzip enforce these restrictions.

Also: copied cmd/go/internal/txtar to internal/txtar for testing.

Updates golang/go#31302
Updates golang/go#33312
Updates golang/go#33778

Change-Id: I6fedb8b839a0cd991c9b210e73bafedc4b286ec5
Reviewed-on: https://go-review.googlesource.com/c/mod/+/202042
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@shaunco
Copy link

shaunco commented Nov 30, 2020

While the new zip tool creates a mechanism for packaging modules, it doesn't answer the original request about managing and consuming the module zips...

It would be amazing if a module's vanity URL could redirect to an https hosted zip file. Something like setting the VCS to zip:

<meta name="go-import" content="go.vanity.dev/package zip https://somesite.com/go.vanity.dev-{version}.zip">
<meta name="go-tags" content="go.vanity.dev/package https://somesite.com/go.vanity.dev-tags.json">

Where go.vanity.dev-tags.json would be something like:

["1.2.3", "1.2.4", "1.2.5"]

This would allow for quite a bit of flexibility in hosting module zips.

@jayconrod
Copy link
Contributor

@shaunco That should already work. In Finding a repository for a module path:

The mod scheme instructs the go command to download the module from the given URL using the GOPROXY protocol. This allows developers to distribute modules without exposing source repositories.

So the server needs to return a tag like the one below, and the URL needs to point to a server that implements the proxy protocol and provides the module.

<meta name="go-import" content="go.vanity.dev/package mod https://somesite.com/go.vanity.dev">

This only works in module mode; GOPATH users will not be able to depend on a module like this.

@shaunco
Copy link

shaunco commented Dec 10, 2020

@jayconrod

point to a server that implements the proxy protocol

Good to know, but the ask in my was specifically to avoid needing a full blown GOPROXY server. Posting a zip file an manifest would be substantially easier.

@bcmills
Copy link
Contributor

bcmills commented Jun 15, 2022

Note that x/mod/zip now exists and contains functions that can be used to construct a zip file. That should be fairly straightforward to stitch into a standalone executable if desired.

I don't think we need to add that to cmd/go proper, but feel free to file a separate proposal to make the case for it.

@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Jun 15, 2022
@golang golang locked and limited conversation to collaborators Jun 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FeatureRequest FrozenDueToAge modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants