cmd/go: [modules + integration] go mod build, build all the binaries provided by a module

[nim-nim]

This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.

Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.

Needed feature

Go needs an official go mod build command that builds all the binaries provided by a particular Go module.

Constrains

• the input should be:

  • a module path, or
  • a module path + module version, or
  • the filesystem path of a specific mod module descriptor
    • either the go.mod file at the root of an unpacked module tree, or
    • a mod file inside a goproxy hierarchy

• all the usual go build flags should apply

• the destination should be any binary directory path the user specifies

• for compatibility with existing tooling, a separate optional prefix flag should allow pre-pending a path to the destination:

  • ie pretend working on destination, actually work on filepath.Join(prefix, destination)
  • that is typically used to prepare deployment to canonical_path, using a /prefix/canonical_path staging directory

• to allow integration with CI/CD software, the command should optionally output a list of the files it created, in machine-readable format, to a user-specified result file

  • the result file path is not affected by destination
  • the result file lists paths without prefix, since the command is pretending to write directly into destination
  • this is similar to the behaviour of go mod pack (cmd/go: [modules + integration] go mod pack, pack sources as module files #31302) and should use the same conventions

• the command should allow selecting (or not) the following binary sets:

  • target binaries (binaries the module was created for, intended for general use)
  • unit test helpers (ancillary binaries, created for the needs of unit tests, that only require the compiler)
  • integration test helpers (ancillary binaries, created for the needs of integration tests, that require more things than just the compiler to run see also cmd/go: [modules + integration] go mod buildrequires, list the build requirements of a set of unpacked modules #31300)
  • example binaries, that have no use in production or tests
    • though most of them won’t build as is, and projects should be incentivised to split them in separate example modules

• the command should work in secure no-internet-download mode. In that mode it should fail if one of the modules it needs for building is not available in the cache or configured goproxy (cmd/go: [modules + integration] use several goproxy sources simultaneously #31304) sources

  • it should not get stuck forever waiting on the CI/CD firewall to let go get download from the internet

Motivation

Go modules are wonderful and exciting, until you realise they only deal with the code download part. One stills need to dissect individual Go packages in the next stages of a CI/CD Go-related job.

The Go module concept should be extended to the rest of Go code processing phases.

[flaviostutz]

I got to really need this in Container related environments.

Normally, I use a previous STEP in Dockerfile in order to download all module dependencies and then I compile the program, so that Docker will cache the module download step and then only execute the program compilation. In newer Golang versions, some modules binary becomes "incompatible" so that Golang will have to compile them along with our program, so that build time becomes very slow.

If we have the before mentioned "go mod build", it would compile all dependencies objects as ".a" files and them only link it to our program at the later stage, making build time during development or CI/CD much faster.

See a sample Dockerfile I use in almost every Golang project of mine (it used to run in less than 5secs before "incompatibilities"):

FROM golang:1.14.3-alpine3.11 AS BUILD

RUN apk add build-base

ENV GO111MODULE 'on'

WORKDIR /app
ADD /go.mod /app
ADD /go.sum /app
RUN go mod download

#now build source code
ADD / /app
RUN go build -x -o /bin/userme-demo-api

[seankhliao]

i believe ./... covers this.