Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: allow versioned go build commands #44469

Open
hyangah opened this issue Feb 20, 2021 · 33 comments
Open

proposal: cmd/go: allow versioned go build commands #44469

hyangah opened this issue Feb 20, 2021 · 33 comments

Comments

@hyangah
Copy link
Contributor

hyangah commented Feb 20, 2021

Update 2024-08-21 by matloob: The proposal has been updated to the following:

The proposal is to allow package@version arguments to go build.

They would behave similarly to how they do for go run: the module containing package at the given version would be loaded outside of the context of the current module. It would use cmd/go/internal/load.PackagsAndErrorsOutsideModule, so it would load the given module as if it were a requirement of an otherwise empty module. This also enforces that the version of the module the package belongs to doesn't have replace or exclude directives.

Similar to go run, if mulitple packages are supplied, they would be required to belong to the same module at the same version. (Though providing multiple package@version arguments wouldn't be useful in practice because the main use case would be to produce the build outputs.)


Alternative: allow to change the installed main package name in go install command

Since go1.16, go install accepts arguments with version suffixes. go install places the output in $GOPATH/bin or $GOBIN, but does not allow the output binary's name change.

When a user wants to use multiple versions of tools, the user need to be careful not to overwrite other versions.

Different versions may be coming from different module paths.
e.g. VS Code Go extension uses two different versions of gocode (stamblerre/gocode and mdempsky/gocode). In order to install the former, it does switching GOBIN to use a temporary directory, installing it, and renaming and copying it to the real GOBIN.

Alternative: There are other proposals for applying the same to go run (#42088) and go list(#44203). Extending go build in the same way (or extending go install) will help simplifying the tool installation workflow.

cc @jayconrod @bcmills @matloob

@seankhliao seankhliao added GoCommand cmd/go modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 21, 2021
@mvdan
Copy link
Member

mvdan commented Feb 21, 2021

On #44203, @jayconrod said:

Though I think we should talk about allowing @version on all commands that take packages (where it makes sense; not go generate).

If the plan is indeed to support versioned variants of multiple commands, perhaps we should use a single issue for all of them at once instead of one per command.

@bcmills
Copy link
Contributor

bcmills commented Feb 22, 2021

Alternative: allow to change the installed main package name in go install command

I think we should definitely do that, regardless of whether we add @version support for go build. go install ought to support the same -o flag as go build and go test -c.

Can we make this issue about that, and discuss generalizing @version support in a separate issue?

@hyangah
Copy link
Contributor Author

hyangah commented Feb 23, 2021

Assuming the scope of #44203 is not limited to go list but all applicable commands including go build, I am fine with repurposing this issue to discuss the possibility of adding -o to go install.

What will be the expected behavior of -o? For my use case, I hope to see

go install -o gopls-master golang.org/x/tools/gopls@master

which installs the output in $GOBIN/gopls-master (or $GOBIN\gopls-master.exe).
That can be achievable with go build -o "$(go env GOBIN)/gopls-master" golang.org/x/tools/gopls@master in some platforms if go build has version support, but still not as convenient as the above command.

@bcmills
Copy link
Contributor

bcmills commented Feb 23, 2021

I would expect -o to mean exactly the same thing for go install that it means for go build. So you would still need the command to be something like:

$ go install -o $(go env GOBIN)/gopls-master golang.org/x/tools/gopls@master

Although honestly, in my own use I would probably spell that as:

$ go install -o ~/bin/gopls-master golang.org/x/tools/gopls@master

@mvdan
Copy link
Member

mvdan commented Feb 23, 2021

go install -o $(go env GOBIN)/gopls-master

The downside of that is that it depends on POSIX Shell syntax, so it wouldn't be portable to e.g. Windows. I see little benefit of go install having a -o flag if one can use go build in exactly the same way, to be honest.

I do see the point in flags being consistent between commands, but the whole point of go install is to put the binary in GOBIN, hence the flag could be more of a "rename the binary when installing it". That could always be a separate flag, though.

@bcmills
Copy link
Contributor

bcmills commented Feb 23, 2021

If there were such a flag, I would not want to name it -o. GOFLAGS=-o=a.out should mean the same thing whether it is applying to go build or go install.

@mvdan
Copy link
Member

mvdan commented Feb 23, 2021

Maybe. The flip side of the coin is that go install -o would make little sense, since go build -o would be much clearer :)

@hyangah
Copy link
Contributor Author

hyangah commented Feb 23, 2021

go build -o foo leaves the output foo in the current directory if foo doesn't include the directory path.
Is it too different if go install -o foo installs the main package as foo in the default install directory?

@bcmills
Copy link
Contributor

bcmills commented Feb 24, 2021

Is it too different if go install -o foo installs the main package as foo in the default install directory?

IMO yes, but I'd like to see what others think. 😅

@jayconrod
Copy link
Contributor

I think if go install gets an -o flag, it would be confusing if it didn't do the same thing as go build. If it only changes the file name, it should be called something else. I think it would be better to just have -o though.

Agree that it's redundant with go build -o if go build gets an @version form though (which I think is a good idea).

@hyangah
Copy link
Contributor Author

hyangah commented Feb 24, 2021

Currently we are using the sequence of

cd $(mktemp -d)
go mod init something
go get -d <tool>
go build -o <destination> <tool>
and cleanup... 

go build -o <destination> tool@version will definitely help us, but still computing <destination> for all platforms needs a bit of work. (checking GOBIN, GOPATH, and adding the right file extension, ...)

@jayconrod
Copy link
Contributor

If it helps, <destination> can be a directory. The executable will be named after the package in that case, but it will have the .exe on Windows at least.

When you say "for all platforms", are you cross-compiling? If GOOS or GOARCH are set to something other than the default, go install puts the result in ${GOBIN}/${GOOS}_${GOARCH}/ instead of ${GOBIN}.

@hyangah
Copy link
Contributor Author

hyangah commented Feb 24, 2021

In our specific case, we don't do cross compiling which simplifies things.
We will need to rename the executable (e.g. gocode -> gocode-gomod, dlv -> dlv-dap, ...).
Currently we are using various hacks
https://github.com/golang/vscode-go/blob/master/build/all.bash#L108-L111 and so on
and also in our js/ts code (where I just found another bug in computing the directory :-) .

To me -o semantic isn't too different from go build's -o.

  • if it's the full path, leave the binary there
  • if it's a binary name (without directory), leave the binary with the name in the default directory. For go build, the default directory is the current directory where the command runs. For go install, the default directory is the default installation directory determined by GOBIN, GOPATH (and GOOS and GOARCH if you cross compile).

@seankhliao
Copy link
Member

Do I have this right? Though I'm not sure when you'd want go build pkg@version

cmd code output
go build pkg current module local directory
go build -o out pkg current module explicit path
go build pkg@version remote code local directory
go build -o out pkg@version remote code explicit path
go install pkg current module global directory
go install -o out pkg current module explicit path
go install pkg@version remote code global directory
go install -o out pkg@version remote code explicit path

@meling
Copy link

meling commented Mar 1, 2021

IMO, the go install -o out pkg should install in the default location as suggested by @hyangah above, if out is just a file name. Otherwise, the build and install cases become the same... And there isn’t a convenient way to install a differently named binary in the default location.

@bcmills
Copy link
Contributor

bcmills commented Mar 1, 2021

@meling

Otherwise, the build and install cases become the same... And there isn’t a convenient way to install a differently named binary in the default location.

I would rather we have consistent meanings for flags than make every subcommand completely orthogonal.

It's not obvious to me that “installing a differently named binary in the default location” really needs to be convenient.

@hyangah
Copy link
Contributor Author

hyangah commented Mar 1, 2021

Thanks @seankhliao.
go install -o out pkg@version and go build -o out pkg@version are indeed same except that - go install uses the global directory as the default directory while go build uses the local directory as the default directory. If out is the full path, yes, they are identical.

Is there any command that can provide the default installation directory? If so, I am happy to use it to formulate the right go build command. (Furthermore, if there is a command that returns the default installation directory, maybe one would argue to deprecate go install)

@bcmills
Copy link
Contributor

bcmills commented Mar 1, 2021

Is there any command that can provide the default installation directory?

Within a module, something like go get -d golang.org/x/tools/gopls && go list -f {{.Target}} golang.org/x/tools/gopls would do the trick.

If we accept #44203, that would shorten to go list -f {{.Target}} golang.org/x/tools/gopls@latest.

svengreb added a commit to svengreb/wand that referenced this issue Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official
deprecation [3] of `gobin` [4] in favor of the new Go 1.16
`go install pkg@version` [5] syntax feature should have been handled for
this project. The idea was to replace the `gobin` task runner [6] with a
one that leverages "bingo" [7], a project similar to `gobin`, that comes
with many great features and also allows to manage development tools on
a per-module basis. The problem is that `bingo` uses some non-default
and nontransparent mechanisms under the hood and automatically generates
files in the repository without the option to disable this behavior.
It does not make use of the `go install` command but relies on custom
dependency resolution mechanisms, making it prone to future changes in
the Go toolchain and therefore not a good choice for the maintainability
of projects.

>>> `go install` is still not perfect

Support for the new `go install` features, which allow to install
commands without affecting the `main` module, have already been added in
GH-71 [8] as an alternative to `gobin`, but one significant problem was
still not addressed: install module/package executables globally without
overriding already installed executables of different versions.
Since `go install` will always place compiled binaries in the path
defined by `go env GOBIN`, any already existing executable with the same
name will be replaced. It is not possible to install a module command
with two different versions since `go install` still messes up the local
user environment.

>>> The Workaround: Hybrid `go install` task runner

This commit therefore implements the solution through a custom
`Runner` [9] that uses `go install` under the hood, but places the
compiled executable in a custom cache directory instead of
`go env GOBIN`. The runner checks if the executable already exists,
installs it if not so, and executes it afterwards.

The concept of storing dependencies locally on a per-project basis is
well-known from the `node_modules` directory [10] of the "Node" [11]
package manager "npm" [12]. Storing executables in a cache directory
within the repository (not tracked by Git) allows to use `go install`
mechanisms while not affect the global user environment and executables
stored in `go env GOBIN`. The runner achieves this by changing the
`GOBIN` environment variable to the custom cache directory during the
execution of `go install`. This way it bypasses the need for
"dirty hacks" while using a custom output path.

The only known disadvantage is the increased usage of storage disk
space, but since most Go executables are small in size anyway, this is
perfectly acceptable compared to the clearly outweighing advantages.

Note that the runner dynamically runs executables based on the given
task so `Validate() error` is a NOOP.

>>> Upcoming Changes

The solution described above works totally fine, but is still not a
clean solution that uses the Go toolchain without any special logic so
as soon as the following changes are made to the
Go toolchain (Go 1.17 or later), the custom runner will be removed
again:

- golang/go/issues#42088 [13] — tracks the process of adding support for
  the Go module syntax to the `go run` command. This will allow to let
  the Go toolchain handle the way how compiled executable are stored,
  located and executed.
- golang/go#44469 [14] — tracks the process of making `go install`
  aware of the `-o` flag like the `go build` command which is the only
  reason why the custom runner has been implemented.

>>> Further Adjustments

Because the new custom task runner dynamically runs executables based on
the given task, the `Bootstrap` method [15] of the `Wand` [16] reference
implementation `Elder` [17] additionally allows to pass Go module import
paths, optionally including a version suffix (`pkg@version`), to install
executables from Go module-based `main` packages into the local cache
directory. This way the local development environment can be set up,
for e.g. by running it as startup task [18] in "JetBrains" IDEs.
The method also ensures that the local cache directory exists and
creates a `.gitignore` file that includes ignore pattern for the cache
directory.

[1]: #89
[2]: #78
[3]: myitcv/gobin#103
[4]: https://github.com/myitcv/gobin
[5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner
[7]: https://github.com/bwplotka/bingo
[8]: #71
[9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner
[10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules
[11]: https://nodejs.org
[12]: https://www.npmjs.com
[13]: golang/go#42088
[14]: golang/go#44469 (comment)
[15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap
[16]: https://pkg.go.dev/github.com/svengreb/wand#Wand
[17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder
[18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html

GH-89
svengreb added a commit to svengreb/wand that referenced this issue Apr 26, 2021
GH-89 [1] supersedes GH-78 [2] which documents how the official
deprecation [3] of `gobin` [4] in favor of the new Go 1.16
`go install pkg@version` [5] syntax feature should have been handled for
this project. The idea was to replace the `gobin` task runner [6] with a
one that leverages "bingo" [7], a project similar to `gobin`, that comes
with many great features and also allows to manage development tools on
a per-module basis. The problem is that `bingo` uses some non-default
and nontransparent mechanisms under the hood and automatically generates
files in the repository without the option to disable this behavior.
It does not make use of the `go install` command but relies on custom
dependency resolution mechanisms, making it prone to future changes in
the Go toolchain and therefore not a good choice for the maintainability
of projects.

>>> `go install` is still not perfect

Support for the new `go install` features, which allow to install
commands without affecting the `main` module, have already been added in
GH-71 [8] as an alternative to `gobin`, but one significant problem was
still not addressed: install module/package executables globally without
overriding already installed executables of different versions.
Since `go install` will always place compiled binaries in the path
defined by `go env GOBIN`, any already existing executable with the same
name will be replaced. It is not possible to install a module command
with two different versions since `go install` still messes up the local
user environment.

>>> The Workaround: Hybrid `go install` task runner

This commit therefore implements the solution through a custom
`Runner` [9] that uses `go install` under the hood, but places the
compiled executable in a custom cache directory instead of
`go env GOBIN`. The runner checks if the executable already exists,
installs it if not so, and executes it afterwards.

The concept of storing dependencies locally on a per-project basis is
well-known from the `node_modules` directory [10] of the "Node" [11]
package manager "npm" [12]. Storing executables in a cache directory
within the repository (not tracked by Git) allows to use `go install`
mechanisms while not affect the global user environment and executables
stored in `go env GOBIN`. The runner achieves this by changing the
`GOBIN` environment variable to the custom cache directory during the
execution of `go install`. This way it bypasses the need for
"dirty hacks" while using a custom output path.

The only known disadvantage is the increased usage of storage disk
space, but since most Go executables are small in size anyway, this is
perfectly acceptable compared to the clearly outweighing advantages.

Note that the runner dynamically runs executables based on the given
task so `Validate() error` is a NOOP.

>>> Upcoming Changes

The solution described above works totally fine, but is still not a
clean solution that uses the Go toolchain without any special logic so
as soon as the following changes are made to the
Go toolchain (Go 1.17 or later), the custom runner will be removed
again:

- golang/go/issues#42088 [13] — tracks the process of adding support for
  the Go module syntax to the `go run` command. This will allow to let
  the Go toolchain handle the way how compiled executable are stored,
  located and executed.
- golang/go#44469 [14] — tracks the process of making `go install`
  aware of the `-o` flag like the `go build` command which is the only
  reason why the custom runner has been implemented.

>>> Further Adjustments

Because the new custom task runner dynamically runs executables based on
the given task, the `Bootstrap` method [15] of the `Wand` [16] reference
implementation `Elder` [17] additionally allows to pass Go module import
paths, optionally including a version suffix (`pkg@version`), to install
executables from Go module-based `main` packages into the local cache
directory. This way the local development environment can be set up,
for e.g. by running it as startup task [18] in "JetBrains" IDEs.
The method also ensures that the local cache directory exists and
creates a `.gitignore` file that includes ignore pattern for the cache
directory.

[1]: #89
[2]: #78
[3]: myitcv/gobin#103
[4]: https://github.com/myitcv/gobin
[5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner
[7]: https://github.com/bwplotka/bingo
[8]: #71
[9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner
[10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules
[11]: https://nodejs.org
[12]: https://www.npmjs.com
[13]: golang/go#42088
[14]: golang/go#44469 (comment)
[15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap
[16]: https://pkg.go.dev/github.com/svengreb/wand#Wand
[17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder
[18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html

GH-89
svengreb added a commit to svengreb/wand that referenced this issue Apr 26, 2021
…#90)

GH-89 [1] supersedes GH-78 [2] which documents how the official
deprecation [3] of `gobin` [4] in favor of the new Go 1.16
`go install pkg@version` [5] syntax feature should have been handled for
this project. The idea was to replace the `gobin` task runner [6] with a
one that leverages "bingo" [7], a project similar to `gobin`, that comes
with many great features and also allows to manage development tools on
a per-module basis. The problem is that `bingo` uses some non-default
and nontransparent mechanisms under the hood and automatically generates
files in the repository without the option to disable this behavior.
It does not make use of the `go install` command but relies on custom
dependency resolution mechanisms, making it prone to future changes in
the Go toolchain and therefore not a good choice for the maintainability
of projects.

>>> `go install` is still not perfect

Support for the new `go install` features, which allow to install
commands without affecting the `main` module, have already been added in
GH-71 [8] as an alternative to `gobin`, but one significant problem was
still not addressed: install module/package executables globally without
overriding already installed executables of different versions.
Since `go install` will always place compiled binaries in the path
defined by `go env GOBIN`, any already existing executable with the same
name will be replaced. It is not possible to install a module command
with two different versions since `go install` still messes up the local
user environment.

>>> The Workaround: Hybrid `go install` task runner

This commit therefore implements the solution through a custom
`Runner` [9] that uses `go install` under the hood, but places the
compiled executable in a custom cache directory instead of
`go env GOBIN`. The runner checks if the executable already exists,
installs it if not so, and executes it afterwards.

The concept of storing dependencies locally on a per-project basis is
well-known from the `node_modules` directory [10] of the "Node" [11]
package manager "npm" [12]. Storing executables in a cache directory
within the repository (not tracked by Git) allows to use `go install`
mechanisms while not affect the global user environment and executables
stored in `go env GOBIN`. The runner achieves this by changing the
`GOBIN` environment variable to the custom cache directory during the
execution of `go install`. This way it bypasses the need for
"dirty hacks" while using a custom output path.

The only known disadvantage is the increased usage of storage disk
space, but since most Go executables are small in size anyway, this is
perfectly acceptable compared to the clearly outweighing advantages.

Note that the runner dynamically runs executables based on the given
task so `Validate() error` is a NOOP.

>>> Upcoming Changes

The solution described above works totally fine, but is still not a
clean solution that uses the Go toolchain without any special logic so
as soon as the following changes are made to the
Go toolchain (Go 1.17 or later), the custom runner will be removed
again:

- golang/go/issues#42088 [13] — tracks the process of adding support for
  the Go module syntax to the `go run` command. This will allow to let
  the Go toolchain handle the way how compiled executable are stored,
  located and executed.
- golang/go#44469 [14] — tracks the process of making `go install`
  aware of the `-o` flag like the `go build` command which is the only
  reason why the custom runner has been implemented.

>>> Further Adjustments

Because the new custom task runner dynamically runs executables based on
the given task, the `Bootstrap` method [15] of the `Wand` [16] reference
implementation `Elder` [17] additionally allows to pass Go module import
paths, optionally including a version suffix (`pkg@version`), to install
executables from Go module-based `main` packages into the local cache
directory. This way the local development environment can be set up,
for e.g. by running it as startup task [18] in "JetBrains" IDEs.
The method also ensures that the local cache directory exists and
creates a `.gitignore` file that includes ignore pattern for the cache
directory.

[1]: #89
[2]: #78
[3]: myitcv/gobin#103
[4]: https://github.com/myitcv/gobin
[5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner
[7]: https://github.com/bwplotka/bingo
[8]: #71
[9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner
[10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules
[11]: https://nodejs.org
[12]: https://www.npmjs.com
[13]: golang/go#42088
[14]: golang/go#44469 (comment)
[15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap
[16]: https://pkg.go.dev/github.com/svengreb/wand#Wand
[17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder
[18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html

Closes GH-89
svengreb added a commit to svengreb/wand that referenced this issue Apr 26, 2021
…#90)

GH-89 [1] supersedes GH-78 [2] which documents how the official
deprecation [3] of `gobin` [4] in favor of the new Go 1.16
`go install pkg@version` [5] syntax feature should have been handled for
this project. The idea was to replace the `gobin` task runner [6] with a
one that leverages "bingo" [7], a project similar to `gobin`, that comes
with many great features and also allows to manage development tools on
a per-module basis. The problem is that `bingo` uses some non-default
and nontransparent mechanisms under the hood and automatically generates
files in the repository without the option to disable this behavior.
It does not make use of the `go install` command but relies on custom
dependency resolution mechanisms, making it prone to future changes in
the Go toolchain and therefore not a good choice for the maintainability
of projects.

>>> `go install` is still not perfect

Support for the new `go install` features, which allow to install
commands without affecting the `main` module, have already been added in
GH-71 [8] as an alternative to `gobin`, but one significant problem was
still not addressed: install module/package executables globally without
overriding already installed executables of different versions.
Since `go install` will always place compiled binaries in the path
defined by `go env GOBIN`, any already existing executable with the same
name will be replaced. It is not possible to install a module command
with two different versions since `go install` still messes up the local
user environment.

>>> The Workaround: Hybrid `go install` task runner

This commit therefore implements the solution through a custom
`Runner` [9] that uses `go install` under the hood, but places the
compiled executable in a custom cache directory instead of
`go env GOBIN`. The runner checks if the executable already exists,
installs it if not so, and executes it afterwards.

The concept of storing dependencies locally on a per-project basis is
well-known from the `node_modules` directory [10] of the "Node" [11]
package manager "npm" [12]. Storing executables in a cache directory
within the repository (not tracked by Git) allows to use `go install`
mechanisms while not affect the global user environment and executables
stored in `go env GOBIN`. The runner achieves this by changing the
`GOBIN` environment variable to the custom cache directory during the
execution of `go install`. This way it bypasses the need for
"dirty hacks" while using a custom output path.

The only known disadvantage is the increased usage of storage disk
space, but since most Go executables are small in size anyway, this is
perfectly acceptable compared to the clearly outweighing advantages.

Note that the runner dynamically runs executables based on the given
task so `Validate() error` is a NOOP.

>>> Upcoming Changes

The solution described above works totally fine, but is still not a
clean solution that uses the Go toolchain without any special logic so
as soon as the following changes are made to the
Go toolchain (Go 1.17 or later), the custom runner will be removed
again:

- golang/go/issues#42088 [13] — tracks the process of adding support for
  the Go module syntax to the `go run` command. This will allow to let
  the Go toolchain handle the way how compiled executable are stored,
  located and executed.
- golang/go#44469 [14] — tracks the process of making `go install`
  aware of the `-o` flag like the `go build` command which is the only
  reason why the custom runner has been implemented.

>>> Further Adjustments

Because the new custom task runner dynamically runs executables based on
the given task, the `Bootstrap` method [15] of the `Wand` [16] reference
implementation `Elder` [17] additionally allows to pass Go module import
paths, optionally including a version suffix (`pkg@version`), to install
executables from Go module-based `main` packages into the local cache
directory. This way the local development environment can be set up,
for e.g. by running it as startup task [18] in "JetBrains" IDEs.
The method also ensures that the local cache directory exists and
creates a `.gitignore` file that includes ignore pattern for the cache
directory.

[1]: #89
[2]: #78
[3]: myitcv/gobin#103
[4]: https://github.com/myitcv/gobin
[5]: https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies
[6]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/task/gobin#Runner
[7]: https://github.com/bwplotka/bingo
[8]: #71
[9]: https://pkg.go.dev/github.com/svengreb/wand/pkg/task#Runner
[10]: https://docs.npmjs.com/cli/v7/configuring-npm/folders#node-modules
[11]: https://nodejs.org
[12]: https://www.npmjs.com
[13]: golang/go#42088
[14]: golang/go#44469 (comment)
[15]: https://pkg.go.dev/github.com/svengreb/wand@v0.5.0/pkg/elder#Elder.Bootstrap
[16]: https://pkg.go.dev/github.com/svengreb/wand#Wand
[17]: https://pkg.go.dev/github.com/svengreb/wand/pkg/elder#Elder
[18]: https://www.jetbrains.com/help/idea/settings-tools-startup-tasks.html

Closes GH-89
gopherbot pushed a commit to golang/vscode-go that referenced this issue Oct 19, 2021
`go install` with the version specifier is THE way to install a binary
in recent versions of Go. From go1.18, `go get` will not do build & install
so shouldn't be used.

For `dlv-dap` or `gocode-gomod`, we use `go get` + `go build` because
`go install` does not allow to specify output path.
(golang/go#44469)

Use of `go install` allows us to address another issue: we no longer
require a dummy main module in a temp directory, and we can run it from
the workspace directory. The use of temp directory broke direnv, asdf
users who configured their go to pick a different version of go based on
the directory. Thus, we pinned the go binary by selecting the go binary
from the current GOROOT's bin directory. Unfortunately, that broke
another group of users who are using GOROOT for different purpose.
(#1691)
Now we can remove the hack to pin the go binary and use the go binary
path as it is, and simplifies the installation logic.

Reduced the verbose logging printed in the console.
Instead, we log the command and the directory path so users could
reproduce. In case of errors, the exception still includes all the stdout
and stderr.

Updates #1825

Change-Id: Idb1604e4484cd73832c24c0077220f8cc57dfaa0
Reviewed-on: https://go-review.googlesource.com/c/vscode-go/+/355974
Trust: Hyang-Ah Hana Kim <hyangah@gmail.com>
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Suzy Mueller <suzmue@golang.org>
@rlanhellas
Copy link
Contributor

We're not even sure if we want this feature or how it would work, see the previous discussion and the NeedsInvestigation label. This is also not a particularly good issue to do your first contribution on, as it's somewhat complex. I'd suggest looking at issues with the labels help wanted or NeedsFix, which are often better candidates.

ok @mvdan , thanks for the feedback.

@ldemailly
Copy link

ldemailly commented Feb 22, 2023

Repeating here the problem I raised earlier in the hope to get a solution or traction:

In order to take advantage of the new in 1.18 fantastic BuildInfo it's ideal (imo) to replace the before classic best practice:
go build with -ldflags -X main.version=1.2.3 by go install pkg@v1.2.3

But go install doesn't simply let you specify where to install the binary. It puts it in GOPATH/bin when on the native architecture and GOPATH/bin/$arch/ when not; which makes it hacky and hard to just produce the binaries in a multi arch CIs

Example: before:

CGO_ENABLED=0 GOOS=windows go build -a -ldflags \
'-s -X fortio.org/fortio/version.version=1.28.0 -X "fortio.org/fortio/version.buildInfo=2022-04-26 00:40 57e9d8f01c342a4c1b5d96f883f81128b04d991a"' \
 -o /tmp/fortio.exe fortio.org/fortio

after/now

GOPATH=/build CGO_ENABLED=0 GOOS=darwin /usr/local/go/bin/go install -a -ldflags -s fortio.org/fortio@v1.29.0-pre16
mv -f /build/bin/*_*/fortio* /build/bin
rmdir /build/bin/*_*

I end up having to use GOPATH to try to set where the binary ends up as well as using some hacks to deal with native vs non native arch

@ahyangyi
Copy link

ahyangyi commented Jun 26, 2023

@meling

Otherwise, the build and install cases become the same... And there isn’t a convenient way to install a differently named binary in the default location.

I would rather we have consistent meanings for flags than make every subcommand completely orthogonal.

It's not obvious to me that “installing a differently named binary in the default location” really needs to be convenient.

I'm working on a non-golang project that depends on a handful of golang binaries. Which unfortunately all have the name cmd. Please enlighten me why I don't need a go install -o <foo> command.

Since my project isn't a go module, I can't even use go build repo@version by definition.

And if your concern is consistency of options, then what about making it a new option other than -o? like -command-name or something: when we want to change the name when running go install, we are not specifying a new file name (e.g. foo.exe under Windows), we are specifying a new command name (e.g., foo). That's self-explanatory enough and clear enough, right?

@bcap
Copy link

bcap commented Jul 29, 2023

TL;DR: Usecase showing how name clashing during go install will cause silent overwrites of previous executables, which will lead to terrible user experience (a tool now is another binary that does a different thing). That coupled with the lack of ability to change the tool name will also lead to some awkward hacks


To leave a practical example of how cumbersome the user experience can be with go install today:

Usecase: Run the tool defined at github.com/Jille/raft-grpc-example/cmd/hammer plus some other raft-related tooling

I can go install github.com/Jille/raft-grpc-example/cmd/hammer, but then what happens is that will overwrite another hammer tool that I already have on my ~/go/bin. It is name clashing and the overwrite happens silenty, which is really bad (this very likely deserves an issue of its own)

Anyway, ideally I would want to build it as "raft-hammer"

So this was my user experience:

Lets try installing it with a different name. How do I do so?

% go help install

usage: go install [build flags] [packages]

Install compiles and installs the packages named by the import paths.

...

Ok, so it takes build flags. Lets try:

% go install -o raft-hammer github.com/Jille/raft-grpc-example/cmd/hammer@latest
flag provided but not defined: -o
usage: go install [build flags] [packages]
Run 'go help install' for details.

Well, thats weird, takes build flags but not this one build flag. Lets google it. I ended up finding this GH Issue

Fair enough, seems like this discussion went a lot around go install becoming the same as go build and how bad would it be to have more flags on go install, which IMO doesnt hold as a strong argument. It seems like we cant add -o to go install to avoid commands having specific flags, but -o is already special anyway because it is not passed to go build by go install. So -o is basically a flag sitting on some kind of a limbo.

Anyway, what if we try to simple go build then?

go build -o raft-hammer github.com/Jille/raft-grpc-example/cmd/hammer@latest
package github.com/Jille/raft-grpc-example/cmd/hammer@latest: can only use path@version syntax with 'go get' and 'go install' in module-aware mode

Ok, so go build is not version aware. Not sure why but that is confusing.

Well I guess we need to go for go install + minor hacking:

% mv ~/go/bin/{hammer,hammer-swap}
% go install github.com/Jille/raft-grpc-example/cmd/hammer@latest
% mv ~/go/bin/{hammer,raft-hammer}
% mv ~/go/bin/{hammer-swap,hammer}

Now I have my tool without losing the previous one

@oliverpool
Copy link

@bcap I had the same issue, but used the workaround of @hyangah: add a go.mod file.

You can then use go get and go build:

go get github.com/Jille/raft-grpc-example/cmd/hammer@latest
go build -o ~/go/bin/raft-hammer github.com/Jille/raft-grpc-example/cmd/hammer

It even works for cross-compilation (go install currently refuses a custom GOBIN for cross-compilation)

@lyda
Copy link

lyda commented Apr 12, 2024

Sigh. It would be really nice if something like this existed. Reading through the objections, how about using -O (an upper-case "O")?

go install -O foozle some.git.host/flibble,burble/foozle.git@v1.3-kitten

That way it doesn't clash with the -o flag elsewhere. Don't allow a slash in foozle and put it the same place that foozle.git currently goes. This doesn't solve all problems, but it solves a lot of them.

@ldemailly
Copy link

being able to specify a full destination path including directory (ie with slashes) is what I need for ci builds or predictable installs but maybe #52898 can be reopen if the direction for this one is to only address the binary basename

@matloob
Copy link
Contributor

matloob commented Jun 27, 2024

I think that since we support go run pkg@version it makes sense to also support go build pkg@version like others have suggested. Though I wonder if we might want to follow @mvdan's suggestion in #44469 (comment) to make a single issue to add pkg@version to all commands where it makes sense?

@matloob
Copy link
Contributor

matloob commented Jul 2, 2024

@hyangah Should we turn this into a proposal issue?

@hyangah
Copy link
Contributor Author

hyangah commented Aug 21, 2024

Do we need more than marking this as a proposal?

@matloob
Copy link
Contributor

matloob commented Aug 21, 2024

I think we could update the first comment with the current version of the proposal?

I'll mark this as a proposal.

@matloob matloob added Proposal and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 21, 2024
@matloob matloob changed the title cmd/go: allow versioned go build commands proposal: cmd/go: allow versioned go build commands Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Incoming
Development

No branches or pull requests