bzlmod: offer "go install" like capabilities

[malt3]

What version of rules_go are you using?

v0.41.0

What version of gazelle are you using?

v0.32.0

What version of Bazel are you using?

6.3.1

Does this issue reproduce with the latest releases of all the above?

yes

What operating system and processor architecture are you using?

Ubuntu 22.04 / amd64

Any other potentially useful information about your toolchain?

n/a

What did you do?

A lot of developer tools are written in go. Often, those are installed like this on a developers laptop: go install example.com/module@v1.2.3.
With bzlmod, I think rules_go could allow users to specify a dependency on a go tool from a third party module in the MODULE.bazel file.

Consider the following (imaginary) syntax:

bazel_dep(name = "rules_go", version = "0.39.1")
bazel_dep(name = "gazelle", version = "0.31.0")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")

go_sdk.download(version = "1.20.3")

go_tool = use_extension("@gazelle//:extensions.bzl", "go_tool")

go_tool.module(
    name = "buildifier",
    path = "github.com/bazelbuild/buildtools/buildifier",
    sum = "h1:ZapXFRHO5mOeXXZentMaSmpWbMNWNxiQE8K+pm4n8Cw=",
    version = "v0.0.0-20230807152244-48385dd1e219",
)

The tool could then be used in genrules, to build custom rules (things like gazelle) or can be executed via bazel run.
Most importantly, I would want this to use the isolated dependency mechanism discussed in #3458, so you can depend on a tool written in go that uses one set of dependencies (and dep versions) that is disconnected from your own code.

This is a feature request / idea. I would potentially like to work on this but first want to know if this is actually something rules_go wants to offer.
Also note that I am not sure if this belongs in rules_go or gazelle.

What did you expect to see?

The ability to easily use third party go tools in bazel.

What did you see instead?

Currently, you have to jump through some hoops. One possibility is to depend on the main package of a tool in your monorepo and create your own binary. This has downsides, like introducing the tools dependencies into your own source code.

[fmeum]

I really like this idea, it could provide a clean solution to bazel-contrib/bazel-gazelle#1585. I will happily offer support to anyone who wants to work on this, but we first need to discuss the design.

Whether this should live in rules_go or gazelle ultimately depends on how we want to implement it. For example, we probably need to answer these questions:

1. Do we want to build the tools with Bazel (that is, generate go_repositorys) or via go install?
2. Do we want to support builds where host and exec platform aren't equal (I think we should)?
3. How does this relate to upcoming Bazel features such as using repository rules in MODULE.bazel files and the "true repository cache"?

[malt3]

Do we want to build the tools with Bazel (that is, generate go_repositorys) or via go install?

Is it possible to have a shared bazel cache for the tools if we go with the go install implementation?
I think the implementation should be able to use the bazel cache effectively, which makes me think that generating go_repository rules is probably better, but this is under the assumption that using go install will result in no / bad shared caching.

Do we want to support builds where host and exec platform aren't equal (I think we should)?

I also think we should.

How does this relate to upcoming Bazel features such as bazelbuild/bazel#17141 and the bazelbuild/bazel#12227?

I'm afraid I don't understand the internals of bzlmod / Bazel well enough to make a statement on this. I think that being able to share go tools caches / binaries between workspaces would be great if possible.

[fmeum]

We should probably wait for the repository cache design doc. Since different tools would use different repos for their deps anyway (due to isolation), we may not be able to cache anything but the final build output anyway.

[fmeum]

I thought about this more and can offer a design proposal: Instead of introducing a new module extension or even new API at all, how about making the existing go_deps.module more powerful by having it fetch all transitive dependencies automatically? More concretely, we could:

1. In the go_deps extension, use the go_first_host_compatible_sdk_label repo provided by the go_sdk extension to get access to a go tool.
2. Collect all dependencies specified via go_deps.module and use go list -m to get all their transitive module dependencies (module paths, versions and sums). Inject these dependencies into the regular Minimum Version Selection logic in go_deps.
3. Make the sum attribute on go_deps.module optional. The sums are obtained from go list -m anyway and are automatically persisted in MODULE.bazel.lock (if used).

By default, the tool would share its transitive dependencies with all other Go deps, resulting in cache sharing for common dependencies, both during load and build time. If you need specific dependency version to build a tool, you can rely on a separate isolated usage of go_deps. Using regular Gazelle-generated BUILD files instead of building at load time also means that tools can be built for any exec platform.

Let me know what you think about this approach. If someone is interested in working on this, please let me know and I can offer support.

CC @tyler-french @cgrindel

[malt3]

Sounds like a good solution. I can't promise to work on this anytime soon, but may pick this up if I have time.

[cgrindel]

@fmeum So, instead of trying to shoehorn all of the Go deps into go.mod, the extension will perform the selection logic. Is that the crux of the proposal? If so, I wonder what the impact will be on other tooling (e.g. gopls).

When I have tackled this for my own repos, I thought the shortcoming was with how I could tell go mod to include the deps from Go tools that I wanted to use. In other words, this problem does not feel like a rules_go problem, but a go mod problem. I have toyed around with writing a utility that would create a go.mod file that included all of the Go deps for the repository, both library and executable. This would allow rules_go and all of the regular Go tooling to work properly.

[fmeum]

I just came across the semi-official recommendation in golang/go#25922 (comment). The tools.go approach should work with Bzlmod and update-repos.

@cgrindel @malt3 What do you think about this solution?

[malt3]

I'm not sure if I fully understand your proposal. Would I, as a user still have one set of go dependencies for tools and my own go code with the tools.go approach?

If the end result is still minimal version selection on all deps of my code and the deps of my Go tools (such as gazelle and rules_go binaries), I don't gain a lot from this.
If this allows me to use a separate set of dependencies in my code (compared to the deps of my tools), then this looks fine to me.

[fmeum]

Ideally you could do both. Here is what I envision the tools.go approach to look like with a separate dependency closure: bazel-contrib/bazel-gazelle#1678
As you can see, this doesn't quite work yet as it fails with duplicate package errors. I will look into that.

[fmeum]

The tricky part is making go_proto_library work in an isolated dependency tree: It references the protobuf runtime from the global dependency tree, which will usually not match the version in the isolated one. Either gazelle needs to override the runtime for isolated repos or the checked in generated code has to be used.

[fmeum]

I fixed the linker issues. Could you give bazel-contrib/bazel-gazelle#1678 a try?

[malt3]

Thank you so much @fmeum! That looks very nice. Was not able to give it a try yet but I’m pretty sure this fits my use case well.