go_repository: add 'clean' build_file_generation

[TvdW]

What type of PR is this?

Feature

What package or component does this PR mostly affect?

go_repository

What does this PR do? Why is it needed?

Before bzlmod, repositories were global, and BUILD files inside upstream repositories could load from the repositories defined at the root-level WORKSPACE file.

With the introduction of visibility rules, these result in errors that cannot trivially be worked around. One pattern that has emerged is ignoring existing build files by specifying gazelle:build_file_name BUILD.bazel, however this does not work for anything already using BUILD.bazel filenames.

This does affect real-world users: an example is https://github.com/google/go-jsonnet which has a BUILD.bazel file to genrule its AST. The module is pure-go, but the build step contains cpp code, and when imported under bzlmod-gazelle will not work because the cpp code is invoked. As a workaround, there is now a 'jsonnet' BCR module.

This patch introduces a build_file_generation = clean option, which removes existing build files before generating new ones. It allows loading the jsonnet code once again, and also makes it possible to load some of the complex protobuf repositories.

Which issues(s) does this PR fix?

Fixes #1773, #1769, #1535, bazelbuild/rules_go#3685

Other notes for review

I have not made 'clean' the default, though I think that for most use cases it will be the correct behavior.

An example for fixing jsonnet:

go_deps.gazelle_override(
    build_file_generation = "clean",
    path = "github.com/google/go-jsonnet",
)

With some gymnastics this does also make it possible to import googleapis as if it were a regular Go dependency, without any switched_rules:

go_deps.module(
    path = "github.com/googleapis/googleapis",
    sum = "unused",
    version = "0.0.0",
)
go_deps.gazelle_override(
    build_file_generation = "clean",
    directives = [
        "gazelle:proto file",
    ],
    path = "github.com/googleapis/googleapis",
)
go_deps.archive_override(
    path = "github.com/googleapis/googleapis",
    sha256 = "7faad14f34810d1165a9873c140b378bc783a1448e46ebdd20a65ca66afa7bb1",
    strip_prefix = "googleapis-7002406181eb300da880701035a25157a5099abb",
    urls = ["https://github.com/googleapis/googleapis/archive/7002406181eb300da880701035a25157a5099abb.zip"],
)

# resolution
 "gazelle:resolve proto google/rpc/code.proto @com_github_googleapis_googleapis//google/rpc:code_proto",
 "gazelle:resolve proto google/rpc/status.proto @com_github_googleapis_googleapis//google/rpc:status_proto",
 "gazelle:resolve proto google/api/annotations.proto @com_github_googleapis_googleapis//google/api:annotations_proto",
 "gazelle:resolve proto google/api/field_behavior.proto @com_github_googleapis_googleapis//google/api:field_behavior_proto",

[TvdW]

I've marked this PR as draft as I have not yet added tests, but I am planning to.

[fmeum]

This is highly appreciated, let me know if you get stuck anywhere.

[fmeum]

Cc @linzhp for googleapis

[TvdW]

Ok, I've added tests. Turns out it broke on Windows, so I fixed that and then it broke on Bazel 6.x, so I ended up adding the code into cmd/fetch_repo. My initial attempt was to add it into Gazelle itself, however between fetch_repo and Gazelle the go_repository code first writes a BUILD file with directives, and that would have been removed/ignored by my changes. Tests are green now 😄

This adds a test to see if this error is fixed:

ERROR: no such package '@@[unknown repo 'cpp_jsonnet' requested from @@gazelle~~go_deps~com_github_google_go_jsonnet]//stdlib': The repository '@@[unknown repo 'cpp_jsonnet' requested from @@gazelle~~go_deps~com_github_google_go_jsonnet]' could not be resolved: No repository visible as '@cpp_jsonnet' from repository '@@gazelle~~go_deps~com_github_google_go_jsonnet'
ERROR: /private/var/tmp/_bazel_tom.vanderwoerdt/c65fedc35a8ffb2abdd111f841d4694d/external/gazelle~~go_deps~com_github_google_go_jsonnet/astgen/BUILD.bazel:3:8: no such package '@@[unknown repo 'cpp_jsonnet' requested from @@gazelle~~go_deps~com_github_google_go_jsonnet]//stdlib': The repository '@@[unknown repo 'cpp_jsonnet' requested from @@gazelle~~go_deps~com_github_google_go_jsonnet]' could not be resolved: No repository visible as '@cpp_jsonnet' from repository '@@gazelle~~go_deps~com_github_google_go_jsonnet' and referenced by '@@gazelle~~go_deps~com_github_google_go_jsonnet//astgen:dumpstdlibast'

I'm not testing any of the googleapis things I mentioned: since googleapis is technically not a Go repository, getting it working is a happy side-effect.

[TvdW]

I've added one more commit on top, reordering (and deduplicating) some code so that fetch_repo is always called. From the commit message:

go_repository: always call fetch_repo, even for urls

After my changes to remove build files as part of fetch_repo, I realized
that when urls=[] is specified the code path doesn't use fetch_repo at
all. In this case, fetch_repo_args would remain None, and raise an
error. Not ideal!

This left me with two general options: either I move the code into a
separate command invocation, or I have all code go through fetch_repo.

The latter seems to be the pragmatic solution: gazelle by itself doesn't
generate code with urls=[] (in either bzlmod or vanilla update-repos),
so it's the option with lower performance impact (also considering we're
stacking it on top of a network download).

[fmeum]

@tyler-french Could you test this?

[fmeum]

Thanks for getting rid of some unnecessary special casing, the code is easier to follow now even though it's longer.