-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the buildtagger tool #187
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ulucinar
force-pushed
the
tagger
branch
2 times, most recently
from
March 6, 2024 03:40
0f84410
to
1833111
Compare
This was referenced Mar 6, 2024
- buildtagger can be used to generate the desired build tags (constraints) for the official provider families. - Each resource provider's source modules can share a unique build tag so that tag-aware Go tools (including golangci-lint) can load only those source modules. Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
… exit non-zero Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
…w to conditionally skip the "lint" job. Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
…exity Signed-off-by: Alper Rifat Ulucinar <ulucinar@users.noreply.github.com>
sergenyalcin
approved these changes
Mar 7, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @ulucinar LGTM!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of your changes
buildtagger
can be used to generate the desired build tags (constraints) for the official provider families. Each resource provider's source modules can share a unique build tag so that the tag-aware Go tools (includinggolangci-lint
linter runner) can load only those source modules belonging to that resource provider, which results in reduced memory consumption.This tool can add the desired build tags to the specified Go source files. Sample invocations of the tool are as follows:
buildtagger --parent-dir ./apis --regex "(.+)/.+/.+\.go" --tag-format "(%s || all) && !ignore_autogenerated" --mode dir
This will run the tagger on the
apis
folder and its sub-folders recursively, matching relative paths against the regular expression supplied with the--regex
argument. The directory structure under theapis
folder for theupjet
-based providers is as follows:So this command matches the relative path (
--mode
set todir
) wrt to the parent path./apis
of every source file under an API group, capturing the API groups of them (such asaccessanalyzer
oraccount
) with a regular expression group. This capturing group is then substituted into format string specified with the--tag-format
argument, e.g.,(%s || all) && !ignore_autogenerated
. So the generated build constraint for the source fileaccount/v1beta1/zz_alternatecontact_terraformed.go
is//go:build (account || all) && !ignore_autogenerated
.This allows the linter runner to only analyze the source files with the
account
build constraint when linting theaccount
resource provider (i.e., the providerupbound/provider-aws-account
) when it's run with theaccount
build tag. Similarly, if the linter runner analyze cache is already populated, and thus, the analysis phase of the linter runner will be cheap (in terms of compute resources), one can just run the linter runner with theall
build tag to lint all the source files (by convention we tag all the source files with theall
tag). The!ignore_autogenerated
constraint is inherited from the controller-gen tool.buildtagger --parent-dir ./internal/controller --regex "zz_(.+)_setup\.go" --tag-format "(%s || all) && !ignore_autogenerated" --mode file
This invocation is very similar to the above one, it just uses the
file
mode instead of the defaultdir
mode. In thefile
mode, the specified regular expression with the--regex
parameter is matched against the base filename instead of the relative path of the discovered file wrt to the specified parent directory. Rest of the semantics are exactly the same with the above example with thedir
mode. This mode can be used to capture the API group name (the resource provider name) from the names of the generated source files instead of the directory names. For example,upjet
generates a setup file for each API group to setup the controller manager with all the reconcilers under that API group:For these files, we need to capture the resource provider name (the API group) from the file name. This is where the
file
mode can be used.buildtagger --parent-dir ./internal/controller/eks/clusterauth/controller.go --tag-format "eks || all" --mode file
This another usage example where the specified regex does not define any capturing groups and the specified tag format does not have any format specifiers. This can be used to tag manually added (non-generated) files.
internal/controller/eks/clusterauth/controller.go
is a manually maintained file for theClusterAuth.eks
resource and we tag it with theeks
constraint so that it's linted as part of the resource providerupbound/provider-aws-eks
.Here's a formal description of the command-line arguments for the tool:
As explained above, the number of format specifiers in the
--tag-format
option must match the number of capturing groups in the--regex
option, and they must all be%s
string specifiers.Go version bump
This PR also bumps the Go module version and the Go version used in the repo's build pipelines to
v1.21
.Alternatives Considered
We've also considered generating these build constraints for the family providers via upjet, which is one of the underlying code generation frameworks that we use to generate the official providers including
upbound/provider-aws
. We did not choose this approach because of the following reasons:upbound/provider-aws
is reducing the compute resources required by the linter runner by running the linters on each API group (resource provider likeec2
oracm
) in isolation. For non-family providers, this idea is inherently more complex to implement across different API groups because there are no logical boundaries that we build this solution on in non-family providers. Adding this capability that we will use only for a family provider toupjet
seems like introducing unnecessary complexity there. Having this capability implemented as a separate tool is thus better for managing complexity.upjet
,upjet
is not the only code generation tool we are using in the upjet-based provider repositories. We also rely on angryjet and controller-gen in those repos. We would also need these tools to be able to generate the build constraints for the resource providers and handle them. A standalone tool likebuildtagger
can implement this feature regardless of the underlying code generators we are using, i.e., this is a cross-cutting concern for us across different code generation frameworks.I have:
make reviewable test
to ensure this PR is ready for review.How has this code been tested
This tool has been tested with the official AWS provider in this test PR.