-
Notifications
You must be signed in to change notification settings - Fork 536
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gazelle: Visibilities are incorrect for projects with multiple # gazelle:python_root
directives
#1682
Comments
I think this is something that you may want only if the gazelle roots are using the same requirements files, so this ask is not necessarily something that is useful for all scenarios and I am not sure it should be the default. |
@aignas I posted a reply here: #1683 (comment). Regarding the requirements files, that's true regarding external dependencies. Our main issue has to do with internal dependencies between the different Python projects in our Bazel workspace themselves. |
Continuing the discussion her. I think the best way would be to support the default visibility directive that gazelle has to support your usecase. Just to ensure that we are not missing anything here - if you manually modify the visibility of a talget you want to expose, does it get overwritten in the consecutive gazelle runs? |
I agree that supporting the
If you manually modify the visibility, it's not overwritten. That's true (and good!). But this saves a lot of manual effort the first time the monorepo gets BUILD files generated by Gazelle, in making those updates. The problem is that the Gazelle-inserted visibility can be too restrictive, but it overrules any package-level visibility declarations. Every single target would need to have the visibility altered after Gazelle is run. And any targets that are added later also need to be corrected. The calculus that motivated this was: a one-line |
I think the best way forward is to support the |
I don't have bandwidth to work on this either. |
It seems the current behavior is erroneous, and either path would ameliorate this. The one in PR #1683 fixes the bug, and processing The Gazelle source code (in a comment) says that if there's a package(default_visibility=...) stanza, then no visibility declaration should be generated. This helps build the case that the current behavior is wrong. Even if we don't "fix" it by respecting default_visibility directives, our PR resolves that. |
This code was added 6 years ago: bazelbuild/bazel-gazelle@cdc77d0. This is when only the Go generation existed. This is not a rule. If it were, it would be enforced by the framework. |
@f0rmiga @aignas we're currently blocked from using gazelle python in our codebase due to this issue. What is your suggestion for moving forward assuming we do not have the bandwidth to learn Go and your codebase to make the ideal changes? Is there any chance you'd approve the current PR or a variation of it or should we look into other solutions that do not involve gazelle-python? And to be clear, the main difference between the
instead of this one (which the current PR enables):
Could you please explain why this is a bad solution? Or at least why you think it's worse than what gazelle-python currently does which is enforcing a stronger opinion that is not configurable at all for the rule visibilities? |
I appreciate the time you've put into this discussion and raising a point that makes your adoption of the Python extension for Gazelle difficult. As an open-source project, we have to be careful with changes to the behaviour of this extension, especially one that affects everyone else, in order to satisfy your use case. I'm not saying that your use case is wrong or that it doesn't deserve attention, but the optimal solution has been laid out, and you are not committed to helping improve the code generator (even though you are blocked on it). A few companies that provide Bazel consulting would be happy to contribute to this issue if you hire them.
Find someone who has the bandwidth to contribute (whatever it means - including learning Go).
You are asking for everyone in the community to fit your workflow and not the other way around. The Gazelle code generator is optional and it is opinionated. |
It also occurred to me that you could probably just use https://github.com/aspect-build/plugin-fix-visibility to fix your visibility issues. |
One more reason why it may be a sub-optimal solution - sharing dependencies between different workspaces than have different requirements files can lead to unpredictable behaviour when two versions of the same package are in the dependency graph. The default visibility settings are there to enforce a sane and deterministic behaviour. If your use case does not involve mixing of third party dependencies then it would be great if you could describe it here. |
@f0rmiga thanks we'll try to use that plugin. @aignas thanks for that reasoning! This makes sense and it's not a scenario I had in mind. Though I'd like to point that with the changes in the PR the default visibility will be Regarding our use case, we have a large monorepo with a single set of external Python/Rust/etc. dependencies for the whole repo. Then, each project in that repo lives in a different directory and only pulls in the dependencies it needs. We perform dependency resolution globally for the whole repo and so do not run into the situation of having different projects depend on different versions of the same package. |
@eaplatanios, one problem with the Regarding the usecase, as I understand correctly your setup is as follows:
I think this is a real feature request that the |
@aignas what you describe as our use case is accurate. Thanks for summarizing it! I only have a couple comments on the rest of your message. You mentioned that:
I believe this is not entirely correct and you only need to add
Yeah I'm also not sure without trying but unfortunately we don't have the necessary background to try this out (i.e., neither familiarity with Go nor with Gazelle implementation details like how to use that extension from upstream Gazelle here). :( |
Related: adding a new Example usage:
Would result in this visibility def:
|
Fixes #1783. Provides a way to fix #1682. Add the `python_default_visibility` directive. This directive sets the `visibility` attribute on all generated `py_*` rules. It accepts a comma-separated list of labels to add as visibility targets, similar to how the base `default_visibility` directive works. Setting this directive multiple times will override previous values. Two special values are also accepted: `NONE` and `DEFAULT`. See ./gazelle/README.md#directive-python_default_visibility for details. The directive also accepts a special string `"$python_root"` that gets replaced with the `python_root` value, if set. If not set, `"$python_root"` is replaced with the empty string. --------- Co-authored-by: Thulio Ferraz Assis <3149049+f0rmiga@users.noreply.github.com>
🐞 bug report
Affected Rule
The issue is caused by the rule: https://github.com/bazelbuild/rules_python/blob/b106f91c9da7e31d73a7293e88ac78eedcad2057/gazelle/python/generate.go#L215Is this a regression?
NoDescription
This is necessary in monorepo situations where there are multiple
# gazelle:python_root
directives used, e.g. in the structure below:As it stands, we write a narrowly scoped
visibility
attribute for every generated rule; the visibility is//{python-root-package}:__subpackages__
. Bazel will fail to build targets inproj_1
because of this.🔥 Exception or Error
🌍 Your Environment
Operating System:
Output of
bazel version
:Rules_python version:
Anything else relevant?
The desired implementation is not to set everything to
//visibility:public
, rather, it's to resolve cross-project dependencies in a monorepo the same way as they are within one directory.Consider a working example here: main...aryamccarthy:rules_python:main
The text was updated successfully, but these errors were encountered: