Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deps: updates for bazel 8 compatibility #2379

Merged
merged 7 commits into from
Nov 12, 2024

Conversation

rickeylev
Copy link
Collaborator

@rickeylev rickeylev commented Nov 6, 2024

Various changes to support Bazel 8. An important note is dependencies have forced
us to change the versions of Bazel we support.

Summary of changes:

  • rules_cc 0.0.14: Releases after 0.0.9 have some Bazel 8 fixes, but also broke
    some things. Things seemed to have settled by 0.0.14.
  • protobuf 29.0-rc1: Technically 28.0 works, however:
    1. 29.0-rc1 is coming via a transitive dependency anyways, and
    2. In protobuf 28.0, compile warnings are treated as errors, which our Debian CI
      respects (and thus fails), while other platforms ignore.
  • stardoc 0.7.1: Fixes an issue with Bazel 8 and stardoc using empty globs.
  • Bazel 7.4 is now the minimum supported Bazel version. This requirements comes via
    dependencies.
  • Drop Bazel 6 bzlmod support. This requirement comes via dependencies.
  • Add a presubmit job for last_rc Bazel (currently the 8.x RC).
  • Use a local patch so Gazelle works with Bazel 8. This can be removed
    once native.sh_binary reference in def.bzl blocks Bazel 8 integration bazel-contrib/bazel-gazelle#1959 is fixed and released.
  • Fix a $(rpathlocation) call in bootstrap tests.
  • Update bzl_library deps after upgrading deps: the set of targets that provide
    bzl sources changed in rules_cc and protobuf in these newer versions.

Sorting this all out and finding the right combination of dependency versions was
fairly involved. The details of that are in
#2378.

Work towards #2378, #2387

@rickeylev rickeylev force-pushed the fix.bazel8 branch 30 times, most recently from 2eb3ccd to d7b780a Compare November 8, 2024 22:07
@rickeylev rickeylev force-pushed the fix.bazel8 branch 3 times, most recently from dcf2d27 to bfe467c Compare November 8, 2024 23:08
@rickeylev rickeylev marked this pull request as ready for review November 9, 2024 00:21
@rickeylev rickeylev requested a review from aignas as a code owner November 9, 2024 00:21
@rickeylev rickeylev changed the title chore: updates for bazel 8 compatibility deps: updates for bazel 8 compatibility Nov 9, 2024
@rickeylev rickeylev enabled auto-merge November 9, 2024 02:50
Copy link
Collaborator

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only real question is the defs.bzl usage from rules_cc. I remember there was a PR here #2293, but I can still find usage of defs.bzl in our git repo in this PR.

@@ -59,3 +59,6 @@ pip.parse(

# example test dependencies
bazel_dep(name = "rules_shell", version = "0.2.0", dev_dependency = True)

# Only needed to make rules_python's CI happy
bazel_dep(name = "rules_java", version = "8.3.0")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little bit mysterious. Will we be able to remove it at some point?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the comment to explain why it's here.

The example sets --java_runtime_version=remotejdk_11, which is broken with Bazel 8 until rules_java 8.3.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if I remember correctly this is only to make the coverage collection not depend on system Java. So technically all of our consumers will need to add this dep if they want coverage.

examples/bzlmod/MODULE.bazel Show resolved Hide resolved
MODULE.bazel Show resolved Hide resolved
srcs = [
# It appears something in the CC rules loads cc_proto_library, but
# where isn't clear.
"@com_google_protobuf//:bzl_srcs",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove the usage of defs.bzl, maybe that will fix it?

$ git grep "defs.bzl.*cc" **/*.bzl
python/private/common_bazel.bzl:load("@rules_cc//cc:defs.bzl", "CcInfo", "cc_common")
python/private/hermetic_runtime_repo_setup.bzl:load("@rules_cc//cc:defs.bzl", "cc_import", "cc_library")
python/private/local_runtime_repo_setup.bzl:load("@rules_cc//cc:defs.bzl", "cc_library")
python/private/py_executable.bzl:load("@rules_cc//cc:defs.bzl", "cc_common")
python/private/runtime_env_toolchain.bzl:load("@rules_cc//cc:defs.bzl", "cc_library")
python/private/whl_filegroup/whl_filegroup.bzl:load("@rules_cc//cc:defs.bzl", "cc_library")
tests/support/cc_toolchains/fake_cc_toolchain_config.bzl:load("@rules_cc//cc:defs.bzl", "cc_common")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, yeah, removing defs.bzl might fix this (depends on if the more specific targets properly specify their bzl_libraries 😅 )

I wrote this comment before (1) I unwound the whole defs.bzl removing cc_proto_library thing, and (2) I gave up trying to make bazel 6 with bzlmod work.

I've updated the comment to better reflect this.

@aignas
Copy link
Collaborator

aignas commented Nov 11, 2024

Ah, I read the PR I linked - I somewhat think that maybe we could couple the PRs together and see if we can remove the extra protobuf deps that we had to add. rules_rust in our examples looks extremely concerning.

@aignas
Copy link
Collaborator

aignas commented Nov 11, 2024

nit: I think we should also resolve the following warnings in the rules_python repo:

WARNING: For repository 'bazel_features', the root module requires module version bazel_features@1.9.1, but got bazel_features@1.19.0 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.4, but got platforms@0.0.10 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'rules_proto', the root module requires module version rules_proto@6.0.0-rc1, but got rules_proto@6.0.2 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off
WARNING: For repository 'com_google_protobuf', the root module requires module version protobuf@29.0-rc1, but got protobuf@29.0-rc2 in the resolved dependency graph. Please update the version in your MODULE.bazel or set --check_direct_dependencies=off

@rickeylev
Copy link
Collaborator Author

The only real question is the defs.bzl usage from rules_cc. I remember there was a PR here #2293, but I can still find usage of defs.bzl in our git repo in this PR.

Yeah, this whole part is a mess. Our usage of rules_cc/defs.bzl is OK, sort of -- deprecated, but not broken, per-se. The problem comes when other dependencies come into play. Basically, rules_cc removed a symbol (cc_proto_library) from defs.bzl. However, some things still loaded that symbol, so fixes were sent out. While all this is going on, various versions of modules were released with various problems with Bazel 6, Bazel 8, and/or the removal of that symbol.

So, our usage of rules_cc/defs.bzl is only a problem depending on what combinations of what dependencies you end up with. Us switching off rules_cc/defs.bzl makes it less likely to occur (since we wouldn't be a potential cause of loading that file when an incompatible dependency is being used).

re: coupling this PR and the defs.bzl removal: Well, I was going to say this PR has been a chore and I don't want to expand it's scope, but now CI is failing with yet-another-rules-cc-defs.bzl looking failure. So maybe it'll be easier to put those in one PR. I'll have a look at that failure to see which is gonna be easier.

re: module versions:

  • bazel_features and platforms are coming from dev dependencies, I think. I don't see those warnings when building the bzlmod example.
  • rules_proto, protobuf: I've updated their versions

@rickeylev
Copy link
Collaborator Author

CI is failing with yet-another-rules-cc-defs.bzl looking failure

The failure was: Bazel 9 with workspace was failing because Bazel 9 removes native.cc_proto_library

I've "fixed" this by removing the bazel-in-bazel integration tests for rolling, A bzlmod invocation is OK because it uses protobuf 29.0-rc1+, which has a fix for that. However, Bazel 9 with workspace is using an earlier protobuf version. I tried updating that, but it looks like it has further dependencies that need to be setup and/or upgraded.

Given that Bazel 9 is a ways off, I think it makes sense to temporarily disable that test. I'll need some time to try and figure out the necessary WORKSPACE updates that need to be made.

@aignas
Copy link
Collaborator

aignas commented Nov 12, 2024

Thanks for all of this untangling of the mess. This is sure a bumpy one - I remember bazel 7 not being nearly as bad.

Copy link
Collaborator

@aignas aignas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving since the CI is green and we can go bit by bit fixing all of the remaining things. My main concerns are:

  • Can we remove some of the deps added here by switching away from rules_cc:defs.bzl.
  • The java flag in the bazelrc was there for bazel coverage support and I don't understand if we can safely remove it and the coverage will just work or we have to document it to the example code readers, that they need to have that dep as well?

Regardless of the outcome of the above, this is moving in the right direction.

Merged via the queue into bazelbuild:main with commit 4864d63 Nov 12, 2024
4 checks passed
@rickeylev rickeylev deleted the fix.bazel8 branch November 12, 2024 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants