feat: Expose Python C headers through the toolchain.

This allows getting a build's `cc_library` of Python headers through
toolchain resolution instead of having to use the underlying toolchain's
repository `:python_headers` target directly.

Without this feature, it's not possible to reliably and correctly get
the C information about the runtime a build is going to use. Existing
solutions require carefully setting up repo names, external state,
and/or using specific build rules. In comparison, with this feature,
consumers are able to simply ask for the current headers via a special
target or manually lookup the toolchain and pull the relevant
information; toolchain resolution handles finding the correct headers.

The basic way this works is by registering a second toolchain to carry
C/C++ related information; as such, it is named `py_cc_toolchain`. The
py cc toolchain has the same constraint settings as the regular py
toolchain; an expected invariant is that there is a 1:1 correspondence
between the two. This base functionality allows a consuming rule
implementation to use toolchain resolution to find the Python C
toolchain information.

Usually what downstream consumers need are the headers to feed into
another `cc_library` (or equivalent) target, so, rather than have every
project reimplement the same "lookup and forward cc_library info" logic,
this is provided by the `//python/cc:current_py_cc_headers` target.
Targets that need the headers can then depend on that target as if it
was a `cc_library` target.

Work towards bazelbuild#824

python/cc/BUILD.bazel

@@ -0,0 +1,20 @@
+# Package for C/C++ specific functionality of the Python rules.
+
+load("//python/private:current_py_cc_headers.bzl", "current_py_cc_headers")
+load("//python/private:util.bzl", "BZLMOD_ENABLED")
+
+# This target provides the C headers for whatever the current toolchain is
+# for the consuming rule. It basically acts like a cc_library by forwarding
+# on the providers for the underlying cc_library that the toolchain is using.
+current_py_cc_headers(
+    name = "current_py_cc_headers",
+    # Building this directly will fail unless a py cc toolchain is registered,
+    # and it's only under bzlmod that one is registered by default.
+    tags = [] if BZLMOD_ENABLED else ["manual"],
+    visibility = ["//visibility:public"],
+)
+
+toolchain_type(
+    name = "toolchain_type",
+    visibility = ["//visibility:public"],
+)

python/cc/py_cc_toolchain.bzl

@@ -0,0 +1,19 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Public entry point for py_cc_toolchain rule."""
+
+load("//python/private:py_cc_toolchain_macro.bzl", _py_cc_toolchain = "py_cc_toolchain")
+
+py_cc_toolchain = _py_cc_toolchain

python/cc/py_cc_toolchain_info.bzl

@@ -0,0 +1,19 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Public entry point for PyCcToolchainInfo."""
+
+load("//python/private:py_cc_toolchain_info.bzl", _PyCcToolchainInfo = "PyCcToolchainInfo")
+
+PyCcToolchainInfo = _PyCcToolchainInfo

python/private/current_py_cc_headers.bzl

@@ -0,0 +1,25 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Implementation of current_py_cc_headers rule."""
+
+def _current_py_cc_headers_impl(ctx):
+    py_cc_toolchain = ctx.toolchains["//python/cc:toolchain_type"].py_cc_toolchain
+    return py_cc_toolchain.headers.providers_map.values()
+
+current_py_cc_headers = rule(
+    implementation = _current_py_cc_headers_impl,
+    toolchains = ["//python/cc:toolchain_type"],
+    provides = [CcInfo],
+)

python/private/py_cc_toolchain_info.bzl

@@ -0,0 +1,43 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Implementation of PyCcToolchainInfo."""
+
+PyCcToolchainInfo = provider(
+    doc = "C/C++ information about the Python runtime.",
+    fields = {
+        "headers": """\
+(struct) Information about the header files, with fields:
+  * providers_map: a dict of string to provider instances. The key should be
+    a fully qualified name (e.g. `@rules_foo//bar:baz.bzl#MyInfo`) of the
+    provider to uniquely identify its type.
+
+    The following keys are always present:
+      * CcInfo: the CcInfo provider instance for the headers.
+      * DefaultInfo: the DefaultInfo provider instance for the headers.
+
+    A map is used to allow additional providers from the originating headers
+    target (typically a `cc_library`) to be propagated to consumers (directly
+    exposing a Target object can cause memory issues and is an anti-pattern).
+
+    When consuming this map, it's suggested to use `providers_map.values()` to
+    return all providers; or copy the map and filter out or replace keys as
+    appropriate. Note that any keys begining with `_` (underscore) are
+    considered private and should be forward along as-is (this better allows
+    e.g. `:current_py_cc_headers` to act as the underlying headers target it
+    represents).
+""",
+        "python_version": "(str) The Python Major.Minor version.",
+    },
+)

python/private/py_cc_toolchain_macro.bzl

@@ -0,0 +1,25 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Fronting macro for the py_cc_toolchain rule."""
+
+load(":py_cc_toolchain_rule.bzl", _py_cc_toolchain = "py_cc_toolchain")
+load(":util.bzl", "add_tag")
+
+# A fronting macro is used because macros have user-observable behavior;
+# using one from the onset avoids introducing those changes in the future.
+def py_cc_toolchain(**kwargs):
+    #  This tag is added to easily identify usages through other macros.
+    add_tag(kwargs, "@rules_python//python:py_cc_toolchain")
+    _py_cc_toolchain(**kwargs)

python/private/py_cc_toolchain_rule.bzl

@@ -0,0 +1,47 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Implementation of py_cc_toolchain rule."""
+
+load(":py_cc_toolchain_info.bzl", "PyCcToolchainInfo")
+
+def _py_cc_toolchain_impl(ctx):
+    py_cc_toolchain = PyCcToolchainInfo(
+        headers = struct(
+            providers_map = {
+                "CcInfo": ctx.attr.headers[CcInfo],
+                "DefaultInfo": ctx.attr.headers[DefaultInfo],
+            },
+        ),
+        python_version = ctx.attr.python_version,
+    )
+    return [platform_common.ToolchainInfo(
+        py_cc_toolchain = py_cc_toolchain,
+    )]
+
+py_cc_toolchain = rule(
+    implementation = _py_cc_toolchain_impl,
+    attrs = {
+        "headers": attr.label(
+            doc = ("Target that provides the Python headers. Typically this " +
+                   "is a cc_library target."),
+            providers = [CcInfo],
+            mandatory = True,
+        ),
+        "python_version": attr.string(
+            doc = "The Major.minor Python version, e.g. 3.11",
+            mandatory = True,
+        ),
+    },
+)

python/private/toolchains_repo.bzl

@@ -83,6 +83,15 @@ toolchain(
     toolchain = "@{user_repository_name}_{platform}//:python_runtimes",
     toolchain_type = "@bazel_tools//tools/python:toolchain_type",
 )
+
+toolchain(
+    name = "{prefix}{platform}_py_cc_toolchain",
+    target_compatible_with = {compatible_with},
+    target_settings = {target_settings},
+    toolchain = "@{user_repository_name}_{platform}//:py_cc_toolchain",
+    toolchain_type = "@rules_python//python/cc:toolchain_type",
+
+)
 """.format(
             compatible_with = meta.compatible_with,
             platform = platform,

python/private/util.bzl

@@ -1,7 +1,25 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 """Functionality shared by multiple pieces of code."""
 
 load("@bazel_skylib//lib:types.bzl", "types")
 
+# When bzlmod is enabled, canonical repos names have @@ in them, while under
+# workspace builds, there is never a @@ in labels.
+BZLMOD_ENABLED = "@@" in str(Label("//:unused"))
+
 def copy_propagating_kwargs(from_kwargs, into_kwargs = None):
     """Copies args that must be compatible between two targets with a dependency relationship.
 
@@ -46,15 +64,26 @@ def add_migration_tag(attrs):
     Returns:
         The same `attrs` object, but modified.
     """
+    add_tag(attrs, _MIGRATION_TAG)
+    return attrs
+
+def add_tag(attrs, tag):
+    """Adds `tag` to `attrs["tags"]`.
+
+    Args:
+        attrs: dict of keyword args. It is modified in place.
+        tag: str, the tag to add.
+    """
     if "tags" in attrs and attrs["tags"] != None:
         tags = attrs["tags"]
 
         # Preserve the input type: this allows a test verifying the underlying
         # rule can accept the tuple for the tags argument.
         if types.is_tuple(tags):
-            attrs["tags"] = tags + (_MIGRATION_TAG,)
+            attrs["tags"] = tags + (tag,)
         else:
-            attrs["tags"] = tags + [_MIGRATION_TAG]
+            # List concatenation is necessary because the original value
+            # may be a frozen list.
+            attrs["tags"] = tags + [tag]
     else:
-        attrs["tags"] = [_MIGRATION_TAG]
-    return attrs
+        attrs["tags"] = [tag]

python/repositories.bzl

@@ -265,6 +265,7 @@ def _python_repository_impl(rctx):
 # Generated by python/repositories.bzl
 
 load("@bazel_tools//tools/python:toolchain.bzl", "py_runtime_pair")
+load("@rules_python//python/cc:py_cc_toolchain.bzl", "py_cc_toolchain")
 
 package(default_visibility = ["//visibility:public"])
 
@@ -336,6 +337,12 @@ py_runtime_pair(
     py2_runtime = None,
     py3_runtime = ":py3_runtime",
 )
+
+py_cc_toolchain(
+    name = "py_cc_toolchain",
+    headers = ":python_headers",
+    python_version = "{python_version}",
+)
 """.format(
         glob_exclude = repr(glob_exclude),
         glob_include = repr(glob_include),

tests/cc/BUILD.bazel

@@ -0,0 +1,45 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Tests for current_py_cc_headers rule
+
+load("@rules_testing//lib:util.bzl", "PREVENT_IMPLICIT_BUILDING_TAGS")
+load("//python/cc:py_cc_toolchain.bzl", "py_cc_toolchain")
+
+package(default_visibility = ["//:__subpackages__"])
+
+toolchain(
+    name = "fake_py_cc_toolchain",
+    tags = PREVENT_IMPLICIT_BUILDING_TAGS,
+    toolchain = ":fake_py_cc_toolchain_impl",
+    toolchain_type = "@rules_python//python/cc:toolchain_type",
+)
+
+py_cc_toolchain(
+    name = "fake_py_cc_toolchain_impl",
+    testonly = True,
+    headers = ":fake_headers",
+    python_version = "3.999",
+    tags = PREVENT_IMPLICIT_BUILDING_TAGS,
+)
+
+# buildifier: disable=native-cc
+cc_library(
+    name = "fake_headers",
+    testonly = True,
+    hdrs = ["fake_header.h"],
+    data = ["data.txt"],
+    includes = ["fake_include"],
+    tags = PREVENT_IMPLICIT_BUILDING_TAGS,
+)

tests/cc/current_py_cc_headers/BUILD.bazel

@@ -0,0 +1,17 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+load(":current_py_cc_headers_tests.bzl", "current_py_cc_headers_test_suite")
+
+current_py_cc_headers_test_suite(name = "current_py_cc_headers_tests")