feat(cc): introduce py_extension rule for C/C++ Python extension modules - #3973
Draft
rickeylev wants to merge 30 commits into
Draft
feat(cc): introduce py_extension rule for C/C++ Python extension modules#3973rickeylev wants to merge 30 commits into
rickeylev wants to merge 30 commits into
Conversation
...and sycophantic toasters are bad collaborators
…into py-extension
This builds on the existing `py-extension` branch. It adds support for platform and abi tags in the resulting library filename. The `:py_extension_test`, `:py_extension_analysis_tests`, and `:py_limited_api_tests` test targets in `//tests/cc/py_extension` now build and pass. Relates to #3283
…tension (#3875) Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain. ### Description This PR contains updates to the experimental `py_extension` implementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanism to delegate to Bazel's native `cc_shared_library` and `cc_library` rules. This PR targets the main repository's `py-extension` branch (not `main`). ### Key Changes #### 1. Platform Tag & ABI Tag Derivation * **Introduced `abi_tag` to `PyCcToolchainInfo`**: Added the `abi_tag` field to the `PyCcToolchainInfo` provider, populated by the `py_cc_toolchain` rule. It defaults to deriving the tag from `python_version` (e.g. `cpython-311`) for backward compatibility. * **Constraint-Based Detection**: Replaced legacy parsing of `cc_toolchain` CPU names with direct lookup in rules_python's central `PLATFORMS` registry using platform constraints. * **Limited API Configuration**: Updated how limited API (`.abi3.so`) suffixes are appended based on the toolchain configuration. #### 2. Compilation & Linking Delegation (Refactor to `cc_shared_library`) * **Macro Fusion**: Replaced the custom linking logic in the `py_extension` rule with a macro of the same name. It now accepts C/C++ source/header files directly (`srcs`, `hdrs`, `copts`, `defines`), implicitly wrapping them in a private `cc_library` under the hood. * **Consolidated Attributes**: Consolidated static linkage under standard `deps` (removing the redundant `static_deps` attribute) and aligned linker arguments with `cc_shared_library`'s `user_link_flags`. * **Wrapper Rule (`_py_extension_wrapper`)**: Added a lightweight, private rule that wraps the `cc_shared_library` output to: * Query the Python toolchain for platform/ABI tags to resolve PEP 3149 compliant filenames (e.g. `.cpython-311-x86_64-linux-gnu.so` or `.abi3.so`). * Create a cheap `symlink` from the CSL output to the PEP 3149 name. * Propagate `PyInfo` (for python rules) and `CcSharedLibraryInfo` (for dynamic C++ dependencies). * **ODR Validation**: Leveraging native `cc_shared_library` ensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains). #### 3. Runfiles and Data Support * **Runfiles Propagation**: Re-implemented runfiles collection using the repository's standard `builders.RunfilesBuilder` to support runtime assets (`data` attribute) and dynamic library dependency propagation. ### Verification & Testing * Added parity test cases in `dependency_graph_tests.bzl` to verify dynamic dependency chains and static sharing behavior. * Added runfiles validation in `py_extension_tests.bzl` to verify data asset propagation. * Updated limited API tests to match CSL output structure. * All 14 tests in `//tests/cc/py_extension/...` are compiling and passing. ────── TAG=agy CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Why: Merge the latest upstream changes into py-extension to keep the branch up to date and compatible with Starlarkified rules_cc in Bazel 9. How: - Resolve conflict in dev/pyproject.toml and remove stale docs/requirements.txt. - Explicitly load CcSharedLibraryInfo from rules_cc. - Use cc_common.link without main_output to avoid private API restrictions, naming the intermediate library with a leading underscore and .pyextimpl suffix. - Add missing bzl_library targets and fix header includes in tests.
Why: Integrate upstream/py-extension (#3875) platform detection logic and cc_shared_library macro wrapper architecture into our local branch. How: - Resolve conflict in py_extension_rule.bzl in favor of upstream wrapper implementation. - Preserve runfiles dynamic lookup test in py_extension_test.py. - Fix function docstring header formatting in py_cc_toolchain_rule.bzl.
…e type annotations
Why:
Mark py_extension as a public but volatile API in documentation and improve docstring parameter type readability.
How:
- Add volatile API inclusion block to py_extension macro docstring.
- Cross-reference py_library.imports using {attr}.
- Add {type} annotations for all parameters and remove redundant Optional wording.
…3971) Per feedback from #3875 , this PR makes a few changes: * Removed an unnecessary hasattr check on py_cc_toolchain * Renamed the arbitrary "os" and "cpu" identifiers to terms standardized in PEP 508. Also marks `py_extension` as experimental. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Why: The py_cc_toolchain_rule.bzl loads flags.bzl, but :flags was missing from the deps attribute of //python/private:py_cc_toolchain_rule, breaking starlark doc extraction in docs build test. How: Add :flags to deps of //python/private:py_cc_toolchain_rule in python/private/BUILD.bazel.
…brary Why: py_cc_toolchain_macro.bzl loads //python/private/pypi:pep508_env.bzl, but the dependency was missing from the py_cc_toolchain_macro target in //python/private:BUILD.bazel. How: Add //python/private/pypi:pep508_env to deps of py_cc_toolchain_macro in python/private/BUILD.bazel.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
rickeylev
marked this pull request as draft
July 29, 2026 03:48
…D file Why: CI failed because tests/cc/py_extension/BUILD.bazel failed Buildifier checks and python/private/cc/BUILD.bazel referenced invalid bzl_library target names from @rules_cc. How: - Run buildifier to format tests/cc/py_extension/BUILD.bazel. - Update py_extension_macro and py_extension_rule bzl_library deps in python/private/cc/BUILD.bazel to use correct target names.
Why: WORKSPACE mode builds failed on Buildkite because bzlmod-only target names were used for rules_cc bzl_library dependencies. How: Use @rules_cc//cc:core_rules and @rules_cc//cc/common in python/private/cc/BUILD.bazel, which are supported in both WORKSPACE mode and bzlmod mode.
Why: Linking C extension shared libraries (.dylib) on macOS failed with undefined Python C symbols because ld requires -undefined dynamic_lookup when symbols are resolved at runtime. How: Add default user_link_flags with -undefined dynamic_lookup for @platforms//os:osx and @platforms//os:macos in py_extension_macro.bzl.
Why: GitHub actions CI failed on the ruff format check due to an extra blank line in py_extension_pkg_test.py. How: Reformat tests/cc/py_extension/py_extension_pkg_test.py using ruff format.
Collaborator
Author
|
cc @rsartor-cmd : something that you might run into integrating on the protobuf side: 65886db tldr, some flags necessary for building on mac are missing (that commit adds them) |
Why: py_extension_bzl is superseded by the py_extension bzl_library target in //python/cc:BUILD.bazel. How: Remove the py_extension_bzl bzl_library target definition from python/cc/BUILD.bazel.
Why: pyelftools is ELF-specific and fails when parsing Mach-O (.dylib on macOS) or PE (.dll on Windows) dynamic libraries. How: Decorate test_inspect_elf with @unittest.skipIf(sys.platform != 'linux', ...) in tests/cc/py_extension/py_extension_test.py.
…ames Why: Analysis tests in py_extension_tests.bzl and py_cc_toolchain_tests.bzl failed on macOS because extension file basenames and toolchain target platform tags vary across operating systems. How: Set explicit platform_machine and sys_platform on fake_py_cc_toolchain_impl in tests/support/cc_toolchains/BUILD.bazel, and use file_path_matches pattern matching for extension filenames in tests/cc/py_extension/py_extension_tests.bzl.
Why: On macOS, Building ext_static under config_settings libc=musl still targets macOS (producing ext_static.cpython-311-darwin.so). How: Update test_musl_platform_impl in tests/cc/py_extension/py_extension_tests.bzl to match ext_static.cpython-311-*.
Why: MSVC link.exe failed to link C extension DLLs on Windows because no symbols were exported, causing LNK1104. How: Add /EXPORT:PyInit_<module_name> to user_link_flags for @platforms//os:windows in py_extension_macro.bzl.
Why: MSVC link.exe failed on Windows with LNK1120 unresolved externals because Python import libraries (.lib) were not included in cc_shared_library deps. How: Append @rules_python//python/cc:current_py_cc_libs to cc_shared_library csl_deps for @platforms//os:windows in py_extension_macro.bzl.
Why: MSVC link.exe failed with LNK1120 unresolved externals because the implicit _impl cc_library lacked Python import libraries on Windows. How: Add current_py_cc_libs to py_cc_headers_and_win_libs for Windows platform in py_extension_macro.bzl.
…encies Why: MSVC link.exe on Windows failed with LNK1120 unresolved externals because default exports_filter excluded external C import libraries (@...). How: Set default exports_filter to ["//...", "@..."] for @platforms//os:windows in py_extension_macro.bzl when not explicitly provided.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(cc): introduce py_extension rule for C/C++ Python extension modules
Building C/C++ Python extension modules currently requires manually
configuring complex
cc_binaryorcc_shared_librarytargets andsetting up platform, ABI3, and extension tags. The new
py_extensionrule provides a high-level Starlark API for compilingnative extensions that integrate seamlessly with
rules_pythontoolchains and
py_librarydependencies.Closes #3283.