Migrate pybind11 to nanobind

[justinchuby]

Migrate Python bindings to nanobind and align abi3 packaging

• replace pybind11 with nanobind v2.13.0 in CMake and dependency manifest
• port pyop bindings and C API wrappers from pybind11 APIs to nanobind APIs
• update wheel build matrices to build cp310/cp311/cp312 only for abi3 strategy
• add py_limited_api metadata and wheel tagging for cp312+ stable ABI
• add abi3audit checks (cp312 baseline) in cibuildwheel test scripts

[justinchuby]

@copilot please fix:

E       onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: /mnt/azure_nvme_temp/_work/1/s/onnxruntime_extensions/_extensions_pydll.abi3.so: undefined symbol: RegisterCustomOps

test_string_ops.py:978: Fail
_________________ TestPythonOpString.test_string_split_cc_sep0 _________________

self = <test_string_ops.TestPythonOpString testMethod=test_string_split_cc_sep0>

    def test_string_split_cc_sep0(self):
        so = _ort.SessionOptions()
>       so.register_custom_ops_library(_get_library_path())
E       onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Failed to get symbol RegisterCustomOps with error: /mnt/azure_nvme_temp/_work/1/s/onnxruntime_extensions/_extensions_pydll.abi3.so: undefined symbol: RegisterCustomOps

test_string_ops.py:1079: Fail

[Copilot]

Pull request overview

This PR migrates the project’s Python extension module from pybind11 to nanobind and adjusts packaging/CI to support an abi3-oriented wheel strategy (cp310–cp312) with additional verification for stable-ABI compliance.

Changes:

• Port C++ binding code from pybind11 to nanobind for the _extensions_pydll module (including the optional C API wrapper).
• Update CMake external dependency wiring from pybind11 to nanobind and build the Python module via nanobind_add_module.
• Adjust wheel build matrices to drop cp313 and add runtime checks (including abi3audit on Python ≥ 3.12).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/test_cibuildwheel.sh	Adds import smoke test and runs abi3audit on Python ≥ 3.12.
tools/test_cibuildwheel.bat	Windows equivalent of the import smoke test + abi3audit gating.
tools/ci_build/github/azure-pipeline/wheels_macos.yml	Narrows cibuildwheel build targets to cp310–cp312.
tools/ci_build/github/azure-pipeline/wheels_linux.yml	Narrows cibuildwheel build targets to cp310–cp312 (x86_64 + aarch64).
tools/ci_build/github/azure-pipeline/templates/windows-build-stage.yml	Narrows Windows cibuildwheel build targets to cp310–cp312.
setup.py	Adds limited-API metadata/macros (gated) and sets python_requires>=3.10.
pyop/pykernel.h	Updates C-API module signature to nanobind and forward-declares nanobind::module_.
pyop/pyfunc.cc	Migrates main binding module to nanobind; refactors numpy/tensor conversions and invocation plumbing.
pyop/py_c_api.cc	Migrates C API Python wrapper bindings to nanobind; refactors numpy conversion.
cmake/externals/pybind11.cmake	Removes pybind11 FetchContent integration.
cmake/externals/nanobind.cmake	Adds nanobind FetchContent integration (v2.13.0).
cmake/ext_python.cmake	Switches to find_package(Python ...) + nanobind_add_module (stable ABI).
cgmanifest.json	Replaces pybind11 component entry with nanobind (v2.13.0).
.pyproject/cmdclass.py	Adds bdist_wheel customization to tag cp312 abi3 wheels when applicable.
.pipelines/wheels_win32.yml	Narrows pipeline wheel targets to cp310–cp312.
.pipelines/wheels_macos.yml	Narrows pipeline wheel targets to cp310–cp312.
.pipelines/wheels_linux.yml	Narrows pipeline wheel targets to cp310–cp312 (x86_64 + aarch64).
.pipelines/templates/onebranch-windows-build-stage.yml	Narrows OneBranch Windows wheel targets to cp310–cp312.
.github/workflows/linux_arm64_wheel.yaml	Updates GH Actions arm64 wheel matrix to Python 3.10–3.12.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@copilot please fix:

...

Fixed in 25d7636. I exported RegisterCustomOps (and GetActiveOrtAPIVersion) with default visibility in include/onnxruntime_extensions.h, then verified _extensions_pydll.abi3.so exports RegisterCustomOps and re-ran test/test_string_ops.py -k string_split_cc_sep0 successfully.

[justinchuby]

@copilot find all python 3.8 CI envs and update them to 3.10

[Copilot]

@copilot find all python 3.8 CI envs and update them to 3.10

Updated the remaining Python 3.8 CI environment to Python 3.10 in tools/ci_build/github/linux/docker/Dockerfile.ubi8_cuda11_8_tensorrt8_6 (commit b95ca64). I also verified there are no 3.8/cp38 references left under .github/workflows and tools/ci_build/github.

[justinchuby]

@copilot for some reason I still get + python3 -m pip install --upgrade pip Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (25.0) in [onnxruntime-extensions.CI (LinuxCUDABuilds LinuxGPU)](https://github.com/microsoft/onnxruntime-extensions/pull/1089/checks?check_run_id=84849656132) etc. is it because the python call was not pointinng to the py3.10 bin, or that the image hasn't been updated yet?

[Copilot]

@copilot for some reason I still get `+ python3 -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /usr/...

It was the image content, not the python3 invocation. The LinuxGPU job uses tools/ci_build/github/linux/docker/Dockerfile.ubuntu_cuda11_8_tensorrt8_6 (from .pipelines/ci.yml), and that Dockerfile was still installing default Ubuntu 20.04 Python (3.8). I fixed that Dockerfile to install/use Python 3.10 in commit d823c80. The next CI run should rebuild/pull the updated image and python3 -m pip should resolve to 3.10.