Conversation
luhenry
changed the title
pyarrow is Cython bindings over the Apache Arrow C++ libraries. Build the Arrow C++ libs once from source with bundled third-party deps (ARROW_DEPENDENCY_SOURCE=BUNDLED — the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then build a wheel per Python (cp312/313/314/314t) against that shared C++ build. Mirrors apache/arrow's ci/scripts/python_wheel_xlinux_build.sh, minus vcpkg, and tests like ci/scripts/python_wheel_unix_test.sh (pytest --pyargs pyarrow, feature-gated via PYARROW_TEST_*). Feature set (workflow env, single source of truth): Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression. S3/GCS/Azure/Gandiva off for now — their dep trees (aws-sdk/google-cloud-cpp/azure-sdk/LLVM) are unverified on riscv64 and will be enabled incrementally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
With the OpenSSL fix in, the build got 1h37m deep and then failed compiling bundled c-ares: "redefinition of hash_func/bucket_key/bucket_free/key_eq", all under c-ares.dir/Unity/unity_*.c. CMAKE_UNITY_BUILD=ON (copied from upstream) merges each target's .c files into a single translation unit; c-ares reuses the same file-local `static` symbol names across ares_htable_vpvp.c / ares_htable_vpstr.c / etc. — fine per-file, a redefinition once unity-merged. Upstream never hits this because their c-ares comes prebuilt from the vcpkg cache; our BUNDLED build compiles it from source, and Arrow adds a unity-OFF guard for re2/grpc/protobuf but not for c-ares. Unity build is only a compile-speed optimization, so disable it globally — robust across the whole bundled dep tree, which will keep including deps upstream never had to make unity-clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `: "${VAR:=default}"` idiom traces under `set -x` as a bare `+ : ON`,
with no hint which flag `ON` belongs to — so the build log's feature-flag
dump was an unreadable column of `+ : ON` / `+ : OFF`. Switch to
`VAR="${VAR:-default}"` (and self-assignment for the `:?` required-var
guards): identical semantics — default applied when unset, hard fail when a
required var is missing — but the trace now reads `+ ARROW_DATASET=ON`,
naming each variable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
luhenry
marked this pull request as draft
August 21, 2026 14:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.github/workflows/build-pyarrow.ymlto build riscv64 pyarrow wheels (25.0.1) and publish them topypi.riseproject.dev.pyarrow is Cython bindings over the Apache Arrow C++ libraries, so this is a heavy build: the Arrow C++ libs are built once from source with bundled third-party deps (
ARROW_DEPENDENCY_SOURCE=BUNDLED— the manylinux riscv64 image has no vcpkg binary cache, unlike upstream's x86/arm images), then a wheel is built per Python against that shared C++ build. This mirrors apache/arrow's ownci/scripts/python_wheel_xlinux_build.sh(minus vcpkg) and tests likeci/scripts/python_wheel_unix_test.sh.Shape
docker run(not cibuildwheel): pyarrow's C++ sources are a sibling of the Python package, which cibuildwheel's copy-package-dir model can't handle, and the manylinux riscv64 image ships no Node (so acontainer:job can't run JS actions). Checkout/upload run on the riscv host; the heavy build runs indocker runinsidequay.io/pypa/manylinux_2_39_riscv64on the native riscv64 runner.cp312 / cp313 / cp314 / cp314tbuilding wheels (scikit-build-core,PYARROW_BUNDLE_ARROW_CPP=ON), thenauditwheel repair --strip.pytest -r s --pyargs pyarrow, feature-gated viaPYARROW_TEST_*.Feature set (workflow
env, single source of truth)ON: Parquet, Dataset, Acero, Compute, CSV, JSON, Filesystem, HDFS, ORC, Substrait, Flight, Parquet-encryption + all compression (snappy/zlib/zstd/lz4/brotli/bz2), mimalloc.
OFF (enabled incrementally later): S3, GCS, Azure, Gandiva, jemalloc, OpenTelemetry, TensorFlow. The network-storage stacks (aws-sdk / google-cloud-cpp / azure-sdk) and LLVM-based Gandiva have dep trees unverified on riscv64; enabling them is a one-line flip in the
envblock per feature.Notes for review
-DARROW_OPENSSL_USE_SHARED=ONis pinned — without it,ARROW_DEPENDENCY_USE_SHARED=OFFcascades to a static-OpenSSL lookup that fails and breaks the bundled gRPC/parquet link. (Found and fixed via a riscv64 qemu configure check.)numpy/libcst/pandasare pulled frompypi.riseproject.dev(no public-PyPI riscv64 wheels);pandasis optional in the test step and gates the pandas-dependent tests (matters for cp314t, which may lack a free-threaded pandas wheel).🤖 Generated with Claude Code