pyemscripten/pyodide wheel builds#14787
Conversation
Enable building pyemscripten/pyodide wheels. Now that https://peps.python.org/pep-0783/ is accepted, Pyodide/PyEmscripten wheels are allowed to be published on PyPI. This enables the last stable release of the Pyodide/Pyemscripten ABI (2025_0), but, will be revved to pyemscripten_2026_0* when it is available. pyca-boringbot would need to be updated to automatically bump the OpenSSL version. Happy to add that after this.
|
We don't ship wheels for any platform for which we don't run tests, so that's the first step here if you're interested in pursuing this. |
Definitely interested seeing this through. Started #14788 to track discussion and remaining TODOs. Just pushed tests and looking to improve them. |
|
To be clear, when we say running the tests, we mean as a part of our CI workflow, on every PR, with acceptable performance. |
| - uses: dtolnay/rust-toolchain@e97e2d8cc328f1b50210efc529dca0028893a2d9 | ||
| with: | ||
| # Pyodide's emscripten build passes nightly-only `-Z` flags | ||
| # (link-native-libraries, emscripten-wasm-eh). |
There was a problem hiding this comment.
It should be possible to build on stable soon, since neither of these flags are needed any longer. It seems like it might still be passing link-native-libraries=yes somewhere which is the default but it's still nightly only.
There was a problem hiding this comment.
Wrote a TODO near this line and in ci.yml to make sure we validate if we still need nightly by the time pyodide 314.0 is released.
Also updated the issue task list to make sure we don't forget about this.
pypa/cibuildwheel#2812 was merged, so we no longer need the fork pin. Switch to pypa/cibuildwheel@main until the changes ship in a tagged release (latest is v3.4.1, which predates the merge).
3f4ee99 to
089badf
Compare
|
|
||
| - uses: ./.github/actions/upload-coverage | ||
|
|
||
| pyemscripten: |
There was a problem hiding this comment.
I've taken a read through of this, and here is my concern: this is basically a significant amount of configuration, that none of hte pyca/cryptography knows anything about. We don't know when versions should be bumped, don't know what compatibility looks like, don't know how to debug. It duplicates a ton of our other CI code as well.
For this to be a realistic thing we were willing to maintain, we need to get this down much closer to "this is a new entry in the existing linux matrix that shares the vast majority of the code with the other CI workflows"
There was a problem hiding this comment.
Thanks for the read. Maintainability concerns are fair. Two paths I'd like your call on before cutting code:
Option A — Fold into the linux matrix
New row:
- VERSION: "3.14"
NOXSESSION: "tests-pyemscripten"
OPENSSL: {TYPE: "pyemscripten", VERSION: "3.6.2"}
PYODIDE: {VERSION: "314.0.0a1", EMSDK: "5.0.3"}
RUST: "nightly"
RUST_TARGETS: "wasm32-unknown-emscripten"Pros: checkout/setup-python/rust-toolchain/cache/upload-coverage reuse linux's steps; YAML shrinks; pyemscripten appears as a row in the linux matrix.
Cons: linux job grows 4-5 if: matrix.PYTHON.PYODIDE branches (install + Tests must diverge — nox can't drive a pyodide venv since uv has no wasm build).
Option B — Standalone job, shrunk via shared building blocks
- New composite action
.github/actions/setup-pyodide-build/for thepyodide-build[resolve]+xbuildenv install+install-emscriptentrio. - Existing
build_openssl.sh(TYPE=pyemscripten already supported). - New
.github/bin/test_pyemscripten.shforpyodide build .→pyodide venv→ install wheel →pytest --cov. Same script runs locally. - Local reproducer (Containerfile + driver, already prototyped) so a maintainer can repro CI's pipeline with one command.
Pros: YAML shrinks; no conditional proliferation; unique logic in one action + one script, both grep-able and locally runnable.
Cons: still its own job rather than a row in the linux matrix.
I lean B — fewer conditionals, better local-debug story — but it's your call.
Independent of A vs B, on the maintainability gap: I'd add docs/development/pyemscripten.rst covering the Pyodide ↔ Emscripten ↔ Python compatibility triple (and where Pyodide pins it in Makefile.envs), bump cadence, and 4-5 failure-triage entries. I'll do this regardless of which option we go with.
What do you think?
There was a problem hiding this comment.
I think one option could be to just use cibuildwheel in all workflow files, as we ensure in our Pyodide platform code that the right Emscripten toolchain is picked up, and we won't need to worry about the pyodide-build version, etc. All dependencies are pinned/constrained, too, making it more reliable and reducing the bespoke configuration involved. The only thing to pick up and update would be based on changes between cibuildwheel releases, i.e., which Pyodide ABIs are available to build and whether to enable or disable them as needed. That said, I'd recommend building only for the 2026 ABI, as the 314.0.0 stable release is coming soon (issue, milestone). The second alpha is out as of two days ago.
First tagged release with cp314-pyodide_wasm32 / pyemscripten_2026_0_wasm32 support (pypa/cibuildwheel#2812). Replaces the git+main pin and unblocks reproducible wheel builds.
Enable building pyemscripten/pyodide wheels. Now that https://peps.python.org/pep-0783/ is accepted, Pyodide/PyEmscripten wheels (specifically the Pyemscripten ABI (2026_0) ) are allowed to be published on PyPI.
This should also remove the need for pyodide-recipies to maintain a cryptography wheel build script out of tree.
Remaning tasks / discussion in #14788