Modernization roadmap: multi-phase init, stable ABI, free-threading, GC support, deprecated-API cleanup (discussion)

[devdanzin]

Summary

A discussion / tracking issue for five modernization topics that are all feasible on zstandard's current codebase but involve design trade-offs better agreed up-front before PR work begins. Each sub-topic could be its own issue + PR if you'd prefer smaller surfaces; grouping here so the full landscape is visible.

The five topics:

1. Multi-phase module init (sub-interpreter readiness).
2. Stable ABI (abi3) adoption — blocked only by one private API.
3. Py_MOD_GIL_NOT_USED is declared but structurally premature.
4. GC support missing on 15 types that store PyObject * fields.
5. Deprecated-API cleanup — small, mechanical.

1. Multi-phase module init

Current state: m_size = -1 + single-phase init blocks sub-interpreter use. 19 global PyTypeObject *, 1 global PyObject *ZstdError, and a features set all live in C globals rather than per-module state.

Migration path: PyModuleDef_Slot array with Py_mod_create / Py_mod_exec; types and other globals move into a per-module state struct, looked up via PyType_GetModuleByDef(...).

Effort: Medium. The global-to-state migration is the bulk of the work; slot adoption is mechanical once state is in place.

2. Stable ABI (abi3) feasibility

Current state: Close to feasible — only one real blocker.

• Heap types already in place via PyType_FromSpec.
• pythoncapi_compat.h already included.
• ~6 PyBytes_AS_STRING + list/tuple macros → replace with their non-limited equivalents (mechanical).
• _PyBytes_Resize is the actual blocker — private API used for copy-avoidance in safe_pybytes_resize.

Target: Py_LIMITED_API = 0x030A0000 (3.10 floor); guard _PyBytes_Resize with #ifndef Py_LIMITED_API, falling back to a copy; replace unlimited macros with limited-API equivalents.

Impact: Wheel matrix collapses from per-minor-version to one abi3 wheel per platform. Minor perf cost on _PyBytes_Resize fallback path (one extra copy per resize), which is only hit on the abi3 build.

3. Py_MOD_GIL_NOT_USED — premature?

The declaration asserts free-threading safety. The current structural reality:

• 19 global PyTypeObject * (not per-interpreter).
• No per-object locking (no Py_BEGIN_CRITICAL_SECTION).
• Single-phase init (see topic 1).
• CFFI backend has no FT story.

Options:

• (A) Gate the declaration behind topic 1 + a critical-section audit.
• (B) #if 0 it so maintainers flip it intentionally after the structural work lands.

Today the declaration is a promise the code doesn't fully keep. Free-threaded CPython users who load zstandard into a shared-type program can hit races on the global type pointers during multi-interpreter or multi-thread setup/teardown.

4. GC support missing — 15 types

15 of the 19 heap types store PyObject * fields (compressor refs, source refs, result refs, etc.) without Py_TPFLAGS_HAVE_GC. Reference cycles through these fields are uncollectible.

Not actively leaking today — the fields are typically short-lived — but user code that creates a cycle through one of these fields (e.g., a callback closure that retains the compressor) leaks until process exit.

Fix: Add Py_TPFLAGS_HAVE_GC to each spec; implement tp_traverse visiting every PyObject * field; tp_clear using Py_CLEAR on each.

Interaction: The separately-filed heap-type refcount issue migrates 4 factories from PyObject_New to tp_alloc; tp_alloc already handles GC registration if the type is flagged HAVE_GC. The two changes compose cleanly.

5. Deprecated-API cleanup

Small, mechanical; can be done in a single PR independent of the above.

• Remove 9 dead #if PY_VERSION_HEX < 0x03090000 guards in c-ext/bufferutil.c — the supported-Python floor is 3.9.
• Drop #include "structmember.h" (deprecated in 3.12; pythoncapi_compat.h supplies it).
• PyModule_AddObject → PyModule_AddObjectRef — covered in the separately-filed OOM-hardening issue.
• 16 PyObject_CallObject → PyObject_CallNoArgs / PyObject_Call — clarity; CallObject is still stable ABI, so this is cosmetic.
• PY_SSIZE_T_CLEAN is a no-op from 3.13; can be removed alongside the 3.13+ bump if/when it happens.
• pythoncapi_compat.h is included but largely unused — PyObject_HasAttrStringWithError, Py_XSETREF, and others in it could replace existing less-robust patterns (the HasAttrString → HasAttrStringWithError migration is in the separately-filed MemoryError-swallowing issue).

Filing as one discussion vs. 5 issues

Grouped here because these decisions are correlated: "where does zstandard want to be in 2-3 Python releases?". Multi-phase init is a precondition for (3) meaningfully; abi3 is orthogonal but valuable to plan alongside; GC and deprecated-API are standalone.

Happy to split into 5 separate issues + staged PRs if you prefer smaller surfaces for review. Also happy to take on any of these as a PR once you've indicated priority — my suggestion would be topic 5 first (small mechanical cleanup, low risk), topic 2 next (abi3 is high-leverage at modest cost), topic 4 (GC support), and save topics 1 + 3 for last since they interact and benefit from an aligned design.

Methodology

Found via cext-review-toolkit — the module-state, stable-abi, free-threading, type-slot, and version-compat scanners collectively covered these topics. All five items are static observations about the codebase's current shape vs. modern CPython C-extension conventions; no live reproducer is applicable.

Discovery, root-cause analysis, and issue drafting were performed by Claude Code and reviewed by a human before filing.

Full report

Complete multi-agent analysis (48 FIX findings across 13 categories, plus a reproducer appendix): https://gist.github.com/devdanzin/b86039ac097141579590c1a0f3a43605