python3t.dll living in abi3t_compat needs special handling for Python embedders using the MSI install

[ngoldbaum]

Bug report

Bug description:

Currently, any abi3t extension that is loaded by CPython automatically handles this inside the dynamic extension loader:

cpython/Python/dynload_win.c

Lines 222 to 227 in 038495d

	#if defined(PY3_DLLNAME) && PY_MAJOR_VERSION==3 && PY_MINOR_VERSION==15
	/* GIL-enabled builds of 3.15 might have a python3t.dll adjacent that is for
	a free-threaded build. So we first check in a subdirectory in that case.
	If it's not there, we'll look adjacent. */
	#define ABI3T_COMPAT_DLLNAME L"abi3t-compat\\" ABI3T_DLLNAME L".dll"
	#endif

However, embedders that statically link python3t.lib can't use this code path. This includes use-cases like the PyO3 cargo unit tests, which link with libpython but are not run inside a running interpreter.

See for example this PyO3 CI job which is hitting the issue.

This fails with the following error:

cargo test --no-fail-fast --features=abi3t --no-default-features --all-targets
     Compiling pyo3-ffi v0.28.3 (D:\a\pyo3\pyo3\pyo3-ffi)
     Compiling pyo3 v0.28.3 (D:\a\pyo3\pyo3)
      Finished `test` profile [unoptimized + debuginfo] target(s) in 13.80s
       Running unittests src\lib.rs (target\debug\deps\pyo3-6e64eab80964a6ee.exe)
  error: test failed, to rerun pass `--lib`
  
  Caused by:
    process didn't exit successfully: `D:\a\pyo3\pyo3\target\debug\deps\pyo3-6e64eab80964a6ee.exe` (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)

The discussion about the abi3t-compat folder placement (gh-148690) didn't consider this case unfortunately and I also somehow missed it in my testing before beta2.

One practical way we can minimize the blast radius here is by only creating the abi3t_compat folder for the truly mixed ft+gil-enabled install and otherwise installing python3t.dll next to python3.dll for a GIL-enabled-only MSI install. That will fix the default case using github actions setup-python, for example.

If we make no other changes, we also need to add some coverage of this issue to the abi3t HOWTO, advising to add the necessary folder to the PATH on windows: https://docs.python.org/3.15/howto/abi3t-migration.html

cc @zooba @encukou @chris-eibl

CPython versions tested on:

3.15

Operating systems tested on:

Windows

[zooba]

Can we instruct embedders to only embed the variant they want? Why are embedders trying to distribute both free-threaded and GIL-ful binaries in the same directory? They can choose one and it should be fine.

[ngoldbaum]

Can we instruct embedders to only embed the variant they want? Why are embedders trying to distribute both free-threaded and GIL-ful binaries in the same directory? They can choose one and it should be fine.

That seems fine to me, along with documenting that "just add Python's install directory to PATH" is no longer sufficient.

[zooba]

Well, it'll be sufficient for a Python install manager install, it's just not sufficient for 3.15 with the legacy installer.

[zooba]

Adding to PATH and loading a DLL isn't sufficient anyway - more than one Windows configuration will ignore PATH for DLL loads - you need to use the full path to the DLL.