feat: Multiprocessing support in packaged desktop apps#6662
Open
ndonkoHenri wants to merge 20 commits into
Open
feat: Multiprocessing support in packaged desktop apps#6662ndonkoHenri wants to merge 20 commits into
ndonkoHenri wants to merge 20 commits into
Conversation
Python's multiprocessing (spawn/forkserver start methods and the resource tracker) launches child processes by re-executing sys.executable with a CPython command line — and in a flet-built app sys.executable is the app binary itself. Each worker therefore booted a full Flutter GUI instance that treated '-B' as a dev-mode page URL, never ran the spawn payload, and never exited: one stray window per worker, hung pools, and "resource_tracker: process died unexpectedly, relaunching" loops. Add macos/Runner/main.swift as the explicit entry point (AppDelegate drops @main; the xib still instantiates and wires the delegate): before NSApplicationMain, it checks argv against dart_bridge >= 1.5.0's serious_python_is_mp_invocation and, on a match, exits with serious_python_main's return code — the embedded interpreter services the child headlessly (Py_BytesMain) with no AppKit/Flutter initialization. The child resolves the bundled stdlib/site-packages via the PYTHONHOME/PYTHONPATH the parent already setenv'd process-wide. Both entry points are resolved via dlsym from the current process image (dart_bridge is a static archive force-loaded into the host binary), so apps built against an older dart_bridge still link and launch — they just don't get the interception. Verified end-to-end on macOS with an instrumented probe app: Process+Queue round-trip with a main.py-defined worker, and a ProcessPoolExecutor run (4 workers, reused across 8 tasks) — headless children, correct results, ~4x speedup over sequential, no leftover processes.
Same mechanism as the macOS runner: multiprocessing spawn children (and the resource tracker) re-execute sys.executable — the app .exe — with a CPython command line, which previously booted one GUI window per worker in dev-mode and never ran the spawn payload. At the top of wWinMain, before any console/COM/window/Flutter work, MaybeRunPythonChild() parses the wide command line (CommandLineToArgvW), loads dart_bridge.dll (falling back to dart_bridge_d.dll for debug-CRT builds), and resolves the wide-char entry points serious_python_is_mp_invocation_w / serious_python_main_w (dart_bridge >= 1.5.0; wide variants because decoding wWinMain argv through the ANSI code page would be lossy — serious_python_main_w wraps Py_Main). On a match, the process runs as a headless interpreter and returns its exit code. Resolution is dynamic (GetProcAddress with graceful fallthrough), so apps built against an older dart_bridge launch unchanged. Eagerly loading the DLL costs nothing on the normal startup path — the plugin loads it moments later anyway. Note: not yet runtime-verified on Windows (the macOS equivalent is verified end-to-end); needs a probe run on a Windows machine.
Same mechanism as the macOS/Windows runners, with an extra reason to
care on Linux: Python 3.14 changed the default start method from fork
to forkserver (gh-84559), and the forkserver's server process is
launched through the same sys.executable re-exec as spawn — so Linux
apps that previously happened to work via raw fork break by default on
the bundled 3.14.
At the top of main(), before any GTK/Flutter initialization,
maybe_run_python_child() dlopen()s libdart_bridge.so — resolved through
the runner's $ORIGIN/lib RUNPATH, exactly the way the Dart FFI's
DynamicLibrary.open() finds it later — and resolves
serious_python_is_mp_invocation / serious_python_main (dart_bridge >=
1.5.0). On a match, the process runs as a headless interpreter and
returns its exit code.
Resolution is dynamic with graceful fallthrough, so apps built against
an older dart_bridge launch unchanged. ${CMAKE_DL_LIBS} is added to the
runner link for dlopen/dlsym (a no-op with glibc >= 2.34, where libdl
is merged into libc).
Note: not yet runtime-verified on Linux (the macOS equivalent is
verified end-to-end); needs a probe run on a Linux machine.
…ssing at the host binary
Two independent multiprocessing breakages lived in the boot script, and
either alone was fatal even with the runners' child interception in
place:
1. __main__ identity. runpy.run_module(run_name="__main__") executes the
app module in a scratch namespace that is never installed in
sys.modules, so pickling any function defined in the app module
failed IN THE PARENT with "Can't pickle <fn>: it's not found as
__main__.<name>" — before a child was ever spawned. Replace it with
_sp_run_module_as_main(): resolve the module spec (with `python -m`
package semantics — pkg runs pkg.__main__ — and clear ImportErrors
for loaderless/codeless specs), build a fresh module with
__spec__/__file__/__cached__/__loader__/__package__ set, install it
as sys.modules["__main__"], and exec the module code in its dict.
Spawn children then re-import the app module as __mp_main__ via the
standard init_main_from_name path — which is why the documented
`if __name__ == "__main__":` guard around ft.run() is now mandatory
for multiprocessing users, exactly as in plain CPython.
The module is also aliased as sys.modules["__mp_main__"] in the
parent, matching what multiprocessing does in children: objects
defined in the app module and pickled by a child carry
__module__ == "__mp_main__", and without the alias the parent fails
to unpickle them (ModuleNotFoundError: __mp_main__).
2. Re-exec target. sys.executable / sys._base_executable are set to the
host binary (new {host_executable} placeholder, filled from
Platform.resolvedExecutable in native_runtime.dart; JSON string
literals are valid Python string literals). On macOS/Windows CPython
already computes the host binary itself, but on Linux bare
Py_Initialize() PATH-guesses an unrelated "python3" (or none) — this
makes the target the runner binary, whose argv interception services
the children, on all three desktops deterministically. Set before
any user import because multiprocessing snapshots sys.executable at
import time.
PYTHONINSPECT is also dropped from the inherited environment: it did
nothing in the embedded parent, but a real interpreter child would
stay open in interactive mode after its -c command completed.
(Defense in depth — serious_python >= 4.3.0 stops setting it and
dart_bridge's serious_python_main unsets it too.)
Verified end-to-end on macOS together with the runner interception; the
rendered boot script is byte-checked to compile after all placeholder
substitutions.
Start every new project with the standard entry-point guard:
if __name__ == "__main__":
ft.run(main)
With multiprocessing now working in packaged desktop apps, spawn
children re-import the app's main module (as __mp_main__) exactly like
plain CPython — an unguarded ft.run() would start a whole new app
session in every worker process. The guard has always been Python best
practice; the scaffold now models it so multiprocessing users don't
learn it the hard way.
Also type the scaffold's event handler parameter
(e: ft.Event[ft.FloatingActionButton]) to model typed event handlers.
JSON string/array literals are valid Python literals, so jsonEncode gives
correct escaping for free. The previous hand-rolled escaping was uneven:
{outLogFilename} doubled backslashes but not quotes, and {argv} escaped
quotes but not backslashes — a Windows-style path in either would corrupt
the generated Python source. {host_executable} already used jsonEncode;
now all three share the one convention. Empty argv still renders as [""]
(CPython always has a sys.argv[0]).
Verified: boot script rendered through the real Dart substitution with
hostile inputs (backslashes, quotes, non-ASCII) compiles as valid Python
and every value round-trips exactly; full flet build macos + the
multiprocessing probe suite passes unchanged.
New cookbook page documenting multiprocessing support in Flet apps, now that packaged desktop apps service the spawn re-exec protocol: - when to reach for processes vs async/threads, and the platform/version support matrix (desktop-only, Flet >= 0.86.0; iOS/Android/browser unsupported); - the rules that are standard Python multiprocessing discipline but MANDATORY in packaged apps: the `if __name__ == "__main__":` guard (spawn/forkserver children re-import the main module), importable and picklable worker functions (top-level, plain data, no controls/page/ lambdas), and no GUI access from workers; - how it works in a `flet build` app: the embedded interpreter, the app binary recognizing CPython helper command lines and running them as a headless interpreter, and the practical consequences — sys.executable points at the app binary by design (don't set_executable() over it), freeze_support() is unnecessary-but-harmless, worker stdout isn't the app console log, and forcing fork on Linux is unsafe under a running Flutter engine; - a runnable ProcessPoolExecutor example (parallel chunk sorting with live progress) driven off the UI thread via page.run_thread. Listed in the cookbook sidebar after Subprocess.
The boot script opened the console-log tee file without an explicit encoding, so Windows used the locale codepage (e.g. cp1252) and the first non-ASCII character an app printed raised UnicodeEncodeError inside the stdout tee — crashing the printing thread. macOS/Linux never hit it because their default filesystem encoding is UTF-8 already. Found while verifying the multiprocessing fix on a Windows VM: the demo app's status line contains "→" and its print() died mid-benchmark. errors="backslashreplace" additionally guarantees that even malformed data (e.g. lone surrogates) degrades to an escaped representation instead of ever breaking the log.
flet/version.py resolves the package version at import time; in a source checkout (empty baked flet_version) it falls back to `git describe`. On Windows, git.exe is a console-subsystem program, so spawning it from a windowless GUI process pops a visible conhost window. In a dev-built app this meant one console flash at app start — and, with multiprocessing now working, one more flash per spawned worker, since every spawn child re-imports flet and re-runs the fallback. Traced on a Windows VM via Win32_ProcessStartTrace: each worker pid parented a git.exe (+ conhost.exe) at click time. Pass CREATE_NO_WINDOW on Windows so the fallback stays invisible. Released packages are unaffected either way (CI bakes flet_version, so the git path never runs).
Enhance docstrings for `debug`, `test`, `create`, `build`, `run`, and `publish` CLI commands with links to detailed usage guides and examples from the Flet documentation.
…port, and usage guidance
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Deploying flet-website-v2 with
|
| Latest commit: |
66f6dec
|
| Status: | 🚫 Build failed. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds end-to-end support for Python multiprocessing in packaged Flet desktop apps by intercepting multiprocessing child re-execs in the native runners and making the embedded Python bootstrap behave like a real python -m __main__ module (including __mp_main__ aliasing), plus supporting docs/examples and a couple of Windows-specific fixes found during validation.
Changes:
- Add early “multiprocessing child invocation” detection in macOS/Windows/Linux runners to run the embedded interpreter headlessly instead of launching additional GUI instances.
- Update the embedded Python bootstrap script to execute the app module as the real
sys.modules["__main__"](and__mp_main__), setsys.executable, sanitize env, and harden console logging. - Add cookbook documentation + runnable multiprocessing examples and a changelog entry.
Reviewed changes
Copilot reviewed 26 out of 35 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| website/sidebars.yml | Adds cookbook nav entry for the new multiprocessing page. |
| website/docs/cookbook/multiprocessing.md | New cookbook page describing supported platforms, required patterns, and examples. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/windows/runner/main.cpp | Early argv-based interception for multiprocessing child re-execs via dart_bridge.dll. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/macos/Runner/main.swift | New explicit entry point to intercept multiprocessing child re-execs before AppKit/Flutter. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/macos/Runner/AppDelegate.swift | Removes @main usage to defer the entry point to main.swift. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/macos/Runner.xcodeproj/project.pbxproj | Registers main.swift in the macOS Runner Xcode project. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/linux/main.cc | Early interception for multiprocessing child re-execs via dlopen/dlsym. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/linux/CMakeLists.txt | Links ${CMAKE_DL_LIBS} to support the new dlopen usage. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/lib/python.dart | Reworks bootstrap to run app module as real __main__/__mp_main__, set sys.executable, and make console log UTF-8. |
| sdk/python/templates/build/{{cookiecutter.out_dir}}/lib/native_runtime.dart | Uses jsonEncode for safer boot-script value splicing and injects host executable path. |
| sdk/python/templates/app/app/{{cookiecutter.out_dir}}/src/main.py | Adds if __name__ == \"__main__\": guard in scaffold to be multiprocessing-safe. |
| sdk/python/packages/flet/src/flet/version.py | Prevents console-window flashes on Windows when running git describe fallback. |
| sdk/python/examples/cookbook/multiprocessing/worker_progress.py | New example demonstrating worker-to-UI progress reporting via multiprocessing.Queue. |
| sdk/python/examples/cookbook/multiprocessing/persistent_worker.py | New example showing a long-lived worker process using a Pipe. |
| sdk/python/examples/cookbook/multiprocessing/parallel_sort.py | New example demonstrating a ProcessPoolExecutor with UI progress updates. |
| sdk/python/examples/cookbook/multiprocessing/cancel_task.py | New example showing cancellation via Process.terminate(). |
| CHANGELOG.md | Adds a release note entry for packaged-desktop multiprocessing support. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+108
to
+115
| // JSON literals are valid Python literals, so every dynamic value is | ||
| // spliced into the boot script through jsonEncode: it correctly escapes | ||
| // backslashes (Windows paths), quotes, and non-ASCII. | ||
| var script = pythonScript | ||
| .replaceAll("{outLogFilename}", outLogFilename.replaceAll("\\", "\\\\")) | ||
| .replaceAll('{outLogFilename}', jsonEncode(outLogFilename)) | ||
| .replaceAll('{module_name}', moduleName) | ||
| .replaceAll('{argv}', argv); | ||
| .replaceAll('{argv}', jsonEncode(args.isNotEmpty ? args : [""])) | ||
| .replaceAll('{host_executable}', jsonEncode(Platform.resolvedExecutable)); |
The jsonEncode refactor's comment promised "every dynamic value is
spliced into the boot script through jsonEncode", but {module_name} was
still inserted raw inside hand-written quotes. No live bug — the module
name must already survive the cookiecutter render and find_spec(), so
hostile values can't reach it — but the exception made the comment
wrong and left a footgun for future entry-point changes (e.g. dotted or
path-like module names).
Encode moduleName like the other values; the Python template now calls
_sp_run_module_as_main({module_name}) without surrounding quotes, since
the JSON string literal brings its own.
Addresses the review note on #6662 (discussion_r3544535152). The
rendered boot script is byte-checked to compile after all substitutions.
calc_worker still carried two print() calls from local testing. They would also be misleading in a packaged app, where worker stdout isn't connected to the app's console log — which the cookbook page itself points out.
Moving the v0.85/v0.86 breaking-change pages into v0-85-0/ and v0-86-0/ subfolders (876a006) was a pure rename, so every relative link inside them started resolving one directory too shallow and the site build failed its broken-links check ("Docusaurus found broken links!"): - "](.)" pointed at the (non-existent) version-folder index instead of the breaking-changes index -> now ../index.md - ../release-notes.md -> ../../release-notes.md - ../../{cli,reference,cookbook}/... -> ../../../{cli,reference,cookbook}/... All nine moved pages updated; every relative .md link target verified to exist, and a full local `docusaurus build` passes the broken-links check again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4283. Companion PRs: flet-dev/dart-bridge#9 (the headless-interpreter entry points) and flet-dev/serious-python#228 (env fix, dead-strip keep-alives, staging fix).
Root cause
multiprocessingwas broken inflet builddesktop apps in four independent ways, each fatal on its own.sys.executable, which in a packaged app is the Flutter host binary (measured on macOS/Windows; on Linux barePy_InitializePATH-guesses an unrelatedpython3instead, which then dies at interpreter init on the bundle's inheritedPYTHONHOMEpointing at a bytecode-only stdlib of a different version).-c "from multiprocessing.spawn import spawn_main;..."payload was silently swallowed while a full GUI booted, giving one stray window per worker that never computed and never exited.runpy.run_module(run_name="__main__"), which executes in a scratch namespace never installed insys.modules, so pickling any function defined in the app'smain.pyfailed in the parent withPicklingError: not found as __main__.<name>before a child was even spawned.multiprocessing.freeze_support()cannot help (sys.frozenis unset and the child never reached Python anyway), and no python executable ships in the bundle forset_executable()to point at. Python 3.14 raised the stakes on Linux by making forkserver (which uses the same re-exec) the default start method.Changes
Native runners (build template): all three desktop runners now check argv before any UI/engine initialization — a new
macos/Runner/main.swiftbeforeNSApplicationMain, the top ofwWinMaininwindows/runner/main.cpp(wide-char variants,LoadLibraryWof the adjacentdart_bridge.dll), and the top ofmaininlinux/main.cc(dlopen("libdart_bridge.so")through the runner's$ORIGIN/libRUNPATH, with${CMAKE_DL_LIBS}added to the link). When dart_bridge >= 1.5.0'sserious_python_is_mp_invocationmatches, the process runs as a headless interpreter viaserious_python_mainand exits with its code; resolution is dynamic with graceful fallthrough, so apps built against older dart_bridge launch unchanged (the fix just stays dormant).Boot script (
python.dart): the app module now runs withpython -msemantics as the realsys.modules["__main__"](spec/loader/__file__/__cached__set, package case handled, clear ImportErrors for loaderless/codeless specs), which fixes parent-side pickling; it is also aliased assys.modules["__mp_main__"], matching what multiprocessing does in children, so objects defined in the app module survive the child-to-parent pickle direction.sys.executableandsys._base_executableare set to the host binary via a new{host_executable}splice fromPlatform.resolvedExecutable(required for determinism on Linux, hardening elsewhere), andPYTHONINSPECTis dropped from the inherited environment. All dynamic boot-script values are now spliced viajsonEncode. Because spawn children re-import the main module (as__mp_main__) exactly like plain CPython, the standardif __name__ == "__main__":guard aroundft.run()is now required for multiprocessing users — theflet createscaffold models it, and the docs call it out.Fixes found while verifying on VMs:
console.logis now opened as UTF-8 witherrors="backslashreplace"(Windows previously used the locale codepage, so the first non-ASCII character an app printed raisedUnicodeEncodeErrorinside the stdout tee), and the dev-modegit describeversion fallback passesCREATE_NO_WINDOWon Windows (each multiprocessing child re-imports flet, and in a source checkout every git invocation from the windowless GUI process flashed a console window; released packages never run that path).Docs and examples: new cookbook page (
website/docs/cookbook/multiprocessing.md) covering when to use processes vs threads/async, the platform support matrix, the mandatory rules, how the packaged-app mechanism works, and practical notes (sys.executablepoints at the app binary by design,freeze_support()is unnecessary but harmless, noforkon Linux under a running Flutter engine), plus four runnable examples undersdk/python/examples/cookbook/multiprocessing/(parallel sort with progress, worker progress reporting, cancelling a running task, persistent worker process). Changelog entry included. The branch also carries a small unrelated docs commit adding usage-guide links to CLI command docstrings.Verification
Before/after apps (identical source) were packaged and run on all three desktops. macOS arm64: before shows the classic window-per-worker pile-up; after passes everything. Windows 11 (arm64 host, x64 app): after passes with an approximately 3.5x pool speedup and headless children; this run surfaced the UTF-8 and console-flash fixes above. Ubuntu 22.04 aarch64: before fails with PATH-python children dying at init (
bad magic number in 'encodings', the pyc version-mismatch shape) and parent-side PicklingErrors; after passes withsys.executablecorrectly pointing at the app binary, under both the spawn context and the 3.14 forkserver default. In all after-runs: main.py-defined workers pickle and round-trip, pools complete with worker reuse, the resource tracker performs semaphore cleanup, no extra windows appear, and no processes leak.Before
After
Activation and scope
The runner interception activates once the template's serious_python pin reaches 4.3.0 (dart_bridge 1.5.0); that pin bump follows the release chain in flet-dev/dart-bridge#9 and flet-dev/serious-python#228 and is not part of this PR. Development mode (
flet run) always worked (a real interpreter runs the script) and is unaffected. Mobile remains unsupported by OS design (no child process spawning on iOS/Android); web is unaffected.Summary by Sourcery
Add full support for Python multiprocessing in packaged Flet desktop apps by intercepting child process invocations in native runners and running the embedded interpreter headlessly, while updating the Python bootstrap to behave like a real main module and tightening logging and argument handling.
New Features:
Bug Fixes:
Enhancements:
Documentation: