Multi-session/connection support with a Worker-backed PGlite postmaster

[samwillis]

Multi-session/connection support

This PR adds an opt-in, Node-only architecture for running PGlite with PostgreSQL's normal postmaster and multiple real backend sessions. It is the first implementation of multi-session/connection support: concurrent clients get independent PostgreSQL backends, transaction state, session state, locks, notifications, cancellation, and protocol connections rather than multiplexing requests through the existing single-user runtime.

The existing PGlite constructor and classic single-user Wasm artifact remain available and unchanged as a separate runtime. The new architecture is selected explicitly with PGlitePostmaster.create(), and sessions expose the normal PGlite interface.

Architecture

PostgreSQL processes mapped to Workers

• Each PostgreSQL process is one Node Worker with its own Wasm instance and function table.
• PostgreSQL uses its EXEC_BACKEND process-start path, analogous to the non-fork() model used on Windows. Backends and auxiliary processes start cleanly and reattach to shared cluster state.
• A Node supervisor owns Worker lifecycle, synthetic PIDs, process registries, memory domains, signal routing, filesystem coordination, and session creation.
• Signals, latches, waits, process exit, and wakeups are carried through control SharedArrayBuffers and Atomics.wait/notify without unwinding active Wasm stacks.

Multi-memory model

The postmaster build separates PostgreSQL address-space lifetimes into three Wasm memory domains:

1. Backend-private memory — independently owned by one PostgreSQL process and reclaimable when its Worker exits.
2. Cluster-global shared memory — shared buffers, lock tables, process arrays, WAL coordination, and other cluster-wide state.
3. Root/scoped shared memory — state shared by a backend group, including hierarchical session, transaction, query, and parallel-operation scopes.

The build retains PostgreSQL's 32-bit C pointers using a tagged memory32 ABI. A pinned Binaryen post-link transformer rewrites scalar, SIMD, bulk-memory, and atomic memory operations to the correct Wasm memory. Pointer-provenance analysis specializes known-private and known-shared accesses, while a sound generic dispatch remains for unknown provenance. Host calls decode tagged pointers through the PGlite libc boundary.

All compiler, transformer, auditor, archive, and wrapper-generation tooling runs in the pinned Docker build image. The currently qualified builder path is native Linux ARM64.

Connections, server, and filesystem

• Postmaster sessions use bounded shared protocol rings and present the normal PGliteInterface surface.
• @electric-sql/pglite-server provides the Node TCP/Unix-socket orchestration layer, and the pglite distribution exposes initdb, server/postgres, psql, dump/restore, and administration commands.
• Each external socket connection owns a real PostgreSQL backend, including native libpq cancellation and backpressure behavior.
• Direct NODEFS is supported first. The VFS design remains pluggable through cloneable factories or a filesystem broker for non-cloneable third-party implementations; WasmFS is not required.

Extension artifacts

Native extensions are selected by an explicit runtime target rather than loading the nearest compatible-looking module. The forward-compatible matrix covers pointer width and memory topology, while this PR completes only the wasm32-initial profile:

• wasm32-classic
• wasm32-multi-memory

Artifacts have generated descriptors, manifests, ABI identities, hashes, bounded archive extraction, atomic publication, lifecycle hooks, dependency/preload ordering, and actual-Wasm validation. Dynamic side modules are transformed and audited at build time; no client-side Wasm rewriting is required. Vector and PostGIS are the initial qualified extension inventory.

What is implemented

• PGlitePostmaster.create() and normal PGlite-compatible session objects.
• One Worker/Wasm instance per postmaster, backend, auxiliary process, and supported background/parallel worker.
• PostgreSQL EXEC_BACKEND startup and parameter transport.
• Private, global, and scoped shared memory, tagged pointers, scope registries, and memory high-water telemetry.
• Worker process control, signals, timers, latches, semaphores, cancellation, crash cleanup, and restart handling.
• Concurrent SQL sessions with transaction isolation, locks, deadlock/cancellation behavior, LISTEN/NOTIFY, COPY, and connection churn coverage.
• Parallel-query/root-scope proof paths and dedicated or compact scoped-memory modes.
• Pluggable Node filesystem handling, including direct NODEFS and brokered filesystems.
• TCP and Unix-socket frontend, native libpq clients, CLI lifecycle, and embedded initdb tooling.
• Deterministic multi-memory build, Binaryen transformer, provenance optimization, module auditing, source-map handling, and native ARM64 build image.
• Exact-target extension registry, safe materialization, lifecycle/dependency handling, third-party build commands, and two-target Vector/PostGIS artifacts.
• PostgreSQL regression-provider integration for make check and make check-world, with an explicit capability ledger for unsupported and blocked upstream cases.
• The main repository records the exact postgres-pglite commit containing the minimal fork-side build and PGlite-libc changes.

Validation

• PostgreSQL core regression schedule: 230/230 passed.
• PostgreSQL isolation schedule: 119/119 passed.
• Supported check-world clusters: 185/185 passed, with zero supported failures.
• Packed CLI lifecycle, including initdb, server startup, native clients, and shutdown: passed.
• Postmaster integration primitives and scenarios, including background workers and extension loading: passed.
• 10,000 backend/session churn run completed without leaked backend memories; peak RSS was approximately 1.7 GiB in the measured configuration.
• wasm32-initial release validation passed for Vector and PostGIS; combined compressed artifacts are 43,408,184 bytes and remain below the frozen 48 MiB budget.
• PGlite/CLI typechecks, formatter, shell syntax, transformer determinism, actual-module audits, and extension packaging checks pass.

The upstream capability ledger is intentional and visible, but it records test qualification rather than declaring every blocked feature architecturally absent. PostgreSQL crash recovery and WAL replay after a forced Worker failure are implemented and tested. WAL sender/receiver, logical-replication workers, recovery code, and replication protocol paths are present in the generic EXEC_BACKEND process model; parts of physical backup, logical decoding, and recovery may already work. The complete multi-cluster physical/logical replication, subscription, standby promotion, archive recovery, and PITR workflows have not yet been qualified end to end. Their current blockers include backend outbound connections to another postmaster, multi-cluster test orchestration, and host-command execution for conventional archive_command/restore_command workflows. pg_upgrade and tests which require real OS child PIDs remain genuinely unsupported by the current artifact/provider. One libpq test which launches three complete postmasters is recorded as blocked because it exceeds Docker Desktop's 7.65 GiB VM memory limit; supported libpq behavior passes.

What is not implemented

• Browser/Web Worker support. This is planned follow-up work. The public API and artifact model already reserve browser-compatible targets so multi-session support can remain under @electric-sql/pglite; this implementation phase is Node-only.
• The wasm32 faceted fallback. This is planned to support environments that provide shared Wasm memory/SharedArrayBuffer but do not provide Wasm multi-memory. Those environments are not supported by PGlitePostmaster in this PR, but the target identity, extension matrix, and fallback selection model reserve this route.
• Wasm memory64 / wasm64 artifacts. Full end-to-end i64-addressed memory remains a later milestone.
• The complete six-target extension matrix. This PR ships only wasm32-classic and wasm32-multi-memory, and only Vector/PostGIS are in the initial postmaster extension inventory.
• AMD64 qualification and cross-architecture parity. The Dockerfile retains architecture selection, but this proof was built and audited natively on ARM64 to avoid emulation.
• Complete qualification of physical/logical replication, subscriptions, standby recovery/promotion, archive recovery, and PITR. These are partially implemented or architecturally present rather than blanket-unsupported. Crash recovery and WAL replay already pass. End-to-end multi-cluster streaming/apply currently needs cross-postmaster outbound socket routing, and conventional PITR tests need an approved replacement for rejected host archive_command/restore_command execution.
• Every native process-management semantic. Basic pg_ctl create/start/status/reload/restart/stop flows and many native client utilities pass. Operations tied specifically to host child PIDs, a forked logging collector, multiple PostgreSQL binary versions (pg_upgrade), or direct host signals remain unsupported or require provider-specific handling.
• A security boundary for mutually untrusted extensions, live session migration between Workers, transparent connection multiplexing, or runtime transformation of arbitrary unmodified side modules.
• Automatic replacement of classic PGlite. Multi-session mode remains an explicit constructor/runtime choice.

Design documents

• multi-session-worker-multi-memory-design.md describes the postmaster, Worker process model, memory domains, pointer ABI, signals, scopes, sockets, VFS, testing, and implementation phases.
• multi-runtime-extension-artifacts-design.md defines the multi-runtime extension target matrix, manifests, wrapper and lifecycle contracts, safe materialization, build pipeline, and the completed wasm32-initial milestone.

This is a large architectural proof and is intentionally opened as a draft for review of the execution model, memory ABI, PostgreSQL-fork boundary, and the path from the current Node/ARM64 qualification to broader runtime support.

[noudadrichem]

Looking forward to trying this out!