diff --git a/README.md b/README.md index 5a2da3c..6db8ce3 100644 --- a/README.md +++ b/README.md @@ -103,6 +103,15 @@ poison jobs, and shows that removing any load-bearing safeguard produces a violation the explorer finds and replays from a seed. The bug table lives in [examples/jobqueue/README.md](https://github.com/dhruvl/simloop/blob/main/examples/jobqueue/README.md). +## Performance + +Simulation is cheap: SimLoop schedules a task step in ~4.4 µs (trace +recording included — about 3.6× faster than the stock loop, which pays a +selector syscall per iteration), compresses sleep-heavy workloads ~2,000× +against wall clock, and the explorer runs the full jobqueue chaos scenario +at ~55 seeds/second on an M4 MacBook Air. Methodology and numbers: +[benchmarks/README.md](https://github.com/dhruvl/simloop/blob/main/benchmarks/README.md). + ## Honest limits Code that goes through the event-loop API is supported; code that @@ -111,6 +120,14 @@ signals, TLS, and `getaddrinfo` raise `SimulationFenceError` rather than silently breaking determinism. Write-side flow control is not simulated. The full contract is in [docs/supported-api.md](https://github.com/dhruvl/simloop/blob/main/docs/supported-api.md). +## Design + +Why the loop is a from-scratch `AbstractEventLoop` instead of an +instrumented stock loop, why one seed feeds three separate RNG streams, +why streams never lose bytes but datagrams do — the decisions and the +alternatives they beat are written up in +[docs/design.md](https://github.com/dhruvl/simloop/blob/main/docs/design.md). + ## License MIT diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..4d803f3 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,71 @@ +# Benchmarks + +Three numbers matter for a simulation harness: what the simulated loop costs +per scheduling step, how much simulated time it covers per wall-clock second, +and how fast the explorer burns through seeds on a real test. Measured on a +MacBook Air (Apple M4, 16 GB), macOS, CPython 3.12.13. Every number is the +median of 5 runs after one warmup run, on an otherwise idle machine. Rerun +them with the commands below; expect the ratios, not the absolute times, to +transfer to other machines. + +## Scheduling overhead + +``` +python benchmarks/overhead.py +``` + +A token circulates around a ring of 100 queue-connected tasks for 200 rounds +(20,000 hops), so the measurement is almost purely task switching and queue +hand-off — no I/O, no timers. + +| loop | median | per hop | +|---|---|---| +| stock asyncio | 0.319 s | 16.0 µs | +| SimLoop | 0.088 s | 4.4 µs | + +SimLoop comes out about **3.6× faster per scheduling step**, trace recording +included, and the ratio holds from 10×2000 to 500×200 task/round shapes +(0.26–0.29× across the sweep). That is not because simloop is a faster event +loop in any general sense — it is because a simulated loop never touches the +OS. Profiling the stock run shows about half its time inside +`select.kqueue.control`: the real loop pays a selector syscall on every +iteration even when no I/O is pending, while SimLoop's iteration is pure +Python — pop the PRNG-chosen callback, run it, append a trace event. The +practical reading: replayable scheduling costs nothing at test time. (For +contrast, trio's experimental deterministic-scheduling hook measured ~15% +overhead on top of its normal loop — python-trio/trio#890; simloop sidesteps +the comparison by replacing the loop instead of instrumenting it.) + +The stock-loop baseline is macOS/kqueue; an epoll or io_uring machine will +price the syscall differently. + +## Time compression + +``` +python benchmarks/time_compression.py +``` + +100 tasks each tick on their own staggered 1–2 s interval, 3600 ticks — the +shape of heartbeats, lease renewals, and retry backoffs. Just under two hours +of simulated time: + +| simulated | wall | compression | +|---|---|---| +| 7164 s (1.99 h) | 3.55 s | **~2,000×** | + +Virtual time never sleeps: between timers the clock jumps, so a suite full of +`await asyncio.sleep(300)` costs only its callback processing. This is what +makes timeout- and lease-expiry bugs cheap to search for. + +## Explorer throughput + +``` +pytest examples/jobqueue/tests/test_campaign.py -q -m slow +``` + +The jobqueue chaos campaign runs one full distributed scenario per seed — +a broker, 3 workers, and 2 clients submitting 8 jobs (some poisoned) under +randomized partitions and a worker crash, then settles for up to 600 +simulated seconds and checks every invariant. 300 seeds complete in +**5.4–6.0 s**, about **55 seeds/second**. A thousand-seed overnight search is +a 20-second coffee break. diff --git a/benchmarks/overhead.py b/benchmarks/overhead.py index e93919a..6e84a6d 100644 --- a/benchmarks/overhead.py +++ b/benchmarks/overhead.py @@ -1,16 +1,21 @@ """Compare scheduling overhead: SimLoop against the stock asyncio loop. The workload passes a token around a ring of queue-connected tasks, so the -measured cost is almost purely task switching and queue hand-off. Run with -``python benchmarks/overhead.py [--tasks N] [--rounds M]``. SimLoop's number -includes trace recording — that is the honest price of replayability. +measured cost is almost purely task switching and queue hand-off. Each loop +gets one warmup run plus ``--repeats`` measured runs on a fresh loop; the +reported overhead is the ratio of medians. Run with +``python benchmarks/overhead.py [--tasks N] [--rounds M] [--repeats K]``. +SimLoop's number includes trace recording — that is the honest price of +replayability. """ from __future__ import annotations import argparse import asyncio +import statistics import time +from collections.abc import Callable from simloop import SimLoop @@ -28,13 +33,22 @@ async def worker(index: int) -> None: await asyncio.gather(*workers) -def _measure(loop: asyncio.AbstractEventLoop, n_tasks: int, rounds: int) -> float: - start = time.perf_counter() - try: - loop.run_until_complete(_token_ring(n_tasks, rounds)) - finally: - loop.close() - return time.perf_counter() - start +def _measure( + make_loop: Callable[[], asyncio.AbstractEventLoop], + n_tasks: int, + rounds: int, + repeats: int, +) -> list[float]: + times: list[float] = [] + for _ in range(repeats + 1): # first run is warmup, dropped below + loop = make_loop() + start = time.perf_counter() + try: + loop.run_until_complete(_token_ring(n_tasks, rounds)) + finally: + loop.close() + times.append(time.perf_counter() - start) + return times[1:] def main() -> None: @@ -43,17 +57,28 @@ def main() -> None: ) parser.add_argument("--tasks", type=int, default=100) parser.add_argument("--rounds", type=int, default=200) + parser.add_argument("--repeats", type=int, default=5) args = parser.parse_args() hops = args.tasks * args.rounds - stock = _measure(asyncio.new_event_loop(), args.tasks, args.rounds) - simulated = _measure(SimLoop(seed=0), args.tasks, args.rounds) + stock = _measure(asyncio.new_event_loop, args.tasks, args.rounds, args.repeats) + simulated = _measure( + lambda: SimLoop(seed=0), args.tasks, args.rounds, args.repeats + ) - print(f"{args.tasks} tasks x {args.rounds} rounds = {hops} hops") - print(f"{'loop':<10}{'total s':>10}{'us/hop':>10}") - for label, seconds in (("asyncio", stock), ("SimLoop", simulated)): - print(f"{label:<10}{seconds:>10.4f}{seconds / hops * 1e6:>10.2f}") - print(f"overhead: {simulated / stock:.2f}x") + print( + f"{args.tasks} tasks x {args.rounds} rounds = {hops} hops, " + f"median of {args.repeats} runs" + ) + print(f"{'loop':<10}{'median s':>10}{'min s':>10}{'us/hop':>10}") + for label, times in (("asyncio", stock), ("SimLoop", simulated)): + median = statistics.median(times) + print( + f"{label:<10}{median:>10.4f}{min(times):>10.4f}" + f"{median / hops * 1e6:>10.2f}" + ) + overhead = statistics.median(simulated) / statistics.median(stock) + print(f"overhead: {overhead:.2f}x") if __name__ == "__main__": diff --git a/benchmarks/time_compression.py b/benchmarks/time_compression.py new file mode 100644 index 0000000..124c824 --- /dev/null +++ b/benchmarks/time_compression.py @@ -0,0 +1,70 @@ +"""Measure how much simulated time SimLoop covers per wall-clock second. + +The workload is sleep-heavy: a fleet of tasks each ticking on its own +staggered interval, the way heartbeats, lease renewals, and retry backoffs +do in a real system. On the stock loop this would take the full simulated +duration in wall time; under SimLoop the clock jumps between timers, so the +whole thing costs only the callback processing. Run with +``python benchmarks/time_compression.py [--tasks N] [--ticks M] [--repeats K]``. +The reported ratio is simulated seconds per wall-clock second, median of +``--repeats`` runs after one warmup. +""" + +from __future__ import annotations + +import argparse +import asyncio +import statistics +import time + +from simloop import SimLoop + + +async def _tick_fleet(n_tasks: int, ticks: int) -> None: + async def ticker(index: int) -> None: + # Stagger intervals so timers do not all collide on the same instant. + interval = 1.0 + index / n_tasks + for _ in range(ticks): + await asyncio.sleep(interval) + + await asyncio.gather(*(ticker(i) for i in range(n_tasks))) + + +def _measure(n_tasks: int, ticks: int, repeats: int) -> tuple[float, list[float]]: + simulated = 0.0 + walls: list[float] = [] + for _ in range(repeats + 1): # first run is warmup, dropped below + loop = SimLoop(seed=0) + start = time.perf_counter() + try: + loop.run_until_complete(_tick_fleet(n_tasks, ticks)) + simulated = loop.time() + finally: + loop.close() + walls.append(time.perf_counter() - start) + return simulated, walls[1:] + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Measure simulated seconds covered per wall-clock second." + ) + parser.add_argument("--tasks", type=int, default=100) + parser.add_argument("--ticks", type=int, default=3600) + parser.add_argument("--repeats", type=int, default=5) + args = parser.parse_args() + + simulated, walls = _measure(args.tasks, args.ticks, args.repeats) + median = statistics.median(walls) + + print( + f"{args.tasks} tasks x {args.ticks} ticks, " + f"median of {args.repeats} runs" + ) + print(f"simulated: {simulated:>10.1f} s ({simulated / 3600:.2f} h)") + print(f"wall: {median:>10.4f} s (min {min(walls):.4f})") + print(f"compression: {simulated / median:,.0f}x") + + +if __name__ == "__main__": + main() diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..3359d62 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,291 @@ +# simloop design notes + +How simloop works, and — more useful — why it works this way. Every section +that describes a decision also names the alternatives it beat and what would +have gone wrong with them. For the user-facing contract see +[supported-api.md](supported-api.md); this page is about the machinery. + +## The gap + +Deterministic simulation testing has a distinguished lineage — FoundationDB +built its reputation on it, TigerBeetle and Antithesis made it a selling +point, and Rust has two mature frameworks in +[madsim](https://github.com/madsim-rs/madsim) and +[turmoil](https://github.com/tokio-rs/turmoil). Python had the pieces but +never the whole: [looptime](https://github.com/nolar/looptime) and trio's +autojump clock do virtual time only, an abandoned Ethereum project mocked +asyncio sockets without determinism, and anysystem runs simulated processes +against its own API rather than real asyncio code. + +The closest anyone came is instructive. Trio users asked for tools to find +scheduler-dependent heisenbugs in 2017 +([trio#239](https://github.com/python-trio/trio/issues/239)); deterministic +scheduling landed in 2019 as an undocumented internal hook +([trio#890](https://github.com/python-trio/trio/pull/890)) after the +maintainers measured ~15% overhead for instrumenting the live scheduler; in +2021 users were still asking for a public version +([trio#2022](https://github.com/python-trio/trio/issues/2022)). Two lessons +carried into simloop: instrumentation must cost nothing when it is not in +use, and a deterministic mode bolted onto a production scheduler fights that +scheduler forever. + +simloop's answer is to not touch the production loop at all. The simulation +is a separate `asyncio.AbstractEventLoop` implementation you only construct +in tests; production code runs on stock asyncio, unchanged and uninstrumented. + +## The core bet: `call_soon` is the only door + +Everything asyncio schedules goes through the loop's `call_soon` / `call_at` +family. Crucially that includes coroutines: `asyncio.Task` drives every step +of every coroutine by scheduling its `__step` callback via `call_soon`. Own +dispatch and you own task interleaving — without touching `Task`, without +instrumenting await points, without import hooks. + +So `SimLoop` implements `AbstractEventLoop` directly (~500 lines, +`src/simloop/_loop.py`) around three structures: a list of ready callbacks, a +timer heap, and a virtual clock starting at 0.0. The scheduling core is one +method: + +- If callbacks are ready, draw the next one to run from a seeded + `random.Random` — this draw is **the only source of nondeterminism in the + entire loop**, so a seed pins the entire execution. +- If none are ready, jump the clock to the earliest timer deadline and move + every timer due at that instant onto the ready list. Time never advances + while work is pending, and never waits on the wall clock. +- If there are no ready callbacks *and* no timers, nothing can ever run + again: raise `SimulationDeadlockError`. Lost-wakeup bugs surface as a + diagnosis instead of a hang. + +Alternatives considered and rejected: + +- **Subclassing `BaseEventLoop`** — it is welded to a selector; every + iteration wants to poll the OS. Fighting that buys nothing, since the + simulation has no file descriptors. +- **Instrumenting the stock loop** (the trio#890 shape) — pays overhead in + production or demands an opt-in mode inside code you don't control, and + the loop's real selector timing keeps leaking into test behavior. +- **Randomizing elsewhere** (task creation order, fuzzing timer durations) — + either misses interleavings or distorts the program's actual timing. + Permuting the ready queue explores exactly the reorderings asyncio itself + is allowed to perform: anything simloop finds is a legal schedule of your + program. + +## Deterministic identity, or: never compare objects + +A seeded draw only helps if everything around it is order-stable too. Three +rules hold everywhere: + +1. **Every scheduled callback gets a `seq`** from a global counter at + scheduling time. The timer heap is keyed `(deadline, seq)` so equal + deadlines break ties by creation order — handles themselves are never + compared, and no scheduling decision ever depends on `id()`. +2. **Trace labels are qualified names, never `repr()`** — a repr can embed a + memory address, which would make traces differ between identical runs. +3. **No scheduling decision iterates a set or hash-ordered mapping.** The + loop's structures are lists, heaps, and insertion-ordered dicts. + +This is what makes the replay proof cross-process. The hardening suite +(`tests/test_hardening.py`, `tests/test_net_hardening.py`) re-runs reference +workloads 100 times per seed in-process and then again in fresh interpreters +under *different* `PYTHONHASHSEED` values, asserting bit-identical trace +hashes throughout. + +## The trace is the proof + +Every decision lands in an append-only recorder (`src/simloop/_trace.py`): +schedule, run, clock advance, cancellation, and every network verdict. Two +properties are load-bearing: + +- **Completeness.** Even a *cancelled* handle's draw is recorded — the draw + consumed PRNG state, so it is a scheduling decision, and a replay that made + a different set of draws must produce a different hash. +- **Injectivity.** Events serialize as `kind|when|seq|label` lines into a + SHA-256. Labels are qualified callback names or network labels built from + validated host names, and host names may not contain `|`, `>` or newline — + so two distinct event streams cannot collide onto one byte sequence. + +Hash equality is therefore a cheap, sufficient check that a replay was +*exact*, not merely same-outcome. It is asserted all over the test suite and +printed with every failure report. + +## Three RNG streams, never one + +`SimLoop` derives three independent generators from one seed: the +scheduler's draw stream, a user-facing stream behind `sim.random` / +`sim.uuid4`, and the network's fault stream. They are string-seeded +(`Random(f"{seed}:net")` — stable across processes, since string seeding +hashes via SHA-512 rather than `hash()`). + +One shared stream would have been simpler and subtly awful: a program that +draws one extra `sim.random()` value would shift every subsequent scheduling +decision and fault verdict, so unrelated behavior changes would reshuffle +the failure you were chasing. Isolation keeps the blast radius of a change +to its own stream. + +`sim` itself (`src/simloop/_sim.py`) is a deliberate concession. Direct +`random` / `uuid` / `time.time()` calls in code under test are honest +nondeterminism simloop cannot intercept without monkeypatching the stdlib — +which was rejected: patching globals leaks across test boundaries and lies +about what production will do. Instead `sim.random` et al. resolve to +seed-derived streams inside a simulation and fall through to the stdlib +outside one, so the same line is deterministic under test and real in +production. + +## Fail loudly: the fence policy + +Anything that would reach outside the simulation — executors and threads, +signals, subprocesses, raw sockets, `add_reader`/`add_writer`, `getaddrinfo`, +TLS, pipes, `sendfile` — raises `SimulationFenceError` naming the exact call, +and optional stdlib kwargs that would smuggle those in (`ssl=`, `sock=`, …) +are rejected the same way. + +The tempting alternative was best-effort passthrough: let `run_in_executor` +actually run things, keep most libraries importable, appear more compatible. +That is the worst possible failure mode for this tool — a harness that +*claims* determinism while real threads race underneath produces +unreproducible "reproducible" failures, and every hour a user spends on a +replay that doesn't replay is trust that never comes back. A loud fence +converts silent wrongness into a documented boundary +([supported-api.md](supported-api.md)) plus an honest error message. + +The same posture applies to errors inside the simulation: a run must not +look green while something failed. Unhandled exceptions from fire-and-forget +tasks are collected by the loop's exception handler and re-raised from +`run_until_complete`. One wrinkle earned its own comment in the source: a +failed orphaned task can keep itself alive through a reference cycle (its +exception's traceback pins its own frame), so its failure only reaches the +handler when the cycle collector finalizes it — `run_until_complete` forces +a `gc.collect()` before declaring success so that failure cannot slip past +the boundary. + +## The network: packets, not pipes + +`SimNetwork` (`src/simloop/_net.py`, `_transports.py`) sits behind the +loop's own `create_connection` / `create_server` / +`create_datagram_endpoint`, so `asyncio.open_connection` and friends work +unchanged. Addresses are host names: `loop.net.host("broker")` declares a +machine, and multi-node systems run as tasks in one process. + +The transport layer could have been a pair of byte queues per connection — +far simpler, and how most asyncio test mocks do it. It models packets +instead, with a small TCP-shaped vocabulary (`syn` / `accept` / `refuse` / +`data` / `fin` / `rst`), because the faults worth testing live at packet +granularity: per-packet latency draws, per-datagram drop and duplication, +partitions that catch some packets mid-flight and not others. Byte pipes +cannot express "the acknowledgment was in flight when the partition landed" +— which is precisely the kind of schedule the jobqueue bugs need. + +Decisions inside that model, each doing real work: + +- **Streams reorder packets, deliver bytes in order.** Independent latency + draws mean packet N+1 can land before packet N; a per-direction sequence + number and reassembly buffer restore order. Faithful jitter above, honest + stream semantics below. +- **Only datagrams are lossy.** A "reliable stream" that loses bytes would + be lying about being a stream; there is no retransmission model, so stream + loss under random drop would otherwise wedge every connection. Drop and + duplication apply to datagram endpoints, where at-most-once is the real + contract. +- **Partitions blackhole silently.** Datagrams crossing a cut are gone; + stream packets are *held* and released on heal — with no retransmission, + permanently dropping a mid-stream packet would leave the receiver waiting + on a sequence gap forever, so held-then-released is what "the bytes stop, + then the connection resumes intact" has to mean. Nothing errors: only your + own timeouts fire, exactly like a real partition. +- **Crashed hosts send no reset.** Their tasks are cancelled, their + listeners close, and they go silent. Peers cannot distinguish a crash from + a partition except by timeout — which is the entire epistemology of + distributed failure detection, enforced by construction. +- **The accept is sequence 0.** The server builds its transport and sends + `accept` before its protocol's `connection_made` can write; the client + transport is built when the accept is *dispatched*, not when the connector + resumes. Data written from `connection_made` (seq 1+) therefore can never + outrun connection establishment, whatever the latency draws say — an + ordering race ruled out by construction rather than patched on discovery. +- **Tasks are pinned to hosts through a `ContextVar`**, inherited by every + task a host's tasks spawn; packet delivery runs the receiving protocol + under the *destination* host's context, so work spawned from + `datagram_received` belongs to the receiving machine, and `crash` knows + exactly which tasks to kill. + +What was cut, deliberately: write-side flow control (`drain()` never blocks +— buffers are unbounded), retransmission and congestion modeling, IP +addresses, TLS. Each would deepen the simulation without widening the class +of bugs it can catch; the supported-subset contract beats chasing 100% of +the asyncio surface. + +## Explorer and pytest plugin: a thin shell by design + +The explorer (`src/simloop/_explore.py`) is deliberately boring: run the +test coroutine on a fresh `SimLoop` per seed, stop at the first failure, +return a report carrying the seed, the trace tail, the trace hash, and every +still-pending task with its await site — the "where was everyone stuck" +snapshot that makes a distributed timeout debuggable. `@sim_test(seeds=N)` +wraps an `async def` into a plain synchronous test; the report is attached +to the original exception as a note, so pytest shows the real traceback +*and* the replay command. + +Two structural choices: + +- `import simloop` pulls in zero third-party code. The explorer never + imports pytest; the plugin (`_pytest_plugin.py`, loaded via the `pytest11` + entry point) feeds `--simloop-seeds` / `--simloop-replay` in through a + session-scoped overrides object. CLI flags override decorator defaults, so + CI can crank a 10-seed test to 1,000 without touching code. +- Failure reporting is part of the library, not the plugin, so the same + report renders anywhere a coroutine can run. + +Schedule shrinking — reducing a failing seed's trace to a minimal +interleaving — was considered and deferred; a seed plus an exact replay has +proven sufficient to debug every failure so far. + +## What replay actually guarantees + +Claimed precisely: **same seed, same code, same interpreter version ⇒ +identical execution**, proven by trace-hash equality — across re-runs, +across processes, across hash-randomization seeds. What can still break +replay is *user* code that consults nondeterminism simloop does not control: +direct `random` / `time` / `uuid` calls (use `sim.*`), iteration over sets +of strings (hash-order varies per process unless `PYTHONHASHSEED` is +pinned), and anything fenced that you catch and route around. The fences +make the third category loud; the first two are documented contract. GC +timing is a non-issue for the proof because collection points depend only on +allocation behavior, which identical execution makes identical — and the +suite's cross-process runs would have caught drift there. + +## Performance + +Numbers and methodology in [benchmarks/](../benchmarks/README.md); the shape +matters more than the digits. A simulated loop never blocks in a selector, +so SimLoop schedules a task step in ~4.4 µs where the stock macOS loop +spends ~16 µs — about half of that inside the per-iteration `kqueue` call — +making the simulation roughly 3.6× faster per step *including* trace +recording. Sleep-heavy workloads compress ~2,000× against wall clock, and +the full jobqueue chaos scenario explores ~55 seeds/second on a laptop. +The trio thread priced deterministic scheduling at ~15% overhead; replacing +the loop instead of instrumenting it turned the overhead negative. + +## Case study: jobqueue + +`examples/jobqueue/` is the proof of usefulness: an exactly-once job +scheduler (leases, fencing tokens, idempotency keys, backoff, dead-letters) +written in plain stdlib asyncio with no simloop imports, tested entirely +under simulation. Its campaign runs 300 seeds of randomized partitions, +worker crashes, and poison jobs with four invariants checked after every +run; its ablation matrix shows that removing any load-bearing safeguard +yields a seed-replayable violation, while each redundant defense also holds +alone. The full bug table is in +[examples/jobqueue/README.md](../examples/jobqueue/README.md). + +The demo also disciplined the harness: carrying a real distributed system +is what shaped the deadlock diagnostics, the pending-task reports, and the +crash and partition semantics described above — nothing exposes a +simulation's blind spots like making it hold up something that matters. + +## Future work + +Schedule shrinking (greedy trace reduction toward a minimal failing +interleaving); trace diffing between the last passing and first failing +seed; compatibility probes for popular pure-asyncio libraries; a +`sock_*`-level simulation if real demand appears. Each is additive — the +core contract above is meant to stay small and true.