Benchmark numbers and design write-up

README.md

@@ -103,6 +103,15 @@ poison jobs, and shows that removing any load-bearing safeguard produces
 a violation the explorer finds and replays from a seed. The bug table
 lives in [examples/jobqueue/README.md](https://github.com/dhruvl/simloop/blob/main/examples/jobqueue/README.md).
 
+## Performance
+
+Simulation is cheap: SimLoop schedules a task step in ~4.4 µs (trace
+recording included — about 3.6× faster than the stock loop, which pays a
+selector syscall per iteration), compresses sleep-heavy workloads ~2,000×
+against wall clock, and the explorer runs the full jobqueue chaos scenario
+at ~55 seeds/second on an M4 MacBook Air. Methodology and numbers:
+[benchmarks/README.md](https://github.com/dhruvl/simloop/blob/main/benchmarks/README.md).
+
 ## Honest limits
 
 Code that goes through the event-loop API is supported; code that
@@ -111,6 +120,14 @@ signals, TLS, and `getaddrinfo` raise `SimulationFenceError` rather than
 silently breaking determinism. Write-side flow control is not simulated.
 The full contract is in [docs/supported-api.md](https://github.com/dhruvl/simloop/blob/main/docs/supported-api.md).
 
+## Design
+
+Why the loop is a from-scratch `AbstractEventLoop` instead of an
+instrumented stock loop, why one seed feeds three separate RNG streams,
+why streams never lose bytes but datagrams do — the decisions and the
+alternatives they beat are written up in
+[docs/design.md](https://github.com/dhruvl/simloop/blob/main/docs/design.md).
+
 ## License
 
 MIT

benchmarks/README.md

@@ -0,0 +1,71 @@
+# Benchmarks
+
+Three numbers matter for a simulation harness: what the simulated loop costs
+per scheduling step, how much simulated time it covers per wall-clock second,
+and how fast the explorer burns through seeds on a real test. Measured on a
+MacBook Air (Apple M4, 16 GB), macOS, CPython 3.12.13. Every number is the
+median of 5 runs after one warmup run, on an otherwise idle machine. Rerun
+them with the commands below; expect the ratios, not the absolute times, to
+transfer to other machines.
+
+## Scheduling overhead
+
+```
+python benchmarks/overhead.py
+```
+
+A token circulates around a ring of 100 queue-connected tasks for 200 rounds
+(20,000 hops), so the measurement is almost purely task switching and queue
+hand-off — no I/O, no timers.
+
+| loop | median | per hop |
+|---|---|---|
+| stock asyncio | 0.319 s | 16.0 µs |
+| SimLoop | 0.088 s | 4.4 µs |
+
+SimLoop comes out about **3.6× faster per scheduling step**, trace recording
+included, and the ratio holds from 10×2000 to 500×200 task/round shapes
+(0.26–0.29× across the sweep). That is not because simloop is a faster event
+loop in any general sense — it is because a simulated loop never touches the
+OS. Profiling the stock run shows about half its time inside
+`select.kqueue.control`: the real loop pays a selector syscall on every
+iteration even when no I/O is pending, while SimLoop's iteration is pure
+Python — pop the PRNG-chosen callback, run it, append a trace event. The
+practical reading: replayable scheduling costs nothing at test time. (For
+contrast, trio's experimental deterministic-scheduling hook measured ~15%
+overhead on top of its normal loop — python-trio/trio#890; simloop sidesteps
+the comparison by replacing the loop instead of instrumenting it.)
+
+The stock-loop baseline is macOS/kqueue; an epoll or io_uring machine will
+price the syscall differently.
+
+## Time compression
+
+```
+python benchmarks/time_compression.py
+```
+
+100 tasks each tick on their own staggered 1–2 s interval, 3600 ticks — the
+shape of heartbeats, lease renewals, and retry backoffs. Just under two hours
+of simulated time:
+
+| simulated | wall | compression |
+|---|---|---|
+| 7164 s (1.99 h) | 3.55 s | **~2,000×** |
+
+Virtual time never sleeps: between timers the clock jumps, so a suite full of
+`await asyncio.sleep(300)` costs only its callback processing. This is what
+makes timeout- and lease-expiry bugs cheap to search for.
+
+## Explorer throughput
+
+```
+pytest examples/jobqueue/tests/test_campaign.py -q -m slow
+```
+
+The jobqueue chaos campaign runs one full distributed scenario per seed —
+a broker, 3 workers, and 2 clients submitting 8 jobs (some poisoned) under
+randomized partitions and a worker crash, then settles for up to 600
+simulated seconds and checks every invariant. 300 seeds complete in
+**5.4–6.0 s**, about **55 seeds/second**. A thousand-seed overnight search is
+a 20-second coffee break.

benchmarks/overhead.py

@@ -1,16 +1,21 @@
 """Compare scheduling overhead: SimLoop against the stock asyncio loop.
 
 The workload passes a token around a ring of queue-connected tasks, so the
-measured cost is almost purely task switching and queue hand-off. Run with
-``python benchmarks/overhead.py [--tasks N] [--rounds M]``. SimLoop's number
-includes trace recording — that is the honest price of replayability.
+measured cost is almost purely task switching and queue hand-off. Each loop
+gets one warmup run plus ``--repeats`` measured runs on a fresh loop; the
+reported overhead is the ratio of medians. Run with
+``python benchmarks/overhead.py [--tasks N] [--rounds M] [--repeats K]``.
+SimLoop's number includes trace recording — that is the honest price of
+replayability.
 """
 
 from __future__ import annotations
 
 import argparse
 import asyncio
+import statistics
 import time
+from collections.abc import Callable
 
 from simloop import SimLoop
 
@@ -28,13 +33,22 @@ async def worker(index: int) -> None:
     await asyncio.gather(*workers)
 
 
-def _measure(loop: asyncio.AbstractEventLoop, n_tasks: int, rounds: int) -> float:
-    start = time.perf_counter()
-    try:
-        loop.run_until_complete(_token_ring(n_tasks, rounds))
-    finally:
-        loop.close()
-    return time.perf_counter() - start
+def _measure(
+    make_loop: Callable[[], asyncio.AbstractEventLoop],
+    n_tasks: int,
+    rounds: int,
+    repeats: int,
+) -> list[float]:
+    times: list[float] = []
+    for _ in range(repeats + 1):  # first run is warmup, dropped below
+        loop = make_loop()
+        start = time.perf_counter()
+        try:
+            loop.run_until_complete(_token_ring(n_tasks, rounds))
+        finally:
+            loop.close()
+        times.append(time.perf_counter() - start)
+    return times[1:]
 
 
 def main() -> None:
@@ -43,17 +57,28 @@ def main() -> None:
     )
     parser.add_argument("--tasks", type=int, default=100)
     parser.add_argument("--rounds", type=int, default=200)
+    parser.add_argument("--repeats", type=int, default=5)
     args = parser.parse_args()
     hops = args.tasks * args.rounds
 
-    stock = _measure(asyncio.new_event_loop(), args.tasks, args.rounds)
-    simulated = _measure(SimLoop(seed=0), args.tasks, args.rounds)
+    stock = _measure(asyncio.new_event_loop, args.tasks, args.rounds, args.repeats)
+    simulated = _measure(
+        lambda: SimLoop(seed=0), args.tasks, args.rounds, args.repeats
+    )
 
-    print(f"{args.tasks} tasks x {args.rounds} rounds = {hops} hops")
-    print(f"{'loop':<10}{'total s':>10}{'us/hop':>10}")
-    for label, seconds in (("asyncio", stock), ("SimLoop", simulated)):
-        print(f"{label:<10}{seconds:>10.4f}{seconds / hops * 1e6:>10.2f}")
-    print(f"overhead: {simulated / stock:.2f}x")
+    print(
+        f"{args.tasks} tasks x {args.rounds} rounds = {hops} hops, "
+        f"median of {args.repeats} runs"
+    )
+    print(f"{'loop':<10}{'median s':>10}{'min s':>10}{'us/hop':>10}")
+    for label, times in (("asyncio", stock), ("SimLoop", simulated)):
+        median = statistics.median(times)
+        print(
+            f"{label:<10}{median:>10.4f}{min(times):>10.4f}"
+            f"{median / hops * 1e6:>10.2f}"
+        )
+    overhead = statistics.median(simulated) / statistics.median(stock)
+    print(f"overhead: {overhead:.2f}x")
 
 
 if __name__ == "__main__":

benchmarks/time_compression.py

@@ -0,0 +1,70 @@
+"""Measure how much simulated time SimLoop covers per wall-clock second.
+
+The workload is sleep-heavy: a fleet of tasks each ticking on its own
+staggered interval, the way heartbeats, lease renewals, and retry backoffs
+do in a real system. On the stock loop this would take the full simulated
+duration in wall time; under SimLoop the clock jumps between timers, so the
+whole thing costs only the callback processing. Run with
+``python benchmarks/time_compression.py [--tasks N] [--ticks M] [--repeats K]``.
+The reported ratio is simulated seconds per wall-clock second, median of
+``--repeats`` runs after one warmup.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import statistics
+import time
+
+from simloop import SimLoop
+
+
+async def _tick_fleet(n_tasks: int, ticks: int) -> None:
+    async def ticker(index: int) -> None:
+        # Stagger intervals so timers do not all collide on the same instant.
+        interval = 1.0 + index / n_tasks
+        for _ in range(ticks):
+            await asyncio.sleep(interval)
+
+    await asyncio.gather(*(ticker(i) for i in range(n_tasks)))
+
+
+def _measure(n_tasks: int, ticks: int, repeats: int) -> tuple[float, list[float]]:
+    simulated = 0.0
+    walls: list[float] = []
+    for _ in range(repeats + 1):  # first run is warmup, dropped below
+        loop = SimLoop(seed=0)
+        start = time.perf_counter()
+        try:
+            loop.run_until_complete(_tick_fleet(n_tasks, ticks))
+            simulated = loop.time()
+        finally:
+            loop.close()
+        walls.append(time.perf_counter() - start)
+    return simulated, walls[1:]
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Measure simulated seconds covered per wall-clock second."
+    )
+    parser.add_argument("--tasks", type=int, default=100)
+    parser.add_argument("--ticks", type=int, default=3600)
+    parser.add_argument("--repeats", type=int, default=5)
+    args = parser.parse_args()
+
+    simulated, walls = _measure(args.tasks, args.ticks, args.repeats)
+    median = statistics.median(walls)
+
+    print(
+        f"{args.tasks} tasks x {args.ticks} ticks, "
+        f"median of {args.repeats} runs"
+    )
+    print(f"simulated: {simulated:>10.1f} s ({simulated / 3600:.2f} h)")
+    print(f"wall:      {median:>10.4f} s (min {min(walls):.4f})")
+    print(f"compression: {simulated / median:,.0f}x")
+
+
+if __name__ == "__main__":
+    main()