Skip to content

bench: re-validate write path on electric#4710 (recovery hardening) — no regression #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
balegas merged 1 commit into from
Jul 14, 2026
Merged

bench: re-validate write path on electric#4710 (recovery hardening) — no regression #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion results/PROVENANCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
# Provenance — canonical campaign 2026-07-14

- **Server build:** electric-sql/electric#4697 head `d5589289d` (perf/wal-checkpoint-syncfs), image `durable-streams:dev` digest `0e8bfe920065`.
**Split build provenance.** The write/durability suites were re-run on the
electric#4710 recovery-hardening build; the rest keep their #4697 values
(#4710 changes no read/ursula/mixed-cal/mixed-writes code path).

- **Write-path suites (`canonical-write`, `canonical-mixed-delivery`):**
electric-sql/electric#4710 head `abca3a5f7` (fix/recovery-hardening), image
`durable-streams:dev` digest `93803a261706`. Re-validation of the durability
hardening — no write regression vs #4697 (wal ±1%, memory flat).
- **All other suites (`canonical-write-ursula`, `canonical-reads-*`,
`canonical-mixed-cal`, `canonical-mixed-writes`):** electric#4697 head
`d5589289d` (perf/wal-checkpoint-syncfs), image digest `0e8bfe920065`.
- **Ursula:** upstream `ghcr.io/tonbo-io/ursula:v0.2.0`.
- **Hardware:** canonical-write on `c4d-standard-64-lssd` raw-block (STATIC_CPU=1, GUARANTEED=1, splitlane3x3 manifest); ursula/reads/mixed on `c4d-standard-16-lssd` stock manifests. Clients `n2d-standard-32` Spot. europe-west4.
- **Harness:** ds-bench bench/canonical-suites @ 2b0ada5.
Expand Down
47 changes: 32 additions & 15 deletions results/REPORT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,35 @@ First run of the canonical suite set, on the electric#4697 merged-state build
(syncfs checkpoints, per-shard triggers, stream lanes; see WAL_TUNING.md).
Provenance in `PROVENANCE.md`; per-suite grids in each subdirectory.

> **Write-path re-validation for electric#4710 (2026-07-14).** The two
> write/durability suites — `canonical-write` and `canonical-mixed-delivery` —
> were re-run on the #4710 recovery-hardening build (fail-stop durability
> barriers, always-sync sidecar, extra parent-dir fsyncs on the WAL path). Those
> two sections below now carry **#4710** numbers; the read/ursula/mixed-cal/
> mixed-writes suites are unchanged code paths under #4710 and keep their #4697
> values. **Verdict: no write regression** (see the per-section deltas). Split
> provenance recorded in `PROVENANCE.md`.

## 1. Write saturation (`canonical-write`, `canonical-write-ursula`)

Peak append/s at saturation (256 B payloads):
Peak append/s at saturation (256 B payloads). **durable-streams columns = #4710**;
Δ vs the #4697 baseline in parentheses:

| streams | wal-ideal | memory | ursula-mem | ursula-disk |
|---|---|---|---|---|
| 100 | — | — | 52.0k | 4.5k |
| 1k | — | — | 54.7k | 7.4k |
| 10k | 417.0k | 680.4k | 49.0k | 8.3k |
| 100k | **382.3k** | **631.5k** | — | — |
| 10k | 413.0k (−1.0%) | 669.0k (−1.6%) | 49.0k | 8.3k |
| 100k | **386.0k (+1.0%)** | **628.0k (−0.5%)** | — | — |

- All four durable-streams cells are true plateaus (pinned rungs). **No
cardinality cliff: −8% (wal) / −7% (memory) from 10k→100k.** Regression gates
(wal@100k > 250k, memory@100k > 400k) passed with wide margin.
cardinality cliff: −6.5% (wal) / −6.1% (memory) from 10k→100k.** Regression
gates (wal@100k > 250k, memory@100k > 400k) passed with wide margin.
- **#4710's durability barriers cost nothing measurable on the write path**:
wal is within ±1% of #4697 (100k even +1%), and memory is flat (it takes no
fsync barrier, so the hardening cannot touch it) — the two move exactly as the
code predicts. The ~5% dips seen mid-sweep were the 8-pod ladder rung; both wal
cells peak at the 4-pod rung (client/contention-bound, not server-bound).
- wal-ideal = the WAL_TUNING.md configuration (3 stream lanes + 3 WAL lanes,
1 GiB checkpoint budget, pinned cores). memory benefits from the same pinned
cores (previous best 512k on shared cores).
Expand All @@ -43,19 +58,21 @@ conns, p99 2–3 ms. ursula matches at 10 streams but degrades at 100 streams
49.9–50.0k at 0 / 1k / 10k / 100k readers while serving up to 4,986 replays/s
at 303 MiB/s. **The premise holds: 100k concurrent catch-up readers cost the
write path nothing.**
- **Delivery under write load (2000 SSE subscribers):**
- **Delivery under write load (2000 SSE subscribers) — #4710:**

| writes/s | wal del/s (p99 ms) | memory del/s (p99 ms) |
|---|---|---|
| 4k | 3.3k (138) | 3.3k (151) |
| 16k | 15.9k (54) | 13.3k (5) |
| 40k | 33.3k (86) | 33.2k (2) |
| 66k | 54.9k (107) | 65.7k (8) |
| max | 63.8k @ 85k writes (106) | **126.7k @ 127k writes (45)** |

**The 2026-07-02 memory-mode delivery collapse is GONE** on this build:
memory delivery tracks writes 1:1 all the way to 127k/s. wal delivery keeps
pace to ~40k writes/s and caps at ~64k del/s at full write saturation.
| 4k | 4.0k (147) | 3.3k (140) |
| 16k | 15.9k (51) | 15.9k (51) |
| 40k | 33.3k (86) | 39.8k (5) |
| 66k | 65.7k (92) | 65.7k (7) |
| max | 64.1k @ 84k writes (69) | **139.3k @ 140k writes (63)** |

**The 2026-07-02 memory-mode delivery collapse stays GONE** on #4710:
memory delivery tracks writes 1:1 all the way to 140k/s (baseline #4697 was
127k — the difference is run-to-run headroom, not a code change). wal delivery
keeps pace to ~66k writes/s and caps at ~64k del/s at full write saturation —
same shape as #4697, no regression from the added durability barriers.

## Known gaps & artifacts

Expand Down
20 changes: 10 additions & 10 deletions results/canonical-mixed-delivery/aggregate.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
mode,stream_count,sweep,level,readers,subscribers,writer_rate,read_rate,write_ops_per_sec,write_p50,write_p99,read_ops_per_sec,read_mib_per_sec,read_p50,read_p99,events_per_sec,delivery_p50,delivery_p99,write_bp,write_err,read_bp,read_err,status,reason
memory,2000,writer_rate,0,0,2000,0,0,127139.0,14.527,41.279,0.0,0.0,,,126729.93333333333,12.015,45.023,0,0,0,0,ok,complete
memory,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.682,129.407,0.0,0.0,,,3341.3333333333335,0.924,150.655,0,0,0,0,ok,complete
memory,2000,writer_rate,8,0,2000,8,0,16047.333333333334,1.654,3.761,0.0,0.0,,,13305.3,0.954,5.063,0,0,0,0,ok,complete
memory,2000,writer_rate,20,0,2000,20,0,39984.833333333336,1.822,3.033,0.0,0.0,,,33226.433333333334,1.09,2.333,0,0,0,0,ok,complete
memory,2000,writer_rate,33,0,2000,33,0,65920.43333333333,2.101,93.759,0.0,0.0,,,65691.26666666666,1.214,8.367,0,0,0,0,ok,complete
wal,2000,writer_rate,0,0,2000,0,0,85168.3,21.935,95.871,0.0,0.0,,,63848.26666666667,35.807,105.599,0,0,0,0,ok,complete
wal,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.67,121.151,0.0,0.0,,,3336.4666666666667,1.011,137.471,0,0,0,0,ok,complete
wal,2000,writer_rate,8,0,2000,8,0,16048.966666666667,1.693,58.143,0.0,0.0,,,15944.433333333332,1.063,54.175,0,0,0,0,ok,complete
wal,2000,writer_rate,20,0,2000,20,0,39977.566666666666,2.251,135.807,0.0,0.0,,,33261.933333333334,1.437,86.079,0,0,0,0,ok,complete
wal,2000,writer_rate,33,0,2000,33,0,65915.2,10.119,262.399,0.0,0.0,,,54853.333333333336,10.063,107.071,0,0,0,0,ok,complete
memory,2000,writer_rate,0,0,2000,0,0,139737.13333333333,9.423,47.615,0.0,0.0,,,139265.1,11.679,62.783,0,0,0,0,ok,complete
memory,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.705,122.943,0.0,0.0,,,3339.0,0.929,139.519,0,0,0,0,ok,complete
memory,2000,writer_rate,8,0,2000,8,0,16051.533333333333,1.799,53.471,0.0,0.0,,,15946.133333333333,1.078,51.135,0,0,0,0,ok,complete
memory,2000,writer_rate,20,0,2000,20,0,39976.4,1.925,37.727,0.0,0.0,,,39816.73333333333,1.111,4.515,0,0,0,0,ok,complete
memory,2000,writer_rate,33,0,2000,33,0,65893.53333333334,2.189,63.167,0.0,0.0,,,65674.93333333333,1.242,6.623,0,0,0,0,ok,complete
wal,2000,writer_rate,0,0,2000,0,0,84265.16666666667,22.399,70.463,0.0,0.0,,,64053.333333333336,36.735,68.991,0,0,0,0,ok,complete
wal,2000,writer_rate,2,0,2000,2,0,4066.6666666666665,1.704,146.687,0.0,0.0,,,4000.0,1.015,147.327,0,0,0,0,ok,complete
wal,2000,writer_rate,8,0,2000,8,0,16051.533333333333,1.799,53.471,0.0,0.0,,,15946.133333333333,1.078,51.135,0,0,0,0,ok,complete
wal,2000,writer_rate,20,0,2000,20,0,39980.7,2.363,131.071,0.0,0.0,,,33258.73333333333,1.517,85.567,0,0,0,0,ok,complete
wal,2000,writer_rate,33,0,2000,33,0,65920.33333333333,10.735,304.383,0.0,0.0,,,65681.7,10.279,91.519,0,0,0,0,ok,complete
116 changes: 58 additions & 58 deletions results/canonical-mixed-delivery/aggregate.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@
"subscribers": 2000,
"writer_rate": 0,
"read_rate": 0,
"write_ops_per_sec": 127139.0,
"write_p50": 14.527,
"write_p99": 41.279,
"write_ops_per_sec": 139737.13333333333,
"write_p50": 9.423,
"write_p99": 47.615,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 126729.93333333333,
"delivery_p50": 12.015,
"delivery_p99": 45.023,
"events_per_sec": 139265.1,
"delivery_p50": 11.679,
"delivery_p99": 62.783,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -35,15 +35,15 @@
"writer_rate": 2,
"read_rate": 0,
"write_ops_per_sec": 4066.6666666666665,
"write_p50": 1.682,
"write_p99": 129.407,
"write_p50": 1.705,
"write_p99": 122.943,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 3341.3333333333335,
"delivery_p50": 0.924,
"delivery_p99": 150.655,
"events_per_sec": 3339.0,
"delivery_p50": 0.929,
"delivery_p99": 139.519,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -60,16 +60,16 @@
"subscribers": 2000,
"writer_rate": 8,
"read_rate": 0,
"write_ops_per_sec": 16047.333333333334,
"write_p50": 1.654,
"write_p99": 3.761,
"write_ops_per_sec": 16051.533333333333,
"write_p50": 1.799,
"write_p99": 53.471,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 13305.3,
"delivery_p50": 0.954,
"delivery_p99": 5.063,
"events_per_sec": 15946.133333333333,
"delivery_p50": 1.078,
"delivery_p99": 51.135,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -86,16 +86,16 @@
"subscribers": 2000,
"writer_rate": 20,
"read_rate": 0,
"write_ops_per_sec": 39984.833333333336,
"write_p50": 1.822,
"write_p99": 3.033,
"write_ops_per_sec": 39976.4,
"write_p50": 1.925,
"write_p99": 37.727,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 33226.433333333334,
"delivery_p50": 1.09,
"delivery_p99": 2.333,
"events_per_sec": 39816.73333333333,
"delivery_p50": 1.111,
"delivery_p99": 4.515,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -112,16 +112,16 @@
"subscribers": 2000,
"writer_rate": 33,
"read_rate": 0,
"write_ops_per_sec": 65920.43333333333,
"write_p50": 2.101,
"write_p99": 93.759,
"write_ops_per_sec": 65893.53333333334,
"write_p50": 2.189,
"write_p99": 63.167,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 65691.26666666666,
"delivery_p50": 1.214,
"delivery_p99": 8.367,
"events_per_sec": 65674.93333333333,
"delivery_p50": 1.242,
"delivery_p99": 6.623,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -138,16 +138,16 @@
"subscribers": 2000,
"writer_rate": 0,
"read_rate": 0,
"write_ops_per_sec": 85168.3,
"write_p50": 21.935,
"write_p99": 95.871,
"write_ops_per_sec": 84265.16666666667,
"write_p50": 22.399,
"write_p99": 70.463,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 63848.26666666667,
"delivery_p50": 35.807,
"delivery_p99": 105.599,
"events_per_sec": 64053.333333333336,
"delivery_p50": 36.735,
"delivery_p99": 68.991,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -165,15 +165,15 @@
"writer_rate": 2,
"read_rate": 0,
"write_ops_per_sec": 4066.6666666666665,
"write_p50": 1.67,
"write_p99": 121.151,
"write_p50": 1.704,
"write_p99": 146.687,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 3336.4666666666667,
"delivery_p50": 1.011,
"delivery_p99": 137.471,
"events_per_sec": 4000.0,
"delivery_p50": 1.015,
"delivery_p99": 147.327,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -190,16 +190,16 @@
"subscribers": 2000,
"writer_rate": 8,
"read_rate": 0,
"write_ops_per_sec": 16048.966666666667,
"write_p50": 1.693,
"write_p99": 58.143,
"write_ops_per_sec": 16051.533333333333,
"write_p50": 1.799,
"write_p99": 53.471,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 15944.433333333332,
"delivery_p50": 1.063,
"delivery_p99": 54.175,
"events_per_sec": 15946.133333333333,
"delivery_p50": 1.078,
"delivery_p99": 51.135,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -216,16 +216,16 @@
"subscribers": 2000,
"writer_rate": 20,
"read_rate": 0,
"write_ops_per_sec": 39977.566666666666,
"write_p50": 2.251,
"write_p99": 135.807,
"write_ops_per_sec": 39980.7,
"write_p50": 2.363,
"write_p99": 131.071,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 33261.933333333334,
"delivery_p50": 1.437,
"delivery_p99": 86.079,
"events_per_sec": 33258.73333333333,
"delivery_p50": 1.517,
"delivery_p99": 85.567,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand All @@ -242,16 +242,16 @@
"subscribers": 2000,
"writer_rate": 33,
"read_rate": 0,
"write_ops_per_sec": 65915.2,
"write_p50": 10.119,
"write_p99": 262.399,
"write_ops_per_sec": 65920.33333333333,
"write_p50": 10.735,
"write_p99": 304.383,
"read_ops_per_sec": 0.0,
"read_mib_per_sec": 0.0,
"read_p50": null,
"read_p99": null,
"events_per_sec": 54853.333333333336,
"delivery_p50": 10.063,
"delivery_p99": 107.071,
"events_per_sec": 65681.7,
"delivery_p50": 10.279,
"delivery_p99": 91.519,
"write_bp": 0,
"write_err": 0,
"read_bp": 0,
Expand Down
Loading