Skip to content

bench: add ioxide 0.4.169 verbatim as an A/B reference - #1376

Closed
MDA2AV wants to merge 2 commits into
mainfrom
bench/ioxide-0-4-169-ab
Closed

bench: add ioxide 0.4.169 verbatim as an A/B reference#1376
MDA2AV wants to merge 2 commits into
mainfrom
bench/ioxide-0-4-169-ab

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 29, 2026

Copy link
Copy Markdown
Owner

baseline-4096 fell from 4,108,420 rps at 6401.5% CPU (0.4.169, 2026-08-09) to 3,896,541 at 5983.6% (0.7.211, 2026-08-28). Nothing distinguishes a library regression from a change in the machine, because no measurement exists for 0.7.210 on its own and the two runs are three weeks apart.

This is the entry as it stood at cc794d0, restored file for file, so both can run back to back in one session. Same numbers as today's ioxide means the box moved; ~4.1M again means it is the library.

Disabled: it never sends FIN on Connection: close, which is what #1289 disabled the live entry for and 0.7.210 fixed, so it does not pass validate.sh. benchmark.sh does not check enabled and the workflow's filter only applies to framework=all, so it still runs by name.

Two deviations from verbatim: nuget.config drops the local developer feed for then-unpublished packages, and tests is trimmed to baseline and json since static, crud, api-4 and api-16 are no longer profiles.

Claude-Session: https://claude.ai/code/session_016iyXstHcyyy1y47jAoBHid

Description

PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run every test the framework subscribes to
/benchmark -f <framework> -t <test> Run one test only
/benchmark -f <framework> --save Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save Run one test and save results
/benchmark -f <framework> --compare <other> Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,... Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test> Benchmark all enabled frameworks subscribed to <test> and save the results

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp-11 --compare genhttp-11-kestrel

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.

Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.

@MDA2AV
bench: add ioxide 0.4.169 verbatim as an A/B reference
0d49503 
baseline-4096 fell from 4,108,420 rps at 6401.5% CPU (0.4.169, 2026-08-09) to
3,896,541 at 5983.6% (0.7.211, 2026-08-28). Nothing distinguishes a library
regression from a change in the machine, because no measurement exists for
0.7.210 on its own and the two runs are three weeks apart.

This is the entry as it stood at cc794d0, restored file for file, so both can
run back to back in one session. Same numbers as today's ioxide means the box
moved; ~4.1M again means it is the library.

Disabled: it never sends FIN on Connection: close, which is what #1289 disabled
the live entry for and 0.7.210 fixed, so it does not pass validate.sh.
benchmark.sh does not check enabled and the workflow's filter only applies to
framework=all, so it still runs by name.

Two deviations from verbatim: nuget.config drops the local developer feed for
then-unpublished packages, and tests is trimmed to baseline and json since
static, crud, api-4 and api-16 are no longer profiles.

Claude-Session: https://claude.ai/code/session_016iyXstHcyyy1y47jAoBHid
@MDA2AV

MDA2AV commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

/benchmark -f ioxide-0-4-169 -t baseline

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: ioxide-0-4-169 | Test: baseline

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 3,836,794 5899.0% 296MiB ~0% ~0%
baseline 4096 3,865,280 5743.6% 485MiB ~0% ~0%
Full log 
#12 DONE 3.2s

#13 [stage-1 2/3] WORKDIR /app
#13 CACHED

#14 [stage-1 3/3] COPY --from=build /app/out ./
#14 DONE 0.1s

#15 exporting to image
#15 exporting layers
#15 exporting layers 0.3s done
#15 exporting manifest sha256:716e8eaa52ca725ce2a537822f90fdf19e66803de8d659a2f2851b669ac2cef4 0.0s done
#15 exporting config sha256:b619ea05d73418caf59ae7160b2275140841ea30bc47d43fb5277652dc9495f2 0.0s done
#15 exporting attestation manifest sha256:3c9477a3660f5a4158351a9a658c5e5b16b95ef3063607d06637f0645cc42a67 0.0s done
#15 exporting manifest list sha256:d1299f9405090d6f04c520411d2d4437f3a2130e7c57ea06897aa23061089ac5
#15 exporting manifest list sha256:d1299f9405090d6f04c520411d2d4437f3a2130e7c57ea06897aa23061089ac5 0.0s done
#15 naming to docker.io/library/httparena-ioxide-0-4-169:latest done
#15 unpacking to docker.io/library/httparena-ioxide-0-4-169:latest 0.1s done
#15 DONE 0.5s
[info] tuning host for benchmark runs
[info] CPU governor → performance
[info] setting kernel socket limits
[info] setting UDP buffer sizes for QUIC
[info] setting loopback MTU to 1500 (realistic Ethernet)
[info] restarting docker daemon
[info] dropping kernel caches

==============================================
=== ioxide-0-4-169 / baseline / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    137us    120us    194us    482us   1.51ms

  18561585 requests in 5.00s, 18561584 responses
  Throughput: 3.71M req/s
  Bandwidth:  233.57MB/s
  Status codes: 2xx=18561584, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 18561554 / 18561584 responses (100.0%)
  Per-template: 6155266,6173349,6232939
  Per-template-ok: 6155266,6173349,6232939
[info] CPU 5810.8% | Mem 275MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    133us    119us    186us    397us   1.43ms

  19130071 requests in 5.00s, 19130069 responses
  Throughput: 3.82M req/s
  Bandwidth:  240.74MB/s
  Status codes: 2xx=19130069, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19130003 / 19130069 responses (100.0%)
  Per-template: 6269051,6389054,6471898
  Per-template-ok: 6269051,6389054,6471898
[info] CPU 5974.9% | Mem 298MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    132us    119us    185us    423us   1.81ms

  19183972 requests in 5.00s, 19183971 responses
  Throughput: 3.83M req/s
  Bandwidth:  241.35MB/s
  Status codes: 2xx=19183971, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19190625 / 19183971 responses (100.0%)
  Per-template: 6265111,6431438,6487338
  Per-template-ok: 6265111,6431437,6487338
[info] CPU 5899.0% | Mem 296MiB

=== Best: 3836794 req/s (CPU: 5899.0%, Mem: 296MiB) ===
[info] input BW: 296.38MB/s (avg template: 81 bytes)
[info] saved results/baseline/512/ioxide-0-4-169.json
httparena-bench-ioxide-0-4-169
httparena-bench-ioxide-0-4-169

==============================================
=== ioxide-0-4-169 / baseline / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.16ms    958us   1.31ms   4.86ms   11.20ms

  17684661 requests in 5.00s, 17684661 responses
  Throughput: 3.54M req/s
  Bandwidth:  222.55MB/s
  Status codes: 2xx=17684661, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 17684661 / 17684661 responses (100.0%)
  Per-template: 5876897,5930220,5877544
  Per-template-ok: 5876897,5930220,5877544
[info] CPU 5318.4% | Mem 347MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.07ms    961us   1.21ms   4.16ms   7.24ms

  19066318 requests in 5.00s, 19065038 responses
  Throughput: 3.81M req/s
  Bandwidth:  239.91MB/s
  Status codes: 2xx=19065038, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19065017 / 19065038 responses (100.0%)
  Per-template: 6388093,6329483,6347441
  Per-template-ok: 6388093,6329483,6347441
[info] CPU 5931.1% | Mem 476MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.06ms    979us   1.22ms   4.12ms   6.65ms

  19326977 requests in 5.00s, 19326401 responses
  Throughput: 3.86M req/s
  Bandwidth:  243.22MB/s
  Status codes: 2xx=19326401, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19326308 / 19326401 responses (100.0%)
  Per-template: 6416877,6433947,6475484
  Per-template-ok: 6416877,6433947,6475484
[info] CPU 5743.6% | Mem 485MiB

=== Best: 3865280 req/s (CPU: 5743.6%, Mem: 485MiB) ===
[info] input BW: 298.58MB/s (avg template: 81 bytes)
[info] saved results/baseline/4096/ioxide-0-4-169.json
httparena-bench-ioxide-0-4-169
httparena-bench-ioxide-0-4-169
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/ioxide-0.4.169.json - 2 new, 2 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@MDA2AV
bench: pin the .NET image too, the runtime was the uncontrolled variable
29d85a0 
The first cut of this entry pinned ioxide at 0.4.169 and left the base image on
the floating 11.0-preview tag, so both arms ran the same runtime. The old entry
regressed as well, which says nothing about the library.

That tag moved between the two measurements:

  11.0.0-preview.6   published 2026-07-28   preview.6.26359.118
  11.0.0-preview.7   published 2026-08-28   preview.7.26381.103

baseline-4096 was 4,108,420 on 2026-08-09, when 11.0-preview was preview.6, and
3,896,541 on 2026-08-28 -- the day preview.7 shipped.

Both images are now pinned, through separate args because the repositories
version differently: 11.0.100-preview.N for the SDK, 11.0.0-preview.N for the
runtime. Verified in the built image: preview.6.26359.118, and it serves.

Claude-Session: https://claude.ai/code/session_016iyXstHcyyy1y47jAoBHid
@MDA2AV

MDA2AV commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

/benchmark -f ioxide-0-4-169 -t baseline

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: ioxide-0-4-169 | Test: baseline

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 4,137,276 6256.6% 266MiB ~0% ~0%
baseline 4096 4,116,724 6159.3% 312MiB ~0% ~0%
Full log 
#12 DONE 4.1s

#13 [stage-1 2/3] WORKDIR /app
#13 CACHED

#14 [stage-1 3/3] COPY --from=build /app/out ./
#14 DONE 0.1s

#15 exporting to image
#15 exporting layers
#15 exporting layers 0.3s done
#15 exporting manifest sha256:de67aae13d5ceb5779626954228592147bc6d05e4f3a5ff73d72085648c005e1 0.0s done
#15 exporting config sha256:908449e73d08ea74933e0d0ae8d692adf13aff69f676212d2d4e687da8860c2b 0.0s done
#15 exporting attestation manifest sha256:7b8e77d6c80834e2c462e59624732855a4c6529e19872c83b42488316a681a0c 0.0s done
#15 exporting manifest list sha256:679cce21a90ed360af2e67a0af5fc44ec472c5581dfb4646215fea5db21bfdba
#15 exporting manifest list sha256:679cce21a90ed360af2e67a0af5fc44ec472c5581dfb4646215fea5db21bfdba 0.0s done
#15 naming to docker.io/library/httparena-ioxide-0-4-169:latest done
#15 unpacking to docker.io/library/httparena-ioxide-0-4-169:latest 0.1s done
#15 DONE 0.6s
[info] tuning host for benchmark runs
[info] CPU governor → performance
[info] setting kernel socket limits
[info] setting UDP buffer sizes for QUIC
[info] setting loopback MTU to 1500 (realistic Ethernet)
[info] restarting docker daemon
[info] dropping kernel caches

==============================================
=== ioxide-0-4-169 / baseline / 512c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    128us    117us    185us    342us    672us

  19881035 requests in 5.00s, 19881035 responses
  Throughput: 3.98M req/s
  Bandwidth:  250.20MB/s
  Status codes: 2xx=19881035, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19881018 / 19881035 responses (100.0%)
  Per-template: 6550520,6569146,6761352
  Per-template-ok: 6550520,6569146,6761352
[info] CPU 6037.1% | Mem 264MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    124us    119us    176us    252us    457us

  20525582 requests in 5.00s, 20525581 responses
  Throughput: 4.10M req/s
  Bandwidth:  258.31MB/s
  Status codes: 2xx=20525581, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 20525514 / 20525581 responses (100.0%)
  Per-template: 6905222,6734568,6885724
  Per-template-ok: 6905222,6734568,6885724
[info] CPU 6332.2% | Mem 264MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     512 (8/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    123us    117us    179us    238us    446us

  20686380 requests in 5.00s, 20686380 responses
  Throughput: 4.13M req/s
  Bandwidth:  260.20MB/s
  Status codes: 2xx=20686380, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 20686300 / 20686380 responses (100.0%)
  Per-template: 6706874,7004169,6975257
  Per-template-ok: 6706874,7004169,6975257
[info] CPU 6256.6% | Mem 266MiB

=== Best: 4137276 req/s (CPU: 6256.6%, Mem: 266MiB) ===
[info] input BW: 319.59MB/s (avg template: 81 bytes)
[info] saved results/baseline/512/ioxide-0-4-169.json
httparena-bench-ioxide-0-4-169
httparena-bench-ioxide-0-4-169

==============================================
=== ioxide-0-4-169 / baseline / 4096c (tool=gcannon) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency   1.03ms    986us   1.20ms   2.31ms   6.35ms

  19868489 requests in 5.00s, 19868489 responses
  Throughput: 3.97M req/s
  Bandwidth:  250.02MB/s
  Status codes: 2xx=19868489, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 19868489 / 19868489 responses (100.0%)
  Per-template: 6611323,6598592,6658574
  Per-template-ok: 6611323,6598592,6658574
[info] CPU 5974.9% | Mem 298MiB

[run 2/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    999us    987us   1.18ms   1.33ms   4.16ms

  20485114 requests in 5.00s, 20483258 responses
  Throughput: 4.09M req/s
  Bandwidth:  257.70MB/s
  Status codes: 2xx=20483258, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 20483258 / 20483258 responses (100.0%)
  Per-template: 6831445,6794716,6857097
  Per-template-ok: 6831445,6794716,6857097
[info] CPU 6426.8% | Mem 315MiB

[run 3/3]
gcannon v0.5.3
  Target:    localhost:8080/
  Threads:   64
  Conns:     4096 (64/thread)
  Pipeline:  1
  Req/conn:  unlimited (keep-alive)
  Templates: 3
  Expected:  200
  Duration:  5s


  Thread Stats   Avg      p50      p90      p99    p99.9
    Latency    994us    990us   1.17ms   1.28ms   1.93ms

  20584518 requests in 5.00s, 20583622 responses
  Throughput: 4.12M req/s
  Bandwidth:  259.02MB/s
  Status codes: 2xx=20583622, 3xx=0, 4xx=0, 5xx=0
  Latency samples: 20583622 / 20583622 responses (100.0%)
  Per-template: 6853562,6866324,6863736
  Per-template-ok: 6853562,6866324,6863736
[info] CPU 6159.3% | Mem 312MiB

=== Best: 4116724 req/s (CPU: 6159.3%, Mem: 312MiB) ===
[info] input BW: 318.01MB/s (avg template: 81 bytes)
[info] saved results/baseline/4096/ioxide-0-4-169.json
httparena-bench-ioxide-0-4-169
httparena-bench-ioxide-0-4-169
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/ioxide-0.4.169.json - 2 new, 2 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@MDA2AV MDA2AV closed this Aug 29, 2026
@MDA2AV
MDA2AV deleted the bench/ioxide-0-4-169-ab branch August 29, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Kaliumhexacyanoferrat Kaliumhexacyanoferrat Awaiting requested review from Kaliumhexacyanoferrat Kaliumhexacyanoferrat is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MDA2AV