README: DeepSWE benchmark results

README.md

@@ -32,6 +32,10 @@ hand work off, see what is moving across every project, and step in only where
 your judgment is needed. A factory, on your own machine, and the more you hand it
 the more it does.
 
+**#1 on DeepSWE** of ten coding harnesses on the same model, ahead of Claude Code,
+Codex, OpenCode, Kilo and DeepSeek's own harness, at the lowest cost per solved
+issue ([benchmarks](#benchmarks)).
+
 Written in Go as one small binary, with nothing else to install or run. Apache
 2.0. By [AgentField AI](https://agentfield.ai?utm_source=github-readme&utm_campaign=codeaf-readme&utm_id=codeaf-readme-byline).
 
@@ -51,6 +55,9 @@ curl -fsSL https://agentfield.ai/get/codeaf | bash
 codeaf
 ```
 
+<details>
+<summary>Pin a version, or build from source</summary>
+
 The script puts the release binary for your platform in `~/.codeaf/bin`. To
 pin a version, give it a tag from the
 [releases page](https://github.com/Agent-Field/codeaf/releases), where the
@@ -63,6 +70,8 @@ curl -fsSL https://agentfield.ai/get/codeaf | VERSION=<tag> bash
 To build it yourself: `git clone`, `make build`, `bin/codeaf`
 ([guide](docs/GUIDE.md#install)).
 
+</details>
+
 On first start it connects OpenRouter in your browser, or takes a key. Codex signs in
 a ChatGPT plan from `/connect` or `codeaf connect codex`; DeepSeek, GLM, Kimi, MiniMax
 and Qwen take keys; Ollama needs none.
@@ -133,19 +142,21 @@ A run is a task like any other, on `home`, with a room and a stop.
 
 - **Coming soon, native:** [PR-AF](https://github.com/Agent-Field/pr-af), the #1
   open-source code reviewer on Martian Code-Review-Bench.
-- **Coming soon, in the benchmark below:** the developer subharness against
-  general harnesses on the same open model.
+- **Native now:** `/senior-dev`, the developer subharness. First of ten
+  harnesses on DeepSWE, in the [benchmark below](#benchmarks).
 - **Your own:** "make me a harness for triaging flaky tests" designs one, saves
   it, and `/subharness` runs it.
 
 ## Benchmarks
 
-Coming soon. The run is held-out GitHub issues, several seeds each, through
-CodeAF's developer subharness and the general harnesses on the same open model:
-pass rate, cost per issue and time per issue, with every failure, timeout and
-unpriced call written up in [BENCHMARKS.md](BENCHMARKS.md). The chart and the
-table land here when the run completes, and `bench/` runs it on your own
-repository.
+<img src="assets/readme/benchmark-deepswe.webp" alt="First on DeepSWE: senior-dev, CodeAF's developer subharness, solved the most tasks (54.9%) at the lowest cost per solved task (1x). Every other harness solved less and paid more per solve: mini-swe-agent 1.9x, codex 2.1x, pi 2.4x, claude-code 3.4x, omp, kilo and opencode about 4.5x, muse-code 11.3x, deepseek-harness 26.6x." width="100%">
+
+`/senior-dev`, CodeAF's developer subharness, against nine other coding harnesses
+on the same model, DeepSeek V4 Flash: 113 real GitHub issues from DeepSWE, graded
+by the official verifiers. It solved the most issues and paid the least for each
+one it solved.
+
+Every number, the method and the limits: [docs/benchmarks/deepswe](docs/benchmarks/deepswe/).
 
 ## The right model for each call
 

docs/TELEMETRY.md

@@ -17,15 +17,8 @@ codeaf sends anonymous usage counts to AgentField.
   Turn off:                 CODEAF_TELEMETRY=off
 ```
 
-The installer prints a three-line form of the same notice, to stderr, after the
-`installed codeaf …` receipt and before the `export PATH` line. The full notice
-above still arrives at the first session:
-
-```
-codeaf shares anonymous performance data with AgentField
-codeaf does NOT share your prompts, code, files, or any private information
-see what is shared: codeaf telemetry info · turn off: CODEAF_TELEMETRY=off
-```
+The installer prints nothing about telemetry; the notice above arrives with the
+first session, before anything is sent.
 
 ## What is sent
 

docs/benchmarks/deepswe/README.md

@@ -0,0 +1,61 @@
+# DeepSWE harness comparison
+
+Ten coding harnesses, one model, the same 113 tasks, one attempt each.
+
+| harness | solved | cost per task | cost per solved issue | mean time |
+| --- | --- | --- | --- | --- |
+| **senior-dev** | **62 of 113, 54.9%** | **22¢** | **1x** | 54 min |
+| mini-swe-agent | 56, 49.6% | 38¢ | 1.9x | 44 min |
+| codex | 51, 45.1% | 37¢ | 2.1x | 46 min |
+| pi | 42, 37.2% | 35¢ | 2.4x | 52 min |
+| omp | 31, 27.4% | 50¢ | 4.5x | 49 min |
+| opencode | 30, 26.6% | 50¢ | 4.8x | 48 min |
+| kilo | 30, 26.6% | 48¢ | 4.6x | 54 min |
+| claude-code | 16, 14.2% | 19¢ | 3.4x | 32 min |
+| deepseek-harness | 16, 14.2% | 150¢ | 26.6x | 94 min |
+| muse-code | 3, 2.7% | 12¢ | 11.3x | 16 min |
+
+senior-dev solved the most issues and paid the least for each one it solved:
+nearly 4x the issues claude-code solved, at about half the cost per solve of the
+next best harness.
+
+Since then, on the same 113 tasks: 88 solved (77.9%, exact 95% CI 69.1% to 85.1%)
+with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. Those runs
+are senior-dev alone, not a comparison.
+
+## Setup
+
+| | |
+| --- | --- |
+| Benchmark | full DeepSWE set, 113 tasks, one seed per harness |
+| Model | `deepseek/deepseek-v4-flash-0731` through OpenRouter |
+| Verifiers | official DeepSWE at `0b9fabb` |
+| Budget | 3 h per task |
+| Isolation | four shards per harness, a dedicated OpenRouter key per harness |
+| Ran | nine harnesses on 2026-09-11; senior-dev on 2026-09-12 |
+
+[`arms.csv`](arms.csv) has one row per harness: solved, reward rate, valid
+grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task,
+mean agent seconds and model.
+
+## How the columns in the README are derived
+
+- **solved**: tasks the verifier passed, out of 113.
+- **cost per task**: billed OpenRouter spend divided by 113, so tasks without a
+  verifier result stay in the denominator.
+- **cost per solved issue**: spend divided by tasks solved, shown as a multiple
+  of senior-dev's (39.9¢).
+- **mean time**: mean agent wall time per task.
+
+## Limits
+
+- One seed per harness. senior-dev's 62 against mini-swe-agent's 56 is not a
+  statistically resolved difference.
+- senior-dev departs from the sampling contract: the other nine sent temperature
+  1.0 and top-p 0.95, senior-dev sent neither, so provider defaults applied.
+- Five tasks produced no verifier outcome: codex 1, pi 1, omp 2, opencode 1.
+  They count as unsolved.
+- omp, opencode, kilo, deepseek-harness, claude-code and muse-code did not record
+  per-attempt cost; their spend is known at harness level only.
+- senior-dev ran under an earlier name for the binary; values are rewritten to
+  `senior-dev`.

docs/benchmarks/deepswe/arms.csv

@@ -0,0 +1,11 @@
+harness,solved,tasks,reward_rate,valid_grades,invalid,mean_f2p,mean_p2p,openrouter_spend_usd,cost_per_task_usd,mean_agent_seconds,model
+senior-dev,62,113,0.5487,113,0,0.8864,0.9942,24.73,0.2188,3258,openrouter/deepseek/deepseek-v4-flash-0731
+mini-swe-agent,56,113,0.4956,113,0,0.8817,0.9966,43.09,0.3813,2651,openrouter/deepseek/deepseek-v4-flash-0731
+codex,51,113,0.4513,112,1,0.8681,0.9876,42.23,0.3737,2746,deepseek/deepseek-v4-flash-0731
+pi,42,113,0.3717,112,1,0.7163,0.8909,39.61,0.3505,3101,openrouter/deepseek/deepseek-v4-flash-0731
+omp,31,113,0.2743,111,2,0.7191,0.8526,55.98,0.4954,2920,openrouter/deepseek/deepseek-v4-flash-0731
+opencode,30,113,0.2655,112,1,0.7113,0.8679,56.96,0.5041,2871,openrouter/deepseek/deepseek-v4-flash-0731
+kilo,30,113,0.2655,113,0,0.6903,0.9051,54.65,0.4836,3215,openrouter/deepseek/deepseek-v4-flash-0731
+deepseek-harness,16,113,0.1416,113,0,0.5339,0.9145,169.47,1.4997,5613,openrouter/deepseek/deepseek-v4-flash-0731
+claude-code,16,113,0.1416,113,0,0.3972,0.916,21.72,0.1922,1909,deepseek/deepseek-v4-flash-0731
+muse-code,3,113,0.0265,113,0,0.0514,0.9901,13.54,0.1198,952,deepseek/deepseek-v4-flash-0731

docs/changes/unreleased/1511-readme-deepswe-benchmark.md

@@ -0,0 +1,11 @@
+---
+kind: changed
+title: the README's Benchmarks section carries the DeepSWE result
+pr: 1511
+surface: [build, docs]
+invalidates:
+  - "The README's Benchmarks section said the run was coming soon. It shows the ten-harness DeepSWE result now, and every number lives in docs/benchmarks/deepswe."
+  - "The installer printed a three-line telemetry notice after the `installed codeaf` receipt. It prints none now; the binary's full notice still arrives before the first session's events are sent."
+  - "When the install marker could not be written, the installer printed a line naming its telemetry folder. The marker is still written when it can be, and a failure is now silent."
+  - "docs/TELEMETRY.md carried a second fenced block, the installer's three-line form, and test/installer-telemetry.sh compared the installer against it. That block is gone and the test asserts the installer prints no notice."
+---

scripts/install.sh

@@ -7,14 +7,6 @@ LEGACY_REPOSITORY="Agent-Field/aforge-v2" # Remove after the one-release reposit
 CHANNEL="${CHANNEL:-stable}"
 INSTALL_NAME="${CODEAF_INSTALL_NAME:-codeaf}"
 VERSION="${VERSION:-}"
-# The installer's three-line telemetry notice, verbatim from docs/TELEMETRY.md.
-# The binary prints the full notice before the first session's events leave;
-# the installer says the fact, the inspector and the switch. It writes a local
-# install marker and prints this text, never sends telemetry, and makes no
-# request that the download steps did not already make.
-TELEMETRY_NOTICE='codeaf shares anonymous performance data with AgentField
-codeaf does NOT share your prompts, code, files, or any private information
-see what is shared: codeaf telemetry info · turn off: CODEAF_TELEMETRY=off'
 VERBOSE="${VERBOSE:-0}"
 NO_MODIFY_PATH="${CODEAF_NO_MODIFY_PATH:-${AFORGE_NO_MODIFY_PATH:-0}}" # legacy-name
 INSTALL_DIR="${CODEAF_INSTALL_DIR:-${AFORGE_INSTALL_DIR:-${HOME}/.codeaf/bin}}" # legacy-name
@@ -51,7 +43,6 @@ Environment:
   CODEAF_NO_MODIFY_PATH, VERBOSE
   GITHUB_TOKEN or GH_TOKEN: GitHub answers anonymous API calls sixty times an hour per address; a token raises that.
   CODEAF_GITHUB_API and CODEAF_GITHUB_DOWNLOAD for mirrors and tests
-  CODEAF_TELEMETRY=off, or DO_NOT_TRACK=1, turns the anonymous usage counts off
 EOF
 }
 
@@ -66,18 +57,6 @@ usage_error() {
   exit 2
 }
 
-telemetry_off() {
-  local value
-  case "${DO_NOT_TRACK:-}" in
-    1|[Tt]|[Tt][Rr][Uu][Ee]) return 0 ;;
-  esac
-  value=$(echo "${CODEAF_TELEMETRY:-}" | tr '[:upper:]' '[:lower:]')
-  case "$value" in
-    off|0|false) return 0 ;;
-  esac
-  return 1
-}
-
 # The install marker is a local record only: it names how and when this
 # machine's codeaf was installed so the usage counts can bucket by channel,
 # never who installed it. Nothing here contacts the network.
@@ -92,22 +71,19 @@ write_install_marker() {
     stable|rc|staging|dev) channel="$CHANNEL" ;;
     *) channel="unknown" ;;
   esac
+  # The marker is background bookkeeping: a failure skips it without a word.
   if ! mkdir -p "$directory/telemetry" 2>/dev/null; then
-    printf 'codeaf: could not create %s/telemetry; skipping the install marker\n' "$directory" >&2
     return 0
   fi
   chmod 0700 "$directory/telemetry"
   file="$directory/telemetry/install.json"
   if ! { printf '{"install_method":"script","channel":"%s","installed_at":"%s"}' \
     "$channel" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"; } > "$file" 2>/dev/null; then
-    printf 'codeaf: could not write %s; skipping the install marker\n' "$file" >&2
     return 0
   fi
   chmod 0600 "$file"
 }
 
-# Printed once, at the very end of a successful install. The binary repeats it
-# before the first session's counts are ever sent.
 # The one line a person still has to paste, printed last of all, between a
 # blank line above and a blank line below, bold green on a terminal. Bare
 # `export PATH=...` and nothing else, so it can be selected and pasted without
@@ -124,14 +100,6 @@ print_path_hint() {
   printf '\n%s%s%s\n\n' "$on" "$hint" "$off"
 }
 
-print_telemetry_notice() {
-  if telemetry_off; then
-    printf 'codeaf: anonymous usage counts are off (CODEAF_TELEMETRY=off or DO_NOT_TRACK=1)\n' >&2
-    return 0
-  fi
-  printf '\n%s\n' "$TELEMETRY_NOTICE" >&2
-}
-
 while [[ $# -gt 0 ]]; do
   case "$1" in
     --stable) CHANNEL="stable"; shift ;;
@@ -509,7 +477,7 @@ append_path_line() {
 }
 
 # The PATH line is not printed here. It is the last thing the installer says,
-# after `codeaf version` and the telemetry notice, so the one line a person
+# after `codeaf version`, so the one line a person
 # has to paste sits at the bottom of the screen where their eye already is.
 PATH_HINT=""
 if [[ "$OS" != "windows" ]] && ! path_has_dir; then
@@ -552,5 +520,4 @@ printf 'installed %s\n' "$version_line"
 if [[ "$RUN_BOOT_ADOPTION" == "1" || -d "$STATE_ROOT" ]]; then
   write_install_marker "$STATE_ROOT"
 fi
-print_telemetry_notice
 print_path_hint "$PATH_HINT"

test/installer-telemetry.sh

@@ -1,11 +1,11 @@
 #!/usr/bin/env bash
 # THE INSTALLER'S TELEMETRY DUTIES, PROVED WITHOUT A NETWORK.
 #
-# The notice text lives once, byte for byte, in docs/TELEMETRY.md: the full
-# form quoted by the binary and the README, and the three-line form the
-# installer prints. Only a test notices when one of them drifts. The installer's main body
+# The installer writes the local install marker and prints no telemetry notice:
+# the binary shows the notice, quoted byte for byte in docs/TELEMETRY.md and the
+# README, before the first session's events are sent. The installer's main body
 # downloads a release, so this test never sources it whole: it lifts out the
-# three telemetry functions and runs them against a temporary state root.
+# marker and PATH functions and runs them against a temporary state root.
 # Nothing here opens a socket.
 set -euo pipefail
 cd "$(git rev-parse --show-toplevel)"
@@ -14,11 +14,8 @@ script=scripts/install.sh
 doc=docs/TELEMETRY.md
 test -f "$doc" || { echo "docs/TELEMETRY.md is missing; this test reads the notice from it"; exit 1; }
 
-eval "$(awk '/^TELEMETRY_NOTICE=/{f=1} /^VERBOSE=/{f=0} f' "$script")"
-eval "$(sed -n '/^telemetry_off()/,/^}/p; /^write_install_marker()/,/^}/p; /^print_telemetry_notice()/,/^}/p; /^print_path_hint()/,/^}/p' "$script")"
-type telemetry_off >/dev/null
+eval "$(sed -n '/^write_install_marker()/,/^}/p; /^print_path_hint()/,/^}/p' "$script")"
 type write_install_marker >/dev/null
-type print_telemetry_notice >/dev/null
 type print_path_hint >/dev/null
 
 pass=0
@@ -78,56 +75,24 @@ CHANNEL='we"ird' write_install_marker "$tmp/state"
 ok "unexpected channel written as unknown" 'grep -q "\"channel\":\"unknown\"" "$f"'
 ok "unexpected-channel marker is valid JSON" 'json_valid'
 ok "unwritable state root does not fail the install" 'write_install_marker /proc/nonexistent-root'
+ok "an unwritable state root says nothing" '[ -z "$(write_install_marker /proc/nonexistent-root 2>&1)" ]'
 
-# --- the notice -------------------------------------------------------------
+# --- no notice from the installer --------------------------------------------
 
-notice="$tmp/notice.txt"
-print_telemetry_notice 2> "$notice"
-# The first fenced block under "The notice" is the binary's full notice, the
-# second is the installer's three-line form; awk counts fences to tell them apart.
+# The binary shows the notice at the first session; the installer shows none.
+ok "installer prints no telemetry notice" '! grep -qE "anonymous (performance data|usage counts)|TELEMETRY_NOTICE|print_telemetry_notice" "$script"'
+ok "no printed line in the installer mentions telemetry" '! grep -E "printf|echo" "$script" | grep -qi telemetry'
 expected=$(awk '
 	/^## The notice$/ {f=1; next}
 	f && /^```$/ {f++; next}
 	f == 2 {print}
 ' "$doc")
-installer_expected=$(awk '
-	/^## The notice$/ {f=1; next}
-	f && /^```$/ {f++; next}
-	f == 4 {print}
-' "$doc")
-body=$(sed 1d "$notice")
-ok "one blank line before the notice" '[ -z "$(head -n 1 "$notice")" ]'
-ok "installer notice matches docs/TELEMETRY.md verbatim" '[ "$body" = "$installer_expected" ]'
-ok "installer notice is three lines" '[ "$(printf "%s\n" "$body" | wc -l | tr -d " ")" = 3 ]'
-ok "installer notice names the inspector and the switch" 'case "$body" in *"codeaf telemetry info"*CODEAF_TELEMETRY=off*) true;; *) false;; esac'
-ok "installer notice names what is never shared" 'case "$body" in *"does NOT share your prompts, code, files"*) true;; *) false;; esac'
 readme_block=$(awk '
 	/^```text$/ {f = 1; buf = ""; next}
 	/^```$/     {if (f && buf ~ /codeaf sends anonymous usage counts/) {print buf; exit} f = 0; next}
 	f           {buf = buf $0 "\n"}
 ' README.md)
 ok "README quotes the notice verbatim" '[ -n "$readme_block" ] && [ "$(printf "%s\n" "$expected")" = "$readme_block" ]'
-ok "notice goes to stderr, nothing to stdout" '[ -z "$(print_telemetry_notice 2>/dev/null)" ]'
-
-for v in off 0 false OFF False; do
-	export CODEAF_TELEMETRY="$v"
-	out=$( print_telemetry_notice 2>&1 )
-	ok "CODEAF_TELEMETRY=$v opts out" 'case "$out" in *"off"*) true;; *) false;; esac'
-	ok "opt-out prints no notice body" 'case "$out" in *"anonymous performance data"*) false;; *) true;; esac'
-	unset CODEAF_TELEMETRY
-done
-for v in 1 true TRUE; do
-	export DO_NOT_TRACK="$v"
-	out=$( print_telemetry_notice 2>&1 )
-	ok "DO_NOT_TRACK=$v opts out" 'case "$out" in *"off"*) true;; *) false;; esac'
-	unset DO_NOT_TRACK
-done
-out=$( print_telemetry_notice 2>&1 )
-ok "unset prints the notice" 'case "$out" in *"anonymous performance data with AgentField"*) true;; *) false;; esac'
-export CODEAF_TELEMETRY=1
-out=$( print_telemetry_notice 2>&1 )
-ok "CODEAF_TELEMETRY=1 prints the notice" 'case "$out" in *"anonymous performance data with AgentField"*) true;; *) false;; esac'
-unset CODEAF_TELEMETRY
 
 # --- the PATH line comes last -------------------------------------------------
 # The line a person has to paste is the installer's final word: bare, after a