diff --git a/README.md b/README.md index dee128d01d..c555736d03 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,10 @@ hand work off, see what is moving across every project, and step in only where your judgment is needed. A factory, on your own machine, and the more you hand it the more it does. +**#1 on DeepSWE** of ten coding harnesses on the same model, ahead of Claude Code, +Codex, OpenCode, Kilo and DeepSeek's own harness, at the lowest cost per solved +issue ([benchmarks](#benchmarks)). + Written in Go as one small binary, with nothing else to install or run. Apache 2.0. By [AgentField AI](https://agentfield.ai?utm_source=github-readme&utm_campaign=codeaf-readme&utm_id=codeaf-readme-byline). @@ -51,6 +55,9 @@ curl -fsSL https://agentfield.ai/get/codeaf | bash codeaf ``` +
+Pin a version, or build from source + The script puts the release binary for your platform in `~/.codeaf/bin`. To pin a version, give it a tag from the [releases page](https://github.com/Agent-Field/codeaf/releases), where the @@ -63,6 +70,8 @@ curl -fsSL https://agentfield.ai/get/codeaf | VERSION= bash To build it yourself: `git clone`, `make build`, `bin/codeaf` ([guide](docs/GUIDE.md#install)). +
+ On first start it connects OpenRouter in your browser, or takes a key. Codex signs in a ChatGPT plan from `/connect` or `codeaf connect codex`; DeepSeek, GLM, Kimi, MiniMax and Qwen take keys; Ollama needs none. @@ -133,19 +142,21 @@ A run is a task like any other, on `home`, with a room and a stop. - **Coming soon, native:** [PR-AF](https://github.com/Agent-Field/pr-af), the #1 open-source code reviewer on Martian Code-Review-Bench. -- **Coming soon, in the benchmark below:** the developer subharness against - general harnesses on the same open model. +- **Native now:** `/senior-dev`, the developer subharness. First of ten + harnesses on DeepSWE, in the [benchmark below](#benchmarks). - **Your own:** "make me a harness for triaging flaky tests" designs one, saves it, and `/subharness` runs it. ## Benchmarks -Coming soon. The run is held-out GitHub issues, several seeds each, through -CodeAF's developer subharness and the general harnesses on the same open model: -pass rate, cost per issue and time per issue, with every failure, timeout and -unpriced call written up in [BENCHMARKS.md](BENCHMARKS.md). The chart and the -table land here when the run completes, and `bench/` runs it on your own -repository. +First on DeepSWE: senior-dev, CodeAF's developer subharness, solved the most tasks (54.9%) at the lowest cost per solved task (1x). Every other harness solved less and paid more per solve: mini-swe-agent 1.9x, codex 2.1x, pi 2.4x, claude-code 3.4x, omp, kilo and opencode about 4.5x, muse-code 11.3x, deepseek-harness 26.6x. + +`/senior-dev`, CodeAF's developer subharness, against nine other coding harnesses +on the same model, DeepSeek V4 Flash: 113 real GitHub issues from DeepSWE, graded +by the official verifiers. It solved the most issues and paid the least for each +one it solved. + +Every number, the method and the limits: [docs/benchmarks/deepswe](docs/benchmarks/deepswe/). ## The right model for each call diff --git a/assets/readme/benchmark-deepswe.webp b/assets/readme/benchmark-deepswe.webp new file mode 100644 index 0000000000..0c6404068e Binary files /dev/null and b/assets/readme/benchmark-deepswe.webp differ diff --git a/docs/TELEMETRY.md b/docs/TELEMETRY.md index a0d1e1a665..17bc05bd17 100644 --- a/docs/TELEMETRY.md +++ b/docs/TELEMETRY.md @@ -17,15 +17,8 @@ codeaf sends anonymous usage counts to AgentField. Turn off: CODEAF_TELEMETRY=off ``` -The installer prints a three-line form of the same notice, to stderr, after the -`installed codeaf …` receipt and before the `export PATH` line. The full notice -above still arrives at the first session: - -``` -codeaf shares anonymous performance data with AgentField -codeaf does NOT share your prompts, code, files, or any private information -see what is shared: codeaf telemetry info · turn off: CODEAF_TELEMETRY=off -``` +The installer prints nothing about telemetry; the notice above arrives with the +first session, before anything is sent. ## What is sent diff --git a/docs/benchmarks/deepswe/README.md b/docs/benchmarks/deepswe/README.md new file mode 100644 index 0000000000..9e5fed668c --- /dev/null +++ b/docs/benchmarks/deepswe/README.md @@ -0,0 +1,61 @@ +# DeepSWE harness comparison + +Ten coding harnesses, one model, the same 113 tasks, one attempt each. + +| harness | solved | cost per task | cost per solved issue | mean time | +| --- | --- | --- | --- | --- | +| **senior-dev** | **62 of 113, 54.9%** | **22¢** | **1x** | 54 min | +| mini-swe-agent | 56, 49.6% | 38¢ | 1.9x | 44 min | +| codex | 51, 45.1% | 37¢ | 2.1x | 46 min | +| pi | 42, 37.2% | 35¢ | 2.4x | 52 min | +| omp | 31, 27.4% | 50¢ | 4.5x | 49 min | +| opencode | 30, 26.6% | 50¢ | 4.8x | 48 min | +| kilo | 30, 26.6% | 48¢ | 4.6x | 54 min | +| claude-code | 16, 14.2% | 19¢ | 3.4x | 32 min | +| deepseek-harness | 16, 14.2% | 150¢ | 26.6x | 94 min | +| muse-code | 3, 2.7% | 12¢ | 11.3x | 16 min | + +senior-dev solved the most issues and paid the least for each one it solved: +nearly 4x the issues claude-code solved, at about half the cost per solve of the +next best harness. + +Since then, on the same 113 tasks: 88 solved (77.9%, exact 95% CI 69.1% to 85.1%) +with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. Those runs +are senior-dev alone, not a comparison. + +## Setup + +| | | +| --- | --- | +| Benchmark | full DeepSWE set, 113 tasks, one seed per harness | +| Model | `deepseek/deepseek-v4-flash-0731` through OpenRouter | +| Verifiers | official DeepSWE at `0b9fabb` | +| Budget | 3 h per task | +| Isolation | four shards per harness, a dedicated OpenRouter key per harness | +| Ran | nine harnesses on 2026-09-11; senior-dev on 2026-09-12 | + +[`arms.csv`](arms.csv) has one row per harness: solved, reward rate, valid +grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task, +mean agent seconds and model. + +## How the columns in the README are derived + +- **solved**: tasks the verifier passed, out of 113. +- **cost per task**: billed OpenRouter spend divided by 113, so tasks without a + verifier result stay in the denominator. +- **cost per solved issue**: spend divided by tasks solved, shown as a multiple + of senior-dev's (39.9¢). +- **mean time**: mean agent wall time per task. + +## Limits + +- One seed per harness. senior-dev's 62 against mini-swe-agent's 56 is not a + statistically resolved difference. +- senior-dev departs from the sampling contract: the other nine sent temperature + 1.0 and top-p 0.95, senior-dev sent neither, so provider defaults applied. +- Five tasks produced no verifier outcome: codex 1, pi 1, omp 2, opencode 1. + They count as unsolved. +- omp, opencode, kilo, deepseek-harness, claude-code and muse-code did not record + per-attempt cost; their spend is known at harness level only. +- senior-dev ran under an earlier name for the binary; values are rewritten to + `senior-dev`. diff --git a/docs/benchmarks/deepswe/arms.csv b/docs/benchmarks/deepswe/arms.csv new file mode 100644 index 0000000000..ba8c10e980 --- /dev/null +++ b/docs/benchmarks/deepswe/arms.csv @@ -0,0 +1,11 @@ +harness,solved,tasks,reward_rate,valid_grades,invalid,mean_f2p,mean_p2p,openrouter_spend_usd,cost_per_task_usd,mean_agent_seconds,model +senior-dev,62,113,0.5487,113,0,0.8864,0.9942,24.73,0.2188,3258,openrouter/deepseek/deepseek-v4-flash-0731 +mini-swe-agent,56,113,0.4956,113,0,0.8817,0.9966,43.09,0.3813,2651,openrouter/deepseek/deepseek-v4-flash-0731 +codex,51,113,0.4513,112,1,0.8681,0.9876,42.23,0.3737,2746,deepseek/deepseek-v4-flash-0731 +pi,42,113,0.3717,112,1,0.7163,0.8909,39.61,0.3505,3101,openrouter/deepseek/deepseek-v4-flash-0731 +omp,31,113,0.2743,111,2,0.7191,0.8526,55.98,0.4954,2920,openrouter/deepseek/deepseek-v4-flash-0731 +opencode,30,113,0.2655,112,1,0.7113,0.8679,56.96,0.5041,2871,openrouter/deepseek/deepseek-v4-flash-0731 +kilo,30,113,0.2655,113,0,0.6903,0.9051,54.65,0.4836,3215,openrouter/deepseek/deepseek-v4-flash-0731 +deepseek-harness,16,113,0.1416,113,0,0.5339,0.9145,169.47,1.4997,5613,openrouter/deepseek/deepseek-v4-flash-0731 +claude-code,16,113,0.1416,113,0,0.3972,0.916,21.72,0.1922,1909,deepseek/deepseek-v4-flash-0731 +muse-code,3,113,0.0265,113,0,0.0514,0.9901,13.54,0.1198,952,deepseek/deepseek-v4-flash-0731 diff --git a/docs/changes/unreleased/1511-readme-deepswe-benchmark.md b/docs/changes/unreleased/1511-readme-deepswe-benchmark.md new file mode 100644 index 0000000000..c952c66400 --- /dev/null +++ b/docs/changes/unreleased/1511-readme-deepswe-benchmark.md @@ -0,0 +1,11 @@ +--- +kind: changed +title: the README's Benchmarks section carries the DeepSWE result +pr: 1511 +surface: [build, docs] +invalidates: + - "The README's Benchmarks section said the run was coming soon. It shows the ten-harness DeepSWE result now, and every number lives in docs/benchmarks/deepswe." + - "The installer printed a three-line telemetry notice after the `installed codeaf` receipt. It prints none now; the binary's full notice still arrives before the first session's events are sent." + - "When the install marker could not be written, the installer printed a line naming its telemetry folder. The marker is still written when it can be, and a failure is now silent." + - "docs/TELEMETRY.md carried a second fenced block, the installer's three-line form, and test/installer-telemetry.sh compared the installer against it. That block is gone and the test asserts the installer prints no notice." +--- diff --git a/scripts/install.sh b/scripts/install.sh index ee503684fb..84f690001c 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -7,14 +7,6 @@ LEGACY_REPOSITORY="Agent-Field/aforge-v2" # Remove after the one-release reposit CHANNEL="${CHANNEL:-stable}" INSTALL_NAME="${CODEAF_INSTALL_NAME:-codeaf}" VERSION="${VERSION:-}" -# The installer's three-line telemetry notice, verbatim from docs/TELEMETRY.md. -# The binary prints the full notice before the first session's events leave; -# the installer says the fact, the inspector and the switch. It writes a local -# install marker and prints this text, never sends telemetry, and makes no -# request that the download steps did not already make. -TELEMETRY_NOTICE='codeaf shares anonymous performance data with AgentField -codeaf does NOT share your prompts, code, files, or any private information -see what is shared: codeaf telemetry info · turn off: CODEAF_TELEMETRY=off' VERBOSE="${VERBOSE:-0}" NO_MODIFY_PATH="${CODEAF_NO_MODIFY_PATH:-${AFORGE_NO_MODIFY_PATH:-0}}" # legacy-name INSTALL_DIR="${CODEAF_INSTALL_DIR:-${AFORGE_INSTALL_DIR:-${HOME}/.codeaf/bin}}" # legacy-name @@ -51,7 +43,6 @@ Environment: CODEAF_NO_MODIFY_PATH, VERBOSE GITHUB_TOKEN or GH_TOKEN: GitHub answers anonymous API calls sixty times an hour per address; a token raises that. CODEAF_GITHUB_API and CODEAF_GITHUB_DOWNLOAD for mirrors and tests - CODEAF_TELEMETRY=off, or DO_NOT_TRACK=1, turns the anonymous usage counts off EOF } @@ -66,18 +57,6 @@ usage_error() { exit 2 } -telemetry_off() { - local value - case "${DO_NOT_TRACK:-}" in - 1|[Tt]|[Tt][Rr][Uu][Ee]) return 0 ;; - esac - value=$(echo "${CODEAF_TELEMETRY:-}" | tr '[:upper:]' '[:lower:]') - case "$value" in - off|0|false) return 0 ;; - esac - return 1 -} - # The install marker is a local record only: it names how and when this # machine's codeaf was installed so the usage counts can bucket by channel, # never who installed it. Nothing here contacts the network. @@ -92,22 +71,19 @@ write_install_marker() { stable|rc|staging|dev) channel="$CHANNEL" ;; *) channel="unknown" ;; esac + # The marker is background bookkeeping: a failure skips it without a word. if ! mkdir -p "$directory/telemetry" 2>/dev/null; then - printf 'codeaf: could not create %s/telemetry; skipping the install marker\n' "$directory" >&2 return 0 fi chmod 0700 "$directory/telemetry" file="$directory/telemetry/install.json" if ! { printf '{"install_method":"script","channel":"%s","installed_at":"%s"}' \ "$channel" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"; } > "$file" 2>/dev/null; then - printf 'codeaf: could not write %s; skipping the install marker\n' "$file" >&2 return 0 fi chmod 0600 "$file" } -# Printed once, at the very end of a successful install. The binary repeats it -# before the first session's counts are ever sent. # The one line a person still has to paste, printed last of all, between a # blank line above and a blank line below, bold green on a terminal. Bare # `export PATH=...` and nothing else, so it can be selected and pasted without @@ -124,14 +100,6 @@ print_path_hint() { printf '\n%s%s%s\n\n' "$on" "$hint" "$off" } -print_telemetry_notice() { - if telemetry_off; then - printf 'codeaf: anonymous usage counts are off (CODEAF_TELEMETRY=off or DO_NOT_TRACK=1)\n' >&2 - return 0 - fi - printf '\n%s\n' "$TELEMETRY_NOTICE" >&2 -} - while [[ $# -gt 0 ]]; do case "$1" in --stable) CHANNEL="stable"; shift ;; @@ -509,7 +477,7 @@ append_path_line() { } # The PATH line is not printed here. It is the last thing the installer says, -# after `codeaf version` and the telemetry notice, so the one line a person +# after `codeaf version`, so the one line a person # has to paste sits at the bottom of the screen where their eye already is. PATH_HINT="" if [[ "$OS" != "windows" ]] && ! path_has_dir; then @@ -552,5 +520,4 @@ printf 'installed %s\n' "$version_line" if [[ "$RUN_BOOT_ADOPTION" == "1" || -d "$STATE_ROOT" ]]; then write_install_marker "$STATE_ROOT" fi -print_telemetry_notice print_path_hint "$PATH_HINT" diff --git a/test/installer-telemetry.sh b/test/installer-telemetry.sh index 2a4bd12f0e..28a1e17427 100755 --- a/test/installer-telemetry.sh +++ b/test/installer-telemetry.sh @@ -1,11 +1,11 @@ #!/usr/bin/env bash # THE INSTALLER'S TELEMETRY DUTIES, PROVED WITHOUT A NETWORK. # -# The notice text lives once, byte for byte, in docs/TELEMETRY.md: the full -# form quoted by the binary and the README, and the three-line form the -# installer prints. Only a test notices when one of them drifts. The installer's main body +# The installer writes the local install marker and prints no telemetry notice: +# the binary shows the notice, quoted byte for byte in docs/TELEMETRY.md and the +# README, before the first session's events are sent. The installer's main body # downloads a release, so this test never sources it whole: it lifts out the -# three telemetry functions and runs them against a temporary state root. +# marker and PATH functions and runs them against a temporary state root. # Nothing here opens a socket. set -euo pipefail cd "$(git rev-parse --show-toplevel)" @@ -14,11 +14,8 @@ script=scripts/install.sh doc=docs/TELEMETRY.md test -f "$doc" || { echo "docs/TELEMETRY.md is missing; this test reads the notice from it"; exit 1; } -eval "$(awk '/^TELEMETRY_NOTICE=/{f=1} /^VERBOSE=/{f=0} f' "$script")" -eval "$(sed -n '/^telemetry_off()/,/^}/p; /^write_install_marker()/,/^}/p; /^print_telemetry_notice()/,/^}/p; /^print_path_hint()/,/^}/p' "$script")" -type telemetry_off >/dev/null +eval "$(sed -n '/^write_install_marker()/,/^}/p; /^print_path_hint()/,/^}/p' "$script")" type write_install_marker >/dev/null -type print_telemetry_notice >/dev/null type print_path_hint >/dev/null pass=0 @@ -78,56 +75,24 @@ CHANNEL='we"ird' write_install_marker "$tmp/state" ok "unexpected channel written as unknown" 'grep -q "\"channel\":\"unknown\"" "$f"' ok "unexpected-channel marker is valid JSON" 'json_valid' ok "unwritable state root does not fail the install" 'write_install_marker /proc/nonexistent-root' +ok "an unwritable state root says nothing" '[ -z "$(write_install_marker /proc/nonexistent-root 2>&1)" ]' -# --- the notice ------------------------------------------------------------- +# --- no notice from the installer -------------------------------------------- -notice="$tmp/notice.txt" -print_telemetry_notice 2> "$notice" -# The first fenced block under "The notice" is the binary's full notice, the -# second is the installer's three-line form; awk counts fences to tell them apart. +# The binary shows the notice at the first session; the installer shows none. +ok "installer prints no telemetry notice" '! grep -qE "anonymous (performance data|usage counts)|TELEMETRY_NOTICE|print_telemetry_notice" "$script"' +ok "no printed line in the installer mentions telemetry" '! grep -E "printf|echo" "$script" | grep -qi telemetry' expected=$(awk ' /^## The notice$/ {f=1; next} f && /^```$/ {f++; next} f == 2 {print} ' "$doc") -installer_expected=$(awk ' - /^## The notice$/ {f=1; next} - f && /^```$/ {f++; next} - f == 4 {print} -' "$doc") -body=$(sed 1d "$notice") -ok "one blank line before the notice" '[ -z "$(head -n 1 "$notice")" ]' -ok "installer notice matches docs/TELEMETRY.md verbatim" '[ "$body" = "$installer_expected" ]' -ok "installer notice is three lines" '[ "$(printf "%s\n" "$body" | wc -l | tr -d " ")" = 3 ]' -ok "installer notice names the inspector and the switch" 'case "$body" in *"codeaf telemetry info"*CODEAF_TELEMETRY=off*) true;; *) false;; esac' -ok "installer notice names what is never shared" 'case "$body" in *"does NOT share your prompts, code, files"*) true;; *) false;; esac' readme_block=$(awk ' /^```text$/ {f = 1; buf = ""; next} /^```$/ {if (f && buf ~ /codeaf sends anonymous usage counts/) {print buf; exit} f = 0; next} f {buf = buf $0 "\n"} ' README.md) ok "README quotes the notice verbatim" '[ -n "$readme_block" ] && [ "$(printf "%s\n" "$expected")" = "$readme_block" ]' -ok "notice goes to stderr, nothing to stdout" '[ -z "$(print_telemetry_notice 2>/dev/null)" ]' - -for v in off 0 false OFF False; do - export CODEAF_TELEMETRY="$v" - out=$( print_telemetry_notice 2>&1 ) - ok "CODEAF_TELEMETRY=$v opts out" 'case "$out" in *"off"*) true;; *) false;; esac' - ok "opt-out prints no notice body" 'case "$out" in *"anonymous performance data"*) false;; *) true;; esac' - unset CODEAF_TELEMETRY -done -for v in 1 true TRUE; do - export DO_NOT_TRACK="$v" - out=$( print_telemetry_notice 2>&1 ) - ok "DO_NOT_TRACK=$v opts out" 'case "$out" in *"off"*) true;; *) false;; esac' - unset DO_NOT_TRACK -done -out=$( print_telemetry_notice 2>&1 ) -ok "unset prints the notice" 'case "$out" in *"anonymous performance data with AgentField"*) true;; *) false;; esac' -export CODEAF_TELEMETRY=1 -out=$( print_telemetry_notice 2>&1 ) -ok "CODEAF_TELEMETRY=1 prints the notice" 'case "$out" in *"anonymous performance data with AgentField"*) true;; *) false;; esac' -unset CODEAF_TELEMETRY # --- the PATH line comes last ------------------------------------------------- # The line a person has to paste is the installer's final word: bare, after a