diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 4c0dba2b4b..2e9982f362 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -7,3 +7,9 @@ Classified by what is being measured. a long session, and behaviour through interrupts and crashes. Seven agent CLIs, one harness, every table regenerable from the scripts beside the results. +- [deepswe/](deepswe/): task-solving outcomes for senior-dev, the coding + harness vendored into this product, on the 113 scored tasks of the DeepSWE + v1.1 corpus. Three campaigns — a ten-harness comparison and two single-arm + model sweeps — as one per-task table each, with reward, F2P, P2P, wall time + and cost. The protocol for this repository's own `codeaf` runs on the same + corpus is separate, in [`bench/deepswe/`](../../bench/deepswe/). diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/README.md b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/README.md new file mode 100644 index 0000000000..fc11d4a870 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/README.md @@ -0,0 +1,92 @@ +# Harness comparison — DeepSWE 113, DeepSeek V4 Flash + +Ten coding harnesses, same model, same 113 tasks, one attempt each. +**senior-dev finished first on reward and on F2P.** + +| Harness | Solved | Reward | Mean F2P | Mean P2P | Spend | Per task | Mean time | +|---|---|---|---|---|---|---|---| +| [**senior-dev**](tasks/senior-dev.csv) | **62/113** | 54.87% | 88.64% | 99.42% | $24.73 | $0.22 | 54 min | +| [mini-swe-agent](tasks/mini-swe-agent.csv) | 56/113 | 49.56% | 88.17% | 99.66% | $43.09 | $0.38 | 44 min | +| [codex](tasks/codex.csv) | 51/113 | 45.13% | 86.81% | 98.76% | $42.23 | $0.37 | 45 min | +| [pi](tasks/pi.csv) | 42/113 | 37.17% | 71.63% | 89.09% | $39.61 | $0.35 | 51 min | +| [omp](tasks/omp.csv) | 31/113 | 27.43% | 71.91% | 85.26% | $55.98 | $0.50 | 48 min | +| [opencode](tasks/opencode.csv) | 30/113 | 26.55% | 71.13% | 86.79% | $56.96 | $0.50 | 47 min | +| [kilo](tasks/kilo.csv) | 30/113 | 26.55% | 69.03% | 90.51% | $54.65 | $0.48 | 53 min | +| [deepseek-harness](tasks/deepseek-harness.csv) | 16/113 | 14.16% | 53.39% | 91.45% | $169.47 | $1.50 | 93 min | +| [claude-code](tasks/claude-code.csv) | 16/113 | 14.16% | 39.72% | 91.60% | $21.72 | $0.19 | 31 min | +| [muse-code](tasks/muse-code.csv) | 3/113 | 2.65% | 5.14% | 99.01% | $13.54 | $0.12 | 15 min | + +Reward is the binary verifier result. F2P and P2P are macro-averages over all 113 +scheduled tasks — the five invalid outcomes contribute zero rather than being dropped. +Per task is spend ÷ 113, so invalid outcomes stay in the denominator too. + +## Setup + +| | | +|---|---| +| Benchmark | full DeepSWE set, 113 tasks, one seed per harness | +| Model | `deepseek/deepseek-v4-flash-0731` via OpenRouter | +| Verifiers | official DeepSWE at `0b9fabb` | +| Budget | 3 h per task | +| Isolation | four GCP shards per harness, dedicated OpenRouter key per harness | +| Ran | nine arms 2026-09-11; senior-dev arm 2026-09-12 03:10Z–08:20Z | +| Total spend | ≈ $521.98 | + +**The senior-dev arm departs from the frozen sampling contract.** The other nine sent +temperature 1.0 and top-p 0.95; senior-dev sent neither, so OpenRouter and provider +defaults applied. Everything else matched. senior-dev was deliberately excluded from +the nine-harness campaign and run separately a day later. + +## Invalid outcomes + +Five tasks across four arms produced no usable verifier outcome. They have an empty +`reward` in `tasks.csv`, stay in the 113-task denominator and contribute zero. + +| Harness | Task | Why | +|---|---|---| +| [codex](tasks/codex.csv) | `valibot-recursive-schema-composition` | patch not accepted | +| [pi](tasks/pi.csv) | `langchain-request-coalescing` | agent and verifier both timed out | +| [omp](tasks/omp.csv) | `actionlint-action-pinning-lint` | agent killed, exit 137, no verifier result | +| [omp](tasks/omp.csv) | `kombu-virtual-queue-dead-lettering` | verifier timeout, exit 124 | +| [opencode](tasks/opencode.csv) | `pwntools-tube-multiplexing` | verifier timeout, exit 124 | + +A non-zero harness exit is not an invalid result — a task can time out and still +produce a patch the verifier grades. Only tasks with no verifier outcome count here. + +## The files + +`arms.csv` — one row per harness: solved, reward rate, valid grades, invalid count, +mean F2P/P2P, OpenRouter spend, cost per task, mean agent seconds, model. + +`tasks/.csv` — one file per harness, 113 rows each, ordered by difficulty +rank. All ten share a column layout, so `cat` gives the whole campaign back: + +``` +awk 'FNR>1 || NR==1' tasks/*.csv > all-arms.csv +``` + +Columns: + +| Column | Meaning | +|---|---| +| `harness` `task` `band` `difficulty_rank` `language` `dev_set` | which arm, which task, and how the task is classified | +| `reward` | 1 pass, 0 fail, empty = no verifier outcome | +| `f2p` `p2p` | fraction of issue tests now passing / pre-existing tests still passing | +| `f2p_passed` `f2p_total` `p2p_passed` `p2p_total` | the underlying test counts | +| `seconds` `started` `finished` | agent wall time and timestamps | +| `cost_usd` `model_calls` | per-attempt model spend and call count | +| `exit` `timed_out` | harness exit status | +| `model` `harness_version` `attempt_id` | what produced it | + +**`cost_usd` and `model_calls` are empty for six arms** — omp, opencode, kilo, +deepseek-harness, claude-code and muse-code did not record per-attempt usage. Their +spend is known only at arm level, in `arms.csv`. + +## Source + +Published as [Harness Comparison — DeepSWE](https://app.notion.com/p/Harness-Comparison-DeepSWE-3d603ed71f1080a5bcf3fddf8d7b2925). + +The per-attempt artifacts behind these rows — `events.ndjson`, `run.log`, traces, +verifier stdout, patches — are about 19 GB and stay on the machines that produced +them. These campaigns ran while the binary was still called `codeaf`; the values +here are rewritten to `senior-dev`. diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/arms.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/arms.csv new file mode 100644 index 0000000000..ba8c10e980 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/arms.csv @@ -0,0 +1,11 @@ +harness,solved,tasks,reward_rate,valid_grades,invalid,mean_f2p,mean_p2p,openrouter_spend_usd,cost_per_task_usd,mean_agent_seconds,model +senior-dev,62,113,0.5487,113,0,0.8864,0.9942,24.73,0.2188,3258,openrouter/deepseek/deepseek-v4-flash-0731 +mini-swe-agent,56,113,0.4956,113,0,0.8817,0.9966,43.09,0.3813,2651,openrouter/deepseek/deepseek-v4-flash-0731 +codex,51,113,0.4513,112,1,0.8681,0.9876,42.23,0.3737,2746,deepseek/deepseek-v4-flash-0731 +pi,42,113,0.3717,112,1,0.7163,0.8909,39.61,0.3505,3101,openrouter/deepseek/deepseek-v4-flash-0731 +omp,31,113,0.2743,111,2,0.7191,0.8526,55.98,0.4954,2920,openrouter/deepseek/deepseek-v4-flash-0731 +opencode,30,113,0.2655,112,1,0.7113,0.8679,56.96,0.5041,2871,openrouter/deepseek/deepseek-v4-flash-0731 +kilo,30,113,0.2655,113,0,0.6903,0.9051,54.65,0.4836,3215,openrouter/deepseek/deepseek-v4-flash-0731 +deepseek-harness,16,113,0.1416,113,0,0.5339,0.9145,169.47,1.4997,5613,openrouter/deepseek/deepseek-v4-flash-0731 +claude-code,16,113,0.1416,113,0,0.3972,0.916,21.72,0.1922,1909,deepseek/deepseek-v4-flash-0731 +muse-code,3,113,0.0265,113,0,0.0514,0.9901,13.54,0.1198,952,deepseek/deepseek-v4-flash-0731 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/claude-code.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/claude-code.csv new file mode 100644 index 0000000000..0b4ce8c21e --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/claude-code.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +claude-code,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,88,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-10T18:45:33+00:00,2026-09-10T18:47:47+00:00 +claude-code,gql-incremental-graphql-delivery,hard,2,python,0,0,0.764706,0.998767,13,17,810,811,3454,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:00+00:00,2026-09-11T14:29:36+00:00 +claude-code,bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,2490,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:44:47+00:00,2026-09-11T14:26:18+00:00 +claude-code,termenv-preserve-ansi-resets,hard,4,go,0,0,0.0,1.0,0,35,87,87,1487,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:06+00:00,2026-09-11T14:09:55+00:00 +claude-code,updo-policy-alerting,hard,5,go,0,0,0.0,1.0,0,17,123,123,25,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:31:24+00:00,2026-09-11T13:31:53+00:00 +claude-code,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.0,1.0,0,47,70,70,2285,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:00+00:00,2026-09-11T14:13:53+00:00 +claude-code,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,1.0,0,6,62,62,280,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:42+00:00,2026-09-11T13:50:30+00:00 +claude-code,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.0,1.0,0,32,1273,1273,5548,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:37+00:00,2026-09-11T15:19:07+00:00 +claude-code,helm-array-merge-strategies,hard,9,go,0,0,0.957447,0.833333,45,47,10,12,4467,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:31:55+00:00,2026-09-11T14:46:28+00:00 +claude-code,meriyah-explicit-resource-declarations,hard,10,typescript,0,1,1.0,1.0,49,49,51469,51469,2909,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:34:59+00:00,2026-09-11T14:26:10+00:00 +claude-code,pest-character-class-coalescing,hard,11,rust,0,0,0.0,0.0,0,104,0,250,1008,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:57+00:00,2026-09-11T14:03:55+00:00 +claude-code,testem-bail-on-test-failure,hard,12,javascript,0,0,0.677778,0.997955,61,90,488,489,2023,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:44:50+00:00,2026-09-11T14:19:06+00:00 +claude-code,koota-pair-relation-tracking,hard,13,typescript,0,0,0.0,1.0,0,38,172,172,27,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:31:22+00:00,2026-09-11T13:32:27+00:00 +claude-code,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.878049,0,15,72,82,585,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:34:18+00:00,2026-09-11T13:44:10+00:00 +claude-code,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.95,1.0,57,60,1131,1131,1836,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:01+00:00,2026-09-11T14:17:27+00:00 +claude-code,koota-query-predicates,hard,16,typescript,0,0,0.953488,1.0,41,43,172,172,6279,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:05+00:00,2026-09-11T15:30:17+00:00 +claude-code,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.769231,1.0,10,13,22,22,1333,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:07+00:00,2026-09-11T13:56:52+00:00 +claude-code,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.0,1.0,0,71,128,128,19,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:34:26+00:00,2026-09-11T13:35:16+00:00 +claude-code,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.710526,1.0,54,76,1412,1412,2301,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:42+00:00,2026-09-11T14:24:06+00:00 +claude-code,ink-grid-box-layout,hard,20,typescript,0,0,0.84,1.0,21,25,49,49,3254,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T18:28:21+00:00,2026-09-11T19:23:13+00:00 +claude-code,awilix-async-container-initialization,hard,21,typescript,0,0,0.0,1.0,0,24,162,162,84,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:31:32+00:00,2026-09-11T13:33:32+00:00 +claude-code,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.928571,1.0,13,14,9,9,1493,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:20+00:00,2026-09-11T13:58:34+00:00 +claude-code,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.822785,1.0,65,79,16715,16715,2190,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:46+00:00,2026-09-11T14:22:24+00:00 +claude-code,superjson-error-stack-serialization,hard,24,typescript,0,0,0.95,1.0,76,80,116,116,7120,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:23+00:00,2026-09-11T15:44:55+00:00 +claude-code,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.0,1.0,0,12,139,139,110,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:06+00:00,2026-09-11T13:34:42+00:00 +claude-code,vulture-persistent-analysis-cache,hard,26,python,0,0,1.0,0.986441,24,24,291,295,3357,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:00+00:00,2026-09-11T14:27:59+00:00 +claude-code,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.0,1.0,0,94,599,599,11,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:27+00:00,2026-09-11T13:48:28+00:00 +claude-code,expr-try-catch-errors,hard,28,go,0,0,0.0,0.005961,0,79,395,66265,1580,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:24+00:00,2026-09-11T14:11:47+00:00 +claude-code,koota-composite-trait-aspects,hard,29,typescript,0,1,1.0,1.0,51,51,172,172,6851,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:06+00:00,2026-09-11T15:26:55+00:00 +claude-code,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.0,0.0,0,82,0,6,4562,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:35:26+00:00,2026-09-11T14:51:30+00:00 +claude-code,eicrud-keyset-pagination-cursor,hard,31,typescript,0,1,1.0,1.0,14,14,168,168,3185,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:18+00:00,2026-09-11T14:40:38+00:00 +claude-code,participle-grammar-conflict-analysis,hard,32,go,0,0,0.021978,1.0,2,91,153,153,1502,,,1,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:44:48+00:00,2026-09-11T14:09:52+00:00 +claude-code,python-statemachine-state-data-scoping,hard,33,python,0,0,0.041667,0.2986,3,72,384,1286,1942,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:12+00:00,2026-09-11T14:05:36+00:00 +claude-code,clack-async-autocomplete-options,hard,34,typescript,0,0,0.560976,1.0,46,82,643,643,1385,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:35:12+00:00,2026-09-11T13:58:46+00:00 +claude-code,langchain-request-coalescing,hard,35,python,0,0,0.0,1.0,0,50,232,232,47,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:47:15+00:00,2026-09-11T13:48:28+00:00 +claude-code,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,4404,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:24+00:00,2026-09-11T14:59:50+00:00 +claude-code,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.555556,1.0,20,36,2034,2034,4339,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:20+00:00,2026-09-11T14:46:10+00:00 +claude-code,textual-kitty-key-phases,hard,38,python,0,0,0.478261,1.0,11,23,57,57,2194,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:35:40+00:00,2026-09-11T14:12:22+00:00 +claude-code,pwntools-tube-multiplexing,medium,39,python,0,0,0.0,1.0,0,73,1,1,1538,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:45:18+00:00,2026-09-11T14:11:00+00:00 +claude-code,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.0,1.0,0,25,1679,1679,94,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:12+00:00,2026-09-11T13:50:04+00:00 +claude-code,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.80597,1.0,54,67,2035,2035,2715,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:32:08+00:00,2026-09-11T14:17:25+00:00 +claude-code,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,1300,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:37:06+00:00,2026-09-11T13:58:53+00:00 +claude-code,bandit-incremental-cache-control,medium,43,python,0,0,0.0,1.0,0,88,275,275,18,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:57+00:00,2026-09-11T13:47:21+00:00 +claude-code,fastapi-deprecation-response-headers,medium,44,python,0,0,0.0,1.0,0,137,3134,3134,8,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:46+00:00,2026-09-11T13:47:05+00:00 +claude-code,go-critic-doc-link-checker,medium,45,go,0,0,0.0,1.0,0,3,16,16,34,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:07+00:00,2026-09-11T13:33:45+00:00 +claude-code,igel-persist-feature-schema,medium,46,python,0,0,0.0,1.0,0,24,2,2,37,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:37:20+00:00,2026-09-11T13:37:59+00:00 +claude-code,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.5,0.993174,33,66,291,293,598,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:49:15+00:00,2026-09-11T13:59:14+00:00 +claude-code,anko-typed-variable-bindings,medium,48,go,0,0,0.0,1.0,0,9,94,94,92,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:40+00:00,2026-09-11T13:48:13+00:00 +claude-code,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.954545,1.0,63,66,30014,30014,3383,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:33:21+00:00,2026-09-11T14:29:47+00:00 +claude-code,dasel-html-document-format,medium,50,go,0,0,0.0,1.0,0,146,1012,1012,1702,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:35:44+00:00,2026-09-11T14:04:07+00:00 +claude-code,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,1.0,0,17,7,7,32,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:48:30+00:00,2026-09-11T13:49:39+00:00 +claude-code,fastapi-implicit-head-options,medium,52,python,0,0,0.162791,1.0,7,43,3134,3134,1182,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:47+00:00,2026-09-11T14:06:40+00:00 +claude-code,prometheus-typed-label-sorting,medium,53,go,0,0,0.0,1.0,0,17,28,28,57,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:34:19+00:00,2026-09-11T13:35:21+00:00 +claude-code,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,1834,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:37:40+00:00,2026-09-11T14:08:56+00:00 +claude-code,pebble-durability-wait-apis,medium,55,go,0,0,0.0,1.0,0,59,44,44,65,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:49:14+00:00,2026-09-11T13:50:22+00:00 +claude-code,cliffy-config-file-parsing,medium,56,typescript,0,0,0.0,1.0,0,37,451,451,216,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:53+00:00,2026-09-11T13:50:32+00:00 +claude-code,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.9,1.0,54,60,1038,1038,2597,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:34:19+00:00,2026-09-11T14:17:39+00:00 +claude-code,kombu-single-active-consumer-priority,medium,58,python,0,0,0.858824,0.999296,73,85,1420,1421,3529,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:37:42+00:00,2026-09-11T14:36:33+00:00 +claude-code,skrub-duration-encoding,medium,59,python,0,0,0.976923,1.0,127,130,2784,2784,1388,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:49:06+00:00,2026-09-11T14:12:16+00:00 +claude-code,scc-bounded-memory-spilling,medium,60,go,0,0,0.0,0.0,0,31,0,286,2038,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T13:46:58+00:00,2026-09-11T14:21:00+00:00 +claude-code,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,5214,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:18+00:00,2026-09-11T17:05:13+00:00 +claude-code,tengo-destructuring-bindings,medium,62,go,0,0,0.868132,0.212121,79,91,28,132,1856,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:06:57+00:00,2026-09-11T15:37:55+00:00 +claude-code,httpx-multipart-response-parsing,medium,63,python,0,0,0.991803,1.0,121,122,1272,1272,3155,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:02:56+00:00,2026-09-11T15:55:33+00:00 +claude-code,opa-rego-rule-profiling,medium,64,go,0,0,0.88,1.0,22,25,6,6,7208,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:15+00:00,2026-09-11T17:54:48+00:00 +claude-code,ipython-session-bundle-replay,medium,65,python,0,0,0.0,1.0,0,17,29,29,101,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:24+00:00,2026-09-11T15:40:09+00:00 +claude-code,scriggo-method-declarations,medium,66,go,0,0,0.0,1.0,0,48,1049,1049,314,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:06:55+00:00,2026-09-11T15:12:14+00:00 +claude-code,opa-template-string-reconstruction,medium,67,go,0,0,0.0,1.0,0,5,4,4,1111,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:03:30+00:00,2026-09-11T15:22:26+00:00 +claude-code,mobly-grouped-test-barriers,medium,68,python,0,0,0.0,0.830446,0,79,671,808,337,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:15+00:00,2026-09-11T15:59:53+00:00 +claude-code,tengo-callable-instance-isolation,medium,69,go,0,0,0.0,1.0,0,23,122,122,1824,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:42+00:00,2026-09-11T16:09:08+00:00 +claude-code,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,1.0,0,43,109,109,144,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:07:18+00:00,2026-09-11T15:10:08+00:00 +claude-code,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.0,1.0,0,69,7,7,675,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:03:40+00:00,2026-09-11T15:14:57+00:00 +claude-code,tomlkit-toml-table-converters,medium,72,python,0,0,0.0,1.0,0,60,964,964,13,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:42+00:00,2026-09-11T15:55:00+00:00 +claude-code,etree-xml-diff-patch,medium,73,go,0,0,0.0,1.0,0,52,15,15,8,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:40+00:00,2026-09-11T15:38:49+00:00 +claude-code,textual-richlog-follow-state,medium,74,python,0,0,0.95,1.0,19,20,6,6,4079,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:06+00:00,2026-09-11T16:16:14+00:00 +claude-code,yaegi-go-embed-directives,medium,75,go,0,0,0.0,1.0,0,38,58,58,2072,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:03:44+00:00,2026-09-11T15:38:21+00:00 +claude-code,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.0,1.0,0,254,22,22,1475,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:41+00:00,2026-09-11T16:21:38+00:00 +claude-code,numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,5582,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:58+00:00,2026-09-11T17:12:04+00:00 +claude-code,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.0,1.0,0,10,209,209,1668,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:07:40+00:00,2026-09-11T15:36:54+00:00 +claude-code,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,0.0,0,17,0,2,4978,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:03:55+00:00,2026-09-11T16:26:56+00:00 +claude-code,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,4094,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:56+00:00,2026-09-11T17:03:13+00:00 +claude-code,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,1.0,0,33,1133,1133,1068,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:45+00:00,2026-09-11T15:57:31+00:00 +claude-code,ts-pattern-match-each,easy,82,typescript,0,0,0.988235,1.0,84,85,6,6,816,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:07:51+00:00,2026-09-11T15:22:04+00:00 +claude-code,anko-default-function-arguments,easy,83,go,0,0,0.0,1.0,0,2,119,119,71,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:03:57+00:00,2026-09-11T15:05:14+00:00 +claude-code,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.0,1.0,0,37,1267,1267,619,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:54:55+00:00,2026-09-11T16:06:18+00:00 +claude-code,abs-stepped-slices,easy,85,go,0,0,0.166667,0.666667,1,6,4,6,3423,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:38+00:00,2026-09-11T16:35:43+00:00 +claude-code,testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,2024,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:07:58+00:00,2026-09-11T15:42:22+00:00 +claude-code,query-persist-restored-query-state,easy,87,typescript,0,0,0.0,1.0,0,8,42,42,197,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:06:06+00:00,2026-09-11T15:15:06+00:00 +claude-code,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,1.0,0,22,58,58,1312,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:23+00:00,2026-09-11T16:17:29+00:00 +claude-code,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1663,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:38:35+00:00,2026-09-11T16:06:52+00:00 +claude-code,geo-shapeindex-serialization,easy,90,go,0,0,0.041667,1.0,1,24,599,599,3772,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:07:40+00:00,2026-09-11T16:10:34+00:00 +claude-code,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,1.0,0,82,2,2,92,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:13+00:00,2026-09-11T15:09:52+00:00 +claude-code,abs-module-cache-flags,easy,92,go,0,0,0.85,1.0,17,20,3,3,3427,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:25+00:00,2026-09-11T16:52:35+00:00 +claude-code,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.90566,0.875,48,53,7,8,1103,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:39:40+00:00,2026-09-11T15:58:05+00:00 +claude-code,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,3615,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:09+00:00,2026-09-11T16:08:26+00:00 +claude-code,vitest-duration-sharding,easy,95,typescript,0,0,0.0,1.0,0,56,24,24,7,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:04:29+00:00,2026-09-11T15:07:50+00:00 +claude-code,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,0,1.0,0.998421,31,31,1265,1267,4601,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:49+00:00,2026-09-11T17:13:24+00:00 +claude-code,task-task-graph-export,easy,97,go,0,0,0.0,0.941176,0,20,16,17,2506,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:39:39+00:00,2026-09-11T16:21:28+00:00 +claude-code,go-genai-streamed-function-args,easy,98,go,0,0,0.0,0.0,0,6,0,62,4626,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:15+00:00,2026-09-11T16:25:23+00:00 +claude-code,ytt-jsonpath-query-api,easy,99,go,0,0,0.980583,1.0,101,103,1,1,2542,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:06:16+00:00,2026-09-11T15:48:43+00:00 +claude-code,helm-unified-manifest-stream,easy,100,go,1,0,0.0,1.0,0,5,2,2,39,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:49+00:00,2026-09-11T15:56:36+00:00 +claude-code,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,2149,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:40:05+00:00,2026-09-11T16:16:23+00:00 +claude-code,goreleaser-retry-publish-auditing,easy,102,go,0,0,0.068966,1.0,2,29,29,29,140,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:10:27+00:00,2026-09-11T15:12:49+00:00 +claude-code,psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,2266,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:19+00:00,2026-09-11T15:46:09+00:00 +claude-code,wazero-multi-module-snapshots,easy,104,go,0,0,0.102564,0.0,8,78,0,2,2130,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:53+00:00,2026-09-11T16:31:56+00:00 +claude-code,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.961538,1.0,25,26,5709,5709,1945,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:39:56+00:00,2026-09-11T16:13:18+00:00 +claude-code,actionlint-action-pinning-lint,easy,106,go,0,0,0.0,1.0,0,55,145,145,26,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:18+00:00,2026-09-11T15:08:48+00:00 +claude-code,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.982609,1.0,113,115,1281,1281,1603,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:41+00:00,2026-09-11T15:35:27+00:00 +claude-code,returns-validated-error-accumulation,easy,108,python,0,1,1.0,1.0,159,159,61,61,2145,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:55:54+00:00,2026-09-11T16:31:42+00:00 +claude-code,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,2301,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:39:59+00:00,2026-09-11T16:18:45+00:00 +claude-code,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.0,1.0,0,130,566,566,108,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:11:14+00:00,2026-09-11T15:14:54+00:00 +claude-code,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,3670,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:08:42+00:00,2026-09-11T16:11:27+00:00 +claude-code,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,1289,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:56:21+00:00,2026-09-11T16:17:53+00:00 +claude-code,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.0,1.0,0,96,561,561,51,,,0,,deepseek/deepseek-v4-flash-0731,2.1.267,verifier,2026-09-11T15:40:23+00:00,2026-09-11T15:42:03+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/codex.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/codex.csv new file mode 100644 index 0000000000..719ff0b1a2 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/codex.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +codex,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,2086,0.220131573,104,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:41Z,2026-09-11T02:51:46Z +codex,gql-incremental-graphql-delivery,hard,2,python,0,0,0.941176,1.0,16,17,811,811,2493,0.40035326,165,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:30Z,2026-09-11T02:58:22Z +codex,bandit-structured-nosec-directives,hard,3,python,0,0,0.971014,0.992908,67,69,280,282,2952,0.202748684,95,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:25Z,2026-09-11T03:08:47Z +codex,termenv-preserve-ansi-resets,hard,4,go,0,0,0.828571,1.0,29,35,87,87,2370,0.210927937,89,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:47Z,2026-09-11T02:56:31Z +codex,updo-policy-alerting,hard,5,go,0,1,1.0,1.0,17,17,123,123,1725,0.173124121,88,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:25Z,2026-09-11T02:46:22Z +codex,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.978723,1.0,46,47,70,70,5857,0.961327179,306,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:31Z,2026-09-11T03:55:29Z +codex,oxvg-structural-selector-preservation,hard,7,rust,0,1,1.0,1.0,6,6,62,62,3290,0.585856639,261,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:26Z,2026-09-11T03:12:43Z +codex,sqlfmt-create-table-ddl-formatting,hard,8,python,0,1,1.0,1.0,32,32,1273,1273,3388,0.42744038,185,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:56Z,2026-09-11T03:14:30Z +codex,helm-array-merge-strategies,hard,9,go,0,0,1.0,0.833333,47,47,10,12,2746,0.351150865,151,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:33Z,2026-09-11T03:04:29Z +codex,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.979592,1.0,48,49,51469,51469,4854,0.586449518,280,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:01Z,2026-09-11T03:38:08Z +codex,pest-character-class-coalescing,hard,11,rust,0,0,0.817308,0.996,85,104,249,250,2156,0.218900243,118,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:30Z,2026-09-11T02:56:40Z +codex,testem-bail-on-test-failure,hard,12,javascript,0,0,0.622222,1.0,56,90,489,489,2718,0.451627575,149,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:47Z,2026-09-11T03:04:12Z +codex,koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,1952,0.52157668,207,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:20Z,2026-09-11T02:50:01Z +codex,prometheus-transactional-reload-status,hard,14,typescript,0,0,1.0,0.878049,15,15,72,82,3246,0.462447721,190,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:53Z,2026-09-11T03:14:20Z +codex,obsidian-linter-link-format-conversion,hard,15,typescript,0,1,1.0,1.0,60,60,1131,1131,1437,0.210883754,124,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:09Z,2026-09-11T02:44:15Z +codex,koota-query-predicates,hard,16,typescript,0,0,0.906977,1.0,39,43,172,172,3318,0.484129387,220,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:58Z,2026-09-11T03:12:26Z +codex,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,2940,0.410316248,129,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:04Z,2026-09-11T03:07:23Z +codex,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.901408,1.0,64,71,128,128,2387,0.24029862899999999,97,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:44Z,2026-09-11T03:00:41Z +codex,kombu-virtual-queue-dead-lettering,hard,19,python,0,1,1.0,1.0,76,76,1412,1412,2696,0.254672527,107,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:53Z,2026-09-11T03:02:55Z +codex,ink-grid-box-layout,hard,20,typescript,0,1,1.0,1.0,25,25,49,49,4391,0.676361825,204,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:09Z,2026-09-11T03:30:31Z +codex,awilix-async-container-initialization,hard,21,typescript,0,0,0.958333,1.0,23,24,162,162,1743,0.144426963,75,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:15Z,2026-09-11T02:47:24Z +codex,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.0,1.0,0,14,9,9,105,0,49,1,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:59Z,2026-09-11T02:21:01Z +codex,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.974684,1.0,77,79,16715,16715,3625,0.243689262,109,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:30Z,2026-09-11T03:20:03Z +codex,superjson-error-stack-serialization,hard,24,typescript,0,0,0.975,0.982759,78,80,114,116,1535,0.084966947,50,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:53Z,2026-09-11T02:43:37Z +codex,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.916667,1.0,11,12,139,139,3557,0.382837732,121,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:13Z,2026-09-11T03:16:38Z +codex,vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,2779,0.185907952,73,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:10Z,2026-09-11T03:06:38Z +codex,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.180851,1.0,17,94,599,599,4094,0.585777177,196,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:13Z,2026-09-11T03:26:43Z +codex,expr-try-catch-errors,hard,28,go,0,0,0.987342,1.0,78,79,66265,66265,3127,0.581330336,233,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:38Z,2026-09-11T03:09:54Z +codex,koota-composite-trait-aspects,hard,29,typescript,0,0,0.980392,1.0,50,51,172,172,3661,0.769578805,279,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:03Z,2026-09-11T03:18:14Z +codex,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.182927,1.0,15,82,6,6,1397,0.132592969,67,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:21Z,2026-09-11T02:41:44Z +codex,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,2550,0.30335785,181,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:54Z,2026-09-11T03:02:38Z +codex,participle-grammar-conflict-analysis,hard,32,go,0,0,0.978022,1.0,89,91,153,153,1791,0.132758818,56,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:35Z,2026-09-11T02:47:33Z +codex,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,2504,0.632741443,206,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:38Z,2026-09-11T03:00:29Z +codex,clack-async-autocomplete-options,hard,34,typescript,0,0,0.97561,1.0,80,82,643,643,2355,0.433123762,147,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:05Z,2026-09-11T02:59:29Z +codex,langchain-request-coalescing,hard,35,python,0,0,0.98,1.0,49,50,232,232,1618,0.279842708,136,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:21:08Z,2026-09-11T02:48:15Z +codex,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,1683,0.134097997,95,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:17Z,2026-09-11T02:45:27Z +codex,optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,2949,0.435008842,189,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:48Z,2026-09-11T03:06:10Z +codex,textual-kitty-key-phases,hard,38,python,0,0,0.478261,1.0,11,23,57,57,3129,0.204608907,97,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:33Z,2026-09-11T03:12:49Z +codex,pwntools-tube-multiplexing,medium,39,python,0,0,0.972603,1.0,71,73,1,1,3761,0.360165613,124,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:22Z,2026-09-11T03:22:10Z +codex,arktype-json-schema-refs-dependencies,medium,40,typescript,1,1,1.0,1.0,25,25,1679,1679,6435,1.040910565,469,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:27Z,2026-09-11T04:05:51Z +codex,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.955224,1.0,64,67,2035,2035,3262,0.362979434,144,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:19Z,2026-09-11T03:11:48Z +codex,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,2639,0.565057128,233,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:09Z,2026-09-11T03:02:27Z +codex,bandit-incremental-cache-control,medium,43,python,0,0,0.977273,1.0,86,88,275,275,2186,0.202981798,100,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:21:13Z,2026-09-11T02:57:44Z +codex,fastapi-deprecation-response-headers,medium,44,python,0,0,0.978102,1.0,134,137,3134,3134,1740,0.160620844,101,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:43Z,2026-09-11T02:46:51Z +codex,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,2910,0.370228577,161,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:17Z,2026-09-11T03:05:55Z +codex,igel-persist-feature-schema,medium,46,python,0,0,0.25,1.0,6,24,2,2,1296,0.125177131,86,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:45Z,2026-09-11T02:38:29Z +codex,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.848485,1.0,56,66,293,293,1989,0.276627419,106,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:56Z,2026-09-11T02:51:13Z +codex,anko-typed-variable-bindings,medium,48,go,0,0,0.555556,1.0,5,9,94,94,1767,0.206280245,117,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:40Z,2026-09-11T02:48:15Z +codex,mashumaro-flattened-dataclass-fields,medium,49,python,0,1,1.0,1.0,66,66,30014,30014,4038,0.571424961,244,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:50Z,2026-09-11T03:24:18Z +codex,dasel-html-document-format,medium,50,go,0,0,0.986301,1.0,144,146,1012,1012,1778,0.126297685,63,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:16Z,2026-09-11T02:50:01Z +codex,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,1,1.0,1.0,17,17,7,7,3508,0.576015587,207,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:13Z,2026-09-11T03:17:56Z +codex,fastapi-implicit-head-options,medium,52,python,0,1,1.0,1.0,43,43,3134,3134,3020,0.312525139,153,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:28Z,2026-09-11T03:08:56Z +codex,prometheus-typed-label-sorting,medium,53,go,0,1,1.0,1.0,17,17,28,28,3195,0.313495085,109,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:19:36Z,2026-09-11T03:13:07Z +codex,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,770,0.070935845,58,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:44Z,2026-09-11T02:31:43Z +codex,pebble-durability-wait-apis,medium,55,go,0,1,1.0,1.0,59,59,44,44,5680,0.537911839,204,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:20:40Z,2026-09-11T03:55:26Z +codex,cliffy-config-file-parsing,medium,56,typescript,0,0,0.972973,1.0,36,37,451,451,1423,0.24251416,131,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:34Z,2026-09-11T02:42:27Z +codex,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.983333,1.0,59,60,1038,1038,2557,0.461368485,176,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:17Z,2026-09-11T03:01:04Z +codex,kombu-single-active-consumer-priority,medium,58,python,0,0,0.858824,1.0,73,85,1421,1421,1756,0.186014313,95,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:18:26Z,2026-09-11T02:47:52Z +codex,skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,2853,0.4483325,155,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:17:48Z,2026-09-11T03:05:26Z +codex,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,2483,0.377431643,163,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T02:16:42Z,2026-09-11T02:58:24Z +codex,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,1087,0.151589687,72,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:24:46Z,2026-09-11T03:43:00Z +codex,tengo-destructuring-bindings,medium,62,go,0,0,0.967033,1.0,88,91,132,132,2303,0.458417605,171,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:57Z,2026-09-11T04:37:26Z +codex,httpx-multipart-response-parsing,medium,63,python,0,0,0.991803,1.0,121,122,1272,1272,1372,0.222595099,131,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:56:57Z,2026-09-11T04:19:59Z +codex,opa-rego-rule-profiling,medium,64,go,0,0,0.88,1.0,22,25,6,6,1904,0.525094346,213,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:16Z,2026-09-11T04:40:17Z +codex,ipython-session-bundle-replay,medium,65,python,0,0,0.823529,1.0,14,17,29,29,1748,0.165329559,82,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:22Z,2026-09-11T03:54:40Z +codex,scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,10370,1.929752489,777,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:40Z,2026-09-11T06:48:38Z +codex,opa-template-string-reconstruction,medium,67,go,0,1,1.0,1.0,5,5,4,4,1999,0.723021139,335,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:59:25Z,2026-09-11T04:32:56Z +codex,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2630,0.250075769,113,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:22Z,2026-09-11T04:52:22Z +codex,tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,4089,0.368794745,137,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:26:12Z,2026-09-11T04:34:28Z +codex,fd-deterministic-multi-key-sorting,medium,70,rust,1,1,1.0,1.0,43,43,109,109,5750,0.643600894,278,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:53Z,2026-09-11T05:31:54Z +codex,cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,2709,0.194582977,97,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:24Z,2026-09-11T04:43:42Z +codex,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,3158,0.282215005,134,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:07:08Z,2026-09-11T04:59:54Z +codex,etree-xml-diff-patch,medium,73,go,0,0,0.980769,1.0,51,52,15,15,1094,0.06972611199999999,32,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:26:07Z,2026-09-11T03:44:30Z +codex,textual-richlog-follow-state,medium,74,python,0,0,0.75,1.0,15,20,6,6,3405,0.256692752,139,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:57:24Z,2026-09-11T04:54:21Z +codex,yaegi-go-embed-directives,medium,75,go,0,1,1.0,1.0,38,38,58,58,5229,0.529795165,227,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:10Z,2026-09-11T05:25:29Z +codex,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.0,1.0,0,254,22,22,2045,0.286186911,161,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:47Z,2026-09-11T04:43:04Z +codex,numba-stencil-boundary-modes,easy,77,python,0,0,0.793103,1.0,23,29,827,827,3143,0.227043205,105,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:41Z,2026-09-11T04:18:14Z +codex,valibot-recursive-schema-composition,easy,78,typescript,0,,,,,,,,,,,,,,,,, +codex,go-git-worktree-merge-conflicts,easy,79,go,0,1,1.0,1.0,17,17,2,2,2058,0.337399569,162,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:01Z,2026-09-11T04:32:33Z +codex,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,3694,0.593916776,272,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:05:55Z,2026-09-11T05:07:37Z +codex,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,1,1.0,1.0,33,33,1133,1133,2420,0.366744286,130,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:45Z,2026-09-11T04:06:13Z +codex,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,1472,0.179600235,116,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:57:47Z,2026-09-11T04:22:30Z +codex,anko-default-function-arguments,easy,83,go,0,1,1.0,1.0,2,2,119,119,4242,0.498435086,231,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:04Z,2026-09-11T05:08:55Z +codex,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.945946,1.0,35,37,1267,1267,6913,0.992326764,426,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:50Z,2026-09-11T06:04:13Z +codex,abs-stepped-slices,easy,85,go,0,0,0.833333,1.0,5,6,6,6,1649,0.119849589,89,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:24:24Z,2026-09-11T03:51:59Z +codex,testem-per-launcher-reports,easy,86,javascript,0,0,0.984615,1.0,64,65,469,469,1500,0.123116473,89,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:36Z,2026-09-11T04:20:43Z +codex,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2261,0.330928948,173,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:56:49Z,2026-09-11T04:34:43Z +codex,wasmi-trap-coredumps,easy,88,rust,0,0,0.772727,1.0,17,22,58,58,6918,1.391192259,555,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:06:30Z,2026-09-11T06:02:03Z +codex,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1164,0.200598328,113,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:24:43Z,2026-09-11T03:44:17Z +codex,geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,2990,0.290840711,133,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:50Z,2026-09-11T04:48:46Z +codex,arcane-drift-detection-baselines,easy,91,go,0,0,0.987805,1.0,81,82,2,2,1665,0.39744903,191,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:57:55Z,2026-09-11T04:26:00Z +codex,abs-module-cache-flags,easy,92,go,0,0,0.8,1.0,16,20,3,3,1123,0.185470187,115,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:06:53Z,2026-09-11T04:25:44Z +codex,aiomonitor-task-snapshots-diff,easy,93,python,0,1,1.0,1.0,53,53,8,8,1827,0.211653904,99,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:24:30Z,2026-09-11T03:55:04Z +codex,kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,7041,0.223211715,79,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:53Z,2026-09-11T05:56:22Z +codex,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,1741,0.367776482,163,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:52Z,2026-09-11T04:28:10Z +codex,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,0,0.0,1.0,0,31,1267,1267,104,0,49,1,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:07:03Z,2026-09-11T04:09:00Z +codex,task-task-graph-export,easy,97,go,0,1,1.0,1.0,20,20,17,17,1444,0.339120927,143,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:26:03Z,2026-09-11T03:50:20Z +codex,go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,2795,0.317869834,138,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:59Z,2026-09-11T04:42:41Z +codex,ytt-jsonpath-query-api,easy,99,go,0,1,1.0,1.0,103,103,1,1,1653,0.101292182,66,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:34Z,2026-09-11T04:23:16Z +codex,helm-unified-manifest-stream,easy,100,go,1,0,0.6,1.0,3,5,2,2,2636,0.435885717,220,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:06:10Z,2026-09-11T04:50:19Z +codex,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,3683,0.322170501,131,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:16Z,2026-09-11T04:26:44Z +codex,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,1727,0.352383278,181,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:57:19Z,2026-09-11T04:26:25Z +codex,psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,2428,0.229493231,130,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:21Z,2026-09-11T04:38:58Z +codex,wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,1911,0.069837152,51,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:55Z,2026-09-11T04:40:56Z +codex,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,1144,0.130471568,74,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:11Z,2026-09-11T03:44:25Z +codex,actionlint-action-pinning-lint,easy,106,go,0,0,0.981818,1.0,54,55,145,145,1660,0.312072217,147,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:47Z,2026-09-11T04:26:34Z +codex,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.965217,1.0,111,115,1281,1281,2351,0.179995362,91,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:56:58Z,2026-09-11T04:36:17Z +codex,returns-validated-error-accumulation,easy,108,python,0,0,0.993711,1.0,158,159,61,61,2720,0.433079207,190,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:08:49Z,2026-09-11T04:54:19Z +codex,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,1278,0.117304523,69,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:25:30Z,2026-09-11T03:46:57Z +codex,drizzle-orm-window-function-builders,easy,110,typescript,0,1,1.0,1.0,130,130,566,566,1494,0.286798704,177,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:58:22Z,2026-09-11T04:23:33Z +codex,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,5327,0.545081976,215,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:55:46Z,2026-09-11T05:24:46Z +codex,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,2381,0.504645421,224,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T04:06:48Z,2026-09-11T04:46:38Z +codex,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.958333,1.0,92,96,561,561,1818,0.220791661,128,0,,deepseek/deepseek-v4-flash-0731,0.154.0,s1,2026-09-11T03:24:59Z,2026-09-11T03:55:27Z diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/deepseek-harness.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/deepseek-harness.csv new file mode 100644 index 0000000000..b24739cb27 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/deepseek-harness.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +deepseek-harness,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.95122,1.0,39,41,1131,1131,8334,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:41+00:00,2026-09-11T10:21:36+00:00 +deepseek-harness,gql-incremental-graphql-delivery,hard,2,python,0,0,0.058824,0.998767,1,17,810,811,7396,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:02+00:00,2026-09-11T10:06:19+00:00 +deepseek-harness,bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,7883,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:57+00:00,2026-09-11T10:14:21+00:00 +deepseek-harness,termenv-preserve-ansi-resets,hard,4,go,0,0,0.857143,1.0,30,35,87,87,8375,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:37+00:00,2026-09-11T10:22:14+00:00 +deepseek-harness,updo-policy-alerting,hard,5,go,0,1,1.0,1.0,17,17,123,123,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:42+00:00,2026-09-11T11:04:46+00:00 +deepseek-harness,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.893617,1.0,42,47,70,70,10871,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:02+00:00,2026-09-11T11:04:16+00:00 +deepseek-harness,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,0.0,0,6,0,62,3441,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:55+00:00,2026-09-11T09:00:17+00:00 +deepseek-harness,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.0,1.0,0,32,1273,1273,2296,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:37+00:00,2026-09-11T08:40:55+00:00 +deepseek-harness,helm-array-merge-strategies,hard,9,go,0,0,0.0,0.0,0,47,0,12,7372,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:03+00:00,2026-09-11T10:05:56+00:00 +deepseek-harness,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.938776,0.999981,46,49,51468,51469,4959,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:20+00:00,2026-09-11T09:26:02+00:00 +deepseek-harness,pest-character-class-coalescing,hard,11,rust,0,0,0.0,1.0,0,104,250,250,2671,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:53+00:00,2026-09-11T08:47:26+00:00 +deepseek-harness,testem-bail-on-test-failure,hard,12,javascript,0,0,0.866667,1.0,78,90,489,489,4190,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:37+00:00,2026-09-11T09:12:29+00:00 +deepseek-harness,koota-pair-relation-tracking,hard,13,typescript,0,0,0.973684,1.0,37,38,172,172,4698,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:09+00:00,2026-09-11T09:21:31+00:00 +deepseek-harness,prometheus-transactional-reload-status,hard,14,typescript,0,0,1.0,0.878049,15,15,72,82,10921,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:22+00:00,2026-09-11T11:06:28+00:00 +deepseek-harness,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.0,1.0,0,60,1131,1131,4175,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:07+00:00,2026-09-11T09:12:43+00:00 +deepseek-harness,koota-query-predicates,hard,16,typescript,0,0,0.0,1.0,0,43,172,172,2374,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:49+00:00,2026-09-11T08:42:24+00:00 +deepseek-harness,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.615385,1.0,8,13,22,22,4449,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:59+00:00,2026-09-11T09:18:09+00:00 +deepseek-harness,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.380282,1.0,27,71,128,128,4799,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:22+00:00,2026-09-11T09:24:22+00:00 +deepseek-harness,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.960526,1.0,73,76,1412,1412,3219,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:10+00:00,2026-09-11T08:56:50+00:00 +deepseek-harness,ink-grid-box-layout,hard,20,typescript,0,0,0.92,1.0,23,25,49,49,4683,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:02:58+00:00,2026-09-11T09:21:02+00:00 +deepseek-harness,awilix-async-container-initialization,hard,21,typescript,0,0,0.75,1.0,18,24,162,162,8981,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:59+00:00,2026-09-11T10:33:44+00:00 +deepseek-harness,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.785714,1.0,11,14,9,9,1551,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:37+00:00,2026-09-11T08:30:29+00:00 +deepseek-harness,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.860759,1.0,68,79,16715,16715,2286,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:29+00:00,2026-09-11T08:41:36+00:00 +deepseek-harness,superjson-error-stack-serialization,hard,24,typescript,0,0,0.7625,0.991379,61,80,115,116,2668,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:12+00:00,2026-09-11T08:47:41+00:00 +deepseek-harness,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.75,1.0,9,12,139,139,6143,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:08+00:00,2026-09-11T09:46:32+00:00 +deepseek-harness,vulture-persistent-analysis-cache,hard,26,python,0,0,0.5,1.0,12,24,295,295,6956,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:37+00:00,2026-09-11T10:00:35+00:00 +deepseek-harness,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.0,1.0,0,94,599,599,8990,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:52+00:00,2026-09-11T10:33:46+00:00 +deepseek-harness,expr-try-catch-errors,hard,28,go,0,0,0.0,1.0,0,79,66265,66265,2003,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:13+00:00,2026-09-11T08:36:37+00:00 +deepseek-harness,koota-composite-trait-aspects,hard,29,typescript,0,0,0.941176,1.0,48,51,172,172,3355,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:08+00:00,2026-09-11T09:00:04+00:00 +deepseek-harness,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.0,1.0,0,82,6,6,8919,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:52+00:00,2026-09-11T10:33:35+00:00 +deepseek-harness,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,7381,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:13+00:00,2026-09-11T10:07:17+00:00 +deepseek-harness,participle-grammar-conflict-analysis,hard,32,go,0,0,0.142857,1.0,13,91,153,153,9027,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:15+00:00,2026-09-11T10:33:46+00:00 +deepseek-harness,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,7148,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:12+00:00,2026-09-11T10:03:22+00:00 +deepseek-harness,clack-async-autocomplete-options,hard,34,typescript,0,0,0.0,1.0,0,82,643,643,2766,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:56+00:00,2026-09-11T08:51:05+00:00 +deepseek-harness,langchain-request-coalescing,hard,35,python,0,0,0.82,1.0,41,50,232,232,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:17+00:00,2026-09-11T11:06:23+00:00 +deepseek-harness,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,5088,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:24+00:00,2026-09-11T09:28:14+00:00 +deepseek-harness,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.944444,1.0,34,36,2034,2034,8953,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:19+00:00,2026-09-11T10:33:35+00:00 +deepseek-harness,textual-kitty-key-phases,hard,38,python,0,0,0.478261,1.0,11,23,57,57,6606,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:08+00:00,2026-09-11T09:55:19+00:00 +deepseek-harness,pwntools-tube-multiplexing,medium,39,python,0,0,0.342466,1.0,25,73,1,1,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:21+00:00,2026-09-11T11:06:22+00:00 +deepseek-harness,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.0,1.0,0,25,1679,1679,8982,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:58+00:00,2026-09-11T10:33:44+00:00 +deepseek-harness,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.985075,1.0,66,67,2035,2035,3945,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:20+00:00,2026-09-11T09:10:06+00:00 +deepseek-harness,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,0.995327,0,2,213,214,4877,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:56+00:00,2026-09-11T09:27:14+00:00 +deepseek-harness,bandit-incremental-cache-control,medium,43,python,0,0,0.931818,1.0,82,88,275,275,5634,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:23+00:00,2026-09-11T09:38:28+00:00 +deepseek-harness,fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,3378,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:03:58+00:00,2026-09-11T09:00:18+00:00 +deepseek-harness,go-critic-doc-link-checker,medium,45,go,0,0,0.0,0.0,0,3,0,16,8049,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:23+00:00,2026-09-11T10:18:34+00:00 +deepseek-harness,igel-persist-feature-schema,medium,46,python,0,1,1.0,1.0,24,24,2,2,7389,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:06:06+00:00,2026-09-11T10:09:16+00:00 +deepseek-harness,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.560606,1.0,37,66,293,293,2173,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:26+00:00,2026-09-11T08:40:39+00:00 +deepseek-harness,anko-typed-variable-bindings,medium,48,go,0,0,0.0,1.0,0,9,94,94,1342,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:17+00:00,2026-09-11T08:26:39+00:00 +deepseek-harness,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.0,1.0,0,66,30014,30014,7288,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:24+00:00,2026-09-11T10:05:54+00:00 +deepseek-harness,dasel-html-document-format,medium,50,go,0,0,0.965753,1.0,141,146,1012,1012,5247,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:06:25+00:00,2026-09-11T09:33:54+00:00 +deepseek-harness,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,0.0,0,17,0,7,5583,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:52+00:00,2026-09-11T09:39:00+00:00 +deepseek-harness,fastapi-implicit-head-options,medium,52,python,0,0,0.790698,1.0,34,43,3134,3134,4099,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:17+00:00,2026-09-11T09:12:38+00:00 +deepseek-harness,prometheus-typed-label-sorting,medium,53,go,0,0,0.882353,1.0,15,17,28,28,6424,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:29+00:00,2026-09-11T09:52:34+00:00 +deepseek-harness,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,0,0.0,1.0,0,7,31,31,4801,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:06:44+00:00,2026-09-11T09:26:46+00:00 +deepseek-harness,pebble-durability-wait-apis,medium,55,go,0,0,0.0,0.0,0,59,0,44,10601,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:54+00:00,2026-09-11T11:02:36+00:00 +deepseek-harness,cliffy-config-file-parsing,medium,56,typescript,0,0,0.837838,1.0,31,37,451,451,6147,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:23+00:00,2026-09-11T09:46:51+00:00 +deepseek-harness,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.766667,1.0,46,60,1038,1038,3489,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:05:29+00:00,2026-09-11T09:03:39+00:00 +deepseek-harness,kombu-single-active-consumer-priority,medium,58,python,0,0,0.894118,1.0,76,85,1421,1421,4936,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:06:45+00:00,2026-09-11T09:29:03+00:00 +deepseek-harness,skrub-duration-encoding,medium,59,python,0,0,0.976923,1.0,127,130,2784,2784,8084,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:06:16+00:00,2026-09-11T10:21:01+00:00 +deepseek-harness,scc-bounded-memory-spilling,medium,60,go,0,0,0.903226,1.0,28,31,286,286,4058,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:04:27+00:00,2026-09-11T09:12:05+00:00 +deepseek-harness,mnamer-daemon-watch-lifecycle,medium,61,python,0,0,0.607843,1.0,31,51,319,319,6463,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:00:26+00:00,2026-09-11T10:48:10+00:00 +deepseek-harness,tengo-destructuring-bindings,medium,62,go,0,1,1.0,1.0,91,91,132,132,3061,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:30:34+00:00,2026-09-11T09:21:36+00:00 +deepseek-harness,httpx-multipart-response-parsing,medium,63,python,0,0,0.0,1.0,0,122,1272,1272,3405,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:40:50+00:00,2026-09-11T09:37:36+00:00 +deepseek-harness,opa-rego-rule-profiling,medium,64,go,0,1,1.0,1.0,25,25,6,6,6241,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:27:50+00:00,2026-09-11T10:11:53+00:00 +deepseek-harness,ipython-session-bundle-replay,medium,65,python,0,0,0.764706,1.0,13,17,29,29,6336,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:04:13+00:00,2026-09-11T10:49:51+00:00 +deepseek-harness,scriggo-method-declarations,medium,66,go,0,0,0.0,1.0,0,48,1049,1049,1476,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:51:11+00:00,2026-09-11T09:15:48+00:00 +deepseek-harness,opa-template-string-reconstruction,medium,67,go,0,0,0.0,1.0,0,5,4,4,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:42:17+00:00,2026-09-11T11:44:18+00:00 +deepseek-harness,mobly-grouped-test-barriers,medium,68,python,0,0,0.0,1.0,0,79,808,808,7011,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:36:41+00:00,2026-09-11T10:33:35+00:00 +deepseek-harness,tengo-callable-instance-isolation,medium,69,go,0,0,0.0,1.0,0,23,122,122,4775,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:10:25+00:00,2026-09-11T10:30:01+00:00 +deepseek-harness,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,0.0,0,43,0,109,3836,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:16:08+00:00,2026-09-11T10:20:05+00:00 +deepseek-harness,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.985507,1.0,68,69,7,7,2625,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:47:30+00:00,2026-09-11T09:31:16+00:00 +deepseek-harness,tomlkit-toml-table-converters,medium,72,python,0,0,0.983333,1.0,59,60,964,964,6133,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:41:04+00:00,2026-09-11T10:23:18+00:00 +deepseek-harness,etree-xml-diff-patch,medium,73,go,0,0,0.038462,1.0,2,52,15,15,4506,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:18:26+00:00,2026-09-11T10:33:36+00:00 +deepseek-harness,textual-richlog-follow-state,medium,74,python,0,0,0.95,1.0,19,20,6,6,9157,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:22:09+00:00,2026-09-11T11:54:46+00:00 +deepseek-harness,yaegi-go-embed-directives,medium,75,go,0,0,0.0,0.0,0,38,0,58,5110,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:57:40+00:00,2026-09-11T10:22:52+00:00 +deepseek-harness,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.0,1.0,0,254,22,22,2489,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:42:53+00:00,2026-09-11T09:24:23+00:00 +deepseek-harness,numba-stencil-boundary-modes,easy,77,python,0,0,0.862069,1.0,25,29,827,827,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:22:08+00:00,2026-09-11T12:24:10+00:00 +deepseek-harness,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.0,1.0,0,10,209,209,4121,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:25:00+00:00,2026-09-11T10:33:45+00:00 +deepseek-harness,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,0.0,0,17,0,2,5590,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:00:30+00:00,2026-09-11T10:33:45+00:00 +deepseek-harness,adaptix-name-mapping-aliases,easy,80,python,1,0,0.0,1.0,0,44,2738,2738,6347,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T08:47:53+00:00,2026-09-11T10:33:44+00:00 +deepseek-harness,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,1.0,0,33,1133,1133,2811,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:46:50+00:00,2026-09-11T10:33:45+00:00 +deepseek-harness,ts-pattern-match-each,easy,82,typescript,0,1,1.0,1.0,85,85,6,6,3527,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:26:56+00:00,2026-09-11T10:25:43+00:00 +deepseek-harness,anko-default-function-arguments,easy,83,go,0,0,0.0,1.0,0,2,119,119,1486,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:12:48+00:00,2026-09-11T09:37:34+00:00 +deepseek-harness,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.0,1.0,0,37,1267,1267,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:01:52+00:00,2026-09-11T12:03:53+00:00 +deepseek-harness,abs-stepped-slices,easy,85,go,0,1,1.0,1.0,6,6,6,6,10005,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:54:08+00:00,2026-09-11T12:40:53+00:00 +deepseek-harness,testem-per-launcher-reports,easy,86,javascript,0,0,0.707692,1.0,46,65,469,469,2645,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:27:57+00:00,2026-09-11T10:12:02+00:00 +deepseek-harness,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,5730,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:32:42+00:00,2026-09-11T11:08:13+00:00 +deepseek-harness,wasmi-trap-coredumps,easy,88,rust,0,0,0.136364,1.0,3,22,58,58,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:13:07+00:00,2026-09-11T12:15:09+00:00 +deepseek-harness,koota-entity-snapshot-rollback,easy,89,python,0,0,0.761905,1.0,64,84,47,47,1792,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:04:14+00:00,2026-09-11T10:34:09+00:00 +deepseek-harness,geo-shapeindex-serialization,easy,90,go,0,0,0.583333,1.0,14,24,599,599,4578,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:29:58+00:00,2026-09-11T10:46:18+00:00 +deepseek-harness,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,0.0,0,82,0,2,4227,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:38:32+00:00,2026-09-11T10:49:00+00:00 +deepseek-harness,abs-module-cache-flags,easy,92,go,0,0,0.8,1.0,16,20,3,3,4118,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:13:13+00:00,2026-09-11T10:21:53+00:00 +deepseek-harness,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.830189,1.0,44,53,8,8,2435,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:06:02+00:00,2026-09-11T10:46:38+00:00 +deepseek-harness,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,3608,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:33:33+00:00,2026-09-11T10:33:45+00:00 +deepseek-harness,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,2296,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:39:16+00:00,2026-09-11T10:17:33+00:00 +deepseek-harness,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,10141,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:14:01+00:00,2026-09-11T12:03:02+00:00 +deepseek-harness,task-task-graph-export,easy,97,go,0,0,0.0,1.0,0,20,17,17,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:09:24+00:00,2026-09-11T13:11:25+00:00 +deepseek-harness,go-genai-streamed-function-args,easy,98,go,0,0,0.833333,1.0,5,6,62,62,1678,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:34:51+00:00,2026-09-11T10:02:50+00:00 +deepseek-harness,ytt-jsonpath-query-api,easy,99,go,0,0,0.941748,1.0,97,103,1,1,2392,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:39:50+00:00,2026-09-11T10:19:44+00:00 +deepseek-harness,helm-unified-manifest-stream,easy,100,go,1,0,0.0,0.5,0,5,1,2,5020,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:21:49+00:00,2026-09-11T10:45:30+00:00 +deepseek-harness,yjs-map-conflict-detection,easy,101,javascript,0,0,0.777778,1.0,7,9,231,231,862,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:19:08+00:00,2026-09-11T10:33:34+00:00 +deepseek-harness,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,8434,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:57:03+00:00,2026-09-11T12:17:38+00:00 +deepseek-harness,psd-tools-blend-range-api,easy,103,python,0,0,0.711111,1.0,32,45,979,979,4000,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:39:55+00:00,2026-09-11T10:46:36+00:00 +deepseek-harness,wazero-multi-module-snapshots,easy,104,go,0,0,0.705128,1.0,55,78,2,2,4113,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:25:21+00:00,2026-09-11T10:33:58+00:00 +deepseek-harness,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:22:32+00:00,2026-09-11T13:24:33+00:00 +deepseek-harness,actionlint-action-pinning-lint,easy,106,go,0,0,0.090909,1.0,5,55,145,145,1967,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:00:54+00:00,2026-09-11T10:33:44+00:00 +deepseek-harness,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.991304,1.0,114,115,1281,1281,2731,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:10:05+00:00,2026-09-11T10:55:38+00:00 +deepseek-harness,returns-validated-error-accumulation,easy,108,python,0,0,0.0,1.0,0,159,61,61,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:28:38+00:00,2026-09-11T12:30:39+00:00 +deepseek-harness,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,0,0.0,1.0,0,47,13,13,199,,,1,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:30:13+00:00,2026-09-11T10:33:35+00:00 +deepseek-harness,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.961538,1.0,125,130,566,566,10920,,,124,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:04:22+00:00,2026-09-11T13:06:25+00:00 +deepseek-harness,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,4412,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:14:59+00:00,2026-09-11T11:28:33+00:00 +deepseek-harness,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,2656,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T09:47:41+00:00,2026-09-11T10:31:59+00:00 +deepseek-harness,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.604167,1.0,58,96,561,561,5161,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.1.5-rc.1,verifier,2026-09-11T10:34:06+00:00,2026-09-11T12:00:09+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/kilo.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/kilo.csv new file mode 100644 index 0000000000..039c8e10b1 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/kilo.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +kilo,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,3393,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:01+00:00,2026-09-11T15:33:35+00:00 +kilo,gql-incremental-graphql-delivery,hard,2,python,0,0,0.941176,0.998767,16,17,810,811,2598,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:36:48+00:00,2026-09-11T15:20:07+00:00 +kilo,bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,2114,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:49+00:00,2026-09-11T13:25:05+00:00 +kilo,termenv-preserve-ansi-resets,hard,4,go,0,0,0.8,1.0,28,35,87,87,3123,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:44+00:00,2026-09-11T13:41:49+00:00 +kilo,updo-policy-alerting,hard,5,go,0,0,0.764706,1.0,13,17,123,123,1669,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:36:49+00:00,2026-09-11T15:04:39+00:00 +kilo,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.851064,1.0,40,47,70,70,5254,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:11+00:00,2026-09-11T14:17:47+00:00 +kilo,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,1.0,0,6,62,62,2373,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:49+00:00,2026-09-11T13:29:25+00:00 +kilo,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.84375,0.995287,27,32,1267,1273,4444,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:48:49+00:00,2026-09-11T17:02:53+00:00 +kilo,helm-array-merge-strategies,hard,9,go,0,0,0.744681,1.0,35,47,12,12,4326,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:01+00:00,2026-09-11T14:02:08+00:00 +kilo,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.959184,0.999961,47,49,51467,51469,2353,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:11+00:00,2026-09-11T13:29:25+00:00 +kilo,pest-character-class-coalescing,hard,11,rust,0,0,0.0,0.0,0,104,0,250,1872,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:18+00:00,2026-09-11T15:08:30+00:00 +kilo,testem-bail-on-test-failure,hard,12,javascript,0,0,0.922222,0.997955,83,90,488,489,4839,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:45+00:00,2026-09-11T14:10:27+00:00 +kilo,koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,3784,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:02+00:00,2026-09-11T13:53:07+00:00 +kilo,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.0,0,15,0,82,2506,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:46+00:00,2026-09-11T15:19:33+00:00 +kilo,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.983333,1.0,59,60,1131,1131,1651,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:19+00:00,2026-09-11T15:04:51+00:00 +kilo,koota-query-predicates,hard,16,typescript,0,0,0.069767,1.0,3,43,172,172,3092,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:49:01+00:00,2026-09-11T16:40:35+00:00 +kilo,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.615385,1.0,8,13,22,22,2769,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:19+00:00,2026-09-11T13:36:29+00:00 +kilo,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.971831,1.0,69,71,128,128,3000,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:47+00:00,2026-09-11T15:27:49+00:00 +kilo,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.881579,0.997167,67,76,1408,1412,1923,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:49+00:00,2026-09-11T13:21:54+00:00 +kilo,ink-grid-box-layout,hard,20,typescript,0,0,0.92,1.0,23,25,49,49,4571,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:49:13+00:00,2026-09-11T17:05:25+00:00 +kilo,awilix-async-container-initialization,hard,21,typescript,0,1,1.0,1.0,24,24,162,162,1711,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:05+00:00,2026-09-11T15:05:41+00:00 +kilo,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.785714,1.0,11,14,9,9,2002,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:05+00:00,2026-09-11T15:11:28+00:00 +kilo,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.936709,1.0,74,79,16715,16715,3551,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:50+00:00,2026-09-11T13:49:03+00:00 +kilo,superjson-error-stack-serialization,hard,24,typescript,0,0,0.0,1.0,0,80,116,116,542,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:49:13+00:00,2026-09-11T15:58:16+00:00 +kilo,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.0,1.0,0,12,139,139,1843,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:00+00:00,2026-09-11T13:20:45+00:00 +kilo,vulture-persistent-analysis-cache,hard,26,python,0,0,1.0,0.986441,24,24,291,295,2077,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:10+00:00,2026-09-11T13:24:50+00:00 +kilo,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.0,1.0,0,94,599,599,2151,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:41+00:00,2026-09-11T15:13:34+00:00 +kilo,expr-try-catch-errors,hard,28,go,0,0,0.974684,1.0,77,79,66265,66265,4015,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:43+00:00,2026-09-11T13:56:41+00:00 +kilo,koota-composite-trait-aspects,hard,29,typescript,0,0,0.960784,1.0,49,51,172,172,5483,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:00+00:00,2026-09-11T14:21:25+00:00 +kilo,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.743902,0.833333,61,82,5,6,5212,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:10+00:00,2026-09-11T14:17:03+00:00 +kilo,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,1403,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:03+00:00,2026-09-11T15:01:26+00:00 +kilo,participle-grammar-conflict-analysis,hard,32,go,0,0,0.967033,1.0,88,91,153,153,3159,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:49:13+00:00,2026-09-11T16:41:54+00:00 +kilo,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,3189,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:00+00:00,2026-09-11T13:43:12+00:00 +kilo,clack-async-autocomplete-options,hard,34,typescript,0,1,1.0,1.0,82,82,643,643,2793,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:09+00:00,2026-09-11T15:24:43+00:00 +kilo,langchain-request-coalescing,hard,35,python,0,0,0.98,1.0,49,50,232,232,4443,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:50+00:00,2026-09-11T14:03:55+00:00 +kilo,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,1426,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:44+00:00,2026-09-11T13:13:33+00:00 +kilo,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.972222,1.0,35,36,2034,2034,4752,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:01+00:00,2026-09-11T14:09:14+00:00 +kilo,textual-kitty-key-phases,hard,38,python,0,0,0.478261,1.0,11,23,57,57,6012,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:10+00:00,2026-09-11T14:30:24+00:00 +kilo,pwntools-tube-multiplexing,medium,39,python,0,0,0.986301,1.0,72,73,1,1,4162,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:03+00:00,2026-09-11T15:47:26+00:00 +kilo,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.92,1.0,23,25,1679,1679,4038,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:48+00:00,2026-09-11T13:57:08+00:00 +kilo,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.820896,1.0,55,67,2035,2035,4110,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:01+00:00,2026-09-11T13:58:33+00:00 +kilo,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,0.0,0,2,0,214,2092,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:10+00:00,2026-09-11T13:25:04+00:00 +kilo,bandit-incremental-cache-control,medium,43,python,0,0,0.988636,1.0,87,88,275,275,2960,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:05+00:00,2026-09-11T15:27:26+00:00 +kilo,fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,1631,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:49:14+00:00,2026-09-11T16:16:27+00:00 +kilo,go-critic-doc-link-checker,medium,45,go,0,0,0.0,0.0,0,3,0,16,2391,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:00+00:00,2026-09-11T13:29:52+00:00 +kilo,igel-persist-feature-schema,medium,46,python,0,0,0.916667,1.0,22,24,2,2,1083,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:30+00:00,2026-09-11T14:56:33+00:00 +kilo,bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,2597,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:49+00:00,2026-09-11T13:33:07+00:00 +kilo,anko-typed-variable-bindings,medium,48,go,0,0,0.0,1.0,0,9,94,94,10157,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:44+00:00,2026-09-11T15:39:04+00:00 +kilo,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.909091,1.0,60,66,30014,30014,2180,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:00+00:00,2026-09-11T13:26:21+00:00 +kilo,dasel-html-document-format,medium,50,go,0,0,0.986301,1.0,144,146,1012,1012,2324,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:35+00:00,2026-09-11T15:17:20+00:00 +kilo,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,0.0,0,17,0,7,6181,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:26+00:00,2026-09-11T14:33:29+00:00 +kilo,fastapi-implicit-head-options,medium,52,python,0,0,0.813953,1.0,35,43,3134,3134,5475,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:44+00:00,2026-09-11T14:21:02+00:00 +kilo,prometheus-typed-label-sorting,medium,53,go,0,0,0.0,1.0,0,17,28,28,1365,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:37:50+00:00,2026-09-11T15:00:36+00:00 +kilo,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,1343,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:50:11+00:00,2026-09-11T13:12:35+00:00 +kilo,pebble-durability-wait-apis,medium,55,go,0,0,0.0,1.0,0,59,44,44,2201,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:15+00:00,2026-09-11T15:14:57+00:00 +kilo,cliffy-config-file-parsing,medium,56,typescript,0,0,0.0,1.0,0,37,451,451,1273,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:44+00:00,2026-09-11T13:10:59+00:00 +kilo,sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,3127,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:59+00:00,2026-09-11T13:42:09+00:00 +kilo,kombu-single-active-consumer-priority,medium,58,python,0,0,0.882353,1.0,75,85,1421,1421,2830,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:30+00:00,2026-09-11T15:25:42+00:00 +kilo,skrub-duration-encoding,medium,59,python,0,0,0.976923,1.0,127,130,2784,2784,5538,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T14:38:48+00:00,2026-09-11T16:11:07+00:00 +kilo,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,2171,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T12:49:46+00:00,2026-09-11T13:25:59+00:00 +kilo,mnamer-daemon-watch-lifecycle,medium,61,python,0,0,0.666667,1.0,34,51,319,319,1657,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:21+00:00,2026-09-11T16:02:59+00:00 +kilo,tengo-destructuring-bindings,medium,62,go,0,0,0.945055,1.0,86,91,132,132,2711,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:00+00:00,2026-09-11T16:13:12+00:00 +kilo,httpx-multipart-response-parsing,medium,63,python,0,1,1.0,1.0,122,122,1272,1272,1372,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:22+00:00,2026-09-11T16:44:15+00:00 +kilo,opa-rego-rule-profiling,medium,64,go,0,0,0.88,1.0,22,25,6,6,3001,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:06:53+00:00,2026-09-11T17:56:56+00:00 +kilo,ipython-session-bundle-replay,medium,65,python,0,0,0.352941,1.0,6,17,29,29,7292,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:35+00:00,2026-09-11T17:36:07+00:00 +kilo,scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,10031,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:17+00:00,2026-09-11T18:15:30+00:00 +kilo,opa-template-string-reconstruction,medium,67,go,0,0,0.0,0.0,0,5,0,4,2828,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:14+00:00,2026-09-11T17:08:24+00:00 +kilo,mobly-grouped-test-barriers,medium,68,python,0,0,0.936709,1.0,74,79,808,808,1540,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:05:45+00:00,2026-09-11T17:31:25+00:00 +kilo,tengo-callable-instance-isolation,medium,69,go,0,0,0.956522,1.0,22,23,122,122,3651,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:30+00:00,2026-09-11T16:35:22+00:00 +kilo,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,0.0,0,43,0,109,1700,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:17+00:00,2026-09-11T15:56:38+00:00 +kilo,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.985507,1.0,68,69,7,7,2327,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:25+00:00,2026-09-11T17:00:13+00:00 +kilo,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,3481,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:06:53+00:00,2026-09-11T18:04:55+00:00 +kilo,etree-xml-diff-patch,medium,73,go,0,0,0.0,1.0,0,52,15,15,1838,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:25+00:00,2026-09-11T16:05:03+00:00 +kilo,textual-richlog-follow-state,medium,74,python,0,1,1.0,1.0,20,20,6,6,3177,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:31:05+00:00,2026-09-11T16:24:04+00:00 +kilo,yaegi-go-embed-directives,medium,75,go,0,0,0.973684,1.0,37,38,58,58,5305,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:24+00:00,2026-09-11T17:49:49+00:00 +kilo,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.984252,1.0,250,254,22,22,1950,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:06:12+00:00,2026-09-11T17:38:46+00:00 +kilo,numba-stencil-boundary-modes,easy,77,python,0,0,0.862069,1.0,25,29,827,827,4674,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:19+00:00,2026-09-11T16:53:16+00:00 +kilo,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.0,1.0,0,10,209,209,6259,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:40+00:00,2026-09-11T17:13:01+00:00 +kilo,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,1.0,0,17,2,2,2414,,,1,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:44+00:00,2026-09-11T17:01:58+00:00 +kilo,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,1872,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:55+00:00,2026-09-11T17:39:09+00:00 +kilo,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,1,1.0,1.0,33,33,1133,1133,3927,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:27+00:00,2026-09-11T16:39:55+00:00 +kilo,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,979,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:50+00:00,2026-09-11T15:45:10+00:00 +kilo,anko-default-function-arguments,easy,83,go,0,0,0.0,1.0,0,2,119,119,10800,,,124,1,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:23:39+00:00,2026-09-11T19:23:40+00:00 +kilo,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.972973,0.996054,36,37,1262,1267,10800,,,124,1,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:05+00:00,2026-09-11T20:07:06+00:00 +kilo,abs-stepped-slices,easy,85,go,0,0,0.333333,1.0,2,6,6,6,1418,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:23+00:00,2026-09-11T15:58:02+00:00 +kilo,testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,1721,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:41+00:00,2026-09-11T15:57:23+00:00 +kilo,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2015,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:25:02+00:00,2026-09-11T16:58:39+00:00 +kilo,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,0.0,0,22,0,58,3571,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:38+00:00,2026-09-11T18:07:11+00:00 +kilo,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1880,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:38+00:00,2026-09-11T16:06:59+00:00 +kilo,geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,3038,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:54+00:00,2026-09-11T16:19:34+00:00 +kilo,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,0.0,0,82,0,2,1242,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:23:39+00:00,2026-09-11T16:44:22+00:00 +kilo,abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,1995,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:10+00:00,2026-09-11T17:40:32+00:00 +kilo,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.90566,1.0,48,53,8,8,1742,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:34:41+00:00,2026-09-11T16:03:44+00:00 +kilo,kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,2917,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:28:52+00:00,2026-09-11T16:17:29+00:00 +kilo,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,6400,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:25:46+00:00,2026-09-11T18:12:29+00:00 +kilo,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,5113,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:37+00:00,2026-09-11T18:32:52+00:00 +kilo,task-task-graph-export,easy,97,go,0,0,0.0,1.0,0,20,17,17,4505,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:36:34+00:00,2026-09-11T16:51:40+00:00 +kilo,go-genai-streamed-function-args,easy,98,go,0,0,0.666667,1.0,4,6,62,62,2491,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:31:05+00:00,2026-09-11T16:12:38+00:00 +kilo,ytt-jsonpath-query-api,easy,99,go,0,0,0.0,0.0,0,103,0,1,2412,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:21:39+00:00,2026-09-11T17:01:52+00:00 +kilo,helm-unified-manifest-stream,easy,100,go,1,0,0.0,0.5,0,5,1,2,2301,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:55+00:00,2026-09-11T17:46:16+00:00 +kilo,yjs-map-conflict-detection,easy,101,javascript,0,0,0.888889,1.0,8,9,231,231,4085,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:49+00:00,2026-09-11T16:43:56+00:00 +kilo,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,2862,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:31:05+00:00,2026-09-11T16:18:49+00:00 +kilo,psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,1213,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:25:10+00:00,2026-09-11T16:45:25+00:00 +kilo,wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,1729,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:08:03+00:00,2026-09-11T17:36:55+00:00 +kilo,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.961538,1.0,25,26,5709,5709,1965,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:36:02+00:00,2026-09-11T16:08:48+00:00 +kilo,actionlint-action-pinning-lint,easy,106,go,0,0,0.854545,1.0,47,55,145,145,1683,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:31:45+00:00,2026-09-11T15:59:49+00:00 +kilo,httpx-deterministic-cookie-store,easy,107,typescript,0,1,1.0,1.0,115,115,1281,1281,4171,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:25:10+00:00,2026-09-11T17:34:41+00:00 +kilo,returns-validated-error-accumulation,easy,108,python,0,0,0.987421,1.0,157,159,61,61,1954,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:07:57+00:00,2026-09-11T17:40:31+00:00 +kilo,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,0,0.978723,1.0,46,47,13,13,2298,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:43+00:00,2026-09-11T16:14:03+00:00 +kilo,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.161538,1.0,21,130,566,566,2255,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:31:45+00:00,2026-09-11T16:09:22+00:00 +kilo,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,2644,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T16:25:55+00:00,2026-09-11T17:09:59+00:00 +kilo,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,5086,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T17:08:25+00:00,2026-09-11T18:33:11+00:00 +kilo,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.770833,1.0,74,96,561,561,2314,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,7.6.2,verifier,2026-09-11T15:35:10+00:00,2026-09-11T16:13:45+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/mini-swe-agent.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/mini-swe-agent.csv new file mode 100644 index 0000000000..c55dadadc8 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/mini-swe-agent.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +mini-swe-agent,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,1506,0.08360655960000002,117,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3112,2026-09-10T18:50:38Z,2026-09-10T19:15:53Z +mini-swe-agent,gql-incremental-graphql-delivery,hard,2,python,0,0,1.0,0.998767,17,17,810,811,2190,0.13690820979999993,177,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3114,2026-09-10T18:50:40Z,2026-09-10T19:28:35Z +mini-swe-agent,bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,1839,0.2490900480000001,145,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213657-247089,2026-09-10T21:36:57Z,2026-09-10T22:07:58Z +mini-swe-agent,termenv-preserve-ansi-resets,hard,4,go,0,0,0.885714,1.0,31,35,87,87,1358,0.0564377052,48,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3143,2026-09-10T18:50:38Z,2026-09-10T19:13:47Z +mini-swe-agent,updo-policy-alerting,hard,5,go,0,0,0.941176,1.0,16,17,123,123,2150,0.125868136,108,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3113,2026-09-10T18:50:38Z,2026-09-10T19:27:17Z +mini-swe-agent,effect-sse-httpapi-streaming,hard,6,typescript,0,1,1.0,1.0,47,47,70,70,8305,1.7064809288,435,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3115,2026-09-10T18:50:40Z,2026-09-10T21:10:37Z +mini-swe-agent,oxvg-structural-selector-preservation,hard,7,rust,0,1,1.0,1.0,6,6,62,62,3147,0.7097812137999999,229,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3134,2026-09-10T18:50:48Z,2026-09-10T19:45:07Z +mini-swe-agent,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.875,0.996858,28,32,1269,1273,2036,0.43634072699999993,166,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222642-142466,2026-09-10T22:26:42Z,2026-09-10T23:00:47Z +mini-swe-agent,helm-array-merge-strategies,hard,9,go,0,1,1.0,1.0,47,47,12,12,2540,0.2408197367999999,209,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3114,2026-09-10T18:50:38Z,2026-09-10T19:33:45Z +mini-swe-agent,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.979592,1.0,48,49,51469,51469,2677,0.5268188490000002,197,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3116,2026-09-10T18:50:40Z,2026-09-10T19:36:48Z +mini-swe-agent,pest-character-class-coalescing,hard,11,rust,0,0,1.0,0.996,104,104,249,250,4109,0.4352379926000001,220,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3135,2026-09-10T18:50:48Z,2026-09-10T20:01:09Z +mini-swe-agent,testem-bail-on-test-failure,hard,12,javascript,0,0,0.666667,1.0,60,90,489,489,3052,0.36010074899999994,108,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3145,2026-09-10T18:50:38Z,2026-09-10T19:42:02Z +mini-swe-agent,koota-pair-relation-tracking,hard,13,typescript,0,0,0.973684,1.0,37,38,172,172,4094,0.29172684660000003,171,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3115,2026-09-10T18:50:38Z,2026-09-10T19:59:39Z +mini-swe-agent,prometheus-transactional-reload-status,hard,14,typescript,0,0,1.0,0.878049,15,15,72,82,3715,0.6438112160000001,257,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3117,2026-09-10T18:50:40Z,2026-09-10T19:54:07Z +mini-swe-agent,obsidian-linter-link-format-conversion,hard,15,typescript,0,1,1.0,1.0,60,60,1131,1131,2312,0.25263631000000003,131,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213608-247092,2026-09-10T21:36:08Z,2026-09-10T22:14:50Z +mini-swe-agent,koota-query-predicates,hard,16,typescript,0,0,0.906977,1.0,39,43,172,172,2235,0.13148473580000003,159,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3146,2026-09-10T18:50:38Z,2026-09-10T19:28:25Z +mini-swe-agent,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,1780,0.30539385300000005,140,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211829-350830,2026-09-10T21:18:29Z,2026-09-10T21:48:20Z +mini-swe-agent,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.0,1.0,0,71,128,128,4717,0.5601602730000002,158,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3118,2026-09-10T18:50:40Z,2026-09-10T20:10:49Z +mini-swe-agent,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.894737,1.0,68,76,1412,1412,2996,0.312991898,150,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213615-247091,2026-09-10T21:36:15Z,2026-09-10T22:26:32Z +mini-swe-agent,ink-grid-box-layout,hard,20,typescript,0,0,0.68,1.0,17,25,49,49,3901,0.6463614339999999,216,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222625-142459,2026-09-10T22:26:25Z,2026-09-10T23:31:36Z +mini-swe-agent,awilix-async-container-initialization,hard,21,typescript,0,0,0.958333,1.0,23,24,162,162,1654,0.10591759400000005,88,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3117,2026-09-10T18:50:38Z,2026-09-10T19:19:01Z +mini-swe-agent,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.785714,1.0,11,14,9,9,6652,0.4409767392000001,165,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3119,2026-09-10T18:50:40Z,2026-09-10T20:43:23Z +mini-swe-agent,csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,2785,0.2591460310000001,135,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3138,2026-09-10T18:50:48Z,2026-09-10T19:38:45Z +mini-swe-agent,superjson-error-stack-serialization,hard,24,typescript,0,1,1.0,1.0,80,80,116,116,949,0.11173308900000001,78,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222638-142467,2026-09-10T22:26:38Z,2026-09-10T22:42:41Z +mini-swe-agent,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.75,1.0,9,12,139,139,2939,0.2003528257999999,142,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3118,2026-09-10T18:50:38Z,2026-09-10T19:40:26Z +mini-swe-agent,vulture-persistent-analysis-cache,hard,26,python,0,0,0.958333,1.0,23,24,295,295,2504,0.194642399,87,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224939-203758,2026-09-10T22:49:39Z,2026-09-10T23:31:35Z +mini-swe-agent,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.87234,1.0,82,94,599,599,2660,0.18778814099999994,188,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3139,2026-09-10T18:50:48Z,2026-09-10T19:36:41Z +mini-swe-agent,expr-try-catch-errors,hard,28,go,0,0,0.924051,1.0,73,79,66265,66265,5483,0.6196304858000004,199,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3149,2026-09-10T18:50:38Z,2026-09-10T20:22:26Z +mini-swe-agent,koota-composite-trait-aspects,hard,29,typescript,0,0,0.901961,1.0,46,51,172,172,2666,0.17137808959999992,150,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3119,2026-09-10T18:50:38Z,2026-09-10T19:35:53Z +mini-swe-agent,onedump-dump-encryption-pipeline,hard,30,go,0,1,1.0,1.0,82,82,6,6,1787,0.11192638599999999,70,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224910-203754,2026-09-10T22:49:10Z,2026-09-10T23:19:07Z +mini-swe-agent,eicrud-keyset-pagination-cursor,hard,31,typescript,0,1,1.0,1.0,14,14,168,168,2744,0.15789531780000002,148,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3140,2026-09-10T18:50:48Z,2026-09-10T19:38:04Z +mini-swe-agent,participle-grammar-conflict-analysis,hard,32,go,0,0,0.989011,1.0,90,91,153,153,1625,0.4443119529999999,148,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222517-142464,2026-09-10T22:25:17Z,2026-09-10T22:52:36Z +mini-swe-agent,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,3799,0.675727362,169,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3120,2026-09-10T18:50:38Z,2026-09-10T19:54:43Z +mini-swe-agent,clack-async-autocomplete-options,hard,34,typescript,0,0,0.963415,0.975117,79,82,627,643,2088,0.10724656739999998,114,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3122,2026-09-10T18:50:40Z,2026-09-10T19:26:59Z +mini-swe-agent,langchain-request-coalescing,hard,35,python,0,0,0.9,1.0,45,50,232,232,1984,0.08930158360000004,129,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3141,2026-09-10T18:50:48Z,2026-09-10T19:25:24Z +mini-swe-agent,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,1709,0.083033045,118,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3151,2026-09-10T18:50:38Z,2026-09-10T19:19:38Z +mini-swe-agent,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.944444,1.0,34,36,2034,2034,2163,0.3807282119999999,167,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211755-350829,2026-09-10T21:17:55Z,2026-09-10T21:54:10Z +mini-swe-agent,textual-kitty-key-phases,hard,38,python,0,0,0.478261,1.0,11,23,57,57,960,0.2777421769999999,142,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224924-203755,2026-09-10T22:49:24Z,2026-09-10T23:05:37Z +mini-swe-agent,pwntools-tube-multiplexing,medium,39,python,0,0,0.972603,1.0,71,73,1,1,4130,0.2244443440000001,102,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3142,2026-09-10T18:50:48Z,2026-09-10T20:01:08Z +mini-swe-agent,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.96,1.0,24,25,1679,1679,4542,0.2958691810000001,282,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3152,2026-09-10T18:50:38Z,2026-09-10T20:07:11Z +mini-swe-agent,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.985075,1.0,66,67,2035,2035,2578,0.3613925164,165,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3122,2026-09-10T18:50:38Z,2026-09-10T19:34:14Z +mini-swe-agent,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,2913,0.5685896772000001,218,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3124,2026-09-10T18:50:40Z,2026-09-10T19:40:45Z +mini-swe-agent,bandit-incremental-cache-control,medium,43,python,0,0,0.977273,1.0,86,88,275,275,1203,0.200685787,108,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213653-247088,2026-09-10T21:36:53Z,2026-09-10T21:57:06Z +mini-swe-agent,fastapi-deprecation-response-headers,medium,44,python,0,0,0.985401,1.0,135,137,3134,3134,2335,0.43077675299999996,220,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222627-142458,2026-09-10T22:26:27Z,2026-09-10T23:05:31Z +mini-swe-agent,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,1848,0.08318340559999998,131,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3123,2026-09-10T18:50:38Z,2026-09-10T19:22:15Z +mini-swe-agent,igel-persist-feature-schema,medium,46,python,0,1,1.0,1.0,24,24,2,2,1170,0.06491063840000001,93,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3125,2026-09-10T18:50:40Z,2026-09-10T19:11:40Z +mini-swe-agent,bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,2675,0.310510406,144,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3145,2026-09-10T18:50:48Z,2026-09-10T19:36:55Z +mini-swe-agent,anko-typed-variable-bindings,medium,48,go,0,1,1.0,1.0,9,9,94,94,1472,0.4526873659999999,173,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222653-142456,2026-09-10T22:26:53Z,2026-09-10T22:51:32Z +mini-swe-agent,mashumaro-flattened-dataclass-fields,medium,49,python,0,1,1.0,1.0,66,66,30014,30014,3648,0.4112226242,167,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3124,2026-09-10T18:50:38Z,2026-09-10T19:52:14Z +mini-swe-agent,dasel-html-document-format,medium,50,go,0,1,1.0,1.0,146,146,1012,1012,2113,0.1571617608,152,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3127,2026-09-10T18:50:40Z,2026-09-10T19:27:25Z +mini-swe-agent,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,1.0,0,17,7,7,3194,0.18871511259999987,240,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3147,2026-09-10T18:50:48Z,2026-09-10T19:45:55Z +mini-swe-agent,fastapi-implicit-head-options,medium,52,python,0,1,1.0,1.0,43,43,3134,3134,2705,0.35458784900000007,170,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3156,2026-09-10T18:50:38Z,2026-09-10T19:36:14Z +mini-swe-agent,prometheus-typed-label-sorting,medium,53,go,0,0,0.941176,1.0,16,17,28,28,2392,0.0963558004,147,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3126,2026-09-10T18:50:38Z,2026-09-10T19:31:48Z +mini-swe-agent,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,907,0.12813329999999998,65,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224841-203750,2026-09-10T22:48:41Z,2026-09-10T23:04:01Z +mini-swe-agent,pebble-durability-wait-apis,medium,55,go,0,1,1.0,1.0,59,59,44,44,4227,0.5627047156,239,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3150,2026-09-10T18:50:48Z,2026-09-10T20:02:46Z +mini-swe-agent,cliffy-config-file-parsing,medium,56,typescript,0,1,1.0,1.0,37,37,451,451,1068,0.21384024599999996,118,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222647-142457,2026-09-10T22:26:47Z,2026-09-10T22:44:47Z +mini-swe-agent,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.916667,1.0,55,60,1038,1038,2465,0.31849232600000005,152,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211734-350832,2026-09-10T21:17:34Z,2026-09-10T21:58:46Z +mini-swe-agent,kombu-single-active-consumer-priority,medium,58,python,0,1,1.0,1.0,85,85,1421,1421,2290,0.151390251,88,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185040-3131,2026-09-10T18:50:40Z,2026-09-10T19:30:41Z +mini-swe-agent,skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,1864,0.13562915179999996,173,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185048-3154,2026-09-10T18:50:48Z,2026-09-10T19:23:41Z +mini-swe-agent,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,2109,0.3644871440000001,181,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-185038-3161,2026-09-10T18:50:38Z,2026-09-10T19:26:18Z +mini-swe-agent,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,2795,0.31799011,143,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211732-350826,2026-09-10T21:17:32Z,2026-09-10T22:04:13Z +mini-swe-agent,tengo-destructuring-bindings,medium,62,go,0,0,0.846154,1.0,77,91,132,132,2241,0.5594130030000002,206,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116189,2026-09-10T21:10:39Z,2026-09-10T21:48:29Z +mini-swe-agent,httpx-multipart-response-parsing,medium,63,python,0,0,0.991803,1.0,121,122,1272,1272,1991,0.141082166,89,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80118,2026-09-10T20:02:49Z,2026-09-10T20:36:41Z +mini-swe-agent,opa-rego-rule-profiling,medium,64,go,0,0,0.96,1.0,24,25,6,6,3005,0.4132917230000002,176,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222611-142463,2026-09-10T22:26:11Z,2026-09-10T23:16:26Z +mini-swe-agent,ipython-session-bundle-replay,medium,65,python,0,0,0.764706,1.0,13,17,29,29,2091,0.1189652752,175,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132532,2026-09-10T19:59:41Z,2026-09-10T20:35:22Z +mini-swe-agent,scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,5012,2.6251966189999987,511,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116190,2026-09-10T21:10:39Z,2026-09-10T22:34:41Z +mini-swe-agent,opa-template-string-reconstruction,medium,67,go,0,0,0.8,1.0,4,5,4,4,2437,0.8620873219999999,321,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213648-247093,2026-09-10T21:36:48Z,2026-09-10T22:17:40Z +mini-swe-agent,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,909,0.31332707899999995,138,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222514-142461,2026-09-10T22:25:14Z,2026-09-10T22:40:29Z +mini-swe-agent,tengo-callable-instance-isolation,medium,69,go,0,0,0.956522,1.0,22,23,122,122,1624,0.11202202880000003,113,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132533,2026-09-10T19:59:42Z,2026-09-10T20:27:19Z +mini-swe-agent,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.976744,1.0,42,43,109,109,740,0.32156840400000003,160,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224826-203752,2026-09-10T22:48:26Z,2026-09-10T23:00:57Z +mini-swe-agent,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.985507,1.0,68,69,7,7,3127,0.18425308940000001,132,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80120,2026-09-10T20:02:49Z,2026-09-10T20:55:08Z +mini-swe-agent,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,4399,0.2782464029999999,145,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74043,2026-09-10T20:22:28Z,2026-09-10T21:36:18Z +mini-swe-agent,etree-xml-diff-patch,medium,73,go,0,1,1.0,1.0,52,52,15,15,1815,0.12650455240000003,76,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132534,2026-09-10T19:59:42Z,2026-09-10T20:30:29Z +mini-swe-agent,textual-richlog-follow-state,medium,74,python,0,1,1.0,1.0,20,20,6,6,3038,0.7059042739999999,262,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116192,2026-09-10T21:10:39Z,2026-09-10T22:01:46Z +mini-swe-agent,yaegi-go-embed-directives,medium,75,go,0,1,1.0,1.0,38,38,58,58,4410,0.6543900197999994,295,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80121,2026-09-10T20:02:49Z,2026-09-10T21:16:43Z +mini-swe-agent,kysely-window-grouping-helpers,medium,76,typescript,0,1,1.0,1.0,254,254,22,22,2246,0.34059140299999996,148,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222539-142460,2026-09-10T22:25:39Z,2026-09-10T23:03:13Z +mini-swe-agent,numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,3575,0.2585002662000001,208,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132535,2026-09-10T19:59:42Z,2026-09-10T20:59:51Z +mini-swe-agent,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.2,1.0,2,10,209,209,2848,1.1038042939999995,300,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224859-203757,2026-09-10T22:48:59Z,2026-09-10T23:36:40Z +mini-swe-agent,go-git-worktree-merge-conflicts,easy,79,go,0,1,1.0,1.0,17,17,2,2,4155,0.5881671446,205,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80122,2026-09-10T20:02:49Z,2026-09-10T21:12:45Z +mini-swe-agent,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,3483,0.5442065030000001,204,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74045,2026-09-10T20:22:28Z,2026-09-10T21:21:03Z +mini-swe-agent,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,0.999117,0,33,1132,1133,3105,0.278715672,103,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211714-350827,2026-09-10T21:17:14Z,2026-09-10T22:09:07Z +mini-swe-agent,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,1220,0.12952470299999996,87,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224933-203756,2026-09-10T22:49:33Z,2026-09-10T23:10:04Z +mini-swe-agent,anko-default-function-arguments,easy,83,go,0,0,0.5,1.0,1,2,119,119,4405,1.2974186990000005,310,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80123,2026-09-10T20:02:49Z,2026-09-10T21:16:54Z +mini-swe-agent,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,6374,1.1699734860000008,297,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74046,2026-09-10T20:22:28Z,2026-09-10T22:09:14Z +mini-swe-agent,abs-stepped-slices,easy,85,go,0,0,1.0,0.833333,6,6,5,6,1259,0.07147781680000001,113,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132537,2026-09-10T19:59:42Z,2026-09-10T20:21:15Z +mini-swe-agent,testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,1819,0.154301516,91,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116195,2026-09-10T21:10:39Z,2026-09-10T21:41:27Z +mini-swe-agent,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,3309,0.6102901864000001,241,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80124,2026-09-10T20:02:49Z,2026-09-10T20:59:23Z +mini-swe-agent,wasmi-trap-coredumps,easy,88,rust,0,1,1.0,1.0,22,22,58,58,4821,1.4018095141999998,359,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74047,2026-09-10T20:22:28Z,2026-09-10T21:43:35Z +mini-swe-agent,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1835,0.2501200240000001,120,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211711-350825,2026-09-10T21:17:11Z,2026-09-10T21:47:57Z +mini-swe-agent,geo-shapeindex-serialization,easy,90,go,0,0,0.916667,1.0,22,24,599,599,2149,0.232493381,123,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-224914-203753,2026-09-10T22:49:15Z,2026-09-10T23:25:10Z +mini-swe-agent,arcane-drift-detection-baselines,easy,91,go,0,1,1.0,1.0,82,82,2,2,2030,0.3679867619999999,166,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213755-247087,2026-09-10T21:37:55Z,2026-09-10T22:12:02Z +mini-swe-agent,abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,1898,0.32773373799999994,184,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74048,2026-09-10T20:22:28Z,2026-09-10T20:54:38Z +mini-swe-agent,aiomonitor-task-snapshots-diff,easy,93,python,0,1,1.0,1.0,53,53,8,8,2479,0.18091542640000002,100,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132539,2026-09-10T19:59:42Z,2026-09-10T20:41:34Z +mini-swe-agent,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,3623,0.23581243100000004,98,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116197,2026-09-10T21:10:39Z,2026-09-10T22:11:31Z +mini-swe-agent,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,1409,0.427370877,201,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213606-247094,2026-09-10T21:36:06Z,2026-09-10T21:59:47Z +mini-swe-agent,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,0,0.677419,1.0,21,31,1267,1267,3841,0.7828126020000004,243,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74049,2026-09-10T20:22:28Z,2026-09-10T21:27:01Z +mini-swe-agent,task-task-graph-export,easy,97,go,0,1,1.0,1.0,20,20,17,17,1695,0.10804480320000001,140,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132540,2026-09-10T19:59:42Z,2026-09-10T20:28:45Z +mini-swe-agent,go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,3258,0.5645121419999999,222,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116198,2026-09-10T21:10:39Z,2026-09-10T22:05:26Z +mini-swe-agent,ytt-jsonpath-query-api,easy,99,go,0,0,0.961165,1.0,99,103,1,1,1429,0.22587837699999996,137,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213805-247095,2026-09-10T21:38:05Z,2026-09-10T22:02:00Z +mini-swe-agent,helm-unified-manifest-stream,easy,100,go,1,1,1.0,1.0,5,5,2,2,3685,0.43080401780000016,242,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-202228-74050,2026-09-10T20:22:28Z,2026-09-10T21:24:26Z +mini-swe-agent,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,3277,0.3760222479999999,181,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211837-350833,2026-09-10T21:18:37Z,2026-09-10T22:13:23Z +mini-swe-agent,goreleaser-retry-publish-auditing,easy,102,go,0,0,0.862069,1.0,25,29,29,29,1817,0.4948836939999998,214,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116199,2026-09-10T21:10:39Z,2026-09-10T21:42:51Z +mini-swe-agent,psd-tools-blend-range-api,easy,103,python,0,0,0.755556,1.0,34,45,979,979,2288,0.15320772500000002,104,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80129,2026-09-10T20:02:49Z,2026-09-10T20:41:36Z +mini-swe-agent,wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,2328,0.5365283079999998,202,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222657-142469,2026-09-10T22:26:57Z,2026-09-10T23:05:58Z +mini-swe-agent,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,2363,0.249755533,119,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211747-350831,2026-09-10T21:17:47Z,2026-09-10T21:57:24Z +mini-swe-agent,actionlint-action-pinning-lint,easy,106,go,0,1,1.0,1.0,55,55,145,145,1391,0.329211295,145,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211039-116200,2026-09-10T21:10:39Z,2026-09-10T21:34:20Z +mini-swe-agent,httpx-deterministic-cookie-store,easy,107,typescript,0,1,1.0,1.0,115,115,1281,1281,2230,0.13737260399999998,89,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-213654-247090,2026-09-10T21:36:54Z,2026-09-10T22:14:13Z +mini-swe-agent,returns-validated-error-accumulation,easy,108,python,0,1,1.0,1.0,159,159,61,61,1012,0.24751526199999999,110,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222542-142465,2026-09-10T22:25:42Z,2026-09-10T22:42:40Z +mini-swe-agent,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,1639,0.15941252299999997,98,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-211701-350828,2026-09-10T21:17:01Z,2026-09-10T21:44:28Z +mini-swe-agent,drizzle-orm-window-function-builders,easy,110,typescript,0,1,1.0,1.0,130,130,566,566,1475,0.323383619,159,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-225022-203751,2026-09-10T22:50:22Z,2026-09-10T23:15:08Z +mini-swe-agent,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,2842,0.5419176789999998,201,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-200249-80135,2026-09-10T20:02:49Z,2026-09-10T20:50:52Z +mini-swe-agent,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,2023,0.7046330470000003,237,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-222510-142462,2026-09-10T22:25:10Z,2026-09-10T22:59:03Z +mini-swe-agent,true-myth-iterable-collection-combinators,easy,113,typescript,1,1,1.0,1.0,96,96,561,561,1595,0.11129807259999994,116,0,,openrouter/deepseek/deepseek-v4-flash-0731,2.4.6,20260910-195941-132545,2026-09-10T19:59:42Z,2026-09-10T20:26:50Z diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/muse-code.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/muse-code.csv new file mode 100644 index 0000000000..0c176ec3da --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/muse-code.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +muse-code,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,686,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-10T18:45:08+00:00,2026-09-10T18:56:34+00:00 +muse-code,gql-incremental-graphql-delivery,hard,2,python,0,0,0.0,1.0,0,17,811,811,384,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:00+00:00,2026-09-11T20:10:24+00:00 +muse-code,bandit-structured-nosec-directives,hard,3,python,0,0,0.0,1.0,0,69,282,282,344,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:13+00:00,2026-09-11T20:09:58+00:00 +muse-code,termenv-preserve-ansi-resets,hard,4,go,0,0,0.0,1.0,0,35,87,87,293,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:03:41+00:00,2026-09-11T20:08:35+00:00 +muse-code,updo-policy-alerting,hard,5,go,0,0,0.0,1.0,0,17,123,123,297,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:04:17+00:00 +muse-code,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.0,1.0,0,47,70,70,300,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:08:12+00:00 +muse-code,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,1.0,0,6,62,62,280,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:07:39+00:00 +muse-code,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.0,1.0,0,32,1273,1273,281,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:16+00:00,2026-09-11T19:06:59+00:00 +muse-code,helm-array-merge-strategies,hard,9,go,0,0,0.0,1.0,0,47,12,12,1634,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:26:34+00:00 +muse-code,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.0,1.0,0,49,51469,51469,2121,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:38:33+00:00 +muse-code,pest-character-class-coalescing,hard,11,rust,0,0,0.0,1.0,0,104,250,250,286,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:07:45+00:00 +muse-code,testem-bail-on-test-failure,hard,12,javascript,0,0,0.0,1.0,0,90,489,489,1610,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:16+00:00,2026-09-11T19:29:08+00:00 +muse-code,koota-pair-relation-tracking,hard,13,typescript,0,0,0.0,1.0,0,38,172,172,353,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:05:12+00:00 +muse-code,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.878049,0,15,72,82,1360,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:12+00:00,2026-09-11T19:25:53+00:00 +muse-code,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.0,1.0,0,60,1131,1131,792,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:03:53+00:00,2026-09-11T20:17:07+00:00 +muse-code,koota-query-predicates,hard,16,typescript,0,0,0.0,1.0,0,43,172,172,284,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:19+00:00,2026-09-11T19:07:05+00:00 +muse-code,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.0,1.0,0,13,22,22,2869,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:05:27+00:00,2026-09-11T20:53:16+00:00 +muse-code,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.0,1.0,0,71,128,128,448,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:10:39+00:00 +muse-code,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.0,1.0,0,76,1412,1412,1494,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:27:53+00:00 +muse-code,ink-grid-box-layout,hard,20,typescript,0,0,0.0,1.0,0,25,49,49,291,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:16+00:00,2026-09-11T19:07:10+00:00 +muse-code,awilix-async-container-initialization,hard,21,typescript,0,0,0.0,1.0,0,24,162,162,3060,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:17+00:00,2026-09-11T19:50:18+00:00 +muse-code,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.0,1.0,0,14,9,9,644,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:36+00:00,2026-09-11T20:15:21+00:00 +muse-code,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.0,1.0,0,79,16715,16715,289,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:58+00:00,2026-09-11T19:07:49+00:00 +muse-code,superjson-error-stack-serialization,hard,24,typescript,0,0,0.0,1.0,0,80,116,116,360,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:18+00:00,2026-09-11T19:08:19+00:00 +muse-code,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.0,1.0,0,12,139,139,1153,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:18:34+00:00 +muse-code,vulture-persistent-analysis-cache,hard,26,python,0,0,0.0,1.0,0,24,295,295,333,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:03:57+00:00,2026-09-11T20:09:30+00:00 +muse-code,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.0,1.0,0,94,599,599,765,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:05:13+00:00,2026-09-11T20:17:58+00:00 +muse-code,expr-try-catch-errors,hard,28,go,0,0,0.0,1.0,0,79,66265,66265,291,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:16+00:00,2026-09-11T19:07:10+00:00 +muse-code,koota-composite-trait-aspects,hard,29,typescript,0,0,0.0,1.0,0,51,172,172,300,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:04:21+00:00 +muse-code,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.890244,1.0,73,82,6,6,2167,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:39:19+00:00 +muse-code,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.0,1.0,0,14,168,168,295,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:07:54+00:00 +muse-code,participle-grammar-conflict-analysis,hard,32,go,0,0,0.0,1.0,0,91,153,153,408,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:17+00:00,2026-09-11T19:09:08+00:00 +muse-code,python-statemachine-state-data-scoping,hard,33,python,0,0,0.0,1.0,0,72,1286,1286,282,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:17+00:00,2026-09-11T19:04:02+00:00 +muse-code,clack-async-autocomplete-options,hard,34,typescript,0,0,0.0,1.0,0,82,643,643,311,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:08:22+00:00 +muse-code,langchain-request-coalescing,hard,35,python,0,0,0.0,1.0,0,50,232,232,1366,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:25:46+00:00 +muse-code,httpx-streaming-json-iteration,hard,36,python,0,0,0.0,1.0,0,108,1404,1404,1081,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:39+00:00,2026-09-11T20:22:42+00:00 +muse-code,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.0,1.0,0,36,2034,2034,323,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:38+00:00,2026-09-11T20:10:01+00:00 +muse-code,textual-kitty-key-phases,hard,38,python,0,0,0.0,1.0,0,23,57,57,707,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:14+00:00,2026-09-11T20:16:03+00:00 +muse-code,pwntools-tube-multiplexing,medium,39,python,0,0,0.0,1.0,0,73,1,1,295,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:03:55+00:00,2026-09-11T20:08:51+00:00 +muse-code,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.0,1.0,0,25,1679,1679,318,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:25+00:00,2026-09-11T20:09:43+00:00 +muse-code,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.0,1.0,0,67,2035,2035,320,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:59:18+00:00,2026-09-11T19:04:40+00:00 +muse-code,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,745,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:15:37+00:00 +muse-code,bandit-incremental-cache-control,medium,43,python,0,0,0.0,1.0,0,88,275,275,1183,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:57+00:00,2026-09-11T19:22:42+00:00 +muse-code,fastapi-deprecation-response-headers,medium,44,python,0,0,0.328467,0.999681,45,137,3133,3134,2779,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:24+00:00,2026-09-11T20:50:45+00:00 +muse-code,go-critic-doc-link-checker,medium,45,go,0,0,0.0,0.0,0,3,0,16,1925,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:36+00:00,2026-09-11T20:36:42+00:00 +muse-code,igel-persist-feature-schema,medium,46,python,0,0,0.0,1.0,0,24,2,2,346,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:28+00:00,2026-09-11T20:10:16+00:00 +muse-code,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.0,1.0,0,66,293,293,374,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:15+00:00,2026-09-11T20:10:30+00:00 +muse-code,anko-typed-variable-bindings,medium,48,go,0,0,0.0,1.0,0,9,94,94,295,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:18+00:00,2026-09-11T19:07:15+00:00 +muse-code,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.0,1.0,0,66,30014,30014,314,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:36+00:00,2026-09-11T20:09:51+00:00 +muse-code,dasel-html-document-format,medium,50,go,0,0,0.0,1.0,0,146,1012,1012,1533,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:10+00:00,2026-09-11T19:28:45+00:00 +muse-code,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,1.0,0,17,7,7,1627,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:19+00:00,2026-09-11T19:30:27+00:00 +muse-code,fastapi-implicit-head-options,medium,52,python,0,0,0.0,1.0,0,43,3134,3134,301,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:17+00:00,2026-09-11T19:07:20+00:00 +muse-code,prometheus-typed-label-sorting,medium,53,go,0,0,0.0,1.0,0,17,28,28,1022,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:35+00:00,2026-09-11T20:21:39+00:00 +muse-code,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,0,0.0,1.0,0,7,31,31,307,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:03:57+00:00,2026-09-11T20:09:04+00:00 +muse-code,pebble-durability-wait-apis,medium,55,go,0,0,0.0,1.0,0,59,44,44,322,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:56+00:00,2026-09-11T19:08:20+00:00 +muse-code,cliffy-config-file-parsing,medium,56,typescript,0,0,0.0,1.0,0,37,451,451,301,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:02:17+00:00,2026-09-11T19:07:20+00:00 +muse-code,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.0,1.0,0,60,1038,1038,322,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T18:58:54+00:00,2026-09-11T19:04:17+00:00 +muse-code,kombu-single-active-consumer-priority,medium,58,python,0,0,0.0,1.0,0,85,1421,1421,294,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T19:03:11+00:00,2026-09-11T19:08:06+00:00 +muse-code,skrub-duration-encoding,medium,59,python,0,0,0.0,1.0,0,130,2784,2784,325,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:54+00:00,2026-09-11T20:10:21+00:00 +muse-code,scc-bounded-memory-spilling,medium,60,go,0,0,0.0,1.0,0,31,286,286,306,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:04:41+00:00,2026-09-11T20:09:48+00:00 +muse-code,mnamer-daemon-watch-lifecycle,medium,61,python,0,0,0.0,1.0,0,51,319,319,305,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:30+00:00,2026-09-11T20:59:36+00:00 +muse-code,tengo-destructuring-bindings,medium,62,go,0,0,0.0,1.0,0,91,132,132,1002,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:16:25+00:00,2026-09-11T20:33:07+00:00 +muse-code,httpx-multipart-response-parsing,medium,63,python,0,0,0.0,1.0,0,122,1272,1272,328,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:36+00:00,2026-09-11T20:25:05+00:00 +muse-code,opa-rego-rule-profiling,medium,64,go,0,0,0.0,1.0,0,25,6,6,304,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:51+00:00,2026-09-11T20:58:56+00:00 +muse-code,ipython-session-bundle-replay,medium,65,python,0,0,0.117647,1.0,2,17,29,29,1400,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:55:04+00:00,2026-09-11T21:18:25+00:00 +muse-code,scriggo-method-declarations,medium,66,go,0,0,0.0,1.0,0,48,1049,1049,972,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:09+00:00,2026-09-11T20:35:23+00:00 +muse-code,opa-template-string-reconstruction,medium,67,go,0,0,0.0,1.0,0,5,4,4,335,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:24:50+00:00,2026-09-11T20:30:27+00:00 +muse-code,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,5733,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:52:40+00:00,2026-09-11T22:28:15+00:00 +muse-code,tengo-callable-instance-isolation,medium,69,go,0,0,0.0,1.0,0,23,122,122,1471,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:47+00:00,2026-09-11T21:18:19+00:00 +muse-code,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,1.0,0,43,109,109,346,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:35+00:00,2026-09-11T20:25:25+00:00 +muse-code,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.0,1.0,0,69,7,7,1731,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:31+00:00,2026-09-11T20:48:23+00:00 +muse-code,tomlkit-toml-table-converters,medium,72,python,0,0,0.0,1.0,0,60,964,964,296,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:52:46+00:00,2026-09-11T20:57:45+00:00 +muse-code,etree-xml-diff-patch,medium,73,go,0,0,0.0,1.0,0,52,15,15,324,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:49+00:00,2026-09-11T21:00:13+00:00 +muse-code,textual-richlog-follow-state,medium,74,python,0,0,0.0,1.0,0,20,6,6,954,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:16+00:00,2026-09-11T20:35:11+00:00 +muse-code,yaegi-go-embed-directives,medium,75,go,0,0,0.0,1.0,0,38,58,58,363,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:35+00:00,2026-09-11T20:25:39+00:00 +muse-code,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.0,1.0,0,254,22,22,2283,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:40+00:00,2026-09-11T21:32:44+00:00 +muse-code,numba-stencil-boundary-modes,easy,77,python,0,0,0.0,1.0,0,29,827,827,416,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:44+00:00,2026-09-11T21:01:42+00:00 +muse-code,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.0,1.0,0,10,209,209,380,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:58+00:00,2026-09-11T20:26:20+00:00 +muse-code,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,1.0,0,17,2,2,1608,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:24:10+00:00,2026-09-11T20:51:00+00:00 +muse-code,adaptix-name-mapping-aliases,easy,80,python,1,0,0.0,1.0,0,44,2738,2738,699,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:16+00:00,2026-09-11T21:04:56+00:00 +muse-code,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,1.0,0,33,1133,1133,326,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:46+00:00,2026-09-11T21:00:14+00:00 +muse-code,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,365,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:16:23+00:00,2026-09-11T20:22:30+00:00 +muse-code,anko-default-function-arguments,easy,83,go,0,0,0.0,1.0,0,2,119,119,818,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:30+00:00,2026-09-11T20:33:10+00:00 +muse-code,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.243243,1.0,9,37,1267,1267,1848,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:55:08+00:00,2026-09-11T21:25:58+00:00 +muse-code,abs-stepped-slices,easy,85,go,0,0,0.0,1.0,0,6,6,6,2229,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:04+00:00,2026-09-11T21:31:15+00:00 +muse-code,testem-per-launcher-reports,easy,86,javascript,0,0,0.276923,1.0,18,65,469,469,1276,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:16:17+00:00,2026-09-11T20:37:35+00:00 +muse-code,query-persist-restored-query-state,easy,87,typescript,0,0,0.0,1.0,0,8,42,42,344,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:23:58+00:00,2026-09-11T20:29:46+00:00 +muse-code,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,1.0,0,22,58,58,1469,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:14+00:00,2026-09-11T21:17:44+00:00 +muse-code,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,3338,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:29+00:00,2026-09-11T21:50:08+00:00 +muse-code,geo-shapeindex-serialization,easy,90,go,0,0,0.0,1.0,0,24,599,599,1249,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:10+00:00,2026-09-11T20:40:01+00:00 +muse-code,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,1.0,0,82,2,2,324,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:22:31+00:00,2026-09-11T20:27:55+00:00 +muse-code,abs-module-cache-flags,easy,92,go,0,0,0.0,1.0,0,20,3,3,587,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:52:09+00:00,2026-09-11T21:01:56+00:00 +muse-code,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.0,1.0,0,53,8,8,294,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:18+00:00,2026-09-11T20:59:13+00:00 +muse-code,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,2637,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:12+00:00,2026-09-11T21:03:09+00:00 +muse-code,vitest-duration-sharding,easy,95,typescript,0,0,0.0,1.0,0,56,24,24,319,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:20:08+00:00,2026-09-11T20:25:28+00:00 +muse-code,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,0,0.0,1.0,0,31,1267,1267,320,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:55:08+00:00,2026-09-11T21:00:29+00:00 +muse-code,task-task-graph-export,easy,97,go,0,0,0.0,1.0,0,20,17,17,333,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:55:36+00:00,2026-09-11T21:01:10+00:00 +muse-code,go-genai-streamed-function-args,easy,98,go,0,0,0.0,1.0,0,6,62,62,309,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:09+00:00,2026-09-11T20:24:20+00:00 +muse-code,ytt-jsonpath-query-api,easy,99,go,0,0,0.951456,1.0,98,103,1,1,2373,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:23:59+00:00,2026-09-11T21:03:33+00:00 +muse-code,helm-unified-manifest-stream,easy,100,go,1,0,0.0,1.0,0,5,2,2,739,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:15+00:00,2026-09-11T21:06:35+00:00 +muse-code,yjs-map-conflict-detection,easy,101,javascript,0,0,0.0,1.0,0,9,231,231,380,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:11+00:00,2026-09-11T21:00:32+00:00 +muse-code,goreleaser-retry-publish-auditing,easy,102,go,0,0,0.0,1.0,0,29,29,29,323,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:09+00:00,2026-09-11T20:24:33+00:00 +muse-code,psd-tools-blend-range-api,easy,103,python,0,0,0.0,1.0,0,45,979,979,326,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:20:18+00:00,2026-09-11T20:25:45+00:00 +muse-code,wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,4025,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:14+00:00,2026-09-11T22:00:20+00:00 +muse-code,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.0,1.0,0,26,5709,5709,323,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:53:46+00:00,2026-09-11T20:59:09+00:00 +muse-code,actionlint-action-pinning-lint,easy,106,go,0,0,0.0,1.0,0,55,145,145,2228,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:09+00:00,2026-09-11T20:56:19+00:00 +muse-code,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.0,1.0,0,115,1281,1281,425,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:17+00:00,2026-09-11T20:26:24+00:00 +muse-code,returns-validated-error-accumulation,easy,108,python,0,0,0.0,1.0,0,159,61,61,340,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:40+00:00,2026-09-11T21:00:21+00:00 +muse-code,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,0,0.0,1.0,0,47,13,13,1183,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:57+00:00,2026-09-11T21:14:40+00:00 +muse-code,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.0,1.0,0,130,566,566,6341,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:20:47+00:00,2026-09-11T22:06:29+00:00 +muse-code,happy-dom-abort-pending-body-reads,easy,111,typescript,0,0,0.0,1.0,0,14,165,165,1448,,,0,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:19:30+00:00,2026-09-11T20:43:39+00:00 +muse-code,narwhals-rolling-window-suite,easy,112,python,0,0,0.0,1.0,0,103,10093,10093,382,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:52:40+00:00,2026-09-11T20:59:04+00:00 +muse-code,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.0,1.0,0,96,561,561,1021,,,1,0,deepseek/deepseek-v4-flash-0731,1.1.1-R2514.1,verifier,2026-09-11T20:54:00+00:00,2026-09-11T21:11:02+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/omp.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/omp.csv new file mode 100644 index 0000000000..e1e2e4e911 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/omp.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +omp,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.926829,1.0,38,41,1131,1131,3482,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:27+00:00,2026-09-11T01:37:34+00:00 +omp,gql-incremental-graphql-delivery,hard,2,python,0,0,0.176471,1.0,3,17,811,811,1178,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:33+00:00,2026-09-11T01:03:05+00:00 +omp,bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,1120,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:36+00:00,2026-09-11T00:58:33+00:00 +omp,termenv-preserve-ansi-resets,hard,4,go,0,0,0.942857,1.0,33,35,87,87,2806,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:16+00:00,2026-09-11T01:27:17+00:00 +omp,updo-policy-alerting,hard,5,go,0,0,0.882353,1.0,15,17,123,123,685,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:37+00:00,2026-09-11T00:47:32+00:00 +omp,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.0,1.0,0,47,70,70,5947,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:29+00:00,2026-09-11T02:47:47+00:00 +omp,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,0.0,0,6,0,62,2207,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:06+00:00,2026-09-11T01:28:39+00:00 +omp,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.9375,0.997643,30,32,1270,1273,3156,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:06+00:00,2026-09-11T01:47:24+00:00 +omp,helm-array-merge-strategies,hard,9,go,0,0,0.468085,0.583333,22,47,7,12,2257,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:47+00:00,2026-09-11T01:17:35+00:00 +omp,meriyah-explicit-resource-declarations,hard,10,typescript,0,1,1.0,1.0,49,49,51469,51469,2916,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:11+00:00,2026-09-11T01:43:49+00:00 +omp,pest-character-class-coalescing,hard,11,rust,0,0,0.0,0.0,0,104,0,250,851,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:52+00:00,2026-09-11T00:53:52+00:00 +omp,testem-bail-on-test-failure,hard,12,javascript,0,0,0.866667,1.0,78,90,489,489,3101,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:15+00:00,2026-09-11T01:30:36+00:00 +omp,koota-pair-relation-tracking,hard,13,typescript,0,0,0.973684,1.0,37,38,172,172,2326,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:47+00:00,2026-09-11T01:26:17+00:00 +omp,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.0,0,15,0,82,2775,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:31:28+00:00,2026-09-11T01:33:32+00:00 +omp,obsidian-linter-link-format-conversion,hard,15,typescript,0,1,1.0,1.0,60,60,1131,1131,1720,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:14+00:00,2026-09-11T01:04:39+00:00 +omp,koota-query-predicates,hard,16,typescript,0,0,0.953488,1.0,41,43,172,172,2603,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:31+00:00,2026-09-11T01:35:40+00:00 +omp,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,3065,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:37+00:00,2026-09-11T01:38:35+00:00 +omp,koota-deferred-mutation-buffer,hard,18,typescript,0,1,1.0,1.0,71,71,128,128,2980,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:31:45+00:00,2026-09-11T01:58:37+00:00 +omp,kombu-virtual-queue-dead-lettering,hard,19,python,0,,,,,,,,,,,,,,,,, +omp,ink-grid-box-layout,hard,20,typescript,0,0,0.92,1.0,23,25,49,49,2349,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T05:53:59+00:00,2026-09-11T06:51:14+00:00 +omp,awilix-async-container-initialization,hard,21,typescript,0,0,0.958333,1.0,23,24,162,162,1122,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:32+00:00,2026-09-11T00:59:10+00:00 +omp,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,1,1.0,1.0,14,14,9,9,1785,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:00+00:00,2026-09-11T01:13:54+00:00 +omp,csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,1117,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:15+00:00,2026-09-11T00:58:29+00:00 +omp,superjson-error-stack-serialization,hard,24,typescript,0,0,0.925,1.0,74,80,116,116,999,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:23+00:00,2026-09-11T00:54:13+00:00 +omp,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.916667,1.0,11,12,139,139,4073,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:53+00:00,2026-09-11T02:06:14+00:00 +omp,vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,2091,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:30+00:00,2026-09-11T01:16:00+00:00 +omp,katex-multicolumn-array-spans,hard,27,javascript,0,1,1.0,1.0,94,94,599,599,5449,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:36+00:00,2026-09-11T02:27:44+00:00 +omp,expr-try-catch-errors,hard,28,go,0,0,0.987342,1.0,78,79,66265,66265,2979,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:22+00:00,2026-09-11T01:39:53+00:00 +omp,koota-composite-trait-aspects,hard,29,typescript,0,0,0.960784,1.0,49,51,172,172,2510,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:45+00:00,2026-09-11T01:33:35+00:00 +omp,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.146341,1.0,12,82,6,6,2385,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:29:25+00:00,2026-09-11T01:26:42+00:00 +omp,eicrud-keyset-pagination-cursor,hard,31,typescript,0,1,1.0,1.0,14,14,168,168,3236,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:36+00:00,2026-09-11T01:35:54+00:00 +omp,participle-grammar-conflict-analysis,hard,32,go,0,0,0.989011,1.0,90,91,153,153,3223,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:32+00:00,2026-09-11T01:40:50+00:00 +omp,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,3879,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:05+00:00,2026-09-11T01:49:49+00:00 +omp,clack-async-autocomplete-options,hard,34,typescript,0,0,0.853659,1.0,70,82,643,643,2090,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:31:28+00:00,2026-09-11T01:26:08+00:00 +omp,langchain-request-coalescing,hard,35,python,0,0,0.94,1.0,47,50,232,232,2589,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:01+00:00,2026-09-11T01:29:26+00:00 +omp,httpx-streaming-json-iteration,hard,36,python,0,0,0.712963,1.0,77,108,1404,1404,975,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:06+00:00,2026-09-11T00:46:43+00:00 +omp,optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,6394,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:16+00:00,2026-09-11T02:28:09+00:00 +omp,textual-kitty-key-phases,hard,38,python,0,0,0.608696,1.0,14,23,57,57,4420,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:56+00:00,2026-09-11T01:59:29+00:00 +omp,pwntools-tube-multiplexing,medium,39,python,0,1,1.0,1.0,73,73,1,1,2840,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:43+00:00,2026-09-11T01:30:11+00:00 +omp,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.92,1.0,23,25,1679,1679,7004,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:08+00:00,2026-09-11T02:51:54+00:00 +omp,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.970149,1.0,65,67,2035,2035,1579,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:25:52+00:00,2026-09-11T01:08:15+00:00 +omp,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,0.0,0,2,0,214,1251,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:53+00:00,2026-09-11T01:02:02+00:00 +omp,bandit-incremental-cache-control,medium,43,python,0,0,0.943182,1.0,83,88,275,275,1493,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:29:18+00:00,2026-09-11T01:02:19+00:00 +omp,fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,1739,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:36+00:00,2026-09-11T01:05:19+00:00 +omp,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,3010,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:50+00:00,2026-09-11T01:33:22+00:00 +omp,igel-persist-feature-schema,medium,46,python,0,1,1.0,1.0,24,24,2,2,1047,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:42+00:00,2026-09-11T00:55:36+00:00 +omp,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.212121,1.0,14,66,293,293,2361,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:45+00:00,2026-09-11T01:18:02+00:00 +omp,anko-typed-variable-bindings,medium,48,go,0,0,0.555556,1.0,5,9,94,94,6129,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:09+00:00,2026-09-11T02:54:06+00:00 +omp,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.69697,0.999933,46,66,30012,30014,3117,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:06+00:00,2026-09-11T01:36:36+00:00 +omp,dasel-html-document-format,medium,50,go,0,0,0.986301,1.0,144,146,1012,1012,2109,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:11+00:00,2026-09-11T01:15:31+00:00 +omp,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,0.0,0,17,0,7,5720,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:29:18+00:00,2026-09-11T02:26:06+00:00 +omp,fastapi-implicit-head-options,medium,52,python,0,1,1.0,1.0,43,43,3134,3134,2644,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:40+00:00,2026-09-11T01:25:42+00:00 +omp,prometheus-typed-label-sorting,medium,53,go,0,0,0.0,0.0,0,17,0,28,2265,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:47+00:00,2026-09-11T01:26:04+00:00 +omp,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,1699,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:29:16+00:00,2026-09-11T01:08:35+00:00 +omp,pebble-durability-wait-apis,medium,55,go,0,0,0.0,0.0,0,59,0,44,2626,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:28:08+00:00,2026-09-11T01:32:23+00:00 +omp,cliffy-config-file-parsing,medium,56,typescript,0,0,0.864865,1.0,32,37,451,451,2141,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:06+00:00,2026-09-11T01:13:48+00:00 +omp,sqlite-utils-safe-import-checkpoints,medium,57,python,1,0,0.95,1.0,57,60,1038,1038,2344,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:26:54+00:00,2026-09-11T01:22:57+00:00 +omp,kombu-single-active-consumer-priority,medium,58,python,0,0,0.988235,1.0,84,85,1421,1421,1455,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:31:51+00:00,2026-09-11T01:11:51+00:00 +omp,skrub-duration-encoding,medium,59,python,0,0,0.976923,1.0,127,130,2784,2784,2906,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:29:41+00:00,2026-09-11T01:30:47+00:00 +omp,scc-bounded-memory-spilling,medium,60,go,0,0,0.83871,1.0,26,31,286,286,1737,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T00:27:40+00:00,2026-09-11T01:04:35+00:00 +omp,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,1354,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:01+00:00,2026-09-11T03:08:22+00:00 +omp,tengo-destructuring-bindings,medium,62,go,0,0,0.967033,1.0,88,91,132,132,4849,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:18+00:00,2026-09-11T05:05:12+00:00 +omp,httpx-multipart-response-parsing,medium,63,python,0,0,0.991803,1.0,121,122,1272,1272,4304,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:14:18+00:00,2026-09-11T04:48:35+00:00 +omp,opa-rego-rule-profiling,medium,64,go,0,0,0.0,1.0,0,25,6,6,3523,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:09:51+00:00,2026-09-11T04:31:09+00:00 +omp,ipython-session-bundle-replay,medium,65,python,0,0,0.941176,1.0,16,17,29,29,2008,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:37+00:00,2026-09-11T03:18:02+00:00 +omp,scriggo-method-declarations,medium,66,go,0,0,0.895833,1.0,43,48,1049,1049,9847,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:26+00:00,2026-09-11T06:55:23+00:00 +omp,opa-template-string-reconstruction,medium,67,go,0,0,0.0,0.0,0,5,0,4,2878,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:14:06+00:00,2026-09-11T04:26:54+00:00 +omp,mobly-grouped-test-barriers,medium,68,python,0,0,0.924051,1.0,73,79,808,808,1982,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:08:32+00:00,2026-09-11T03:56:58+00:00 +omp,tengo-callable-instance-isolation,medium,69,go,0,0,0.956522,1.0,22,23,122,122,1245,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:46+00:00,2026-09-11T03:14:58+00:00 +omp,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,0.0,0,43,0,109,2619,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:16+00:00,2026-09-11T04:09:55+00:00 +omp,cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,2262,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:14:06+00:00,2026-09-11T04:05:19+00:00 +omp,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,6748,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:23+00:00,2026-09-11T05:39:38+00:00 +omp,etree-xml-diff-patch,medium,73,go,0,0,0.980769,1.0,51,52,15,15,3025,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:36:44+00:00,2026-09-11T03:52:39+00:00 +omp,textual-richlog-follow-state,medium,74,python,0,0,0.7,1.0,14,20,6,6,3570,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:03:21+00:00,2026-09-11T04:23:40+00:00 +omp,yaegi-go-embed-directives,medium,75,go,0,1,1.0,1.0,38,38,58,58,4432,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:14+00:00,2026-09-11T04:52:00+00:00 +omp,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.0,0.0,0,254,0,22,1098,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:09:25+00:00,2026-09-11T03:40:26+00:00 +omp,numba-stencil-boundary-modes,easy,77,python,0,0,0.793103,1.0,23,29,827,827,2274,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:00+00:00,2026-09-11T03:27:43+00:00 +omp,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.9,1.0,9,10,209,209,6295,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:47+00:00,2026-09-11T05:43:29+00:00 +omp,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,0.0,0,17,0,2,4685,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:14:29+00:00,2026-09-11T05:02:32+00:00 +omp,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,3034,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:43+00:00,2026-09-11T04:18:40+00:00 +omp,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,0.999117,0,33,1132,1133,4674,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:36:46+00:00,2026-09-11T04:18:38+00:00 +omp,ts-pattern-match-each,easy,82,typescript,0,1,1.0,1.0,85,85,6,6,1045,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:49+00:00,2026-09-11T03:28:47+00:00 +omp,anko-default-function-arguments,easy,83,go,0,0,0.0,1.0,0,2,119,119,10803,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:14:19+00:00,2026-09-11T07:02:28+00:00 +omp,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,4602,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:08:32+00:00,2026-09-11T04:43:18+00:00 +omp,abs-stepped-slices,easy,85,go,0,0,0.833333,0.833333,5,6,5,6,1799,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:38:05+00:00,2026-09-11T03:18:55+00:00 +omp,testem-per-launcher-reports,easy,86,javascript,0,0,0.969231,1.0,63,65,469,469,989,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:02:59+00:00,2026-09-11T03:28:18+00:00 +omp,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2089,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:15:44+00:00,2026-09-11T04:01:51+00:00 +omp,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,0.0,0,22,0,58,5161,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:08:55+00:00,2026-09-11T05:04:30+00:00 +omp,koota-entity-snapshot-rollback,easy,89,python,0,0,0.964286,1.0,81,84,47,47,2822,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:38:00+00:00,2026-09-11T03:39:02+00:00 +omp,geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,4976,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:03:24+00:00,2026-09-11T05:05:21+00:00 +omp,arcane-drift-detection-baselines,easy,91,go,0,0,0.817073,1.0,67,82,2,2,2033,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:15+00:00,2026-09-11T04:05:57+00:00 +omp,abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,1598,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:09:52+00:00,2026-09-11T03:50:50+00:00 +omp,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.943396,1.0,50,53,8,8,1704,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:45+00:00,2026-09-11T03:13:21+00:00 +omp,kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,4420,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:05:08+00:00,2026-09-11T04:54:08+00:00 +omp,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,3885,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:40+00:00,2026-09-11T04:38:25+00:00 +omp,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,0,0.225806,1.0,7,31,1267,1267,3986,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:42+00:00,2026-09-11T04:31:37+00:00 +omp,task-task-graph-export,easy,97,go,0,0,0.0,1.0,0,20,17,17,2485,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:25+00:00,2026-09-11T03:35:57+00:00 +omp,go-genai-streamed-function-args,easy,98,go,0,0,0.0,0.0,0,6,0,62,1576,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:03:17+00:00,2026-09-11T03:44:14+00:00 +omp,ytt-jsonpath-query-api,easy,99,go,0,0,0.0,0.0,0,103,0,1,2946,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:15+00:00,2026-09-11T04:24:30+00:00 +omp,helm-unified-manifest-stream,easy,100,go,1,0,0.6,1.0,3,5,2,2,1851,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:09+00:00,2026-09-11T03:52:16+00:00 +omp,yjs-map-conflict-detection,easy,101,javascript,0,0,0.888889,1.0,8,9,231,231,4359,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:38:30+00:00,2026-09-11T04:11:52+00:00 +omp,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,2913,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:05:08+00:00,2026-09-11T04:09:11+00:00 +omp,psd-tools-blend-range-api,easy,103,python,0,0,0.955556,1.0,43,45,979,979,1535,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:50+00:00,2026-09-11T03:51:27+00:00 +omp,wazero-multi-module-snapshots,easy,104,go,0,0,0.935897,1.0,73,78,2,2,2442,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:11+00:00,2026-09-11T04:04:37+00:00 +omp,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.961538,1.0,25,26,5709,5709,2414,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:37:27+00:00,2026-09-11T03:31:55+00:00 +omp,actionlint-action-pinning-lint,easy,106,go,0,,,,,,,,,,,,,,,,, +omp,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.965217,1.0,111,115,1281,1281,915,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:17:51+00:00,2026-09-11T03:43:15+00:00 +omp,returns-validated-error-accumulation,easy,108,python,0,0,0.0,1.0,0,159,61,61,1534,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:09+00:00,2026-09-11T03:47:45+00:00 +omp,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,2917,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:38:24+00:00,2026-09-11T03:38:53+00:00 +omp,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.961538,1.0,125,130,566,566,837,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:05:34+00:00,2026-09-11T03:32:40+00:00 +omp,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,1758,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:18:06+00:00,2026-09-11T04:01:37+00:00 +omp,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,4676,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T03:10:42+00:00,2026-09-11T04:50:16+00:00 +omp,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.729167,1.0,70,96,561,561,2259,,,0,,openrouter/deepseek/deepseek-v4-flash-0731,18.1.17,verifier,2026-09-11T02:38:19+00:00,2026-09-11T03:30:04+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/opencode.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/opencode.csv new file mode 100644 index 0000000000..257202be61 --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/opencode.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +opencode,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,1677,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:58+00:00,2026-09-11T07:55:57+00:00 +opencode,gql-incremental-graphql-delivery,hard,2,python,0,0,0.588235,0.998767,10,17,810,811,1794,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:30+00:00,2026-09-11T07:59:25+00:00 +opencode,bandit-structured-nosec-directives,hard,3,python,0,0,0.971014,0.992908,67,69,280,282,2214,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:50+00:00,2026-09-11T08:03:45+00:00 +opencode,termenv-preserve-ansi-resets,hard,4,go,0,0,0.657143,1.0,23,35,87,87,3183,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:12+00:00,2026-09-11T08:20:17+00:00 +opencode,updo-policy-alerting,hard,5,go,0,1,1.0,1.0,17,17,123,123,1426,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:03+00:00,2026-09-11T07:51:51+00:00 +opencode,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.808511,1.0,38,47,70,70,6268,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:28+00:00,2026-09-11T09:13:56+00:00 +opencode,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,0.0,0,6,0,62,2651,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:52+00:00,2026-09-11T08:11:04+00:00 +opencode,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.9375,0.994501,30,32,1266,1273,4648,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:49+00:00,2026-09-11T08:44:18+00:00 +opencode,helm-array-merge-strategies,hard,9,go,0,0,0.531915,0.416667,25,47,5,12,1807,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:19+00:00,2026-09-11T07:57:28+00:00 +opencode,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.938776,0.999981,46,49,51468,51469,3519,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:10+00:00,2026-09-11T08:28:50+00:00 +opencode,pest-character-class-coalescing,hard,11,rust,0,0,0.0,0.0,0,104,0,250,2997,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:21+00:00,2026-09-11T08:17:20+00:00 +opencode,testem-bail-on-test-failure,hard,12,javascript,0,0,0.944444,0.997955,85,90,488,489,3728,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:49+00:00,2026-09-11T08:28:58+00:00 +opencode,koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,3403,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:58+00:00,2026-09-11T08:24:42+00:00 +opencode,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.0,0,15,0,82,2760,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:59+00:00,2026-09-11T08:14:00+00:00 +opencode,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.983333,1.0,59,60,1131,1131,1308,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:22+00:00,2026-09-11T07:49:11+00:00 +opencode,koota-query-predicates,hard,16,typescript,0,0,0.906977,1.0,39,43,172,172,2050,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:40+00:00,2026-09-11T08:01:52+00:00 +opencode,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,2834,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:58+00:00,2026-09-11T08:15:13+00:00 +opencode,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.971831,1.0,69,71,128,128,3296,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:53+00:00,2026-09-11T08:21:51+00:00 +opencode,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.894737,0.997167,68,76,1408,1412,1028,,,137,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:30+00:00,2026-09-11T07:44:42+00:00 +opencode,ink-grid-box-layout,hard,20,typescript,0,0,0.92,1.0,23,25,49,49,4815,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:04+00:00,2026-09-11T08:47:21+00:00 +opencode,awilix-async-container-initialization,hard,21,typescript,0,1,1.0,1.0,24,24,162,162,1039,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:20+00:00,2026-09-11T07:45:41+00:00 +opencode,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,1,1.0,1.0,14,14,9,9,2295,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:53+00:00,2026-09-11T08:05:10+00:00 +opencode,csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,1638,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:26+00:00,2026-09-11T07:54:44+00:00 +opencode,superjson-error-stack-serialization,hard,24,typescript,0,0,0.7875,0.974138,63,80,113,116,846,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:10+00:00,2026-09-11T07:41:18+00:00 +opencode,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.666667,1.0,8,12,139,139,3369,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:14+00:00,2026-09-11T08:24:24+00:00 +opencode,vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,2629,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:26+00:00,2026-09-11T08:14:17+00:00 +opencode,katex-multicolumn-array-spans,hard,27,javascript,0,1,1.0,1.0,94,94,599,599,2972,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:27+00:00,2026-09-11T08:18:00+00:00 +opencode,expr-try-catch-errors,hard,28,go,0,0,0.924051,1.0,73,79,66265,66265,5926,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:49+00:00,2026-09-11T09:05:36+00:00 +opencode,koota-composite-trait-aspects,hard,29,typescript,0,0,0.960784,1.0,49,51,172,172,4277,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:26:56+00:00,2026-09-11T08:38:15+00:00 +opencode,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.817073,0.333333,67,82,2,6,1330,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:49+00:00,2026-09-11T07:51:00+00:00 +opencode,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,1836,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:01+00:00,2026-09-11T07:58:39+00:00 +opencode,participle-grammar-conflict-analysis,hard,32,go,0,0,0.978022,1.0,89,91,153,153,1959,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:40+00:00,2026-09-11T08:00:20+00:00 +opencode,python-statemachine-state-data-scoping,hard,33,python,0,0,0.972222,1.0,70,72,1286,1286,2711,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:24+00:00,2026-09-11T08:13:36+00:00 +opencode,clack-async-autocomplete-options,hard,34,typescript,0,0,0.890244,1.0,73,82,643,643,1239,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:28+00:00,2026-09-11T07:50:08+00:00 +opencode,langchain-request-coalescing,hard,35,python,0,0,0.92,1.0,46,50,232,232,6167,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:30+00:00,2026-09-11T09:11:18+00:00 +opencode,httpx-streaming-json-iteration,hard,36,python,0,0,0.990741,1.0,107,108,1404,1404,1793,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:56+00:00,2026-09-11T07:57:49+00:00 +opencode,optique-conditional-option-dependencies,hard,37,typescript,0,0,0.972222,1.0,35,36,2034,2034,1762,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:32+00:00,2026-09-11T07:57:56+00:00 +opencode,textual-kitty-key-phases,hard,38,python,0,0,0.26087,1.0,6,23,57,57,862,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:50+00:00,2026-09-11T07:44:15+00:00 +opencode,pwntools-tube-multiplexing,medium,39,python,0,,,,,,,,,,,,,,,,, +opencode,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.92,1.0,23,25,1679,1679,10800,,,124,1,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:34+00:00,2026-09-11T10:28:35+00:00 +opencode,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.985075,1.0,66,67,2035,2035,2853,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:30+00:00,2026-09-11T08:17:04+00:00 +opencode,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,0.0,0,2,0,214,1560,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:48+00:00,2026-09-11T07:54:50+00:00 +opencode,bandit-incremental-cache-control,medium,43,python,0,0,0.943182,1.0,83,88,275,275,1400,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:32+00:00,2026-09-11T07:51:55+00:00 +opencode,fastapi-deprecation-response-headers,medium,44,python,0,0,0.927007,1.0,127,137,3134,3134,1567,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:47+00:00,2026-09-11T07:53:55+00:00 +opencode,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,2908,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:34+00:00,2026-09-11T08:17:04+00:00 +opencode,igel-persist-feature-schema,medium,46,python,0,0,0.958333,1.0,23,24,2,2,1985,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:45+00:00,2026-09-11T08:02:50+00:00 +opencode,bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,3745,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:08+00:00,2026-09-11T08:32:35+00:00 +opencode,anko-typed-variable-bindings,medium,48,go,0,0,0.111111,1.0,1,9,94,94,10148,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:27:58+00:00,2026-09-11T10:17:08+00:00 +opencode,mashumaro-flattened-dataclass-fields,medium,49,python,0,1,1.0,1.0,66,66,30014,30014,2660,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:36+00:00,2026-09-11T08:12:56+00:00 +opencode,dasel-html-document-format,medium,50,go,0,0,0.0,1.0,0,146,1012,1012,2009,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:09+00:00,2026-09-11T08:03:40+00:00 +opencode,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,0.0,0,17,0,7,2668,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:04+00:00,2026-09-11T08:14:34+00:00 +opencode,fastapi-implicit-head-options,medium,52,python,0,0,0.976744,1.0,42,43,3134,3134,2717,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:39+00:00,2026-09-11T08:13:57+00:00 +opencode,prometheus-typed-label-sorting,medium,53,go,0,0,0.941176,1.0,16,17,28,28,4288,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:30+00:00,2026-09-11T08:41:00+00:00 +opencode,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,966,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:26+00:00,2026-09-11T07:46:34+00:00 +opencode,pebble-durability-wait-apis,medium,55,go,0,0,0.0,0.0,0,59,0,44,2200,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:06+00:00,2026-09-11T08:06:47+00:00 +opencode,cliffy-config-file-parsing,medium,56,typescript,0,0,0.0,1.0,0,37,451,451,1464,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:34+00:00,2026-09-11T07:53:00+00:00 +opencode,sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,2865,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:29:30+00:00,2026-09-11T08:17:15+00:00 +opencode,kombu-single-active-consumer-priority,medium,58,python,0,0,0.882353,1.0,75,85,1421,1421,1226,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:30+00:00,2026-09-11T07:50:58+00:00 +opencode,skrub-duration-encoding,medium,59,python,0,0,0.976923,1.0,127,130,2784,2784,3105,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:30:36+00:00,2026-09-11T08:22:21+00:00 +opencode,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,1375,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T07:28:38+00:00,2026-09-11T07:51:35+00:00 +opencode,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,2568,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:42:26+00:00,2026-09-11T09:25:15+00:00 +opencode,tengo-destructuring-bindings,medium,62,go,0,0,0.857143,0.219697,78,91,29,132,1450,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:45+00:00,2026-09-11T09:38:55+00:00 +opencode,httpx-multipart-response-parsing,medium,63,python,0,0,0.352459,1.0,43,122,1272,1272,909,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:12:59+00:00,2026-09-11T09:28:12+00:00 +opencode,opa-rego-rule-profiling,medium,64,go,0,0,0.0,0.0,0,25,0,6,1025,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:31:22+00:00,2026-09-11T10:48:29+00:00 +opencode,ipython-session-bundle-replay,medium,65,python,0,0,0.823529,1.0,14,17,29,29,1692,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:42:31+00:00,2026-09-11T09:10:45+00:00 +opencode,scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,6189,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:33+00:00,2026-09-11T10:58:43+00:00 +opencode,opa-template-string-reconstruction,medium,67,go,0,0,0.4,1.0,2,5,4,4,4298,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:12:54+00:00,2026-09-11T10:24:35+00:00 +opencode,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2602,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:31:27+00:00,2026-09-11T11:14:49+00:00 +opencode,tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,2918,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:42:48+00:00,2026-09-11T09:31:27+00:00 +opencode,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,0.0,0,43,0,109,2919,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:45+00:00,2026-09-11T10:03:24+00:00 +opencode,cattrs-partial-structuring-recovery,medium,71,python,0,0,0.565217,1.0,39,69,7,7,3128,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:11:59+00:00,2026-09-11T10:04:08+00:00 +opencode,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,3174,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:31:23+00:00,2026-09-11T11:24:17+00:00 +opencode,etree-xml-diff-patch,medium,73,go,0,0,0.0,1.0,0,52,15,15,3133,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:42:46+00:00,2026-09-11T09:34:59+00:00 +opencode,textual-richlog-follow-state,medium,74,python,0,0,0.7,1.0,14,20,6,6,5510,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:26+00:00,2026-09-11T10:46:18+00:00 +opencode,yaegi-go-embed-directives,medium,75,go,0,0,0.973684,1.0,37,38,58,58,5400,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:11:57+00:00,2026-09-11T10:41:58+00:00 +opencode,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.984252,1.0,250,254,22,22,1783,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:24+00:00,2026-09-11T11:02:09+00:00 +opencode,numba-stencil-boundary-modes,easy,77,python,0,0,0.931034,1.0,27,29,827,827,5585,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:42:44+00:00,2026-09-11T10:15:50+00:00 +opencode,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.0,1.0,0,10,209,209,8167,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:05+00:00,2026-09-11T11:31:14+00:00 +opencode,go-git-worktree-merge-conflicts,easy,79,go,0,1,1.0,1.0,17,17,2,2,7918,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:10+00:00,2026-09-11T11:26:09+00:00 +opencode,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,1594,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:31:25+00:00,2026-09-11T10:58:01+00:00 +opencode,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,1,1.0,1.0,33,33,1133,1133,3831,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:26+00:00,2026-09-11T09:47:18+00:00 +opencode,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,490,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:24+00:00,2026-09-11T09:23:34+00:00 +opencode,anko-default-function-arguments,easy,83,go,0,0,0.5,1.0,1,2,119,119,9034,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:12:16+00:00,2026-09-11T11:42:51+00:00 +opencode,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,7020,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:31:38+00:00,2026-09-11T12:28:39+00:00 +opencode,abs-stepped-slices,easy,85,go,0,0,0.833333,0.833333,5,6,5,6,1827,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:31+00:00,2026-09-11T09:14:00+00:00 +opencode,testem-per-launcher-reports,easy,86,javascript,0,0,0.984615,1.0,64,65,469,469,1912,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:39+00:00,2026-09-11T09:47:33+00:00 +opencode,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,1231,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:07+00:00,2026-09-11T09:34:40+00:00 +opencode,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,0.0,0,22,0,58,7012,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:04+00:00,2026-09-11T12:28:57+00:00 +opencode,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1575,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:26+00:00,2026-09-11T09:09:42+00:00 +opencode,geo-shapeindex-serialization,easy,90,go,0,0,0.375,1.0,9,24,599,599,2551,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:25+00:00,2026-09-11T09:57:56+00:00 +opencode,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,0.0,0,82,0,2,1020,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:16:40+00:00,2026-09-11T09:33:40+00:00 +opencode,abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,1391,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:25+00:00,2026-09-11T10:55:36+00:00 +opencode,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.962264,1.0,51,53,8,8,1847,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:10+00:00,2026-09-11T09:13:58+00:00 +opencode,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,869,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:17:02+00:00,2026-09-11T09:31:31+00:00 +opencode,vitest-duration-sharding,easy,95,typescript,0,0,0.982143,0.958333,55,56,23,24,2150,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:48+00:00,2026-09-11T09:50:40+00:00 +opencode,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,5257,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:25+00:00,2026-09-11T12:00:04+00:00 +opencode,task-task-graph-export,easy,97,go,0,0,0.0,0.941176,0,20,16,17,2146,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:56+00:00,2026-09-11T09:19:43+00:00 +opencode,go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,1128,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:30+00:00,2026-09-11T09:34:19+00:00 +opencode,ytt-jsonpath-query-api,easy,99,go,0,0,0.990291,1.0,102,103,1,1,1175,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:14:20+00:00,2026-09-11T09:33:57+00:00 +opencode,helm-unified-manifest-stream,easy,100,go,1,0,0.0,0.5,0,5,1,2,780,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:53+00:00,2026-09-11T10:45:53+00:00 +opencode,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,3838,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:43:56+00:00,2026-09-11T09:47:56+00:00 +opencode,goreleaser-retry-publish-auditing,easy,102,go,0,0,0.068966,1.0,2,29,29,29,1362,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:18:13+00:00,2026-09-11T09:40:56+00:00 +opencode,psd-tools-blend-range-api,easy,103,python,0,0,0.933333,1.0,42,45,979,979,778,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:16:55+00:00,2026-09-11T09:29:54+00:00 +opencode,wazero-multi-module-snapshots,easy,104,go,0,0,0.935897,1.0,73,78,2,2,3196,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:54+00:00,2026-09-11T11:26:12+00:00 +opencode,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.961538,1.0,25,26,5709,5709,1805,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:44:16+00:00,2026-09-11T09:14:22+00:00 +opencode,actionlint-action-pinning-lint,easy,106,go,0,0,0.0,0.0,0,55,0,145,2526,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:15:41+00:00,2026-09-11T09:57:48+00:00 +opencode,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.965217,1.0,111,115,1281,1281,1392,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:17:17+00:00,2026-09-11T09:40:29+00:00 +opencode,returns-validated-error-accumulation,easy,108,python,0,0,0.974843,1.0,155,159,61,61,3881,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:32:56+00:00,2026-09-11T11:37:37+00:00 +opencode,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,1757,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:44:03+00:00,2026-09-11T09:13:21+00:00 +opencode,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.8,1.0,104,130,566,566,1125,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:17:01+00:00,2026-09-11T09:35:49+00:00 +opencode,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,3854,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T09:17:17+00:00,2026-09-11T10:21:33+00:00 +opencode,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,1754,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T10:33:27+00:00,2026-09-11T11:02:42+00:00 +opencode,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.479167,1.0,46,96,561,561,1553,,,0,0,openrouter/deepseek/deepseek-v4-flash-0731,1.18.30,verifier,2026-09-11T08:44:26+00:00,2026-09-11T09:10:20+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/pi.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/pi.csv new file mode 100644 index 0000000000..d0011b71fb --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/pi.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +pi,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.878049,1.0,36,41,1131,1131,2238,0.0029774709999999998,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185428Z-5554,2026-09-10T18:54:28+00:00,2026-09-10T19:32:33+00:00 +pi,gql-incremental-graphql-delivery,hard,2,python,0,0,0.117647,0.998767,2,17,810,811,1858,0.00692975,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185421Z-5425,2026-09-10T18:54:21+00:00,2026-09-10T19:26:53+00:00 +pi,bandit-structured-nosec-directives,hard,3,python,0,0,0.971014,0.992908,67,69,280,282,1995,0.00825352,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185418Z-5424,2026-09-10T18:54:18+00:00,2026-09-10T19:29:01+00:00 +pi,termenv-preserve-ansi-resets,hard,4,go,0,0,0.857143,1.0,30,35,87,87,1203,0.005986567,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185423Z-5165,2026-09-10T18:54:23+00:00,2026-09-10T19:14:59+00:00 +pi,updo-policy-alerting,hard,5,go,0,0,0.823529,1.0,14,17,123,123,1174,0.00431127,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185419Z-5653,2026-09-10T18:54:19+00:00,2026-09-10T19:14:50+00:00 +pi,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.978723,1.0,46,47,70,70,7316,0.005894362,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185500Z-5528,2026-09-10T18:55:01+00:00,2026-09-10T20:57:52+00:00 +pi,oxvg-structural-selector-preservation,hard,7,rust,0,0,0.0,0.0,0,6,0,62,3350,0.014589814999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185442Z-5496,2026-09-10T18:54:42+00:00,2026-09-10T19:51:39+00:00 +pi,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.90625,0.990573,29,32,1261,1273,3763,0.012289675,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185419Z-5266,2026-09-10T18:54:19+00:00,2026-09-10T19:57:38+00:00 +pi,helm-array-merge-strategies,hard,9,go,0,0,0.744681,0.833333,35,47,10,12,3937,0.02058102,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T221039Z-160916,2026-09-10T22:10:39+00:00,2026-09-10T23:16:22+00:00 +pi,meriyah-explicit-resource-declarations,hard,10,typescript,0,1,1.0,1.0,49,49,51469,51469,2910,0.004464964,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185444Z-5669,2026-09-10T18:54:44+00:00,2026-09-10T19:44:21+00:00 +pi,pest-character-class-coalescing,hard,11,rust,0,0,0.0,0.0,0,104,0,250,1683,0.003524073,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185448Z-5653,2026-09-10T18:54:48+00:00,2026-09-10T19:23:52+00:00 +pi,testem-bail-on-test-failure,hard,12,javascript,0,0,0.611111,1.0,55,90,489,489,2962,0.003942789,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185430Z-5364,2026-09-10T18:54:30+00:00,2026-09-10T19:44:18+00:00 +pi,koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,2602,0.008499850000000002,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185429Z-5925,2026-09-10T18:54:29+00:00,2026-09-10T19:38:34+00:00 +pi,prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.0,0,15,0,82,3654,0.018958775,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185519Z-5781,2026-09-10T18:55:19+00:00,2026-09-10T19:56:45+00:00 +pi,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.95,1.0,57,60,1131,1131,1449,0.004714635,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185431Z-5798,2026-09-10T18:54:31+00:00,2026-09-10T19:19:56+00:00 +pi,koota-query-predicates,hard,16,typescript,0,0,0.953488,1.0,41,43,172,172,3312,0.008450708,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185434Z-5498,2026-09-10T18:54:34+00:00,2026-09-10T19:50:10+00:00 +pi,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,4584,0.011073095,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185459Z-6034,2026-09-10T18:54:59+00:00,2026-09-10T20:11:39+00:00 +pi,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.0,1.0,0,71,128,128,2646,0.015857034,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185433Z-5855,2026-09-10T18:54:33+00:00,2026-09-10T19:40:01+00:00 +pi,kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.842105,1.0,64,76,1412,1412,4554,0.011039420000000001,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185421Z-5873,2026-09-10T18:54:21+00:00,2026-09-10T20:11:41+00:00 +pi,ink-grid-box-layout,hard,20,typescript,0,0,0.72,1.0,18,25,49,49,6939,0.01616012,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185435Z-5576,2026-09-10T18:54:35+00:00,2026-09-10T20:50:36+00:00 +pi,awilix-async-container-initialization,hard,21,typescript,0,0,0.875,1.0,21,24,162,162,1601,0.006700721999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185421Z-6177,2026-09-10T18:54:21+00:00,2026-09-10T19:21:56+00:00 +pi,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.928571,1.0,13,14,9,9,2461,0.0031811879999999997,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185438Z-5963,2026-09-10T18:54:38+00:00,2026-09-10T19:36:55+00:00 +pi,csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,2373,0.00612694,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185419Z-6037,2026-09-10T18:54:19+00:00,2026-09-10T19:35:19+00:00 +pi,superjson-error-stack-serialization,hard,24,typescript,0,0,0.8625,0.982759,69,80,114,116,1576,0.005565931,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185434Z-5659,2026-09-10T18:54:34+00:00,2026-09-10T19:21:14+00:00 +pi,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.75,1.0,9,12,139,139,3715,0.01416533,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185428Z-6285,2026-09-10T18:54:28+00:00,2026-09-10T19:57:09+00:00 +pi,vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,3641,0.01266237,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185421Z-6089,2026-09-10T18:54:21+00:00,2026-09-10T19:56:35+00:00 +pi,katex-multicolumn-array-spans,hard,27,javascript,0,0,0.180851,1.0,17,94,599,599,3702,0.016679270000000003,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185441Z-6149,2026-09-10T18:54:41+00:00,2026-09-10T19:57:29+00:00 +pi,expr-try-catch-errors,hard,28,go,0,0,0.949367,1.0,75,79,66265,66265,3708,0.017111595,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185422Z-5806,2026-09-10T18:54:22+00:00,2026-09-10T19:56:44+00:00 +pi,koota-composite-trait-aspects,hard,29,typescript,0,1,1.0,1.0,51,51,172,172,3158,0.0075695450000000004,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185429Z-6372,2026-09-10T18:54:29+00:00,2026-09-10T19:47:52+00:00 +pi,onedump-dump-encryption-pipeline,hard,30,go,0,0,0.182927,1.0,15,82,6,6,3345,0.008929555,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185432Z-6205,2026-09-10T18:54:32+00:00,2026-09-10T19:51:39+00:00 +pi,eicrud-keyset-pagination-cursor,hard,31,typescript,0,1,1.0,1.0,14,14,168,168,2152,0.002122838,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185451Z-6239,2026-09-10T18:54:51+00:00,2026-09-10T19:31:39+00:00 +pi,participle-grammar-conflict-analysis,hard,32,go,0,0,0.021978,1.0,2,91,153,153,2049,0.009119765,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185421Z-5911,2026-09-10T18:54:21+00:00,2026-09-10T19:29:04+00:00 +pi,python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,3415,0.005427454999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185419Z-6480,2026-09-10T18:54:19+00:00,2026-09-10T19:52:08+00:00 +pi,clack-async-autocomplete-options,hard,34,typescript,0,0,0.987805,1.0,81,82,643,643,2238,0.0107393,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185430Z-6298,2026-09-10T18:54:30+00:00,2026-09-10T19:33:10+00:00 +pi,langchain-request-coalescing,hard,35,python,0,,,,,,,,,,,,,,,,, +pi,httpx-streaming-json-iteration,hard,36,python,0,0,0.925926,1.0,100,108,1404,1404,1041,0.003350722,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185430Z-6046,2026-09-10T18:54:30+00:00,2026-09-10T19:12:18+00:00 +pi,optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,4386,0.010263745000000001,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185425Z-6571,2026-09-10T18:54:25+00:00,2026-09-10T20:08:22+00:00 +pi,textual-kitty-key-phases,hard,38,python,0,0,0.608696,1.0,14,23,57,57,2005,0.010134651,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185429Z-6373,2026-09-10T18:54:29+00:00,2026-09-10T19:29:19+00:00 +pi,pwntools-tube-multiplexing,medium,39,python,0,0,0.958904,1.0,70,73,1,1,3258,0.008315563,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260911T010048Z-550815,2026-09-11T01:00:48+00:00,2026-09-11T01:55:09+00:00 +pi,arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.92,1.0,23,25,1679,1679,9540,0.01553578,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185458Z-6121,2026-09-10T18:54:58+00:00,2026-09-10T21:33:59+00:00 +pi,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.985075,1.0,66,67,2035,2035,1979,0.005622492,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185413Z-6645,2026-09-10T18:54:13+00:00,2026-09-10T19:28:12+00:00 +pi,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,1770,0.0043804790000000005,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185508Z-6482,2026-09-10T18:55:08+00:00,2026-09-10T19:25:21+00:00 +pi,bandit-incremental-cache-control,medium,43,python,0,0,0.931818,1.0,82,88,275,275,1493,0.0042120249999999995,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185417Z-6512,2026-09-10T18:54:17+00:00,2026-09-10T19:20:39+00:00 +pi,fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,1529,0.003669246,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185426Z-6194,2026-09-10T18:54:26+00:00,2026-09-10T19:20:26+00:00 +pi,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,1987,0.004387949,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185416Z-6737,2026-09-10T18:54:16+00:00,2026-09-10T19:28:22+00:00 +pi,igel-persist-feature-schema,medium,46,python,0,1,1.0,1.0,24,24,2,2,1468,0.00547653,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185437Z-6570,2026-09-10T18:54:37+00:00,2026-09-10T19:20:19+00:00 +pi,bandit-interprocedural-taint-checks,medium,47,python,0,0,0.969697,1.0,64,66,293,293,1936,0.003856755,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185417Z-6600,2026-09-10T18:54:17+00:00,2026-09-10T19:28:02+00:00 +pi,anko-typed-variable-bindings,medium,48,go,0,1,1.0,1.0,9,9,94,94,8652,0.023839865000000002,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185424Z-6271,2026-09-10T18:54:24+00:00,2026-09-10T21:19:07+00:00 +pi,mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.924242,1.0,61,66,30014,30014,4051,0.014316045,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185411Z-6816,2026-09-10T18:54:11+00:00,2026-09-10T20:02:45+00:00 +pi,dasel-html-document-format,medium,50,go,0,1,1.0,1.0,146,146,1012,1012,1932,0.004242132,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185423Z-6644,2026-09-10T18:54:23+00:00,2026-09-10T19:28:03+00:00 +pi,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,0,0.0,0.0,0,17,0,7,3310,0.005011234,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185547Z-6676,2026-09-10T18:55:47+00:00,2026-09-10T19:50:59+00:00 +pi,fastapi-implicit-head-options,medium,52,python,0,0,0.883721,1.0,38,43,3134,3134,3175,0.01074913,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185426Z-6356,2026-09-10T18:54:26+00:00,2026-09-10T19:47:51+00:00 +pi,prometheus-typed-label-sorting,medium,53,go,0,0,0.823529,1.0,14,17,28,28,2330,0.003724695,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185454Z-6919,2026-09-10T18:54:54+00:00,2026-09-10T19:34:04+00:00 +pi,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,884,0.002421152,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185441Z-6750,2026-09-10T18:54:41+00:00,2026-09-10T19:10:39+00:00 +pi,pebble-durability-wait-apis,medium,55,go,0,0,0.0,0.0,0,59,0,44,4554,0.01863801,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185428Z-6752,2026-09-10T18:54:28+00:00,2026-09-10T20:11:43+00:00 +pi,cliffy-config-file-parsing,medium,56,typescript,0,0,0.0,1.0,0,37,451,451,1370,0.002601329,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185426Z-6428,2026-09-10T18:54:26+00:00,2026-09-10T19:17:48+00:00 +pi,sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,3339,0.013769115,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185412Z-7134,2026-09-10T18:54:12+00:00,2026-09-10T19:50:54+00:00 +pi,kombu-single-active-consumer-priority,medium,58,python,0,0,0.976471,1.0,83,85,1421,1421,2564,0.00976611,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185423Z-6877,2026-09-10T18:54:23+00:00,2026-09-10T19:38:37+00:00 +pi,skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,5242,0.01188582,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T185445Z-6884,2026-09-10T18:54:45+00:00,2026-09-10T20:23:09+00:00 +pi,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,1423,0.0065530449999999995,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T232319Z-412518,2026-09-10T23:23:19+00:00,2026-09-10T23:47:04+00:00 +pi,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,4138,0.007959615,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201822Z-93044,2026-09-10T20:18:22+00:00,2026-09-10T21:27:21+00:00 +pi,tengo-destructuring-bindings,medium,62,go,0,1,1.0,1.0,91,91,132,132,2493,0.006897254,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211044Z-123905,2026-09-10T21:10:44+00:00,2026-09-10T21:52:19+00:00 +pi,httpx-multipart-response-parsing,medium,63,python,0,0,0.983607,1.0,120,122,1272,1272,2138,0.008431857999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T223850Z-60176,2026-09-10T22:38:50+00:00,2026-09-10T23:14:32+00:00 +pi,opa-rego-rule-profiling,medium,64,go,0,0,0.96,1.0,24,25,6,6,1388,0.0037835150000000003,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214007Z-294596,2026-09-10T21:40:07+00:00,2026-09-10T22:03:28+00:00 +pi,ipython-session-bundle-replay,medium,65,python,0,0,0.882353,1.0,15,17,29,29,2164,0.005351160000000001,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201827Z-93199,2026-09-10T20:18:27+00:00,2026-09-10T20:54:35+00:00 +pi,scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,3522,0.02012284,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211048Z-124118,2026-09-10T21:10:48+00:00,2026-09-10T22:09:34+00:00 +pi,opa-template-string-reconstruction,medium,67,go,0,0,0.0,0.0,0,5,0,4,4777,0.029743095,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224002Z-60276,2026-09-10T22:40:02+00:00,2026-09-10T23:59:52+00:00 +pi,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2381,0.008424369999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214014Z-294671,2026-09-10T21:40:14+00:00,2026-09-10T22:19:59+00:00 +pi,tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,5175,0.004969395,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201828Z-93688,2026-09-10T20:18:28+00:00,2026-09-10T21:44:46+00:00 +pi,fd-deterministic-multi-key-sorting,medium,70,rust,1,0,0.0,0.0,0,43,0,109,1097,0.012440291,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211105Z-124616,2026-09-10T21:11:05+00:00,2026-09-10T21:29:32+00:00 +pi,cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,3377,0.009095565,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224009Z-60366,2026-09-10T22:40:09+00:00,2026-09-10T23:36:31+00:00 +pi,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,2042,0.013960914999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214019Z-294723,2026-09-10T21:40:19+00:00,2026-09-10T22:14:23+00:00 +pi,etree-xml-diff-patch,medium,73,go,0,0,0.0,1.0,0,52,15,15,5578,0.01776983,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201831Z-93957,2026-09-10T20:18:31+00:00,2026-09-10T21:51:30+00:00 +pi,textual-richlog-follow-state,medium,74,python,0,0,0.75,1.0,15,20,6,6,3363,0.004339307,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211055Z-124984,2026-09-10T21:10:55+00:00,2026-09-10T22:07:15+00:00 +pi,yaegi-go-embed-directives,medium,75,go,0,0,0.973684,1.0,37,38,58,58,7800,0.021994865000000002,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224013Z-60522,2026-09-10T22:40:13+00:00,2026-09-11T00:50:16+00:00 +pi,kysely-window-grouping-helpers,medium,76,typescript,0,0,0.984252,1.0,250,254,22,22,3318,0.003889156,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214046Z-294797,2026-09-10T21:40:46+00:00,2026-09-10T22:36:12+00:00 +pi,numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,4365,0.00967182,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201843Z-94830,2026-09-10T20:18:44+00:00,2026-09-10T21:31:35+00:00 +pi,valibot-recursive-schema-composition,easy,78,typescript,0,0,0.2,1.0,2,10,209,209,8689,0.024725459999999998,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211120Z-125224,2026-09-10T21:11:20+00:00,2026-09-10T23:36:25+00:00 +pi,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.0,0.0,0,17,0,2,2272,0.009106177,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224018Z-60935,2026-09-10T22:40:18+00:00,2026-09-10T23:18:13+00:00 +pi,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,4035,0.0081829,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214050Z-294848,2026-09-10T21:40:50+00:00,2026-09-10T22:48:10+00:00 +pi,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,0.997352,0,33,1130,1133,3721,0.014618995,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201837Z-95275,2026-09-10T20:18:37+00:00,2026-09-10T21:20:44+00:00 +pi,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,1241,0.00503875,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211108Z-125544,2026-09-10T21:11:08+00:00,2026-09-10T21:31:56+00:00 +pi,anko-default-function-arguments,easy,83,go,0,1,1.0,1.0,2,2,119,119,2887,0.011483566,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224021Z-60995,2026-09-10T22:40:21+00:00,2026-09-10T23:28:30+00:00 +pi,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,0,0.972973,1.0,36,37,1267,1267,5309,0.013523984,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214059Z-294900,2026-09-10T21:40:59+00:00,2026-09-10T23:09:35+00:00 +pi,abs-stepped-slices,easy,85,go,0,1,1.0,1.0,6,6,6,6,1766,0.005066941,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201845Z-95499,2026-09-10T20:18:45+00:00,2026-09-10T20:48:14+00:00 +pi,testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,2071,0.00643188,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211110Z-125659,2026-09-10T21:11:11+00:00,2026-09-10T21:45:45+00:00 +pi,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2527,0.006825197,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224122Z-61050,2026-09-10T22:41:22+00:00,2026-09-10T23:23:46+00:00 +pi,wasmi-trap-coredumps,easy,88,rust,0,0,0.0,0.0,0,22,0,58,2812,0.008018002,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214118Z-294953,2026-09-10T21:41:18+00:00,2026-09-10T22:28:19+00:00 +pi,koota-entity-snapshot-rollback,easy,89,python,0,0,0.964286,1.0,81,84,47,47,1359,0.004161521,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201854Z-95610,2026-09-10T20:18:54+00:00,2026-09-10T20:41:46+00:00 +pi,geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,4595,0.012729705,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211107Z-125795,2026-09-10T21:11:07+00:00,2026-09-10T22:27:50+00:00 +pi,arcane-drift-detection-baselines,easy,91,go,0,0,0.0,0.0,0,82,0,2,912,0.004221536,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224220Z-61104,2026-09-10T22:42:20+00:00,2026-09-10T22:57:51+00:00 +pi,abs-module-cache-flags,easy,92,go,0,0,0.8,1.0,16,20,3,3,1502,0.003592737,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214127Z-295005,2026-09-10T21:41:27+00:00,2026-09-10T22:06:32+00:00 +pi,aiomonitor-task-snapshots-diff,easy,93,python,0,1,1.0,1.0,53,53,8,8,2330,0.008233049999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201851Z-96125,2026-09-10T20:18:51+00:00,2026-09-10T20:57:55+00:00 +pi,kcp-go-multiplexed-kcp-streams,easy,94,go,0,0,0.0,1.0,0,30,12,12,5659,0.022905705,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211110Z-126177,2026-09-10T21:11:10+00:00,2026-09-10T22:45:35+00:00 +pi,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,2014,0.003471712,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224245Z-61164,2026-09-10T22:42:45+00:00,2026-09-10T23:16:23+00:00 +pi,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,4967,0.0052221030000000005,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214127Z-295086,2026-09-10T21:41:27+00:00,2026-09-10T23:04:17+00:00 +pi,task-task-graph-export,easy,97,go,0,0,0.0,0.941176,0,20,16,17,2114,0.005380615,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201908Z-96466,2026-09-10T20:19:08+00:00,2026-09-10T20:54:29+00:00 +pi,go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,1695,0.0038545920000000004,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T235838Z-308878,2026-09-10T23:58:38+00:00,2026-09-11T00:26:57+00:00 +pi,ytt-jsonpath-query-api,easy,99,go,0,1,1.0,1.0,103,103,1,1,2500,0.00987021,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224256Z-61216,2026-09-10T22:42:56+00:00,2026-09-10T23:24:42+00:00 +pi,helm-unified-manifest-stream,easy,100,go,1,0,0.6,1.0,3,5,2,2,3526,0.003773287,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214141Z-295147,2026-09-10T21:41:41+00:00,2026-09-10T22:40:33+00:00 +pi,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,5027,0.012552109999999998,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201857Z-97205,2026-09-10T20:18:57+00:00,2026-09-10T21:42:54+00:00 +pi,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,3035,0.003073335,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211231Z-126898,2026-09-10T21:12:31+00:00,2026-09-10T22:03:13+00:00 +pi,psd-tools-blend-range-api,easy,103,python,0,0,0.822222,1.0,37,45,979,979,2143,0.008977714999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224304Z-61268,2026-09-10T22:43:04+00:00,2026-09-10T23:18:51+00:00 +pi,wazero-multi-module-snapshots,easy,104,go,0,0,0.0,1.0,0,78,2,2,2087,0.013416589999999999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214146Z-295208,2026-09-10T21:41:46+00:00,2026-09-10T22:16:35+00:00 +pi,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,0,0.0,1.0,0,26,5709,5709,2666,0.010276,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201911Z-97485,2026-09-10T20:19:11+00:00,2026-09-10T21:03:41+00:00 +pi,actionlint-action-pinning-lint,easy,106,go,0,0,0.0,0.0,0,55,0,145,2446,0.00757999,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211118Z-128228,2026-09-10T21:11:18+00:00,2026-09-10T21:52:22+00:00 +pi,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.930435,1.0,107,115,1281,1281,2597,0.007468541,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224305Z-61321,2026-09-10T22:43:05+00:00,2026-09-10T23:26:26+00:00 +pi,returns-validated-error-accumulation,easy,108,python,0,1,1.0,1.0,159,159,61,61,4876,0.005624966,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214148Z-295365,2026-09-10T21:41:48+00:00,2026-09-10T23:03:06+00:00 +pi,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,2848,0.008606928000000002,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T221024Z-160917,2026-09-10T22:10:24+00:00,2026-09-10T22:57:57+00:00 +pi,drizzle-orm-window-function-builders,easy,110,typescript,0,1,1.0,1.0,130,130,566,566,2595,0.01040814,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T211155Z-128742,2026-09-10T21:11:55+00:00,2026-09-10T21:55:35+00:00 +pi,happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,2359,0.012696325,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T224317Z-61374,2026-09-10T22:43:17+00:00,2026-09-10T23:22:42+00:00 +pi,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,3546,0.01537156,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T214204Z-295437,2026-09-10T21:42:04+00:00,2026-09-10T22:41:16+00:00 +pi,true-myth-iterable-collection-combinators,easy,113,typescript,1,0,0.604167,1.0,58,96,561,561,2976,0.012021315000000001,,0,,openrouter/deepseek/deepseek-v4-flash-0731,0.85.1,20260910T201912Z-97932,2026-09-10T20:19:12+00:00,2026-09-10T21:08:51+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/senior-dev.csv b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/senior-dev.csv new file mode 100644 index 0000000000..d7b5ce369c --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-12-harness-comparison-v4-flash/tasks/senior-dev.csv @@ -0,0 +1,114 @@ +harness,task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,seconds,cost_usd,model_calls,exit,timed_out,model,harness_version,attempt_id,started,finished +senior-dev,obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.902439,1.0,37,41,1131,1131,2208,0.292758,133,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030334-13592,2026-09-12T03:03:35+00:00,2026-09-12T03:40:26+00:00 +senior-dev,gql-incremental-graphql-delivery,hard,2,python,0,0,0.058824,1.0,1,17,811,811,3001,0.670467,214,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030426-22083,2026-09-12T03:04:27+00:00,2026-09-12T03:54:31+00:00 +senior-dev,bandit-structured-nosec-directives,hard,3,python,0,1,1.0,1.0,69,69,282,282,2605,0.449804,135,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030517-12869,2026-09-12T03:05:17+00:00,2026-09-12T03:48:45+00:00 +senior-dev,termenv-preserve-ansi-resets,hard,4,go,0,0,0.771429,1.0,27,35,87,87,1535,0.118809,59,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030609-12338,2026-09-12T03:06:09+00:00,2026-09-12T03:31:46+00:00 +senior-dev,updo-policy-alerting,hard,5,go,0,0,0.882353,1.0,15,17,123,123,1703,0.264829,111,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030337-13992,2026-09-12T03:03:38+00:00,2026-09-12T03:32:03+00:00 +senior-dev,effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.893617,1.0,42,47,70,70,7554,1.044218,365,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030429-22509,2026-09-12T03:04:30+00:00,2026-09-12T05:10:27+00:00 +senior-dev,oxvg-structural-selector-preservation,hard,7,rust,0,1,1.0,1.0,6,6,62,62,4650,0.726019,218,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030520-13271,2026-09-12T03:05:20+00:00,2026-09-12T04:22:53+00:00 +senior-dev,sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.9375,0.974863,30,32,1241,1273,3851,0.50432,136,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030612-12757,2026-09-12T03:06:12+00:00,2026-09-12T04:10:26+00:00 +senior-dev,helm-array-merge-strategies,hard,9,go,0,0,1.0,0.833333,47,47,10,12,3500,0.403012,147,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030340-14393,2026-09-12T03:03:41+00:00,2026-09-12T04:02:04+00:00 +senior-dev,meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.979592,1.0,48,49,51469,51469,4094,0.690071,212,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030432-22925,2026-09-12T03:04:33+00:00,2026-09-12T04:12:52+00:00 +senior-dev,pest-character-class-coalescing,hard,11,rust,0,0,1.0,0.996,104,104,249,250,2116,0.287506,128,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030523-13684,2026-09-12T03:05:23+00:00,2026-09-12T03:40:42+00:00 +senior-dev,testem-bail-on-test-failure,hard,12,javascript,0,0,0.5,0.985685,45,90,482,489,4384,0.6641,187,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030615-13167,2026-09-12T03:06:15+00:00,2026-09-12T04:19:22+00:00 +senior-dev,koota-pair-relation-tracking,hard,13,typescript,0,0,0.894737,1.0,34,38,172,172,4784,0.737066,212,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030343-14798,2026-09-12T03:03:44+00:00,2026-09-12T04:23:30+00:00 +senior-dev,prometheus-transactional-reload-status,hard,14,typescript,0,0,1.0,0.878049,15,15,72,82,6378,0.458262,203,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030435-23345,2026-09-12T03:04:36+00:00,2026-09-12T04:50:58+00:00 +senior-dev,obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.966667,1.0,58,60,1131,1131,2361,0.357635,119,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030526-14086,2026-09-12T03:05:26+00:00,2026-09-12T03:44:50+00:00 +senior-dev,koota-query-predicates,hard,16,typescript,0,1,1.0,1.0,43,43,172,172,4479,0.775097,226,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030618-13584,2026-09-12T03:06:18+00:00,2026-09-12T04:21:00+00:00 +senior-dev,quill-shared-toolbar-focus,hard,17,typescript,0,0,0.615385,1.0,8,13,22,22,2808,0.217647,85,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030346-15217,2026-09-12T03:03:47+00:00,2026-09-12T03:50:37+00:00 +senior-dev,koota-deferred-mutation-buffer,hard,18,typescript,0,0,0.901408,1.0,64,71,128,128,3097,0.436139,141,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030438-23766,2026-09-12T03:04:39+00:00,2026-09-12T03:56:18+00:00 +senior-dev,kombu-virtual-queue-dead-lettering,hard,19,python,0,1,1.0,1.0,76,76,1412,1412,2479,0.313904,116,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030529-14504,2026-09-12T03:05:29+00:00,2026-09-12T03:46:53+00:00 +senior-dev,ink-grid-box-layout,hard,20,typescript,0,1,1.0,1.0,25,25,49,49,4255,0.437953,162,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030621-13997,2026-09-12T03:06:21+00:00,2026-09-12T04:17:19+00:00 +senior-dev,awilix-async-container-initialization,hard,21,typescript,0,1,1.0,1.0,24,24,162,162,2165,0.218503,103,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030349-15638,2026-09-12T03:03:50+00:00,2026-09-12T03:39:57+00:00 +senior-dev,happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.642857,1.0,9,14,9,9,3322,0.368442,191,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030441-24173,2026-09-12T03:04:42+00:00,2026-09-12T04:00:07+00:00 +senior-dev,csstree-shorthand-expansion-compression,hard,23,javascript,1,0,0.974684,1.0,77,79,16715,16715,2304,0.2783,115,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030532-14924,2026-09-12T03:05:32+00:00,2026-09-12T03:44:00+00:00 +senior-dev,superjson-error-stack-serialization,hard,24,typescript,0,0,0.9375,0.982759,75,80,114,116,1176,0.10357,59,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030624-14405,2026-09-12T03:06:24+00:00,2026-09-12T03:26:02+00:00 +senior-dev,kea-atomic-signal-selectors,hard,25,typescript,0,0,0.75,1.0,9,12,139,139,4733,0.519643,166,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030352-16069,2026-09-12T03:03:53+00:00,2026-09-12T04:22:48+00:00 +senior-dev,vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,2498,0.240913,96,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030444-24594,2026-09-12T03:04:45+00:00,2026-09-12T03:46:26+00:00 +senior-dev,katex-multicolumn-array-spans,hard,27,javascript,0,1,1.0,1.0,94,94,599,599,2966,0.508291,197,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030535-15356,2026-09-12T03:05:35+00:00,2026-09-12T03:55:04+00:00 +senior-dev,expr-try-catch-errors,hard,28,go,0,0,0.987342,1.0,78,79,66265,66265,3733,0.464137,170,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030627-14808,2026-09-12T03:06:27+00:00,2026-09-12T04:08:43+00:00 +senior-dev,koota-composite-trait-aspects,hard,29,typescript,0,0,0.588235,1.0,30,51,172,172,3046,0.465523,165,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030355-16494,2026-09-12T03:03:56+00:00,2026-09-12T03:54:44+00:00 +senior-dev,onedump-dump-encryption-pipeline,hard,30,go,0,1,1.0,1.0,82,82,6,6,2245,0.106409,72,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030447-25006,2026-09-12T03:04:48+00:00,2026-09-12T03:42:16+00:00 +senior-dev,eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,2514,0.278236,154,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030538-15774,2026-09-12T03:05:38+00:00,2026-09-12T03:47:36+00:00 +senior-dev,participle-grammar-conflict-analysis,hard,32,go,0,1,1.0,1.0,91,91,153,153,2345,0.139824,55,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030630-15211,2026-09-12T03:06:30+00:00,2026-09-12T03:45:38+00:00 +senior-dev,python-statemachine-state-data-scoping,hard,33,python,0,0,0.944444,1.0,68,72,1286,1286,3770,0.790962,221,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030358-16891,2026-09-12T03:03:59+00:00,2026-09-12T04:06:51+00:00 +senior-dev,clack-async-autocomplete-options,hard,34,typescript,0,0,0.97561,1.0,80,82,643,643,2430,0.238109,109,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030450-25417,2026-09-12T03:04:51+00:00,2026-09-12T03:45:23+00:00 +senior-dev,langchain-request-coalescing,hard,35,python,0,0,0.94,1.0,47,50,232,232,2980,0.397264,148,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030541-16192,2026-09-12T03:05:41+00:00,2026-09-12T03:55:24+00:00 +senior-dev,httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,2202,0.18555,84,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030633-15624,2026-09-12T03:06:33+00:00,2026-09-12T03:43:18+00:00 +senior-dev,optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,3344,0.601294,183,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030401-17328,2026-09-12T03:04:02+00:00,2026-09-12T03:59:48+00:00 +senior-dev,textual-kitty-key-phases,hard,38,python,0,0,0.652174,1.0,15,23,57,57,3457,0.334924,124,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030453-25823,2026-09-12T03:04:54+00:00,2026-09-12T04:02:34+00:00 +senior-dev,pwntools-tube-multiplexing,medium,39,python,0,0,0.972603,1.0,71,73,1,1,6054,0.412717,142,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030544-16605,2026-09-12T03:05:46+00:00,2026-09-12T04:46:43+00:00 +senior-dev,arktype-json-schema-refs-dependencies,medium,40,typescript,1,1,1.0,1.0,25,25,1679,1679,6117,0.607309,257,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030636-16041,2026-09-12T03:06:36+00:00,2026-09-12T04:48:36+00:00 +senior-dev,dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.835821,1.0,56,67,2035,2035,2136,0.234053,93,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030404-17758,2026-09-12T03:04:05+00:00,2026-09-12T03:39:45+00:00 +senior-dev,kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,2672,0.375667,156,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030456-26227,2026-09-12T03:04:57+00:00,2026-09-12T03:49:32+00:00 +senior-dev,bandit-incremental-cache-control,medium,43,python,0,0,0.988636,1.0,87,88,275,275,1906,0.26697,133,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030547-16957,2026-09-12T03:05:47+00:00,2026-09-12T03:37:36+00:00 +senior-dev,fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,2447,0.328509,157,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030639-16448,2026-09-12T03:06:39+00:00,2026-09-12T03:47:29+00:00 +senior-dev,go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,3212,0.323751,164,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030407-18194,2026-09-12T03:04:08+00:00,2026-09-12T03:57:42+00:00 +senior-dev,igel-persist-feature-schema,medium,46,python,0,0,0.25,1.0,6,24,2,2,1798,0.20074,86,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030459-26659,2026-09-12T03:05:00+00:00,2026-09-12T03:35:01+00:00 +senior-dev,bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,2836,0.375013,146,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030550-17454,2026-09-12T03:05:50+00:00,2026-09-12T03:53:09+00:00 +senior-dev,anko-typed-variable-bindings,medium,48,go,0,1,1.0,1.0,9,9,94,94,2109,0.312068,100,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030642-16885,2026-09-12T03:06:42+00:00,2026-09-12T03:41:53+00:00 +senior-dev,mashumaro-flattened-dataclass-fields,medium,49,python,0,1,1.0,1.0,66,66,30014,30014,3871,0.426588,159,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030410-18605,2026-09-12T03:04:11+00:00,2026-09-12T04:08:44+00:00 +senior-dev,dasel-html-document-format,medium,50,go,0,1,1.0,1.0,146,146,1012,1012,1741,0.176821,106,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030502-27101,2026-09-12T03:05:03+00:00,2026-09-12T03:34:07+00:00 +senior-dev,boa-hierarchical-evaluation-cancellation,medium,51,rust,0,1,1.0,1.0,17,17,7,7,3148,0.535929,180,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030553-17914,2026-09-12T03:05:53+00:00,2026-09-12T03:58:24+00:00 +senior-dev,fastapi-implicit-head-options,medium,52,python,0,0,0.953488,1.0,41,43,3134,3134,3868,0.621404,171,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030645-17314,2026-09-12T03:06:45+00:00,2026-09-12T04:11:17+00:00 +senior-dev,prometheus-typed-label-sorting,medium,53,go,0,1,1.0,1.0,17,17,28,28,3374,0.189636,104,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030413-19038,2026-09-12T03:04:14+00:00,2026-09-12T04:00:32+00:00 +senior-dev,claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,1788,0.192912,67,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030505-27529,2026-09-12T03:05:06+00:00,2026-09-12T03:34:57+00:00 +senior-dev,pebble-durability-wait-apis,medium,55,go,0,1,1.0,1.0,59,59,44,44,5516,0.727753,186,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030556-18337,2026-09-12T03:05:56+00:00,2026-09-12T04:37:55+00:00 +senior-dev,cliffy-config-file-parsing,medium,56,typescript,0,0,0.972973,1.0,36,37,451,451,2229,0.283721,156,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030648-17749,2026-09-12T03:06:48+00:00,2026-09-12T03:44:00+00:00 +senior-dev,sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,2859,0.360942,147,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030416-19454,2026-09-12T03:04:17+00:00,2026-09-12T03:51:58+00:00 +senior-dev,kombu-single-active-consumer-priority,medium,58,python,0,0,0.988235,1.0,84,85,1421,1421,2855,0.391801,121,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030508-27983,2026-09-12T03:05:09+00:00,2026-09-12T03:52:47+00:00 +senior-dev,skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,4664,0.596434,193,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030559-18776,2026-09-12T03:05:59+00:00,2026-09-12T04:23:46+00:00 +senior-dev,scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,2064,0.314286,127,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-030651-18208,2026-09-12T03:06:51+00:00,2026-09-12T03:41:18+00:00 +senior-dev,mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,2317,0.211338,87,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043516-225457,2026-09-12T04:35:17+00:00,2026-09-12T05:13:57+00:00 +senior-dev,tengo-destructuring-bindings,medium,62,go,0,1,1.0,1.0,91,91,132,132,10081,0.361849,190,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052715-235123,2026-09-12T05:27:16+00:00,2026-09-12T08:15:20+00:00 +senior-dev,httpx-multipart-response-parsing,medium,63,python,0,1,1.0,1.0,122,122,1272,1272,2129,0.485142,103,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051525-174406,2026-09-12T05:15:26+00:00,2026-09-12T05:50:58+00:00 +senior-dev,opa-rego-rule-profiling,medium,64,go,0,0,0.96,1.0,24,25,6,6,2506,0.367731,156,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045932-336317,2026-09-12T04:59:33+00:00,2026-09-12T05:41:21+00:00 +senior-dev,ipython-session-bundle-replay,medium,65,python,0,1,1.0,1.0,17,17,29,29,1897,0.182366,107,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043519-225931,2026-09-12T04:35:20+00:00,2026-09-12T05:07:00+00:00 +senior-dev,scriggo-method-declarations,medium,66,go,0,0,0.895833,1.0,43,48,1049,1049,9910,0.971157,634,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052718-235605,2026-09-12T05:27:19+00:00,2026-09-12T08:12:34+00:00 +senior-dev,opa-template-string-reconstruction,medium,67,go,0,1,1.0,1.0,5,5,4,4,4738,1.400465,255,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051528-174893,2026-09-12T05:15:29+00:00,2026-09-12T06:34:31+00:00 +senior-dev,mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2775,0.549513,106,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045935-336817,2026-09-12T04:59:41+00:00,2026-09-12T05:46:00+00:00 +senior-dev,tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,2342,0.292547,104,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043522-226321,2026-09-12T04:35:23+00:00,2026-09-12T05:14:28+00:00 +senior-dev,fd-deterministic-multi-key-sorting,medium,70,rust,1,1,1.0,1.0,43,43,109,109,3129,0.401283,272,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052721-236012,2026-09-12T05:27:22+00:00,2026-09-12T06:19:34+00:00 +senior-dev,cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,3381,0.646877,130,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051531-175340,2026-09-12T05:15:37+00:00,2026-09-12T06:12:02+00:00 +senior-dev,tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,2220,0.246244,96,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045938-336881,2026-09-12T04:59:41+00:00,2026-09-12T05:36:44+00:00 +senior-dev,etree-xml-diff-patch,medium,73,go,0,0,0.980769,1.0,51,52,15,15,3097,0.345063,80,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043525-226733,2026-09-12T04:35:26+00:00,2026-09-12T05:27:06+00:00 +senior-dev,textual-richlog-follow-state,medium,74,python,0,0,0.75,1.0,15,20,6,6,3317,0.194296,179,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052724-236431,2026-09-12T05:27:25+00:00,2026-09-12T06:22:46+00:00 +senior-dev,yaegi-go-embed-directives,medium,75,go,0,0,0.815789,0.948276,31,38,55,58,3629,0.815202,176,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051534-175407,2026-09-12T05:15:37+00:00,2026-09-12T06:16:11+00:00 +senior-dev,kysely-window-grouping-helpers,medium,76,typescript,0,1,1.0,1.0,254,254,22,22,2305,0.384189,171,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045941-337168,2026-09-12T04:59:42+00:00,2026-09-12T05:38:11+00:00 +senior-dev,numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,4804,0.745899,251,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043528-227125,2026-09-12T04:35:29+00:00,2026-09-12T05:55:36+00:00 +senior-dev,valibot-recursive-schema-composition,easy,78,typescript,0,1,1.0,1.0,10,10,209,209,3548,0.281757,203,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052727-236847,2026-09-12T05:27:28+00:00,2026-09-12T06:26:39+00:00 +senior-dev,go-git-worktree-merge-conflicts,easy,79,go,0,0,0.823529,1.0,14,17,2,2,3841,0.895669,151,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051537-175688,2026-09-12T05:15:38+00:00,2026-09-12T06:19:44+00:00 +senior-dev,adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,2832,0.541963,189,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045944-338036,2026-09-12T04:59:45+00:00,2026-09-12T05:47:00+00:00 +senior-dev,obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,0,0.0,0.999117,0,33,1132,1133,3429,0.467006,163,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043531-227526,2026-09-12T04:35:32+00:00,2026-09-12T05:32:44+00:00 +senior-dev,ts-pattern-match-each,easy,82,typescript,0,0,0.0,1.0,0,85,6,6,1260,0.098893,91,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052730-237264,2026-09-12T05:27:31+00:00,2026-09-12T05:48:33+00:00 +senior-dev,anko-default-function-arguments,easy,83,go,0,1,1.0,1.0,2,2,119,119,2777,0.827614,174,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051540-176550,2026-09-12T05:15:41+00:00,2026-09-12T06:02:01+00:00 +senior-dev,dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,6092,0.598381,213,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045947-338470,2026-09-12T04:59:48+00:00,2026-09-12T06:41:24+00:00 +senior-dev,abs-stepped-slices,easy,85,go,0,0,0.5,0.833333,3,6,5,6,1947,0.286674,127,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043534-227942,2026-09-12T04:35:35+00:00,2026-09-12T05:08:05+00:00 +senior-dev,testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,2428,0.10595,117,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052733-237678,2026-09-12T05:27:34+00:00,2026-09-12T06:08:06+00:00 +senior-dev,query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,5096,1.127116,200,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051543-176951,2026-09-12T05:15:44+00:00,2026-09-12T06:40:43+00:00 +senior-dev,wasmi-trap-coredumps,easy,88,rust,0,0,0.909091,1.0,20,22,58,58,4154,1.043595,369,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045950-338912,2026-09-12T04:59:51+00:00,2026-09-12T06:09:08+00:00 +senior-dev,koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,1882,0.246131,119,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043537-228351,2026-09-12T04:35:38+00:00,2026-09-12T05:07:03+00:00 +senior-dev,geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,2796,0.258587,192,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052736-238088,2026-09-12T05:27:36+00:00,2026-09-12T06:14:16+00:00 +senior-dev,arcane-drift-detection-baselines,easy,91,go,0,1,1.0,1.0,82,82,2,2,1925,0.556208,138,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051546-177370,2026-09-12T05:15:47+00:00,2026-09-12T05:47:55+00:00 +senior-dev,abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,1323,0.126811,87,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045953-339339,2026-09-12T04:59:54+00:00,2026-09-12T05:21:59+00:00 +senior-dev,aiomonitor-task-snapshots-diff,easy,93,python,0,0,0.962264,1.0,51,53,8,8,1704,0.338715,127,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043540-228769,2026-09-12T04:35:41+00:00,2026-09-12T05:04:08+00:00 +senior-dev,kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,5920,0.380697,255,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052739-238533,2026-09-12T05:27:40+00:00,2026-09-12T07:06:23+00:00 +senior-dev,vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,3404,0.925801,203,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051549-177810,2026-09-12T05:15:50+00:00,2026-09-12T06:12:38+00:00 +senior-dev,dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,4474,0.711701,272,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045956-339743,2026-09-12T04:59:57+00:00,2026-09-12T06:14:34+00:00 +senior-dev,task-task-graph-export,easy,97,go,0,1,1.0,1.0,20,20,17,17,2406,0.428137,163,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043543-229184,2026-09-12T04:35:44+00:00,2026-09-12T05:15:53+00:00 +senior-dev,go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,4075,0.25624,149,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052742-238980,2026-09-12T05:27:43+00:00,2026-09-12T06:35:40+00:00 +senior-dev,ytt-jsonpath-query-api,easy,99,go,0,1,1.0,1.0,103,103,1,1,1811,0.203172,109,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051552-178249,2026-09-12T05:15:53+00:00,2026-09-12T05:46:08+00:00 +senior-dev,helm-unified-manifest-stream,easy,100,go,1,1,1.0,1.0,5,5,2,2,2474,0.185822,97,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-045959-340160,2026-09-12T05:00:00+00:00,2026-09-12T05:41:18+00:00 +senior-dev,yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,5071,0.659571,225,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043546-229599,2026-09-12T04:35:47+00:00,2026-09-12T06:00:21+00:00 +senior-dev,goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,3166,0.166281,160,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052745-239439,2026-09-12T05:27:46+00:00,2026-09-12T06:20:34+00:00 +senior-dev,psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,1786,0.522373,110,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051555-178702,2026-09-12T05:15:56+00:00,2026-09-12T05:45:45+00:00 +senior-dev,wazero-multi-module-snapshots,easy,104,go,0,0,0.935897,1.0,73,78,2,2,1554,0.101368,64,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-050002-340586,2026-09-12T05:00:03+00:00,2026-09-12T05:26:01+00:00 +senior-dev,sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,3272,0.340386,135,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043549-229995,2026-09-12T04:35:50+00:00,2026-09-12T05:30:24+00:00 +senior-dev,actionlint-action-pinning-lint,easy,106,go,0,1,1.0,1.0,55,55,145,145,1391,0.208596,165,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052748-239953,2026-09-12T05:27:49+00:00,2026-09-12T05:51:03+00:00 +senior-dev,httpx-deterministic-cookie-store,easy,107,typescript,0,0,0.991304,1.0,114,115,1281,1281,3238,0.665674,144,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051558-179129,2026-09-12T05:15:59+00:00,2026-09-12T06:10:01+00:00 +senior-dev,returns-validated-error-accumulation,easy,108,python,0,0,0.0,1.0,0,159,61,61,3367,0.322056,136,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-050005-341008,2026-09-12T05:00:06+00:00,2026-09-12T05:56:16+00:00 +senior-dev,ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,1908,0.114574,53,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043552-230417,2026-09-12T04:35:53+00:00,2026-09-12T05:07:47+00:00 +senior-dev,drizzle-orm-window-function-builders,easy,110,typescript,0,0,0.530769,1.0,69,130,566,566,5209,0.14123,128,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-052751-240835,2026-09-12T05:27:52+00:00,2026-09-12T06:54:45+00:00 +senior-dev,happy-dom-abort-pending-body-reads,easy,111,typescript,0,0,0.071429,1.0,1,14,165,165,3610,0.391544,157,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-051601-179556,2026-09-12T05:16:02+00:00,2026-09-12T06:16:15+00:00 +senior-dev,narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,3995,0.622587,210,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-050008-341457,2026-09-12T05:00:09+00:00,2026-09-12T06:06:48+00:00 +senior-dev,true-myth-iterable-collection-combinators,easy,113,typescript,1,1,1.0,1.0,96,96,561,561,2126,0.425493,123,0,,openrouter/deepseek/deepseek-v4-flash-0731,8ae964c,20260912-043555-230835,2026-09-12T04:35:56+00:00,2026-09-12T05:11:31+00:00 diff --git a/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/README.md b/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/README.md new file mode 100644 index 0000000000..f60dc7e4fc --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/README.md @@ -0,0 +1,91 @@ +# senior-dev `278a076` — DeepSWE 113, DeepSeek V4.1 Flash + +**88/113 solved — 77.9%**, one attempt per task, official verifiers. + +| Metric | Value | +|---|---| +| Solved | **88/113 — 77.9%** (exact 95% CI 69.1%–85.1%) | +| Mean F2P | 0.9549 | +| Mean P2P | 0.9822 | +| Valid grades | 113/113 — no invalid outcomes | +| Model cost | **$56.52 billed** — $0.50 per attempt | +| Mean agent time | 33 min (median 26 min, longest 103 min) | +| Runs that hit the 3 h budget | 0/113 | +| Model calls | 16,951 total, 150 mean | +| First attempt started | 2026-09-14T22:48:36+00:00 | +| Last attempt finished | 2026-09-15T04:53:10+00:00 | + +## Setup + +| | | +|---|---| +| Binary | senior-dev `278a076`, sha256 `1a2f212f4042f57e3cc86c76338731e75a488ad648850ebf6f9af0664978fa69` | +| Binary, re-authored | the same tree is `712d980` after the branch history was re-authored; other documents cite that SHA for this arm | +| Model | `deepseek/deepseek-v4.1-flash` via OpenRouter | +| Routing | pinned to two fp8 1M-context endpoints, `{"only": ["novita", "gmicloud"]}` | +| Compaction | `window` policy, capacity 500,000 | +| Effort | max, 131,072 output-token cap, `--variant max` | +| Budget | 3 h per task, hard timeout 10,920 s | +| Verifiers | official DeepSWE at `0b9fabb` | +| Hosts | four `e2-standard-32`, 6 concurrent containers, 2 CPU / 8 GiB, contained egress | + +## By difficulty + +| Cut | Passes | Rate | +|---|---|---| +| Hard (38 tasks) | 23/38 | 60.5% | +| Medium (38 tasks) | 29/38 | 76.3% | +| Easy (37 tasks) | 36/37 | 97.3% | + +## By language + +| Language | Passes | Rate | +|---|---|---| +| `go` | 32/34 | 94.1% | +| `javascript` | 5/5 | 100.0% | +| `python` | 21/34 | 61.8% | +| `rust` | 4/5 | 80.0% | +| `typescript` | 26/35 | 74.3% | + +Per-language rows are 5–35 tasks wide and separate nothing. + +## The tuned split + +| Cut | Passes | Rate | +|---|---|---| +| Canonical eight (development set) | 7/8 | 87.5% | +| **Excluding the canonical eight** | **81/105 | 77.1%** | + +Removing the eight development tasks moves the headline by 0.7 pp, so the +result is not an artifact of tuning on them. `dev_set` in `tasks.csv` marks which eight. + +## The file + +`tasks.csv` — one row per task: + +| Column | Meaning | +|---|---| +| `task` `band` `difficulty_rank` `language` `dev_set` | which task, and how it is classified | +| `reward` | 1 pass, 0 fail | +| `f2p` `p2p` | fraction of issue tests now passing / pre-existing tests still passing | +| `f2p_passed` `f2p_total` `p2p_passed` `p2p_total` | the underlying test counts | +| `started` `finished` `seconds` | when the attempt ran and how long it took | +| `cost_usd` `model_calls` | per-attempt model spend and call count | +| `input_tokens` `output_tokens` `cache_read_tokens` `reasoning_tokens` | usage | +| `exit` `patch_files` `patch_bytes` | how it ended and what it wrote | +| `model` `harness_version` `attempt_id` | what produced it | + +**$56.52 is the billed figure** and the one quoted: the dedicated key's counter +moved $534.28 → $590.80 over the sweep. The per-attempt `cost_usd` column sums to +$34.83 instead, because that telemetry prices tokens at model-level list rates +(models.dev, DeepSeek first-party) rather than at what the two pinned endpoints +actually charge. Treat the column as a relative measure across tasks, not as spend. + +## Source + +Published as the first row of the DeepSeek V4.1 Flash table in +[Coding Agent Benchmark Baselines](https://app.notion.com/p/Coding-Agent-Benchmark-Baselines-DeepSWE-Terminal-Bench-Aider-3ce03ed71f108147b0bdf596b800a731#50cc22a0368e4931a4139628001678b3). + +The per-attempt artifacts behind these rows — `events.ndjson`, `run.log`, traces, +verifier stdout, patches — are about 3.7 GB and stay on the machines that produced +them. This run predates the rename; the values here are rewritten from `codeaf`. diff --git a/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/tasks.csv b/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/tasks.csv new file mode 100644 index 0000000000..ed63650c0a --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-15-senior-dev-278a076-v41-flash/tasks.csv @@ -0,0 +1,114 @@ +task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,started,finished,seconds,cost_usd,model_calls,input_tokens,output_tokens,cache_read_tokens,reasoning_tokens,exit,patch_files,patch_bytes,model,harness_version,attempt_id +obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,2026-09-14T22:48:36+00:00,2026-09-14T23:11:05+00:00,1344,0.17841,107,335357,24272,14815360,115162,0,5,33491,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224835-5102 +gql-incremental-graphql-delivery,hard,2,python,0,0,1.0,0.998767,17,17,810,811,2026-09-14T22:49:11+00:00,2026-09-14T23:13:06+00:00,1430,0.27624,165,603118,47114,29875200,113131,0,11,61512,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224910-4557 +bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,2026-09-14T22:49:22+00:00,2026-09-14T23:11:03+00:00,1295,0.199394,116,474661,28355,14637184,112117,0,8,42617,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224921-4443 +termenv-preserve-ansi-resets,hard,4,go,0,1,1.0,1.0,35,35,87,87,2026-09-14T22:49:54+00:00,2026-09-14T23:04:06+00:00,846,0.143196,67,200927,23755,7949440,124926,0,8,39118,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224953-4892 +updo-policy-alerting,hard,5,go,0,1,1.0,1.0,17,17,123,123,2026-09-14T22:48:39+00:00,2026-09-14T23:05:57+00:00,1033,0.146282,84,270715,26355,9547776,102031,0,6,29127,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224838-5510 +effect-sse-httpapi-streaming,hard,6,typescript,0,1,1.0,1.0,47,47,70,70,2026-09-14T22:49:14+00:00,2026-09-14T23:28:57+00:00,2374,0.282706,182,662992,43814,29606784,113581,0,9,53984,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224913-4976 +oxvg-structural-selector-preservation,hard,7,rust,0,1,1.0,1.0,6,6,62,62,2026-09-14T22:49:25+00:00,2026-09-14T23:42:45+00:00,3194,0.325117,206,804857,44941,35277568,119318,0,13,34069,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224924-4848 +sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.9375,0.998429,30,32,1271,1273,2026-09-14T22:49:57+00:00,2026-09-14T23:20:14+00:00,1810,0.341941,137,962626,36983,24249728,171013,0,5,37661,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224956-5314 +helm-array-merge-strategies,hard,9,go,0,1,1.0,1.0,47,47,12,12,2026-09-14T22:48:42+00:00,2026-09-14T23:42:12+00:00,3201,0.345764,229,722936,58387,38958336,142360,0,19,83839,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224841-5925 +meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.918367,1.0,45,49,51469,51469,2026-09-14T22:49:17+00:00,2026-09-14T23:25:15+00:00,2152,0.305296,230,663250,44475,36864256,114218,0,10,63297,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224916-5354 +pest-character-class-coalescing,hard,11,rust,0,0,0.971154,1.0,101,104,250,250,2026-09-14T22:49:28+00:00,2026-09-14T23:21:16+00:00,1901,0.208291,142,468316,28526,18688640,108104,0,5,28517,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224927-5244 +testem-bail-on-test-failure,hard,12,javascript,0,1,1.0,1.0,90,90,489,489,2026-09-14T22:50:00+00:00,2026-09-14T23:27:16+00:00,2231,0.325929,177,828642,44221,32997376,126846,0,20,41032,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224959-5724 +koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,2026-09-14T22:48:45+00:00,2026-09-14T23:18:29+00:00,1778,0.313278,202,628817,56799,30937088,153442,0,30,116893,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224844-6316 +prometheus-transactional-reload-status,hard,14,typescript,0,0,0.0,0.0,0,15,0,82,2026-09-14T22:49:20+00:00,2026-09-14T23:28:43+00:00,2357,0.145126,145,301143,33096,16258432,52202,0,18,75729,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224919-5783 +obsidian-linter-link-format-conversion,hard,15,typescript,0,1,1.0,1.0,60,60,1131,1131,2026-09-14T22:49:31+00:00,2026-09-14T23:09:19+00:00,1180,0.176976,98,442209,24303,12451840,97846,0,6,36002,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224930-5669 +koota-query-predicates,hard,16,typescript,0,-1,0,0,,,,,2026-09-14T22:50:03+00:00,2026-09-14T23:17:08+00:00,1619,0.295766,159,643604,35731,29824512,147188,0,17,68563,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-225002-6137 +quill-shared-toolbar-focus,hard,17,typescript,0,0,0.846154,1.0,11,13,22,22,2026-09-14T22:48:48+00:00,2026-09-14T23:20:25+00:00,1891,0.251398,159,443748,35579,26848640,138238,0,6,42144,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224847-6751 +koota-deferred-mutation-buffer,hard,18,typescript,0,1,1.0,1.0,71,71,128,128,2026-09-14T22:49:23+00:00,2026-09-14T23:15:08+00:00,1540,0.247655,146,409238,47775,24655616,139395,0,10,107861,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224922-6189 +kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.881579,1.0,67,76,1412,1412,2026-09-14T22:49:34+00:00,2026-09-14T23:12:20+00:00,1361,0.218943,144,394577,39412,22739328,113152,0,6,47327,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224933-6086 +ink-grid-box-layout,hard,20,typescript,0,0,0.88,1.0,22,25,49,49,2026-09-14T22:50:06+00:00,2026-09-14T23:26:50+00:00,2200,0.255808,134,685068,41625,22833536,99287,0,7,33974,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-225005-6552 +awilix-async-container-initialization,hard,21,typescript,0,1,1.0,1.0,24,24,162,162,2026-09-14T22:48:51+00:00,2026-09-14T23:05:03+00:00,966,0.140441,100,226311,30461,10604288,94008,0,5,55187,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224850-7183 +happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,0,0.928571,1.0,13,14,9,9,2026-09-14T22:49:27+00:00,2026-09-14T23:33:19+00:00,2626,0.182088,129,295300,31999,16105728,117128,0,5,41777,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224925-6633 +csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,2026-09-14T22:49:37+00:00,2026-09-14T23:06:13+00:00,990,0.18804,90,410309,31621,11105664,123674,0,3,48843,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-224936-6503 +superjson-error-stack-serialization,hard,24,typescript,0,0,0.9875,1.0,79,80,116,116,2026-09-14T22:50:09+00:00,2026-09-14T23:02:57+00:00,763,0.133737,75,243669,24889,8028416,96947,0,9,47973,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-225008-6992 +kea-atomic-signal-selectors,hard,25,typescript,0,1,1.0,1.0,12,12,139,139,2026-09-14T23:47:41+00:00,2026-09-15T01:16:22+00:00,5315,0.320307,177,651428,45800,33072384,159826,0,8,65836,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234740-112167 +vulture-persistent-analysis-cache,hard,26,python,0,1,1.0,1.0,24,24,295,295,2026-09-14T23:36:58+00:00,2026-09-15T00:01:45+00:00,1482,0.351711,158,1171130,44017,25003904,124366,0,8,50924,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233657-82587 +katex-multicolumn-array-spans,hard,27,javascript,0,1,1.0,1.0,94,94,599,599,2026-09-14T23:51:47+00:00,2026-09-15T00:18:16+00:00,1582,0.322242,158,892180,34624,28400256,137399,0,7,35507,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235146-123485 +expr-try-catch-errors,hard,28,go,0,1,1.0,1.0,79,79,66265,66265,2026-09-14T23:59:10+00:00,2026-09-15T00:35:23+00:00,2168,0.335417,180,859335,43734,34337280,128775,0,15,43834,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235909-133049 +koota-composite-trait-aspects,hard,29,typescript,0,1,1.0,1.0,51,51,172,172,2026-09-14T23:47:44+00:00,2026-09-15T00:26:03+00:00,2292,0.434879,206,1296690,70583,30821120,175938,0,20,159420,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234743-112647 +onedump-dump-encryption-pipeline,hard,30,go,0,1,1.0,1.0,82,82,6,6,2026-09-14T23:37:01+00:00,2026-09-14T23:57:58+00:00,1251,0.166538,118,291626,41402,14106880,92721,0,14,58120,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233700-83068 +eicrud-keyset-pagination-cursor,hard,31,typescript,0,1,1.0,1.0,14,14,168,168,2026-09-14T23:51:50+00:00,2026-09-15T00:24:58+00:00,1983,0.16548,143,461979,24111,14375552,64316,0,8,27867,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235149-123934 +participle-grammar-conflict-analysis,hard,32,go,0,1,1.0,1.0,91,91,153,153,2026-09-14T23:59:13+00:00,2026-09-15T00:15:14+00:00,956,0.13789,62,294497,19729,6965760,101634,0,4,31865,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235912-133528 +python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,2026-09-14T23:47:47+00:00,2026-09-15T00:28:10+00:00,2417,0.414965,241,1204058,59459,45278976,104740,0,19,76304,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234746-113058 +clack-async-autocomplete-options,hard,34,typescript,0,0,0.97561,1.0,80,82,643,643,2026-09-14T23:37:04+00:00,2026-09-14T23:55:21+00:00,1092,0.197498,110,567010,34593,12171904,91958,0,7,57341,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233703-83465 +langchain-request-coalescing,hard,35,python,0,1,1.0,1.0,50,50,232,232,2026-09-14T23:51:53+00:00,2026-09-15T00:18:04+00:00,1561,0.311529,157,837608,46977,24761856,139027,0,5,80934,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235152-124347 +httpx-streaming-json-iteration,hard,36,python,0,1,1.0,1.0,108,108,1404,1404,2026-09-14T23:59:16+00:00,2026-09-15T00:25:50+00:00,1589,0.143475,114,254887,27754,12061184,87344,0,7,31616,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235915-133933 +optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,2026-09-14T23:47:50+00:00,2026-09-15T00:17:07+00:00,1751,0.414378,224,1305208,54711,35122816,134003,0,8,67797,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234749-113470 +textual-kitty-key-phases,hard,38,python,0,0,0.782609,1.0,18,23,57,57,2026-09-14T23:37:07+00:00,2026-09-15T00:17:37+00:00,2421,0.26218,128,851500,28000,16583168,113175,0,6,27950,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233706-83869 +pwntools-tube-multiplexing,medium,39,python,0,0,0.986301,1.0,72,73,1,1,2026-09-14T23:51:56+00:00,2026-09-15T00:17:45+00:00,1542,0.228795,98,655025,31920,12813312,121582,0,6,38484,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235155-124747 +arktype-json-schema-refs-dependencies,medium,40,typescript,1,1,1.0,1.0,25,25,1679,1679,2026-09-14T23:59:19+00:00,2026-09-15T00:38:03+00:00,2317,0.333204,234,715690,58190,37525760,130598,0,14,36681,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235918-134336 +dateutil-rfc5545-timezone-interop,medium,41,python,0,0,0.970149,1.0,65,67,2035,2035,2026-09-14T23:47:53+00:00,2026-09-15T00:15:40+00:00,1661,0.311378,137,867593,48445,22777728,139731,0,2,51508,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234752-113886 +kgateway-consistent-hash-policy,medium,42,go,0,0,0.0,1.0,0,2,214,214,2026-09-14T23:37:10+00:00,2026-09-15T00:22:31+00:00,2713,0.324417,172,1068825,34305,29174784,93310,0,11,71329,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233709-84258 +bandit-incremental-cache-control,medium,43,python,0,1,1.0,1.0,88,88,275,275,2026-09-14T23:51:59+00:00,2026-09-15T00:17:26+00:00,1521,0.249217,162,677713,55908,22527360,77389,0,11,81747,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235158-125139 +fastapi-deprecation-response-headers,medium,44,python,0,0,0.992701,1.0,136,137,3134,3134,2026-09-14T23:59:22+00:00,2026-09-15T00:20:09+00:00,1239,0.177744,122,455540,29873,13247488,86245,0,6,94578,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235921-134723 +go-critic-doc-link-checker,medium,45,go,0,1,1.0,1.0,3,3,16,16,2026-09-14T23:47:56+00:00,2026-09-15T00:15:30+00:00,1647,0.257278,132,711480,24227,18135936,136020,0,6,16498,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-234755-114301 +igel-persist-feature-schema,medium,46,python,0,0,0.916667,1.0,22,24,2,2,2026-09-14T23:37:13+00:00,2026-09-15T00:02:09+00:00,1491,0.270662,128,869743,35922,16964224,112925,0,5,31492,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-233712-84667 +bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,2026-09-14T23:52:03+00:00,2026-09-15T00:13:57+00:00,1309,0.23343,127,643851,47674,16375936,98533,0,14,61769,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235201-125589 +anko-typed-variable-bindings,medium,48,go,0,1,1.0,1.0,9,9,94,94,2026-09-14T23:59:25+00:00,2026-09-15T00:24:36+00:00,1505,0.243446,149,462344,29473,22261504,149377,0,9,110840,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260914-235924-135117 +mashumaro-flattened-dataclass-fields,medium,49,python,0,0,0.984848,1.0,65,66,30014,30014,2026-09-15T01:18:43+00:00,2026-09-15T01:54:07+00:00,2118,0.41329,144,1593851,41510,24030336,128692,0,6,53231,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011842-238516 +dasel-html-document-format,medium,50,go,0,1,1.0,1.0,146,146,1012,1012,2026-09-15T00:31:50+00:00,2026-09-15T00:46:14+00:00,859,0.171339,95,454669,27404,11387904,87555,0,8,47371,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003149-192205 +boa-hierarchical-evaluation-cancellation,medium,51,rust,0,1,1.0,1.0,17,17,7,7,2026-09-15T00:31:29+00:00,2026-09-15T02:15:30+00:00,6232,0.285881,193,548338,43338,33888896,126602,0,12,80014,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003128-155909 +fastapi-implicit-head-options,medium,52,python,0,1,1.0,1.0,43,43,3134,3134,2026-09-15T00:44:03+00:00,2026-09-15T01:15:00+00:00,1852,0.299876,183,677392,37668,29307008,146243,0,5,70216,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004402-232112 +prometheus-typed-label-sorting,medium,53,go,0,1,1.0,1.0,17,17,28,28,2026-09-15T01:18:46+00:00,2026-09-15T02:19:11+00:00,3615,0.289268,192,537991,39160,31973376,148588,0,9,39412,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011845-238986 +claude-code-by-agents-recursive-delegation,medium,54,typescript,0,1,1.0,1.0,7,7,31,31,2026-09-15T00:31:53+00:00,2026-09-15T00:44:18+00:00,741,0.152193,78,400059,21112,8991232,87572,0,4,35335,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003152-192700 +pebble-durability-wait-apis,medium,55,go,0,0,0.983051,1.0,58,59,44,44,2026-09-15T00:31:32+00:00,2026-09-15T01:44:14+00:00,4352,0.458806,194,1712363,40624,33962496,126150,0,11,63151,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003131-156365 +cliffy-config-file-parsing,medium,56,typescript,0,1,1.0,1.0,37,37,451,451,2026-09-15T00:44:06+00:00,2026-09-15T01:10:48+00:00,1597,0.274881,204,696547,37228,26842496,112557,0,38,57206,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004405-232613 +sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,2026-09-15T01:18:49+00:00,2026-09-15T01:42:58+00:00,1442,0.210295,137,356620,45166,21023104,111055,0,7,95664,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011848-239361 +kombu-single-active-consumer-priority,medium,58,python,0,0,0.976471,1.0,83,85,1421,1421,2026-09-15T00:31:56+00:00,2026-09-15T00:53:00+00:00,1259,0.217508,124,497908,34159,18018048,113787,0,4,56819,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003155-193109 +skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,2026-09-15T00:31:35+00:00,2026-09-15T01:40:58+00:00,4156,0.202611,129,519459,35677,17681152,83737,0,10,46123,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003134-156760 +scc-bounded-memory-spilling,medium,60,go,0,1,1.0,1.0,31,31,286,286,2026-09-15T00:44:09+00:00,2026-09-15T01:05:55+00:00,1301,0.203462,112,516914,34864,16990848,90057,0,6,51718,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004408-233019 +mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,2026-09-15T01:18:52+00:00,2026-09-15T01:41:52+00:00,1374,0.177529,92,423043,35461,11176832,98775,0,5,59919,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011851-239795 +tengo-destructuring-bindings,medium,62,go,0,1,1.0,1.0,91,91,132,132,2026-09-15T00:31:59+00:00,2026-09-15T00:50:40+00:00,1116,0.205169,120,357633,36781,20081664,115351,0,7,37531,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003158-193530 +httpx-multipart-response-parsing,medium,63,python,0,0,0.991803,1.0,121,122,1272,1272,2026-09-15T00:31:38+00:00,2026-09-15T00:59:15+00:00,1648,0.256105,145,603983,31224,21324032,138001,0,6,27525,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003137-157195 +opa-rego-rule-profiling,medium,64,go,0,1,1.0,1.0,25,25,6,6,2026-09-15T00:44:12+00:00,2026-09-15T01:12:33+00:00,1687,0.178495,141,408779,27325,16723968,84352,0,7,34638,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004411-233417 +ipython-session-bundle-replay,medium,65,python,0,1,1.0,1.0,17,17,29,29,2026-09-15T01:18:55+00:00,2026-09-15T01:38:04+00:00,1143,0.126565,86,303191,25145,7560064,72198,0,3,55155,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011854-240182 +scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,2026-09-15T00:32:02+00:00,2026-09-15T01:18:24+00:00,2772,0.526632,375,1308960,71006,53068928,214129,0,19,56601,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003201-193945 +opa-template-string-reconstruction,medium,67,go,0,1,1.0,1.0,5,5,4,4,2026-09-15T00:31:41+00:00,2026-09-15T01:37:03+00:00,3904,0.466437,282,1314528,69380,38518016,186793,0,4,29098,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003140-157629 +mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2026-09-15T00:44:14+00:00,2026-09-15T01:05:04+00:00,1244,0.236894,114,549188,40619,16881152,132502,0,4,71412,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004414-233849 +tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,2026-09-15T01:18:58+00:00,2026-09-15T01:39:04+00:00,1201,0.219665,106,532021,30856,16908288,117706,0,5,42053,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011857-240587 +fd-deterministic-multi-key-sorting,medium,70,rust,1,1,1.0,1.0,43,43,109,109,2026-09-15T00:32:05+00:00,2026-09-15T00:50:07+00:00,1078,0.177072,137,445739,32218,15458304,74176,0,9,48698,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003204-194348 +cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,2026-09-15T00:31:44+00:00,2026-09-15T00:56:28+00:00,1479,0.199073,136,464373,30599,17763328,96279,0,7,45362,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-003143-158032 +tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,2026-09-15T00:44:18+00:00,2026-09-15T01:06:57+00:00,1352,0.293842,112,829524,57171,17613696,137116,0,4,43210,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-004417-234268 +etree-xml-diff-patch,medium,73,go,0,1,1.0,1.0,52,52,15,15,2026-09-15T02:24:31+00:00,2026-09-15T02:47:53+00:00,1396,0.479305,116,472302,53335,16876416,143628,0,5,66161,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022430-287261 +textual-richlog-follow-state,medium,74,python,0,1,1.0,1.0,20,20,6,6,2026-09-15T01:21:46+00:00,2026-09-15T02:19:37+00:00,3460,0.304144,215,586579,47165,34628864,139953,0,7,39855,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012145-268051 +yaegi-go-embed-directives,medium,75,go,0,1,1.0,1.0,38,38,58,58,2026-09-15T02:38:44+00:00,2026-09-15T03:27:24+00:00,2911,0.558796,201,616446,40037,30338688,119821,0,10,38499,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023843-318837 +kysely-window-grouping-helpers,medium,76,typescript,0,1,1.0,1.0,254,254,22,22,2026-09-15T01:18:54+00:00,2026-09-15T01:42:21+00:00,1400,0.204426,125,505814,47459,14876032,92417,0,21,68453,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011853-342001 +numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,2026-09-15T02:24:34+00:00,2026-09-15T03:39:50+00:00,4508,0.549845,190,577260,48580,30433280,113143,0,5,55889,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022433-287749 +valibot-recursive-schema-composition,easy,78,typescript,0,1,1.0,1.0,10,10,209,209,2026-09-15T01:21:49+00:00,2026-09-15T02:29:39+00:00,4060,0.367759,202,828326,77704,31915520,168568,0,13,48965,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012148-268511 +go-git-worktree-merge-conflicts,easy,79,go,0,1,1.0,1.0,17,17,2,2,2026-09-15T02:38:47+00:00,2026-09-15T03:32:52+00:00,3237,0.654086,172,756592,69179,26034304,156573,0,6,67375,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023846-319290 +adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,2026-09-15T01:18:57+00:00,2026-09-15T01:50:49+00:00,1905,0.331335,196,740057,52849,33497216,146876,0,8,27932,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011856-342456 +obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,1,1.0,1.0,33,33,1133,1133,2026-09-15T02:24:37+00:00,2026-09-15T02:52:27+00:00,1665,0.548297,136,661103,32892,23872512,139384,0,4,24345,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022436-288137 +ts-pattern-match-each,easy,82,typescript,0,1,1.0,1.0,85,85,6,6,2026-09-15T01:21:52+00:00,2026-09-15T01:37:04+00:00,908,0.144245,95,447683,23681,8373120,62941,0,5,35562,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012151-268917 +anko-default-function-arguments,easy,83,go,0,1,1.0,1.0,2,2,119,119,2026-09-15T02:38:50+00:00,2026-09-15T03:11:18+00:00,1940,0.688632,166,863676,41277,30004608,166641,0,10,33871,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023849-319706 +dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,2026-09-15T01:19:00+00:00,2026-09-15T02:48:36+00:00,5369,0.430102,285,1177339,74635,42461952,135557,0,73,118151,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011859-342868 +abs-stepped-slices,easy,85,go,0,1,1.0,1.0,6,6,6,6,2026-09-15T02:24:40+00:00,2026-09-15T02:42:29+00:00,1063,0.408615,109,573044,26578,12600576,107671,0,3,17983,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022439-288606 +testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,2026-09-15T01:21:55+00:00,2026-09-15T01:44:15+00:00,1336,0.132697,107,324161,26070,9853952,64782,0,13,42784,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012154-269327 +query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2026-09-15T02:38:53+00:00,2026-09-15T03:45:19+00:00,3972,0.579237,213,620016,40464,33331712,120571,0,10,45361,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023852-320127 +wasmi-trap-coredumps,easy,88,rust,0,1,1.0,1.0,22,22,58,58,2026-09-15T01:19:03+00:00,2026-09-15T02:03:50+00:00,2675,0.405977,260,1113470,59718,38391424,146586,0,15,87327,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011902-343275 +koota-entity-snapshot-rollback,easy,89,python,0,0,0.988095,1.0,83,84,47,47,2026-09-15T02:24:43+00:00,2026-09-15T02:44:28+00:00,1179,0.321883,122,284566,31207,14804608,91864,0,12,84660,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022442-289019 +geo-shapeindex-serialization,easy,90,go,0,1,1.0,1.0,24,24,599,599,2026-09-15T01:21:58+00:00,2026-09-15T01:55:35+00:00,2011,0.235831,151,410020,37657,28024576,112766,0,13,59974,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012157-269772 +arcane-drift-detection-baselines,easy,91,go,0,1,1.0,1.0,82,82,2,2,2026-09-15T02:38:56+00:00,2026-09-15T03:19:26+00:00,2421,0.374585,142,476210,43088,17517696,62425,0,21,93623,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023855-320527 +abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,2026-09-15T01:19:06+00:00,2026-09-15T01:35:29+00:00,978,0.140437,117,252284,29003,13636864,73804,0,5,34173,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011905-343697 +aiomonitor-task-snapshots-diff,easy,93,python,0,1,1.0,1.0,53,53,8,8,2026-09-15T02:24:46+00:00,2026-09-15T02:41:55+00:00,1025,0.341484,118,358565,33645,14227200,90148,0,11,61477,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-022445-289490 +kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,2026-09-15T01:22:01+00:00,2026-09-15T01:54:04+00:00,1919,0.149386,77,292839,32374,8569728,100545,0,4,65032,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-012200-270218 +vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,2026-09-15T02:38:59+00:00,2026-09-15T03:25:38+00:00,2788,0.414096,140,559858,43788,16299264,79831,0,18,89212,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023858-320926 +dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,2026-09-15T01:19:09+00:00,2026-09-15T02:16:01+00:00,3407,0.377402,283,862829,60914,47683712,113964,0,55,105855,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-011908-344145 +task-task-graph-export,easy,97,go,0,1,1.0,1.0,20,20,17,17,2026-09-15T03:50:20+00:00,2026-09-15T04:14:43+00:00,1456,0.431798,142,556240,27458,19274496,96941,0,5,25318,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035019-333299 +go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,2026-09-15T02:31:51+00:00,2026-09-15T02:54:18+00:00,1342,0.35577,137,321608,35748,17190272,94374,0,6,52461,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023150-357869 +ytt-jsonpath-query-api,easy,99,go,0,1,1.0,1.0,103,103,1,1,2026-09-15T03:56:33+00:00,2026-09-15T04:09:41+00:00,781,0.269545,86,295585,34978,7182336,79835,0,6,62880,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035632-555189 +helm-unified-manifest-stream,easy,100,go,1,1,1.0,1.0,5,5,2,2,2026-09-15T02:53:01+00:00,2026-09-15T03:40:27+00:00,2838,0.531825,181,602166,33711,29895552,109457,0,30,60918,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-025300-448635 +yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,2026-09-15T03:50:23+00:00,2026-09-15T04:11:35+00:00,1268,0.449034,133,507910,35904,20835968,107134,0,11,49984,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035022-333758 +goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,2026-09-15T02:31:54+00:00,2026-09-15T03:15:25+00:00,2602,0.511177,160,648655,39261,24852480,100294,0,10,24650,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023153-358343 +psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,2026-09-15T03:56:36+00:00,2026-09-15T04:14:19+00:00,1058,0.340532,105,441947,34160,11696384,80648,0,7,44346,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035635-555639 +wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,2026-09-15T02:53:04+00:00,2026-09-15T03:16:54+00:00,1411,0.330448,94,387458,37132,9656192,93096,0,8,54856,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-025303-449082 +sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,2026-09-15T03:50:27+00:00,2026-09-15T04:10:45+00:00,1213,0.390635,118,551456,20892,13191040,100818,0,9,17850,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035025-334184 +actionlint-action-pinning-lint,easy,106,go,0,1,1.0,1.0,55,55,145,145,2026-09-15T02:31:57+00:00,2026-09-15T02:55:19+00:00,1396,0.473186,155,519277,52636,24054400,91594,0,15,70557,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023156-358738 +httpx-deterministic-cookie-store,easy,107,typescript,0,1,1.0,1.0,115,115,1281,1281,2026-09-15T03:56:39+00:00,2026-09-15T04:22:21+00:00,1537,0.36126,111,429944,34580,12378880,97090,0,7,55663,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035638-556034 +returns-validated-error-accumulation,easy,108,python,0,1,1.0,1.0,159,159,61,61,2026-09-15T02:53:07+00:00,2026-09-15T03:18:46+00:00,1533,0.415574,126,530936,29617,17234176,97790,0,8,36479,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-025306-449508 +ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,2026-09-15T03:50:29+00:00,2026-09-15T04:03:14+00:00,760,0.221018,77,194644,24274,7067264,75910,0,6,47412,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035028-334601 +drizzle-orm-window-function-builders,easy,110,typescript,0,1,1.0,1.0,130,130,566,566,2026-09-15T02:32:00+00:00,2026-09-15T03:01:05+00:00,1736,0.370131,121,526299,31901,15169792,69118,0,20,59138,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-023159-359148 +happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,2026-09-15T03:56:42+00:00,2026-09-15T04:53:10+00:00,3380,0.606204,229,578817,51264,39198592,113209,0,11,36668,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035641-556479 +narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,2026-09-15T02:53:10+00:00,2026-09-15T03:35:36+00:00,2539,0.728832,231,1080518,58727,35964160,98683,0,21,80396,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-025309-449917 +true-myth-iterable-collection-combinators,easy,113,typescript,1,1,1.0,1.0,96,96,561,561,2026-09-15T03:50:32+00:00,2026-09-15T04:10:17+00:00,1180,0.425645,116,428707,32605,18586240,121991,0,5,61579,openrouter/deepseek/deepseek-v4.1-flash,278a076b,20260915-035031-335008 diff --git a/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/README.md b/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/README.md new file mode 100644 index 0000000000..a42af110dd --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/README.md @@ -0,0 +1,98 @@ +# senior-dev `f3b9716` — DeepSWE 113, Kimi K3 + +**78/113 solved — 69.0%**, one attempt per task, official verifiers. + +The question was whether the harness holds up when the backend model is swapped to +Kimi K3, with everything the rig can pin held at the values of the +[V4.1 Flash run](../2026-09-15-senior-dev-278a076-v41-flash/). + +| Metric | Value | +|---|---| +| Solved | **78/113 — 69.0%** (exact Clopper-Pearson 95% CI 59.6%–77.4%) | +| Mean F2P | 0.9485 | +| Mean P2P | 0.9876 | +| Valid grades | 113/113 — no invalid outcomes | +| Model cost | **$562.47** billed — $4.98 per attempt | +| Mean agent time | 58 min (longest 172 min) | +| Runs that hit the 3 h budget | 0/113 | +| Model calls | 11,311 total | +| Ran | 2026-09-22, launched 14:37Z, last attempt 17:30Z, graded 17:53Z | + +## By difficulty + +| Cut | Passes | Rate | +|---|---|---| +| Hard (38 tasks) | 12/38 | 31.6% | +| Medium (38 tasks) | 31/38 | 81.6% | +| Easy (37 tasks) | 35/37 | 94.6% | + +## Setup + +| | | +|---|---| +| Binary | senior-dev `f3b9716` (branch `zeropoint95/improvements`), linux/amd64, SHA-256 `3f570d77d2f4d1bb07cdcf50360f32cd9e7ff4830dff3595cf5301439532a104` | +| Coder prompt | `prompt_sha256` `32cda36e…`, identical on all 113 attempts | +| Model | `openrouter/moonshotai/kimi-k3`, reasoning effort max | +| Routing | `{"only": ["modal", "moonshotai"]}` — the only two Kimi K3 endpoints sharing one quantisation (mxfp4), price, 1,048,576 context and an output cap above 131,072, both supporting tools | +| Compaction | window policy, 500,000 / 300,000 / 200,000, 60,000 tail | +| Output cap / budget | 131,072 tokens per call; 3 h agent cap, 10,920 s watchdog | +| Population | the frozen 113-task list, hash-pinned, four shards (29/28/28/28), one attempt per task, one seed | +| Verifiers | official DeepSWE at `0b9fabb` | +| Substrate | four `n2-standard-64`, `us-central1-a`, 2 CPU / 8 GiB per container, contained egress, 113 containers concurrent | + +**Delivery was clean.** All 113 attempts passed the delivery gate — routing, +compaction watermarks, effort, model id, output cap, a single campaign-wide prompt +hash, and the control-plane record. Zero rate limits, zero attempts ended by +anything external, zero empty patches, no upstream outside the allowlist, and no +run reached the 3 h cap. + +## The comparison, and why it resolves nothing + +**The intended control was never run.** A DeepSeek V4.1 Flash re-anchor on this same +binary would have isolated the model; it was deprioritised. The only available +reference is the [88/113 V4.1 Flash run](../2026-09-15-senior-dev-278a076-v41-flash/), +which used a **different model**, a **different binary** and a **different coder +prompt** (`008bd219…` against `32cda36e…`, the rename having moved six lines of +`coder.md`). Those three differences are not separable. + +| Reading | Result | +|---|---| +| Unpaired | 69.0% [59.6, 77.4] against 77.9% [69.1, 85.1] — intervals **overlap**, not resolved | +| Paired, 112 tasks valid in both | 77 against 88; both pass 65, neither 12, discordant 12 / 23; exact McNemar **two-sided p = 0.0895** — not resolved | + +**No resolved difference, on either reading.** The preregistered decision rule is +the two-sided test. A one-sided p of 0.0448 exists and was registered as +supplementary; it is not the decision rule, and reading the result off it would be +choosing the test after seeing which one clears 0.05. This is also not a finding +that the models are equivalent, and it is not a ranking. + +## Honest limits + +1. **No re-anchor exists**, so nothing here isolates the model from the binary and the coder prompt. +2. **One seed, one attempt per task** — no within-configuration variance estimate. Four runs of one identical configuration previously measured **SD 0.183** mean-of-means, so some discordance would appear pairing an arm against *itself*; McNemar cannot see that. +3. **A ~10-point difference is not resolvable at n = 113.** Two arms must differ by roughly fifteen points before their exact intervals separate. +4. **Concurrency and host type were not held constant** — 113-way on `n2-standard-64` here, 24-way and 60-way on `e2-standard-32` in the references. Neither changes what is sent to the model. +5. **The earlier Kimi K3 campaign is not a control.** It scored **77/113**, three era lines back (binary `4bfb927`, default routing, `adopt_image_tree` off). Some documents cite it as 78/113; the store and that campaign's own result both say 77. +6. **The serving endpoint per call is not recorded** by this binary. +7. **No per-task claims** — per-task behaviour on this corpus is bimodal. +8. **This result exists because another arm was stopped for it.** A companion GLM 5.3 lane was killed mid-run at 15:26:32Z to free the shared credit pool. That lane has no corpus rate and must not be quoted: its partial salvage is length-biased and excludes the 58 longest tasks. + +## The file + +`tasks.csv` — one row per task, same columns as the other runs in `runs/`: reward, +F2P, P2P and the underlying test counts, `started` / `finished` / `seconds`, +`cost_usd`, `model_calls`, token counts, `exit`, `patch_files`, `patch_bytes`, and +the model, binary and attempt id, alongside the task's band, difficulty rank, +language and development-set flag. + +## Source + +Preregistration and amendments: `PREREGISTRATION-full113-kimi-k3-sweep.md` at commit +`7a509fa` on branch `kimi-k3-sweep/full113-kimi-k3`. Manifest: +`full113/manifest-kimi-k3-sweep.json`. Profile +`deepswe-full113-senior-dev-kimi-k3-sweep-v2`, labels +`full113-kimi-k3-sweep-s1r2-shard{1..4}-wave1`. Reproduce the headline with +`FULL113_MANIFEST=manifest-kimi-k3-sweep.json ./full113/report.py --require-complete`. + +The per-attempt artifacts stay on the machines that produced them. This campaign ran +while the binary was called `swe-pro`; the values here are rewritten to `senior-dev`. diff --git a/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/tasks.csv b/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/tasks.csv new file mode 100644 index 0000000000..fa5304e3ed --- /dev/null +++ b/docs/benchmarks/deepswe/2026-09-22-senior-dev-f3b9716-kimi-k3/tasks.csv @@ -0,0 +1,114 @@ +task,band,difficulty_rank,language,dev_set,reward,f2p,p2p,f2p_passed,f2p_total,p2p_passed,p2p_total,started,finished,seconds,cost_usd,model_calls,input_tokens,output_tokens,cache_read_tokens,reasoning_tokens,exit,patch_files,patch_bytes,model,harness_version,attempt_id +obsidian-linter-auto-table-of-contents,hard,1,typescript,1,0,0.0,1.0,0,41,1131,1131,2026-09-22T14:37:19+00:00,2026-09-22T15:24:29+00:00,2828,3.310792,71,138098,20052,5793792,57172,0,4,36041,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143718-5634 +gql-incremental-graphql-delivery,hard,2,python,0,0,0.941176,0.998767,16,17,810,811,2026-09-22T14:37:18+00:00,2026-09-22T15:27:36+00:00,3016,5.564805,108,187885,37443,12908800,37791,0,11,94775,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143718-5477 +bandit-structured-nosec-directives,hard,3,python,0,0,1.0,0.996454,69,69,281,282,2026-09-22T14:37:18+00:00,2026-09-22T15:23:35+00:00,2773,3.41963,75,132519,22674,5950976,59778,0,9,36822,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143718-5487 +termenv-preserve-ansi-resets,hard,4,go,0,0,0.942857,1.0,33,35,87,87,2026-09-22T14:37:19+00:00,2026-09-22T14:55:01+00:00,1061,2.103379,35,275563,18899,1513984,35934,0,7,36407,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143718-5513 +updo-policy-alerting,hard,5,go,0,0,0.941176,1.0,16,17,123,123,2026-09-22T14:37:22+00:00,2026-09-22T15:07:03+00:00,1779,1.978445,55,94689,25318,3350528,20630,0,11,64246,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143721-6062 +effect-sse-httpapi-streaming,hard,6,typescript,0,0,0.978723,1.0,46,47,70,70,2026-09-22T14:37:21+00:00,2026-09-22T16:01:35+00:00,5050,7.794528,141,489254,37919,16645120,50963,0,10,48268,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143721-5912 +oxvg-structural-selector-preservation,hard,7,rust,0,0,0.666667,1.0,4,6,62,62,2026-09-22T14:37:21+00:00,2026-09-22T16:59:57+00:00,8553,11.664217,163,300291,34827,28045312,121823,0,12,40769,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143721-5921 +sqlfmt-create-table-ddl-formatting,hard,8,python,0,0,0.9375,0.995287,30,32,1267,1273,2026-09-22T14:37:22+00:00,2026-09-22T15:56:55+00:00,4771,6.906104,130,199397,35332,15134976,82496,0,18,61726,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143721-5939 +helm-array-merge-strategies,hard,9,go,0,0,1.0,0.833333,47,47,10,12,2026-09-22T14:37:25+00:00,2026-09-22T15:39:33+00:00,3725,4.961987,110,306251,34499,9070848,53633,0,23,69566,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143724-6479 +meriyah-explicit-resource-declarations,hard,10,typescript,0,0,0.979592,1.0,48,49,51469,51469,2026-09-22T14:37:25+00:00,2026-09-22T15:59:28+00:00,4921,9.75779,126,279348,30101,23515136,94246,0,10,80481,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143724-6349 +pest-character-class-coalescing,hard,11,rust,0,0,0.990385,0.996,103,104,249,250,2026-09-22T14:37:24+00:00,2026-09-22T15:35:33+00:00,3486,5.108006,99,176502,31254,10464768,64684,0,4,31348,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143724-6355 +testem-bail-on-test-failure,hard,12,javascript,0,1,1.0,1.0,90,90,489,489,2026-09-22T14:37:25+00:00,2026-09-22T15:40:46+00:00,3798,4.739857,128,153077,37001,10621952,35935,0,27,57085,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143724-6373 +koota-pair-relation-tracking,hard,13,typescript,0,1,1.0,1.0,38,38,172,172,2026-09-22T14:37:28+00:00,2026-09-22T16:41:35+00:00,7445,15.649206,228,299890,61054,40888320,104482,0,29,129339,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143727-6901 +prometheus-transactional-reload-status,hard,14,typescript,0,0,1.0,0.878049,15,15,72,82,2026-09-22T14:37:27+00:00,2026-09-22T15:44:10+00:00,3999,4.449052,140,137616,33931,10308864,28972,0,22,79968,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143727-6786 +obsidian-linter-link-format-conversion,hard,15,typescript,0,0,0.983333,1.0,59,60,1131,1131,2026-09-22T14:37:27+00:00,2026-09-22T15:02:19+00:00,1488,3.575291,69,272468,23136,5293056,54862,0,5,38185,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143727-6798 +koota-query-predicates,hard,16,typescript,0,1,1.0,1.0,43,43,172,172,2026-09-22T14:37:28+00:00,2026-09-22T15:37:08+00:00,3578,5.910879,107,333727,30839,11560960,65357,0,23,104745,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143727-6810 +quill-shared-toolbar-focus,hard,17,typescript,0,0,0.923077,1.0,12,13,22,22,2026-09-22T14:37:31+00:00,2026-09-22T16:09:49+00:00,5536,6.271334,90,217118,29754,11682816,111255,0,4,39360,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143730-7333 +koota-deferred-mutation-buffer,hard,18,typescript,0,1,1.0,1.0,71,71,128,128,2026-09-22T14:37:31+00:00,2026-09-22T15:29:21+00:00,3108,3.674038,69,153462,26558,6351872,60648,0,11,79863,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143730-7222 +kombu-virtual-queue-dead-lettering,hard,19,python,0,0,0.894737,1.0,68,76,1412,1412,2026-09-22T14:37:30+00:00,2026-09-22T15:29:40+00:00,3127,4.083963,90,146855,38320,8117760,42218,0,9,65664,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143730-7225 +ink-grid-box-layout,hard,20,typescript,0,0,0.92,1.0,23,25,49,49,2026-09-22T14:37:31+00:00,2026-09-22T15:52:25+00:00,4493,5.784688,124,173618,40454,12742912,55610,0,11,48294,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143730-7245 +awilix-async-container-initialization,hard,21,typescript,0,0,0.958333,1.0,23,24,162,162,2026-09-22T14:37:34+00:00,2026-09-22T15:19:36+00:00,2520,2.715593,63,109685,22934,4406528,48038,0,5,54254,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143733-7767 +happy-dom-deterministic-intersectionobserver,hard,22,typescript,0,1,1.0,1.0,14,14,9,9,2026-09-22T14:37:34+00:00,2026-09-22T15:28:20+00:00,3043,2.69371,83,105186,26019,4873472,35055,0,7,48038,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143733-7648 +csstree-shorthand-expansion-compression,hard,23,javascript,1,1,1.0,1.0,79,79,16715,16715,2026-09-22T14:37:33+00:00,2026-09-22T15:18:49+00:00,2473,2.413964,50,106726,26119,3148288,50501,0,3,55358,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143733-7669 +superjson-error-stack-serialization,hard,24,typescript,0,0,0.9875,1.0,79,80,116,116,2026-09-22T14:37:34+00:00,2026-09-22T15:15:59+00:00,2303,2.193666,48,95613,19064,3056640,46925,0,7,43009,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143733-7664 +kea-atomic-signal-selectors,hard,25,typescript,0,0,0.916667,1.0,11,12,139,139,2026-09-22T14:37:37+00:00,2026-09-22T15:52:49+00:00,4510,6.040529,98,280427,30998,11685376,81911,0,6,56114,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143736-8219 +vulture-persistent-analysis-cache,hard,26,python,0,0,1.0,0.986441,24,24,291,295,2026-09-22T14:37:36+00:00,2026-09-22T15:31:30+00:00,3232,3.733158,79,129897,31669,6627840,58672,0,8,42879,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143736-8116 +katex-multicolumn-array-spans,hard,27,javascript,0,1,1.0,1.0,94,94,599,599,2026-09-22T14:37:36+00:00,2026-09-22T15:29:04+00:00,3085,3.892165,97,134678,26116,7861504,49196,0,6,34491,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143736-8131 +expr-try-catch-errors,hard,28,go,0,0,0.987342,1.0,78,79,66265,66265,2026-09-22T14:37:37+00:00,2026-09-22T15:50:18+00:00,4360,6.72245,120,201400,41350,15495168,56630,0,17,58121,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143736-8123 +koota-composite-trait-aspects,hard,29,typescript,0,1,1.0,1.0,51,51,172,172,2026-09-22T14:37:40+00:00,2026-09-22T16:11:04+00:00,5602,9.901096,166,242096,56207,23513344,85180,0,26,122245,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143739-8642 +onedump-dump-encryption-pipeline,hard,30,go,0,0,0.182927,1.0,15,82,6,6,2026-09-22T14:37:39+00:00,2026-09-22T15:19:35+00:00,2513,1.530501,55,94861,20887,2178560,18603,0,16,48970,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143739-8568 +eicrud-keyset-pagination-cursor,hard,31,typescript,0,0,0.714286,0.97619,10,14,164,168,2026-09-22T14:37:39+00:00,2026-09-22T15:36:54+00:00,3553,4.156516,98,151900,22215,9199104,40524,0,10,27238,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143739-8571 +participle-grammar-conflict-analysis,hard,32,go,0,0,0.989011,1.0,90,91,153,153,2026-09-22T14:37:40+00:00,2026-09-22T15:09:02+00:00,1880,2.993082,63,175145,21259,4970240,45649,0,5,43649,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143739-8566 +python-statemachine-state-data-scoping,hard,33,python,0,1,1.0,1.0,72,72,1286,1286,2026-09-22T14:37:43+00:00,2026-09-22T15:43:40+00:00,3956,6.832117,135,198348,37621,16306944,52045,0,15,63499,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143742-9073 +clack-async-autocomplete-options,hard,34,typescript,0,1,1.0,1.0,82,82,643,643,2026-09-22T14:37:42+00:00,2026-09-22T15:15:59+00:00,2293,2.715462,62,109006,25466,4513280,43498,0,7,60997,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143742-9005 +langchain-request-coalescing,hard,35,python,0,0,0.96,1.0,48,50,232,232,2026-09-22T14:37:42+00:00,2026-09-22T16:00:10+00:00,4944,5.920385,93,200158,34434,12135936,77508,0,5,61585,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143742-9003 +httpx-streaming-json-iteration,hard,36,python,0,0,0.972222,1.0,105,108,1404,1404,2026-09-22T14:37:43+00:00,2026-09-22T15:40:45+00:00,3780,4.323361,90,145469,31221,8356864,60772,0,5,42985,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143742-9008 +optique-conditional-option-dependencies,hard,37,typescript,0,1,1.0,1.0,36,36,2034,2034,2026-09-22T14:37:46+00:00,2026-09-22T16:00:47+00:00,4980,10.029502,131,277163,39688,24543744,82638,0,5,76630,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143745-9532 +textual-kitty-key-phases,hard,38,python,0,1,1.0,1.0,23,23,57,57,2026-09-22T14:37:45+00:00,2026-09-22T15:38:35+00:00,3647,3.325898,86,124321,19780,6063616,55810,0,8,27582,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143745-9456 +pwntools-tube-multiplexing,medium,39,python,0,0,0.986301,1.0,72,73,1,1,2026-09-22T14:37:45+00:00,2026-09-22T15:37:47+00:00,3598,4.24467,79,157478,30859,7832320,63977,0,6,42758,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143745-9442 +arktype-json-schema-refs-dependencies,medium,40,typescript,1,0,0.92,1.0,23,25,1679,1679,2026-09-22T14:37:46+00:00,2026-09-22T15:43:08+00:00,3920,7.022899,139,318393,37260,14531584,79397,0,15,319863,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143745-9452 +dateutil-rfc5545-timezone-interop,medium,41,python,0,1,1.0,1.0,67,67,2035,2035,2026-09-22T14:37:49+00:00,2026-09-22T15:21:11+00:00,2598,3.921577,65,171542,32655,7281152,48852,0,3,53512,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143748-9959 +kgateway-consistent-hash-policy,medium,42,go,0,1,1.0,1.0,2,2,214,214,2026-09-22T14:37:49+00:00,2026-09-22T15:33:25+00:00,3334,4.819979,122,160188,35697,11322368,27150,0,12,87889,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143748-9903 +bandit-incremental-cache-control,medium,43,python,0,1,1.0,1.0,88,88,275,275,2026-09-22T14:37:49+00:00,2026-09-22T15:13:32+00:00,2142,2.548964,70,107330,27865,4599296,28614,0,6,59395,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143748-9881 +fastapi-deprecation-response-headers,medium,44,python,0,1,1.0,1.0,137,137,3134,3134,2026-09-22T14:37:49+00:00,2026-09-22T15:16:26+00:00,2315,3.633499,78,145273,23404,7896832,31838,0,5,77366,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143748-9883 +go-critic-doc-link-checker,medium,45,go,0,0,0.666667,0.9375,2,3,15,16,2026-09-22T14:37:52+00:00,2026-09-22T15:38:59+00:00,3663,3.868993,97,131858,23909,7405312,59546,0,8,24248,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143751-10388 +igel-persist-feature-schema,medium,46,python,0,1,1.0,1.0,24,24,2,2,2026-09-22T14:37:52+00:00,2026-09-22T15:11:15+00:00,2000,2.108158,63,88796,24592,3369984,30793,0,6,52140,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143751-10351 +bandit-interprocedural-taint-checks,medium,47,python,0,1,1.0,1.0,66,66,293,293,2026-09-22T14:37:51+00:00,2026-09-22T15:16:02+00:00,2287,2.649401,60,121486,18712,4440576,44806,0,14,33782,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143751-10323 +anko-typed-variable-bindings,medium,48,go,0,1,1.0,1.0,9,9,94,94,2026-09-22T14:37:52+00:00,2026-09-22T14:49:14+00:00,680,1.913038,56,108859,19859,3089152,24315,0,11,119482,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143751-10314 +mashumaro-flattened-dataclass-fields,medium,49,python,0,1,1.0,1.0,66,66,30014,30014,2026-09-22T14:37:55+00:00,2026-09-22T15:13:23+00:00,2126,5.42426,121,289058,33897,10855936,55602,0,4,51117,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143754-10828 +dasel-html-document-format,medium,50,go,0,1,1.0,1.0,146,146,1012,1012,2026-09-22T14:37:55+00:00,2026-09-22T15:19:21+00:00,2484,2.460378,62,100572,25276,3933440,39966,0,9,47347,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143754-10807 +boa-hierarchical-evaluation-cancellation,medium,51,rust,0,1,1.0,1.0,17,17,7,7,2026-09-22T14:37:54+00:00,2026-09-22T16:14:39+00:00,5801,11.154476,155,278915,44982,27417088,94525,0,10,68291,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143754-10753 +fastapi-implicit-head-options,medium,52,python,0,1,1.0,1.0,43,43,3134,3134,2026-09-22T14:37:55+00:00,2026-09-22T15:48:17+00:00,4220,5.811521,119,188309,37204,12196096,68647,0,4,93196,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143754-10787 +prometheus-typed-label-sorting,medium,53,go,0,1,1.0,1.0,17,17,28,28,2026-09-22T14:37:58+00:00,2026-09-22T15:39:15+00:00,3673,2.726285,54,113797,27360,3671296,58207,0,8,50591,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143757-11258 +claude-code-by-agents-recursive-delegation,medium,54,typescript,0,0,0.285714,1.0,2,7,31,31,2026-09-22T14:37:57+00:00,2026-09-22T15:02:11+00:00,1452,1.527616,50,79389,16860,2347264,22158,0,4,32229,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143757-11251 +pebble-durability-wait-apis,medium,55,go,0,1,1.0,1.0,59,59,44,44,2026-09-22T14:37:57+00:00,2026-09-22T15:43:04+00:00,3903,3.840466,87,139316,27095,6943744,62198,0,8,52921,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143757-11186 +cliffy-config-file-parsing,medium,56,typescript,0,1,1.0,1.0,37,37,451,451,2026-09-22T14:37:58+00:00,2026-09-22T14:58:30+00:00,1229,3.442178,95,158972,24257,6718208,41520,0,8,43531,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143757-11230 +sqlite-utils-safe-import-checkpoints,medium,57,python,1,1,1.0,1.0,60,60,1038,1038,2026-09-22T14:38:01+00:00,2026-09-22T15:24:48+00:00,2805,3.323416,101,115875,37604,6391552,32951,0,6,79846,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143800-11698 +kombu-single-active-consumer-priority,medium,58,python,0,1,1.0,1.0,85,85,1421,1421,2026-09-22T14:38:00+00:00,2026-09-22T15:11:02+00:00,1979,2.536454,64,116995,26793,4584448,27216,0,10,52018,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143800-11690 +skrub-duration-encoding,medium,59,python,0,1,1.0,1.0,130,130,2784,2784,2026-09-22T14:38:00+00:00,2026-09-22T15:42:15+00:00,3852,5.000036,105,389274,31946,9448448,34566,0,14,56083,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143800-11647 +scc-bounded-memory-spilling,medium,60,go,0,0,0.903226,1.0,28,31,286,286,2026-09-22T14:38:01+00:00,2026-09-22T15:21:53+00:00,2630,3.010166,65,121903,23060,5630208,40633,0,6,32486,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143800-11668 +mnamer-daemon-watch-lifecycle,medium,61,python,0,1,1.0,1.0,51,51,319,319,2026-09-22T14:38:04+00:00,2026-09-22T15:25:02+00:00,2816,2.524222,57,181781,22629,3667712,35942,0,5,49069,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143803-12155 +tengo-destructuring-bindings,medium,62,go,0,1,1.0,1.0,91,91,132,132,2026-09-22T14:38:03+00:00,2026-09-22T15:46:27+00:00,4101,6.224008,110,192378,30587,13536512,75141,0,9,35220,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143803-12150 +httpx-multipart-response-parsing,medium,63,python,0,1,1.0,1.0,122,122,1272,1272,2026-09-22T14:38:03+00:00,2026-09-22T15:29:55+00:00,3109,3.576863,75,129311,21288,6541568,60476,0,5,34775,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143803-12080 +opa-rego-rule-profiling,medium,64,go,0,1,1.0,1.0,25,25,6,6,2026-09-22T14:38:04+00:00,2026-09-22T15:33:55+00:00,3346,4.287514,77,171613,25589,9048832,44946,0,9,43095,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143803-12149 +ipython-session-bundle-replay,medium,65,python,0,0,0.941176,1.0,16,17,29,29,2026-09-22T14:38:07+00:00,2026-09-22T15:25:49+00:00,2860,3.667469,63,159933,24118,7164416,45105,0,5,60211,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143806-13049 +scriggo-method-declarations,medium,66,go,0,1,1.0,1.0,48,48,1049,1049,2026-09-22T14:38:06+00:00,2026-09-22T17:29:55+00:00,10305,23.382387,401,502539,80111,62348800,131231,0,19,5442458,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143806-13028 +opa-template-string-reconstruction,medium,67,go,0,1,1.0,1.0,5,5,4,4,2026-09-22T14:38:06+00:00,2026-09-22T16:44:14+00:00,7561,10.933594,199,255947,61055,25938944,97883,0,5,52859,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143806-12936 +mobly-grouped-test-barriers,medium,68,python,0,1,1.0,1.0,79,79,808,808,2026-09-22T14:38:07+00:00,2026-09-22T15:26:52+00:00,2923,3.245107,57,139110,27762,5018624,60384,0,3,72242,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143806-13091 +tengo-callable-instance-isolation,medium,69,go,0,1,1.0,1.0,23,23,122,122,2026-09-22T14:38:10+00:00,2026-09-22T15:51:29+00:00,4397,5.81034,99,196040,32623,12236800,70789,0,4,34222,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143809-13608 +fd-deterministic-multi-key-sorting,medium,70,rust,1,1,1.0,1.0,43,43,109,109,2026-09-22T14:38:09+00:00,2026-09-22T15:21:42+00:00,2611,3.425297,91,130175,25013,7095808,35389,0,9,50969,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143809-13645 +cattrs-partial-structuring-recovery,medium,71,python,0,1,1.0,1.0,69,69,7,7,2026-09-22T14:38:09+00:00,2026-09-22T15:20:21+00:00,2529,2.559706,52,120812,22634,4310784,37635,0,6,39072,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143809-13540 +tomlkit-toml-table-converters,medium,72,python,0,1,1.0,1.0,60,60,964,964,2026-09-22T14:38:12+00:00,2026-09-22T15:57:29+00:00,4752,6.414402,96,199051,39522,13146880,85357,0,5,51839,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143809-13609 +etree-xml-diff-patch,medium,73,go,0,1,1.0,1.0,52,52,15,15,2026-09-22T14:38:13+00:00,2026-09-22T15:10:28+00:00,1933,1.816824,27,101101,26351,1886720,36816,0,3,74428,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143812-14060 +textual-richlog-follow-state,medium,74,python,0,1,1.0,1.0,20,20,6,6,2026-09-22T14:38:12+00:00,2026-09-22T15:53:00+00:00,4484,5.469068,125,162736,38530,11465216,64223,0,8,51764,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143812-14096 +yaegi-go-embed-directives,medium,75,go,0,0,0.0,0.0,0,38,0,58,2026-09-22T14:38:15+00:00,2026-09-22T16:05:37+00:00,5238,6.705077,168,177191,43484,15560448,56874,0,16,43454,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143812-14027 +kysely-window-grouping-helpers,medium,76,typescript,0,1,1.0,1.0,254,254,22,22,2026-09-22T14:38:15+00:00,2026-09-22T15:37:42+00:00,3564,5.357534,117,171974,52871,11264256,44618,0,23,120223,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143812-13898 +numba-stencil-boundary-modes,easy,77,python,0,1,1.0,1.0,29,29,827,827,2026-09-22T14:38:16+00:00,2026-09-22T15:32:14+00:00,3236,4.393258,85,165911,25714,9334784,47292,0,4,40167,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143815-14468 +valibot-recursive-schema-composition,easy,78,typescript,0,1,1.0,1.0,10,10,209,209,2026-09-22T14:38:16+00:00,2026-09-22T16:42:13+00:00,7434,10.945712,192,243707,59895,25614336,108791,0,15,47605,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143815-14548 +go-git-worktree-merge-conflicts,easy,79,go,0,1,1.0,1.0,17,17,2,2,2026-09-22T14:38:17+00:00,2026-09-22T15:24:33+00:00,2771,4.41553,82,341884,26790,6732544,64768,0,4,51295,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143815-14208 +adaptix-name-mapping-aliases,easy,80,python,1,1,1.0,1.0,44,44,2738,2738,2026-09-22T14:38:16+00:00,2026-09-22T15:43:05+00:00,3886,6.664721,144,181893,32917,16019456,54630,0,12,50403,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143815-14302 +obsidian-linter-scoped-ignore-markers,easy,81,typescript,0,1,1.0,1.0,33,33,1133,1133,2026-09-22T14:38:19+00:00,2026-09-22T15:31:24+00:00,3183,3.888884,87,142388,24474,7567616,54955,0,7,46916,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143818-14944 +ts-pattern-match-each,easy,82,typescript,0,1,1.0,1.0,85,85,6,6,2026-09-22T14:38:18+00:00,2026-09-22T15:07:44+00:00,1763,1.890387,46,126965,19342,2301440,35700,0,5,41745,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143818-14993 +anko-default-function-arguments,easy,83,go,0,1,1.0,1.0,2,2,119,119,2026-09-22T14:38:18+00:00,2026-09-22T15:25:04+00:00,2803,5.999153,123,385107,32379,11219456,66154,0,6,50913,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143818-14951 +dynamodb-toolbox-lazy-recursive-schemas,easy,84,typescript,0,1,1.0,1.0,37,37,1267,1267,2026-09-22T14:38:19+00:00,2026-09-22T16:54:03+00:00,8141,13.39661,227,281020,72727,33794816,88280,0,79,133939,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143818-14976 +abs-stepped-slices,easy,85,go,0,0,0.833333,1.0,5,6,6,6,2026-09-22T14:38:22+00:00,2026-09-22T15:25:01+00:00,2797,3.272095,86,121347,27839,5784064,50350,0,7,30913,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143821-15364 +testem-per-launcher-reports,easy,86,javascript,0,1,1.0,1.0,65,65,469,469,2026-09-22T14:38:21+00:00,2026-09-22T15:24:14+00:00,2750,2.616617,81,135462,28435,4997888,18956,0,13,49860,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143821-15483 +query-persist-restored-query-state,easy,87,typescript,0,1,1.0,1.0,8,8,42,42,2026-09-22T14:38:21+00:00,2026-09-22T16:04:22+00:00,5157,7.182372,145,196327,37074,17086720,60751,0,10,45491,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143821-15551 +wasmi-trap-coredumps,easy,88,rust,0,1,1.0,1.0,22,22,58,58,2026-09-22T14:38:22+00:00,2026-09-22T16:36:06+00:00,7061,14.942747,215,421662,59481,37810688,96156,0,15,78826,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143821-15426 +koota-entity-snapshot-rollback,easy,89,python,0,1,1.0,1.0,84,84,47,47,2026-09-22T14:38:25+00:00,2026-09-22T15:07:25+00:00,1738,2.068126,63,95973,20850,3612672,25577,0,15,78485,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143824-15796 +geo-shapeindex-serialization,easy,90,go,0,0,0.875,1.0,21,24,599,599,2026-09-22T14:38:24+00:00,2026-09-22T16:18:04+00:00,5977,7.371077,130,273706,39985,16100096,74677,0,13,53079,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143824-15934 +arcane-drift-detection-baselines,easy,91,go,0,1,1.0,1.0,82,82,2,2,2026-09-22T14:38:25+00:00,2026-09-22T15:19:25+00:00,2458,3.663616,84,146470,34266,7857152,23538,0,18,88366,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143824-16067 +abs-module-cache-flags,easy,92,go,0,1,1.0,1.0,20,20,3,3,2026-09-22T14:38:25+00:00,2026-09-22T15:23:23+00:00,2695,3.519044,108,119739,27116,7268608,38167,0,5,38123,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143824-15904 +aiomonitor-task-snapshots-diff,easy,93,python,0,1,1.0,1.0,53,53,8,8,2026-09-22T14:38:28+00:00,2026-09-22T15:07:49+00:00,1760,2.118272,55,100672,25484,3295488,29690,0,8,62822,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143827-16236 +kcp-go-multiplexed-kcp-streams,easy,94,go,0,1,1.0,1.0,30,30,12,12,2026-09-22T14:38:27+00:00,2026-09-22T15:15:01+00:00,2192,2.858294,50,225775,28819,3300096,50751,0,3,58684,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143827-16390 +vitest-duration-sharding,easy,95,typescript,0,1,1.0,1.0,56,56,24,24,2026-09-22T14:38:28+00:00,2026-09-22T15:46:13+00:00,4062,4.213667,127,138445,34130,9069056,37711,0,14,66393,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143827-16531 +dynamodb-toolbox-conditional-attribute-requirements,easy,96,typescript,0,1,1.0,1.0,31,31,1267,1267,2026-09-22T14:38:28+00:00,2026-09-22T15:27:36+00:00,2947,7.202828,174,419934,44613,15340288,46444,0,50,115524,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143827-16391 +task-task-graph-export,easy,97,go,0,1,1.0,1.0,20,20,17,17,2026-09-22T14:38:31+00:00,2026-09-22T15:11:14+00:00,1961,2.751276,73,117867,20852,5516800,28657,0,14,41655,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143830-16672 +go-genai-streamed-function-args,easy,98,go,0,1,1.0,1.0,6,6,62,62,2026-09-22T14:38:30+00:00,2026-09-22T15:14:15+00:00,2143,2.134349,65,96124,19964,3520256,32696,0,5,42719,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143830-16875 +ytt-jsonpath-query-api,easy,99,go,0,1,1.0,1.0,103,103,1,1,2026-09-22T14:38:31+00:00,2026-09-22T15:11:53+00:00,1999,2.162695,58,90640,28145,3241984,33067,0,6,61967,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143830-17009 +helm-unified-manifest-stream,easy,100,go,1,1,1.0,1.0,5,5,2,2,2026-09-22T14:38:31+00:00,2026-09-22T16:05:50+00:00,5235,7.642218,147,211062,28400,18455040,69768,0,38,83897,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143830-16846 +yjs-map-conflict-detection,easy,101,javascript,0,1,1.0,1.0,9,9,231,231,2026-09-22T14:38:34+00:00,2026-09-22T15:32:12+00:00,3216,4.039893,66,163119,29128,7102720,65520,0,11,56222,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143833-17187 +goreleaser-retry-publish-auditing,easy,102,go,0,1,1.0,1.0,29,29,29,29,2026-09-22T14:38:33+00:00,2026-09-22T15:26:42+00:00,2886,3.392406,104,122019,31629,7105280,28022,0,14,56145,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143833-17332 +psd-tools-blend-range-api,easy,103,python,0,1,1.0,1.0,45,45,979,979,2026-09-22T14:38:33+00:00,2026-09-22T15:10:26+00:00,1910,3.114647,84,124545,26520,6718208,21850,0,9,46248,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143833-17504 +wazero-multi-module-snapshots,easy,104,go,0,1,1.0,1.0,78,78,2,2,2026-09-22T14:38:34+00:00,2026-09-22T15:05:18+00:00,1598,1.198592,43,56330,18008,1470208,21228,0,9,42214,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143833-17316 +sql-formatter-bigquery-pipe-formatting,easy,105,typescript,0,1,1.0,1.0,26,26,5709,5709,2026-09-22T14:38:37+00:00,2026-09-22T15:25:00+00:00,2781,3.149116,67,127406,16700,5682944,54101,0,9,20016,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143836-18282 +actionlint-action-pinning-lint,easy,106,go,0,1,1.0,1.0,55,55,145,145,2026-09-22T14:38:37+00:00,2026-09-22T15:20:35+00:00,2515,4.03117,91,151259,32938,8626944,33016,0,15,63009,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143836-18475 +httpx-deterministic-cookie-store,easy,107,typescript,0,1,1.0,1.0,115,115,1281,1281,2026-09-22T14:38:36+00:00,2026-09-22T15:19:47+00:00,2468,3.241844,69,137408,24483,6221568,39727,0,5,49602,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143836-18619 +returns-validated-error-accumulation,easy,108,python,0,1,1.0,1.0,159,159,61,61,2026-09-22T14:38:37+00:00,2026-09-22T15:33:46+00:00,3307,4.032522,99,195352,38372,7813120,35130,0,26,68134,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143836-18541 +ofetch-per-origin-circuit-breaker,easy,109,typescript,0,1,1.0,1.0,47,47,13,13,2026-09-22T14:38:40+00:00,2026-09-22T15:17:47+00:00,2345,2.510901,61,102541,23181,3957760,44549,0,5,44479,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143839-18935 +drizzle-orm-window-function-builders,easy,110,typescript,0,1,1.0,1.0,130,130,566,566,2026-09-22T14:38:39+00:00,2026-09-22T15:30:39+00:00,3116,4.284468,128,139167,36716,9379840,33485,0,19,69976,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143839-19143 +happy-dom-abort-pending-body-reads,easy,111,typescript,0,1,1.0,1.0,14,14,165,165,2026-09-22T14:38:39+00:00,2026-09-22T15:48:44+00:00,4201,6.685675,128,359335,29450,14623232,51930,0,9,25502,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143839-19297 +narwhals-rolling-window-suite,easy,112,python,0,1,1.0,1.0,103,103,10093,10093,2026-09-22T14:38:40+00:00,2026-09-22T16:01:41+00:00,4978,10.708462,167,280271,51698,27256064,61024,0,19,78898,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143839-19169 +true-myth-iterable-collection-combinators,easy,113,typescript,1,1,1.0,1.0,96,96,561,561,2026-09-22T14:38:43+00:00,2026-09-22T15:13:48+00:00,2103,3.534874,61,173401,28210,6967552,33417,0,8,67824,openrouter/moonshotai/kimi-k3,f3b9716,20260922-143842-19463 diff --git a/docs/benchmarks/deepswe/README.md b/docs/benchmarks/deepswe/README.md new file mode 100644 index 0000000000..60425ef75e --- /dev/null +++ b/docs/benchmarks/deepswe/README.md @@ -0,0 +1,78 @@ +# DeepSWE — senior-dev + +Task-solving outcomes for **senior-dev**, the coding harness vendored into this +product, measured on the DeepSWE v1.1 corpus. One directory per campaign, tables only. + +| Run | What | Headline | +|---|---|---| +| [`2026-09-12-harness-comparison-v4-flash`](2026-09-12-harness-comparison-v4-flash/) | Ten harnesses, DeepSeek V4 Flash, 113 tasks each | **senior-dev 1st — 62/113** | +| [`2026-09-15-senior-dev-278a076-v41-flash`](2026-09-15-senior-dev-278a076-v41-flash/) | senior-dev `278a076`, DeepSeek V4.1 Flash | **88/113 — 77.9%** | +| [`2026-09-22-senior-dev-f3b9716-kimi-k3`](2026-09-22-senior-dev-f3b9716-kimi-k3/) | senior-dev `f3b9716`, Kimi K3 | **78/113 — 69.0%** | + +Each directory holds a `README.md` with the run's setup and results, and a per-task +table: one row per task carrying reward, F2P, P2P and the underlying test counts, +wall time and timestamps, cost, model calls, exit status, and which binary and model +produced it. + +The two single-arm runs each have one `tasks.csv`. The comparison has one table per +harness under `tasks/`, 113 rows each, plus an `arms.csv` summarising the ten. + +Nothing else is committed. The per-attempt artifacts — `events.ndjson`, `run.log`, +traces, verifier stdout, patches — come to roughly 23 GB across the two 2026-09 +campaigns and stay on the machines that produced them. + +## Two `codeaf`s — read this before the tables + +The artifacts behind these runs were produced by a binary that **used to be called +`codeaf`, and is not this repository's `codeaf`.** Two Agent-Field projects built a +binary of that name; the other one renamed away, first to `swe-pro` and then to +`senior-dev`, which is the name used throughout this directory. Wherever the history +below says `codeaf`, it means senior-dev's own former name. + +## Relationship to `bench/deepswe/` + +[`bench/deepswe/`](../../../bench/deepswe/) is this repository's own DeepSWE rig, +which measures **this product's** `codeaf` binary and writes to a gitignored +`bench/deepswe/results/`. **These campaigns did not come from it.** They were run on +senior-dev's own rig, against the official DeepSWE verifiers at `0b9fabb`, and are +reproduced here as the committed record. Each run's README names its manifest, +preregistration and reproduction command. + +Corpus note: the two rigs count differently. `bench/deepswe/` works from the v1.1 +checkout of `github.com/datacurve-ai/deep-swe` at 117 tasks; these campaigns use the +**113 scored** tasks of that corpus. A 113 figure here and a 117 figure there are not +the same denominator. + +## Reading the grades + +`reward` is the DeepSWE verifier's binary pass. `f2p` is the fraction of +issue-specific fail-to-pass tests that pass after the patch; `p2p` is the fraction of +pre-existing pass-to-pass tests still passing. An empty `reward` means the task +produced no usable verifier outcome — it stays in the 113-task denominator and +contributes zero. Published F2P and P2P figures are macro-averages over all 113 +scheduled tasks on the same basis. + +## Binary identities + +These commit SHAs belong to senior-dev's repository, not to this one. Its branch +history was re-authored on 2026-09-15, so the same tree carries two SHAs and +different documents cite different ones. Each pair below was confirmed identical by +tree hash, not by assertion: + +| Recorded by the rig | After re-authoring | Tree | +|---|---|---| +| `8ae964c` | `45df90b` | `2fdf00a94890` | +| `278a076` | `712d980` | `3832b6376029` | + +Each run directory is named for **what the rig recorded at the time**, so the +artifacts match their own provenance. The Kimi K3 campaign's binary `f3b9716` sits +two commits after `712d980`: `4c3084f` renamed the namespace, `f3b9716` made the +control plane optional. + +## A note on names + +senior-dev has been renamed twice: `codeaf`, then `swe-pro`, now `senior-dev`. The +2026-09-12 and 2026-09-15 campaigns ran under `codeaf`, the 2026-09-22 campaign under +`swe-pro`. The values here are rewritten to the current name throughout. The originals +on the run machines are untouched, and checksums recorded inside these runs describe +those originals. diff --git a/docs/changes/unreleased/1512-senior-dev-deepswe-results.md b/docs/changes/unreleased/1512-senior-dev-deepswe-results.md new file mode 100644 index 0000000000..edf3a78c79 --- /dev/null +++ b/docs/changes/unreleased/1512-senior-dev-deepswe-results.md @@ -0,0 +1,25 @@ +--- +kind: added +title: senior-dev's DeepSWE results are in the repository, three campaigns as per-task tables +pr: 1512 +surface: [docs] +invalidates: + - "There was no committed record of how senior-dev does on DeepSWE — the only DeepSWE material in the repository was `bench/deepswe/`, the rig, whose `results/` is gitignored, so every number lived on the machines that produced it. `docs/benchmarks/deepswe/` now holds three campaigns as per-task tables: a ten-harness comparison on DeepSeek V4 Flash where senior-dev came first at 62/113, senior-dev `278a076` on DeepSeek V4.1 Flash at 88/113, and senior-dev `f3b9716` on Kimi K3 at 78/113. Each row carries reward, F2P, P2P and the test counts behind them, wall time, cost, model calls and the binary that produced it." + - "`docs/benchmarks/` measured one thing, the CLI's own footprint, so `performance/` was its only section. It now has two, and the index says which is which: `performance/` for the footprint, `deepswe/` for task-solving." + - "`codeaf` named exactly one binary as far as this repository was concerned. The DeepSWE artifacts arriving here were produced by a *different* binary of that name — the other Agent-Field project renamed away, first to `swe-pro` and then to `senior-dev` — so a `codeaf` in those campaign records is senior-dev's former name, not this product's. The values are rewritten to `senior-dev` throughout and the index states the collision outright." + - "A DeepSWE figure could be read as one denominator. It cannot: `bench/deepswe/` works from the 117-task v1.1 checkout and these campaigns from the 113 *scored* tasks of the same corpus, so 113 here and 117 there are different sets." +--- + +The rig is not the record. `bench/deepswe/` measures this product's own `codeaf` and +writes where nothing is committed, which is right for a rig and useless for anyone +asking later what the numbers were. These campaigns ran on senior-dev's rig against +the official DeepSWE verifiers at `0b9fabb`, and what is committed is only the +outcome: 308 KB of tables, against roughly 23 GB of per-attempt artifacts left behind. + +The caveats travel with the numbers rather than sitting in a footnote. Five tasks +across four arms produced no usable verifier outcome and are empty rather than zero, +while still counting against 113. Six of the nine other harnesses recorded no +per-attempt cost. And the Kimi K3 run resolves no difference against the V4.1 Flash +run — the arms differ in model, binary and coder prompt inseparably, the re-anchor +that would have separated them was never run, and the preregistered two-sided McNemar +gives p = 0.0895.