Open-source offensive-security harness. Real findings across frontier, cheap API, and local models.
Sometimes, yes - but not by pretending one tiny model can replace every frontier-model decision.
The data-backed play is a conductor pattern: route the work to the model that belongs on that stage. Use cheap or local subagents for broad recon, inventory, tool-readiness checks, and first-pass evidence gathering; spend frontier-model budget only where deeper reasoning, validation, or report synthesis actually pays for itself.
That is the bet behind ExploitHunter.app. In the Hard Juice Shop sweep below, qwen-3.6-flash found 7 evidence-backed bug categories for $0.007. Frontier Claude rows in the same table cost roughly $2.00-$2.35 per run, so the right lane can be 280x+ cheaper while still producing useful evidence. Local LM Studio routes add another mode: useful signals with $0 API cost when the target, privacy constraints, or run volume make hosted inference the bottleneck.
The answer is only partially "use cheaper models." The real answer is orchestration: a security-research conductor that keeps target scope, memory, approvals, tools, and artifacts stable while sending each slice of work to the right model route.
What the results show:
- Frontier models are still worth paying for on deeper investigation lanes and harder validation.
- Cheap hosted models can produce broad, evidence-backed web findings for cents when the harness gives them the right rails.
- Local models can run offline with no API key and still find useful signals.
- Model behavior varies by lane: broad web sweeps, disciplined tool use, Docker network attack paths, and slower high-score investigations do not all favor the same route.
Eval map:
eval:juice/ Hard Juice Shop broad sweep: asks a model to handle an authorized vulnerable web target from a realistic prompt; probes broad web-hunting, evidence capture, cost, runtime, and whether findings are backed by artifacts.eval:webapp/ guided local webapp: gives a more directed local-target prompt; probes whether a model can follow the harness loop, update a useful system map, and produce findings when the task shape is clearer.eval:attack-vectors: runs focused prompts such as URL discovery, XSS, account controls, or package-risk review; probes whether the agent can investigate one security theme at a time instead of only doing broad sweeps.eval:network: runs read-only commands against the multi-service Docker network lab; probes service inventory, misconfiguration, command discipline, approval boundaries, and network evidence handling.eval:docker-labs: batches several network-lab scenarios into a composite comparison; probes consistency across access-control bugs, server-side request bugs, archive-handling bugs, and related Docker target tasks.eval:model-tools: uses synthetic local tools, not a real target; proves basic tool-calling discipline, budget behavior, repeated-call avoidance, and evidence triage before spending hunting budget.eval:prompt-improvement: asks a model to rewrite vague human instructions into scoped internal research prompts; probes planning quality without leaking known answers into model-comparison runs.eval:e2e:threadsandeval:e2e:browser: exercise the actual chat/runtime path; prove project/thread persistence, model routing, target authorization metadata, approval surfaces, real-LLM evidence, and stream/report plumbing.- Skill recall, benchmark-pack, and publish audits: check workspace skill loading, benchmark hygiene, dataset schema, leakage guards, provenance, and report completeness; these are harness-quality gates, not vulnerability-finding leaderboards.
If you're newer to appsec, the shorthand is:
- Vulnerability class: the kind of bug, such as XSS, IDOR, SSRF, weak auth, exposed files, unsafe archive handling, or leaking debug data.
- Evidence-backed: the model did not merely guess; the run saved a probe, response, command transcript, screenshot, or artifact that supports the claim.
- XSS / cross-site scripting: a web page accepts attacker-controlled input and can run it as script in someone else's browser.
- IDOR / insecure direct object reference: changing an id, filename, or object reference lets a user access something they should not, such as another user's report.
- SSRF / server-side request forgery: the app's server can be tricked into making a request to an internal or unintended URL.
- Tool calls / step budget: how much the agent used tools such as HTTP probes, file reads, and lab commands. One agent step can produce more than one tool call, so this is pressure telemetry, not a perfect quota.
These tables compare frontier model routes on the same two local labs as the harness improved over time: Anthropic Sonnet 5 low/high, Anthropic Opus 4.8 low/high, GPT-5.5 low/xhigh, GLM 5.2, and Kimi K2.7 Code.
Hard Juice Shop frontier rows:
| Model | Earlier score | Latest score | Earlier cost | Latest cost | Earlier tools | Latest tools |
|---|---|---|---|---|---|---|
anthropic-sonnet-5-low |
19/21 | 2/21 | $0.960786 | $0.113632 | 15/12 | 11/12 |
anthropic-sonnet-5-high |
9/21 | 16/21 | $0.944890 | $0.402350 | 15/12 | 23/12 |
anthropic-opus-4.8-low |
21/21 | 21/21 | $2.984985 | $0.647230 | 14/12 | 25/12 |
anthropic-opus-4.8-high |
19/21 | 19/21 | $2.419235 | $0.510570 | 13/12 | 30/12 |
gpt-5.5-low |
16/21 | 21/21 | $2.581800 | $0.414665 | 14/96 | 1/96 |
gpt-5.5-xhigh |
18/21 | 21/21 | $2.487445 | $0.408200 | 13/96 | 1/96 |
glm-5.2 |
14/21 | 19/21 | $0.114890 | $0.032168 | 19/12 | 19/12 |
kimi-k2.7-code |
7/21 | 19/21 | $0.005878 | $0.044358 | 16/10 | 60/10 |
Network access-control frontier rows:
| Model | Earlier score | Latest score | Earlier cost | Latest cost | Earlier tools | Latest tools |
|---|---|---|---|---|---|---|
anthropic-sonnet-5-low |
6/21 | 6/21 | $1.551972 | $0.116152 | 23/96 | 39/96 |
anthropic-sonnet-5-high |
13/21 | 6/21 | $1.428010 | $0.108802 | 24/96 | 8/96 |
anthropic-opus-4.8-low |
14/21 | 11/21 | $2.597560 | $0.364765 | 24/96 | 46/96 |
anthropic-opus-4.8-high |
21/21 | 15/21 | $8.095270 | $0.389125 | 36/96 | 41/96 |
gpt-5.5-low |
10/21 | 11/21 | $2.186540 | $0.279125 | 23/96 | 51/96 |
gpt-5.5-xhigh |
9/21 | 13/21 | $1.830995 | $0.264770 | 19/96 | 47/96 |
glm-5.2 |
11/21 | 15/21 | $0.020042 | $0.015369 | 20/96 | 48/96 |
kimi-k2.7-code |
4/21 | 11/21 | $0.006963 | $0.014711 | 101/96 | 82/96 |
The main signal is the self-tuning eval loop: run comparable tasks, measure score/cost/runtime/tool pressure, tighten the harness, then rerun. Newer runs cut cost sharply and improve several model rows, while Opus 4.8 high remains strongest on the network access-control comparison. Evidence backing remains incomplete in the scorer for many rows, so treat this as harness-learning and model-routing evidence, not a final product-quality leaderboard.
Single-run results against "Hard" Juice Shop, a deliberately vulnerable local web app with the easy hints and mitigations disabled. Classes means distinct vulnerability categories found; Evidence-backed means the run saved supporting artifacts for that category.
| Model | Route | Classes | Evidence-backed | Runtime | Cost |
|---|---|---|---|---|---|
qwen-3.6-flash |
OpenRouter | 7 | 7 | 21.4s | $0.007 |
gpt-oss-120b |
OpenRouter | 7 | 1 | 84.3s | $0.020 |
claude-opus-4.8-low |
Anthropic | 7 | 5 | 94.0s | $1.977 |
claude-opus-4.8-high |
Anthropic | 8 | 4 | 91.7s | $2.346 |
claude-sonnet-5-low |
Anthropic | 5 | 3 | 108.5s | $1.596 |
claude-fable-5 |
OpenRouter | filtered | 0 | 5.8s | $0.379 |
gpt-5.5-xhigh |
OpenRouter | 6 | 1 | 138.3s | $1.422 |
gpt-5.5-low |
OpenRouter | 1 | 1 | 60.8s | $0.555 |
gemma-4-26b-a4b-it |
OpenRouter | 7 | 4 | 13.1s | $0.002 |
kimi-k2.7-code |
OpenRouter | 6 | 6 | 20.2s | $0.021 |
gemini-3.1-flash-lite |
OpenRouter | 5 | 3 | 6.0s | $0.004 |
qwopus3.5-9b-v3 |
LM Studio (RTX 3090) | 5 | 5 | 27.7s | $0.00 |
gemma-4-e4b |
LM Studio (RTX 3090) | 7 | 7 | 49.0s | $0.00 |
glm-4.7-flash |
LM Studio (RTX 3090) | 6 | 6 | 160.8s | $0.00 |
claude-fable-5 is a negative-control result in this comparison: the model returned content-filter, made 0 tool calls, and produced no findings. The harness treats this as a filtered response, not a valid result.
Installed LM Studio models are useful, but lane-specific. The best local results are not all from the same model, which is exactly why the harness treats model choice as a route decision instead of a fixed backend.
| Model | Why it matters | Eval lane | Result | Tool calls / budget | Runtime | API cost |
|---|---|---|---|---|---|---|
gemma-4-e4b |
Best small local hunting balance in the installed-model pass | Hard Juice Shop | 5 answer findings, 6 signals | 4/8 | 74.8s | $0.00 |
qwen3.6-27b |
Cleanest local tool-behavior model; passed every synthetic tool scenario | Tool behavior | 109/109 | 17/340 | mixed | $0.00 |
gemma-4-12b-it@iq4_nl |
Strongest guided Juice Shop score | Guided Juice Shop | 21/21, 2 evidence-backed | 7/96 | 283.4s | $0.00 |
gemma-4-26b-a4b@q4_k_m |
Higher-capacity guided run with broad final findings | Guided Juice Shop | 16/21, 2 evidence-backed | 6/96 | 213.2s | $0.00 |
glm-4.7-flash |
Interesting Docker-lab signal finder despite weaker tool-readiness | Network access-control | 0.67 score, 4 signals | 40/96 | 441.5s | $0.00 |
Recommended local use: gemma-4-e4b for cheap/offline broad sweeps, qwen3.6-27b when disciplined tool use matters, and the 12B/26B Gemma variants for guided Juice Shop follow-up runs that still need broader coverage.
Composite of three Docker-lab scenarios: report access control (IDOR-style), SSRF URL preview, and archive extraction preview.
| Model | Route | Scorer score | Evidence-backed | Tool calls / step budget | Runtime | Cost |
|---|---|---|---|---|---|---|
gpt-5.5-low |
OpenRouter | 58/63 | 13 | 120/288 | 572.8s | $10.979 |
claude-opus-4.8-low |
Anthropic | 52/63 | 3 | 77/288 | 383.1s | $8.938 |
gpt-5.5-xhigh |
OpenRouter | 42/63 | 11 | 100/288 | 435.2s | $7.999 |
claude-sonnet-5-low |
Anthropic | 44/63 | 3 | 86/288 | 766.6s | $5.055 |
gpt-oss-120b |
OpenRouter | 46/63 | 7 | 75/288 | 218.6s | $0.049 |
Tool calls / step budget means actual tool invocations over configured agent steps; one step can emit multiple tool calls.
The full eval rollup, including per-scenario rows, lives in docs/model-comparison.md.
The practical read: ExploitHunter is not tied to one expensive model path. Qwen, Kimi, Gemma, Gemini, OpenAI, Anthropic, GLM, and local LM Studio routes all show useful behavior somewhere in the matrix. The right default depends on the lane: cheap broad sweeps, local/offline work, current harness smoke, or slower high-score network investigation.
ExploitHunter.app is an AI-agent security workspace: a local app where a model can plan, use tools, save evidence, and ask before risky actions. It gives an AI agent:
- a persistent project memory across sessions and threads
- a durable target authorization ledger (no scope drift, no "yeah go ahead" chat history)
- an evidence pipeline that stores every probe, transcript, screenshot, and finding
- an approval gate before any active scan, credential test, shell command, or file write
- a local lab runtime — spin up Juice Shop or a multi-service network target with one command
It is not a scanner that wraps an LLM. It is not a chat UI with some tools bolted on. It is a research loop: scope → passive review → approval → active probe → evidence → validate → patch.
git clone https://github.com/justsml/ExploitHunter.app.git
cd ExploitHunter.app
pnpm install
cp .env.example .env
pnpm devOpen http://localhost:3210.
Mastra Studio runs separately:
pnpm studioOpen http://localhost:4111. The older pnpm dev:mastra command is kept as an alias.
No API key? Leave .env blank. The app auto-starts Ollama with a RAM-aware Gemma 4 tag and runs fully offline.
Have OpenRouter?
OPENROUTER_API_KEY=sk-or-v1-...
MODEL_DEFAULT=llm://openrouter/deepseek/deepseek-v4-flashSpin up the bundled target and let the agent loose:
pnpm juice-shop:hard # starts the hardened Juice Shop at http://127.0.0.1:3323In a project thread:
I want to authorize http://127.0.0.1:3323 for testing. Hack it.
That's it. The agent plans, probes, saves evidence, and surfaces findings with the receipts attached.
Security work is tool-heavy. The agent issues HTTP probes, reads responses, chains findings, and summarizes evidence — a very different workload from coding benchmarks or reasoning tests.
The comparison covers Qwen 3.6 Flash, Kimi K2.7, Gemma, Gemini, GPT-OSS, and local LM Studio routes against the same target and tool surface. Additional sweeps include DeepSeek and GLM.
What I found:
qwen-3.6-flashproduced the densest evidence: 7 finding classes, all 7 evidence-backed, for $0.007.deepseek-v4-flashhit 7 answer classes in a separate sweep, but it is not part of the visible Hard Juice Shop table above.lmstudio-glm-4.7-flash(running local, offline, $0) found 6 classes with 6 evidence-backed findings — just slower.gpt-oss-120bremains cheap, but the current comparison row took 84.3s and produced only 1 evidence-backed finding; useqwen-3.6-flashwhen evidence density matters.
The model routing system (llm://openrouter/... or llm://lmstudio/...) makes switching trivial. Run the same prompt against five models in parallel and pick the one that fits your budget.
The tool-behavior readiness sweep passed 120 synthetic scenarios across 30 model routes with 150/150 expected tool calls, 0 invalid tool calls, and 0 budget misses. Reported cost was about $0.045 with mixed exact and estimated row costs. It is not a hunting-quality leaderboard; it shows that a broad model set can use the harness tool surface well enough to be selected by task, latency, and cost.
If you have an M-series Mac or a GPU:
MODEL_DEFAULT=llm://lmstudio/lmstudio-gemma-4-e4b
MODEL_DEFAULT=llm://ollama/gemma4:e4bNo API key. No data leaves your machine. The same eval loop that runs against OpenRouter works against a local LM Studio server. lmstudio-gemma-4-e4b found 7 bug categories with 7 evidence-backed findings with $0 API cost.
Useful for: air-gapped environments, sensitive targets, fixed local hardware budgets, or high-volume exploratory sweeps where hosted model costs would dominate.
1. Record target and authorization
2. Passive review first — no active probes yet
3. Approval gate — the agent asks before touching anything
4. Active probing with evidence capture
5. Validate or reject each finding
6. Dedupe, trace reachability, report
7. Patch in an isolated workspace, retest
Every artifact — HTTP responses, command transcripts, screenshots, patch diffs — is stored, indexed, and retrievable. The agent cites its evidence. You can replay any finding.
- Kali Linux containers and custom tool images
- Hard Juice Shop (bundled — one command)
- Multi-service network target: HTTP, FTP, SSH, Redis (bundled)
- Wireless: RF/IoT/Bluetooth/Zigbee
- Behind-firewall VPCs and private ranges over SSH
- Cloud instances (test your own infra end-to-end)
- ICS/SCADA/PLC
- Local CTF and training labs
Model refs are canonical llm://... strings. Switch by changing one env var:
# Fastest / cheapest hosted
MODEL_DEFAULT=llm://openrouter/deepseek/deepseek-v4-flash
# Best evidence density (hosted)
MODEL_DEFAULT=llm://openrouter/qwen/qwen3.6-flash
# DeepSeek
MODEL_DEFAULT=llm://openrouter/deepseek/deepseek-v4-flash
# Free, offline
MODEL_DEFAULT=llm://lmstudio/lmstudio-gemma-4-e4b
MODEL_DEFAULT=llm://ollama/gemma4Supported providers: OpenRouter, OpenAI, Anthropic, Gemini, Mistral, DeepSeek, Qwen, Ollama, LM Studio, any OpenAI-compatible endpoint.
Do not treat any hosted LLM as a discreet accomplice.
Recent evaluations — SnitchBench, Anthropic's agentic misalignment research, Simon Willison's recreation — show that tool-enabled models can decide to report, expose, or escalate behavior they interpret as illegal or dangerous, especially when given external communication tools.
The practical rule: only do authorized work, minimize sensitive third-party data in prompts, and prefer local/offline models for sensitive investigations. For high-sensitivity targets, use llm://ollama/... or llm://lmstudio/... so nothing leaves your machine.
ExploitHunter.app is for:
- systems, accounts, networks, and data you own
- penetration tests, audits, bug bounty, red-team exercises where you have explicit written permission
- CTFs, training ranges, and isolated labs
- defensive investigation of artifacts, malware samples, and suspicious services where you are authorized
Unauthorized access, scanning, credential testing, and data extraction can violate the CFAA, UK Computer Misuse Act, EU Directive 2013/40/EU, and other laws. The project does not provide legal advice. You are responsible for your engagements.
pnpm typecheck && pnpm test && pnpm build
pnpm audit --audit-level moderate- docs/model-comparison.md — full model snapshot with source eval artifacts
- docs/fable5-model-eval-comparison-2026-07-01.md — Fable 5 side-by-side eval comparison and refusal analysis
- evals/results/juice-shop/cheap-model-tuning-report-2026-06-25.md — tuning notes
- docs/eval-honesty.md — how evidence-backing is scored
- docs/architecture.md — system architecture
- docs/durable-approvals.md — approval model
- docs/getting-started.md — guided first run
Star it. Try it against the bundled lab. Open the sharpest issue you can.