Skip to content

Document DeepSeek agent loop 50 query eval #131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SonAIengine merged 1 commit into from
Jul 2, 2026
Merged

Document DeepSeek agent loop 50 query eval #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions examples/ablation/diagnostics/agent_loop_20260702_201106.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Agent Loop Retrieval Benchmark — Synaptic

- Run at: 2026-07-02 20:11:06 KST
- Dataset path: tests/benchmark/data/msmarco_passage_full.json
- SQLite DB path: tests/benchmark/data/msmarco_full.db
- Subset: 50
- Corpus limit: 8841823
- LLM base URL: https://api.deepseek.com/v1
- Model: deepseek-v4-flash
- Max turns: 5
- Sufficiency gate: yes
- Force first tool: yes
- Incremental JSONL: examples/ablation/diagnostics/agent_loop_deepseek_v4_flash_50.jsonl
- SQLite FTS AND-first threshold: 20
- SQLite FTS lexical rerank pool: 500

This measures LLM-planned exploration. The agent can change follow-up queries and tool choices based on evidence from earlier turns. The main metric is document reach, not ranked MRR, because the agent loop returns a cumulative evidence set.

## Summary

- Reach: 23/50 (0.460)
- Mean turns: 4.14
- Mean tool calls: 5.78
- Mean first relevant turn: 1.70
- Mean first relevant tool calls: 2.22
- Mean elapsed: 50.8s
- P50/P90 elapsed: 48.7s / 67.0s
- Mean prompt tokens: 19616
- Mean completion tokens: 960
- Mean unique tools: 2.38
- Mean unique search targets: 5.40
- Mean query rewrites: 4.20
- Queries with >1 tool type: 48/50
- Queries with query rewrites: 48/50
- Duplicate tool calls: 0
- Empty tool calls: 11

## Per Query

| QID | Reach | Turns | Calls | Tools | Targets | Rewrites | First Rel Turn | First Rel Calls | Found Relevant | Elapsed | Query |
|-----|:-----:|------:|------:|------:|--------:|---------:|---------------:|----------------:|----------------|--------:|-------|
| 300674 | yes | 4 | 4 | 2 | 3 | 3 | 1 | 1 | 7067032 | 36.2s | how many years did william bradford serve as governor of plymouth colony? |
| 125705 | no | 3 | 3 | 2 | 2 | 2 | - | - | - | 38.5s | define preventive |
| 94798 | yes | 4 | 5 | 2 | 5 | 4 | 1 | 1 | 7067181 | 46.4s | color overlay photoshop |
| 9083 | yes | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 7067274 | 28.3s | ____________________ is considered the father of modern medicine. |
| 174249 | no | 4 | 3 | 2 | 3 | 3 | - | - | - | 39.9s | does xpress bet charge to deposit money in your account |
| 320792 | no | 5 | 7 | 3 | 7 | 7 | - | - | - | 61.3s | how much is a cost to run disneyland |
| 1090270 | yes | 4 | 3 | 2 | 1 | 0 | 1 | 1 | 7067796 | 40.0s | botulinum definition |
| 1101279 | no | 5 | 6 | 2 | 6 | 4 | - | - | - | 53.8s | do physicians pay for insurance from their salaries? |
| 201376 | yes | 5 | 8 | 3 | 8 | 2 | 1 | 1 | 7068066 | 60.4s | here there be dragons comic |
| 54544 | no | 5 | 10 | 3 | 10 | 10 | - | - | - | 67.0s | blood diseases that are sexually transmitted |
| 118457 | no | 3 | 3 | 2 | 2 | 1 | - | - | - | 38.9s | define bona fides |
| 178627 | yes | 5 | 13 | 3 | 13 | 12 | 3 | 6 | 7068519 | 75.8s | effects of detox juice cleanse |
| 1101278 | no | 3 | 4 | 2 | 4 | 1 | - | - | - | 41.6s | do prince harry and william have last names |
| 68095 | yes | 4 | 8 | 2 | 8 | 4 | 1 | 1 | 7069266 | 59.2s | can hives be a sign of pregnancy |
| 87892 | yes | 5 | 10 | 3 | 8 | 8 | 3 | 5 | 7069601 | 64.6s | causes of petechial hemorrhage |
| 257309 | no | 4 | 5 | 3 | 5 | 5 | - | - | - | 54.1s | how long does it take to get your bsrn if you already have a bachelors degree |
| 1090242 | yes | 5 | 14 | 3 | 14 | 13 | 3 | 7 | 7070556 | 70.1s | symptoms of ptsd in vietnam veterans |
| 211691 | no | 5 | 5 | 3 | 4 | 3 | - | - | - | 45.3s | how coffee works quote |
| 165002 | yes | 4 | 4 | 2 | 4 | 1 | 1 | 1 | 7070877 | 44.6s | does contraction of the ciliary muscles shorten the lens |
| 1101276 | yes | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 7070950 | 34.6s | do spiders eat other animals |
| 264827 | yes | 3 | 2 | 2 | 2 | 2 | 1 | 1 | 7071066 | 32.8s | how long is the flight from chicago to cairo |
| 342285 | no | 4 | 6 | 2 | 6 | 6 | - | - | - | 48.3s | how titanic facts |
| 372586 | no | 4 | 6 | 2 | 6 | 5 | - | - | - | 49.3s | how to play blu ray discs |
| 89786 | yes | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 7071501 | 33.0s | central city definition |
| 118448 | no | 4 | 4 | 2 | 4 | 1 | - | - | - | 42.5s | define body muscular endurance |
| 92542 | yes | 4 | 5 | 2 | 5 | 4 | 1 | 1 | 7072003 | 45.6s | circulation money definition |
| 206117 | yes | 5 | 8 | 3 | 8 | 3 | 1 | 1 | 7072155, 7072160 | 56.9s | hotels in thornton co |
| 141472 | yes | 3 | 3 | 2 | 2 | 1 | 1 | 1 | 7072290 | 41.1s | derriere definition |
| 293992 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 79.4s | how many product lines does coca cola have |
| 196232 | no | 2 | 1 | 1 | 1 | 0 | - | - | - | 32.2s | government does do |
| 352818 | no | 5 | 7 | 2 | 6 | 5 | - | - | - | 62.2s | how to cook string beans |
| 45924 | yes | 5 | 7 | 3 | 7 | 6 | 5 | 7 | 5167800 | 62.5s | average temperatures las vegas by month |
| 208145 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 71.0s | how bicycle tire tubes are sized |
| 79891 | no | 5 | 7 | 2 | 7 | 3 | - | - | - | 60.9s | can you substitute chocolate chips for semi-sweet |
| 208494 | yes | 4 | 5 | 3 | 5 | 5 | 2 | 2 | 7073272 | 51.1s | how big do newfypoo's get |
| 319564 | no | 3 | 3 | 2 | 3 | 3 | - | - | - | 37.7s | how much fiber is in carrots |
| 155234 | no | 3 | 3 | 2 | 3 | 2 | - | - | - | 40.0s | do bigger tires affect gas mileage |
| 14151 | no | 5 | 13 | 3 | 13 | 12 | - | - | - | 83.9s | age requirements for name change |
| 67802 | yes | 3 | 4 | 2 | 4 | 3 | 1 | 1 | 7074071 | 45.9s | can green tea cause stomach problems |
| 1090184 | yes | 5 | 7 | 3 | 7 | 1 | 1 | 1 | 7074235 | 52.4s | synonym of subordinate |
| 323382 | no | 3 | 6 | 2 | 6 | 1 | - | - | - | 45.1s | how much is the stamp to send a card |
| 323998 | yes | 5 | 5 | 3 | 5 | 5 | 3 | 4 | 7074377 | 47.8s | how much magnesium in kidney beans |
| 91711 | no | 5 | 8 | 3 | 3 | 2 | - | - | - | 60.6s | child psychiatrist salary 2016 |
| 125898 | no | 5 | 6 | 2 | 5 | 5 | - | - | - | 57.9s | define prosthetic device |
| 289812 | no | 3 | 3 | 2 | 3 | 3 | - | - | - | 32.6s | how many mm is a nickel coin |
| 333486 | yes | 5 | 4 | 3 | 4 | 4 | 4 | 4 | 7075218 | 56.1s | how old do you have to be to get a job in idaho |
| 1090171 | yes | 4 | 7 | 2 | 4 | 3 | 1 | 1 | 7075317 | 50.9s | synonyms for the word discipline |
| 73257 | no | 5 | 6 | 2 | 6 | 6 | - | - | - | 48.7s | can seizure meds cause low sodium? |
| 1090170 | no | 5 | 6 | 4 | 6 | 6 | - | - | - | 48.7s | synonyms for the word, decline |
| 237373 | no | 5 | 9 | 3 | 9 | 9 | - | - | - | 64.1s | how is soil created from rocks |
Loading
Loading