A2: freeze time-bounded adaptive read experiment#308
Draft
witbrock wants to merge 3 commits into
Draft
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a disposable, non-production “A2” thin read-only turn candidate plus a focused test suite to freeze the experimental condition for a time-bounded adaptive read experiment (elapsed-time budget, tool-parallelism within a batch, and tool-free final synthesis).
Changes:
- Adds
scripts/try_thin_readonly_turn.py: a standalone A2 candidate runner using the OpenAI Responses API with a frozen read-only tool palette and elapsed-time termination (research deadline + final synthesis reserve). - Adds
tests/test_thin_readonly_turn.py: tests for protocol boundaries (read-only enforcement, provider call-id correlation), elapsed-time budgeting, parallel within-batch tool execution, and late-answer marking.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/try_thin_readonly_turn.py | Implements the time-bounded adaptive read-only A2 candidate, parallel tool execution, transcript emission, and read-only boundary enforcement. |
| tests/test_thin_readonly_turn.py | Adds behavioural tests covering the candidate’s time-budget semantics, tool execution behaviour, and protocol/guardrail handling. |
Comment on lines
+409
to
+413
| if phase == "research": | ||
| if text: | ||
| transcript["last_partial_text"] = text | ||
| continue | ||
| transcript.update( |
Comment on lines
+496
to
+499
| executor = ThreadPoolExecutor( | ||
| max_workers=len(prepared), | ||
| thread_name_prefix="a2-read", | ||
| ) |
Comment on lines
+515
to
+519
| remaining_research_seconds = max(0.0, research_deadline - clock()) | ||
| completed_futures, _ = wait( | ||
| [future for _, _, _, future in pending], | ||
| timeout=remaining_research_seconds, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A1 showed that its fixed two-tool-batch ceiling directly caused all three thin-candidate failures, while current Von produced one substantive answer from three cases in 52 minutes. A2 tests the more general hypothesis: ordinary read-only administrative and scientific work should leave route selection and recovery to the model, bounded by trusted identity, read-only capability and elapsed time rather than counts of model calls, tool batches, retries or routes.
A1 commit
207178f0and draft PR #307 remain immutable evidence. This branch descends from that frozen condition but defines a new candidate and a fresh experiment.What changed
scripts/try_thin_readonly_turn.pycandidate and its case-independent tests; no production runtime file or route is changed.tool_choice="none"for final synthesis.The tests cover capability/protocol boundaries and observable outcomes. They do not prescribe a tool order, route, workflow identity, intent class or answer wording.
Frozen comparison protocol
After this commit/PR freeze, Michael supplies three fresh unseen low-risk read cases. Each exact case is submitted once to this candidate and once to current Von in a fresh chat: exactly six runs. Within-turn adaptation is allowed; after-result retry, rephrasing, tuning, replacement cases and a seventh run are not. The A1 cases and entities are forbidden from A2 prompts, fixtures and acceptance logic. Every outcome is retained.
The comparison baseline is
origin/mainat95c6a4caad1f50b0845f2e654b97522cc586426b.Validation
pytest -q tests/test_thin_readonly_turn.py— 12 passedtool_choice="none"completed with textStopping point
Do not merge, route production traffic, delete production controller code, tune against evaluation cases or begin A3. A2 stops after the six frozen runs with evidence ready for Michael's H2 decision.