feat(agent-harness): recover queued runs and stream checkpointed steps - #5688
feat(agent-harness): recover queued runs and stream checkpointed steps#5688iscekic wants to merge 1 commit into
Conversation
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| fail('limit_exceeded', 'The model inference cost limit is exhausted.'); | ||
| } else { | ||
| if ( | ||
| previous.filter(item => item.kind === 'tool' && item.status !== 'released').length >= |
There was a problem hiding this comment.
WARNING: Interrupted tool leases consume limits.calls, so recovery cannot re-reserve the same pending call at capacity
commitModel and validateResponse treat limits.calls as the max stored tool calls, but reserve() counts every non-released tool reservation against the same ceiling. claim() marks an expired lease interrupted (still counted) and then immediately tries to reserve again for that pending call. After a crash during the last in-flight tool attempt (a 32-call checkpoint at the default cap, or calls: 1), that retry hits length >= limits.calls and fails the run with limit_exceeded without dispatching the stored call. Model steps explicitly allow two attempts per step; tool recovery has no equivalent budget.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| signal.addEventListener('abort', onAbort, { once: true }); | ||
| }); | ||
| try { | ||
| return await Promise.race([work(), aborted]); |
There was a problem hiding this comment.
WARNING: Promise.race leaves work() unhandled after abort, which can fail the isolate after durable writes
When the abort promise wins, work() is still running with no .catch(). execute() uses this helper around authorize, runModelStep, and dispatch. An abort (Stop or deadline) then finishFailure / maintainAlarm can commit, after which a later rejection from streamText or the adapter is an unhandled rejection on the Durable Object alarm. That can fail the Worker after the epoch has already moved, and the loser can still be in flight when the next alarm claims new work.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| }; | ||
| }); | ||
| if (!dispatched) return true; | ||
| fence(job); |
There was a problem hiding this comment.
WARNING: Post-commit fence() treats Stop as epoch loss after the mutation is already executing, so dispatch never runs and Stop cannot cancel
insertAttempt and compareAndSetCall(..., executing) already committed inside store.transition. blockConcurrencyWhile is released before this fence(job) (which does not allow stopping). A cancelRun in that gap makes fence throw; finishFailure then stopRun sees an executing non-read and keeps the lease instead of cancelling. Dispatch never starts. At deadline the call is recovered as outcome_unknown / reconciliation for an effect that was never sent, so a named Stop can strand the run waiting for a13 instead of cancelling.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 3 Issues Found | Recommendation: Address before merge Fix these issues in Kilo Cloud Overview
Issue Details (click to expand)WARNING
Files Reviewed (4 files)
Reviewed by grok-4.6 · Input: 258.4K · Output: 25.2K · Cached: 1.3M Review guidance: REVIEW.md from base branch |
1 participant
No new behavior — the product does not use this new work yet.
Summary
SchedulerStateSchemapersists ownership epochs and reservations so recovery resumes stored work, rejects stale results, and never blindly replays dispatched mutations.SchedulerAdapterrequires fixed model selection, authorization, policy, token counting, and dispatch; deadlines bound external waits even when adapters ignore cancellation.Named Stop aborts inference and reads, cancels remaining calls, and preserves completed effects or waits for mutation reconciliation.
Files
services/agent-harness/src/scheduler.ts— Source, added (+862/-0 lines). Reserves step zero for scheduler state and starts executable model steps at one. Prearms recovery before external work, preserves projection deadlines, and serializes alarm deletion after checking for remaining work or leases. Atomically commits validated checkpoints and immutable ordered calls; persists partial text, input digests, definition versions, and dispatch attempts. Rechecks epochs, deadlines, context, call order, and current authority before dispatch. Requires the admitted model and variant without fallback or a client bearer. Keeps approval, question, client, and reconciliation waits ahead of queued runs without polling. Marks interrupted reads failed and retains uncertain mutations with provider references for reconciliation. Stop preserves a live mutation lease for confirmed completion, cancels remaining calls even after definitions disappear, and leaves unrelated queued runs unchanged. Gives each call a separate bounded message and a stable timestamp after the step text, including after restart.ModelTooldefinitions omit executors, andTokenCountermust bound canonical history, including system instructions and tool schemas.PartialStepSchemakeeps display text non-executable;CompleteStepSchemarequires one fully drained, validated response before calls can execute.Legacy text stays untrusted; failed or cancelled turns retain canonical text and settled call/result pairs, but their unfinished calls cannot enter later prompts.
Files
services/agent-harness/src/model-step.ts— Source, added (+486/-0 lines). Strips executors and input callbacks from tool definitions; rejects duplicate, provider-defined, and provider-executed tools. Enforces the admitted model identifier, a single model step, and no automatic library retries. Validates stream events, invalid-call flags, finish reasons, ordered calls, schemas, token usage, and stored tool outcomes before checkpointing or reuse. Reads stream errors without logging provider objects. Throttles partial writes at 250 milliseconds or 4 kibibytes, and caps display text at 64 kibibytes. Caps streamed output and checkpoints at 256 kibibytes and validates up to 32 citations. Reads at most four 50-message history pages, excludes display partials and future queued inputs, and trims only whole prior turns. Preserves system instructions and the current turn; unresolved active calls block inference. Wraps legacy text, including old assistant messages, as untrusted user data until legacy writers and imported records disappear. Stop can preserve validated outcomes without requiring their old tool definitions, but this exception cannot authorize dispatch or inference.AdmissionandRunLimitsreuse acceptedSendResultSchemavalues; missing price bounds or mismatched admissions raiseRuntimeErrorbefore execution.ReservationSchemadefinesReservationrecords that retain token, request, time, and cost ceilings across retries and restarts without adding gateway billing charges.Each model step permits at most two attempts; lost responses keep their request and cost reservations rather than applying new defaults.
Files
services/agent-harness/src/limits.ts— Source, added (+125/-0 lines). Validates the stored admission against the run, accepted message, and conversation context. Calculates model input/output reservations from trusted token counts, remaining context space, and admitted prices. Bounds model requests, per-step regeneration, tool calls, web requests, each attempt, and total active execution time. Tracks reserved, finished, interrupted, and released attempts. Keeps crashed-attempt time reservations conservative, while finished attempts record elapsed time. Released non-dispatch requests retain elapsed time but no longer consume tool or web request slots; durable waits consume no active execution time. Adds byte sizing and portable errors, with retryability disabled unless explicitly allowed.Tests: 1 test file added —
services/agent-harness/src/model-step.test.ts(+1,971/-0 lines). Fake model streams and real SQLite cover crash recovery, partials, validation, budgets, Stop, unresponsive waits, and terminal-failure history. Shared selectors verify event, snapshot, and paged-history ordering after restart.Generated: 0 files changed.
Verification
Manual runtime verification did not run because this level adds no user interface or production wiring; verification remains pending at the completed stack tip.
Visual Changes
Visual Changes: N/A
Reviewer Notes
Human steps
No human steps are required before merge or after merge for this level.
Recorded checks
The handoff records passing results for the focused Vitest suite, changed-file oxlint, oxfmt, and
git diff --check.Scope
Kilo-Org/cloud./Users/igor/Projects/.worktrees/shared-agent-harness-3bb0.shared-agent-harness-3bb0-s11toshared-agent-harness-3bb0-s12.Notes
Runtime verification remains pending at the completed stack tip. This level adds no user interface or production wiring.
Stacked PRs — merge bottom to top. Each level shows only its own diff.
Runtime verification (E2E, user advocacy, simplify) runs on the tip PR over every level.
Every level keeps its own checks, its own bot review, and its own threads; each one is answered on its own PR.
Each level is its own deliverable: it builds and passes its own checks alone.
A finding on a level is repaired on that level, then carried upward with stack.sh forward.
shared-agent-harness-3bb0— chore(agent-harness): register workspaces and enforce CI boundaries #5632shared-agent-harness-3bb0-s2— feat(agent-harness): define portable domain and snapshots #5637shared-agent-harness-3bb0-s3— feat(agent-harness): define commands tools and permission policy #5639shared-agent-harness-3bb0-s4— feat(agent-harness): share client state and cursor recovery #5643shared-agent-harness-3bb0-s5— feat(agent-harness): persist command intents and execution receipts #5647shared-agent-harness-3bb0-s6— feat(db): add harness ingress grants and retirement fences #5655shared-agent-harness-3bb0-s7— feat(agent-harness): deliver legacy history and project durable text #5659shared-agent-harness-3bb0-s8— feat(agent-harness): authorize durable grants and registered clients #5662shared-agent-harness-3bb0-s9— feat(agent-harness): fence retirement and retry payload cleanup #5667shared-agent-harness-3bb0-s10— feat(agent-harness): persist authoritative state in SQLite #5675shared-agent-harness-3bb0-s11— feat(agent-harness): admit durable runs and revisioned commands #5678shared-agent-harness-3bb0-s12— feat(agent-harness): recover queued runs and stream checkpointed steps #5688 ← this PRshared-agent-harness-3bb0-s13— feat(agent-harness): resolve interactions and dispatch tools sequentially #5693 (tip)