Let one Bot hand work to another, and reach a person when no Bot will do - #266
Open
davidmckayv wants to merge 15 commits into
Open
Let one Bot hand work to another, and reach a person when no Bot will do#266davidmckayv wants to merge 15 commits into
davidmckayv wants to merge 15 commits into
Conversation
Issue #192. This is what decides a hop, not what delivers one. Resolving who is being addressed, refusing when it should, writing the row that says what happened, and putting durable work on the queue #216 shipped. The runner that claims that work and runs the other Bot comes next, and the split is the point: deciding happens inside somebody's run and has to be fast and fail closed, while delivering is a whole agent turn that has to survive the pod it started on. THE ENVELOPE IS TYPED, WHICH IS THE ONE DEPARTURE FROM THE ISSUE. It proposed `message_bot(target, message)`. Free text is the commonest way a multi-agent system goes quietly wrong: the receiving Bot infers the intent, re-derives the constraints and guesses what shape of answer was wanted, and when it guesses wrong it does not fail, it returns something else confidently. Naming the task, its constraints and what a good answer looks like costs the asking model a little effort and removes most of that. THE GRANT IS AN ORDINARY GRANT. `plugin_grants` gains a `bot` kind rather than getting a table of its own, because an administrator already understands "this Bot may use that" and a fork's policy layer already applies to grants. That widening also caught a ternary labelling everything that was not an MCP tool a skill, which would have filed a Bot grant in the trail as one. DEPTH AND THE CONVERSATION TRAVEL IN THE SIGNED ASSERTION. A chain is three runs on up to three pods, so a counter in a variable stops applying the moment the second hop lands elsewhere, which is also when a loop starts costing money. And where an answer lands cannot come from the model, or one Bot could drop a turn into a conversation it was never part of. BOTH CAPS FAIL CLOSED AND ARE COUNTED FROM ROWS. The fan-out cap counts the hops this run has already offered, because counting in a process counts one pod and a run whose hops land on several is exactly what it exists to bound. They are configuration rather than constants, and mean by default: one level deep and three per run. A Bot is refused, in the same words, whether the Bot it named does not exist or is one its person cannot see, so this cannot be used to enumerate the roster. Every refusal is a sentence the Bot can say rather than an exception, because a throw ends the run with nothing said and reads to the person as the Bot ignoring them. And every outcome leaves an audit row: the refused one matters more, since a hop that happened is visible in the transcript and one that was refused is invisible everywhere else.
The decision half is wired in. A Bot that has been granted another is offered `message_bot`; one that has not is offered nothing, which is the correct default and better than a tool whose every call is refused. MADE PER RUN, NOT PER REQUEST, and that is the whole reason this touches the runtime. The tool has to know how deep the chain already is and which conversation an answer belongs in, and both are facts about the run rather than the request: a request is earlier, with a Bot and a person and no message. The per-run wrapper that already existed for narrowing tools is exactly that seam, so it does both now and is named for what it does rather than for one of its reasons. Depth comes from the assertion this deployment signed, never from the Bot id the runtime happens to be building. On a hop those agree; taking it from the signed value rather than the build is what stops a stale assertion aiming the next hop at another Bot's grants. The grant is read on every run and every hop rather than held, so one made a minute ago counts and one revoked a minute ago stops counting. A read that fails is treated as no grant: failing closed costs a hop, failing open would let a Bot address one nobody gave it because the database blinked. Driven against Postgres rather than fakes, because three of the four properties are the database's own: whether a second offer of the same hop collides, whether the fan-out count sees rows another replica wrote, and whether a grant read now reflects one written a moment ago. A fake answers all three the way its author expected, which is the wrong witness for the questions worth asking. Booted the server too: every wiring bug in this shape lives in module construction, where no unit test goes.
… of it The other half of #192. Deciding a hop happens inside somebody's run and has to be quick and fail closed; delivering one is a whole agent turn against a model. They are separated by the queue rather than by a function call, which is what lets any replica take any hop: on a cluster the Bot being addressed is very unlikely to be on the pod that addressed it. THE LEASE IS RENEWED FOR AS LONG AS THE RUN TAKES. A run is minutes and a lease that lapses mid-answer hands the same hop to a second replica, which runs the same Bot again and bills for it twice. That is the failure this queue exists to prevent and the one it is easiest to reintroduce by forgetting a heartbeat. THROUGH THE PLATFORM'S OWN RUNNER, not by calling the agent and writing the answer somewhere. The runner is what persists a turn to a thread, so a delivered answer is the same kind of object as one a person's run produced: in the transcript, in the history the next run reads, and surviving whichever pod made it. Calling `agent.run` directly would produce an answer nothing recorded, which is the failure nobody can debug. The addressed Bot reads the conversation before the ask, because it is joining something already in progress: one handed only the task answers a question whose other half was settled three messages ago. Who is asking is stamped from the row this deployment wrote, never from anything a model produced, or a Bot could claim to be another. And the parts stay parts: the asking model was made to name the task, its constraints and what a good answer looks like precisely so this one need not infer them, and flattening them into prose at the last step would throw that away. A run that ended in an error is not a delivery. Treating it as one finishes the work and leaves the person waiting for an answer that will never come. A run that completed IS one, whatever the Bot said: "I could not find that" is an answer, and retrying spends another model call on the same non-answer. A second attempt says so in the trail before it runs anything, because it may already have run that Bot and posted an answer before its owner died, and somebody looking at two similar answers should be able to tell a duplicate from a mystery.
Slice two of #192 is connected. A Bot calls the tool, a row lands on the queue, and whichever replica gets there first runs the addressed Bot and lets its answer into the conversation. THE ADDRESSED BOT IS BUILT BY THE RUNTIME MOUNT, not by wiring assembled beside it. `agentFor` and `history` are handed out from where the runtime already knows how to make a Bot for a person, because "built exactly the way a person's run builds it" is worth guaranteeing structurally: a Bot assembled by parallel wiring drifts the first time one of those arguments changes, and the drift is invisible. It runs, and quietly holds different tools or a different role from the one the person is talking to. One Intelligence client serves both, so a hop reads the history a person's run would read rather than a second view of it that could disagree. A LOOP RATHER THAN A SCHEDULE. A hop is somebody waiting for an answer, not housekeeping, so the culler's minute-granularity CronJob would be an unexplainable pause in a conversation. Every replica sweeps and the queue decides who gets what, so a replica added is delivery capacity rather than contention. It does not run at all where the depth cap is zero: a deployment that has switched the capability off has no hop to find, and polling for work that cannot exist is a query a second for nothing. The end-to-end test is the one worth having. The two halves never speak: deciding happens in one run and delivering in another process, and the only thing between them is a row. Unit tests on either side pass while the row they agree on is written by one and unreadable by the other. Only the model call is faked, because running a real one is slow, expensive and non-deterministic for a question the files either side already answer. History is passed through untouched rather than converted. The platform holds a thread's messages in its own shape and takes them back in the same one, so a stricter type in the middle would mean inventing a conversion between two things that already agree, and a conversion is a place to lose a message.
The caps are chart values and documented variables rather than folklore, and always rendered including the zeroes, so a deployment that has switched the capability off says so rather than relying on the image's default staying what it is today. AND THE HOP IS DRAWN IN THE TRANSCRIPT. Without this the call still appears, as a generic tool call named `message_bot` with its arguments as JSON: technically visible and practically not. What the issue asks for is that a person can see their Bot bringing in another one and read what it asked for, because a conversation that quietly fans out to four Bots and bills for all of them is the thing to avoid. The renderer registers no tool. `message_bot` runs on the server, where the grant, the caps and the audit row are, and a frontend registration would be a second place that decides. It draws a refusal differently from a handoff, since one is a Bot bringing in help and the other is a boundary holding, and drawing them alike would make a working cap look like a working handoff. The parts stay parts there too: what was asked, what bounded it, and what was wanted back.
Two found by driving this on a real cluster rather than by reading it. THE CULLER HAD NO CEILING ON ONE SWEEP. With `concurrencyPolicy: Forbid` above it, a sweep that hangs holds the lock for ever and Kubernetes never starts the next one, so culling simply ceases: no error, no restart, no alert, and the first sign is a bill for a fleet of browsers nobody has used in a fortnight. That is the failure the feature exists to prevent, arriving through the mechanism meant to prevent overlapping runs. Guido flagged it; it was on the follow-up list and should not have been. AND THE TRANSCRIPT DREW EVERY SUCCESSFUL HANDOFF AS BLOCKED. A server-side tool's result reaches the surface as a tool message whose content is JSON-encoded, so the renderer saw `"Handed to Knowledge…"` with the quotes and matched none of them. Worse than not drawing it: a working boundary and a working handoff looked identical, and the reassuring one was the wrong one. The prefix the two sides agree on is now named in one place, which is not a contract to be proud of but does stop them drifting apart in silence. The runner is also built from the client's own address and token rather than from configuration, the way the runtime builds it. That was a real bug and not the one it looked like: it has not fixed the delivery failure, which is recorded on the PR.
A delivery presented the lock's join token as the runner socket's credential. That token is what a browser presents to watch a conversation; the runner's socket has its own. Overridden with it the socket was refused, and because the runner treats a socket that will not connect as something to retry rather than as a failed run, nothing was emitted and nothing ever completed. Every hop hung in total silence. Taking the lock is what makes the run legitimate; the gateway checks the run id on every event, and nothing else needs presenting. Three more faults sat behind it, each of which would have been the next one: A hop was unbounded. Nobody watches a hop, so a run that stalls holds the conversation's lock and its place on the queue for as long as the process lives. It now has a deadline, and says how far the Bot got before it passed. The history handed across carried the asking Bot's tool traffic. A thread's stored history is what a person is shown, not a prompt: the assistant message that made a tool call is not kept, so its result is stored alone with a toolCallId matching nothing. The asking Bot's last act is always the call that handed the work on, so every hop carried one. What crosses now is what was said. The lock was released on the conversation that asked rather than the one it was taken on, and each attempt made a fresh conversation to answer in, so a retried hop left a row of empty channels behind it. The rest is what a person sees. The addressed Bot answers in the conversation they already have with it, which now moves up the roster when it does, because the browser is what writes that and no browser is watching. Its transcript keeps one line saying who asked and what for, rather than the whole prompt. And a hop that fails for good sends the asking Bot back to say so, so a question handed on and never answered stops being silence. Asking a person is now a tool of its own, sitting beside the one for handing work sideways and competing with it. A model with no named way to stop takes the one it has: it guesses, or it asks a Bot that cannot settle it either. Who "a person" is, is a seam; this template answers the person in the conversation.
`runAgent` takes runId, tools, context and forwardedProps. AG-UI keeps the messages and the thread on the agent itself and builds the run's input from them, so a messages array passed as a run parameter is ignored in silence. Nothing failed. The Bot ran, read an empty conversation, and answered "how can I help?" to a question printed directly above its reply. Told to answer with one particular word, it asked what it could help with instead, which is how this was finally pinned down.
The cap counted the run's hops and then wrote one, which holds only while nothing else is writing. The case it exists for is the opposite: a model asked to do several things emits several tool calls in one turn and they run at once, each reading a count taken before any of the others had committed. Five hops passed a cap of three, every time, on a single pod, with no unusual timing. The count and the write are one step now, in the queue where the rows are, under an advisory lock on the run's own prefix. A key already on the queue still counts as offered rather than as refused: a retried offer of queued work is not a new hop. The integration test drives five concurrent offers against a real PostgreSQL, because a stub that awaits one call at a time cannot fail the way this did.
…apart `ask_person` appeared in the transcript as a raw tool call with its arguments as JSON. A Bot that decided it could not settle something and stopped to ask has done the right thing; drawn that way it reads as a malfunction. Both lines read their outcome out of the tool's own prose, which is what a server-side tool leaves available, and both now decode through `asText` rather than stripping quotes by hand. Matched against the raw value the prefix never matches, which is how every accepted handoff came to be drawn as Blocked.
…cript A Bot going back to its own conversation to report a failed hop had the instruction that made it speak persisted alongside its answer, in a bubble that looks like something the person typed and then had read back to them. Its own sentence is the whole message.
# Conflicts: # server/src/work/queue.ts
davidmckayv
requested review from
MikeRyanDev,
guidovizoso and
tylerslaton
as code owners
August 26, 2026 21:50
Contributor
Author
|
CI here is red on |
Contributor
Author
|
Known limitation worth naming before review: a hop that fails and is retried leaves one "asked …" line per attempt in the addressed Bot's conversation, with no answer under any of them. There is no clean fix on this side. Every attempt pushes Persisting the ask only on the first attempt was considered and rejected: if that attempt loses the lock race the line is never written at all, and a later successful answer then appears with nothing saying why the Bot spoke. A duplicate on a failure path is the better of the two. Cosmetic, and only on repeated failure, which the asking conversation now explains in words. A proper fix needs the platform to discard a run that produced no assistant message, or to allow a message to be withdrawn. |
…voke one Three faults found by review and reproduced against a real PostgreSQL before being touched. The suite was green through all of them, which is the point: two are about time passing, and every stub of this queue answers whatever it is told. A claim leases the whole batch from one moment and the batch is delivered one at a time, so a heartbeat covering only the hop in flight left the rest on a lease quietly running out. A delivery is minutes and a lease is one: the tail of every batch expired, was claimed by another replica, and was delivered by both. Two model calls, two answers in somebody's conversation, and both replicas reporting success, because `finish` returns a boolean saying the lease had gone and nothing read it. Every claimed hop is renewed now, and each is renewed once more immediately before its delivery starts. That renewal is the question and the answer at once: consulting the heartbeat's own bookkeeping would only catch a refusal it had already seen, and a process paused long enough to lose the lease never asked. `finish` answering false no longer reads as success. `ChannelStore.direct` looked and then made, which is not find-or-create. Two hops delivered at the same moment each found nothing and each made a conversation, so one person had two channels with that Bot and their answers split between them. A Bot asked for several things in one turn produces exactly that. Find and make now share one transaction, serialised on the person and the Bot. The grant table learned a `bot` kind and the API did not. Revoke rejected it outright, so the capability could only be enabled by writing a row by hand and could not be turned off at all, while the design rests on a revoked grant applying to the very next hop. Both endpoints take it, one Bot reaching another is an administrator's decision like an MCP tool rather than a skill somebody attaches to their own Bot, and `kind` is checked against the three that exist rather than trusted from a JSON body. A fan-out cap of zero switched the capability off everywhere except the model's tool list, so every call was refused and the Bot told the person it had tried. Both zeros close the same door now.
Contributor
Author
|
All four confirmed, reproduced against a real PostgreSQL before anything was touched, and fixed in 1395278. 1. The tail of a batch was delivered twice. A claim leases the whole batch from one moment and the batch is delivered one at a time, so a heartbeat covering only the hop in flight left the rest on a lease quietly running out. Reproduced with two replicas: bot-1 was safe, bot-2 and bot-3 expired, replica B took and delivered them, replica A delivered them again, and A reported all three as delivered because Every claimed hop is renewed now, and each is renewed once more immediately before its own delivery starts. That second renewal is deliberately the check as well: consulting the heartbeat's bookkeeping would only catch a refusal it had already seen, and a process paused long enough to lose its lease never asked. My first attempt at this got it wrong and the second regression test caught it. 2. Duplicate conversations. Confirmed: two concurrent 3. Administrators could not configure it. Worse than described: 4. Each of the four has a regression test that fails without its fix — I checked by reverting each one in turn. The two races needed integration tests against a real database and a heartbeat interval that could be injected; a stub answers whatever it is told, which is exactly why 1,781 green tests said nothing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #192.
One Bot can address another and the addressed one answers for itself, with its own role, tools and grants, as the same person. Every hop is a claimed row on the queue #216 shipped, so it survives the pod it started on and lands on whichever replica gets to it.
What a person sees
A Bot with a
botgrant is offeredmessage_bot. It names the task, anything that bounds it and what a good answer looks like, and the deployment decides everything else: who is being addressed (against the roster that person may see), who is asking, where the answer lands, and how deep the chain has gone.The answer lands in the addressed Bot's own conversation with that person. Not the one that asked, and this is the platform rather than a choice: an Intelligence thread is owned by exactly one agent, and
assertThreadAgentOwnershipis unconditional. So the conversation that asked says where the work went, and the one that answers moves to the top of the roster with an unread mark. Its transcript keeps one line saying who asked and what for, not the envelope: the model needs the constraints and the shape of a good answer, a person scrolling needs to know why that Bot suddenly spoke.A hop that fails for good sends the asking Bot back into the conversation the person is watching to say plainly that nothing came back. Otherwise a question handed on and never answered looks exactly like a slow one.
ask_personsits besidemessage_botand competes with it. A Bot that needs judgement should stop rather than guess or hand the question to a Bot that cannot settle it either, and a model with no named way to stop takes one of the two it has. It is offered to every run, granted anybody or not: asking the person already in the conversation costs nothing, and a deployment able to switch off the safe exit while keeping the expensive one would be backwards. Who "a person" is is a seam.What driving it on a cluster found
Every one of these was silent, and each would have been the next one:
POST /api/threads/:id/lockhands one back and it reads like the thing to present. It is what a browser presents; the runner's socket has its own. Passed in, the socket is refused, and the runner treats a socket that will not connect as something to retry rather than as a failed run: nothing is emitted, nothing completes, and every hop hangs for ever in total silence.runAgenttakesrunId,tools,contextandforwardedProps. Amessagesarray passed as a run parameter is ignored without error, so the Bot ran against an empty conversation and answered "how can I help?" to a question printed directly above its reply. Told to answer with one particular word, it asked what it could help with, which is how this was finally pinned down.toolCallIdmatching nothing. The asking Bot's last act is always the call that handed the work on, so every hop carried one.Driven
On EKS, in a browser, against the real cluster:
ask_personchosen over guessing on a request only a person could settle, with the reason on the record;The queue's new prefix cap has an integration test that drives five concurrent offers against a real PostgreSQL. It fails against the old
offer.After review
Four more, all found by review, all reproduced against a real PostgreSQL before anything was touched, all fixed in 1395278. The suite was green through every one of them.
bot-1was safe,bot-2andbot-3expired, replica B took and delivered them, replica A delivered them again — and A reported all three as delivered, becausefinishreturns a boolean saying the lease had gone and nothing read it. Every claimed hop is renewed now, and each is renewed once more immediately before its own delivery starts. That second renewal is deliberately the check as well: consulting the heartbeat's own bookkeeping would only catch a refusal it had already seen, and a process paused long enough to lose its lease never asked.ChannelStore.directlooked and then made, which is not find-or-create: each of two concurrent deliveries found nothing and made a channel, so one person had two conversations with that Bot and their answers split between them. A Bot asked for several things in one turn produces exactly that. Find and make share one transaction now, serialised on the person and the Bot. An advisory lock rather than a unique constraint, because what has to be unique is not a column: it is "this person's channel whose whole roster is this one Bot", a count over another table.botkind and the API did not. Worse,kindwas never checked at runtime, so an admin could grant one by accident of the missing check while revoke rejected it outright: enable by hand, no way to turn off, against a design that rests on a revoked grant applying to the very next hop. Both endpoints take it, one Bot reaching another is an administrator's decision like an MCP tool rather than a skill somebody attaches to a Bot they own, andkindis checked against the three that exist.BOT_HANDOFF_MAX_PER_RUN=0switched the capability off everywhere except the model's tool list, so every call was refused and the Bot told the person it had tried. Both zeros close the same door now.Each has a regression test that fails without its fix, checked by reverting each in turn. The two races needed integration tests against a real database and a heartbeat interval that could be injected: a stub answers whatever it is told, which is why the green suite said nothing.
Driven again on the cluster afterwards: a hop delivered end to end, and a
botgrant revoked and re-granted through the API rather than by hand, attributed to the administrator who did it.Known limitation
A hop that fails and is retried leaves one "asked …" line per attempt in the addressed Bot's conversation, with no answer under any of them. Every attempt pushes
RUN_STARTEDcarrying the ask before the failure happens inside the agent, and the transcript is replayed per run, so no message id collapses them and a thread message cannot be withdrawn. Persisting the ask only on the first attempt was considered and rejected: if that attempt loses the lock race the line is never written at all, and a later successful answer then appears with nothing saying why the Bot spoke. Cosmetic, only on repeated failure, and the asking conversation now explains it in words. A proper fix needs the platform to discard a run that produced no assistant message.