Routines: let a Bot work on a schedule with nobody watching

[davidmckayv]

Raised by Mark alongside #192. Recording it; not scheduled.

Revised after the Kubernetes work in #216. Two things changed: the claiming mechanism this needs turns out to be the same one two other pieces need, and a routine that drives a browser now has a cold start to account for. The horizontal section is concrete rather than aspirational as a result.

The topic

Every run today starts because a person typed something. A routine is a Bot doing its job on a schedule with nobody in the room: check the overnight exceptions at 07:00, summarise what changed in the document library every Friday, sweep the open items each morning and post what needs a human.

The result lands in the Bot's own history, so a person reads it when they arrive rather than having to ask for it.

This is the feature where governance stops being paperwork. An interactive run has somebody watching who will notice a wrong tool call. An unattended one does not.

The claiming primitive is built. Use it, and read its sharp edges

Shipped in #216 (merged 26 August). server/src/work/queue.ts and work_items. Everything the
old version of this section asked for is there: Postgres, select ... for update skip locked, a
lease with an expiry, renewal while the work runs, an attempts count so a reclaimed item can be
told from a fresh one, and idempotence on the key. Do not rebuild it, and do not fork it.

createWorkQueue(database) gives you offer, claim, renew, finish, release and purge. The
culler in server/src/work/culler.ts is a worked example of the whole shape: decide what is due,
offer it, claim it, renew while acting, finish or release, sweep.

The six things that are not obvious, every one of which was a real bug found in review. They are
written down because a routine that gets any of them wrong spends money twice with nobody watching,
which is the exact failure this feature is most exposed to.

• Every moment is named in SQL, never Date.now(). Leases were computed on the replica and
  compared against now() in Postgres, which is two clocks pretending to be one. A node ninety
  seconds behind wrote a live sixty-second lease that arrived already expired, the next replica took
  the item, and both ran it. If you add a queue method, name its times in SQL.
• finish and release are owner-scoped, and so is anything you add. A replica whose lease had
  quietly expired could otherwise delete or reschedule work another replica was in the middle of.
  Both return a boolean; false means it is not yours any more, and stopping is the correct
  response rather than an error.
• finish keeps the row. Do not delete it. This is the subtle one. Idempotence lives on the key,
  so a routine due at 07:00 must collide with itself, and deleting on completion hands that key
  straight back to the next replica to wake late. The recovery path becomes the duplicate-run path.
  Rows are swept by purge on a retention window, which is also what reaps an item that ran out of
  attempts, so a permanently failing key is not wedged for ever.
• Renewal is the caller's job, and a routine run is long. claim hands you a lease; nothing
  refreshes it for you. The culler renews before each item because suspending a sandbox is quick. A
  routine run is a whole agent turn, minutes at a time, so it needs a heartbeat for as long as it
  runs. Forgetting it is how the tail of a batch gets executed twice.
• There is an attempt cap, and it is not the routine-level policy this ticket asks for.
  DEFAULT_MAX_ATTEMPTS stops an item being handed out and leaves it visible with its count and its
  last_error. "Disabled with a reason a person can read", plus backoff, is still yours to build on
  top.
• The queue claims; it does not schedule. Deciding what is due and calling offer is a separate
  piece, and it is where the idempotence actually comes from: the key has to carry the scheduled
  minute. offerIdleComputers is the culler's version of it. Routines need the cron equivalent.

The wrong build is still the obvious one. setInterval in the server is how we do the
audit-retention sweep (server/src/audit-retention.ts), and it is safe only there because deleting
old rows twice is the same as deleting them once. A routine fired by every replica is the same run
billed N times, N times the tool calls, N posts in the channel. On a cluster that is not a corner
case, it is Tuesday.

An unattended run may have to wake a computer

This is new since #216 and it changes the shape.

A Bot's computer is a browser holding a profile. On Kubernetes it is a Sandbox that gets suspended when idle — pod gone, volume kept — and resumed on demand. A routine that drives a browser at 07:00 is therefore the thing that wakes a computer that has been asleep since yesterday evening, with nobody watching it happen.

What follows:

• Cold start is part of the schedule. A resume costs a pod schedule and a browser launch. A routine that assumes its computer is warm will look like it hung. Warm pools exist upstream for exactly this and are worth wiring for scheduled work specifically, because the wake-up time is known in advance.
• The resume path gets exercised at 07:00 with nobody watching, which is the worst time to discover that a ref from before the suspend resolves against the page after it. sessionOf is what prevents that and it wants a test that actually suspends and resumes, not an assumption.
• Routines that only call tools have none of this, and they are most of the useful ones. Build the schedule so it does not assume a computer, and let the browser-shaped ones follow the fleet work rather than holding this back.

Governance, which is the point rather than the trim

A routine runs as a person, and the trail says nobody was watching. There is no live actor, so it must carry one: its owner. It sees what that person may see and no more. If the owner loses access to a connector, the routine loses it on the next run. An audit row for an unattended run has to be distinguishable from an interactive one, because "why did this Bot read that at 3am" is a question somebody will ask.

Same grants, same policy, same audit as any other run. No separate path. A routine is a run nobody typed, not a run with fewer checks.

Caps that refuse rather than truncate. Concurrent routine runs per deployment and per Bot. Unattended work is the easiest way to spend money without noticing, and a schedule that overruns its own interval piles up on itself.

Failure stops rather than retries forever. Backoff, a cap on consecutive failures, then disabled with a reason a person can read. A routine failing every ten minutes for a week is a bill and a noisy trail and nobody notices either.

Delivery recorded apart from the outcome. A run can succeed and its result reach nowhere — the channel is gone, the post failed. Collapsing those produces a trail that says a routine has been fine while nobody has seen a result in a month, which is worse than one that says nothing.

A person can list, pause and stop them, including somebody else's, for an administrator. A routine somebody set up and forgot is the thing that outlives them.

What a fork needs to be able to change

This gets adopted by companies who will run it on their own infrastructure and want it to fit what they already have. The seams that matter:

• Where the runner lives. worker/ exists and does nothing but report status. If a routine runner should be its own process rather than the API server's, that is where it goes — and because work is claimed rather than assigned, that is a replica count, not a rewrite.
• Where routines come from. Authored in the product, or shipped in the tenant package the way skills now are after Ship a package's skills, so tool selection works on a clone #181. A template that ships useful routines demonstrates this on a clone; one that ships none waits for somebody to think of it.
• What a schedule is. A cron expression is the obvious default and somebody will want to drive it from their own scheduler instead. The firing decision and the run should be separable enough that an external trigger is a supported way in rather than a fork of the runner.
• Where output goes. Its own history, a channel, or somewhere the company already reads. Worth an explicit seam rather than a hardcoded destination.

What we already have

• The schema opens with "bots, skills, routines, bot-to-bot handoff" (server/src/db/schema/coworker.ts). Routines were anticipated and never built.
• Postgres already carries everything that has to survive a replica, including the snapshot store, which exists precisely because holding it in one process broke across replicas.
• Grants, policy and audit are already the path every tool call takes.
• Skills now ship in the tenant package (Ship a package's skills, so tool selection works on a clone #181), which is the pattern a shipped routine would follow.

What is not decided

• Whether a routine may use Bot-to-bot messaging: let a Bot hand work to another Bot #192 to bring in another Bot. Probably yes eventually, and the caps have to compose, because unattended plus fan-out is the expensive corner.
• Whether a missed window is skipped or made up. A machine that was down for an hour either runs nothing or runs everything at once, and neither is obviously right.

[zopeVaibhav]

Both this and #192 are marked "not scheduled", so this is a question about whether the shared piece underneath them is worth taking, not a claim on the issue.

Reading around it, the two issues do not need two builds. Both need one thing that does not exist yet: a way to start a run nobody typed. Today every run is driven by the browser — copilotkit.runAgent in app/src/components/channels/channel-chat.tsx:276. There is no worker, no queue and no leased work anywhere in server/src: skip locked and for update return nothing, and the only setInterval is the audit-retention sweep at audit-retention.ts:156, which this issue already cites as the thing not to copy.

So #192's "the message is a row, claimed the way any other queued work is" has nothing to sit on yet. The claimable-work runner is the whole of the overlap: a table of due work, a lease with an expiry, a claim that is idempotent, and a run that carries an owner rather than a live actor. This issue is that runner plus a schedule and the failure caps; #192 is that runner plus a tool and a target.

One thing worth knowing if the order is ever discussed: #192 opens by saying several Bots can be addressed in a channel with @ today. That is not true on main. A channel is pinned to one runtimeAgentId for the life of its thread, and channel-chat.tsx:379 says so directly — the mention is carried in the draft but nothing routes on it. routing/classify.ts picks the coworker at channel creation, not per message. So a second Bot's reply currently has nowhere to land, which is a change to the thread binding rather than to messaging. This issue has no such problem: a routine's result landing in the Bot's own history is exactly what one pinned thread already is.

Two questions:

1. Is the design here settled enough for somebody to take, or is it still being decided?
2. If it is taken, would you want the runner landing as its own change that both issues then sit on, rather than folded into whichever feature is built first?