Skip to content

feat(automation): durable run history with failure reasons (run observability)#2581

Merged
os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history
Jul 4, 2026
Merged

feat(automation): durable run history with failure reasons (run observability)#2581
os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Why

Automation runs were observable only in memory. The engine kept the last N runs in a ring buffer, so after a process restart (or once the buffer evicted the entry) you could not answer "did this flow run, and why did it fail?" — and a failed run surfaced no reason at all. This was the single biggest silent-trust gap for anyone authoring automations: a flow could stop firing or start failing with nothing durable to inspect.

This is the #1 UX recommendation from the online-automation dogfood: run observability — durable, queryable runs with status + failure reason.

What

sys_automation_run — previously the ADR-0019 store for live suspended runs only — becomes a durable run-history table.

  • enginerecordLog mirrors every terminal run (completed / failed) to the store via recordTerminal (fire-and-forget). listRuns() merges durable history with the in-memory buffer (in-memory wins on id, newest-first) so the Studio "Runs" surface shows runs that predate the current process.
  • storerecordTerminal / listHistory on both SuspendedRunStore impls. Terminal history rows use a run_-prefixed id, disjoint from live suspended runs (raw runId, status: 'paused', deleted on completion), so the suspend save/load/delete/list path is untouched and resume sweeps (list() filters status: 'paused') never see history rows.
  • schemafinished_at / duration_ms / error (all optional) + the status enum gains running / completed / failed alongside paused.

Safe & additive

  • Distinct id namespace ⇒ zero collision with the ADR-0019 suspend/resume lifecycle.
  • Persisting is best-effort + fire-and-forget — a history write/read failure is logged and swallowed, never breaking the run that produced it.
  • New fields optional; new store methods optional (in-memory/test defaults still work).

Verification

Unit — new run-history.test.ts: completed persistence, failed persistence with error, read-across-restart (fresh engine over the same store), best-effort isolation (a throwing store never breaks the run). Full suite green (231 passed), DTS typecheck clean.

Runtime, end-to-end on a clean showcase instance (the standard that caught the earlier no-op #2560 fix — a fake-service unit test alone can mask a runtime-broken change):

  • schedule-triggered showcase_scheduled_digest + 7 task-completion flows → durable completed rows with finished_at;
  • showcase_resilient_sync → a failed row carrying its try_catch failure reason;
  • a live showcase_task_follow_up paused suspended run coexisted without collision;
  • after a full process restart, the failed row — reason intact — was still queryable via /api/v1/data/sys_automation_run.

🤖 Generated with Claude Code

…vability)

Mirror every terminal automation run (completed / failed) to the
sys_automation_run table so "did it run, and why did it fail?" survives a
process restart and the in-memory ring-buffer eviction — previously runs were
observable only in memory and failures surfaced no reason.

- engine: `recordLog` fire-and-forget `recordTerminal` on terminal runs;
  `listRuns` merges durable history with the in-memory buffer (freshest wins).
- store: `recordTerminal` / `listHistory` on both SuspendedRunStore impls,
  keyed by a `run_`-prefixed id disjoint from live suspended runs so the
  ADR-0019 suspend/resume path is untouched and resume sweeps skip history.
- schema: sys_automation_run gains `finished_at` / `duration_ms` / `error`
  (all optional) and a `completed` / `failed` / `running` status alongside
  `paused`; becomes a durable run-history table, not suspend-only.
- best-effort throughout: a history write/read failure is logged and swallowed,
  never breaking the run that produced it.

Verified end-to-end on a clean showcase: completed + failed runs persisted
durable rows (failure reason intact), a live paused run coexisted without
collision, and the failed row survived a full process restart. New
run-history.test.ts covers persistence, read-across-restart, best-effort isolation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
spec Ready Ready Preview, Comment Jul 4, 2026 1:56pm

Request Review

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling size/m labels Jul 4, 2026
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): packages/services.

5 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/guides/packages.mdx (via packages/services)
  • content/docs/guides/runtime-services/audit-service.mdx (via packages/services)
  • content/docs/guides/runtime-services/index.mdx (via packages/services)
  • content/docs/guides/runtime-services/settings-service.mdx (via packages/services)
  • content/docs/protocol/objectos/i18n-standard.mdx (via packages/services)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@os-zhuang
os-zhuang merged commit ffafb30 into main Jul 4, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the feat/automation-run-history branch July 4, 2026 14:01
This was referenced Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant