Skip to content

Commit 4d22836

Browse files
jonathanpeterwuStackMemory Bot (CLI)
andauthored
feat(webhook): add persistent retry with exponential backoff (#19)
* feat(teleop): add native voice control prototype * feat(cli): add scaffold command for Company OS folder structure stackmemory scaffold creates company/, wiki/, skills/, clients/, raw/, and .stackmemory/config.yml. Enables local context management with file-based skill rot detection and tenant isolation. * feat(daemon): add opt-out telemetry service New DaemonTelemetryService collects anonymous usage snapshots: - Daemon health (uptime, context saves, memory triggers, errors) - Session counts (total heartbeats, active now) - Skill audit entries, handoff counts - No PII — instance ID is random hex Runs daily (default 24h interval, first at boot+30s). Stores rolling 90-snapshot history in ~/.stackmemory/telemetry.json. Opt out: STACKMEMORY_TELEMETRY=0 or telemetry.enabled: false in config. * feat(hooks): add self-healing daemon health check for SessionStart * feat(daemon): add desire-path detector — auto-discover workflows, suggest skills Three-component system in DaemonDesirePathService: 1. ActionStreamLogger — PostToolUse hook captures tool:target pairs to ~/.stackmemory/desire-paths/action-stream.jsonl (no data/content) 2. PatternDetector — sliding window extracts repeated sequences, filters by min 3 occurrences across 2+ sessions, scores by freq×sessions 3. SkillSuggester — generates skill.md files from top patterns with inputs/outputs inferred from sequence endpoints - 10MB JSONL rotation, 10K entry scan cap for performance - Opt out: STACKMEMORY_DESIRE_PATHS=0 or desirePaths.enabled: false - Scans every 6h, first at boot+2m - Suggestions written to ~/.stackmemory/desire-paths/suggestions/ - 3 adversarial review rounds: fixed separator injection, added scan cap, improved skill naming with target directory context * fix(desire-paths): adaptive backoff — hourly when active, exponential to 12h when idle * feat(cli): add hermes-sm wrapper with StackMemory integration - Auto-starts daemon on session boot - Writes session heartbeats for telemetry tracking - Restores handoff context from previous sessions - Sets STACKMEMORY_SESSION env for desire-path hook - Determinism watcher + tracing - bin/hermes-sm and bin/hermes-smd registered in package.json * feat(desire-paths): auto-promote skills above 0.8 confidence + 5 sessions * feat(daemon): add research stream scanner for market signal detection * fix(test): replace bun:test import with vitest in desire-path-service test * feat(tokens): replace char/4 heuristic with js-tiktoken (cl100k_base) Centralizes token estimation across 14 files through src/core/cache/token-estimator.ts and packages/sdk/src/token-estimator.ts. Lazy-loads cl100k_base encoder with char/4 fallback if WASM fails. Also ports context-budget hook to codex-sm exit handler for compact/restart nudges matching Claude Code behavior. * feat(skill-packs): add content licenses (CC-BY-4.0) to registry metadata Skills are often prompt text (content) not code — content licenses like CC-BY-4.0 fit better than MIT for these. Adds KnownLicenseSchema enum with both code (MIT, Apache-2.0, ISC, BSD) and content (CC-BY-4.0, CC-BY-SA-4.0, CC0-1.0) licenses while keeping the field open for custom SPDX identifiers. * feat(tasks): add local-first master-tasks.md task management Markdown table parser + CLI commands + MCP tools for local-first task steering. Tasks live in master-tasks.md, optionally sync to Linear/GH. - Parser: parse/serialize/update/add/getNext for pipe-delimited md tables - CLI: stackmemory tasks init/md list/md next/md add/md update - MCP: get_next_master_task, update_master_task, create_master_task - 19 tests covering parse, round-trip, priority sorting, file ops * feat(hooks): token optimization — dedup escalation, auto-route, prewarm, script-suggest - dedup-reads: escalate to [STOP] at 5+ reads (was soft-only at 3+) - desire-path-hook: auto-route Bash→Glob/Read/Grep with inline suggestions - prewarm-tools: SessionStart hook emits top deferred tool pre-fetch hint - script-suggest: detects multi-tool patterns matching existing scripts * feat(bench): add hook benchmark script + baseline report Replays 7,589 action-stream entries through hook logic. Result: 324K token savings projected (22% waste reduction). * feat(hooks): weekly skill-mine reminder on SessionStart Emits reminder when >7 days since last mine and new suggested skills exist. Points to /workflow-skill-miner. * fix(desire-paths): filter trivial patterns from suggestions + auto-delete - Skip patterns with <2 unique tools or <3 steps in generateSuggestions() - Add cleanTrivialSuggestion() to auto-delete stale suggestion files - Remove duplicate quality gate from autoPromote (filtering now upstream) - Prevents bloat: 19/20 current patterns were trivial (git×2, Edit×3, etc.) * feat(subagent): design delegation — Claude-first routing for frontend/UI tasks Design tasks bypass subscription-first cascade (Codex→Grok→API) and route directly to Claude CLI, which excels at creative UI/UX decisions. - Add 'design' task type to SubagentRequest, TaskType, ModelRouterConfig - Add forceProvider field for explicit provider override - Add design prompt (opinionated, production-ready, no-ask-just-decide) - Add delegateDesign() convenience method for wrapper CLIs * docs: rewrite CLAUDE.md as tool-agnostic agent guide Consolidate from StackMemory-specific config to a generic agent reference covering stack, structure, commands, and key patterns. * feat(hooks): project-aware prewarm tool cache Filter action-stream by current project directory so prewarm suggestions are scoped to the repo you're working in. Falls back to global stats when no project-specific data exists. * feat(hooks): memory-loader SessionStart hook Reads MEMORY.md index at session start, scores entries by relevance to current project context, and surfaces the most useful memories. * feat(hooks): image preprocessing + MCP vision extraction image-preprocess: PreToolUse hook that intercepts Read calls on image files and routes them through vision-capable models. image-extract-mcp: stdio MCP server providing a describe_image tool for text extraction from images via vision model APIs. * feat(gepa): daemon watcher + session hook updates daemon.js: persistent file watcher for all GEPA targets — triggers optimization on CLAUDE.md changes. Session hook and .before-optimize baseline updated for current optimization state. * chore(gepa): gen-001 variant updates from optimization run * feat(browser): Stagehand workflow capture, cache, replay + benchmark harness End-to-end integration bridging Stagehand browser automation with StackMemory's desire-path system for workflow discovery and replay. - StagehandWorkflowCapture: wraps act/extract/observe, emits to action-stream - WorkflowCache: persists workflows, bridges to desire-path patterns.json - WorkflowReplayer: cached (0 tokens), AI (self-healing), hybrid modes - WorkflowBenchmark: compare stagehand-ai vs cached vs playwright-code - 4 MCP tools: workflow_list, workflow_get, workflow_replay, workflow_benchmarks - Benchmark script with 3 test workflows (GitHub, HN, NPM) Stagehand is a peer dependency — not required for core functionality. * feat(browser): CLI browser agent + benchmark updates CliBrowserAgent: Playwright + claude/codex CLI hybrid that routes AI understanding through subscription CLIs instead of direct API. Falls back to Anthropic API when CLI hooks interfere. - Playwright handles browser control (fast, deterministic) - claude --print / codex -q handles extraction/action interpretation - Results cached locally for zero-LLM replay on subsequent runs - Benchmark script supports --cli mode vs --api mode Known: claude --print triggers SessionEnd hooks that timeout. TODO: fix hook interference or add CLAUDE_CODE_SKIP_HOOKS support. * fix(hooks): add DISABLE_HOOKS skip guard to all Stop/SessionEnd hooks When DISABLE_HOOKS=1 env var is set, all session lifecycle hooks exit immediately. Prevents timeout/cancellation when claude --print is invoked as a subprocess (e.g., from CliBrowserAgent). Hooks patched: chime-on-stop.sh, stop-checkpoint.js, session-rescue.sh, wiki-update.js, token-meter-finalize.js, gepa-session-hook.js * fix(browser): accept CLI output on non-zero exit from hook failures claude --print produces valid output before SessionEnd hooks fire. Exit 143 from hook cancellation shouldn't reject — check stdout content instead of exit code. * chore(deps): add stagehand + playwright for browser workflows * docs(research): agent-readable web standards landscape 2026 * chore(gepa): update generation variants + daemon state * feat(mcp): wire workflow tools into main MCP server Register workflow_list, workflow_get, workflow_replay, workflow_benchmarks in MCPToolDefinitions and MCPHandlerFactory. Routes through handleWorkflowTool from browser/workflow-mcp-tools.ts. * feat(hooks): add cd-thrash, linear-dedup, and bash-dominance guardrails - cd-thrash-guard: warns on 3+ cd commands in 10 tool calls - linear-dedup: detects duplicate Linear API calls within 60s - bash-dominance-guard: suggests Read/Grep/Glob/Edit over Bash equivalents * fix(benchmark): fix NPM selector timeout + use page.evaluate extraction * chore(gepa): update daemon state + generation variants * chore(gepa): update hook state + scores * chore(gepa): auto-optimizer state + eval results * chore(gepa): daemon state update * chore(gepa): auto-optimizer state update * feat(skill-packs): add ops/log-investigation pack Distributable investigation skill teaching agents to debug production issues from structured wide-format logs. Encodes the tenant-context + domain-extras + timeline-reconstruction pattern. * fix(tests): update token estimator + subagent routing test expectations Token estimator tests assumed ceil(length/4) heuristic but implementation now uses js-tiktoken (cl100k_base). Subagent routing tests failed because codex CLI is installed locally, causing isCodexAvailable() to short-circuit multiProvider/batch/Kimi paths — fixed by spying on private methods. * feat(tracing): instrument MCP server with Raindrop Workshop Every MCP tool call now emits begin/finish traces to Raindrop Workshop when RAINDROP_LOCAL_DEBUGGER env is set. Conditional — zero overhead when env var is absent. Flush on SIGINT shutdown. * chore(gepa): auto-optimizer state update * chore: add Raindrop Workshop agent config files * refactor(mcp): extract handleTool from IIFE wrapper Replace async IIFE wrapping the tool dispatch switch with a named handleTool() function. Removes one indentation level from ~500 lines, makes the Raindrop tracing wrapper cleaner, and shrinks the bundle. * feat(operator): autonomous Claude Code driver via screen control Three adapter modes behind a unified ScreenAdapter interface: - TmuxAdapter: reads pane buffer as text, sends keys (CLI, no API needed) - DesktopAdapter: macOS screenshots + AppleScript + Haiku VLM - BrowserAdapter: Playwright DOM reads for claude.ai/code web app Rule-based state machine detects: IDLE, WORKING, PERMISSION_PROMPT, ERROR, RATE_LIMITED, STUCK, SESSION_ENDED from screen content. LLM decision layer (Haiku) handles ambiguous states and generates nudges for stuck sessions instead of blind restarts. CLI: stackmemory operator start/stop/status/attach - Drains master-tasks.md queue overnight on Max plan - Auto-approves permission prompts, exponential backoff on rate limits - JSONL logging + checkpoint file for monitoring - 31 tests passing * fix(operator): nudge-then-escalate for stuck sessions + barrel export - STUCK now nudges twice before marking blocked (was: immediate block) - Track nudgeCount per task in checkpoint, reset on new task - Fix ScreenshotAdapter adapterType to match union type - Add index.ts barrel export for clean imports - 32 tests passing * feat(patterns): first-party learned patterns system (observe → learn → apply) Repackages the external instincts/continuous-learning concept into StackMemory as a native feature backed by SQLite. New modules: - PatternStore: CRUD with confidence scoring, decay, pruning - PatternObserver: extracts patterns from trace events at session end (tool sequences, error→fix pairs, tool preferences) - PatternApplier: surfaces relevant patterns in context retrieval - CLI: stackmemory patterns list|learn|stats|prune|export|import Schema: adds `patterns` table with domain, trigger, action, confidence scoring (0.3→0.85 based on observation count), project-scoping, and weekly confidence decay. 16 tests passing. * feat(patterns): add promote, projects, evolve commands — full CL-v2 parity Fills 3 feature gaps vs continuous-learning-v2: - promote: project→global (manual or auto-detect cross-project patterns) - projects: list projects with pattern counts - evolve: cluster related patterns, identify skill/command candidates PatternStore gains: promote(), projects(), promotionCandidates(), findClusters() CL-v2 for stackmemory is now retired — observer was already disabled, no hooks registered. Patterns system is the native replacement. * chore: handoff checkpoint on main * chore: handoff checkpoint on main * chore: handoff checkpoint on main * fix(test): skip OpenRouter live tests on invalid/missing API key Tests already skipped when OPENROUTER_API_KEY env var is unset, but still failed with 401 when the key existed but was expired or invalid. Now catch auth errors (401/403) and skip gracefully via ctx.skip(). * feat(webhook): add persistent retry with exponential backoff Replace in-memory webhook event queue with SQLite-backed delivery queue. Failed webhook events are now retried up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s, capped at 300s) and jitter. Deliveries persist across process restarts. - WebhookDeliveryQueue: SQLite persistence, atomic claim, background worker - Reuses existing calculateBackoff() from core/errors/recovery.ts - Health endpoint now returns delivery stats by status - 11 tests covering enqueue, retry, dead letter, concurrency, stats * fix(lint): resolve 46 pre-existing eslint errors across 8 files - Fix prettier formatting in hermes-sm, daemon-config, research-stream-service, telemetry-service - Add eslint-disable for dynamic require() in operator, adapter-factory - Replace inline require() with top-level imports in operator-logger, subagent-client * fix(storage): make redis import lazy in two-tier-storage Redis is optional (only needed for remote tier). The hard import crashed when redis wasn't installed, causing test suite failures. Now uses lazy require() with graceful fallback. * chore: handoff checkpoint on feature/webhook-retry-backoff * fix: auto-load nvm in all scripts to prevent Node version mismatch Ensures every bash script that invokes stackmemory or node loads the correct Node version from .nvmrc before executing. Prevents better-sqlite3 NODE_MODULE_VERSION errors when hooks, wrappers, or daemons run in contexts without nvm initialized (git GUIs, IDE integrations, background processes, subprocesses). * docs: mark webhook retry backoff acceptance criteria as complete --------- Co-authored-by: StackMemory Bot (CLI) <bot@stackmemory.ai>
1 parent 64a3674 commit 4d22836

1,186 files changed

Lines changed: 63491 additions & 775 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/skills/instrument-agent

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../.agents/skills/instrument-agent

.claude/skills/setup-agent-replay

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../.agents/skills/setup-agent-replay

.codex/config.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[mcp_servers.raindrop]
2+
command = "/Users/jwu/.raindrop/bin/raindrop"
3+
args = [ "workshop", "mcp" ]

.cursor/mcp.json

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"mcpServers": {
3+
"raindrop": {
4+
"command": "/Users/jwu/.raindrop/bin/raindrop",
5+
"args": [
6+
"workshop",
7+
"mcp"
8+
]
9+
}
10+
}
11+
}

.mcp.json

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"mcpServers": {
3+
"raindrop": {
4+
"command": "/Users/jwu/.raindrop/bin/raindrop",
5+
"args": [
6+
"workshop",
7+
"mcp"
8+
]
9+
}
10+
}
11+
}

CLAUDE.md

Lines changed: 65 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,27 @@
1-
# StackMemory - Project Configuration
1+
You are a senior Node.js/Express engineer working on this codebase. Write working code over explanations. Run commands before asserting state — never assume branch, file, or test status without verification.
2+
3+
# croissant.ai — Agent Guide
4+
5+
Tool-agnostic reference for AI coding agents working in this repository.
6+
7+
## Stack
8+
9+
Node.js / Express / PostgreSQL / Redis
10+
Railway deployment | Stripe / Salesforce / QuickBooks integrations
211

312
## Project Structure
413

514
```
615
src/
7-
cli/ # CLI commands and entry point
8-
core/ # Core business logic
9-
config/ # Config types and manager
10-
context/ # Frame management, enrichment, rehydration
11-
database/ # SQLite adapter, migrations, query cache
12-
digest/ # Digest generation (hybrid, chronological)
13-
errors/ # Error types and recovery
14-
merge/ # Stack merge and conflict resolution
15-
models/ # Model routing, complexity scoring
16-
monitoring/ # Logging, metrics, session monitor
17-
performance/ # Caching, profiling, benchmarks
18-
query/ # Query parsing and routing
19-
retrieval/ # Context retrieval, LLM provider
20-
session/ # Handoff, session management
21-
skills/ # Skill storage and types
22-
storage/ # Tiered storage, remote sync
23-
trace/ # Debug tracing, trace detection
24-
integrations/ # External integrations
25-
claude-code/ # Agent bridge, post-task hooks
26-
linear/ # Linear sync, webhooks, OAuth
27-
mcp/ # MCP server, 56 tool handlers
28-
ralph/ # Multi-agent swarm orchestration
29-
daemon/ # Unified daemon, session daemon
30-
features/ # Analytics, browser, sweep, TUI
31-
hooks/ # Claude Code hook handlers
32-
skills/ # Built-in skill implementations
33-
utils/ # Shared utilities
34-
scripts/ # Build and utility scripts
35-
docs/ # Documentation
16+
api/ # Route handlers
17+
core/ # monitoring-service, cache-service, queue-service, master-agent, api-validation
18+
features/ # Feature modules
19+
shared/ # Shared utilities
20+
integrations/ # Third-party connectors
21+
docs/ # Documentation
22+
scripts/ # Automation scripts
23+
docker/ # Container configs
24+
prompts/ # Externalized LLM prompt templates
3625
```
3726

3827
## Key Files
@@ -69,81 +58,64 @@ Full documentation (docs/):
6958
## Commands
7059

7160
```bash
72-
npm run build # Compile TypeScript (esbuild)
73-
npm run lint # ESLint check
74-
npm run lint:fix # Auto-fix lint issues
75-
npm run lint:fast # Fast lint via oxlint
76-
npm run typecheck # tsc --noEmit (8GB heap, avoids OOM)
77-
npm test # Run Vitest (watch)
78-
npm run test:run # Run tests once
79-
npm run linear:sync # Sync with Linear
80-
81-
# StackMemory CLI
82-
stackmemory capture # Save session state for handoff
83-
stackmemory restore # Restore from captured state
84-
stackmemory snapshot save # Post-run context snapshot (alias: snap)
85-
stackmemory snapshot list # List recent snapshots
86-
stackmemory preflight # File overlap check for parallel tasks (alias: pf)
87-
stackmemory conductor start # Autonomous Linear→worktree→agent orchestrator
88-
stackmemory conductor learn # Analyze agent outcomes (success rate, failure phases, error patterns)
89-
stackmemory conductor learn --evolve # Auto-mutate prompt template from failure data (GEPA)
90-
stackmemory conductor status # Live agent status dashboard
91-
92-
# GEPA Optimizer (scripts/gepa/optimize.js)
93-
node scripts/gepa/optimize.js run [gens] [--auto-apply] # Full optimization loop
94-
node scripts/gepa/optimize.js score [--auto-apply] # Score variants, select best
95-
node scripts/gepa/optimize.js run --target skill:start # Optimize specific target
96-
node scripts/gepa/optimize.js mutate --auto-phase # Auto-detect worst phase
97-
# Flags: --auto-apply (deploy winner), --no-cache (fresh eval), --target <name>, --phase <name>
98-
stackmemory conductor monitor # Real-time TUI with phase tracking
99-
stackmemory conductor finalize # Clean up dead/stale agents
100-
stackmemory conductor traces <issue-id> # View conversation traces for an agent run
101-
stackmemory conductor replay <session-id> # Replay full agent conversation from traces
102-
stackmemory conductor trace-stats # Aggregate trace statistics
103-
stackmemory loop "<cmd>" --until "<pattern>" # Poll until condition met (alias: watch)
61+
npm run dev # Start dev server
62+
npm run test # Run test suites (3 parallel Jest workers, maxWorkers=4)
63+
npm run lint # Lint check
64+
npm run migrate # Run DB migrations
65+
docker-compose up -d # Start local DBs
10466
```
10567

106-
## Working Directory
68+
## Git Conventions
69+
70+
- Branch prefixes: `feature/`, `fix/`, `chore/`
71+
- Commit format: `type(scope): message`
72+
- Do NOT add `Co-Authored-By` lines to commits
73+
- Pre-commit hook runs: `npm run lint` + `npm run test` + E2E browser screenshots
74+
75+
## Testing Rules
10776

108-
- PRIMARY: /Users/jwu/Dev/stackmemory
109-
- ALLOWED: All subdirectories
110-
- TEMP: /tmp for temporary operations
77+
- **Framework**: Jest + SWC
78+
- **DB mocking**: Use dependency injection (DI), not global mocks
79+
- **Supertest**: Pass `app` (NOT `server`) to supertest
80+
- **Global jest**: src/ tests use global `jest` — do NOT import from `@jest/globals` (causes redeclaration errors)
81+
- **Mock reset**: `jest.clearAllMocks()` resets `mockReturnValue` — always re-set mocks in `beforeEach`
82+
- **Test runner**: `npm test` is long-running; run in a background process or sub-agent, not inline
11183

112-
## Validation
84+
## ESLint Rules
11385

114-
Verify each step after code changes — pre-commit hooks catch 80% of CI failures locally:
115-
1. `npm run lint` - fix any errors AND warnings
116-
2. `npm run test:run` - verify no regressions
117-
3. `npm run build` - ensure compilation
118-
4. Run code to verify it works
86+
- Use `catch {}` not `catch (_err) {}` — underscore prefix not in the allowed pattern
87+
- CJS format for JS files in `src/`
11988

120-
Test coverage:
121-
- New features require tests in `src/**/__tests__/`
122-
- Maintain or improve coverage (no untested code paths)
123-
- Critical paths: context management, handoff, Linear sync
89+
## Key Patterns
12490

125-
Testing rules:
126-
- Run `npm run test:run` via subagent or background task — never inline (blocks context)
127-
- ESLint: use `catch {}` not `catch (_err) {}` (lint rule)
128-
- `vi.clearAllMocks()` resets `mockReturnValue` — re-set mocks in `beforeEach`
129-
- Pre-commit hook runs: lint + parallel vitest + build — fix issues before commit, never skip
91+
- Provenance tracking: every data point includes source, timestamp, lineage
92+
- Multi-tenant container isolation
93+
- DI route factories for testability
94+
- Error handling: return undefined over throwing; log and continue over crashing
95+
- Add `.js` extension to relative ESM imports
13096

131-
## Git Rules
97+
## Task Steering
13298

133-
The pre-commit hook enforces lint + test + build. Fix the underlying issue rather than bypassing it.
99+
**`master-tasks.md`** is the single source of truth for what to build. Agents must:
134100

135-
- Do not use `--no-verify` on git push or commit — fix the hook failure instead
136-
- Fix lint/test errors before pushing
137-
- If pre-push hooks fail, fix the underlying issue
138-
- Run `npm run lint && npm run test:run` before pushing
139-
- Commit message format: `type(scope): message`
140-
- Branch naming: `feature/STA-XXX-description` | `fix/STA-XXX-description` | `chore/description`
101+
1. Read `master-tasks.md` before starting work (especially via `/next`)
102+
2. Pick the highest-priority (`P0` > `P1` > `P2`) non-blocked `todo` task
103+
3. Prefer tasks with `owner=@agent` over `owner=@me` (unless user overrides)
104+
4. Update task status to `active` when starting, `done` when complete
105+
5. Add branch/PR info to the table row
106+
6. Never create tasks in Linear or GitHub unless `sync` column says so
141107

142-
## Task Management
108+
## StackMemory Context Rule
143109

144-
- Use TodoWrite for 3+ steps or multiple requests
145-
- Keep one task in_progress at a time
146-
- Update task status immediately on completion
110+
- When an agent fetches conversation context for active work, it must pass the exact current assignment or question as `task_query`.
111+
- Prefer the MCP shape:
112+
- `org_id`
113+
- `conversation_id`
114+
- `worker_mode: true`
115+
- `task_query`
116+
- `recover_on_low_signal: true`
117+
- Do not fetch raw `get_conversation` context for worker execution unless full transcript behavior is explicitly required.
118+
- The current assignment is persisted under `.stackmemory/worker-context/current-assignment.json` so wrappers and hooks can auto-fill or enforce `task_query`.
147119

148120
## Security
149121

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Plan: Webhook Retry with Exponential Backoff
2+
3+
## Summary
4+
5+
Add persistent retry with exponential backoff to webhook event processing. Replace the in-memory `eventQueue` in `webhook-server.ts` with a SQLite-backed delivery queue that tracks attempts, applies exponential backoff with jitter, and respects circuit breaker state.
6+
7+
## Existing Infrastructure to Leverage
8+
9+
- **`src/core/errors/recovery.ts`**: `retry()`, `calculateBackoff()`, `CircuitBreaker` — all production-ready
10+
- **`src/integrations/linear/webhook-server.ts`**: Current in-memory queue (`eventQueue[]`, `processQueue()`)
11+
- **`src/core/database/sqlite-adapter.ts`**: SQLite persistence layer
12+
- **Error codes**: `LINEAR_WEBHOOK_FAILED`, `LINEAR_API_ERROR` already exist
13+
14+
## Files to Change
15+
16+
| File | Action | Purpose |
17+
|---|---|---|
18+
| `src/integrations/linear/webhook-retry.ts` | CREATE | Delivery queue + retry worker |
19+
| `src/integrations/linear/webhook-server.ts` | MODIFY | Replace in-memory queue with persistent queue |
20+
| `src/integrations/linear/__tests__/webhook-retry.test.ts` | CREATE | Tests for retry logic |
21+
22+
## Data Model
23+
24+
New table: `webhook_deliveries` (added inline in webhook-retry.ts, not in global migrations — this is integration-scoped)
25+
26+
```sql
27+
CREATE TABLE IF NOT EXISTS webhook_deliveries (
28+
id TEXT PRIMARY KEY,
29+
event_type TEXT NOT NULL,
30+
payload TEXT NOT NULL,
31+
status TEXT NOT NULL DEFAULT 'pending', -- pending | processing | completed | failed | dead
32+
attempts INTEGER NOT NULL DEFAULT 0,
33+
max_attempts INTEGER NOT NULL DEFAULT 5,
34+
next_retry_at INTEGER, -- unix ms
35+
last_error TEXT,
36+
created_at INTEGER NOT NULL,
37+
updated_at INTEGER NOT NULL
38+
);
39+
CREATE INDEX IF NOT EXISTS idx_webhook_deliveries_status_retry
40+
ON webhook_deliveries(status, next_retry_at);
41+
```
42+
43+
## Implementation Steps
44+
45+
### Step 1: Create `webhook-retry.ts`
46+
47+
- `WebhookDeliveryQueue` class
48+
- `constructor(dbPath: string, options?: RetryConfig)` — opens/creates SQLite DB, ensures table
49+
- `enqueue(eventType: string, payload: object): string` — inserts delivery, returns ID
50+
- `processNext(): Promise<boolean>` — picks oldest `pending` or retriable `failed` delivery where `next_retry_at <= now`, marks `processing`, calls handler, updates status
51+
- `startWorker(intervalMs?: number): void` — setInterval loop calling `processNext()`
52+
- `stopWorker(): void` — clearInterval
53+
- `getStats(): { pending, processing, completed, failed, dead }` — counts by status
54+
- Uses `calculateBackoff()` from `recovery.ts` for next_retry_at computation
55+
- Marks delivery `dead` after max_attempts exceeded
56+
- Config: `{ maxAttempts: 5, initialDelay: 1000, maxDelay: 300_000, backoffFactor: 2 }`
57+
58+
### Step 2: Modify `webhook-server.ts`
59+
60+
- Replace `eventQueue: LinearWebhookPayload[]` with `WebhookDeliveryQueue` instance
61+
- In webhook endpoint handler: call `queue.enqueue()` instead of `eventQueue.push()`
62+
- Start worker in `start()`, stop in `stop()`
63+
- Remove `processQueue()` method and `isProcessing` flag
64+
65+
### Step 3: Write tests
66+
67+
- Unit tests for `WebhookDeliveryQueue`:
68+
- enqueue creates a delivery record
69+
- processNext picks the oldest pending delivery
70+
- failed delivery gets exponential backoff schedule
71+
- delivery marked dead after max_attempts
72+
- concurrent processNext doesn't double-process (status = processing guard)
73+
- getStats returns correct counts
74+
75+
## Acceptance Criteria
76+
77+
- [x] Failed webhook events are retried up to 5 times with exponential backoff
78+
- [x] Backoff schedule: 1s, 2s, 4s, 8s, 16s (capped at 300s)
79+
- [x] Delivery state persisted in SQLite — survives process restart
80+
- [x] Dead deliveries (exceeded max attempts) are logged but not retried
81+
- [x] Existing webhook signature verification unchanged
82+
- [x] Tests pass with 80%+ coverage on new code
83+
84+
## Risks
85+
86+
- **LOW**: SQLite write contention if webhook volume is high — mitigated by WAL mode (already used)
87+
- **LOW**: Worker interval drift — acceptable for webhook retry cadence (not real-time)
88+
89+
## Non-Goals
90+
91+
- Redis/BullMQ queue (overkill for single-process webhook handler)
92+
- Webhook delivery UI/dashboard
93+
- Dead letter queue notification
94+
- Outbound webhook sending (this is for processing *received* webhooks)

0 commit comments

Comments
 (0)