Skip to content

feat: connect orchestrator and frontend#1564

Open
Scra3 wants to merge 172 commits into
mainfrom
feat/prd-214-server-step-mapper
Open

feat: connect orchestrator and frontend#1564
Scra3 wants to merge 172 commits into
mainfrom
feat/prd-214-server-step-mapper

Conversation

@Scra3
Copy link
Copy Markdown
Member

@Scra3 Scra3 commented Apr 20, 2026

MAIN BRANCH TO INTRODUCE WORKFLOW EXECUTOR.

image

Note

Connect workflow orchestrator to frontend via a new workflow-executor package and agent proxy routes

  • Introduces @forestadmin/workflow-executor, a standalone service that polls the Forest server for pending workflow runs and executes them step-by-step using AI-driven executors for condition, read-record, update-record, trigger-action, load-related-record, MCP, and guidance step types.
  • Adds a Koa HTTP server (ExecutorHttpServer) exposing /health, GET /runs/:runId, and POST /runs/:runId/trigger so the frontend can query step state and submit pending-data confirmations.
  • Adds WorkflowExecutorProxyRoute to the agent, which forwards /_internal/workflow-executions/:runId (GET/POST) to the configured workflowExecutorUrl; the route is registered only when workflowExecutorUrl is set in agent options.
  • Adds persistence adapters including an in-memory store and a SQLite/Postgres DatabaseStore (with Umzug migrations), plus activity-log, agent-client, and AI-proxy adapters with retry logic.
  • Extracts AiClient, createBaseChatModel, and getAiConfiguration into @forestadmin/ai-proxy and exports LangChain types, enabling the executor to support both external AI configs and a server-routed AI proxy.
  • Risk: the agent now exposes unauthenticated-at-the-agent-level proxy routes under /_internal/workflow-executions/; access control relies on JWT middleware inside the executor HTTP server.

Macroscope summarized 2b7a1b5.

matthv and others added 30 commits March 17, 2026 15:00
…premature deps, add smoke test

- Rewrite CLAUDE.md with project overview and architecture principles, remove changelog
- Remove unused dependencies (ai-proxy, sequelize, zod) per YAGNI
- Add smoke test so CI passes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… document system architecture

- Lint now covers src and test directories
- Replace require() with import, use stronger assertion (toHaveLength)
- Add System Architecture section describing Front/Orchestrator/Executor/Agent
- Mark Architecture Principles as planned (not yet implemented)
- Remove redundant test/.gitkeep
- Make index.ts a valid module with export {}

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…erver (#1504)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: alban bertolini <albanb@forestadmin.com>
…ain (#1512)

Co-authored-by: alban bertolini <albanb@forestadmin.com>
- Remove McpClient.tools property, loadTools() returns local array
- Rename closeConnections() → dispose()
- Rename testConnections() → checkConnection()
- Add McpServers type export
- Rename mcpServerConfigs → toolConfigs in create-ai-provider
- Update all tests accordingly

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…eability

Add sourceId to McpToolRef so that persisted execution data (pendingData,
executionParams) tracks which MCP server provided the tool. This fixes
tool lookup on re-entry (confirmation flow) when multiple servers expose
tools with the same name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… collection schema

- Replace non-null assertion with explicit McpToolNotFoundError when AI
  selects a tool name that doesn't match any available tool
- Resolve related collection name from parent schema before looking up
  the related schema in cache, fixing cases where relation name differs
  from target collection name

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge main into feature branch, resolve conflicts by taking main's
versions of router.ts, create-ai-provider.ts and their tests.
Remove deleted mcp-config-checker.ts. Bump workflow-executor internal
deps to match main.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
alban bertolini and others added 5 commits April 30, 2026 12:52
…ed signal

Replace idempotencyPhase === undefined with executionResult === undefined
in StepSummaryBuilder. The previous check produced false positives for
non-mutating steps (load-related-record, skipped paths, trigger-action via
saveFrontendResult) that never set idempotencyPhase but do set executionResult.

Any normal executor completion always writes executionResult before reporting
the step as done, making its absence the correct invariant.

Add regression tests covering all false-positive paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lder test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cordStepExecutor

getAvailableRecordRefs, selectRecordRef, toRecordIdentifier, getCollectionSchema,
and findField were only used by the 4 record executors that already extend
RecordStepExecutor. Moving them keeps BaseStepExecutor generic and confines
record-domain logic to the correct layer.

Also extract idempotencyPhase into MutatingStepExecutionData so non-mutating
step types (condition, read-record, guidance, load-related-record) no longer
carry a field that can never be set on them.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…TCH body

The server validator (Joi) only accepts { status } — extra fields cause a
400 ValidationFailedError. The error message is still logged locally via
stepErrorMessage on failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nt camelCase deserialization

The JSON:API deserializer in agent-client converts all attribute keys to camelCase
(card_number → cardNumber). ReadRecordStepExecutor looks up values by original field name,
so snake_case fields came back undefined and were stripped from executionResult.

Add restoreFieldNames() in AgentClientAgentPort.getRecord to reverse the camelCase
mapping using the original field names from the query, ensuring executors receive
values keyed by the field name they requested.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread packages/workflow-executor/src/adapters/agent-client-agent-port.ts Outdated
alban bertolini and others added 16 commits April 30, 2026 16:41
…d and values

Same camelCase deserialization issue as getRecord: apply restoreFieldNames to
related records using both primaryKeyFields and requested fields so that
extractRecordId finds the correct PK values and callers receive snake_case keys.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lues

Same camelCase deserialization issue as getRecord/getRelatedData: agent-client's
JSON:API deserializer converts response keys to camelCase. Restore original names
using the input values keys, which are already in the caller's original format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…client to 1.39.5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pending data

When the executor picks up a guidance step from the polling loop (before
the user has submitted anything), it now returns awaiting-input instead of
throwing StepStateError. This keeps the step pending so the user trigger
can process it with the submitted data.

Also makes userInput optional so users can submit a guidance step without
providing any text input.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…restoreFieldNames

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… rethrow path

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ect to fix OpenAI rejection

OpenAI requires type: "object" at the root of a tool schema. Using
z.discriminatedUnion directly as schema serialized to anyOf, causing
a 400 "got type: None" error on update-data steps with ≥2 fields.

Wrap in z.object({ input: ... }) (same pattern as the frontend) so
the root is always type: "object". Switch to z.union for the field
variants — discriminatedUnion brought no benefit for LLM selection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… root schema shape

Add two tests to prevent the OpenAI 400 regression from silently
re-appearing: one asserting the root schema is a ZodObject (not a
union), and one covering the multi-field z.union path with a flat
payload rejection check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…apper for update-record-field

The integration tests were still passing the old flat args shape
{ fieldName, value, reasoning } to the mock model. After wrapping
the tool schema in z.object({ input: ... }), the executor now
destructures result.input — so the mocks need the wrapper too.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r) when finding pending step

After a back change, errored steps are stored as done:false + context.error instead of done:true.
The find() must exclude them so the executor picks the next genuinely pending step and not the
already-failed one.

Also includes a temporary throw in ReadRecordStepExecutor for manual front-end error testing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… first non-done

The orchestrator is the source of truth for which step to execute next.
Always pick the last entry in workflowHistory rather than scanning for
the first non-done/non-cancelled/non-errored step.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
run: ServerHydratedWorkflowRun,
err: WorkflowExecutorError,
): MalformedRunInfo {
const pending = run.workflowHistory.at(-1) ?? null;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium adapters/forest-server-workflow-port.ts:134

In toMalformedInfo, accessing run.workflowHistory.at(-1) throws TypeError: Cannot read properties of undefined (reading 'at') when workflowHistory is null or undefined. Since this method is only called from error handlers during malformed run recovery, a missing workflowHistory causes the error handling itself to crash. Consider guarding the access with optional chaining: run.workflowHistory?.at(-1).

Suggested change
const pending = run.workflowHistory.at(-1) ?? null;
const pending = run.workflowHistory?.at(-1) ?? null;
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file packages/workflow-executor/src/adapters/forest-server-workflow-port.ts around line 134:

In `toMalformedInfo`, accessing `run.workflowHistory.at(-1)` throws `TypeError: Cannot read properties of undefined (reading 'at')` when `workflowHistory` is `null` or `undefined`. Since this method is only called from error handlers during malformed run recovery, a missing `workflowHistory` causes the error handling itself to crash. Consider guarding the access with optional chaining: `run.workflowHistory?.at(-1)`.

Evidence trail:
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts line 134: `run.workflowHistory.at(-1)` without optional chaining
packages/workflow-executor/src/adapters/server-types.ts line 120: `workflowHistory: ServerStepHistory[]` (non-nullable type, but data from network)
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 63-76: catch block calling `toMalformedInfo` at line 69 when error is `WorkflowExecutorError`
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 107-128: `toDispatch` throws `InvalidStepDefinitionError` at lines 109 and 118 BEFORE `workflowHistory` is accessed
packages/workflow-executor/src/adapters/forest-server-workflow-port.ts lines 55-57: data comes from server HTTP response with no runtime type validation

alban bertolini and others added 2 commits May 18, 2026 12:08
… run)

at(-1) picks the orchestrator's current step but must still return null
when that step is already done — the run is complete and nothing to execute.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Scra3 Scra3 changed the base branch from main to feat/prd-214-setup-workflow-executor-package May 18, 2026 13:00
@Scra3 Scra3 changed the base branch from feat/prd-214-setup-workflow-executor-package to main May 18, 2026 13:03
alban bertolini and others added 2 commits May 19, 2026 18:37
The model sometimes returns snake_case (first_name) when the actual
displayName is camelCase or unseparated (firstname). Add a normalized
fallback that strips separators and lowercases before matching, so a
hallucinated variation doesn't kill an otherwise correct step.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…allback

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants