Skip to content

file transfer (ts): parse images/files, wire directive sink to Rust parity - #346

Merged
brentrager merged 1 commit into
mainfrom
ft-ts
Aug 11, 2026
Merged

file transfer (ts): parse images/files, wire directive sink to Rust parity#346
brentrager merged 1 commit into
mainfrom
ft-ts

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Problem

The file-transfer protocol contract merged in #342 (send_message.files[] + images[], and the send_file directive convention on eventual_response.directive) was spec-only. The Rust server already implements images/directive; the TypeScript server parsed message only, built the user message text-only, and never emitted a directive.

Solution — TS server to Rust parity

  • Regen protocol types (typescript/src/generated/types.ts via scripts/generate.ts): send_message gains images[]/files[]; eventual_response gains directive.
  • Images → model: parsed in the dispatcher and attached to the turn's user message as OpenAI image_url content parts. The published @smooai/smooth-operator-core engine takes a plain-string message and reuses it for retrieval/memory, so a new withUserImages() wrapper rewrites the outgoing request body's current user message ("text"[{type:'text'},{type:'image_url'}]) — the exact shape Rust's with_user_images produces — while leaving text-based retrieval untouched.
  • Files → tool context: parsed and surfaced on a new per-turn ToolContext (parallel to images), never sent to the model — mirroring the Rust ToolProviderContext.
  • Directive sink: a new optional toolProvider seam (mirrors Rust ToolProvider::tools_for) hands host tools the per-turn context. A host tool writes ctx.directive; the dispatcher drains it after the turn and passes it to the eventual_response builder (last-write-wins, omitted when none).
  • All attachment parsing is fail-soft (a malformed entry is dropped, never rejects the turn).

Notes

  • The TS core engine (1.7.1) has no multimodal seam; the request-body wrapper is the workaround, with a ponytail: upgrade path to a core withUserImages when the engine gains one (as Rust has).
  • Additive + back-compat: no attachments / no provider ⇒ behaviour byte-for-byte unchanged. Existing conformance/scenario-parity suites stay green with no edits.

Tests / gates

  • New withUserImages + parseImages/parseFiles unit tests, plus end-to-end (real WS): images-attach, files-on-context, tool → directive → eventual_response, back-compat omit, and fail-soft. 12 new tests.
  • Green: typecheck + test + build for @smooai/smooth-operator and @smooai/smooth-operator-server (245 server + 41 SDK tests pass).
  • Changeset added (minor on both packages).

🤖 Generated with Claude Code

…arity

Implement the PR #342 file-transfer contract in the TypeScript smooth-operator
server, to parity with the Rust reference:

- Regenerate the client SDK protocol types so send_message gains images[]/files[]
  and eventual_response gains directive.
- images[] are attached to the turn's user message as OpenAI image_url content
  parts. The published core takes a plain-string message and reuses it for
  retrieval, so a withUserImages() wrapper rewrites the outgoing request body's
  current user message instead (retrieval/memory stay text-based).
- files[] are surfaced on a new per-turn ToolContext (never sent to the model),
  mirroring the Rust ToolProviderContext.
- New optional toolProvider seam (mirrors Rust ToolProvider): host tools bound to
  the turn's context can read the attachments and write ctx.directive; the
  dispatcher drains it onto eventual_response.directive (last-write-wins).
- All attachment parsing is fail-soft.
- Tests: withUserImages/parse unit tests + end-to-end images-attach,
  files-on-context, tool->directive->eventual_response, and fail-soft.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: bbd1c0f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@smooai/smooth-operator-server Minor
@smooai/smooth-operator Minor
@smooai/smooth-operator-web-chat-example Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@brentrager
brentrager merged commit d41643e into main Aug 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant