file transfer (ts): parse images/files, wire directive sink to Rust parity - #346
Merged
Conversation
…arity Implement the PR #342 file-transfer contract in the TypeScript smooth-operator server, to parity with the Rust reference: - Regenerate the client SDK protocol types so send_message gains images[]/files[] and eventual_response gains directive. - images[] are attached to the turn's user message as OpenAI image_url content parts. The published core takes a plain-string message and reuses it for retrieval, so a withUserImages() wrapper rewrites the outgoing request body's current user message instead (retrieval/memory stay text-based). - files[] are surfaced on a new per-turn ToolContext (never sent to the model), mirroring the Rust ToolProviderContext. - New optional toolProvider seam (mirrors Rust ToolProvider): host tools bound to the turn's context can read the attachments and write ctx.directive; the dispatcher drains it onto eventual_response.directive (last-write-wins). - All attachment parsing is fail-soft. - Tests: withUserImages/parse unit tests + end-to-end images-attach, files-on-context, tool->directive->eventual_response, and fail-soft. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: bbd1c0f The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The file-transfer protocol contract merged in #342 (
send_message.files[]+images[], and thesend_filedirective convention oneventual_response.directive) was spec-only. The Rust server already implements images/directive; the TypeScript server parsedmessageonly, built the user message text-only, and never emitted adirective.Solution — TS server to Rust parity
typescript/src/generated/types.tsviascripts/generate.ts):send_messagegainsimages[]/files[];eventual_responsegainsdirective.image_urlcontent parts. The published@smooai/smooth-operator-coreengine takes a plain-string message and reuses it for retrieval/memory, so a newwithUserImages()wrapper rewrites the outgoing request body's current user message ("text"→[{type:'text'},{type:'image_url'}]) — the exact shape Rust'swith_user_imagesproduces — while leaving text-based retrieval untouched.ToolContext(parallel to images), never sent to the model — mirroring the RustToolProviderContext.toolProviderseam (mirrors RustToolProvider::tools_for) hands host tools the per-turn context. A host tool writesctx.directive; the dispatcher drains it after the turn and passes it to theeventual_responsebuilder (last-write-wins, omitted when none).Notes
ponytail:upgrade path to a corewithUserImageswhen the engine gains one (as Rust has).Tests / gates
withUserImages+parseImages/parseFilesunit tests, plus end-to-end (real WS): images-attach, files-on-context,tool → directive → eventual_response, back-compat omit, and fail-soft. 12 new tests.@smooai/smooth-operatorand@smooai/smooth-operator-server(245 server + 41 SDK tests pass).🤖 Generated with Claude Code