feat(askui-nodejs)!: replace WebSocket UI Controller with gRPC AgentOS client#879
Merged
Merged
Conversation
…S client Connect to the AskUI AgentOS (AskUI Remote Device Controller) via gRPC on localhost:23000 (standalone) or localhost:26000 (AskuiCoreService-managed), mirroring the connection approach of the AskUI Python SDK (vision-agent). - add AgentOsClient with vendored Controller_V1.proto (@grpc/grpc-js, @grpc/proto-loader); handshake: StartSession -> StartExecution -> SetActiveDisplay; actions via RunRecordedAction + Poll(ActionFinished) - detect the Windows AskuiCoreService and prefer the service-managed port - convert CaptureScreen BGRA bitmaps to base64 PNG via sharp - remove the legacy WebSocket client, runner protocol, and UiController binary download/spawn stack; drop ws, fkill, and wait-port dependencies - add test.ts (npm run test:agentos) as a manual end-to-end check against a running AgentOS BREAKING CHANGE: startVideoRecording, stopVideoRecording, readVideoRecording, annotateInteractively, and the UiController export have been removed. Android input events throw AgentOsActionNotSupportedError until Android support is reintroduced. Default uiControllerUrl changed from http://127.0.0.1:6769 to localhost:23000. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add an AndroidAdbClient that drives an Android device by shelling out to adb
directly (input tap/swipe/keyevent, screencap, wm size), replacing the legacy
UI Controller's WebSocket/adbkit path. No separate controller process or
bundled binary is required.
- introduce a transport-agnostic DeviceClient interface that both
AgentOsClient (desktop, gRPC) and AndroidAdbClient (Android, adb) implement;
ExecutionRuntime now depends on the interface
- select the client by a new ClientArgs.runtime ('desktop' | 'android'); the
optional android options object carries device id, adb path, and keyboard
mode. Desktop usage is unchanged (runtime defaults to 'desktop')
- getStartingArguments() reports runtime:'android', which re-activates the
existing Android agent tool set and system prompt
- add test-android.ts (npm run test:android); verified end-to-end on a real
device (tap, swipe, type, keyevents, screenshot, lifecycle)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…OsUrl - add ClientArgs.agentOsUrl; keep uiControllerUrl as a deprecated alias (used only when agentOsUrl is unset) that logs a deprecation warning - default the AgentOS address to the OS service (localhost:26000), which is the recommended install; connect to a standalone AgentOS by setting agentOsUrl to localhost:23000 explicitly - drop the standalone auto-fallback and the Windows sc-query detection: the SDK does not download or spawn a standalone binary, so it now connects to a single configured address - on connection failure, point users to the AgentOS installation docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Android can now be automated either through the AskUI legacy UI Controller (WebSocket runner-protocol) or directly via adb, selected by ClientArgs.android.transport. - add LegacyAndroidClient implementing DeviceClient over the legacy controller's WebSocket runner-protocol (CONTROL_REQUEST, CAPTURE_SCREENSHOT_REQUEST, GET_STARTING_ARGUMENTS_REQUEST), default ws://127.0.0.1:6769; single request in flight to match the controller - default the Android transport to 'legacy-controller'; opt into direct adb with android.transport: 'adb' - the controller must be started separately in Android mode (AskUI-StartController ... -r android) - extend test-android.ts to cover both transports (TRANSPORT env var) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the legacy WebSocket "UI Controller" connection (port 6769) in `askui-nodejs` with a gRPC client for the new AskUI AgentOS (AskUI Remote Device Controller), mirroring how the AskUI Python SDK (`vision-agent`) connects.
The SDK now speaks gRPC to `localhost:23000` (standalone AgentOS) or `localhost:26000` (when the Windows `AskuiCoreService` manages it), using the vendored `Controller_V1.proto` (`Askui.API.TDKv1.ControllerAPI`).
What changed
Breaking changes
Testing
Follow-ups
🤖 Generated with Claude Code