Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions .agents/skills/update-models/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: update-models
description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
description: Discover latest OpenAI, Anthropic, Google/Gemini, Meta, xAI, Moonshot, Tzafon, and Yutori models and verify computer-use support. Use when updating CUA model defaults, checking new model releases, auditing provider-native computer tool actions, or comparing provider metadata, official examples, and smoke-test results.
---

# Update Models
Expand All @@ -9,14 +9,14 @@ Use this workflow to keep CUA current with provider model releases and computer-

## Quick Start

1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
1. Verify credentials are available: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY` or `GEMINI_API_KEY`, `META_API_KEY`, `XAI_API_KEY`, `MOONSHOT_API_KEY`, `TZAFON_API_KEY`, and `YUTORI_API_KEY`.
2. If credentials live in `~/AGENTS.md`, load them into the current shell without printing them:

```bash
eval "$(python3 - <<'PY'
import pathlib, re, shlex
text = pathlib.Path('~/AGENTS.md').expanduser().read_text()
for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
for key in ['OPENAI_API_KEY', 'ANTHROPIC_API_KEY', 'GOOGLE_API_KEY', 'META_API_KEY', 'XAI_API_KEY', 'MOONSHOT_API_KEY', 'TZAFON_API_KEY', 'YUTORI_API_KEY']:
m = re.search(r'export\s+' + re.escape(key) + r'=(?:"([^"]+)"|([^\s\n]+))', text)
if m:
print(f'export {key}={shlex.quote(m.group(1) or m.group(2))}')
Expand Down Expand Up @@ -114,6 +114,16 @@ xAI:
- Use CUA's normalized 0-1000 coordinate instructions, `parallel_tool_calls: false`, `store: true`, and `previous_response_id` for browser loops. xAI accepts encrypted reasoning replay in these requests.
- Use `reasoning: { effort: "low" }` for low-latency smoke tests; Grok 4.5 also supports `medium` and `high`, cannot disable reasoning, and defaults to `high`.

Moonshot:

- Discover with the OpenAI SDK against `https://api.moonshot.ai/v1` using `MOONSHOT_API_KEY`.
- Smoke-test the chat completions API with screenshot input and explicit function tools matching CUA's canonical actions.
- Pass condition: response `choices[0].message.tool_calls[]` contains one of the supplied browser actions.
- Treat Kimi computer use as custom-function-tool support, not a provider-native computer tool. Moonshot documents vision input and function calling but no coordinate protocol.
- Kimi grounding emits width/height fractions from 0 to 1 regardless of prompt or schema wording; keep CUA's fractional coordinate contract and verify emitted values stay in 0-1.
- Set `parallel_tool_calls: false` because browser actions mutate shared state. There is no response threading; the full context replays each turn.
- Kimi K3 launched with max-only thinking effort. pi-ai's registry entry clamps other levels away; re-check `thinkingLevelMap` when Moonshot ships low/high modes.

Tzafon:

- Discover with `new Lightcone({ apiKey }).models.list()` from `@tzafon/lightcone` when available.
Expand Down Expand Up @@ -187,6 +197,7 @@ All CUA model and adapter support lives in `packages/ai` (`@onkernel/cua-ai`). W
- Anthropic: update the `ANTHROPIC_CUA_ACTION_TYPES` set in `packages/ai/src/providers/anthropic/actions.ts` and `index.ts`. The computer tool version and `computer-use-*` beta header are selected by `pi-ai` per model, so a new dated tool version usually means bumping `@earendil-works/pi-ai`, not editing this package.
- Gemini: update `packages/ai/src/providers/gemini/index.ts`, including coordinate handling if needed.
- xAI: update `packages/ai/src/providers/xai/index.ts` and `provider.ts`, including normalized coordinate instructions, Responses threading, and reasoning compatibility.
- Moonshot: update `packages/ai/src/providers/moonshot/index.ts`, including the fractional coordinate instructions and payload middleware. Streaming rides pi-ai's builtin `openai-completions` transport, so wire-format changes usually mean bumping `@earendil-works/pi-ai`.
- Tzafon: update `packages/ai/src/providers/tzafon/index.ts` and `provider.ts`, including coordinate/action handling.
- Yutori: update `packages/ai/src/providers/yutori/actions.ts`, `index.ts`, and `provider.ts`, including payload filtering and coordinate/action handling.
- Shared canonical action semantics go in `packages/ai/src/providers/common.ts`.
Expand Down
3 changes: 2 additions & 1 deletion .agents/skills/update-models/reference/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ These scripts support the `update-models` skill. Run them from the repository ro
- `GOOGLE_API_KEY` or `GEMINI_API_KEY`
- `META_API_KEY`
- `XAI_API_KEY`
- `MOONSHOT_API_KEY`
- `TZAFON_API_KEY`
- `YUTORI_API_KEY`

The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta and xAI use supplied function tools; other providers may use provider-native computer tools.
The scripts never print API keys. Smoke tests are non-destructive: they ask each model to emit a computer-use tool call, then inspect the response without executing the action. Meta, xAI, and Moonshot use supplied function tools; other providers may use provider-native computer tools.

## Common Commands

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { basename, join, resolve } from "node:path";
import { spawnSync } from "node:child_process";
import process from "node:process";

type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "yutori";
type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori";

interface ExampleRepo {
provider: Provider;
Expand Down Expand Up @@ -83,6 +83,7 @@ const ACTION_REGEXES: Record<Provider, RegExp[]> = {
gemini: [/\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g],
meta: [/\b(screenshot|left_click|right_click|middle_click|double_click|triple_click|left_click_drag|mouse_move|key|type|scroll|hold_key|wait|left_mouse_down|left_mouse_up)\b/g],
xai: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g],
moonshot: [/\b(screenshot|click|double_click|mouse_down|mouse_up|scroll|type|keypress|drag|move|wait)\b/g],
yutori: [/\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|goto_url|mouse_move|middle_click|mouse_down|mouse_up|go_forward|hold_key|extract_elements|find|set_element_value|execute_js)\b/g],
};

Expand Down
101 changes: 97 additions & 4 deletions .agents/skills/update-models/reference/discover-models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import { join } from "node:path";
import process from "node:process";
import { parse as parseToml } from "smol-toml";

type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "tzafon" | "yutori";
type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "tzafon" | "yutori";

interface Args {
provider: Provider | "all";
Expand Down Expand Up @@ -39,7 +39,7 @@ interface ModelResult {
cua?: Record<string, unknown>;
}

const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "tzafon", "yutori"];
const PROVIDERS: Provider[] = ["openai", "anthropic", "gemini", "meta", "xai", "moonshot", "tzafon", "yutori"];
const TZAFON_KNOWN_MODELS = [
"tzafon.northstar-cua-fast",
];
Expand Down Expand Up @@ -110,7 +110,7 @@ function usage(): never {
npx tsx .agents/skills/update-models/reference/discover-models.ts --provider openai --models gpt-5.5,gpt-5.4

Options:
--provider <all|openai|anthropic|gemini|meta|xai|tzafon|yutori>
--provider <all|openai|anthropic|gemini|meta|xai|moonshot|tzafon|yutori>
--models <comma-separated model ids> Smoke-test explicit models instead of inferred candidates.
--candidate-limit <n> Max inferred candidates per provider. Default: 20.
--no-smoke Only list metadata.
Expand Down Expand Up @@ -140,6 +140,7 @@ async function runProvider(provider: Provider, args: Args): Promise<Record<strin
if (provider === "gemini") return await discoverGemini(args);
if (provider === "meta") return await discoverMeta(args);
if (provider === "xai") return await discoverXai(args);
if (provider === "moonshot") return await discoverMoonshot(args);
if (provider === "tzafon") return await discoverTzafon(args);
if (provider === "yutori") return await discoverYutori(args);
throw new Error(`unknown provider ${provider satisfies never}`);
Expand Down Expand Up @@ -334,6 +335,98 @@ function likelyXaiGenerationModel(id: string): boolean {
return lower.startsWith("grok-") && !lower.includes("imagine");
}

async function discoverMoonshot(args: Args): Promise<Record<string, unknown>> {
const OpenAI = await importDefault("openai", "OpenAI");
const client = new OpenAI({ apiKey: process.env.MOONSHOT_API_KEY, baseURL: "https://api.moonshot.ai/v1" });
const rawModels = await collectAsync(client.models.list());
const models: ModelResult[] = rawModels.map((m) => ({
id: String(m.id),
display_name: String(m.id),
created_at: typeof m.created === "number" && m.created > 0 ? new Date(m.created * 1000).toISOString() : null,
raw: m,
supports_generation: String(m.id).toLowerCase().startsWith("kimi-"),
model_docs: {
url: "https://platform.kimi.ai/docs/api/tool-use",
chat_completions_endpoint: "supported",
function_calling: "supported",
image_input: "verify-per-model",
coordinate_space: "CUA-defined 0-1 fractions",
},
}));
const candidates = explicitOrCandidates(
args,
models
.filter((model) => model.supports_generation)
.sort(compareMoonshotCandidates)
.map((model) => model.id),
);
if (args.smoke) {
await Promise.all(candidates.map(async (id) => {
const model = models.find((candidate) => candidate.id === id) ?? { id, display_name: id, supports_generation: true };
model.computer_use = await smokeMoonshot(client, id);
if (!models.find((candidate) => candidate.id === id)) models.unshift(model);
}));
}
await annotateCuaSupport("moonshot", models);
return { provider: "moonshot", metadata_source: "Moonshot models.list()", models, candidates };
}

async function smokeMoonshot(client: any, model: string): Promise<SmokeResult> {
try {
const screenshot = await readFile(fixtureScreenshotPath());
const response = await client.chat.completions.create({
model,
parallel_tool_calls: false,
max_tokens: 8192,
messages: [
{
role: "system",
content: "Coordinates are fractions of the screenshot, normalized from 0 to 1.",
},
{
role: "user",
content: [
{ type: "text", text: "Call the click tool for the sign in link. Do not answer only in text." },
{ type: "image_url", image_url: { url: `data:image/png;base64,${screenshot.toString("base64")}` } },
],
},
],
tools: [{
type: "function",
function: {
name: "click",
description: "Click at coordinates given as 0-1 fractions of the screenshot.",
parameters: {
type: "object",
properties: { x: { type: "number" }, y: { type: "number" } },
required: ["x", "y"],
additionalProperties: false,
},
},
}],
});
const calls: any[] = response.choices?.[0]?.message?.tool_calls ?? [];
return {
status: calls.length > 0 ? "pass" : "inconclusive",
tool_name: "function_tools",
tool_version: null,
beta_header: null,
observed_actions: unique(calls.map((call) => call?.function?.name).filter(Boolean)),
response_item_types: unique([response.choices?.[0]?.finish_reason].filter(Boolean)),
coordinate_space: "0-1 fractions",
error: null,
};
} catch (err) {
return smokeError(err, { tool_name: "function_tools" });
}
}

function compareMoonshotCandidates(a: ModelResult, b: ModelResult): number {
if (a.id === "kimi-k3") return -1;
if (b.id === "kimi-k3") return 1;
return String(b.created_at ?? "").localeCompare(String(a.created_at ?? ""));
}

function compareXaiCandidates(a: ModelResult, b: ModelResult): number {
if (a.id === "grok-4.5") return -1;
if (b.id === "grok-4.5") return 1;
Expand Down Expand Up @@ -839,7 +932,7 @@ const TZAFON_FUNCTION_TOOLS = [
];

async function annotateCuaSupport(provider: Provider, models: ModelResult[]): Promise<void> {
const piProvider = provider === "gemini" ? "google" : provider;
const piProvider = provider === "gemini" ? "google" : provider === "moonshot" ? "moonshotai" : provider;
const getBuiltinModel = await import("@earendil-works/pi-ai/providers/all").then((mod) => mod.getBuiltinModel).catch(() => undefined);
for (const model of models) {
const inRegistry = getBuiltinModel ? !!getBuiltinModel(piProvider as never, model.id as never) : false;
Expand Down
16 changes: 13 additions & 3 deletions .agents/skills/update-models/reference/provider-doc-drift.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import { readFile, writeFile } from "node:fs/promises";
import process from "node:process";

type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "yutori";
type Provider = "openai" | "anthropic" | "gemini" | "meta" | "xai" | "moonshot" | "yutori";

interface Args {
examples: string;
Expand Down Expand Up @@ -40,6 +40,11 @@ const DOCS: Record<Provider, string[]> = {
"https://docs.x.ai/developers/tools/function-calling",
"https://docs.x.ai/developers/model-capabilities/images/understanding",
],
moonshot: [
"https://www.kimi.com/blog/kimi-k3",
"https://platform.kimi.ai/docs/api/tool-use",
"https://platform.kimi.ai/docs/guide/use-kimi-vision-model",
],
yutori: [
"https://docs.yutori.com/reference/navigator",
"https://docs.yutori.com/reference/n1",
Expand All @@ -54,6 +59,7 @@ const LOCAL_FILES: Record<Provider, string> = {
gemini: "packages/ai/src/providers/gemini/index.ts",
meta: "packages/ai/src/providers/meta/index.ts",
xai: "packages/ai/src/providers/xai/index.ts",
moonshot: "packages/ai/src/providers/moonshot/index.ts",
yutori: "packages/ai/src/providers/yutori/actions.ts",
};

Expand All @@ -63,6 +69,7 @@ const ACTION_REGEXES: Record<Provider, RegExp> = {
gemini: /\b(open_web_browser|open_web|wait_5_seconds|go_back|go_forward|search|navigate|click_at|hover_at|type_text_at|key_combination|scroll_document|scroll_at|drag_and_drop)\b/g,
meta: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position|left_click|right_click|middle_click|triple_click|left_click_drag|mouse_move|key|hold_key|left_mouse_down|left_mouse_up)\b/g,
xai: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g,
moonshot: /\b(click|double_click|mouse_down|mouse_up|scroll|type|wait|keypress|drag|move|screenshot|goto|back|forward|url|cursor_position)\b/g,
yutori: /\b(left_click|double_click|triple_click|right_click|scroll|type|key_press|hover|drag|wait|refresh|go_back|go_forward|goto_url|mouse_move|middle_click|mouse_down|mouse_up|hold_key|extract_elements|find|set_element_value|execute_js)\b/g,
};

Expand Down Expand Up @@ -137,8 +144,8 @@ async function checkProvider(provider: Provider, examples: any): Promise<Record<
documented_actions: sorted(documentedActions),
example_repo_actions: sorted(exampleActions),
repo_supported_actions: sorted(localActions),
unknown_documented_actions: provider === "meta" || provider === "xai" ? [] : sorted(difference(new Set(documentedActions), new Set(localActions))),
unknown_example_actions: provider === "meta" || provider === "xai" ? [] : sorted(difference(new Set(exampleActions), new Set(localActions))),
unknown_documented_actions: provider === "meta" || provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(documentedActions), new Set(localActions))),
unknown_example_actions: provider === "meta" || provider === "xai" || provider === "moonshot" ? [] : sorted(difference(new Set(exampleActions), new Set(localActions))),
response_fields_from_examples: sorted(example.response_fields ?? []),
notes: notesFor(provider, documentedToolVersions, exampleToolVersions, localToolVersions),
};
Expand Down Expand Up @@ -174,6 +181,9 @@ function notesFor(provider: Provider, documentedToolVersions: string[], exampleT
if (provider === "xai") {
notes.push("xAI uses developer-defined function tools and does not document a native coordinate protocol; compare Responses compatibility, reasoning controls, and CUA's normalized coordinate contract instead.");
}
if (provider === "moonshot") {
notes.push("Moonshot uses developer-defined function tools over OpenAI-compatible chat completions and does not document a coordinate protocol; Kimi grounding emits 0-1 width/height fractions, so compare against CUA's fractional coordinate contract.");
}
if (provider === "yutori") {
notes.push("Yutori Navigator emits OpenAI-compatible tool_calls for built-in browser actions; local AgentTools should execute those names but outbound payloads should not duplicate the built-in browser schemas.");
notes.push("Track n1 vs n1.5 separately because n1.5 can add tool_set/disable_tools and expanded browser actions.");
Expand Down
1 change: 1 addition & 0 deletions .agents/skills/update-models/reference/report-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Use this shape for JSON reports and the same fields when writing a Markdown summ
"gemini": {},
"meta": {},
"xai": {},
"moonshot": {},
"tzafon": {},
"yutori": {}
},
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ jobs:
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
META_API_KEY: ${{ secrets.META_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
run: npm run test:integration --workspace @onkernel/cua-ai
Expand Down Expand Up @@ -162,6 +163,7 @@ jobs:
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
META_API_KEY: ${{ secrets.META_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
TZAFON_API_KEY: ${{ secrets.TZAFON_API_KEY }}
YUTORI_API_KEY: ${{ secrets.YUTORI_API_KEY }}
run: npm test --workspace @onkernel/cua-agent -- test/e2e.live.test.ts
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # for claude-opus-4-7
export GOOGLE_API_KEY=... # for gemini-3-flash-preview
export META_API_KEY=... # for muse-spark-1.1
export XAI_API_KEY=xai-... # for grok-4.5
export MOONSHOT_API_KEY=sk-... # for kimi-k3
export YUTORI_API_KEY=yt_... # for n1.5-latest
export KERNEL_API_KEY=sk_... # always required

Expand All @@ -127,6 +128,9 @@ cua -p --model meta:muse-spark-1.1 "Same prompt"
# xAI Grok 4.5
cua -p --model xai:grok-4.5 "Same prompt"

# Moonshot Kimi K3
cua -p --model moonshotai:kimi-k3 "Same prompt"

# Yutori Navigator
cua -p --model n1.5-latest "Same prompt"

Expand Down
Loading
Loading