diff --git a/EVENTS.md b/EVENTS.md index 4229767..99563e8 100644 --- a/EVENTS.md +++ b/EVENTS.md @@ -538,12 +538,20 @@ These events are additive in schema version `"1"` and emitted only when The schema is validated before any model request, with Ajv defaults and `allErrors: true` (draft-07, or Ajv2020 for a declared draft 2020-12). Validation does not coerce types, insert defaults or remove extra properties. - -`--output-schema-retries ` permits N additional corrective attempts (default 2). -Zero permits exactly one attempt. `--output-result ` writes only accepted -JSON using two-space indentation and a trailing newline, with a temporary file -in the destination directory followed by an atomic rename. Failures preserve the -file. Both flags require `--output-schema`; the destination directory must exist and be writable at configuration time. +Serialized schemas are limited to 64 KiB, depth 64 and 10,000 nested objects. +The schema file is trusted instruction input: annotations (`description`, `title`, +`examples`) are shown to the model in the StructuredOutput tool description. + +`--output-schema-retries ` permits N additional corrective turns (0–10, default 2). +Zero permits only the initial turn; N permits the initial turn and at most N +corrective turns. A step can contain multiple StructuredOutput +calls; every rejected call is reported and counted even after the budget is spent. +`--output-result ` writes only accepted JSON using two-space indentation and +a trailing newline, with a temporary file in the destination directory followed +by an atomic rename. Failures preserve the file. Both flags require +`--output-schema`; the destination directory must exist and be writable at +configuration time. Relative paths resolve against `--dir` when it is given, and +configuration errors name the resolved path after the reason. Without `--output-result`, formatted mode prints final JSON; NDJSON includes the value in the terminal event below. @@ -562,7 +570,8 @@ and a model turn that finishes in prose without a final tool call: ``` The usual event envelope fields also apply. `attempt` is one-based; -`maxAttempts` equals `retryCount + 1`. Diagnostics include at most ten errors, +`maxAttempts` equals `retryCount + 1`, the corrective turn budget; `attempt` can +exceed it when a step contains multiple calls. Diagnostics include at most ten errors, with bounded JSON-pointer paths, keywords and messages. Their encoded payload is under 2 KB and never includes submitted values or raw JSON arguments. A `path` can contain property names submitted by the model (for example, in map or @@ -578,7 +587,8 @@ validation always uses the canonical schema. ### `structured_output` -Exactly one terminal event per schema-enabled session, before `session_complete`: +Exactly one terminal event per schema-enabled prompt run, before `session_complete`. +A headless `aictrl run` has one prompt run: ```json { "type": "structured_output", "status": "accepted", "attempts": 2, "value": { "result": "ok" } } @@ -595,8 +605,9 @@ and prose-only finishes without a final call. Other tool calls do not count. Compaction turns do not consume a missing-result attempt. Discarded provider streams restore the budget and captured result; their rejection events remain as telemetry. A valid call in a parallel step wins regardless of call order if -that step started with budget remaining, and its attempt count stays within the -configured maximum. +that step started with budget remaining. The attempt count includes every call +in that step and can exceed `maxAttempts`; the N+1 bound applies to corrective +turns. When the budget is spent, the last rejection determines the reason: `missing` for a prose-only finish, otherwise `exhausted`. Failure reasons are: diff --git a/README.md b/README.md index 7258cef..e90d8f6 100644 --- a/README.md +++ b/README.md @@ -52,8 +52,13 @@ aictrl run --format json "review this PR" | jq '.type' ### Validated Final Results `aictrl run --output-schema ` validates the final JSON result. -Use `--output-schema-retries ` for additional corrective attempts (default 2) +Use `--output-schema-retries ` for additional corrective turns (0–10, default 2) and `--output-result ` to atomically write the accepted value. +Zero retries permits only the initial turn; N permits the initial turn and at +most N corrective turns. Serialized schemas are limited to 64 KiB, depth 64 and 10,000 +nested objects. +Schema mode emits exactly one terminal `structured_output` event per prompt run; +a headless `aictrl run` has one prompt run. Exit codes: **0** accepted, **2** configuration error, **3** missing or invalid result; provider failures and stream timeouts use **1**, signals use **130**/**143**. See the [full contract](packages/cli/README.md#validated-final-results) diff --git a/packages/cli/README.md b/packages/cli/README.md index 6788afe..812e2c4 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -34,6 +34,7 @@ aictrl run --format json "scan for secrets" | jq '.properties.part.text' while the agent can continue using tools. The schema root must be `type: "object"`. Schemas use Ajv defaults (strict validation, draft-07); a declared draft 2020-12 schema uses Ajv2020. Invalid configuration exits before any model request. +Serialized schemas are limited to 64 KiB, depth 64 and 10,000 nested objects. ```bash aictrl run --output-schema result.schema.json --output-schema-retries 2 \ @@ -41,27 +42,35 @@ aictrl run --output-schema result.schema.json --output-schema-retries 2 \ ``` - `--output-schema `: JSON Schema file, forwarded in local and `--attach` runs. -- `--output-schema-retries `: additional corrective attempts, integer >= 0, - default 2. Zero permits one attempt; N permits at most N+1 attempts. + The file is trusted instruction input: annotations (`description`, `title`, + `examples`) are shown to the model in the StructuredOutput tool description. +- `--output-schema-retries `: additional corrective turns, integer from 0 to 10, + default 2. Zero permits only the initial turn; N permits the initial turn and + at most N corrective turns. Each rejected call is counted, including multiple + calls in one step. - `--output-result `: atomically write only the validated JSON value, indented with two spaces and a trailing newline. Failures preserve an existing file and leave an absent file absent. The destination directory must exist. +Relative `--output-schema` and `--output-result` paths resolve against `--dir` when +it is given (as `--file` paths do); configuration errors print the resolved path. + Ajv strict mode rejects unknown schema keywords and unregistered `format` values such as `date-time` and `uri` as configuration errors (exit **2**). The result parent directory must exist and be writable at configuration time. The retries and result flags require `--output-schema`. Without a result file, formatted mode prints validated JSON at the end; `--format json` emits it in a -terminal `structured_output` event. Rejections emit `structured_output_rejected` +terminal `structured_output` event, exactly one per prompt run. A headless +`aictrl run` has one prompt run. Rejections emit `structured_output_rejected` with bounded JSON-pointer diagnostics. See [EVENTS.md](../../EVENTS.md) for the contract. No valid result means failure, including missing output and the agent step limit; there is no prose fallback. A prose-only finish consumes one attempt and receives a reminder to call StructuredOutput while retries remain. Earlier tools are not -replayed for corrective attempts. +replayed for corrective turns. Exit codes: **0** for accepted results, **2** for schema configuration errors, -**3** for exhausted attempts, missing output or a step limit without a result. +**3** for exhausted corrective turns, missing output or a step limit without a result. Provider failures and stream timeouts retain **1**; SIGINT/SIGTERM retain **130**/**143**. Invocations without `--output-schema` keep their existing output. diff --git a/packages/cli/src/cli/cmd/run.output.ts b/packages/cli/src/cli/cmd/run.output.ts index 5f93079..a01b50c 100644 --- a/packages/cli/src/cli/cmd/run.output.ts +++ b/packages/cli/src/cli/cmd/run.output.ts @@ -1,4 +1,5 @@ import path from "path" +// Bun content I/O preserves createPath: false; fs/promises supplies metadata and atomic rename/removal. import fs from "fs/promises" import { OutputSchema } from "@/session/output-schema" @@ -12,32 +13,37 @@ export async function outputSchema(input: { schema?: string; retries?: number; r throw new Error("--output-schema-retries and --output-result require --output-schema") return } - if (input.retries !== undefined && (!Number.isSafeInteger(input.retries) || input.retries < 0)) - throw new Error("--output-schema-retries must be an integer >= 0") - const text = await Bun.file(input.schema) + if (input.retries !== undefined && (!Number.isSafeInteger(input.retries) || input.retries < 0 || input.retries > 10)) + throw new Error("--output-schema-retries must be an integer between 0 and 10") + // Relative paths resolve against --dir (run changes into it first), like --file. + // Report the resolved path after the reason, so clipping (300 chars) drops path + // text rather than the cause. + const file = path.resolve(input.schema) + const text = await Bun.file(file) .text() .catch(() => { - throw new Error(`${input.schema}: cannot read schema file`) + throw new Error(`cannot read schema file (${file})`) }) const schema: unknown = (() => { try { return JSON.parse(text) } catch { - throw new Error(`${input.schema}: invalid JSON`) + throw new Error(`invalid JSON (${file})`) } })() if (!schema || typeof schema !== "object" || Array.isArray(schema)) - throw new Error(`${input.schema}: schema root must have type "object"`) + throw new Error(`schema root must be a JSON object (${file})`) const canonical = schema as Record const validate = (() => { try { return OutputSchema.compile(canonical) } catch (error) { - throw new Error(`${input.schema}: ${error instanceof Error ? error.message : "cannot compile schema"}`) + throw new Error(`${error instanceof Error ? error.message : "cannot compile schema"} (${file})`) } })() if (input.result) { - const directory = path.dirname(input.result) + const result = path.resolve(input.result) + const directory = path.dirname(result) const writable = await fs .stat(directory) .then(async (stat) => { @@ -46,7 +52,7 @@ export async function outputSchema(input: { schema?: string; retries?: number; r return true }) .catch(() => false) - if (!writable) throw new Error(`${input.result}: output-result parent directory must exist and be writable`) + if (!writable) throw new Error(`output-result parent directory must exist and be writable (${result})`) } return { format: { type: "json_schema" as const, schema: canonical, retryCount: input.retries ?? 2 }, validate } } diff --git a/packages/cli/src/cli/cmd/run.ts b/packages/cli/src/cli/cmd/run.ts index a150510..3296655 100644 --- a/packages/cli/src/cli/cmd/run.ts +++ b/packages/cli/src/cli/cmd/run.ts @@ -350,7 +350,7 @@ export const RunCommand = cmd({ describe: "format: default (formatted) or json (raw JSON events)", }) .option("output-schema", { type: "string", describe: "JSON Schema file for the final result (object root)" }) - .option("output-schema-retries", { type: "number", describe: "additional corrective attempts (default: 2)" }) + .option("output-schema-retries", { type: "number", describe: "additional corrective turns (0-10, default: 2)" }) .option("output-result", { type: "string", describe: "atomically write only the validated JSON result to this file", @@ -561,7 +561,8 @@ export const RunCommand = cmd({ return false } - const events = await sdk.event.subscribe() + const subscriptionAbort = new AbortController() + const events = await sdk.event.subscribe({ signal: subscriptionAbort.signal }) let error: string | undefined const startTime = Date.now() const childSessions = new Set() @@ -1014,12 +1015,21 @@ export const RunCommand = cmd({ }) } - const loopDone = loop() - .then(() => complete()) - .catch(reject) + const loopDone = loop().then( + () => ({ ok: true as const }), + (cause: unknown) => ({ ok: false as const, cause }), + ) + const finish = async (result: Awaited) => { + if (result.ok) return complete() + return reject(result.cause) + } + let failure: { cause: unknown } | undefined + const failed = (cause: unknown) => { + failure = { cause } + } if (control.current) { - await loopDone + await finish(await loopDone) await Shutdown.flush() return } @@ -1034,7 +1044,7 @@ export const RunCommand = cmd({ ...(schema ? { format: schema.format } : {}), } const request = sdk.session.command(input(sessionID, opts)) - await (schema ? request.catch(reject) : request) + await (schema ? request.catch(failed) : request) } else { const model = args.model ? Provider.parseModel(args.model) : undefined const opts = { @@ -1045,11 +1055,15 @@ export const RunCommand = cmd({ ...(schema ? { format: schema.format } : {}), } const request = sdk.session.prompt(input(sessionID, opts)) - await (schema ? request.catch(reject) : request) + await (schema ? request.catch(failed) : request) } - if (schema && error) { + if (schema && (failure || error)) { abort() + subscriptionAbort.abort() + const result = await loopDone + if (failure) await reject(failure.cause) + if (!failure) await finish(result) await Shutdown.flush() return } @@ -1059,7 +1073,19 @@ export const RunCommand = cmd({ // (e.g., Session.get() fails, model not found), no session.status idle event // is emitted, so loopDone would hang forever. Racing ensures we surface // the error and exit. - await Promise.race([loopDone, promptResult.then(() => loopDone, reject)]) + const result = await Promise.race([ + loopDone, + promptResult.then( + () => loopDone, + (cause: unknown) => ({ ok: false as const, cause }), + ), + ]) + if (schema && !result.ok) { + abort() + subscriptionAbort.abort() + await loopDone + } + await finish(result) await Shutdown.flush() } @@ -1124,7 +1150,7 @@ export const RunCommand = cmd({ }, }, event: { - async subscribe() { + async subscribe(opts: { signal: AbortSignal }) { const queue: any[] = [] let resolve: (() => void) | null = null const handler = (event: { payload: any }) => { @@ -1132,6 +1158,11 @@ export const RunCommand = cmd({ resolve?.() } GlobalBus.on("event", handler) + const stop = () => { + GlobalBus.off("event", handler) + resolve?.() + } + opts.signal.addEventListener("abort", stop) const stream = (async function* () { try { @@ -1139,12 +1170,14 @@ export const RunCommand = cmd({ while (queue.length > 0) { yield queue.shift() } + if (opts.signal.aborted) break await new Promise((r) => { resolve = r }) } } finally { GlobalBus.off("event", handler) + opts.signal.removeEventListener("abort", stop) } })() return { stream } diff --git a/packages/cli/src/session/llm.ts b/packages/cli/src/session/llm.ts index e6b5d12..64a6fe6 100644 --- a/packages/cli/src/session/llm.ts +++ b/packages/cli/src/session/llm.ts @@ -9,6 +9,7 @@ import { type StreamTextResult, type Tool, type ToolSet, + type ToolChoice, tool, jsonSchema, } from "ai" @@ -40,7 +41,7 @@ export namespace LLM { small?: boolean tools: Record retries?: number - toolChoice?: "auto" | "required" | "none" + toolChoice?: ToolChoice structured?: { reject: (input: string) => Promise; checkpoint: () => () => void; repairs: Set } } diff --git a/packages/cli/src/session/message-v2.ts b/packages/cli/src/session/message-v2.ts index 972f6f3..18a0dbe 100644 --- a/packages/cli/src/session/message-v2.ts +++ b/packages/cli/src/session/message-v2.ts @@ -83,7 +83,12 @@ export namespace MessageV2 { .object({ type: z.literal("json_schema"), schema: z.record(z.string(), z.any()).meta({ ref: "JSONSchema" }), - retryCount: z.number().int().min(0).default(2), + retryCount: z + .number() + .int() + .min(0) + .transform((n) => Math.min(n, 10)) + .default(2), }) .meta({ ref: "OutputFormatJsonSchema", diff --git a/packages/cli/src/session/output-schema.ts b/packages/cli/src/session/output-schema.ts index 72eeff7..0a6c05b 100644 --- a/packages/cli/src/session/output-schema.ts +++ b/packages/cli/src/session/output-schema.ts @@ -1,6 +1,7 @@ import Ajv, { type ErrorObject } from "ajv" import Ajv2020 from "ajv/dist/2020" import z from "zod" +import { createHash } from "crypto" export namespace OutputSchema { export const Diagnostic = z.object({ path: z.string(), keyword: z.string(), message: z.string() }) @@ -19,9 +20,34 @@ export namespace OutputSchema { // acceptance, including schemas cloned by Zod or loaded from session storage. const cache = new Map>() + // Ajv codegen recurses per nesting level: ~2,000 levels fit in 16 KB yet overflow + // the stack, so bound structure as well as bytes. Iterative, so the check itself + // cannot overflow. + const MAX_DEPTH = 64 + const MAX_NODES = 10_000 + function bound(schema: unknown) { + const stack: [unknown, number][] = [[schema, 0]] + let nodes = 0 + while (stack.length) { + const [value, depth] = stack.pop()! + if (!value || typeof value !== "object") continue + if (depth > MAX_DEPTH) throw new Error(`schema nesting must not exceed ${MAX_DEPTH} levels`) + if (++nodes > MAX_NODES) throw new Error(`schema must not exceed ${MAX_NODES} nested objects`) + for (const child of Object.values(value)) stack.push([child, depth + 1]) + } + } + + export function canonical(schema: Record) { + bound(schema) + const text = JSON.stringify(schema) + const bytes = Buffer.byteLength(text) + if (bytes > 64 * 1024) throw new Error("serialized schema must not exceed 64 KiB") + return { text, bytes } + } + export function compile(schema: Record) { if (schema.type !== "object") throw new Error('schema root must have type "object"') - const key = JSON.stringify(schema) + const key = createHash("sha256").update(canonical(schema).text).digest("hex") const cached = cache.get(key) if (cached) return cached const ajv = diff --git a/packages/cli/src/session/prompt.ts b/packages/cli/src/session/prompt.ts index 97c8c2b..9e8bb72 100644 --- a/packages/cli/src/session/prompt.ts +++ b/packages/cli/src/session/prompt.ts @@ -725,16 +725,14 @@ export namespace SessionPrompt { const rejection = async (errors: OutputSchema.Diagnostic[]) => { if (!contract) throw new Error("Missing output schema") - if (attempts < contract.format.retryCount + 1) { - attempts++ - rejected++ - await Bus.publish(Session.Event.StructuredOutputRejected, { - sessionID, - attempt: attempts, - maxAttempts: contract.format.retryCount + 1, - errors, - }) - } + attempts++ + rejected++ + await Bus.publish(Session.Event.StructuredOutputRejected, { + sessionID, + attempt: attempts, + maxAttempts: contract.format.retryCount + 1, + errors, + }) return OutputSchema.error(errors).message } @@ -752,8 +750,8 @@ export namespace SessionPrompt { model, async onSuccess(output) { if (structuredOutput !== undefined) return - if (!available) throw new Error("StructuredOutput attempt budget exhausted") - attempts = Math.min(attempts + 1, format.retryCount + 1) + if (!available) throw new Error("StructuredOutput corrective turn budget exhausted") + attempts++ structuredOutput = output }, }) @@ -823,7 +821,7 @@ export namespace SessionPrompt { ], tools, model, - toolChoice: contract ? "required" : undefined, + toolChoice: contract ? (reminder ? { type: "tool", toolName: "StructuredOutput" } : "required") : undefined, structured: contract ? { reject, @@ -1125,10 +1123,16 @@ export namespace SessionPrompt { model?: Provider.Model onSuccess: (output: unknown) => void | Promise }): AITool & { id: "StructuredOutput" } { + OutputSchema.canonical(input.schema) const validate = input.validate ?? OutputSchema.compile(input.schema) const schema = Object.fromEntries(Object.entries(input.schema).filter(([key]) => key !== "$schema")) + const canonical = OutputSchema.canonical(schema) + const description = + canonical.bytes <= 8 * 1024 + ? `Canonical JSON Schema: ${canonical.text}` + : `Canonical JSON Schema (truncated; the validator enforces the full schema): ${new TextDecoder().decode(Buffer.from(canonical.text).subarray(0, 8 * 1024))}` const result = tool({ - description: STRUCTURED_OUTPUT_DESCRIPTION, + description: `${STRUCTURED_OUTPUT_DESCRIPTION}\n\n${description}`, inputSchema: jsonSchema(input.model ? ProviderTransform.schema(input.model, schema) : schema, { validate(value) { return validate(value) diff --git a/packages/cli/test/cli/run-event-race.test.ts b/packages/cli/test/cli/run-event-race.test.ts index 9bece20..55f9c42 100644 --- a/packages/cli/test/cli/run-event-race.test.ts +++ b/packages/cli/test/cli/run-event-race.test.ts @@ -7,10 +7,12 @@ describe("run.ts GlobalBus listener race (OPS-05)", () => { test("GlobalBus.on is registered before async generator in subscribe", async () => { const source = await Bun.file(RUN_SRC).text() // Find the subscribe method in the local SDK object - const subscribeIdx = source.indexOf("async subscribe()") + const subscribeIdx = source.indexOf("async subscribe(") expect(subscribeIdx).toBeGreaterThan(-1) // Extract the subscribe method body - const subscribeBody = source.slice(subscribeIdx, subscribeIdx + 600) + const end = source.indexOf("return { stream }", subscribeIdx) + expect(end).toBeGreaterThan(-1) + const subscribeBody = source.slice(subscribeIdx, end) // GlobalBus.on must appear BEFORE the generator definition const onIdx = subscribeBody.indexOf('GlobalBus.on("event"') const generatorIdx = subscribeBody.indexOf("(async function* ()") @@ -21,9 +23,11 @@ describe("run.ts GlobalBus listener race (OPS-05)", () => { test("GlobalBus.off cleanup is still in generator finally block", async () => { const source = await Bun.file(RUN_SRC).text() - const subscribeIdx = source.indexOf("async subscribe()") + const subscribeIdx = source.indexOf("async subscribe(") expect(subscribeIdx).toBeGreaterThan(-1) - const subscribeBody = source.slice(subscribeIdx, subscribeIdx + 800) + const end = source.indexOf("return { stream }", subscribeIdx) + expect(end).toBeGreaterThan(-1) + const subscribeBody = source.slice(subscribeIdx, end) // Generator must still have finally with GlobalBus.off expect(subscribeBody).toContain("finally") expect(subscribeBody).toMatch(/GlobalBus\.off\("event"/) diff --git a/packages/cli/test/cli/run-output-schema.test.ts b/packages/cli/test/cli/run-output-schema.test.ts index 49a160b..7882c4c 100644 --- a/packages/cli/test/cli/run-output-schema.test.ts +++ b/packages/cli/test/cli/run-output-schema.test.ts @@ -18,6 +18,7 @@ async function run( timeout?: boolean abort?: boolean failure?: boolean + queued?: boolean attach?: boolean locked?: boolean denied?: boolean @@ -107,6 +108,26 @@ async function run( if (options.existing) await Bun.write(path.join(tmp.path, "result.json"), options.existing) const home = path.join(tmp.path, "home") await fs.mkdir(home) + if (options.queued) { + // Inject a synchronous provider failure after a burst of real bus events so + // the local subscription still has a rejection queued when it is aborted. + await Bun.write( + path.join(tmp.path, "failure.ts"), + `import { SessionPrompt } from ${JSON.stringify(path.resolve(import.meta.dir, "../../src/session/prompt.ts"))} +import { GlobalBus } from ${JSON.stringify(path.resolve(import.meta.dir, "../../src/bus/global.ts"))} +import { MessageV2 } from ${JSON.stringify(path.resolve(import.meta.dir, "../../src/session/message-v2.ts"))} +SessionPrompt.prompt = async (input) => { + for (let n = 0; n < 32; n++) GlobalBus.emit("event", { payload: { type: "fixture.queued", properties: {} } }) + GlobalBus.emit("event", { payload: { + type: "session.structured_output_rejected", + properties: { sessionID: input.sessionID, attempt: 1, maxAttempts: 3, + errors: [{ path: "", keyword: "required", message: "must have required property 'result'" }] } + } }) + throw new MessageV2.APIError({ message: "Fixture provider failed", statusCode: 400, isRetryable: false }) +} +`, + ) + } const attached: { url: string; body: Record }[] = [] // The headless package has no HTTP server; bridge its real session APIs rather // than synthesizing responses or terminal events for the attach regression. @@ -171,6 +192,7 @@ async function run( "bun", "run", "--conditions=browser", + ...(options.queued ? ["--preload", path.join(tmp.path, "failure.ts")] : []), entry, "run", ...(bridge ? ["--attach", bridge.url.origin, "--dir", tmp.path] : []), @@ -232,9 +254,31 @@ async function run( for (const [name, options, reason] of [ ["unreadable", { flags: ["--output-schema", "absent.json"] }, "cannot read"], + [ + // run changes into --dir first, so relative paths resolve there (as --file does) + // and the error names the resolved path rather than the bare argument. + "relative schema under --dir", + { flags: ["--dir", "home", "--output-schema", "schema.json"] }, + `${path.sep}home${path.sep}schema.json)`, + ], ["invalid JSON", { content: "{broken" }, "invalid JSON"], ["uncompilable", { content: '{"type":"object","unknownKeyword":true}' }, "strict mode"], - ["non-object root", { content: '{"type":"array"}' }, "object"], + ["non-object root", { content: '{"type":"array"}' }, 'schema root must have type "object"'], + ["array JSON root", { content: "[]" }, "schema root must be a JSON object"], + ["null JSON root", { content: "null" }, "schema root must be a JSON object"], + ["scalar JSON root", { content: '"schema"' }, "schema root must be a JSON object"], + ["oversized schema", { content: JSON.stringify({ type: "object", description: "x".repeat(64 * 1024) }) }, "64 KiB"], + [ + // ~2,000 levels is only ~16 KB but overflows Ajv's recursive codegen. + "deeply nested schema", + { + content: JSON.stringify({ + type: "object", + properties: { a: Array.from({ length: 2000 }).reduce((inner) => ({ not: inner }), { type: "string" }) }, + }), + }, + "nesting must not exceed 64 levels", + ], [ "unknown format", { content: '{"type":"object","properties":{"result":{"type":"string","format":"date-time"}}}' }, @@ -257,11 +301,20 @@ for (const [name, options, reason] of [ ], ["retries without schema", { flags: ["--output-schema-retries", "1"] }, "require --output-schema"], ["result without schema", { flags: ["--output-result", "result.json"] }, "require --output-schema"], - ["negative retries", { flags: ["--output-schema", "schema.json", "--output-schema-retries", "-1"] }, "integer >= 0"], + [ + "excessive retries", + { flags: ["--output-schema", "schema.json", "--output-schema-retries", "11"] }, + "integer between 0 and 10", + ], + [ + "negative retries", + { flags: ["--output-schema", "schema.json", "--output-schema-retries", "-1"] }, + "integer between 0 and 10", + ], [ "fractional retries", { flags: ["--output-schema", "schema.json", "--output-schema-retries", "1.5"] }, - "integer >= 0", + "integer between 0 and 10", ], [ "unsupported draft", @@ -279,8 +332,8 @@ for (const [name, options, reason] of [ expect(result.stderr).toContain(reason) const event = result.events.find((event) => event.type === "invocation_error") expect((event!.message as string).length).toBeLessThanOrEqual(300) - if (name === "negative retries" || name === "fractional retries") - expect(event?.message).toBe("--output-schema-retries must be an integer >= 0") + if (name === "negative retries" || name === "fractional retries" || name === "excessive retries") + expect(event?.message).toBe("--output-schema-retries must be an integer between 0 and 10") expect(event).toMatchObject({ code: "OUTPUT_SCHEMA_CONFIG", message: expect.stringContaining(reason) }) if ("content" in options) expect(result.stderr).toContain("schema.json") expect(result.requests).toEqual([]) @@ -324,7 +377,10 @@ for (const json of [true, false]) { }) expect(result.exit, result.stderr).toBe(0) expect(result.requests).toHaveLength(2) - expect(result.requests.map((request) => request.tool_choice)).toEqual(["required", "required"]) + expect(result.requests.map((request) => request.tool_choice)).toEqual([ + "required", + { type: "function", function: { name: "StructuredOutput" } }, + ]) expect(JSON.stringify(result.requests[1].messages)).toContain("call StructuredOutput with the final result") expect(result.result).toBe(JSON.stringify({ result: "accepted" }, null, 2) + "\n") if (!json) return @@ -464,7 +520,10 @@ test("schema attach repairs a prose finish through the HTTP session boundary", a retryCount: 2, }) expect(result.requests).toHaveLength(2) - expect(result.requests.map((request) => request.tool_choice)).toEqual(["required", "required"]) + expect(result.requests.map((request) => request.tool_choice)).toEqual([ + "required", + { type: "function", function: { name: "StructuredOutput" } }, + ]) expect(result.result).toBe(JSON.stringify({ result: "accepted" }, null, 2) + "\n") expect(result.events.filter((event) => event.type === "structured_output_rejected")).toMatchObject([ { attempt: 1, errors: [{ path: "", keyword: "missing", message: "call StructuredOutput with the final result" }] }, @@ -490,12 +549,17 @@ for (const [name, value, failure, code] of [ ["malformed outcome", { result: "attached" }, false, 3], ["structured failure followed by loop rejection", {}, false, 3], ["provider rejection", { result: "submitted-secret" }, false, 1], + ["provider rejection after loop failure", { result: "submitted-secret" }, false, 1], ] as const) { test(`attach ${name} forwards the canonical schema and validates final acceptance`, async () => { await using tmp = await tmpdir() await Bun.write(path.join(tmp.path, "schema.json"), JSON.stringify(schema)) + const rejection = name.startsWith("provider rejection") const requests: { url: string; body: Record }[] = [] - let controller: ReadableStreamDefaultController | undefined + let ready: (value: ReadableStreamDefaultController) => void = () => {} + const connected = new Promise>((resolve) => { + ready = resolve + }) const server = Bun.serve({ port: 0, idleTimeout: 0, @@ -505,8 +569,8 @@ for (const [name, value, failure, code] of [ return new Response( new ReadableStream({ start(value) { - controller = value - controller.enqueue(new TextEncoder().encode(": ready\n\n")) + value.enqueue(new TextEncoder().encode(": ready\n\n")) + ready(value) }, }), { headers: { "Content-Type": "text/event-stream" } }, @@ -517,10 +581,13 @@ for (const [name, value, failure, code] of [ requests.push({ url, body }) if (url === "/session") return Response.json({ id: "ses_fixture" }) if (url === "/session/ses_fixture/message") { + const controller = await connected // Provider rejection answers before any SSE event, the ordering that // once let a stale exit code 3 survive a genuine provider failure. - for (const event of name === "provider rejection" - ? [] + for (const event of rejection + ? name === "provider rejection after loop failure" + ? [{ type: "session.status", properties: { sessionID: "ses_fixture", status: null } }] + : [] : [ { type: "session.structured_output_rejected", @@ -572,12 +639,14 @@ for (const [name, value, failure, code] of [ ] : [{ type: "session.status", properties: { sessionID: "ses_fixture", status: { type: "idle" } } }]), ]) - controller!.enqueue(new TextEncoder().encode(`data: ${JSON.stringify(event)}\n\n`)) - if (name === "provider rejection") + controller.enqueue(new TextEncoder().encode(`data: ${JSON.stringify(event)}\n\n`)) + if (rejection) { + if (name === "provider rejection after loop failure") await Bun.sleep(25) return Response.json( { name: "APIError", data: { message: "Fixture provider failed", statusCode: 400, isRetryable: false } }, { status: 400 }, ) + } return Response.json({}) } return new Response("unexpected request", { status: 400 }) @@ -618,8 +687,9 @@ for (const [name, value, failure, code] of [ proc.exited, ]) expect(exit, stderr + stdout).toBe(code) - if (name === "provider rejection") { - expect(stderr).toContain('"code":"400"') + if (rejection) { + expect(stderr.split("\n").filter((line) => line.startsWith("{") && line.includes('"reason":'))).toHaveLength(1) + expect(stderr.split("\n").filter((line) => line.includes('"code":"400"'))).toHaveLength(1) expect(stderr).toContain('"reason":"unknown"') expect(stdout + stderr).not.toContain("submitted-secret") } @@ -640,7 +710,7 @@ for (const [name, value, failure, code] of [ : { status: "failed", reason: "error", - attempts: name === "malformed outcome" ? 1 : name === "provider rejection" ? 0 : 2, + attempts: name === "malformed outcome" ? 1 : rejection ? 0 : 2, }, ]) expect(events.filter((event) => event.type === "structured_output")).toHaveLength(1) @@ -685,6 +755,25 @@ test("schema config NDJSON clips the specific compilation cause", async () => { expect(result.requests).toEqual([]) const event = result.events.find((event) => event.type === "invocation_error") expect(event?.code).toBe("OUTPUT_SCHEMA_CONFIG") - expect(event?.message).toStartWith("schema.json: strict mode: unknown keyword:") + // The reason leads so the 300-char clip removes path text, never the cause. + expect(event?.message).toStartWith("strict mode: unknown keyword:") expect((event?.message as string).length).toBe(300) }, 25000) + +test("local abort drains rejection queued immediately before provider failure", async () => { + const result = await run({ queued: true }) + expect(result.exit, result.stderr + result.stdout).toBe(1) + expect(result.events.filter((event) => event.type === "structured_output_rejected")).toMatchObject([ + { attempt: 1, maxAttempts: 3, errors: [{ keyword: "required" }] }, + ]) + expect(result.events.filter((event) => event.type === "structured_output")).toMatchObject([ + { status: "failed", reason: "error", attempts: 1 }, + ]) + expect(result.events.filter((event) => event.type === "structured_output")).toHaveLength(1) + expect(result.events.filter((event) => event.type === "session_complete")).toHaveLength(1) + expect(result.stderr.split("\n").filter((line) => line.startsWith("{") && line.includes('"reason":'))).toHaveLength(1) + const types = result.events.map((event) => event.type) + expect(types.indexOf("structured_output_rejected")).toBeLessThan(types.indexOf("structured_output")) + expect(types.indexOf("structured_output")).toBeLessThan(types.indexOf("session_complete")) + expect(types.at(-1)).toBe("invocation_complete") +}, 25000) diff --git a/packages/cli/test/provider/models.test.ts b/packages/cli/test/provider/models.test.ts index 7256c35..7532421 100644 --- a/packages/cli/test/provider/models.test.ts +++ b/packages/cli/test/provider/models.test.ts @@ -29,7 +29,13 @@ test("model catalog refresh handles a timeout while reading the response body", stderr: "pipe", }, ) - using cleanup = { [Symbol.dispose]: () => proc.kill() } + const timer = setTimeout(() => proc.kill("SIGKILL"), 20000) + using cleanup = { + [Symbol.dispose]() { + clearTimeout(timer) + proc.kill("SIGKILL") + }, + } const [stdout, stderr, exit] = await Promise.all([ new Response(proc.stdout).text(), new Response(proc.stderr).text(), diff --git a/packages/cli/test/session/output-schema.test.ts b/packages/cli/test/session/output-schema.test.ts index 1076f40..b010a59 100644 --- a/packages/cli/test/session/output-schema.test.ts +++ b/packages/cli/test/session/output-schema.test.ts @@ -491,7 +491,8 @@ test("Gemini tool schema is transformed but canonical numeric enum remains autho test("prose-only finish is a counted corrective attempt before a valid result", async () => { const result = await run(["prose", '{"result":"repaired"}'], 1) expect(result.requests).toHaveLength(2) - result.requests.forEach((request) => expect(request.tool_choice).toBe("required")) + expect(result.requests[0].tool_choice).toBe("required") + expect(result.requests[1].tool_choice).toEqual({ type: "function", function: { name: "StructuredOutput" } }) expect(result.info.structured).toEqual({ result: "repaired" }) expect(result.info.error).toBeUndefined() expect(result.rejected).toEqual([ @@ -583,7 +584,8 @@ for (const parallel of [ expect(result.requests).toHaveLength(1) expect(result.info.error).toBeUndefined() expect(result.info.structured).toEqual({ result: "accepted" }) - expect(result.outcomes).toEqual([{ status: "accepted", attempts: 1, value: { result: "accepted" } }]) + expect(result.outcomes).toEqual([{ status: "accepted", attempts: 2, value: { result: "accepted" } }]) + expect(result.rejected).toMatchObject([{ attempt: parallel[0] === "{}" ? 1 : 2, maxAttempts: 1 }]) }, 15000) } @@ -645,3 +647,84 @@ test("structured error summaries preserve bounded identity without submitted dat name: "x".repeat(96), }) }) + +test("serialized schema limit counts bytes and accepts exactly 64 KiB", () => { + const schema = { type: "object", description: "" } + const size = Buffer.byteLength(JSON.stringify(schema)) + schema.description = "x".repeat(64 * 1024 - size) + expect(OutputSchema.compile(schema)({})).toBe(true) + expect(() => OutputSchema.compile({ ...schema, description: schema.description + "x" })).toThrow("64 KiB") + expect(() => OutputSchema.compile({ ...schema, description: "é".repeat(33 * 1024) })).toThrow("64 KiB") + expect(OutputSchema.compile(schema)).toBe(OutputSchema.compile(JSON.parse(JSON.stringify(schema)))) +}) + +test("prompt path rejects an oversized schema before any provider request", async () => { + await expect(run(["unused"], 0, { schema: { type: "object", description: "x".repeat(64 * 1024) } })).rejects.toThrow( + "64 KiB", + ) +}) + +test("retryCount accepts the upper bound and clamps legacy values above ten", () => { + expect(MessageV2.OutputFormatJsonSchema.parse({ type: "json_schema", schema, retryCount: 10 }).retryCount).toBe(10) + expect(MessageV2.OutputFormatJsonSchema.parse({ type: "json_schema", schema, retryCount: 50 }).retryCount).toBe(10) +}) + +test("every parallel rejection is counted after the corrective budget is spent", async () => { + const result = await run(["unused"], 0, { parallel: ["{}", "{}", "{}"] }) + expect(result.requests).toHaveLength(1) + expect(result.rejected).toMatchObject([ + { attempt: 1, maxAttempts: 1 }, + { attempt: 2, maxAttempts: 1 }, + { attempt: 3, maxAttempts: 1 }, + ]) + expect(result.outcomes).toEqual([{ status: "failed", reason: "exhausted", attempts: 3 }]) +}) + +test("provider tool description includes canonical pattern constraints", async () => { + const canonical = { + type: "object", + properties: { result: { type: "string", pattern: "^OK-[0-9]{4}$" } }, + required: ["result"], + additionalProperties: false, + } + const result = await run(['{"result":"OK-1234"}'], 0, { schema: canonical, gemini: true }) + const tools = result.requests[0].tools as { function: { name: string; description: string } }[] + expect(tools.find((tool) => tool.function.name === "StructuredOutput")?.function.description).toContain( + JSON.stringify(canonical), + ) +}) + +for (const [name, description] of [ + ["ASCII", "x".repeat(9 * 1024)], + ["multibyte", "x" + "€".repeat(4 * 1024)], +]) { + test(`tool description caps ${name} schema at 8 KiB and explains truncation`, () => { + const canonical = { type: "object", description } + const tool = SessionPrompt.createStructuredOutputTool({ schema: canonical, onSuccess() {} }) + const text = JSON.stringify(canonical) + const expected = new TextDecoder().decode(Buffer.from(text).subarray(0, 8 * 1024)) + expect(tool.description).toContain("truncated; the validator enforces the full schema") + expect(tool.description?.split(": ").at(-1)).toBe(expected) + if (name === "multibyte") expect(expected).toEndWith("�") + }) +} + +for (const [name, schema, reason] of [ + ["oversized", { type: "object", description: "é".repeat(33 * 1024) }, "64 KiB"], + [ + "over-deep", + { type: "object", properties: { a: Array.from({ length: 65 }).reduce((inner) => ({ not: inner }), {}) } }, + "nesting must not exceed 64 levels", + ], + ["too many objects", { type: "object", examples: Array.from({ length: 10_000 }, () => ({})) }, "10000"], +] as const) { + test(`custom validator cannot bypass ${name} schema bounds`, () => { + expect(() => + SessionPrompt.createStructuredOutputTool({ + schema, + validate: OutputSchema.compile({ type: "object" }), + onSuccess() {}, + }), + ).toThrow(reason) + }) +} diff --git a/packages/cli/test/session/structured-output.test.ts b/packages/cli/test/session/structured-output.test.ts index 7d1ca1f..de8f5cb 100644 --- a/packages/cli/test/session/structured-output.test.ts +++ b/packages/cli/test/session/structured-output.test.ts @@ -213,6 +213,8 @@ describe("structured-output.createStructuredOutputTool", () => { // AI SDK wraps schema in { jsonSchema: {...} } const inputSchema = tool.inputSchema as any expect(inputSchema.jsonSchema?.$schema).toBeUndefined() + expect(tool.description).toContain(`Canonical JSON Schema: ${JSON.stringify(inputSchema.jsonSchema)}`) + expect(tool.description).not.toContain("$schema") }) test("execute calls onSuccess with valid args", async () => {