From bc5a85df3ec6fcb13553056778e162ee056620c2 Mon Sep 17 00:00:00 2001 From: ozymandiashh <234437643+ozymandiashh@users.noreply.github.com> Date: Wed, 5 Aug 2026 04:17:26 +0300 Subject: [PATCH] fix(kiro): estimate chat-file input tokens from every human turn, and invalidate cached history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit decodeKiroChatFile estimated input tokens from pendingUserMessage - the last human turn sliced to 500 chars - while output summed every bot char, so any multi-turn session or final prompt over 500 chars under-reported input tokens (and therefore costUSD) severalfold. The estimate is now the sum of every human turn's full character count, with the 500-char cap kept for the display userMessage only. That closes most of the gap but does NOT reach parity with the modern-execution, CLI-session and V2 arms: those count tool and system records as input - their code comments state tool results are fed back to the model - while the chat arm still counts only human records. Tool content demonstrably exists in the format (the G2 fixture carries a tool record), so the chat arm still under-reports; it just under-reports far less than before. The blast radius is the chat-file arm alone: the IDE-file dispatcher routes any record carrying a chat array plus metadata to decodeKiroChatFile, so this is chat-shaped Kiro IDE files, not every Kiro prompt. The identity-preamble exclusion now trims leading whitespace before its startsWith match. Pre-fix, a near miss (a leading newline, a BOM, a wrapper) was nearly harmless, because the preamble only mattered if it happened to be the last human turn; post-fix, every unmatched system-injected human record adds its FULL length to input, and preambles are large - a missed match is a silent multi-thousand-token inflation on every affected session. Leading whitespace tolerance is cheap (a genuine prompt never starts with whitespace plus an identity tag) and the failure asymmetry favours exclusion: a false negative inflates tokens, a false positive only skips a preamble. A renamed preamble remains a residual risk, noted in the near-miss regression test. Cached history is affected, which is what a user actually sees. session-cache serves unchanged files without invoking the provider parser, so bump kiro's PROVIDER_PARSE_VERSIONS fingerprint (ide-parsing-v1 -> v2) to force one re-parse of every already-cached kiro session; without it the pre-fix token and cost numbers would be served forever. The daily rollup ALSO needs invalidating for this fix to be fully visible: days finalized before the fix keep their pre-fix kiro cost in the daily cache, and ensureCacheHydrated re-derives them only on a version bump, a savings-config change, a timezone change, or an incomplete cache — the session-cache re-parse alone leaves finalized day totals untouched. So this commit bumps BOTH layers: the session-cache PROVIDER_PARSE_VERSIONS fingerprint above forces the one re-parse of every already-cached kiro session, and DAILY_CACHE_VERSION (15 -> 17, MIN_SUPPORTED_VERSION raised with it; 16 is skipped because main already claimed it for the codex structural-discovery fix, and claiming 16 here would load a main-built v16 cache as current and complete, so the invalidation would never fire) forces the daily rollup's one-time re-derivation, so finalized day totals are rebuilt under the corrected estimate. The re-derive reaches every day whose kiro chat files still exist; sourceless days carry forward with their pre-fix totals under the v14 NEVER-LOSE rule (a carry-forward, not a refresh — nothing can reconstruct them once the files are gone). Update the G2 parity golden: A1 was pinned at 125 tokens for a 3000-char prompt (the 500-char slice / 4); the corrected value is 750 (3000 / 4), with a comment marking 125 as a pre-fix value so it is not restored. 3000, 2400 and 1000+1000 are all exact multiples of four, so add G2b pinning the estimator's rounding with an odd length (3001 chars -> 751 tokens; round and floor would both give 750). Add a money-path regression test (2400-char prompt -> 600 tokens, userMessage still 500-capped for display), a multi-turn accumulation test (an identical resubmitted prompt counts again - a real second model input; identity messages stay excluded), a near-miss identity test (leading-newline and BOM preambles stay excluded), extend the kiro cache-invalidation test to pin the v1 -> v2 fingerprint bump, and add a daily-cache regression test seeding a complete pre-fix v15 cache (unchanged savings hash and timezone, so nothing but the version bump can invalidate it) and proving the bump forces the re-derive that lands the corrected kiro cost — while the v15 file is never rewritten. --- CHANGELOG.md | 1 + docs/providers/kiro.md | 2 +- packages/cli/src/daily-cache.ts | 22 +++- packages/cli/src/session-cache.ts | 6 +- packages/cli/tests/daily-cache.test.ts | 76 +++++++++++ .../cli/tests/kiro-cache-invalidation.test.ts | 122 +++++++++++++++++- .../cli/tests/providers/kiro-golden.test.ts | 34 ++++- packages/cli/tests/providers/kiro.test.ts | 98 ++++++++++++++ packages/core/src/providers/kiro/decode.ts | 25 +++- 9 files changed, 375 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98c38076..8a431c6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Fixed - Claude Desktop and Cowork sessions are discovered for Windows Microsoft Store (MSIX) installs. (#611) +- **Kiro chat sessions no longer under-report input.** The token estimate for chat-file sessions (legacy `.chat` and chat-shaped IDE files) now counts every human turn's full text instead of only the last 500 characters, so multi-turn sessions and long prompts stop under-reporting input (and cost) severalfold. The estimate still counts human turns only — tool and system content, and resent context, are not included — and the summary still shows the last prompt, capped at 500 chars. Already-cached sessions re-parse once so the corrected numbers replace the old ones, and the daily cache bumps to v17 with this fix (v16 was skipped — main already claimed it for the codex structural-discovery fix), so finalized day totals re-derive too — for days whose kiro chat files still exist: since the v14 NEVER-LOSE carry-forward, a bump re-derives what sources survive and carries every sourceless (day, provider) slice forward with its old values, so a day whose kiro chat files have since been deleted keeps the pre-fix cost after the bump (minor in practice, since kiro IDE sources usually persist, but a carry-forward, not a refresh). (#909) ## 0.9.19 - 2026-07-20 diff --git a/docs/providers/kiro.md b/docs/providers/kiro.md index 0252e901..72c98368 100644 --- a/docs/providers/kiro.md +++ b/docs/providers/kiro.md @@ -57,7 +57,7 @@ The stores are disjoint (v2 sessions use `sess_`-prefixed IDs in a separate dire - **Model ID normalization.** Kiro stores models like `claude-1.2`; the parser rewrites the dot to a hyphen so they match `claude-1-2` in the pricing snapshot. Add new versions here when Kiro ships them. - **Tool name extraction accepts text and structured calls.** Kiro can embed tool calls inside message text as `...` or expose structured `toolCalls` / `tool_calls` / `tools` entries. - Token counts are estimated via char count (`CHARS_PER_TOKEN = 4`). -- **Credits are the cost source; tokens stay estimated.** Kiro bills in credits ($20/mo for 1,000; overage $0.04/credit). CLI (`metering_usage`), v1 executions (`usageSummary[].usage`), and v2 (`usage_summary.promptTurnSummaries[].usage`) turns record real credits, converted to USD at `USD_PER_KIRO_CREDIT = 0.04` (the public overage rate — the same never-understate approach as Codebuff). Turns without credit data fall back to token-estimated cost (`costIsEstimated: true`); legacy `.chat` and workspace-session records carry no usage data, so they are always token-estimated. Note: an earlier CLI implementation summed credit values directly as dollars, overstating cost 25×. Token *counts* remain char-estimated everywhere (input undercounts: only visible transcript text is seen, not the full resent context; v2's `session_metadata.contextUsage.usagePercentage` × context window is a better input proxy if ever needed). v2 does keep the real `modelId`, so unlike the v1 execution-file path it is not mislabeled `kiro-auto`. +- **Credits are the cost source; tokens stay estimated.** Kiro bills in credits ($20/mo for 1,000; overage $0.04/credit). CLI (`metering_usage`), v1 executions (`usageSummary[].usage`), and v2 (`usage_summary.promptTurnSummaries[].usage`) turns record real credits, converted to USD at `USD_PER_KIRO_CREDIT = 0.04` (the public overage rate — the same never-understate approach as Codebuff). Turns without credit data fall back to token-estimated cost (`costIsEstimated: true`); legacy `.chat` and workspace-session records carry no usage data, so they are always token-estimated. Note: an earlier CLI implementation summed credit values directly as dollars, overstating cost 25×. Token *counts* remain char-estimated everywhere. The chat-file arm now sums every human turn's full text (identity preamble excluded) instead of only the last 500-char slice, so its input no longer undercounts for long or multi-turn chats; it still skips `tool`/`system` records, which the modern-execution, CLI, and v2 arms count as input (tool results are fed back to the model), and no arm sees the full resent context. v2's `session_metadata.contextUsage.usagePercentage` × context window is a better input proxy if ever needed. v2 does keep the real `modelId`, so unlike the v1 execution-file path it is not mislabeled `kiro-auto`. - **Cost is frozen at parse time.** Kiro is on the `costUSD` pass-through allowlist in `providerCallToCachedCall` (alongside mistral-vibe, devin, hermes, …), so its credit-based cost survives the session cache instead of being re-priced from estimated tokens — token re-pricing understated/overstated real kiro spend by up to 16× per model. The tradeoff, shared with all allowlisted providers: `codeburn price-override` and `model-alias` do not affect kiro dollar amounts (token *counts* are unaffected). Historical caches from before this change re-parse via the `CACHE_VERSION` bump to 5. ## When fixing a bug here diff --git a/packages/cli/src/daily-cache.ts b/packages/cli/src/daily-cache.ts index c5439c34..379f5a11 100644 --- a/packages/cli/src/daily-cache.ts +++ b/packages/cli/src/daily-cache.ts @@ -5,7 +5,23 @@ import { homedir } from 'os' import { join } from 'path' import type { DateRange, ProjectSummary } from './types.js' -// Bumped to 15: per-project daily rollups. Days and provider slices now carry +// Bumped to 17 — 16 is skipped on purpose: main already spent it on the codex +// structural-discovery fix (eece4cf), so a v16 cache in the wild is a +// main-owned cache meaning the codex fix, not this one. Claiming 16 here too +// would let a user who has ever run a main build load its v16 cache as +// CURRENT and COMPLETE, and this invalidation would never fire. 17 is the +// first version that also means the kiro chat-file fix. +// +// kiro chat-file input tokens are now estimated from every human turn's full +// text instead of the last 500 chars (#909, this PR), so a v15 rollup +// finalized by the pre-fix binary carries kiro costs off by up to +// severalfold. Nothing downstream can notice on its own: `usage-aggregator` +// serves every day before today from this cache, and retention is ten years, +// so an upgrading user with a warm complete cache would keep the stale +// pre-fix kiro day totals forever while freshly reparsed sessions disagreed +// with them. Raising MIN_SUPPORTED_VERSION forces the one-time re-derivation. +// +// v15: per-project daily rollups. Days and provider slices now carry // a `projects` breakdown (cost/calls/savings/sessions per project) so project // history outlives the session files, like models and categories already do. // This bump is the first to ride the v14 carry-forward: the old cache is @@ -57,8 +73,8 @@ import type { DateRange, ProjectSummary } from './types.js' // that older binaries skipped. v8 added local-model savings to the daily // rollup; the `savingsConfigHash` field is invalidated separately when the // user changes their `localModelSavings` mapping. -export const DAILY_CACHE_VERSION = 15 -const MIN_SUPPORTED_VERSION = 15 +export const DAILY_CACHE_VERSION = 17 +const MIN_SUPPORTED_VERSION = 17 // Version-suffixed so different binaries each own a distinct file and never // clobber an incompatible schema. Bumping the version mints a fresh filename; // adoptOlderDailyCaches then unions days out of every previous file (including diff --git a/packages/cli/src/session-cache.ts b/packages/cli/src/session-cache.ts index 461dd461..0b35f7b1 100644 --- a/packages/cli/src/session-cache.ts +++ b/packages/cli/src/session-cache.ts @@ -221,7 +221,11 @@ export const PROVIDER_PARSE_VERSIONS: Record = { hermes: 'reasoning-output-accounting-v1-est-cost', 'lingtai-tui': 'token-ledger-registry-activity-v3', 'ibm-bob': 'worktree-project-grouping-v1', - kiro: 'ide-parsing-v1-est-cost', + // v2: chat-file input tokens now estimate from the FULL prompt (sum of every + // human turn), not the last 500-char slice — so costUSD changed for chat-arm + // sessions. Bump so already-cached kiro sessions re-parse once and the + // corrected tokens/cost land instead of being served pre-fix forever. + kiro: 'ide-parsing-v2-est-cost', quickdesk: 'emf-sqlite-v2-est-cost', kimicode: 'wire-usage-v1-est-cost', 'kilo-code': 'worktree-project-grouping-v1', diff --git a/packages/cli/tests/daily-cache.test.ts b/packages/cli/tests/daily-cache.test.ts index 6ae07079..c50b330a 100644 --- a/packages/cli/tests/daily-cache.test.ts +++ b/packages/cli/tests/daily-cache.test.ts @@ -466,3 +466,79 @@ describe('ensureCacheHydrated: timezone invalidation', () => { expect(preserved.days[0]!.date).toBe(twoDaysAgoStr) }) }) + +// A complete v15 cache is trusted as-is (unchanged savings hash, matching tz, +// complete marker set) — so without the v17 bump, an upgrading user keeps the +// pre-fix kiro day totals forever while freshly reparsed sessions disagree +// with them. The bump mints a fresh filename, adoption marks the result +// incomplete, and the next hydration re-derives the window; the corrected kiro +// estimate (every human turn's full text, #909) wins wherever the sources +// survive, and sourceless days carry forward under the v14 NEVER-LOSE rule. +describe('ensureCacheHydrated: schema version invalidation (#909)', () => { + function kiroDay(date: string, cost: number, calls: number, inputTokens: number): DailyEntry { + return { + date, + cost, + savingsUSD: 0, + calls, + sessions: 1, + inputTokens, + outputTokens: 200, + cacheReadTokens: 0, + cacheWriteTokens: 0, + editTurns: 0, + oneShotTurns: 0, + models: { + 'claude-haiku-4-5': { calls, cost, savingsUSD: 0, inputTokens, outputTokens: 200, cacheReadTokens: 0, cacheWriteTokens: 0 }, + }, + categories: {}, + providers: { + kiro: { cost, calls, savingsUSD: 0, sessions: 1, inputTokens, outputTokens: 200 }, + }, + } + } + + it('re-derives a warm complete v15 cache instead of serving its pre-fix kiro totals', async () => { + vi.useFakeTimers() + vi.setSystemTime(new Date('2026-06-12T12:00:00.000Z')) + + const { writeFile, mkdir } = await import('fs/promises') + await mkdir(TMP_CACHE_ROOT, { recursive: true }) + // A cache exactly as a pre-fix release left it: current schema at the + // time, finalized off a complete parse, watermark at yesterday, matching + // tz and savings hash. Nothing but the version bump can invalidate it. + const v15 = { + version: 15, + savingsConfigHash: '', + tzKey: currentTzKey(), + lastComputedDate: '2026-06-11', + days: [kiroDay('2026-06-11', 4.55, 1, 500)], + complete: true, + } + await writeFile(join(TMP_CACHE_ROOT, 'daily-cache.v15.json'), JSON.stringify(v15), 'utf-8') + + let parseCalls = 0 + const hydrated = await ensureCacheHydrated( + async () => { + parseCalls += 1 + return [] + }, + // The corrected kiro accounting: 750 input tokens (3000 chars / 4) + // instead of the 500-char-slice 125, i.e. the full-prompt estimate. + () => [kiroDay('2026-06-11', 18.2, 1, 750)], + ) + + // The whole point: the window is re-parsed rather than served frozen. + expect(parseCalls).toBe(1) + // ...and the fresh derivation wins over the stale v15 day, at the + // provider-slice level where the pre-fix kiro cost actually lived. + const day = hydrated.days.find(d => d.date === '2026-06-11') + expect(day?.cost).toBe(18.2) + expect(day?.providers['kiro']?.cost).toBe(18.2) + expect(day?.providers['kiro']?.inputTokens).toBe(750) + expect(hydrated.version).toBe(DAILY_CACHE_VERSION) + expect(hydrated.complete).toBe(true) + // The v15 file is never rewritten or deleted — old binaries still own it. + expect(JSON.parse(await readFile(join(TMP_CACHE_ROOT, 'daily-cache.v15.json'), 'utf-8')).version).toBe(15) + }) +}) diff --git a/packages/cli/tests/kiro-cache-invalidation.test.ts b/packages/cli/tests/kiro-cache-invalidation.test.ts index 0046dd7a..0ae2231c 100644 --- a/packages/cli/tests/kiro-cache-invalidation.test.ts +++ b/packages/cli/tests/kiro-cache-invalidation.test.ts @@ -57,11 +57,18 @@ function kiroAgentDir(): string { // What computeEnvFingerprint('kiro') returned before kiro had an entry in // PROVIDER_PARSE_VERSIONS: no env vars, no parser version, i.e. a hash of -// zero parts. This is the fingerprint sitting in every pre-fix cache. +// zero parts. This is the fingerprint sitting in every pre-registration cache. function preFixFingerprint(): string { return createHash('sha256').update([].join('\0')).digest('hex').slice(0, 16) } +// What computeEnvFingerprint('kiro') returned under the FIRST parser version +// ('ide-parsing-v1-est-cost'): the fingerprint sitting in every cache written +// by the release that shipped the 500-char-slice input-token estimate. +function v1Fingerprint(): string { + return createHash('sha256').update('parser=ide-parsing-v1-est-cost').digest('hex').slice(0, 16) +} + // Writes one IDE execution file in the context.messages[].entries format that // the pre-fix parser turned into 0 turns, and returns its path. async function seedExecutionFile(): Promise { @@ -84,6 +91,33 @@ async function seedExecutionFile(): Promise { return path } +// Writes one chat-shaped IDE file (chat array + metadata) with a 3000-char +// human prompt, and returns its path. Routes to decodeKiroChatFile. +async function seedChatFile(): Promise { + const dir = join(kiroAgentDir(), 'c'.repeat(32)) + await mkdir(dir, { recursive: true }) + const path = join(dir, 'chat-stale-001.chat') + await writeFile(path, JSON.stringify({ + executionId: 'exec-chat-stale-001', + actionId: 'act', + chat: [ + { role: 'human', content: '\nYou are Kiro.\n' }, + { role: 'bot', content: 'I will follow these instructions.' }, + { role: 'human', content: 'x'.repeat(3000) }, + { role: 'bot', content: 'short' }, + ], + metadata: { + modelId: 'claude-haiku-4-5', + modelProvider: 'qdev', + workflow: 'act', + workflowId: 'wf-chat-stale-001', + startTime: 1780000000000, + endTime: 1780000001000, + }, + })) + return path +} + async function seedCache(execPath: string, envFingerprint: string): Promise { const fp = await fingerprintFile(execPath) if (!fp) throw new Error('failed to fingerprint seeded execution file') @@ -102,6 +136,60 @@ async function seedCache(execPath: string, envFingerprint: string): Promise { + const path = await seedChatFile() + const fp = await fingerprintFile(path) + if (!fp) throw new Error('failed to fingerprint seeded chat file') + const cache: SessionCache = { + version: CACHE_VERSION, + providers: { + kiro: { + envFingerprint, + files: { + [path]: { + fingerprint: fp, + mcpInventory: [], + turns: [{ + timestamp: '2026-05-27T00:00:00.000Z', + sessionId: 'wf-chat-stale-001', + userMessage: 'x'.repeat(500), + calls: [{ + provider: 'kiro', + model: 'claude-haiku-4-5', + usage: { + inputTokens: 125, + outputTokens: 2, + cacheCreationInputTokens: 0, + cacheReadInputTokens: 0, + cachedInputTokens: 0, + reasoningTokens: 0, + webSearchRequests: 0, + cacheCreationOneHourTokens: 0, + }, + speed: 'standard', + timestamp: '2026-05-27T00:00:00.000Z', + tools: [], + bashCommands: [], + skills: [], + subagentTypes: [], + deduplicationKey: 'kiro:wf-chat-stale-001:exec-chat-stale-001', + }], + }], + }, + }, + }, + }, + } + await mkdir(CACHE_DIR, { recursive: true }) + await writeFile(sessionCachePath(), JSON.stringify(cache)) + return path +} + async function parseKiroCalls() { const projects = await parseAllSessions(undefined, 'kiro') return projects @@ -127,6 +215,26 @@ describe('Kiro session cache invalidation', () => { expect(computeEnvFingerprint('kiro')).not.toBe(preFixFingerprint()) }) + it('bumps the kiro parser version again for the full-prompt input-token fix', () => { + // The v1 fingerprint is what the release that shipped the 500-char-slice + // estimate wrote into every user cache. It must NOT match the current one, + // or cached chat files would keep the pre-fix 125-token estimate forever. + expect(computeEnvFingerprint('kiro')).not.toBe(v1Fingerprint()) + }) + + it('control: a chat cache at the CURRENT fingerprint is honored (stale 125 stays)', async () => { + await seedChatCache(computeEnvFingerprint('kiro')) + + const calls = await parseKiroCalls() + + // The seeded cache is structurally valid and trusted: the unchanged chat + // file is not re-parsed, so the pre-fix 125-token estimate survives + // verbatim. This proves the seed is real (not silently ignored) — and that + // WITHOUT a fingerprint bump, the stale estimate would be served forever. + expect(calls).toHaveLength(1) + expect(calls[0]!.usage.inputTokens).toBe(125) + }) + it('control: a zero-turn cache entry at the current fingerprint is honored', async () => { const execPath = await seedExecutionFile() await seedCache(execPath, computeEnvFingerprint('kiro')) @@ -140,6 +248,18 @@ describe('Kiro session cache invalidation', () => { expect(calls).toHaveLength(0) }) + it('regression: a v1 cache fingerprint forces a re-parse that lands the corrected tokens', async () => { + await seedChatCache(v1Fingerprint()) + + const calls = await parseKiroCalls() + + // The v1 fingerprint no longer matches, the stale section is discarded, + // the unchanged chat file re-parses, and the full-prompt estimate (3000 + // chars / 4 = 750) replaces the cached 125 — the money-path correction. + expect(calls).toHaveLength(1) + expect(calls[0]!.usage.inputTokens).toBe(750) + }) + it('regression: a pre-fix cache fingerprint forces a re-parse that recovers the calls', async () => { const execPath = await seedExecutionFile() await seedCache(execPath, preFixFingerprint()) diff --git a/packages/cli/tests/providers/kiro-golden.test.ts b/packages/cli/tests/providers/kiro-golden.test.ts index 413ca3e9..f627b640 100644 --- a/packages/cli/tests/providers/kiro-golden.test.ts +++ b/packages/cli/tests/providers/kiro-golden.test.ts @@ -340,11 +340,11 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => { expect(blankCalls[0]!.deduplicationKey).toBe('kiro:wf-g1b2:') }) - it('G2 — A1 input tokens derived from truncated prompt; A2 from full prompt', async () => { + it('G2 — A1 input tokens from the full prompt (500 cap is display-only); A2 modern arm from full prompt', async () => { const wsHashA1 = 'a'.repeat(32) const wsDirA1 = join(tmpDir, 'g2a1', wsHashA1) await mkdir(wsDirA1, { recursive: true }) - const chatPath = join(wsDirA1, 'trunc.chat') + const chatPath = join(wsDirA1, 'long.chat') await writeFile(chatPath, makeChatFile({ executionId: 'exec-g2a1', workflowId: 'wf-g2a1', @@ -353,7 +353,13 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => { })) const a1 = await parseSource({ path: chatPath, project: 'p', provider: 'kiro' }) expect(a1).toHaveLength(1) - expect(a1[0]!.inputTokens).toBe(125) + // 3000 fixture chars / 4 per token = 750. The prior pin (125) encoded the + // pre-fix behaviour: input tokens were estimated from the last human turn + // sliced to 500 chars (500 / 4 = 125), under-reporting cost for any longer + // prompt. That bug was fixed by porting upstream 6c4645a ('fix(kiro): + // estimate input tokens from the full prompt, not a 500-char slice'); + // do not restore 125. + expect(a1[0]!.inputTokens).toBe(750) expect(a1[0]!.userMessage.length).toBe(500) const wsHashA2 = 'b'.repeat(32) @@ -371,6 +377,28 @@ describe('kiro golden pins (raw calls, unmodified provider)', () => { expect(a2[0]!.inputTokens).toBe(500) }) + it('G2b — A1 rounding is pinned: an odd length forces ceil, not round/floor', async () => { + // 3000 (G2), 2400 (money-path regression) and 1000+1000 (multi-turn + // accumulation) are all exact multiples of four, so they cannot + // distinguish ceil from round or floor in estimateTokensFromChars. + // 3001 chars / 4 = 750.25: ceil gives 751, round and floor both give + // 750 — only the ceil pin passes here. + const wsHash = 'd'.repeat(32) + const wsDir = join(tmpDir, 'g2b', wsHash) + await mkdir(wsDir, { recursive: true }) + const chatPath = join(wsDir, 'odd.chat') + await writeFile(chatPath, makeChatFile({ + executionId: 'exec-g2b', + workflowId: 'wf-g2b', + userPrompt: 'x'.repeat(3001), + botResponses: ['short'], + })) + const calls = await parseSource({ path: chatPath, project: 'p', provider: 'kiro' }) + expect(calls).toHaveLength(1) + expect(calls[0]!.inputTokens).toBe(751) + expect(calls[0]!.userMessage.length).toBe(500) + }) + it('G3 — A1 single-tool toolSequence is present but undefined', async () => { const wsHash = 'c'.repeat(32) const wsDir = join(tmpDir, 'g3', wsHash) diff --git a/packages/cli/tests/providers/kiro.test.ts b/packages/cli/tests/providers/kiro.test.ts index 808e3596..92c32f27 100644 --- a/packages/cli/tests/providers/kiro.test.ts +++ b/packages/cli/tests/providers/kiro.test.ts @@ -220,6 +220,104 @@ describe('kiro provider - chat file parsing', () => { expect(calls[0]!.outputTokens).toBe(109) }) + it('estimates input tokens from the full prompt, not a 500-char slice (money-path)', async () => { + // Regression: decodeKiroChatFile estimated input tokens from + // pendingUserMessage (the last human turn sliced to 500 chars) while + // output summed every bot char, so a long prompt undercounted input + // tokens - and cost - severalfold. + const wsHash = 'n'.repeat(32) + const wsDir = join(tmpDir, wsHash) + await mkdir(wsDir, { recursive: true }) + const chatPath = join(wsDir, 'long.chat') + const prompt = 'x'.repeat(2400) // 2400 chars / 4 = 600 tokens; a 500-slice would give 125 + await writeFile(chatPath, makeChatFile({ userPrompt: prompt, botResponses: ['ok'] })) + + const source = { path: chatPath, project: 'test', provider: 'kiro' } + const calls: ParsedProviderCall[] = [] + for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call)) + + expect(calls).toHaveLength(1) + expect(calls[0]!.inputTokens).toBe(600) + // userMessage stays capped for display; only the token estimate uses the full length. + expect(calls[0]!.userMessage.length).toBe(500) + }) + + it('accumulates EVERY human turn - including an identical resubmit - and skips identity messages', async () => { + // A resubmitted prompt is a real second model input (the model consumed + // its tokens twice), so it must be counted - the same semantics every + // other kiro arm uses (modern-execution, ws-session, cli, v2 all sum + // every human/user message). The one non-input human message, the + // `` system preamble, stays excluded. + const wsHash = 'o'.repeat(32) + const wsDir = join(tmpDir, wsHash) + await mkdir(wsDir, { recursive: true }) + const chatPath = join(wsDir, 'multi.chat') + const prompt = 'y'.repeat(1000) // 1000 chars / 4 = 250 tokens + const chat = [ + { role: 'human', content: '\nYou are Kiro.\n' }, // excluded + { role: 'bot', content: 'I will follow these instructions.' }, + { role: 'human', content: prompt }, // 250 + { role: 'bot', content: 'first answer' }, + { role: 'human', content: prompt }, // resubmit - identical text, counted again: +250 + { role: 'bot', content: 'second answer' }, + ] + await writeFile(chatPath, JSON.stringify({ + executionId: 'exec-multi-001', + actionId: 'act', + chat, + metadata: { modelId: 'claude-haiku-4-5', modelProvider: 'qdev', workflow: 'act', workflowId: 'wf-multi-001', startTime: 1777333000000, endTime: 1777333010000 }, + })) + + const source = { path: chatPath, project: 'test', provider: 'kiro' } + const calls: ParsedProviderCall[] = [] + for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call)) + + expect(calls).toHaveLength(1) + expect(calls[0]!.inputTokens).toBe(500) // 250 + 250, identity not counted + // Display shows the LAST human turn, capped at 500. + expect(calls[0]!.userMessage).toBe(prompt.slice(0, 500)) + }) + + it('excludes a near-miss identity preamble (leading newline / BOM) instead of counting it as input', async () => { + // The preamble is a large system-injected human record. With + // the full-length accumulator, a near-miss match - a leading newline or + // BOM before the tag - would silently add the preamble's whole length to + // input tokens on every affected session. decodeKiroChatFile trims + // leading whitespace before the startsWith match; pin that here so a + // regression cannot re-inflate tokens. (A renamed preamble remains a + // residual risk: no prefix match can catch it, and it is out of scope.) + const wsHash = 'r'.repeat(32) + const wsDir = join(tmpDir, wsHash) + await mkdir(wsDir, { recursive: true }) + const chatPath = join(wsDir, 'near-miss.chat') + const prompt = 'z'.repeat(1000) // 1000 chars / 4 = 250 tokens + const chat = [ + { role: 'human', content: '\n\nYou are Kiro.\n' }, // leading newline - still excluded + { role: 'bot', content: 'I will follow these instructions.' }, + { role: 'human', content: '\uFEFF\nYou are Kiro.\n' }, // BOM - still excluded + { role: 'bot', content: 'I will follow these instructions too.' }, + { role: 'human', content: prompt }, + { role: 'bot', content: 'answer' }, + ] + await writeFile(chatPath, JSON.stringify({ + executionId: 'exec-near-001', + actionId: 'act', + chat, + metadata: { modelId: 'claude-haiku-4-5', modelProvider: 'qdev', workflow: 'act', workflowId: 'wf-near-001', startTime: 1777333000000, endTime: 1777333010000 }, + })) + + const source = { path: chatPath, project: 'test', provider: 'kiro' } + const calls: ParsedProviderCall[] = [] + for await (const call of kiro.createSessionParser(source, new Set()).parse()) calls.push(priceProviderCall(call)) + + expect(calls).toHaveLength(1) + // Only the real prompt counts: 250. If either near-miss preamble were + // treated as input, tokens would jump by both preamble lengths. + expect(calls[0]!.inputTokens).toBe(250) + // Display shows the last human turn, capped at 500. + expect(calls[0]!.userMessage).toBe(prompt.slice(0, 500)) + }) + it('normalizes dot-versioned model IDs to dashes', async () => { const wsHash = 'h'.repeat(32) const wsDir = join(tmpDir, wsHash) diff --git a/packages/core/src/providers/kiro/decode.ts b/packages/core/src/providers/kiro/decode.ts index 24a663ea..cf48be7e 100644 --- a/packages/core/src/providers/kiro/decode.ts +++ b/packages/core/src/providers/kiro/decode.ts @@ -186,12 +186,33 @@ export function decodeKiroChatFile(input: { if (modelId === 'auto' || !modelId) modelId = 'kiro-auto' let pendingUserMessage = '' + // Accumulate every human turn's full length for the input-token estimate, + // mirroring the modern-execution path's inputChars accumulator. The prior + // code estimated input tokens from pendingUserMessage.length alone - the + // LAST human turn truncated to 500 chars - so a multi-turn session, or any + // final prompt over 500 chars, undercounted input tokens (and therefore + // costUSD) severalfold, while output correctly summed all bot chars. Note + // the chat arm still counts only human records: tool and system records + // stay excluded, unlike the modern-execution, CLI and v2 arms (whose + // comments state tool results are fed back to the model as input), so this + // arm still under-reports - just far less than before. + let inputChars = 0 const allTools: string[] = [] const toolSequence: KiroToolCall[][] = [] for (const msg of chat) { if (msg.role === 'human') { - if (msg.content.startsWith('')) continue + // The system preamble is the one non-input human record. + // Trim leading whitespace before matching: preambles are large, and + // with the full-length accumulator below a missed match (a leading + // newline or BOM before the tag) would silently add the preamble's + // whole length to input tokens on every affected session. Tolerating + // leading whitespace costs nothing real - a genuine prompt never + // starts with whitespace plus an identity tag - and the failure + // asymmetry favours exclusion: a false negative inflates tokens by + // the preamble length, a false positive only skips a preamble. + if (msg.content.trimStart().startsWith('')) continue + inputChars += msg.content.length pendingUserMessage = msg.content.slice(0, 500) } if (msg.role === 'bot') { @@ -210,7 +231,7 @@ export function decodeKiroChatFile(input: { if (seen.has(dedupKey)) return { calls, diagnostics: [] } const outputTokens = estimateTokensFromChars(totalOutputChars) - const inputTokens = estimateTokensFromChars(pendingUserMessage.length) + const inputTokens = estimateTokensFromChars(inputChars) const tsDate = parseKiroTimestamp(metadata.startTime) if (!tsDate) return { calls, diagnostics: [] } const timestamp = tsDate.toISOString()