diff --git a/agent/README.md b/agent/README.md index ecfc6bdde..088a4d38f 100644 --- a/agent/README.md +++ b/agent/README.md @@ -119,11 +119,11 @@ The `run.sh` script overrides the container's default CMD to run `python /app/sr | `AWS_SECRET_ACCESS_KEY` | Conditional† | | Explicit keys, if you are not using CLI-based resolution | | `AWS_SESSION_TOKEN` | No | | For temporary credentials | | `AWS_PROFILE` | No | | Profile for `aws configure export-credentials` in `run.sh`, or default profile when using the `~/.aws` mount fallback | -| `ANTHROPIC_MODEL` | No | `us.anthropic.claude-sonnet-4-6` | Bedrock **inference profile** or model ID for `InvokeModel` (see [inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-use.html)) | +| `ANTHROPIC_MODEL` | No | `us.anthropic.claude-opus-4-8` | Bedrock **inference profile** ID for `InvokeModel` (see [inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-use.html)). Must be the `us.`-prefixed profile ID, not a bare foundation-model ID — see [Model configuration](../docs/guides/DEVELOPER_GUIDE.md#model-configuration) | | `MAX_TURNS` | No | `100` | Max agent turns before stopping | | `MAX_BUDGET_USD` | No | | **Local batch only** (shell env when running `entrypoint.py` directly). Range 0.01–100; agent stops when the budget is reached. For deployed AgentCore **server** mode and production tasks, set **`max_budget_usd`** on task creation (REST API, CLI `--max-budget`, or Blueprint default); the orchestrator sends it in the `/invocations` JSON body — server mode does not read `MAX_BUDGET_USD` from the environment. | | `DRY_RUN` | No | | Set to `1` to validate config and print the prompt without running the agent | -| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | No | `anthropic.claude-haiku-4-5-20251001-v1:0` | Bedrock model ID for the pre-flight safety check (see below) | +| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | No | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | Bedrock **inference profile** ID for the small/fast auxiliary model — the pre-flight safety check and WebFetch summarization (see below). Set by the CDK stack (`cdk/src/stacks/agent.ts` (the runtime environment block)); the `us.` prefix is required | | `NUDGES_TABLE_NAME` | No | | **Phase 2.** DynamoDB table for mid-task user nudges (`` XML blocks injected between turns). If unset, the agent runs without nudge support — `nudge_reader.read_pending()` returns `[]` and logs a WARN once. Set automatically by the CDK stack on both AgentCore runtimes. | | `JIRA_APP_ACTOR_PROXY_URL` | No | | Resolved per-task from the Jira tenant secret. Forge v2 web-trigger URL used for app-authored Jira comments and transitions. | | `JIRA_APP_ACTOR_SHARED_SECRET` | No | | Resolved per-task from the Jira tenant secret. HMAC key for the Forge proxy; redacted from agent diagnostics. | @@ -133,7 +133,7 @@ The `run.sh` script overrides the container's default CMD to run `python /app/sr including non-Jira tasks, so a warm AgentCore process cannot expose one tenant's OAuth or Forge credential to the next task. -**Bedrock model access (main model):** Configuring `ANTHROPIC_MODEL` and IAM credentials is not enough. Your AWS account must be able to **invoke** that model in Amazon Bedrock: follow [Request access to models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) (Marketplace permissions on first use, Anthropic first-time use where required, valid payment method for Marketplace-backed models). Use an inference profile ID such as `us.anthropic.claude-sonnet-4-6` when Bedrock requires it. If the CLI stops with a message that the model is not available on your Bedrock deployment, fix model access in the console or switch `ANTHROPIC_MODEL` to an entitled profile, then retry. +**Bedrock model access (main model):** Configuring `ANTHROPIC_MODEL` and IAM credentials is not enough. Your AWS account must be able to **invoke** that model in Amazon Bedrock: follow [Request access to models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) (Marketplace permissions on first use, Anthropic first-time use where required, valid payment method for Marketplace-backed models). Always use an inference profile ID such as `us.anthropic.claude-opus-4-8`: a bare foundation-model ID cannot be invoked with on-demand throughput and Bedrock rejects it with `ValidationException`. IAM must also grant the model — see [Model configuration](../docs/guides/DEVELOPER_GUIDE.md#model-configuration) for the full layering. If the CLI stops with a message that the model is not available on your Bedrock deployment, fix model access in the console or switch `ANTHROPIC_MODEL` to an entitled profile, then retry. **Pre-flight check model**: Claude Code runs a quick safety verification using a small Haiku model before executing each tool command. On Bedrock, the default Haiku model ID may not be enabled in your account, causing the check to time out with *"Pre-flight check is taking longer than expected"* warnings. The agent sets `ANTHROPIC_DEFAULT_HAIKU_MODEL` to a known-available Bedrock Haiku model ID to avoid this. If you see pre-flight timeout warnings, verify that this model is enabled in your Bedrock model access settings. @@ -145,7 +145,8 @@ tenant's OAuth or Forge credential to the next task. # Dry run — validate config, fetch issue, print assembled prompt, then exit DRY_RUN=1 ./agent/run.sh "owner/repo" 42 -# Run with a specific model +# Run with a specific model (overrides the us.anthropic.claude-opus-4-8 default). +# Must be a `us.`-prefixed inference profile that IAM grants — see Model configuration. ANTHROPIC_MODEL="us.anthropic.claude-sonnet-4-6" ./agent/run.sh "owner/repo" 42 # Limit agent to 50 turns diff --git a/cdk/test/contracts/model-default-docs-parity.test.ts b/cdk/test/contracts/model-default-docs-parity.test.ts new file mode 100644 index 000000000..77fa43dcf --- /dev/null +++ b/cdk/test/contracts/model-default-docs-parity.test.ts @@ -0,0 +1,192 @@ +/** + * MIT No Attribution + * + * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * the Software without restriction, including without limitation the rights to + * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of + * the Software, and to permit persons to whom the Software is furnished to do so. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +import * as fs from 'fs'; +import * as path from 'path'; + +/** + * DOCS CONTRACT: the model defaults the docs advertise must equal the defaults + * the agent actually uses. + * + * The agent's default model is a Python literal in `agent/src/config.py` with no + * CDK prop or environment knob in front of it, so a model bump is a one-line + * source edit — and every doc that quotes the old value silently becomes a lie. + * That is exactly what happened: four docs advertised a Sonnet-4.6 default long + * after the code moved to Opus 4.8, and `agent/README.md` advertised a BARE + * haiku id that cannot be invoked on-demand at all, contradicting the `us.` + * -prefixed inference-profile id the stack actually deploys. Nothing guarded + * either one, so both rotted unnoticed across several releases (#742). + * + * `cdk/test/constructs/bedrock-models.test.ts` already proves this + * cross-language regex-grep pattern for the code→IAM half of the invariant (the + * agent fallback must be in `DEFAULT_BEDROCK_MODEL_IDS` or every task fails at + * turn 0 with AccessDenied). This file closes the code→docs half: the next model + * bump fails CI here instead of quietly rotting the documentation. + * + * Deliberately asserts against the *rendered doc text* (the value inside + * backticks in the env-var tables) rather than a shared constant, because the + * failure mode being guarded is precisely a human reading a stale doc. + */ + +const REPO_ROOT = path.resolve(__dirname, '..', '..', '..'); + +function read(relPath: string): string { + return fs.readFileSync(path.join(REPO_ROOT, relPath), 'utf8'); +} + +/** + * Extracts an env-var fallback literal from `agent/src/config.py`, i.e. the + * second argument of `os.environ.get("", "")`. Tolerates the + * line wrapping Ruff applies to the call. + */ +function agentDefaultFor(envVar: string): string { + const configPy = read('agent/src/config.py'); + const match = configPy.match(new RegExp(`"${envVar}",\\s*"([^"]+)"`)); + expect(match).not.toBeNull(); + return match![1]; +} + +/** + * Extracts the `Default` cell of a markdown env-var table row keyed by + * `` `` `` — the value the doc advertises to a reader. Returns every + * match so a doc with more than one such table fails loudly rather than having + * the second table silently unguarded. + */ +function documentedDefaults(markdown: string, envVar: string): string[] { + const rows = markdown + .split('\n') + .filter((line) => line.trimStart().startsWith('|') && line.includes(`\`${envVar}\``)); + const found: string[] = []; + for (const row of rows) { + // Cells, minus the leading/trailing empties produced by the outer pipes. + const cells = row.split('|').slice(1, -1).map((c) => c.trim()); + // The default is the first backticked cell AFTER the one naming the env var. + const nameIdx = cells.findIndex((c) => c === `\`${envVar}\``); + if (nameIdx === -1) continue; + for (const cell of cells.slice(nameIdx + 1)) { + const literal = cell.match(/^`([^`]+)`$/); + if (literal) { + found.push(literal[1]); + break; + } + } + } + return found; +} + +describe('documented model defaults match the agent runtime defaults', () => { + // The docs that advertise a default in an env-var table. Both are read by + // humans configuring a deployment, so both must track config.py. + const DOCS_WITH_ENV_TABLES = [ + 'docs/guides/DEVELOPER_GUIDE.md', + 'agent/README.md', + ] as const; + + it.each(DOCS_WITH_ENV_TABLES)('%s documents the real ANTHROPIC_MODEL default', (docPath) => { + const expected = agentDefaultFor('ANTHROPIC_MODEL'); + const documented = documentedDefaults(read(docPath), 'ANTHROPIC_MODEL'); + // A doc that stops documenting the default at all is also a regression: the + // guard would silently pass on an empty list. + expect(documented.length).toBeGreaterThan(0); + for (const value of documented) { + expect(value).toBe(expected); + } + }); + + it('agent/README.md documents the real ANTHROPIC_DEFAULT_HAIKU_MODEL default', () => { + const expected = agentDefaultFor('ANTHROPIC_DEFAULT_HAIKU_MODEL'); + const documented = documentedDefaults(read('agent/README.md'), 'ANTHROPIC_DEFAULT_HAIKU_MODEL'); + expect(documented.length).toBeGreaterThan(0); + for (const value of documented) { + expect(value).toBe(expected); + } + }); + + it('both agent defaults are inference-profile ids, not bare foundation-model ids', () => { + // A bare `anthropic.…` id cannot be invoked with on-demand throughput + // (Bedrock returns ValidationException), so documenting one sends readers + // down a dead end. Guards the specific bug fixed in agent/README.md. + for (const envVar of ['ANTHROPIC_MODEL', 'ANTHROPIC_DEFAULT_HAIKU_MODEL']) { + expect(agentDefaultFor(envVar)).toMatch(/^(us|eu|apac|global)\./); + } + }); + + /** + * The env-var-table assertions above only reach the two docs that HAVE such a + * table. Four other places quote a model literal — the `model_id` rows in + * USER_GUIDE / REPO_ONBOARDING and their two generated Starlight mirrors — and + * nothing read them, so on the next model bump CI would force the two guarded + * docs to update while those four quietly went stale again. That is precisely + * the rot this file exists to stop, so sweep every model literal in the doc set + * instead of enumerating table shapes: any `us.`/`eu.`/`apac.`/`global.`-prefixed + * Claude id in a guarded doc must be one the agent actually defaults to. + * + * Deliberately literal-based rather than row-based: it survives someone + * re-wording a table, moving the value into prose, or adding a doc — none of + * which a row parser would follow. + */ + it('no guarded doc presents a stale model id AS the default', () => { + const allowed = new Set([ + agentDefaultFor('ANTHROPIC_MODEL'), + agentDefaultFor('ANTHROPIC_DEFAULT_HAIKU_MODEL'), + ]); + // Only lines that CLAIM to state the default are in scope. A doc legitimately + // names other models as illustrative examples — a per-repo override snippet, a + // cost-comparison row, "switch to a lighter model such as X" — and failing those + // would make the guard unmaintainable, so it would end up deleted rather than + // fixed. Match on the claim, not on the mere presence of an id. + // Two shapes claim a default: prose/cells saying so, and a `model_id` table row + // whose Default CELL carries the literal with no such word on the line at all + // (REPO_ONBOARDING's blueprint-defaults table and USER_GUIDE's per-repo table + // are both this shape — mutation-tested, and a keyword-only rule misses them). + const CLAIMS_DEFAULT = /\bdefaults?\b|\bfallback\b|^\s*\|\s*`model_id`\s*\|/i; + // Prefixed inference-profile ids only. Bare `anthropic.claude-…` ids appear + // legitimately when the docs explain WHY a bare id is not invocable, and the + // IAM grant list in bedrock-models.ts is bare-by-contract — both out of scope + // here and already covered by bedrock-models.test.ts. + const MODEL_ID = /\b(?:us|eu|apac|global)\.anthropic\.claude-[a-z0-9-]+(?::[0-9]+)?/g; + // Hand-authored sources plus every generated mirror that actually quotes a + // prefixed id (enumerated from the tree, not guessed — `using/Overview.md` + // mirrors USER_GUIDE but carries no literal, so listing it would assert + // nothing while implying coverage). + const GUARDED_DOCS = [ + 'docs/guides/DEVELOPER_GUIDE.md', + 'docs/guides/USER_GUIDE.md', + 'docs/design/REPO_ONBOARDING.md', + 'agent/README.md', + 'docs/src/content/docs/architecture/Repo-onboarding.md', + 'docs/src/content/docs/customizing/Per-repo-overrides.md', + 'docs/src/content/docs/developer-guide/Model-configuration.md', + 'docs/src/content/docs/developer-guide/Repository-preparation.md', + 'docs/src/content/docs/developer-guide/Installation.md', + 'docs/src/content/docs/getting-started/Quick-start.mdx', + ] as const; + + const offenders: string[] = []; + for (const docPath of GUARDED_DOCS) { + for (const [i, line] of read(docPath).split('\n').entries()) { + if (!CLAIMS_DEFAULT.test(line)) continue; + for (const id of line.match(MODEL_ID) ?? []) { + if (!allowed.has(id)) offenders.push(`${docPath}:${i + 1} → ${id}`); + } + } + } + expect(offenders).toEqual([]); + }); +}); diff --git a/cdk/test/stacks/agent.test.ts b/cdk/test/stacks/agent.test.ts index e7c917486..f7acc8a37 100644 --- a/cdk/test/stacks/agent.test.ts +++ b/cdk/test/stacks/agent.test.ts @@ -294,15 +294,23 @@ describe('AgentStack', () => { // and the per-task session role (the coding agent's task-model grants). The // override replaces the model set for the WORKLOAD; these are its surfaces. // - // Deliberately EXCLUDES the Linear webhook processor's policy: the - // deterministic-revise interpreter (linear-integration.ts) makes one tiny - // "which plan-edit did they mean?" classification call pinned to a FIXED - // model (DEFAULT_REVISE_MODEL_ID = sonnet), by design independent of the - // per-task ``bedrockModels`` override — you don't want a cheap classification - // running on whatever heavyweight coding model an operator selected. That - // grant is scoped to its single fixed model (asserted in the linear - // integration tests), so it's not a wildcard/drift risk; it just isn't part - // of the override contract this test checks. + // The prefix filter, not a blanket scan, is what this test asserts against. + // On main those two roles are in fact the ONLY policies in the stack holding + // a bedrock:InvokeModel statement — the Linear webhook processor deliberately + // has none (`linear-integration.ts`: "No bedrock:InvokeModel grant: this + // processor never calls a model directly"; its only Bedrock action is + // ApplyGuardrail). So the filter is currently a no-op belt-and-braces guard + // that keeps this assertion honest if a future construct adds an + // InvokeModel grant that the ``bedrockModels`` override is not meant to + // govern — e.g. a cheap fixed-model classification call, which you would not + // want running on whatever heavyweight coding model an operator selected. + // + // (An earlier revision of this comment cited a fixed-model revise grant via a + // `DEFAULT_REVISE_MODEL_ID` constant. That constant and its + // orchestration-plan-revise-interpret module exist only on the unmerged + // #299 branch and never landed on main, so the reference was dangling — see + // #742. Corrected rather than deleted to record that the exclusion describes + // a hypothetical, not a live grant.) const OVERRIDE_GOVERNED_POLICY_PREFIXES = ['RuntimeExecutionRole', 'AgentSessionRole']; const policies = overridden.findResources('AWS::IAM::Policy'); const bedrockResources: unknown[] = []; diff --git a/docs/design/REPO_ONBOARDING.md b/docs/design/REPO_ONBOARDING.md index f7c4d4a75..912f90c51 100644 --- a/docs/design/REPO_ONBOARDING.md +++ b/docs/design/REPO_ONBOARDING.md @@ -120,7 +120,7 @@ From lowest to highest priority: |---|---|---| | `compute_type` | `agentcore` | Platform constant | | `runtime_arn` | Stack-level env var | CDK stack props | -| `model_id` | Claude Sonnet 4 | CDK stack props | +| `model_id` | `us.anthropic.claude-opus-4-8` | Python literal in `agent/src/config.py` (no CDK prop or env knob today) — see [Model configuration](../guides/DEVELOPER_GUIDE.md#model-configuration) | | `max_turns` | 100 | Platform constant | | `max_budget_usd` | None (unlimited) | - | | `memory_token_budget` | 2000 | Platform constant | diff --git a/docs/guides/DEVELOPER_GUIDE.md b/docs/guides/DEVELOPER_GUIDE.md index b821b8973..e5814f239 100644 --- a/docs/guides/DEVELOPER_GUIDE.md +++ b/docs/guides/DEVELOPER_GUIDE.md @@ -114,6 +114,95 @@ See the [Cedar policy guide](./CEDAR_POLICY_GUIDE.md) for the full authoring ref - **Stack name** - The default is `backgroundagent-dev` (set in `cdk/src/main.ts`). If you rename it, update all `--stack-name` references. - **Making repos agent-friendly** - Add `CLAUDE.md`, `.claude/rules/`, and clear build commands. See the [Prompt guide](./PROMPT_GUIDE.md#repo-level-instructions) for details. +## Model configuration + +**This is the canonical reference for which model the agent uses and where to change it.** The model ID is configured across five independent layers in three languages, so read this section before changing a default — a mismatch between the layers fails every task on the stack at turn 0, not just an edge case. + +### The five layers + +| # | Layer | What it controls | Where | ID form | +|---|---|---|---|---| +| 1 | **IAM invoke allowlist** | Which models the agent's roles may invoke at all. The outer gate — everything below fails without it. | `DEFAULT_BEDROCK_MODEL_IDS` (`cdk/src/constructs/bedrock-models.ts:34`); override with CDK context `bedrockModels` (key at `:48`, resolver at `:67`) | **Bare** (`anthropic.claude-…`) | +| 2 | **Platform default model** | The model used when nothing narrower is set. A **Python literal only** — there is no CDK prop or environment knob in front of it today. | `agent/src/config.py:563` (the `ANTHROPIC_MODEL` fallback) and `agent/src/models.py:157` (`TaskConfig.anthropic_model`) | Prefixed (`us.anthropic.…`) | +| 3 | **Auxiliary / fast model** | The small model Claude Code uses for auxiliary work (WebFetch page summarization, the pre-flight safety check). | Stack env `ANTHROPIC_DEFAULT_HAIKU_MODEL` (`cdk/src/stacks/agent.ts` (the runtime environment block)); agent-side fallback at `agent/src/config.py:569` | Prefixed (`us.anthropic.…`) | +| 4 | **Per-repo override** | One repository's model, with no agent redeploy. | Blueprint `agent.modelId` (`cdk/src/constructs/blueprint.ts`, `BlueprintProps.agent.modelId`) → RepoTable `model_id` (`cdk/src/handlers/shared/repo-config.ts:37`) → ECS injects `ANTHROPIC_MODEL` (`cdk/src/handlers/shared/strategies/ecs-strategy.ts:217`) | Prefixed (`us.anthropic.…`) | +| 5 | **Per-task / local** | One task's model. Payload `model_id` is aliased to `anthropic_model` (`agent/src/pipeline.py`, `_PAYLOAD_KEY_ALIASES`); local batch runs read `ANTHROPIC_MODEL` from the shell via `agent/run.sh`. | Task payload `model_id`; shell `ANTHROPIC_MODEL` | Prefixed (`us.anthropic.…`) | + +### Environment variables + +| Variable | Who sets it | ID form | Purpose | +|---|---|---|---| +| `ANTHROPIC_MODEL` | ECS strategy from the repo Blueprint (layer 4); you, in the shell, for local batch runs (layer 5) | Prefixed inference profile | The main coding model. Unset → the `agent/src/config.py` fallback. | +| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | The CDK stack, hardcoded at `cdk/src/stacks/agent.ts` (the runtime environment block) | Prefixed inference profile | The small/fast auxiliary model. Must be a granted profile, or the pre-flight check times out with *"Pre-flight check is taking longer than expected"*. | +| `CLAUDE_CODE_USE_BEDROCK` | The CDK stack (`='1'`) and `agent/run.sh` | — | Routes Claude Code to Bedrock instead of the Anthropic API. ABCA always runs on Bedrock. | + +### Precedence — narrowest wins + +```text +per-task payload model_id (layer 5) + > blueprint agent.modelId (layer 4, arrives as stack env ANTHROPIC_MODEL) + > stack env ANTHROPIC_MODEL (layer 3-adjacent / local shell) + > agent/src/config.py fallback (layer 2 — us.anthropic.claude-opus-4-8) +``` + +Every one of those is gated by the **IAM invoke allowlist** (layer 1), which is itself gated by **account-level Bedrock model access**. Both gates are silent until invocation: a model that resolves fine through precedence still fails at turn 0 with `AccessDenied` if it is not in the grant list, and fails again if your account has not completed [Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for it. + +### Bare vs. prefixed IDs — the one rule that bites + +Layer 1 takes **bare foundation-model IDs**; every other layer takes the **prefixed inference-profile ID**. This asymmetry is deliberate: both grant sites derive the inference-profile ARN by *adding* the `us.` prefix themselves, so a prefixed entry in `bedrockModels` would produce an invalid `us.us.anthropic.…` ARN. The resolver rejects a `us.`/`eu.`/`apac.`-prefixed entry at `cdk/src/constructs/bedrock-models.ts:84` so the typo fails at synth rather than at runtime. + +In the other direction, a **bare** ID cannot be invoked on demand at all. Verified: + +```console +$ aws bedrock-runtime invoke-model --model-id anthropic.claude-opus-5 ... +ValidationException: Invocation of model ID anthropic.claude-opus-5 with on-demand +throughput isn't supported. Retry your request with the ID or ARN of an inference +profile that contains this model. +``` + +So: `bedrockModels` context → `anthropic.claude-opus-4-8`. Everywhere else → `us.anthropic.claude-opus-4-8`. + +### Bumping the default model + +1. Add the **bare** ID to `DEFAULT_BEDROCK_MODEL_IDS` (`cdk/src/constructs/bedrock-models.ts`) and deploy, so the grant exists before anything tries to use it. +2. Confirm account-level Bedrock access for the model in the target Region. +3. Update the **prefixed** ID in `agent/src/config.py` and `agent/src/models.py`. +4. **Verify the SDK price table recognizes the model.** The `max_budget_usd` guardrail is computed from a price table bundled into the Claude Agent SDK at build time, so an unrecognized model silently degrades budget enforcement. Run `agent/scripts/diagnostics/test_sdk_smoke.py` with `ANTHROPIC_MODEL` set to the new ID, divide the reported cost by the input-token count, and confirm the implied rate matches [published Bedrock pricing](https://aws.amazon.com/bedrock/pricing/). A `$0.00` or wildly-off result means the table does not know the model and budgets cannot be trusted. +5. The doc-drift test (`cdk/test/contracts/model-default-docs-parity.test.ts`) fails until the documented defaults here and in `agent/README.md` match `config.py`. That failure is the reminder, not a nuisance — update both. + +### Cost and model selection + +Model choice is a **cost** decision, which is why it is adjustable per repo and per task without a code change. + +**Per-token rate vs. token volume.** Measured on the pinned toolchain, same one-turn prompt, same system prompt: + +| Model | Input tokens | Reported `cost_usd` | Implied input rate | +|---|---|---|---| +| `us.anthropic.claude-opus-4-8` | 32,145 | $0.160850 | **$5.00/MTok** | +| `us.anthropic.claude-opus-5` | 37,584 | $0.188020 | **$5.00/MTok** | + +Token ratio 1.169; cost ratio 1.169 — identical. **The per-token rate is unchanged; the whole delta is token volume on an identical prompt.** Read it that way: "Opus 5 costs ~17% more per task" invites the wrong remedy (switch models), while "same rate, more tokens" points at the real levers — prompt size, prompt caching, and `max_turns`. + +**Where can I set `max_budget_usd`?** + +| Surface | How | Status | +|---|---|---| +| Per task, CLI | `bgagent submit --max-budget ` (`cli/src/commands/submit.ts:69`), range 0.01–100 | Works | +| Per task, REST | `max_budget_usd` in the `POST /v1/tasks` body | Works | +| Local batch only | `MAX_BUDGET_USD` shell env, when running `entrypoint.py` directly | Works locally; **ignored** by the deployed AgentCore **server** mode, which reads the budget from the `/invocations` JSON body | +| Per repo, Blueprint | `agent.maxBudgetUsd` | **Not implemented** — `cdk/src/constructs/blueprint.ts` has no such prop (it implements `maxTurns`). Tracked in [#748](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/748), which owns that documentation. | +| Platform default | — | None by design: **unset means unlimited** | + +**Unlimited-by-default is deliberate — pair it with the escape hatch.** Because no platform budget ceiling applies, the documented mitigation for cost is choosing a lighter-token model rather than relying on a cap: + +- **Per repo:** Blueprint `agent.modelId` (e.g. `us.anthropic.claude-sonnet-4-6`) — no code change, no agent redeploy +- **Per task:** `model_id` in the task payload +- **Platform-wide:** the `bedrockModels` context plus the layer-2 call sites above + +The model must be in the IAM grant list (layer 1) or the task fails at turn 0 with `AccessDenied` — the grant is the gate, so a lighter model is only reachable if it is granted. + +**Trust boundary on the number.** `cost_usd` is the Claude Agent SDK's **client-side estimate** from that bundled price table — not authoritative billing. It drifts when Bedrock pricing changes, when the SDK version does not recognize a model, or when discounts and commitments apply. See [Cost attribution](./COST_ATTRIBUTION.md) (the warning at line 6); authoritative cost comes from AWS Cost Explorer / CUR 2.0. + ## Installation Follow the [Quick Start](./QUICK_START.mdx) to clone, install, deploy, and submit your first task. It covers prerequisites, toolchain setup, deployment, PAT configuration, Cognito user creation, and a smoke test. @@ -247,12 +336,12 @@ The `--local-events` flag connects the agent container to DynamoDB Local on the | Variable | Default | Description | |---|---|---| -| `ANTHROPIC_MODEL` | `us.anthropic.claude-sonnet-4-6` | Bedrock model ID | +| `ANTHROPIC_MODEL` | `us.anthropic.claude-opus-4-8` | Bedrock inference-profile ID for the main coding model | | `MAX_TURNS` | `100` | Max agent turns before stopping | | `MAX_BUDGET_USD` | | Cost ceiling for local batch runs only (production uses the API field) | | `DRY_RUN` | | Set to `1` to validate and print prompt without running the agent | -For the full list, see `agent/README.md`. +For the full list, see `agent/README.md`. For how the model default is layered, overridden, and priced, see [Model configuration](./DEVELOPER_GUIDE.md#model-configuration). #### Troubleshooting diff --git a/docs/guides/USER_GUIDE.md b/docs/guides/USER_GUIDE.md index 44a9391a6..7b37407e9 100644 --- a/docs/guides/USER_GUIDE.md +++ b/docs/guides/USER_GUIDE.md @@ -215,13 +215,13 @@ Contact your platform administrator to onboard a new repository. For details on ## Per-repo overrides -Blueprints can configure per-repository settings that override platform defaults: +Blueprints can configure per-repository settings that override platform defaults. For how `model_id` is layered against the platform default, per-task overrides, and the IAM invoke allowlist — plus the cost tradeoffs of picking a different model — see [Model configuration](./DEVELOPER_GUIDE.md#model-configuration). | Setting | Description | Default | |---|---|---| | `compute_type` | Compute strategy (`agentcore` or `ecs`) | `agentcore` | | `runtime_arn` | AgentCore runtime ARN override | Platform default | -| `model_id` | Foundation model ID | Platform default | +| `model_id` | Bedrock inference-profile ID (`us.`-prefixed) | `us.anthropic.claude-opus-4-8` | | `max_turns` | Default turn limit for tasks | 100 | | `max_budget_usd` | Default cost budget in USD per task | None (unlimited) | | `system_prompt_overrides` | Additional system prompt instructions | None | diff --git a/docs/scripts/sync-starlight.mjs b/docs/scripts/sync-starlight.mjs index 56b082c20..b7522539a 100644 --- a/docs/scripts/sync-starlight.mjs +++ b/docs/scripts/sync-starlight.mjs @@ -49,11 +49,17 @@ function rewriteDocsLinkTarget(target) { DEPLOY_PREVIEW_SCREENSHOTS_GUIDE: '/using/deploy-preview-screenshots-guide', CEDAR_POLICY_GUIDE: '/customizing/cedar-policies', DEPLOYMENT_GUIDE: '/getting-started/deployment-guide', + // Mirrored to getting-started/Cost-attribution.md (see the copy list below), NOT + // architecture/. Without this entry a relative `./COST_ATTRIBUTION.md` link misses + // the `/guides/` bail-out and falls through to the `/architecture/${slug}` default, + // producing a 404 on the published site. + COST_ATTRIBUTION: '/getting-started/cost-attribution', }; /** `splitGuide` emits each `##` from DEVELOPER_GUIDE as its own page — map #anchors to those routes. */ const developerGuideAnchorRoutes = { 'repository-preparation': '/developer-guide/repository-preparation', + 'model-configuration': '/developer-guide/model-configuration', }; if (stem === 'DEVELOPER_GUIDE' && anchor) { const splitRoute = developerGuideAnchorRoutes[anchor.toLowerCase()]; diff --git a/docs/src/content/docs/architecture/Cost-model.md b/docs/src/content/docs/architecture/Cost-model.md index d9606fb20..d0044f923 100644 --- a/docs/src/content/docs/architecture/Cost-model.md +++ b/docs/src/content/docs/architecture/Cost-model.md @@ -92,12 +92,12 @@ These estimates assume Claude Sonnet with prompt caching enabled and average tas For multi-user deployments, cost should be attributable to individual users and repositories: -- **Per-task:** Token usage and compute duration are captured in task metadata (`agent.cost_usd`, `agent.turns` - see [OBSERVABILITY.md](/sample-autonomous-cloud-coding-agents/architecture/observability)). Note: `agent.cost_usd` is the Claude Agent SDK's **client-side estimate** (a build-time price table), not authoritative billing — use it for guardrails, and AWS Cost Explorer / CUR 2.0 for the real bill (see [COST_ATTRIBUTION.md](../guides/COST_ATTRIBUTION.md)). +- **Per-task:** Token usage and compute duration are captured in task metadata (`agent.cost_usd`, `agent.turns` - see [OBSERVABILITY.md](/sample-autonomous-cloud-coding-agents/architecture/observability)). Note: `agent.cost_usd` is the Claude Agent SDK's **client-side estimate** (a build-time price table), not authoritative billing — use it for guardrails, and AWS Cost Explorer / CUR 2.0 for the real bill (see [COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/getting-started/cost-attribution)). - **Per-user:** Aggregate task costs by `user_id`. - **Per-repo:** Aggregate task costs by `repo`. - **Dashboard:** Cost attribution dashboards should be built from the same task-level metrics. -For **AWS-native** chargeback of Bedrock spend (Cost Explorer / CUR 2.0 by `user_id` / `repo`, plus per-call invocation-log forensics) — beyond the in-app `cost_usd` meter above — see the operator guide [COST_ATTRIBUTION.md](../guides/COST_ATTRIBUTION.md) and the platform design [BEDROCK_COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/architecture/bedrock-cost-attribution). +For **AWS-native** chargeback of Bedrock spend (Cost Explorer / CUR 2.0 by `user_id` / `repo`, plus per-call invocation-log forensics) — beyond the in-app `cost_usd` meter above — see the operator guide [COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/getting-started/cost-attribution) and the platform design [BEDROCK_COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/architecture/bedrock-cost-attribution). ## Cost guardrails (current) diff --git a/docs/src/content/docs/architecture/Repo-onboarding.md b/docs/src/content/docs/architecture/Repo-onboarding.md index 05286b689..61b33220b 100644 --- a/docs/src/content/docs/architecture/Repo-onboarding.md +++ b/docs/src/content/docs/architecture/Repo-onboarding.md @@ -124,7 +124,7 @@ From lowest to highest priority: |---|---|---| | `compute_type` | `agentcore` | Platform constant | | `runtime_arn` | Stack-level env var | CDK stack props | -| `model_id` | Claude Sonnet 4 | CDK stack props | +| `model_id` | `us.anthropic.claude-opus-4-8` | Python literal in `agent/src/config.py` (no CDK prop or env knob today) — see [Model configuration](/sample-autonomous-cloud-coding-agents/developer-guide/model-configuration) | | `max_turns` | 100 | Platform constant | | `max_budget_usd` | None (unlimited) | - | | `memory_token_budget` | 2000 | Platform constant | diff --git a/docs/src/content/docs/customizing/Per-repo-overrides.md b/docs/src/content/docs/customizing/Per-repo-overrides.md index 7a03ede92..360e10ff1 100644 --- a/docs/src/content/docs/customizing/Per-repo-overrides.md +++ b/docs/src/content/docs/customizing/Per-repo-overrides.md @@ -2,13 +2,13 @@ title: Per-repo overrides --- -Blueprints can configure per-repository settings that override platform defaults: +Blueprints can configure per-repository settings that override platform defaults. For how `model_id` is layered against the platform default, per-task overrides, and the IAM invoke allowlist — plus the cost tradeoffs of picking a different model — see [Model configuration](/sample-autonomous-cloud-coding-agents/developer-guide/model-configuration). | Setting | Description | Default | |---|---|---| | `compute_type` | Compute strategy (`agentcore` or `ecs`) | `agentcore` | | `runtime_arn` | AgentCore runtime ARN override | Platform default | -| `model_id` | Foundation model ID | Platform default | +| `model_id` | Bedrock inference-profile ID (`us.`-prefixed) | `us.anthropic.claude-opus-4-8` | | `max_turns` | Default turn limit for tasks | 100 | | `max_budget_usd` | Default cost budget in USD per task | None (unlimited) | | `system_prompt_overrides` | Additional system prompt instructions | None | diff --git a/docs/src/content/docs/developer-guide/Installation.md b/docs/src/content/docs/developer-guide/Installation.md index 6fc81d2b3..6811e81de 100644 --- a/docs/src/content/docs/developer-guide/Installation.md +++ b/docs/src/content/docs/developer-guide/Installation.md @@ -133,12 +133,12 @@ The `--local-events` flag connects the agent container to DynamoDB Local on the | Variable | Default | Description | |---|---|---| -| `ANTHROPIC_MODEL` | `us.anthropic.claude-sonnet-4-6` | Bedrock model ID | +| `ANTHROPIC_MODEL` | `us.anthropic.claude-opus-4-8` | Bedrock inference-profile ID for the main coding model | | `MAX_TURNS` | `100` | Max agent turns before stopping | | `MAX_BUDGET_USD` | | Cost ceiling for local batch runs only (production uses the API field) | | `DRY_RUN` | | Set to `1` to validate and print prompt without running the agent | -For the full list, see `agent/README.md`. +For the full list, see `agent/README.md`. For how the model default is layered, overridden, and priced, see [Model configuration](/sample-autonomous-cloud-coding-agents/developer-guide/model-configuration). #### Troubleshooting diff --git a/docs/src/content/docs/developer-guide/Model-configuration.md b/docs/src/content/docs/developer-guide/Model-configuration.md new file mode 100644 index 000000000..cf03195aa --- /dev/null +++ b/docs/src/content/docs/developer-guide/Model-configuration.md @@ -0,0 +1,90 @@ +--- +title: Model configuration +--- + +**This is the canonical reference for which model the agent uses and where to change it.** The model ID is configured across five independent layers in three languages, so read this section before changing a default — a mismatch between the layers fails every task on the stack at turn 0, not just an edge case. + +### The five layers + +| # | Layer | What it controls | Where | ID form | +|---|---|---|---|---| +| 1 | **IAM invoke allowlist** | Which models the agent's roles may invoke at all. The outer gate — everything below fails without it. | `DEFAULT_BEDROCK_MODEL_IDS` (`cdk/src/constructs/bedrock-models.ts:34`); override with CDK context `bedrockModels` (key at `:48`, resolver at `:67`) | **Bare** (`anthropic.claude-…`) | +| 2 | **Platform default model** | The model used when nothing narrower is set. A **Python literal only** — there is no CDK prop or environment knob in front of it today. | `agent/src/config.py:563` (the `ANTHROPIC_MODEL` fallback) and `agent/src/models.py:157` (`TaskConfig.anthropic_model`) | Prefixed (`us.anthropic.…`) | +| 3 | **Auxiliary / fast model** | The small model Claude Code uses for auxiliary work (WebFetch page summarization, the pre-flight safety check). | Stack env `ANTHROPIC_DEFAULT_HAIKU_MODEL` (`cdk/src/stacks/agent.ts` (the runtime environment block)); agent-side fallback at `agent/src/config.py:569` | Prefixed (`us.anthropic.…`) | +| 4 | **Per-repo override** | One repository's model, with no agent redeploy. | Blueprint `agent.modelId` (`cdk/src/constructs/blueprint.ts`, `BlueprintProps.agent.modelId`) → RepoTable `model_id` (`cdk/src/handlers/shared/repo-config.ts:37`) → ECS injects `ANTHROPIC_MODEL` (`cdk/src/handlers/shared/strategies/ecs-strategy.ts:217`) | Prefixed (`us.anthropic.…`) | +| 5 | **Per-task / local** | One task's model. Payload `model_id` is aliased to `anthropic_model` (`agent/src/pipeline.py`, `_PAYLOAD_KEY_ALIASES`); local batch runs read `ANTHROPIC_MODEL` from the shell via `agent/run.sh`. | Task payload `model_id`; shell `ANTHROPIC_MODEL` | Prefixed (`us.anthropic.…`) | + +### Environment variables + +| Variable | Who sets it | ID form | Purpose | +|---|---|---|---| +| `ANTHROPIC_MODEL` | ECS strategy from the repo Blueprint (layer 4); you, in the shell, for local batch runs (layer 5) | Prefixed inference profile | The main coding model. Unset → the `agent/src/config.py` fallback. | +| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | The CDK stack, hardcoded at `cdk/src/stacks/agent.ts` (the runtime environment block) | Prefixed inference profile | The small/fast auxiliary model. Must be a granted profile, or the pre-flight check times out with *"Pre-flight check is taking longer than expected"*. | +| `CLAUDE_CODE_USE_BEDROCK` | The CDK stack (`='1'`) and `agent/run.sh` | — | Routes Claude Code to Bedrock instead of the Anthropic API. ABCA always runs on Bedrock. | + +### Precedence — narrowest wins + +```text +per-task payload model_id (layer 5) + > blueprint agent.modelId (layer 4, arrives as stack env ANTHROPIC_MODEL) + > stack env ANTHROPIC_MODEL (layer 3-adjacent / local shell) + > agent/src/config.py fallback (layer 2 — us.anthropic.claude-opus-4-8) +``` + +Every one of those is gated by the **IAM invoke allowlist** (layer 1), which is itself gated by **account-level Bedrock model access**. Both gates are silent until invocation: a model that resolves fine through precedence still fails at turn 0 with `AccessDenied` if it is not in the grant list, and fails again if your account has not completed [Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for it. + +### Bare vs. prefixed IDs — the one rule that bites + +Layer 1 takes **bare foundation-model IDs**; every other layer takes the **prefixed inference-profile ID**. This asymmetry is deliberate: both grant sites derive the inference-profile ARN by *adding* the `us.` prefix themselves, so a prefixed entry in `bedrockModels` would produce an invalid `us.us.anthropic.…` ARN. The resolver rejects a `us.`/`eu.`/`apac.`-prefixed entry at `cdk/src/constructs/bedrock-models.ts:84` so the typo fails at synth rather than at runtime. + +In the other direction, a **bare** ID cannot be invoked on demand at all. Verified: + +```console +$ aws bedrock-runtime invoke-model --model-id anthropic.claude-opus-5 ... +ValidationException: Invocation of model ID anthropic.claude-opus-5 with on-demand +throughput isn't supported. Retry your request with the ID or ARN of an inference +profile that contains this model. +``` + +So: `bedrockModels` context → `anthropic.claude-opus-4-8`. Everywhere else → `us.anthropic.claude-opus-4-8`. + +### Bumping the default model + +1. Add the **bare** ID to `DEFAULT_BEDROCK_MODEL_IDS` (`cdk/src/constructs/bedrock-models.ts`) and deploy, so the grant exists before anything tries to use it. +2. Confirm account-level Bedrock access for the model in the target Region. +3. Update the **prefixed** ID in `agent/src/config.py` and `agent/src/models.py`. +4. **Verify the SDK price table recognizes the model.** The `max_budget_usd` guardrail is computed from a price table bundled into the Claude Agent SDK at build time, so an unrecognized model silently degrades budget enforcement. Run `agent/scripts/diagnostics/test_sdk_smoke.py` with `ANTHROPIC_MODEL` set to the new ID, divide the reported cost by the input-token count, and confirm the implied rate matches [published Bedrock pricing](https://aws.amazon.com/bedrock/pricing/). A `$0.00` or wildly-off result means the table does not know the model and budgets cannot be trusted. +5. The doc-drift test (`cdk/test/contracts/model-default-docs-parity.test.ts`) fails until the documented defaults here and in `agent/README.md` match `config.py`. That failure is the reminder, not a nuisance — update both. + +### Cost and model selection + +Model choice is a **cost** decision, which is why it is adjustable per repo and per task without a code change. + +**Per-token rate vs. token volume.** Measured on the pinned toolchain, same one-turn prompt, same system prompt: + +| Model | Input tokens | Reported `cost_usd` | Implied input rate | +|---|---|---|---| +| `us.anthropic.claude-opus-4-8` | 32,145 | $0.160850 | **$5.00/MTok** | +| `us.anthropic.claude-opus-5` | 37,584 | $0.188020 | **$5.00/MTok** | + +Token ratio 1.169; cost ratio 1.169 — identical. **The per-token rate is unchanged; the whole delta is token volume on an identical prompt.** Read it that way: "Opus 5 costs ~17% more per task" invites the wrong remedy (switch models), while "same rate, more tokens" points at the real levers — prompt size, prompt caching, and `max_turns`. + +**Where can I set `max_budget_usd`?** + +| Surface | How | Status | +|---|---|---| +| Per task, CLI | `bgagent submit --max-budget ` (`cli/src/commands/submit.ts:69`), range 0.01–100 | Works | +| Per task, REST | `max_budget_usd` in the `POST /v1/tasks` body | Works | +| Local batch only | `MAX_BUDGET_USD` shell env, when running `entrypoint.py` directly | Works locally; **ignored** by the deployed AgentCore **server** mode, which reads the budget from the `/invocations` JSON body | +| Per repo, Blueprint | `agent.maxBudgetUsd` | **Not implemented** — `cdk/src/constructs/blueprint.ts` has no such prop (it implements `maxTurns`). Tracked in [#748](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/748), which owns that documentation. | +| Platform default | — | None by design: **unset means unlimited** | + +**Unlimited-by-default is deliberate — pair it with the escape hatch.** Because no platform budget ceiling applies, the documented mitigation for cost is choosing a lighter-token model rather than relying on a cap: + +- **Per repo:** Blueprint `agent.modelId` (e.g. `us.anthropic.claude-sonnet-4-6`) — no code change, no agent redeploy +- **Per task:** `model_id` in the task payload +- **Platform-wide:** the `bedrockModels` context plus the layer-2 call sites above + +The model must be in the IAM grant list (layer 1) or the task fails at turn 0 with `AccessDenied` — the grant is the gate, so a lighter model is only reachable if it is granted. + +**Trust boundary on the number.** `cost_usd` is the Claude Agent SDK's **client-side estimate** from that bundled price table — not authoritative billing. It drifts when Bedrock pricing changes, when the SDK version does not recognize a model, or when discounts and commitments apply. See [Cost attribution](/sample-autonomous-cloud-coding-agents/getting-started/cost-attribution) (the warning at line 6); authoritative cost comes from AWS Cost Explorer / CUR 2.0. \ No newline at end of file diff --git a/docs/src/content/docs/getting-started/Deployment-guide.md b/docs/src/content/docs/getting-started/Deployment-guide.md index 01f6e2ce5..e7eccb25b 100644 --- a/docs/src/content/docs/getting-started/Deployment-guide.md +++ b/docs/src/content/docs/getting-started/Deployment-guide.md @@ -233,5 +233,5 @@ For users without AWS CLI access. - [User guide](/sample-autonomous-cloud-coding-agents/using/overview) -- API reference, CLI usage, task management. - [DEPLOYMENT_ROLES.md](/sample-autonomous-cloud-coding-agents/architecture/deployment-roles) -- Least-privilege IAM policies for CloudFormation execution. - [COST_MODEL.md](/sample-autonomous-cloud-coding-agents/architecture/cost-model) -- Per-task costs, cost guardrails, cost at scale. -- [COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/architecture/cost-attribution) -- Operator FinOps setup for per-user/per-repo Bedrock chargeback (Cost Explorer / CUR 2.0, invocation-log forensics). +- [COST_ATTRIBUTION.md](/sample-autonomous-cloud-coding-agents/getting-started/cost-attribution) -- Operator FinOps setup for per-user/per-repo Bedrock chargeback (Cost Explorer / CUR 2.0, invocation-log forensics). - [COMPUTE.md](/sample-autonomous-cloud-coding-agents/architecture/compute) -- Compute backend architecture and trade-offs.