diff --git a/agent/README.md b/agent/README.md index ecfc6bdde..2d7bdac67 100644 --- a/agent/README.md +++ b/agent/README.md @@ -218,18 +218,37 @@ Immediate response (acceptance): Final metrics (PR URL, cost, turns, build status, etc.) appear in **container logs**, in **DynamoDB** when configured, and in the **REST API** for deployed tasks (`GET /v1/tasks/{task_id}` via the `bgagent` CLI or HTTP client). -### AWS Lambda MicroVMs lifecycle hooks (ADR-021 P1) +### AWS Lambda MicroVMs lifecycle hooks (ADR-021 P1 + P2) The same uvicorn process also serves the **Lambda MicroVMs** lifecycle hooks, on the same port (8080 — the port declared in the image's `hooks.port`). On that backend there is no `InvokeAgentRuntime` and no orchestrator→agent HTTP path at all: the task payload arrives as the `/run` hook body and nothing else dials in. -**`POST /aws/lambda-microvms/runtime/v1/ready`** — Build hook. Returns `{"status": "ready"}` as soon as the server is up, which is the signal the service waits for before taking the snapshot. **Mandatory**, not optional: `CreateMicrovmImage` refuses an image that enables *any* lifecycle hook without `/ready`, and with the hook enabled but unserved every build fails with `Ready hook check failed: the application returned a client error (HTTP 4xx) response`. +**`POST /aws/lambda-microvms/runtime/v1/ready`** — Build hook. **Mandatory**, not optional: `CreateMicrovmImage` refuses an image that enables *any* lifecycle hook without `/ready`, and with the hook enabled but unserved every build fails with `Ready hook check failed: the application returned a client error (HTTP 4xx) response`. -**`POST /aws/lambda-microvms/runtime/v1/run`** — Payload delivery. Validates the body, starts the pipeline in a background thread (the same `_extract_invocation_params` → `_spawn_background` path `/invocations` uses), and returns 200 inside the 1–60 s hook budget. Body: +Its 200 is the signal the service waits for before **taking the snapshot**, which makes it the only hook that can put a warm page in that snapshot — so it does two things: + +1. reports that uvicorn is bound and `server` imported cleanly (which pulls in `pipeline` → `runner` → the policy engine, so a missing policy file fails the *build* rather than the first task); +2. **warms the heavyweight binaries** by exec'ing `claude --version` (required), plus `git` and `node` best-effort. + +That warm-up exists because of a live defect (ADR-021 P2-F5): `claude` is a **225 MiB statically-linked ELF**, and on a MicroVM restored from a snapshot that never touched it, the first `exec` had to fault every page in from lazily-restored storage — `runner.py`'s version probe timed out at 10 s and **every task died at turn 0**, reproducibly, while the same binary in the same image answers in under a second locally. If a *required* warm-up fails, `/ready` returns **503** (the hook contract's "not ready yet"), so the service keeps asking within the `/ready` budget and, failing that, fails the image build — the right trade, since a snapshot that cannot exec `claude` cannot run one task. The `/ready` hook budget is 300 s for the same reason (`READY_HOOK_TIMEOUT_SECONDS` in `cdk/src/constructs/lambda-microvm-compute.ts`); build hooks are allowed up to 3600 s. Budgets are composed, not stacked: `claude` runs **first** with its own 120 s, the best-effort commands then **share** what is left of a 240 s total ceiling (and are skipped once the remainder is too small to warm anything), so the whole warm-up stays inside the hook budget and a hung `git`/`node` can never hold up a 200 the required warm-up has already earned. All three numbers live in `contracts/constants.json` → `microvm_hook_budgets`, read by this server *and* by the CDK construct, because `warmup_required < warmup_total < ready_hook` is a relationship neither side can enforce alone — `scripts/check-constants-sync.ts` asserts the ordering and bans a literal re-declaration on either side. A `--version` exec is not an AWS call and touches no network, so the hook stays AWS-silent (below). + +The warm-up is the *primary* fix; the probe that failed is also now non-fatal. `runner.py`'s `_log_claude_cli_version` keeps its loosened 60 s bound but additionally downgrades every `subprocess.SubprocessError`/`OSError` to a `WARN` — a diagnostic whose entire output is a log line must not be able to end a task at turn 0, however it fails. + +**`POST /aws/lambda-microvms/runtime/v1/validate`** — Build hook (P2). A **shallow self-check only**: server alive, every hook route registered, interpreter floor, `platform_config` contract loaded. Returns 200 with the individual check results, or 503 while still initialising (which fails the build if it never clears — the right outcome for a broken snapshot). That 503 branch is a **refactor tripwire**, not a state you can reach today: `_module_initialized` is set as the module's last statement and uvicorn accepts no request until the import completes, so it only becomes reachable once someone moves warm-up work behind the bind — and a hook with only a 200 path would then report a still-initialising snapshot as valid. + +It runs under the **build role**, which deliberately holds no Bedrock / Secrets Manager / DynamoDB grants, so it **makes zero AWS API calls and must keep making zero** — including its own logging (both build hooks log to stdout via `_build_hook_log`, never through the CloudWatch writer). Two reasons: a Logs write under the build role can only fail, and each failure pollutes the shared `_debug_cw_failures` alarm signal; and `boto3.client(...)` populates `boto3.DEFAULT_SESSION`, a module global holding a resolved credential chain plus the build-time region, which the snapshot would then freeze in for every MicroVM launched from that image version. "Deeper warm-up assertions" (Bedrock reachability, Memory access, tool availability) are therefore *not* implementable here. The one member of that list that turned out to be partly implementable — proving the local `claude` binary execs — lives on `/ready` instead, as a side effect of warming it (above): a local `exec` is not an AWS call, and it belongs to the hook whose 200 gates the snapshot. + +Baked secrets are **reported, not enforced**: `warnings` lists the names (never values) of any credential-shaped env var present in the snapshot, because the build environment's own credentials may legitimately be in that env and failing here would fail every build. + +**`POST /aws/lambda-microvms/runtime/v1/terminate`** — Runtime hook (P2). Best-effort: emits one final structured log line and returns 200 — always, inside the hook budget, even with nothing running, and for **any body**: malformed JSON, a wrong content-type, an empty body or no body at all. That is why the handler takes the raw request instead of a typed body model — FastAPI validates a typed body *before* the handler runs, so a truncated body would answer 422 and report a hook failure for a teardown that actually succeeded. It does **not** join the pipeline thread (that is `lifespan`'s job on graceful shutdown) and it **never writes terminal task status**: the orchestrator finalizes the task and *then* calls `TerminateMicrovm`, so a status write here would race that finalization. Nothing is buffered to flush — `ProgressWriter` does a synchronous `put_item` per event, so progress is already durable at call time. + +`microvmId` is parsed defensively and **arrives empty in practice**: the service sends `""` here, unlike `/run` where it is populated (live-verified, ADR-021 P2-F8). So an empty id is expected-normal, not a degraded read — and this hook therefore **cannot** join the guest's record to the control-plane one. `/run`'s `hook accepted task_id=… microvm_id=…` line carries that correlation; `/terminate`'s value is the pipeline-state snapshot it reports. + +**`POST /aws/lambda-microvms/runtime/v1/run`** — Payload delivery. Validates the body, installs `platform_config` (below), starts the pipeline in a background thread (the same `_extract_invocation_params` → `_spawn_background` path `/invocations` uses), and returns 200 inside the 1–60 s hook budget. Body: ```json { "microvmId": "microvm-b44b69d9-…", - "runHookPayload": "{\"agent_payload_s3_uri\": \"s3://bucket//payload.json\"}" + "runHookPayload": "{\"agent_payload_s3_uri\": \"s3://bucket//payload.json\", \"platform_config\": {…}}" } ``` @@ -237,14 +256,42 @@ The same uvicorn process also serves the **Lambda MicroVMs** lifecycle hooks, on | Envelope | When | |---|---| -| `{"agent_payload": {…}}` | the whole orchestrator payload inline — only when it fits | -| `{"agent_payload_s3_uri": "s3://bucket/key"}` | pointer to the payload in the platform payload bucket | +| `{"agent_payload": {…}, "platform_config": {…}}` | the whole orchestrator payload inline — only when it fits | +| `{"agent_payload_s3_uri": "s3://bucket/key", "platform_config": {…}}` | pointer to the payload in the platform payload bucket | The service caps `runHookPayload` at **4 096 bytes**, so the **pointer form is the normal one** — a hydrated payload is essentially always larger. Fetching it needs no new env var: the MicroVM execution role holds read-only access to that bucket and the URI carries bucket + key. -Rejections are structured so they are readable in the MicroVM log group: `400 MICROVM_RUN_PAYLOAD_INVALID` (unusable envelope — retrying the same body cannot help), `500 MICROVM_RUN_PAYLOAD_UNREADABLE` (the S3 fetch failed), `400 TASK_RECORD_INCOMPLETE` (same validator and vocabulary as `/invocations`). +#### `platform_config` — the agent's env, delivered per task (P2) + +On AgentCore and ECS the agent's non-secret platform env arrives as runtime env / container overrides. A MicroVM boots from a **snapshot**, so its env is frozen at *image build* time; baking the deployment's identifiers in would make every image version describe a deployment that may since have been redeployed. They therefore travel with the task, as a **sibling** of `agent_payload` (per-task fields like `memory_id` stay *inside* `agent_payload` — `platform_config` configures the process, `agent_payload` describes the task): + +```json +{ "platform_config": { "task_table_name": "…", "github_token_secret_arn": "arn:…" } } +``` -`/validate` (build) and `/suspend`, `/resume`, `/terminate` (runtime) are deliberately **not** served — declaring a hook nothing answers fails the corresponding build or lifecycle transition, so the CDK construct declares exactly `/ready` + `/run`. `/terminate` and `/validate` land in P2; `/suspend` + `/resume` in P3 with the ComputeStrategy interface widening. +Each snake_case key installs into its UPPER_SNAKE env var, and a payload value **wins** over any image/pre-existing value (the payload describes the live deployment; the snapshot describes a past one). Installation happens **before** any credential or pipeline initialisation — the very next step resolves the GitHub token from `GITHUB_TOKEN_SECRET_ARN`. Everything the hook logs before that point goes to stdout only (`[server/run-pre-config]`), for the same reason the build hooks do: the CloudWatch writer resolves AWS credentials and pins `boto3.DEFAULT_SESSION` (region included), and until the install has run the only environment available is whatever the snapshot baked. The single AWS call allowed before the install is the S3 payload fetch, because the config is inside the object being fetched. The allowlist lives in `contracts/constants.json` → `microvm_platform_config` (produced by the orchestrator, consumed here; shape enforced by `mise run check:constants-sync`): + +| Key | Env var | Required | +|---|---|---| +| `task_table_name` | `TASK_TABLE_NAME` | ✅ | +| `task_events_table_name` | `TASK_EVENTS_TABLE_NAME` | ✅ | +| `github_token_secret_arn` | `GITHUB_TOKEN_SECRET_ARN` | ✅ | +| `agent_session_role_arn` | `AGENT_SESSION_ROLE_ARN` | ✅ | +| `task_approvals_table_name` | `TASK_APPROVALS_TABLE_NAME` | | +| `nudges_table_name` | `NUDGES_TABLE_NAME` | | +| `log_group_name` | `LOG_GROUP_NAME` | | +| `artifacts_bucket_name` | `ARTIFACTS_BUCKET_NAME` | | +| `trace_artifacts_bucket_name` | `TRACE_ARTIFACTS_BUCKET_NAME` | | +| `linear_oauth_secret_arn` | `LINEAR_OAUTH_SECRET_ARN` | | +| `jira_oauth_secret_arn` | `JIRA_OAUTH_SECRET_ARN` | | +| `aws_sdk_ua_app_id` | `AWS_SDK_UA_APP_ID` | | +| `anthropic_default_haiku_model` | `ANTHROPIC_DEFAULT_HAIKU_MODEL` | | + +Values are **non-secret identifiers only** — secrets are still fetched at `/run` time from Secrets Manager using the ARNs delivered here, so the snapshot stays secret-free. The allowlist **fails closed**: these values land in `os.environ` of the process that spawns the agent's tool subprocesses, so an unrecognised key is an env-injection attempt (`LD_PRELOAD`, `AWS_ENDPOINT_URL`, …) and the whole run is rejected with nothing installed. Blank/`null` values for optional keys are skipped rather than clobbering an image value; blank required keys are rejected. An envelope with no `platform_config` at all is accepted with a loud warning (the image and the orchestrator deploy on independent cadences). + +Rejections are structured so they are readable in the MicroVM log group: `400 MICROVM_RUN_PAYLOAD_INVALID` (unusable envelope — retrying the same body cannot help), `500 MICROVM_RUN_PAYLOAD_UNREADABLE` (the S3 fetch failed), `400 MICROVM_RUN_PLATFORM_CONFIG_INVALID` (key off the allowlist, non-object block, or non-string value — fix the producer), `400 MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE` (a required key missing or blank — fix the deployment wiring), `400 TASK_RECORD_INCOMPLETE` (same validator and vocabulary as `/invocations`). + +`/suspend` and `/resume` are deliberately **not** served — declaring a hook nothing answers fails the corresponding lifecycle transition, so the CDK construct declares exactly the hooks the agent serves. They land in P3 with the ComputeStrategy interface widening. ### Testing Server Mode Locally diff --git a/agent/src/runner.py b/agent/src/runner.py index c88f1194f..6ec08ce49 100644 --- a/agent/src/runner.py +++ b/agent/src/runner.py @@ -435,6 +435,68 @@ def _resolve_setting_sources(config: TaskConfig) -> list[Literal["user", "projec return ["project"] if config.repo_url else [] +#: Timeout (seconds) for the ``claude --version`` diagnostic probe. +#: +#: 60, not the original 10 (ADR-021 P2-F5, live 2026-08-07). This probe killed +#: EVERY task on the ``lambda-microvm`` backend at turn 0, reproducibly: +#: +#: TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +#: +#: The binary was fine — the same image answers ``2.1.191 (Claude Code)`` in under +#: a second locally. It is a 225 MiB (236,305,136-byte) statically linked ELF, and +#: on a MicroVM restored from a snapshot that never touched it, the first ``exec`` +#: must fault all of its pages in from lazily-restored storage. +#: +#: The real fix is warming the binary BEFORE the snapshot is captured +#: (``_warm_snapshot_binaries`` in ``server.py``'s ``/ready`` hook); this bound is +#: the belt to that braces, and it is deliberately loose because a tight bound buys +#: NOTHING here: the call prints a version string into a log line, and its failure +#: mode is a dead task. Any cold-start environment (a fresh container, a cold page +#: cache, a throttled volume) has to fit inside it. 60 s still fails loudly on a +#: genuinely broken binary. +_CLAUDE_VERSION_PROBE_TIMEOUT_S = 60 + +#: Timeout (seconds) for ``which claude``. Unchanged at 5: a PATH lookup touches no +#: page of the 225 MiB binary, so it is not subject to the hydration cost above. +_WHICH_CLAUDE_TIMEOUT_S = 5 + + +def _log_claude_cli_version() -> None: + """Log the resolved ``claude`` path and version, for protocol-mismatch triage. + + Diagnostics only — nothing branches on the result, which is exactly why the + timeout above is loose. Extracted from ``run_agent`` so the probe is assertable + in a unit test without standing up the SDK client: this line, and only this + line, is what failed every task on the MicroVM backend (P2-F5). + + **Diagnostics-only means it cannot fail the task.** Raising the timeout to 60 s + made P2-F5 unlikely; it did not make it impossible, and a probe whose entire + output is a log line has no business propagating. So every failure mode of the + two ``subprocess.run`` calls is caught and downgraded to a ``WARN``: + ``TimeoutExpired`` (``SubprocessError``) for a still-hydrating binary, + ``FileNotFoundError`` / ``PermissionError`` (``OSError``) for a missing or + non-executable ``which``/``claude``. The real ``claude`` invocation happens + inside the SDK client below and still fails loudly — this line does not. + """ + try: + cli_path = subprocess.run( + ["which", "claude"], capture_output=True, text=True, timeout=_WHICH_CLAUDE_TIMEOUT_S + ) + if cli_path.returncode != 0: + log("WARN", "claude CLI not found on PATH") + return + cli_ver = subprocess.run( + ["claude", "--version"], + capture_output=True, + text=True, + timeout=_CLAUDE_VERSION_PROBE_TIMEOUT_S, + ) + except (subprocess.SubprocessError, OSError) as exc: + log("WARN", f"claude CLI version probe failed (non-fatal): {type(exc).__name__}: {exc}") + return + log("AGENT", f"claude CLI: {cli_path.stdout.strip()} version={cli_ver.stdout.strip()}") + + async def run_agent( prompt: str, system_prompt: str, @@ -470,14 +532,7 @@ def _on_stderr(line: str) -> None: sdk_version = getattr(_sdk, "__version__", "unknown") log("AGENT", f"claude-agent-sdk version: {sdk_version}") - cli_path = subprocess.run(["which", "claude"], capture_output=True, text=True, timeout=5) - if cli_path.returncode == 0: - cli_ver = subprocess.run( - ["claude", "--version"], capture_output=True, text=True, timeout=10 - ) - log("AGENT", f"claude CLI: {cli_path.stdout.strip()} version={cli_ver.stdout.strip()}") - else: - log("WARN", "claude CLI not found on PATH") + _log_claude_cli_version() # SDK tool surface — see _resolve_allowed_tools for the policy. allowed_tools = _resolve_allowed_tools(config) diff --git a/agent/src/server.py b/agent/src/server.py index 831696555..7feef404a 100644 --- a/agent/src/server.py +++ b/agent/src/server.py @@ -14,6 +14,9 @@ import json import logging import os +import re +import subprocess +import sys import threading import time as _time_for_debug import traceback @@ -30,6 +33,7 @@ from models import TaskResult from observability import propagate_correlation_context from pipeline import run_task +from shared_constants import SHARED_CONSTANTS # --- _debug_cw / _warn_cw failure counter ------------------------------- # Shared counter for BOTH the debug and warn CloudWatch writers. AgentCore @@ -819,7 +823,7 @@ async def invoke_agent(request: Request, body: InvocationRequest): # -------------------------------------------------------------------------- -# AWS Lambda MicroVMs lifecycle hooks (ADR-021 P1) +# AWS Lambda MicroVMs lifecycle hooks (ADR-021 P1 + P2) # -------------------------------------------------------------------------- # The MicroVM backend has NO orchestrator→agent HTTP path: the task payload # arrives as the ``/run`` hook's request body and nothing else dials in. The @@ -827,23 +831,356 @@ async def invoke_agent(request: Request, body: InvocationRequest): # (8080 — the same uvicorn process that serves /invocations and /ping), so the # hooks live here rather than in a sidecar. # -# Only ``/ready`` and ``/run`` are implemented, and both are P1: -# * ``/ready`` is MANDATORY. ``CreateMicrovmImage`` refuses an image that -# enables ANY lifecycle hook without it ("The ready (/ready) MicroVM image -# hook must be enabled when any MicroVM lifecycle hook … is enabled"), and an -# image with no hooks at all cannot receive a ``runHookPayload``. So ADR-021's -# original "declare /run in P1, serve it in P2" split was not a reachable -# service state. -# * ``/run`` is the payload-delivery channel. +# Four hooks are served; ``/suspend`` + ``/resume`` are still P3: +# * ``/ready`` (build, P1) is MANDATORY. ``CreateMicrovmImage`` refuses an image +# that enables ANY lifecycle hook without it ("The ready (/ready) MicroVM +# image hook must be enabled when any MicroVM lifecycle hook … is enabled"), +# and an image with no hooks at all cannot receive a ``runHookPayload``. So +# ADR-021's original "declare /run in P1, serve it in P2" split was not a +# reachable service state. +# * ``/run`` (runtime, P1) is the payload-delivery channel — and, since P2, the +# platform-configuration channel (see ``platform_config`` below). +# * ``/validate`` (build, P2) is the snapshot self-check. It runs under the +# BUILD role and makes ZERO AWS calls — see ``microvm_validate``. +# * ``/terminate`` (runtime, P2) is a best-effort final flush. It never writes +# terminal task status — the orchestrator owns terminal state. # ``/suspend`` and ``/resume`` are P3 (they need the ComputeStrategy interface -# widening), and ``/validate`` + ``/terminate`` are P2 polish. Declaring a hook -# the agent does not answer fails the corresponding build or lifecycle -# transition, which is why the construct declares exactly these two. +# widening). Declaring a hook the agent does not answer fails the corresponding +# build or lifecycle transition, which is why the construct declares exactly the +# hooks served here. MICROVM_HOOK_PREFIX = "/aws/lambda-microvms/runtime/v1" #: ``s3://`` scheme prefix for the out-of-band payload pointer. _S3_URI_SCHEME = "s3://" +# --- platform_config allowlist (ADR-021 P2) -------------------------------- +# WHY the agent's platform env arrives in the ``/run`` payload at all, instead of +# being baked into the image like it is on AgentCore/ECS: the MicroVM image is a +# SNAPSHOT. Its process environment is frozen at build time and then replayed by +# every MicroVM launched from that image version, so image env is +# version-frozen — it describes the deployment as it looked when the snapshot was +# taken. The orchestrator's values describe the LIVE deployment (tables, buckets, +# secret ARNs, the session role it just provisioned). When the two disagree the +# live one is right, so a payload-supplied value WINS over any pre-existing / +# image value (see ``_install_platform_config``). It also keeps the build hooks +# AWS-silent: with no ``LOG_GROUP_NAME`` in the snapshot there is nothing for a +# build-time hook to write to (see ``_build_hook_log``). +# +# WHY an allowlist, and why it fails closed: these values are installed into +# ``os.environ`` of the process that spawns the agent's tool subprocesses. An +# unrecognised key is therefore an attempt to set an arbitrary environment +# variable in the agent (``AWS_ENDPOINT_URL``, ``LD_PRELOAD``, ``PATH``, …), i.e. +# an injection attempt, not a forward-compatibility nicety — so an unknown key +# REJECTS the whole run rather than being skipped. Values are non-secret +# identifiers only; secrets are still fetched at ``/run`` time from Secrets +# Manager using the ARNs delivered here. +# +# SOURCE OF TRUTH: ``contracts/constants.json`` → +# ``microvm_platform_config``. The PRODUCER of this block is the orchestrator's +# MicroVM run-hook envelope builder (``cdk/src/handlers/shared/orchestrator.ts``, +# ADR-021 P2 Stage B), which must read the SAME contract file (TypeScript gets +# compile-time enforcement of the key names via ``resolveJsonModule``) rather +# than re-declaring the key names. ``mise run check:constants-sync`` +# (``scripts/check-constants-sync.ts``) validates this block's shape and rejects a +# literal re-declaration of either constant below in the Python consumers. +_PLATFORM_CONFIG_CONTRACT: dict[str, Any] = SHARED_CONSTANTS["microvm_platform_config"] + +#: ``platform_config`` key (snake_case) → environment variable it becomes. +MICROVM_PLATFORM_CONFIG_ENV_BY_KEY: dict[str, str] = dict(_PLATFORM_CONFIG_CONTRACT["env_by_key"]) + +#: Subset without which a task cannot run: no task/event tables means no status +#: or progress writes, no GitHub secret ARN means no clone/PR, and no session +#: role ARN means no tenant-scoped credentials. Missing or blank → 400. +MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS: frozenset[str] = frozenset( + _PLATFORM_CONFIG_CONTRACT["required"] +) + +_PLATFORM_CONFIG_KEY_RE = re.compile(r"^[a-z][a-z0-9_]*$") +_PLATFORM_CONFIG_ENV_RE = re.compile(r"^[A-Z][A-Z0-9_]*$") + + +def _validate_platform_config_contract() -> None: + """Fail-fast on a malformed ``microvm_platform_config`` contract. + + Runs at import time, so a corrupt contract fails the IMAGE BUILD (uvicorn + never binds → the ``/ready`` hook never answers) instead of the first task — + the same posture the policy-file load already has. + """ + where = "contracts/constants.json: microvm_platform_config" + if not MICROVM_PLATFORM_CONFIG_ENV_BY_KEY: + raise ValueError(f"{where}.env_by_key must not be empty") + for key, env_name in MICROVM_PLATFORM_CONFIG_ENV_BY_KEY.items(): + if not _PLATFORM_CONFIG_KEY_RE.match(key): + raise ValueError(f"{where}.env_by_key key {key!r} is not snake_case") + if not isinstance(env_name, str) or not _PLATFORM_CONFIG_ENV_RE.match(env_name): + raise ValueError( + f"{where}.env_by_key[{key!r}] must be an UPPER_SNAKE env var name, got {env_name!r}" + ) + env_names = list(MICROVM_PLATFORM_CONFIG_ENV_BY_KEY.values()) + if len(set(env_names)) != len(env_names): + raise ValueError(f"{where}.env_by_key maps two keys onto the same env var") + if not MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS: + raise ValueError(f"{where}.required must not be empty") + unknown_required = sorted( + MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS - set(MICROVM_PLATFORM_CONFIG_ENV_BY_KEY) + ) + if unknown_required: + raise ValueError( + f"{where}.required names key(s) absent from env_by_key: {unknown_required}" + ) + + +_validate_platform_config_contract() + + +#: MicroVM lifecycle-hook budgets, shared with the CDK construct that declares them. +#: +#: ``contracts/constants.json`` → ``microvm_hook_budgets``. The agent's ``/ready`` +#: warm-up ceiling and the service-side ``/ready`` hook timeout +#: (``READY_HOOK_TIMEOUT_SECONDS`` in +#: ``cdk/src/constructs/lambda-microvm-compute.ts``) are not two independent +#: numbers: the warm-up must finish inside the hook budget or a fix for a runtime +#: failure becomes a build failure. An invariant between two values cannot be +#: enforced from one side, so both live in the contract and +#: ``scripts/check-constants-sync.ts`` asserts ``warmup_total < ready_hook`` and +#: rejects a literal re-declaration on either side. +_HOOK_BUDGETS: dict[str, int] = SHARED_CONSTANTS["microvm_hook_budgets"] + + +def _validate_hook_budget_contract() -> None: + """Fail-fast on a ``microvm_hook_budgets`` block that cannot hold. + + Import time, so a contract whose warm-up no longer fits inside the hook budget + fails the IMAGE BUILD rather than producing a ``/ready`` that times out — the + same posture as :func:`_validate_platform_config_contract`. + """ + where = "contracts/constants.json: microvm_hook_budgets" + for name in ("ready_hook_timeout_seconds", "warmup_total_budget_seconds"): + value = _HOOK_BUDGETS.get(name) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"{where}.{name} must be a positive integer, got {value!r}") + required = _HOOK_BUDGETS.get("warmup_required_timeout_seconds") + if not isinstance(required, int) or isinstance(required, bool) or required <= 0: + raise ValueError( + f"{where}.warmup_required_timeout_seconds must be a positive integer, got {required!r}" + ) + if _HOOK_BUDGETS["warmup_total_budget_seconds"] >= _HOOK_BUDGETS["ready_hook_timeout_seconds"]: + raise ValueError( + f"{where}: warmup_total_budget_seconds " + f"({_HOOK_BUDGETS['warmup_total_budget_seconds']}) must be < " + f"ready_hook_timeout_seconds ({_HOOK_BUDGETS['ready_hook_timeout_seconds']}) — " + "/ready has to answer inside the hook budget." + ) + if required >= _HOOK_BUDGETS["warmup_total_budget_seconds"]: + raise ValueError( + f"{where}: warmup_required_timeout_seconds ({required}) must be < " + f"warmup_total_budget_seconds " + f"({_HOOK_BUDGETS['warmup_total_budget_seconds']}) — the required command " + "must leave the optional ones something to share." + ) + + +_validate_hook_budget_contract() + + +class _PlatformConfigError(Exception): + """A ``platform_config`` block the agent refuses to install (fail closed). + + Carries the wire ``code`` the ``/run`` hook returns, so the handler maps one + exception type onto the two distinct operator remedies: a producer bug / + injection attempt (``…_INVALID``) versus a deployment wiring gap + (``…_INCOMPLETE``). + """ + + def __init__(self, code: str, message: str) -> None: + super().__init__(message) + self.code = code + + +def _install_platform_config(raw: Any) -> list[str]: + """Validate ``platform_config`` against the allowlist and install it into env. + + Returns the sorted env var names actually installed. Rules, all deliberate: + + * ``None`` / absent → install nothing and return ``[]``. This is the P1 + envelope (no ``platform_config`` sibling), where the snapshot's own env is + all there is; a MicroVM image can be launched by an orchestrator that + predates Stage B, and the two deploy on independent cadences. + * present but not an object, or carrying ANY key outside the allowlist, or + carrying a non-string value → reject the run (``…_INVALID``). Unknown keys + are an env-injection attempt, not a compatibility gap (see the allowlist + comment above), so the whole block is refused rather than filtered. + * ``None`` / blank / whitespace-only values are treated as ABSENT, not as an + instruction to clear the variable: the natural TypeScript producer + (``process.env.X ?? ''``) emits an empty string for a resource the + deployment does not have, and clobbering an image value with ``""`` would + turn "not configured over there" into "unconfigured here". + * every required key must survive that filter, else reject + (``…_INCOMPLETE``). An explicitly-sent-but-empty ``{}`` therefore fails — + a producer with nothing to say must omit the key entirely. + + Payload values WIN over pre-existing/image env (see the block comment above: + image env is version-frozen, the payload describes the live deployment). + """ + if raw is None: + return [] + if not isinstance(raw, dict): + raise _PlatformConfigError( + "MICROVM_RUN_PLATFORM_CONFIG_INVALID", + f"platform_config must be an object, got {type(raw).__name__}", + ) + + unknown = sorted(key for key in raw if key not in MICROVM_PLATFORM_CONFIG_ENV_BY_KEY) + if unknown: + raise _PlatformConfigError( + "MICROVM_RUN_PLATFORM_CONFIG_INVALID", + f"platform_config carries key(s) outside the allowlist: {unknown}. " + "These would become process environment variables, so an unrecognised " + "key is refused rather than ignored. Allowed keys: " + f"{sorted(MICROVM_PLATFORM_CONFIG_ENV_BY_KEY)}", + ) + + resolved: dict[str, str] = {} + bad_types: list[str] = [] + for key, value in raw.items(): + if value is None: + continue + if not isinstance(value, str): + bad_types.append(f"{key}:{type(value).__name__}") + continue + if value.strip(): + resolved[key] = value + + if bad_types: + raise _PlatformConfigError( + "MICROVM_RUN_PLATFORM_CONFIG_INVALID", + "platform_config values become environment variables, so they must be " + f"strings; got non-string value(s) for {sorted(bad_types)}", + ) + + missing = sorted(MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS - resolved.keys()) + if missing: + raise _PlatformConfigError( + "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE", + f"platform_config is missing or blank for required key(s): {missing}. " + "The orchestrator must populate these before starting the MicroVM — " + "without them the agent cannot write task status/progress, resolve the " + "GitHub token, or scope its credentials to the tenant.", + ) + + installed: list[str] = [] + for key, value in sorted(resolved.items()): + env_name = MICROVM_PLATFORM_CONFIG_ENV_BY_KEY[key] + os.environ[env_name] = value + installed.append(env_name) + return installed + + +def _aws_silent_log(msg: str, *, tag: str) -> None: + """Emit one log line WITHOUT touching any AWS seam. + + The shared sink for the two hook paths that must stay AWS-silent (the build + hooks, and ``/run`` before ``platform_config`` is installed). Deliberately NOT + ``_debug_cw`` / ``_warn_cw``: those writers spawn a daemon thread that builds a + CloudWatch Logs client whenever ``LOG_GROUP_NAME`` is set, which drags in AWS + credential resolution and — worse — populates ``boto3.DEFAULT_SESSION``, a + module global holding a resolved credential chain plus the region that was + current when it was created. + + Routes through the same ``os.write`` sink and credential redaction as + ``_debug_cw``, so local runs and the ``capfd``-based tests still see the line. + """ + _emit_stdout_line(f"[server/{tag}] {_redact_cached_credentials(msg)}") + + +def _build_hook_log(msg: str) -> None: + """stdout-only log line for the BUILD hooks (``/ready``, ``/validate``). + + A build hook must make ZERO AWS calls. Two structural reasons, on top of + ``_aws_silent_log``'s general one: + + 1. The build role has no Logs grant, so the write can only FAIL — and its + failure bumps the shared ``_debug_cw_failures`` counter, i.e. every image + build would poison the "debug path is blind" signal with a false positive. + 2. ``boto3.DEFAULT_SESSION`` created during ``/ready`` freezes the BUILD + environment's session (credentials + region) into the snapshot, where every + launched MicroVM would inherit it. + + Being AWS-silent by construction rather than by "``LOG_GROUP_NAME`` happens + not to be baked" is what keeps that true if a future image ever bakes it. + """ + _aws_silent_log(msg, tag="build-hook") + + +def _pre_config_log(msg: str) -> None: + """stdout-only log line for the part of ``/run`` that precedes the install. + + Same class of defect as logging from a build hook, one phase later: until + ``_install_platform_config`` has run, ``LOG_GROUP_NAME`` is whatever the + snapshot happens to carry — normally nothing, but a legacy or hand-built image + could bake it, and then a ``_debug_cw`` on this path would resolve credentials + and pin ``boto3.DEFAULT_SESSION`` *before* the orchestrator's own values + (region, ``AWS_SDK_UA_APP_ID``, session role) are in the environment. The one + AWS call this phase is allowed to make is the S3 payload fetch, because + ``platform_config`` is inside the object it fetches. + + Nothing observable is lost. In the intended deployment there is no baked + ``LOG_GROUP_NAME``, so ``_debug_cw`` would have degraded to exactly this + stdout line anyway; and the *reason* for every pre-install rejection also + travels in the structured 4xx/5xx response body, which is what the MicroVM + service surfaces to the operator. + """ + _aws_silent_log(msg, tag="run-pre-config") + + +def _parse_terminate_microvm_id(raw: bytes) -> str: + """Best-effort MicroVM id out of the raw ``/terminate`` body. + + ``/terminate`` must answer 200 for ANY body, so it cannot use a Pydantic body + model: FastAPI validates that *before* the handler runs and answers 422 on + malformed JSON, a wrong content-type, or (for a required model) no body at all + — reporting a hook failure for a teardown that actually succeeded. So the + handler takes the raw request and this function degrades instead of raising: + anything unparseable, non-object, or missing simply yields ``""`` and the hook + still acknowledges. + + **``""`` is the EXPECTED result in production, not a degraded one** (ADR-021 + P2-F8, live 2026-08-07). The service sends a body whose ``microvmId`` is the + empty string — ``{"microvm_id": ""}`` in the guest's own breadcrumb — unlike + ``/run``, where it is populated. So an empty id is normal-and-uninteresting and + is logged without comment; only a body that is genuinely unreadable (not JSON, + not an object) earns a warning, because that would mean the wire contract + changed shape rather than merely omitting a value. + + The consequence for correlation, stated plainly because the hook's original + rationale claimed the opposite: **this hook cannot join the guest record to the + control-plane one.** ``/run``'s "hook accepted task_id=… microvm_id=…" line + carries that join (it always has, and its id IS populated); ``/terminate``'s + value is the pipeline-state snapshot it reports, not the id. + + Accepts both spellings of the field (``microvmId`` is the service's camelCase + wire name; ``microvm_id`` is tolerated because it costs one ``or``). The id is + used only as a log/response correlation string, never as an authorization or + lookup key, so degrading to empty has no security consequence. + """ + if not raw.strip(): + return "" + try: + parsed = json.loads(raw) + except (json.JSONDecodeError, UnicodeDecodeError) as exc: + # Not silent: an unreadable body is worth a breadcrumb even though it + # cannot change the outcome. + _emit_stdout_line(f"[server/warn] /terminate hook body is not JSON ({exc}); ignoring it") + return "" + if not isinstance(parsed, dict): + _emit_stdout_line( + f"[server/warn] /terminate hook body is {type(parsed).__name__}, " + "expected an object; ignoring it" + ) + return "" + candidate = parsed.get("microvmId") or parsed.get("microvm_id") or "" + return candidate if isinstance(candidate, str) else "" + class MicrovmRunHookRequest(BaseModel): """Body the MicroVM service POSTs to the ``/run`` hook. @@ -853,12 +1190,19 @@ class MicrovmRunHookRequest(BaseModel): string (``lambda-microvm-strategy.ts``) is one of two shapes, mirroring the ECS container env contract (``AGENT_PAYLOAD`` / ``AGENT_PAYLOAD_S3_URI``): - * ``{"agent_payload": {...}}`` — the whole orchestrator payload, inline. - * ``{"agent_payload_s3_uri": "s3://bucket/key"}`` — a pointer to it. + * ``{"agent_payload": {...}, "platform_config": {...}}`` — inline. + * ``{"agent_payload_s3_uri": "s3://bucket/key", "platform_config": {...}}`` — + a pointer; the object at the URI carries the task payload (and a + ``platform_config`` copy, so either end of the fetch yields it). The pointer form is the DOMINANT one: the service caps ``runHookPayload`` at 4 096 bytes and a hydrated payload is essentially always larger. + ``platform_config`` (ADR-021 P2) is a SIBLING of ``agent_payload``, not a + field inside it: it configures the agent's *process*, whereas + ``agent_payload`` describes the *task* (``memory_id`` and friends stay + inside ``agent_payload``, unchanged). See ``_install_platform_config``. + Both fields default to empty so a malformed call produces this module's structured 400 rather than FastAPI's 422 — the service surfaces a 4xx as a generic "client error" hook failure either way, and our own body is what ends @@ -877,16 +1221,27 @@ def _fetch_microvm_payload_from_s3(uri: str) -> dict: payload bucket. Errors propagate to the caller, which turns them into a structured 400/500 — silently starting a pipeline with no payload would produce a task that runs with an empty prompt. + + Built through ``aws_session.platform_client`` so the call carries the ABCA + ``md/`` solution-attribution segment (#319). Platform, not tenant: the bucket + is platform-owned and — decisively — this is the ONE call that must happen + BEFORE ``platform_config`` is installed (the config is inside the object + being fetched), so ``AGENT_SESSION_ROLE_ARN`` may not be set yet and a + tenant-scoped client could not be built. ``platform_client`` does not touch + the cached session, so this call also cannot pin an unscoped session for the + rest of the task. The ``app/`` UA segment (native, from ``AWS_SDK_UA_APP_ID``) + is the one attribution field this single call can miss for the same + chicken-and-egg reason. """ remainder = uri[len(_S3_URI_SCHEME) :] bucket, _, key = remainder.partition("/") if not bucket or not key: raise ValueError(f"agent_payload_s3_uri is not a bucket/key URI: {uri!r}") - import boto3 + from aws_session import platform_client region = os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") - client = boto3.client("s3", region_name=region) + client = platform_client("s3", region_name=region) body = client.get_object(Bucket=bucket, Key=key)["Body"].read() payload = json.loads(body) if not isinstance(payload, dict): @@ -894,8 +1249,12 @@ def _fetch_microvm_payload_from_s3(uri: str) -> dict: return payload -def _resolve_microvm_run_payload(run_hook_payload: str) -> dict: - """Turn the ``runHookPayload`` string into the orchestrator payload dict. +def _resolve_microvm_run_payload(run_hook_payload: str) -> tuple[dict, Any]: + """Split the ``runHookPayload`` string into (agent payload, platform config). + + The second element is returned RAW (unvalidated) — ``_install_platform_config`` + owns its allowlist checks so the two failure classes get distinct wire codes. + ``None`` means the envelope carried no ``platform_config`` at all. Raises ``ValueError`` for every shape the agent cannot act on, so the caller has exactly one failure branch to map onto a 400. @@ -915,11 +1274,35 @@ def _resolve_microvm_run_payload(run_hook_payload: str) -> dict: if inline is not None: if not isinstance(inline, dict): raise ValueError(f"agent_payload must be an object, got {type(inline).__name__}") - return inline + return inline, envelope.get("platform_config") uri = envelope.get("agent_payload_s3_uri") if isinstance(uri, str) and uri.startswith(_S3_URI_SCHEME): - return _fetch_microvm_payload_from_s3(uri) + fetched = _fetch_microvm_payload_from_s3(uri) + # ``platform_config`` may sit beside the pointer (outer envelope) or + # inside the fetched object — the producer writes it in BOTH places on + # this path deliberately, so the agent gets it whichever end it reads. + # Inner first, outer as the fallback. + platform_config = fetched.get("platform_config") + if platform_config is None: + platform_config = envelope.get("platform_config") + # The fetched object is EITHER the same envelope shape as the inline form + # ({"agent_payload": …}) or the task payload itself with ``platform_config`` + # merged in at the top level (what the strategy writes today, and what P1 + # wrote without the config). Both are accepted because the image snapshot + # and the orchestrator Lambda deploy on independent cadences — a new image + # must not require a same-instant orchestrator. Discriminating on the + # ``agent_payload`` key is unambiguous: no orchestrator task payload has a + # field by that name. A stray ``platform_config`` key left in the bare + # form is inert — ``_extract_invocation_params`` reads named fields only. + nested = fetched.get("agent_payload") + if nested is None: + return fetched, platform_config + if not isinstance(nested, dict): + raise ValueError( + f"agent_payload in the S3 payload must be an object, got {type(nested).__name__}" + ) + return nested, platform_config if uri is not None: raise ValueError(f"agent_payload_s3_uri must be an s3:// URI, got {uri!r}") @@ -929,6 +1312,165 @@ def _resolve_microvm_run_payload(run_hook_payload: str) -> dict: ) +#: The ONE executable whose warm-up gates the snapshot, exec'd FIRST. +#: +#: WHY THIS EXISTS (ADR-021 P2-F5, live 2026-08-07). ``/ready`` used to answer 200 +#: the moment uvicorn was bound, and its own docstring said the point of the hook +#: was that "the snapshot is taken with a warm server". The snapshot was warm for +#: uvicorn and stone cold for the binary that does all the work: ``claude`` is a +#: **225 MiB (236,305,136-byte) statically-linked ELF** whose pages had never been +#: touched when the snapshot was captured. On a guest restored from that snapshot, +#: the FIRST ``exec`` of it has to fault those pages in from lazily-restored +#: storage — and ``runner.py``'s version probe timed out at 10 s, failing EVERY +#: task at turn 0, reproducibly, while the same binary in the same image answers +#: in under a second locally. Exec'ing it here means those pages are resident when +#: the snapshot is taken, so every MicroVM cloned from it inherits them warm. +#: +#: It is REQUIRED (a failure answers 503 and ultimately fails the image build) +#: because a snapshot that cannot exec ``claude`` cannot run a single task, and it +#: goes FIRST so no best-effort warm-up can eat the budget it needs. +#: +#: ``--version`` is the cheapest argv that still exec's the real binary: it touches +#: no network, writes nothing, and needs no credentials — which is what keeps +#: ``/ready`` AWS-silent (see ``_build_hook_log``). +_READY_WARMUP_REQUIRED: tuple[str, ...] = ("claude", "--version") + +#: Best-effort warm-ups, exec'd AFTER the required one and only with what is left +#: of :data:`_READY_WARMUP_TOTAL_BUDGET_SECONDS`. +#: +#: Same mechanism, no measured problem: neither was observed to blow a timeout, and +#: a snapshot missing them is still a snapshot that can start a task — so neither +#: may fail a build, and neither may delay the 200 that a successful required +#: warm-up has already earned. Nothing is added here on speculation: every entry +#: costs build time and, more importantly, snapshot memory. +_READY_WARMUP_OPTIONAL: tuple[tuple[str, ...], ...] = ( + ("git", "--version"), + ("node", "--version"), +) + +#: Budget for the REQUIRED warm-up alone, in seconds. +#: +#: Deliberately generous, and generosity is nearly free: the hook runs once per +#: image build (twice per image — one build per chipset). A cold 225 MiB ``exec`` +#: is exactly the operation whose duration nobody here can predict, which is the +#: whole lesson of P2-F5, where a tight bound on a version probe cost every task. +#: +#: Contract-sourced (see :data:`_HOOK_BUDGETS`) so that it, the total ceiling and +#: the CDK hook budget are one edit rather than three. +_READY_WARMUP_REQUIRED_TIMEOUT_SECONDS: int = _HOOK_BUDGETS["warmup_required_timeout_seconds"] + +#: Ceiling for the WHOLE warm-up (required + every optional), in seconds. +#: +#: The hook's own budget (``READY_HOOK_TIMEOUT_SECONDS`` in +#: ``cdk/src/constructs/lambda-microvm-compute.ts``) comes from the SAME contract +#: block, and this ceiling must stay strictly below it — ``/ready`` has to answer +#: inside the hook budget, and the margin covers uvicorn scheduling plus the +#: request itself. That margin is the point: per-command budgets do NOT compose — +#: three commands at 120 s each is 360 s, which would blow a 300 s hook budget and +#: turn a warm-up meant to prevent a runtime failure into a build failure. So the +#: required command takes its own budget and the optional ones SHARE whatever is +#: left of this ceiling, meaning the warm-up's worst case is bounded by one number +#: that can be compared against the hook budget by eye — and by +#: ``scripts/check-constants-sync.ts``, which rejects a contract where it is not. +_READY_WARMUP_TOTAL_BUDGET_SECONDS: int = _HOOK_BUDGETS["warmup_total_budget_seconds"] + +#: Below this many seconds of remaining budget an optional warm-up is skipped +#: rather than started: a sub-second timeout cannot warm a large binary, it can +#: only manufacture a scary log line. +_READY_WARMUP_MIN_TIMEOUT_SECONDS = 1 + + +def _warm_one_binary(argv: tuple[str, ...], timeout: float) -> str | None: + """Exec ``argv`` once, bounded by ``timeout``. Returns a failure tag or ``None``. + + Every failure mode of ``subprocess.run`` is caught here rather than propagating + — a warm-up defect must produce the hook's own honest 503, not a FastAPI 500 + that reports a hook failure with no explanation in the build log. The caller + decides whether the returned tag is fatal. + """ + name = argv[0] + started = _time_for_debug.monotonic() + try: + # Fixed argv from the module constants above, `shell=False`, and no user- or + # payload-derived input anywhere on this path (`/ready` takes no request + # body at all) — so there is no injection surface here. + completed = subprocess.run( + list(argv), + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except Exception as exc: + # FileNotFoundError (not on PATH), TimeoutExpired, PermissionError, … + elapsed = _time_for_debug.monotonic() - started + _build_hook_log(f"/ready hook: warm-up of {name!r} FAILED after {elapsed:.1f}s: {exc!r}") + return f"{name}:{type(exc).__name__}" + + elapsed = _time_for_debug.monotonic() - started + # Version strings only — no repo content, no credentials — and the first line + # is enough to identify the binary that was warmed. + detail = (completed.stdout or completed.stderr or "").strip().splitlines() + version = detail[0] if detail else "" + if completed.returncode != 0: + _build_hook_log( + f"/ready hook: warm-up of {name!r} exited {completed.returncode} " + f"after {elapsed:.1f}s ({version!r})" + ) + return f"{name}:exit{completed.returncode}" + + _build_hook_log(f"/ready hook: warmed {name!r} in {elapsed:.1f}s (version={version!r})") + return None + + +def _warm_snapshot_binaries() -> list[str]: + """Warm the snapshot's heavyweight binaries; return the REQUIRED failures. + + An empty list means the snapshot is warm enough to capture. Best-effort + failures are logged and never appear in the return value. + + Ordering and budgeting are the contract, not an implementation detail: + + * the REQUIRED command runs FIRST, with its own + :data:`_READY_WARMUP_REQUIRED_TIMEOUT_SECONDS`, so no best-effort warm-up can + starve the one that decides whether the snapshot is usable; + * if it fails, the optional ones are SKIPPED — the image build is already going + to fail, so spending more of the hook budget warming ``git`` buys nothing and + delays the 503 the service is waiting for; + * the optional ones then SHARE what is left of + :data:`_READY_WARMUP_TOTAL_BUDGET_SECONDS`, each bounded by the remaining + budget, so a hung optional command can delay the 200 by at most the remainder + and can never prevent it. Once the remainder falls below + :data:`_READY_WARMUP_MIN_TIMEOUT_SECONDS` the rest are skipped with a log line. + + Makes ZERO AWS calls and no network calls: ``--version`` execs plus stdout + lines, so ``/ready``'s AWS-silence property is intact. + """ + deadline = _time_for_debug.monotonic() + _READY_WARMUP_TOTAL_BUDGET_SECONDS + + failure = _warm_one_binary(_READY_WARMUP_REQUIRED, _READY_WARMUP_REQUIRED_TIMEOUT_SECONDS) + if failure is not None: + _build_hook_log( + f"/ready hook: skipping best-effort warm-ups after {_READY_WARMUP_REQUIRED[0]!r} " + "failed — the image build cannot succeed, so the 503 should not wait" + ) + return [failure] + + for argv in _READY_WARMUP_OPTIONAL: + remaining = deadline - _time_for_debug.monotonic() + if remaining < _READY_WARMUP_MIN_TIMEOUT_SECONDS: + _build_hook_log( + f"/ready hook: skipping best-effort warm-up of {argv[0]!r} — " + f"{remaining:.1f}s left of the {_READY_WARMUP_TOTAL_BUDGET_SECONDS}s warm-up " + "ceiling. The required warm-up already succeeded, so this does not " + "block the snapshot." + ) + continue + _warm_one_binary(argv, remaining) + + return [] + + @app.post(f"{MICROVM_HOOK_PREFIX}/ready") def microvm_ready(): """MicroVM image ``/ready`` build hook — "the application has initialised". @@ -938,22 +1480,264 @@ def microvm_ready(): hook enabled and nothing serving it, both chipset builds fail with "Ready hook check failed: the application returned a client error (HTTP 4xx) response". - Reaching this handler already proves everything P1 needs: uvicorn is bound on - the hook port and ``server`` imported cleanly (which pulls in ``pipeline`` → - ``runner`` → the policy engine, so a missing policy file or a broken import - fails the BUILD instead of the first task). Deeper warm-up assertions — - Bedrock reachability, Memory access, tool availability — belong to P2's - ``/validate``, which is deliberately still not declared: a hook that 404s or - reports failure fails every image build. - - Declared ``def`` rather than ``async def`` on purpose: Starlette runs sync - handlers in a threadpool, so this never competes with the event loop that has - to keep ``GET /ping`` fast. + Reaching this handler already proves everything the ORIGINAL ``/ready`` + contract needed: uvicorn is bound on the hook port and ``server`` imported + cleanly (which pulls in ``pipeline`` → ``runner`` → the policy engine, so a + missing policy file or a broken import fails the BUILD instead of the first + task). The *shape* checks — hook routes registered, interpreter and contract + sanity — belong to ``/validate``, which reports them individually. + + **It then WARMS the snapshot** (:func:`_warm_snapshot_binaries`, ADR-021 + P2-F5). This is the hook's second job and the reason it can now take seconds + rather than microseconds: the snapshot is only as warm as the pages that were + touched before it was captured, and the 225 MiB ``claude`` binary was never + among them — which failed every P2 smoke task at turn 0. A **required** + warm-up failure returns **503**, the hook contract's "not ready yet" signal, + so the service keeps asking within the ``/ready`` budget and — if it never + clears — fails the image build. Failing the build is right: a snapshot that + cannot exec ``claude`` cannot run a single task, and discovering that at build + time costs one build instead of every task. + + The whole warm-up is bounded by + :data:`_READY_WARMUP_TOTAL_BUDGET_SECONDS`, which the shared contract keeps + strictly inside the hook's own budget with margin to spare — the required + command takes its own share and the best-effort ones split the remainder, so + this handler cannot talk itself past the deadline the service is holding it to. + + Makes ZERO AWS calls, including its own logging (``_build_hook_log``): this + runs under the build role, and a client built here would freeze a build-time + boto3 session into the snapshot. A ``--version`` subprocess is not an AWS call + and touches no network, so the warm-up does not weaken that property. + + Declared ``def`` rather than ``async def`` on purpose, and now load-bearing + rather than stylistic: Starlette runs sync handlers in a threadpool, so a + warm-up that blocks for seconds never competes with the event loop that has to + keep ``GET /ping`` fast. """ - _debug_cw("/ready hook: server is up, reporting ready for snapshot") + _build_hook_log("/ready hook: server is up, warming the snapshot before it is taken") + failures = _warm_snapshot_binaries() + if failures: + _build_hook_log( + f"/ready hook: NOT ready — required warm-up failed for {failures}. " + "The snapshot would launch MicroVMs that cannot exec these binaries." + ) + return JSONResponse( + status_code=503, + content={"status": "not_ready", "failed_warmups": failures}, + ) + _build_hook_log("/ready hook: reporting ready for snapshot") return {"status": "ready"} +#: Env var names that must never be baked into a MicroVM snapshot (ADR-021 +#: sub-decision 3: "the image build shall not embed secrets, tokens, or per-task +#: identity in the snapshot"). ``/validate`` REPORTS these rather than failing on +#: them — see ``microvm_validate``. +_SNAPSHOT_FORBIDDEN_SECRET_ENV = ( + "GITHUB_TOKEN", + "GH_TOKEN", + "LINEAR_API_TOKEN", + "JIRA_API_TOKEN", + "JIRA_APP_ACTOR_SHARED_SECRET", + "ANTHROPIC_API_KEY", + "AWS_SECRET_ACCESS_KEY", + "AWS_SESSION_TOKEN", +) + +#: Interpreter floor, mirroring ``requires-python`` in ``agent/pyproject.toml``. +_MIN_PYTHON_VERSION = (3, 13) + +# Set once, as the LAST statement of this module. ``/validate`` reports 503 until +# then: it is the only honest "still initialising" signal available to a build +# hook. Unreachable in practice today (uvicorn accepts no request until the import +# completes) — it is a tripwire for the refactor that moves warm-up work behind the +# bind, at which point the flag is the difference between a 503 and a snapshot +# reported valid while still initialising. See ``microvm_validate``. +_module_initialized = False + + +@app.post(f"{MICROVM_HOOK_PREFIX}/validate") +def microvm_validate(): + """MicroVM image ``/validate`` build hook — shallow snapshot self-check. + + **THIS HOOK MAKES ZERO AWS API CALLS, AND MUST KEEP MAKING ZERO.** It runs + during ``CreateMicrovmImage`` under the **build role**, which deliberately + holds no Bedrock / Secrets Manager / DynamoDB grants (ADR-021 sub-decision 4: + the build path's only extra privilege is port 80 egress for ``apt-get``). So + the "deeper warm-up assertions" ADR-021 originally sketched for this hook — + Bedrock reachability, Memory access, tool availability — are NOT implementable + here: every one of them would AccessDenied and fail every image build. They + belong to the first task's own error handling, not to a build hook. + + The one member of that list that turned out to be *partly* implementable landed + on ``/ready`` instead, and for a different reason: ``/ready`` warms (and so + incidentally proves the execability of) the local ``claude`` binary, because a + cold 225 MiB ELF in the snapshot killed every task at turn 0 (ADR-021 P2-F5). + That is a local ``exec``, not an AWS call, so it violates nothing above — but it + belongs to the hook whose 200 gates the snapshot, not to the hook that reports + check results. + + It must also not touch credential resolution: ``platform_config`` has not + arrived yet (it comes with ``/run``), and any client built here would leave a + resolved boto3 session — with the build role's credentials and the build + region — frozen in the snapshot for every MicroVM launched from it. Hence + ``_build_hook_log`` instead of ``_debug_cw``, and no import of + ``aws_session``. + + What it CAN prove, all in-process: + + * the server is alive and this route is reachable at all; + * every hook route the image declares is registered (a typo'd prefix would + otherwise surface as a failed lifecycle transition on the first real task); + * interpreter floor, and that the cross-package ``platform_config`` contract + loaded — the thing ``/run`` will validate the payload against. + + Returns 200 with the individual check results. Returns **503** while the + module has not finished initialising, per the hook contract's + still-initialising semantics; a permanently failing check therefore fails the + image build, which is the correct outcome for a genuinely broken snapshot. + + **The 503 branch is a REFACTOR TRIPWIRE, not a reachable state today.** + ``_module_initialized`` is set as the last statement of this module and uvicorn + does not accept a request until the import completes, so under the current + import-time-only initialisation the check cannot be observed False from a real + hook call. It is kept because that is a property of *how the module happens to + initialise*, not of the contract: the moment anyone moves warm-up work behind + the bind — a lifespan startup task, a lazily-built cache, a thread the first + request has to wait on — the honest answer becomes 503, and a hook that had + only a 200 path would report a broken snapshot as a valid one. Cheap to keep, + silently wrong to delete. + + Baked-secret detection is REPORT-ONLY (``warnings``): the build environment's + own credentials may legitimately be in this process's env, so failing here + would fail every build. Names only — never values. + """ + expected_routes = { + f"{MICROVM_HOOK_PREFIX}/{hook}" for hook in ("ready", "validate", "run", "terminate") + } + registered = {getattr(route, "path", None) for route in app.routes} + missing_routes = sorted(expected_routes - registered) + + checks = { + "server_initialized": _module_initialized, + "hook_routes_registered": not missing_routes, + "python_version_supported": sys.version_info[:2] >= _MIN_PYTHON_VERSION, + "platform_config_contract_loaded": bool(MICROVM_PLATFORM_CONFIG_ENV_BY_KEY), + } + failed = sorted(name for name, ok in checks.items() if not ok) + + baked_secrets = [name for name in _SNAPSHOT_FORBIDDEN_SECRET_ENV if os.environ.get(name)] + + body: dict[str, Any] = { + "status": "valid" if not failed else "not_ready", + "checks": checks, + "python_version": ".".join(str(part) for part in sys.version_info[:3]), + "hook_prefix": MICROVM_HOOK_PREFIX, + "platform_config_keys": len(MICROVM_PLATFORM_CONFIG_ENV_BY_KEY), + "warnings": [f"secret_env_present_in_snapshot:{name}" for name in baked_secrets], + } + if missing_routes: + body["missing_routes"] = missing_routes + + if failed: + body["failed_checks"] = failed + _build_hook_log(f"/validate hook: NOT ready, failed checks={failed}") + return JSONResponse(status_code=503, content=body) + + _build_hook_log( + f"/validate hook: ok (python={body['python_version']}, " + f"platform_config_keys={body['platform_config_keys']}, warnings={len(baked_secrets)})" + ) + return body + + +@app.post(f"{MICROVM_HOOK_PREFIX}/terminate") +async def microvm_terminate(request: Request): + """MicroVM ``/terminate`` runtime hook — best-effort flush, always 200. + + Called as the MicroVM is torn down. Three hard constraints: + + * **It must not write terminal task status.** The orchestrator owns terminal + state: it finalizes the task and THEN calls ``TerminateMicrovm``, so a + terminate hook that wrote ``FAILED``/``COMPLETED`` would race the + finalization it follows and could clobber the real outcome with a + substrate-shutdown artifact. The pipeline thread's own crash path + (``_run_task_background``) remains the only in-guest terminal writer. + * **It must return 200 inside the hook budget, even with nothing running.** + So it never joins the pipeline thread (a drain could take minutes — that is + ``lifespan``'s job on a graceful shutdown) and every best-effort step is + wrapped: a failure here must not turn a clean teardown into a hook failure. + * **It must return 200 for any BODY too.** That is why this handler takes the + raw ``Request`` instead of a Pydantic body model: FastAPI validates a typed + body BEFORE the handler runs, so malformed JSON, a wrong content-type, or a + missing body would produce a 422 this function never gets a chance to + prevent — a reported hook failure on a successful teardown. Parsing is + deferred to ``_parse_terminate_microvm_id``, which degrades to ``""``. + + ``async def`` (unlike ``/ready`` and ``/run``) because reading the raw body + requires awaiting it. Safe on the event loop: the work is a JSON parse, a + thread-count read and a fire-and-forget log — no blocking AWS call. + + On flushing: there is nothing buffered to flush. ``_ProgressWriter`` performs + a synchronous DynamoDB ``put_item`` per event, and ``task_state`` writes + inline, so every progress/status write is already durable at call time — this + hook has no queue to drain, which is why it is a log-and-acknowledge rather + than a flush loop. (ADR-021 sub-decision 2's "flush progress events before + returning 200" applies to ``/suspend`` in P3 for the same reason: durability + is per-write, so the hook only has to observe it.) + """ + raw = b"" + try: + raw = await request.body() + except Exception as exc: + # A truncated/aborted body must not become a 5xx: the VM is going away and + # the id is only a correlation string. Logged, not swallowed. + _emit_stdout_line(f"[server/warn] /terminate hook could not read its body: {exc!r}") + microvm_id = _parse_terminate_microvm_id(raw) + + # `None`, not `0`: if the thread-count read below raises, "we could not tell" + # must not be reported as the confident "nothing was running" — the one reading + # an operator would use to conclude a clean teardown. + active: int | None = None + try: + with _threads_lock: + active = sum(1 for t in _active_threads if t.is_alive()) + # Final structured line. Fire-and-forget CloudWatch (LOG_GROUP_NAME is + # present at runtime when the orchestrator delivered it via /run's + # platform_config; on the legacy no-config path `_debug_cw` degrades to + # stdout) so a terminated MicroVM leaves a last breadcrumb in the task's log + # group; stdout regardless. + # + # `microvm_id` is normally `""` here — that is what the service sends on + # this hook (P2-F8, see `_parse_terminate_microvm_id`), not a parse + # failure. The load-bearing fields are the pipeline-state ones. + _debug_cw( + "/terminate hook: " + + json.dumps( + { + "event": "microvm_terminate", + "microvm_id": microvm_id, + "active_pipeline_threads": active, + "background_pipeline_failed": _background_pipeline_failed, + "timestamp": datetime.now(UTC).isoformat(), + }, + sort_keys=True, + ) + ) + except Exception as exc: + # Best-effort by contract: the VM is going away either way, and a 5xx + # here would report a hook failure for a teardown that succeeded. Logged + # (not swallowed) so the failure is still findable. + _emit_stdout_line(f"[server/warn] /terminate hook best-effort step failed: {exc!r}") + + return { + "status": "acknowledged", + "microvm_id": microvm_id, + "active_pipeline_threads": active, + "timestamp": datetime.now(UTC).isoformat(), + } + + @app.post(f"{MICROVM_HOOK_PREFIX}/run") def microvm_run(request: Request, body: MicrovmRunHookRequest): """MicroVM ``/run`` lifecycle hook — accept the task and start the pipeline. @@ -966,6 +1750,14 @@ def microvm_run(request: Request, body: MicrovmRunHookRequest): substrates (AgentCore receives it as ``input``, ECS as ``AGENT_PAYLOAD``, MicroVMs inside this envelope), which is what makes that reuse correct. + ``platform_config`` (P2) is installed into ``os.environ`` FIRST — before + ``_extract_invocation_params``, which calls ``resolve_github_token()`` and so + reads ``GITHUB_TOKEN_SECRET_ARN``, and before any pipeline/credential + initialisation that reads ``AGENT_SESSION_ROLE_ARN`` or the table names. + Installing after that point would resolve the whole task against the + snapshot's version-frozen env and silently ignore the values the orchestrator + sent. + Session/workload headers are absent here (there is no AgentCore Runtime in front of this call), so ``_extract_invocation_params`` resolves an empty ``session_id`` / workload token — the same posture the ECS backend already @@ -974,15 +1766,24 @@ def microvm_run(request: Request, body: MicrovmRunHookRequest): Sync ``def`` for the same reason as ``/ready``, and additionally because the S3 payload fetch is a blocking boto3 call: in a threadpool it cannot stall the event loop. + + **Every log line before the install goes through ``_pre_config_log``** (stdout + only). Until ``platform_config`` is in the environment, a ``_debug_cw`` here + would resolve AWS credentials and pin ``boto3.DEFAULT_SESSION`` off whatever + the snapshot happens to carry — the same defect the build hooks avoid, one + phase later. The single AWS call this phase is allowed to make is the S3 + payload fetch, because the config is inside the object being fetched. """ - _debug_cw(f"/run hook received: microvm_id={body.microvmId!r} bytes={len(body.runHookPayload)}") + _pre_config_log( + f"/run hook received: microvm_id={body.microvmId!r} bytes={len(body.runHookPayload)}" + ) try: - payload = _resolve_microvm_run_payload(body.runHookPayload) + payload, platform_config = _resolve_microvm_run_payload(body.runHookPayload) except ValueError as exc: # Bad envelope — the orchestrator built something this agent cannot act # on. 400 (not 500) because retrying an identical body cannot help. - _warn_cw(f"/run hook rejected: {exc}") + _pre_config_log(f"/run hook rejected: {exc}") return JSONResponse( status_code=400, content={ @@ -993,8 +1794,12 @@ def microvm_run(request: Request, body: MicrovmRunHookRequest): except Exception as exc: # Payload fetch failed (S3 AccessDenied / NoSuchKey / transient). 500 so # the failure is distinguishable from a malformed body, and loud enough to - # find in the MicroVM log group. - _debug_cw_exc("/run hook payload fetch FAILED", exc) + # find in the MicroVM log group — via the response body, since the + # CloudWatch writer is off-limits until the config is installed. + _pre_config_log( + f"/run hook payload fetch FAILED [{type(exc).__name__}: {exc}]\n" + f"{traceback.format_exc()}" + ) return JSONResponse( status_code=500, content={ @@ -1004,6 +1809,40 @@ def microvm_run(request: Request, body: MicrovmRunHookRequest): ) task_id_log = str(payload.get("task_id", "")) + + try: + installed_env = _install_platform_config(platform_config) + except _PlatformConfigError as exc: + _pre_config_log(f"/run hook rejected: {exc}") + return JSONResponse( + status_code=400, + content={"code": exc.code, "message": str(exc)}, + ) + + if installed_env: + # Names only: the values are non-secret identifiers, but the list is what + # an operator needs to see when the agent behaves as if a table or bucket + # were missing. Emitted AFTER the install so it can reach the log group + # the payload just named. + _debug_cw( + f"/run hook installed platform_config env: {installed_env}", + task_id=task_id_log or None, + ) + else: + # STILL pre-install: nothing was installed, so this branch has exactly the + # rights the lines above it had — stdout only. A `_warn_cw` here would spawn + # the CloudWatch writer thread and pin `boto3.DEFAULT_SESSION` off the + # snapshot's baked env, which is the very defect this compatibility branch is + # reporting. The warning is not lost: on the intended deployment (no baked + # `LOG_GROUP_NAME`) `_warn_cw` would have degraded to this same stdout line, + # and on a legacy image the log group would be the wrong one anyway. + _pre_config_log( + "/run hook received no platform_config; running on the image snapshot's " + "own environment, which is frozen at build time. Expected only from an " + "orchestrator that predates ADR-021 P2." + + (f" task_id={task_id_log!r}" if task_id_log else "") + ) + try: params = _extract_invocation_params(payload, request) except Exception as exc: @@ -1032,10 +1871,22 @@ def microvm_run(request: Request, body: MicrovmRunHookRequest): _spawn_background(params) task_id = params["task_id"] - _debug_cw(f"/run hook accepted task_id={task_id!r}", task_id=task_id or None) + # Carries microvm_id as well as task_id: the "/run hook received" line that + # used to correlate the two is stdout-only now (pre-install), so this is the + # first line that reaches the task's log group and it has to join the CloudWatch + # record to the control-plane one on its own. + _debug_cw( + f"/run hook accepted task_id={task_id!r} microvm_id={body.microvmId!r}", + task_id=task_id or None, + ) return { "status": "accepted", "task_id": task_id, "microvm_id": body.microvmId, "timestamp": datetime.now(UTC).isoformat(), } + + +# LAST statement in the module, on purpose: ``/validate`` reports 503 until this +# flips, so the flag means "import completed" rather than "the flag exists". +_module_initialized = True diff --git a/agent/tests/test_runner.py b/agent/tests/test_runner.py index 1462764dd..4b5b6a13d 100644 --- a/agent/tests/test_runner.py +++ b/agent/tests/test_runner.py @@ -9,14 +9,21 @@ from __future__ import annotations +import asyncio +import subprocess from typing import Any from unittest.mock import MagicMock, patch +import pytest + +import runner from models import TaskConfig from runner import ( + _CLAUDE_VERSION_PROBE_TIMEOUT_S, _DISALLOWED_TOOLS, _FULL_TOOL_SURFACE, _initialize_policy_engine_and_hooks, + _log_claude_cli_version, _resolve_allowed_tools, _resolve_setting_sources, _setup_agent_env, @@ -420,3 +427,111 @@ def test_config_default_haiku_model_is_an_inference_profile(self): # The platform default (no override) must be a us.* profile, never a bare # foundation-model id — the whole point of the fix. assert _config().haiku_model.startswith("us.") + + +class TestClaudeCliVersionProbe: + """The diagnostic probe that killed every MicroVM task at turn 0 (P2-F5). + + ``agent/src/runner.py`` used ``timeout=10`` on ``claude --version``. On the + ``lambda-microvm`` backend that failed EVERY task, reproducibly: + + TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds + + The binary answers in under a second in the same image locally; it is a 225 MiB + statically linked ELF whose pages had never been touched before the snapshot was + captured, so the first ``exec`` on a restored guest had to fault them all in. + The primary fix is warming it in ``/ready`` (``server._warm_snapshot_binaries``); + this bound is the backstop, and the point of these tests is that it stays loose. + """ + + def test_the_version_probe_bound_is_loose_enough_for_a_cold_start(self): + # A version string for a log line: nothing branches on it, and the failure + # mode is a dead task. Any cold-start environment must fit inside it. + assert _CLAUDE_VERSION_PROBE_TIMEOUT_S >= 60 + + def test_the_probe_passes_that_timeout_to_the_version_call_only(self, monkeypatch): + calls: list[tuple[list[str], float]] = [] + + def fake_run(argv, **kwargs): + calls.append((list(argv), kwargs["timeout"])) + return MagicMock(returncode=0, stdout="/usr/bin/claude\n") + + monkeypatch.setattr(runner.subprocess, "run", fake_run) + _log_claude_cli_version() + + assert calls[0][0] == ["which", "claude"] + assert calls[1][0] == ["claude", "--version"] + # The PATH lookup keeps its tight bound — it touches none of the 225 MiB — + # while the exec that DOES gets the loose one. Sharing one number would + # either loosen a lookup that cannot hang or re-tighten the exec that can. + assert calls[1][1] == _CLAUDE_VERSION_PROBE_TIMEOUT_S + assert calls[0][1] < calls[1][1] + + def test_a_missing_cli_warns_and_skips_the_version_call(self, monkeypatch): + calls: list[list[str]] = [] + + def fake_run(argv, **_kwargs): + calls.append(list(argv)) + return MagicMock(returncode=1, stdout="") + + monkeypatch.setattr(runner.subprocess, "run", fake_run) + _log_claude_cli_version() + # No point exec'ing a binary `which` could not find — and no exception: + # this is diagnostics, so it must never be the thing that fails a task. + assert calls == [["which", "claude"]] + + @pytest.mark.parametrize( + "error", + [ + subprocess.TimeoutExpired(["claude", "--version"], 60), + FileNotFoundError(2, "No such file or directory", "claude"), + PermissionError(13, "Permission denied", "claude"), + ], + ids=["timeout", "missing-binary", "not-executable"], + ) + def test_a_failing_version_call_warns_instead_of_killing_the_task( + self, monkeypatch, capfd, error + ): + # The other half of P2-F5. A looser timeout makes the 10 s failure unlikely, + # not impossible — and a probe whose entire output is a log line must not be + # able to end a task at turn 0 no matter how it fails. + def fake_run(argv, **_kwargs): + if argv[0] == "which": + return MagicMock(returncode=0, stdout="/usr/bin/claude\n") + raise error + + monkeypatch.setattr(runner.subprocess, "run", fake_run) + + _log_claude_cli_version() # must not raise + + out = capfd.readouterr().out + assert "claude CLI version probe failed (non-fatal)" in out + assert type(error).__name__ in out + # Degraded, not silent — and NOT reported as a successful probe. + assert "claude CLI:" not in out + + def test_a_failing_path_lookup_also_only_warns(self, monkeypatch, capfd): + # `which` itself can be absent on a minimal image; same rule applies. + def fake_run(_argv, **_kwargs): + raise FileNotFoundError(2, "No such file or directory", "which") + + monkeypatch.setattr(runner.subprocess, "run", fake_run) + + _log_claude_cli_version() + + assert "claude CLI version probe failed (non-fatal)" in capfd.readouterr().out + + def test_run_agent_invokes_the_cli_version_probe(self): + sentinel = RuntimeError("stop after version probe") + with ( + patch.object(runner, "_setup_agent_env"), + patch.object(runner, "_log_claude_cli_version", side_effect=sentinel) as probe, + ): + try: + asyncio.run(runner.run_agent("prompt", "system", _config())) + except RuntimeError as error: + assert error is sentinel + else: + raise AssertionError("run_agent continued past the version probe") + + probe.assert_called_once_with() diff --git a/agent/tests/test_server.py b/agent/tests/test_server.py index dcec9de59..51a094bf7 100644 --- a/agent/tests/test_server.py +++ b/agent/tests/test_server.py @@ -3,8 +3,13 @@ from __future__ import annotations import json +import os +import subprocess +import sys import threading import time +from pathlib import Path +from types import SimpleNamespace from typing import Any from unittest.mock import MagicMock @@ -438,6 +443,23 @@ def test_debug_cw_write_blocking_no_log_group_is_noop(monkeypatch): server._debug_cw("hello", task_id="t") +def test_debug_cw_exc_appends_the_traceback(monkeypatch, capfd): + """``_debug_cw_exc`` is the exception-carrying variant used by the error paths. + + Still reached from ``/invocations`` and from ``/run`` AFTER the config install + (the pre-install paths deliberately use the stdout-only ``_pre_config_log``), + so its formatting stays under test on its own rather than incidentally. + """ + monkeypatch.delenv("LOG_GROUP_NAME", raising=False) + try: + raise RuntimeError("kaboom") + except RuntimeError as exc: + server._debug_cw_exc("something FAILED", exc, task_id="t") + out = capfd.readouterr().out + assert "something FAILED [RuntimeError: kaboom]" in out + assert "Traceback" in out + + def test_debug_cw_write_blocking_bumps_failure_counter_on_boto_error(monkeypatch): """On boto errors the failure counter increments so operators can alarm. @@ -839,6 +861,19 @@ def test_none_stays_none(self): READY_HOOK = f"{server.MICROVM_HOOK_PREFIX}/ready" RUN_HOOK = f"{server.MICROVM_HOOK_PREFIX}/run" +VALIDATE_HOOK = f"{server.MICROVM_HOOK_PREFIX}/validate" +TERMINATE_HOOK = f"{server.MICROVM_HOOK_PREFIX}/terminate" + + +def _platform_config(**overrides) -> dict: + """A ``platform_config`` block carrying exactly the required subset. + + Built from the contract rather than a literal key list so a contract change + cannot leave these tests asserting a stale required set. + """ + config = {key: f"value-for-{key}" for key in server.MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS} + config.update(overrides) + return config def _run_hook_body(envelope: dict, microvm_id: str = "microvm-abc") -> dict: @@ -851,17 +886,35 @@ def _run_hook_body(envelope: dict, microvm_id: str = "microvm-abc") -> dict: return {"microvmId": microvm_id, "runHookPayload": json.dumps(envelope)} +#: A warm-up command that always succeeds, on any box, in well under a second. +#: +#: ``claude`` is absent from a CI checkout and PRESENT on some developer machines, +#: so a test that lets the real warm-up table run would pass or fail depending on +#: whose laptop it is. Every ``/ready`` test therefore pins the table. +_PORTABLE_WARMUP = (sys.executable, "--version") + + +@pytest.fixture +def warm_ready(monkeypatch): + """Pin ``/ready``'s warm-up to :data:`_PORTABLE_WARMUP` and nothing optional.""" + monkeypatch.setattr(server, "_READY_WARMUP_REQUIRED", _PORTABLE_WARMUP) + monkeypatch.setattr(server, "_READY_WARMUP_OPTIONAL", ()) + + class TestMicrovmReadyHook: """``/ready`` is what makes a MicroVM image buildable at all. ``CreateMicrovmImage`` refuses an image that enables any lifecycle hook without ``/ready``, and with the hook enabled but unserved both chipset builds fail ("Ready hook check failed: the application returned a client - error (HTTP 4xx) response"). A 200 from a booted server is the whole - contract in P1 — deeper warm-up checks are P2's ``/validate``. + error (HTTP 4xx) response"). + + Since ADR-021 P2-F5 a 200 means TWO things: the server is up **and** the + snapshot's heavyweight binaries have been exec'd, so their pages are resident + when the snapshot is captured. See :class:`TestMicrovmReadyHookWarmUp`. """ - def test_ready_returns_200_once_the_server_is_up(self, client): + def test_ready_returns_200_once_the_server_is_up_and_warm(self, client, warm_ready): r = client.post(READY_HOOK) assert r.status_code == 200 assert r.json() == {"status": "ready"} @@ -871,23 +924,352 @@ def test_ready_is_mounted_under_the_service_hook_prefix(self): routes = {getattr(r, "path", None) for r in server.app.routes} assert READY_HOOK in routes assert RUN_HOOK in routes + assert VALIDATE_HOOK in routes + assert TERMINATE_HOOK in routes - def test_ready_does_not_start_a_pipeline(self, client, monkeypatch): + def test_ready_does_not_start_a_pipeline(self, client, monkeypatch, warm_ready): # A build hook must never run task work: the snapshot is taken right # after it answers, so anything it starts would be frozen into the image. + # (The warm-up subprocesses are not "task work": they are ``--version`` + # execs that exit before the handler returns, and they join no thread.) monkeypatch.setattr(server, "run_task", MagicMock()) client.post(READY_HOOK) with server._threads_lock: assert server._active_threads == [] - def test_validate_and_suspend_resume_terminate_are_NOT_served(self, client): + def test_suspend_and_resume_are_NOT_served(self, client): # Declaring a hook nothing answers fails the corresponding build or - # lifecycle transition, so the construct declares exactly /ready + /run. - # This asserts the agent side of that: the others must 404. - for hook in ("validate", "suspend", "resume", "terminate"): + # lifecycle transition, so the construct declares exactly the hooks the + # agent serves. /validate + /terminate joined that set in P2; /suspend + + # /resume need the ComputeStrategy interface widening (P3), so they must + # still 404 — the assertion that keeps the construct honest. + for hook in ("suspend", "resume"): assert client.post(f"{server.MICROVM_HOOK_PREFIX}/{hook}").status_code == 404 +class TestMicrovmReadyHookWarmUp: + """``/ready`` warms the snapshot's heavyweight binaries (ADR-021 P2-F5). + + THE DEFECT THIS EXISTS FOR, because it is not guessable from the code: on the + P2 live run every task died at turn 0 with + + TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds + + reproducibly, while the same binary in the same image answers in under a second + locally. ``claude`` is a 225 MiB statically-linked ELF whose pages had never + been touched when the snapshot was taken, so the first ``exec`` on a restored + guest had to fault all of them in from lazily-restored storage. ``/ready`` is + the only hook that runs BEFORE the snapshot is captured, which makes it the + only place a warm page can be created. + + Three properties are load-bearing and all three are asserted here: the warm-up + actually EXECS the binary (a stat or a file read would not populate the same + pages), a required failure returns **503** rather than a 200 that would freeze + a cold — or broken — snapshot into every future MicroVM, and the whole thing + stays inside the hook budget the service is holding it to (see + :class:`TestMicrovmReadyWarmUpBudget`). + """ + + def test_the_warm_up_execs_every_command_once_required_first(self, client, monkeypatch): + calls: list[list[str]] = [] + + def fake_run(argv, **kwargs): + calls.append(list(argv)) + assert kwargs["capture_output"] is True + assert kwargs["check"] is False + return subprocess.CompletedProcess(argv, 0, stdout="2.1.191 (Claude Code)\n", stderr="") + + monkeypatch.setattr(server.subprocess, "run", fake_run) + assert client.post(READY_HOOK).status_code == 200 + # The real constants, not a stand-in: `claude` must be warmed or the fix is + # a no-op, and it must go FIRST so no best-effort command can eat the budget + # the required one needs. + assert calls[0] == ["claude", "--version"] + assert calls == [list(server._READY_WARMUP_REQUIRED)] + [ + list(argv) for argv in server._READY_WARMUP_OPTIONAL + ] + + def test_claude_is_the_only_REQUIRED_warm_up(self): + # git/node are warmed on the same mechanism but must never fail a build: + # neither was measured to blow a timeout, and a snapshot missing them is + # still a snapshot that can start a task. `claude` is the opposite — hence + # two constants rather than one table with a boolean, so the ordering + # guarantee is structural instead of conventional. + assert server._READY_WARMUP_REQUIRED[0] == "claude" + assert "claude" not in [argv[0] for argv in server._READY_WARMUP_OPTIONAL] + # Every warm-up is a bare `--version`: no network, no credentials, nothing + # written — which is what keeps /ready AWS-silent. + for argv in (server._READY_WARMUP_REQUIRED, *server._READY_WARMUP_OPTIONAL): + assert argv[1:] == ("--version",) + + def test_a_required_warm_up_timeout_reports_503_not_ready(self, client, monkeypatch, capfd): + # 503 is the hook contract's "still initialising", so the service keeps + # asking within the /ready budget and — if it never clears — fails the + # IMAGE BUILD. That is the correct trade: one failed build instead of every + # task failing at turn 0 on a snapshot that cannot exec its own agent. + def slow(argv, **kwargs): + raise subprocess.TimeoutExpired(argv, kwargs["timeout"]) + + monkeypatch.setattr(server.subprocess, "run", slow) + r = client.post(READY_HOOK) + assert r.status_code == 503 + assert r.json()["status"] == "not_ready" + assert any("claude" in f for f in r.json()["failed_warmups"]) + # The attempt is logged (stdout only — build role has no Logs grant), or a + # failed build gives the operator nothing to read. + out = capfd.readouterr().out + assert "warm-up of 'claude' FAILED" in out + assert "TimeoutExpired" in out + + def test_a_missing_binary_reports_503(self, client, monkeypatch): + # A snapshot without `claude` on PATH cannot run one task, so this must fail + # the build rather than be smoothed over. + def missing(argv, **kwargs): + raise FileNotFoundError(argv[0]) + + monkeypatch.setattr(server.subprocess, "run", missing) + assert client.post(READY_HOOK).status_code == 503 + + def test_a_nonzero_exit_reports_503(self, client, monkeypatch): + def broken(argv, **kwargs): + return subprocess.CompletedProcess(argv, 1, stdout="", stderr="Exec format error") + + monkeypatch.setattr(server.subprocess, "run", broken) + r = client.post(READY_HOOK) + assert r.status_code == 503 + assert r.json()["failed_warmups"] == ["claude:exit1"] + + def test_a_BEST_EFFORT_failure_still_reports_ready(self, client, monkeypatch, capfd): + # The other half of the required/best-effort split: git or node missing is + # logged and moves on. Failing the build on them would make the warm-up + # mechanism itself a liability. + def selective(argv, **kwargs): + if argv[0] == "claude": + return subprocess.CompletedProcess(argv, 0, stdout="2.1.191\n", stderr="") + raise FileNotFoundError(argv[0]) + + monkeypatch.setattr(server.subprocess, "run", selective) + r = client.post(READY_HOOK) + assert r.status_code == 200 + assert r.json() == {"status": "ready"} + out = capfd.readouterr().out + assert "warmed 'claude'" in out + assert "warm-up of 'git' FAILED" in out + + def test_an_unexpected_subprocess_error_becomes_503_not_500(self, client, monkeypatch): + # A warm-up defect must surface as the hook's own honest "not ready", never + # as a FastAPI 500 — the service reports both as a hook failure, but only + # one of them puts the reason in the build log. + def exploding(argv, **kwargs): + raise OSError("resource temporarily unavailable") + + monkeypatch.setattr(server.subprocess, "run", exploding) + assert client.post(READY_HOOK).status_code == 503 + + def test_the_warm_up_makes_no_aws_call_even_with_a_log_group_baked( + self, client, monkeypatch, capfd, warm_ready + ): + # /ready runs under the BUILD role: a Logs write can only fail (and each + # failure pollutes the shared _debug_cw_failures alarm), and any boto3 + # client built here freezes the build role's credential chain and the build + # region into the snapshot. Adding a subprocess must not have changed that. + monkeypatch.setenv("LOG_GROUP_NAME", "/abca/agent") + + def forbidden(*_args, **_kwargs): + raise AssertionError("the /ready warm-up must not make AWS calls") + + monkeypatch.setattr(server, "_debug_cw", forbidden) + monkeypatch.setattr(server, "_warn_cw", forbidden) + assert client.post(READY_HOOK).status_code == 200 + assert "warmed" in capfd.readouterr().out + + def test_warm_snapshot_binaries_returns_only_required_failures(self, monkeypatch): + # Unit-level, because the return value is the whole contract between the + # warm-up and the hook's status code. + monkeypatch.setattr(server, "_READY_WARMUP_REQUIRED", ("nope-required", "--version")) + monkeypatch.setattr(server, "_READY_WARMUP_OPTIONAL", (("nope-optional", "--version"),)) + + def missing(argv, **kwargs): + raise FileNotFoundError(argv[0]) + + monkeypatch.setattr(server.subprocess, "run", missing) + assert server._warm_snapshot_binaries() == ["nope-required:FileNotFoundError"] + + +class _FakeClock: + """Monotonic clock the tests advance by hand. + + The warm-up's budget arithmetic is about elapsed time, and the only honest way + to test "three slow commands cannot exceed the ceiling" without burning four + minutes of wall clock is to make time itself a test input. Patched onto + ``server._time_for_debug`` (the module-local ``time`` alias), so only the + server's view of the clock changes and monkeypatch restores it. + """ + + def __init__(self) -> None: + self.now = 1000.0 + + def monotonic(self) -> float: + return self.now + + def advance(self, seconds: float) -> None: + self.now += seconds + + +class TestMicrovmReadyWarmUpBudget: + """The warm-up must fit inside the hook budget the SERVICE is enforcing. + + Per-command timeouts do not compose. Three commands at 120 s each is 360 s + against a 300 s ``/ready`` budget (``READY_HOOK_TIMEOUT_SECONDS``), so a + warm-up added to prevent a RUNTIME failure could instead produce a BUILD + failure — and a hung best-effort command could starve the required one that + decides whether the snapshot is usable at all. Hence: required first with its + own budget, optional ones sharing the remainder of a total ceiling. + """ + + @pytest.fixture + def clock(self, monkeypatch): + fake = _FakeClock() + monkeypatch.setattr(server, "_time_for_debug", fake) + return fake + + @staticmethod + def _timeout_recorder(clock: _FakeClock, *, hang: tuple[str, ...] = (), fail: bool = False): + """subprocess.run stand-in that BURNS its whole timeout for hung commands.""" + seen: list[tuple[str, float]] = [] + + def fake_run(argv, **kwargs): + timeout = kwargs["timeout"] + seen.append((argv[0], timeout)) + if argv[0] in hang: + clock.advance(timeout) + raise subprocess.TimeoutExpired(argv, timeout) + if fail: + clock.advance(timeout) + raise subprocess.TimeoutExpired(argv, timeout) + clock.advance(0.1) + return subprocess.CompletedProcess(argv, 0, stdout="v1\n", stderr="") + + return fake_run, seen + + def test_the_ceiling_leaves_margin_inside_the_hook_budget(self): + # The numbers have to be comparable by eye against the CDK constant, so this + # pins the relationship rather than the values: total warm-up < hook budget, + # and the required command's own budget fits inside the total. + assert server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS >= 60 + assert ( + server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS + < server._READY_WARMUP_TOTAL_BUDGET_SECONDS + ) + # Read from the contract, NOT re-declared here. `READY_HOOK_TIMEOUT_SECONDS` + # in `cdk/src/constructs/lambda-microvm-compute.ts` imports the same field, + # so this assertion compares the agent against the value the service is + # actually configured with — a hardcoded 300 would keep passing after + # someone lowered the real budget. + from shared_constants import SHARED_CONSTANTS + + budgets = SHARED_CONSTANTS["microvm_hook_budgets"] + ready_hook_budget = budgets["ready_hook_timeout_seconds"] + assert ready_hook_budget > server._READY_WARMUP_TOTAL_BUDGET_SECONDS + # Both agent-side constants come from that same block, so a single edit moves + # the pair rather than half of it. + assert budgets["warmup_total_budget_seconds"] == (server._READY_WARMUP_TOTAL_BUDGET_SECONDS) + assert budgets["warmup_required_timeout_seconds"] == ( + server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS + ) + # Real margin, not a rounding error: enough for uvicorn scheduling plus the + # request itself. + assert ready_hook_budget - server._READY_WARMUP_TOTAL_BUDGET_SECONDS >= 30 + + def test_ALL_commands_slow_still_answers_within_the_ceiling(self, client, clock, monkeypatch): + # The aggregate-budget property. Every command hangs for its full timeout; + # the handler must still answer, and the total elapsed must not exceed the + # ceiling (which is what keeps it inside the hook budget). + fake_run, seen = self._timeout_recorder(clock, fail=True) + monkeypatch.setattr(server.subprocess, "run", fake_run) + started = clock.now + + r = client.post(READY_HOOK) + + assert r.status_code == 503 + elapsed = clock.now - started + assert elapsed <= server._READY_WARMUP_TOTAL_BUDGET_SECONDS + # The required command hung, so the optional ones were skipped entirely: + # the build cannot succeed now, and making the service wait longer for the + # 503 buys nothing. + assert [name for name, _ in seen] == ["claude"] + assert seen[0][1] == server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS + + def test_every_command_slow_but_none_skipped_stays_under_the_ceiling( + self, client, clock, monkeypatch + ): + # Same aggregate property with the required command SUCCEEDING slowly, so the + # optional ones do run: their timeouts must be the shrinking remainder, never + # a fresh full budget each. + fake_run, seen = self._timeout_recorder(clock, hang=("git", "node")) + monkeypatch.setattr(server.subprocess, "run", fake_run) + started = clock.now + + assert client.post(READY_HOOK).status_code == 200 + + assert clock.now - started <= server._READY_WARMUP_TOTAL_BUDGET_SECONDS + # Strictly decreasing budgets after the required one — the signature of a + # SHARED remainder rather than per-command budgets that sum past the hook's. + optional_timeouts = [timeout for name, timeout in seen if name != "claude"] + assert optional_timeouts == sorted(optional_timeouts, reverse=True) + assert sum(t for _, t in seen) <= ( + server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS + + server._READY_WARMUP_TOTAL_BUDGET_SECONDS + ) + + def test_a_HUNG_optional_command_never_blocks_the_200(self, client, clock, monkeypatch, capfd): + # The starvation guard, stated as the property that matters: once the + # REQUIRED warm-up has succeeded the snapshot is usable, so no best-effort + # command may talk the handler out of saying so. + fake_run, seen = self._timeout_recorder(clock, hang=("git", "node")) + monkeypatch.setattr(server.subprocess, "run", fake_run) + + r = client.post(READY_HOOK) + + assert r.status_code == 200 + assert r.json() == {"status": "ready"} + out = capfd.readouterr().out + assert "warmed 'claude'" in out + assert "warm-up of 'git' FAILED" in out + # `git` consumed the entire remainder, so `node` was skipped rather than + # started with a useless sub-second budget. + assert [name for name, _ in seen] == ["claude", "git"] + assert "skipping best-effort warm-up of 'node'" in out + + def test_the_required_command_gets_its_OWN_budget_not_a_share(self, client, clock, monkeypatch): + # It runs first precisely so the number it gets cannot be reduced by anything + # else: the one warm-up that decides whether the snapshot is usable must not + # be squeezed by a best-effort neighbour. + fake_run, seen = self._timeout_recorder(clock) + monkeypatch.setattr(server.subprocess, "run", fake_run) + assert client.post(READY_HOOK).status_code == 200 + assert seen[0] == ("claude", server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS) + + def test_an_optional_command_is_skipped_when_the_remainder_is_useless( + self, clock, monkeypatch, capfd + ): + # A sub-second timeout cannot warm a large binary; it can only manufacture a + # scary log line in a build that actually succeeded. + monkeypatch.setattr(server, "_READY_WARMUP_REQUIRED", ("req", "--version")) + monkeypatch.setattr(server, "_READY_WARMUP_OPTIONAL", (("opt", "--version"),)) + started = clock.now + + def fake_run(argv, **kwargs): + # The required command eats the entire ceiling but SUCCEEDS. + clock.advance(server._READY_WARMUP_TOTAL_BUDGET_SECONDS) + return subprocess.CompletedProcess(argv, 0, stdout="v\n", stderr="") + + monkeypatch.setattr(server.subprocess, "run", fake_run) + assert server._warm_snapshot_binaries() == [] + assert clock.now - started == server._READY_WARMUP_TOTAL_BUDGET_SECONDS + assert "skipping best-effort warm-up of 'opt'" in capfd.readouterr().out + + class TestMicrovmRunHookInlinePayload: """Inline envelope: ``{"agent_payload": {...}}``. @@ -1269,3 +1651,1335 @@ def test_merge_branches_non_string_entries_filtered(self): self._fake_req(), ) assert params["merge_branches"] == ["ok", "ok2"] + + +# -------------------------------------------------------------------------- +# platform_config: payload-sourced platform env (ADR-021 P2) +# -------------------------------------------------------------------------- + + +@pytest.fixture +def env_guard(): + """Snapshot/restore ``os.environ`` around a test that installs into it. + + ``_install_platform_config`` writes to the REAL process environment (that is + its job), and ``monkeypatch`` cannot undo a write it did not make — so + without this, one platform_config test would leak table names and a bogus + ``AGENT_SESSION_ROLE_ARN`` into every test that runs after it (the conftest + ``_clean_env`` fixture only strips the subset it knows about). + """ + before = dict(os.environ) + yield + os.environ.clear() + os.environ.update(before) + + +class TestPlatformConfigContract: + """The allowlist is a CROSS-PACKAGE contract, not an agent-local constant. + + The producer is the orchestrator's run-hook envelope builder + (``cdk/src/handlers/shared/orchestrator.ts``); both sides read + ``contracts/constants.json``. These tests are the agent-side tripwire: an + edit to the contract that the CDK side has not followed shows up here. + """ + + def test_allowlist_is_sourced_from_the_shared_contract(self): + from shared_constants import SHARED_CONSTANTS + + contract = SHARED_CONSTANTS["microvm_platform_config"] + assert contract["env_by_key"] == server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY + assert frozenset(contract["required"]) == server.MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS + + def test_wire_contract_is_exactly_the_documented_key_set(self): + # Spelled out on purpose: this is the wire contract Stage B's producer is + # written against, so a silent add/remove/rename must fail a test rather + # than merely change a JSON file. + assert server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY == { + "task_table_name": "TASK_TABLE_NAME", + "task_events_table_name": "TASK_EVENTS_TABLE_NAME", + "task_approvals_table_name": "TASK_APPROVALS_TABLE_NAME", + "nudges_table_name": "NUDGES_TABLE_NAME", + "log_group_name": "LOG_GROUP_NAME", + "artifacts_bucket_name": "ARTIFACTS_BUCKET_NAME", + "trace_artifacts_bucket_name": "TRACE_ARTIFACTS_BUCKET_NAME", + "github_token_secret_arn": "GITHUB_TOKEN_SECRET_ARN", + "linear_oauth_secret_arn": "LINEAR_OAUTH_SECRET_ARN", + "jira_oauth_secret_arn": "JIRA_OAUTH_SECRET_ARN", + "agent_session_role_arn": "AGENT_SESSION_ROLE_ARN", + "aws_sdk_ua_app_id": "AWS_SDK_UA_APP_ID", + "anthropic_default_haiku_model": "ANTHROPIC_DEFAULT_HAIKU_MODEL", + } + + def test_required_subset_is_exactly_the_four_run_blocking_keys(self): + assert ( + frozenset( + { + "task_table_name", + "task_events_table_name", + "github_token_secret_arn", + "agent_session_role_arn", + } + ) + == server.MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS + ) + + def test_required_keys_are_all_on_the_allowlist(self): + assert ( + set(server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY) + >= server.MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS + ) + + def test_env_names_are_upper_snake_and_unique(self): + env_names = list(server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY.values()) + assert len(set(env_names)) == len(env_names) + for name in env_names: + assert server._PLATFORM_CONFIG_ENV_RE.match(name), name + + def test_memory_id_is_not_a_platform_config_key(self): + # memory_id stays inside agent_payload: it is per-task state, not process + # configuration. Asserted so a future "it's an env var too" refactor has + # to argue with a test. + assert "memory_id" not in server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY + + def test_contract_validator_rejects_a_non_snake_case_key(self, monkeypatch): + monkeypatch.setattr( + server, "MICROVM_PLATFORM_CONFIG_ENV_BY_KEY", {"Task-Table": "TASK_TABLE_NAME"} + ) + with pytest.raises(ValueError, match="not snake_case"): + server._validate_platform_config_contract() + + def test_contract_validator_rejects_a_non_env_name_value(self, monkeypatch): + monkeypatch.setattr( + server, "MICROVM_PLATFORM_CONFIG_ENV_BY_KEY", {"task_table_name": "task table"} + ) + with pytest.raises(ValueError, match="UPPER_SNAKE"): + server._validate_platform_config_contract() + + def test_contract_validator_rejects_two_keys_on_one_env_var(self, monkeypatch): + monkeypatch.setattr( + server, + "MICROVM_PLATFORM_CONFIG_ENV_BY_KEY", + {"a_name": "SAME_ENV", "b_name": "SAME_ENV"}, + ) + monkeypatch.setattr(server, "MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS", frozenset({"a_name"})) + with pytest.raises(ValueError, match="same env var"): + server._validate_platform_config_contract() + + def test_contract_validator_rejects_a_required_key_off_the_allowlist(self, monkeypatch): + monkeypatch.setattr( + server, "MICROVM_PLATFORM_CONFIG_ENV_BY_KEY", {"task_table_name": "TASK_TABLE_NAME"} + ) + monkeypatch.setattr(server, "MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS", frozenset({"nope"})) + with pytest.raises(ValueError, match="absent from env_by_key"): + server._validate_platform_config_contract() + + def test_contract_validator_rejects_an_empty_allowlist(self, monkeypatch): + monkeypatch.setattr(server, "MICROVM_PLATFORM_CONFIG_ENV_BY_KEY", {}) + with pytest.raises(ValueError, match="must not be empty"): + server._validate_platform_config_contract() + + def test_contract_validator_rejects_an_empty_required_set(self, monkeypatch): + monkeypatch.setattr(server, "MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS", frozenset()) + with pytest.raises(ValueError, match="required must not be empty"): + server._validate_platform_config_contract() + + +class TestHookBudgetContract: + """The `/ready` budgets are a RELATIONSHIP, so the contract owns both halves. + + ``warmup_required < warmup_total < ready_hook``: the agent's warm-up must finish + inside the budget the MicroVM service holds the hook to, or the P2-F5 fix (warm + the 225 MiB ``claude`` binary before the snapshot) trades a runtime failure for a + build failure. Neither side can enforce an ordering it only knows half of, which + is why ``READY_HOOK_TIMEOUT_SECONDS`` in the CDK construct and these two + constants read the same block. ``scripts/check-constants-sync.ts`` is the other + tripwire; this validator makes the same violation fail the IMAGE BUILD (uvicorn + never binds, so ``/ready`` never answers) rather than a task. + """ + + def test_budgets_are_sourced_from_the_shared_contract(self): + from shared_constants import SHARED_CONSTANTS + + budgets = SHARED_CONSTANTS["microvm_hook_budgets"] + assert budgets["warmup_total_budget_seconds"] == server._READY_WARMUP_TOTAL_BUDGET_SECONDS + assert ( + budgets["warmup_required_timeout_seconds"] + == server._READY_WARMUP_REQUIRED_TIMEOUT_SECONDS + ) + + def test_the_shipped_contract_satisfies_its_own_invariant(self): + server._validate_hook_budget_contract() + + @pytest.mark.parametrize( + "budgets,match", + [ + ( + { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": 300, + "warmup_required_timeout_seconds": 120, + }, + "must be < ready_hook_timeout_seconds", + ), + ( + { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": 240, + "warmup_required_timeout_seconds": 240, + }, + "must be < warmup_total_budget_seconds", + ), + ( + { + "ready_hook_timeout_seconds": 0, + "warmup_total_budget_seconds": 240, + "warmup_required_timeout_seconds": 120, + }, + "must be a positive integer", + ), + ( + { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": "240", + "warmup_required_timeout_seconds": 120, + }, + "must be a positive integer", + ), + ( + { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": 240, + "warmup_required_timeout_seconds": None, + }, + "warmup_required_timeout_seconds must be a positive integer", + ), + ], + ids=["total-equals-hook", "required-equals-total", "zero", "string", "missing"], + ) + def test_a_contract_that_cannot_hold_fails_the_image_build(self, monkeypatch, budgets, match): + monkeypatch.setattr(server, "_HOOK_BUDGETS", budgets) + with pytest.raises(ValueError, match=match): + server._validate_hook_budget_contract() + + +class TestInstallPlatformConfig: + """Installing into ``os.environ`` is an env-injection surface — fail closed.""" + + def test_absent_block_installs_nothing(self, env_guard): + # The P1 envelope shape. A MicroVM image must still boot under an + # orchestrator that predates Stage B: the snapshot env is all there is. + assert server._install_platform_config(None) == [] + assert "TASK_TABLE_NAME" not in os.environ + + def test_installs_the_allowlisted_keys_as_upper_snake_env_vars(self, env_guard): + installed = server._install_platform_config(_platform_config()) + assert installed == sorted( + server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY[key] + for key in server.MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS + ) + assert os.environ["TASK_TABLE_NAME"] == "value-for-task_table_name" + assert os.environ["AGENT_SESSION_ROLE_ARN"] == "value-for-agent_session_role_arn" + + def test_every_allowlisted_key_is_installable(self, env_guard): + full = {key: f"v-{key}" for key in server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY} + installed = server._install_platform_config(full) + assert installed == sorted(server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY.values()) + for key, env_name in server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY.items(): + assert os.environ[env_name] == f"v-{key}" + + def test_payload_wins_over_a_pre_existing_image_env_value(self, env_guard): + # The load-bearing precedence rule: image env is frozen at snapshot time, + # the payload describes the live deployment. + os.environ["TASK_TABLE_NAME"] = "baked-into-the-snapshot" + server._install_platform_config(_platform_config(task_table_name="live-table")) + assert os.environ["TASK_TABLE_NAME"] == "live-table" + + def test_unknown_key_rejects_the_whole_block_and_installs_nothing(self, env_guard): + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config(_platform_config(ld_preload="/tmp/evil.so")) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INVALID" + assert "ld_preload" in str(excinfo.value) + # Fail CLOSED: not one key from a rejected block reaches the environment, + # so a hostile key cannot ride along with valid ones. + assert "TASK_TABLE_NAME" not in os.environ + + def test_unknown_key_message_lists_the_allowlist(self, env_guard): + with pytest.raises(server._PlatformConfigError, match="task_table_name"): + server._install_platform_config({"nope": "x"}) + + @pytest.mark.parametrize("raw", ["a string", ["a", "list"], 42, True]) + def test_a_non_object_block_is_rejected(self, raw, env_guard): + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config(raw) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INVALID" + assert "must be an object" in str(excinfo.value) + + @pytest.mark.parametrize("value", [42, 1.5, ["x"], {"a": 1}, True]) + def test_a_non_string_value_is_rejected(self, value, env_guard): + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config(_platform_config(log_group_name=value)) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INVALID" + assert "must be" in str(excinfo.value) + assert "LOG_GROUP_NAME" not in os.environ + + @pytest.mark.parametrize("value", ["", " ", None]) + def test_a_blank_optional_value_is_treated_as_absent(self, value, env_guard): + # The natural producer (`process.env.X ?? ''`) emits an empty string for a + # resource the deployment does not have. Skipping is right; clobbering an + # image value with "" would turn "absent there" into "unconfigured here". + os.environ["LOG_GROUP_NAME"] = "from-the-image" + installed = server._install_platform_config(_platform_config(log_group_name=value)) + assert "LOG_GROUP_NAME" not in installed + assert os.environ["LOG_GROUP_NAME"] == "from-the-image" + + @pytest.mark.parametrize("value", ["", " ", None]) + def test_a_blank_required_value_is_rejected(self, value, env_guard): + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config(_platform_config(task_table_name=value)) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE" + assert "task_table_name" in str(excinfo.value) + assert "TASK_EVENTS_TABLE_NAME" not in os.environ + + def test_a_missing_required_key_is_rejected(self, env_guard): + partial = _platform_config() + partial.pop("agent_session_role_arn") + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config(partial) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE" + assert "agent_session_role_arn" in str(excinfo.value) + + def test_an_explicitly_empty_block_is_incomplete_not_absent(self, env_guard): + # Sending the key with nothing in it is a producer bug; omitting the key + # is the documented "I have nothing to say". + with pytest.raises(server._PlatformConfigError) as excinfo: + server._install_platform_config({}) + assert excinfo.value.code == "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE" + + def test_the_two_codes_are_distinct(self, env_guard): + # One exception type, two operator remedies: fix the producer vs. fix the + # deployment wiring. Collapsing them would send operators to the wrong one. + with pytest.raises(server._PlatformConfigError) as invalid: + server._install_platform_config({"bogus_key": "x"}) + with pytest.raises(server._PlatformConfigError) as incomplete: + server._install_platform_config({"log_group_name": "lg"}) + assert invalid.value.code != incomplete.value.code + + +class TestMicrovmRunHookPlatformConfig: + """``platform_config`` arrives on the ``/run`` hook as a SIBLING of ``agent_payload``.""" + + def _payload(self, **extra) -> dict: + return { + "task_id": "t-pc", + "repo_url": "org/repo", + "prompt": "do it", + "github_token": "ghp_x", + **extra, + } + + def test_inline_envelope_installs_the_config_and_accepts_the_task( + self, client, monkeypatch, env_guard + ): + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload": self._payload(), + "platform_config": _platform_config(log_group_name="/abca/agent"), + } + ), + ) + + assert r.status_code == 200 + assert r.json()["status"] == "accepted" + assert os.environ["TASK_TABLE_NAME"] == "value-for-task_table_name" + assert os.environ["LOG_GROUP_NAME"] == "/abca/agent" + + def test_the_config_is_installed_BEFORE_credential_resolution( + self, client, monkeypatch, env_guard + ): + # The ordering that makes the whole feature work: _extract_invocation_params + # resolves the GitHub token, which reads GITHUB_TOKEN_SECRET_ARN. Installing + # after that point would resolve the task against the snapshot's frozen env + # and silently ignore everything the orchestrator sent. + seen: dict = {} + + def fake_resolve_github_token(): + seen["gh_arn"] = os.environ.get("GITHUB_TOKEN_SECRET_ARN") + seen["session_role"] = os.environ.get("AGENT_SESSION_ROLE_ARN") + seen["threads_at_resolve"] = len(server._active_threads) + return "ghp_resolved" + + monkeypatch.setattr(server, "resolve_github_token", fake_resolve_github_token) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + payload = self._payload() + payload.pop("github_token") # force the resolver to run + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload": payload, + "platform_config": _platform_config( + github_token_secret_arn="arn:aws:secretsmanager:::secret:live" + ), + } + ), + ) + + assert r.status_code == 200 + assert seen["gh_arn"] == "arn:aws:secretsmanager:::secret:live" + assert seen["session_role"] == "value-for-agent_session_role_arn" + # ...and before the pipeline thread existed at all. + assert seen["threads_at_resolve"] == 0 + + def test_an_unknown_key_returns_400_and_starts_nothing(self, client, monkeypatch, env_guard): + monkeypatch.setattr(server, "run_task", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload": self._payload(), + "platform_config": _platform_config(aws_endpoint_url="http://attacker"), + } + ), + ) + + assert r.status_code == 400 + assert r.json()["code"] == "MICROVM_RUN_PLATFORM_CONFIG_INVALID" + assert "aws_endpoint_url" in r.json()["message"] + assert "TASK_TABLE_NAME" not in os.environ + with server._threads_lock: + assert server._active_threads == [] + + def test_a_missing_required_key_returns_400_and_starts_nothing( + self, client, monkeypatch, env_guard + ): + monkeypatch.setattr(server, "run_task", MagicMock()) + partial = _platform_config() + partial.pop("task_events_table_name") + + r = client.post( + RUN_HOOK, + json=_run_hook_body({"agent_payload": self._payload(), "platform_config": partial}), + ) + + assert r.status_code == 400 + assert r.json()["code"] == "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE" + assert "task_events_table_name" in r.json()["message"] + with server._threads_lock: + assert server._active_threads == [] + + def test_a_non_object_block_returns_400(self, client, monkeypatch, env_guard): + monkeypatch.setattr(server, "run_task", MagicMock()) + r = client.post( + RUN_HOOK, + json=_run_hook_body( + {"agent_payload": self._payload(), "platform_config": "TASK_TABLE_NAME=x"} + ), + ) + assert r.status_code == 400 + assert r.json()["code"] == "MICROVM_RUN_PLATFORM_CONFIG_INVALID" + + def test_an_envelope_without_platform_config_is_still_accepted( + self, client, monkeypatch, env_guard, capfd + ): + # P1 compatibility: image snapshot and orchestrator Lambda deploy on + # independent cadences, so a new image must not require a Stage-B + # orchestrator. It warns, loudly, rather than rejecting. + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post(RUN_HOOK, json=_run_hook_body({"agent_payload": self._payload()})) + + assert r.status_code == 200 + # Still pre-install (nothing was installed), so the warning is stdout-only: + # `_warn_cw` here would spawn the CloudWatch thread off the snapshot's own + # baked env — the very thing it is warning about. See `TestMicrovmRunHook + # PreInstallAwsSilence`. + assert "[server/run-pre-config] /run hook received no platform_config" in ( + capfd.readouterr().out + ) + + def test_s3_pointer_takes_the_config_from_the_outer_envelope( + self, client, monkeypatch, env_guard + ): + # The producer's pointer form: the bare task payload lands in S3 and the + # config rides beside the pointer, inside the 4 KB hook body. + monkeypatch.setattr( + server, + "_fetch_microvm_payload_from_s3", + lambda _uri: self._payload(task_id="t-outer"), + ) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload_s3_uri": "s3://bucket/t-outer/payload.json", + "platform_config": _platform_config(task_table_name="outer-table"), + } + ), + ) + + assert r.status_code == 200 + assert r.json()["task_id"] == "t-outer" + assert os.environ["TASK_TABLE_NAME"] == "outer-table" + + def test_s3_pointer_takes_the_config_merged_into_the_fetched_object( + self, client, monkeypatch, env_guard + ): + # The producer ALSO merges the config into the S3 object, so the agent + # gets it whichever end of the fetch it reads. A stray platform_config key + # left in the bare payload is inert — the extractor reads named fields. + fetched = self._payload(task_id="t-inner") + fetched["platform_config"] = _platform_config(task_table_name="inner-table") + monkeypatch.setattr(server, "_fetch_microvm_payload_from_s3", lambda _uri: fetched) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body({"agent_payload_s3_uri": "s3://bucket/t-inner/payload.json"}), + ) + + assert r.status_code == 200 + assert r.json()["task_id"] == "t-inner" + assert os.environ["TASK_TABLE_NAME"] == "inner-table" + + def test_s3_object_may_itself_be_the_full_envelope(self, client, monkeypatch, env_guard): + monkeypatch.setattr( + server, + "_fetch_microvm_payload_from_s3", + lambda _uri: { + "agent_payload": self._payload(task_id="t-nested"), + "platform_config": _platform_config(task_table_name="nested-table"), + }, + ) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body({"agent_payload_s3_uri": "s3://bucket/t-nested/payload.json"}), + ) + + assert r.status_code == 200 + assert r.json()["task_id"] == "t-nested" + assert os.environ["TASK_TABLE_NAME"] == "nested-table" + + def test_the_fetched_object_wins_over_the_outer_envelope(self, client, monkeypatch, env_guard): + fetched = self._payload(task_id="t-prec") + fetched["platform_config"] = _platform_config(task_table_name="inner-wins") + monkeypatch.setattr(server, "_fetch_microvm_payload_from_s3", lambda _uri: fetched) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload_s3_uri": "s3://bucket/t-prec/payload.json", + "platform_config": _platform_config(task_table_name="outer-loses"), + } + ), + ) + + assert r.status_code == 200 + assert os.environ["TASK_TABLE_NAME"] == "inner-wins" + + def test_a_nested_agent_payload_of_the_wrong_type_is_a_400(self, client, monkeypatch): + monkeypatch.setattr( + server, + "_fetch_microvm_payload_from_s3", + lambda _uri: {"agent_payload": "not-an-object"}, + ) + monkeypatch.setattr(server, "run_task", MagicMock()) + + r = client.post(RUN_HOOK, json=_run_hook_body({"agent_payload_s3_uri": "s3://b/k"})) + + assert r.status_code == 400 + assert r.json()["code"] == "MICROVM_RUN_PAYLOAD_INVALID" + assert "agent_payload in the S3 payload must be an object" in r.json()["message"] + + def test_resolve_returns_the_config_alongside_the_payload(self): + payload, config = server._resolve_microvm_run_payload( + json.dumps({"agent_payload": {"task_id": "t"}, "platform_config": {"a": "b"}}) + ) + assert payload == {"task_id": "t"} + assert config == {"a": "b"} + + def test_resolve_returns_none_for_an_envelope_without_a_config(self): + _payload, config = server._resolve_microvm_run_payload( + json.dumps({"agent_payload": {"task_id": "t"}}) + ) + assert config is None + + +# -------------------------------------------------------------------------- +# /validate + /terminate (ADR-021 P2) +# -------------------------------------------------------------------------- + + +class TestMicrovmValidateHook: + """A BUILD hook running under the BUILD role — so: shallow, and AWS-silent. + + ``CreateMicrovmImage`` runs ``/validate`` with the build role, which + deliberately holds no Bedrock / Secrets Manager / DynamoDB grants. Every + "deeper warm-up assertion" ADR-021 originally sketched here would therefore + AccessDenied and fail every image build. The hook is a self-check, not a + reachability probe. + """ + + def test_returns_200_with_the_individual_checks(self, client): + r = client.post(VALIDATE_HOOK) + assert r.status_code == 200 + body = r.json() + assert body["status"] == "valid" + assert body["checks"] == { + "server_initialized": True, + "hook_routes_registered": True, + "python_version_supported": True, + "platform_config_contract_loaded": True, + } + assert body["hook_prefix"] == server.MICROVM_HOOK_PREFIX + assert body["platform_config_keys"] == len(server.MICROVM_PLATFORM_CONFIG_ENV_BY_KEY) + + def test_makes_zero_aws_calls_even_with_a_log_group_configured( + self, client, monkeypatch, capfd + ): + # The whole point. _debug_cw / _warn_cw build a CloudWatch Logs client + # whenever LOG_GROUP_NAME is set, so a build hook must not use them; and + # boto3.client() would populate boto3.DEFAULT_SESSION — a module global + # holding the BUILD role's resolved credentials and region — which the + # snapshot would then freeze in for every MicroVM launched from it. + monkeypatch.setenv("LOG_GROUP_NAME", "/abca/agent") + + def forbidden(*_args, **_kwargs): + raise AssertionError("a build hook must not make AWS calls") + + import boto3 + + import aws_session + + monkeypatch.setattr(boto3, "client", forbidden) + monkeypatch.setattr(boto3, "Session", forbidden) + monkeypatch.setattr(aws_session, "platform_client", forbidden) + monkeypatch.setattr(aws_session, "get_session", forbidden) + monkeypatch.setattr(server, "_debug_cw", forbidden) + monkeypatch.setattr(server, "_warn_cw", forbidden) + + assert client.post(VALIDATE_HOOK).status_code == 200 + # ...and it still logs, to stdout only. + assert "/validate hook: ok" in capfd.readouterr().out + + def test_ready_is_also_aws_silent_with_a_log_group_configured( + self, client, monkeypatch, capfd, warm_ready + ): + # /ready runs under the same build role, so the same rule applies. It used + # to route through _debug_cw, whose write can only FAIL under a role with + # no Logs grant — and each failure bumps the shared _debug_cw_failures + # counter, poisoning the "debug path is blind" signal on every build. + monkeypatch.setenv("LOG_GROUP_NAME", "/abca/agent") + + def forbidden(*_args, **_kwargs): + raise AssertionError("a build hook must not make AWS calls") + + monkeypatch.setattr(server, "_debug_cw", forbidden) + monkeypatch.setattr(server, "_warn_cw", forbidden) + + assert client.post(READY_HOOK).status_code == 200 + assert "/ready hook" in capfd.readouterr().out + + def test_does_not_touch_credential_resolution(self, client, monkeypatch): + import aws_session + + def forbidden(*_args, **_kwargs): + raise AssertionError("/validate must not resolve credentials") + + monkeypatch.setattr(aws_session, "get_session", forbidden) + monkeypatch.setattr(server, "resolve_github_token", forbidden) + + assert client.post(VALIDATE_HOOK).status_code == 200 + assert aws_session._session is None + assert aws_session._scoped is None + + def test_starts_no_pipeline(self, client, monkeypatch): + # The snapshot is taken right after the build hooks answer, so anything + # started here would be frozen into the image. + monkeypatch.setattr(server, "run_task", MagicMock()) + client.post(VALIDATE_HOOK) + with server._threads_lock: + assert server._active_threads == [] + + def test_returns_503_while_the_module_is_still_initialising(self, client, monkeypatch): + # Per the hook contract, 503 means "not ready yet". A permanently failing + # check therefore fails the image build — the right outcome for a snapshot + # that is genuinely broken. + monkeypatch.setattr(server, "_module_initialized", False) + r = client.post(VALIDATE_HOOK) + assert r.status_code == 503 + assert r.json()["status"] == "not_ready" + assert r.json()["failed_checks"] == ["server_initialized"] + + def test_reports_a_missing_hook_route(self, client, monkeypatch): + monkeypatch.setattr(server, "MICROVM_HOOK_PREFIX", "/typo/prefix") + r = client.post(VALIDATE_HOOK) + assert r.status_code == 503 + assert "hook_routes_registered" in r.json()["failed_checks"] + assert r.json()["missing_routes"] == [ + "/typo/prefix/ready", + "/typo/prefix/run", + "/typo/prefix/terminate", + "/typo/prefix/validate", + ] + + def test_reports_an_unsupported_interpreter(self, client, monkeypatch): + monkeypatch.setattr(server, "_MIN_PYTHON_VERSION", (99, 0)) + r = client.post(VALIDATE_HOOK) + assert r.status_code == 503 + assert "python_version_supported" in r.json()["failed_checks"] + + def test_reports_baked_secret_env_as_a_warning_not_a_failure(self, client, monkeypatch): + # ADR-021 sub-decision 3: the snapshot must stay secret-free. REPORT-ONLY, + # because the build environment's own credentials may legitimately be in + # this process's env and failing here would fail every build. Names only. + for name in server._SNAPSHOT_FORBIDDEN_SECRET_ENV: + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("GITHUB_TOKEN", "ghp_should_not_be_baked") + r = client.post(VALIDATE_HOOK) + assert r.status_code == 200 + assert r.json()["warnings"] == ["secret_env_present_in_snapshot:GITHUB_TOKEN"] + assert "ghp_should_not_be_baked" not in r.text + + def test_no_warnings_on_a_clean_snapshot_env(self, client, monkeypatch): + for name in server._SNAPSHOT_FORBIDDEN_SECRET_ENV: + monkeypatch.delenv(name, raising=False) + assert client.post(VALIDATE_HOOK).json()["warnings"] == [] + + +class TestMicrovmTerminateHook: + """Best-effort flush. Always 200, never a terminal status write.""" + + def test_returns_200_with_no_body_at_all(self, client): + # The hook must not turn a body-less call into FastAPI's 422: a 4xx here + # reports a hook failure for a teardown that actually succeeded. + r = client.post(TERMINATE_HOOK) + assert r.status_code == 200 + assert r.json()["status"] == "acknowledged" + assert r.json()["active_pipeline_threads"] == 0 + + def test_echoes_the_microvm_id(self, client): + r = client.post(TERMINATE_HOOK, json={"microvmId": "microvm-zzz"}) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "microvm-zzz" + + def test_an_EMPTY_microvm_id_is_expected_normal_and_earns_no_warning(self, client, capfd): + # What the service ACTUALLY sends (live 2026-08-07, ADR-021 P2-F8): the id + # is the empty string on this hook, unlike /run where it is populated. So it + # must not look like a defect in the guest's last log line — an operator + # reading a warning here would go hunting for a wire-contract break that + # does not exist. Only a genuinely unreadable body warns (tests below). + r = client.post(TERMINATE_HOOK, json={"microvmId": ""}) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + out = capfd.readouterr().out + assert "[server/warn]" not in out + # ...and the breadcrumb still lands, carrying the pipeline state that IS + # this hook's value (correlation rides /run's accepted line instead). + assert '"microvm_id": ""' in out + assert '"active_pipeline_threads": 0' in out + + def test_never_writes_terminal_task_status(self, client, monkeypatch): + # The orchestrator owns terminal state: it finalizes the task and THEN + # calls TerminateMicrovm, so a terminate hook that wrote a status would + # race the finalization it follows and could clobber the real outcome. + write_terminal = MagicMock() + monkeypatch.setattr(server.task_state, "write_terminal", write_terminal) + write_heartbeat = MagicMock() + monkeypatch.setattr(server.task_state, "write_heartbeat", write_heartbeat) + + client.post(TERMINATE_HOOK, json={"microvmId": "m-1"}) + + write_terminal.assert_not_called() + write_heartbeat.assert_not_called() + + def test_returns_200_without_joining_a_running_pipeline(self, client, monkeypatch): + # A drain can take minutes (that is lifespan's job on graceful shutdown); + # the hook budget is 1-60 s, so /terminate must observe and return. + release = threading.Event() + entered = threading.Event() + + def slow_run_task(**_kwargs): + entered.set() + release.wait(timeout=10.0) + + monkeypatch.setattr(server, "run_task", slow_run_task) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + try: + client.post( + RUN_HOOK, + json=_run_hook_body( + {"agent_payload": {"task_id": "t-term", "repo_url": "o/r", "prompt": "x"}} + ), + ) + assert entered.wait(timeout=5.0) + + r = client.post(TERMINATE_HOOK, json={"microvmId": "m-live"}) + assert r.status_code == 200 + assert r.json()["active_pipeline_threads"] == 1 + # Still running: the hook did not stop or join it. + with server._threads_lock: + assert any(t.is_alive() for t in server._active_threads) + finally: + release.set() + + def test_emits_a_final_structured_log_line(self, client, capfd): + client.post(TERMINATE_HOOK, json={"microvmId": "m-log"}) + out = capfd.readouterr().out + assert "/terminate hook:" in out + assert '"event": "microvm_terminate"' in out + assert '"microvm_id": "m-log"' in out + + def test_still_returns_200_when_the_best_effort_step_fails(self, client, monkeypatch, capfd): + def boom(*_args, **_kwargs): + raise RuntimeError("log sink exploded") + + monkeypatch.setattr(server, "_debug_cw", boom) + r = client.post(TERMINATE_HOOK, json={"microvmId": "m-boom"}) + assert r.status_code == 200 + # Best-effort, but not silent. + assert "best-effort step failed" in capfd.readouterr().out + + +class TestMicrovmPayloadFetchAttribution: + """Every outbound AWS call carries ABCA's solution attribution (#319).""" + + def test_the_s3_payload_fetch_goes_through_the_attributed_factory(self, monkeypatch): + captured: dict = {} + + class _Body: + @staticmethod + def read(): + return b'{"task_id": "t-1"}' + + def fake_platform_client(service_name, **kwargs): + captured["service"] = service_name + captured["kwargs"] = kwargs + return SimpleNamespace(get_object=lambda **_kw: {"Body": _Body}) + + import aws_session + + monkeypatch.setattr(aws_session, "platform_client", fake_platform_client) + + assert server._fetch_microvm_payload_from_s3("s3://b/k") == {"task_id": "t-1"} + assert captured["service"] == "s3" + + def test_the_fetch_client_carries_the_md_user_agent_segment(self, monkeypatch): + # A naked boto3.client('s3') would silently drop the md/ segment. Assert on + # the OUTCOME (the UA on the config) rather than on which helper was used. + captured: dict = {} + + class _Body: + @staticmethod + def read(): + return b'{"task_id": "t-1"}' + + def fake_boto3_client(service_name, **kwargs): + captured["service"] = service_name + captured["config"] = kwargs.get("config") + return SimpleNamespace(get_object=lambda **_kw: {"Body": _Body}) + + import boto3 + + monkeypatch.setattr(boto3, "client", fake_boto3_client) + + server._fetch_microvm_payload_from_s3("s3://b/k") + + import ua + + assert captured["service"] == "s3" + assert ua.static_user_agent_extra() in captured["config"].user_agent_extra + + +class TestSnapshotCredentialHygiene: + """Nothing on the server's import or /ready path may cache an SDK session. + + The MicroVM image is a SNAPSHOT: whatever module state exists when the + snapshot is taken is replayed by every MicroVM launched from that image + version. A boto3 session created during import or a build hook would freeze + the BUILD role's resolved credential chain and the BUILD-time region into the + image — inherited, stale and cross-role, by every task. + + Runs in a SUBPROCESS because the assertion is about process-global state + (``sys.modules``, ``boto3.DEFAULT_SESSION``, ``aws_session._session``) that + dozens of earlier tests in this suite have already populated in-process. + """ + + PROBE = """ +import sys, time, threading +sys.path.insert(0, "src") +import server +import aws_session +from fastapi.testclient import TestClient + +client = TestClient(server.app) +# /ready warms the snapshot's heavyweight binaries by exec'ing them (ADR-021 +# P2-F5). `claude` is not installed in a CI checkout, so pin the warm-up to a +# binary that always exists: the property under test is that a build hook's +# subprocess exec creates no boto3 session, not WHICH binary it warms. +server._READY_WARMUP_REQUIRED = (sys.executable, "--version") +server._READY_WARMUP_OPTIONAL = () +assert client.post("/aws/lambda-microvms/runtime/v1/ready").status_code == 200 +client.post("/aws/lambda-microvms/runtime/v1/validate") + +# Any AWS work would happen on a fire-and-forget daemon thread, so give one a +# chance to run before concluding that none exists. +for _ in range(20): + if "boto3" in sys.modules: + break + time.sleep(0.05) + +findings = { + "boto3_imported": "boto3" in sys.modules, + "botocore_imported": "botocore" in sys.modules, + "cached_session": aws_session._session is not None, + "scoped_resolved": aws_session._scoped is not None, + "log_writer_threads": [ + t.name for t in threading.enumerate() if "cw-write" in t.name + ], +} +print("FINDINGS:" + repr(findings)) +""" + + def test_import_and_build_hooks_create_no_boto3_session(self): + agent_dir = Path(__file__).resolve().parent.parent + env = { + **os.environ, + # The hostile case: a snapshot that DID bake the log group would make + # the old _debug_cw-based /ready spawn a CloudWatch writer. + "LOG_GROUP_NAME": "/abca/agent", + "AWS_REGION": "us-east-1", + "PYTHONPATH": str(agent_dir / "src"), + } + proc = subprocess.run( + [sys.executable, "-c", self.PROBE], + cwd=str(agent_dir), + env=env, + capture_output=True, + text=True, + timeout=90, + check=False, + ) + assert proc.returncode == 0, proc.stderr + line = next( + (ln for ln in proc.stdout.splitlines() if ln.startswith("FINDINGS:")), + None, + ) + assert line is not None, proc.stdout + findings = eval(line[len("FINDINGS:") :]) # noqa: S307 — our own repr() + assert findings == { + "boto3_imported": False, + "botocore_imported": False, + "cached_session": False, + "scoped_resolved": False, + "log_writer_threads": [], + } + + +class TestMicrovmRunHookPreInstallAwsSilence: + """Before ``platform_config`` is installed, ``/run`` may touch exactly ONE AWS seam. + + Same defect class the build hooks avoid, one phase later: until the install has + run, ``LOG_GROUP_NAME`` is whatever the snapshot happens to carry, so a + ``_debug_cw`` on this path would resolve credentials and pin + ``boto3.DEFAULT_SESSION`` *before* the orchestrator's own region / + ``AWS_SDK_UA_APP_ID`` / session role are in the environment. The sole permitted + pre-install call is the S3 payload fetch, because the config is inside the + object being fetched. + + Every test here runs with a **baked ``LOG_GROUP_NAME``** — the hostile case the + fix exists for. Without it, ``_debug_cw`` degrades to stdout on its own and the + assertions would pass vacuously. + """ + + def _payload(self, **extra) -> dict: + return { + "task_id": "t-silent", + "repo_url": "org/repo", + "prompt": "do it", + "github_token": "ghp_x", + **extra, + } + + @pytest.fixture + def seam_guard(self, monkeypatch): + """Arm every AWS/credential seam to raise until the pre-install phase is OVER. + + Two things end that phase, and only two: + + * ``_install_platform_config`` returning a **non-empty** env list — a real + install. Flipping on *any* return would be a hole big enough to drive B2 + through: the ``raw is None`` early return installs nothing and returns + ``[]``, so treating it as "installed" disarms the guard for the entire + legacy no-``platform_config`` path — which is exactly where a ``_warn_cw`` + was spawning the CloudWatch thread off the snapshot's baked env. + * ``_extract_invocation_params`` being entered. Past that point the legacy + path is *allowed* to talk to AWS: running on the snapshot's own env is the + documented P1-compatibility behaviour, so the accepted-line ``_debug_cw`` + and the pipeline below it are legitimate. Everything the handler does + *before* it — including the "no platform_config" warning — is not. + + A rejection path reaches neither, so the seams stay armed for the whole + request: a rejected run installed nothing and has no more right to an AWS + call than it had before. + """ + state: dict[str, Any] = { + "install_phase_done": False, + "installed_env": None, + "violations": [], + } + real_install = server._install_platform_config + real_extract = server._extract_invocation_params + + def spy_install(raw): + result = real_install(raw) + state["installed_env"] = result + if result: + state["install_phase_done"] = True + return result + + def spy_extract(*args, **kwargs): + state["install_phase_done"] = True + return real_extract(*args, **kwargs) + + monkeypatch.setattr(server, "_install_platform_config", spy_install) + monkeypatch.setattr(server, "_extract_invocation_params", spy_extract) + + def guard(name): + def _seam(*_args, **_kwargs): + if not state["install_phase_done"]: + state["violations"].append(name) + raise AssertionError(f"{name} touched before platform_config was installed") + return MagicMock() + + return _seam + + import boto3 + + import aws_session + + # Kept so a test can re-enable exactly the ONE permitted pre-install seam + # (the S3 payload fetch) and assert on it positively. + state["real_platform_client"] = aws_session.platform_client + + for module, attr in ( + (boto3, "client"), + (boto3, "Session"), + (aws_session, "platform_client"), + (aws_session, "tenant_client"), + (aws_session, "tenant_resource"), + (aws_session, "get_session"), + (server, "_debug_cw"), + (server, "_warn_cw"), + (server, "_debug_cw_exc"), + ): + monkeypatch.setattr(module, attr, guard(f"{module.__name__}.{attr}")) + + monkeypatch.setenv("LOG_GROUP_NAME", "/abca/agent") + return state + + @pytest.mark.parametrize("with_config", [True, False], ids=["with-config", "no-config"]) + def test_no_cloudwatch_or_credential_seam_is_touched_before_the_install( + self, client, monkeypatch, env_guard, seam_guard, capfd, with_config + ): + # The ``no-config`` arm is the legacy P1 envelope, and it is the harder case: + # nothing is ever installed, so EVERY line up to param extraction — including + # the "running on the snapshot's frozen env" warning itself — is still + # pre-install. A ``_warn_cw`` there would spawn the CloudWatch writer thread + # and pin ``boto3.DEFAULT_SESSION`` off the baked ``LOG_GROUP_NAME`` this + # fixture sets, which is precisely the defect the warning is reporting. + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + envelope: dict[str, Any] = {"agent_payload": self._payload()} + if with_config: + envelope["platform_config"] = _platform_config() + + r = client.post(RUN_HOOK, json=_run_hook_body(envelope)) + + assert r.status_code == 200 + assert seam_guard["violations"] == [] + assert seam_guard["install_phase_done"] is True + if with_config: + assert seam_guard["installed_env"] + else: + # Vacuously "installed": the early return the flag must NOT trust. + assert seam_guard["installed_env"] == [] + # The warning still reaches an operator — stdout, via the pre-install sink. + assert ( + "[server/run-pre-config] /run hook received no platform_config" + in capfd.readouterr().out + ) + + def test_the_received_line_is_stdout_only( + self, client, monkeypatch, env_guard, seam_guard, capfd + ): + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + client.post( + RUN_HOOK, + json=_run_hook_body( + {"agent_payload": self._payload(), "platform_config": _platform_config()}, + microvm_id="microvm-quiet", + ), + ) + + out = capfd.readouterr().out + assert "[server/run-pre-config] /run hook received:" in out + assert "microvm-quiet" in out + + def test_the_s3_payload_fetch_is_the_only_pre_install_aws_call( + self, client, monkeypatch, env_guard, seam_guard + ): + # The permitted exception, asserted positively: exactly one client, for s3, + # while the CloudWatch/credential seams stay armed. + services: list[str] = [] + + class _Body: + @staticmethod + def read(): + return json.dumps(self._payload(task_id="t-from-s3")).encode() + + def recording_client(service_name, **_kwargs): + services.append(service_name) + return SimpleNamespace(get_object=lambda **_kw: {"Body": _Body}) + + import boto3 + + import aws_session + + # Re-enable the one permitted seam, and only it: the fetch must still go + # through the attributed factory (#319), which delegates to boto3.client. + monkeypatch.setattr(aws_session, "platform_client", seam_guard["real_platform_client"]) + monkeypatch.setattr(boto3, "client", recording_client) + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body( + { + "agent_payload_s3_uri": "s3://payload-bucket/t-from-s3/payload.json", + "platform_config": _platform_config(), + } + ), + ) + + assert r.status_code == 200 + assert r.json()["task_id"] == "t-from-s3" + assert services == ["s3"] + assert seam_guard["violations"] == [] + + def test_a_malformed_envelope_is_rejected_without_touching_a_seam( + self, client, monkeypatch, seam_guard, capfd + ): + monkeypatch.setattr(server, "run_task", MagicMock()) + + r = client.post(RUN_HOOK, json={"microvmId": "m", "runHookPayload": "not json"}) + + assert r.status_code == 400 + assert r.json()["code"] == "MICROVM_RUN_PAYLOAD_INVALID" + assert seam_guard["violations"] == [] + assert seam_guard["install_phase_done"] is False + # The reason still reaches an operator: stdout here, and the response body + # (which the MicroVM service surfaces) in every case. + assert "[server/run-pre-config] /run hook rejected:" in capfd.readouterr().out + + def test_a_failed_payload_fetch_is_reported_without_touching_a_seam( + self, client, monkeypatch, seam_guard, capfd + ): + def boom(_uri): + raise RuntimeError("AccessDenied") + + monkeypatch.setattr(server, "_fetch_microvm_payload_from_s3", boom) + monkeypatch.setattr(server, "run_task", MagicMock()) + + r = client.post(RUN_HOOK, json=_run_hook_body({"agent_payload_s3_uri": "s3://b/k"})) + + assert r.status_code == 500 + assert r.json()["code"] == "MICROVM_RUN_PAYLOAD_UNREADABLE" + assert seam_guard["violations"] == [] + out = capfd.readouterr().out + assert "[server/run-pre-config] /run hook payload fetch FAILED" in out + # The traceback is preserved on the stdout line (it is the only diagnostic + # the response body does not carry). + assert "Traceback" in out + + @pytest.mark.parametrize( + "config,expected_code", + [ + ({"ld_preload": "/tmp/evil.so"}, "MICROVM_RUN_PLATFORM_CONFIG_INVALID"), + ({"log_group_name": "lg"}, "MICROVM_RUN_PLATFORM_CONFIG_INCOMPLETE"), + ], + ) + def test_a_rejected_platform_config_touches_no_seam( + self, client, monkeypatch, env_guard, seam_guard, config, expected_code + ): + monkeypatch.setattr(server, "run_task", MagicMock()) + + r = client.post( + RUN_HOOK, + json=_run_hook_body({"agent_payload": self._payload(), "platform_config": config}), + ) + + assert r.status_code == 400 + assert r.json()["code"] == expected_code + # Nothing was installed, so nothing earned the right to an AWS call. + assert seam_guard["violations"] == [] + assert seam_guard["install_phase_done"] is False + + def test_the_accepted_line_correlates_task_and_microvm_ids( + self, client, monkeypatch, env_guard, capfd + ): + # The pre-install "received" line is stdout-only now, so the first line that + # reaches the task's log group has to join both ids by itself. + monkeypatch.setattr(server, "run_task", MagicMock()) + monkeypatch.setattr(server.task_state, "write_terminal", MagicMock()) + + client.post( + RUN_HOOK, + json=_run_hook_body( + {"agent_payload": self._payload(), "platform_config": _platform_config()}, + microvm_id="microvm-joined", + ), + ) + + out = capfd.readouterr().out + assert "/run hook accepted task_id='t-silent' microvm_id='microvm-joined'" in out + + +class TestTerminateHookBodyTolerance: + """``/terminate`` must answer 200 for ANY body — that is why it takes the raw request. + + A Pydantic body model is validated BEFORE the handler runs, so malformed JSON, + a wrong content-type or a missing body would produce a 422 the handler never + gets to prevent: a reported hook failure on a teardown that actually succeeded. + """ + + def test_malformed_json_still_returns_200(self, client, capfd): + r = client.post( + TERMINATE_HOOK, + content=b'{"microvmId": "m-1"', # truncated + headers={"content-type": "application/json"}, + ) + assert r.status_code == 200 + assert r.json()["status"] == "acknowledged" + assert r.json()["microvm_id"] == "" + # Degraded, not silent. + assert "/terminate hook body is not JSON" in capfd.readouterr().out + + def test_a_wrong_content_type_still_returns_200(self, client): + r = client.post( + TERMINATE_HOOK, + content=b"microvmId=m-1", + headers={"content-type": "text/plain"}, + ) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + + def test_a_json_body_under_the_wrong_content_type_is_still_read(self, client): + # The handler reads bytes, so it does not care what the sender declared. + r = client.post( + TERMINATE_HOOK, + content=b'{"microvmId": "m-ct"}', + headers={"content-type": "text/plain"}, + ) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "m-ct" + + def test_an_empty_body_still_returns_200(self, client): + r = client.post(TERMINATE_HOOK, content=b"") + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + + def test_a_whitespace_only_body_still_returns_200(self, client): + r = client.post(TERMINATE_HOOK, content=b" \n ") + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + + @pytest.mark.parametrize("body", [[1, 2, 3], "a string", 42, True]) + def test_a_non_object_json_body_still_returns_200(self, client, body): + r = client.post(TERMINATE_HOOK, json=body) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + + @pytest.mark.parametrize("value", [42, None, ["m"], {"nested": "x"}]) + def test_a_non_string_microvm_id_degrades_to_empty(self, client, value): + r = client.post(TERMINATE_HOOK, json={"microvmId": value}) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + + def test_no_typed_body_model_is_left_on_the_route(self): + # Structural guard: re-introducing a Pydantic body model would silently + # restore the 422. FastAPI records body params in the route's dependant. + routes: Any = server.app.routes + route = next(r for r in routes if getattr(r, "path", None) == TERMINATE_HOOK) + assert route.dependant.body_params == [] + + def test_a_body_read_failure_still_returns_200(self, client, monkeypatch, capfd): + # e.g. the service aborts mid-body as the VM goes down. + async def boom(): + raise RuntimeError("connection reset") + + import starlette.requests + + monkeypatch.setattr(starlette.requests.Request, "body", lambda _self: boom()) + + r = client.post(TERMINATE_HOOK, json={"microvmId": "m-x"}) + assert r.status_code == 200 + assert r.json()["microvm_id"] == "" + assert "could not read its body" in capfd.readouterr().out + + +class TestParseTerminateMicrovmId: + """Unit-level: the parser degrades, never raises.""" + + def test_reads_the_service_camel_case_field(self): + assert server._parse_terminate_microvm_id(b'{"microvmId": "m-1"}') == "m-1" + + def test_tolerates_the_snake_case_spelling(self): + assert server._parse_terminate_microvm_id(b'{"microvm_id": "m-2"}') == "m-2" + + def test_camel_case_wins_when_both_are_present(self): + raw = b'{"microvmId": "camel", "microvm_id": "snake"}' + assert server._parse_terminate_microvm_id(raw) == "camel" + + @pytest.mark.parametrize( + "raw", + [ + b"", + b" ", + b"{", + b"not json", + b"[1,2,3]", + b'"a string"', + b"{}", + b'{"microvmId": null}', + b'{"microvmId": 7}', + b"\xff\xfe\x00bad utf8", + ], + ) + def test_every_unusable_body_yields_empty(self, raw): + assert server._parse_terminate_microvm_id(raw) == "" + + def test_ignores_unrelated_fields(self): + assert server._parse_terminate_microvm_id(b'{"reason": "idle", "x": 1}') == "" diff --git a/cdk/AGENTS.md b/cdk/AGENTS.md index 40fc722b3..396c326f9 100644 --- a/cdk/AGENTS.md +++ b/cdk/AGENTS.md @@ -94,4 +94,6 @@ beforeAll(() => { - **Lambda bundling in unit tests** — `Template.fromStack()` synths the stack but bundling is disabled via `CDK_CONTEXT_JSON`. Do not re-enable globally; opt in per-test with `postCliContext` only when asserting on bundle output. Details: `test/setup/disable-bundling.ts`, #366. - **Cedar engine drift** — `@cedar-policy/cedar-wasm` and `cedarpy` share a Rust core. Bump both + parity fixtures in one commit. See `docs/design/CEDAR_HITL_GATES.md` §15.6 and `mise.toml` parity banner. - **Types out of sync** — `cdk/src/handlers/shared/types.ts` and `cli/src/types.ts` must match; CI runs `check-types-sync`. +- **Trusting an L1's field type for a new service** — generated L1s for freshly launched services often type an enum field as `string` with no documented allowed values. `tsc` and every unit test will accept a wrong value; only a real change set rejects it, and change-set *early validation* fails before the stack is touched, so there is no rollback and no runtime symptom to trace back. ADR-021 P2-F2 sent four hook route paths into `AWS::Lambda::MicrovmImage.Hooks.*` (an `ENABLED`/`DISABLED` enum) and `arm64` where `ARM_64` was required, which made the whole CDK-managed image path non-functional. Validate new-service enum fields against the equivalent CLI/API call or a throwaway change set, and pin the accepted value in a construct test. +- **Statement-level bootstrap additions still need a re-bootstrap** — adding a statement to an existing `cdk/src/bootstrap/policies/*.ts` file changes the policy operators have *already deployed*, so it does not reach an account until `mise //cdk:bootstrap` runs again. Bump `cdk/src/bootstrap/version.ts` (MINOR), regenerate with `mise //cdk:bootstrap:generate` (policies JSON + template YAML + `BOOTSTRAP_HASH`/`BOOTSTRAP_VERSION`), and say "re-bootstrap to bundle ≥ x.y.z" in the operator-facing surface — otherwise the deploy fails with an AccessDenied that looks like a code bug. See `MicrovmPassRoles` (ADR-021 P2r2-F9, bundle 1.4.0). - **Un-attributed AWS SDK client (#319)** — build clients via `makeClient(Ctor, cfg)` / `makeDocClient(cfg)` from `src/handlers/shared/ua.ts`; a naked `new XxxClient({})` silently drops solution attribution. diff --git a/cdk/bootstrap/BOOTSTRAP_HASH b/cdk/bootstrap/BOOTSTRAP_HASH index 35b70e2bd..e28919cf1 100644 --- a/cdk/bootstrap/BOOTSTRAP_HASH +++ b/cdk/bootstrap/BOOTSTRAP_HASH @@ -1 +1 @@ -40d0a8b2343663084f614423fc3e6210761377dedf0bc079956ba7d4cea84c5e +99e9bd35471ed2397f8de0f97dd6fd7bc5b8d83891592448f7c2813be0a3ad0a diff --git a/cdk/bootstrap/BOOTSTRAP_VERSION b/cdk/bootstrap/BOOTSTRAP_VERSION index f0bb29e76..88c5fb891 100644 --- a/cdk/bootstrap/BOOTSTRAP_VERSION +++ b/cdk/bootstrap/BOOTSTRAP_VERSION @@ -1 +1 @@ -1.3.0 +1.4.0 diff --git a/cdk/bootstrap/bootstrap-template.yaml b/cdk/bootstrap/bootstrap-template.yaml index a2dcd08b4..419252f42 100644 --- a/cdk/bootstrap/bootstrap-template.yaml +++ b/cdk/bootstrap/bootstrap-template.yaml @@ -1,7 +1,7 @@ # GENERATED FILE - DO NOT EDIT DIRECTLY # This template is generated by: npx tsx scripts/generate-bootstrap-template.ts -# ABCA Bootstrap Policy Version: 1.3.0 -# ABCA Bootstrap Policy Hash: 40d0a8b2343663084f614423fc3e6210761377dedf0bc079956ba7d4cea84c5e +# ABCA Bootstrap Policy Version: 1.4.0 +# ABCA Bootstrap Policy Hash: 99e9bd35471ed2397f8de0f97dd6fd7bc5b8d83891592448f7c2813be0a3ad0a # # Based on the default CDK bootstrap template with the following modifications: # - BootstrapVariant set to "ABCA: Least-Privilege Bootstrap" @@ -1389,6 +1389,12 @@ Resources: Effect: Allow Resource: '*' Sid: LambdaMicrovms + - Action: iam:PassRole + Effect: Allow + Resource: + - arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild* + - arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector* + Sid: MicrovmPassRoles Version: '2012-10-17' Description: 'ABCA Bootstrap: IaCRole-ABCA-Compute-LambdaMicrovms permissions for CloudFormation execution role' Condition: IncludeComputeLambdaMicrovms @@ -1420,10 +1426,10 @@ Outputs: Value: '32' BootstrapPolicyVersion: Description: The version of the ABCA bootstrap policy bundle - Value: 1.3.0 + Value: 1.4.0 BootstrapPolicyHash: Description: SHA-256 hash of the ABCA bootstrap policy bundle for drift detection - Value: 40d0a8b2343663084f614423fc3e6210761377dedf0bc079956ba7d4cea84c5e + Value: 99e9bd35471ed2397f8de0f97dd6fd7bc5b8d83891592448f7c2813be0a3ad0a BootstrapPolicySet: Description: Comma-separated list of active ABCA bootstrap policy names Value: diff --git a/cdk/bootstrap/policies/compute-lambda-microvm.json b/cdk/bootstrap/policies/compute-lambda-microvm.json index 302a385a8..dd7656ad6 100644 --- a/cdk/bootstrap/policies/compute-lambda-microvm.json +++ b/cdk/bootstrap/policies/compute-lambda-microvm.json @@ -25,6 +25,15 @@ "Effect": "Allow", "Resource": "*", "Sid": "LambdaMicrovms" + }, + { + "Action": "iam:PassRole", + "Effect": "Allow", + "Resource": [ + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*", + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*" + ], + "Sid": "MicrovmPassRoles" } ], "Version": "2012-10-17" diff --git a/cdk/scripts/package-microvm-artifact.sh b/cdk/scripts/package-microvm-artifact.sh index 526e40f70..fc73aae67 100755 --- a/cdk/scripts/package-microvm-artifact.sh +++ b/cdk/scripts/package-microvm-artifact.sh @@ -78,14 +78,14 @@ # contracts/ <- cross-language constants the agent reads at runtime # # --------------------------------------------------------------------------- -# !! A P1 IMAGE IS RUNNABLE, BUT NOT SMOKE-VERIFIED !! +# !! A P2 IMAGE IS FULLY WIRED, BUT NOT SMOKE-VERIFIED !! # --------------------------------------------------------------------------- # This script packages and uploads a real artifact, and the image the service # builds from it will reach ACTIVE, accept a `runHookPayload`, and launch. What # it does NOT have is any smoke-parity guarantee. # # ADR-021 sub-decision 3's hook-phasing table (corrected after the live P1 -# verification run) is now: +# verification run, then completed in P2) is now: # # /ready, /run declared by the CDK construct AND served by # the agent in P1 (agent/src/server.py). @@ -94,15 +94,29 @@ # image with no hooks at all cannot receive a # runHookPayload — so "declare /run in P1, # serve it in P2" was never a reachable state. -# /validate P2 (a /validate that 404s fails every build) -# /terminate P2; /suspend, /resume P3 +# /validate, /terminate declared AND served in P2. /validate is a +# build-time self-check that makes ZERO AWS +# calls (it runs under the build role, which +# holds no Bedrock/Secrets/DynamoDB grants); +# /terminate is a best-effort in-guest +# breadcrumb that must not write terminal task +# status — the orchestrator finalizes the task +# and THEN calls TerminateMicrovm. +# /suspend, /resume P3. A hook the service calls but nothing +# answers fails its lifecycle transition, so +# each is enabled only once it is served. # -# Still unverified and owned by P2: AgentCore Memory grants + MEMORY_ID delivery, -# the agent's non-secret env parity inside the snapshot, egress specifics from a -# running MicroVM, and heartbeat/progress behaviour end to end. So clone → change -# → PR on this backend is untested. Keep production repos on -# compute_type=agentcore or ecs until P2 (smoke parity) lands. The Dockerfile is -# copied unmodified deliberately — adapting it to a MicroVM base image is P2 work. +# A P2 smoke run HAS now completed clone → change → PR on this substrate +# (2026-08-07: two tasks COMPLETED with pull requests, live progress events, and +# the 45 s agent heartbeat observed). What is still missing is a run with NO manual +# intervention: that smoke needed a live IAM workaround, and the two defects behind +# it (ADR-021 P2r2-F9 / P2r2-F10 — the `iam:PassedToService` condition on both +# `iam:PassRole` paths) are fixed in source but not yet re-exercised live. Keep +# production repos on compute_type=agentcore or ecs until a clean run is on record, +# and note that the CDK-managed image path additionally needs bootstrap policy +# bundle >= 1.4.0 (see the banner after upload). The Dockerfile is the P2 tuned base +# and is copied unmodified — further customization (e.g., Alpine adoption) would be +# P2.5 work. # # Requires: awscli v2, zip, rsync, python3 (none of which are installed by this script). @@ -251,15 +265,18 @@ echo " log group : ${LOG_GROUP}" print_p1_reminder() { cat <<'EOF' -!! REMINDER (ADR-021 P1): a P1 image is runnable but NOT smoke-verified. - The image IS creatable and launchable and the agent DOES serve /ready + /run, - so a lambda-microvm task can start and receive its payload. NOT verified: - AgentCore Memory grants + MEMORY_ID delivery, the agent's non-secret env - parity inside the snapshot, egress specifics from a running MicroVM, and - heartbeat/progress behaviour. clone -> change -> PR on this backend is - untested. Keep production repos on compute_type=agentcore or ecs until P2 - (smoke parity) lands. CDK synth emits the same warning - (abca:microvm-image-p1-smoke-unverified) on every deploy that configures an image. +!! REMINDER (ADR-021 P2): smoke-verified ONCE, and only WITH a manual workaround. + The image is creatable and launchable, the agent serves all four declared hooks + (/ready + /validate on the build path, /run + /terminate at runtime), the + execution role holds its full runtime permission set, and a 2026-08-07 run took + two tasks clone -> change -> PR to COMPLETED with a live 45 s heartbeat. + NOT verified: an UNATTENDED run. That smoke needed a live IAM workaround, and the + two defects behind it (ADR-021 P2r2-F9 / P2r2-F10) are fixed in source but not + re-exercised. The CDK-managed image path also needs bootstrap bundle >= 1.4.0. + Keep production repos on compute_type=agentcore or ecs until a clean run is on + record. /suspend and /resume stay disabled until P3. CDK synth emits the same + warning (abca:microvm-image-p1-smoke-unverified) on every deploy that configures + an image. EOF } @@ -309,7 +326,28 @@ if [[ "${CREATE_IMAGE}" -eq 0 ]]; then ==> Artifact uploaded. Next: create (or update) the image. - CDK-managed (recommended) — redeploy with the base image pinned: + CDK-managed (recommended) — redeploy with the base image pinned. + + !! RE-BOOTSTRAP REQUIRED (bootstrap policy bundle >= 1.4.0) !! + This path took two live-verified fixes to work. The first (ADR-021 P2-F2: the L1 + sent hook paths and \`arm64\` where CloudFormation wants ENABLED / ARM_64) is + DISCHARGED — change-set early validation now passes. The second (ADR-021 + P2r2-F9) is a BOOTSTRAP change: CloudFormation could not pass the MicroVM build + role, because the deploy role's \`iam:PassRole\` carried an + \`iam:PassedToService\` condition the Lambda MicroVMs service does not satisfy. + The fix is the \`MicrovmPassRoles\` statement in the conditional + IaCRole-ABCA-Compute-LambdaMicrovms policy, which only reaches your account when + you re-bootstrap: + + aws cloudformation describe-stacks --stack-name CDKToolkit \\ + --query 'Stacks[0].Outputs[?OutputKey==\`BootstrapPolicyVersion\`].OutputValue' --output text + # if that is below 1.4.0: + MISE_EXPERIMENTAL=1 mise //cdk:bootstrap # ComputeTypes must include lambda-microvm + + Without it the image resource fails with + "is not authorized to perform: iam:PassRole on resource: + ...LambdaMicrovmComputeBuildRole... (Service: LambdaMicrovms, Status Code: 403)". + Then: aws lambda-microvms list-managed-microvm-images MISE_EXPERIMENTAL=1 mise //cdk:deploy -- \\ @@ -344,13 +382,48 @@ echo "==> Creating MicroVM image '${IMAGE_NAME}' (${MEMORY_MIB} MiB baseline)" # Flags use the Lambda MicroVMs service API shape (which differs from the # CloudFormation shape used by CfnMicrovmImage), all confirmed against the live # CLI/SDK model on 2026-07-31: -# * `ARM_64` is the only documented architecture value. +# * `ARM_64` is the only accepted architecture value — on BOTH surfaces (see +# below); `arm64` is rejected. # * hooks are ENABLED/DISABLED with timeouts, NOT paths. +# * CORRECTION (live 2026-08-06, ADR-021 P2-F2): this comment used to claim the +# hook/architecture fields were "the ONE place the two shapes genuinely +# disagree" — that `CfnMicrovmImage` routes on a path string while this API +# takes a flag, and that both were correct for their own surface. That was +# WRONG, and it made the CDK-managed image path non-functional. CloudFormation +# enforces the SAME enums at change-set early validation, and rejected all +# five values the construct was sending: +# "/aws/lambda-microvms/runtime/v1/run is not a valid enum value. Supported +# values: [DISABLED, ENABLED]" (x4, one per hook) +# "arm64 is not a valid enum value. Supported values: [ARM_64]" +# The L1 types them as plain strings and documents no allowed values, which is +# how the wrong shape survived synth, unit tests and cdk-nag. The construct now +# sends ENABLED / ARM_64, i.e. exactly what this script always sent. Neither +# surface takes a hook path at all: the service calls fixed well-known routes, +# which the agent serves and `MICROVM_AGENT_HOOK_ROUTES` in +# `cdk/src/constructs/lambda-microvm-compute.ts` records for that purpose only. # * `/ready` is MANDATORY whenever any lifecycle hook is enabled: # "The ready (/ready) MicroVM image hook must be enabled when any MicroVM # lifecycle hook (run, resume, suspend, or terminate) is enabled." -# `/validate` stays disabled — the agent serves no validation endpoint, and a -# 404 there fails every build. +# * all four hooks the agent serves are enabled: `/ready` + `/validate` (build) +# and `/run` + `/terminate` (runtime). `/suspend` and `/resume` stay DISABLED +# until P3 implements them — a hook the service calls but nothing answers +# fails the corresponding build or lifecycle transition. +# * the timeouts mirror the construct's constants +# (`RUN_/READY_/VALIDATE_/TERMINATE_HOOK_TIMEOUT_SECONDS` in +# `cdk/src/constructs/lambda-microvm-compute.ts`), which carry the rationale +# for each value. A bash helper cannot import them, and "keep the two in step" +# as prose already FAILED once — `readyTimeoutInSeconds` stayed at 60 here when +# the construct went to 300 — so the invariant is now enforced by a unit test +# that parses this exact `--hooks` string and compares it against the +# synthesized template (`cdk/test/constructs/lambda-microvm-compute.test.ts`, +# "the out-of-band script's API request matches the CDK-managed image"). If that +# test fails, one of the two moved: fix the one that is wrong, do not relax the +# test. An out-of-band image built here must behave like a CDK-built one. +# * `readyTimeoutInSeconds` is 300, NOT 60: as of ADR-021 P2-F5 the `/ready` hook +# warms the 225 MiB `claude` binary before the snapshot is captured, so it does +# real work whose duration is a cold `exec`. Build hooks are allowed up to +# 3600 s. `validateTimeoutInSeconds` deliberately stays at 60 — /validate gained +# no warm-up, so sharing a number would size it for work it does not do. # * the BUILD connector (443 + 80) is used here, not the runtime one. CREATE_RESPONSE="$(aws lambda-microvms create-microvm-image \ --name "${IMAGE_NAME}" \ @@ -363,7 +436,7 @@ CREATE_RESPONSE="$(aws lambda-microvms create-microvm-image \ --resources "[{\"minimumMemoryInMiB\":${MEMORY_MIB}}]" \ --egress-network-connectors "${BUILD_EGRESS_CONNECTORS}" \ --logging "{\"cloudWatch\":{\"logGroup\":\"${LOG_GROUP}\"}}" \ - --hooks '{"port":8080,"microvmHooks":{"run":"ENABLED","runTimeoutInSeconds":60},"microvmImageHooks":{"ready":"ENABLED","readyTimeoutInSeconds":60}}' \ + --hooks '{"port":8080,"microvmHooks":{"run":"ENABLED","runTimeoutInSeconds":60,"terminate":"ENABLED","terminateTimeoutInSeconds":15},"microvmImageHooks":{"ready":"ENABLED","readyTimeoutInSeconds":300,"validate":"ENABLED","validateTimeoutInSeconds":60}}' \ --tags "abca:compute-backend=lambda-microvm" \ --output json)" diff --git a/cdk/src/bootstrap/policies/compute-lambda-microvm.ts b/cdk/src/bootstrap/policies/compute-lambda-microvm.ts index 8e1c120d6..122de1fa4 100644 --- a/cdk/src/bootstrap/policies/compute-lambda-microvm.ts +++ b/cdk/src/bootstrap/policies/compute-lambda-microvm.ts @@ -40,10 +40,10 @@ import { aws_iam as iam } from 'aws-cdk-lib'; * would silently never match and the deploy would fail with AccessDenied. * * Dependent actions that are already covered elsewhere in the bundle and are - * therefore not repeated: `iam:PassRole` (the build role, → - * `lambda.amazonaws.com`) and `iam:CreateServiceLinkedRole` (network-connector - * ENI management) live in the `infrastructure` policy; `lambda:TagResource` / - * `lambda:UntagResource` live in `application`. + * therefore not repeated: `iam:CreateServiceLinkedRole` (network-connector ENI + * management) lives in the `infrastructure` policy; `lambda:TagResource` / + * `lambda:UntagResource` live in `application`. `iam:PassRole` USED to be in that + * list — see the second statement below for why it is not. */ export function computeLambdaMicrovmPolicy(): iam.PolicyDocument { return new iam.PolicyDocument({ @@ -87,6 +87,76 @@ export function computeLambdaMicrovmPolicy(): iam.PolicyDocument { ], resources: ['*'], }), + + // --- iam:PassRole for the two MicroVM roles CloudFormation hands to the + // service, WITHOUT a service condition (ADR-021 P2r2-F9) --- + // + // Why this statement exists at all, when `infrastructure`'s `IAMPassRole` + // already covers `role/backgroundagent-dev-*`: that statement carries a + // `iam:PassedToService` allowlist, and the Lambda MicroVMs service does not + // present a usable value for that key. Live 2026-08-07 (run 2), the + // CDK-managed image path died on exactly this, one step past the enum fix + // that unblocked change-set validation: + // + // LambdaMicrovmComputeImage… CREATE_FAILED + // "User: …/cdk-hnb659fds-cfn-exec-role-…/AWSCloudFormation is not + // authorized to perform: iam:PassRole on resource: + // …role/backgroundagent-dev-LambdaMicrovmComputeBuildRoleF0-… because no + // identity-based policy allows the iam:PassRole action + // (Service: LambdaMicrovms, Status Code: 403)" + // + // Three pieces of evidence pin it to the CONDITION rather than to a stale + // bootstrap or a wrong resource pattern: + // 1. the live `IaCRole-ABCA-Infrastructure` policy was byte-identical to + // this branch's `bootstrap/policies/infrastructure.json`, so + // `bootstrap --force` would have changed nothing; + // 2. `simulate-principal-policy` on the deploy role returned `allowed` WITH + // `iam:PassedToService=lambda.amazonaws.com` supplied and `implicitDeny` + // with no context — so the resource pattern matches and the condition is + // the only remaining variable; + // 3. the CONTROL: the out-of-band `create-microvm-image` call passed **the + // same build role** to the same service successfully, using operator + // credentials that carry no such condition. So the role's trust is fine + // and the denial is genuinely caller-side. + // + // This is the CloudFormation-side twin of P2r2-F10 (the orchestrator's + // `RunMicrovm` PassRole, `task-orchestrator.ts`): one root cause — the + // service presents no usable `iam:PassedToService` — with two symptoms, one + // per PassRole path. + // + // WHY HERE rather than editing `infrastructure`'s `IAMPassRole`: + // - this policy is CONDITIONAL on `ComputeTypes` including + // `lambda-microvm` (template condition `IncludeComputeLambdaMicrovms`), so + // an agentcore-only or ECS-only bootstrap gains nothing — the + // unconditioned pass simply does not exist there; + // - the existing allowlisted statement stays untouched, so every other role + // in the stack keeps its `iam:PassedToService` constraint. Relaxing the + // shared statement would have dropped that constraint for ~30 roles to + // fix two. + // + // SCOPE. Two name-prefix patterns, not `role/backgroundagent-dev-*`, and + // deliberately NOT the execution role — CloudFormation never passes that one + // (the orchestrator does, at `RunMicrovm`), so including it here would widen + // the unconditioned pass to the role that runs untrusted repo code for no + // reason. The patterns match the physical names CloudFormation generates from + // the construct's logical ids (`LambdaMicrovmComputeBuildRole…`, + // `LambdaMicrovmComputeConnectorOperatorRole…`), which it truncates to fit + // 64 characters before appending a random suffix — verified against the live + // ARNs. If a future rename or a longer stack name pushed the discriminating + // part out of that window, the failure is the loud AccessDenied above naming + // the exact ARN, not a silent widening. + new iam.PolicyStatement({ + sid: 'MicrovmPassRoles', + effect: iam.Effect.ALLOW, + actions: ['iam:PassRole'], + resources: [ + // Passed as `buildRoleArn` on AWS::Lambda::MicrovmImage. + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*', + // Passed as `operatorRole` on AWS::Lambda::NetworkConnector (required + // for VPC_EGRESS connectors). + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*', + ], + }), ], }); } diff --git a/cdk/src/bootstrap/resource-action-map.ts b/cdk/src/bootstrap/resource-action-map.ts index cb117290f..a1c829fea 100644 --- a/cdk/src/bootstrap/resource-action-map.ts +++ b/cdk/src/bootstrap/resource-action-map.ts @@ -85,8 +85,19 @@ export const RESOURCE_ACTION_MAP: Record = { // `--context compute_type=lambda-microvm`, so the default-context // synth-coverage test never sees these — they are mapped anyway so the map // stays a complete statement of what the bootstrap bundle must cover. - 'AWS::Lambda::MicrovmImage': ['lambda:CreateMicrovmImage'], - 'AWS::Lambda::NetworkConnector': ['lambda:CreateNetworkConnector'], + // + // `iam:PassRole` is listed on BOTH because CloudFormation hands a role to the + // Lambda MicroVMs service for each: `buildRoleArn` on the image and + // `operatorRole` on the (VPC_EGRESS) connector. Its absence here is what let + // ADR-021 P2r2-F9 through — the bundle's shared `IAMPassRole` statement carries + // an `iam:PassedToService` allowlist that this service presents no usable value + // for, so the deploy failed with `iam:PassRole … not authorized` on the build + // role while every mapped action was covered. The unconditioned pass now lives in + // the conditional `compute-lambda-microvm` policy; listing the action here is + // what makes its removal a test failure instead of a redeploy failure. Evidence + // inlined in ADR-021 §4. + 'AWS::Lambda::MicrovmImage': ['lambda:CreateMicrovmImage', 'iam:PassRole'], + 'AWS::Lambda::NetworkConnector': ['lambda:CreateNetworkConnector', 'iam:PassRole'], 'AWS::Logs::Delivery': ['logs:CreateDelivery'], 'AWS::Logs::DeliveryDestination': ['logs:PutDeliveryDestination'], 'AWS::Logs::DeliverySource': ['logs:PutDeliverySource'], diff --git a/cdk/src/bootstrap/version.ts b/cdk/src/bootstrap/version.ts index 17ad37ae7..bd2a1050a 100644 --- a/cdk/src/bootstrap/version.ts +++ b/cdk/src/bootstrap/version.ts @@ -25,11 +25,20 @@ import { allPolicies } from './policies'; * Semantic version of the bootstrap policy bundle. * * Bump history: 1.0.0 → 1.1.0 added the `compute-ecs` policy (#162), 1.1.0 → - * 1.2.0 refreshed policies for a full deploy (#350), 1.2.0 → 1.3.0 adds the - * `compute-lambda-microvm` policy (#645 / ADR-021). Adding a policy to the - * bundle is a minor bump — that is the precedent `compute-ecs` set. + * 1.2.0 refreshed policies for a full deploy (#350), 1.2.0 → 1.3.0 added the + * `compute-lambda-microvm` policy (#645 / ADR-021), 1.3.0 → 1.4.0 adds that + * policy's `MicrovmPassRoles` statement (#645, ADR-021 P2r2-F9). + * + * On the 1.4.0 bump specifically: it is a *statement* addition, not a new policy, + * and it is still a MINOR bump for the reason 1.2.0 was — **an operator must + * re-bootstrap to pick it up**, and the version is the only signal that says so. + * Without it the CDK-managed MicroVM image path fails at deploy with a caller-side + * `iam:PassRole` AccessDenied on the build role (live-verified; see + * `policies/compute-lambda-microvm.ts`), which is exactly the class of breakage a + * patch-level bump would under-advertise. Adding a whole policy remains a minor + * bump too — the precedent `compute-ecs` set. */ -export const BOOTSTRAP_VERSION = '1.3.0'; +export const BOOTSTRAP_VERSION = '1.4.0'; /** * Computes a SHA-256 hash over all bootstrap policies. diff --git a/cdk/src/constructs/bedrock-models.ts b/cdk/src/constructs/bedrock-models.ts index 9f3d34932..6e396cde2 100644 --- a/cdk/src/constructs/bedrock-models.ts +++ b/cdk/src/constructs/bedrock-models.ts @@ -19,6 +19,24 @@ import { Node } from 'constructs'; +/** + * The small/fast model the agent uses for cheap side-calls, as a BARE + * foundation-model id. + * + * Named separately from {@link DEFAULT_BEDROCK_MODEL_IDS} because it has a second + * consumer: the agent needs it as a runtime *value* + * (`ANTHROPIC_DEFAULT_HAIKU_MODEL`), not just as an IAM grant. Splicing it out of + * that list means the granted model and the delivered model id cannot drift — a + * mismatch would AccessDenied every Haiku call at run time while synth stayed + * green. + * + * Declared ABOVE the list rather than beside its inference-profile sibling below + * it because the list interpolates it: a `const` referenced before its + * declaration is a TDZ `ReferenceError` at module load, and re-inlining the + * literal into the list is exactly the drift this constant exists to prevent. + */ +export const DEFAULT_HAIKU_MODEL_ID = 'anthropic.claude-haiku-4-5-20251001-v1:0'; + /** * Single source of truth for the Bedrock **foundation-model IDs** the agent * runtime may invoke. Both grant sites — the AgentCore runtime in @@ -41,9 +59,21 @@ export const DEFAULT_BEDROCK_MODEL_IDS: readonly string[] = [ // this entry and that default in the same change — a fallback the role cannot // invoke fails every task on the stack, not just an edge case. 'anthropic.claude-opus-4-8', - 'anthropic.claude-haiku-4-5-20251001-v1:0', + DEFAULT_HAIKU_MODEL_ID, ]; +/** + * `ANTHROPIC_DEFAULT_HAIKU_MODEL` value delivered to the agent on every backend + * (AgentCore runtime env, and `platform_config` for lambda-microvm). + * + * The **cross-region inference-profile** id, not the bare foundation-model id: + * Claude 4.x cannot be invoked on-demand by bare id (400 "on-demand throughput + * isn't supported"). The `us.` prefix matches how both grant sites derive their + * inference-profile ARNs, so the value is always one of the granted profiles. + * (`agent/src/runner.py` re-sets this at spawn time from the same value.) + */ +export const DEFAULT_HAIKU_INFERENCE_PROFILE_ID = `us.${DEFAULT_HAIKU_MODEL_ID}`; + /** CDK context key whose value (a string array) overrides the model set. */ export const BEDROCK_MODELS_CONTEXT_KEY = 'bedrockModels'; diff --git a/cdk/src/constructs/lambda-microvm-compute.ts b/cdk/src/constructs/lambda-microvm-compute.ts index d7590a2c4..ec40f9ae4 100644 --- a/cdk/src/constructs/lambda-microvm-compute.ts +++ b/cdk/src/constructs/lambda-microvm-compute.ts @@ -23,15 +23,25 @@ import * as iam from 'aws-cdk-lib/aws-iam'; import * as lambda from 'aws-cdk-lib/aws-lambda'; import * as logs from 'aws-cdk-lib/aws-logs'; import * as s3 from 'aws-cdk-lib/aws-s3'; +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager'; import { NagSuppressions } from 'cdk-nag'; import { Construct } from 'constructs'; +// Cross-language contract (S9): `microvm_hook_budgets` couples THIS construct's +// `/ready` hook timeout to the agent's own warm-up ceiling in +// `agent/src/server.py`. Imported (not copied) so `tsc` fails on a renamed field, +// and `scripts/check-constants-sync.ts` enforces the `warmup_total < ready_hook` +// invariant plus the no-literal-redeclaration rule on both sides. See +// `contracts/constants.md`. // Single source of truth for the supported-Region list. ADR-021's // `microvm-regions.ts` header is explicit that the list "is the ONLY place the // list is declared — do not copy it", so the synth-time gate IMPORTS it rather // than duplicating it. The module is a dependency-free pair of pure constants // (no AWS SDK, no Lambda-runtime code), so pulling it into the CDK app tree // costs nothing and cannot drift. +import { AgentMemory } from './agent-memory'; import { AgentSessionRole } from './agent-session-role'; +import { resolveBedrockModelIds } from './bedrock-models'; +import sharedConstants from '../../../contracts/constants.json'; import { LAMBDA_MICROVM_SUPPORTED_REGIONS, isLambdaMicrovmRegionSupported } from '../handlers/shared/microvm-regions'; /** @@ -81,34 +91,82 @@ export const MICROVM_ARTIFACT_OBJECT_KEY = 'microvm-images/agent-artifact.zip'; const AGENT_HOOK_PORT = 8080; /** - * `/run` hook path, as SERVED by `agent/src/server.py`. + * Value every hook field on `AWS::Lambda::MicrovmImage` takes to turn a hook ON. * - * Load-bearing in P1: it is how the task payload reaches the agent - * (`runHookPayload`, ADR-021 sub-decision 3) — there is no other - * orchestrator→agent channel on this backend. + * **The field is an ENUM, not a path** — `[DISABLED, ENABLED]`. The CDK L1 types + * `hooks.microvmHooks.run` and its three siblings as plain `string` and documents + * no allowed values, which is why this construct originally sent the agent's + * route there. CloudFormation **rejected all four at change-set early validation** + * (live 2026-08-06, ADR-021 P2-F2 — the stack was never touched, so there was no + * rollback to read): * - * The value is the service's fixed lifecycle-hook route, not a path of our - * choosing. Live verification (2026-07-31, issue #645) found that the - * `CreateMicrovmImage` **API** model has no hook-path field at all — it takes - * only `ENABLED`/`DISABLED` plus a timeout — while the generated - * CloudFormation type still types these as strings. Setting the string to the - * route the agent actually answers is therefore correct under either reading: - * if CloudFormation genuinely routes on it, it points at the real endpoint; if - * it is ignored (as the API model implies), the value is inert. Keep in lockstep - * with `MICROVM_HOOK_PREFIX` in `agent/src/server.py`. + * > /aws/lambda-microvms/runtime/v1/run is not a valid enum value. Supported + * > values: [DISABLED, ENABLED] (at + * > /Resources/…/Properties/Hooks/MicrovmHooks/Run) + * + * So the CloudFormation surface is IDENTICAL to the `CreateMicrovmImage` API + * surface (`--hooks '{"microvmHooks":{"run":"ENABLED",…}}'`), not different from + * it as the previous comment here claimed. There is no hook-path field on either + * surface: the service calls fixed well-known routes, which + * {@link MICROVM_AGENT_HOOK_ROUTES} records and the live build/run logs confirm. + * + * `DISABLED` is never emitted: a hook the agent does not serve is OMITTED rather + * than disabled, so the exactness test can assert the declared set in both + * directions (see `/suspend` + `/resume`, P3). + */ +const HOOK_ENABLED = 'ENABLED'; + +/** + * Route prefix the MicroVM service POSTs its lifecycle hooks to, and the prefix + * the agent mounts them under (`MICROVM_HOOK_PREFIX` in `agent/src/server.py`). + */ +const MICROVM_HOOK_ROUTE_PREFIX = '/aws/lambda-microvms/runtime/v1'; + +/** + * The service's fixed hook ROUTES, keyed by hook name — the paths + * `agent/src/server.py` must serve. + * + * ## ⚠️ These are AGENT ROUTE CONSTANTS ONLY. Never send them to an AWS API. + * + * They were previously passed as the `hooks.*` property VALUES on the L1, on the + * reasoning that the generated CloudFormation type accepts strings and documents + * no allowed-value constraint. CloudFormation refused every one of them + * (P2-F2 — see {@link HOOK_ENABLED} for the verbatim early-validation output): + * the hook fields are `ENABLED`/`DISABLED` enums on BOTH the CloudFormation and + * the API surface, and neither surface accepts a path at all. + * + * The routes are still worth declaring here because they are a CROSS-PACKAGE + * CONTRACT that nothing else in the CDK tree records: the service POSTs to these + * exact paths (live 2026-08-06 — `"POST /aws/lambda-microvms/runtime/v1/ready + * HTTP/1.1" 200 OK`, and the same for `/validate`, `/run` and `/terminate`), so a + * prefix drift between this map and the agent's `MICROVM_HOOK_PREFIX` surfaces as + * a failed image build (`/ready`, `/validate`) or a failed lifecycle transition on + * a real task (`/run`, `/terminate`). The construct test compares THIS map — not + * the rendered template, which no longer contains a path — against the routes the + * agent serves. */ -const RUN_HOOK_PATH = '/aws/lambda-microvms/runtime/v1/run'; +export const MICROVM_AGENT_HOOK_ROUTES = { + ready: `${MICROVM_HOOK_ROUTE_PREFIX}/ready`, + validate: `${MICROVM_HOOK_ROUTE_PREFIX}/validate`, + run: `${MICROVM_HOOK_ROUTE_PREFIX}/run`, + terminate: `${MICROVM_HOOK_ROUTE_PREFIX}/terminate`, +} as const; -/** `/run` hook budget (seconds). The hook only validates + starts the pipeline - * asynchronously, so it stays well inside the service's 1–60 s hook window. */ +/** + * `/run` runtime-hook budget (seconds). + * + * `/run` is how the task payload reaches the agent (`runHookPayload`, ADR-021 + * sub-decision 3) — there is no other orchestrator→agent channel on this + * backend. The hook only validates + starts the pipeline asynchronously, so it + * stays well inside the service's 1–60 s runtime-hook window. + */ const RUN_HOOK_TIMEOUT_SECONDS = 60; /** - * `/ready` build hook path, as SERVED by `agent/src/server.py` (see - * {@link RUN_HOOK_PATH} for why the string is the real route). + * `/ready` build-hook budget (seconds). * - * **Mandatory, not optional.** `CreateMicrovmImage` rejects an image that - * enables ANY lifecycle hook without `/ready` (live 2026-07-31): + * `/ready` is **mandatory, not optional**: `CreateMicrovmImage` rejects an image + * that enables ANY lifecycle hook without it (live 2026-07-31): * * > The ready (/ready) MicroVM image hook must be enabled when any MicroVM * > lifecycle hook (run, resume, suspend, or terminate) is enabled. The ready @@ -116,13 +174,93 @@ const RUN_HOOK_TIMEOUT_SECONDS = 60; * > is taken in a ready state. * * So ADR-021's original "declare `/run` in P1, serve it in P2" plan was not a - * reachable service state: `/ready` + `/run` now land together in P1. + * reachable service state: `/ready` + `/run` land together in P1. + * + * **The budget is 300 s, not 60 s, because as of the P2-F5 fix `/ready` does real + * work.** It no longer just reports that uvicorn is bound: it warms the 225 MiB + * `claude` binary (`agent/src/server.py` → `_warm_snapshot_binaries`) so the + * binary's pages are resident when the snapshot is taken, instead of being faulted + * in lazily on the first task and blowing a timeout there (the defect that failed + * every P2 smoke task at turn 0). A cold 225 MiB `exec` is the one thing on this + * path that can plausibly take tens of seconds, and the build hook window allows + * up to 3600 s, so a 60 s budget would trade the runtime failure for a build + * failure. It stays far below the service ceiling: the cost of a too-generous + * budget is only how long the service waits before calling a permanently-wedged + * snapshot broken. + * + * The number is chosen against the agent's own warm-up ceiling, not guessed — and + * it is not declared here either. Both this budget and the agent's + * `_READY_WARMUP_TOTAL_BUDGET_SECONDS` come from `contracts/constants.json` → + * `microvm_hook_budgets`, because the relationship between them is the invariant + * that matters and a relationship cannot be enforced from one side. The agent's + * ceiling bounds the WHOLE warm-up (required command + every best-effort one, + * which share the remainder) at 240 s, leaving ~60 s here for uvicorn scheduling + * and the request itself. Per-command timeouts deliberately do NOT compose on the + * agent side — three commands at 120 s each would be 360 s and would blow this + * budget, turning a fix for a runtime failure into a build failure. + * `scripts/check-constants-sync.ts` fails the build if the contract ever stops + * satisfying `warmup_total < ready_hook`, so the two numbers cannot drift apart in + * a single-sided edit. + */ +const READY_HOOK_TIMEOUT_SECONDS = sharedConstants.microvm_hook_budgets.ready_hook_timeout_seconds; + +/** + * `/validate` build-hook budget (seconds). + * + * `/validate` is declared as of P2, when the agent started serving it. What it + * asserts is narrower than ADR-021 first sketched, and the narrowing is a + * consequence of THIS construct's IAM: it runs during the image build under + * {@link LambdaMicrovmCompute.buildRole}, which holds only `s3:GetObject` on the + * artifact plus log writes. So the "deeper warm-up assertions" (Bedrock + * reachability, Memory access, tool availability) are not implementable here — + * each would `AccessDenied` and fail every build. The agent's hook is therefore an + * in-process self-check (server alive, every declared hook route registered, + * interpreter floor, cross-package `platform_config` contract loaded), which is + * exactly the class of failure a build hook CAN catch: a typo'd hook prefix would + * otherwise surface as a failed lifecycle transition on the first real task + * instead of as a failed build. + * + * The checks themselves are sub-millisecond (no AWS calls, no I/O beyond a stdout + * line), so the budget is not sized for the work: it is sized for the + * still-initialising path, where the agent answers **503** until module import + * completes. 60 s covers that with orders of magnitude to spare. This used to be + * an alias for {@link READY_HOOK_TIMEOUT_SECONDS} on the argument that one number + * should cover both build hooks; the two DECOUPLED when `/ready` gained the + * binary warm-up (P2-F5) and `/validate` did not, so sharing a number would now + * mean sizing `/validate` for work it does not do. Set explicitly rather than + * relying on the service's 30 s default: a permanently failing check SHOULD fail + * the image build, and the budget is what decides how long the service waits + * before calling it that. */ -const READY_HOOK_PATH = '/aws/lambda-microvms/runtime/v1/ready'; +const VALIDATE_HOOK_TIMEOUT_SECONDS = 60; -/** `/ready` build-hook budget (seconds). The agent answers as soon as uvicorn is - * bound, so the snapshot is taken with a warm server. */ -const READY_HOOK_TIMEOUT_SECONDS = 60; +/** + * `/terminate` runtime-hook budget (seconds). + * + * `/terminate` is declared as of P2. It is a log-and-acknowledge breadcrumb, NOT + * a shutdown mechanism: the orchestrator finalizes the task and *then* calls + * `TerminateMicrovm`, so the hook must not write terminal task status (it would + * race the finalization it follows) and must not join the pipeline thread. Its + * value is the last structured line in the task's log group from inside the guest. + * + * This is the one hook where a GENEROUS budget buys nothing and costs something. + * There is nothing to drain — `_ProgressWriter` does a synchronous `put_item` per + * event, so every progress write is already durable when this hook is called — + * and the handler never joins the pipeline thread, so it completes in + * milliseconds by construction. Meanwhile the budget bounds how long teardown + * waits on a guest that is WEDGED, and a MicroVM that has not finished + * terminating is still holding the account memory quota that gates admission for + * everyone else. + * + * So this is set near the bottom of the service's 1–60 s window rather than at + * it: 15 s is ~three orders of magnitude above the measured work, which absorbs + * a scheduling delay on a guest still saturated by a build (the realistic reason + * a fast handler answers slowly), while keeping teardown prompt. Exceeding it + * costs only a reported hook failure — the task is already finalized and + * `TerminateMicrovm` removes the VM regardless — which is why erring tight is + * the safe direction here and erring generous is not. + */ +const TERMINATE_HOOK_TIMEOUT_SECONDS = 15; /** * BASELINE memory sizes (MiB) the service accepts for a MicroVM image. @@ -190,8 +328,18 @@ const HTTPS_PORT = 443; */ const HTTP_PORT = 80; -/** Graviton/ARM64: the agent image is ARM64 on every backend. */ -const CPU_ARCHITECTURE = 'arm64'; +/** + * Graviton/ARM64: the agent image is ARM64 on every backend. + * + * The value is the service's **enum member spelling**, `ARM_64` — not the + * lowercase `arm64` Docker/CDK use elsewhere. The CDK L1 types + * `cpuConfigurations[].architecture` as a plain `string` and documents no allowed + * values, and `arm64` was rejected at change-set early validation (live + * 2026-08-06, ADR-021 P2-F2): *"arm64 is not a valid enum value. Supported + * values: [ARM_64]"*. Matches `--cpu-configurations '[{"architecture":"ARM_64"}]'` + * in `cdk/scripts/package-microvm-artifact.sh`, which had it right all along. + */ +const CPU_ARCHITECTURE = 'ARM_64'; /** * Resource-name half of the Lambda-managed **`NO_INGRESS`** network connector @@ -351,6 +499,60 @@ export interface LambdaMicrovmComputeProps extends LambdaMicrovmImageInputs { */ readonly agentSessionRole?: AgentSessionRole; + /** + * GitHub PAT secret. When provided, the MicroVM **execution role** gets + * `grantRead` on it. + * + * This grant stays on the execution role rather than moving to the SessionRole + * because of WHEN it is used: the agent resolves the token at startup, before + * it has assumed the SessionRole — the same ordering that keeps the grant on the + * ECS task role and the AgentCore runtime role. Without it the MicroVM cannot + * clone, push, or open a PR, which is the whole task. + * + * Omitted in isolated construct tests → no grant. + */ + readonly githubTokenSecret?: secretsmanager.ISecret; + + /** + * AgentCore Memory for cross-task learning. When provided, the execution role + * gets read+write so the agent's `write_task_episode` / `write_repo_learnings` + * (`bedrock-agentcore:CreateEvent`) succeed on this substrate. + * + * Exactly the prop `EcsAgentCluster` takes, for exactly the same reason: the + * `MEMORY_ID` the agent receives (in `agent_payload`, unchanged by ADR-021 P2) + * makes it ATTEMPT the write, and without the grant that attempt fails closed on + * AccessDenied. `memory.py` treats that as an infra failure — logged, + * non-fatal — so learning would silently never persist on a MicroVM-only + * deployment. Omitted in isolated construct tests / memory-less deployments. + */ + readonly agentMemory?: AgentMemory; + + /** + * The platform's APPLICATION_LOGS group — the same log group whose NAME travels + * to the guest as `platform_config.log_group_name` (`stacks/agent.ts` → + * `TaskOrchestrator.agentPlatformConfig` → `LOG_GROUP_NAME`). When provided, the + * MicroVM **execution role** gets `logs:CreateLogStream` + `logs:PutLogEvents` + * on it. + * + * Not optional in spirit — omitted only in isolated construct tests. P2 wired + * the name into `platform_config`, which makes the agent ATTEMPT the write, and + * shipped without the matching grant, so every structured per-task log line was + * denied (live 2026-08-07, ADR-021 P2-F4): + * + * > User: …:assumed-role/…LambdaMicrovmComputeExecutionRo…/Lambda-microvmsExecutor-… + * > is not authorized to perform: logs:CreateLogStream on resource: + * > …:log-group:/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/… + * + * The role's OTHER logs grant ({@link LambdaMicrovmCompute.grantMicrovmLogWrites}) + * is scoped to the service's own `/aws/lambda-microvms/*` namespace and cannot + * cover this group — the two namespaces are unrelated. Non-fatal (the agent + * degrades to stdout, which the MicroVM log group captures) but it empties the + * platform's canonical per-task observability streams, `METRICS_REPORT` + * included, on this backend only. Exactly the omission class the P2 Bedrock / + * Secrets Manager / Memory grants exist to close. + */ + readonly applicationLogGroup?: logs.ILogGroup; + /** * ARN of the Lambda-managed base MicroVM image to build on * (`aws lambda-microvms list-managed-microvm-images`), e.g. @@ -426,11 +628,15 @@ export interface LambdaMicrovmComputeProps extends LambdaMicrovmImageInputs { /** * Non-secret environment variables baked into the snapshot at build time. * - * Deliberately empty by default. ADR-021 sub-decision 3 forbids secrets, - * tokens, and per-task identity in the snapshot; the agent's non-secret - * configuration parity with the ECS container (table names, `MEMORY_ID`, - * `ARTIFACTS_BUCKET_NAME`, …) is P2 "smoke parity" work and is wired here - * when it lands. + * Deliberately empty by default, and expected to STAY empty. ADR-021 + * sub-decision 3 forbids secrets, tokens, and per-task identity in the snapshot + * — and P2 resolved the remaining question (where the agent's non-secret + * configuration parity with the ECS container comes from) in favour of the + * `/run` payload's `platform_config` block, NOT this prop. A snapshot is shared + * across every task and every deployment that reuses it, so a table or bucket + * name baked in here would be a deploy-time value frozen at image-build time — + * stale the moment the stack is redeployed. Reach for this only for genuinely + * image-invariant settings (a locale, a toolchain path). * @default {} — no baked configuration */ readonly imageEnvironmentVariables?: Record; @@ -459,9 +665,14 @@ export interface LambdaMicrovmComputeProps extends LambdaMicrovmImageInputs { * 3. **Build role** — assumed by Lambda during image creation: `s3:GetObject` * on the artifact object and CloudWatch Logs writes. Without it Lambda * cannot emit build logs, which makes a failed snapshot build undebuggable. - * 4. **Execution role** — assumed by the running MicroVM: CloudWatch Logs, - * read-only on the payload bucket, and (when a SessionRole is wired) - * admission to the per-task SessionRole for tenant-data access. + * 4. **Execution role** — assumed by the running MicroVM: CloudWatch Logs (both + * the service's own `/aws/lambda-microvms/*` namespace and the platform + * APPLICATION_LOGS group whose name `platform_config` delivers), read-only on + * the payload bucket, the P2 runtime-parity grants (GitHub PAT + + * channel-OAuth secret reads, scoped Bedrock invocation, AgentCore Memory, + * `ec2:DescribeAvailabilityZones` for a CDK repo's synth gate), and — when a + * SessionRole is wired — admission to the per-task SessionRole, which is the + * ONLY path to tenant data. * 5. **MicroVM image** (`AWS::Lambda::MicrovmImage`) — see {@link baseImageArn} * for why this is conditional. * @@ -482,37 +693,46 @@ export interface LambdaMicrovmComputeProps extends LambdaMicrovmImageInputs { * fails fast with the strategy's own "stack deployed without the MicroVM * substrate" error, which names the remedy. * - * ## ⚠️ A P1 image is runnable, but NOT smoke-verified + * ## ⚠️ A P2 substrate is fully wired, but still NOT smoke-verified * * Reaching state 1 or 2 provisions a complete substrate, a buildable image, and * a payload-deliverable `/run` path: P1 declares AND the agent serves `/ready` * and `/run` (`agent/src/server.py`), because live verification proved the * original "declare in P1, serve in P2" split was not a reachable service state * — `CreateMicrovmImage` refuses any lifecycle hook without `/ready` (see - * {@link READY_HOOK_PATH}), and an image with no hooks at all cannot receive a - * `runHookPayload`. + * {@link READY_HOOK_TIMEOUT_SECONDS}), and an image with no hooks at all cannot + * receive a `runHookPayload`. + * + * P2 adds the two halves an agent needs to actually finish a task: the runtime + * IAM parity on the execution role (see item 4 above) and non-secret + * configuration delivery through the `/run` payload's `platform_config` block + * (`handlers/shared/strategies/lambda-microvm-strategy.ts`) — the substitute for + * the env block the other two backends get at deploy time, since the snapshot must + * not bake it in. It also declares the two hooks the agent gained in the same + * phase: `/validate` (build-time self-check — see + * {@link VALIDATE_HOOK_TIMEOUT_SECONDS} for why the build role's permissions bound + * what it can assert) and `/terminate` (in-guest teardown breadcrumb — + * {@link TERMINATE_HOOK_TIMEOUT_SECONDS}). * - * What is still unverified is everything P2 owns: AgentCore Memory grants + - * `MEMORY_ID` delivery, the non-secret env parity the agent needs inside the - * snapshot, egress specifics from a running MicroVM, and heartbeat/progress - * behaviour end to end. So a `lambda-microvm` task can start and receive its - * payload, but clone → change → PR is **not** covered by any test or live run - * yet. That is what the `abca:microvm-image-p1-smoke-unverified` warning below - * says, and it is repeated in `cdk/scripts/package-microvm-artifact.sh`. - * `/validate` (build) and `/suspend`, `/resume`, `/terminate` (runtime) are - * still deliberately not declared: a hook the service calls but nothing answers - * fails the corresponding build or lifecycle transition. + * What is still unverified is the thing no amount of wiring can assert: an + * end-to-end clone → change → PR run on this substrate, plus egress specifics and + * heartbeat/progress behaviour from a live MicroVM. That is what the + * `abca:microvm-image-p1-smoke-unverified` warning below says, and it is repeated + * in `cdk/scripts/package-microvm-artifact.sh`. Only `/suspend` and `/resume` + * remain undeclared, until P3 implements them: a hook the service calls but + * nothing answers fails the corresponding lifecycle transition. * - * ## Deliberately NOT here (P1 scope) + * ## Deliberately NOT here * - * The execution role gets **no** Bedrock, Secrets Manager, AgentCore Memory, or - * artifacts-bucket grants. On the ECS backend those exist because the agent - * actually runs there today; ADR-021 puts agent parity on this backend in P2 - * ("smoke parity … AgentCore Memory parity (IAM grant + MEMORY_ID)"). Adding - * them now would hand a role permissions nothing exercises, and would have to - * be reviewed twice. `lambda:SuspendMicrovm` / `lambda:ResumeMicrovm` are - * likewise absent (P3) and `lambda:CreateMicrovmAuthToken` is granted to no - * role in any phase — no JWE consumer exists (sub-decision 3). + * The execution role gets no artifacts-bucket grant and no DynamoDB grant: an + * artifact delivery write goes through the SessionRole's + * `artifacts/${task_id}/*` statement (the AgentCore runtime role has no direct + * grant either) and every table the agent touches is `task_id`-partitioned + * SessionRole territory. It also has no UserConcurrencyTable grant — that counter + * is orchestrator/reconciler-owned and the agent path never writes it. + * `lambda:SuspendMicrovm` / `lambda:ResumeMicrovm` are absent (P3), and + * `lambda:CreateMicrovmAuthToken` is granted to no role in any phase — no JWE + * consumer exists (sub-decision 3). */ export class LambdaMicrovmCompute extends Construct { /** S3 bucket holding the zip + Dockerfile the snapshot is built from. */ @@ -683,19 +903,38 @@ export class LambdaMicrovmCompute extends Construct { // // The permission set below is the minimal recipe validated standalone in that // run: `AWSLambdaVPCAccessExecutionRole` plus the ENI/tag/private-IP actions - // the managed policy omits. Trust mirrors the build/execution roles - // (`lambda.amazonaws.com` + `aws:SourceAccount`), so the confused-deputy - // posture is identical on all three. + // the managed policy omits. // // ONE role for BOTH connectors: they differ only in security group, both are // created and owned by this construct in the same VPC, and a second identical // role would double the IAM surface a reviewer has to check for no isolation // gain (the role manages ENIs, not traffic). - const microvmAssumedBy = new iam.ServicePrincipal('lambda.amazonaws.com', { - conditions: { - StringEquals: { 'aws:SourceAccount': stack.account }, - }, - }); + // + // --- TRUST POLICY: bare service principal, NO source conditions (P2-F1/F3) --- + // + // Shared by all three MicroVM-facing roles (this one, `buildRole`, + // `executionRole`) because the decision is one decision. `lambda.amazonaws.com` + // is the correct principal — there is no `microvms.lambda.amazonaws.com`, and + // using one is rejected at role-creation time with MalformedPolicyDocument. + // + // ⚠️ **`aws:SourceAccount`/`aws:SourceArn` are deliberately absent. Adding one + // back re-breaks the deploy.** The Lambda MicroVMs service populates NO source + // condition key when it assumes these roles, so a trust policy carrying one is + // unassumable: both network connectors CREATE_FAILED deterministically, and + // `RunMicrovm` surfaced the same root cause as a misleading caller-side + // `iam:PassRole` denial on the orchestrator. Removing the conditions fixed both + // within seconds. This looks like a regression to anyone applying the standard + // confused-deputy pattern — and it already WAS one in the other direction: P1's + // working probe had no conditions, and the P1 F2 fix added them "to mirror the + // build/execution roles". `sts:TagSession` stays; it was never implicated. + // + // ADR-021 §4 is authoritative for the rest: the live evidence (P2-F1 / P2-F3, + // verbatim failures, the two-arm PassRole experiment, the contaminated control + // that produced run 1's false negative), the per-role compensating-controls + // table, and the conditions under which the condition could be restored. The + // trust shape here is asserted by `test/constructs/lambda-microvm-compute.test.ts` + // ("NO source-key condition on any MicroVM-facing role trust"). + const microvmAssumedBy = new iam.ServicePrincipal('lambda.amazonaws.com'); this.connectorOperatorRole = new iam.Role(this, 'ConnectorOperatorRole', { assumedBy: microvmAssumedBy, description: @@ -827,30 +1066,15 @@ export class LambdaMicrovmCompute extends Construct { // --- Roles --- // - // TRUST POLICY (verified against the AWS developer guide, "Lambda MicroVMs - // → Security and permissions"): the build role, the execution role and the - // connector operator role are all assumed by the ORDINARY Lambda service - // principal `lambda.amazonaws.com` (`microvmAssumedBy`, declared with the - // connector above because the operator role needs it first). The build and - // execution roles additionally need `sts:TagSession` alongside - // `sts:AssumeRole`. There is no `microvms.lambda.amazonaws.com` principal — - // using one is rejected at role-creation time with MalformedPolicyDocument. - // - // CONFUSED-DEPUTY: `aws:SourceAccount` is pinned to this account, which is - // the meaningful protection here — `lambda.amazonaws.com` is shared with - // every other Lambda feature, so without it any caller who could make - // Lambda act in *some* account could target these roles. - // - // `aws:SourceArn` is deliberately NOT added. Field reports of this exact - // pattern (an `ArnLike` on `…:microvm-image/*`) have the service failing to - // satisfy the condition — the image does not exist yet at build time — so - // both `create-microvm-image` and `run-microvm` fail with "unable to assume - // role". The AWS docs themselves are inconsistent about the separator in - // MicroVM image ARNs (`microvm-image:` vs `microvm-image/`), - // which is a second reason an ARN condition here is a deploy-time - // foot-gun. The live 2026-07-31 run confirmed the observed image ARN uses - // the `microvm-image:` (colon) form; narrowing to `aws:SourceArn` - // stays a P2 candidate rather than a P1 change. + // TRUST POLICY: all three MicroVM-facing roles share `microvmAssumedBy` — + // the BARE `lambda.amazonaws.com` service principal, with NO source + // conditions. The warning and the pointer live at that constant's + // declaration, above the connector operator role that needs it first; ADR-021 + // §4 carries the evidence (P2-F1 / P2-F3) and the per-role compensating + // controls. The + // build and execution roles additionally need `sts:TagSession` alongside + // `sts:AssumeRole` (developer guide, "Trust policies"), which + // {@link grantTagSession} adds. this.buildRole = new iam.Role(this, 'BuildRole', { assumedBy: microvmAssumedBy, @@ -878,6 +1102,17 @@ export class LambdaMicrovmCompute extends Construct { grantTagSession(this.executionRole, microvmAssumedBy); this.grantMicrovmLogWrites(this.executionRole); + // The APPLICATION_LOGS group the agent is TOLD to write to (P2-F4). Separate + // from `grantMicrovmLogWrites` above and not reachable from it: that grant + // covers the service-owned `/aws/lambda-microvms/*` namespace, while + // `platform_config.log_group_name` points at the platform's vended + // `/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/` group — + // the one the dashboard and every per-task log query read. Scoped to that one + // group (CDK `grantWrite` → `logs:CreateLogStream` + `logs:PutLogEvents` on the + // group's ARN, stream wildcard only), so this adds no cross-log-group reach. + // See `applicationLogGroup` for the denial this fixes. + props.applicationLogGroup?.grantWrite(this.executionRole); + // READ-ONLY on the payload bucket (ADR-021: "The MicroVM execution role // shall hold read-only access to the payload bucket, scoped to that // bucket"). Read-only is not a nicety: the MicroVM runs untrusted repo @@ -890,10 +1125,125 @@ export class LambdaMicrovmCompute extends Construct { // that construct: there is no `else` branch granting DynamoDB directly — // this backend has no legacy deployments to keep working, so a missing // SessionRole means no tenant-data access rather than broad access. + // + // This is ALSO why nothing below grants DynamoDB: every table the agent + // touches is `task_id`-partitioned and reachable only through the + // SessionRole's `dynamodb:LeadingKeys` condition. `admitComputeRole` wires + // both halves that needs (trust on the SessionRole + `sts:AssumeRole` / + // `sts:TagSession` here), so P2 adds nothing to this seam — asserted by a + // unit test, because a "convenience" direct grant is exactly how per-tenant + // isolation gets lost. if (props.agentSessionRole) { props.agentSessionRole.admitComputeRole(this.executionRole); } + // --- P2 runtime parity on the EXECUTION role (ADR-021 "smoke parity") --- + // + // Feature-derived, not copied from `ecs-agent-cluster`: each grant below + // exists because a specific agent code path fails without it on THIS + // substrate. The ECS task role's remaining grants are deliberately absent — + // the UserConcurrencyTable (orchestrator/reconciler-owned; the agent path + // never writes it) and any artifacts-bucket access (delivery writes go + // through the SessionRole's `artifacts/${task_id}/*` statement, so the + // AgentCore runtime role has no direct grant either and neither does this). + + // Secrets Manager, part 1: the GitHub PAT, read at startup before the agent + // assumes the SessionRole. + if (props.githubTokenSecret) { + props.githubTokenSecret.grantRead(this.executionRole); + } + + // Secrets Manager, part 2: per-workspace Linear/Jira OAuth tokens. Same shape + // and same reason as `ecs-agent-cluster`'s grant (ABCA-488): the CLI creates + // `bgagent-linear-oauth-` / `bgagent-jira-oauth-` at setup, so + // the name is unknown at synth and a PREFIX grant is the only expressible + // scope. For a Linear/Jira-channel task the agent resolves that token at + // startup (`config.resolve_linear_api_token` / + // `resolve_jira_oauth_token`) to fire the 👀→✅ reaction and drive the channel + // MCP; without the grant the fetch hits AccessDenied and both silently no-op + // (logged by the token resolver, but invisible to the user in the channel). + // + // `GetSecretValue` ONLY — the agent reads; the orchestrator owns refresh / + // PutSecretValue. + this.executionRole.addToPrincipalPolicy(new iam.PolicyStatement({ + actions: ['secretsmanager:GetSecretValue'], + resources: [ + stack.formatArn({ + service: 'secretsmanager', + resource: 'secret', + arnFormat: ArnFormat.COLON_RESOURCE_NAME, + resourceName: 'bgagent-linear-oauth-*', + }), + stack.formatArn({ + service: 'secretsmanager', + resource: 'secret', + arnFormat: ArnFormat.COLON_RESOURCE_NAME, + resourceName: 'bgagent-jira-oauth-*', + }), + ], + })); + + // Bedrock model invocation — scoped to explicit foundation-model and + // cross-Region inference-profile ARNs (parity with the AgentCore runtime and + // the ECS task role), NEVER `Resource: '*'`. The model set comes from the + // shared, context-overridable list (`constructs/bedrock-models.ts`) so no + // backend can drift from the others. + // + // Required on the COMPUTE role even though the SessionRole carries a + // session-tagged Bedrock grant for cost attribution (#215): that attribution + // is designed to FAIL OPEN — Claude Code's credential helper falls back to + // ambient compute-role credentials when the assume-role fails — so without + // this grant the fallback path AccessDenies and the task dies at turn 0. + const bedrockResources: string[] = []; + for (const modelId of resolveBedrockModelIds(this.node)) { + bedrockResources.push( + stack.formatArn({ + service: 'bedrock', + region: '*', + account: '', + resource: 'foundation-model', + resourceName: modelId, + arnFormat: ArnFormat.SLASH_RESOURCE_NAME, + }), + stack.formatArn({ + service: 'bedrock', + resource: 'inference-profile', + resourceName: `us.${modelId}`, + arnFormat: ArnFormat.SLASH_RESOURCE_NAME, + }), + ); + } + this.executionRole.addToPrincipalPolicy(new iam.PolicyStatement({ + actions: [ + 'bedrock:InvokeModel', + 'bedrock:InvokeModelWithResponseStream', + ], + resources: bedrockResources, + })); + + // AgentCore Memory read+write, so cross-task learning actually persists on + // this substrate. `MEMORY_ID` already reaches the agent inside + // `agent_payload` (unchanged by P2 — it is task data, not platform config), + // which means the agent ATTEMPTS the write regardless; the grant is what + // decides whether it lands or fails closed. + if (props.agentMemory) { + props.agentMemory.grantReadWrite(this.executionRole); + } + + // A CDK-based target repo's build gate runs `cdk synth`, and a stack wired to + // a concrete env ({account, region}) does a synth-time availability-zone + // context lookup. On a developer box the gitignored cdk.context.json caches + // the answer; the agent clones fresh, so there is no cache and synth fires the + // live lookup. Without this grant the role hits AccessDenied → "Synthesis + // finished with errors" → a FALSE build-gate failure on code that builds fine + // everywhere else (the exact regression the ECS task role hit). Read-only + // describe with no resource-level scoping in IAM, so `Resource: '*'` is + // mandatory (suppressed below); it grants no mutation and no data access. + this.executionRole.addToPrincipalPolicy(new iam.PolicyStatement({ + actions: ['ec2:DescribeAvailabilityZones'], + resources: ['*'], + })); + // --- Image --- // Branch through the shared predicate's two components rather than an // ad-hoc condition, so this construct and the stack's pre-TaskApi decision @@ -926,22 +1276,41 @@ export class LambdaMicrovmCompute extends Construct { hooks: { port: AGENT_HOOK_PORT, microvmHooks: { - // `/run` is the payload-delivery channel, and the agent SERVES it as - // of P1 (`agent/src/server.py`). `/suspend` and `/resume` land with - // the P3 interface widening, and `/terminate` with P2: declaring a - // runtime hook the agent does not answer fails the corresponding - // lifecycle transition. P1 termination is the orchestrator's - // `TerminateMicrovm`, which needs no in-guest cooperation. - run: RUN_HOOK_PATH, + // Values are the `ENABLED` enum, NOT the hook route: CloudFormation + // rejects a path on every one of these four fields (P2-F2 — see + // {@link HOOK_ENABLED}). The routes are fixed and service-owned; + // {@link MICROVM_AGENT_HOOK_ROUTES} records them for the agent's + // benefit and must never be sent here again. + // + // `/run` is the payload-delivery channel (and, since P2, the + // platform-configuration channel); `/terminate` is the in-guest + // teardown breadcrumb. The agent serves BOTH — enabling a runtime + // hook nothing answers fails the corresponding lifecycle transition, + // so each is enabled only once it is served. + // + // `/suspend` and `/resume` stay OMITTED (not `DISABLED`) until P3, + // where the suspend/resume interface widening lands across all three + // strategies. Note that termination does NOT depend on this hook: + // `TerminateMicrovm` removes the VM with or without in-guest + // cooperation, which is what makes a best-effort `/terminate` safe to + // declare. + run: HOOK_ENABLED, runTimeoutInSeconds: RUN_HOOK_TIMEOUT_SECONDS, + terminate: HOOK_ENABLED, + terminateTimeoutInSeconds: TERMINATE_HOOK_TIMEOUT_SECONDS, }, microvmImageHooks: { // `/ready` is MANDATORY whenever any lifecycle hook is enabled — the - // service refuses the create otherwise (see READY_HOOK_PATH), which - // is why it moved from P2 to P1. `/validate` stays out: the agent has - // no validation endpoint, and one that 404s fails every image build. - ready: READY_HOOK_PATH, + // service refuses the create otherwise (see + // READY_HOOK_TIMEOUT_SECONDS), which is why it moved from P2 to P1. + // `/validate` joins it in P2, now that the agent serves a real + // (AWS-call-free — see VALIDATE_HOOK_TIMEOUT_SECONDS) self-check: a + // hook that 404s or reports failure fails every image build, so it + // could not be enabled before there was something behind it. + ready: HOOK_ENABLED, readyTimeoutInSeconds: READY_HOOK_TIMEOUT_SECONDS, + validate: HOOK_ENABLED, + validateTimeoutInSeconds: VALIDATE_HOOK_TIMEOUT_SECONDS, }, }, }); @@ -1001,22 +1370,31 @@ export class LambdaMicrovmCompute extends Construct { if (this.imageIdentifier) { // Emitted on EVERY deploy that configures an image, in both image states. - // Not a throw and not suppressible: P1 now provisions a substrate, a - // buildable image AND a payload-deliverable /run path, which makes it look - // even MORE like a working backend than before — while nothing in P1 has - // exercised clone → change → PR on this substrate. The warning's job is to - // keep "deploy succeeded" from reading as "backend works". + // Not a throw and not suppressible: the substrate now looks like a working + // backend in every observable way — the image builds, launches, receives a + // payload, and the execution role holds the full runtime permission set — + // while nothing has exercised clone → change → PR on it. The warning's job is + // to keep "deploy succeeded" from reading as "backend works". + // + // The id is deliberately UNCHANGED across P1→P2 (operators grep for it, and a + // rename would read as "the old warning is gone, so it must be fine"). Annotations.of(this).addWarningV2( 'abca:microvm-image-p1-smoke-unverified', - 'A MicroVM image is configured. As of ADR-021 P1 the image IS creatable and launchable and ' - + 'the agent DOES serve the /ready and /run hooks, so a lambda-microvm task can start and ' - + 'receive its payload — but the backend has NO smoke-parity guarantee: AgentCore Memory ' - + 'grants and MEMORY_ID delivery, the agent\'s non-secret env parity inside the snapshot, ' - + 'egress specifics from a running MicroVM, and heartbeat/progress behaviour are all P2 and ' - + 'untested here. Keep production repos on compute_type=agentcore or ecs until P2 (smoke ' - + 'parity) lands. The /validate build hook and the /suspend, /resume and /terminate runtime ' - + 'hooks are deliberately still not declared: a hook the service calls but nothing answers ' - + 'fails the corresponding build or lifecycle transition.', + 'A MicroVM image is configured. A P2 smoke run HAS now completed clone -> change -> PR on ' + + 'this substrate (2026-08-07: two tasks COMPLETED with pull requests, progress streaming to ' + + 'bgagent watch, and the 45s agent heartbeat observed live), the agent serves the /ready, ' + + '/validate, /run and /terminate hooks, and the execution role holds its full runtime ' + + 'permission set. What is still MISSING is a run with no manual intervention: that smoke ' + + 'needed a live IAM workaround, and the two defects behind it (ADR-021 P2r2-F9 / P2r2-F10 — ' + + 'the iam:PassedToService condition on both PassRole paths) are fixed in source but NOT yet ' + + 're-exercised live. ALSO REQUIRED: re-bootstrap to policy bundle 1.4.0 or the CDK-managed ' + + 'image path fails with iam:PassRole AccessDenied on the build role. So the backend still ' + + 'carries no smoke-parity guarantee for an unattended deployment - keep production repos on ' + + 'compute_type=agentcore or ecs until a clean run is on record. Only the /suspend and ' + + '/resume runtime hooks remain undeclared, until P3 implements them: a hook the service ' + + 'calls but nothing answers fails the corresponding lifecycle transition. ' + + "(This warning's id still reads p1- by design: it is frozen across phases so operator " + + 'greps and suppression lists keep matching — read the text, not the id, for the phase.)', ); } @@ -1039,7 +1417,23 @@ export class LambdaMicrovmCompute extends Construct { + `${MICROVM_LOG_GROUP_PREFIX}/* namespace (log stream names are minted per MicroVM, so no ` + 'synth-time ARN exists); S3 object/* wildcard comes from CDK grantRead on the dedicated ' + 'payload bucket (read-only, scoped to that bucket — ADR-021 sub-decision 3). The build ' - + 'role\'s s3:GetObject is scoped to a single object key, not a wildcard.', + + 'role\'s s3:GetObject is scoped to a single object key, not a wildcard. On the execution ' + + 'role (ADR-021 P2 runtime parity, mirroring the ECS task role): the second Logs grant is ' + + 'CDK grantWrite (CreateLogStream + PutLogEvents only) on the SINGLE platform ' + + 'APPLICATION_LOGS group whose name platform_config delivers to the guest, whose ARN ends ' + + 'in a log-stream wildcard because streams are minted per task (ADR-021 P2-F4); ' + + 'Secrets Manager wildcards are CDK grantRead on the GitHub PAT secret plus the ' + + 'bgagent-linear-oauth-*/' + + 'bgagent-jira-oauth-* prefix grant (ABCA-488 — per-workspace channel OAuth tokens are ' + + 'created by the CLI at setup, so the name is unknown at synth; GetSecretValue only); ' + + 'AgentCore Memory wildcards are CDK grantRead/grantWrite on the single platform Memory ' + + 'resource; Bedrock InvokeModel is scoped to explicit foundation-model and ' + + 'inference-profile ARNs from the shared model list (no wildcard resource); ' + + 'ec2:DescribeAvailabilityZones requires Resource:* because EC2 describe actions have no ' + + 'resource-level scoping — read-only, no mutation and no data access, needed so a CDK ' + + 'target repo\'s `cdk synth` build gate can resolve AZ context on a fresh clone. No ' + + 'DynamoDB grant is issued to either role: tenant-data access goes exclusively through ' + + 'the per-task SessionRole\'s task_id-scoped policy.', }, ], true); @@ -1061,9 +1455,12 @@ export class LambdaMicrovmCompute extends Construct { id: 'AwsSolutions-IAM5', reason: 'EC2 network-interface APIs are not meaningfully resource-scopable here: ' + 'CreateNetworkInterface is authorized before the ENI exists, and the Describe* calls ' - + 'take no resource at all. The role is assumable ONLY by lambda.amazonaws.com with ' - + 'aws:SourceAccount pinned to this account, holds no data-plane permission, and is used ' - + 'solely to attach the two platform-owned connectors to the platform VPC. The ' + + 'take no resource at all. The role is assumable ONLY by lambda.amazonaws.com, holds no ' + + 'data-plane permission, and is used solely to attach the two platform-owned connectors ' + + 'to the platform VPC. An aws:SourceAccount confused-deputy condition is NOT available ' + + 'on this trust: the Lambda MicroVMs service presents no source key when it assumes the ' + + 'role, and adding one makes the connector un-creatable (live-verified, ADR-021 P2-F1 — ' + + 'see ADR-021 section 4 for the per-role compensating controls). The ' + 'AWS-managed VPC-access policy uses the same wildcard for the same reason.', }, ], true); @@ -1106,9 +1503,12 @@ export class LambdaMicrovmCompute extends Construct { * * The MicroVM service needs BOTH actions (developer guide, "Trust policies"), * but `iam.Role`'s `assumedBy` only renders `sts:AssumeRole`. Passing a second - * statement through `assumeRolePolicy` keeps the `aws:SourceAccount` condition - * identical on both actions — dropping it on the `TagSession` half would leave - * the confused-deputy hole half-open. + * statement through `assumeRolePolicy` keeps the two halves identical — which + * since P2-F1/F3 means "identical and unconditioned": `principal.policyFragment. + * conditions` is now empty, and the pass-through is kept deliberately rather + * than hardcoding `{}` so that if a source-condition key ever becomes usable on + * this path (see the trust-policy block in the constructor), adding it to the + * principal fixes BOTH actions instead of half-closing the hole. */ function grantTagSession(role: iam.Role, principal: iam.ServicePrincipal): void { role.assumeRolePolicy?.addStatements(new iam.PolicyStatement({ diff --git a/cdk/src/constructs/task-orchestrator.ts b/cdk/src/constructs/task-orchestrator.ts index fcaf4d282..113e7171a 100644 --- a/cdk/src/constructs/task-orchestrator.ts +++ b/cdk/src/constructs/task-orchestrator.ts @@ -190,6 +190,80 @@ export interface TaskOrchestratorProps { */ readonly attachmentsBucket?: s3.IBucket; + /** + * Non-secret platform identifiers the orchestrator FORWARDS to the in-guest + * agent, for backends that have no deploy-time env block of their own. + * + * ## Why the orchestrator carries values it never uses itself + * + * On AgentCore these live in the runtime's `environmentVariables` and on ECS in + * the container's `environment` — CDK sets them directly on the compute. A + * Lambda MicroVM snapshot cannot have them: ADR-021 sub-decision 3 forbids + * baking configuration into an image that is shared across tasks and + * deployments, so the only channel is the `/run` payload the orchestrator + * writes (`platform_config`, assembled by + * `handlers/shared/strategies/lambda-microvm-strategy.ts`). That makes the + * orchestrator's own environment the transport, which is why these appear here + * rather than on `LambdaMicrovmCompute`. + * + * ## Names, ARNs — and NO grants + * + * Every field is an identifier, never a secret value, and NONE of them adds an + * IAM grant to the orchestrator role: it forwards these strings and never calls + * the resources they name (the agent does, through its own execution role / + * SessionRole). The approvals and nudges tables in particular stay ungranted to + * the orchestrator, which is asserted by a unit test — a "while I'm here" grant + * would hand the orchestration plane tenant-data access it has never needed. + * + * ## All-or-nothing, and wired unconditionally + * + * Every field is required so a partial configuration is unrepresentable (same + * rationale as `ecsConfig` / `microvmConfig`). The stack wires it for EVERY + * compute type rather than under the `lambda-microvm` gate: the strategy fails + * the session start when a required identifier is missing, and that guard should + * only ever fire for a hand-edited Lambda environment — never because a + * deploy-time gate and a per-repo `compute_type` disagreed. + * + * Optional as a prop only so isolated construct tests can omit it. Four of the + * thirteen `platform_config` keys come from env vars the orchestrator already + * carries for its own work (`TASK_TABLE_NAME`, `TASK_EVENTS_TABLE_NAME`, + * `GITHUB_TOKEN_SECRET_ARN`) or from the stack-wide `SolutionUaAspect` + * (`AWS_SDK_UA_APP_ID`), so they are deliberately NOT repeated here. + */ + readonly agentPlatformConfig?: { + /** + * Cedar HITL approvals table (`TASK_APPROVALS_TABLE_NAME`). The agent's + * approval primitives write PENDING rows here; absent, the PreToolUse hook + * fails closed with `approval_write_failed`. + */ + readonly taskApprovalsTableName: string; + /** Nudges table (`NUDGES_TABLE_NAME`) the agent polls for mid-task nudges. */ + readonly nudgesTableName: string; + /** Application log group (`LOG_GROUP_NAME`) the agent writes progress logs to. */ + readonly logGroupName: string; + /** + * Bucket a `deliver_artifact` step uploads to (`ARTIFACTS_BUCKET_NAME`). + * Without it an artifact workflow fails at delivery with + * "ARTIFACTS_BUCKET_NAME is not configured". + */ + readonly artifactsBucketName: string; + /** Bucket the `--trace` trajectory upload targets (`TRACE_ARTIFACTS_BUCKET_NAME`). */ + readonly traceArtifactsBucketName: string; + /** + * Per-task SessionRole ARN (`AGENT_SESSION_ROLE_ARN`). The sharpest field in + * this block: when the agent does not receive it, it falls back to ambient + * compute-role credentials and per-tenant scoping is silently OFF. + */ + readonly agentSessionRoleArn: string; + /** + * Cross-region inference-profile id for the small/fast model + * (`ANTHROPIC_DEFAULT_HAIKU_MODEL`). Must be a `us.`-prefixed profile id, not + * a bare foundation-model id — Claude 4.x rejects on-demand invocation by + * bare id — and must match a granted profile (`constructs/bedrock-models.ts`). + */ + readonly anthropicDefaultHaikuModel: string; + }; + /** * AWS Lambda MicroVMs compute strategy configuration (ADR-021 sub-decision 4). * When provided, the `MICROVM_*` env vars and the MicroVM lifecycle IAM @@ -395,6 +469,21 @@ export class TaskOrchestrator extends Construct { }), }), ...(props.attachmentsBucket && { ATTACHMENTS_BUCKET_NAME: props.attachmentsBucket.bucketName }), + // ADR-021 P2: non-secret identifiers the orchestrator FORWARDS to the + // in-guest agent as `platform_config` on the MicroVM /run payload, because + // a MicroVM snapshot must not bake configuration in. Names match the + // AgentCore runtime env block in `stacks/agent.ts` and the strategy's + // PLATFORM_CONFIG_ENV_VARS map verbatim — one stack value, one name, three + // backends. NO IAM grant accompanies any of these (see the prop docs). + ...(props.agentPlatformConfig && { + TASK_APPROVALS_TABLE_NAME: props.agentPlatformConfig.taskApprovalsTableName, + NUDGES_TABLE_NAME: props.agentPlatformConfig.nudgesTableName, + LOG_GROUP_NAME: props.agentPlatformConfig.logGroupName, + ARTIFACTS_BUCKET_NAME: props.agentPlatformConfig.artifactsBucketName, + TRACE_ARTIFACTS_BUCKET_NAME: props.agentPlatformConfig.traceArtifactsBucketName, + AGENT_SESSION_ROLE_ARN: props.agentPlatformConfig.agentSessionRoleArn, + ANTHROPIC_DEFAULT_HAIKU_MODEL: props.agentPlatformConfig.anthropicDefaultHaikuModel, + }), }, bundling: orchestratorBundling, }); @@ -565,15 +654,57 @@ export class TaskOrchestrator extends Construct { // execution roles to ecs-tasks.amazonaws.com. ADR-021's grant list omits // it because it enumerates MicroVM actions, not the IAM plumbing they // imply; without it RunMicrovm fails on the role hand-off. + // + // ⚠️ NO `iam:PassedToService` CONDITION HERE, AND THAT IS DELIBERATE + // (ADR-021 P2r2-F10, live 2026-08-07 run 2). This reverses what an earlier + // revision of this comment asserted — that the condition "was explicitly + // EXONERATED live … so it stays". It was not. It is a second, independent + // blocker of exactly the same class as the trust-policy source keys: the + // Lambda MicroVMs service does not present a usable `iam:PassedToService` + // value on the `RunMicrovm` PassRole path, so a grant carrying the condition + // is denied. + // + // Proven by a controlled two-arm experiment — SAME exact-ARN resource, SAME + // ~5-minute IAM settle, one variable: + // + // | grant | result | + // |--------------------------------------------------|--------------------| + // | exact ARN + iam:PassedToService (as written then) | DENIED (twice) | + // | exact ARN, no condition | RUNNING in 9 s | + // + // …with this denial on the CALLER, which is what makes it so misleading: + // "User: …/backgroundagent-dev-TaskOrchestratorOrchestratorFn-… is not + // authorized to perform: iam:PassRole on resource: + // …role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-… because no + // identity-based policy allows the iam:PassRole action" + // even though the statement below names that exact ARN and + // `simulate-principal-policy` answers `allowed`. + // + // WHY RUN 1 GOT THIS WRONG, because the failure mode is worth knowing: run 1 + // "exonerated" the condition by attaching a temporary UNCONDITIONED + // `iam:PassRole` and observing that the task still failed — but that + // temporary grant was **still attached** for the later submissions that + // reached `RUNNING`, so the conditioned grant was never once tested against + // a working trust policy. A false negative from a contaminated control. + // Run 2 removed the workaround first (submission 4: denied) and only then + // added back the unconditioned grant on the same resource (submission 5: + // `RUNNING`). + // + // Compensating control that REMAINS: the grant is scoped to the execution + // role's EXACT ARN (`props.microvmConfig.executionRoleArn`), not a name + // prefix and not `*` — so this Lambda can pass exactly one role, the one + // this deployment created for this backend. That is now the whole of the + // scoping, which is why the resource must never be relaxed to a wildcard. + // + // If AWS documents (or a bounded probe finds) the value the service does + // present, add it back as a condition — `microvms.lambda.amazonaws.com`, + // `lambda-microvms.amazonaws.com` and `microvms.amazonaws.com` are all + // candidates that were `implicitDeny` against the conditioned policy, so any + // of them would work as the allowlist entry if it is the right one. this.fn.addToRolePolicy(new iam.PolicyStatement({ sid: 'MicrovmPassExecutionRole', actions: ['iam:PassRole'], resources: [props.microvmConfig.executionRoleArn], - conditions: { - StringEquals: { - 'iam:PassedToService': 'lambda.amazonaws.com', - }, - }, })); } diff --git a/cdk/src/handlers/shared/orchestrator.ts b/cdk/src/handlers/shared/orchestrator.ts index f91950f1b..01a6abced 100644 --- a/cdk/src/handlers/shared/orchestrator.ts +++ b/cdk/src/handlers/shared/orchestrator.ts @@ -78,6 +78,50 @@ const AGENT_HEARTBEAT_GRACE_SEC = 120; /** If `agent_heartbeat_at` exists and is older than this, the session is treated as lost. */ const AGENT_HEARTBEAT_STALE_SEC = 240; +/** + * Whether a backend's liveness is (partly) inferred from `agent_heartbeat_at`. + * + * The agent writes that timestamp UNCONDITIONALLY on every substrate — it is a + * DynamoDB write from the pipeline, with no backend awareness — so this predicate + * decides only whether the ORCHESTRATOR acts on it. + * + * - `agentcore` — yes, and it is the ONLY liveness signal there: + * `AgentCoreComputeStrategy.pollSession` is an explicit stub that always + * reports `running`, so a crashed container is invisible without the heartbeat. + * - `lambda-microvm` — yes, and it is the SECOND of two complementary signals + * (ADR-021 P2). The substrate `GetMicrovm` check catches a VM that DIED; it + * cannot catch a VM that is alive and healthy while the in-guest pipeline is + * hung, deadlocked, or OOM-killed inside the guest — nothing self-terminates on + * this substrate (live-verified: a MicroVM with a broken hook sat in `RUNNING` + * indefinitely with no `stateReason`). Without the heartbeat check such a task + * would burn the full ~8.5 h poll window, billing an 8-hour MicroVM + * reservation, before the safety net fired. So liveness here is substrate state + * AND agent heartbeat. + * - `ecs` — no, deliberately unchanged. `DescribeTasks` reports a real container + * exit (including OOM-kill, exit 137) with an exit code, and the ECS poll block + * in `orchestrate-task.ts` already interprets it with its own patience + * counters. Layering the heartbeat on top would give one backend two + * independently-tuned kill paths for the same failure and could fail a task + * whose container is provably still running. + * + * A `switch` rather than a set membership test so a fourth backend cannot be + * added without making an explicit, compile-checked decision here — the culture + * ADR-021 sub-decision 1 asks for. + */ +function heartbeatLivenessApplies(computeType: ComputeType): boolean { + switch (computeType) { + case 'agentcore': + case 'lambda-microvm': + return true; + case 'ecs': + return false; + default: { + const _exhaustive: never = computeType; + throw new Error(`Unknown compute type for heartbeat liveness: ${String(_exhaustive)}`); + } + } +} + /** * Load a task record from DynamoDB. * @param taskId - the task to load. @@ -847,6 +891,15 @@ export async function hydrateAndTransition(task: TaskRecord, blueprintConfig?: B /** * Poll the task record in DynamoDB to check if the agent wrote a terminal status. * Returns the updated PollState; the waitStrategy decides whether to continue. + * + * Heartbeat staleness is evaluated for the backends + * {@link heartbeatLivenessApplies} names — `agentcore` (its only liveness signal) + * and `lambda-microvm` (the in-guest half of "substrate state AND agent + * heartbeat"). Thresholds are shared across both on purpose: the timestamp is + * written by the same pipeline code at the same cadence regardless of substrate, + * so a backend-specific grace window would encode a difference that does not + * exist. + * * @param taskId - the task to poll. * @param state - current poll state. * @param computeType - the compute backend for this task (controls heartbeat checks). @@ -869,7 +922,7 @@ export async function pollTaskStatus( let sessionUnhealthy = false; if ( - computeType === 'agentcore' + heartbeatLivenessApplies(computeType) && currentStatus === TaskStatus.RUNNING && item?.session_id && typeof item.started_at === 'string' @@ -888,6 +941,7 @@ export async function pollTaskStatus( sessionUnhealthy = true; logger.warn('Agent heartbeat stale while task RUNNING', { task_id: taskId, + compute_type: computeType, agent_heartbeat_at: item.agent_heartbeat_at, heartbeat_age_sec: Math.round(hbAgeSec), }); @@ -899,6 +953,7 @@ export async function pollTaskStatus( sessionUnhealthy = true; logger.warn('Agent never sent heartbeat while task RUNNING past grace period', { task_id: taskId, + compute_type: computeType, running_age_sec: Math.round(runningAgeSec), }); } diff --git a/cdk/src/handlers/shared/strategies/lambda-microvm-strategy.ts b/cdk/src/handlers/shared/strategies/lambda-microvm-strategy.ts index cb736e3af..6358e583e 100644 --- a/cdk/src/handlers/shared/strategies/lambda-microvm-strategy.ts +++ b/cdk/src/handlers/shared/strategies/lambda-microvm-strategy.ts @@ -25,6 +25,10 @@ import { TerminateMicrovmCommand, } from '@aws-sdk/client-lambda-microvms'; import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'; +// Cross-language contract (S9): `microvm_platform_config` is read by BOTH this +// producer and `agent/src/server.py`'s `/run` consumer. Imported (not copied) so +// `tsc` fails on a renamed field — see `contracts/constants.md`. +import sharedConstants from '../../../../../contracts/constants.json'; import type { ComputeStrategy, SessionHandle, SessionStatus } from '../compute-strategy'; import { logger } from '../logger'; import type { BlueprintConfig } from '../repo-config'; @@ -123,6 +127,173 @@ export const MICROVM_MAX_DURATION_SECONDS = 28_800; */ const RUN_HOOK_PAYLOAD_LIMIT_BYTES = 4_096; +/** + * The `platform_config` contract (ADR-021 P2): the non-secret platform + * identifiers the in-guest agent needs, and the EXACT wire keys it reads. + * + * ## Why this block exists at all + * + * On AgentCore the same values arrive as runtime `environmentVariables`, and on + * ECS as container `environment` — both set at deploy time by CDK. A MicroVM + * snapshot cannot carry them: ADR-021 sub-decision 3 forbids baking + * configuration into the image (it is shared across every task and every + * deployment that reuses the snapshot, and its env is frozen at build time), so + * the `/run` hook payload is the only channel. `platform_config` is that channel + * — the third backend's equivalent of the other two backends' env blocks. + * + * ## Cross-language, and therefore contract-sourced + * + * The key set is NOT declared here. It is read from + * `contracts/constants.json` → `microvm_platform_config.env_by_key`, the same + * object `agent/src/server.py` reads to decide which keys it will install into + * the guest's `os.environ` — so producer and consumer cannot disagree about a + * key, its environment-variable name, or the required subset. `tsc` enforces + * this side (the JSON is imported, so a renamed field fails compilation); + * `scripts/check-constants-sync.ts` validates the contract's shape and rejects a + * Python-side literal re-declaration. See `contracts/constants.md`. + * + * That indirection is load-bearing on the agent side for a security reason: the + * values land in `os.environ`, so a key outside the allow-list is an + * env-injection attempt and the agent **refuses the whole block** rather than + * filtering it. A producer that invented a key would therefore fail every task, + * not silently drop a field. + * + * ## What may and may not go in here + * + * NON-SECRET IDENTIFIERS ONLY — table names, bucket names, log-group names, and + * secret/role **ARNs**. Never a token, never a secret *value*: the envelope is + * written to an S3 object and echoed into MicroVM logs on a hook failure, and + * the agent resolves an ARN itself through its own (SessionRole / + * execution-role) credentials. The producer below is a map over exactly the + * contract's keys, so a value can only reach the wire by being added to the + * contract — an unrelated `process.env` entry (`GITHUB_TOKEN`, + * `ANTHROPIC_API_KEY`, …) cannot leak in by accident. + * + * ## Ordering is part of the contract + * + * The contract's declaration order is the emission order (`JSON.stringify` + * preserves insertion order for string keys), which keeps the serialized + * envelope — and therefore the 4 KB inline/S3 branch decision — deterministic + * for a given environment. + */ +const PLATFORM_CONFIG_CONTRACT = sharedConstants.microvm_platform_config; + +/** + * Wire key → the environment variable the orchestrator carries it in, AND the + * name the agent installs it as in the guest. + * + * The env-var names are the ones `TaskOrchestrator` injects + * (`constructs/task-orchestrator.ts`), which are in turn the names the AgentCore + * runtime env block in `stacks/agent.ts` uses — so one stack-level value feeds + * all three backends under one name. + * + * Two entries have no CDK-injected source today, deliberately: + * `LINEAR_OAUTH_SECRET_ARN` / `JIRA_OAUTH_SECRET_ARN` name **per-workspace** + * secrets created by the CLI at setup, so no single ARN exists at synth time + * (which is why every consumer role gets a `bgagent-linear-oauth-*` / + * `bgagent-jira-oauth-*` PREFIX grant instead). The agent's normal source is + * `channel_metadata.{linear,jira}_oauth_secret_arn` inside `agent_payload`; + * these keys are the env-var fallback `agent/src/config.py` already reads, so + * they are forwarded when an operator sets them and omitted otherwise. + */ +const PLATFORM_CONFIG_ENV_VARS = PLATFORM_CONFIG_CONTRACT.env_by_key; + +/** One of the `platform_config` wire keys. */ +export type MicrovmPlatformConfigKey = keyof typeof PLATFORM_CONFIG_ENV_VARS; + +/** + * Every `platform_config` wire key, in contract (and therefore serialization) + * order. + */ +export const MICROVM_PLATFORM_CONFIG_KEYS = Object.keys( + PLATFORM_CONFIG_ENV_VARS, +) as readonly MicrovmPlatformConfigKey[]; + +/** + * The `platform_config` block as it appears on the wire. Every key is optional + * in the TYPE because the producer omits what the orchestrator's environment + * does not carry; {@link MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS} is the subset + * whose absence fails the session start instead. + */ +export type MicrovmPlatformConfig = Partial>; + +/** + * Keys the agent cannot start a task without, so a missing one fails the session + * start here rather than producing a task that dies in-guest (the agent rejects + * the same set with an `…_INCOMPLETE` 400 — failing at the orchestrator is the + * cheaper, better-attributed half of the same rule). + * + * Each earns its place by what breaks without it: + * - `task_table_name` / `task_events_table_name` — every status transition, + * heartbeat and progress event the orchestrator polls for. Without them the + * task looks hung to the poller and gets failed ~8.5 h later. + * - `github_token_secret_arn` — no clone, no push, no PR. + * - `agent_session_role_arn` — the agent falls back to AMBIENT execution-role + * credentials and per-tenant scoping is silently OFF. That is the failure mode + * `ecs-agent-cluster`'s reserved-env list calls the sharpest one in the + * platform, and exactly the kind of security control that must not degrade + * quietly. + * + * Everything else is genuinely optional: a deployment may have no approvals + * table wired, no artifacts bucket and no channel OAuth, and the agent's own + * fallbacks cover it. + * + * The cast is safe by contract: `scripts/check-constants-sync.ts` fails the build + * unless `required` is a duplicate-free subset of `env_by_key`. + */ +export const MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS = + PLATFORM_CONFIG_CONTRACT.required as readonly MicrovmPlatformConfigKey[]; + +/** + * Assemble the `platform_config` block from the ORCHESTRATOR Lambda's own + * environment. + * + * Read at CALL time rather than module load (unlike the `MICROVM_*` constants + * above) for two reasons: the required-key check has to throw *during* + * `startSession` so the failure lands on the task with a remedy, and these are + * forwarded values rather than substrate identity — so there is nothing to + * freeze at import and one env lookup per session start costs nothing. + * + * @param env - environment to read; defaults to `process.env`. Injectable so + * tests can vary it without reloading the module. + * @returns the block, with absent optional keys OMITTED (not `undefined`) so the + * agent's `key in platform_config` checks mean what they say and the serialized + * envelope carries no dead weight against the 4 KB cap. + * @throws Error naming every missing {@link MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS} + * entry, its environment variable, and the redeploy remedy. + */ +export function buildMicrovmPlatformConfig( + env: NodeJS.ProcessEnv = process.env, +): MicrovmPlatformConfig { + const config: Record = {}; + for (const key of MICROVM_PLATFORM_CONFIG_KEYS) { + const value = env[PLATFORM_CONFIG_ENV_VARS[key]]; + // Empty/whitespace-only is treated as ABSENT, matching the agent's own rule: + // CloudFormation renders an unresolved optional value as `''`, and sending + // that would either clobber an image value with nothing or build a request + // against a nameless table. Omitting says "this deployment has no such + // resource", which is the truth. + if (value !== undefined && value.trim() !== '') { + config[key] = value; + } + } + + const missing = MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS.filter(key => !(key in config)); + if (missing.length > 0) { + throw new Error( + 'Cannot start a lambda-microvm session: the orchestrator environment is missing platform ' + + `configuration the in-guest agent cannot run without (${missing + .map(key => `${key} <- ${PLATFORM_CONFIG_ENV_VARS[key]}`) + .join(', ')}). A MicroVM snapshot must not bake these in (ADR-021 sub-decision 3), so the ` + + '/run payload is the only channel for them. TaskOrchestrator injects every one of these ' + + 'from stack-level values, so this indicates the orchestrator function\'s environment was ' + + 'edited outside CDK — redeploy the stack to restore it.', + ); + } + + return config as MicrovmPlatformConfig; +} + /** * Stable marker prefixed onto every error this strategy lets escape, via * {@link wrapMicrovmError}. Load-bearing, not cosmetic: ``error-classifier`` @@ -288,15 +459,26 @@ export class LambdaMicrovmComputeStrategy implements ComputeStrategy { // inline branch is the exception. The MicroVM EXECUTION role holds the read // grant, exactly as the ECS task role does today. // - // Two keys, deliberately mirroring the ECS container env contract + // Three keys, deliberately mirroring the ECS container env contract // (AGENT_PAYLOAD / AGENT_PAYLOAD_S3_URI) so the agent's `/run` hook has one // self-describing shape to branch on: - // { "agent_payload": {...} } — inline - // { "agent_payload_s3_uri": "s3://..." } — pointer - const inlineEnvelope = JSON.stringify({ agent_payload: payload }); + // { "agent_payload": {...}, "platform_config": {...} } — inline + // { "agent_payload_s3_uri": "…", "platform_config": {...} } — pointer + // + // `platform_config` (see MICROVM_PLATFORM_CONFIG_KEYS) rides in BOTH forms, + // and is ALSO merged into the S3 object on the pointer path: + // s3://…//payload.json = { ...agent_payload, "platform_config": {…} } + // The duplication is deliberate and cheap (a few hundred bytes). It is the + // agent's env-block substitute — nothing else delivers it, because the + // snapshot must not bake it in — so it must be reachable whether the agent + // reads it off the hook body before fetching S3 or out of the fetched object. + const platformConfig = buildMicrovmPlatformConfig(); + const inlineEnvelope = JSON.stringify({ agent_payload: payload, platform_config: platformConfig }); // Measure the SERIALIZED envelope, not the bare payload: the envelope is - // what the service counts against the 4 KB cap. Byte length (not - // String.length) because a multi-byte prompt/diff makes chars an undercount. + // what the service counts against the 4 KB cap, and `platform_config` is part + // of it — which is precisely why nearly everything lands on the S3 path. + // Byte length (not String.length) because a multi-byte prompt/diff makes + // chars an undercount. const inlineBytes = Buffer.byteLength(inlineEnvelope, 'utf8'); let runHookPayload: string; @@ -307,7 +489,32 @@ export class LambdaMicrovmComputeStrategy implements ComputeStrategy { runHookPayload = inlineEnvelope; } else { const key = microvmPayloadKey(taskId); - const payloadJson = JSON.stringify(payload); + const uri = `s3://${MICROVM_PAYLOAD_BUCKET}/${key}`; + const pointerEnvelope = JSON.stringify({ + agent_payload_s3_uri: uri, + platform_config: platformConfig, + }); + // The pointer envelope is the LAST RESORT — there is no smaller shape to + // fall back to — so check it BEFORE the upload (an upload followed by a + // throw would leave an orphan object for the lifecycle rule to reap) and + // name the one thing an operator can actually act on. Unreachable in + // practice: the pointer plus all thirteen identifiers is well under 4 KB. + const pointerBytes = Buffer.byteLength(pointerEnvelope, 'utf8'); + if (pointerBytes > RUN_HOOK_PAYLOAD_LIMIT_BYTES) { + throw new Error( + `The MicroVM /run pointer envelope is ${pointerBytes} bytes, over the service's ` + + `${RUN_HOOK_PAYLOAD_LIMIT_BYTES}-byte runHookPayload cap, with the payload already moved ` + + 'to S3. The remaining size is the S3 URI plus the platform_config identifiers, so a ' + + 'pathologically long table/bucket/ARN name is the only possible cause — shorten the ' + + 'stack name (physical resource names derive from it) and redeploy.', + ); + } + // The S3 object carries the payload with `platform_config` merged in at the + // top level, so an agent that fetches the object gets the config with it. + // Platform config wins on a key collision — the payload has no + // `platform_config` key today, and if one ever appeared the platform's + // value is the authoritative one. + const payloadJson = JSON.stringify({ ...payload, platform_config: platformConfig }); try { await getS3Client().send(new PutObjectCommand({ Bucket: MICROVM_PAYLOAD_BUCKET, @@ -320,8 +527,8 @@ export class LambdaMicrovmComputeStrategy implements ComputeStrategy { // rather than letting a bare S3 exception name fall through to UNKNOWN. throw wrapMicrovmError('payload upload', err); } - payloadS3Uri = `s3://${MICROVM_PAYLOAD_BUCKET}/${key}`; - runHookPayload = JSON.stringify({ agent_payload_s3_uri: payloadS3Uri }); + payloadS3Uri = uri; + runHookPayload = pointerEnvelope; logger.info('Wrote MicroVM run-hook payload to S3', { task_id: taskId, bytes: Buffer.byteLength(payloadJson, 'utf8'), @@ -426,6 +633,11 @@ export class LambdaMicrovmComputeStrategy implements ComputeStrategy { maximum_duration_seconds: MICROVM_MAX_DURATION_SECONDS, payload_delivery: payloadS3Uri ? 's3_pointer' : 'inline', ...(payloadS3Uri && { payload_s3_uri: payloadS3Uri }), + // KEY NAMES only, never values: this is the one operator-visible record of + // which optional platform identifiers a given session actually received, and + // "the agent said ARTIFACTS_BUCKET_NAME is not configured" is otherwise a + // half-hour of guessing. Values stay out — see MICROVM_PLATFORM_CONFIG_KEYS. + platform_config_keys: Object.keys(platformConfig), }); return { diff --git a/cdk/src/handlers/shared/types.ts b/cdk/src/handlers/shared/types.ts index c73621447..b34b925b0 100644 --- a/cdk/src/handlers/shared/types.ts +++ b/cdk/src/handlers/shared/types.ts @@ -373,6 +373,22 @@ export interface TaskDetail { readonly updated_at: string; readonly started_at: string | null; readonly completed_at: string | null; + /** + * ISO timestamp of the agent's last heartbeat, written by the in-guest pipeline + * every 45 s on every compute backend (``agent/src/server.py`` + * ``_heartbeat_worker``). ``null`` before the first beat, on tasks that never + * ran, and on records predating the field. + * + * Surfaced (ADR-021 P2r2-F11) because it was the platform's only in-guest + * liveness signal and the API hid it: the orchestrator reads it for + * hang detection (``orchestrator.ts`` ``heartbeatLivenessApplies``) but + * ``toTaskDetail`` never mapped it, so ``bgagent status`` reported ``None`` + * while DynamoDB held a 6-second-old value. That gap produced a WRONG + * live-verification conclusion ("heartbeats not observed", attributed to a + * different defect entirely), which is the cost of an internal signal no + * operator can see. Keep in sync with ``cli/src/types.ts::TaskDetail``. + */ + readonly agent_heartbeat_at: string | null; readonly duration_s: number | null; readonly cost_usd: number | null; readonly build_passed: boolean | null; @@ -834,6 +850,9 @@ export function toTaskDetail( updated_at: record.updated_at, started_at: record.started_at ?? null, completed_at: record.completed_at ?? null, + // ADR-021 P2r2-F11: written by every backend, consumed by the orchestrator for + // hang detection, and — until this line — invisible to every API consumer. + agent_heartbeat_at: record.agent_heartbeat_at ?? null, duration_s: coerceNumericOrNull(record.duration_s, { ...ctx, field: 'duration_s' }, logger), cost_usd: coerceNumericOrNull(record.cost_usd, { ...ctx, field: 'cost_usd' }, logger), build_passed: record.build_passed ?? null, diff --git a/cdk/src/stacks/agent.ts b/cdk/src/stacks/agent.ts index 41c0351e0..bf0c6dbfc 100644 --- a/cdk/src/stacks/agent.ts +++ b/cdk/src/stacks/agent.ts @@ -36,7 +36,7 @@ import { AgentVpc } from '../constructs/agent-vpc'; import { ApiKeyTable } from '../constructs/api-key-table'; import { ApprovalMetricsPublisherConsumer } from '../constructs/approval-metrics-publisher-consumer'; import { AttachmentsBucket } from '../constructs/attachments-bucket'; -import { resolveBedrockModelIds } from '../constructs/bedrock-models'; +import { DEFAULT_HAIKU_INFERENCE_PROFILE_ID, resolveBedrockModelIds } from '../constructs/bedrock-models'; import { Blueprint } from '../constructs/blueprint'; import { CedarWasmLayer } from '../constructs/cedar-wasm-layer'; import { ConcurrencyReconciler } from '../constructs/concurrency-reconciler'; @@ -387,9 +387,11 @@ export class AgentStack extends Stack { ANTHROPIC_LOG: 'debug', // Cross-region inference-profile id (``us.`` prefix), NOT the bare // foundation-model id: Claude 4.x can't be invoked on-demand by bare id - // (400 "on-demand throughput isn't supported"). Must match a granted - // profile (see bedrock-models.ts). runner.py re-sets this at spawn time. - ANTHROPIC_DEFAULT_HAIKU_MODEL: 'us.anthropic.claude-haiku-4-5-20251001-v1:0', + // (400 "on-demand throughput isn't supported"). Read from bedrock-models.ts + // so the granted model and the delivered id cannot drift, and so the + // lambda-microvm `platform_config` block below carries the same value. + // runner.py re-sets this at spawn time. + ANTHROPIC_DEFAULT_HAIKU_MODEL: DEFAULT_HAIKU_INFERENCE_PROFILE_ID, TASK_TABLE_NAME: taskTable.table.tableName, TASK_EVENTS_TABLE_NAME: taskEventsTable.table.tableName, NUDGES_TABLE_NAME: taskNudgesTable.table.tableName, @@ -778,6 +780,22 @@ export class AgentStack extends Stack { // to the same per-task SessionRole the AgentCore runtime and the Fargate // task role use, so tenant-data access is tag-scoped on every substrate. agentSessionRole, + // ADR-021 P2 runtime parity on the MicroVM execution role. Same two props + // EcsAgentCluster takes, for the same reasons: the PAT is read at startup + // before the SessionRole is assumed, and MEMORY_ID (already delivered in + // agent_payload) makes the agent ATTEMPT a memory write that fails closed + // without the grant. The remaining parity grants (channel OAuth, Bedrock, + // AZ describe) need no stack input and are wired inside the construct. + githubTokenSecret, + agentMemory, + // ADR-021 P2-F4: the SAME log group whose name travels to the guest in + // `agentPlatformConfig.logGroupName` below (→ `LOG_GROUP_NAME`). P2 + // delivered the name without the grant, so the agent's structured per-task + // lines and its METRICS_REPORT were AccessDenied on + // logs:CreateLogStream and the platform's canonical observability streams + // were empty on this backend. Passing the construct (not the name) keeps the + // grant and the delivered value derived from one object. + applicationLogGroup, // Resolved above TaskApi — see `microvmImageInputs`. ...microvmImageInputs, }) @@ -860,6 +878,41 @@ export class AgentStack extends Stack { guardrailId: inputGuardrail.guardrailId, guardrailVersion: inputGuardrail.guardrailVersion, attachmentsBucket: attachmentsBucket.bucket, + // ADR-021 P2: non-secret platform identifiers the orchestrator forwards to + // the in-guest agent as `platform_config` on the MicroVM /run payload — the + // MicroVM equivalent of the AgentCore runtime env block above and the ECS + // container env, because a snapshot must not bake configuration in. + // + // Sourced from the SAME stack-level values that block uses, deliberately, so + // an agent behaves identically on all three substrates and a value can only + // be changed in one place. Wired unconditionally (not under the + // lambda-microvm gate) so the strategy's required-identifier guard can only + // ever fire for an environment edited outside CDK. + // + // No grant rides along: the orchestrator forwards these names and calls none + // of the resources they identify. + agentPlatformConfig: { + taskApprovalsTableName: taskApprovalsTable.table.tableName, + nudgesTableName: taskNudgesTable.table.tableName, + logGroupName: applicationLogGroup.logGroupName, + // INTENTIONAL, not a wiring bug: both keys resolve to the SAME bucket + // (`traceArtifactsBucket`), exactly as `ARTIFACTS_BUCKET_NAME` and + // `TRACE_ARTIFACTS_BUCKET_NAME` do in the AgentCore runtime env block above + // — a live P2 run flagged the coincidence (ADR-021 P2-F8) so it is recorded + // here rather than re-derived. They stay two keys because the agent reads + // them from two independent code paths with two different prefixes + // (`deliver_artifact` → `artifacts//`, `telemetry.py --trace` → + // `traces//.jsonl.gz`), and the per-task SessionRole + // scopes each prefix separately. Collapsing them to one key would make + // splitting the buckets later a cross-package contract change; sending one + // bucket through two keys costs nothing today. + artifactsBucketName: traceArtifactsBucket.bucket.bucketName, + traceArtifactsBucketName: traceArtifactsBucket.bucket.bucketName, + // The SessionRole is created above (before the orchestrator), so this needs + // no Lazy — it is the same CFN token the runtime env receives. + agentSessionRoleArn: agentSessionRole.role.roleArn, + anthropicDefaultHaikuModel: DEFAULT_HAIKU_INFERENCE_PROFILE_ID, + }, // Route ``compute_type: 'ecs'`` repos to the Fargate cluster above — // only when the cluster was synthesized (deploy --context compute_type=ecs). ...(ecsCluster && { diff --git a/cdk/test/bootstrap/__snapshots__/version.test.ts.snap b/cdk/test/bootstrap/__snapshots__/version.test.ts.snap index f8b1a2e7e..37981f70f 100644 --- a/cdk/test/bootstrap/__snapshots__/version.test.ts.snap +++ b/cdk/test/bootstrap/__snapshots__/version.test.ts.snap @@ -1,3 +1,3 @@ // Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing -exports[`bootstrap version module hash is stable 1`] = `"40d0a8b2343663084f614423fc3e6210761377dedf0bc079956ba7d4cea84c5e"`; +exports[`bootstrap version module hash is stable 1`] = `"99e9bd35471ed2397f8de0f97dd6fd7bc5b8d83891592448f7c2813be0a3ad0a"`; diff --git a/cdk/test/bootstrap/bootstrap-template.test.ts b/cdk/test/bootstrap/bootstrap-template.test.ts index f028501a1..80f185fe8 100644 --- a/cdk/test/bootstrap/bootstrap-template.test.ts +++ b/cdk/test/bootstrap/bootstrap-template.test.ts @@ -90,6 +90,30 @@ describe('Bootstrap template', () => { .toBe('IncludeComputeLambdaMicrovms'); }); + it('IaCRoleABCAComputeLambdaMicrovms carries the unconditioned MicrovmPassRoles statement', () => { + // ADR-021 P2r2-F9, asserted on the artifact operators actually deploy rather + // than only on the TypeScript source: CloudFormation cannot pass the MicroVM + // build role while an `iam:PassedToService` condition is in force, so the + // CDK-managed image path depends on this statement reaching the YAML with no + // Condition key. It lives in the CONDITIONAL per-backend policy, so an + // agentcore-only bootstrap never gains the unconditioned pass at all. + const statements = template.Resources.IaCRoleABCAComputeLambdaMicrovms + .Properties.PolicyDocument.Statement as Array<{ + Sid: string; + Action: string | string[]; + Resource: string | string[]; + Condition?: unknown; + }>; + const passRole = statements.find((s) => s.Sid === 'MicrovmPassRoles'); + expect(passRole).toBeDefined(); + expect(passRole!.Action).toBe('iam:PassRole'); + expect(passRole!.Condition).toBeUndefined(); + expect(passRole!.Resource).toEqual([ + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*', + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*', + ]); + }); + it('non-optional-compute policies do not have a condition', () => { const unconditional = expectedPolicies.filter( (p) => p !== 'IaCRoleABCAComputeEcs' && p !== 'IaCRoleABCAComputeLambdaMicrovms', diff --git a/cdk/test/bootstrap/policies.test.ts b/cdk/test/bootstrap/policies.test.ts index e3cdae579..eb9c5020f 100644 --- a/cdk/test/bootstrap/policies.test.ts +++ b/cdk/test/bootstrap/policies.test.ts @@ -64,6 +64,24 @@ describe('infrastructurePolicy', () => { expect(unique.size).toBe(sids.length); }); + it('KEEPS the iam:PassedToService allowlist on IAMPassRole', () => { + // ADR-021 P2r2-F9 added an UNCONDITIONED `iam:PassRole` to the + // `compute-lambda-microvm` policy, because the Lambda MicroVMs service presents + // no usable value for this key. That fix must not spread: this statement covers + // every role in the stack by prefix, so dropping its condition here would + // relax ~30 roles to fix two. The narrow statement lives in the conditional + // per-backend policy precisely so this one can stay as it is. + const resolvedDoc = stack.resolve(doc); + const statements = resolvedDoc.Statement as Array<{ + Sid: string; + Condition?: { StringEquals?: Record }; + }>; + const passRole = statements.find((st) => st.Sid === 'IAMPassRole')!; + const services = passRole.Condition?.StringEquals?.['iam:PassedToService']; + expect(services).toBeDefined(); + expect(services).toContain('lambda.amazonaws.com'); + }); + it('covers the expected service prefixes', () => { const resolvedDoc = stack.resolve(doc); const statements = resolvedDoc.Statement as Array<{ Action: string | string[] }>; @@ -419,7 +437,7 @@ describe('computeLambdaMicrovmPolicy', () => { const resolvedDoc = stack.resolve(doc); const statements = resolvedDoc.Statement as Array<{ Sid: string }>; - expect(statements.map((s) => s.Sid)).toEqual(['LambdaMicrovms']); + expect(statements.map((s) => s.Sid)).toEqual(['LambdaMicrovms', 'MicrovmPassRoles']); }); it('covers the expected service prefixes', () => { @@ -430,7 +448,84 @@ describe('computeLambdaMicrovmPolicy', () => { ); const prefixes = new Set(allActions.map((a) => a.split(':')[0])); - expect(prefixes).toEqual(new Set(['lambda'])); + // `iam` joins `lambda` as of the MicrovmPassRoles statement (ADR-021 P2r2-F9). + expect(prefixes).toEqual(new Set(['lambda', 'iam'])); + }); + + describe('MicrovmPassRoles (ADR-021 P2r2-F9)', () => { + function passRoleStatement() { + const resolvedDoc = stack.resolve(doc); + const statements = resolvedDoc.Statement as Array<{ + Sid: string; + Action: string | string[]; + Resource: string | string[]; + Condition?: unknown; + }>; + return statements.find((st) => st.Sid === 'MicrovmPassRoles')!; + } + + it('carries NO iam:PassedToService condition — the service presents no usable value', () => { + // The whole point of the statement. `infrastructure`'s IAMPassRole already + // matches these roles by prefix, but its `iam:PassedToService` allowlist is + // DENIED on this path: live 2026-08-07, CloudFormation could not pass the + // build role to CreateMicrovmImage ("...is not authorized to perform: + // iam:PassRole ... (Service: LambdaMicrovms, Status Code: 403)") while the + // out-of-band create-image call passed the SAME role successfully with + // unconditioned operator credentials. Re-adding a condition here re-breaks + // the CDK-managed image path. + expect(passRoleStatement().Condition).toBeUndefined(); + }); + + it('grants only iam:PassRole', () => { + expect(passRoleStatement().Action).toBe('iam:PassRole'); + }); + + it('is scoped to the build + connector-operator role prefixes only', () => { + const resources = passRoleStatement().Resource as string[]; + expect(resources).toEqual([ + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*', + 'arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*', + ]); + // NOT the stack-wide role prefix the conditioned statement uses: an + // unconditioned pass on `backgroundagent-dev-*` would drop the + // iam:PassedToService constraint for every role in the stack to fix two. + expect(resources).not.toContain('arn:aws:iam::*:role/backgroundagent-dev-*'); + }); + + it('does NOT cover the MicroVM execution role', () => { + // CloudFormation never passes it — the orchestrator does, at RunMicrovm, + // under its own exact-ARN grant (task-orchestrator.ts). Including it here + // would extend an unconditioned pass to the role that runs untrusted repo + // code, for no deploy-time reason. The live physical name is + // `backgroundagent-dev-LambdaMicrovmComputeExecutionRo-`. + const resources = passRoleStatement().Resource as string[]; + const executionRoleArn = + 'arn:aws:iam::123456789012:role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-abc123'; + const matches = resources.some((pattern) => { + const re = new RegExp(`^${pattern.replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`); + return re.test(executionRoleArn); + }); + expect(matches).toBe(false); + }); + + it('matches the live physical names CloudFormation generated for both roles', () => { + // Regression guard on the truncation window: CFN truncates the logical id to + // fit 64 chars before appending a random suffix, so a pattern that reaches + // past the cut silently matches nothing. These two ARNs are verbatim from the + // live run (the build role's is the one in the AccessDenied above). + const resources = passRoleStatement().Resource as string[]; + const live = [ + 'arn:aws:iam::704224321915:role/backgroundagent-dev-LambdaMicrovmComputeBuildRoleF0-9FxjQbiJC3px', + 'arn:aws:iam::704224321915:role/backgroundagent-dev-LambdaMicrovmComputeConnectorOp-Ab12Cd34Ef56', + ]; + for (const arn of live) { + const matched = resources.some((pattern) => { + const re = new RegExp(`^${pattern.replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*')}$`); + return re.test(arn); + }); + expect(matched).toBe(true); + } + }); }); it('covers both CFN resource types the construct synthesizes', () => { diff --git a/cdk/test/constructs/lambda-microvm-compute.test.ts b/cdk/test/constructs/lambda-microvm-compute.test.ts index 6e70e595a..5d47ed8aa 100644 --- a/cdk/test/constructs/lambda-microvm-compute.test.ts +++ b/cdk/test/constructs/lambda-microvm-compute.test.ts @@ -17,16 +17,24 @@ * SOFTWARE. */ +import { readFileSync } from 'node:fs'; +import { resolve } from 'node:path'; import { App, Stack } from 'aws-cdk-lib'; import { Match, Template } from 'aws-cdk-lib/assertions'; import * as dynamodb from 'aws-cdk-lib/aws-dynamodb'; import * as ec2 from 'aws-cdk-lib/aws-ec2'; import * as iam from 'aws-cdk-lib/aws-iam'; +import * as logs from 'aws-cdk-lib/aws-logs'; import * as s3 from 'aws-cdk-lib/aws-s3'; +import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager'; +import sharedConstants from '../../../contracts/constants.json'; +import { AgentMemory } from '../../src/constructs/agent-memory'; import { AgentSessionRole } from '../../src/constructs/agent-session-role'; +import { DEFAULT_BEDROCK_MODEL_IDS } from '../../src/constructs/bedrock-models'; import { DEFAULT_MINIMUM_MEMORY_MIB, LambdaMicrovmCompute, + MICROVM_AGENT_HOOK_ROUTES, MICROVM_ARTIFACT_OBJECT_KEY, MICROVM_BACKEND_TAG_KEY, MICROVM_BACKEND_TAG_VALUE, @@ -40,8 +48,20 @@ import { microvmNoIngressConnectorArn, } from '../../src/constructs/lambda-microvm-compute'; import { LAMBDA_MICROVM_SUPPORTED_REGIONS } from '../../src/handlers/shared/microvm-regions'; +// The `/ready` budgets are a cross-language RELATIONSHIP, so the tests read the +// same contract the construct and `agent/src/server.py` do — a literal here would +// keep passing after someone lowered the real budget. const BASE_IMAGE_ARN = 'arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1'; +const GITHUB_TOKEN_SECRET_ARN = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:abca/github-token-AbCdEf'; +/** + * Physical name of the stand-in APPLICATION_LOGS group, spelled like the real one + * (`stacks/agent.ts` → `RuntimeApplicationLogGroup`) so the grant assertions read + * against a recognisable ARN rather than a CDK-generated one. + */ +const APPLICATION_LOG_GROUP_NAME = + '/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/TestStack'; interface BuildOptions { readonly region?: string; @@ -50,6 +70,8 @@ interface BuildOptions { readonly externalImageIdentifier?: string; readonly externalImageVersion?: string; readonly withSessionRole?: boolean; + /** Wire the P2 runtime-parity props (GitHub PAT secret + AgentCore Memory). */ + readonly withRuntimeParity?: boolean; readonly regionAgnostic?: boolean; readonly minimumMemoryInMiB?: number; } @@ -104,6 +126,18 @@ function instantiate(options: BuildOptions = {}): Omit { const construct = new LambdaMicrovmCompute(stack, 'LambdaMicrovmCompute', { vpc, agentSessionRole, + ...(options.withRuntimeParity && { + githubTokenSecret: secretsmanager.Secret.fromSecretCompleteArn( + stack, 'GitHubTokenSecret', GITHUB_TOKEN_SECRET_ARN, + ), + agentMemory: new AgentMemory(stack, 'AgentMemory'), + // Stands in for the stack's RuntimeApplicationLogGroup — the group whose + // NAME travels to the guest as platform_config.log_group_name, so the grant + // and the delivered value must come from one object (ADR-021 P2-F4). + applicationLogGroup: new logs.LogGroup(stack, 'ApplicationLogGroup', { + logGroupName: APPLICATION_LOG_GROUP_NAME, + }), + }), ...(options.withImage && { baseImageArn: BASE_IMAGE_ARN, baseImageVersion: '1', @@ -127,7 +161,7 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', let template: Template; beforeAll(() => { - built = build({ withImage: true, withSessionRole: true }); + built = build({ withImage: true, withSessionRole: true, withRuntimeParity: true }); template = built.template; }); @@ -154,8 +188,15 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', // are: [512, 1024, 2048, 4096, 8192]." Note this configures the BASELINE — // the service scales vertically to a 32 GiB / 16 vCPU peak on its own, which // is why nothing here asks for the peak. + // + // `ARM_64`, not `arm64`: the CDK L1 types Architecture as a plain string and + // documents no allowed values, and CloudFormation rejected the lowercase + // spelling at change-set early validation — "arm64 is not a valid enum value. + // Supported values: [ARM_64]" (ADR-021 P2-F2). The literal is spelled out here + // rather than imported from the construct so the test fails if the constant is + // "corrected" back to Docker's spelling. template.hasResourceProperties('AWS::Lambda::MicrovmImage', { - CpuConfigurations: [{ Architecture: 'arm64' }], + CpuConfigurations: [{ Architecture: 'ARM_64' }], Resources: [{ MinimumMemoryInMiB: 8192 }], }); expect(DEFAULT_MINIMUM_MEMORY_MIB).toBe(8192); @@ -163,28 +204,195 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', expect(Math.max(...MICROVM_SUPPORTED_MEMORY_MIB)).toBe(DEFAULT_MINIMUM_MEMORY_MIB); }); - test('configures /ready + /run and NOTHING else (the rest fail their transition)', () => { + test('declares EXACTLY the four hooks the agent serves (P2), and no more', () => { + // `toEqual` on the whole object, not per-key assertions: the invariant runs in + // BOTH directions. A hook the agent serves but the image does not declare is + // never called (the P2 R2 regression this replaces — the agent gained + // /validate and /terminate while the construct still advertised two hooks); + // a hook the image declares but the agent does not serve fails the + // corresponding build or lifecycle transition. Only an exact set catches both. const images = template.findResources('AWS::Lambda::MicrovmImage'); const hooks = Object.values(images)[0]!.Properties.Hooks; expect(hooks.Port).toBe(8080); - // Paths are the routes `agent/src/server.py` actually serves (the API model - // has no path field at all, so a made-up path would be a silent lie). + + // RUNTIME hooks. The VALUE of each hook field is the `ENABLED` enum, NOT the + // agent's route: CloudFormation rejected all four paths at change-set early + // validation — "/aws/lambda-microvms/runtime/v1/run is not a valid enum value. + // Supported values: [DISABLED, ENABLED]" (ADR-021 P2-F2). The CFN surface is + // identical to the API surface here; the routes are fixed and service-owned + // (asserted separately against MICROVM_AGENT_HOOK_ROUTES below). expect(hooks.MicrovmHooks).toEqual({ - Run: '/aws/lambda-microvms/runtime/v1/run', + Run: 'ENABLED', RunTimeoutInSeconds: 60, + Terminate: 'ENABLED', + // Near the BOTTOM of the service's 1–60 s window on purpose: the handler is + // a log-and-acknowledge with nothing to drain (progress writes are already + // durable per event), and the budget bounds how long teardown waits on a + // WEDGED guest that is still holding admission-gating memory quota. + TerminateTimeoutInSeconds: 15, }); - // /ready is MANDATORY: create-microvm-image refuses ANY lifecycle hook - // without it, so "declare /run in P1, serve it in P2" was unreachable. + + // BUILD (image) hooks. /ready is MANDATORY: create-microvm-image refuses ANY + // lifecycle hook without it, so "declare /run in P1, serve it in P2" was + // unreachable. expect(hooks.MicrovmImageHooks).toEqual({ - Ready: '/aws/lambda-microvms/runtime/v1/ready', - ReadyTimeoutInSeconds: 60, + Ready: 'ENABLED', + // 300 s, not 60: since the P2-F5 fix /ready warms the 225 MiB `claude` + // binary before the snapshot is captured, so it does real work whose + // duration is a cold exec. Build hooks allow up to 3600 s, so a tight + // budget here would trade a runtime failure for a build failure. Read from + // the contract, not re-typed: the value is half of the cross-language + // invariant `warmup_total < ready_hook` (see the dedicated test below). + ReadyTimeoutInSeconds: sharedConstants.microvm_hook_budgets.ready_hook_timeout_seconds, + Validate: 'ENABLED', + // Decoupled from /ready by that same change: /validate's checks are still + // sub-millisecond, so its budget is sized only for the still-initialising + // 503 path and must NOT inherit /ready's warm-up allowance. + ValidateTimeoutInSeconds: 60, }); - // A hook the service calls but the agent does not serve fails the lifecycle - // transition (or every build), so nothing else may be advertised. + }); + + test('sends NO hook path as a property value — the fields are enums (P2-F2)', () => { + // The regression guard for the defect that made the whole CDK-managed image + // path non-functional: the L1 types every hook field as `string` and documents + // no allowed values, which is how four route strings got sent as property + // values and were rejected at change-set validation — before the stack was + // touched, so there was no rollback and no runtime symptom to trace back. + // Asserting on the whole rendered resource (not just Hooks) also catches a + // route leaking into Description, a tag, or a future property. + const images = template.findResources('AWS::Lambda::MicrovmImage'); + const rendered = JSON.stringify(Object.values(images)[0]!); + expect(rendered).not.toContain('/aws/lambda-microvms/runtime/v1'); + for (const route of Object.values(MICROVM_AGENT_HOOK_ROUTES)) { + expect(rendered).not.toContain(route); + } + }); + + test("the out-of-band script's API request matches the CDK-managed image", () => { + // ADR-021 P2-F2/P2-F5 drift guard, and the reason it exists is that the prose + // version of it FAILED: `cdk/scripts/package-microvm-artifact.sh` said "the + // timeouts mirror the construct's constants … keep the two in step", and when + // READY_HOOK_TIMEOUT_SECONDS went 60 → 300 for the /ready warm-up, the script + // kept sending 60. Nothing caught it, because the two paths never meet in code: + // a bash helper cannot import a TypeScript constant. + // + // The two requests MUST agree. An operator reaches for --create-image exactly + // when the CDK path is failing, i.e. while debugging — so an out-of-band image + // that behaves differently from a CDK-built one turns the fallback into a + // second variable. This test parses the script's real flag values and compares + // them to the synthesized template. + const script = readFileSync( + resolve(__dirname, '../../scripts/package-microvm-artifact.sh'), 'utf8', + ); + + /** Value of a single-quoted `--flag ''` argument in the script. */ + const flagJson = (flag: string): unknown => { + const match = new RegExp(`--${flag} '([^']+)'`).exec(script); + expect(match).not.toBeNull(); + return JSON.parse(match![1]!); + }; + + /** The API's camelCase keys → CloudFormation's PascalCase, recursively. */ + const toCfnKeys = (value: unknown): unknown => { + if (Array.isArray(value)) return value.map(toCfnKeys); + if (value === null || typeof value !== 'object') return value; + return Object.fromEntries( + Object.entries(value as Record) + .map(([key, inner]) => [key[0]!.toUpperCase() + key.slice(1), toCfnKeys(inner)]), + ); + }; + + const image = Object.values(template.findResources('AWS::Lambda::MicrovmImage'))[0]!; + // Hooks: all four states AND all four timeouts, in one comparison — which is + // precisely the assertion the missing one would have been. + expect(toCfnKeys(flagJson('hooks'))).toEqual(image.Properties.Hooks); + // ...and the architecture enum, the other half of P2-F2. + expect(toCfnKeys(flagJson('cpu-configurations'))).toEqual(image.Properties.CpuConfigurations); + }); + + test('does NOT declare /suspend or /resume — they are P3 and nothing answers them yet', () => { + // The remaining half of the exactness rule, called out separately because it + // is the one that must survive P3 landing suspend/resume in ONE commit across + // all three strategies: until then, declaring either fails the corresponding + // lifecycle transition on a real suspend attempt. + const images = template.findResources('AWS::Lambda::MicrovmImage'); + const hooks = Object.values(images)[0]!.Properties.Hooks; expect(hooks.MicrovmHooks.Suspend).toBeUndefined(); + expect(hooks.MicrovmHooks.SuspendTimeoutInSeconds).toBeUndefined(); expect(hooks.MicrovmHooks.Resume).toBeUndefined(); - expect(hooks.MicrovmHooks.Terminate).toBeUndefined(); - expect(hooks.MicrovmImageHooks.Validate).toBeUndefined(); + expect(hooks.MicrovmHooks.ResumeTimeoutInSeconds).toBeUndefined(); + }); + + test('the agent hook routes are exactly the four the service calls, under one prefix', () => { + // The cross-package contract that used to be checked against the rendered + // template. It cannot be any more: the template carries `ENABLED`, not a path + // (P2-F2), so the routes now have a dedicated source — MICROVM_AGENT_HOOK_ROUTES + // — and this asserts THAT against `MICROVM_HOOK_PREFIX` in + // `agent/src/server.py`. A prefix drift is invisible at synth and at deploy; it + // surfaces as a failed image build (/ready, /validate) or a failed lifecycle + // transition on a real task (/run, /terminate). Live 2026-08-06 confirmed the + // service POSTs to exactly these paths ("POST /aws/lambda-microvms/runtime/v1/ + // ready HTTP/1.1" 200 OK, and the same for the other three). + const routes = Object.values(MICROVM_AGENT_HOOK_ROUTES); + expect(routes).toHaveLength(4); + for (const route of routes) { + expect(route).toMatch(/^\/aws\/lambda-microvms\/runtime\/v1\/(ready|validate|run|terminate)$/); + } + expect([...routes].sort()).toEqual([ + '/aws/lambda-microvms/runtime/v1/ready', + '/aws/lambda-microvms/runtime/v1/run', + '/aws/lambda-microvms/runtime/v1/terminate', + '/aws/lambda-microvms/runtime/v1/validate', + ]); + // The map's KEYS are the service's hook names, i.e. the same names the L1's + // Hooks properties use — so "the agent serves every hook the image enables" + // stays checkable from one place. + expect(Object.keys(MICROVM_AGENT_HOOK_ROUTES).sort()) + .toEqual(['ready', 'run', 'terminate', 'validate']); + }); + + test('every declared hook timeout sits inside the service window for its kind', () => { + // Runtime hooks are capped at 60 s; build hooks allow up to 3600 s. A value + // outside its window is rejected at image-create time — minutes into a build, + // after the artifact has already been packaged and uploaded. The build-hook + // ceiling is what makes /ready's 300 s warm-up budget legal (P2-F5). + const images = template.findResources('AWS::Lambda::MicrovmImage'); + const hooks = Object.values(images)[0]!.Properties.Hooks; + + for (const key of ['RunTimeoutInSeconds', 'TerminateTimeoutInSeconds']) { + const value = (hooks.MicrovmHooks as Record)[key]!; + expect(value).toBeGreaterThanOrEqual(1); + expect(value).toBeLessThanOrEqual(60); + } + for (const key of ['ReadyTimeoutInSeconds', 'ValidateTimeoutInSeconds']) { + const value = (hooks.MicrovmImageHooks as Record)[key]!; + expect(value).toBeGreaterThanOrEqual(1); + expect(value).toBeLessThanOrEqual(3_600); + } + }); + + test("/ready's budget comes from the contract and outlasts the agent's warm-up", () => { + // The invariant that used to be asserted twice against two literals — a 300 in + // this tree and a 300 in `agent/tests/test_server.py`. Neither side could see + // the other, so lowering the hook budget would have silently made the agent's + // warm-up (P2-F5: warm the 225 MiB `claude` binary before the snapshot) unable + // to finish inside it, turning a runtime fix into a build failure. Both halves + // now live in `contracts/constants.json` → `microvm_hook_budgets`; + // `scripts/check-constants-sync.ts` enforces the ordering and bans a literal + // re-declaration on either side, and this test confirms the value the contract + // holds is the value that reaches CloudFormation. + const budgets = sharedConstants.microvm_hook_budgets; + const images = template.findResources('AWS::Lambda::MicrovmImage'); + const imageHooks = Object.values(images)[0]!.Properties.Hooks.MicrovmImageHooks; + + expect(imageHooks.ReadyTimeoutInSeconds).toBe(budgets.ready_hook_timeout_seconds); + expect(budgets.warmup_total_budget_seconds).toBeLessThan(budgets.ready_hook_timeout_seconds); + expect(budgets.warmup_required_timeout_seconds) + .toBeLessThan(budgets.warmup_total_budget_seconds); + // Real margin for uvicorn scheduling plus the request itself, not a rounding + // error — the same floor `agent/tests/test_server.py` asserts from its side. + expect(budgets.ready_hook_timeout_seconds - budgets.warmup_total_budget_seconds) + .toBeGreaterThanOrEqual(30); }); test('bakes NO environment variables into the snapshot (ADR-021: no secrets in the image)', () => { @@ -261,14 +469,16 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', const [, role] = Object.entries(template.findResources('AWS::IAM::Role')) .find(([id]) => id.includes('LambdaMicrovmComputeConnectorOperatorRole'))!; - // Same confused-deputy posture as the build/execution roles. + // Same trust posture as the build/execution roles, and the same reason it + // carries NO condition — see the dedicated test below. This role is where the + // defect was FIRST observed: with aws:SourceAccount present, both connectors + // CREATE_FAILED deterministically with "The service is unable to assume the + // provided NetworkConnectorOperatorRole" (ADR-021 P2-F1). const statements = role.Properties.AssumeRolePolicyDocument.Statement; expect(statements).toHaveLength(1); expect(statements[0].Action).toBe('sts:AssumeRole'); expect(statements[0].Principal).toEqual({ Service: 'lambda.amazonaws.com' }); - expect(statements[0].Condition).toEqual({ - StringEquals: { 'aws:SourceAccount': '123456789012' }, - }); + expect(statements[0].Condition).toBeUndefined(); // The AWS-managed policy for exactly this job... expect(JSON.stringify(role.Properties.ManagedPolicyArns)) @@ -394,7 +604,7 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', expect(rules[0].ExpirationInDays).toBeUndefined(); }); - test('both roles are trusted by lambda.amazonaws.com for AssumeRole AND TagSession, pinned to this account', () => { + test('both roles are trusted by lambda.amazonaws.com for AssumeRole AND TagSession', () => { const roles = Object.entries(template.findResources('AWS::IAM::Role')) .filter(([id]) => id.includes('LambdaMicrovmComputeBuildRole') || id.includes('LambdaMicrovmComputeExecutionRole')); expect(roles).toHaveLength(2); @@ -407,16 +617,52 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', // `microvms.lambda.amazonaws.com` does not exist — using it is rejected // with MalformedPolicyDocument. expect(statement.Principal).toEqual({ Service: 'lambda.amazonaws.com' }); - // Confused-deputy: lambda.amazonaws.com is shared with every other - // Lambda feature, so aws:SourceAccount must be on BOTH actions. - expect(statement.Condition).toEqual({ - StringEquals: { 'aws:SourceAccount': '123456789012' }, - }); - expect(JSON.stringify(statement.Condition)).not.toContain('aws:SourceArn'); } } }); + test('NO source-key condition on any MicroVM-facing role trust (P2-F1/F3)', () => { + // The sharpest IAM assertion in this file, and the one most likely to be + // "fixed" back by a reviewer applying the standard service-principal + // confused-deputy pattern. It must not be. + // + // The Lambda MicroVMs service presents NO source condition key when it assumes + // these roles, so a trust policy carrying one is unassumable. Live 2026-08-06/07 + // (evidence inlined in ADR-021 §4; `docs/verification/645-p2-smoke-runbook.md` + // is the raw session log, additional detail rather than the sole proof), one + // root cause, two symptoms: + // both network connectors CREATE_FAILED deterministically on a freshly deleted + // stack (P2-F1), and RunMicrovm reported a MISLEADING caller-side + // `iam:PassRole` AccessDenied on the orchestrator (P2-F3) — with the grant + // present, `simulate-principal-policy` returning `allowed`, no permissions + // boundary, and an unconditioned PassRole ALSO denied. Removing the execution + // role's trust conditions made the next submission reach RUNNING in 6 s. + // + // What compensates is asserted elsewhere in this file and in + // `test/constructs/task-orchestrator.test.ts`: the EXECUTION role is passable by + // the orchestrator only, scoped to its EXACT ARN — and with NO + // `iam:PassedToService` condition either, because the same missing-context-key + // root cause blocks that path too (P2r2-F10), which is why the exact ARN is the + // whole of the scoping. Every resource these roles reach is account-scoped by + // ARN apart from two justified `Resource: '*'` statements. + const roles = Object.entries(template.findResources('AWS::IAM::Role')) + .filter(([id]) => id.includes('LambdaMicrovmComputeBuildRole') + || id.includes('LambdaMicrovmComputeExecutionRole') + || id.includes('LambdaMicrovmComputeConnectorOperatorRole')); + expect(roles).toHaveLength(3); + + for (const [, role] of roles) { + const trust = role.Properties.AssumeRolePolicyDocument; + for (const statement of trust.Statement) { + expect(statement.Condition).toBeUndefined(); + } + const rendered = JSON.stringify(trust); + expect(rendered).not.toContain('aws:SourceAccount'); + expect(rendered).not.toContain('aws:SourceArn'); + expect(rendered).not.toContain('aws:SourceOrgID'); + } + }); + test('build role reads exactly the one artifact object and writes MicroVM logs', () => { const policies = Object.entries(template.findResources('AWS::IAM::Policy')) .filter(([id]) => id.includes('LambdaMicrovmComputeBuildRole')); @@ -453,14 +699,204 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', } }); - test('execution role has NO Bedrock / Secrets Manager / DynamoDB grants (P2 scope)', () => { - const policies = Object.entries(template.findResources('AWS::IAM::Policy')) - .filter(([id]) => id.includes('LambdaMicrovmComputeExecutionRole')); - const rendered = JSON.stringify(policies); + /** + * Every statement on the execution role's inline policies, flattened. The role's + * grants arrive from several sources (CDK `grantRead`/`grantReadWrite` plus + * hand-written statements), and CDK may split them across policies, so the + * assertions below work from one flattened list rather than a policy index. + */ + function executionRoleStatements(): Array<{ + Action: string | string[]; + Resource?: unknown; + Condition?: unknown; + }> { + return Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('LambdaMicrovmComputeExecutionRole')) + .flatMap(([, p]) => p.Properties.PolicyDocument.Statement); + } + + /** Statements whose action set includes `action`. */ + function statementsWithAction(action: string) { + return executionRoleStatements().filter((statement) => { + const actions = Array.isArray(statement.Action) ? statement.Action : [statement.Action]; + return actions.includes(action); + }); + } + + // --- ADR-021 P2 runtime parity on the execution role --- + // + // These replace P1's "the execution role has NO Bedrock / Secrets Manager / + // DynamoDB grants" assertion, which was a scope marker rather than a property: + // P2 is the phase that adds them. What remains a real, permanent property — and + // is still asserted below — is that DynamoDB is NOT among them. + + test('execution role reads the GitHub PAT secret (needed before the SessionRole is assumed)', () => { + const statements = statementsWithAction('secretsmanager:GetSecretValue'); + const rendered = JSON.stringify(statements); + expect(rendered).toContain(GITHUB_TOKEN_SECRET_ARN); + // grantRead, not write: the agent consumes the PAT, it never rotates it. + expect(rendered).not.toContain('secretsmanager:PutSecretValue'); + expect(rendered).not.toContain('secretsmanager:UpdateSecret'); + }); + + test('execution role gets the channel-OAuth PREFIX grant, GetSecretValue only', () => { + // Per-workspace secrets are created by the CLI at setup, so the name is + // unknown at synth and a prefix is the only expressible scope (mirroring + // ecs-agent-cluster). Without it a Linear/Jira task's 👀→✅ reaction and the + // channel MCP silently no-op. + const prefixStatement = executionRoleStatements().find( + statement => JSON.stringify(statement.Resource).includes('bgagent-linear-oauth-*'), + )!; + expect(prefixStatement).toBeDefined(); + expect(prefixStatement.Action).toBe('secretsmanager:GetSecretValue'); + const resources = JSON.stringify(prefixStatement.Resource); + expect(resources).toContain('bgagent-linear-oauth-*'); + expect(resources).toContain('bgagent-jira-oauth-*'); + // Scoped to THIS account/Region's secrets, and to those two prefixes only — + // never `secret:*`. + expect(resources).not.toContain('secret:*'); + }); + + test('execution role Bedrock grant is scoped to explicit model + inference-profile ARNs', () => { + const statements = statementsWithAction('bedrock:InvokeModel'); + expect(statements).toHaveLength(1); + const statement = statements[0]!; + expect(statement.Action).toEqual([ + 'bedrock:InvokeModel', + 'bedrock:InvokeModelWithResponseStream', + ]); + + const resources = statement.Resource as unknown[]; + // Two ARNs per model: the all-Regions foundation model and its `us.` + // cross-Region inference profile — the same derivation the AgentCore runtime + // and the ECS task role use, from the same shared model list. + expect(resources).toHaveLength(DEFAULT_BEDROCK_MODEL_IDS.length * 2); + const rendered = JSON.stringify(resources); + for (const modelId of DEFAULT_BEDROCK_MODEL_IDS) { + expect(rendered).toContain(`:bedrock:*::foundation-model/${modelId}`); + expect(rendered).toContain(`inference-profile/us.${modelId}`); + } + // NEVER a wildcard resource — this role runs untrusted repo code. + expect(resources).not.toContain('*'); + }); + + test('execution role gets AgentCore Memory read+write so learning actually persists', () => { + // MEMORY_ID reaches the agent in agent_payload either way, so without the + // grant the write is ATTEMPTED and fails closed (AccessDenied → logged, + // non-fatal), i.e. learning silently never persists on this substrate. + const rendered = JSON.stringify(executionRoleStatements()); + expect(rendered).toContain('bedrock-agentcore:CreateEvent'); + expect(rendered).toContain('bedrock-agentcore:RetrieveMemoryRecords'); + }); + + test('execution role can write to the APPLICATION_LOGS group platform_config names', () => { + // ADR-021 P2-F4. P2 delivered `log_group_name` in platform_config — which makes + // the agent ATTEMPT the write — without the matching grant, so every structured + // per-task line AND the METRICS_REPORT were denied live: + // "…LambdaMicrovmComputeExecutionRo…/Lambda-microvmsExecutor-… is not + // authorized to perform: logs:CreateLogStream on resource: + // …:/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/…" + // Non-fatal (stdout fallback lands in the MicroVM log group) which is exactly + // why it survived P2: the substrate looked fine while the platform's canonical + // observability streams were empty. + const statements = executionRoleStatements().filter((statement) => { + const resource = JSON.stringify(statement.Resource); + return resource.includes('ApplicationLogGroup'); + }); + expect(statements).toHaveLength(1); + // Write-only, and only the two actions the agent's writer calls — no + // CreateLogGroup (the stack owns the group and its retention), no read. + expect(statements[0]!.Action).toEqual(['logs:CreateLogStream', 'logs:PutLogEvents']); + // Scoped to that ONE group's ARN (whose trailing `:*` is the log-STREAM + // wildcard — streams are minted per task), never to a log-group prefix. + const rendered = JSON.stringify(statements[0]!.Resource); + expect(rendered).toContain('ApplicationLogGroup'); + expect(rendered).not.toContain(`${MICROVM_LOG_GROUP_PREFIX}/*`); + }); + + test('the two logs grants stay separate — one namespace cannot cover the other', () => { + // The service's own `/aws/lambda-microvms/*` grant and the platform's + // APPLICATION_LOGS grant are unrelated namespaces, so neither can be widened + // into the other. Asserting both are present keeps a future "consolidation" + // from silently dropping one. + const logsStatements = executionRoleStatements().filter((statement) => { + const actions = Array.isArray(statement.Action) ? statement.Action : [statement.Action]; + return actions.some(action => action.startsWith('logs:')); + }); + expect(logsStatements).toHaveLength(2); + const rendered = JSON.stringify(logsStatements); + expect(rendered).toContain(`${MICROVM_LOG_GROUP_PREFIX}/*`); + expect(rendered).toContain('ApplicationLogGroup'); + // `logs:*` would satisfy both and is exactly what must not happen. + expect(rendered).not.toContain('"logs:*"'); + }); + + test('execution role can describe AZs, for a CDK target repo\'s synth build gate', () => { + const statements = statementsWithAction('ec2:DescribeAvailabilityZones'); + expect(statements).toHaveLength(1); + // EC2 describe actions have no resource-level scoping, so Resource:* is + // mandatory; it is read-only with no mutation and no data access. Without it + // `cdk synth` in a freshly-cloned CDK repo AccessDenies the AZ context lookup + // and the build gate fails on code that builds fine everywhere else. + expect(statements[0]!.Resource).toBe('*'); + // ...and it is the ONLY ec2 action granted — the connector operator role owns + // ENI management, not this role. + const ec2Actions = executionRoleStatements() + .flatMap(st => (Array.isArray(st.Action) ? st.Action : [st.Action])) + .filter(action => action.startsWith('ec2:')); + expect(ec2Actions).toEqual(['ec2:DescribeAvailabilityZones']); + }); + + test('execution role still has NO direct DynamoDB grant — tenant data goes via the SessionRole', () => { + // The permanent property. Every table the agent touches is task_id-partitioned + // and reachable only through the SessionRole's `dynamodb:LeadingKeys` + // condition; a direct grant here would hand a role running untrusted repo code + // cross-task read/write and quietly break per-tenant isolation. Note the + // asymmetry with ecs-agent-cluster, which keeps a legacy no-SessionRole + // fallback branch — this backend has none. + expect(JSON.stringify(executionRoleStatements())).not.toContain('dynamodb:'); + }); + + test('execution role gets no artifacts/trace bucket grant (delivery rides the SessionRole)', () => { + // The only S3 the execution role may reach is the payload bucket (read-only, + // asserted above). Artifact delivery writes go through the SessionRole's + // `artifacts/${task_id}/*` statement — the AgentCore runtime role has no direct + // grant either, and granting the whole bucket here would let one task read or + // clobber another's artifacts, traces and attachments. + const s3Resources = executionRoleStatements() + .flatMap(st => (Array.isArray(st.Action) ? st.Action : [st.Action])) + .filter(action => action.startsWith('s3:')); + expect(s3Resources).toEqual(['s3:GetObject*', 's3:GetBucket*', 's3:List*']); + const rendered = JSON.stringify( + executionRoleStatements().filter((st) => { + const actions = Array.isArray(st.Action) ? st.Action : [st.Action]; + return actions.some(action => action.startsWith('s3:')); + }), + ); + expect(rendered).toContain('LambdaMicrovmComputePayloadBucket'); + expect(rendered).not.toContain('TraceBucket'); + expect(rendered).not.toContain('AttachmentsBucket'); + }); + + test('the BUILD role gains none of the P2 runtime grants', () => { + // The build role runs the /ready (and future /validate) hooks — i.e. code from + // the repo under build, at image-build time — so it must stay at + // "one artifact object + logs". A P2 grant leaking onto it would give + // build-time repo code Bedrock/Secrets/Memory reach. + const rendered = JSON.stringify( + Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('LambdaMicrovmComputeBuildRole')), + ); expect(rendered).not.toContain('bedrock:'); + expect(rendered).not.toContain('bedrock-agentcore:'); expect(rendered).not.toContain('secretsmanager:'); expect(rendered).not.toContain('dynamodb:'); - expect(rendered).not.toContain('bedrock-agentcore:'); + expect(rendered).not.toContain('ec2:'); + // ...including the P2-F4 application-log-group grant: the build hooks log to + // stdout only (`_build_hook_log`), precisely so no build-time Logs write is + // attempted, and the build role's own `/aws/lambda-microvms/*` grant is what + // carries the service's build logs. + expect(rendered).not.toContain('ApplicationLogGroup'); }); test('execution role is admitted to the per-task SessionRole (tenant-data delegation)', () => { @@ -483,11 +919,11 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', expect(JSON.stringify(template.toJSON())).not.toContain('CreateMicrovmAuthToken'); }); - test('warns that a P1 image has no smoke-parity guarantee (hook phasing)', () => { - // ADR-021 sub-decision 3, as corrected by the live P1 run: P1 declares AND - // serves /ready + /run, so the image is creatable, launchable and - // payload-deliverable — which makes it look even more like a working backend - // than before, while nothing has exercised clone → change → PR on it. + test('warns that a configured image has no smoke-parity guarantee (hook phasing)', () => { + // ADR-021 sub-decision 3, as corrected by the live P1 run and completed in P2: + // all four served hooks are declared, so the image is creatable, launchable and + // payload-deliverable — which makes it look even MORE like a working backend, + // while nothing has exercised clone → change → PR on it. const warnings = built.construct.node.metadata.filter(m => m.type === 'aws:cdk:warning'); const message = warnings.map(w => String(w.data)).join('\n'); expect(JSON.stringify(built.construct.node.metadata)) @@ -497,19 +933,33 @@ describe('LambdaMicrovmCompute — image provisioned from a managed base image', .not.toContain('abca:microvm-image-p1-not-runnable'); expect(message).toContain('smoke'); expect(message).toContain('P2'); - // It must state what IS true now, or it reads as the old (wrong) claim. - expect(message).toContain('/run'); - expect(message).toContain('/ready'); + // It must state what IS true now, or it reads as the old (wrong) claim — and + // the hook list here is what an operator compares against a failed build or a + // failed lifecycle transition, so all four have to be named. + for (const hook of ['/ready', '/validate', '/run', '/terminate']) { + expect(message).toContain(hook); + } + // ...and it must still say which two are NOT declared, or the enumeration + // above reads as "everything is wired". + expect(message).toContain('/suspend'); + expect(message).toContain('/resume'); }); - test('declares no hook the agent does not serve yet', () => { + test('enables every hook the agent serves, and only those (rendered form)', () => { + // Same invariant as the structural assertions above, checked against the + // rendered template — this is the shape an operator reads in a `cdk diff`, and + // the shape CloudFormation validates. `"ENABLED"`, never a path (P2-F2). const images = template.findResources('AWS::Lambda::MicrovmImage'); const rendered = JSON.stringify(Object.values(images)[0]!.Properties.Hooks); - for (const hook of ['Suspend', 'Resume', 'Terminate', 'Validate']) { + for (const hook of ['Run', 'Terminate', 'Ready', 'Validate']) { + expect(rendered).toContain(`"${hook}":"ENABLED"`); + } + // P3, and nothing answers them yet. OMITTED rather than "DISABLED", so the + // absence assertion stays meaningful. + for (const hook of ['Suspend', 'Resume']) { expect(rendered).not.toContain(hook); } - expect(rendered).toContain('"Run":"/aws/lambda-microvms/runtime/v1/run"'); - expect(rendered).toContain('"Ready":"/aws/lambda-microvms/runtime/v1/ready"'); + expect(rendered).not.toContain('DISABLED'); }); test('tags every MicroVM resource with the backend cost-allocation tag', () => { @@ -634,6 +1084,48 @@ describe('LambdaMicrovmCompute — first deploy, no image configured', () => { }); }); +describe('LambdaMicrovmCompute — optional runtime-parity props omitted', () => { + // Isolated-construct posture: the two PROP-driven parity grants disappear, while + // the three that need no stack input stay. Worth pinning because "the grant is + // conditional" is only half a contract — which half is conditional matters. + let template: Template; + + beforeAll(() => { + template = build({ withImage: true, withSessionRole: true }).template; + }); + + function executionRolePolicies(): string { + return JSON.stringify( + Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('LambdaMicrovmComputeExecutionRole')), + ); + } + + test('no GitHub PAT read and no AgentCore Memory grant without the props', () => { + expect(executionRolePolicies()).not.toContain(GITHUB_TOKEN_SECRET_ARN); + expect(executionRolePolicies()).not.toContain('bedrock-agentcore:'); + }); + + test('no APPLICATION_LOGS grant without the log group, and the MicroVM one remains', () => { + // The application-log-group grant is prop-driven (isolated construct tests have + // no stack log group), so its absence here is the contract — but the service's + // own /aws/lambda-microvms/* grant must NOT be conditional, or a MicroVM cannot + // write build/run logs at all. + const rendered = executionRolePolicies(); + expect(rendered).not.toContain('ApplicationLogGroup'); + expect(rendered).toContain(`${MICROVM_LOG_GROUP_PREFIX}/*`); + }); + + test('the input-free parity grants are still there (Bedrock, channel OAuth, AZ describe)', () => { + const rendered = executionRolePolicies(); + // These derive from the shared model list / a fixed secret-name prefix / no + // resource at all, so nothing about a deployment can make them optional. + expect(rendered).toContain('bedrock:InvokeModel'); + expect(rendered).toContain('bgagent-linear-oauth-*'); + expect(rendered).toContain('ec2:DescribeAvailabilityZones'); + }); +}); + describe('LambdaMicrovmCompute — memory sizing', () => { // TEST-CONVENTION EXEMPTION (cdk/AGENTS.md "synth once in beforeAll"): the // rejection cases assert the CONSTRUCTOR throws, so there is no template to diff --git a/cdk/test/constructs/task-orchestrator.test.ts b/cdk/test/constructs/task-orchestrator.test.ts index 418617928..9734ef017 100644 --- a/cdk/test/constructs/task-orchestrator.test.ts +++ b/cdk/test/constructs/task-orchestrator.test.ts @@ -33,6 +33,16 @@ interface StackOverrides { memoryId?: string; guardrailId?: string; guardrailVersion?: string; + /** ADR-021 P2: the identifiers the orchestrator forwards as `platform_config`. */ + agentPlatformConfig?: { + taskApprovalsTableName: string; + nudgesTableName: string; + logGroupName: string; + artifactsBucketName: string; + traceArtifactsBucketName: string; + agentSessionRoleArn: string; + anthropicDefaultHaikuModel: string; + }; ecsConfig?: { clusterArn: string; taskDefinitionArn: string; @@ -723,13 +733,23 @@ describe('TaskOrchestrator with the Lambda MicroVMs backend (ADR-021)', () => { expect(pass.Resource).toBe('*'); }); - test('passes the execution role to lambda.amazonaws.com only', () => { + test('passes ONLY the exact execution-role ARN, with NO iam:PassedToService condition', () => { + // ADR-021 P2r2-F10, and the sharpest IAM assertion in this file: the condition + // must NOT come back. A controlled two-arm experiment (same exact-ARN resource, + // same ~5-minute settle, one variable) showed the Lambda MicroVMs service does + // not present a usable `iam:PassedToService` value on the RunMicrovm PassRole + // path — with the condition every submission was DENIED, without it the next + // one reached RUNNING in 9 s. An earlier revision asserted the opposite + // ("exonerated live"); that was a false negative from a contaminated control + // (run 1's temporary unconditioned grant was still attached during its + // "control" arm). See the comment on the statement in task-orchestrator.ts. const passRole = microvmStatements(template).find(s => s.Sid === 'MicrovmPassExecutionRole')!; expect(passRole.Action).toBe('iam:PassRole'); + expect(passRole.Condition).toBeUndefined(); + // With the condition gone, the exact-ARN resource is the WHOLE of the scoping — + // so a widening here (a name prefix, or `*`) would leave the grant unbounded. expect(passRole.Resource).toBe(EXECUTION_ROLE_ARN); - expect(passRole.Condition).toEqual({ - StringEquals: { 'iam:PassedToService': 'lambda.amazonaws.com' }, - }); + expect(JSON.stringify(passRole.Resource)).not.toContain('*'); }); test('grants NO suspend/resume (P3) and NO auth-token minting (never)', () => { @@ -762,3 +782,134 @@ describe('TaskOrchestrator with the Lambda MicroVMs backend (ADR-021)', () => { expect(orchestratorEnv(noMicrovmTemplate).MICROVM_IMAGE_IDENTIFIER).toBeUndefined(); }); }); + +describe('TaskOrchestrator agentPlatformConfig (ADR-021 P2 platform_config transport)', () => { + const SESSION_ROLE_ARN = 'arn:aws:iam::123456789012:role/AbcaAgentSessionRole'; + const HAIKU_PROFILE = 'us.anthropic.claude-haiku-4-5-20251001-v1:0'; + + /** + * Stack with real approvals/nudges tables and buckets, so the "forwards the + * NAME, grants nothing" property can be asserted against actual logical IDs + * rather than string literals. + */ + function createPlatformConfigStack(withConfig: boolean): { template: Template } { + const app = new App(); + const stack = new Stack(app, 'TestStack', { + env: { account: '123456789012', region: 'us-east-1' }, + }); + const mkTable = (id: string) => new dynamodb.Table(stack, id, { + partitionKey: { name: 'task_id', type: dynamodb.AttributeType.STRING }, + }); + const approvalsTable = mkTable('TaskApprovalsTable'); + const nudgesTable = mkTable('TaskNudgesTable'); + const traceBucket = new s3.Bucket(stack, 'TraceArtifactsBucket'); + + new TaskOrchestrator(stack, 'TaskOrchestrator', { + taskTable: mkTable('TaskTable'), + taskEventsTable: mkTable('TaskEventsTable'), + userConcurrencyTable: new dynamodb.Table(stack, 'UserConcurrencyTable', { + partitionKey: { name: 'user_id', type: dynamodb.AttributeType.STRING }, + }), + runtimeArn: 'arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/test-runtime', + ...(withConfig && { + agentPlatformConfig: { + taskApprovalsTableName: approvalsTable.tableName, + nudgesTableName: nudgesTable.tableName, + logGroupName: '/aws/abca/application', + artifactsBucketName: traceBucket.bucketName, + traceArtifactsBucketName: traceBucket.bucketName, + agentSessionRoleArn: SESSION_ROLE_ARN, + anthropicDefaultHaikuModel: HAIKU_PROFILE, + }, + }), + }); + + return { template: Template.fromStack(stack) }; + } + + function orchestratorEnvVars(template: Template): Record { + const [, fn] = Object.entries(template.findResources('AWS::Lambda::Function')) + .find(([id]) => id.includes('OrchestratorFn'))!; + return fn.Properties.Environment.Variables as Record; + } + + let template: Template; + let withoutConfigTemplate: Template; + + beforeAll(() => { + template = createPlatformConfigStack(true).template; + withoutConfigTemplate = createPlatformConfigStack(false).template; + }); + + test('injects the seven forwarded identifiers under the names the strategy reads', () => { + // These names are a CONTRACT with + // `handlers/shared/strategies/lambda-microvm-strategy.ts`'s + // PLATFORM_CONFIG_ENV_VARS map, and with the AgentCore runtime env block in + // `stacks/agent.ts` — one stack value, one name, three backends. Renaming one + // side silently strips a key from every MicroVM task's platform_config. + const env = orchestratorEnvVars(template); + expect(env.TASK_APPROVALS_TABLE_NAME).toEqual({ Ref: expect.stringMatching(/^TaskApprovalsTable/) }); + expect(env.NUDGES_TABLE_NAME).toEqual({ Ref: expect.stringMatching(/^TaskNudgesTable/) }); + expect(env.LOG_GROUP_NAME).toBe('/aws/abca/application'); + expect(env.ARTIFACTS_BUCKET_NAME).toEqual({ Ref: expect.stringMatching(/^TraceArtifactsBucket/) }); + expect(env.TRACE_ARTIFACTS_BUCKET_NAME).toEqual({ Ref: expect.stringMatching(/^TraceArtifactsBucket/) }); + expect(env.AGENT_SESSION_ROLE_ARN).toBe(SESSION_ROLE_ARN); + expect(env.ANTHROPIC_DEFAULT_HAIKU_MODEL).toBe(HAIKU_PROFILE); + }); + + test('carries the four REQUIRED platform_config sources together (never a partial set)', () => { + // The strategy refuses to start a lambda-microvm session without these four. + // Three come from the orchestrator's own wiring and one from this block, so + // this is the assertion that they are all reachable from ONE deploy. + const env = orchestratorEnvVars(createStack({ + githubTokenSecretArn: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:github-token-abc123', + agentPlatformConfig: { + taskApprovalsTableName: 'approvals', + nudgesTableName: 'nudges', + logGroupName: '/aws/abca/application', + artifactsBucketName: 'artifacts', + traceArtifactsBucketName: 'traces', + agentSessionRoleArn: SESSION_ROLE_ARN, + anthropicDefaultHaikuModel: HAIKU_PROFILE, + }, + }).template); + expect(env.TASK_TABLE_NAME).toBeDefined(); + expect(env.TASK_EVENTS_TABLE_NAME).toBeDefined(); + expect(env.GITHUB_TOKEN_SECRET_ARN).toBeDefined(); + expect(env.AGENT_SESSION_ROLE_ARN).toBe(SESSION_ROLE_ARN); + }); + + test('omits every one of them when the prop is absent (isolated-construct posture)', () => { + const env = orchestratorEnvVars(withoutConfigTemplate); + for (const key of [ + 'TASK_APPROVALS_TABLE_NAME', + 'NUDGES_TABLE_NAME', + 'LOG_GROUP_NAME', + 'ARTIFACTS_BUCKET_NAME', + 'TRACE_ARTIFACTS_BUCKET_NAME', + 'AGENT_SESSION_ROLE_ARN', + 'ANTHROPIC_DEFAULT_HAIKU_MODEL', + ]) { + expect(env[key]).toBeUndefined(); + } + }); + + test('grants NOTHING for the forwarded resources — names only, no new reach', () => { + // The load-bearing property of this block. The orchestrator transports these + // identifiers to the agent and never calls the resources itself, so a + // "while I'm here" grant would hand the orchestration plane approvals/nudges + // tenant-data access it has never needed. The agent reaches them through its + // own execution role / the task-scoped SessionRole. + const orchestratorPolicies = JSON.stringify( + Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('TaskOrchestrator')), + ); + expect(orchestratorPolicies).not.toContain('TaskApprovalsTable'); + expect(orchestratorPolicies).not.toContain('TaskNudgesTable'); + expect(orchestratorPolicies).not.toContain('TraceArtifactsBucket'); + // ...and no sts:AssumeRole on the SessionRole either: forwarding the ARN is + // not assuming the role. + expect(orchestratorPolicies).not.toContain(SESSION_ROLE_ARN); + expect(orchestratorPolicies).not.toContain('sts:AssumeRole'); + }); +}); diff --git a/cdk/test/handlers/get-task.test.ts b/cdk/test/handlers/get-task.test.ts index 1cc2bd4c0..7e4b2f2cc 100644 --- a/cdk/test/handlers/get-task.test.ts +++ b/cdk/test/handlers/get-task.test.ts @@ -119,6 +119,32 @@ describe('get-task handler', () => { expect(body.data.channel_source).toBe('api'); }); + test('surfaces agent_heartbeat_at so the in-guest liveness signal is observable', async () => { + // ADR-021 P2r2-F11. The agent writes this every 45 s on every backend and the + // orchestrator reads it for hang detection, but `toTaskDetail` dropped it — so + // `bgagent status` showed `None` while DynamoDB held a 6-second-old value, and + // a live verification run concluded "heartbeats not observed" and blamed an + // unrelated defect. Exactly the class of bug the channel_source assertion above + // exists for. + mockSend.mockReset(); + mockSend.mockResolvedValueOnce({ + Item: { ...TASK_RECORD, agent_heartbeat_at: '2025-03-15T10:30:45Z' }, + }); + + const result = await handler(makeEvent()); + + expect(result.statusCode).toBe(200); + expect(JSON.parse(result.body).data.agent_heartbeat_at).toBe('2025-03-15T10:30:45Z'); + }); + + test('reports agent_heartbeat_at as null when the agent has not beaten yet', async () => { + // Present-but-null, not absent: scripts and the CLI branch on the field, so it + // must exist on every response (the TaskDetail contract is non-optional). + const body = JSON.parse((await handler(makeEvent())).body); + expect(body.data).toHaveProperty('agent_heartbeat_at'); + expect(body.data.agent_heartbeat_at).toBeNull(); + }); + test('surfaces channel_source=webhook for tasks created via the webhook path', async () => { mockSend.mockReset(); mockSend.mockResolvedValueOnce({ diff --git a/cdk/test/handlers/orchestrate-task-microvm.test.ts b/cdk/test/handlers/orchestrate-task-microvm.test.ts index 75c5fc881..1004e3600 100644 --- a/cdk/test/handlers/orchestrate-task-microvm.test.ts +++ b/cdk/test/handlers/orchestrate-task-microvm.test.ts @@ -110,6 +110,15 @@ process.env.TASK_EVENTS_TABLE_NAME = 'TaskEvents'; process.env.USER_CONCURRENCY_TABLE_NAME = 'UserConcurrency'; process.env.TASK_RETENTION_DAYS = '90'; +// platform_config (ADR-021 P2): the four REQUIRED identifiers the MicroVM +// strategy refuses to start a session without — they are the agent's only +// channel for them, since a snapshot must not bake configuration in. Read at +// call time by `buildMicrovmPlatformConfig`, but set here alongside the rest +// for clarity. +process.env.GITHUB_TOKEN_SECRET_ARN = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:abca/github-token-AbCdEf'; +process.env.AGENT_SESSION_ROLE_ARN = 'arn:aws:iam::123456789012:role/AbcaAgentSessionRole'; + import { TaskStatus } from '../../src/constructs/task-status'; import { handler } from '../../src/handlers/orchestrate-task'; import { LambdaMicrovmComputeStrategy } from '../../src/handlers/shared/strategies/lambda-microvm-strategy'; diff --git a/cdk/test/handlers/orchestrate-task.test.ts b/cdk/test/handlers/orchestrate-task.test.ts index 42247705c..8ff0a3ff6 100644 --- a/cdk/test/handlers/orchestrate-task.test.ts +++ b/cdk/test/handlers/orchestrate-task.test.ts @@ -553,6 +553,14 @@ describe('pollTaskStatus', () => { expect(result.lastStatus).toBeUndefined(); }); + test('rejects an unknown compute type instead of silently skipping heartbeat liveness', async () => { + mockDdbSend.mockResolvedValueOnce({ Item: { status: 'RUNNING' } }); + + await expect(pollTaskStatus('TASK001', { attempts: 0 }, 'unknown' as never)).rejects.toThrow( + 'Unknown compute type for heartbeat liveness: unknown', + ); + }); + test('sets sessionUnhealthy when agent heartbeat is stale (RUNNING)', async () => { const old = new Date(Date.now() - 400_000).toISOString(); mockDdbSend.mockResolvedValueOnce({ @@ -645,6 +653,119 @@ describe('pollTaskStatus', () => { const result = await pollTaskStatus('TASK001', { attempts: 1 }, 'agentcore'); expect(result.sessionUnhealthy).toBe(true); }); + + // --- ADR-021 P2: heartbeat liveness on lambda-microvm --- + // + // The gap this closes: the agent writes `agent_heartbeat_at` on every substrate, + // but the orchestrator only ACTED on it for agentcore. On a MicroVM the + // substrate `GetMicrovm` cross-check catches a VM that DIED and nothing else — + // an alive, healthy VM whose in-guest pipeline is hung or was OOM-killed inside + // the guest looks perfectly fine (nothing self-terminates on this substrate, and + // it stays RUNNING with no stateReason). That task would burn the full ~8.5 h + // poll window while billing an 8-hour reservation. So liveness here is substrate + // state AND agent heartbeat. + + test('sets sessionUnhealthy for lambda-microvm when the heartbeat is stale', async () => { + const old = new Date(Date.now() - 400_000).toISOString(); + mockDdbSend.mockResolvedValueOnce({ + Item: { + status: 'RUNNING', + session_id: 'mvm-0123456789abcdef', + started_at: old, + agent_heartbeat_at: old, + }, + }); + const result = await pollTaskStatus('TASK001', { attempts: 1 }, 'lambda-microvm'); + expect(result.sessionUnhealthy).toBe(true); + }); + + test('sets sessionUnhealthy for lambda-microvm when the agent never heartbeat past the window', async () => { + // The in-guest early-crash case: the /run hook returned 200, the pipeline died + // before its first heartbeat, and the MicroVM is still happily RUNNING. + mockDdbSend.mockResolvedValueOnce({ + Item: { + status: 'RUNNING', + session_id: 'mvm-0123456789abcdef', + started_at: new Date(Date.now() - 400_000).toISOString(), + }, + }); + const result = await pollTaskStatus('TASK001', { attempts: 1 }, 'lambda-microvm'); + expect(result.sessionUnhealthy).toBe(true); + }); + + test('lambda-microvm uses the SAME thresholds as agentcore — a fresh heartbeat is healthy', async () => { + // Shared thresholds on purpose: the timestamp is written by the same pipeline + // code at the same cadence regardless of substrate, so a backend-specific grace + // window would encode a difference that does not exist. + const started = new Date(Date.now() - 200_000).toISOString(); + const heartbeat = new Date(Date.now() - 30_000).toISOString(); + const item = { + status: 'RUNNING', + session_id: 'mvm-0123456789abcdef', + started_at: started, + agent_heartbeat_at: heartbeat, + }; + + mockDdbSend.mockResolvedValueOnce({ Item: item }); + const microvm = await pollTaskStatus('TASK001', { attempts: 1 }, 'lambda-microvm'); + mockDdbSend.mockResolvedValueOnce({ Item: item }); + const agentcore = await pollTaskStatus('TASK001', { attempts: 1 }, 'agentcore'); + + expect(microvm.sessionUnhealthy).toBe(false); + expect(microvm.sessionUnhealthy).toBe(agentcore.sessionUnhealthy); + }); + + test('lambda-microvm within the grace period is healthy, exactly as agentcore is', async () => { + const item = { + status: 'RUNNING', + session_id: 'mvm-0123456789abcdef', + started_at: new Date(Date.now() - 60_000).toISOString(), + }; + mockDdbSend.mockResolvedValueOnce({ Item: item }); + const microvm = await pollTaskStatus('TASK001', { attempts: 1 }, 'lambda-microvm'); + mockDdbSend.mockResolvedValueOnce({ Item: item }); + const agentcore = await pollTaskStatus('TASK001', { attempts: 1 }, 'agentcore'); + + expect(microvm.sessionUnhealthy).toBe(false); + expect(agentcore.sessionUnhealthy).toBe(false); + }); + + test('a stale heartbeat on a NON-RUNNING lambda-microvm task is not unhealthy', async () => { + // AWAITING_APPROVAL is the case that matters: from P3 the orchestrator SUSPENDS + // the MicroVM during an approval wait, which stops the in-guest pipeline (and + // its heartbeats) by design. Failing that task would be the worst possible + // regression — the check is scoped to RUNNING for exactly this reason. + const old = new Date(Date.now() - 400_000).toISOString(); + mockDdbSend.mockResolvedValueOnce({ + Item: { + status: 'AWAITING_APPROVAL', + session_id: 'mvm-0123456789abcdef', + started_at: old, + agent_heartbeat_at: old, + }, + }); + const result = await pollTaskStatus('TASK001', { attempts: 1 }, 'lambda-microvm'); + expect(result.sessionUnhealthy).toBe(false); + expect(result.lastStatus).toBe('AWAITING_APPROVAL'); + }); + + test('ECS remains untouched by the widening', async () => { + // The one backend deliberately left out: DescribeTasks reports a real container + // exit (including OOM-kill / exit 137) with an exit code, and the ECS poll block + // interprets it with its own patience counters. Layering the heartbeat on top + // would give one backend two independently-tuned kill paths for one failure. + const old = new Date(Date.now() - 400_000).toISOString(); + mockDdbSend.mockResolvedValueOnce({ + Item: { + status: 'RUNNING', + session_id: 'arn:aws:ecs:us-east-1:123456789012:task/agent/abc', + started_at: old, + agent_heartbeat_at: old, + }, + }); + const result = await pollTaskStatus('TASK001', { attempts: 1 }, 'ecs'); + expect(result.sessionUnhealthy).toBe(false); + }); }); describe('loadBlueprintConfig', () => { diff --git a/cdk/test/handlers/shared/strategies/lambda-microvm-strategy.test.ts b/cdk/test/handlers/shared/strategies/lambda-microvm-strategy.test.ts index bf9ad7792..4ae0bd4bc 100644 --- a/cdk/test/handlers/shared/strategies/lambda-microvm-strategy.test.ts +++ b/cdk/test/handlers/shared/strategies/lambda-microvm-strategy.test.ts @@ -31,6 +31,18 @@ const PAYLOAD_BUCKET = 'test-microvm-payload-bucket'; const MICROVM_ID = 'mvm-0123456789abcdef'; const ENDPOINT = 'https://mvm-0123456789abcdef.microvm.lambda.us-east-1.amazonaws.com'; +// --- platform_config (ADR-021 P2) --- +// The FOUR required identifiers, and only those, are set for the main describes, +// so the default `platform_config` block is small and its exact serialized size is +// known — which the 4 KB boundary probes below depend on. The nine optional keys +// get their own describe (and are deleted here so a leaked env var from another +// suite cannot silently change the envelope's byte length). +const TASK_TABLE_NAME = 'abca-task-table'; +const TASK_EVENTS_TABLE_NAME = 'abca-task-events-table'; +const GITHUB_TOKEN_SECRET_ARN = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:abca/github-token-AbCdEf'; +const AGENT_SESSION_ROLE_ARN = 'arn:aws:iam::123456789012:role/AbcaAgentSessionRole'; + // Set env vars BEFORE import — LambdaMicrovmComputeStrategy reads them as // module-level constants (same pattern as ecs-strategy). The top-of-file import // is the FULLY-CONFIGURED substrate; the missing-config describe block below @@ -46,6 +58,24 @@ process.env.MICROVM_PAYLOAD_BUCKET = PAYLOAD_BUCKET; process.env.AWS_REGION = 'us-east-1'; delete process.env.MICROVM_INGRESS_CONNECTOR_ARNS; +process.env.TASK_TABLE_NAME = TASK_TABLE_NAME; +process.env.TASK_EVENTS_TABLE_NAME = TASK_EVENTS_TABLE_NAME; +process.env.GITHUB_TOKEN_SECRET_ARN = GITHUB_TOKEN_SECRET_ARN; +process.env.AGENT_SESSION_ROLE_ARN = AGENT_SESSION_ROLE_ARN; +for (const optional of [ + 'TASK_APPROVALS_TABLE_NAME', + 'NUDGES_TABLE_NAME', + 'LOG_GROUP_NAME', + 'ARTIFACTS_BUCKET_NAME', + 'TRACE_ARTIFACTS_BUCKET_NAME', + 'LINEAR_OAUTH_SECRET_ARN', + 'JIRA_OAUTH_SECRET_ARN', + 'AWS_SDK_UA_APP_ID', + 'ANTHROPIC_DEFAULT_HAIKU_MODEL', +]) { + delete process.env[optional]; +} + const mockSend = jest.fn(); jest.mock('@aws-sdk/client-lambda-microvms', () => ({ LambdaMicrovmsClient: jest.fn(() => ({ send: mockSend })), @@ -76,12 +106,16 @@ jest.mock('@aws-sdk/client-s3', () => ({ const mockLogger = { info: jest.fn(), warn: jest.fn(), error: jest.fn(), child: jest.fn() }; jest.mock('../../../../src/handlers/shared/logger', () => ({ logger: mockLogger })); +import sharedConstants from '../../../../../contracts/constants.json'; import type { BlueprintConfig } from '../../../../src/handlers/shared/repo-config'; import { LambdaMicrovmComputeStrategy, MICROVM_ERROR_MARKER, MICROVM_MAX_DURATION_SECONDS, + MICROVM_PLATFORM_CONFIG_KEYS, + MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS, MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES, + buildMicrovmPlatformConfig, microvmNoIngressConnectorArnForRegion, microvmPayloadKey, } from '../../../../src/handlers/shared/strategies/lambda-microvm-strategy'; @@ -89,15 +123,38 @@ import { const BLUEPRINT: BlueprintConfig = { compute_type: 'lambda-microvm', runtime_arn: '' }; /** - * Build a payload whose serialized `{"agent_payload": …}` envelope is EXACTLY - * `targetBytes` long, so the 4 KB boundary can be probed on both sides. Asserts - * its own arithmetic — if the envelope shape ever changes, this fails loudly - * rather than silently testing the wrong boundary. + * The `platform_config` block every startSession in this file's main describes + * must produce, given the module-level environment above. + */ +const EXPECTED_PLATFORM_CONFIG = { + task_table_name: TASK_TABLE_NAME, + task_events_table_name: TASK_EVENTS_TABLE_NAME, + github_token_secret_arn: GITHUB_TOKEN_SECRET_ARN, + agent_session_role_arn: AGENT_SESSION_ROLE_ARN, +}; + +/** + * Build a payload whose serialized `{"agent_payload": …, "platform_config": …}` + * envelope is EXACTLY `targetBytes` long, so the 4 KB boundary can be probed on + * both sides. + * + * `platform_config` is part of the counted envelope (ADR-021 P2), so its bytes are + * subtracted from the payload's budget here. Asserts its own arithmetic — if the + * envelope shape or the platform block ever changes, this fails loudly rather than + * silently testing the wrong boundary. */ function payloadWithEnvelopeBytes(targetBytes: number): Record { - const overhead = Buffer.byteLength(JSON.stringify({ agent_payload: { p: '' } }), 'utf8'); + const overhead = Buffer.byteLength( + JSON.stringify({ agent_payload: { p: '' }, platform_config: EXPECTED_PLATFORM_CONFIG }), + 'utf8', + ); const payload = { p: 'x'.repeat(targetBytes - overhead) }; - expect(Buffer.byteLength(JSON.stringify({ agent_payload: payload }), 'utf8')).toBe(targetBytes); + expect( + Buffer.byteLength( + JSON.stringify({ agent_payload: payload, platform_config: EXPECTED_PLATFORM_CONFIG }), + 'utf8', + ), + ).toBe(targetBytes); return payload; } @@ -167,6 +224,41 @@ async function withRegionAsync( } } +/** + * Run `body` with the named environment variables DELETED, restoring them after. + * + * `buildMicrovmPlatformConfig` reads the environment at CALL time (unlike the + * `MICROVM_*` substrate constants, which are frozen at import), so a missing + * platform identifier needs no `jest.isolateModules` module reload — which is + * exactly why it is written that way. + */ +async function withoutEnvAsync(keys: string[], body: () => Promise): Promise { + const saved = Object.fromEntries(keys.map(key => [key, process.env[key]])); + try { + for (const key of keys) delete process.env[key]; + await body(); + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +} + +/** {@link withoutEnvAsync}'s inverse: run `body` with extra env vars SET. */ +async function withEnvAsync(env: Record, body: () => Promise): Promise { + const saved = Object.fromEntries(Object.keys(env).map(key => [key, process.env[key]])); + try { + Object.assign(process.env, env); + await body(); + } finally { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + } +} + beforeEach(() => { jest.clearAllMocks(); }); @@ -330,6 +422,10 @@ describe('LambdaMicrovmComputeStrategy', () => { const envelope = JSON.parse(mockSend.mock.calls[0][0].input.runHookPayload); expect(envelope.agent_payload).toEqual({ repo_url: 'org/repo', prompt: 'Fix the bug', max_turns: 50 }); expect(envelope.agent_payload_s3_uri).toBeUndefined(); + // The MicroVM's substitute for the env block the other two backends get at + // deploy time — a snapshot must not bake it in (ADR-021 sub-decision 3), so + // it rides the envelope alongside the payload. + expect(envelope.platform_config).toEqual(EXPECTED_PLATFORM_CONFIG); }); test('uploads an oversized payload to S3 and inlines only the pointer', async () => { @@ -351,12 +447,18 @@ describe('LambdaMicrovmComputeStrategy', () => { expect(put.input.Bucket).toBe(PAYLOAD_BUCKET); expect(put.input.Key).toBe('TASK001/payload.json'); expect(put.input.ContentType).toBe('application/json'); - expect(JSON.parse(put.input.Body)).toEqual(big); + // The S3 object carries the payload with platform_config merged in at the top + // level, so an agent that resolves the pointer gets the config with it. + expect(JSON.parse(put.input.Body)).toEqual({ ...big, platform_config: EXPECTED_PLATFORM_CONFIG }); const runHookPayload = mockSend.mock.calls[0][0].input.runHookPayload; const envelope = JSON.parse(runHookPayload); expect(envelope.agent_payload_s3_uri).toBe(`s3://${PAYLOAD_BUCKET}/TASK001/payload.json`); expect(envelope.agent_payload).toBeUndefined(); + // ...and ALSO inline on the pointer envelope, deliberately duplicated: the + // agent must be able to read its platform configuration whether it takes it + // off the hook body before fetching S3 or out of the fetched object. + expect(envelope.platform_config).toEqual(EXPECTED_PLATFORM_CONFIG); // The whole point: the hook body must sit far under the 4 KB cap. expect(Buffer.byteLength(runHookPayload, 'utf8')).toBeLessThan(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES); }); @@ -429,7 +531,7 @@ describe('LambdaMicrovmComputeStrategy', () => { // 3-byte UTF-8 characters: 2000 chars is ~6 KB of bytes but only 2 KB of // chars, so measuring String.length would have wrongly inlined this. const payload = { prompt: '\u4f60'.repeat(2_000) }; - expect(JSON.stringify({ agent_payload: payload }).length) + expect(JSON.stringify({ agent_payload: payload, platform_config: EXPECTED_PLATFORM_CONFIG }).length) .toBeLessThan(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES); await new LambdaMicrovmComputeStrategy().startSession({ taskId: 'TASK001', @@ -538,6 +640,96 @@ describe('LambdaMicrovmComputeStrategy', () => { expect(mockSend.mock.calls.filter(c => c[0]._type === 'TerminateMicrovm')).toHaveLength(0); }); + test('platform_config COUNTS toward the 4 KB boundary — an otherwise-inlineable payload goes to S3', async () => { + // The regression this locks: measuring `{agent_payload}` alone and then + // sending `{agent_payload, platform_config}` would inline an envelope the + // service rejects outright. So the branch decision has to be made on the + // FULL envelope. This payload is exactly 4 096 bytes WITHOUT the platform + // block — i.e. the old code would have inlined it — and must now upload. + mockS3Send.mockResolvedValueOnce({}); + runMicrovmOk(); + + const overheadWithoutConfig = Buffer.byteLength(JSON.stringify({ agent_payload: { p: '' } }), 'utf8'); + const payload = { p: 'x'.repeat(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES - overheadWithoutConfig) }; + expect(Buffer.byteLength(JSON.stringify({ agent_payload: payload }), 'utf8')) + .toBe(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES); + + await new LambdaMicrovmComputeStrategy().startSession({ + taskId: 'TASK001', + userId: 'cognito-test', + payload, + blueprintConfig: BLUEPRINT, + }); + + expect(mockS3Send).toHaveBeenCalledTimes(1); + const runHookPayload = mockSend.mock.calls[0][0].input.runHookPayload; + expect(JSON.parse(runHookPayload).agent_payload_s3_uri).toBeDefined(); + expect(Buffer.byteLength(runHookPayload, 'utf8')) + .toBeLessThanOrEqual(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES); + }); + + test.each(MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS.map(key => [key]))( + 'refuses to start — before any AWS call — when required platform config %s is missing', + async (key) => { + const envVar = sharedConstants.microvm_platform_config.env_by_key[key]; + + await withoutEnvAsync([envVar], async () => { + const start = new LambdaMicrovmComputeStrategy().startSession({ + taskId: 'TASK001', + userId: 'cognito-test', + // Oversized on purpose: the guard must fire before the payload upload, + // or a misconfiguration leaves orphan objects in the payload bucket. + payload: payloadWithEnvelopeBytes(20_000), + blueprintConfig: BLUEPRINT, + }); + + await expect(start).rejects.toThrow(new RegExp(`${key} <- ${envVar}`)); + await expect(start).rejects.toThrow(/redeploy the stack/); + expect(mockSend).not.toHaveBeenCalled(); + expect(mockS3Send).not.toHaveBeenCalled(); + }); + }, + ); + + test('logs which platform_config KEYS a session received, and never their values', async () => { + runMicrovmOk(); + + await new LambdaMicrovmComputeStrategy().startSession({ + taskId: 'TASK001', + userId: 'cognito-test', + payload: { repo_url: 'org/repo' }, + blueprintConfig: BLUEPRINT, + }); + + const started = mockLogger.info.mock.calls + .find(([message]) => message === 'Lambda MicroVM session started')!; + expect(started[1].platform_config_keys).toEqual(Object.keys(EXPECTED_PLATFORM_CONFIG)); + // Key names are the diagnostic; values are not, and one of them is a secret + // ARN. Nothing resembling a value may appear on the log line. + expect(JSON.stringify(started[1])).not.toContain(GITHUB_TOKEN_SECRET_ARN); + }); + + test('fails BEFORE the upload when even the POINTER envelope cannot fit (no orphan object)', async () => { + // The one shape with no smaller fallback: the payload has already been moved + // to S3, so if `{pointer + platform_config}` still exceeds 4 096 bytes there + // is nothing left to shed. Only a pathological identifier length can cause it + // — hence the check, and hence its placement BEFORE the PutObject so a + // misconfiguration cannot leave objects behind for the lifecycle rule to reap. + await withEnvAsync({ LOG_GROUP_NAME: `/aws/${'x'.repeat(5_000)}` }, async () => { + const start = new LambdaMicrovmComputeStrategy().startSession({ + taskId: 'TASK001', + userId: 'cognito-test', + payload: payloadWithEnvelopeBytes(20_000), + blueprintConfig: BLUEPRINT, + }); + + await expect(start).rejects.toThrow(/pointer envelope is \d+ bytes/); + await expect(start).rejects.toThrow(/shorten the stack name/); + expect(mockS3Send).not.toHaveBeenCalled(); + expect(mockSend).not.toHaveBeenCalled(); + }); + }); + test('microvmPayloadKey matches the ECS payload key shape', () => { expect(microvmPayloadKey('TASK001')).toBe('TASK001/payload.json'); }); @@ -912,3 +1104,226 @@ describe('LambdaMicrovmComputeStrategy image-identifier validation', () => { expect(mockSend.mock.calls[0][0].input.imageIdentifier).toBe(IMAGE_IDENTIFIER); }); }); + +describe('buildMicrovmPlatformConfig — the MicroVM substitute for a deploy-time env block', () => { + /** A fully-populated orchestrator environment: all thirteen keys present. */ + const FULL_ENV: NodeJS.ProcessEnv = { + TASK_TABLE_NAME: 'tasks', + TASK_EVENTS_TABLE_NAME: 'events', + TASK_APPROVALS_TABLE_NAME: 'approvals', + NUDGES_TABLE_NAME: 'nudges', + LOG_GROUP_NAME: '/aws/abca/application', + ARTIFACTS_BUCKET_NAME: 'artifacts-bucket', + TRACE_ARTIFACTS_BUCKET_NAME: 'trace-bucket', + GITHUB_TOKEN_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:gh-AbCdEf', + LINEAR_OAUTH_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:bgagent-linear-oauth-acme-XyZ', + JIRA_OAUTH_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:bgagent-jira-oauth-cloud1-XyZ', + AGENT_SESSION_ROLE_ARN: 'arn:aws:iam::123456789012:role/SessionRole', + AWS_SDK_UA_APP_ID: 'uksb-wt64nei4u6#backgroundagent-dev', + ANTHROPIC_DEFAULT_HAIKU_MODEL: 'us.anthropic.claude-haiku-4-5-20251001-v1:0', + }; + + test('sources the wire key allow-list from the cross-language contract', () => { + // The pin is MECHANICAL, not this test: both this producer and + // `agent/src/server.py`'s consumer read + // `contracts/constants.json → microvm_platform_config`, and + // `scripts/check-constants-sync.ts` validates its shape and rejects a + // Python-side literal re-declaration. What this asserts is that the CDK side + // really is reading it (a local copy would pass every other test in this file) + // and, below, that its contents are what review approved — a contract edit is + // a wire-format change on both sides, so it must not slip through unnoticed. + expect([...MICROVM_PLATFORM_CONFIG_KEYS]) + .toEqual(Object.keys(sharedConstants.microvm_platform_config.env_by_key)); + expect([...MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS]) + .toEqual(sharedConstants.microvm_platform_config.required); + + // Order is part of the contract: it is the serialization order, which the 4 KB + // inline/S3 branch decision is computed against. + expect([...MICROVM_PLATFORM_CONFIG_KEYS]).toEqual([ + 'task_table_name', + 'task_events_table_name', + 'task_approvals_table_name', + 'nudges_table_name', + 'log_group_name', + 'artifacts_bucket_name', + 'trace_artifacts_bucket_name', + 'github_token_secret_arn', + 'linear_oauth_secret_arn', + 'jira_oauth_secret_arn', + 'agent_session_role_arn', + 'aws_sdk_ua_app_id', + 'anthropic_default_haiku_model', + ]); + expect(MICROVM_PLATFORM_CONFIG_KEYS).toHaveLength(13); + // snake_case on the wire, matching every other key in the /run envelope. + for (const key of MICROVM_PLATFORM_CONFIG_KEYS) { + expect(key).toMatch(/^[a-z][a-z0-9_]*$/); + } + }); + + test('pins the REQUIRED subset — these four are what a task cannot start without', () => { + expect([...MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS]).toEqual([ + 'task_table_name', + 'task_events_table_name', + 'github_token_secret_arn', + 'agent_session_role_arn', + ]); + // Every required key must be a real wire key, or the guard would demand + // something the producer never emits. + for (const key of MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS) { + expect(MICROVM_PLATFORM_CONFIG_KEYS).toContain(key); + } + }); + + test('emits all thirteen keys, in declaration order, from a full environment', () => { + const config = buildMicrovmPlatformConfig(FULL_ENV); + expect(Object.keys(config)).toEqual([...MICROVM_PLATFORM_CONFIG_KEYS]); + expect(config.task_table_name).toBe('tasks'); + expect(config.nudges_table_name).toBe('nudges'); + expect(config.agent_session_role_arn).toBe('arn:aws:iam::123456789012:role/SessionRole'); + expect(config.anthropic_default_haiku_model).toBe('us.anthropic.claude-haiku-4-5-20251001-v1:0'); + }); + + test('OMITS optional keys the orchestrator does not carry (no `undefined` placeholders)', () => { + const config = buildMicrovmPlatformConfig({ + TASK_TABLE_NAME: 'tasks', + TASK_EVENTS_TABLE_NAME: 'events', + GITHUB_TOKEN_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:gh-AbCdEf', + AGENT_SESSION_ROLE_ARN: 'arn:aws:iam::123456789012:role/SessionRole', + }); + + // Omitted, not present-and-undefined: the agent's `key in platform_config` + // checks must mean what they say, and a `"k":null` costs bytes against 4 KB. + expect(Object.keys(config)).toEqual([ + 'task_table_name', + 'task_events_table_name', + 'github_token_secret_arn', + 'agent_session_role_arn', + ]); + expect('nudges_table_name' in config).toBe(false); + expect(JSON.stringify(config)).not.toContain('null'); + }); + + test('treats an EMPTY value as absent', () => { + // CloudFormation renders an unresolved optional value as ''. A nameless table + // is worse than a missing one — the agent would build a request against it. + const config = buildMicrovmPlatformConfig({ ...FULL_ENV, NUDGES_TABLE_NAME: '' }); + expect('nudges_table_name' in config).toBe(false); + }); + + test('is a closed map: an environment variable outside the allow-list can never leak', () => { + const config = buildMicrovmPlatformConfig({ + ...FULL_ENV, + // The reason this matters: the envelope is written to an S3 object and echoed + // into MicroVM logs on a hook failure. Secret VALUES must never be reachable + // from this producer, only the ARNs that name them. + GITHUB_TOKEN: 'ghp_averysecrettokenvalue', + ANTHROPIC_API_KEY: 'dummy-anthropic-credential-do-not-log', + AWS_SECRET_ACCESS_KEY: 'wJalrXUtnFEMI', + MICROVM_PAYLOAD_BUCKET: 'some-bucket', + }); + + expect(Object.keys(config)).toEqual([...MICROVM_PLATFORM_CONFIG_KEYS]); + const rendered = JSON.stringify(config); + expect(rendered).not.toContain('ghp_'); + expect(rendered).not.toContain('dummy-anthropic-credential-do-not-log'); + expect(rendered).not.toContain('wJalrXUtnFEMI'); + }); + + test.each([ + ['TASK_TABLE_NAME', 'task_table_name'], + ['TASK_EVENTS_TABLE_NAME', 'task_events_table_name'], + ['GITHUB_TOKEN_SECRET_ARN', 'github_token_secret_arn'], + ['AGENT_SESSION_ROLE_ARN', 'agent_session_role_arn'], + ])('throws naming %s when it is missing', (envVar, wireKey) => { + const env = { ...FULL_ENV }; + delete env[envVar]; + + // The message must carry the wire key, its env var, and the remedy — an + // operator reading a failed task should not have to open this file. + expect(() => buildMicrovmPlatformConfig(env)).toThrow(new RegExp(`${wireKey} <- ${envVar}`)); + expect(() => buildMicrovmPlatformConfig(env)).toThrow(/redeploy the stack/); + expect(() => buildMicrovmPlatformConfig(env)).toThrow(/ADR-021 sub-decision 3/); + }); + + test('names EVERY missing required key at once, not just the first', () => { + // One redeploy should fix all of them; reporting one per attempt turns a + // misconfiguration into four round-trips. + expect(() => buildMicrovmPlatformConfig({})).toThrow( + /task_table_name.*task_events_table_name.*github_token_secret_arn.*agent_session_role_arn/s, + ); + }); + + test('does NOT throw for a missing OPTIONAL key', () => { + const env = { ...FULL_ENV }; + for (const optional of [ + 'TASK_APPROVALS_TABLE_NAME', 'NUDGES_TABLE_NAME', 'LOG_GROUP_NAME', + 'ARTIFACTS_BUCKET_NAME', 'TRACE_ARTIFACTS_BUCKET_NAME', 'LINEAR_OAUTH_SECRET_ARN', + 'JIRA_OAUTH_SECRET_ARN', 'AWS_SDK_UA_APP_ID', 'ANTHROPIC_DEFAULT_HAIKU_MODEL', + ]) { + delete env[optional]; + } + expect(() => buildMicrovmPlatformConfig(env)).not.toThrow(); + expect(Object.keys(buildMicrovmPlatformConfig(env))).toEqual([ + ...MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS, + ]); + }); + + test('defaults to process.env when no environment is passed', () => { + // The production call site passes nothing; this is the path that actually runs. + const config = buildMicrovmPlatformConfig(); + expect(config).toEqual(EXPECTED_PLATFORM_CONFIG); + }); + + test('the full thirteen-key block fits the pointer envelope inside the 4 KB cap', () => { + // The one shape with no smaller fallback: if the pointer envelope itself + // exceeded 4 096 bytes there would be nothing left to move to S3. This asserts + // the design has real headroom rather than relying on the guard. + const pointerEnvelope = JSON.stringify({ + agent_payload_s3_uri: `s3://${PAYLOAD_BUCKET}/TASK001/payload.json`, + platform_config: buildMicrovmPlatformConfig(FULL_ENV), + }); + expect(Buffer.byteLength(pointerEnvelope, 'utf8')) + .toBeLessThan(MICROVM_RUN_HOOK_PAYLOAD_LIMIT_BYTES / 2); + }); +}); + +describe('LambdaMicrovmComputeStrategy with the FULL platform_config environment', () => { + const OPTIONAL_ENV: Record = { + TASK_APPROVALS_TABLE_NAME: 'approvals', + NUDGES_TABLE_NAME: 'nudges', + LOG_GROUP_NAME: '/aws/abca/application', + ARTIFACTS_BUCKET_NAME: 'artifacts-bucket', + TRACE_ARTIFACTS_BUCKET_NAME: 'trace-bucket', + LINEAR_OAUTH_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:bgagent-linear-oauth-acme-XyZ', + JIRA_OAUTH_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:bgagent-jira-oauth-cloud1-XyZ', + AWS_SDK_UA_APP_ID: 'uksb-wt64nei4u6#backgroundagent-dev', + ANTHROPIC_DEFAULT_HAIKU_MODEL: 'us.anthropic.claude-haiku-4-5-20251001-v1:0', + }; + + beforeEach(() => { + Object.assign(process.env, OPTIONAL_ENV); + }); + + afterEach(() => { + for (const key of Object.keys(OPTIONAL_ENV)) delete process.env[key]; + }); + + test('delivers every configured identifier on the wire, in both envelope halves', async () => { + mockS3Send.mockResolvedValueOnce({}); + runMicrovmOk(); + + await new LambdaMicrovmComputeStrategy().startSession({ + taskId: 'TASK001', + userId: 'cognito-test', + payload: { repo_url: 'org/repo', hydrated_context: { blob: 'x'.repeat(10_000) } }, + blueprintConfig: BLUEPRINT, + }); + + const expected = { ...EXPECTED_PLATFORM_CONFIG, ...buildMicrovmPlatformConfig() }; + const envelope = JSON.parse(mockSend.mock.calls[0][0].input.runHookPayload); + expect(envelope.platform_config).toEqual(expected); + expect(Object.keys(envelope.platform_config)).toHaveLength(13); + expect(JSON.parse(mockS3Send.mock.calls[0][0].input.Body).platform_config).toEqual(expected); + }); +}); diff --git a/cdk/test/handlers/start-session-composition.test.ts b/cdk/test/handlers/start-session-composition.test.ts index 43f7e146a..1d2e816b9 100644 --- a/cdk/test/handlers/start-session-composition.test.ts +++ b/cdk/test/handlers/start-session-composition.test.ts @@ -93,6 +93,15 @@ process.env.MICROVM_EXECUTION_ROLE_ARN = 'arn:aws:iam::123456789012:role/AbcaMic process.env.MICROVM_EGRESS_CONNECTOR_ARNS = 'arn:aws:lambda:us-east-1:123456789012:network-connector/egress-1'; process.env.MICROVM_PAYLOAD_BUCKET = 'test-microvm-payload-bucket'; +// platform_config (ADR-021 P2): the four REQUIRED identifiers the MicroVM +// strategy refuses to start a session without — they are the agent's only +// channel for them, since a snapshot must not bake configuration in. Read at +// call time by `buildMicrovmPlatformConfig`, but set here alongside the rest +// for clarity. +process.env.GITHUB_TOKEN_SECRET_ARN = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:abca/github-token-AbCdEf'; +process.env.AGENT_SESSION_ROLE_ARN = 'arn:aws:iam::123456789012:role/AbcaAgentSessionRole'; + import { TaskStatus } from '../../src/constructs/task-status'; import { resolveComputeStrategy } from '../../src/handlers/shared/compute-strategy'; import { transitionTask, emitTaskEvent, failTask, buildComputeMetadata } from '../../src/handlers/shared/orchestrator'; diff --git a/cdk/test/stacks/agent.test.ts b/cdk/test/stacks/agent.test.ts index 19fb077b5..028c67950 100644 --- a/cdk/test/stacks/agent.test.ts +++ b/cdk/test/stacks/agent.test.ts @@ -77,6 +77,63 @@ describe('AgentStack', () => { }); }); + test('the orchestrator carries the platform_config transport env on EVERY compute type', () => { + // Wired unconditionally rather than under the lambda-microvm gate: the strategy + // fails a session start when a required identifier is missing, and that guard + // must only ever fire for a hand-edited Lambda environment — never because a + // deploy-time gate and a per-repo `compute_type` disagreed. These are the + // MicroVM's substitute for the AgentCore runtime env block / ECS container env, + // since a snapshot must not bake configuration in (ADR-021 sub-decision 3). + const [, orchestrator] = Object.entries(template.findResources('AWS::Lambda::Function')) + .find(([id]) => id.includes('TaskOrchestratorOrchestratorFn'))!; + const env = orchestrator.Properties.Environment.Variables as Record; + + for (const key of [ + 'TASK_APPROVALS_TABLE_NAME', + 'NUDGES_TABLE_NAME', + 'LOG_GROUP_NAME', + 'ARTIFACTS_BUCKET_NAME', + 'TRACE_ARTIFACTS_BUCKET_NAME', + 'AGENT_SESSION_ROLE_ARN', + 'ANTHROPIC_DEFAULT_HAIKU_MODEL', + ]) { + expect(env[key]).toBeDefined(); + } + // Plus the three the orchestrator already carried for its own work — together + // these cover all four identifiers the MicroVM strategy treats as required. + expect(env.TASK_TABLE_NAME).toBeDefined(); + expect(env.TASK_EVENTS_TABLE_NAME).toBeDefined(); + expect(env.GITHUB_TOKEN_SECRET_ARN).toBeDefined(); + }); + + test('the forwarded identifiers are the SAME stack values the AgentCore runtime gets', () => { + // One stack value, one env-var name, three backends — so an agent behaves + // identically on every substrate and a value can only be changed in one place. + // A drift here would mean a MicroVM agent writing approvals to a different + // table than an AgentCore agent on the same deployment. + const [, orchestrator] = Object.entries(template.findResources('AWS::Lambda::Function')) + .find(([id]) => id.includes('TaskOrchestratorOrchestratorFn'))!; + const orchestratorEnv = orchestrator.Properties.Environment.Variables as Record; + + const runtimes = template.findResources('AWS::BedrockAgentCore::Runtime'); + const runtimeEnv = Object.values(runtimes)[0]!.Properties.EnvironmentVariables as Record; + + for (const key of [ + 'TASK_APPROVALS_TABLE_NAME', + 'NUDGES_TABLE_NAME', + 'LOG_GROUP_NAME', + 'ARTIFACTS_BUCKET_NAME', + 'TRACE_ARTIFACTS_BUCKET_NAME', + 'AGENT_SESSION_ROLE_ARN', + 'ANTHROPIC_DEFAULT_HAIKU_MODEL', + 'TASK_TABLE_NAME', + 'TASK_EVENTS_TABLE_NAME', + 'GITHUB_TOKEN_SECRET_ARN', + ]) { + expect(JSON.stringify(orchestratorEnv[key])).toEqual(JSON.stringify(runtimeEnv[key])); + } + }); + test('outputs ComputeSubstrate=agentcore on the default (no-gate) deploy', () => { // The CLI reads this to refuse onboarding a repo as compute_type=ecs on a // stack that never provisioned the ECS substrate. @@ -924,6 +981,51 @@ describe('AgentStack with the Lambda MicroVMs substrate gate (--context compute_ expect(rendered).not.toContain('microvm-image:*'); }); + test('wires the P2 runtime-parity grants onto the MicroVM execution role', () => { + // Construct-level scoping is asserted in test/constructs/lambda-microvm-compute + // test; what only the STACK can get wrong is passing the props at all — a + // missing `githubTokenSecret` or `agentMemory` here synthesizes cleanly and + // fails at run time (no clone / silently-dropped memory writes). + const policies = JSON.stringify( + Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('LambdaMicrovmComputeExecutionRole')), + ); + // The platform GitHub PAT secret, by reference to the real stack secret. + expect(policies).toContain('secretsmanager:GetSecretValue'); + expect(policies).toContain('GitHubTokenSecret'); + // AgentCore Memory, so cross-task learning persists on this substrate. + expect(policies).toContain('bedrock-agentcore:CreateEvent'); + // Bedrock, scoped to the shared model list rather than a wildcard. + expect(policies).toContain('bedrock:InvokeModel'); + expect(policies).toContain('inference-profile/us.anthropic.claude-opus-4-8'); + // Tenant data stays on the SessionRole: no direct DynamoDB, ever. + expect(policies).not.toContain('dynamodb:'); + expect(policies).toContain('sts:TagSession'); + }); + + test('grants the MicroVM execution role writes on the SAME log group platform_config names', () => { + // ADR-021 P2-F4, and a stack-level property by construction: the construct can + // only grant against the log group the stack hands it, and the orchestrator can + // only deliver the name the stack puts in `agentPlatformConfig`. If those two + // ever came from different objects the deploy would still succeed and every + // per-task log line would AccessDenied — which is exactly what the live P2 run + // hit. So this asserts they are the SAME logical resource. + const policies = JSON.stringify( + Object.entries(template.findResources('AWS::IAM::Policy')) + .filter(([id]) => id.includes('LambdaMicrovmComputeExecutionRole')), + ); + expect(policies).toContain('logs:CreateLogStream'); + expect(policies).toContain('RuntimeApplicationLogGroup'); + + // ...and the orchestrator delivers that group's NAME as LOG_GROUP_NAME. + const orchestrator = Object.entries(template.findResources('AWS::Lambda::Function')) + .find(([id]) => id.includes('TaskOrchestratorOrchestratorFn'))!; + const logGroupEnv = JSON.stringify( + orchestrator[1].Properties.Environment.Variables.LOG_GROUP_NAME, + ); + expect(logGroupEnv).toContain('RuntimeApplicationLogGroup'); + }); + test('MicroVM resources carry the backend cost-allocation tag', () => { template.hasResourceProperties('AWS::Lambda::MicrovmImage', { Tags: Match.arrayWith([{ Key: 'abca:compute-backend', Value: 'lambda-microvm' }]), diff --git a/cli/src/format.ts b/cli/src/format.ts index 3283aad34..eb6bb69bd 100644 --- a/cli/src/format.ts +++ b/cli/src/format.ts @@ -88,6 +88,15 @@ export function formatTaskDetail(task: TaskDetail): string { if (task.completed_at) { lines.push(`Completed: ${task.completed_at}`); } + // In-guest liveness (ADR-021 P2r2-F11). Shown only while the task is still + // going: that is the window in which "is the agent alive?" is a live question, + // and where a stale value is the actionable signal — the orchestrator's own hang + // detector reads the same field. On a terminal task the last beat is just noise + // next to Completed/Duration. The relative age is what an operator actually + // needs, so it is rendered alongside the timestamp rather than instead of it. + if (task.agent_heartbeat_at && !isTerminalStatus(task.status)) { + lines.push(`Heartbeat: ${task.agent_heartbeat_at} (${heartbeatAge(task.agent_heartbeat_at)})`); + } if (task.duration_s !== null) { lines.push(`Duration: ${task.duration_s}s`); } @@ -249,6 +258,16 @@ export function formatStatusSnapshot( if (task.artifact_uri) { lines.push(` Artifact: ${task.artifact_uri}`); } + // In-guest liveness, next to the control-plane freshness line it complements + // (ADR-021 P2r2-F11). `Last event:` says when the platform last heard *about* + // the task; `Heartbeat:` says when the agent last said it was alive — the same + // field the orchestrator's hang detector reads, on a fixed 45 s cadence, so its + // AGE is the diagnostic. Suppressed on a terminal task (the last beat is noise + // beside a final status) and when the agent has not beaten yet. + if (task.agent_heartbeat_at && !isTerminalStatus(task.status)) { + const age = relativeTime(task.agent_heartbeat_at, now) ?? PLACEHOLDER; + lines.push(` Heartbeat: ${age} ago`); + } lines.push(` Last event: ${lastEventLine}`); return lines.join('\n'); @@ -654,6 +673,20 @@ function readNumberField(meta: Record | undefined, key: string) return typeof v === 'number' && Number.isFinite(v) ? v : null; } +/** + * Age of the agent's last heartbeat, as the detail view renders it. + * + * Wraps {@link relativeTime} with `Date.now()` and a placeholder, because the + * heartbeat's VALUE is the age rather than the timestamp: the beat is written every + * 45 s, so anything much past that is the signal an operator is looking for (it is + * the same field the orchestrator's own hang detector reads). Unparseable input + * degrades to a dash rather than throwing — a malformed timestamp must not take out + * `bgagent status`. + */ +function heartbeatAge(isoTimestamp: string): string { + return relativeTime(isoTimestamp, Date.now()) ?? PLACEHOLDER; +} + /** * Compact relative time like "42s", "3m 14s", "1h 02m". Returns null if * the timestamp does not parse — callers fall back to a placeholder. diff --git a/cli/src/types.ts b/cli/src/types.ts index 90a64c4a7..5f6944103 100644 --- a/cli/src/types.ts +++ b/cli/src/types.ts @@ -111,6 +111,13 @@ export interface TaskDetail { readonly updated_at: string; readonly started_at: string | null; readonly completed_at: string | null; + /** ISO timestamp of the agent's last heartbeat (45 s cadence, every compute + * backend); ``null`` before the first beat or on tasks that never ran. The + * platform's only in-guest liveness signal, surfaced through the API as of + * ADR-021 P2r2-F11 — it was written and consumed internally but hidden from + * every operator, which produced a wrong live-verification conclusion. Mirrors + * ``cdk/src/handlers/shared/types.ts::TaskDetail``. */ + readonly agent_heartbeat_at: string | null; readonly duration_s: number | null; readonly cost_usd: number | null; readonly build_passed: boolean | null; diff --git a/cli/test/format-status-snapshot.test.ts b/cli/test/format-status-snapshot.test.ts index c057673c7..53bb856fa 100644 --- a/cli/test/format-status-snapshot.test.ts +++ b/cli/test/format-status-snapshot.test.ts @@ -46,6 +46,10 @@ function buildTask(overrides: Partial = {}): TaskDetail { updated_at: '2026-04-29T15:30:00Z', started_at: '2026-04-29T15:27:06Z', // 3m 14s before NOW completed_at: null, + // Default null: the exact-output assertion below must not have to carry a + // Heartbeat line, and "the agent has not beaten yet" is the honest default for + // a freshly-built fixture. + agent_heartbeat_at: null, duration_s: null, cost_usd: null, build_passed: null, @@ -271,6 +275,48 @@ describe('formatStatusSnapshot', () => { ); }); + describe('agent_heartbeat_at — in-guest liveness (ADR-021 P2r2-F11)', () => { + // `bgagent status` renders this snapshot, so this is the surface that decides + // whether the heartbeat is observable at all. It was not: the field lived in + // DynamoDB, drove the orchestrator's hang detector, and never reached the API — + // so a live verification run polled `status`, saw nothing, and recorded + // "heartbeats NOT observed" against the wrong defect. + + test('shows the age beside the Last event line while the task is live', () => { + const task = buildTask({ agent_heartbeat_at: '2026-04-29T15:29:36Z' }); // 44s before NOW + const rendered = formatStatusSnapshot(task, [], NOW); + expect(rendered).toContain(' Heartbeat: 44s ago'); + // Ordering is the point of putting it here: control-plane freshness + // (`Last event`) and in-guest freshness (`Heartbeat`) read together. + expect(rendered.indexOf('Heartbeat:')).toBeLessThan(rendered.indexOf('Last event:')); + }); + + test('a stale beat is visible as a large age rather than being hidden', () => { + // The actionable case: the substrate says RUNNING and the guest went quiet. + // 45 s is the write cadence, so ~4 minutes is unambiguous. + const task = buildTask({ agent_heartbeat_at: '2026-04-29T15:26:20Z' }); + expect(formatStatusSnapshot(task, [], NOW)).toContain(' Heartbeat: 4m 00s ago'); + }); + + test('is suppressed on a terminal task', () => { + const task = buildTask({ + status: 'COMPLETED', + completed_at: '2026-04-29T15:29:50Z', + agent_heartbeat_at: '2026-04-29T15:29:38Z', + }); + expect(formatStatusSnapshot(task, [], NOW)).not.toContain('Heartbeat:'); + }); + + test('is omitted entirely when the agent has not beaten yet', () => { + expect(formatStatusSnapshot(buildTask(), [], NOW)).not.toContain('Heartbeat:'); + }); + + test('degrades to a placeholder on an unparseable timestamp', () => { + const task = buildTask({ agent_heartbeat_at: 'garbage' }); + expect(formatStatusSnapshot(task, [], NOW)).toContain(' Heartbeat: — ago'); + }); + }); + test('missing / non-string timestamp degrades "Last event" to placeholder', () => { // The event table is weakly typed at the storage layer: a malformed // agent write could produce a row without ``timestamp``. Without the diff --git a/cli/test/format.test.ts b/cli/test/format.test.ts index 5d66e3d91..5e4293870 100644 --- a/cli/test/format.test.ts +++ b/cli/test/format.test.ts @@ -40,6 +40,7 @@ describe('format', () => { updated_at: '2026-01-01T01:00:00Z', started_at: '2026-01-01T00:01:00Z', completed_at: '2026-01-01T01:00:00Z', + agent_heartbeat_at: '2026-01-01T00:59:48Z', duration_s: 3540, cost_usd: 0.1234, build_passed: true, @@ -132,6 +133,57 @@ describe('format', () => { expect(output).not.toContain('Max Turns:'); }); + describe('agent_heartbeat_at (ADR-021 P2r2-F11)', () => { + // The field exists in DynamoDB and drives the orchestrator's hang detector, + // but was never projected into the API response — so `bgagent status` + // printed nothing while a 6-second-old beat sat in the table, and a live + // verification run concluded "heartbeats not observed". The CLI is the + // consumer that makes the signal actionable, so it renders the AGE (the beat + // is written every 45 s; the age is what tells you something is wrong). + + test('shows the heartbeat and its age while the task is still running', () => { + const running: TaskDetail = { + ...task, + status: 'RUNNING', + completed_at: null, + agent_heartbeat_at: new Date(Date.now() - 12_000).toISOString(), + }; + const output = formatTaskDetail(running); + expect(output).toContain('Heartbeat: '); + expect(output).toMatch(/Heartbeat:.*\(1[0-9]s\)/); + }); + + test('hides it on a terminal task, where the last beat is noise', () => { + // The fixture is COMPLETED and DOES carry a heartbeat, so this asserts the + // suppression rather than an absent value. + expect(task.agent_heartbeat_at).not.toBeNull(); + expect(formatTaskDetail(task)).not.toContain('Heartbeat:'); + }); + + test('hides it when the agent has not beaten yet', () => { + const running: TaskDetail = { + ...task, + status: 'RUNNING', + completed_at: null, + agent_heartbeat_at: null, + }; + expect(formatTaskDetail(running)).not.toContain('Heartbeat:'); + }); + + test('degrades to a placeholder rather than throwing on a malformed value', () => { + // A bad timestamp must not take out `bgagent status` — the command a user + // reaches for when something is already wrong. + const running: TaskDetail = { + ...task, + status: 'RUNNING', + completed_at: null, + agent_heartbeat_at: 'not-a-timestamp', + }; + expect(() => formatTaskDetail(running)).not.toThrow(); + expect(formatTaskDetail(running)).toContain('Heartbeat: not-a-timestamp (—)'); + }); + }); + test('renders a repo-less placeholder when repo is null (#248 Phase 3)', () => { const repoless: TaskDetail = { ...task, diff --git a/contracts/constants.json b/contracts/constants.json index 2bd422306..96c14c035 100644 --- a/contracts/constants.json +++ b/contracts/constants.json @@ -16,5 +16,33 @@ "jira_app_actor": { "min_secret_length": 32, "forge_webtrigger_suffix": ".webtrigger.atlassian.app" + }, + "microvm_platform_config": { + "env_by_key": { + "task_table_name": "TASK_TABLE_NAME", + "task_events_table_name": "TASK_EVENTS_TABLE_NAME", + "task_approvals_table_name": "TASK_APPROVALS_TABLE_NAME", + "nudges_table_name": "NUDGES_TABLE_NAME", + "log_group_name": "LOG_GROUP_NAME", + "artifacts_bucket_name": "ARTIFACTS_BUCKET_NAME", + "trace_artifacts_bucket_name": "TRACE_ARTIFACTS_BUCKET_NAME", + "github_token_secret_arn": "GITHUB_TOKEN_SECRET_ARN", + "linear_oauth_secret_arn": "LINEAR_OAUTH_SECRET_ARN", + "jira_oauth_secret_arn": "JIRA_OAUTH_SECRET_ARN", + "agent_session_role_arn": "AGENT_SESSION_ROLE_ARN", + "aws_sdk_ua_app_id": "AWS_SDK_UA_APP_ID", + "anthropic_default_haiku_model": "ANTHROPIC_DEFAULT_HAIKU_MODEL" + }, + "required": [ + "task_table_name", + "task_events_table_name", + "github_token_secret_arn", + "agent_session_role_arn" + ] + }, + "microvm_hook_budgets": { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": 240, + "warmup_required_timeout_seconds": 120 } } diff --git a/contracts/constants.md b/contracts/constants.md index d6fed37c0..ddb7dba74 100644 --- a/contracts/constants.md +++ b/contracts/constants.md @@ -21,7 +21,10 @@ the contract. This is the neutral location both runtimes read. |---|---|---| | `agent/src/shared_constants.py` | `/app/contracts/constants.json` | import-time | | `agent/src/policy.py`, `agent/src/jira_reactions.py` | `SHARED_CONSTANTS` | import-time | +| `agent/src/server.py` | `SHARED_CONSTANTS["microvm_platform_config"]`, `SHARED_CONSTANTS["microvm_hook_budgets"]` | import-time | | `cdk/src/handlers/shared/types.ts`, `jira-app-actor.ts` | `../../../../contracts/constants.json` | synth-time `import` | +| `cdk/src/handlers/shared/strategies/lambda-microvm-strategy.ts` | `microvm_platform_config` | synth-time `import`, read per session start | +| `cdk/src/constructs/lambda-microvm-compute.ts` | `microvm_hook_budgets` | synth-time `import` | | `cdk/src/constructs/blueprint.ts` | re-exports from `types.ts` | synth-time | | `cli/test/constants-parity.test.ts` | package-safe literal parity | test-time | @@ -51,6 +54,16 @@ JSON at TypeScript compile time via `resolveJsonModule`. "jira_app_actor": { "min_secret_length": 32, "forge_webtrigger_suffix": ".webtrigger.atlassian.app" + }, + "microvm_platform_config": { + "env_by_key": { "task_table_name": "TASK_TABLE_NAME", "...": "..." }, + "required": ["task_table_name", "task_events_table_name", + "github_token_secret_arn", "agent_session_role_arn"] + }, + "microvm_hook_budgets": { + "ready_hook_timeout_seconds": 300, + "warmup_total_budget_seconds": 240, + "warmup_required_timeout_seconds": 120 } } ``` @@ -81,6 +94,49 @@ JSON at TypeScript compile time via `resolveJsonModule`. accepted by the agent, CDK, and CLI Jira app-actor clients. - **`jira_app_actor.forge_webtrigger_suffix`** — hostname suffix required by app-actor proxy URL validation to prevent operator-supplied SSRF targets. +- **`microvm_platform_config.env_by_key`** — the Lambda MicroVMs `platform_config` + allowlist (ADR-021 P2): each wire key (snake_case) mapped to the environment + variable the agent installs it as (UPPER_SNAKE). This block is unlike the + others — it is a **security allowlist**, not a tuning bound. The MicroVM image + is a snapshot whose env is frozen at build time, so the agent's non-secret + platform env arrives in the `/run` hook payload instead; the values land in + `os.environ`, which makes an unrecognised key an env-injection attempt. The + consumer (`agent/src/server.py`) therefore **rejects** any `platform_config` + carrying a key that is not in this map. Values are non-secret identifiers + (table/bucket names, secret ARNs, role ARNs) only. +- **`microvm_platform_config.required`** — the subset without which a task cannot + run (task + event tables, GitHub secret ARN, session role ARN). A `/run` hook + whose `platform_config` misses or blanks any of these is rejected with HTTP 400. + +Both `microvm_platform_config` fields are validated for shape (snake_case keys, +UPPER_SNAKE unique env names, `required ⊆ env_by_key`) by +`scripts/check-constants-sync.ts` **and** by `agent/src/server.py` at import time, +so a malformed contract fails the drift check *and* the MicroVM image build. + +- **`microvm_hook_budgets.ready_hook_timeout_seconds`** — the `/ready` build-hook + budget the CDK construct declares to `CreateMicrovmImage` + (`READY_HOOK_TIMEOUT_SECONDS` in `cdk/src/constructs/lambda-microvm-compute.ts`). + 300 s, not 60 s, because as of ADR-021 P2-F5 `/ready` does real work: it warms the + 225 MiB `claude` binary so its pages are resident when the snapshot is taken. +- **`microvm_hook_budgets.warmup_total_budget_seconds`** — the agent's ceiling for + the WHOLE `/ready` warm-up (`_READY_WARMUP_TOTAL_BUDGET_SECONDS` in + `agent/src/server.py`): required command plus every best-effort one, which share + the remainder rather than each getting a fresh budget. +- **`microvm_hook_budgets.warmup_required_timeout_seconds`** — the required + warm-up's own slice (`_READY_WARMUP_REQUIRED_TIMEOUT_SECONDS`). Generous on + purpose: a cold 225 MiB `exec` has no predictable duration, which is the lesson of + P2-F5. + +Unlike every other block here, these three are not independent tuning bounds — they +are a **relationship**: `warmup_required < warmup_total < ready_hook`. The warm-up +must finish inside the budget the service holds the hook to, or a fix for a runtime +failure turns into a build failure. A relationship cannot be enforced from one side, +which is why both halves live in the contract even though each has a single +consumer. `scripts/check-constants-sync.ts` asserts the ordering and rejects a +literal re-declaration on **either** side — the Python constants *and* +`READY_HOOK_TIMEOUT_SECONDS` in the TypeScript construct — and +`agent/src/server.py` re-checks the same ordering at import time, so a bad contract +fails the drift check *and* the image build. The published CLI package contains only `lib/`, so it cannot load the repository contract at runtime. It mirrors these values as literals and diff --git a/docs/decisions/ADR-021-lambda-microvms-compute-backend.md b/docs/decisions/ADR-021-lambda-microvms-compute-backend.md index 9da6bc927..fe376a7f0 100644 --- a/docs/decisions/ADR-021-lambda-microvms-compute-backend.md +++ b/docs/decisions/ADR-021-lambda-microvms-compute-backend.md @@ -26,7 +26,7 @@ ABCA selects a per-repo compute backend through the Blueprint's `compute_type` f | Resources | AgentCore-managed | 16 vCPU / 120 GB / 20–200 GB disk | **Baseline 8 GiB RAM / 4 vCPU, auto-scaling to a 32 GiB / 16 vCPU peak**; 32 GB disk. `minimumMemoryInMiB` configures the BASELINE (max 8,192 MiB); the service scales vertically on demand — capacity is baseline-priced with 4× burst headroom | | Packaging | ECR image ≤ 2 GB | ECR image, no hard cap | **Zip + Dockerfile in S3 → service-built snapshot image** (versioned, storage billed) | | Invocation | `InvokeAgentRuntime` (SigV4) | `RunTask` + container overrides | `RunMicrovm` (image **ARN** required — a bare name is rejected) → dedicated HTTPS endpoint + JWE token (`CreateMicrovmAuthToken`, ≤ 60 min TTL) | -| Liveness | Agent heartbeat + `/ping` | `DescribeTasks` | MicroVM state (RUNNING / SUSPENDED / TERMINATED) via control-plane API | +| Liveness | Agent heartbeat + `/ping` | `DescribeTasks` | MicroVM state (RUNNING / SUSPENDED / TERMINATED) via control-plane API **and** agent heartbeat (see sub-decision 1) | | Session storage | `/mnt/workspace` FUSE (no `flock()`) | Ephemeral disk | Native disk in snapshot — **survives suspend/resume, `flock()` works** | | Architecture | ARM64 | ARM64 | ARM64 (Graviton) | | Regions (launch) | Broad | Broad | 5 (us-east-1/2, us-west-2, eu-west-1, ap-northeast-1) | @@ -68,6 +68,10 @@ The `ComputeStrategy` interface gains **mandatory** `suspendSession(handle)` / ` **Poll semantics — the strategy reports, the orchestrator interprets.** `pollSession(handle)` receives only the session handle and cannot see task state, so the health rules must live where the DynamoDB status lives. `SessionStatus` gains a `'suspended'` variant; the strategy maps `GetMicrovm` state mechanically and the **orchestrator** cross-references against the task row — the same division of labor `finalPollState` already uses for ECS (substrate stopped + non-terminal DynamoDB status → failed) and `pollTaskStatus` uses for agentcore heartbeats: substrate `suspended` + task `AWAITING_APPROVAL` is healthy (orchestrator-intended suspend); `suspended` with any other task status is an anomaly to surface, not fail-fast; substrate terminal + non-terminal task status → classify failed. +**Liveness on this backend is substrate state AND agent heartbeat.** The substrate cross-check above answers one question — "is the VM still there?" — and P2 established that it is not sufficient on its own. `GetMicrovm` catches a MicroVM that *died*; it cannot catch a MicroVM that is alive and reporting `RUNNING` while the pipeline **inside the guest** is hung, deadlocked, or was OOM-killed. That state is not hypothetical or self-correcting on this substrate, and the P2 live run narrowed *why* without weakening the conclusion. The service does reap a VM whose run hook FAILS (a 4xx makes it terminate within ~12 s — see sub-decision 2), so the P1 evidence for this paragraph (a hook-less image sitting in `RUNNING` indefinitely with no `stateReason`) no longer describes an ABCA image. What the service reaps is a hook *result*; it has no view into the guest afterwards. So the surviving — and more realistic — hang case is a `/run` hook that returned **200** and a pipeline that then hung, deadlocked or was OOM-killed behind it: the substrate stays `RUNNING`, the service is satisfied, and nothing else notices. Left to the substrate check alone, such a task would burn the orchestrator's full ~8.5 h poll window — billing an 8-hour reservation — before the safety net fired. + +The in-guest half is already being written: the agent updates `agent_heartbeat_at` on the task row unconditionally, with no backend awareness, so the timestamp exists on every substrate. Only the orchestrator's *reaction* to it was backend-scoped — `pollTaskStatus` evaluated staleness for `agentcore` alone — which is the gap P2 closes by extending it to `lambda-microvm`. The grace and stale thresholds are the SAME on both: the timestamp is written by the same pipeline code at the same cadence, so a backend-specific window would encode a difference that does not exist. The two signals stay complementary rather than redundant — the substrate check is the crash detector, the heartbeat is the hang detector — and the check remains scoped to task status `RUNNING`, which is what keeps a deliberately suspended VM during an approval wait (sub-decision 2, P3) from being read as a dead one. `ecs` is deliberately left out: `DescribeTasks` reports a real container exit *with an exit code* (OOM-kill included) and the ECS poll block already interprets it with its own patience counters, so adding the heartbeat there would give one backend two independently-tuned kill paths for the same failure. + The service's `MicrovmState` enum has **six** members, not three, so the mapping is stated exhaustively (one line of rationale each, mirrored in the strategy's doc comment): | `MicrovmState` | `SessionStatus` | Why | @@ -103,6 +107,8 @@ Normative requirements (EARS, per [ADR-020](./ADR-020-ears-requirements-syntax.m - If `GetMicrovm` reports that the MicroVM does not exist, then the strategy shall report `completed`. - If the strategy reports a terminal substrate state while the task's DynamoDB status is non-terminal, then the orchestrator shall re-read the task row and, if it is still non-terminal, classify the task as failed with a substrate-failure remedy. - If the strategy reports `suspended` while the task's DynamoDB status is not `AWAITING_APPROVAL`, then the orchestrator shall surface an anomaly event and shall not fail-fast the task. +- While a `lambda-microvm` task's DynamoDB status is `RUNNING`, if the task's `agent_heartbeat_at` is stale (or absent past the grace window) by the same thresholds the orchestrator applies to `agentcore`, then the orchestrator shall treat the session as unhealthy and stop polling — the substrate `GetMicrovm` check shall remain the crash detector, and the heartbeat shall be the in-guest hang detector. +- The task-detail API response shall include `agent_heartbeat_at`, and the CLI shall surface it while the task is non-terminal (P2r2-F11: the field drove the orchestrator's hang detector but was never projected, so no operator could observe the signal — and its invisibility produced a wrong verification conclusion). - If `suspendSession` or `resumeSession` is invoked on a strategy that does not support suspension, then the strategy shall return an explicit unsupported result. - When the agent process reaches a terminal state, the agent shall exit. - When the orchestrator finalizes a `lambda-microvm` task, the orchestrator shall call `terminate-microvm` (termination shall not rely on any substrate timeout, and shall not rely on the MicroVM self-terminating — it does not). @@ -120,7 +126,11 @@ The handshake must respect the existing approval mechanics: the agent **discover *Why inline rather than poll-only — codebase precedent:* resume-on-approve is structurally identical to task cancellation — a user-initiated, latency-sensitive action whose purpose is an immediate compute-lifecycle side effect. `cancel-task.ts` already resolves this exact tension: the API-plane handler invokes ECS `StopTask` / AgentCore `StopRuntimeSession` inline, best-effort (a failed stop logs a warning and the state transition stands; a `task_cancel_compute_orphan` event is written when no stoppable compute handle exists, `reason: missing_runtime_handle`) — with the conditional IAM wired in `task-api.ts`. The resume path goes one step further than the precedent by also writing the orphan event on *failed* resume calls, because a failed resume strands a suspended VM awaiting a decision — a stronger liveness consequence than a failed stop of an already-cancelled task. The alternative (orchestrator-poll-only resume) preserves single-owner lifecycle purity but pays up to a full poll interval (~30 s) of latency on every approval, and the purity argument was already litigated and declined for cancel. `approve-task.ts` is deliberately minimal today (security-critical ownership comparison, Cedar finding #6); the resume call is therefore added *after* the transaction commits, cannot alter the decision outcome, and carries one conditional `lambda:ResumeMicrovm` grant — the same blast-radius trade the cancel handler accepted in review. - **Timeout under freeze — the agent re-bases on the wall clock it already owns.** The agent's monotonic gate timer freezes while suspended, so resuming near the deadline is not enough: the frozen timer would still hold its remaining budget and fire the deny minutes *after* the user-visible window — colliding with the approval row's TTL (`created_at + timeout_s + 120s`) and triggering the "row reaped → stranded" fallback on a healthy gate. Instead, the gate expires at **`min(monotonic budget, created_at + timeout_s)`**, evaluated on each poll iteration and on `/resume`. This is not a new principle: Cedar decision #6 is already "min wins" for timeouts, the wall-clock deadline is already durable in the approval row the agent itself writes (`created_at` is in the agent's own clock domain — no skew), and §13.12's late-approval race fix already establishes that the durable row is authoritative over the agent's local timer. Deny authority stays agent-side (the conditional `TIMED_OUT` write + ConsistentRead re-read race protection is untouched); the orchestrator's resume at `deadline − margin` is purely the wake-up mechanism, with no correctness role. -- **Backstops, not mechanisms.** `maximumDurationInSeconds` (mandatory on every `RunMicrovm`, pinned at 28 800 s — see sub-decision 1) is the substrate kill switch bounding running **and** suspended time; the orchestrator's finalization `terminate-microvm` is the active cleanup path; the stranded-approval reconciler retains its role for orphaned waits. No `idlePolicy`-based bound is used in any phase — see sub-decision 1's omit-`idlePolicy` invariant. **The active terminate is mandatory, not belt-and-braces**: a MicroVM whose hook never ran reached `RUNNING` in 12 s and stayed `RUNNING` with no `stateReason` through every checkpoint (live). Nothing self-terminates on this substrate, so nothing cleans up — a leaked VM bills until the 8 h cap. +- **Backstops, not mechanisms.** `maximumDurationInSeconds` (mandatory on every `RunMicrovm`, pinned at 28 800 s — see sub-decision 1) is the substrate kill switch bounding running **and** suspended time; the orchestrator's finalization `terminate-microvm` is the active cleanup path; the stranded-approval reconciler retains its role for orphaned waits. No `idlePolicy`-based bound is used in any phase — see sub-decision 1's omit-`idlePolicy` invariant. + + **The active terminate is still mandatory on the SUCCESS path, and P2 sharpened why.** P1 concluded flatly that "nothing self-terminates": a hook-less MicroVM reached `RUNNING` in 12 s and stayed there with no `stateReason` through every checkpoint. P2 refuted that *for the failure path only* — with `run: ENABLED`, a run hook that answers 4xx makes the **service** terminate the VM within ~12 s, `stateReason: "Run lifecycle hook returned HTTP status 400. Please check your hook endpoint and application logs for more details."`, after which `suspend-microvm` correctly refuses it. That is a real improvement in cost posture and a direct benefit of declaring hooks (see also the failure-path row in the phasing table, sub-decision 3). + + It does **not** relieve the orchestrator of anything, because the two cases are disjoint. The service reaps a hook *result* it did not like; it has no view of the guest once the hook returned 200. So a task that starts normally — the overwhelming majority — has no service-side reaper at all, and a VM whose pipeline finished, crashed after `/run`, or hung is reaped by nobody but `TerminateMicrovm`. A leaked handle therefore remains a cost incident that bills until the 8 h cap; only the "the guest rejected its own payload" corner now cleans itself up. - **Concurrency slot stays held** during suspend. Cedar decision #7's rationale ("container alive, consuming memory") weakens under suspend, and the harder replacement rationale — "AWS counts `SUSPENDED` MicroVMs toward the account memory quota, so releasing ABCA's slot would not free real capacity" — is **undischarged**: the suspended VM stayed in `list-microvms` at every checkpoint, but that only proves *listed*. `L-CD1C0CC4` (1024 GB, account-scoped) exposes no `UsageMetric`, `AWS/Usage` carries only `CallCount` per API, and no MicroVM memory metric exists in any namespace, so consumption is **not observable safely** — proving it would need a large concurrent fleet. The conclusion (hold the slot) stands as the conservative choice, not as a verified fact. Size the arithmetic against the 32 GiB **peak** rather than the 8 GiB baseline: a busy fleet scales up, so peak is what actually competes for the account quota. The agent's `/suspend` hook flushes progress events (durable writes before returning 200, within the 60 s hook budget); `/resume` reseeds CSPRNGs and refreshes cached credentials. @@ -149,25 +159,63 @@ The phasing is therefore: | Hook | Declared by | Served by the agent | Notes | |---|---|---|---| -| `/ready` | **P1** (construct sets `hooks.microvmImageHooks.ready`) | **P1** | MANDATORY, not a quality nicety — see above. A 200 once uvicorn is bound is the whole P1 contract: it also proves `server` imported cleanly (pulling in `pipeline` → `runner` → the policy engine), so a missing policy file fails the BUILD instead of the first task. | -| `/run` | **P1** (construct sets `hooks.microvmHooks.run`) | **P1** | The payload-delivery channel. Must be served in P1 because `/ready` forces hooks to exist at all, and a hook-less image cannot accept `runHookPayload`. | -| `/validate` | **P2** | **P2** | Build-time snapshot-quality hook. Still deliberately NOT declared: a `/validate` that 404s or reports failure fails every image build. Deeper warm-up assertions (Bedrock reachability, Memory access, tool availability) belong here. | -| `/suspend`, `/resume`, `/terminate` | **P3** (suspend/resume), P2 (`/terminate`) | P3 / P2 | Declaring a runtime hook the agent does not serve fails the corresponding lifecycle transition, so each is declared only in the phase that implements it. P1 termination is the orchestrator's `TerminateMicrovm`, which needs no in-guest cooperation. | +| `/ready` | **P1** (construct enables `hooks.microvmImageHooks.ready`) | **P1** | MANDATORY, not a quality nicety — see above. A 200 proves uvicorn is bound and `server` imported cleanly (pulling in `pipeline` → `runner` → the policy engine), so a missing policy file fails the BUILD instead of the first task. **Since P2-F5 it also WARMS the snapshot** — the hook's 200 is what the service waits for before capturing the snapshot, making this the only place a warm page can be created, and the 225 MiB `claude` binary was cold in it (see the P2-F5 correction below). A required warm-up failure answers 503, so a snapshot that cannot exec the agent's own CLI fails the image build instead of every task. Still makes ZERO AWS calls, logging included (a `--version` exec is neither an AWS call nor a network call). | +| `/run` | **P1** (construct sets `hooks.microvmHooks.run`) | **P1** | The payload-delivery channel. Must be served in P1 because `/ready` forces hooks to exist at all, and a hook-less image cannot accept `runHookPayload`. Since P2 it is also the **platform-configuration** channel (see "Platform configuration delivery" below). | +| `/validate` | **P2** (construct sets `hooks.microvmImageHooks.validate`) | **P2** | An **image** (build-time) hook, and a **shallow self-check only**: server alive, hook routes registered, interpreter + contract sanity. It runs under the BUILD role, which deliberately holds no Bedrock / Secrets Manager / DynamoDB grants, so it must make **zero AWS API calls** and must not touch credential resolution — the "deeper warm-up assertions (Bedrock reachability, Memory access, tool availability)" this ADR originally assigned here are **not implementable**: every one of them would `AccessDenied` and fail every image build. They belong to the first task's own error handling. 200 when the checks pass, 503 while still initialising. | +| `/terminate` | **P2** (construct sets `hooks.microvmHooks.terminate`) | **P2** | Best-effort final flush: a final structured log line, then 200 — always, inside the hook budget, even with nothing running. It must **not** write terminal task status (the orchestrator finalizes the task and *then* calls `TerminateMicrovm`, so a terminate hook that wrote a status would race that finalization and could clobber the real outcome) and must not join the pipeline thread. There is nothing buffered to flush: `_ProgressWriter` does a synchronous `put_item` per event, so durability is per-write. "Always 200" also covers the BODY: the handler reads the raw request rather than a typed model, because a typed body is validated before the handler runs and would answer 422 to malformed JSON — a reported hook failure on a successful teardown. Safe to declare because `TerminateMicrovm` removes the VM with or without in-guest cooperation. **Correction (P2-F8):** the service sends `microvmId: ""` on this hook, unlike `/run` where it is populated, so an empty id is expected-normal and this hook cannot join the guest record to the control-plane one — `/run`'s accepted line carries that correlation instead. | +| `/suspend`, `/resume` | **P3** | **P3** | Declaring a runtime hook the agent does not serve fails the corresponding lifecycle transition, so each is declared only in the phase that implements it. P1 termination is the orchestrator's `TerminateMicrovm`, which needs no in-guest cooperation. | Consequence to state plainly, replacing the original "a P1-built MicroVM image is not runnable end to end": **a P1 image is creatable, launchable and payload-deliverable, but carries no smoke-parity guarantee.** P1 delivers the strategy, the construct, the roles/buckets/connectors, the image resource, the packaging script, and the `/ready` + `/run` endpoints — so a `lambda-microvm` task can start a MicroVM and hand it a payload. What P1 has **not** established is anything P2 owns: AgentCore Memory grants and `MEMORY_ID` delivery, the agent's non-secret env parity inside the snapshot, egress specifics from a running MicroVM, and heartbeat/progress behaviour end to end. No clone → change → PR run has happened on this substrate. P2 ("smoke parity") is the phase that closes that gap. The construct and the packaging script both surface exactly this at synth/run time (`abca:microvm-image-p1-smoke-unverified`) so an operator cannot mistake a launchable substrate for a verified one. -**Payload delivery** reuses the ECS strategy's S3-pointer pattern, adapted to `runHookPayload` (**≤ 4 KB** — measured, see below): payloads that fit ride inline; the rest are uploaded by the strategy to a platform payload bucket (the ECS payload bucket pattern in `ecs-agent-cluster.ts`: orchestrator write access, compute-role read-only scoped to the bucket, lifecycle expiry on objects) with only the S3 URI in `runHookPayload` — the MicroVM **execution role** holds the read grant, exactly as the ECS task role does today. +**The `AWS::Lambda::MicrovmImage` L1 enforces the API's enums (P2-F2, live 2026-08-06).** This closes the one item P1 left explicitly open, and it closes it against the construct's own stated reasoning. CloudFormation's generated types make `cpuConfigurations[].architecture` and all four `hooks.*` fields plain strings and document no allowed values, from which P1 concluded that the CloudFormation surface takes a *hook path* while the API takes an `ENABLED`/`DISABLED` flag, and that both were correct for their own surface. CloudFormation refused the change set at **early validation** — the stack was never touched, so there was no rollback and no runtime symptom to trace back — on five values: + +``` +/aws/lambda-microvms/runtime/v1/run is not a valid enum value. Supported values: [DISABLED, ENABLED] + (at /Resources/…/Properties/Hooks/MicrovmHooks/Run) … and the same for Terminate, Ready, Validate +arm64 is not a valid enum value. Supported values: [ARM_64] + (at /Resources/…/Properties/CpuConfigurations/0/Architecture) +``` + +Three consequences. First, the CloudFormation surface is **identical** to the API surface, and the packaging script (`--cpu-configurations '[{"architecture":"ARM_64"}]'`, `--hooks '{"microvmHooks":{"run":"ENABLED",…}}'`) had it right all along. Second, the "CDK-managed (recommended)" bootstrap path was **non-functional** for the whole of P1 and P2 — the out-of-band `--create-image` script was the only working path — and no unit test, `cdk synth` or cdk-nag rule could see it, because the types accept any string. Third, **hook paths are not configurable on either surface**: the service calls fixed well-known routes (proved by the build and run logs, which POST to exactly the `/aws/lambda-microvms/runtime/v1/*` paths the agent serves), so the route constants in the construct are an agent-side cross-package contract ONLY and must never be sent as property values again. Also discharged in passing: the `microvmImageHooks` property name and nesting are correct — CloudFormation resolved `…/Hooks/MicrovmImageHooks/Ready` and objected only to its value. + +**A snapshot is only as warm as the pages touched before it was captured (P2-F5, live 2026-08-07).** This is the defect that stopped the P2 smoke run one step short of a pull request, and it is a property of the substrate rather than a bug in any one file. Every task failed at turn 0, reproducibly: + +``` +TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +``` + +The binary was fine — in the identical image, locally, `claude --version` answers `2.1.191 (Claude Code)` in under a second. It is a **225 MiB (236,305,136-byte) statically-linked ELF** that nothing had exec'd before the snapshot was taken, so on a guest restored ~50 s earlier the first `exec` had to fault all of those pages in from lazily-restored storage, and 10 s was not enough. `/ready` existed precisely so "the snapshot is taken with a warm server", and the snapshot was warm for uvicorn and stone cold for the binary that does all the work. + +Both halves of the fix are kept, because they answer different questions. `/ready` now **exec's the heavyweight binaries before returning 200** (`claude` required, `git`/`node` best-effort), which is the only mechanism that can make the shipped snapshot warm — and its own budget rises to 300 s, well inside the 3600 s build-hook window, because it now does work whose duration is a cold `exec`. Two structural rules keep that honest, because **per-command timeouts do not compose**: the required command runs FIRST with its own budget so no best-effort warm-up can starve the one that decides whether the snapshot is usable, and the best-effort ones then SHARE the remainder of a total warm-up ceiling that sits inside the hook budget with margin (240 s against 300 s). Without them, three commands at 120 s each would be 360 s — a fix for a runtime failure that produces a build failure instead — and a single hung optional command could hold up a 200 that the required warm-up had already earned. Separately, the version probe's timeout goes from 10 s to 60 s: a probe that exists to print a version string into a log line gains nothing from a tight bound and loses the whole task when it trips. The general rule this generalises to, and the reason it belongs in the ADR rather than only in a comment: **on this backend, a first-touch cost that other substrates pay during container start is deferred to the first task instead**, so anything large and lazily-loaded is a turn-0 hazard unless it is touched in `/ready`. + +**Payload delivery** reuses the ECS strategy's S3-pointer pattern, adapted to `runHookPayload` (**≤ 4 KB** — measured, see below): payloads that fit ride inline; the rest are uploaded by the strategy to a platform payload bucket (the ECS payload bucket pattern in `ecs-agent-cluster.ts`: orchestrator write access, compute-role read-only scoped to the bucket, lifecycle expiry on objects) with the S3 URI in `runHookPayload` in place of the payload itself — the MicroVM **execution role** holds the read grant, exactly as the ECS task role does today. Since P2 the hook body also carries `platform_config` in both branches, so `runHookPayload` is never *only* the URI (see the canonical shapes below). The cap is **4 096 bytes**, not the 16 384 the SDK documents. Measured exactly: 4 096 passes, 4 097 is rejected with *"Value at 'runHookPayload' failed to satisfy constraint: Member must have length less than or equal to 4096"*. Two consequences follow. First, the original threshold would have inlined every envelope between 4 097 and 16 384 bytes and had the service reject all of them. Second, and more structurally: **the S3-pointer path is now the dominant one, and inline is the exception.** A hydrated task payload (prompt + issue thread + repo context) essentially always exceeds 4 KB, so "small payloads ride inline" describes tiny repo-less prompts rather than the common case. The payload bucket is therefore not a rarely-exercised overflow valve but a required part of every normal task, which raises its lifecycle rule (`MICROVM_PAYLOAD_TTL_DAYS`) and the execution role's read grant from edge-case plumbing to load-bearing. +**Canonical wire shapes.** Three, and the producer (`lambda-microvm-strategy.ts`) emits exactly these: + +| Where | Exact shape | +|---|---| +| `runHookPayload`, inline branch | `{"agent_payload": {…}, "platform_config": {…}}` | +| `runHookPayload`, pointer branch | `{"agent_payload_s3_uri": "s3://…", "platform_config": {…}}` | +| the object at that S3 URI | `{…agent_payload fields…, "platform_config": {…}}` — the payload's own fields at the TOP level, with the config merged in beside them | + +Two asymmetries are deliberate and must not be "tidied" without changing both sides. First, **the S3 object is not the envelope**: the payload's fields sit at the top level (that is what P1 uploaded, before `platform_config` existed) rather than nested under an `agent_payload` key. Second, **`platform_config` is duplicated** on the pointer path — once beside the pointer, once inside the uploaded object. It costs a few hundred bytes and buys the property that the config is reachable whichever end of the fetch a reader looks at, which matters because it is the agent's only substitute for an env block. + +The agent's reader is deliberately more permissive than this contract: it also accepts an S3 object shaped like the envelope (`agent_payload` nested), and `platform_config` present in only one of the two places (the fetched object wins, the hook body is the fallback). Those are **defensive compatibility** for the independent deploy cadences of a snapshot image and the orchestrator Lambda — a tolerant reader, not an alternative contract. A producer must emit the three shapes above. + +**Platform configuration delivery (P2): payload-sourced, allowlisted, fail-closed.** The other two backends hand the agent its non-secret platform env at launch — AgentCore Runtime env vars, ECS container overrides — and there is no equivalent on this substrate: a MicroVM starts from a **snapshot**, so its process environment is whatever was frozen at *image build* time and is then replayed by every MicroVM launched from that image version. Baking the deployment's identifiers into the snapshot would make them **version-frozen**: a redeploy that renames a table, adds a bucket or rotates the session role would leave every existing image version describing a deployment that no longer exists, and the drift would surface as a task-time `ResourceNotFound` rather than a deploy-time error. So the values travel with the task instead: `platform_config` is a SIBLING of the payload — beside `agent_payload` in the inline branch, beside `agent_payload_s3_uri` in the pointer branch, and merged in beside the payload's own fields inside the S3 object (the canonical shapes above give each one exactly) — whose snake_case keys the agent installs into `os.environ` as their UPPER_SNAKE equivalents. A payload value therefore **wins** over any pre-existing/image value — the orchestrator is describing the live deployment, the snapshot is describing a past one. `platform_config` carries **non-secret identifiers only** (table and bucket names, secret ARNs, the session-role ARN); secrets are still fetched at `/run` time from Secrets Manager using those ARNs, so the snapshot-must-stay-secret-free requirement above is untouched. Per-task fields — `memory_id` and friends — stay inside `agent_payload`: `platform_config` configures the *process*, `agent_payload` describes the *task*. + +Two rules make it safe. First, **the allowlist fails closed**: the agent installs a fixed set of keys and *rejects the entire run* (HTTP 400, nothing spawned, not one key installed) if the block carries anything else. These values become environment variables of the process that spawns the agent's tool subprocesses, so an unrecognised key is an attempt to set an arbitrary variable in the agent (`AWS_ENDPOINT_URL`, `LD_PRELOAD`, `PATH`, …) — an injection attempt, not a forward-compatibility gap, which is why unknown keys are refused rather than filtered out. Second, **installation happens before any credential or pipeline initialisation** on the hook path: the very next step reads `GITHUB_TOKEN_SECRET_ARN` to resolve the GitHub token and `AGENT_SESSION_ROLE_ARN` to scope the task's credentials, so installing later would silently resolve the whole task against the snapshot's frozen env. The one call that must precede installation is the S3 payload fetch (the config is *inside* the fetched object), which therefore runs on the ambient compute role via the attributed platform client — and it is the ONLY one: the same rule covers **logging**, so every `/run` log line before the install is stdout-only. The CloudWatch writer would otherwise resolve credentials and pin a boto3 default session (region included) off whatever a snapshot happened to bake, which is the build-hook defect one phase later. Nothing is lost — in the intended deployment there is no baked `LOG_GROUP_NAME`, so those lines would have gone to stdout anyway, and the reason for every pre-install rejection also travels in the structured 4xx/5xx body the service surfaces. A **required subset** (task table, task-events table, GitHub token secret ARN, session-role ARN) is rejected as `…_INCOMPLETE` when missing or blank — a distinct wire code from the `…_INVALID` allowlist rejection, because the remedies differ (deployment wiring vs. producer bug). A `/run` envelope with *no* `platform_config` at all is still accepted, loudly warned: the image snapshot and the orchestrator Lambda deploy on independent cadences, and a new image must not require a same-instant orchestrator. The key set is a cross-package contract in `contracts/constants.json` (`microvm_platform_config`), consumed by the agent's `/run` hook and produced by the orchestrator, with shape and required-subset invariants enforced by `scripts/check-constants-sync.ts`. + **No orchestrator→agent HTTP path exists in P1–P3**: payload arrives through the `/run` hook, all agent work is outbound, and therefore **no JWE auth tokens are minted at all** — token minting (and its ≤ 60 min TTL refresh problem) is deferred until a real consumer exists (e.g. operator shell access, [#391](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/391)). The `endpoint` stays in the `SessionHandle` because it is genuinely per-session state that becomes load-bearing the day such a consumer appears. But note the service does not agree by default: omitting `ingressNetworkConnectors` on `RunMicrovm` attaches a **public** `HTTP_INGRESS` connector, so the strategy passes the Lambda-managed `NO_INGRESS` connector explicitly on every launch (see sub-decision 4's security table). **Constraint accepted:** the configured **baseline is 8 GiB RAM / 4 vCPU** and the service scales vertically to a **32 GiB / 16 vCPU peak** on its own, with 32 GB of disk. So capacity is baseline-priced with 4× burst headroom — good for the bursty compile-and-test shape of an agent task — but the SUSTAINED ceiling is still 32 GiB, so repos that motivated the 120 GB ECS sizing stay on `ecs`. What the construct configures (and validates) is the baseline; the peak is not something a deployment asks for. -Normative requirements (EARS). All of these are P1 now — the earlier P1/P2 split of this list existed only because the phasing table split declaring a hook from serving it, which the service does not permit: +Normative requirements (EARS). **Each requirement's own `(Pn)` tag is authoritative**; there is no blanket phase for the list. The tags are per-requirement because the original list *was* split P1/P2 on the assumption that a hook could be declared in one phase and served in a later one — which the service does not permit (see the phasing table above), so the hook-serving requirements collapsed into P1 while the P2 items below arrived with the P2 hooks and `platform_config`: - (P1) The image build shall not embed secrets, tokens, or per-task identity in the snapshot. -- (P1) If the task payload exceeds the 4 KB `runHookPayload` limit, then the strategy shall upload the payload to the platform payload bucket and pass only its S3 URI in `runHookPayload`. +- (P1) If the task payload exceeds the 4 KB `runHookPayload` limit, then the strategy shall upload the payload to the platform payload bucket and pass its S3 URI in `runHookPayload` in place of the payload (from P2, alongside `platform_config`). - (P1) The MicroVM execution role shall hold read-only access to the payload bucket, scoped to that bucket. - (P1) Where no ingress is configured for a deployment, the strategy shall pass the Lambda-managed `NO_INGRESS` network connector on every `RunMicrovm` call (the field shall not be omitted). - (P1) Where the image enables any MicroVM lifecycle hook, the image shall also enable the `/ready` hook and the agent shall serve it. @@ -177,6 +225,16 @@ Normative requirements (EARS). All of these are P1 now — the earlier P1/P2 spl - (P1) The agent shall resolve credentials at `/run` time. - (P1) Where a deployment configures a MicroVM image before smoke parity is verified, the platform shall warn that the backend has no smoke-parity guarantee. - (P2) Where the image declares the `/validate` build hook, the agent shall serve it. +- (P2) The `/validate` hook shall make no AWS API calls. +- (P2) When the `/run` hook receives `platform_config`, the agent shall install only allowlisted keys into the environment before pipeline initialization. +- (P2) If `platform_config` carries a key that is not on the allowlist, then the agent shall reject the run with a 400 and shall install none of the block's keys. +- (P2) If a required `platform_config` key is missing, then the agent shall reject the run with a 400. +- (P2) Where a `platform_config` value and an image-baked environment value disagree, the agent shall use the `platform_config` value. +- (P2) Until `platform_config` is installed, the `/run` hook shall make no AWS API call other than the payload fetch, and shall log to stdout only. +- (P2) Where the image declares the `/terminate` hook, the agent shall return 200 within the hook budget for any request body — including a malformed, empty or absent one — and shall not write terminal task status. +- (P2) When the `/ready` hook runs, the agent shall exec the agent CLI binary before returning 200, so that its pages are resident when the snapshot is captured. +- (P2) If a required `/ready` warm-up does not complete successfully, then the agent shall report not-ready (HTTP 503) rather than allow the snapshot to be taken. +- (P2) The `/ready` hook shall make no AWS API call, warm-up included. ### 4. Infra and IAM: conditional resources behind bootstrap `ComputeTypes` @@ -184,12 +242,78 @@ Mirroring the ECS pattern: a `compute-lambda-microvm` bootstrap policy (`cdk/src Two networking facts the construct has to encode, both established live: -- **A `VPC_EGRESS` connector requires an operator role.** CloudFormation's generated L1 types `operatorRole` as optional and this ADR originally assumed Lambda would manage the ENIs with its own service-linked role. It does not: the connector fails to create with *"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type"*. The construct creates one role — trusting `lambda.amazonaws.com` with `aws:SourceAccount` pinned, carrying `AWSLambdaVPCAccessExecutionRole` plus the ENI / tag / private-IP actions that policy omits — and shares it across both connectors, since it manages interfaces rather than traffic. +- **A `VPC_EGRESS` connector requires an operator role.** CloudFormation's generated L1 types `operatorRole` as optional and this ADR originally assumed Lambda would manage the ENIs with its own service-linked role. It does not: the connector fails to create with *"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type"*. The construct creates one role — trusting the bare `lambda.amazonaws.com` service principal (see the trust-policy fact below), carrying `AWSLambdaVPCAccessExecutionRole` plus the ENI / tag / private-IP actions that policy omits — and shares it across both connectors, since it manages interfaces rather than traffic. +- **The MicroVM-facing roles cannot carry a confused-deputy source condition.** All three (build, execution, connector operator) trust the bare `lambda.amazonaws.com` service principal with **no** `aws:SourceAccount` / `aws:SourceArn`, and that is a forced choice, not an oversight: the Lambda MicroVMs service presents no source key when it assumes them, so a trust policy carrying one is unassumable. Two symptoms of the one cause, both live 2026-08-06/07 and both blocking: + + - Both `AWS::Lambda::NetworkConnector` resources `CREATE_FAILED` **deterministically** (on a freshly deleted stack, so not propagation lag — which matters, because *"The service is unable to assume the provided NetworkConnectorOperatorRole. Please verify the trust policy on the role."* is also the classic propagation symptom and a re-run is the obvious wrong guess). Removing the condition → both created within a second. + - `RunMicrovm` failed with a **misleading `iam:PassRole` AccessDenied on the caller**, with the orchestrator's grant present, `simulate-principal-policy` returning `allowed`, no permissions boundary, and a temporary *unconditioned* `iam:PassRole` **also** denied. The real cause was the execution role's trust; removing its conditions made the next submission reach `RUNNING` in 6 s. So the service reports a role it cannot pass-and-assume as an identity-policy denial on the principal passing it. + + Recorded plainly because the fix looks like a regression to anyone applying the standard service-principal pattern — and because it *was* a regression in the other direction: P1's standalone-validated operator-role probe had no conditions and worked, and the P1 F2 fix then added them "to mirror the build/execution roles". `sts:TagSession` stays: the service needs both actions and it was never implicated. + +- **Neither can the `iam:PassRole` grants carry an `iam:PassedToService` condition — same root cause, identity side (P2r2-F9 + P2r2-F10, live 2026-08-07 run 2).** An earlier revision of this ADR recorded the opposite, that the identity-side condition "was exonerated" by run 1's elimination. **That was a false negative**, and its cause is worth recording because it is a general trap: run 1 tested the conditioned grant by *adding* a temporary unconditioned `iam:PassRole` and watching the task still fail — but the temporary grant remained attached through the later submissions that succeeded, so the conditioned grant was never once tested against a working trust policy. A contaminated control. + + Run 2 ran the clean experiment — same exact-ARN resource, same ~5-minute IAM settle, one variable. It removed the run-1 workaround **first** (submission 4: denied) and only then added the unconditioned grant back on the same resource (submission 5: `RUNNING`), which is the ordering run 1 got wrong: + + | Orchestrator `iam:PassRole` on the execution role | Result | + |---|---| + | exact ARN **+ `iam:PassedToService: lambda.amazonaws.com`** | **DENIED** (two independent submissions) | + | exact ARN, **no condition** | **`RUNNING` in 9 s** | + + The denial lands on the **caller**, which is what makes it so misleading — the statement names that exact ARN and `simulate-principal-policy` answers `allowed`: + + ``` + User: arn:aws:sts:::assumed-role/backgroundagent-dev-TaskOrchestratorOrchestratorFn-… + is not authorized to perform: iam:PassRole on resource: + arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-… + because no identity-based policy allows the iam:PassRole action + ``` + + And the same key blocks the *other* PassRole path, which run 1 never reached because the enum defect (P2-F2) stopped it earlier: CloudFormation could not pass the **build role** at `CreateMicrovmImage` under the bootstrap `infrastructure` policy's allowlisted `IAMPassRole`. Verbatim, so the diagnosis does not have to be taken on trust: + + ``` + LambdaMicrovmComputeImage… CREATE_FAILED + User: arn:aws:sts:::assumed-role/cdk-hnb659fds-cfn-exec-role--us-east-1/AWSCloudFormation + is not authorized to perform: iam:PassRole on resource: + arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeBuildRoleF0-… + because no identity-based policy allows the iam:PassRole action + (Service: LambdaMicrovms, Status Code: 403) + ``` + + Three pieces of evidence pin that to the *condition* rather than to a stale bootstrap or a wrong resource pattern: + + 1. the live `IaCRole-ABCA-Infrastructure` policy was byte-identical to this branch's `cdk/bootstrap/policies/infrastructure.json`, so `cdk bootstrap --force` would have changed nothing; + 2. `aws iam simulate-principal-policy --policy-source-arn --action-names iam:PassRole --resource-arns ` returned `allowed` **with** `--context-entries ContextKeyName=iam:PassedToService,ContextKeyValues=lambda.amazonaws.com,ContextKeyType=string` and `implicitDeny` with no context entry — so the resource pattern matches and the condition key is the only remaining variable; + 3. the **control**: the out-of-band `create-microvm-image` call passed the *same build role* to the *same service* successfully, using operator credentials that carry no such condition. The role's trust is therefore fine and the denial is genuinely caller-side. + + So: **the Lambda MicroVMs service presents no usable value for `iam:PassedToService` on either PassRole path** (CloudFormation → build role at `CreateMicrovmImage`; orchestrator → execution role at `RunMicrovm`), exactly as it presents no `aws:SourceAccount` on the assume-role path. One root cause, two more symptoms. Both statements therefore drop the condition, and the fix is deliberately asymmetric so it stays contained: + + - `task-orchestrator.ts` sid `MicrovmPassExecutionRole` — condition removed; the **exact execution-role ARN** is now the whole of the scoping, which is why that resource must never be relaxed to a prefix or `*`. + - a new sid `MicrovmPassRoles` in the **conditional** `compute-lambda-microvm` bootstrap policy — unconditioned `iam:PassRole` on the build- and connector-operator role **name prefixes only** (not the execution role, which CloudFormation never passes). The shared `infrastructure` `IAMPassRole` keeps its allowlist, so no other role in the stack loses that constraint, and an agentcore-only bootstrap never gains an unconditioned pass at all. **Operators must re-bootstrap** (bundle ≥ 1.4.0) for the CDK-managed image path to work. + + If AWS documents the value the service does present, adding it to both statements restores the condition. `microvms.lambda.amazonaws.com`, `lambda-microvms.amazonaws.com` and `microvms.amazonaws.com` were all `implicitDeny` against the conditioned policy, so any one of them would serve as the allowlist entry if it turns out to be right. Note that CloudTrail carries **no `lambda-microvms` management events at all** today, so the value cannot be read out of a log — only confirmed by AWS or found by a bounded sweep. + + Compensating controls, enumerated per role. The deployment-role's shared `IAMPassRole` grant is **name-prefix-scoped**, so it technically covers all three roles; in practice only the orchestrator actively invokes `iam:PassRole` on the execution role (CloudFormation never requests it). The other two roles are passed **to** the deployment role (not to themselves). + + | Role | Who can pass it, and how that grant is scoped | + |---|---| + | **Execution role** | The **orchestrator Lambda only**, at `RunMicrovm` — `iam:PassRole` scoped to this role's **exact ARN**, no condition (`constructs/task-orchestrator.ts`, sid `MicrovmPassExecutionRole`). The referenced comment contains the authoritative two-arm experiment evidence that the condition is the true blocker (not a permissions gap or stale bootstrap). | + | **Build role** | The **CloudFormation deployment role**, at `CreateMicrovmImage` (the L1's `buildRoleArn`) — via the new `MicrovmPassRoles` statement, scoped to `role/backgroundagent-dev-LambdaMicrovmComputeBuild*`, no condition. Also whoever runs `package-microvm-artifact.sh --create-image` out of band, using their own credentials. | + | **Connector operator role** | The **CloudFormation deployment role**, at `AWS::Lambda::NetworkConnector` create/update (`operatorRole`) — the same statement, scoped to `role/backgroundagent-dev-LambdaMicrovmComputeConnector*`. | + + The rest of the posture: every resource these roles can reach is account-scoped by ARN **except two deliberate `Resource: '*'` statements** — `ec2:DescribeAvailabilityZones` on the execution role (EC2 describe actions have no resource-level scoping; read-only, no mutation, no data access, needed so a CDK target repo's `cdk synth` build gate can resolve AZ context on a fresh clone) and the connector operator role's ENI/tag/private-IP statement (`CreateNetworkInterface` is authorized before the ENI exists and the `Describe*` calls take no resource, which is why the AWS-managed VPC-access policy uses `*` too). Both are justified in the construct's cdk-nag `AwsSolutions-IAM5` suppressions, which is where a reviewer should check them rather than here. The Logs grants are prefix-scoped (`/aws/lambda-microvms/*` plus one named log group), i.e. wildcards inside a namespace, not `*`. Separately, the **orchestrator's** `lambda:PassNetworkConnector` is also `Resource: '*'` and unavoidably so — the AWS-managed connectors live in the `aws` account, outside any ARN we could enumerate (justified in `task-orchestrator.ts`, sid `MicrovmPassNetworkConnector`). Finally: none of the three roles holds `iam:*`, none has cross-account trust, and the only `sts:AssumeRole` any of them has is the execution role's, scoped to the per-task SessionRole. + + If AWS later populates a source key on this path, adding it to the shared principal fixes all three roles and both `sts` actions at once. + - **Build-time egress needs port 80; runtime does not.** `agent/Dockerfile` installs Debian packages and `apt-get` fetches over plain HTTP, so a 443-only egress path fails every snapshot build (`Could not connect to deb.debian.org:80 … exit code: 100`). Rather than widen the runtime posture, the construct provisions a **second, build-only** connector on the same private subnets with a 443 + 80 security group, referenced solely by the image resource and the packaging script. The agent at run time still has 443-only egress. - Where the bootstrap `ComputeTypes` parameter includes `lambda-microvm`, the generated template shall attach the `IaCRole-ABCA-Compute-LambdaMicrovms` policy to the CloudFormation execution role. - The orchestrator role shall receive only the MicroVM lifecycle actions it calls (`lambda:RunMicrovm`, `lambda:SuspendMicrovm`, `lambda:ResumeMicrovm`, `lambda:TerminateMicrovm`, `lambda:GetMicrovm` for `pollSession`, and `lambda:PassNetworkConnector`, which is required even for the default connectors), scoped to platform-created images. - Where the `lambda-microvm` backend is enabled, the approve and deny Lambdas shall receive `lambda:ResumeMicrovm` and `lambda:GetMicrovm` — conditionally, mirroring the cancel handler's conditional `RUNTIME_ARN` wiring in `task-api.ts`. +- The trust policy of every MicroVM-facing role shall name `lambda.amazonaws.com` and shall carry no source-condition key (the service presents none; see the trust-policy fact above). +- The `iam:PassRole` grant the orchestrator uses for the MicroVM execution role shall carry no `iam:PassedToService` condition and shall be scoped to that role's exact ARN. +- Where the bootstrap `ComputeTypes` parameter includes `lambda-microvm`, the `IaCRole-ABCA-Compute-LambdaMicrovms` policy shall grant `iam:PassRole` without an `iam:PassedToService` condition, scoped to the MicroVM build- and connector-operator role name prefixes, and shall not extend that grant to the MicroVM execution role. +- The shared `IaCRole-ABCA-Infrastructure` `iam:PassRole` statement shall retain its `iam:PassedToService` allowlist. +- The MicroVM execution role shall hold `logs:CreateLogStream` and `logs:PutLogEvents` on the application log group whose name is delivered in `platform_config`, scoped to that log group. `lambda:CreateMicrovmAuthToken` is granted to no role in P1–P3 (no JWE consumer exists; see sub-decision 3). @@ -205,7 +329,10 @@ Two networking facts the construct has to encode, both established live: | Egress, image build | ECR build outside the platform VPC | ECR build outside the platform VPC | Platform VPC via a **separate build-only connector, TCP 443 + 80** (`apt-get` is plain HTTP) | New surface: build-time egress is wider than runtime egress by one port, on a connector no running MicroVM can use | | Tenant-data scoping | Per-session role (`admitComputeRole`) | Per-session role | Per-session role, execution role admitted identically | None | | Secrets delivery | Runtime env + Identity injection | Task env vars | Fetched at `/run`; never in snapshot | New surface: snapshot must stay secret-free (EARS req., sub-decision 3) | +| Non-secret platform config (table/bucket names, secret + role ARNs) | Runtime env vars | Task env vars | `platform_config` in the `/run` payload, installed into the process env | New surface: the values are attacker-relevant *as env vars* (`LD_PRELOAD`, `AWS_ENDPOINT_URL`), so the agent installs a fixed **allowlist** and rejects the whole run on any other key (EARS req., sub-decision 3) | | Inbound exposure | None (SigV4 invoke only) | None (no endpoint) | **None — but only because the strategy passes `NO_INGRESS` explicitly.** The service default is a PUBLIC `HTTP_INGRESS` connector plus a public `*.lambda-microvm..on.aws` endpoint; no tokens are minted in P1–P3 either way | New surface **and** a new failure mode: "no inbound" is an active control, not an absence. Drop the `NO_INGRESS` argument and every agent MicroVM gets a public endpoint (EARS req., sub-decision 3) | +| IAM condition keys on the compute-role trust **and** on the `iam:PassRole` grants that hand it over | Trust pinned with `aws:SourceAccount`; `PassRole` under the allowlisted bootstrap statement | Trust pinned per-service; `PassRole` under the allowlisted bootstrap statement | **Neither is possible.** All three MicroVM-facing roles trust the bare `lambda.amazonaws.com` with no `aws:SourceAccount`/`aws:SourceArn`, **and** both `iam:PassRole` grants (orchestrator → execution role at `RunMicrovm`; CloudFormation → build role at `CreateMicrovmImage`) carry no `iam:PassedToService` — the service presents no usable value for any of those keys, and each condition is a hard blocker while present (live-verified, blocking, four times across two runs) | **Real, evidenced gap that does not close from our side, and it is wider than the trust policy alone.** `lambda.amazonaws.com` is shared with every other Lambda feature, so neither the account pin nor the passed-to-service pin is available on this path. Compensated per role (table in sub-decision 4): the **execution** role is passable by the **orchestrator only** (at `RunMicrovm`), restricted to its **exact ARN**; the **build** and **connector-operator** roles are passable by the CloudFormation deployment role under a new **conditional, per-backend, name-prefix-scoped** statement (`MicrovmPassRoles`, bootstrap ≥ 1.4.0) that deliberately excludes the execution role. The shared allowlisted `IAMPassRole` (`role/backgroundagent-dev-*`) is left intact to avoid widening the grant for ~30 other roles, so while it technically matches the execution role, only the orchestrator actively reaches for it. Resources are account-scoped by ARN apart from two justified `Resource: \'*\'` statements (`ec2:DescribeAvailabilityZones`; the operator role\'s ENI management — both carry cdk-nag IAM5 suppressions). No `iam:*`, no cross-account trust. Revisit if AWS ever documents the values the service presents; CloudTrail records no `lambda-microvms` events, so they cannot be read from logs | +| Per-task observability writes | Runtime writes to the vended APPLICATION_LOGS group | Task role writes to the task log group | Execution role writes to the SAME APPLICATION_LOGS group, granted against the group `platform_config` names (P2-F4) | None — but only after P2-F4: the name was delivered a phase before the grant, so the agent attempted the write and every per-task line (and `METRICS_REPORT`) was `AccessDenied`, degrading silently to guest stdout | | Session isolation | MicroVM | Task-level | MicroVM (Firecracker) | None (≥ ECS) | | State reuse | None | None | Snapshot shared across MicroVMs | New surface: CSPRNG reseed + credential refresh on `/run`/`/resume` (EARS req.) | | Workload-token injection | Yes (Runtime-coupled) | No (env-var posture) | No (env-var posture) | Shared with ECS; deferred to [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 | @@ -230,7 +357,7 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor ### 5. Rollout: phased, default unchanged - **P1 — strategy + infra + minimal hook serving:** `LambdaMicrovmComputeStrategy` (start/poll/stop), CDK construct, bootstrap policy, types sync, unit + CDK assertion tests, and the agent's `/ready` + `/run` endpoints. No suspend yet. The image IS creatable and launchable and the payload DOES reach the agent — but there is **no smoke-parity guarantee** (sub-decision 3's phasing table). -- **P2 — smoke parity:** the agent serves the remaining hooks (`/terminate`, `/validate`); agent completes clone → change → PR on the backend with progress visible to `bgagent watch`; failure classification entries in `error-classifier.ts`; **AgentCore Memory parity** (IAM grant + `MEMORY_ID` delivery, following the `EcsAgentCluster` pattern — Memory is a standalone service already consumed cross-substrate, and omitting the grant silently no-ops cross-session learning); the agent's remaining non-secret env parity inside the snapshot. +- **P2 — smoke parity:** the agent serves `/terminate` + `/validate` and installs its platform env from the `/run` payload (see sub-decision 3's "Platform configuration delivery"); agent completes clone → change → PR on the backend with progress visible to `bgagent watch`; failure classification entries in `error-classifier.ts`; **AgentCore Memory parity** (IAM grant + `MEMORY_ID` delivery, following the `EcsAgentCluster` pattern — Memory is a standalone service already consumed cross-substrate, and omitting the grant silently no-ops cross-session learning); the agent's remaining non-secret env parity inside the snapshot. - **P3 — suspend/resume:** the interface widening from sub-decision 1 (mandatory methods, all three strategies in one commit), HITL-wait suspend policy, inline resume in the approve/deny Lambda with orchestrator-poll reconciliation (sub-decision 2), timeout-under-freeze wall-clock handling; coordinate with [#491](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/491)'s unified liveness model and update Cedar decision #7's rationale note. - **Out of scope:** replacing AgentCore as default; classic Lambda functions as a runtime; GPU; the Runtime-coupled workload-access-token injection path (delivery mechanism exists only on AgentCore Runtime; MicroVMs adopt the ECS env-var posture until [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 redesign the seam). Gateway integration is orthogonal: ADR-019/[#641](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/641) is substrate-portable by design and applies to this backend when it lands. @@ -247,8 +374,10 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor - (−) **8-hour hard cap includes suspended time**, and with `idlePolicy` omitted there is no tighter substrate-level suspended-TTL — the suspended-state bound is `maximumDurationInSeconds` plus orchestrator termination and the stranded reconciler. A manually suspended VM was observed alive at 1 h with no TTL in sight (observation truncated there), so nothing contradicts this bound, but nothing narrows it either. Under today's 1 h gate ceiling it is comfortably sufficient; any future extension of gate ceilings must revisit the bound (an additive `idlePolicy` change) and give the orchestrator a checkpoint-and-restart path (push branch, new session) beyond the cap. - (!) **Idle-policy foot-gun.** Traffic-based auto-suspend would freeze a busy outbound-only agent; the decision to disable auto-suspend must be enforced in code and covered by tests, not left to configuration discipline. - (!) **Service defaults are not the desired posture.** Two live-caught cases (public `HTTP_INGRESS` by default; `/ready` mandatory) mean an omitted field on this backend does not mean "off" — it can mean "the service picks, and it picks wider than we want". Every new `RunMicrovm` / `CreateMicrovmImage` field should be assumed to have an opinionated default until checked. -- (!) **Nothing self-terminates.** A MicroVM whose hook never ran still reaches `RUNNING` and stays there, billing, until the 8 h cap. The orchestrator's `TerminateMicrovm` on finalize is the only cleanup, so a leaked handle is a cost incident, not just an untidy state. +- (!) **Nothing self-terminates on the paths that matter** — superseding P1's unqualified version of this bullet. With `run: ENABLED` the service DOES reap a VM whose run hook returns 4xx (~12 s, `stateReason: "Run lifecycle hook returned HTTP status 400."`, live-verified), so a guest that rejects its own payload cleans itself up. That is the only self-cleaning case: the service reaps a hook *result*, and once `/run` has answered 200 it has no view of the guest. A VM whose task finished, crashed after `/run`, or hung stays `RUNNING` and billing until the 8 h cap, so the orchestrator's `TerminateMicrovm` on finalize remains the only cleanup for normal operation and a leaked handle is still a cost incident. - (!) **Snapshot uniqueness.** Shared memory snapshots require CSPRNG reseeding and credential refresh in `/run` / `/resume` hooks; missing this is a silent security defect. +- (−) **The agent stack template is at 98.6 % of CloudFormation's 1 MB limit** (985,886 bytes) and 486 of 500 resources with a MicroVM image configured — ~14 KB of headroom, i.e. roughly one more construct, and down from 98.4 % / ~16 KB one run earlier. Not caused by this backend (the MicroVM construct is ~6 KB of it) but reached by it, and it will block deploys for reasons that have nothing to do with MicroVMs. Tracked in [#735](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/735); the candidate remedies are `suppressTemplateIndentation` and a stack split. +- (!) **Snapshot WARMTH is a first-class property, not an optimisation.** A snapshot inherits only the pages something touched before it was captured, so a large lazily-loaded artifact — the 225 MiB `claude` binary, and anything similar added later — pays its first-touch cost on the *first task* instead of at container start. That cost failed every task at turn 0 in the P2 smoke run (P2-F5). Anything heavyweight added to the image must be exec'd in `/ready`, and any timeout guarding a first touch must be sized for a cold page fault rather than for the work itself. - (!) **Regional availability (5 regions at launch, expanding)** — enforced in layers (synth-time static check, onboarding + doctor live probes, orchestration-time classification; see sub-decision 4). The static CDK constant is the one piece that rots as AWS expands; its update path and context-flag escape hatch are deliberate. - (!) **Workload-token injection delta persists** (shared with the ECS backend) until [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 land; document it in the security bar comparison rather than blocking on it. Memory and Gateway are explicitly *not* deltas — both are standalone services consumed via IAM from any substrate. @@ -257,15 +386,21 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor **P1 (start/poll/stop — no suspend):** - Unit tests for the strategy: start/poll/stop mapping (including `SessionStatus` `'suspended'` reported mechanically, without task-state interpretation), payload-size branching (inline vs S3 pointer, at the exact 4 096/4 097-byte boundary), the image-identifier-must-be-an-ARN guard, the explicit `NO_INGRESS` argument (including the blank-env-var fallback, which must never omit the field), error classification (`ServiceQuotaExceededException`, `ThrottlingException`, `ResourceNotFoundException`, regional-unavailability), the omit-`idlePolicy` invariant, and `maximumDurationInSeconds` fixed at 28 800. -- Agent tests: `/ready` returns 200 once the server is up and starts nothing; `/run` accepts both envelope shapes (inline and S3 pointer), starts the pipeline asynchronously through the same mapper `/invocations` uses, returns before the pipeline finishes, and rejects every unusable envelope with a named code before spawning; `/validate`, `/suspend`, `/resume` and `/terminate` are NOT served. +- Agent tests: `/ready` returns 200 once the server is up and starts nothing; `/run` accepts both envelope shapes (inline and S3 pointer), starts the pipeline asynchronously through the same mapper `/invocations` uses, returns before the pipeline finishes, and rejects every unusable envelope with a named code before spawning; `/suspend` and `/resume` are NOT served (`/validate` and `/terminate` joined the served set in P2). - Orchestrator tests: substrate-terminal + non-terminal task status → failed classification; `suspended` + non-`AWAITING_APPROVAL` status → anomaly event, no fail-fast; `compute_metadata` persisted with `microvmId`/`endpoint` after `startSession`. - CDK assertions: MicroVM resources present only when `ComputeTypes` includes the backend; synth failure for unsupported regions (plus the context-flag escape hatch); memory size validated against the accepted list at synth; the connector operator role and its trust; two connectors with the build-only one carrying port 80 and the runtime one not; the `/ready` + `/run` hook declaration and the absence of the others; IAM actions scoped as specified (orchestrator lifecycle set; no `CreateMicrovmAuthToken` anywhere); payload-bucket grants (execution role read-only); backend cost-allocation tags; types-sync check covers the widened `ComputeType`. - CLI tests: onboarding rejection with remedy when the availability probe fails; doctor check present when a blueprint selects the backend. -- P1 verification items (external service facts) — **executed 2026-07-31, us-east-1**; see `docs/verification/645-p1-lambda-microvm-runbook.md` for the full evidence. Discharged: `runHookPayload` limit (**4 096**, not 16 KB), the accepted baseline memory sizes (`[512…8192]` MiB — note the developer guide, not the probe, is what establishes that this is a BASELINE with a 32 GiB peak), image-identifier ARN requirement, IAM action names and the observed image-ARN shape, region probe behaviour, manual suspend/resume without `idlePolicy`, terminate timing and the `TERMINATED`-persists-≥10-min finding, the default public `HTTP_INGRESS`, and the `/ready` requirement. **Not** discharged: account-quota treatment of `SUSPENDED` MicroVMs (not observable safely), suspended TTL beyond 1 h (truncated), the vertical-scaling behaviour itself (no workload here approached the baseline, so the 4× peak is documented rather than observed), and the `AWS::Lambda::MicrovmImage` CloudFormation value shapes (never exercised — the run used the out-of-band script path). Record the closed answers in COMPUTE.md. +- P1 verification items (external service facts) — **executed 2026-07-31, us-east-1**; see `docs/verification/645-p1-lambda-microvm-runbook.md` for the full evidence. Discharged: `runHookPayload` limit (**4 096**, not 16 KB), the accepted baseline memory sizes (`[512…8192]` MiB — note the developer guide, not the probe, is what establishes that this is a BASELINE with a 32 GiB peak), image-identifier ARN requirement, IAM action names and the observed image-ARN shape, region probe behaviour, manual suspend/resume without `idlePolicy`, terminate timing and the `TERMINATED`-persists-≥10-min finding, the default public `HTTP_INGRESS`, and the `/ready` requirement. **Not** discharged: account-quota treatment of `SUSPENDED` MicroVMs (not observable safely), suspended TTL beyond 1 h (truncated), the vertical-scaling behaviour itself (no workload here approached the baseline, so the 4× peak is documented rather than observed), and the `AWS::Lambda::MicrovmImage` CloudFormation value shapes (never exercised — the run used the out-of-band script path; **discharged, and REFUTED, by the P2 run — see P2-F2 in sub-decision 3**). Record the closed answers in COMPUTE.md. **P2 (smoke parity):** -- Smoke (gated like the ECS backend): clone → change → PR with `bgagent watch` progress; Memory write parity (no AccessDenied no-op). +- Agent tests: `/validate` returns 200 with its individual check results, 503 while initialising, reports a missing hook route / unsupported interpreter, starts nothing, and makes **zero AWS calls even with `LOG_GROUP_NAME` set** (asserted by poisoning the boto3 and CloudWatch-writer seams — the same assertion covers `/ready`); `/terminate` returns 200 with no body at all, with a malformed / non-object / wrong-content-type / whitespace-only body, when the body read itself fails, with a pipeline still running (without joining it), and when its own best-effort step raises — and never calls `task_state.write_terminal`; a structural assertion that the route carries no typed body param keeps the 422 from being reintroduced. +- `platform_config` tests: the allowlist and required subset are read from `contracts/constants.json` (the wire key set is additionally asserted literally, as the agent-side tripwire on a contract edit); an unknown key rejects the whole block with nothing installed; a non-object block and a non-string value are rejected; a blank/`null` optional value is skipped without clobbering an image value while a blank required value is rejected; a payload value beats a pre-existing env value; installation is observed to happen before the GitHub-token resolver runs and before any pipeline thread exists; the config is picked up from the inline envelope, from beside the S3 pointer, and from inside the fetched object (inner wins); an envelope with no `platform_config` is still accepted with a warning. +- Snapshot credential hygiene: a subprocess probe asserts that importing `server` and serving `/ready` + `/validate` imports neither `boto3` nor `botocore`, caches no `aws_session` session, and spawns no CloudWatch writer thread — the property that keeps a build-role credential chain and the build-time region out of the snapshot. +- `/run` pre-install silence: with a **baked `LOG_GROUP_NAME`** (the hostile case — without it the assertions pass vacuously) every AWS/credential seam (`boto3.client`/`Session`, the `aws_session` factories, `_debug_cw`/`_warn_cw`) is armed to raise until the install succeeds. Asserted on the accepted path, on all three rejection paths (bad envelope, `platform_config` invalid, `platform_config` incomplete) and on the failed-fetch 500 — where the seams stay armed for the whole request, because a rejected run installed nothing and so earns no AWS call. The permitted exception is asserted POSITIVELY: exactly one client is built pre-install, for `s3`, through the attributed factory. +- `/ready` warm-up tests (P2-F5): the hook exec's each configured binary exactly once with a generous timeout; `claude` is the only REQUIRED entry; a timeout, a missing binary, a non-zero exit and an unexpected `OSError` each produce **503 with the reason logged to stdout** rather than a 200 or a 500; a best-effort failure still reports ready; the warm-up makes zero AWS calls with `LOG_GROUP_NAME` baked. Plus the backstop half: the `claude --version` probe's bound is asserted to be ≥ 60 s and to be applied to the *exec* rather than to the PATH lookup, and a missing CLI warns instead of raising. +- CDK assertions (P2-F1/F2/F4): no source-condition key on any of the three MicroVM-facing role trusts, and no `aws:SourceAccount`/`aws:SourceArn` string anywhere in them; hook properties are `ENABLED` and the architecture is `ARM_64`, with a negative assertion that **no** hook route string appears anywhere in the rendered image resource; the agent hook routes are asserted against their own dedicated constant (the template no longer carries a path to compare); the execution role holds `logs:CreateLogStream`/`PutLogEvents` on the application log group and the two logs grants stay separate; the stack wires the SAME log group it delivers as `platform_config.log_group_name`. +- Smoke (gated like the ECS backend): clone → change → PR with `bgagent watch` progress; Memory write parity (no AccessDenied no-op). **Run 1 (2026-08-06) FAILED at `implement`, turn 0 — no PR. Run 2 (2026-08-07) PASSED: two tasks clone → change → commit → push → PR, `COMPLETED`, 12 turns / $0.279 / 153 s** (`docs/verification/645-p2-smoke-runbook.md`), which also discharged P2-F1, P2-F2, P2-F4, P2-F5 and the dual-signal-liveness item (45 s heartbeat cadence observed across a 181 s `RUNNING` window). **The row is not yet fully closed:** run 2 needed one live IAM workaround, and establishing why produced P2r2-F10 (the identity-side `iam:PassedToService`) and P2r2-F9 (its CloudFormation twin). Both are fixed in source above and neither has been re-exercised live, so what remains is a re-run on a re-bootstrapped account with no workarounds. **P3 (suspend/resume):** diff --git a/docs/design/COMPUTE.md b/docs/design/COMPUTE.md index 66f7eafb3..48670a6df 100644 --- a/docs/design/COMPUTE.md +++ b/docs/design/COMPUTE.md @@ -1,6 +1,6 @@ # Compute -Every task runs in an isolated cloud compute environment. Nothing runs on the user's machine. The agent clones the repo, writes code, runs tests, and opens a PR inside a MicroVM that is created for the task and destroyed when it ends. +Every task runs in an isolated cloud compute environment. Nothing runs on the user's machine. The agent clones the repo, writes code, runs tests, and opens a PR inside a compute session that is created for the task and destroyed when it ends. - **Use this doc for:** understanding the compute environment, agent harness, network architecture, and the constraints that shape the platform's design. - **Related docs:** [ORCHESTRATOR.md](./ORCHESTRATOR.md) for session management and liveness monitoring, [SECURITY.md](./SECURITY.md) for isolation and egress controls, [REPO_ONBOARDING.md](./REPO_ONBOARDING.md) for per-repo compute configuration. @@ -9,17 +9,19 @@ Every task runs in an isolated cloud compute environment. Nothing runs on the us The default runtime is **Amazon Bedrock AgentCore Runtime**, which runs each session in a Firecracker MicroVM with per-session isolation, managed lifecycle, and built-in health monitoring. For repos that exceed AgentCore's constraints (2 GB image limit, no GPU), the `ComputeStrategy` interface allows switching to alternative backends per repo. -| | AgentCore Runtime | ECS on Fargate | ECS on EC2 | EKS | AWS Batch | Lambda | Custom EC2 + Firecracker | -|---|---|---|---|---|---|---|---| -| **Isolation** | MicroVM (Firecracker) | Task-level (Firecracker) | Container on shared nodes | Pod on shared nodes | Backend-dependent | Function env (Firecracker) | MicroVM (you own it) | -| **Image limit** | 2 GB (non-adjustable) | No hard cap | No hard cap | No hard cap | Backend-dependent | 10 GB | N/A (you define) | -| **Filesystem** | Ephemeral + persistent mount (preview) | 20-200 GB ephemeral | Node disk + EBS/EFS | Node disk + PVs | Backend-dependent | 512 MB-10 GB `/tmp` | You choose (EBS/NVMe) | -| **Max duration** | 8 hours | No hard cap | No hard cap | No hard cap | Configurable | **15 minutes** | Unlimited | -| **Startup** | Service-managed | Slim images help | Warm ASGs + pre-pull | Karpenter + pre-pull | Backend-dependent | Provisioned concurrency | Snapshot pools (DIY) | -| **GPU** | No | No | Yes | Yes | Yes (EC2/EKS backend) | No | Yes (with passthrough) | -| **Ops burden** | Low (managed) | Low | Medium | High | Low-Medium | Low | **Very high** | -| **Cost model** | vCPU-hrs + GB-hrs | vCPU + mem/sec | EC2 + EBS | EKS control + EC2 | Underlying compute | Request + duration | EC2 metal + your ops | -| **Fit** | **Default choice** | Repos > 2 GB image | GPU, heavy toolchains | Max flexibility | Queued batch jobs | **Poor** (15 min cap) | Best potential, highest cost | +| | AgentCore Runtime | ECS on Fargate | **Lambda MicroVMs** | ECS on EC2 | EKS | AWS Batch | Lambda (functions) | Custom EC2 + Firecracker | +|---|---|---|---|---|---|---|---|---| +| **Isolation** | MicroVM (Firecracker) | Task-level (Firecracker) | MicroVM (Firecracker) | Container on shared nodes | Pod on shared nodes | Backend-dependent | Function env (Firecracker) | MicroVM (you own it) | +| **Image limit** | 2 GB (non-adjustable) | No hard cap | Zip + Dockerfile → snapshot; snapshot build size and OCI image size are different measures | No hard cap | No hard cap | Backend-dependent | 10 GB | N/A (you define) | +| **Filesystem** | Ephemeral + persistent mount (preview) | 20-200 GB ephemeral | 32 GB native disk in snapshot; survives suspend/resume; `flock()` works | Node disk + EBS/EFS | Node disk + PVs | Backend-dependent | 512 MB-10 GB `/tmp` | You choose (EBS/NVMe) | +| **Max duration** | 8 hours | No hard cap | 8 hours (running + suspended; 28,800s) | No hard cap | No hard cap | Configurable | **15 minutes** | Unlimited | +| **Startup** | Service-managed | Slim images help | Snapshot resume | Warm ASGs + pre-pull | Karpenter + pre-pull | Backend-dependent | Provisioned concurrency | Snapshot pools (DIY) | +| **GPU** | No | No | No | Yes | Yes | Yes (EC2/EKS backend) | No | Yes (with passthrough) | +| **Ops burden** | Low (managed) | Low | Low (managed) | Medium | High | Low-Medium | Low | **Very high** | +| **Cost model** | vCPU-hrs + GB-hrs | vCPU + mem/sec | Baseline-priced (8 GiB / 4 vCPU) with 4× vertical burst (32 GiB / 16 vCPU peak); suspended time is storage-only | EC2 + EBS | EKS control + EC2 | Underlying compute | Request + duration | EC2 metal + your ops | +| **Fit** | **Default choice** | Repos > 2 GB image | Suspend/resume economics; approval-wait-heavy workloads; default-sized repos. Heavy sustained-memory builds stay on ECS | GPU, heavy toolchains | Max flexibility | Queued batch jobs | **Poor** (15 min cap) | Best potential, highest cost | + +> **Lambda MicroVMs are not Lambda functions.** They are a different compute primitive, so the functions column's 15-minute cap and poor-fit verdict do not apply. See [ADR-021](../decisions/ADR-021-lambda-microvms-compute-backend.md). The backend is selected per repo via `compute_type` in the Blueprint config. The orchestrator resolves the strategy and delegates session start, polling, and termination to the strategy implementation. See [REPO_ONBOARDING.md](./REPO_ONBOARDING.md) for the `ComputeStrategy` interface. @@ -73,6 +75,14 @@ The platform works around this by splitting storage: See [ORCHESTRATOR.md](./ORCHESTRATOR.md) for how the orchestrator handles these timeouts. +## Lambda MicroVMs backend + +Lambda MicroVMs are an opt-in third backend, selected per repository with `compute_type: lambda-microvm`; AgentCore remains the default. Image configuration has three states: a managed base-image ARN and version creates the snapshot image in CDK; an external image identifier uses a snapshot built out of band; and supplying neither provisions only the roles, buckets, and connectors needed for the bootstrap deploy. `cdk/scripts/package-microvm-artifact.sh` packages the agent as zip + Dockerfile, uploads it to the artifact bucket, and can create the external image. Lambda MicroVMs are available in five launch regions (us-east-1, us-east-2, us-west-2, eu-west-1, ap-northeast-1) and will expand; the platform enforces regional availability in layers via a synth-time constant, onboarding live probes, and orchestration-time classification. + +Because a snapshot freezes its build-time environment, deployment-specific, non-secret identifiers travel in the `/run` hook's `platform_config` block instead. The strategy sends the canonical inline envelope or, when that envelope exceeds the verified 4,096-byte `runHookPayload` limit, an S3-pointer envelope with the configuration also merged into the uploaded payload. The agent accepts only allowlisted keys and installs them before pipeline initialization; [ADR-021 §3](../decisions/ADR-021-lambda-microvms-compute-backend.md#3-packaging-same-agent-image-source-new-build-path) defines the exact wire shapes and validation rules. + +Networking separates image build from execution: the build-only connector permits TCP 80 and 443 because the Dockerfile uses `apt-get`, while running MicroVMs retain 443-only egress through the platform VPC. Every launch explicitly passes the Lambda-managed `NO_INGRESS` connector; omission would select the service's public-ingress default. The P2 image declares and serves `/ready` and `/validate` at build time and `/run` and `/terminate` at runtime. `/suspend` and `/resume` remain disabled until their P3 implementation. + ## ECS Fargate task sizing (build vs. planning) When a repo is `compute_type: ecs`, `EcsAgentCluster` provisions **two** Fargate task definitions, and the orchestrator picks between them per task by whether the resolved workflow is **read-only**: @@ -98,14 +108,14 @@ The platform uses the [Claude Agent SDK](https://github.com/anthropics/claude-ag **System prompt:** Selected by workflow from a shared base template (`agent/src/prompts/base.py`) with per-workflow sections (`coding/new-task-v1`, `coding/pr-iteration-v1`, `coding/pr-review-v1`). The platform defines what the agent should do; the harness executes it. -**Result contract:** The agent does not call back to the platform. It follows the contract (push work, create PR) and exits. The orchestrator infers the outcome from GitHub state and the agent's poll response. When the agent is stopped by an *environmental* fault (missing secret, egress denial, unreachable dependency, fail-closed policy-engine error), it emits a typed `agent_blocked` event and carries a canonical `BLOCKED[]: …` reason in its terminal error so the orchestrator's classifier attaches a precise remedy — see [Cedar HITL gates §13.16](./CEDAR_HITL_GATES.md#1316-observable-blocker-signal-251). +**Result contract:** The agent follows the contract (push work, create PR), writes task state, and exits. The orchestrator infers the outcome from task state, backend liveness signals, and GitHub state. When the agent is stopped by an *environmental* fault (missing secret, egress denial, unreachable dependency, fail-closed policy-engine error), it emits a typed `agent_blocked` event and carries a canonical `BLOCKED[]: …` reason in its terminal error so the orchestrator's classifier attaches a precise remedy — see [Cedar HITL gates §13.16](./CEDAR_HITL_GATES.md#1316-observable-blocker-signal-251). ### Tool set | Tool | Source | Description | |------|--------|-------------| -| Shell execution | Native (MicroVM) | Build, test, lint via bash | -| File system | Native (MicroVM) | Read/write code | +| Shell execution | Native (compute session) | Build, test, lint via bash | +| File system | Native (compute session) | Read/write code | | GitHub | AgentCore Gateway + Identity | Clone, push, PR, issues | | Web search | AgentCore Gateway | Documentation lookups | @@ -128,7 +138,7 @@ The agent runtime runs inside a VPC with private subnets. AWS service traffic st flowchart TB subgraph VPC["VPC (10.0.0.0/16)"] subgraph Private["Private Subnets"] - RT[AgentCore Runtime] + RT[Compute session] end subgraph Public["Public Subnets"] NAT[NAT Gateway] diff --git a/docs/design/DEPLOYMENT_ROLES.md b/docs/design/DEPLOYMENT_ROLES.md index 8a279edc8..996a3acf8 100644 --- a/docs/design/DEPLOYMENT_ROLES.md +++ b/docs/design/DEPLOYMENT_ROLES.md @@ -743,6 +743,10 @@ When the ECS Fargate compute backend is enabled (set the `ComputeTypes` CFN para When the Lambda MicroVM compute backend is enabled (include `lambda-microvm` in the `ComputeTypes` CFN parameter on the `CDKToolkit` stack), the generated template conditionally attaches this policy to the CloudFormation execution role. It permits CloudFormation to manage MicroVM images and network connectors; runtime session lifecycle permissions remain on the orchestrator role. +The second statement, `MicrovmPassRoles`, is the one exception to the rule that every `iam:PassRole` in this bundle carries an `iam:PassedToService` condition (`IaCRole-ABCA-Infrastructure` → `IAMPassRole`). It has to be: the Lambda MicroVMs service does not present a usable value for that key, so the conditioned statement is **denied** when CloudFormation passes the build role to `CreateMicrovmImage` — live-verified in `us-east-1` (ADR-021 P2r2-F9), with the out-of-band `create-microvm-image` call passing the *same* role successfully as the control. It is deliberately scoped to the two role-name prefixes CloudFormation actually passes (the image build role and the network-connector operator role) and excludes the MicroVM **execution** role, which only the orchestrator passes, at `RunMicrovm`. The shared allowlisted `IAMPassRole` statement (`role/backgroundagent-dev-*`) is left intact to avoid widening the grant for ~30 other roles in the stack, so while it technically matches the execution role, only the orchestrator actively reaches for it. + +> **Operators must re-bootstrap for this.** The statement ships in bootstrap policy bundle **1.4.0**; a CDKToolkit stack bootstrapped at 1.3.0 or earlier will fail the CDK-managed MicroVM image deploy with a caller-side `iam:PassRole` AccessDenied on the build role. Check `CDKToolkit`'s `BootstrapPolicyVersion` output, and re-run `mise //cdk:bootstrap` (with `ComputeTypes` including `lambda-microvm`) if it is behind. + ```json { "Statement": [ @@ -771,6 +775,15 @@ When the Lambda MicroVM compute backend is enabled (include `lambda-microvm` in "Effect": "Allow", "Resource": "*", "Sid": "LambdaMicrovms" + }, + { + "Action": "iam:PassRole", + "Effect": "Allow", + "Resource": [ + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*", + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*" + ], + "Sid": "MicrovmPassRoles" } ], "Version": "2012-10-17" diff --git a/docs/design/ORCHESTRATOR.md b/docs/design/ORCHESTRATOR.md index a3c87584f..188eea2bd 100644 --- a/docs/design/ORCHESTRATOR.md +++ b/docs/design/ORCHESTRATOR.md @@ -28,7 +28,7 @@ The orchestrator is deliberately scoped. It handles coordination and bookkeeping | Task lifecycle | Accept tasks, drive them through the state machine to a terminal state, persist state at each transition | | Admission control | Validate repo onboarding, concurrency limits, rate limits, idempotency | | Context hydration | Assemble the agent prompt from user input, GitHub data, memory, and repo config | -| Session management | Start the compute session, monitor liveness via heartbeat, detect completion | +| Session management | Start the compute session, monitor backend liveness and heartbeats where applicable, detect completion | | Result inference | Determine success or failure from agent response, DynamoDB record, and GitHub state | | Finalization | Update status, emit events, release concurrency, persist audit records | | Cancellation | Stop the session and drive the task to CANCELLED at any point | @@ -40,7 +40,7 @@ The orchestrator is deliberately scoped. It handles coordination and bookkeeping |---|---|---| | Request authentication | Input gateway | [INPUT_GATEWAY.md](./INPUT_GATEWAY.md) | | Agent logic (clone, code, test, PR) | Agent runtime | [COMPUTE.md](./COMPUTE.md) | -| Compute session lifecycle (VM, image pull) | AgentCore Runtime | [COMPUTE.md](./COMPUTE.md) | +| Compute substrate internals (VM/task provisioning, image pull) | Selected compute service | [COMPUTE.md](./COMPUTE.md) | | Memory storage and retrieval | AgentCore Memory | [MEMORY.md](./MEMORY.md) | | Repository onboarding | Blueprint construct | [REPO_ONBOARDING.md](./REPO_ONBOARDING.md) | @@ -116,13 +116,13 @@ stateDiagram-v2 | `QUEUED` | `SUBMITTED` | Queue pickup | Admission-queue pickup Lambda sees free capacity; re-invokes the orchestrator. FIFO by `created_at`. | | `QUEUED` | `CANCELLED` | User cancels | Explicit cancel removes the task from the queue | | `QUEUED` | `FAILED` | Queue-age backstop | Task waited longer than `QUEUE_MAX_AGE_SECONDS` (default 24h) without admission | -| `HYDRATING` | `RUNNING` | Hydration complete | `invoke_agent_runtime` returns session ID | +| `HYDRATING` | `RUNNING` | Hydration complete | Selected `ComputeStrategy.startSession` returns a session handle | | `HYDRATING` | `AWAITING_APPROVAL` | Cedar soft-deny gate fires | Tool call triggers a soft-deny policy rule during hydration | | `HYDRATING` | `FAILED` | Hydration error | GitHub API failure, guardrail blocks content, Bedrock unavailable | | `RUNNING` | `AWAITING_APPROVAL` | Cedar soft-deny gate fires | Tool call triggers a soft-deny policy rule during execution | | `RUNNING` | `FINALIZING` | Session ends | Response received or session terminated | -| `RUNNING` | `TIMED_OUT` | Max duration exceeded | AgentCore terminates the session at its 8h cap; the orchestrator's own safety-net poll window is `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h, after which a still-`RUNNING` task is driven to `TIMED_OUT` | -| `RUNNING` | `FAILED` | Session crash | Heartbeat lost (see Liveness monitoring) | +| `RUNNING` | `TIMED_OUT` | Max duration exceeded | AgentCore and Lambda MicroVMs have an 8h substrate cap; the orchestrator's own safety-net poll window is `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h, after which a still-`RUNNING` task is driven to `TIMED_OUT` | +| `RUNNING` | `FAILED` | Session crash | Heartbeat or substrate liveness lost (see Liveness monitoring) | | `AWAITING_APPROVAL` | `RUNNING` | Approved or denied | Human decision received; agent resumes | | `AWAITING_APPROVAL` | `CANCELLED` | User cancels | Explicit cancel while awaiting approval | | `AWAITING_APPROVAL` | `FAILED` | Stranded reconciler | Approval request orphaned (agent died mid-wait) | @@ -139,19 +139,21 @@ Users can cancel a task at any point. The orchestrator's response depends on how | `QUEUED` | Transition to `CANCELLED`. Removes the task from the admission queue. No compute or concurrency slot to release (a queued task never held one). | | `SUBMITTED` | Transition to `CANCELLED`. No cleanup needed. | | `HYDRATING` | Abort hydration, release concurrency slot, transition to `CANCELLED`. | -| `RUNNING` | Call `stop_runtime_session`, wait for confirmation, release concurrency, transition to `CANCELLED`. Partial work on GitHub remains for the user to inspect. | -| `AWAITING_APPROVAL` | Call `stop_runtime_session`, release concurrency slot, transition to `CANCELLED`. The pending approval row transitions to `STRANDED`. | +| `RUNNING` | Transition to `CANCELLED` and stop the selected compute session best-effort. Partial work on GitHub remains for the user to inspect. | +| `AWAITING_APPROVAL` | Transition to `CANCELLED`. The pending approval row transitions to `STRANDED`. | | `FINALIZING` | Let finalization complete. Mark `CANCELLED` only if the terminal state was not yet written. | | Terminal | Reject the cancel request. | +For a running cancellation, backend dispatch calls ECS `StopTask`, Lambda `TerminateMicrovm`, or AgentCore `StopRuntimeSession`. The `lambda-microvm` branch is evaluated before the AgentCore `RUNTIME_ARN` fallback; otherwise a mixed deployment could stop an unrelated AgentCore session and leave the MicroVM billing until its cap. + ### Timeouts -Multiple timeout mechanisms work together to prevent runaway tasks. Time-based limits (session duration, idle) are enforced by AgentCore; cost-based limits (turns, budget) are enforced by the agent SDK. The orchestrator acts as a safety net when external timeouts fire. +Multiple timeout mechanisms work together to prevent runaway tasks. Substrate time limits vary by backend; cost-based limits (turns, budget) are enforced by the agent SDK. The orchestrator acts as a safety net when external timeouts fire. | Type | Default | Effect | |---|---|---| -| Max session duration | 8 hours | AgentCore terminates the session at its 8h cap. The orchestrator's safety-net poll loop runs up to `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h; a task still `RUNNING` when that window is exhausted is driven to `TIMED_OUT`. | -| Idle timeout | 15 minutes | AgentCore terminates if agent is idle. See Liveness monitoring. | +| Max session duration | 8 hours | AgentCore caps a session at 8h; Lambda MicroVMs use `maximumDurationInSeconds: 28,800`, including suspended time. The orchestrator's safety-net poll loop runs up to `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h; a task still `RUNNING` when that window is exhausted is driven to `TIMED_OUT`. | +| Idle timeout | Backend-specific | AgentCore has an idle timeout. Lambda MicroVMs omit `idlePolicy` because inbound-traffic idleness would suspend an outbound-only agent while it is working. See Liveness monitoring. | | Max turns | 100 (range 1-500) | Agent stops after N model invocations. Configurable per task or per repo. | | Max cost budget | $0.01-$100 | Agent stops when budget is reached. Per-task or per-repo via Blueprint. | | Hydration timeout | 2 minutes | Fail the task if context assembly takes too long. | @@ -177,7 +179,7 @@ Validates the task before any compute is consumed. Checks run in order: 1. **Repo onboarding** - `GetItem` on `RepoTable`. If not found or inactive, reject with `REPO_NOT_ONBOARDED`. This runs at the API handler level (`createTaskCore`) for fast rejection. 2. **User concurrency** - Atomic check-and-increment on `UserConcurrency` counter. If at limit (default 10), the task is **queued, not failed** (#441): it transitions `SUBMITTED → QUEUED` and a scheduled admission-queue pickup Lambda re-attempts admission in FIFO order (by `created_at`) as slots free up, flipping `QUEUED → SUBMITTED` and re-invoking the orchestrator. The pickup Lambda does a read-only capacity pre-check; the orchestrator's atomic increment remains the single writer of the counter, so a pickup that loses the race harmlessly re-queues without losing FIFO position. `GET /tasks/{id}` surfaces `queue_position` and `estimated_wait_s` while queued. -3. **System concurrency** - Compare total running + hydrating tasks to system limit (bounded by AgentCore quotas). +3. **System concurrency** - Compare total running + hydrating tasks to the configured system limit and selected-backend quotas. 4. **Rate limiting** - Sliding window counter (10 tasks/hour per user). Rate-limit rejections happen at submit time and are rejected, not queued (unlike the concurrency cap, which queues). 5. **Idempotency** - If the request includes an idempotency key and a task with that key exists, return the existing task. @@ -185,7 +187,7 @@ On acceptance, the concurrency slot is acquired and the orchestrator proceeds to ### Step 2: Pre-flight checks -Runs as a distinct top-level step (`pre-flight` in `orchestrate-task.ts`, via `runPreflightChecks`) **after** admission and **before** hydration, so external-dependency failures are caught before any prompt assembly or Bedrock screening consumes work. It verifies the GitHub token has sufficient permissions for the task type, catches inaccessible or closed PRs, and confirms GitHub API reachability. On failure it drives the task to `FAILED` and emits a `preflight_failed` event, surfacing clear errors like `INSUFFICIENT_GITHUB_REPO_PERMISSIONS` before AgentCore runtime is consumed. +Runs as a distinct top-level step (`pre-flight` in `orchestrate-task.ts`, via `runPreflightChecks`) **after** admission and **before** hydration, so external-dependency failures are caught before any prompt assembly or Bedrock screening consumes work. It verifies the GitHub token has sufficient permissions for the task type, catches inaccessible or closed PRs, and confirms GitHub API reachability. On failure it drives the task to `FAILED` and emits a `preflight_failed` event, surfacing clear errors like `INSUFFICIENT_GITHUB_REPO_PERMISSIONS` before a compute session is consumed. ### Step 3: Context hydration @@ -199,19 +201,23 @@ Regardless of workflow, the assembled prompt is screened through Amazon Bedrock ### Step 4: Session start -The orchestrator calls `invoke_agent_runtime` with the hydrated payload. The agent receives it, starts the coding task in a background thread (via `add_async_task`), and returns an acknowledgment immediately. The orchestrator records the `(task_id, session_id)` mapping and transitions to `RUNNING`. +The orchestrator resolves the repository's `ComputeStrategy` and calls `startSession` with the hydrated payload. AgentCore invokes the runtime, ECS starts a Fargate task, and Lambda MicroVMs launch a snapshot and deliver the payload through `/run`. The orchestrator persists the returned backend handle in `compute_metadata`, records the `(task_id, session_id)` mapping, and transitions to `RUNNING`. + +AgentCore's session ID is pre-generated and reused on retry. ECS and Lambda MicroVMs use their substrate identifiers as session IDs. -The session ID is pre-generated and reused on retry, making session start idempotent after a crash. +If `RunMicrovm` succeeds but persisting the session handle or emitting the start event fails, the start step terminates the MicroVM best-effort using its in-memory handle before propagating the original error. This orphan reap is required because no later poll or finalization step can recover an unpersisted handle. ### Step 5: Await completion -The orchestrator polls for completion using `waitForCondition` from the Durable Execution SDK. At configurable intervals (default 30s), it re-invokes on the same session (sticky routing). The agent responds with its current status: +The orchestrator polls for completion using `waitForCondition` from the Durable Execution SDK at configurable intervals (default 30s). DynamoDB task status is common to all backends; backend-specific checks supplement it: -- `running` - Orchestrator suspends until next interval (no compute charges) -- `completed` - Orchestrator resumes to finalization with the result -- `failed` - Same, with error payload +| Backend | Additional poll signal | +|---|---| +| AgentCore | Agent heartbeat; `/ping` keeps the runtime healthy while the task thread works | +| ECS | `DescribeTasks`, including container exit status and exit code | +| Lambda MicroVMs | `GetMicrovm` state plus agent heartbeat | -If the session is terminated externally (crash, timeout, cancellation), the poll detects it and the orchestrator proceeds to finalization using GitHub-based result inference as fallback. +While waiting between polls, the durable orchestrator suspends without compute charges. If the session is terminated externally (crash, timeout, cancellation), the poll detects it and the orchestrator proceeds to finalization using GitHub-based result inference as fallback. ### Step 6: Finalization @@ -236,13 +242,13 @@ After the session ends, the orchestrator determines the outcome from multiple si | error | No | any | `FAILED` | | unknown | - | - | `FAILED` | -**Cleanup:** Update task status with metadata (PR URL, cost, duration). Set TTL for data retention (default 90 days). Emit task events. Release concurrency counter. Send notifications. Persist code attribution to memory. +**Cleanup:** Update task status with metadata (PR URL, cost, duration). Set TTL for data retention (default 90 days). Emit task events. Release concurrency counter. Send notifications. Persist code attribution to memory. For `lambda-microvm`, finalization must call `TerminateMicrovm` after writing the terminal outcome: nothing in the guest self-terminates, and the 28,800-second maximum is a safety bound rather than cleanup. ### Step execution contract Every step in the pipeline satisfies these properties: -- **Idempotent** - Safe to retry after crashes. Context hydration produces the same prompt for the same inputs; session start reuses a pre-generated session ID. +- **Idempotent** - Safe to retry after crashes. Context hydration produces the same prompt for the same inputs; session-start retry semantics are implemented by each backend strategy. - **Timeout-bounded** - Each step has a configurable timeout to prevent blocking the pipeline. - **Failure-aware** - Returns `success` or `failed`. Infrastructure failures (throttle, transient errors) trigger exponential backoff retries (default: 2 retries, base 1s, max 10s). Explicit failures transition to `FAILED` without retry. - **Least-privilege input** - Each step receives only the `blueprintConfig` fields it needs. Custom Lambda steps get credential ARNs stripped. @@ -252,7 +258,7 @@ Every step in the pipeline satisfies these properties: Per [REPO_ONBOARDING.md](./REPO_ONBOARDING.md), blueprints customize execution through three layers: -1. **Parameterized strategies** - Select built-in implementations without code. Example: `compute.type: 'agentcore'` vs `compute.type: 'ecs'`. +1. **Parameterized strategies** - Select built-in implementations without code: `agentcore`, `ecs`, or `lambda-microvm`. 2. **Lambda-backed custom steps** - Inject custom logic at `pre-agent` or `post-agent` phases. Example: SAST scan before the agent, custom lint after. 3. **Custom step sequences** - Override the default step order entirely via an ordered `step_sequence` list. @@ -264,9 +270,9 @@ Agent sessions run for minutes to hours inside isolated compute environments. Th ### Liveness monitoring -Liveness detection varies by compute backend. AgentCore sessions use DynamoDB heartbeats and a `/ping` health endpoint; ECS Fargate tasks rely on the ECS `DescribeTasks` API since the ECS entrypoint does not write heartbeats. +Liveness detection varies by compute backend. AgentCore sessions use DynamoDB heartbeats and a `/ping` health endpoint; ECS Fargate tasks rely on the ECS `DescribeTasks` API; Lambda MicroVMs combine control-plane state with the same in-guest heartbeat used by AgentCore. -**DynamoDB heartbeat (AgentCore only).** The agent writes `agent_heartbeat_at` every 45 seconds via a daemon thread. The orchestrator applies two thresholds during polling when `computeType === 'agentcore'`: +**DynamoDB heartbeat (AgentCore and Lambda MicroVMs).** The agent writes `agent_heartbeat_at` every 45 seconds via a daemon thread. The orchestrator applies the same thresholds to both backends, only while task status is `RUNNING`: - **Grace period** (120s) - After entering `RUNNING`, the orchestrator waits before expecting heartbeats (covers container startup). - **Stale threshold** (240s) - If the heartbeat exists but is older than this, the session is treated as lost. @@ -276,6 +282,14 @@ When the session is unhealthy, the task transitions to `FAILED` with "Agent sess **ECS task status polling (ECS only).** The orchestrator calls `computeStrategy.pollSession` (ECS `DescribeTasks`) on each poll cycle. Three failure modes are detected: container failure (immediate `FAILED`), container exit without DynamoDB terminal write (fail after 5 consecutive completed polls), and repeated API failures (fail after 3 consecutive errors). ECS does not have heartbeat-based hung-process detection; a hung but alive container polls for the full `MAX_POLL_ATTEMPTS` window (~8.5h) before timing out. +**Lambda MicroVM state polling.** Liveness is a dual signal. The strategy maps `GetMicrovm` mechanically: `PENDING`/`RUNNING` report `running`, `SUSPENDING`/`SUSPENDED` report `suspended`, and `TERMINATING`/`TERMINATED` report terminal completion. The orchestrator supplies the health interpretation: + +- `suspended` is healthy only while the task is `AWAITING_APPROVAL`; in any other task state it emits an anomaly and keeps polling rather than failing recoverable work. +- A terminal substrate report paired with a non-terminal task is a failure, but the orchestrator first re-reads the task row to confirm the agent did not write a terminal result between the original read and VM termination. +- Substrate state detects a dead VM; heartbeat staleness detects a hung, deadlocked, or OOM-killed pipeline inside a VM that still reports `RUNNING`. + +`TERMINATED` is the normal terminal signal and remains observable for at least 10 minutes. `ResourceNotFoundException` maps to completion only as a late fallback after the control-plane record is eventually reaped; polling does not wait for `NotFound`. + **`/ping` health endpoint (AgentCore only).** The agent's FastAPI server responds to AgentCore's `/ping` calls while the coding task runs in a separate thread. AgentCore sees `HealthyBusy` and keeps the session alive. ### The idle timeout problem @@ -298,9 +312,9 @@ Long-running distributed systems fail. The orchestrator is designed so that ever | Hydration | Memory service unavailable | Proceed without memory (it is enrichment, not required) | | Hydration | Guardrail blocks content | Fail the task (content is adversarial, no retry) | | Hydration | Guardrail API unavailable | Fail the task (fail-closed: unscreened content never reaches agent) | -| Session start | `invoke_agent_runtime` throttled | Exponential backoff. Fail after retries exhausted. | -| Session start | Session crashes immediately | AgentCore: heartbeat never set, detected after 360s grace window. ECS: `DescribeTasks` reports failure on next poll. | -| Running | Agent crashes mid-task | AgentCore: heartbeat goes stale. ECS: `DescribeTasks` reports stopped task. Finalization inspects GitHub for partial work. | +| Session start | Selected compute service throttled | Exponential backoff. Fail after retries exhausted. | +| Session start | Session crashes immediately | AgentCore: heartbeat never set, detected after 360s grace window. ECS: `DescribeTasks` reports failure. Lambda MicroVMs: `GetMicrovm` reports terminal state or the heartbeat never appears. | +| Running | Agent crashes mid-task | AgentCore: heartbeat goes stale. ECS: `DescribeTasks` reports stopped task. Lambda MicroVMs: `GetMicrovm` detects VM death and heartbeat staleness detects an in-guest hang. Finalization inspects GitHub for partial work. | | Running | Agent hits turn or budget limit | Session ends normally. Finalize based on what was produced. | | Running | Idle for 15 min | AgentCore kills session. Task transitions to `TIMED_OUT`. | | Finalization | GitHub API down | Retry 3x. If still failing, mark `FAILED` with infrastructure reason. | @@ -325,7 +339,7 @@ Each task runs in its own isolated compute session with no shared mutable state | `invoke_agent_runtime` TPS | 25 per agent/account | AgentCore quota (adjustable) | | Concurrent sessions | Account-level limit | AgentCore quota | | Per-user concurrency | Configurable (default 3-5) | Platform config | -| System-wide max tasks | Configurable | Bounded by AgentCore session limit | +| System-wide max tasks | Configurable | Bounded by selected-backend quotas | ### Counter management @@ -344,7 +358,7 @@ Key properties: - **Sequential code, not a DSL.** The blueprint maps naturally to TypeScript with durable operations. No Amazon States Language or state machine abstractions. - **Built-in retry with checkpointing.** Steps support configurable retry strategies without re-executing completed work. -### Session monitoring pattern +### AgentCore session monitoring pattern ```mermaid sequenceDiagram @@ -400,7 +414,9 @@ Three DynamoDB tables back the orchestrator: one for task state, one for the aud | `pr_number` | Number? | PR number (required for PR workflows) | | `task_description` | String? | Free-text description | | `branch_name` | String | `bgagent/{task_id}/{slug}` for new tasks; PR's `head_ref` for PR tasks | -| `session_id` | String? | AgentCore session ID | +| `session_id` | String? | Backend session identifier (AgentCore session ID, ECS task ARN, or MicroVM ID) | +| `compute_type` | String? | Selected backend: `agentcore`, `ecs`, or `lambda-microvm` | +| `compute_metadata` | Map? | Backend lifecycle handle; Lambda MicroVMs persist `microvmId` and `endpoint` | | `execution_id` | String? | Durable execution ID | | `pr_url` | String? | PR URL (set during finalization) | | `error_message` | String? | Error reason if FAILED | diff --git a/docs/design/SECURITY.md b/docs/design/SECURITY.md index 8b04e1d82..3935d020e 100644 --- a/docs/design/SECURITY.md +++ b/docs/design/SECURITY.md @@ -39,12 +39,14 @@ Three authentication mechanisms protect the platform, matching its input channel **Agent credentials** - GitHub access currently uses a PAT stored in Secrets Manager. The orchestrator reads the secret at hydration time and passes it to the agent runtime. The model never receives the token in its context. Planned: replace the shared PAT with a GitHub App via AgentCore Identity Token Vault, providing per-task, repo-scoped, short-lived tokens (see [GitHub issues](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues), the [ADR-016](../decisions/ADR-016-pluggable-identity-and-auth.md) two-seam design, and the [IDENTITY_AND_AUTH.md](./IDENTITY_AND_AUTH.md) worked examples). -**Per-session IAM scoping** - The agent does not use its long-lived compute role (the AgentCore Runtime `ExecutionRole` or the ECS Fargate task role) for tenant data. Instead, at task startup it assumes a per-task **SessionRole** via `sts:AssumeRole` with session tags `{user_id, repo, task_id}`, and uses the resulting short-lived credentials for all DynamoDB and S3 tenant-data access. The SessionRole's policies self-constrain on those tags: +**Per-session IAM scoping** - The agent does not use its long-lived compute role (the AgentCore Runtime `ExecutionRole`, ECS Fargate task role, or Lambda MicroVMs execution role) for tenant data. Instead, at task startup it assumes a per-task **SessionRole** via `sts:AssumeRole` with session tags `{user_id, repo, task_id}`, and uses the resulting short-lived credentials for all DynamoDB and S3 tenant-data access. The SessionRole's policies self-constrain on those tags: - **DynamoDB**: item access on the four `task_id`-partitioned tables (task, events, approvals, nudges) is gated by a `dynamodb:LeadingKeys` condition equal to `${aws:PrincipalTag/task_id}`, so a session can read or write only its own task's rows. `Scan` is not granted (it ignores leading-keys). `task_id` is the isolation boundary because it is the base-table partition key — `LeadingKeys` cannot bind to a GSI partition key such as `user_id`. - **S3**: trace writes and attachment reads are scoped to the `/${aws:PrincipalTag/user_id}/` object prefix. -The compute role retains only non-tenant access (Bedrock model invocation — already ARN-scoped; CloudWatch Logs; the GitHub PAT secret, read once before the SessionRole is assumed; AgentCore Memory) plus `sts:AssumeRole`/`sts:TagSession` on the SessionRole. Because the agent runs under credentials that are themselves an assumed role, its `AssumeRole` is *role chaining* — capped at one hour regardless of the role's max session duration — so the agent uses a **refreshable** credential provider that re-assumes before expiry (tasks can run up to the 8-hour `maxLifetime`). The design is backend-agnostic: the same SessionRole and agent code serve both the AgentCore and ECS backends. A compromised agent session is therefore confined to its own task's data, enforced at the IAM layer rather than by application-code conventions. The policy structure (the `dynamodb:LeadingKeys` condition on `${aws:PrincipalTag/task_id}`, per-user S3 prefixes, and `Scan` exclusion) is asserted by CDK template tests, and the refreshable-credential and session-tag flow by agent unit tests; the matching-tag → allow / mismatched-or-absent-tag → deny behaviour was additionally confirmed once via the IAM policy simulator during development. +The compute role retains only non-tenant access (Bedrock model invocation — already ARN-scoped; CloudWatch Logs; the GitHub PAT secret, read once before the SessionRole is assumed; AgentCore Memory) plus `sts:AssumeRole`/`sts:TagSession` on the SessionRole. Because the agent runs under credentials that are themselves an assumed role, its `AssumeRole` is *role chaining* — capped at one hour regardless of the role's max session duration — so the agent uses a **refreshable** credential provider that re-assumes before expiry (tasks can run up to the 8-hour `maxLifetime`). The design is backend-agnostic: the same SessionRole and agent code serve all three backends (AgentCore Runtime, ECS Fargate, Lambda MicroVMs). A compromised agent session is therefore confined to its own task's data, enforced at the IAM layer rather than by application-code conventions. The policy structure (the `dynamodb:LeadingKeys` condition on `${aws:PrincipalTag/task_id}`, per-user S3 prefixes, and `Scan` exclusion) is asserted by CDK template tests, and the refreshable-credential and session-tag flow by agent unit tests; the matching-tag → allow / mismatched-or-absent-tag → deny behaviour was additionally confirmed once via the IAM policy simulator during development. + +**Lambda MicroVMs compute-role delta** - On this backend the compute role additionally holds prefix-scoped `secretsmanager:GetSecretValue` on the per-workspace channel-OAuth secrets (`bgagent-linear-oauth-*`, `bgagent-jira-oauth-*`), read access to the `/run` payload bucket (the task payload arrives as an S3 object, not as environment), and `ec2:DescribeAvailabilityZones` (`Resource: *`, read-only — EC2 describe actions have no resource-level scoping) for a CDK repo's own synth gate. It is also **the only compute role in the platform whose trust policy carries no confused-deputy condition**: the Lambda MicroVMs service populates no `aws:SourceAccount` or `aws:SourceArn` when it assumes the role, so a trust policy carrying one is unassumable — verified live, not assumed (connector creation failed deterministically, and `RunMicrovm` surfaced the same root cause as a misleading caller-side `iam:PassRole` denial). The compensating controls are that each of the three MicroVM roles can be passed to `lambda.amazonaws.com` only by a named principal — the orchestrator, via an `iam:PassRole` scoped to the execution role's exact ARN, and the CloudFormation deployment role for the build and connector-operator roles — that every other resource they reach is account-scoped by ARN apart from two justified `Resource: *` read/create-time statements, and that none of them holds `iam:*`, cross-account trust, or any `sts:AssumeRole` beyond the execution role's scoped hop to the per-task SessionRole. Full evidence, the two-arm PassRole experiment, and the alternatives considered are in [ADR-021 §4](../decisions/ADR-021-lambda-microvms-compute-backend.md#4-infra-and-iam-conditional-resources-behind-bootstrap-computetypes). > Out of scope for this control and tracked separately as GitHub issues: replacing the shared GitHub PAT (GitHub App / Token Vault), binding credentials to the MicroVM via attestation, and scoping AgentCore Memory (namespace isolation by `actorId`/`sessionId` remains its boundary). diff --git a/docs/src/content/docs/architecture/Compute.md b/docs/src/content/docs/architecture/Compute.md index 5b43f1e01..c574d6350 100644 --- a/docs/src/content/docs/architecture/Compute.md +++ b/docs/src/content/docs/architecture/Compute.md @@ -4,7 +4,7 @@ title: Compute # Compute -Every task runs in an isolated cloud compute environment. Nothing runs on the user's machine. The agent clones the repo, writes code, runs tests, and opens a PR inside a MicroVM that is created for the task and destroyed when it ends. +Every task runs in an isolated cloud compute environment. Nothing runs on the user's machine. The agent clones the repo, writes code, runs tests, and opens a PR inside a compute session that is created for the task and destroyed when it ends. - **Use this doc for:** understanding the compute environment, agent harness, network architecture, and the constraints that shape the platform's design. - **Related docs:** [ORCHESTRATOR.md](/sample-autonomous-cloud-coding-agents/architecture/orchestrator) for session management and liveness monitoring, [SECURITY.md](/sample-autonomous-cloud-coding-agents/architecture/security) for isolation and egress controls, [REPO_ONBOARDING.md](/sample-autonomous-cloud-coding-agents/architecture/repo-onboarding) for per-repo compute configuration. @@ -13,17 +13,19 @@ Every task runs in an isolated cloud compute environment. Nothing runs on the us The default runtime is **Amazon Bedrock AgentCore Runtime**, which runs each session in a Firecracker MicroVM with per-session isolation, managed lifecycle, and built-in health monitoring. For repos that exceed AgentCore's constraints (2 GB image limit, no GPU), the `ComputeStrategy` interface allows switching to alternative backends per repo. -| | AgentCore Runtime | ECS on Fargate | ECS on EC2 | EKS | AWS Batch | Lambda | Custom EC2 + Firecracker | -|---|---|---|---|---|---|---|---| -| **Isolation** | MicroVM (Firecracker) | Task-level (Firecracker) | Container on shared nodes | Pod on shared nodes | Backend-dependent | Function env (Firecracker) | MicroVM (you own it) | -| **Image limit** | 2 GB (non-adjustable) | No hard cap | No hard cap | No hard cap | Backend-dependent | 10 GB | N/A (you define) | -| **Filesystem** | Ephemeral + persistent mount (preview) | 20-200 GB ephemeral | Node disk + EBS/EFS | Node disk + PVs | Backend-dependent | 512 MB-10 GB `/tmp` | You choose (EBS/NVMe) | -| **Max duration** | 8 hours | No hard cap | No hard cap | No hard cap | Configurable | **15 minutes** | Unlimited | -| **Startup** | Service-managed | Slim images help | Warm ASGs + pre-pull | Karpenter + pre-pull | Backend-dependent | Provisioned concurrency | Snapshot pools (DIY) | -| **GPU** | No | No | Yes | Yes | Yes (EC2/EKS backend) | No | Yes (with passthrough) | -| **Ops burden** | Low (managed) | Low | Medium | High | Low-Medium | Low | **Very high** | -| **Cost model** | vCPU-hrs + GB-hrs | vCPU + mem/sec | EC2 + EBS | EKS control + EC2 | Underlying compute | Request + duration | EC2 metal + your ops | -| **Fit** | **Default choice** | Repos > 2 GB image | GPU, heavy toolchains | Max flexibility | Queued batch jobs | **Poor** (15 min cap) | Best potential, highest cost | +| | AgentCore Runtime | ECS on Fargate | **Lambda MicroVMs** | ECS on EC2 | EKS | AWS Batch | Lambda (functions) | Custom EC2 + Firecracker | +|---|---|---|---|---|---|---|---|---| +| **Isolation** | MicroVM (Firecracker) | Task-level (Firecracker) | MicroVM (Firecracker) | Container on shared nodes | Pod on shared nodes | Backend-dependent | Function env (Firecracker) | MicroVM (you own it) | +| **Image limit** | 2 GB (non-adjustable) | No hard cap | Zip + Dockerfile → snapshot; snapshot build size and OCI image size are different measures | No hard cap | No hard cap | Backend-dependent | 10 GB | N/A (you define) | +| **Filesystem** | Ephemeral + persistent mount (preview) | 20-200 GB ephemeral | 32 GB native disk in snapshot; survives suspend/resume; `flock()` works | Node disk + EBS/EFS | Node disk + PVs | Backend-dependent | 512 MB-10 GB `/tmp` | You choose (EBS/NVMe) | +| **Max duration** | 8 hours | No hard cap | 8 hours (running + suspended; 28,800s) | No hard cap | No hard cap | Configurable | **15 minutes** | Unlimited | +| **Startup** | Service-managed | Slim images help | Snapshot resume | Warm ASGs + pre-pull | Karpenter + pre-pull | Backend-dependent | Provisioned concurrency | Snapshot pools (DIY) | +| **GPU** | No | No | No | Yes | Yes | Yes (EC2/EKS backend) | No | Yes (with passthrough) | +| **Ops burden** | Low (managed) | Low | Low (managed) | Medium | High | Low-Medium | Low | **Very high** | +| **Cost model** | vCPU-hrs + GB-hrs | vCPU + mem/sec | Baseline-priced (8 GiB / 4 vCPU) with 4× vertical burst (32 GiB / 16 vCPU peak); suspended time is storage-only | EC2 + EBS | EKS control + EC2 | Underlying compute | Request + duration | EC2 metal + your ops | +| **Fit** | **Default choice** | Repos > 2 GB image | Suspend/resume economics; approval-wait-heavy workloads; default-sized repos. Heavy sustained-memory builds stay on ECS | GPU, heavy toolchains | Max flexibility | Queued batch jobs | **Poor** (15 min cap) | Best potential, highest cost | + +> **Lambda MicroVMs are not Lambda functions.** They are a different compute primitive, so the functions column's 15-minute cap and poor-fit verdict do not apply. See [ADR-021](/sample-autonomous-cloud-coding-agents/architecture/adr-021-lambda-microvms-compute-backend). The backend is selected per repo via `compute_type` in the Blueprint config. The orchestrator resolves the strategy and delegates session start, polling, and termination to the strategy implementation. See [REPO_ONBOARDING.md](/sample-autonomous-cloud-coding-agents/architecture/repo-onboarding) for the `ComputeStrategy` interface. @@ -77,6 +79,14 @@ The platform works around this by splitting storage: See [ORCHESTRATOR.md](/sample-autonomous-cloud-coding-agents/architecture/orchestrator) for how the orchestrator handles these timeouts. +## Lambda MicroVMs backend + +Lambda MicroVMs are an opt-in third backend, selected per repository with `compute_type: lambda-microvm`; AgentCore remains the default. Image configuration has three states: a managed base-image ARN and version creates the snapshot image in CDK; an external image identifier uses a snapshot built out of band; and supplying neither provisions only the roles, buckets, and connectors needed for the bootstrap deploy. `cdk/scripts/package-microvm-artifact.sh` packages the agent as zip + Dockerfile, uploads it to the artifact bucket, and can create the external image. Lambda MicroVMs are available in five launch regions (us-east-1, us-east-2, us-west-2, eu-west-1, ap-northeast-1) and will expand; the platform enforces regional availability in layers via a synth-time constant, onboarding live probes, and orchestration-time classification. + +Because a snapshot freezes its build-time environment, deployment-specific, non-secret identifiers travel in the `/run` hook's `platform_config` block instead. The strategy sends the canonical inline envelope or, when that envelope exceeds the verified 4,096-byte `runHookPayload` limit, an S3-pointer envelope with the configuration also merged into the uploaded payload. The agent accepts only allowlisted keys and installs them before pipeline initialization; [ADR-021 §3](/sample-autonomous-cloud-coding-agents/architecture/adr-021-lambda-microvms-compute-backend#3-packaging-same-agent-image-source-new-build-path) defines the exact wire shapes and validation rules. + +Networking separates image build from execution: the build-only connector permits TCP 80 and 443 because the Dockerfile uses `apt-get`, while running MicroVMs retain 443-only egress through the platform VPC. Every launch explicitly passes the Lambda-managed `NO_INGRESS` connector; omission would select the service's public-ingress default. The P2 image declares and serves `/ready` and `/validate` at build time and `/run` and `/terminate` at runtime. `/suspend` and `/resume` remain disabled until their P3 implementation. + ## ECS Fargate task sizing (build vs. planning) When a repo is `compute_type: ecs`, `EcsAgentCluster` provisions **two** Fargate task definitions, and the orchestrator picks between them per task by whether the resolved workflow is **read-only**: @@ -102,14 +112,14 @@ The platform uses the [Claude Agent SDK](https://github.com/anthropics/claude-ag **System prompt:** Selected by workflow from a shared base template (`agent/src/prompts/base.py`) with per-workflow sections (`coding/new-task-v1`, `coding/pr-iteration-v1`, `coding/pr-review-v1`). The platform defines what the agent should do; the harness executes it. -**Result contract:** The agent does not call back to the platform. It follows the contract (push work, create PR) and exits. The orchestrator infers the outcome from GitHub state and the agent's poll response. When the agent is stopped by an *environmental* fault (missing secret, egress denial, unreachable dependency, fail-closed policy-engine error), it emits a typed `agent_blocked` event and carries a canonical `BLOCKED[]: …` reason in its terminal error so the orchestrator's classifier attaches a precise remedy — see [Cedar HITL gates §13.16](/sample-autonomous-cloud-coding-agents/architecture/cedar-hitl-gates#1316-observable-blocker-signal-251). +**Result contract:** The agent follows the contract (push work, create PR), writes task state, and exits. The orchestrator infers the outcome from task state, backend liveness signals, and GitHub state. When the agent is stopped by an *environmental* fault (missing secret, egress denial, unreachable dependency, fail-closed policy-engine error), it emits a typed `agent_blocked` event and carries a canonical `BLOCKED[]: …` reason in its terminal error so the orchestrator's classifier attaches a precise remedy — see [Cedar HITL gates §13.16](/sample-autonomous-cloud-coding-agents/architecture/cedar-hitl-gates#1316-observable-blocker-signal-251). ### Tool set | Tool | Source | Description | |------|--------|-------------| -| Shell execution | Native (MicroVM) | Build, test, lint via bash | -| File system | Native (MicroVM) | Read/write code | +| Shell execution | Native (compute session) | Build, test, lint via bash | +| File system | Native (compute session) | Read/write code | | GitHub | AgentCore Gateway + Identity | Clone, push, PR, issues | | Web search | AgentCore Gateway | Documentation lookups | @@ -132,7 +142,7 @@ The agent runtime runs inside a VPC with private subnets. AWS service traffic st flowchart TB subgraph VPC["VPC (10.0.0.0/16)"] subgraph Private["Private Subnets"] - RT[AgentCore Runtime] + RT[Compute session] end subgraph Public["Public Subnets"] NAT[NAT Gateway] diff --git a/docs/src/content/docs/architecture/Deployment-roles.md b/docs/src/content/docs/architecture/Deployment-roles.md index cdce53038..f99472196 100644 --- a/docs/src/content/docs/architecture/Deployment-roles.md +++ b/docs/src/content/docs/architecture/Deployment-roles.md @@ -747,6 +747,10 @@ When the ECS Fargate compute backend is enabled (set the `ComputeTypes` CFN para When the Lambda MicroVM compute backend is enabled (include `lambda-microvm` in the `ComputeTypes` CFN parameter on the `CDKToolkit` stack), the generated template conditionally attaches this policy to the CloudFormation execution role. It permits CloudFormation to manage MicroVM images and network connectors; runtime session lifecycle permissions remain on the orchestrator role. +The second statement, `MicrovmPassRoles`, is the one exception to the rule that every `iam:PassRole` in this bundle carries an `iam:PassedToService` condition (`IaCRole-ABCA-Infrastructure` → `IAMPassRole`). It has to be: the Lambda MicroVMs service does not present a usable value for that key, so the conditioned statement is **denied** when CloudFormation passes the build role to `CreateMicrovmImage` — live-verified in `us-east-1` (ADR-021 P2r2-F9), with the out-of-band `create-microvm-image` call passing the *same* role successfully as the control. It is deliberately scoped to the two role-name prefixes CloudFormation actually passes (the image build role and the network-connector operator role) and excludes the MicroVM **execution** role, which only the orchestrator passes, at `RunMicrovm`. The shared allowlisted `IAMPassRole` statement (`role/backgroundagent-dev-*`) is left intact to avoid widening the grant for ~30 other roles in the stack, so while it technically matches the execution role, only the orchestrator actively reaches for it. + +> **Operators must re-bootstrap for this.** The statement ships in bootstrap policy bundle **1.4.0**; a CDKToolkit stack bootstrapped at 1.3.0 or earlier will fail the CDK-managed MicroVM image deploy with a caller-side `iam:PassRole` AccessDenied on the build role. Check `CDKToolkit`'s `BootstrapPolicyVersion` output, and re-run `mise //cdk:bootstrap` (with `ComputeTypes` including `lambda-microvm`) if it is behind. + ```json { "Statement": [ @@ -775,6 +779,15 @@ When the Lambda MicroVM compute backend is enabled (include `lambda-microvm` in "Effect": "Allow", "Resource": "*", "Sid": "LambdaMicrovms" + }, + { + "Action": "iam:PassRole", + "Effect": "Allow", + "Resource": [ + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeBuild*", + "arn:aws:iam::*:role/backgroundagent-dev-LambdaMicrovmComputeConnector*" + ], + "Sid": "MicrovmPassRoles" } ], "Version": "2012-10-17" diff --git a/docs/src/content/docs/architecture/Orchestrator.md b/docs/src/content/docs/architecture/Orchestrator.md index 0067ba18b..f1bfc6dd9 100644 --- a/docs/src/content/docs/architecture/Orchestrator.md +++ b/docs/src/content/docs/architecture/Orchestrator.md @@ -32,7 +32,7 @@ The orchestrator is deliberately scoped. It handles coordination and bookkeeping | Task lifecycle | Accept tasks, drive them through the state machine to a terminal state, persist state at each transition | | Admission control | Validate repo onboarding, concurrency limits, rate limits, idempotency | | Context hydration | Assemble the agent prompt from user input, GitHub data, memory, and repo config | -| Session management | Start the compute session, monitor liveness via heartbeat, detect completion | +| Session management | Start the compute session, monitor backend liveness and heartbeats where applicable, detect completion | | Result inference | Determine success or failure from agent response, DynamoDB record, and GitHub state | | Finalization | Update status, emit events, release concurrency, persist audit records | | Cancellation | Stop the session and drive the task to CANCELLED at any point | @@ -44,7 +44,7 @@ The orchestrator is deliberately scoped. It handles coordination and bookkeeping |---|---|---| | Request authentication | Input gateway | [INPUT_GATEWAY.md](/sample-autonomous-cloud-coding-agents/architecture/input-gateway) | | Agent logic (clone, code, test, PR) | Agent runtime | [COMPUTE.md](/sample-autonomous-cloud-coding-agents/architecture/compute) | -| Compute session lifecycle (VM, image pull) | AgentCore Runtime | [COMPUTE.md](/sample-autonomous-cloud-coding-agents/architecture/compute) | +| Compute substrate internals (VM/task provisioning, image pull) | Selected compute service | [COMPUTE.md](/sample-autonomous-cloud-coding-agents/architecture/compute) | | Memory storage and retrieval | AgentCore Memory | [MEMORY.md](/sample-autonomous-cloud-coding-agents/architecture/memory) | | Repository onboarding | Blueprint construct | [REPO_ONBOARDING.md](/sample-autonomous-cloud-coding-agents/architecture/repo-onboarding) | @@ -120,13 +120,13 @@ stateDiagram-v2 | `QUEUED` | `SUBMITTED` | Queue pickup | Admission-queue pickup Lambda sees free capacity; re-invokes the orchestrator. FIFO by `created_at`. | | `QUEUED` | `CANCELLED` | User cancels | Explicit cancel removes the task from the queue | | `QUEUED` | `FAILED` | Queue-age backstop | Task waited longer than `QUEUE_MAX_AGE_SECONDS` (default 24h) without admission | -| `HYDRATING` | `RUNNING` | Hydration complete | `invoke_agent_runtime` returns session ID | +| `HYDRATING` | `RUNNING` | Hydration complete | Selected `ComputeStrategy.startSession` returns a session handle | | `HYDRATING` | `AWAITING_APPROVAL` | Cedar soft-deny gate fires | Tool call triggers a soft-deny policy rule during hydration | | `HYDRATING` | `FAILED` | Hydration error | GitHub API failure, guardrail blocks content, Bedrock unavailable | | `RUNNING` | `AWAITING_APPROVAL` | Cedar soft-deny gate fires | Tool call triggers a soft-deny policy rule during execution | | `RUNNING` | `FINALIZING` | Session ends | Response received or session terminated | -| `RUNNING` | `TIMED_OUT` | Max duration exceeded | AgentCore terminates the session at its 8h cap; the orchestrator's own safety-net poll window is `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h, after which a still-`RUNNING` task is driven to `TIMED_OUT` | -| `RUNNING` | `FAILED` | Session crash | Heartbeat lost (see Liveness monitoring) | +| `RUNNING` | `TIMED_OUT` | Max duration exceeded | AgentCore and Lambda MicroVMs have an 8h substrate cap; the orchestrator's own safety-net poll window is `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h, after which a still-`RUNNING` task is driven to `TIMED_OUT` | +| `RUNNING` | `FAILED` | Session crash | Heartbeat or substrate liveness lost (see Liveness monitoring) | | `AWAITING_APPROVAL` | `RUNNING` | Approved or denied | Human decision received; agent resumes | | `AWAITING_APPROVAL` | `CANCELLED` | User cancels | Explicit cancel while awaiting approval | | `AWAITING_APPROVAL` | `FAILED` | Stranded reconciler | Approval request orphaned (agent died mid-wait) | @@ -143,19 +143,21 @@ Users can cancel a task at any point. The orchestrator's response depends on how | `QUEUED` | Transition to `CANCELLED`. Removes the task from the admission queue. No compute or concurrency slot to release (a queued task never held one). | | `SUBMITTED` | Transition to `CANCELLED`. No cleanup needed. | | `HYDRATING` | Abort hydration, release concurrency slot, transition to `CANCELLED`. | -| `RUNNING` | Call `stop_runtime_session`, wait for confirmation, release concurrency, transition to `CANCELLED`. Partial work on GitHub remains for the user to inspect. | -| `AWAITING_APPROVAL` | Call `stop_runtime_session`, release concurrency slot, transition to `CANCELLED`. The pending approval row transitions to `STRANDED`. | +| `RUNNING` | Transition to `CANCELLED` and stop the selected compute session best-effort. Partial work on GitHub remains for the user to inspect. | +| `AWAITING_APPROVAL` | Transition to `CANCELLED`. The pending approval row transitions to `STRANDED`. | | `FINALIZING` | Let finalization complete. Mark `CANCELLED` only if the terminal state was not yet written. | | Terminal | Reject the cancel request. | +For a running cancellation, backend dispatch calls ECS `StopTask`, Lambda `TerminateMicrovm`, or AgentCore `StopRuntimeSession`. The `lambda-microvm` branch is evaluated before the AgentCore `RUNTIME_ARN` fallback; otherwise a mixed deployment could stop an unrelated AgentCore session and leave the MicroVM billing until its cap. + ### Timeouts -Multiple timeout mechanisms work together to prevent runaway tasks. Time-based limits (session duration, idle) are enforced by AgentCore; cost-based limits (turns, budget) are enforced by the agent SDK. The orchestrator acts as a safety net when external timeouts fire. +Multiple timeout mechanisms work together to prevent runaway tasks. Substrate time limits vary by backend; cost-based limits (turns, budget) are enforced by the agent SDK. The orchestrator acts as a safety net when external timeouts fire. | Type | Default | Effect | |---|---|---| -| Max session duration | 8 hours | AgentCore terminates the session at its 8h cap. The orchestrator's safety-net poll loop runs up to `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h; a task still `RUNNING` when that window is exhausted is driven to `TIMED_OUT`. | -| Idle timeout | 15 minutes | AgentCore terminates if agent is idle. See Liveness monitoring. | +| Max session duration | 8 hours | AgentCore caps a session at 8h; Lambda MicroVMs use `maximumDurationInSeconds: 28,800`, including suspended time. The orchestrator's safety-net poll loop runs up to `MAX_POLL_ATTEMPTS` (1020) × 30s ≈ 8.5h; a task still `RUNNING` when that window is exhausted is driven to `TIMED_OUT`. | +| Idle timeout | Backend-specific | AgentCore has an idle timeout. Lambda MicroVMs omit `idlePolicy` because inbound-traffic idleness would suspend an outbound-only agent while it is working. See Liveness monitoring. | | Max turns | 100 (range 1-500) | Agent stops after N model invocations. Configurable per task or per repo. | | Max cost budget | $0.01-$100 | Agent stops when budget is reached. Per-task or per-repo via Blueprint. | | Hydration timeout | 2 minutes | Fail the task if context assembly takes too long. | @@ -181,7 +183,7 @@ Validates the task before any compute is consumed. Checks run in order: 1. **Repo onboarding** - `GetItem` on `RepoTable`. If not found or inactive, reject with `REPO_NOT_ONBOARDED`. This runs at the API handler level (`createTaskCore`) for fast rejection. 2. **User concurrency** - Atomic check-and-increment on `UserConcurrency` counter. If at limit (default 10), the task is **queued, not failed** (#441): it transitions `SUBMITTED → QUEUED` and a scheduled admission-queue pickup Lambda re-attempts admission in FIFO order (by `created_at`) as slots free up, flipping `QUEUED → SUBMITTED` and re-invoking the orchestrator. The pickup Lambda does a read-only capacity pre-check; the orchestrator's atomic increment remains the single writer of the counter, so a pickup that loses the race harmlessly re-queues without losing FIFO position. `GET /tasks/{id}` surfaces `queue_position` and `estimated_wait_s` while queued. -3. **System concurrency** - Compare total running + hydrating tasks to system limit (bounded by AgentCore quotas). +3. **System concurrency** - Compare total running + hydrating tasks to the configured system limit and selected-backend quotas. 4. **Rate limiting** - Sliding window counter (10 tasks/hour per user). Rate-limit rejections happen at submit time and are rejected, not queued (unlike the concurrency cap, which queues). 5. **Idempotency** - If the request includes an idempotency key and a task with that key exists, return the existing task. @@ -189,7 +191,7 @@ On acceptance, the concurrency slot is acquired and the orchestrator proceeds to ### Step 2: Pre-flight checks -Runs as a distinct top-level step (`pre-flight` in `orchestrate-task.ts`, via `runPreflightChecks`) **after** admission and **before** hydration, so external-dependency failures are caught before any prompt assembly or Bedrock screening consumes work. It verifies the GitHub token has sufficient permissions for the task type, catches inaccessible or closed PRs, and confirms GitHub API reachability. On failure it drives the task to `FAILED` and emits a `preflight_failed` event, surfacing clear errors like `INSUFFICIENT_GITHUB_REPO_PERMISSIONS` before AgentCore runtime is consumed. +Runs as a distinct top-level step (`pre-flight` in `orchestrate-task.ts`, via `runPreflightChecks`) **after** admission and **before** hydration, so external-dependency failures are caught before any prompt assembly or Bedrock screening consumes work. It verifies the GitHub token has sufficient permissions for the task type, catches inaccessible or closed PRs, and confirms GitHub API reachability. On failure it drives the task to `FAILED` and emits a `preflight_failed` event, surfacing clear errors like `INSUFFICIENT_GITHUB_REPO_PERMISSIONS` before a compute session is consumed. ### Step 3: Context hydration @@ -203,19 +205,23 @@ Regardless of workflow, the assembled prompt is screened through Amazon Bedrock ### Step 4: Session start -The orchestrator calls `invoke_agent_runtime` with the hydrated payload. The agent receives it, starts the coding task in a background thread (via `add_async_task`), and returns an acknowledgment immediately. The orchestrator records the `(task_id, session_id)` mapping and transitions to `RUNNING`. +The orchestrator resolves the repository's `ComputeStrategy` and calls `startSession` with the hydrated payload. AgentCore invokes the runtime, ECS starts a Fargate task, and Lambda MicroVMs launch a snapshot and deliver the payload through `/run`. The orchestrator persists the returned backend handle in `compute_metadata`, records the `(task_id, session_id)` mapping, and transitions to `RUNNING`. + +AgentCore's session ID is pre-generated and reused on retry. ECS and Lambda MicroVMs use their substrate identifiers as session IDs. -The session ID is pre-generated and reused on retry, making session start idempotent after a crash. +If `RunMicrovm` succeeds but persisting the session handle or emitting the start event fails, the start step terminates the MicroVM best-effort using its in-memory handle before propagating the original error. This orphan reap is required because no later poll or finalization step can recover an unpersisted handle. ### Step 5: Await completion -The orchestrator polls for completion using `waitForCondition` from the Durable Execution SDK. At configurable intervals (default 30s), it re-invokes on the same session (sticky routing). The agent responds with its current status: +The orchestrator polls for completion using `waitForCondition` from the Durable Execution SDK at configurable intervals (default 30s). DynamoDB task status is common to all backends; backend-specific checks supplement it: -- `running` - Orchestrator suspends until next interval (no compute charges) -- `completed` - Orchestrator resumes to finalization with the result -- `failed` - Same, with error payload +| Backend | Additional poll signal | +|---|---| +| AgentCore | Agent heartbeat; `/ping` keeps the runtime healthy while the task thread works | +| ECS | `DescribeTasks`, including container exit status and exit code | +| Lambda MicroVMs | `GetMicrovm` state plus agent heartbeat | -If the session is terminated externally (crash, timeout, cancellation), the poll detects it and the orchestrator proceeds to finalization using GitHub-based result inference as fallback. +While waiting between polls, the durable orchestrator suspends without compute charges. If the session is terminated externally (crash, timeout, cancellation), the poll detects it and the orchestrator proceeds to finalization using GitHub-based result inference as fallback. ### Step 6: Finalization @@ -240,13 +246,13 @@ After the session ends, the orchestrator determines the outcome from multiple si | error | No | any | `FAILED` | | unknown | - | - | `FAILED` | -**Cleanup:** Update task status with metadata (PR URL, cost, duration). Set TTL for data retention (default 90 days). Emit task events. Release concurrency counter. Send notifications. Persist code attribution to memory. +**Cleanup:** Update task status with metadata (PR URL, cost, duration). Set TTL for data retention (default 90 days). Emit task events. Release concurrency counter. Send notifications. Persist code attribution to memory. For `lambda-microvm`, finalization must call `TerminateMicrovm` after writing the terminal outcome: nothing in the guest self-terminates, and the 28,800-second maximum is a safety bound rather than cleanup. ### Step execution contract Every step in the pipeline satisfies these properties: -- **Idempotent** - Safe to retry after crashes. Context hydration produces the same prompt for the same inputs; session start reuses a pre-generated session ID. +- **Idempotent** - Safe to retry after crashes. Context hydration produces the same prompt for the same inputs; session-start retry semantics are implemented by each backend strategy. - **Timeout-bounded** - Each step has a configurable timeout to prevent blocking the pipeline. - **Failure-aware** - Returns `success` or `failed`. Infrastructure failures (throttle, transient errors) trigger exponential backoff retries (default: 2 retries, base 1s, max 10s). Explicit failures transition to `FAILED` without retry. - **Least-privilege input** - Each step receives only the `blueprintConfig` fields it needs. Custom Lambda steps get credential ARNs stripped. @@ -256,7 +262,7 @@ Every step in the pipeline satisfies these properties: Per [REPO_ONBOARDING.md](/sample-autonomous-cloud-coding-agents/architecture/repo-onboarding), blueprints customize execution through three layers: -1. **Parameterized strategies** - Select built-in implementations without code. Example: `compute.type: 'agentcore'` vs `compute.type: 'ecs'`. +1. **Parameterized strategies** - Select built-in implementations without code: `agentcore`, `ecs`, or `lambda-microvm`. 2. **Lambda-backed custom steps** - Inject custom logic at `pre-agent` or `post-agent` phases. Example: SAST scan before the agent, custom lint after. 3. **Custom step sequences** - Override the default step order entirely via an ordered `step_sequence` list. @@ -268,9 +274,9 @@ Agent sessions run for minutes to hours inside isolated compute environments. Th ### Liveness monitoring -Liveness detection varies by compute backend. AgentCore sessions use DynamoDB heartbeats and a `/ping` health endpoint; ECS Fargate tasks rely on the ECS `DescribeTasks` API since the ECS entrypoint does not write heartbeats. +Liveness detection varies by compute backend. AgentCore sessions use DynamoDB heartbeats and a `/ping` health endpoint; ECS Fargate tasks rely on the ECS `DescribeTasks` API; Lambda MicroVMs combine control-plane state with the same in-guest heartbeat used by AgentCore. -**DynamoDB heartbeat (AgentCore only).** The agent writes `agent_heartbeat_at` every 45 seconds via a daemon thread. The orchestrator applies two thresholds during polling when `computeType === 'agentcore'`: +**DynamoDB heartbeat (AgentCore and Lambda MicroVMs).** The agent writes `agent_heartbeat_at` every 45 seconds via a daemon thread. The orchestrator applies the same thresholds to both backends, only while task status is `RUNNING`: - **Grace period** (120s) - After entering `RUNNING`, the orchestrator waits before expecting heartbeats (covers container startup). - **Stale threshold** (240s) - If the heartbeat exists but is older than this, the session is treated as lost. @@ -280,6 +286,14 @@ When the session is unhealthy, the task transitions to `FAILED` with "Agent sess **ECS task status polling (ECS only).** The orchestrator calls `computeStrategy.pollSession` (ECS `DescribeTasks`) on each poll cycle. Three failure modes are detected: container failure (immediate `FAILED`), container exit without DynamoDB terminal write (fail after 5 consecutive completed polls), and repeated API failures (fail after 3 consecutive errors). ECS does not have heartbeat-based hung-process detection; a hung but alive container polls for the full `MAX_POLL_ATTEMPTS` window (~8.5h) before timing out. +**Lambda MicroVM state polling.** Liveness is a dual signal. The strategy maps `GetMicrovm` mechanically: `PENDING`/`RUNNING` report `running`, `SUSPENDING`/`SUSPENDED` report `suspended`, and `TERMINATING`/`TERMINATED` report terminal completion. The orchestrator supplies the health interpretation: + +- `suspended` is healthy only while the task is `AWAITING_APPROVAL`; in any other task state it emits an anomaly and keeps polling rather than failing recoverable work. +- A terminal substrate report paired with a non-terminal task is a failure, but the orchestrator first re-reads the task row to confirm the agent did not write a terminal result between the original read and VM termination. +- Substrate state detects a dead VM; heartbeat staleness detects a hung, deadlocked, or OOM-killed pipeline inside a VM that still reports `RUNNING`. + +`TERMINATED` is the normal terminal signal and remains observable for at least 10 minutes. `ResourceNotFoundException` maps to completion only as a late fallback after the control-plane record is eventually reaped; polling does not wait for `NotFound`. + **`/ping` health endpoint (AgentCore only).** The agent's FastAPI server responds to AgentCore's `/ping` calls while the coding task runs in a separate thread. AgentCore sees `HealthyBusy` and keeps the session alive. ### The idle timeout problem @@ -302,9 +316,9 @@ Long-running distributed systems fail. The orchestrator is designed so that ever | Hydration | Memory service unavailable | Proceed without memory (it is enrichment, not required) | | Hydration | Guardrail blocks content | Fail the task (content is adversarial, no retry) | | Hydration | Guardrail API unavailable | Fail the task (fail-closed: unscreened content never reaches agent) | -| Session start | `invoke_agent_runtime` throttled | Exponential backoff. Fail after retries exhausted. | -| Session start | Session crashes immediately | AgentCore: heartbeat never set, detected after 360s grace window. ECS: `DescribeTasks` reports failure on next poll. | -| Running | Agent crashes mid-task | AgentCore: heartbeat goes stale. ECS: `DescribeTasks` reports stopped task. Finalization inspects GitHub for partial work. | +| Session start | Selected compute service throttled | Exponential backoff. Fail after retries exhausted. | +| Session start | Session crashes immediately | AgentCore: heartbeat never set, detected after 360s grace window. ECS: `DescribeTasks` reports failure. Lambda MicroVMs: `GetMicrovm` reports terminal state or the heartbeat never appears. | +| Running | Agent crashes mid-task | AgentCore: heartbeat goes stale. ECS: `DescribeTasks` reports stopped task. Lambda MicroVMs: `GetMicrovm` detects VM death and heartbeat staleness detects an in-guest hang. Finalization inspects GitHub for partial work. | | Running | Agent hits turn or budget limit | Session ends normally. Finalize based on what was produced. | | Running | Idle for 15 min | AgentCore kills session. Task transitions to `TIMED_OUT`. | | Finalization | GitHub API down | Retry 3x. If still failing, mark `FAILED` with infrastructure reason. | @@ -329,7 +343,7 @@ Each task runs in its own isolated compute session with no shared mutable state | `invoke_agent_runtime` TPS | 25 per agent/account | AgentCore quota (adjustable) | | Concurrent sessions | Account-level limit | AgentCore quota | | Per-user concurrency | Configurable (default 3-5) | Platform config | -| System-wide max tasks | Configurable | Bounded by AgentCore session limit | +| System-wide max tasks | Configurable | Bounded by selected-backend quotas | ### Counter management @@ -348,7 +362,7 @@ Key properties: - **Sequential code, not a DSL.** The blueprint maps naturally to TypeScript with durable operations. No Amazon States Language or state machine abstractions. - **Built-in retry with checkpointing.** Steps support configurable retry strategies without re-executing completed work. -### Session monitoring pattern +### AgentCore session monitoring pattern ```mermaid sequenceDiagram @@ -404,7 +418,9 @@ Three DynamoDB tables back the orchestrator: one for task state, one for the aud | `pr_number` | Number? | PR number (required for PR workflows) | | `task_description` | String? | Free-text description | | `branch_name` | String | `bgagent/{task_id}/{slug}` for new tasks; PR's `head_ref` for PR tasks | -| `session_id` | String? | AgentCore session ID | +| `session_id` | String? | Backend session identifier (AgentCore session ID, ECS task ARN, or MicroVM ID) | +| `compute_type` | String? | Selected backend: `agentcore`, `ecs`, or `lambda-microvm` | +| `compute_metadata` | Map? | Backend lifecycle handle; Lambda MicroVMs persist `microvmId` and `endpoint` | | `execution_id` | String? | Durable execution ID | | `pr_url` | String? | PR URL (set during finalization) | | `error_message` | String? | Error reason if FAILED | diff --git a/docs/src/content/docs/architecture/Security.md b/docs/src/content/docs/architecture/Security.md index cbe8dbe26..d7a6d04e1 100644 --- a/docs/src/content/docs/architecture/Security.md +++ b/docs/src/content/docs/architecture/Security.md @@ -43,12 +43,14 @@ Three authentication mechanisms protect the platform, matching its input channel **Agent credentials** - GitHub access currently uses a PAT stored in Secrets Manager. The orchestrator reads the secret at hydration time and passes it to the agent runtime. The model never receives the token in its context. Planned: replace the shared PAT with a GitHub App via AgentCore Identity Token Vault, providing per-task, repo-scoped, short-lived tokens (see [GitHub issues](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues), the [ADR-016](/sample-autonomous-cloud-coding-agents/architecture/adr-016-pluggable-identity-and-auth) two-seam design, and the [IDENTITY_AND_AUTH.md](/sample-autonomous-cloud-coding-agents/architecture/identity-and-auth) worked examples). -**Per-session IAM scoping** - The agent does not use its long-lived compute role (the AgentCore Runtime `ExecutionRole` or the ECS Fargate task role) for tenant data. Instead, at task startup it assumes a per-task **SessionRole** via `sts:AssumeRole` with session tags `{user_id, repo, task_id}`, and uses the resulting short-lived credentials for all DynamoDB and S3 tenant-data access. The SessionRole's policies self-constrain on those tags: +**Per-session IAM scoping** - The agent does not use its long-lived compute role (the AgentCore Runtime `ExecutionRole`, ECS Fargate task role, or Lambda MicroVMs execution role) for tenant data. Instead, at task startup it assumes a per-task **SessionRole** via `sts:AssumeRole` with session tags `{user_id, repo, task_id}`, and uses the resulting short-lived credentials for all DynamoDB and S3 tenant-data access. The SessionRole's policies self-constrain on those tags: - **DynamoDB**: item access on the four `task_id`-partitioned tables (task, events, approvals, nudges) is gated by a `dynamodb:LeadingKeys` condition equal to `${aws:PrincipalTag/task_id}`, so a session can read or write only its own task's rows. `Scan` is not granted (it ignores leading-keys). `task_id` is the isolation boundary because it is the base-table partition key — `LeadingKeys` cannot bind to a GSI partition key such as `user_id`. - **S3**: trace writes and attachment reads are scoped to the `/${aws:PrincipalTag/user_id}/` object prefix. -The compute role retains only non-tenant access (Bedrock model invocation — already ARN-scoped; CloudWatch Logs; the GitHub PAT secret, read once before the SessionRole is assumed; AgentCore Memory) plus `sts:AssumeRole`/`sts:TagSession` on the SessionRole. Because the agent runs under credentials that are themselves an assumed role, its `AssumeRole` is *role chaining* — capped at one hour regardless of the role's max session duration — so the agent uses a **refreshable** credential provider that re-assumes before expiry (tasks can run up to the 8-hour `maxLifetime`). The design is backend-agnostic: the same SessionRole and agent code serve both the AgentCore and ECS backends. A compromised agent session is therefore confined to its own task's data, enforced at the IAM layer rather than by application-code conventions. The policy structure (the `dynamodb:LeadingKeys` condition on `${aws:PrincipalTag/task_id}`, per-user S3 prefixes, and `Scan` exclusion) is asserted by CDK template tests, and the refreshable-credential and session-tag flow by agent unit tests; the matching-tag → allow / mismatched-or-absent-tag → deny behaviour was additionally confirmed once via the IAM policy simulator during development. +The compute role retains only non-tenant access (Bedrock model invocation — already ARN-scoped; CloudWatch Logs; the GitHub PAT secret, read once before the SessionRole is assumed; AgentCore Memory) plus `sts:AssumeRole`/`sts:TagSession` on the SessionRole. Because the agent runs under credentials that are themselves an assumed role, its `AssumeRole` is *role chaining* — capped at one hour regardless of the role's max session duration — so the agent uses a **refreshable** credential provider that re-assumes before expiry (tasks can run up to the 8-hour `maxLifetime`). The design is backend-agnostic: the same SessionRole and agent code serve all three backends (AgentCore Runtime, ECS Fargate, Lambda MicroVMs). A compromised agent session is therefore confined to its own task's data, enforced at the IAM layer rather than by application-code conventions. The policy structure (the `dynamodb:LeadingKeys` condition on `${aws:PrincipalTag/task_id}`, per-user S3 prefixes, and `Scan` exclusion) is asserted by CDK template tests, and the refreshable-credential and session-tag flow by agent unit tests; the matching-tag → allow / mismatched-or-absent-tag → deny behaviour was additionally confirmed once via the IAM policy simulator during development. + +**Lambda MicroVMs compute-role delta** - On this backend the compute role additionally holds prefix-scoped `secretsmanager:GetSecretValue` on the per-workspace channel-OAuth secrets (`bgagent-linear-oauth-*`, `bgagent-jira-oauth-*`), read access to the `/run` payload bucket (the task payload arrives as an S3 object, not as environment), and `ec2:DescribeAvailabilityZones` (`Resource: *`, read-only — EC2 describe actions have no resource-level scoping) for a CDK repo's own synth gate. It is also **the only compute role in the platform whose trust policy carries no confused-deputy condition**: the Lambda MicroVMs service populates no `aws:SourceAccount` or `aws:SourceArn` when it assumes the role, so a trust policy carrying one is unassumable — verified live, not assumed (connector creation failed deterministically, and `RunMicrovm` surfaced the same root cause as a misleading caller-side `iam:PassRole` denial). The compensating controls are that each of the three MicroVM roles can be passed to `lambda.amazonaws.com` only by a named principal — the orchestrator, via an `iam:PassRole` scoped to the execution role's exact ARN, and the CloudFormation deployment role for the build and connector-operator roles — that every other resource they reach is account-scoped by ARN apart from two justified `Resource: *` read/create-time statements, and that none of them holds `iam:*`, cross-account trust, or any `sts:AssumeRole` beyond the execution role's scoped hop to the per-task SessionRole. Full evidence, the two-arm PassRole experiment, and the alternatives considered are in [ADR-021 §4](/sample-autonomous-cloud-coding-agents/architecture/adr-021-lambda-microvms-compute-backend#4-infra-and-iam-conditional-resources-behind-bootstrap-computetypes). > Out of scope for this control and tracked separately as GitHub issues: replacing the shared GitHub PAT (GitHub App / Token Vault), binding credentials to the MicroVM via attestation, and scoping AgentCore Memory (namespace isolation by `actorId`/`sessionId` remains its boundary). diff --git a/docs/src/content/docs/decisions/Adr-021-lambda-microvms-compute-backend.md b/docs/src/content/docs/decisions/Adr-021-lambda-microvms-compute-backend.md index a88f76fc5..ebdddd0e9 100644 --- a/docs/src/content/docs/decisions/Adr-021-lambda-microvms-compute-backend.md +++ b/docs/src/content/docs/decisions/Adr-021-lambda-microvms-compute-backend.md @@ -30,7 +30,7 @@ ABCA selects a per-repo compute backend through the Blueprint's `compute_type` f | Resources | AgentCore-managed | 16 vCPU / 120 GB / 20–200 GB disk | **Baseline 8 GiB RAM / 4 vCPU, auto-scaling to a 32 GiB / 16 vCPU peak**; 32 GB disk. `minimumMemoryInMiB` configures the BASELINE (max 8,192 MiB); the service scales vertically on demand — capacity is baseline-priced with 4× burst headroom | | Packaging | ECR image ≤ 2 GB | ECR image, no hard cap | **Zip + Dockerfile in S3 → service-built snapshot image** (versioned, storage billed) | | Invocation | `InvokeAgentRuntime` (SigV4) | `RunTask` + container overrides | `RunMicrovm` (image **ARN** required — a bare name is rejected) → dedicated HTTPS endpoint + JWE token (`CreateMicrovmAuthToken`, ≤ 60 min TTL) | -| Liveness | Agent heartbeat + `/ping` | `DescribeTasks` | MicroVM state (RUNNING / SUSPENDED / TERMINATED) via control-plane API | +| Liveness | Agent heartbeat + `/ping` | `DescribeTasks` | MicroVM state (RUNNING / SUSPENDED / TERMINATED) via control-plane API **and** agent heartbeat (see sub-decision 1) | | Session storage | `/mnt/workspace` FUSE (no `flock()`) | Ephemeral disk | Native disk in snapshot — **survives suspend/resume, `flock()` works** | | Architecture | ARM64 | ARM64 | ARM64 (Graviton) | | Regions (launch) | Broad | Broad | 5 (us-east-1/2, us-west-2, eu-west-1, ap-northeast-1) | @@ -72,6 +72,10 @@ The `ComputeStrategy` interface gains **mandatory** `suspendSession(handle)` / ` **Poll semantics — the strategy reports, the orchestrator interprets.** `pollSession(handle)` receives only the session handle and cannot see task state, so the health rules must live where the DynamoDB status lives. `SessionStatus` gains a `'suspended'` variant; the strategy maps `GetMicrovm` state mechanically and the **orchestrator** cross-references against the task row — the same division of labor `finalPollState` already uses for ECS (substrate stopped + non-terminal DynamoDB status → failed) and `pollTaskStatus` uses for agentcore heartbeats: substrate `suspended` + task `AWAITING_APPROVAL` is healthy (orchestrator-intended suspend); `suspended` with any other task status is an anomaly to surface, not fail-fast; substrate terminal + non-terminal task status → classify failed. +**Liveness on this backend is substrate state AND agent heartbeat.** The substrate cross-check above answers one question — "is the VM still there?" — and P2 established that it is not sufficient on its own. `GetMicrovm` catches a MicroVM that *died*; it cannot catch a MicroVM that is alive and reporting `RUNNING` while the pipeline **inside the guest** is hung, deadlocked, or was OOM-killed. That state is not hypothetical or self-correcting on this substrate, and the P2 live run narrowed *why* without weakening the conclusion. The service does reap a VM whose run hook FAILS (a 4xx makes it terminate within ~12 s — see sub-decision 2), so the P1 evidence for this paragraph (a hook-less image sitting in `RUNNING` indefinitely with no `stateReason`) no longer describes an ABCA image. What the service reaps is a hook *result*; it has no view into the guest afterwards. So the surviving — and more realistic — hang case is a `/run` hook that returned **200** and a pipeline that then hung, deadlocked or was OOM-killed behind it: the substrate stays `RUNNING`, the service is satisfied, and nothing else notices. Left to the substrate check alone, such a task would burn the orchestrator's full ~8.5 h poll window — billing an 8-hour reservation — before the safety net fired. + +The in-guest half is already being written: the agent updates `agent_heartbeat_at` on the task row unconditionally, with no backend awareness, so the timestamp exists on every substrate. Only the orchestrator's *reaction* to it was backend-scoped — `pollTaskStatus` evaluated staleness for `agentcore` alone — which is the gap P2 closes by extending it to `lambda-microvm`. The grace and stale thresholds are the SAME on both: the timestamp is written by the same pipeline code at the same cadence, so a backend-specific window would encode a difference that does not exist. The two signals stay complementary rather than redundant — the substrate check is the crash detector, the heartbeat is the hang detector — and the check remains scoped to task status `RUNNING`, which is what keeps a deliberately suspended VM during an approval wait (sub-decision 2, P3) from being read as a dead one. `ecs` is deliberately left out: `DescribeTasks` reports a real container exit *with an exit code* (OOM-kill included) and the ECS poll block already interprets it with its own patience counters, so adding the heartbeat there would give one backend two independently-tuned kill paths for the same failure. + The service's `MicrovmState` enum has **six** members, not three, so the mapping is stated exhaustively (one line of rationale each, mirrored in the strategy's doc comment): | `MicrovmState` | `SessionStatus` | Why | @@ -107,6 +111,8 @@ Normative requirements (EARS, per [ADR-020](/sample-autonomous-cloud-coding-agen - If `GetMicrovm` reports that the MicroVM does not exist, then the strategy shall report `completed`. - If the strategy reports a terminal substrate state while the task's DynamoDB status is non-terminal, then the orchestrator shall re-read the task row and, if it is still non-terminal, classify the task as failed with a substrate-failure remedy. - If the strategy reports `suspended` while the task's DynamoDB status is not `AWAITING_APPROVAL`, then the orchestrator shall surface an anomaly event and shall not fail-fast the task. +- While a `lambda-microvm` task's DynamoDB status is `RUNNING`, if the task's `agent_heartbeat_at` is stale (or absent past the grace window) by the same thresholds the orchestrator applies to `agentcore`, then the orchestrator shall treat the session as unhealthy and stop polling — the substrate `GetMicrovm` check shall remain the crash detector, and the heartbeat shall be the in-guest hang detector. +- The task-detail API response shall include `agent_heartbeat_at`, and the CLI shall surface it while the task is non-terminal (P2r2-F11: the field drove the orchestrator's hang detector but was never projected, so no operator could observe the signal — and its invisibility produced a wrong verification conclusion). - If `suspendSession` or `resumeSession` is invoked on a strategy that does not support suspension, then the strategy shall return an explicit unsupported result. - When the agent process reaches a terminal state, the agent shall exit. - When the orchestrator finalizes a `lambda-microvm` task, the orchestrator shall call `terminate-microvm` (termination shall not rely on any substrate timeout, and shall not rely on the MicroVM self-terminating — it does not). @@ -124,7 +130,11 @@ The handshake must respect the existing approval mechanics: the agent **discover *Why inline rather than poll-only — codebase precedent:* resume-on-approve is structurally identical to task cancellation — a user-initiated, latency-sensitive action whose purpose is an immediate compute-lifecycle side effect. `cancel-task.ts` already resolves this exact tension: the API-plane handler invokes ECS `StopTask` / AgentCore `StopRuntimeSession` inline, best-effort (a failed stop logs a warning and the state transition stands; a `task_cancel_compute_orphan` event is written when no stoppable compute handle exists, `reason: missing_runtime_handle`) — with the conditional IAM wired in `task-api.ts`. The resume path goes one step further than the precedent by also writing the orphan event on *failed* resume calls, because a failed resume strands a suspended VM awaiting a decision — a stronger liveness consequence than a failed stop of an already-cancelled task. The alternative (orchestrator-poll-only resume) preserves single-owner lifecycle purity but pays up to a full poll interval (~30 s) of latency on every approval, and the purity argument was already litigated and declined for cancel. `approve-task.ts` is deliberately minimal today (security-critical ownership comparison, Cedar finding #6); the resume call is therefore added *after* the transaction commits, cannot alter the decision outcome, and carries one conditional `lambda:ResumeMicrovm` grant — the same blast-radius trade the cancel handler accepted in review. - **Timeout under freeze — the agent re-bases on the wall clock it already owns.** The agent's monotonic gate timer freezes while suspended, so resuming near the deadline is not enough: the frozen timer would still hold its remaining budget and fire the deny minutes *after* the user-visible window — colliding with the approval row's TTL (`created_at + timeout_s + 120s`) and triggering the "row reaped → stranded" fallback on a healthy gate. Instead, the gate expires at **`min(monotonic budget, created_at + timeout_s)`**, evaluated on each poll iteration and on `/resume`. This is not a new principle: Cedar decision #6 is already "min wins" for timeouts, the wall-clock deadline is already durable in the approval row the agent itself writes (`created_at` is in the agent's own clock domain — no skew), and §13.12's late-approval race fix already establishes that the durable row is authoritative over the agent's local timer. Deny authority stays agent-side (the conditional `TIMED_OUT` write + ConsistentRead re-read race protection is untouched); the orchestrator's resume at `deadline − margin` is purely the wake-up mechanism, with no correctness role. -- **Backstops, not mechanisms.** `maximumDurationInSeconds` (mandatory on every `RunMicrovm`, pinned at 28 800 s — see sub-decision 1) is the substrate kill switch bounding running **and** suspended time; the orchestrator's finalization `terminate-microvm` is the active cleanup path; the stranded-approval reconciler retains its role for orphaned waits. No `idlePolicy`-based bound is used in any phase — see sub-decision 1's omit-`idlePolicy` invariant. **The active terminate is mandatory, not belt-and-braces**: a MicroVM whose hook never ran reached `RUNNING` in 12 s and stayed `RUNNING` with no `stateReason` through every checkpoint (live). Nothing self-terminates on this substrate, so nothing cleans up — a leaked VM bills until the 8 h cap. +- **Backstops, not mechanisms.** `maximumDurationInSeconds` (mandatory on every `RunMicrovm`, pinned at 28 800 s — see sub-decision 1) is the substrate kill switch bounding running **and** suspended time; the orchestrator's finalization `terminate-microvm` is the active cleanup path; the stranded-approval reconciler retains its role for orphaned waits. No `idlePolicy`-based bound is used in any phase — see sub-decision 1's omit-`idlePolicy` invariant. + + **The active terminate is still mandatory on the SUCCESS path, and P2 sharpened why.** P1 concluded flatly that "nothing self-terminates": a hook-less MicroVM reached `RUNNING` in 12 s and stayed there with no `stateReason` through every checkpoint. P2 refuted that *for the failure path only* — with `run: ENABLED`, a run hook that answers 4xx makes the **service** terminate the VM within ~12 s, `stateReason: "Run lifecycle hook returned HTTP status 400. Please check your hook endpoint and application logs for more details."`, after which `suspend-microvm` correctly refuses it. That is a real improvement in cost posture and a direct benefit of declaring hooks (see also the failure-path row in the phasing table, sub-decision 3). + + It does **not** relieve the orchestrator of anything, because the two cases are disjoint. The service reaps a hook *result* it did not like; it has no view of the guest once the hook returned 200. So a task that starts normally — the overwhelming majority — has no service-side reaper at all, and a VM whose pipeline finished, crashed after `/run`, or hung is reaped by nobody but `TerminateMicrovm`. A leaked handle therefore remains a cost incident that bills until the 8 h cap; only the "the guest rejected its own payload" corner now cleans itself up. - **Concurrency slot stays held** during suspend. Cedar decision #7's rationale ("container alive, consuming memory") weakens under suspend, and the harder replacement rationale — "AWS counts `SUSPENDED` MicroVMs toward the account memory quota, so releasing ABCA's slot would not free real capacity" — is **undischarged**: the suspended VM stayed in `list-microvms` at every checkpoint, but that only proves *listed*. `L-CD1C0CC4` (1024 GB, account-scoped) exposes no `UsageMetric`, `AWS/Usage` carries only `CallCount` per API, and no MicroVM memory metric exists in any namespace, so consumption is **not observable safely** — proving it would need a large concurrent fleet. The conclusion (hold the slot) stands as the conservative choice, not as a verified fact. Size the arithmetic against the 32 GiB **peak** rather than the 8 GiB baseline: a busy fleet scales up, so peak is what actually competes for the account quota. The agent's `/suspend` hook flushes progress events (durable writes before returning 200, within the 60 s hook budget); `/resume` reseeds CSPRNGs and refreshes cached credentials. @@ -153,25 +163,63 @@ The phasing is therefore: | Hook | Declared by | Served by the agent | Notes | |---|---|---|---| -| `/ready` | **P1** (construct sets `hooks.microvmImageHooks.ready`) | **P1** | MANDATORY, not a quality nicety — see above. A 200 once uvicorn is bound is the whole P1 contract: it also proves `server` imported cleanly (pulling in `pipeline` → `runner` → the policy engine), so a missing policy file fails the BUILD instead of the first task. | -| `/run` | **P1** (construct sets `hooks.microvmHooks.run`) | **P1** | The payload-delivery channel. Must be served in P1 because `/ready` forces hooks to exist at all, and a hook-less image cannot accept `runHookPayload`. | -| `/validate` | **P2** | **P2** | Build-time snapshot-quality hook. Still deliberately NOT declared: a `/validate` that 404s or reports failure fails every image build. Deeper warm-up assertions (Bedrock reachability, Memory access, tool availability) belong here. | -| `/suspend`, `/resume`, `/terminate` | **P3** (suspend/resume), P2 (`/terminate`) | P3 / P2 | Declaring a runtime hook the agent does not serve fails the corresponding lifecycle transition, so each is declared only in the phase that implements it. P1 termination is the orchestrator's `TerminateMicrovm`, which needs no in-guest cooperation. | +| `/ready` | **P1** (construct enables `hooks.microvmImageHooks.ready`) | **P1** | MANDATORY, not a quality nicety — see above. A 200 proves uvicorn is bound and `server` imported cleanly (pulling in `pipeline` → `runner` → the policy engine), so a missing policy file fails the BUILD instead of the first task. **Since P2-F5 it also WARMS the snapshot** — the hook's 200 is what the service waits for before capturing the snapshot, making this the only place a warm page can be created, and the 225 MiB `claude` binary was cold in it (see the P2-F5 correction below). A required warm-up failure answers 503, so a snapshot that cannot exec the agent's own CLI fails the image build instead of every task. Still makes ZERO AWS calls, logging included (a `--version` exec is neither an AWS call nor a network call). | +| `/run` | **P1** (construct sets `hooks.microvmHooks.run`) | **P1** | The payload-delivery channel. Must be served in P1 because `/ready` forces hooks to exist at all, and a hook-less image cannot accept `runHookPayload`. Since P2 it is also the **platform-configuration** channel (see "Platform configuration delivery" below). | +| `/validate` | **P2** (construct sets `hooks.microvmImageHooks.validate`) | **P2** | An **image** (build-time) hook, and a **shallow self-check only**: server alive, hook routes registered, interpreter + contract sanity. It runs under the BUILD role, which deliberately holds no Bedrock / Secrets Manager / DynamoDB grants, so it must make **zero AWS API calls** and must not touch credential resolution — the "deeper warm-up assertions (Bedrock reachability, Memory access, tool availability)" this ADR originally assigned here are **not implementable**: every one of them would `AccessDenied` and fail every image build. They belong to the first task's own error handling. 200 when the checks pass, 503 while still initialising. | +| `/terminate` | **P2** (construct sets `hooks.microvmHooks.terminate`) | **P2** | Best-effort final flush: a final structured log line, then 200 — always, inside the hook budget, even with nothing running. It must **not** write terminal task status (the orchestrator finalizes the task and *then* calls `TerminateMicrovm`, so a terminate hook that wrote a status would race that finalization and could clobber the real outcome) and must not join the pipeline thread. There is nothing buffered to flush: `_ProgressWriter` does a synchronous `put_item` per event, so durability is per-write. "Always 200" also covers the BODY: the handler reads the raw request rather than a typed model, because a typed body is validated before the handler runs and would answer 422 to malformed JSON — a reported hook failure on a successful teardown. Safe to declare because `TerminateMicrovm` removes the VM with or without in-guest cooperation. **Correction (P2-F8):** the service sends `microvmId: ""` on this hook, unlike `/run` where it is populated, so an empty id is expected-normal and this hook cannot join the guest record to the control-plane one — `/run`'s accepted line carries that correlation instead. | +| `/suspend`, `/resume` | **P3** | **P3** | Declaring a runtime hook the agent does not serve fails the corresponding lifecycle transition, so each is declared only in the phase that implements it. P1 termination is the orchestrator's `TerminateMicrovm`, which needs no in-guest cooperation. | Consequence to state plainly, replacing the original "a P1-built MicroVM image is not runnable end to end": **a P1 image is creatable, launchable and payload-deliverable, but carries no smoke-parity guarantee.** P1 delivers the strategy, the construct, the roles/buckets/connectors, the image resource, the packaging script, and the `/ready` + `/run` endpoints — so a `lambda-microvm` task can start a MicroVM and hand it a payload. What P1 has **not** established is anything P2 owns: AgentCore Memory grants and `MEMORY_ID` delivery, the agent's non-secret env parity inside the snapshot, egress specifics from a running MicroVM, and heartbeat/progress behaviour end to end. No clone → change → PR run has happened on this substrate. P2 ("smoke parity") is the phase that closes that gap. The construct and the packaging script both surface exactly this at synth/run time (`abca:microvm-image-p1-smoke-unverified`) so an operator cannot mistake a launchable substrate for a verified one. -**Payload delivery** reuses the ECS strategy's S3-pointer pattern, adapted to `runHookPayload` (**≤ 4 KB** — measured, see below): payloads that fit ride inline; the rest are uploaded by the strategy to a platform payload bucket (the ECS payload bucket pattern in `ecs-agent-cluster.ts`: orchestrator write access, compute-role read-only scoped to the bucket, lifecycle expiry on objects) with only the S3 URI in `runHookPayload` — the MicroVM **execution role** holds the read grant, exactly as the ECS task role does today. +**The `AWS::Lambda::MicrovmImage` L1 enforces the API's enums (P2-F2, live 2026-08-06).** This closes the one item P1 left explicitly open, and it closes it against the construct's own stated reasoning. CloudFormation's generated types make `cpuConfigurations[].architecture` and all four `hooks.*` fields plain strings and document no allowed values, from which P1 concluded that the CloudFormation surface takes a *hook path* while the API takes an `ENABLED`/`DISABLED` flag, and that both were correct for their own surface. CloudFormation refused the change set at **early validation** — the stack was never touched, so there was no rollback and no runtime symptom to trace back — on five values: + +``` +/aws/lambda-microvms/runtime/v1/run is not a valid enum value. Supported values: [DISABLED, ENABLED] + (at /Resources/…/Properties/Hooks/MicrovmHooks/Run) … and the same for Terminate, Ready, Validate +arm64 is not a valid enum value. Supported values: [ARM_64] + (at /Resources/…/Properties/CpuConfigurations/0/Architecture) +``` + +Three consequences. First, the CloudFormation surface is **identical** to the API surface, and the packaging script (`--cpu-configurations '[{"architecture":"ARM_64"}]'`, `--hooks '{"microvmHooks":{"run":"ENABLED",…}}'`) had it right all along. Second, the "CDK-managed (recommended)" bootstrap path was **non-functional** for the whole of P1 and P2 — the out-of-band `--create-image` script was the only working path — and no unit test, `cdk synth` or cdk-nag rule could see it, because the types accept any string. Third, **hook paths are not configurable on either surface**: the service calls fixed well-known routes (proved by the build and run logs, which POST to exactly the `/aws/lambda-microvms/runtime/v1/*` paths the agent serves), so the route constants in the construct are an agent-side cross-package contract ONLY and must never be sent as property values again. Also discharged in passing: the `microvmImageHooks` property name and nesting are correct — CloudFormation resolved `…/Hooks/MicrovmImageHooks/Ready` and objected only to its value. + +**A snapshot is only as warm as the pages touched before it was captured (P2-F5, live 2026-08-07).** This is the defect that stopped the P2 smoke run one step short of a pull request, and it is a property of the substrate rather than a bug in any one file. Every task failed at turn 0, reproducibly: + +``` +TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +``` + +The binary was fine — in the identical image, locally, `claude --version` answers `2.1.191 (Claude Code)` in under a second. It is a **225 MiB (236,305,136-byte) statically-linked ELF** that nothing had exec'd before the snapshot was taken, so on a guest restored ~50 s earlier the first `exec` had to fault all of those pages in from lazily-restored storage, and 10 s was not enough. `/ready` existed precisely so "the snapshot is taken with a warm server", and the snapshot was warm for uvicorn and stone cold for the binary that does all the work. + +Both halves of the fix are kept, because they answer different questions. `/ready` now **exec's the heavyweight binaries before returning 200** (`claude` required, `git`/`node` best-effort), which is the only mechanism that can make the shipped snapshot warm — and its own budget rises to 300 s, well inside the 3600 s build-hook window, because it now does work whose duration is a cold `exec`. Two structural rules keep that honest, because **per-command timeouts do not compose**: the required command runs FIRST with its own budget so no best-effort warm-up can starve the one that decides whether the snapshot is usable, and the best-effort ones then SHARE the remainder of a total warm-up ceiling that sits inside the hook budget with margin (240 s against 300 s). Without them, three commands at 120 s each would be 360 s — a fix for a runtime failure that produces a build failure instead — and a single hung optional command could hold up a 200 that the required warm-up had already earned. Separately, the version probe's timeout goes from 10 s to 60 s: a probe that exists to print a version string into a log line gains nothing from a tight bound and loses the whole task when it trips. The general rule this generalises to, and the reason it belongs in the ADR rather than only in a comment: **on this backend, a first-touch cost that other substrates pay during container start is deferred to the first task instead**, so anything large and lazily-loaded is a turn-0 hazard unless it is touched in `/ready`. + +**Payload delivery** reuses the ECS strategy's S3-pointer pattern, adapted to `runHookPayload` (**≤ 4 KB** — measured, see below): payloads that fit ride inline; the rest are uploaded by the strategy to a platform payload bucket (the ECS payload bucket pattern in `ecs-agent-cluster.ts`: orchestrator write access, compute-role read-only scoped to the bucket, lifecycle expiry on objects) with the S3 URI in `runHookPayload` in place of the payload itself — the MicroVM **execution role** holds the read grant, exactly as the ECS task role does today. Since P2 the hook body also carries `platform_config` in both branches, so `runHookPayload` is never *only* the URI (see the canonical shapes below). The cap is **4 096 bytes**, not the 16 384 the SDK documents. Measured exactly: 4 096 passes, 4 097 is rejected with *"Value at 'runHookPayload' failed to satisfy constraint: Member must have length less than or equal to 4096"*. Two consequences follow. First, the original threshold would have inlined every envelope between 4 097 and 16 384 bytes and had the service reject all of them. Second, and more structurally: **the S3-pointer path is now the dominant one, and inline is the exception.** A hydrated task payload (prompt + issue thread + repo context) essentially always exceeds 4 KB, so "small payloads ride inline" describes tiny repo-less prompts rather than the common case. The payload bucket is therefore not a rarely-exercised overflow valve but a required part of every normal task, which raises its lifecycle rule (`MICROVM_PAYLOAD_TTL_DAYS`) and the execution role's read grant from edge-case plumbing to load-bearing. +**Canonical wire shapes.** Three, and the producer (`lambda-microvm-strategy.ts`) emits exactly these: + +| Where | Exact shape | +|---|---| +| `runHookPayload`, inline branch | `{"agent_payload": {…}, "platform_config": {…}}` | +| `runHookPayload`, pointer branch | `{"agent_payload_s3_uri": "s3://…", "platform_config": {…}}` | +| the object at that S3 URI | `{…agent_payload fields…, "platform_config": {…}}` — the payload's own fields at the TOP level, with the config merged in beside them | + +Two asymmetries are deliberate and must not be "tidied" without changing both sides. First, **the S3 object is not the envelope**: the payload's fields sit at the top level (that is what P1 uploaded, before `platform_config` existed) rather than nested under an `agent_payload` key. Second, **`platform_config` is duplicated** on the pointer path — once beside the pointer, once inside the uploaded object. It costs a few hundred bytes and buys the property that the config is reachable whichever end of the fetch a reader looks at, which matters because it is the agent's only substitute for an env block. + +The agent's reader is deliberately more permissive than this contract: it also accepts an S3 object shaped like the envelope (`agent_payload` nested), and `platform_config` present in only one of the two places (the fetched object wins, the hook body is the fallback). Those are **defensive compatibility** for the independent deploy cadences of a snapshot image and the orchestrator Lambda — a tolerant reader, not an alternative contract. A producer must emit the three shapes above. + +**Platform configuration delivery (P2): payload-sourced, allowlisted, fail-closed.** The other two backends hand the agent its non-secret platform env at launch — AgentCore Runtime env vars, ECS container overrides — and there is no equivalent on this substrate: a MicroVM starts from a **snapshot**, so its process environment is whatever was frozen at *image build* time and is then replayed by every MicroVM launched from that image version. Baking the deployment's identifiers into the snapshot would make them **version-frozen**: a redeploy that renames a table, adds a bucket or rotates the session role would leave every existing image version describing a deployment that no longer exists, and the drift would surface as a task-time `ResourceNotFound` rather than a deploy-time error. So the values travel with the task instead: `platform_config` is a SIBLING of the payload — beside `agent_payload` in the inline branch, beside `agent_payload_s3_uri` in the pointer branch, and merged in beside the payload's own fields inside the S3 object (the canonical shapes above give each one exactly) — whose snake_case keys the agent installs into `os.environ` as their UPPER_SNAKE equivalents. A payload value therefore **wins** over any pre-existing/image value — the orchestrator is describing the live deployment, the snapshot is describing a past one. `platform_config` carries **non-secret identifiers only** (table and bucket names, secret ARNs, the session-role ARN); secrets are still fetched at `/run` time from Secrets Manager using those ARNs, so the snapshot-must-stay-secret-free requirement above is untouched. Per-task fields — `memory_id` and friends — stay inside `agent_payload`: `platform_config` configures the *process*, `agent_payload` describes the *task*. + +Two rules make it safe. First, **the allowlist fails closed**: the agent installs a fixed set of keys and *rejects the entire run* (HTTP 400, nothing spawned, not one key installed) if the block carries anything else. These values become environment variables of the process that spawns the agent's tool subprocesses, so an unrecognised key is an attempt to set an arbitrary variable in the agent (`AWS_ENDPOINT_URL`, `LD_PRELOAD`, `PATH`, …) — an injection attempt, not a forward-compatibility gap, which is why unknown keys are refused rather than filtered out. Second, **installation happens before any credential or pipeline initialisation** on the hook path: the very next step reads `GITHUB_TOKEN_SECRET_ARN` to resolve the GitHub token and `AGENT_SESSION_ROLE_ARN` to scope the task's credentials, so installing later would silently resolve the whole task against the snapshot's frozen env. The one call that must precede installation is the S3 payload fetch (the config is *inside* the fetched object), which therefore runs on the ambient compute role via the attributed platform client — and it is the ONLY one: the same rule covers **logging**, so every `/run` log line before the install is stdout-only. The CloudWatch writer would otherwise resolve credentials and pin a boto3 default session (region included) off whatever a snapshot happened to bake, which is the build-hook defect one phase later. Nothing is lost — in the intended deployment there is no baked `LOG_GROUP_NAME`, so those lines would have gone to stdout anyway, and the reason for every pre-install rejection also travels in the structured 4xx/5xx body the service surfaces. A **required subset** (task table, task-events table, GitHub token secret ARN, session-role ARN) is rejected as `…_INCOMPLETE` when missing or blank — a distinct wire code from the `…_INVALID` allowlist rejection, because the remedies differ (deployment wiring vs. producer bug). A `/run` envelope with *no* `platform_config` at all is still accepted, loudly warned: the image snapshot and the orchestrator Lambda deploy on independent cadences, and a new image must not require a same-instant orchestrator. The key set is a cross-package contract in `contracts/constants.json` (`microvm_platform_config`), consumed by the agent's `/run` hook and produced by the orchestrator, with shape and required-subset invariants enforced by `scripts/check-constants-sync.ts`. + **No orchestrator→agent HTTP path exists in P1–P3**: payload arrives through the `/run` hook, all agent work is outbound, and therefore **no JWE auth tokens are minted at all** — token minting (and its ≤ 60 min TTL refresh problem) is deferred until a real consumer exists (e.g. operator shell access, [#391](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/391)). The `endpoint` stays in the `SessionHandle` because it is genuinely per-session state that becomes load-bearing the day such a consumer appears. But note the service does not agree by default: omitting `ingressNetworkConnectors` on `RunMicrovm` attaches a **public** `HTTP_INGRESS` connector, so the strategy passes the Lambda-managed `NO_INGRESS` connector explicitly on every launch (see sub-decision 4's security table). **Constraint accepted:** the configured **baseline is 8 GiB RAM / 4 vCPU** and the service scales vertically to a **32 GiB / 16 vCPU peak** on its own, with 32 GB of disk. So capacity is baseline-priced with 4× burst headroom — good for the bursty compile-and-test shape of an agent task — but the SUSTAINED ceiling is still 32 GiB, so repos that motivated the 120 GB ECS sizing stay on `ecs`. What the construct configures (and validates) is the baseline; the peak is not something a deployment asks for. -Normative requirements (EARS). All of these are P1 now — the earlier P1/P2 split of this list existed only because the phasing table split declaring a hook from serving it, which the service does not permit: +Normative requirements (EARS). **Each requirement's own `(Pn)` tag is authoritative**; there is no blanket phase for the list. The tags are per-requirement because the original list *was* split P1/P2 on the assumption that a hook could be declared in one phase and served in a later one — which the service does not permit (see the phasing table above), so the hook-serving requirements collapsed into P1 while the P2 items below arrived with the P2 hooks and `platform_config`: - (P1) The image build shall not embed secrets, tokens, or per-task identity in the snapshot. -- (P1) If the task payload exceeds the 4 KB `runHookPayload` limit, then the strategy shall upload the payload to the platform payload bucket and pass only its S3 URI in `runHookPayload`. +- (P1) If the task payload exceeds the 4 KB `runHookPayload` limit, then the strategy shall upload the payload to the platform payload bucket and pass its S3 URI in `runHookPayload` in place of the payload (from P2, alongside `platform_config`). - (P1) The MicroVM execution role shall hold read-only access to the payload bucket, scoped to that bucket. - (P1) Where no ingress is configured for a deployment, the strategy shall pass the Lambda-managed `NO_INGRESS` network connector on every `RunMicrovm` call (the field shall not be omitted). - (P1) Where the image enables any MicroVM lifecycle hook, the image shall also enable the `/ready` hook and the agent shall serve it. @@ -181,6 +229,16 @@ Normative requirements (EARS). All of these are P1 now — the earlier P1/P2 spl - (P1) The agent shall resolve credentials at `/run` time. - (P1) Where a deployment configures a MicroVM image before smoke parity is verified, the platform shall warn that the backend has no smoke-parity guarantee. - (P2) Where the image declares the `/validate` build hook, the agent shall serve it. +- (P2) The `/validate` hook shall make no AWS API calls. +- (P2) When the `/run` hook receives `platform_config`, the agent shall install only allowlisted keys into the environment before pipeline initialization. +- (P2) If `platform_config` carries a key that is not on the allowlist, then the agent shall reject the run with a 400 and shall install none of the block's keys. +- (P2) If a required `platform_config` key is missing, then the agent shall reject the run with a 400. +- (P2) Where a `platform_config` value and an image-baked environment value disagree, the agent shall use the `platform_config` value. +- (P2) Until `platform_config` is installed, the `/run` hook shall make no AWS API call other than the payload fetch, and shall log to stdout only. +- (P2) Where the image declares the `/terminate` hook, the agent shall return 200 within the hook budget for any request body — including a malformed, empty or absent one — and shall not write terminal task status. +- (P2) When the `/ready` hook runs, the agent shall exec the agent CLI binary before returning 200, so that its pages are resident when the snapshot is captured. +- (P2) If a required `/ready` warm-up does not complete successfully, then the agent shall report not-ready (HTTP 503) rather than allow the snapshot to be taken. +- (P2) The `/ready` hook shall make no AWS API call, warm-up included. ### 4. Infra and IAM: conditional resources behind bootstrap `ComputeTypes` @@ -188,12 +246,78 @@ Mirroring the ECS pattern: a `compute-lambda-microvm` bootstrap policy (`cdk/src Two networking facts the construct has to encode, both established live: -- **A `VPC_EGRESS` connector requires an operator role.** CloudFormation's generated L1 types `operatorRole` as optional and this ADR originally assumed Lambda would manage the ENIs with its own service-linked role. It does not: the connector fails to create with *"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type"*. The construct creates one role — trusting `lambda.amazonaws.com` with `aws:SourceAccount` pinned, carrying `AWSLambdaVPCAccessExecutionRole` plus the ENI / tag / private-IP actions that policy omits — and shares it across both connectors, since it manages interfaces rather than traffic. +- **A `VPC_EGRESS` connector requires an operator role.** CloudFormation's generated L1 types `operatorRole` as optional and this ADR originally assumed Lambda would manage the ENIs with its own service-linked role. It does not: the connector fails to create with *"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type"*. The construct creates one role — trusting the bare `lambda.amazonaws.com` service principal (see the trust-policy fact below), carrying `AWSLambdaVPCAccessExecutionRole` plus the ENI / tag / private-IP actions that policy omits — and shares it across both connectors, since it manages interfaces rather than traffic. +- **The MicroVM-facing roles cannot carry a confused-deputy source condition.** All three (build, execution, connector operator) trust the bare `lambda.amazonaws.com` service principal with **no** `aws:SourceAccount` / `aws:SourceArn`, and that is a forced choice, not an oversight: the Lambda MicroVMs service presents no source key when it assumes them, so a trust policy carrying one is unassumable. Two symptoms of the one cause, both live 2026-08-06/07 and both blocking: + + - Both `AWS::Lambda::NetworkConnector` resources `CREATE_FAILED` **deterministically** (on a freshly deleted stack, so not propagation lag — which matters, because *"The service is unable to assume the provided NetworkConnectorOperatorRole. Please verify the trust policy on the role."* is also the classic propagation symptom and a re-run is the obvious wrong guess). Removing the condition → both created within a second. + - `RunMicrovm` failed with a **misleading `iam:PassRole` AccessDenied on the caller**, with the orchestrator's grant present, `simulate-principal-policy` returning `allowed`, no permissions boundary, and a temporary *unconditioned* `iam:PassRole` **also** denied. The real cause was the execution role's trust; removing its conditions made the next submission reach `RUNNING` in 6 s. So the service reports a role it cannot pass-and-assume as an identity-policy denial on the principal passing it. + + Recorded plainly because the fix looks like a regression to anyone applying the standard service-principal pattern — and because it *was* a regression in the other direction: P1's standalone-validated operator-role probe had no conditions and worked, and the P1 F2 fix then added them "to mirror the build/execution roles". `sts:TagSession` stays: the service needs both actions and it was never implicated. + +- **Neither can the `iam:PassRole` grants carry an `iam:PassedToService` condition — same root cause, identity side (P2r2-F9 + P2r2-F10, live 2026-08-07 run 2).** An earlier revision of this ADR recorded the opposite, that the identity-side condition "was exonerated" by run 1's elimination. **That was a false negative**, and its cause is worth recording because it is a general trap: run 1 tested the conditioned grant by *adding* a temporary unconditioned `iam:PassRole` and watching the task still fail — but the temporary grant remained attached through the later submissions that succeeded, so the conditioned grant was never once tested against a working trust policy. A contaminated control. + + Run 2 ran the clean experiment — same exact-ARN resource, same ~5-minute IAM settle, one variable. It removed the run-1 workaround **first** (submission 4: denied) and only then added the unconditioned grant back on the same resource (submission 5: `RUNNING`), which is the ordering run 1 got wrong: + + | Orchestrator `iam:PassRole` on the execution role | Result | + |---|---| + | exact ARN **+ `iam:PassedToService: lambda.amazonaws.com`** | **DENIED** (two independent submissions) | + | exact ARN, **no condition** | **`RUNNING` in 9 s** | + + The denial lands on the **caller**, which is what makes it so misleading — the statement names that exact ARN and `simulate-principal-policy` answers `allowed`: + + ``` + User: arn:aws:sts:::assumed-role/backgroundagent-dev-TaskOrchestratorOrchestratorFn-… + is not authorized to perform: iam:PassRole on resource: + arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-… + because no identity-based policy allows the iam:PassRole action + ``` + + And the same key blocks the *other* PassRole path, which run 1 never reached because the enum defect (P2-F2) stopped it earlier: CloudFormation could not pass the **build role** at `CreateMicrovmImage` under the bootstrap `infrastructure` policy's allowlisted `IAMPassRole`. Verbatim, so the diagnosis does not have to be taken on trust: + + ``` + LambdaMicrovmComputeImage… CREATE_FAILED + User: arn:aws:sts:::assumed-role/cdk-hnb659fds-cfn-exec-role--us-east-1/AWSCloudFormation + is not authorized to perform: iam:PassRole on resource: + arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeBuildRoleF0-… + because no identity-based policy allows the iam:PassRole action + (Service: LambdaMicrovms, Status Code: 403) + ``` + + Three pieces of evidence pin that to the *condition* rather than to a stale bootstrap or a wrong resource pattern: + + 1. the live `IaCRole-ABCA-Infrastructure` policy was byte-identical to this branch's `cdk/bootstrap/policies/infrastructure.json`, so `cdk bootstrap --force` would have changed nothing; + 2. `aws iam simulate-principal-policy --policy-source-arn --action-names iam:PassRole --resource-arns ` returned `allowed` **with** `--context-entries ContextKeyName=iam:PassedToService,ContextKeyValues=lambda.amazonaws.com,ContextKeyType=string` and `implicitDeny` with no context entry — so the resource pattern matches and the condition key is the only remaining variable; + 3. the **control**: the out-of-band `create-microvm-image` call passed the *same build role* to the *same service* successfully, using operator credentials that carry no such condition. The role's trust is therefore fine and the denial is genuinely caller-side. + + So: **the Lambda MicroVMs service presents no usable value for `iam:PassedToService` on either PassRole path** (CloudFormation → build role at `CreateMicrovmImage`; orchestrator → execution role at `RunMicrovm`), exactly as it presents no `aws:SourceAccount` on the assume-role path. One root cause, two more symptoms. Both statements therefore drop the condition, and the fix is deliberately asymmetric so it stays contained: + + - `task-orchestrator.ts` sid `MicrovmPassExecutionRole` — condition removed; the **exact execution-role ARN** is now the whole of the scoping, which is why that resource must never be relaxed to a prefix or `*`. + - a new sid `MicrovmPassRoles` in the **conditional** `compute-lambda-microvm` bootstrap policy — unconditioned `iam:PassRole` on the build- and connector-operator role **name prefixes only** (not the execution role, which CloudFormation never passes). The shared `infrastructure` `IAMPassRole` keeps its allowlist, so no other role in the stack loses that constraint, and an agentcore-only bootstrap never gains an unconditioned pass at all. **Operators must re-bootstrap** (bundle ≥ 1.4.0) for the CDK-managed image path to work. + + If AWS documents the value the service does present, adding it to both statements restores the condition. `microvms.lambda.amazonaws.com`, `lambda-microvms.amazonaws.com` and `microvms.amazonaws.com` were all `implicitDeny` against the conditioned policy, so any one of them would serve as the allowlist entry if it turns out to be right. Note that CloudTrail carries **no `lambda-microvms` management events at all** today, so the value cannot be read out of a log — only confirmed by AWS or found by a bounded sweep. + + Compensating controls, enumerated per role. The deployment-role's shared `IAMPassRole` grant is **name-prefix-scoped**, so it technically covers all three roles; in practice only the orchestrator actively invokes `iam:PassRole` on the execution role (CloudFormation never requests it). The other two roles are passed **to** the deployment role (not to themselves). + + | Role | Who can pass it, and how that grant is scoped | + |---|---| + | **Execution role** | The **orchestrator Lambda only**, at `RunMicrovm` — `iam:PassRole` scoped to this role's **exact ARN**, no condition (`constructs/task-orchestrator.ts`, sid `MicrovmPassExecutionRole`). The referenced comment contains the authoritative two-arm experiment evidence that the condition is the true blocker (not a permissions gap or stale bootstrap). | + | **Build role** | The **CloudFormation deployment role**, at `CreateMicrovmImage` (the L1's `buildRoleArn`) — via the new `MicrovmPassRoles` statement, scoped to `role/backgroundagent-dev-LambdaMicrovmComputeBuild*`, no condition. Also whoever runs `package-microvm-artifact.sh --create-image` out of band, using their own credentials. | + | **Connector operator role** | The **CloudFormation deployment role**, at `AWS::Lambda::NetworkConnector` create/update (`operatorRole`) — the same statement, scoped to `role/backgroundagent-dev-LambdaMicrovmComputeConnector*`. | + + The rest of the posture: every resource these roles can reach is account-scoped by ARN **except two deliberate `Resource: '*'` statements** — `ec2:DescribeAvailabilityZones` on the execution role (EC2 describe actions have no resource-level scoping; read-only, no mutation, no data access, needed so a CDK target repo's `cdk synth` build gate can resolve AZ context on a fresh clone) and the connector operator role's ENI/tag/private-IP statement (`CreateNetworkInterface` is authorized before the ENI exists and the `Describe*` calls take no resource, which is why the AWS-managed VPC-access policy uses `*` too). Both are justified in the construct's cdk-nag `AwsSolutions-IAM5` suppressions, which is where a reviewer should check them rather than here. The Logs grants are prefix-scoped (`/aws/lambda-microvms/*` plus one named log group), i.e. wildcards inside a namespace, not `*`. Separately, the **orchestrator's** `lambda:PassNetworkConnector` is also `Resource: '*'` and unavoidably so — the AWS-managed connectors live in the `aws` account, outside any ARN we could enumerate (justified in `task-orchestrator.ts`, sid `MicrovmPassNetworkConnector`). Finally: none of the three roles holds `iam:*`, none has cross-account trust, and the only `sts:AssumeRole` any of them has is the execution role's, scoped to the per-task SessionRole. + + If AWS later populates a source key on this path, adding it to the shared principal fixes all three roles and both `sts` actions at once. + - **Build-time egress needs port 80; runtime does not.** `agent/Dockerfile` installs Debian packages and `apt-get` fetches over plain HTTP, so a 443-only egress path fails every snapshot build (`Could not connect to deb.debian.org:80 … exit code: 100`). Rather than widen the runtime posture, the construct provisions a **second, build-only** connector on the same private subnets with a 443 + 80 security group, referenced solely by the image resource and the packaging script. The agent at run time still has 443-only egress. - Where the bootstrap `ComputeTypes` parameter includes `lambda-microvm`, the generated template shall attach the `IaCRole-ABCA-Compute-LambdaMicrovms` policy to the CloudFormation execution role. - The orchestrator role shall receive only the MicroVM lifecycle actions it calls (`lambda:RunMicrovm`, `lambda:SuspendMicrovm`, `lambda:ResumeMicrovm`, `lambda:TerminateMicrovm`, `lambda:GetMicrovm` for `pollSession`, and `lambda:PassNetworkConnector`, which is required even for the default connectors), scoped to platform-created images. - Where the `lambda-microvm` backend is enabled, the approve and deny Lambdas shall receive `lambda:ResumeMicrovm` and `lambda:GetMicrovm` — conditionally, mirroring the cancel handler's conditional `RUNTIME_ARN` wiring in `task-api.ts`. +- The trust policy of every MicroVM-facing role shall name `lambda.amazonaws.com` and shall carry no source-condition key (the service presents none; see the trust-policy fact above). +- The `iam:PassRole` grant the orchestrator uses for the MicroVM execution role shall carry no `iam:PassedToService` condition and shall be scoped to that role's exact ARN. +- Where the bootstrap `ComputeTypes` parameter includes `lambda-microvm`, the `IaCRole-ABCA-Compute-LambdaMicrovms` policy shall grant `iam:PassRole` without an `iam:PassedToService` condition, scoped to the MicroVM build- and connector-operator role name prefixes, and shall not extend that grant to the MicroVM execution role. +- The shared `IaCRole-ABCA-Infrastructure` `iam:PassRole` statement shall retain its `iam:PassedToService` allowlist. +- The MicroVM execution role shall hold `logs:CreateLogStream` and `logs:PutLogEvents` on the application log group whose name is delivered in `platform_config`, scoped to that log group. `lambda:CreateMicrovmAuthToken` is granted to no role in P1–P3 (no JWE consumer exists; see sub-decision 3). @@ -209,7 +333,10 @@ Two networking facts the construct has to encode, both established live: | Egress, image build | ECR build outside the platform VPC | ECR build outside the platform VPC | Platform VPC via a **separate build-only connector, TCP 443 + 80** (`apt-get` is plain HTTP) | New surface: build-time egress is wider than runtime egress by one port, on a connector no running MicroVM can use | | Tenant-data scoping | Per-session role (`admitComputeRole`) | Per-session role | Per-session role, execution role admitted identically | None | | Secrets delivery | Runtime env + Identity injection | Task env vars | Fetched at `/run`; never in snapshot | New surface: snapshot must stay secret-free (EARS req., sub-decision 3) | +| Non-secret platform config (table/bucket names, secret + role ARNs) | Runtime env vars | Task env vars | `platform_config` in the `/run` payload, installed into the process env | New surface: the values are attacker-relevant *as env vars* (`LD_PRELOAD`, `AWS_ENDPOINT_URL`), so the agent installs a fixed **allowlist** and rejects the whole run on any other key (EARS req., sub-decision 3) | | Inbound exposure | None (SigV4 invoke only) | None (no endpoint) | **None — but only because the strategy passes `NO_INGRESS` explicitly.** The service default is a PUBLIC `HTTP_INGRESS` connector plus a public `*.lambda-microvm..on.aws` endpoint; no tokens are minted in P1–P3 either way | New surface **and** a new failure mode: "no inbound" is an active control, not an absence. Drop the `NO_INGRESS` argument and every agent MicroVM gets a public endpoint (EARS req., sub-decision 3) | +| IAM condition keys on the compute-role trust **and** on the `iam:PassRole` grants that hand it over | Trust pinned with `aws:SourceAccount`; `PassRole` under the allowlisted bootstrap statement | Trust pinned per-service; `PassRole` under the allowlisted bootstrap statement | **Neither is possible.** All three MicroVM-facing roles trust the bare `lambda.amazonaws.com` with no `aws:SourceAccount`/`aws:SourceArn`, **and** both `iam:PassRole` grants (orchestrator → execution role at `RunMicrovm`; CloudFormation → build role at `CreateMicrovmImage`) carry no `iam:PassedToService` — the service presents no usable value for any of those keys, and each condition is a hard blocker while present (live-verified, blocking, four times across two runs) | **Real, evidenced gap that does not close from our side, and it is wider than the trust policy alone.** `lambda.amazonaws.com` is shared with every other Lambda feature, so neither the account pin nor the passed-to-service pin is available on this path. Compensated per role (table in sub-decision 4): the **execution** role is passable by the **orchestrator only** (at `RunMicrovm`), restricted to its **exact ARN**; the **build** and **connector-operator** roles are passable by the CloudFormation deployment role under a new **conditional, per-backend, name-prefix-scoped** statement (`MicrovmPassRoles`, bootstrap ≥ 1.4.0) that deliberately excludes the execution role. The shared allowlisted `IAMPassRole` (`role/backgroundagent-dev-*`) is left intact to avoid widening the grant for ~30 other roles, so while it technically matches the execution role, only the orchestrator actively reaches for it. Resources are account-scoped by ARN apart from two justified `Resource: \'*\'` statements (`ec2:DescribeAvailabilityZones`; the operator role\'s ENI management — both carry cdk-nag IAM5 suppressions). No `iam:*`, no cross-account trust. Revisit if AWS ever documents the values the service presents; CloudTrail records no `lambda-microvms` events, so they cannot be read from logs | +| Per-task observability writes | Runtime writes to the vended APPLICATION_LOGS group | Task role writes to the task log group | Execution role writes to the SAME APPLICATION_LOGS group, granted against the group `platform_config` names (P2-F4) | None — but only after P2-F4: the name was delivered a phase before the grant, so the agent attempted the write and every per-task line (and `METRICS_REPORT`) was `AccessDenied`, degrading silently to guest stdout | | Session isolation | MicroVM | Task-level | MicroVM (Firecracker) | None (≥ ECS) | | State reuse | None | None | Snapshot shared across MicroVMs | New surface: CSPRNG reseed + credential refresh on `/run`/`/resume` (EARS req.) | | Workload-token injection | Yes (Runtime-coupled) | No (env-var posture) | No (env-var posture) | Shared with ECS; deferred to [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 | @@ -234,7 +361,7 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor ### 5. Rollout: phased, default unchanged - **P1 — strategy + infra + minimal hook serving:** `LambdaMicrovmComputeStrategy` (start/poll/stop), CDK construct, bootstrap policy, types sync, unit + CDK assertion tests, and the agent's `/ready` + `/run` endpoints. No suspend yet. The image IS creatable and launchable and the payload DOES reach the agent — but there is **no smoke-parity guarantee** (sub-decision 3's phasing table). -- **P2 — smoke parity:** the agent serves the remaining hooks (`/terminate`, `/validate`); agent completes clone → change → PR on the backend with progress visible to `bgagent watch`; failure classification entries in `error-classifier.ts`; **AgentCore Memory parity** (IAM grant + `MEMORY_ID` delivery, following the `EcsAgentCluster` pattern — Memory is a standalone service already consumed cross-substrate, and omitting the grant silently no-ops cross-session learning); the agent's remaining non-secret env parity inside the snapshot. +- **P2 — smoke parity:** the agent serves `/terminate` + `/validate` and installs its platform env from the `/run` payload (see sub-decision 3's "Platform configuration delivery"); agent completes clone → change → PR on the backend with progress visible to `bgagent watch`; failure classification entries in `error-classifier.ts`; **AgentCore Memory parity** (IAM grant + `MEMORY_ID` delivery, following the `EcsAgentCluster` pattern — Memory is a standalone service already consumed cross-substrate, and omitting the grant silently no-ops cross-session learning); the agent's remaining non-secret env parity inside the snapshot. - **P3 — suspend/resume:** the interface widening from sub-decision 1 (mandatory methods, all three strategies in one commit), HITL-wait suspend policy, inline resume in the approve/deny Lambda with orchestrator-poll reconciliation (sub-decision 2), timeout-under-freeze wall-clock handling; coordinate with [#491](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/491)'s unified liveness model and update Cedar decision #7's rationale note. - **Out of scope:** replacing AgentCore as default; classic Lambda functions as a runtime; GPU; the Runtime-coupled workload-access-token injection path (delivery mechanism exists only on AgentCore Runtime; MicroVMs adopt the ECS env-var posture until [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 redesign the seam). Gateway integration is orthogonal: ADR-019/[#641](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/641) is substrate-portable by design and applies to this backend when it lands. @@ -251,8 +378,10 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor - (−) **8-hour hard cap includes suspended time**, and with `idlePolicy` omitted there is no tighter substrate-level suspended-TTL — the suspended-state bound is `maximumDurationInSeconds` plus orchestrator termination and the stranded reconciler. A manually suspended VM was observed alive at 1 h with no TTL in sight (observation truncated there), so nothing contradicts this bound, but nothing narrows it either. Under today's 1 h gate ceiling it is comfortably sufficient; any future extension of gate ceilings must revisit the bound (an additive `idlePolicy` change) and give the orchestrator a checkpoint-and-restart path (push branch, new session) beyond the cap. - (!) **Idle-policy foot-gun.** Traffic-based auto-suspend would freeze a busy outbound-only agent; the decision to disable auto-suspend must be enforced in code and covered by tests, not left to configuration discipline. - (!) **Service defaults are not the desired posture.** Two live-caught cases (public `HTTP_INGRESS` by default; `/ready` mandatory) mean an omitted field on this backend does not mean "off" — it can mean "the service picks, and it picks wider than we want". Every new `RunMicrovm` / `CreateMicrovmImage` field should be assumed to have an opinionated default until checked. -- (!) **Nothing self-terminates.** A MicroVM whose hook never ran still reaches `RUNNING` and stays there, billing, until the 8 h cap. The orchestrator's `TerminateMicrovm` on finalize is the only cleanup, so a leaked handle is a cost incident, not just an untidy state. +- (!) **Nothing self-terminates on the paths that matter** — superseding P1's unqualified version of this bullet. With `run: ENABLED` the service DOES reap a VM whose run hook returns 4xx (~12 s, `stateReason: "Run lifecycle hook returned HTTP status 400."`, live-verified), so a guest that rejects its own payload cleans itself up. That is the only self-cleaning case: the service reaps a hook *result*, and once `/run` has answered 200 it has no view of the guest. A VM whose task finished, crashed after `/run`, or hung stays `RUNNING` and billing until the 8 h cap, so the orchestrator's `TerminateMicrovm` on finalize remains the only cleanup for normal operation and a leaked handle is still a cost incident. - (!) **Snapshot uniqueness.** Shared memory snapshots require CSPRNG reseeding and credential refresh in `/run` / `/resume` hooks; missing this is a silent security defect. +- (−) **The agent stack template is at 98.6 % of CloudFormation's 1 MB limit** (985,886 bytes) and 486 of 500 resources with a MicroVM image configured — ~14 KB of headroom, i.e. roughly one more construct, and down from 98.4 % / ~16 KB one run earlier. Not caused by this backend (the MicroVM construct is ~6 KB of it) but reached by it, and it will block deploys for reasons that have nothing to do with MicroVMs. Tracked in [#735](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/735); the candidate remedies are `suppressTemplateIndentation` and a stack split. +- (!) **Snapshot WARMTH is a first-class property, not an optimisation.** A snapshot inherits only the pages something touched before it was captured, so a large lazily-loaded artifact — the 225 MiB `claude` binary, and anything similar added later — pays its first-touch cost on the *first task* instead of at container start. That cost failed every task at turn 0 in the P2 smoke run (P2-F5). Anything heavyweight added to the image must be exec'd in `/ready`, and any timeout guarding a first touch must be sized for a cold page fault rather than for the work itself. - (!) **Regional availability (5 regions at launch, expanding)** — enforced in layers (synth-time static check, onboarding + doctor live probes, orchestration-time classification; see sub-decision 4). The static CDK constant is the one piece that rots as AWS expands; its update path and context-flag escape hatch are deliberate. - (!) **Workload-token injection delta persists** (shared with the ECS backend) until [#249](https://github.com/aws-samples/sample-autonomous-cloud-coding-agents/issues/249)/ADR-016 land; document it in the security bar comparison rather than blocking on it. Memory and Gateway are explicitly *not* deltas — both are standalone services consumed via IAM from any substrate. @@ -261,15 +390,21 @@ Lambda MicroVMs launched in 5 regions (us-east-1/2, us-west-2, eu-west-1, ap-nor **P1 (start/poll/stop — no suspend):** - Unit tests for the strategy: start/poll/stop mapping (including `SessionStatus` `'suspended'` reported mechanically, without task-state interpretation), payload-size branching (inline vs S3 pointer, at the exact 4 096/4 097-byte boundary), the image-identifier-must-be-an-ARN guard, the explicit `NO_INGRESS` argument (including the blank-env-var fallback, which must never omit the field), error classification (`ServiceQuotaExceededException`, `ThrottlingException`, `ResourceNotFoundException`, regional-unavailability), the omit-`idlePolicy` invariant, and `maximumDurationInSeconds` fixed at 28 800. -- Agent tests: `/ready` returns 200 once the server is up and starts nothing; `/run` accepts both envelope shapes (inline and S3 pointer), starts the pipeline asynchronously through the same mapper `/invocations` uses, returns before the pipeline finishes, and rejects every unusable envelope with a named code before spawning; `/validate`, `/suspend`, `/resume` and `/terminate` are NOT served. +- Agent tests: `/ready` returns 200 once the server is up and starts nothing; `/run` accepts both envelope shapes (inline and S3 pointer), starts the pipeline asynchronously through the same mapper `/invocations` uses, returns before the pipeline finishes, and rejects every unusable envelope with a named code before spawning; `/suspend` and `/resume` are NOT served (`/validate` and `/terminate` joined the served set in P2). - Orchestrator tests: substrate-terminal + non-terminal task status → failed classification; `suspended` + non-`AWAITING_APPROVAL` status → anomaly event, no fail-fast; `compute_metadata` persisted with `microvmId`/`endpoint` after `startSession`. - CDK assertions: MicroVM resources present only when `ComputeTypes` includes the backend; synth failure for unsupported regions (plus the context-flag escape hatch); memory size validated against the accepted list at synth; the connector operator role and its trust; two connectors with the build-only one carrying port 80 and the runtime one not; the `/ready` + `/run` hook declaration and the absence of the others; IAM actions scoped as specified (orchestrator lifecycle set; no `CreateMicrovmAuthToken` anywhere); payload-bucket grants (execution role read-only); backend cost-allocation tags; types-sync check covers the widened `ComputeType`. - CLI tests: onboarding rejection with remedy when the availability probe fails; doctor check present when a blueprint selects the backend. -- P1 verification items (external service facts) — **executed 2026-07-31, us-east-1**; see `docs/verification/645-p1-lambda-microvm-runbook.md` for the full evidence. Discharged: `runHookPayload` limit (**4 096**, not 16 KB), the accepted baseline memory sizes (`[512…8192]` MiB — note the developer guide, not the probe, is what establishes that this is a BASELINE with a 32 GiB peak), image-identifier ARN requirement, IAM action names and the observed image-ARN shape, region probe behaviour, manual suspend/resume without `idlePolicy`, terminate timing and the `TERMINATED`-persists-≥10-min finding, the default public `HTTP_INGRESS`, and the `/ready` requirement. **Not** discharged: account-quota treatment of `SUSPENDED` MicroVMs (not observable safely), suspended TTL beyond 1 h (truncated), the vertical-scaling behaviour itself (no workload here approached the baseline, so the 4× peak is documented rather than observed), and the `AWS::Lambda::MicrovmImage` CloudFormation value shapes (never exercised — the run used the out-of-band script path). Record the closed answers in COMPUTE.md. +- P1 verification items (external service facts) — **executed 2026-07-31, us-east-1**; see `docs/verification/645-p1-lambda-microvm-runbook.md` for the full evidence. Discharged: `runHookPayload` limit (**4 096**, not 16 KB), the accepted baseline memory sizes (`[512…8192]` MiB — note the developer guide, not the probe, is what establishes that this is a BASELINE with a 32 GiB peak), image-identifier ARN requirement, IAM action names and the observed image-ARN shape, region probe behaviour, manual suspend/resume without `idlePolicy`, terminate timing and the `TERMINATED`-persists-≥10-min finding, the default public `HTTP_INGRESS`, and the `/ready` requirement. **Not** discharged: account-quota treatment of `SUSPENDED` MicroVMs (not observable safely), suspended TTL beyond 1 h (truncated), the vertical-scaling behaviour itself (no workload here approached the baseline, so the 4× peak is documented rather than observed), and the `AWS::Lambda::MicrovmImage` CloudFormation value shapes (never exercised — the run used the out-of-band script path; **discharged, and REFUTED, by the P2 run — see P2-F2 in sub-decision 3**). Record the closed answers in COMPUTE.md. **P2 (smoke parity):** -- Smoke (gated like the ECS backend): clone → change → PR with `bgagent watch` progress; Memory write parity (no AccessDenied no-op). +- Agent tests: `/validate` returns 200 with its individual check results, 503 while initialising, reports a missing hook route / unsupported interpreter, starts nothing, and makes **zero AWS calls even with `LOG_GROUP_NAME` set** (asserted by poisoning the boto3 and CloudWatch-writer seams — the same assertion covers `/ready`); `/terminate` returns 200 with no body at all, with a malformed / non-object / wrong-content-type / whitespace-only body, when the body read itself fails, with a pipeline still running (without joining it), and when its own best-effort step raises — and never calls `task_state.write_terminal`; a structural assertion that the route carries no typed body param keeps the 422 from being reintroduced. +- `platform_config` tests: the allowlist and required subset are read from `contracts/constants.json` (the wire key set is additionally asserted literally, as the agent-side tripwire on a contract edit); an unknown key rejects the whole block with nothing installed; a non-object block and a non-string value are rejected; a blank/`null` optional value is skipped without clobbering an image value while a blank required value is rejected; a payload value beats a pre-existing env value; installation is observed to happen before the GitHub-token resolver runs and before any pipeline thread exists; the config is picked up from the inline envelope, from beside the S3 pointer, and from inside the fetched object (inner wins); an envelope with no `platform_config` is still accepted with a warning. +- Snapshot credential hygiene: a subprocess probe asserts that importing `server` and serving `/ready` + `/validate` imports neither `boto3` nor `botocore`, caches no `aws_session` session, and spawns no CloudWatch writer thread — the property that keeps a build-role credential chain and the build-time region out of the snapshot. +- `/run` pre-install silence: with a **baked `LOG_GROUP_NAME`** (the hostile case — without it the assertions pass vacuously) every AWS/credential seam (`boto3.client`/`Session`, the `aws_session` factories, `_debug_cw`/`_warn_cw`) is armed to raise until the install succeeds. Asserted on the accepted path, on all three rejection paths (bad envelope, `platform_config` invalid, `platform_config` incomplete) and on the failed-fetch 500 — where the seams stay armed for the whole request, because a rejected run installed nothing and so earns no AWS call. The permitted exception is asserted POSITIVELY: exactly one client is built pre-install, for `s3`, through the attributed factory. +- `/ready` warm-up tests (P2-F5): the hook exec's each configured binary exactly once with a generous timeout; `claude` is the only REQUIRED entry; a timeout, a missing binary, a non-zero exit and an unexpected `OSError` each produce **503 with the reason logged to stdout** rather than a 200 or a 500; a best-effort failure still reports ready; the warm-up makes zero AWS calls with `LOG_GROUP_NAME` baked. Plus the backstop half: the `claude --version` probe's bound is asserted to be ≥ 60 s and to be applied to the *exec* rather than to the PATH lookup, and a missing CLI warns instead of raising. +- CDK assertions (P2-F1/F2/F4): no source-condition key on any of the three MicroVM-facing role trusts, and no `aws:SourceAccount`/`aws:SourceArn` string anywhere in them; hook properties are `ENABLED` and the architecture is `ARM_64`, with a negative assertion that **no** hook route string appears anywhere in the rendered image resource; the agent hook routes are asserted against their own dedicated constant (the template no longer carries a path to compare); the execution role holds `logs:CreateLogStream`/`PutLogEvents` on the application log group and the two logs grants stay separate; the stack wires the SAME log group it delivers as `platform_config.log_group_name`. +- Smoke (gated like the ECS backend): clone → change → PR with `bgagent watch` progress; Memory write parity (no AccessDenied no-op). **Run 1 (2026-08-06) FAILED at `implement`, turn 0 — no PR. Run 2 (2026-08-07) PASSED: two tasks clone → change → commit → push → PR, `COMPLETED`, 12 turns / $0.279 / 153 s** (`docs/verification/645-p2-smoke-runbook.md`), which also discharged P2-F1, P2-F2, P2-F4, P2-F5 and the dual-signal-liveness item (45 s heartbeat cadence observed across a 181 s `RUNNING` window). **The row is not yet fully closed:** run 2 needed one live IAM workaround, and establishing why produced P2r2-F10 (the identity-side `iam:PassedToService`) and P2r2-F9 (its CloudFormation twin). Both are fixed in source above and neither has been re-exercised live, so what remains is a re-run on a re-bootstrapped account with no workarounds. **P3 (suspend/resume):** diff --git a/docs/verification/645-p1-lambda-microvm-runbook.md b/docs/verification/645-p1-lambda-microvm-runbook.md new file mode 100644 index 000000000..12d1d56dd --- /dev/null +++ b/docs/verification/645-p1-lambda-microvm-runbook.md @@ -0,0 +1,2198 @@ +# ADR-021 P1 Lambda MicroVM verification runbook + +Working verification document for issue #645 / PR #689 on branch +`feat/645-lambda-microvm-p1`. This is for a **new, CDK-bootstrapped sandbox +account in which ABCA has never been deployed**. It is not a production deploy +guide. + +`docs/scripts/sync-starlight.mjs` mirrors only selected guides, `docs/design/`, +`docs/decisions/`, `CONTRIBUTING.md`, and assets. It does **not** mirror +`docs/verification/`, so this file intentionally stays here. + +## ⚠️ This runbook predates the Stage D fixes + +The pass recorded under “Live execution results” below ran against the code as of +`0505f914`, and its findings (F1–F14) were then fixed. The **instructions** above +each step have been updated only where a fix renamed something an instruction +asserts on. The following instructions are known to still describe the pre-fix +world and must be adjusted by whoever runs this again: + +| Step | Stale instruction | Post-fix reality | +|---|---|---| +| 1.2 | ~~unescaped `ParameterValue=agentcore,lambda-microvm`; bootstrap without `--force`~~ | **CORRECTED IN PLACE** — the comma is now escaped (`agentcore\,lambda-microvm`) and both bootstrap invocations carry `--force`. F14/F10 are the evidence; nothing is left to adjust here | +| 2.2 | "a 443-only security group" and one `AWS::Lambda::NetworkConnector` | **two** security groups (443-only runtime, 443 + 80 build) and **two** connectors, plus a connector operator role; a seventh output `MicrovmBuildEgressConnectorArns` | +| 2.3 | `list-stack-resources --query "…|[0]"` | needs `--no-paginate` (F14) | +| 4.3 | `export IMAGE_VERSION=1` | the service returns `1.0`; there are two builds per version (one per chipset) | +| 5.1 / 5.9 | `--image-identifier "$IMAGE_NAME"` | an **ARN** is required (F3); use the `imageArn` the script prints | +| 5.9 | 16 384 / 16 385-byte probes | the real boundary is **4 096 / 4 097** (F6) | +| Phase 5 | "the hook-less P1 image" framing | the image now declares AND serves `/ready` + `/run`, so hook behaviour is a different experiment | + +## Important P1 and tooling limits + +- P1 provisions and can start the substrate, and — **as of the Stage D fixes** — + its image declares AND the agent serves `/ready` + `/run`, so the image is + creatable, launchable and payload-deliverable. What P1 has no guarantee of is + smoke parity (Memory grants, snapshot env parity, egress specifics from a + running MicroVM, heartbeats). *Pre-fix, this bullet read "not runnable end to + end" because the plan was to declare `/run` in P1 and serve it in P2; the live + run proved that is not a reachable service state (F1).* +- P1's orchestrator role intentionally has only `RunMicrovm`, `GetMicrovm`, + `TerminateMicrovm`, `PassNetworkConnector`, and the required `iam:PassRole`. + It does **not** have `SuspendMicrovm`, `ResumeMicrovm`, or + `CreateMicrovmAuthToken`. Therefore use the orchestrator role for the P1 IAM + checks when it can be assumed, but use the sandbox administrator identity for + the manual suspend/resume experiment. This is a deliberate P1/brief mismatch. +- The repository's AWS SDK model is + `@aws-sdk/client-lambda-microvms@3.1098.0`. It verifies operation names, + request keys, state enums, and `delete-microvm-image-version`. The local + `aws-cli/2.35.8` does **not** recognize `aws lambda-microvms`; consequently all + `aws lambda-microvms ...` commands below are **best-effort CLI spellings + derived from that SDK model and the repository packaging script**, not locally + CLI-validated. No minimum AWS CLI release containing this service could be + established. The executor must install a CLI build for which + `aws lambda-microvms help` succeeds. Do not proceed with image/lifecycle work + merely because `aws --version` is newer than 2.35.8. +- The packaging-script model drift is resolved: its direct service request now + uses SDK 3.1098.0's `ARM_64` architecture and `ENABLED|DISABLED` hook-state + shape (with port and timeout), rather than CloudFormation's `arm64` and hook + path strings. The CDK L1 remains intentionally unchanged because its generated + CloudFormation types accept string values and document no architecture/hook + allowed-value constraint. Step 4.2 still captures CLI help/input skeleton as + a live check because the local CLI cannot validate this service offline. +- Commands are run from the repository root. `mise` is primary. Commands marked + **raw fallback** are only for a machine without `mise`. +- **Run the whole thing under `set -o pipefail`.** Several steps pipe a command + through `tee`; without `pipefail` the pipeline reports `tee`'s exit status and a + failed command looks like a success. The 2026-07-31 pass recorded `EXIT=0` for a + `package-microvm-artifact.sh` run that had actually failed service validation + (F14). The script now also prints an explicit + `!! package-microvm-artifact.sh FAILED (exit N) !!` marker on any failure, so + the teed log carries the truth either way — but set the option anyway: + + ```bash + set -o pipefail + ``` + +## Variables and evidence directory + +**Purpose:** make every subsequent command target one account, Region, and stack. + +```bash +export AWS_REGION=us-east-1 +export AWS_DEFAULT_REGION="$AWS_REGION" +export CDK_DEFAULT_REGION="$AWS_REGION" +export STACK_NAME=backgroundagent-dev +export EXPECTED_BRANCH=feat/645-lambda-microvm-p1 +export EVIDENCE_DIR="/tmp/abca-645-p1-$(date -u +%Y%m%dT%H%M%SZ)" +mkdir -p "$EVIDENCE_DIR" +``` + +If using a profile, also `export AWS_PROFILE=`. Supported +Regions are `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and +`ap-northeast-1`; this runbook defaults to `us-east-1`. + +**Expected:** the directory exists and all variables print non-empty. + +**Record:** variable values and evidence-directory path. + +**ADR-021 item:** regional availability enforcement and reproducibility. + +--- + +## Phase 0 — Preflight + +### 0.1 Verify identity, branch, and virgin account + +**Purpose:** prevent deploying to the wrong account/branch and fail fast if this +is not the assumed first ABCA deployment. + +```bash +aws sts get-caller-identity | tee "$EVIDENCE_DIR/caller-identity.json" +export ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" +test "$(git branch --show-current)" = "$EXPECTED_BRANCH" +git status --short --branch | tee "$EVIDENCE_DIR/git-status.txt" + +if aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + >"$EVIDENCE_DIR/unexpected-existing-stack.json" 2>"$EVIDENCE_DIR/stack-absence.txt"; then + echo "STOP: $STACK_NAME already exists; this runbook requires a virgin account." >&2 + exit 1 +fi +``` + +The expected CloudFormation error is `ValidationError: Stack ... does not +exist`. Any other error (especially `AccessDenied`) is not proof of absence; +stop and fix credentials. + +**Expected:** account is the intended sandbox, branch test passes, only known +local files are shown, and `backgroundagent-dev` is absent. `CDKToolkit` may and +normally will exist. + +**Record:** account ID, caller ARN, branch SHA (`git rev-parse HEAD`), status, +and the exact stack-absence response. + +**ADR-021 item:** clean first-deploy/substrate bootstrap path. + +### 0.2 Record tools and install dependencies + +**Purpose:** capture the exact client/service models and prepare CDK and CLI. + +```bash +aws --version 2>&1 | tee "$EVIDENCE_DIR/aws-version.txt" +node --version | tee "$EVIDENCE_DIR/node-version.txt" +npx cdk --version | tee "$EVIDENCE_DIR/cdk-version.txt" +mise --version | tee "$EVIDENCE_DIR/mise-version.txt" +python3 --version | tee "$EVIDENCE_DIR/python-version.txt" +zip -v | tee "$EVIDENCE_DIR/zip-version.txt" +rsync --version | tee "$EVIDENCE_DIR/rsync-version.txt" + +MISE_EXPERIMENTAL=1 mise run install +``` + +**Raw fallback if `mise` is absent:** + +```bash +yarn install --check-files +``` + +Then perform the mandatory service-client gate: + +```bash +aws lambda-microvms help >"$EVIDENCE_DIR/lambda-microvms-help.txt" +node -p "require('./node_modules/@aws-sdk/client-lambda-microvms/package.json').version" \ + | tee "$EVIDENCE_DIR/lambda-microvms-sdk-version.txt" +``` + +**Expected:** install succeeds; SDK version is `3.1098.0`; the help command +lists at least `run-microvm`, `get-microvm`, `suspend-microvm`, +`resume-microvm`, `terminate-microvm`, `create-microvm-auth-token`, and +`delete-microvm-image-version`. If help fails, stop and update/install the AWS +CLI distribution that exposes the preview/new service. + +**Record:** every version and whether service help was available. + +**ADR-021 item:** empirical IAM/API action-name verification. + +--- + +## Phase 1 — Least-privilege bootstrap + +### 1.1 Inspect the existing bootstrap + +**Purpose:** determine whether the standard bootstrap lacks ABCA's generated +template and custom compute policies. + +```bash +aws cloudformation describe-stacks --stack-name CDKToolkit \ + | tee "$EVIDENCE_DIR/cdktoolkit-before.json" +aws cloudformation get-template --stack-name CDKToolkit \ + --query TemplateBody --output text >"$EVIDENCE_DIR/cdktoolkit-template-before.txt" +python3 - "$EVIDENCE_DIR/cdktoolkit-template-before.txt" <<'PY' +import pathlib, sys +s = pathlib.Path(sys.argv[1]).read_text() +for needle in ("ComputeTypes", "IaCRoleABCAComputeLambdaMicrovms"): + print(needle, "present" if needle in s else "ABSENT") +PY +``` + +**Expected:** a standard bootstrap may report both markers absent. That is the +reason for the next step, not a failure. + +**Record:** template markers and current CDKToolkit parameters. + +**ADR-021 item:** conditional bootstrap policy exists only with the custom +template. + +### 1.2 Re-bootstrap, then set the CloudFormation parameter + +**Purpose:** replace the standard administrator bootstrap with the repository's +generated least-privilege template, then enable both AgentCore and Lambda +MicroVM deployment permissions. `cdk bootstrap` has no `--parameters`; CDK +context is not a substitute for this CloudFormation parameter. + +```bash +# `--force` is REQUIRED on an already-bootstrapped account: without it the CDK CLI +# refuses to replace the default template and exits 0 ("Bootstrap stack already +# exists, containing 'AWS CDK: Default Resources'. Not overwriting it…"), leaving +# AdministratorAccess attached while looking like a success (F10). Note also that +# BootstrapVariant stays 'AWS CDK: Default Resources' afterwards, so every future +# non-forced bootstrap refuses again. +MISE_EXPERIMENTAL=1 mise //cdk:bootstrap -- --force + +# The comma MUST be backslash-escaped. The CLI's shorthand parser otherwise splits +# on it and rejects the call (F14): +# "Invalid type for parameter Parameters[0].ParameterValue, +# value: ['agentcore', 'lambda-microvm'], type: , +# valid types: " +# The comment above `[tasks.bootstrap]` in cdk/mise.toml shows the same escaping. +aws cloudformation update-stack \ + --stack-name CDKToolkit \ + --use-previous-template \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameters 'ParameterKey=ComputeTypes,ParameterValue=agentcore\,lambda-microvm' +aws cloudformation wait stack-update-complete --stack-name CDKToolkit +aws cloudformation describe-stacks --stack-name CDKToolkit \ + --query 'Stacks[0].Parameters' \ + | tee "$EVIDENCE_DIR/cdktoolkit-parameters-after.json" +``` + +**Raw fallback if `mise` is absent** (run from `cdk/`): + +```bash +npx tsx scripts/generate-bootstrap-artifacts.ts +npx tsx scripts/generate-bootstrap-template.ts +# --force for the same reason as above (F10). +npx cdk bootstrap --template bootstrap/bootstrap-template.yaml --force +``` + +Then run the same `aws cloudformation update-stack` parameter dance above, +including the escaped comma. + +**Expected:** `ComputeTypes` is exactly `agentcore,lambda-microvm`. The custom +template replaces default `AdministratorAccess` with generated ABCA policies. + +**Record:** update stack ID/events and final parameters. + +**ADR-021 item:** “where `ComputeTypes` includes `lambda-microvm`, attach +`IaCRole-ABCA-Compute-LambdaMicrovms`.” + +### 1.3 Verify policy creation and attachment + +**Purpose:** prove the MicroVM CloudFormation permissions are attached to the +actual execution role. + +```bash +export CFN_EXEC_ROLE="$(aws cloudformation describe-stack-resource \ + --stack-name CDKToolkit \ + --logical-resource-id CloudFormationExecutionRole \ + --query StackResourceDetail.PhysicalResourceId --output text)" + +aws iam list-policies --scope Local \ + --query "Policies[?contains(PolicyName, 'IaCRole-ABCA-Compute-LambdaMicrovms')].[PolicyName,Arn]" \ + --output table | tee "$EVIDENCE_DIR/microvm-bootstrap-policy.txt" +aws iam list-attached-role-policies --role-name "$CFN_EXEC_ROLE" \ + | tee "$EVIDENCE_DIR/cfn-exec-attached-policies.json" +``` + +**Expected:** one generated policy whose name contains +`IaCRole-ABCA-Compute-LambdaMicrovms` exists and its ARN is attached to +`$CFN_EXEC_ROLE`. + +**Record:** execution role name, policy ARN, and attachments. + +**ADR-021 item:** conditional bootstrap policy and verified IAM action names. + +**Optional negative deliberately skipped:** a scratch-qualifier bootstrap with +only `agentcore` would create another bootstrap stack, buckets, ECR repository, +roles, and policies merely to prove a template condition already covered by CDK +tests. It is not cheap enough for the core pass and complicates teardown. Run it +only if specifically requested, and destroy every scratch bootstrap resource. + +--- + +## Phase 2 — Substrate-only deploy (no image context) + +### 2.1 Synthesize and deploy the bootstrap state + +**Purpose:** verify the intended first-deploy state: connector, buckets, roles, +and logs exist while no image or orchestrator image configuration exists. + +```bash +MISE_EXPERIMENTAL=1 mise //cdk:synth -- \ + "$STACK_NAME" --context compute_type=lambda-microvm \ + 2>&1 | tee "$EVIDENCE_DIR/substrate-synth.txt" + +MISE_EXPERIMENTAL=1 mise //cdk:deploy -- \ + "$STACK_NAME" --require-approval never \ + --context compute_type=lambda-microvm \ + 2>&1 | tee "$EVIDENCE_DIR/substrate-deploy.txt" +``` + +**Raw fallback if `mise` is absent** (run from `cdk/`): + +```bash +npx cdk synth "$STACK_NAME" --context compute_type=lambda-microvm +npx cdk deploy "$STACK_NAME" --require-approval never --context compute_type=lambda-microvm +``` + +**Expected:** synth includes warning ID +`abca:microvm-image-not-provisioned`; deploy completes. This is intentionally +not `abca:microvm-image-p1-smoke-unverified` yet because no image is configured. +(The 2026-07-31 pass observed the pre-fix id `abca:microvm-image-p1-not-runnable`; +the warning was renamed when F1 was fixed.) + +**Record:** warning, deployment duration, stack ID/status, and failures/retries. + +**ADR-021 item:** conditional substrate and explicit no-image first-deploy +warning. + +### 2.2 Resolve exact outputs and resources + +**Purpose:** prove the script-facing substrate contract and capture physical IDs. + +```bash +aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query 'Stacks[0].Outputs' | tee "$EVIDENCE_DIR/stack-outputs-substrate.json" +aws cloudformation list-stack-resources --stack-name "$STACK_NAME" \ + | tee "$EVIDENCE_DIR/stack-resources-substrate.json" + +for key in ComputeSubstrate MicrovmArtifactBucketName MicrovmArtifactObjectKey \ + MicrovmBuildRoleArn MicrovmExecutionRoleArn MicrovmEgressConnectorArns \ + MicrovmLogGroupName; do + aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='$key'].OutputValue | [0]" --output text +done | tee "$EVIDENCE_DIR/microvm-output-values.txt" +``` + +**Expected:** `ComputeSubstrate=lambda-microvm`; all six `Microvm...` outputs +are populated; artifact key is `microvm-images/agent-artifact.zip`. Stack +resources include two S3 buckets, build/execution roles, a 443-only security +group, `/aws/lambda-microvms/...` log group, and +`AWS::Lambda::NetworkConnector`; no `AWS::Lambda::MicrovmImage` exists. + +**Record:** outputs and physical IDs. + +**ADR-021 item:** construct resources, egress connector, build/execution roles, +artifact/payload buckets. + +### 2.3 Verify no orchestrator `MICROVM_*` environment + +**Purpose:** prove partial image configuration is not injected. + +```bash +export ORCHESTRATOR_FN="$(aws cloudformation list-stack-resources \ + --stack-name "$STACK_NAME" \ + --query "StackResourceSummaries[?ResourceType=='AWS::Lambda::Function' && contains(LogicalResourceId, 'TaskOrchestrator')].PhysicalResourceId | [0]" \ + --output text)" +aws lambda get-function-configuration --function-name "$ORCHESTRATOR_FN" \ + --query 'Environment.Variables' | tee "$EVIDENCE_DIR/orchestrator-env-no-image.json" +aws lambda get-function-configuration --function-name "$ORCHESTRATOR_FN" \ + --query 'Environment.Variables' --output json \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print([k for k in d if k.startswith("MICROVM_")])' +``` + +**Expected:** the final line is `[]`. + +**Record:** function name and environment-key list (do not publish environment +values if later deployments add sensitive configuration). + +**ADR-021 item:** reject tasks when deployed without an image; all-or-nothing +strategy configuration. + +### 2.4 Verify backend cost tags + +**Purpose:** verify deployed, taggable construct resources carry +`abca:compute-backend=lambda-microvm`. + +```bash +aws resourcegroupstaggingapi get-resources \ + --tag-filters Key=abca:compute-backend,Values=lambda-microvm \ + --query 'ResourceTagMappingList[].ResourceARN' --output text \ + | tee "$EVIDENCE_DIR/microvm-tagged-resource-arns.txt" + +aws cloudformation get-template --stack-name "$STACK_NAME" \ + --query TemplateBody --output json >"$EVIDENCE_DIR/deployed-template.json" +python3 - "$EVIDENCE_DIR/deployed-template.json" <<'PY' +import json, pathlib, sys +t = json.loads(pathlib.Path(sys.argv[1]).read_text()) +for logical_id, r in t["Resources"].items(): + if "LambdaMicrovmCompute" not in logical_id: + continue + tags = r.get("Properties", {}).get("Tags") + print(logical_id, r["Type"], tags if tags is not None else "NOT-TAGGABLE/NO-TAGS") +PY +``` + +**Expected:** every taggable construct resource (buckets, roles, security group, +log group, and connector where supported) shows the backend tag. Generated +policies/bucket policies are not independently taggable resources. Save any +service that omits tags as a defect rather than silently accepting it. + +**Record:** tagged ARNs and the per-logical-resource template report. + +**ADR-021 item:** backend-identifying cost-allocation tags. + +--- + +## Phase 3 — Region gate (synth only) + +### 3.1 Reject an unsupported Region + +**Purpose:** prove static fail-fast enforcement without deploying there. + +```bash +env AWS_REGION=eu-central-1 AWS_DEFAULT_REGION=eu-central-1 \ + CDK_DEFAULT_REGION=eu-central-1 CDK_DEFAULT_ACCOUNT="$ACCOUNT_ID" \ + MISE_EXPERIMENTAL=1 mise //cdk:synth -- \ + "$STACK_NAME" --context compute_type=lambda-microvm \ + >"$EVIDENCE_DIR/unsupported-region-synth.txt" 2>&1 && { + echo "ERROR: unsupported-region synth unexpectedly succeeded" >&2; exit 1; + } +``` + +**Raw fallback if `mise` is absent** (run from `cdk/`): + +```bash +env AWS_REGION=eu-central-1 AWS_DEFAULT_REGION=eu-central-1 \ + CDK_DEFAULT_REGION=eu-central-1 CDK_DEFAULT_ACCOUNT="$ACCOUNT_ID" \ + npx cdk synth "$STACK_NAME" --context compute_type=lambda-microvm +``` + +**Expected:** failure names `eu-central-1`, all five supported Regions, and +`--context microvm_region_override=true`. + +**Record:** complete stderr. + +**ADR-021 item:** static unsupported-Region synth failure. + +### 3.2 Exercise the escape hatch + +**Purpose:** prove newly launched Regions can bypass only the static list. + +```bash +env AWS_REGION=eu-central-1 AWS_DEFAULT_REGION=eu-central-1 \ + CDK_DEFAULT_REGION=eu-central-1 CDK_DEFAULT_ACCOUNT="$ACCOUNT_ID" \ + MISE_EXPERIMENTAL=1 mise //cdk:synth -- \ + "$STACK_NAME" --context compute_type=lambda-microvm \ + --context microvm_region_override=true \ + 2>&1 | tee "$EVIDENCE_DIR/unsupported-region-override-synth.txt" +``` + +**Expected:** synth succeeds with warning `abca:microvm-region-override`. + +**Record:** warning and exit status. + +**ADR-021 item:** Region-list escape hatch. + +**Gotcha:** `src/main.ts` reads `CDK_DEFAULT_REGION`, not merely `AWS_REGION`. +An unresolved/region-agnostic CDK token skips the static check by design. The +commands set both account and Region to force the real test. Synth makes no +MicroVM control-plane calls, but CDK context/asset bundling may still require +valid AWS credentials and the bootstrap version parameter. + +--- + +## Phase 4 — Package and build an image + +### 4.1 Select a managed base image + +**Purpose:** pin a real regional base-image ARN/version rather than guessing. + +```bash +aws lambda-microvms list-managed-microvm-images \ + | tee "$EVIDENCE_DIR/managed-images.json" +export BASE_IMAGE_ARN="$(aws lambda-microvms list-managed-microvm-images \ + --query 'items[0].imageArn' --output text)" +aws lambda-microvms list-managed-microvm-image-versions \ + --image-identifier "$BASE_IMAGE_ARN" \ + | tee "$EVIDENCE_DIR/managed-image-versions.json" +# NEWEST FIRST (measured 2026-07-31): items[0] is the latest version, items[-1] +# is the OLDEST. The original `items[-1]` here selected version 0 instead of 1. +export BASE_IMAGE_VERSION="$(aws lambda-microvms list-managed-microvm-image-versions \ + --image-identifier "$BASE_IMAGE_ARN" \ + --query 'items[0].imageVersion' --output text)" +test -n "$BASE_IMAGE_ARN" && test "$BASE_IMAGE_ARN" != None +test -n "$BASE_IMAGE_VERSION" && test "$BASE_IMAGE_VERSION" != None +``` + +**Expected:** the regional probe succeeds and returns at least one ARN/version. +Ordering is newest-to-oldest, so `items[0]` is correct; inspect the timestamps +and explicitly export the desired version if the installed CLI ever differs. + +**Record:** complete catalogs and selected pair. + +**ADR-021 item:** live regional availability probe and managed base-image API. + +### 4.2 Package, upload, and start the out-of-band build + +**Purpose:** exercise the actual script interface and avoid slow CloudFormation +iteration while still using CDK-created bucket, role, connector, and logs. + +Before running it, capture the installed CLI's authoritative request shape: + +```bash +aws lambda-microvms create-microvm-image help \ + >"$EVIDENCE_DIR/create-microvm-image-help.txt" +aws lambda-microvms create-microvm-image --generate-cli-skeleton input \ + >"$EVIDENCE_DIR/create-microvm-image-skeleton.json" +``` + +Confirm that `hooks.microvmHooks.run` is `ENABLED` and +`cpuConfigurations[].architecture` is `ARM_64`, matching SDK 3.1098.0. The CDK +L1 request is a separate CloudFormation surface and legitimately retains its +generated path/string shape. If the installed CLI skeleton differs from the SDK +model or rejects the script request, stop this phase, save the parser/service +error as a model-drift defect, and mark later image/runtime steps blocked. + +```bash +export IMAGE_NAME="${STACK_NAME}-abca-agent" +export BUILD_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +cdk/scripts/package-microvm-artifact.sh \ + --stack-name "$STACK_NAME" \ + --create-image \ + --base-image-arn "$BASE_IMAGE_ARN" \ + --base-image-version "$BASE_IMAGE_VERSION" \ + --image-name "$IMAGE_NAME" \ + 2>&1 | tee "$EVIDENCE_DIR/package-and-create-image.txt" +# Explicit status check — `| tee` reports tee's status, so a bare `$?` here lies +# unless `set -o pipefail` is on (see "Important P1 and tooling limits"). +test "${PIPESTATUS[0]}" -eq 0 +``` + +The script requires `aws`, `zip`, `python3`, and `rsync`; reads outputs +`MicrovmArtifactBucketName`, `MicrovmArtifactObjectKey`, +`MicrovmBuildRoleArn`, `MicrovmBuildEgressConnectorArns`, +`MicrovmEgressConnectorArns`, and `MicrovmLogGroupName`; stages root +`Dockerfile`, `agent/`, and `contracts/`; uploads the zip; and calls +`create-microvm-image` with ARM64, 8,192 MiB, `/ready` **and** `/run` enabled on +port 8080, the **build-time** egress connector (443 + 80), and the backend tag. + +**Expected:** upload succeeds, create returns/starts image version `1.0` in this +virgin image name, and the output contains the conspicuous “P1 image is runnable +but NOT smoke-verified” reminder — printed BOTH before and after the create call, +so a failing create cannot swallow it. + +**Record:** artifact size printed by the script, S3 object size from the command +below, create response, and exact banner. + +```bash +export ARTIFACT_BUCKET="$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='MicrovmArtifactBucketName'].OutputValue | [0]" --output text)" +export ARTIFACT_KEY="$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='MicrovmArtifactObjectKey'].OutputValue | [0]" --output text)" +aws s3api head-object --bucket "$ARTIFACT_BUCKET" --key "$ARTIFACT_KEY" \ + | tee "$EVIDENCE_DIR/artifact-head.json" +``` + +**ADR-021 item:** zip+Dockerfile packaging plane, no secret build inputs, build +role/connector/log wiring, and the `abca:microvm-image-p1-smoke-unverified` +warning (renamed from `…-p1-not-runnable` when F1 was fixed: the image IS +creatable, launchable and payload-deliverable now that `/ready` + `/run` are +declared AND served — what is unverified is smoke parity). + +### 4.3 Poll build and image status to ACTIVE + +**Purpose:** capture real snapshot build duration, state, and component sizes. + +```bash +export IMAGE_VERSION=1 +while :; do + aws lambda-microvms list-microvm-image-builds \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + | tee "$EVIDENCE_DIR/image-builds-latest.json" + STATE="$(aws lambda-microvms list-microvm-image-builds \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --query 'items[0].buildState' --output text)" + date -u '+%Y-%m-%dT%H:%M:%SZ buildState='"$STATE" + case "$STATE" in SUCCESSFUL) break;; FAILED) exit 1;; esac + sleep 30 +done + +export BUILD_ID="$(aws lambda-microvms list-microvm-image-builds \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --query 'items[0].buildId' --output text)" +aws lambda-microvms get-microvm-image-build \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --build-id "$BUILD_ID" | tee "$EVIDENCE_DIR/image-build-final.json" + +while :; do + aws lambda-microvms get-microvm-image-version \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + | tee "$EVIDENCE_DIR/image-version-latest.json" + STATUS="$(aws lambda-microvms get-microvm-image-version \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --query status --output text)" + test "$STATUS" = ACTIVE && break + sleep 30 +done +export BUILD_FINISHED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +``` + +**Expected:** build states may include `PENDING` and `IN_PROGRESS`, then +`SUCCESSFUL`; image-version `state` becomes `SUCCESSFUL` and `status` becomes +`ACTIVE`. On failure, save `stateReason` and tail the exact output log group: + +```bash +export MICROVM_LOG_GROUP="$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='MicrovmLogGroupName'].OutputValue | [0]" --output text)" +aws logs tail "$MICROVM_LOG_GROUP" --since 2h +``` + +**Record:** start/finish timestamps, duration, all states/reasons, build ID, and +`snapshotBuild.memorySnapshotSizeInBytes`, `codeInstallSizeInBytes`, and +`diskSnapshotSizeInBytes`. Compare **code-install size** to AgentCore's 2 GB +container-image limit, while clearly noting that memory/disk snapshots are not +equivalent to an OCI image and must not be summed into a misleading comparison. +Also record the reported image resources/disk facts; the SDK exposes minimum +memory but no explicit disk-capacity field, so verify the 32 GB disk claim in +the service quota/console and report “not exposed” if that remains true. + +**ADR-021 item:** buildability, final image size, 2 GB-limit narrative, and disk +quota external fact. + +### 4.4 Redeploy against the built image and inspect IAM/env + +**Purpose:** hand the out-of-band image to the orchestrator and prove exact-image +IAM scoping. + +```bash +MISE_EXPERIMENTAL=1 mise //cdk:deploy -- \ + "$STACK_NAME" --require-approval never \ + --context compute_type=lambda-microvm \ + --context microvm_image_identifier="$IMAGE_NAME" \ + --context microvm_image_version="$IMAGE_VERSION" \ + 2>&1 | tee "$EVIDENCE_DIR/image-configured-deploy.txt" +``` + +**Expected:** synth/deploy emits `abca:microvm-image-p1-smoke-unverified` (the +2026-07-31 pass saw the pre-fix id `abca:microvm-image-p1-not-runnable`; the +warning was renamed when F1 was fixed). The orchestrator now has +`MICROVM_IMAGE_IDENTIFIER` — **a full image ARN**, not a bare name (F3) — +`MICROVM_IMAGE_VERSION`, `MICROVM_EXECUTION_ROLE_ARN`, +`MICROVM_EGRESS_CONNECTOR_ARNS`, `MICROVM_PAYLOAD_BUCKET`, and +`MICROVM_INGRESS_CONNECTOR_ARNS` carrying the Lambda-managed `NO_INGRESS` +connector (F7 — the pre-fix build had no ingress variable at all). + +```bash +aws lambda get-function-configuration --function-name "$ORCHESTRATOR_FN" \ + --query 'Environment.Variables' --output json \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); print({k:d[k] for k in d if k.startswith("MICROVM_")})' \ + | tee "$EVIDENCE_DIR/orchestrator-microvm-env.txt" + +export ORCHESTRATOR_ROLE_ARN="$(aws lambda get-function-configuration \ + --function-name "$ORCHESTRATOR_FN" --query Role --output text)" +export ORCHESTRATOR_ROLE="${ORCHESTRATOR_ROLE_ARN##*/}" +aws iam list-role-policies --role-name "$ORCHESTRATOR_ROLE" \ + | tee "$EVIDENCE_DIR/orchestrator-inline-policy-names.json" +export ORCH_POLICY_NAME="$(aws iam list-role-policies --role-name "$ORCHESTRATOR_ROLE" \ + --query 'PolicyNames[0]' --output text)" +aws iam get-role-policy --role-name "$ORCHESTRATOR_ROLE" \ + --policy-name "$ORCH_POLICY_NAME" \ + | tee "$EVIDENCE_DIR/orchestrator-inline-policy.json" +``` + +The inline policy's physical name is CDK-generated and therefore cannot be +hard-coded; the actual name to fetch is `$ORCH_POLICY_NAME` returned by +`list-role-policies` (normally the role's `DefaultPolicy`). If more than one is +listed, fetch each and select the document containing `Sid=MicrovmLifecycle`. + +**Expected:** `MicrovmLifecycle` grants exactly `lambda:RunMicrovm`, +`lambda:GetMicrovm`, and `lambda:TerminateMicrovm` against exactly +`arn:...:microvm-image:$IMAGE_NAME` and its `:` suffix sibling; +`MicrovmPassNetworkConnector` has `lambda:PassNetworkConnector` on `*`; no +`SuspendMicrovm`, `ResumeMicrovm`, or `CreateMicrovmAuthToken` exists. + +**Record:** warning, environment-key/value map, role/policy names, statements, +and exact image ARN format observed. + +**ADR-021 item:** the `abca:microvm-image-p1-smoke-unverified` warning (formerly +`…-p1-not-runnable`), exact-ARN lifecycle IAM, no-JWE grant, and all-or-nothing +environment wiring — which now includes `MICROVM_INGRESS_CONNECTOR_ARNS`, always +injected, carrying the `NO_INGRESS` control. + +--- + +## Phase 5 — Manual lifecycle and empirical checklist + +### 5.0 Resolve launch inputs and IAM identity mode + +**Purpose:** use deployed values and distinguish true role-policy evidence from +admin-only lifecycle evidence. + +```bash +export EXECUTION_ROLE_ARN="$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='MicrovmExecutionRoleArn'].OutputValue | [0]" --output text)" +export EGRESS_CONNECTORS="$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs[?OutputKey=='MicrovmEgressConnectorArns'].OutputValue | [0]" --output text)" + +aws sts assume-role --role-arn "$ORCHESTRATOR_ROLE_ARN" \ + --role-session-name abca-645-verification \ + >"$EVIDENCE_DIR/orchestrator-assume-role.json" \ + 2>"$EVIDENCE_DIR/orchestrator-assume-role-error.txt" || true +``` + +Lambda execution-role trust normally allows only `lambda.amazonaws.com`, so +operator assumption is expected to fail unless the sandbox has an explicit +test trust path. **Do not modify production-like trust just for this run.** If +assumption succeeds, open a subshell with those temporary credentials for steps +5.1, 5.2, 5.7, and 5.8(a/b), and mark evidence `ORCHESTRATOR_ROLE`. Otherwise +run as sandbox admin and mark scoping observations `ADMIN — advisory`; the +static inline-policy inspection in 4.4 remains authoritative. + +Example temporary-credential subshell setup: + +```bash +read AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN < **Stale premise.** This step was written against the pre-fix world, where the +> plan was to declare `/run` in P1 and serve it in P2. F1 proved that is not a +> reachable service state, and the agent now serves `/ready` + `/run`. Two +> consequences for a re-run: (i) the image is no longer hook-less, so +> `--run-hook-payload` is ACCEPTED rather than rejected — the interesting +> observation becomes whether the payload reaches the pipeline, not what a +> hook-less VM does; and (ii) `--image-identifier` needs the image **ARN**, not +> `$IMAGE_NAME` (F3). What still holds unchanged, and is worth re-confirming, is +> everything below about the state enum, the 28,800-second bound, the +> omit-`idlePolicy` invariant, and the default public ingress. + +```bash +export RUN_STARTED_EPOCH="$(date +%s)" +aws lambda-microvms run-microvm \ + --image-identifier "$IMAGE_NAME" \ + --image-version "$IMAGE_VERSION" \ + --execution-role-arn "$EXECUTION_ROLE_ARN" \ + --egress-network-connectors "$EGRESS_CONNECTORS" \ + --run-hook-payload '{"verification":"issue-645-p1"}' \ + --maximum-duration-in-seconds 28800 \ + | tee "$EVIDENCE_DIR/run-microvm.json" +export MICROVM_ID="$(python3 -c 'import json,os; print(json.load(open(os.environ["EVIDENCE_DIR"] + "/run-microvm.json"))["microvmId"])')" +export MICROVM_ENDPOINT="$(python3 -c 'import json,os; print(json.load(open(os.environ["EVIDENCE_DIR"] + "/run-microvm.json"))["endpoint"])')" +``` + +Do **not** pass `--idle-policy`. Immediately poll: + +```bash +for delay in 0 2 5 10 20 30 60; do + sleep "$delay" + date -u '+%Y-%m-%dT%H:%M:%SZ' + aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" || true +done | tee "$EVIDENCE_DIR/hookless-state-timeline.txt" +``` + +**Expected:** `run-microvm` should return a `microvmId`, endpoint, image ARN, +version, and initial state if `/run` is asynchronous at control-plane return. +The eventual state is deliberately **not prescribed**: record whether it reaches +`RUNNING`, remains `PENDING`, becomes `TERMINATING/TERMINATED`, or disappears, +plus `stateReason` and elapsed time. If `run-microvm` itself rejects because the +hook returns 404/times out, record that exact exception and timing. This result +feeds the P2 hook/startup design. + +**Record:** full response, endpoint (not an auth token), all states/reasons, +time-to-first-state/time-to-terminal, and relevant log lines. + +**ADR-021 item:** real state enum, 28,800-second bound, and omit-`idlePolicy` +invariant. (The original "P1-not-runnable premise" this step was written to probe +no longer exists — see the Phase 5 row of the stale-instruction table.) + +### 5.2 Explicit `get-microvm` state mapping + +**Purpose:** validate the six SDK states used by strategy mapping. + +```bash +aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" \ + | tee "$EVIDENCE_DIR/get-microvm.json" +``` + +**Expected:** observed values come from `PENDING`, `RUNNING`, `SUSPENDING`, +`SUSPENDED`, `TERMINATING`, `TERMINATED`; a reaped ID returns +`ResourceNotFoundException`. + +**Record:** every distinct state actually observed and any unknown state. + +**ADR-021 item:** mechanical state mapping and future-enum safety premise. + +### 5.3 Manual suspend without `idlePolicy` + +**Purpose:** determine whether explicit suspend works independently of traffic +idle policy and whether a hook-less image survives long enough to suspend. + +Use the sandbox admin identity because P1's orchestrator correctly lacks this +permission: + +```bash +date -u '+%Y-%m-%dT%H:%M:%SZ suspend-request' +aws lambda-microvms suspend-microvm --microvm-identifier "$MICROVM_ID" \ + | tee "$EVIDENCE_DIR/suspend-microvm.json" +for i in 1 2 3 4 5 6; do + aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" || true + sleep 10 +done | tee "$EVIDENCE_DIR/suspend-state-timeline.txt" +``` + +**Expected:** if the VM reached a suspendible state, observe +`SUSPENDING → SUSPENDED`. A conflict/not-found caused by the failed `/run` hook +is a valid P1 result but means 5.4–5.6 cannot discharge TTL/resume empirically; +mark those **BLOCKED-BY-P1-HOOKLESS-IMAGE**, do not invent an answer. + +**Record:** identity, response/exception, states, and suspend latency. + +**ADR-021 item:** manual suspend without idle policy. + +### 5.4 Suspended TTL experiment + +**Purpose:** determine the default lifetime of a manually suspended VM when +`idlePolicy` (and therefore `suspendedDurationSeconds`) is omitted. + +Only run after observing `SUSPENDED`: + +```bash +for seconds in 0 900 3600 14400; do + sleep "$seconds" + printf '\ncheckpoint_after_sleep_seconds=%s at %s\n' "$seconds" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" || true +done | tee "$EVIDENCE_DIR/suspended-ttl-checkpoints.txt" +``` + +The sleeps are incremental (0, then 15 min, then +1 h, then +4 h). A time-boxed +executor may truncate after 15 min or 1 h, but must say so. Never leave the VM +past the 28,800-second maximum; Phase 8 terminates it. + +**Expected:** unknown by design. Record whether it stays `SUSPENDED`, terminates, +or becomes NotFound and at what wall-clock age. `maximumDurationInSeconds=28800` +bounds the worst case even if no separate suspended TTL exists. + +**Record:** complete checkpoints, truncation, start age, and terminal time. + +**ADR-021 item:** P1 external fact — manual-suspend default TTL without +`idlePolicy`. + +### 5.5 Quota treatment while suspended + +**Purpose:** test the rationale that a suspended VM still holds account memory +quota. + +```bash +aws service-quotas list-services \ + --query "Services[?contains(ServiceName, 'Lambda')].[ServiceCode,ServiceName]" \ + --output table | tee "$EVIDENCE_DIR/lambda-service-codes.txt" +aws service-quotas list-service-quotas --service-code lambda \ + --query "Quotas[?contains(QuotaName, 'MicroVM') || contains(QuotaName, 'microVM') || contains(QuotaName, 'memory')]" \ + | tee "$EVIDENCE_DIR/microvm-service-quotas.json" +aws lambda-microvms list-microvms --image-identifier "$IMAGE_NAME" \ + | tee "$EVIDENCE_DIR/microvms-while-suspended.json" +``` + +Also open the Lambda MicroVM **account quota / memory utilization view** in the +AWS console before launch, while RUNNING, while SUSPENDED, and after termination; +capture values/timestamps. The SDK 3.1098 model has no “get account memory +usage” operation, and `service-quotas` normally reports limits rather than live +consumption. If the console has no utilization view and quota is too high to +safely saturate with a second 32 GiB VM, record **NOT OBSERVABLE SAFELY** rather +than launching VMs until failure. + +**Expected:** the suspended VM remains in `list-microvms`; the load-bearing +claim is discharged only if the account quota view continues counting its +32,768 MiB after `SUSPENDED`. + +**Record:** quota names/codes/limits, list response, and four utilization +snapshots. Distinguish “listed” from “proven to consume quota.” + +**ADR-021 item:** account-memory quota treatment of suspended VMs / concurrency +slot-held rationale. + +### 5.6 Resume and state transition + +**Purpose:** verify manual resume and preserved lifecycle identity. + +```bash +aws lambda-microvms resume-microvm --microvm-identifier "$MICROVM_ID" \ + | tee "$EVIDENCE_DIR/resume-microvm.json" +for i in 1 2 3 4 5 6; do + aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" || true + sleep 10 +done | tee "$EVIDENCE_DIR/resume-state-timeline.txt" +``` + +**Expected:** unknown for P1 because no `/resume` hook is declared and `/run` +may already have failed. Record whether resume succeeds and transitions to +`RUNNING`, conflicts, terminates, or disappears, and whether ID/endpoint remain +stable. + +**Record:** response, transitions, latency, ID/endpoint stability. + +**ADR-021 item:** empirical resume lifecycle input for P3 design. + +### 5.7 Terminate and observe reaping + +**Purpose:** validate active cleanup and the `NotFound → completed` strategy +mapping premise. + +```bash +date -u '+%Y-%m-%dT%H:%M:%SZ terminate-request' +aws lambda-microvms terminate-microvm --microvm-identifier "$MICROVM_ID" \ + | tee "$EVIDENCE_DIR/terminate-microvm.json" +for delay in 0 2 5 10 20 30 60 120; do + sleep "$delay" + date -u '+%Y-%m-%dT%H:%M:%SZ' + aws lambda-microvms get-microvm --microvm-identifier "$MICROVM_ID" || true +done | tee "$EVIDENCE_DIR/terminate-reap-timeline.txt" +``` + +**Expected:** `TERMINATING`/`TERMINATED` may be visible, followed by +`ResourceNotFoundException`. Exact reaping time is empirical. + +**Record:** transition and seconds from terminate request to NotFound, including +exception name/message/status code. + +**ADR-021 item:** explicit terminate path and `ResourceNotFoundException → +completed` mapping. + +### 5.8 IAM negative tests (only under assumed orchestrator role) + +**Purpose:** prove the deployed role cannot escape its exact image or mint JWE +tokens. + +Run only if step 5.0 successfully assumed the orchestrator role. Otherwise mark +**SKIPPED — LAMBDA ROLE TRUST DOES NOT ALLOW OPERATOR ASSUMPTION** and rely on +4.4's policy document. + +```bash +aws lambda-microvms run-microvm \ + --image-identifier "${IMAGE_NAME}-different" \ + --execution-role-arn "$EXECUTION_ROLE_ARN" \ + --egress-network-connectors "$EGRESS_CONNECTORS" \ + --maximum-duration-in-seconds 60 \ + 2>&1 | tee "$EVIDENCE_DIR/iam-negative-different-image.txt" + +aws lambda-microvms create-microvm-auth-token \ + --microvm-identifier "$MICROVM_ID" \ + --expiration-in-minutes 5 \ + --allowed-ports '[{"port":8080}]' \ + 2>&1 | tee "$EVIDENCE_DIR/iam-negative-auth-token.txt" +``` + +**Expected:** both return `AccessDeniedException`. If the different-name test +returns `ResourceNotFoundException`, it does not prove exact-ARN denial; create +or use a real second sandbox image only if already available, otherwise mark +that sub-check inconclusive. The `allowed-ports` CLI union syntax is +best-effort; the verified SDK request is +`{microvmIdentifier, expirationInMinutes, allowedPorts:[{port:8080}]}`. A local +argument-parser error is not an IAM result. + +**Record:** assumed caller ARN and full errors. + +**ADR-021 item:** exact-image ARN scoping and no +`CreateMicrovmAuthToken`/no-JWE posture. + +### 5.9 Direct service boundary: 16,384 vs 16,385 bytes + +**Purpose:** independently verify the documented `runHookPayload` service cap. +This is direct-service validation; the ABCA strategy already routes envelopes +larger than 16,384 bytes to S3. + +```bash +python3 - <<'PY' +from pathlib import Path +Path('/tmp/abca-payload-16384.txt').write_bytes(b'x' * 16384) +Path('/tmp/abca-payload-16385.txt').write_bytes(b'x' * 16385) +PY +wc -c /tmp/abca-payload-16384.txt /tmp/abca-payload-16385.txt + +aws lambda-microvms run-microvm \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --execution-role-arn "$EXECUTION_ROLE_ARN" \ + --egress-network-connectors "$EGRESS_CONNECTORS" \ + --run-hook-payload file:///tmp/abca-payload-16384.txt \ + --maximum-duration-in-seconds 60 \ + | tee "$EVIDENCE_DIR/run-payload-16384.json" + +aws lambda-microvms run-microvm \ + --image-identifier "$IMAGE_NAME" --image-version "$IMAGE_VERSION" \ + --execution-role-arn "$EXECUTION_ROLE_ARN" \ + --egress-network-connectors "$EGRESS_CONNECTORS" \ + --run-hook-payload file:///tmp/abca-payload-16385.txt \ + --maximum-duration-in-seconds 60 \ + 2>&1 | tee "$EVIDENCE_DIR/run-payload-16385.txt" +``` + +Immediately terminate the MicroVM returned by the accepted request (if any). + +**Expected:** 16,384 bytes is accepted; 16,385 bytes is rejected, likely with +`ValidationException`. Record the actual exception rather than treating the +predicted name as normative. Confirm the installed CLI expands `file://` to file +contents; if it passes the literal URI, repeat with command substitution and +record the client behavior. + +**Record:** byte counts, both complete responses, exception name/message/status, +and cleanup ID. + +**ADR-021 item:** exact 16 KB `runHookPayload` boundary. + +--- + +## Phase 6 — CLI behavior + +### 6.1 Build and point the CLI at the stack + +**Purpose:** use the repository CLI's real resolution rules: operator commands +take `--region`/`--stack-name`; configured Region is a fallback; API commands +also need Cognito configuration/login. + +```bash +MISE_EXPERIMENTAL=1 mise //cli:build +bgagent() { node cli/lib/bin/bgagent.js "$@"; } +bgagent configure --stack-name "$STACK_NAME" --region "$AWS_REGION" +bgagent platform outputs --stack-name "$STACK_NAME" --region "$AWS_REGION" +``` + +**Expected:** config is written under `${BGAGENT_CONFIG_DIR:-$HOME/.bgagent}`; +stack outputs resolve. No Cognito login is needed for operator AWS commands. + +**Record:** CLI build result and redacted outputs. + +**ADR-021 item:** deploy/CLI substrate discovery contract. + +### 6.2 Onboard, probe, inspect, and clean up a dummy row + +**Purpose:** exercise the live managed-image probe, doctor check, and runtime +grouping without submitting a task. + +```bash +export DUMMY_REPO=verification-only/issue-645 +bgagent repo onboard "$DUMMY_REPO" \ + --compute-type lambda-microvm \ + --stack-name "$STACK_NAME" --region "$AWS_REGION" \ + --output json | tee "$EVIDENCE_DIR/cli-onboard.json" + +bgagent platform doctor --stack-name "$STACK_NAME" --region "$AWS_REGION" \ + --output json | tee "$EVIDENCE_DIR/cli-doctor.json" || true +bgagent runtime status --stack-name "$STACK_NAME" --region "$AWS_REGION" \ + --output json | tee "$EVIDENCE_DIR/cli-runtime-status.json" + +bgagent repo offboard "$DUMMY_REPO" \ + --stack-name "$STACK_NAME" --region "$AWS_REGION" \ + --output json | tee "$EVIDENCE_DIR/cli-offboard.json" +``` + +**Expected:** onboarding's `ListManagedMicrovmImages` probe passes and the row +has `compute_type=lambda-microvm`; doctor contains +`lambda_microvm_availability`; runtime status groups it under +`lambda_microvm_substrates`; offboard marks it removed. Doctor may still exit +non-zero in a virgin sandbox because its GitHub secret is an unpopulated +placeholder or no real repo/token/model access exists—record those independent +failures. + +**SKIP-IF-UNCONFIGURED:** if the sandbox principal lacks DDB or MicroVM catalog +read permission, record the IAM gap and skip the write. Onboarding itself does +not require a valid GitHub token, but a meaningful doctor pass and any task do. + +**Record:** probe result, row, doctor checks, grouping, and cleanup row status. + +**ADR-021 item:** onboarding region probe and doctor availability check. + +--- + +## Phase 7 — OPTIONAL negative task path + +### 7.1 Submit only in a fully configured sandbox + +**Purpose:** observe orchestrator classification and terminate-on-finalize, not +to claim P2 smoke parity. + +This is **OPTIONAL / usually DEFERRED-TO-P2-ENV**. A virgin account is missing a +Cognito user/login, populated GitHub token secret, and a genuinely accessible +onboarded repository unless the executor configures all three. Do not create +those merely for this P1 substrate pass. + +If they already exist: + +```bash +bgagent login +bgagent repo onboard --compute-type lambda-microvm \ + --stack-name "$STACK_NAME" --region "$AWS_REGION" +bgagent submit --repo --task "P1 negative: observe hook-less MicroVM failure" +# Use the returned task ID: +bgagent watch +``` + +**Expected:** the backend does not complete agent work. Capture the task's +failure classification/remedy, persisted `compute_metadata` (`microvmId` and +endpoint), orchestrator logs, and whether finalization calls +`TerminateMicrovm`. If no complete setup exists, write +`DEFERRED-TO-P2-ENV — missing Cognito user/login, GitHub token, and/or real repo +onboarding`. + +**Record:** task ID/evidence or exact deferral reason. + +**ADR-021 item:** defense-in-depth failure classification, handle persistence, +and terminate fire; this is not P2 clone→change→PR smoke parity. + +--- + +## Phase 8 — Teardown + +### 8.1 Terminate every MicroVM + +**Purpose:** stop compute billing before image/stack deletion. + +```bash +aws lambda-microvms list-microvms --image-identifier "$IMAGE_NAME" \ + | tee "$EVIDENCE_DIR/microvms-before-teardown.json" +for id in $(aws lambda-microvms list-microvms --image-identifier "$IMAGE_NAME" \ + --query 'items[].microvmId' --output text); do + aws lambda-microvms terminate-microvm --microvm-identifier "$id" || true +done +sleep 30 +aws lambda-microvms list-microvms --image-identifier "$IMAGE_NAME" \ + | tee "$EVIDENCE_DIR/microvms-after-terminate.json" +``` + +**Expected:** no nonterminal VM remains; wait/retry if necessary. + +**Record:** IDs and final states. + +**ADR-021 item:** explicit cleanup rather than relying on the eight-hour bound. + +### 8.2 Delete out-of-band image versions and image + +**Purpose:** stop snapshot storage charges. The verified operation/CLI command +name is `delete-microvm-image-version` with `imageIdentifier` and +`imageVersion`. + +```bash +aws lambda-microvms list-microvm-image-versions --image-identifier "$IMAGE_NAME" \ + | tee "$EVIDENCE_DIR/image-versions-before-delete.json" +for version in $(aws lambda-microvms list-microvm-image-versions \ + --image-identifier "$IMAGE_NAME" --query 'items[].imageVersion' --output text); do + aws lambda-microvms delete-microvm-image-version \ + --image-identifier "$IMAGE_NAME" --image-version "$version" +done +aws lambda-microvms delete-microvm-image --image-identifier "$IMAGE_NAME" +``` + +**Expected:** all versions enter deletion and the image is deleted. If the +service requires deleting the parent first/last or waiting between operations, +follow the returned conflict remedy and record actual order. + +**Record:** versions, responses, final NotFound/list absence. + +**ADR-021 item:** versioned image lifecycle cleanup. + +### 8.3 Destroy ABCA; retain bootstrap + +**Purpose:** remove the platform and recurring network costs while preserving +the account's reusable CDK bootstrap. + +```bash +MISE_EXPERIMENTAL=1 mise //cdk:destroy -- \ + "$STACK_NAME" --force \ + --context compute_type=lambda-microvm \ + --context microvm_image_identifier="$IMAGE_NAME" \ + --context microvm_image_version="$IMAGE_VERSION" +aws cloudformation wait stack-delete-complete --stack-name "$STACK_NAME" +aws cloudformation describe-stacks --stack-name CDKToolkit \ + --query 'Stacks[0].[StackStatus,Parameters]' \ + | tee "$EVIDENCE_DIR/bootstrap-left-in-place.json" +``` + +**Raw fallback if `mise` is absent** (run from `cdk/`): + +```bash +npx cdk destroy "$STACK_NAME" --force \ + --context compute_type=lambda-microvm \ + --context microvm_image_identifier="$IMAGE_NAME" \ + --context microvm_image_version="$IMAGE_VERSION" +``` + +**Expected:** `backgroundagent-dev` is absent; `CDKToolkit` remains +`CREATE_COMPLETE`/`UPDATE_COMPLETE` with the custom policies. VPC teardown can +lag while service-managed ENIs are reclaimed; wait and retry rather than +force-deleting resources past CloudFormation. + +**Record:** destroy duration/events, leftovers, and final bootstrap status. + +**ADR-021 item:** clean substrate/resource lifecycle. + +**Cost note:** this pass can accrue MicroVM running minutes, snapshot/image +storage, S3 artifact storage, NAT gateway hourly/data charges, VPC endpoint +hourly charges, CloudWatch logs, and brief Lambda/DynamoDB/API usage. Suspended +VMs should stop compute charges but may retain billed snapshot storage and +account memory quota; the experiment determines the latter. NAT gateways and +endpoints continue charging until stack deletion. + +--- + +## Live execution results — 2026-07-31, account , us-east-1 + +Executed against real AWS. Evidence directory: +`/tmp/abca-645-p1-20260731T184822Z`. Wall clock 18:48Z → 23:07Z (4 h 19 min). + +**Execution deviations from the runbook as written** (each is itself a result): + +1. `mise` is **not installed** on the executor; every **raw fallback** was used. +2. The account is **not virgin overall** — `serverless-api-powertools`, + `BuildingServerlessAPIs`, `aws-sam-cli-managed-default`, and `CDKToolkit` + pre-existed. `backgroundagent-dev` was absent, so the ABCA-specific + first-deploy premise held. +3. `docker` is absent; `finch` 1.x (`CDK_DOCKER=finch`) built the AgentCore + container asset. +4. Phase 2 could not deploy from unmodified sources. The stack was deployed from + a **hand-patched cloud assembly** (`/tmp/cdkout-p1*`, a build artifact — no + repository source file was modified). Two patches, both forced by live-service + rejections recorded below: a MicroVM connector **operator role**, and moving + subnets off `us-east-1a`. +5. A temporary **port-80 egress rule** (`sgr-07ed1fa48ef38467a`) was added to the + construct's security group to get any image to build at all (see 4.3). +6. Suspend-TTL observation was **truncated at ~1 h** (runbook allows this). + +### Phase 0 + +**0.1** — Account ``, caller +`arn:aws:sts:::assumed-role/AdminConsoleAccess/aamorosi-Isengard` +(administrator). Branch `feat/645-lambda-microvm-p1`, SHA +`0505f914fd7093cccd067b6346a24e1c40e50643`. Untracked: `docs/verification/`, +`opencode.json`. Stack absence returned exactly the expected error: + +``` +An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id backgroundagent-dev does not exist +``` + +**0.2** — `aws-cli/2.36.13 Python/3.14.6 Darwin/25.5.0 source/arm64`; +`node v24.16.0`; `cdk 2.1129.0`; `mise NOT INSTALLED`; `Python 3.9.6`; +`Zip 3.0`; **`openrsync` (protocol 29, "rsync 2.6.9 compatible")** — the +packaging script's `rsync -a --exclude` usage worked unmodified on macOS. +`@aws-sdk/client-lambda-microvms` = **3.1098.0**. + +`aws lambda-microvms help` **succeeded (exit 0)** and lists **24** commands +including all seven the runbook requires. The CLI command list is an **exact +match** to the SDK 3.1098.0 command list (24 vs 24), including +`create-microvm-shell-auth-token`. **No CLI/SDK operation-name drift.** + +`create-microvm-image --generate-cli-skeleton input` **confirms the packaging +script's shape and refutes the CDK L1 shape**: + +- `cpuConfigurations[].architecture` — help documents exactly one allowed value: + `ARM_64`. The L1's `'arm64'` is not a documented value. +- `hooks` — `{"port": integer, "microvmHooks": {"run": "DISABLED"|"ENABLED", + "runTimeoutInSeconds": integer, ...}}`. There is **no hook-path field at all**; + the L1's `run: '/run'` path string has no counterpart in the service model. +- `run-microvm` skeleton confirms `idlePolicy` + `{maxIdleDurationSeconds, suspendedDurationSeconds, autoResumeEnabled}` and + `runHookPayload` as a plain string. + +The CFN-vs-API question is therefore **half-adjudicated**: the API side is +settled, but the `AWS::Lambda::MicrovmImage` CFN path was never exercised, +because the construct only synthesizes it when `microvm_base_image_arn` + +`microvm_base_image_version` context is supplied, and the runbook's Phase 4 uses +the out-of-band script path. **The CFN value shapes remain untested** — and 4.2 +below shows the request would be rejected on hook semantics regardless of shape. + +### Phase 1 + +**1.1** — `CDKToolkit` `CREATE_COMPLETE`, created 2025-11-24, `BootstrapVariant` += `AWS CDK: Default Resources`. Markers: `ComputeTypes` **ABSENT**, +`IaCRoleABCAComputeLambdaMicrovms` **ABSENT**, `AdministratorAccess` +**present** — exactly the standard-bootstrap starting state the step predicts. + +**1.2 — DEFECT (runbook + `mise //cdk:bootstrap`): the re-bootstrap is a silent +no-op on an already-bootstrapped account.** Verbatim: + +``` +Bootstrap stack already exists, containing 'AWS CDK: Default Resources'. Not overwriting it with a template containing 'ABCA: Least-Privilege Bootstrap' (use --force if you intend to overwrite) +✅ Environment aws:///us-east-1 bootstrapped (no changes). +``` + +Exit status **0**. A pass that trusts this would proceed to 1.3 with +`AdministratorAccess` still attached. `--force` was required. A **durable** +consequence: after the forced bootstrap, `BootstrapVariant` **remains** +`AWS CDK: Default Resources` (the CDK CLI re-sends the previous value rather than +the template default `ABCA: Least-Privilege Bootstrap`), so **every future +non-forced `mise //cdk:bootstrap` will refuse again**. + +**1.2 — DEFECT (runbook): the `ComputeTypes` parameter command as written is +rejected.** Verbatim: + +``` +An error occurred (ParamValidation): Parameter validation failed: +Invalid type for parameter Parameters[0].ParameterValue, value: ['agentcore', 'lambda-microvm'], type: , valid types: +``` + +The CLI shorthand parser splits on the comma. The escaped form +`ParameterValue=agentcore\,lambda-microvm` works — which is exactly what the +comment above `[tasks.bootstrap]` in `cdk/mise.toml` already shows; the runbook +dropped the escapes. Final state: `UPDATE_COMPLETE`, `ComputeTypes` = +`agentcore,lambda-microvm`. + +**1.3 — PASS.** `CFN_EXEC_ROLE` = +`cdk-hnb659fds-cfn-exec-role--us-east-1`. Exactly one local policy +matched: `cdk-hnb659fds-IaCRole-ABCA-Compute-LambdaMicrovms--us-east-1`, +and it is attached. Attachments are the five ABCA policies (Application, +Infrastructure, Observability, Compute-Agentcore, Compute-LambdaMicrovms) and +**no `AdministratorAccess`** — the template's replacement works. The +`LambdaMicrovms` statement grants 19 actions, all image/version/build/ +managed-catalog/network-connector, including `lambda:PassNetworkConnector`. +The optional scratch-qualifier negative was deliberately skipped as the runbook +directs. + +### Phase 2 + +**2.1 — synth PASS, deploy BLOCKED THREE TIMES.** Synth emitted +`abca:microvm-image-not-provisioned` and **not** +`abca:microvm-image-p1-not-runnable`, exactly as specified. Incidental synth +warnings worth noting: `Template size is approaching limit: 893273/1000000` and +`Number of resources: 463 is approaching allowed maximum of 500`. + +*Blocker A (environmental, not an ABCA defect).* The stack carries one Docker +image asset (`agent/Dockerfile`) and no container builder was installed. With +`finch`, the `gh-builder` stage failed twice on upstream flakiness: + +``` +pkg/mod/github.com/cli/go-gh/v2@v2.13.0/internal/yamlmap/yaml_map.go:8:2: unrecognized import path "gopkg.in/yaml.v3": reading https://gopkg.in/yaml.v3?go-get=1: 502 Proxy Error +``` + +`gopkg.in` alternated 200/502 from the host too. A direct `finch build` then +succeeded. **Data point for 4.3's size narrative:** the AgentCore container +image is **1.799 GB uncompressed / 629.7 MB compressed**. + +*Blocker B — **the P1 substrate cannot deploy from unmodified sources**.* +`AWS::Lambda::NetworkConnector` `CREATE_FAILED`, verbatim: + +``` +Resource handler returned message: "NetworkConnectorOperatorRole is required for VPC_EGRESS connector type (Service: Lambda, Status Code: 400, Request ID: 04726267-6c61-4ff5-bb1d-302122e9f955) (SDK Attempt Count: 1)" (RequestToken: 1a1652c3-2166-e865-c49d-6cdb5927bbfe, HandlerErrorCode: InvalidRequest) +``` + +This **directly refutes an explicit design assumption** stated in +`cdk/src/constructs/lambda-microvm-compute.ts` (~line 467): + +> `operatorRole` is left unset so Lambda manages the ENIs with its own +> service-linked role rather than a role we would have to trust. + +The generated L1 also marks `operatorRole` optional +(`readonly operatorRole?: string`) with no note that `VPC_EGRESS` requires it. +An independent probe stack (`abca645-connector-probe`) confirmed the minimal +working recipe: a role trusting `lambda.amazonaws.com` with +`AWSLambdaVPCAccessExecutionRole` plus `ec2:CreateNetworkInterface` / +`DeleteNetworkInterface` / `DescribeNetworkInterfaces` / `DescribeSubnets` / +`DescribeVpcs` / `DescribeSecurityGroups` / `CreateTags` / +`AssignPrivateIpAddresses` / `UnassignPrivateIpAddresses` / +`Describe|ModifyNetworkInterfaceAttribute` → connector `CREATE_COMPLETE`. + +*Blocker C (AgentCore, account-AZ-specific, blocks any ABCA deploy here).* + +``` +Resource handler returned message: "Agent runtime creation failed with status: CREATE_FAILED for runtime: backgroundagentdevRuntimeCC6E3A5A-yiKm9OEVPo. Reason: The following subnets are in unsupported availability zones in region us-east-1: subnet-02b0221802f3fee10 in us-east-1a (ID: use1-az6). Supported availability zones are: use1-az4, use1-az1, use1-az2" +``` + +This account maps `us-east-1a` → `use1-az6`. `AgentVpc` does not constrain AZ +selection, so CDK's default two-AZ pick lands on an AZ AgentCore rejects. +Patched `us-east-1a` → `us-east-1c` (`use1-az2`). + +*Two further teardown/iteration gotchas.* (i) Rollback itself failed once: +`Validation failed during DeleteMemory: Memory is in transitional state +CREATING. Cannot delete memory.` — `AWS::BedrockAgentCore::Memory` cannot be +deleted while creating, leaving `ROLLBACK_FAILED`; a plain `delete-stack` +cleared it. (ii) Post-synth template edits are **silently ignored** if the +template's S3 asset object already exists: the object key is the pre-edit content +hash recorded in `*.assets.json`, so `cdk-assets` skips the upload and CFN +re-uses the stale template. The stale object must be deleted. + +Successful deploy: `CREATE_COMPLETE`, 19:41:53Z → 19:55:35Z = **13 min 42 s**, +464 resources. + +**2.2 — PASS.** `ComputeSubstrate=lambda-microvm`; all six `Microvm…` outputs +populated; artifact key exactly `microvm-images/agent-artifact.zip`. The 13 +`LambdaMicrovmCompute` resources are: artifact + payload buckets (each with a +bucket policy and an auto-delete custom resource), build role + policy, execution +role + policy, `AWS::EC2::SecurityGroup sg-0e662dc0d6f6e9ade`, +`AWS::Logs::LogGroup /aws/lambda-microvms/backgroundagent-dev-abca-agent`, and +`AWS::Lambda::NetworkConnector nc-132ede11-cb63-4dfa-b75b-6a4713023c1a`. +**No `AWS::Lambda::MicrovmImage`** — correct for this state. The security group +has exactly one rule: egress `tcp/443 → 0.0.0.0/0`, *"Allow HTTPS egress (GitHub +API, AWS services)"*. (That single rule is what breaks the image build — 4.3.) + +**2.3 — PASS.** `MICROVM_*` keys = `[]` (14 env keys total). +**DEFECT (runbook): the `ORCHESTRATOR_FN` command is broken by pagination.** +With 464 resources, `list-stack-resources --query "…|[0]"` applies the query +**per page** and printed five lines (`None None None None `), which then +failed `get-function-configuration` on the multi-line value. Needs +`--no-paginate` (or local parsing). Resolved value: +`backgroundagent-dev-TaskOrchestratorOrchestratorFn-gM2sydgNVf1V`. + +**2.4 — PASS.** All **six** taggable construct resources carry +`abca:compute-backend=lambda-microvm`: security group, network connector, log +group, both buckets, and — verified via `iam list-role-tags` — both roles. +`resourcegroupstaggingapi` returned only **5** ARNs; **IAM roles are simply not +returned by that API**, which is an API coverage gap, not a missing tag. +Bucket policies and auto-delete custom resources are not independently taggable, +as the step anticipates. + +### Phase 3 + +**3.1 — PASS** (exit 1). Verbatim: + +``` +Error: AWS Lambda MicroVMs are not available in eu-central-1. The lambda-microvm compute backend is enabled (--context compute_type=lambda-microvm) but the stack Region is not one of: us-east-1, us-east-2, us-west-2, eu-west-1, ap-northeast-1. Either deploy the stack into a supported Region, drop the backend (--context compute_type=agentcore or ecs), or — if AWS has since launched Lambda MicroVMs in eu-central-1 — bypass this static check with --context microvm_region_override=true and add eu-central-1 to LAMBDA_MICROVM_SUPPORTED_REGIONS in cdk/src/handlers/shared/microvm-regions.ts. +``` + +Names the Region, all five supported Regions, and the override flag. + +**3.2 — PASS** (exit 0) with `abca:microvm-region-override` (and, correctly, the +`abca:microvm-image-not-provisioned` warning still present). + +### Phase 4 + +**4.1 — PASS, with a runbook selector bug.** Exactly **one** managed base image +exists in us-east-1: `arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1`, with +versions `1` (created 2026-07-21) and `0` (2026-06-17). **Ordering is +newest-first**, so the runbook's `items[-1].imageVersion` returns **`0`** (the +older version); `items[0]` returns `1`. The runbook's own warning about ordering +is therefore *load-bearing here*. Selected `1`; the service echoes it back as +`baseImageVersion: "1.0"`. + +**4.2 — Artifact plane PASS; image creation FAILED TWICE on service +validation.** The script read all five outputs, staged `Dockerfile` + `agent/` + +`contracts/`, printed **`584K artifact`**, and uploaded successfully (S3 +`ContentLength` **597305**, SSE `AES256`). + +Then, on the exact request the script builds, verbatim: + +``` +An error occurred (ValidationException) when calling the CreateMicrovmImage operation: The ready (/ready) MicroVM image hook must be enabled when any MicroVM lifecycle hook (run, resume, suspend, or terminate) is enabled. The ready hook signals when the application has finished initializing so the snapshot is taken in a ready state. +``` + +This **refutes the P1 hook-phasing plan directly**. The construct comments state +`/ready` and `/validate` are *"omitted in P1 because the agent does not implement +them yet: configuring a `/validate` endpoint that 404s would fail every image +build"* — but the service **will not accept `run: ENABLED` without +`ready: ENABLED`**. "Declare `/run` in P1, serve it in P2" is not a reachable +state. + +Consequences of that failure: **the conspicuous "P1 image is NOT runnable end to +end" banner was never printed**, because the script's banner heredoc comes after +the `create-microvm-image` call. Also, the runbook's `2>&1 | tee` pipeline +reported `EXIT=0` while the script had failed — the tee status masks it. + +Retrying with `/ready` enabled surfaced the **second** rejection: + +``` +An error occurred (ValidationException) when calling the CreateMicrovmImage operation: The requested memory size of 32768 MiB is not supported by base MicroVM image arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1. Supported memory sizes in MiB are: [512, 1024, 2048, 4096, 8192]. +``` + +The script's and construct's `32768` MiB (`DEFAULT_MINIMUM_MEMORY_MIB`, +documented as *"the service ceiling"*) is **not accepted**. The ceiling for this +base image is **8192 MiB (8 GiB)**, a quarter of what ADR-021 claims. + +**4.3 — Image ACTIVE only after two more corrections; the P1 hook shape is +UNBUILDABLE.** + +*Attempt 1* (hooks omitted, 8192 MiB): create succeeded, returning +`imageVersion: "1.0"` — **not `1`**, contradicting the runbook's +`IMAGE_VERSION=1` and the script's `--image-version 1` guidance. Both builds then +**FAILED** with `stateReason: "The container image build failed."` The exact +root cause, from `/aws/lambda-microvms/backgroundagent-dev-abca-agent`: + +``` +Could not connect to deb.debian.org:80 (146.75.38.132), connection timed out +E: Unable to locate package curl +E: Unable to locate package git +E: Unable to locate package build-essential +E: Package 'gnupg' has no installation candidate +ERROR: process "/bin/sh -c apt-get update && ... apt-get install -y --no-install-recommends curl git build-essential ca-certificates gnupg ..." did not complete successfully: exit code: 100 +``` + +**The construct's 443-only security group makes the agent image unbuildable.** +`agent/Dockerfile` runs `apt-get`, which uses HTTP on **port 80**; DNS resolution +worked, so only the port is the problem. Adding a temporary port-80 egress rule +(`sgr-07ed1fa48ef38467a`) fixed it immediately. + +*Attempt 2* (`update-microvm-image`, port 80 open) produced version **`2.0`**, +both builds `SUCCESSFUL`, `state=SUCCESSFUL`, `status=ACTIVE` in +20:11:18Z → 20:17:09Z = **5 min 51 s**. + +Further API-shape observations: + +- **Two builds per image version**, one per `chipsetGeneration` (`3` and `4`, + `chipset: GRAVITON`). The runbook's `items[0].buildState` inspects only one. +- `list-microvm-image-builds --image-identifier ` → + `ValidationException: Invalid ARN format: backgroundagent-dev-abca-agent`. + **An ARN is required.** `--image-version` accepts either `1` or `1.0`. +- `snapshotBuild` is returned by **`get-microvm-image-build`**, not by + `get-microvm-image-version` (which returned `snapshotBuild: null`). + +Sizes (`buildId 2033b7d8-1aa2-44a4-b174-3fc4bffebcea`, GRAVITON gen 4): + +| Field | Bytes | Human | +|---|---|---| +| `codeInstallSizeInBytes` | 2,334,748,672 | **2.17 GiB** | +| `memorySnapshotSizeInBytes` | 1,216,577,536 | 1.13 GiB | +| `diskSnapshotSizeInBytes` | 37,089,280 | 35.4 MiB | + +**Code-install size (2.17 GiB) exceeds AgentCore's 2 GB container-image limit**, +while the equivalent OCI image built locally was 1.799 GB (629.7 MB compressed). +So the same agent tree is *over* the AgentCore ceiling when measured as MicroVM +code-install and *under* it as an OCI image — the two are not interchangeable +measures, and the ADR narrative should say which one it means. Memory and disk +snapshots are deliberately **not** summed into that comparison. + +**Disk capacity: NOT EXPOSED.** No disk quota appears in `service-quotas` +(full list under 5.5) and no image/version field reports disk capacity. The +32 GB disk claim remains unverified; the 32 GB *memory* claim is refuted (8 GiB). + +*The decisive experiment.* A second image (`…-abca-agent-hooks`) was created with +the **exact P1 hook shape plus the service-mandated `/ready`**, and rebuilt after +port 80 was open. Both builds **FAILED**: + +``` +Ready hook check failed: the application returned a client error (HTTP 4xx) response +``` + +The agent **does** answer on port 8080 (an HTTP 4xx, not a connection failure), +but does not implement `/ready`. Combined with 4.2: **a P1 image that declares +`/run` cannot be built at all**, and an image that omits hooks cannot receive a +`runHookPayload` (5.1). P1 as specified is not merely "not runnable end to end" — +its image is **not creatable**. + +**4.4 — PASS, exactly as designed.** `UPDATE_COMPLETE`. Synth emitted +`abca:microvm-image-p1-not-runnable` with the full expected text. Orchestrator +env is exactly the five variables and **no ingress variable**: + +``` +MICROVM_EGRESS_CONNECTOR_ARNS = arn:aws:lambda:us-east-1::network-connector:nc-132ede11-cb63-4dfa-b75b-6a4713023c1a +MICROVM_EXECUTION_ROLE_ARN = arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-ZJu8Y1ybJt1N +MICROVM_IMAGE_IDENTIFIER = backgroundagent-dev-abca-agent +MICROVM_IMAGE_VERSION = 1.0 +MICROVM_PAYLOAD_BUCKET = backgroundagent-dev-lambdamicrovmcomputepayloadbuc-en08fimmvu6h +``` + +Inline policy `TaskOrchestratorOrchestratorFnServiceRoleDefaultPolicyDECF0D43`: + +- `Sid: MicrovmLifecycle` — exactly `lambda:RunMicrovm`, `lambda:GetMicrovm`, + `lambda:TerminateMicrovm` on exactly + `arn:aws:lambda:us-east-1::microvm-image:backgroundagent-dev-abca-agent` + and `…:backgroundagent-dev-abca-agent:*`. **Observed image ARN format matches** + the Service Authorization Reference pattern the construct derives. +- `Sid: MicrovmPassNetworkConnector` — `lambda:PassNetworkConnector` on `*`. +- `Sid: MicrovmPassExecutionRole` — `iam:PassRole` scoped to the execution role + with `iam:PassedToService = lambda.amazonaws.com`. +- **Zero** `SuspendMicrovm` / `ResumeMicrovm` / `CreateMicrovmAuthToken` actions + anywhere in the role (only attached managed policy is + `AWSLambdaBasicDurableExecutionRolePolicy`). + +**Critical mismatch:** `MICROVM_IMAGE_IDENTIFIER` is a **bare name**, and +`RunMicrovm` rejects bare names (5.1). The construct's comment asserts the +opposite — *"`imageIdentifier` may legitimately be a bare image NAME (that is +what `create-microvm-image --name` returns and what `run-microvm +--image-identifier` accepts)"*. `lambda-microvm-strategy.ts:236` passes that env +var straight through, so the P1 orchestrator would fail at `RunMicrovm`. + +### Phase 5 + +**5.0 — ADMIN — advisory.** AssumeRole denied, verbatim: + +``` +An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts:::assumed-role/AdminConsoleAccess/aamorosi-Isengard is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam:::role/backgroundagent-dev-TaskOrchestratorOrchestratorFnS-Bd7rBa2V6Jwf +``` + +Trust policy allows only `Service: lambda.amazonaws.com`. Trust was **not** +modified. All Phase 5 lifecycle evidence is therefore **admin-identity**; +4.4's static policy inspection remains the authoritative scoping evidence. + +**5.1 — Hook-less behaviour MEASURED: the VM runs and stays running.** +Three requests, in order: + +1. Bare name → `ValidationException: Malformed ARN - doesn't start with 'arn:'` +2. ARN + `--run-hook-payload` on the hook-less image → + `ValidationException: The run hook must be enabled in the MicroVM image to pass the run hook payload` +3. ARN, no payload → **success**: + +```json +{ "microvmId": "microvm-b44b69d9-f23b-30d1-97d1-a4ac558cfb5c", + "state": "PENDING", + "endpoint": ".lambda-microvm.us-east-1.on.aws", + "imageArn": "arn:aws:lambda:us-east-1::microvm-image:backgroundagent-dev-abca-agent", + "imageVersion": "2.0", + "maximumDurationInSeconds": 28800, + "ingressNetworkConnectors": ["arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:HTTP_INGRESS"], + "egressNetworkConnectors": ["arn:aws:lambda:us-east-1::network-connector:nc-132ede11-cb63-4dfa-b75b-6a4713023c1a"] } +``` + +`maximumDurationInSeconds=28800` accepted; `idlePolicy` omitted as required. +Timeline: `RUNNING` at **+12 s**, and still `RUNNING` at +15/+21/+32/+53/+84/ +**+145 s**, `stateReason` always `None`. It did **not** terminate, stall in +`PENDING`, or disappear. A hook-less MicroVM is a healthy idle VM that bills. + +**Security finding — unrequested public ingress.** The service **auto-attached** +`arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:HTTP_INGRESS` +even though `--ingress-network-connectors` was never passed, and returned a +public `*.lambda-microvm.us-east-1.on.aws` endpoint. ADR-021's "no ingress" +posture is not what the service defaults to; it is a *default-on* public HTTP +ingress that P1 neither requests nor suppresses. + +**5.2 — five of six states observed.** `PENDING`, `RUNNING`, `SUSPENDED`, +`TERMINATING`, `TERMINATED`. **`SUSPENDING` was never observable** — suspend +reached `SUSPENDED` in under 1 s. No unknown/unmapped state appeared. +`ResourceNotFoundException` for a reaped ID was **not** reached (5.7). + +**5.3 — PASS.** Admin `suspend-microvm` on a hook-less image with no +`idlePolicy`: **empty response body**, `SUSPENDED` at **+1 s**, stable across ++12/+23/+34/+44/+55 s. Explicit suspend works independently of any idle policy, +and a hook-less image is perfectly suspendible. + +**5.4 — Suspended TTL: survived the full observation window; TRUNCATED at ~1 h.** +Suspended at 20:21:12Z. `maximumDurationInSeconds` was 28800 and `startedAt` +stayed 20:18:05Z throughout. + +| Checkpoint | Wall clock | Suspended age | `get-microvm` state | `list-microvms` | Account quota view | +|---|---|---|---|---|---| +| start | 20:21:13Z | 1 s | `SUSPENDED` | present | `L-CD1C0CC4` = 1024 GB (limit only) | +| +15 min | 20:36:27Z | 915 s | `SUSPENDED` | present | 1024 GB (unchanged) | +| +45 min | 21:06:12Z | 2700 s | `SUSPENDED` | present | 1024 GB (unchanged) | +| ~+1 h (cap) | 21:21:29Z | 3617 s | `SUSPENDED` | present | 1024 GB (unchanged) | + +**No suspended TTL was observed within 1 h.** The runbook's 4-hour checkpoint was +**not** run (time-boxed, as the runbook permits). So the answer is bounded, not +final: *a manually suspended MicroVM with no `idlePolicy` survives at least +1 h 0 min 17 s (3617 s)*; whether a TTL exists between 1 h and the 8 h +`maximumDurationInSeconds` bound is **still open**. The VM was terminated in +Phase 8 rather than left to expire. + +**5.5 — Suspended VM stays listed; quota consumption NOT PROVABLE.** +`service-quotas list-service-quotas --service-code lambda` returned 24 MicroVM +quotas. The load-bearing ones: + +| Code | Name | Value | +|---|---|---| +| `L-CD1C0CC4` | Max allocated memory | **1024 Gigabytes** | +| `L-B430C318` | Max Execution Duration of a MicroVM (in Hours) | **8** | +| `L-942E56BE` | Number of MicroVM images | 100 | +| `L-F8BECE9C` | Versions per MicroVM Image | 50 | +| `L-72E0D058` | Number of concurrent MicroVM image builds | 10 | +| `L-535CA9B6` / `L-91B95582` | Rate / burst of `RunMicrovm` | 5 / 5 | +| `L-90045317` / `L-139F9A48` | Rate / burst of `SuspendMicrovm` | 2 / 2 | +| `L-118C44B3` / `L-25EEC0A4` | Rate / burst of `ResumeMicrovm` | 5 / 5 | +| `L-74787B8A` / `L-2CCA0501` | Rate / burst of `TerminateMicrovm` | 10 / 10 | +| `L-7712260B` / `L-D65D9F16` | Rate / burst of `CreateMicrovmAuthToken` | 50 / 50 | +| `L-772D8D8F` … `L-19741F6D` | Concurrent connections per 1/2/4/8/16 vCPU MicroVM | 8 / 16 / 32 / 64 / 128 | + +`L-B430C318 = 8 hours` independently confirms the 28,800-second bound. There is +**no disk quota at all**, corroborating "disk capacity not exposed". +`L-CD1C0CC4` is `QuotaAppliedAtLevel: ACCOUNT`, described as *"The maximum amount +of memory that can be allocated across all MicroVMs per account per region. +Customers can burst up to 4x this limit."* + +**Utilization is not observable.** `get-service-quota` returns **no +`UsageMetric`** for `L-CD1C0CC4`; `AWS/Usage` exposes only `CallCount` per API +name (`RunMicrovm`, `GetMicrovm`, `CreateMicrovmImage`, …) and **no memory +metric**; there is no MicroVM metric in `AWS/Lambda` and no MicroVM CloudWatch +namespace. Saturating the quota would require ~128 × 8 GiB VMs. + +**Verdict: LISTED, NOT PROVEN.** The suspended VM remained in `list-microvms` at +every checkpoint, but the claim *"a suspended VM still holds account memory +quota"* is **NOT OBSERVABLE SAFELY** in this account and remains undischarged. +Note also that the rationale was written for 32,768 MiB per VM against this +1024 GB quota; at the real 8192 MiB ceiling the arithmetic changes by 4×. + +**5.6 — Resume PASSES with no `/resume` hook declared.** Run on a second VM +(`microvm-d4992cba-92a5-328b-b534-d40ef8715ab3`) so the TTL observation was not +disturbed. `resume-microvm` returned an empty body; state `RUNNING` at **+1 s** +and stable through +55 s. **`microvmId` and `endpoint` were byte-identical before +and after** suspend/resume — lifecycle identity is preserved, so a stored +`SessionHandle` survives a suspend/resume cycle. + +**5.7 — Terminate is fast; `NotFound` did NOT arrive.** Terminate requested +20:26:09Z: `TERMINATING` at **+1 s**, `TERMINATED` at **+3 s**, then still +`TERMINATED` at +9/+20/+41/+72/+133/**+254 s**, and still `TERMINATED` at the ++15-min checkpoint (~10 min after termination) and in `list-microvms` at every +later checkpoint. **`ResourceNotFoundException` was never observed.** The +strategy's `NotFound → completed` mapping is not wrong, but it is **not the +near-term signal**: for at least ~10 minutes the observable terminal state is +`TERMINATED`, so `TERMINATED` must map to completed on its own. + +**5.8 — SKIPPED as a role-scoped test** (Lambda role trust does not allow +operator assumption — 5.0). Advisory admin-identity results: + +- Different image ARN → `ResourceNotFoundException: No active version found for + MicroVM image arn:aws:lambda:us-east-1::microvm-image:backgroundagent-dev-abca-agent-different`. + **Inconclusive** for exact-ARN denial, exactly as the runbook predicts. Note + the message is about *no active version*, not a missing image. +- `create-microvm-auth-token --expiration-in-minutes 5 --allowed-ports + '[{"port":8080}]'` → **SUCCEEDED**. The CLI union syntax is valid, and the + request shape matches the SDK. It returned a genuine JWE under key + `X-aws-proxy-auth` (header `{"kid":"9e81880a-…","alg":"dir","enc":"A256GCM"}`), + **minted against a `SUSPENDED` MicroVM**. So tokens are mintable at will by any + principal holding the action, including for suspended VMs; the no-JWE posture + rests entirely on the orchestrator role omitting the action, which 4.4 verified + statically. + +**5.9 — The 16 KB boundary is WRONG: the real cap is 4096.** Both runbook +payloads were rejected identically: + +``` +An error occurred (ValidationException) when calling the RunMicrovm operation: 1 validation error detected: Value at 'runHookPayload' failed to satisfy constraint: Member must have length less than or equal to 4096 +``` + +Re-measured at the real boundary: **4096 bytes passes length validation** (and +then fails only the hook-enabled check), **4097 bytes is rejected** with the +message above. The CLI **does** expand `file://` (a literal URI would have been +33 bytes and passed). Length validation runs **before** the hook-enabled check, +which is why the boundary was measurable on a hook-less image at all. + +`lambda-microvm-strategy.ts:84` sets `RUN_HOOK_PAYLOAD_LIMIT_BYTES = 16_384` and +inlines anything `<= 16384`; ADR-021 states `≤ 16 KB` in four places. Any +envelope between 4,097 and 16,384 bytes would be inlined by the strategy and +**rejected by the service**. No MicroVM was created by these calls, so no +cleanup was needed. + +### Phase 6 + +**6.1 — PASS.** `npx tsc -p cli/tsconfig.json` built cleanly (mise fallback). +`bgagent configure` wrote config (`BGAGENT_CONFIG_DIR=/tmp/abca-645-bgagent`); +`platform outputs` resolved all seven MicroVM outputs. No Cognito login was +needed for operator AWS commands, as documented. + +**6.2 — PASS.** `repo onboard verification-only/issue-645 --compute-type +lambda-microvm` succeeded (the live `ListManagedMicrovmImages` probe passed) with +`status: active`, `compute_type: lambda-microvm`. +`platform doctor` returned `passed: true` with **all six** checks passing, +including `lambda_microvm_availability` — *"Managed MicroVM images are available +in us-east-1"*. (`github_token` also passed: the CFN-generated secret holds a +32-character generated placeholder, which the check cannot distinguish from a +real PAT — worth noting, since the runbook expected doctor to fail here.) +`runtime status` grouped the row under `lambda_microvm_substrates` with +`used_by_repos: ["verification-only/issue-645"]`. +`repo offboard` set `status: removed` with a TTL. Nothing was skipped. + +### Phase 7 + +**7.1 — DEFERRED-TO-P2-ENV — missing Cognito user/login and real repo +onboarding.** The user pool `us-east-1_sYU3Rftw6` has **zero users**, so +`bgagent login` is impossible, and no genuinely accessible repository is +onboarded; the platform GitHub secret is a 32-character generated placeholder. +None of these were created, per the step's instruction. Independently, the task +path could not have produced meaningful classification evidence: `RunMicrovm` +rejects the bare-name identifier the orchestrator injects (4.4 / 5.1), so every +task would fail with a `ValidationException` at launch rather than at +"session start" as the P1 narrative predicts. + +### Phase 8 — teardown (executed) + +**8.1** — `list-microvms` before teardown showed +`microvm-b44b69d9-…` `SUSPENDED` and `microvm-d4992cba-…` `TERMINATED`. +`terminate-microvm` on the suspended VM (a `SUSPENDED` VM terminates directly, +no resume required) → both `TERMINATED`; no non-terminal VM remained. + +**8.2** — Both out-of-band images deleted (`list-microvm-images` now returns +**empty**). **Correction to the runbook's loop:** the *last remaining* version +cannot be deleted individually — +`ValidationException: This is the last version. Please delete the entire image` — +so the correct order is *delete every version except the last, then delete the +image*, which reaps the final version. A version delete in flight also puts the +image in `UPDATING` and makes concurrent calls fail with +`ConflictException: MicroVM Image is already in state: UPDATING` and +`ValidationException: Cannot delete MicroVM image in its current state: `; +both cleared on retry after ~40 s. Versions reaped: `1.0` + `2.0` on +`…-abca-agent` and `1.0` + `2.0` on `…-abca-agent-hooks` — note that **failed +builds still create versions that must be reaped**. `get-microvm-image` on the +first image now returns +`ResourceNotFoundException: MicroVMImage not found for MicroVMImageID: `. + +**8.3 — Stack deletion is INCOMPLETE: `DELETE_FAILED`, blocked by leaked +AgentCore ENIs. All billable resources are confirmed gone.** + +`cdk destroy` ran 21:24:34Z and deleted 460 of 464 resources. It then failed: + +``` +The following resource(s) failed to delete: [AgentVpcRuntimeSG96507CD0, AgentVpcPrivateSubnet1Subnet8051BB57, AgentVpcPrivateSubnet2SubnetC66971D0]. +resource sg-05f50b9950d41572e has a dependent object (Service: Ec2, Status Code: 400 …) +Resource handler returned message: "The subnet 'subnet-0befbffccbb83b718' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400 …)" +``` + +Cause: two ENIs of `InterfaceType: agentic_ai` (AgentCore-managed, requester +`AROA…[redacted]:[redacted]`) — `eni-080353b2356328ed7` and +`eni-04500acbfed377f4a` — remained `in-use` in the private subnets. The runbook's +own gotcha ("VPC teardown can lag while service-managed ENIs are reclaimed; wait +and retry rather than force-deleting resources past CloudFormation") was +followed: **three** `delete-stack` retries spread over ~1 h 40 min (21:48Z, +22:37Z, 23:05Z) all returned `DELETE_FAILED`, and the ENIs were still `in-use` +each time. A direct `delete-network-interface` was attempted once for diagnosis +and correctly refused (`InvalidParameterValue: Network interface … is currently +in use.`); nothing was force-deleted past CloudFormation. + +Final residual state of `backgroundagent-dev` — **4 resources, all zero-cost**: +`AWS::EC2::VPC AgentVpcA6796801` (`vpc-0a12c1a64cc960c6a`), two private subnets, +and one security group, plus the two AgentCore ENIs holding them. + +**Billing is stopped.** Verified after teardown: + +- MicroVMs: both `TERMINATED`; `list-microvm-images` empty (no snapshot storage). +- **NAT gateways: none in the ABCA VPC** (`nat-0c6cdbee97699f4e8` deleted + 21:24:43Z). The two `available` NAT gateways in the account belong to + pre-existing `vpc-01c9984d163d2965e` and were **not** created or touched by + this run. +- **VPC endpoints in the ABCA VPC: none.** +- ABCA S3 buckets: none (auto-delete custom resources emptied them). +- Elastic IPs: no unattached (billable) addresses. +- `/aws/lambda-microvms/backgroundagent-dev-abca-agent`: deleted. + +Retry command for whoever picks this up (should succeed once AgentCore releases +the ENIs): + +```bash +aws cloudformation delete-stack --stack-name backgroundagent-dev +aws cloudformation wait stack-delete-complete --stack-name backgroundagent-dev +``` + +**Verification-only resources created and removed:** the +`abca645-connector-probe` stack (deleted — `Stack with id +abca645-connector-probe does not exist`) and the temporary port-80 egress rule +`sgr-07ed1fa48ef38467a` (removed with its security group when the stack deleted +it). The local `finch` VM was stopped. + +**Bootstrap retained as instructed:** `CDKToolkit` `UPDATE_COMPLETE`, +`ComputeTypes = agentcore,lambda-microvm`, with all five ABCA policies attached +to `cdk-hnb659fds-cfn-exec-role--us-east-1` and **no +`AdministratorAccess`**. + +--- + +## Results table (fill this in) + +Use one row per material observation; add rows as needed. + +| Step | Expected | Observed | ADR item discharged | Feeds back to design? | +|---|---|---|---|---| +| Setup | Vars + evidence dir | `us-east-1`, `backgroundagent-dev`, evidence `/tmp/abca-645-p1-20260731T184822Z`. `mise` absent → all **raw fallbacks** used | Reproducibility | No | +| 0.1 | Virgin account, correct branch | Account ``, admin `AdminConsoleAccess/aamorosi-Isengard`, branch OK, SHA `0505f914`. `backgroundagent-dev` absent (`ValidationError … does not exist`). **Account not virgin overall** — 3 unrelated stacks + `CDKToolkit` pre-existed; ABCA itself never deployed, so the premise holds | Clean first-deploy path | No | +| 0.2 | CLI exposes Lambda MicroVMs; SDK 3.1098.0 | `aws lambda-microvms help` **exit 0**, 24 commands, all 7 required present. CLI command list is an **exact match** to SDK 3.1098.0 (24 vs 24). CLI 2.36.13, cdk 2.1129.0, node 24.16.0, python 3.9.6, **openrsync** (worked). Skeleton confirms `ARM_64`-only and `ENABLED\|DISABLED` hooks with a single `hooks.port` and **no hook-path field** | API/action-name verification | **Yes — CFN L1 shapes (`arm64`, `run:'/run'`) have no counterpart in the service model** | +| 1.2 | Custom template replaces admin bootstrap | **DEFECT: `cdk bootstrap --template …` is a silent no-op** on an already-bootstrapped account (`Not overwriting it with a template containing 'ABCA: Least-Privilege Bootstrap' (use --force …)`, **exit 0**). `--force` required. `BootstrapVariant` then *stays* `AWS CDK: Default Resources`, so every future non-forced bootstrap refuses again. **DEFECT: the runbook's `--parameters` shorthand is rejected** (`Invalid type for parameter Parameters[0].ParameterValue, value: ['agentcore', 'lambda-microvm'] … valid types: `); the escaped `agentcore\,lambda-microvm` works | Conditional bootstrap IAM | **Yes — `mise //cdk:bootstrap` + runbook/docs** | +| 1.3 | Custom policy attached for `agentcore,lambda-microvm` | `ComputeTypes=agentcore,lambda-microvm`; exactly one `IaCRole-ABCA-Compute-LambdaMicrovms` policy, attached to `cdk-hnb659fds-cfn-exec-role-…`; 5 ABCA policies, **no `AdministratorAccess`**; 19 MicroVM/connector actions incl. `PassNetworkConnector` | Conditional bootstrap IAM | No | +| 2.1 (synth) | No-image warning | `abca:microvm-image-not-provisioned` present, `…p1-not-runnable` correctly absent. Incidental: template 893273/1000000, 463/500 resources | First-deploy bootstrap state | No | +| 2.1 (deploy) | Substrate deploys | **BLOCKED — cannot deploy from unmodified sources.** `AWS::Lambda::NetworkConnector` CREATE_FAILED: `"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type (… Status Code: 400 …)" HandlerErrorCode: InvalidRequest`. Refutes the construct's stated *"`operatorRole` is left unset so Lambda manages the ENIs with its own service-linked role"*. Deployed only after patching in an operator role (+ moving off `us-east-1a`): `CREATE_COMPLETE` in **13 min 42 s** | Conditional substrate — **only with a code fix** | **YES — construct must create + pass an operator role; L1 marks it optional** | +| 2.1 (deploy, 2nd) | — | **AgentCore blocker:** `The following subnets are in unsupported availability zones in region us-east-1: subnet-… in us-east-1a (ID: use1-az6). Supported availability zones are: use1-az4, use1-az1, use1-az2`. This account maps `us-east-1a`→`use1-az6`; `AgentVpc` does not constrain AZs | — | **YES — `AgentVpc` should pin AgentCore-supported AZs** | +| 2.1 (rollback) | — | Rollback itself failed: `Validation failed during DeleteMemory: Memory is in transitional state CREATING. Cannot delete memory.` → `ROLLBACK_FAILED`; plain `delete-stack` cleared it | — | Minor — yes (`AgentMemory` delete retry) | +| 2.2 | Outputs/resources present | `ComputeSubstrate=lambda-microvm`; all six `Microvm…` outputs populated; key `microvm-images/agent-artifact.zip`; 2 buckets, build+execution roles, **443-only SG** (`sg-0e662dc0d6f6e9ade`, one rule tcp/443), `/aws/lambda-microvms/…` log group, `AWS::Lambda::NetworkConnector`; **no `AWS::Lambda::MicrovmImage`** | Conditional substrate/config | No | +| 2.3 | No `MICROVM_*` env | `[]` ✓ (14 env keys). **DEFECT: the runbook's `ORCHESTRATOR_FN` query is broken by pagination** — 464 resources → the query ran per page and returned 5 values (`None None None None `), breaking the next call. Needs `--no-paginate` | Reject-without-image config | Yes — runbook | +| 2.4 | Construct resources have backend tag | All **6** taggable resources tagged `abca:compute-backend=lambda-microvm` (SG, connector, log group, 2 buckets, 2 roles). `resourcegroupstaggingapi` returned only 5 — **IAM roles are not returned by that API** (coverage gap, not a missing tag); confirmed via `iam list-role-tags` | Cost attribution | No | +| 3.1–3.2 | Unsupported failure; override warning | 3.1 exit 1 naming `eu-central-1`, all five Regions, and `--context microvm_region_override=true`. 3.2 exit 0 with `abca:microvm-region-override` (plus the not-provisioned warning) | Region gate/escape hatch | No | +| 4.1 | Live regional probe | Exactly **one** base image: `…:aws:microvm-image:al2023-1`, versions `1` and `0`. **Ordering is newest-first, so the runbook's `items[-1]` selects the OLDER version `0`**; `items[0]`=`1` is correct. Service echoes `baseImageVersion: "1.0"` | Regional availability probe | Yes — runbook selector | +| 4.2 (artifact) | Script uploads | Staged + zipped + uploaded fine on macOS/openrsync: script printed `584K artifact`, S3 `ContentLength 597305`, SSE `AES256` | Packaging plane | No | +| 4.2 (create) | Image created; P1 banner shown | **FAILED:** `The ready (/ready) MicroVM image hook must be enabled when any MicroVM lifecycle hook (run, resume, suspend, or terminate) is enabled.` → **the P1 banner was never printed** (it comes after the failing call), and the runbook's `2>&1 \| tee` reported `EXIT=0`, masking the failure | **NOT discharged** — packaging + operator warning | **YES — "declare `/run` in P1, serve it in P2" is not a reachable state** | +| 4.2 (memory) | 32,768 MiB accepted | **FAILED:** `The requested memory size of 32768 MiB is not supported by base MicroVM image …al2023-1. Supported memory sizes in MiB are: [512, 1024, 2048, 4096, 8192].` Real ceiling **8192 MiB (8 GiB)**, ¼ of the documented figure | Refutes sizing premise | **YES — `DEFAULT_MINIMUM_MEMORY_MIB` and ADR-021's "32 GB RAM"** | +| 4.3 (build 1) | Build successful | **FAILED** (`The container image build failed.`). Root cause in the log group: `Could not connect to deb.debian.org:80 (146.75.38.132), connection timed out` → `E: Unable to locate package curl/git/build-essential` → `exit code: 100`. **The construct's 443-only SG makes the agent image unbuildable** (`apt-get` needs port 80; DNS was fine) | **NOT discharged** without a fix | **YES — SG must allow 80, or the Dockerfile must not use HTTP apt** | +| 4.3 (build 2) | Image ACTIVE | After adding a temporary port-80 egress rule: version **`2.0`** `state=SUCCESSFUL`, `status=ACTIVE` in **5 min 51 s**, both builds `SUCCESSFUL` | Buildability (with fixes) | No | +| 4.3 (shapes) | `IMAGE_VERSION=1` | **Version is `1.0`, not `1`.** **Two builds per version** (`chipsetGeneration` 3 and 4, GRAVITON) — the runbook's `items[0]` checks only one. `list-microvm-image-builds --image-identifier ` → `ValidationException: Invalid ARN format: …` (**ARN required**). `snapshotBuild` lives on `get-microvm-image-build`, **not** on the version (which returns `null`) | State/shape mapping | Yes — runbook + script | +| 4.3 (sizes) | Size vs 2 GB narrative | `codeInstallSizeInBytes` **2,334,748,672 (2.17 GiB) — exceeds AgentCore's 2 GB container-image limit**; `memorySnapshotSizeInBytes` 1,216,577,536 (1.13 GiB); `diskSnapshotSizeInBytes` 37,089,280 (35.4 MiB). Same tree as an OCI image = 1.799 GB (629.7 MB compressed). Snapshots deliberately not summed | Sizing narrative | **Yes — state which measure the 2 GB comparison uses** | +| 4.3 (disk) | Verify 32 GB disk | **NOT EXPOSED** — no disk quota in `service-quotas`, no disk field on image/version. 32 GB disk unverified; 32 GB *memory* refuted | Disk external fact | Yes | +| 4.3 (P1 shape) | — | **Decisive:** the exact P1 hook shape + service-mandated `/ready` **FAILED both builds**: `Ready hook check failed: the application returned a client error (HTTP 4xx) response`. The agent *does* answer on 8080 but not `/ready`. **A P1 image declaring `/run` is not creatable at all** | Refutes P1 premise | **YES — P1/P2 hook phasing** | +| 4.4 | Env present; exact image IAM; no JWE grant | **PASS exactly as designed.** `abca:microvm-image-p1-not-runnable` emitted. Exactly 5 `MICROVM_*` vars, no ingress var. `MicrovmLifecycle` = exactly `RunMicrovm`/`GetMicrovm`/`TerminateMicrovm` on `…:microvm-image:backgroundagent-dev-abca-agent` + `:*`; `MicrovmPassNetworkConnector` on `*`; `MicrovmPassExecutionRole` with `iam:PassedToService=lambda.amazonaws.com`; **zero** Suspend/Resume/AuthToken actions | Least privilege | No | +| 4.4 (identifier) | Bare name accepted by RunMicrovm | **REFUTED.** `MICROVM_IMAGE_IDENTIFIER` is the bare name `backgroundagent-dev-abca-agent`; `run-microvm` with a bare name → `ValidationException: Malformed ARN - doesn't start with 'arn:'`. `lambda-microvm-strategy.ts:236` passes it straight through, so P1 would fail at launch | Refutes construct comment | **YES — inject the image ARN, not the name** | +| 5.0 | Assume-role identity or trust denial | **ADMIN — advisory.** `AccessDenied … not authorized to perform: sts:AssumeRole on resource: …TaskOrchestratorOrchestratorFnS-Bd7rBa2V6Jwf`; trust = `lambda.amazonaws.com` only. Trust not modified | Verification confidence | No | +| 5.1 | Hook-less behavior measured, not assumed | **Measured: it runs and keeps running.** `RUNNING` at **+12 s**, still `RUNNING` at +145 s, `stateReason` always `None` — no terminate, no stall, no disappearance. `maximumDurationInSeconds=28800` accepted, `idlePolicy` omitted. A payload on a hook-less image is rejected: `The run hook must be enabled in the MicroVM image to pass the run hook payload` | P1/P2 phase boundary | **Yes — P2 startup/hooks** | +| 5.1 (ingress) | No ingress | **Service auto-attached `…:aws:network-connector:aws-network-connector:HTTP_INGRESS`** with a public `*.lambda-microvm.us-east-1.on.aws` endpoint, though none was requested. "No ingress" is not the service default | Security posture | **YES — P1 must suppress or accept default public ingress** | +| 5.2 | Actual state enum values recorded | Observed 5 of 6: `PENDING`, `RUNNING`, `SUSPENDED`, `TERMINATING`, `TERMINATED`. **`SUSPENDING` never observable** (<1 s). No unknown state. `ResourceNotFoundException` not reached | State mapping | Yes (see 5.7) | +| 5.3 | Manual suspend without idle policy | **PASS.** Admin suspend on a hook-less image, no `idlePolicy`: **empty response body**, `SUSPENDED` at **+1 s**, stable | Explicit suspend external fact | **Yes — P3 lifecycle** | +| 5.4 | Suspended TTL/checkpoint result | **No TTL within 1 h.** `SUSPENDED` at start / +15 min / +45 min / +1 h (3617 s); `startedAt` and `maximumDurationInSeconds=28800` unchanged. **TRUNCATED at ~1 h**; the 4 h checkpoint was NOT run, so a TTL between 1 h and the 8 h bound is **still open** | Partially — bounded below only | **Yes — timeout policy** | +| 5.5 | Suspended quota consumption proven/inconclusive | **LISTED, NOT PROVEN — NOT OBSERVABLE SAFELY.** Suspended VM present in `list-microvms` at every checkpoint. `L-CD1C0CC4 Max allocated memory = 1024 GB` (ACCOUNT, "burst up to 4x"), **no `UsageMetric`**; `AWS/Usage` has only `CallCount`; no MicroVM memory metric anywhere. Proving it needs ~128 × 8 GiB VMs. `L-B430C318 = 8 hours` independently confirms the 28,800 s bound; **no disk quota exists** | **NOT discharged** | **Yes — concurrency policy; rationale was sized on 32 GiB/VM, real is 8 GiB** | +| 5.6 | Resume transitions/result | **PASS with no `/resume` hook declared.** `RUNNING` at **+1 s**, empty response body; **`microvmId` and `endpoint` byte-identical** across suspend→resume, so a stored `SessionHandle` survives | Resume external fact | **Yes — P3 hooks/reconciliation** | +| 5.7 | Terminate→NotFound timing | `TERMINATING` **+1 s** → `TERMINATED` **+3 s**, then `TERMINATED` at +254 s and still `TERMINATED` ~10 min later and at every later checkpoint. **`ResourceNotFoundException` never observed** | Partially — terminate path yes, `NotFound` mapping no | **YES — `TERMINATED` must map to completed; `NotFound` is not the near-term signal** | +| 5.8 | Different image/JWE denied under role | **SKIPPED — LAMBDA ROLE TRUST DOES NOT ALLOW OPERATOR ASSUMPTION.** Advisory (admin): different image ARN → `ResourceNotFoundException: No active version found for MicroVM image …-different` (**inconclusive**, as predicted). `create-microvm-auth-token … --allowed-ports '[{"port":8080}]'` **SUCCEEDED** as admin, returning a real JWE under `X-aws-proxy-auth` (`{"alg":"dir","enc":"A256GCM"}`) **against a SUSPENDED VM**; CLI union syntax valid. No-JWE posture rests solely on 4.4's role omission | Static only (4.4) | Yes — tokens are mintable for suspended VMs | +| 5.9 | 16,384 accepted; 16,385 rejected | **REFUTED — the cap is 4096, not 16,384.** Both 16,384 and 16,385 → `Value at 'runHookPayload' failed to satisfy constraint: Member must have length less than or equal to 4096`. Re-measured: **4096 passes, 4097 rejected**. CLI does expand `file://`. Length validation precedes the hook check. `RUN_HOOK_PAYLOAD_LIMIT_BYTES = 16_384` would inline 4,097–16,384-byte envelopes that the service rejects | **Refutes the documented boundary** | **YES — strategy threshold + ADR-021 (4 places)** | +| 6.1–6.2 | Onboard probe, doctor, grouping, cleanup | **PASS, nothing skipped.** `npx tsc` build clean; `platform outputs` resolved all 7 MicroVM outputs; onboard probe passed with `compute_type=lambda-microvm`, `status=active`; doctor `passed: true` with all 6 checks including `lambda_microvm_availability` ("Managed MicroVM images are available in us-east-1"); `runtime status` grouped under `lambda_microvm_substrates`; offboard `status=removed` + TTL. Note `github_token` **passed** on a 32-char generated placeholder | CLI regional enforcement | Minor — yes (`github_token` can't detect a placeholder) | +| 7.1 | Negative task evidence or explicit deferral | **DEFERRED-TO-P2-ENV — missing Cognito user/login, real GitHub token, and real repo onboarding.** User pool `us-east-1_sYU3Rftw6` has **zero users**; secret is a 32-char placeholder. Not created, per the step. Independently moot: `RunMicrovm` rejects the bare-name identifier, so tasks would fail at launch, not at session start | Not discharged (by design) | **Yes — P2 env** | +| 8.1–8.2 | VMs/images gone | **PASS.** Both MicroVMs `TERMINATED` (a `SUSPENDED` VM terminates directly, no resume needed). `list-microvm-images` **empty**. **Correction:** the last remaining version cannot be deleted alone (`This is the last version. Please delete the entire image`) — delete all but the last, then the image. Concurrent calls during a version delete give `ConflictException: MicroVM Image is already in state: UPDATING`. **Failed builds still create versions that must be reaped** (`1.0`+`2.0` on both images) | Versioned image lifecycle | Yes — runbook loop order | +| 8.3 | Stack gone; bootstrap retained | **PARTIAL — `DELETE_FAILED`.** 460/464 resources deleted; VPC + 2 private subnets + 1 SG remain, blocked by two leaked AgentCore ENIs (`InterfaceType: agentic_ai`) still `in-use` after 3 retries over ~1 h 40 min. **All billable resources confirmed gone** (no ABCA NAT gateway, no VPC endpoints, no buckets, no images/VMs, no unattached EIPs); residual 4 resources are zero-cost. Nothing force-deleted past CFN. `CDKToolkit` retained `UPDATE_COMPLETE`, `ComputeTypes=agentcore,lambda-microvm`, 5 ABCA policies, no `AdministratorAccess`. Verification-only extras (`abca645-connector-probe`, temp rule `sgr-07ed1fa48ef38467a`) removed | Partially — cleanup blocked by AgentCore | **YES — AgentCore ENI reclaim blocks clean `cdk destroy`** | + +## Findings summary + +Live run, 2026-07-31, account ``, `us-east-1`, branch +`feat/645-lambda-microvm-p1` @ `0505f914`. Evidence: +`/tmp/abca-645-p1-20260731T184822Z`. + +**Headline:** the *P1 substrate* is broadly correct — conditional bootstrap IAM, +outputs, tags, region gate, warnings, exact-ARN least privilege, and the CLI +surface all behave as designed. But **P1 cannot deploy, cannot build its image, +and cannot launch a MicroVM from unmodified sources**: five independent +live-service rejections had to be worked around to get any empirical result, and +three documented ADR-021 constants (32 GB memory, 16 KB payload, bare-name image +identifier) are **wrong**. + +### Items discharged (behaved exactly as designed) + +1. **0.2** — CLI/SDK operation names: `aws lambda-microvms` exposes 24 commands, + an exact match to SDK 3.1098.0. No action-name drift; the packaging script's + `ARM_64` / `ENABLED` shapes are confirmed correct against the live model. +2. **1.3** — Conditional bootstrap IAM: `IaCRole-ABCA-Compute-LambdaMicrovms` is + created and attached only with `ComputeTypes` including `lambda-microvm`, and + `AdministratorAccess` really is replaced. +3. **2.2** — Substrate contract: `ComputeSubstrate=lambda-microvm`, all six + `Microvm…` outputs, both buckets, both roles, 443-only SG, `/aws/lambda-microvms/` + log group, network connector, and **no** `AWS::Lambda::MicrovmImage`. +4. **2.3** — All-or-nothing config: zero `MICROVM_*` env vars without an image. +5. **2.4** — Cost tags: all six taggable construct resources carry + `abca:compute-backend=lambda-microvm`. +6. **3.1 / 3.2** — Region gate and escape hatch, verbatim as specified. +7. **4.1** — Live regional availability probe works. +8. **4.2 (artifact half)** — Packaging plane: zip+Dockerfile staging, no secret + build inputs, correct bucket/key, works on macOS with `openrsync`. +9. **4.4** — Least privilege: exactly `RunMicrovm`/`GetMicrovm`/`TerminateMicrovm` + on exactly the image ARN + `:*`; `PassNetworkConnector`; scoped `iam:PassRole`; + **zero** `SuspendMicrovm`/`ResumeMicrovm`/`CreateMicrovmAuthToken`. Both + no-image and image-configured warnings fire correctly. +10. **5.3** — Manual suspend works without any `idlePolicy` (`SUSPENDED` in ~1 s). +11. **5.6** — Manual resume works with no `/resume` hook declared; `microvmId` + **and** `endpoint` are preserved, so `SessionHandle` survives a cycle. +12. **5.7 (terminate half)** — Explicit terminate is near-instant + (`TERMINATING` +1 s → `TERMINATED` +3 s). +13. **6.1 / 6.2** — CLI: outputs discovery, live `ListManagedMicrovmImages` + onboarding probe, `lambda_microvm_availability` doctor check, + `lambda_microvm_substrates` grouping, and offboard all pass. +14. **8.1 / 8.2** — MicroVM and image cleanup paths work (with the version-order + correction below). + +### Items contradicting design assumptions — `feeds-back-to-design: YES` + +Ordered by severity. + +**F1. The P1 image is not creatable at all** (blocks the entire P1 premise). +`create-microvm-image` with the script's/construct's hook shape: + +``` +ValidationException: The ready (/ready) MicroVM image hook must be enabled when any MicroVM lifecycle hook (run, resume, suspend, or terminate) is enabled. The ready hook signals when the application has finished initializing so the snapshot is taken in a ready state. +``` + +And with `/ready` added as demanded, both builds fail: + +``` +Ready hook check failed: the application returned a client error (HTTP 4xx) response +``` + +ADR-021's hook-phasing plan ("declare `/run` in P1, serve it in P2; omit `/ready` +and `/validate` because the agent does not implement them") is **not a reachable +service state**. The only creatable P1 image is one with **no hooks at all**, and +such an image **cannot accept a `runHookPayload`** +(`The run hook must be enabled in the MicroVM image to pass the run hook +payload`) — so P1's payload-delivery path cannot function either. The agent does +answer on port 8080 (HTTP 4xx, not a connection refusal), so serving `/ready` +is the unblocking change. + +**F2. The substrate cannot deploy: the network connector requires an operator +role.** + +``` +"NetworkConnectorOperatorRole is required for VPC_EGRESS connector type (Service: Lambda, Status Code: 400, Request ID: 04726267-6c61-4ff5-bb1d-302122e9f955)" HandlerErrorCode: InvalidRequest +``` + +This refutes the explicit comment in `lambda-microvm-compute.ts` (~L467): +*"`operatorRole` is left unset so Lambda manages the ENIs with its own +service-linked role rather than a role we would have to trust."* The generated L1 +also mis-signals it as optional (`readonly operatorRole?: string`). Proven fix +(validated standalone): a role trusting `lambda.amazonaws.com` with +`AWSLambdaVPCAccessExecutionRole` + `ec2:CreateNetworkInterface` / +`DeleteNetworkInterface` / `DescribeNetworkInterfaces` / `DescribeSubnets` / +`DescribeVpcs` / `DescribeSecurityGroups` / `CreateTags` / +`AssignPrivateIpAddresses` / `UnassignPrivateIpAddresses` / +`Describe|ModifyNetworkInterfaceAttribute`. + +**F3. `RunMicrovm` requires an image ARN; the orchestrator injects a bare name.** + +``` +ValidationException: Malformed ARN - doesn't start with 'arn:' +``` + +`MICROVM_IMAGE_IDENTIFIER` is set to `backgroundagent-dev-abca-agent` and +`lambda-microvm-strategy.ts:236` passes it straight to `RunMicrovm`. The +construct's comment claims *"`run-microvm --image-identifier` accepts"* bare +names — it does not. Every P1 task would fail at launch. The same applies to +`list-microvm-image-builds` (`ValidationException: Invalid ARN format: …`), which +the packaging script's operator instructions also get wrong. Note the construct +already derives the correct ARN for IAM, so the fix is to inject that ARN. + +**F4. The 443-only security group makes the agent image unbuildable.** + +``` +Could not connect to deb.debian.org:80 (146.75.38.132), connection timed out +E: Unable to locate package curl / git / build-essential +… did not complete successfully: exit code: 100 +``` + +`agent/Dockerfile` runs `apt-get`, which uses **HTTP/80**; the construct's SG +allows only 443 (DNS resolution succeeded, so the port is the sole cause). Either +the SG must allow 80 for build-time egress, or the Dockerfile must use an +HTTPS apt transport/mirror. Opening port 80 made the build succeed immediately. + +**F5. Memory: 32,768 MiB is rejected; the real ceiling is 8,192 MiB.** + +``` +ValidationException: The requested memory size of 32768 MiB is not supported by base MicroVM image arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1. Supported memory sizes in MiB are: [512, 1024, 2048, 4096, 8192]. +``` + +`DEFAULT_MINIMUM_MEMORY_MIB = 32768` is documented in the construct as *"the +service ceiling"*, and ADR-021 states **32 GB RAM** in at least three places +(comparison table, constraint paragraph, consequences). The real ceiling for the +only available base image (`al2023-1`) is **8 GiB — one quarter**. This +materially changes the "MicroVMs target the default-sized workload" positioning +*and* the 5.5 concurrency arithmetic (which was sized on 32 GiB/VM). + +**F6. `runHookPayload` cap is 4096 bytes, not 16,384.** + +``` +ValidationException: 1 validation error detected: Value at 'runHookPayload' failed to satisfy constraint: Member must have length less than or equal to 4096 +``` + +Boundary measured exactly: **4096 passes, 4097 fails**. Both of the runbook's +16 KB probes failed. `RUN_HOOK_PAYLOAD_LIMIT_BYTES = 16_384` +(`lambda-microvm-strategy.ts:84`) would inline every envelope from 4,097 to +16,384 bytes and the service would reject all of them; ADR-021 repeats `≤ 16 KB` +in four places, including the P1 requirement statement. + +**F7. `RunMicrovm` attaches a default public HTTP ingress connector.** +Without `--ingress-network-connectors`, the response contained: + +``` +"ingressNetworkConnectors": ["arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:HTTP_INGRESS"] +``` + +plus a public `*.lambda-microvm.us-east-1.on.aws` endpoint. ADR-021's "no +orchestrator→agent HTTP path / no ingress in P1–P3" posture is **not the service +default**; P1 either has to suppress it explicitly or document that a public +ingress endpoint exists on every agent MicroVM. + +**F8. `NotFound` is not the near-term terminal signal.** After terminate, the VM +was `TERMINATED` at +3 s and **still `TERMINATED` ~10 min later** and at every +subsequent checkpoint; `ResourceNotFoundException` was **never observed**. The +strategy's `NotFound → completed` mapping is fine as a fallback, but `TERMINATED` +must map to completed in its own right or the orchestrator will poll a terminal +VM indefinitely. Relatedly, **`SUSPENDING` is not observable** (suspend reaches +`SUSPENDED` in <1 s), so any state machine that waits for `SUSPENDING` will hang. + +**F9. Hook-less MicroVMs run indefinitely and bill.** A P1-style hook-less image +reaches `RUNNING` in **12 s** and stays `RUNNING` (no `stateReason`, no +self-termination) up to its 8 h `maximumDurationInSeconds`. The P1 narrative +("a launch may return an ID and endpoint and then fail its hook or terminate") +is wrong in the safe direction for correctness and wrong in the *expensive* +direction for cost: nothing fails, so nothing cleans up. Active +`TerminateMicrovm` is mandatory, not belt-and-braces. + +**F10. `mise //cdk:bootstrap` silently no-ops on an already-bootstrapped +account** — `Not overwriting it with a template containing 'ABCA: Least-Privilege +Bootstrap' (use --force if you intend to overwrite)` with **exit 0**. Worse, after +a forced bootstrap `BootstrapVariant` remains `AWS CDK: Default Resources`, so +the refusal recurs forever. Any operator following ADR-002's documented flow on an +existing account keeps `AdministratorAccess`. + +**F11. `AgentVpc` picks AZs AgentCore rejects.** +`The following subnets are in unsupported availability zones in region us-east-1: +subnet-… in us-east-1a (ID: use1-az6). Supported availability zones are: +use1-az4, use1-az1, use1-az2`. AZ *names* are account-scoped, so this is a +latent first-deploy failure for any account whose `us-east-1a` maps to `use1-az6`. + +**F12. AgentCore leaks ENIs and blocks `cdk destroy`.** Two +`InterfaceType: agentic_ai` ENIs stayed `in-use` for >1 h 40 min after runtime +deletion, leaving the stack `DELETE_FAILED` with VPC/subnets/SG undeletable. +Also `AWS::BedrockAgentCore::Memory` cannot be deleted while `CREATING` +(`Validation failed during DeleteMemory: Memory is in transitional state +CREATING`), which turned one rollback into `ROLLBACK_FAILED`. + +**F13. `codeInstallSizeInBytes` = 2.17 GiB exceeds AgentCore's 2 GB image +limit**, while the same tree as an OCI image is 1.799 GB (629.7 MB compressed). +The ADR's "compare to AgentCore's 2 GB limit" narrative must say *which* measure +it means, because the two straddle the limit. + +**F14. Runbook/tooling defects found in execution** (lower severity, but they +would silently corrupt a future pass): + +- The `--parameters 'ParameterKey=ComputeTypes,ParameterValue=agentcore,lambda-microvm'` + form is rejected (`Invalid type for parameter … valid types: `); + needs `agentcore\,lambda-microvm`, as `cdk/mise.toml` already shows. +- `list-stack-resources --query "…|[0]"` is **paginated** at 464 resources and + returned five values, breaking `ORCHESTRATOR_FN`. Needs `--no-paginate`. +- `list-managed-microvm-image-versions` is **newest-first**, so `items[-1]` + selects the *older* version. +- Image version is **`1.0`**, not `1`, so `IMAGE_VERSION=1` is wrong. +- There are **two builds per version** (GRAVITON gen 3 and 4); `items[0]` checks + only one. +- `snapshotBuild` comes from `get-microvm-image-build`, not + `get-microvm-image-version` (which returns `null`). +- The script's failure is masked by `2>&1 | tee` (reported `EXIT=0`), and its + "P1 image is NOT runnable" banner never prints because it sits *after* the + `create-microvm-image` call that fails. +- Teardown: the **last** image version cannot be deleted individually + (`This is the last version. Please delete the entire image`). +- Post-synth cloud-assembly template edits are ignored if the template's S3 + asset object already exists (key = pre-edit content hash). + +### Items skipped, blocked, or inconclusive + +| Item | Verdict | Reason | +|---|---|---| +| 1.3 optional negative (scratch-qualifier bootstrap) | **SKIPPED** | The runbook directs skipping it; not cheap enough and complicates teardown. | +| CFN `AWS::Lambda::MicrovmImage` value shapes (`arm64`, `run:'/run'`) | **NOT TESTED** | The construct only synthesizes the L1 with `microvm_base_image_arn`/`_version` context, which the runbook's Phase 4 does not use. The API side is settled (0.2) and the request would be rejected on hook semantics anyway (F1), so the CFN-vs-API shape question stays **open**. | +| 5.4 suspended TTL beyond 1 h | **TRUNCATED / OPEN** | Time-boxed at ~1 h as the runbook permits. `SUSPENDED` held at start/+15/+45/+60 min (3617 s). The 4 h checkpoint was not run; a TTL between 1 h and the 8 h bound remains unknown. Bounded result: **survives ≥ 1 h with no `idlePolicy`**. | +| 5.5 suspended VM consumes account memory quota | **NOT OBSERVABLE SAFELY / UNDISCHARGED** | The VM stays in `list-microvms`, but that only proves *listed*. `L-CD1C0CC4` (1024 GB, ACCOUNT) exposes **no `UsageMetric`**; `AWS/Usage` has only `CallCount`; no MicroVM memory metric exists in any namespace; no console utilization view is reachable from a CLI-only session. Proving it would need ~128 × 8 GiB VMs. | +| 5.8 IAM negatives under the orchestrator role | **SKIPPED — LAMBDA ROLE TRUST DOES NOT ALLOW OPERATOR ASSUMPTION** | `AccessDenied … sts:AssumeRole`; trust is `lambda.amazonaws.com` only and was deliberately not modified. 4.4's static policy inspection is authoritative. | +| 5.8 exact-ARN denial sub-check | **INCONCLUSIVE** | As the runbook predicts, a different image name returns `ResourceNotFoundException: No active version found for MicroVM image …-different`, not `AccessDenied`. Advisory only (admin identity). | +| 5.8 no-JWE posture | **STATIC ONLY** | As admin, `create-microvm-auth-token` **succeeded**, returning a real JWE (`{"alg":"dir","enc":"A256GCM"}`, key `X-aws-proxy-auth`) — **against a `SUSPENDED` VM**. The posture depends entirely on the role omitting the action. | +| 7.1 negative task path | **DEFERRED-TO-P2-ENV** | Cognito pool `us-east-1_sYU3Rftw6` has **zero users**, no real repo onboarded, GitHub secret is a 32-char generated placeholder. Not created, per the step. Independently moot given F3. | +| 8.3 stack deletion | **DELETE_FAILED (billing stopped)** | Leaked AgentCore ENIs (F12). 4 zero-cost resources remain; all billable resources verified gone. Retry command recorded in 8.3. | +| Disk capacity / 32 GB disk claim | **NOT EXPOSED** | No disk quota in `service-quotas`, no disk field on image or version. Unverified. | + +### Elapsed and approximate cost + +**Elapsed:** 18:48Z → 23:07Z = **4 h 19 min** wall clock. Of that, ~1 h was the +suspend-TTL observation (run concurrently with the CLI phase, IAM checks, the +16 KB probes, and the second image build, per instructions — no idle waiting); +~1 h 15 min was consumed by the three blocked deploys plus rollbacks/redeploys; +~1 h 40 min was teardown retries. + +**Approximate cost: well under US$10, dominated by NAT gateways and VPC +endpoints, not by MicroVMs.** + +| Item | Quantity | Est. | +|---|---|---| +| NAT gateways (2 × $0.045/h) | ~2.3 h summed across 4 stack lifetimes | ~$0.21 + trivial data | +| Interface VPC endpoints (7 × $0.01/h × 2 AZ) | ~1.6 h | ~$0.22 | +| MicroVM runtime | VM1 ~3 min `RUNNING` + ~2 h 45 min `SUSPENDED`; VM2 ~3 min | < $0.50 (suspended compute is not billed; snapshot storage was < 3 h) | +| MicroVM image builds | 6 builds (2 versions × 2 chipsets × 2 images), ~6 min each | low single-digit $ at most | +| Snapshot/image storage | ~3.5 GiB × 2 images × < 3 h | negligible | +| AgentCore runtime | created 4×, never invoked | negligible | +| S3 / DynamoDB / Lambda / API GW / Cognito / Secrets / logs | brief, mostly idle | < $1 | +| ECR container asset (retained in bootstrap) | 630 MB stored | ~$0.06/month ongoing | + +The 8 h `maximumDurationInSeconds` worst case was never approached; both VMs were +explicitly terminated. + +### Deliberately left in place + +1. **`CDKToolkit`** — retained as the runbook instructs, now carrying + `ComputeTypes=agentcore,lambda-microvm` and the five ABCA least-privilege + policies **instead of `AdministratorAccess`**. ⚠️ This is a change to a + *shared* account: other CDK apps in `` now deploy through the + ABCA-scoped execution role. The original standard-bootstrap template is + captured at `$EVIDENCE_DIR/cdktoolkit-template-before.txt` if it needs + restoring. +2. **`backgroundagent-dev` in `DELETE_FAILED`** — VPC `vpc-0a12c1a64cc960c6a`, + two private subnets, one security group, and two leaked AgentCore ENIs. Zero + cost; retry `delete-stack` once AgentCore releases the ENIs. +3. **Bootstrap S3/ECR assets** — including the 630 MB agent container image, + normal bootstrap content. +4. **Service-vended log groups** — `/aws/bedrock-agentcore/runtimes/…` and + `/aws/lambda/backgroundagent-dev-…` created outside CloudFormation. + +Untouched and **not** created by this run: the pre-existing stacks +(`serverless-api-powertools`, `BuildingServerlessAPIs`, +`aws-sam-cli-managed-default`) and the two `available` NAT gateways in +`vpc-01c9984d163d2965e`. + +### Recommended follow-up before P1 merges + +F1, F2, F3, F4, F5, and F6 are each independently sufficient to make the +`lambda-microvm` backend non-functional. F1 (the `/ready` requirement) is the one +that changes the *shape* of the phase plan rather than a constant, so it should be +adjudicated first: either P1 grows a minimal `/ready` (and `/run`) responder, or +P1 ships a hook-less image and explicitly defers all payload delivery to P2. + +## Report back + +Attach or summarize the evidence paths, then provide the filled results table to +the orchestrator. Draft it as a comment for issue #645; **do not post it until +the orchestrator reviews it**. Highlight every `BLOCKED`, `INCONCLUSIVE`, +`SKIPPED`, and `DEFERRED-TO-P2-ENV` result and explicitly separate observed AWS +behavior from expectations inferred from the SDK model. diff --git a/docs/verification/645-p2-smoke-runbook.md b/docs/verification/645-p2-smoke-runbook.md new file mode 100644 index 000000000..d7d94e4e5 --- /dev/null +++ b/docs/verification/645-p2-smoke-runbook.md @@ -0,0 +1,1905 @@ +# ADR-021 P2 Stage D — live smoke runbook + +Working verification document for issue #645 on branch +`feat/645-lambda-microvm-p2` @ `3a4b61a97b22b7bcdd9832101f8d61a077fbf103`. +Companion to [`645-p1-lambda-microvm-runbook.md`](./645-p1-lambda-microvm-runbook.md), +whose command incantations this run reuses wholesale (escaped `ComputeTypes` +comma, `bootstrap --force`, newest-first version ordering, image-version `N.0` +spelling, two builds per version, delete-all-but-last then delete-image, +teardown-as-finally). + +`docs/scripts/sync-starlight.mjs` does not mirror `docs/verification/`, so this +file intentionally stays here and is not part of the Starlight site. + +**Live run:** 2026-08-06 22:38Z → 2026-08-07 01:26Z, account ``, +`us-east-1`. Evidence directory: `/tmp/abca-645-p2-20260806`. + +**Teardown: complete.** All MicroVMs terminated, image + version deleted, both +live IAM workarounds reverted, Cognito user deleted, secret died with the stack, +96/100 stack resources deleted (4 zero-cost residual, #702), **all billable +resources confirmed gone**, and every global/host configuration restored. See +Phase 8. + +## What Stage D was for + +P2 (`ab4808c`) wired everything short of the live run. Its commit message names +what it could not assert: + +> Remaining for P2 completion: the live smoke run (clone -> change -> PR with +> `bgagent watch`) and the deferred empirical items (suspend TTL >1h, +> SUSPENDED-vs-quota, `microvmImageHooks` API spelling, `NO_INGRESS` ARN). + +Five jobs, and their verdicts: + +| # | Job | Verdict | +|---|---|---| +| 1 | **THE SMOKE** — clone → change → PR through `bgagent watch` | **BLOCKED at `implement`, turn 0** — reproducible. Clone, branch, `/run`, `platform_config`, progress events and terminate all work; `claude --version` times out (P2-F5). **No PR was created.** | +| 2 | Adjudicate the CFN `AWS::Lambda::MicrovmImage` shape P1 left half-open | **DISCHARGED — the L1 is REFUTED on 5 values** (P2-F2) | +| 3 | `microvmImageHooks` spelling | **DISCHARGED both ways** — property name/nesting correct, hook *values* must be `ENABLED`/`DISABLED` (P2-F2); the API request built by the packaging script is correct and all four hooks were accepted and served | +| 4 | `NO_INGRESS` ARN name | **DISCHARGED** — the injected ARN is right and does suppress P1 F7's default public ingress | +| 5 | Extend suspend-TTL bound; re-probe SUSPENDED-vs-quota | **TTL extension SKIPPED** (time-boxed, see Skipped); quota **re-confirmed NOT OBSERVABLE** | + +**Headline:** the P2 substrate is much closer than P1 — the image builds with all +four hooks, the MicroVM launches, `/run` installs `platform_config`, the repo +clones, progress events stream to `bgagent watch`, and finalization terminates +the VM. But **four independent defects had to be worked around to get that far**, +and the run still ends one step short of a PR on a fifth. None of the four is +visible to `cdk synth`, `mise //cdk:test`, or any unit test: every one is a +live-service contract mismatch. + +--- + +## Variables + +```bash +set -o pipefail # P1's hard-won lesson: `| tee` masks failures +export AWS_PROFILE=aamorosi+workshops-AdminConsoleAccess +export AWS_REGION=us-east-1 +export AWS_DEFAULT_REGION="$AWS_REGION" +export CDK_DEFAULT_REGION="$AWS_REGION" +export CDK_DEFAULT_ACCOUNT="" +export STACK_NAME=backgroundagent-dev +export EXPECTED_BRANCH=feat/645-lambda-microvm-p2 +export SCRATCH_REPO=dreamorosi/batch-sync-triage +export SMOKE_USER="" +export CDK_DOCKER=finch # no docker on this box (P1 deviation 3) +export EVIDENCE_DIR=/tmp/abca-645-p2-20260806 +export BGAGENT_CONFIG_DIR=/tmp/abca-645-p2-bgagent +``` + +### ⚠️ zsh trap that bit this run + +The executor's shell is **zsh**, where `$VAR:latest` is parsed as the +`${VAR:l}` *lowercase modifier*, silently producing `…atest`. It cost one +mis-diagnosed container push. **Always brace: `${VAR}:latest`.** Likewise zsh has +no `PIPESTATUS` (it is `$pipestatus`, 1-indexed) and no `timeout(1)` — P1's +`test "${PIPESTATUS[0]}" -eq 0` silently evaluates to empty here. Use +`set -o pipefail` plus a plain `$?`. + +--- + +## Execution deviations (each is itself a result) + +1. **AZ pinning was required — P1 F11 is UNFIXED on this branch.** + `agent-vpc.ts` still does `maxAzs: 2` with no AZ constraint, and this account + still maps `us-east-1a → use1-az6`, which AgentCore rejects. Rather than + re-derive a finding P1 already recorded, the gitignored CDK context cache + (`cdk/cdk.context.json`, a build artifact — `git status` stayed clean) was + trimmed to lead with `us-east-1b` (`use1-az1`) + `us-east-1c` (`use1-az2`). + Original saved to `$EVIDENCE_DIR/cdk.context.json.ORIGINAL` and **restored at + teardown**. With the pin, AgentCore Memory and Runtime created cleanly, so + this is the whole of F11's remaining impact. +2. **The AgentCore container asset could not be pushed; an existing ECR manifest + was retagged instead.** `finch` (v1.17.2, no docker on this box) *built* the + image fine but every `finch push`/`finch pull` against ECR failed instantly + with `no basic auth credentials`. Root cause established: `finch push` shells + into the Lima VM as `limactl shell finch sudo -E nerdctl push`, and the VM's + `DOCKER_CONFIG` (`/home/aamorosi.guest/.finch-vm-config/config.json`) carries + `credsStore: finchhost`, a helper that cannot resolve this account's + Isengard `credential_process`. A host-side `finch login` succeeded and wrote + to `~/.finch/config.json`, but the VM never consulted it. Since the ECR image + is **only** consumed by the AgentCore runtime — which this run never invokes, + because the smoke runs on the MicroVM substrate built from the S3 zip by the + Lambda MicroVMs service — the required asset tag was added to an existing + manifest with `aws ecr put-image`. **This does not touch the MicroVM image + under test.** Consequence to be honest about: the deployed AgentCore runtime + carries a P1-era agent image. Nothing in this runbook depends on it. + *(All global config touched during that investigation — `~/.finch/config.json` + and the VM's docker config — was reverted; see Teardown.)* +3. **The connector operator role's trust policy had to be patched to deploy at + all** (P2-F1) — via a `/tmp` cloud-assembly patch, P1's technique. No + repository source file was modified. +4. **Two IAM workarounds were applied live to get past P2-F3** (execution-role + trust condition, plus a temporary unconditioned `iam:PassRole` that turned out + to be unnecessary). Both reverted; see Teardown. +5. **The CDK-managed `CfnMicrovmImage` path was attempted first, as briefed, and + failed** (P2-F2). The out-of-band `--create-image` script path was used + instead — the same path P1 used, and currently the only one that works. +6. **The suspend-TTL extension beyond P1's 1 h floor was skipped** (time-boxed). +7. **One self-inflicted error is recorded rather than hidden:** the GitHub PAT was + first written to Secrets Manager with a trailing newline (`gh auth token | + … --secret-string file:///dev/stdin`), which produced a real clone failure. + Corrected; see 3.2. + +--- + +## Phase 0 — Preflight + +### 0.1 Identity, region, branch + +`aws sts get-caller-identity` → +`arn:aws:sts:::assumed-role/AdminConsoleAccess/aamorosi-Isengard`, +account ``. Branch `feat/645-lambda-microvm-p2`, SHA +`3a4b61a97b22b7bcdd9832101f8d61a077fbf103`. Untracked: `docs/verification/`, +`opencode.json` — the same two P1 saw. + +> **Credential note.** The profile's default region is `eu-west-1`, so +> `AWS_REGION`/`AWS_DEFAULT_REGION` must be exported explicitly for every +> command. A bare `aws` call in this account goes to the wrong Region. +> Mid-run the credentials expired once; re-pinning `AWS_PROFILE` (which resolves +> through `credential_process` and auto-refreshes) fixed it. **No global AWS +> config was read or modified at any point.** + +`backgroundagent-dev` was **absent**, exactly as P1's Phase 8 hoped: + +``` +An error occurred (ValidationError) when calling the DescribeStacks operation: Stack with id backgroundagent-dev does not exist +``` + +**P1 F12's stack half is therefore RESOLVED**: P1 ended with `backgroundagent-dev` +in `DELETE_FAILED` (VPC + 2 subnets + 1 SG pinned by two leaked `agentic_ai` +ENIs) and a recorded retry command. AgentCore did eventually release them and the +stack is gone. The three unrelated pre-existing stacks +(`serverless-api-powertools`, `BuildingServerlessAPIs`, +`aws-sam-cli-managed-default`) plus `CDKToolkit` remain. + +### 0.2 Tooling + +| Tool | Version | Note | +|---|---|---| +| `aws-cli` | 2.36.13 | `aws lambda-microvms help` **exit 0**, **24 commands** — identical to P1 | +| `mise` | 2026.8.1 | **present this time** (P1 ran entirely on raw fallbacks) | +| node | v24.16.0 | | +| python3 | 3.9.6 | system python; the *guest* runs 3.13.13 | +| `zip` | 3.0 | | +| `rsync` | openrsync (protocol 29) | packaging script works unmodified | +| docker | **absent** | | +| `finch` | v1.17.2 | builds fine, **cannot push to ECR** (deviation 2) | + +### 0.3 Bootstrap — no re-bootstrap needed + +`CDKToolkit` `UPDATE_COMPLETE` (last updated 2026-07-31T18:54:48Z, i.e. P1's run), +and it already carries what P2 needs: + +- `ComputeTypes` = **`agentcore,lambda-microvm`** ✓ +- `BootstrapVariant` = **`ABCA: Least-Privilege Bootstrap`** +- bootstrap version SSM parameter = `32` +- `cdk-hnb659fds-cfn-exec-role--us-east-1` carries exactly the **five** + ABCA policies (Application, Infrastructure, Observability, Compute-Agentcore, + **Compute-LambdaMicrovms**) and **no `AdministratorAccess`**. + +So `bootstrap --force` was **not** re-run. + +> **P1 F10 is partly self-healing — correction to the P1 record.** P1 reported as +> a "durable consequence" that `BootstrapVariant` *stays* `AWS CDK: Default +> Resources` after a forced bootstrap, so every future non-forced bootstrap +> refuses forever. It now reads `ABCA: Least-Privilege Bootstrap`. The reason is +> the very next command in P1's own runbook: `update-stack +> --use-previous-template` with **only** `ParameterKey=ComputeTypes` supplied +> resets every unspecified parameter to its **template default** (that is +> CloudFormation's documented behaviour absent `UsePreviousValue=true`), and the +> ABCA template's default for `BootstrapVariant` is the ABCA string. F10's +> *first* half (the silent `exit 0` no-op without `--force`) stands; the +> "recurs forever" half does not. + +### 0.4 Managed base image + +Unchanged from P1: exactly **one** managed base image in `us-east-1`, +`arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1`, versions `1` +(2026-07-21) and `0` (2026-06-17), **newest first**, so `items[0]` = `1`. +Selected version `1`; the service echoes `baseImageVersion: "1.0"`. + +--- + +## Phase 1 — Deploy + +### 1.1 Substrate synth — PASS + +`abca:microvm-image-not-provisioned` emitted with the full remedy text; +`abca:microvm-image-p1-smoke-unverified` correctly **absent**. Both incidental +warnings grew since P1 and one is close to the wall: + +| Metric | P1 | P2 | Limit | +|---|---|---|---| +| Template size | 893,273 | **977,650** (substrate) / **983,796** (with image) | 1,000,000 | +| Resource count | 463 | **485** / **486** | 500 | + +**Feeds back to design:** the template is at **98.4 %** of the 1 MB +CloudFormation limit with the image configured. That is ~16 KB of headroom — +roughly one more construct. `suppressTemplateIndentation` or a stack split is +now a near-term requirement, not a nicety. + +### 1.2 Substrate deploy — BLOCKED, then PASS after a trust-policy patch + +**Attempts 1–2 failed on the ECR push** (deviation 2), a purely local tooling +problem. **Attempts 3–4 failed identically on the network connectors** — this is +P2-F1: + +``` +Resource handler returned message: "The service is unable to assume the provided NetworkConnectorOperatorRole. Please verify the trust policy on the role. (Service: Lambda, Status Code: 400, Request ID: dbe1d2f4-dd0c-4319-8f5d-b4be4f076843) (SDK Attempt Count: 1)" (RequestToken: d6e21be9-9857-f452-bf12-3b93f89a4c75, HandlerErrorCode: InvalidRequest) +``` + +Both `LambdaMicrovmCompute/EgressConnector` and +`LambdaMicrovmCompute/BuildEgressConnector` `CREATE_FAILED`. Attempt 4 ran +against a **freshly deleted stack**, so this is **deterministic, not IAM +propagation lag** — an important distinction, because that error message is the +classic propagation symptom and a re-run is the obvious (wrong) first guess. + +**Attempt 5** deployed a `/tmp` cloud assembly with exactly one edit — the +`aws:SourceAccount` condition removed from +`LambdaMicrovmComputeConnectorOperatorRole`'s `AssumeRolePolicyDocument`: + +```json +{"Statement":[{"Action":"sts:AssumeRole","Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"}}],"Version":"2012-10-17"} +``` + +Both connectors went `CREATE_IN_PROGRESS → Resource creation Initiated` within a +second. **`CREATE_COMPLETE` 23:13:43Z → 23:27:03Z = 13 min 20 s**, 485 resources +(P1's comparable figure: 13 min 42 s / 464 resources). + +*Two P1 gotchas recurred verbatim and their remedies still work:* + +- `AWS::BedrockAgentCore::Memory` cannot be deleted while `CREATING` + (`Validation failed during DeleteMemory: Memory is in transitional state + CREATING. Cannot delete memory.`) → rollback ends `ROLLBACK_FAILED`; a plain + `aws cloudformation delete-stack` clears it (~3 min). **P1 F12's Memory half is + unfixed.** +- Post-synth template edits are silently ignored unless the template's S3 asset + object is deleted first (the object key is the *pre-edit* content hash). + +### 1.3 Substrate outputs — PASS + +`ComputeSubstrate = lambda-microvm`; all **seven** MicroVM outputs populated +(the six P1 had, plus `MicrovmBuildEgressConnectorArns`); artifact key exactly +`microvm-images/agent-artifact.zip`. Runtime and build egress connectors are +distinct ARNs, as designed. + +### 1.4 No-image orchestrator env — PASS, and a correction to P1 F14 + +`MICROVM_*` keys = **`[]`** (23 env keys total). All-or-nothing holds. + +> **P1 F14's `--no-paginate` remedy is WRONG and silently truncates.** P1 +> concluded that `list-stack-resources --query "…|[0]"` needs `--no-paginate`. +> With 485 resources, `--no-paginate` returns **only the first page** — it +> yielded 6 Lambda functions and *no* `TaskOrchestrator`, resolving +> `ORCHESTRATOR_FN` to the literal string `None` and producing a confident +> `Function not found: …:function:None`. That is worse than P1's original bug, +> because the original at least printed five values and broke loudly. **The +> correct approach is to let the CLI paginate (its default) and filter +> client-side**, which returns all 485: +> ```bash +> aws cloudformation list-stack-resources --stack-name "$STACK_NAME" --output json \ +> | python3 -c 'import json,sys; print([r["PhysicalResourceId"] for r in json.load(sys.stdin)["StackResourceSummaries"] if "TaskOrchestrator" in r["LogicalResourceId"] and r["ResourceType"]=="AWS::Lambda::Function"])' +> ``` + +**The new `agentPlatformConfig` env vars are present — 11 of 13.** All seven +`agentPlatformConfig` fields plus the four inherited ones landed on the +orchestrator: + +| `platform_config` key | Orchestrator env var | Present | +|---|---|---| +| `task_table_name` | `TASK_TABLE_NAME` | ✓ | +| `task_events_table_name` | `TASK_EVENTS_TABLE_NAME` | ✓ | +| `github_token_secret_arn` | `GITHUB_TOKEN_SECRET_ARN` | ✓ | +| `agent_session_role_arn` | `AGENT_SESSION_ROLE_ARN` | ✓ | +| `task_approvals_table_name` | `TASK_APPROVALS_TABLE_NAME` | ✓ | +| `nudges_table_name` | `NUDGES_TABLE_NAME` | ✓ | +| `log_group_name` | `LOG_GROUP_NAME` | ✓ | +| `artifacts_bucket_name` | `ARTIFACTS_BUCKET_NAME` | ✓ | +| `trace_artifacts_bucket_name` | `TRACE_ARTIFACTS_BUCKET_NAME` | ✓ | +| `aws_sdk_ua_app_id` | `AWS_SDK_UA_APP_ID` | ✓ (`uksb-wt64nei4u6#backgroundagent-dev`) | +| `anthropic_default_haiku_model` | `ANTHROPIC_DEFAULT_HAIKU_MODEL` | ✓ (`us.anthropic.claude-haiku-4-5-20251001-v1:0`) | +| `linear_oauth_secret_arn` | `LINEAR_OAUTH_SECRET_ARN` | — (per-workspace, CLI-created; correctly absent) | +| `jira_oauth_secret_arn` | `JIRA_OAUTH_SECRET_ARN` | — (same) | + +The two absentees are **not** in the contract's `required` list, so the +producer's optional-key handling is exercised and correct. + +*Incidental:* `ARTIFACTS_BUCKET_NAME` and `TRACE_ARTIFACTS_BUCKET_NAME` resolve +to the **same** bucket (`…-traceartifactsbucket8cbd5207-pwjbv0diorys`). Not a +MicroVM issue, but worth a glance — two distinct `platform_config` keys carrying +one bucket makes the trace/artifact separation notional. + +### 1.5 The CDK-managed `CfnMicrovmImage` path — **REFUTED** (P2-F2) + +This is the question P1 left open, and it is now closed. Synth produced the +warning correctly (`abca:microvm-image-p1-smoke-unverified`) and the exact L1 +under test: + +```json +"CpuConfigurations": [{ "Architecture": "arm64" }], +"Hooks": { + "Port": 8080, + "MicrovmHooks": { "Run": "/aws/lambda-microvms/runtime/v1/run", "RunTimeoutInSeconds": 60, + "Terminate": "/aws/lambda-microvms/runtime/v1/terminate", "TerminateTimeoutInSeconds": 15 }, + "MicrovmImageHooks": { "Ready": "/aws/lambda-microvms/runtime/v1/ready", "ReadyTimeoutInSeconds": 60, + "Validate": "/aws/lambda-microvms/runtime/v1/validate", "ValidateTimeoutInSeconds": 60 } +} +``` + +CloudFormation **rejected it at change-set early validation** — the stack was +never touched, so there was no rollback: + +``` +Early validation failed for change set cdk-deploy-change-set: +backgroundagent-dev/LambdaMicrovmCompute/Image (AWS::Lambda::MicrovmImage LambdaMicrovmComputeImage16B48539) + /aws/lambda-microvms/runtime/v1/run is not a valid enum value. Supported values: [DISABLED, ENABLED] (at + /Resources/LambdaMicrovmComputeImage16B48539/Properties/Hooks/MicrovmHooks/Run) + /aws/lambda-microvms/runtime/v1/terminate is not a valid enum value. Supported values: [DISABLED, ENABLED] (at + /Resources/LambdaMicrovmComputeImage16B48539/Properties/Hooks/MicrovmHooks/Terminate) + arm64 is not a valid enum value. Supported values: [ARM_64] (at + /Resources/LambdaMicrovmComputeImage16B48539/Properties/CpuConfigurations/0/Architecture) + /aws/lambda-microvms/runtime/v1/ready is not a valid enum value. Supported values: [DISABLED, ENABLED] (at + /Resources/LambdaMicrovmComputeImage16B48539/Properties/Hooks/MicrovmImageHooks/Ready) + /aws/lambda-microvms/runtime/v1/validate is not a valid enum value. Supported values: [DISABLED, ENABLED] (at + /Resources/LambdaMicrovmComputeImage16B48539/Properties/Hooks/MicrovmImageHooks/Validate) +``` + +**Five rejected values, and the CFN surface is identical to the API surface.** +This refutes the construct's explicit reasoning — *"The CDK L1 remains +intentionally unchanged because its generated CloudFormation types accept string +values and document no architecture/hook allowed-value constraint"*. The types +accept strings; the **service** enforces the enum, at change-set time. + +Two useful corollaries: + +- **The `microvmImageHooks` *spelling* is CORRECT.** The errors are scoped to + `…/Hooks/MicrovmImageHooks/Ready` and `…/Validate`, so CFN resolved the + property name and its children. Only the *values* are wrong. Combined with 2.1 + below (the API accepted the identical structure), the naming question is + discharged in both directions. +- **Hook paths are not configurable anywhere.** Neither CFN nor the API takes a + path; both take `ENABLED`/`DISABLED`. The service calls fixed well-known routes + — 2.2 proves they are exactly the `/aws/lambda-microvms/runtime/v1/*` strings + the agent serves. So `RUN_HOOK_PATH` and friends are correct *as route + constants for the agent* and simply must not be sent as property values. + +### 1.6 Wired deploy (out-of-band image) — PASS + +After the image existed (Phase 2), redeploying with +`--context microvm_image_identifier=` completed in **3 min 33 s** +(00:06:00Z → 00:09:33Z), `UPDATE_COMPLETE`, warning +`abca:microvm-image-p1-smoke-unverified` emitted. **All six `MICROVM_*` vars +present — all-or-nothing WITH the image confirmed:** + +``` +MICROVM_EGRESS_CONNECTOR_ARNS = arn:aws:lambda:us-east-1::network-connector:nc-d306f00f-1bd0-45ea-9457-0fcec0dab2a4 +MICROVM_EXECUTION_ROLE_ARN = arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-tF8Idpc9aT0R +MICROVM_IMAGE_IDENTIFIER = arn:aws:lambda:us-east-1::microvm-image:backgroundagent-dev-abca-agent +MICROVM_IMAGE_VERSION = 1.0 +MICROVM_INGRESS_CONNECTOR_ARNS = arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:NO_INGRESS +MICROVM_PAYLOAD_BUCKET = backgroundagent-dev-lambdamicrovmcomputepayloadbuc-bctl5ej8aazr +``` + +**P1 F3 is FIXED:** `MICROVM_IMAGE_IDENTIFIER` is a full ARN, not a bare name. + +--- + +## Phase 2 — Image + +### 2.1 `create-microvm-image` — PASS with all four hooks + +`package-microvm-artifact.sh` (no `--create-image`) staged and uploaded first: +**904K artifact / 922,056 bytes in S3, SSE `AES256`** (P1: 584K / 597,305 — the +P2 `server.py` growth). Then with `--create-image`, exit **0**, and the service +echoed the request back: + +```json +"cpuConfigurations": [{ "architecture": "ARM_64" }], +"resources": [{ "minimumMemoryInMiB": 8192 }], +"hooks": { + "port": 8080, + "microvmHooks": { "run": "ENABLED", "runTimeoutInSeconds": 60, + "terminate": "ENABLED", "terminateTimeoutInSeconds": 15 }, + "microvmImageHooks": { "ready": "ENABLED", "readyTimeoutInSeconds": 60, + "validate": "ENABLED", "validateTimeoutInSeconds": 60 } +}, +"state": "CREATING", "imageVersion": "1.0", +"imageArn": "arn:aws:lambda:us-east-1::microvm-image:backgroundagent-dev-abca-agent" +``` + +**`microvmImageHooks` with `ready` + `validate`, and `microvmHooks` with `run` + +`terminate`, are all ACCEPTED.** P1 could not test this: it never got a +four-hook image created (F1), and its `/ready`-only attempt failed the build. + +The script's P2 reminder banner printed both before and after the call, and +`8192 MiB` was accepted — P1 F5's ceiling holds. + +### 2.2 Build — PASS in 4 min 35 s, `/ready` **and** `/validate` served + +Two builds per version again (`chipsetGeneration` 3 and 4), both `SUCCESSFUL`; +`state=SUCCESSFUL`, `status=ACTIVE`. **00:00:01Z → 00:04:36Z = 4 min 35 s** +(P1: 5 min 51 s). + +The decisive evidence, from `/aws/lambda-microvms/backgroundagent-dev-abca-agent` +— **this is the item P1 F1 blocked entirely**: + +``` +[server/build-hook] /ready hook: server is up, reporting ready for snapshot +INFO: 127.0.0.1:52856 - "POST /aws/lambda-microvms/runtime/v1/ready HTTP/1.1" 200 OK +[server/build-hook] /validate hook: ok (python=3.13.13, platform_config_keys=13, warnings=0) +INFO: 127.0.0.1:50860 - "POST /aws/lambda-microvms/runtime/v1/validate HTTP/1.1" 200 OK +``` + +(both lines twice, once per chipset). Note what this proves beyond "the hooks +work": the service POSTs to **exactly** the `/aws/lambda-microvms/runtime/v1/*` +paths, confirming the fixed-route model inferred in 1.5; `/validate` reports +`platform_config_keys=13`, so the cross-package contract loaded inside the +snapshot; and `warnings=0`, so the baked-secret scan found nothing. + +**P1 F4 is FIXED:** `apt-get` reached `deb.debian.org` over port 80 through the +dedicated build connector — the build log shows `Get:… http://deb.debian.org/…` +succeeding. No temporary security-group rule was needed this time. + +`agent/Dockerfile` also now installs Go tooling; the build log shows +`go: downloading …` completing, so build-time egress is sufficient. + +### 2.3 Sizes — P1 F13 reconfirmed and slightly worse + +| Field | Bytes | Human | vs P1 | +|---|---|---|---| +| `codeInstallSizeInBytes` | 2,342,203,392 | **2.18 GiB** | 2.17 GiB | +| `memorySnapshotSizeInBytes` | 1,223,421,952 | 1.14 GiB | 1.13 GiB | +| `diskSnapshotSizeInBytes` | 34,959,360 | 33.3 MiB | 35.4 MiB | + +`codeInstallSizeInBytes` still **exceeds AgentCore's 2 GB container-image +limit**, while the same tree as an OCI image is 1.803 GB / 631.1 MB compressed +(measured locally). P1 F13's "say which measure you mean" recommendation stands. +`snapshotBuild` is still only on `get-microvm-image-build`, not on the version. + +--- + +## Phase 3 — Platform user, secret, repo onboarding + +### 3.1 Cognito user — PASS, no first-login dance needed + +`bgagent admin invite-user --stack-name … --region …` +created the user with a **permanent** password (`UserStatus: CONFIRMED`, not +`FORCE_CHANGE_PASSWORD`), wrote credentials to +`$BGAGENT_CONFIG_DIR/invites/…txt` mode `0600`, and printed a `configure` +bundle. `bgagent configure --stack-name` + `bgagent login --username …` → +`Login successful.` **This is the answer to "find the exact bgagent commands": +`admin invite-user` is the whole first-login flow** — there is no +`RespondToAuthChallenge` step to drive, which is why P1's 7.1 blocker +("user pool has zero users") is a two-command fix, not an obstacle. + +### 3.2 GitHub PAT into the stack secret — PASS on the second attempt + +The token was piped from `gh auth token` straight into +`aws secretsmanager put-secret-value` and **never** written to disk, a log, or +this file. + +**Operator error worth recording, because it produced a convincing false +defect.** `gh auth token` emits a trailing newline and +`--secret-string file:///dev/stdin` stores it verbatim, so the secret was **41 +bytes**. My own verification used `.strip()` and reported "length 40", hiding it. +The task then failed inside the guest with: + +``` +RuntimeError: clone failed (non-transient): Post "https://api.github.com/graphql": net/http: invalid … +``` + +— i.e. an invalid HTTP header value, because the token carried `\n`. Rewriting +the secret stripped (40 bytes, no trailing whitespace) fixed the clone +immediately. + +*Verification lesson:* assert on the **raw** secret, never a stripped copy: + +```bash +aws secretsmanager get-secret-value --secret-id "$SECRET_ARN" --output json \ + | python3 -c 'import json,sys; s=json.load(sys.stdin)["SecretString"]; print(len(s), s!=s.strip())' +``` + +*Minor robustness observation (not a defect found by this run's design):* the +agent passes the secret value through to `gh`/`git` unstripped, so any +whitespace an operator introduces surfaces as a confusing `net/http` error rather +than "your token looks malformed". A `.strip()` at the token resolver would turn +a 20-minute misdiagnosis into a non-event. + +### 3.3 Repo onboarding — PASS, gate and probe both behaved + +`bgagent repo onboard dreamorosi/batch-sync-triage --compute-type lambda-microvm`: + +```json +{ "repo": "dreamorosi/batch-sync-triage", "status": "active", + "compute_type": "lambda-microvm", "onboarded_at": "2026-08-07T00:10:43.376Z" } +``` + +Both guards fired as designed and in the documented order: the **ComputeSubstrate +gate** passed because the stack output reads `lambda-microvm`, and the live +**`ListManagedMicrovmImages` availability probe** passed. The command also +printed the two ADR-021 advisory notes, including the smoke-unverified warning — +correct, and still accurate at the end of this run. + +`bgagent platform doctor` → `passed: true`, **all 7 checks**, including +`Managed MicroVM images are available in us-east-1` and — because 3.2 had already +run — `GitHubTokenSecretArn contains a token value`. (P1 noted this check cannot +distinguish a real PAT from the 32-char generated placeholder; that is still +true, it just happens to be a true positive here.) + +--- + +## Phase 4 — THE SMOKE + +Five submissions. Each failure moved the boundary forward, so all five are +recorded. + +| # | Task ID | Outcome | Finding | +|---|---|---|---| +| 1 | `01KZCRY70HRBP236GECR768JJX` | `FAILED` — `RunMicrovm … AccessDeniedException … iam:PassRole` | P2-F3 | +| 2 | `01KZCS6451HRPSXAG33Z4R5XRV` | identical, **with an unconditioned `iam:PassRole` attached** → so PassRole was never the real problem | P2-F3 | +| 3 | `01KZCSCD6PKDZNC8WRTFNRQG3H` | identical, 3 min after the IAM change → **not propagation lag** | P2-F3 | +| 4 | `01KZCSNM8MD4MB17ZTW1VDB6PY` | **`RUNNING`** after removing the execution-role trust condition, then `clone failed … net/http: invalid` | P2-F3 root cause proven; 3.2 token bug | +| 5 | `01KZCSVRHZXVHXQ4T29XYZKBAM` | **`RUNNING` → clone OK → branch OK → `implement` failed at turn 0** | **P2-F5** | +| 5r | `01KZCT8SWZ1DDZC7P0RYS982ES` | identical to #5 — **reproducible** | P2-F5 | + +### 4.1 P2-F3 — the `iam:PassRole` deny that was really a trust-policy deny + +``` +Session start failed: Error: MicroVM RunMicrovm failed: AccessDeniedException: User: arn:aws:sts:::assumed-role/backgroundagent-dev-TaskOrchestratorOrchestratorFnS-kyEY8iz3mrcm/backgroundagent-dev-TaskOrchestratorOrchestratorFn-huSs3tbuFbJs is not authorized to perform: iam:PassRole on resource: arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-tF8Idpc9aT0R because no identity-based policy allows the iam:PassRole action +``` + +The message is actively misleading, and the diagnosis is the useful part of this +section. The orchestrator's inline policy **does** carry the grant, exactly as +P1 4.4 recorded it: + +```json +{ "Sid": "MicrovmPassExecutionRole", "Effect": "Allow", "Action": "iam:PassRole", + "Resource": "arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-tF8Idpc9aT0R", + "Condition": { "StringEquals": { "iam:PassedToService": "lambda.amazonaws.com" } } } +``` + +Elimination sequence: + +1. Attached a **temporary unconditioned** `iam:PassRole` on the same resource → + **still denied** (submission 2). So the `iam:PassedToService` condition was + *not* the cause. +2. `aws iam get-role` → **no permissions boundary**, path `/`. +3. `aws iam simulate-principal-policy … --action-names iam:PassRole` → + **`allowed`**, matching my temporary statement. So IAM itself says yes. +4. Waited 3 minutes and resubmitted → **still denied** (submission 3). Not + propagation. +5. Read the **target** role's trust policy — and found the same + `aws:SourceAccount` condition that had just broken the network connectors: + +```json +{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole", + "Condition":{"StringEquals":{"aws:SourceAccount":""}}} +{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:TagSession", + "Condition":{"StringEquals":{"aws:SourceAccount":""}}} +``` + +6. Removed both conditions → **submission 4 reached `RUNNING` in 6 seconds.** + +**So `RunMicrovm` reports a role it cannot pass-and-assume as an `iam:PassRole` +identity-policy denial on the *caller*.** P2-F1 and P2-F3 are therefore **one +root cause with two symptoms**: the Lambda MicroVMs service does not present +`aws:SourceAccount` when assuming ABCA's MicroVM-facing roles, so every trust +policy carrying that confused-deputy condition is unassumable. + +### 4.2 THE SMOKE (submission 5) — `bgagent watch`, verbatim + +``` +Watching task 01KZCSVRHZXVHXQ4T29XYZKBAM... (Ctrl+C to stop) +[5:28:03 PM] ★ repo_setup_complete: branch=bgagent/01KZCSVRHZXVHXQ4T29XYZKBAM/add-a-codeowners-file-at-the-repository-root-conta build_before=False +[5:28:04 PM] ★ step:implement:start +[5:28:15 PM] ★ step:implement:failed +[5:28:15 PM] ★ agent_execution_complete: status=error turns=0 +Task 01KZCSVRHZXVHXQ4T29XYZKBAM failed. timeout: Build/tests didn't finish in time (timed out) — Workflow run_agent step failed: TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +``` + +**Time-to-RUNNING: ~6 s.** Submitted 00:27:09Z, `started_at` +`2026-08-07T00:27:11`, VM `startedAt` 00:27:12Z. That is materially faster than +AgentCore or ECS cold start and is the backend's main selling point — worth +recording as the one positive performance result of the run. + +**`/run` accepted the envelope, and `platform_config` was installed** — the +required observation, from the guest log group: + +``` +[server/run-pre-config] /run hook received: microvm_id='microvm-8ad29e93-99c2-3ccc-b079-12f8bdab2936' bytes=2120 +[server/debug] /run hook installed platform_config env: ['AGENT_SESSION_ROLE_ARN', 'ANTHROPIC_DEFAULT_HAIKU_MODEL', 'ARTIFACTS_BUCKET_NAME', 'AWS_SDK_UA_APP_ID', 'GITHUB_TOKEN_SECRET_ARN', 'LOG_GROUP_NAME', 'NUDGES_TABLE_NAME', 'TASK_APPROVALS_TABLE_NAME', 'TASK_EVENTS_TABLE_NAME', 'TASK_TABLE_NAME', 'TRACE_ARTIFACTS_BUCKET_NAME'] +[server/debug] /run hook accepted task_id='01KZCSVRHZXVHXQ4T29XYZKBAM' microvm_id='microvm-8ad29e93-99c2-3ccc-b079-12f8bdab2936' +``` + +Everything in the P2 delivery design is confirmed here: **2,120 bytes**, so the +envelope went **inline** and stayed under the real 4,096-byte cap (P1 F6); the +installed set is **exactly the 11 available keys**, names only, no values; the +pre-install line is stdout-only (`run-pre-config`) and the post-install line is +the first to reach CloudWatch, exactly as the snapshot-credential-hygiene work +intended. + +**Progress events streamed** — `repo_setup_complete`, `step:implement:start`, +`step:implement:failed`, `agent_execution_complete` all arrived live in `watch`. + +**Clone → change → PR got exactly one step:** clone ✓, branch ✓, +`build_before=False` ✓ … then `implement` died at turn 0. **No commit, no push, +no PR.** + +**Heartbeats: NOT observed.** `agent_heartbeat_at` was `None` at every poll +across all six submissions. The task was `RUNNING` for only ~12 s and the agent +bumps the heartbeat every 45 s, so it never had a chance to fire. **The +dual-signal liveness path is therefore NOT discharged** — see Skipped. + +### 4.3 P2-F5 — `claude --version` times out in the guest + +``` +[00:28:04] AGENT claude-agent-sdk version: 0.2.110 +[00:28:15] ERROR step 'implement' handler raised: TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +[00:28:15] WORKFLOW step 'implement' failed (on_failure=fail) — workflow FAILED +``` + +`METRICS_REPORT`: `"turns": 0, "duration_s": 12.7, "code_changed": null, +"pr_url": null, "memory_written": true`. + +The probe is `agent/src/runner.py:476`: + +```python +["claude", "--version"], capture_output=True, text=True, timeout=10 +``` + +Characterisation, to separate "broken binary" from "slow substrate": + +- **In the identical image, locally under finch: `claude --version` → `2.1.191 + (Claude Code)` in under 1 second.** So the binary, its symlink and `PATH` are + all fine. +- `/usr/bin/claude` → `../lib/node_modules/@anthropic-ai/claude-code/bin/claude.exe`, + a **236,305,136-byte (225 MiB) statically-linked ELF**. +- The MicroVM had been restored from a snapshot ~50 s earlier and had already + done a full `git clone` over the network. + +The consistent reading is **lazy snapshot hydration**: the first `exec` of a +225 MiB binary that was never touched before the snapshot was taken must fault +its pages in from lazily-restored storage, and that exceeds 10 s. Two +independent fixes suggest themselves, and the second is the more interesting +because P2 already built the mechanism and then deliberately declined to use it: + +1. Raise / make backend-aware the `timeout=10` (it is a *liveness probe for a + version string* — a tight bound buys nothing). +2. **Warm `claude` in the `/ready` build hook so it lands in the memory + snapshot.** P2's `/ready` deliberately does the minimum — `"server is up, + reporting ready for snapshot"` — and its own docstring explains that `/ready` + exists so "the snapshot is taken with a warm server". The snapshot is warm for + *uvicorn* and stone cold for the 225 MiB binary that does all the work. + +**`build_passed: true, lint_passed: false`** in the same report is incidental +noise from the scratch repo (`mise ERROR no tasks defined …`, `unknown command: +lint`) and unrelated to the substrate. + +### 4.4 P2-F4 — the execution role cannot write to the application log group + +Found while reading logs for F5, and independent of it. Every structured agent +log line to the log group that `platform_config` itself delivers is denied: + +``` +[server/debug/self] CloudWatch write failed: AccessDeniedException: … User: arn:aws:sts:::assumed-role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-tF8Idpc9aT0R/Lambda-microvmsExecutor-86cfecce-… is not authorized to perform: logs:CreateLogStream on resource: arn:aws:logs:us-east-1::log-group:/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/backgroundagent-dev:log-stream:server_debug/01KZCSVRHZXVHXQ4T29XYZKBAM because no identity-based policy allows the logs:CreateLogStream action +``` + +Same denial for the `metrics/` stream, so `METRICS_REPORT` never lands +either. P2 wired `log_group_name` into `platform_config` (making the agent +*attempt* the write) but the execution role's logs grant is scoped to +`/aws/lambda-microvms/*` only — the construct's own cdk-nag suppression says so: +*"the `MICROVM_LOG_GROUP_PREFIX/*` namespace"*. + +Not fatal — the agent degrades to stdout, which the MicroVM log group captures, +which is why this run could be debugged at all. But it is a genuine smoke-parity +hole: on `lambda-microvm`, the platform's canonical per-task observability +streams are empty, and anything reading them (rather than the guest's stdout) +sees nothing. Exactly the class of grant P2 added for Bedrock, Secrets Manager +and Memory — this one was missed. + +--- + +## Phase 5 — Post-smoke verification + +### 5.1 Finalization called `TerminateMicrovm` — PASS + +`get-microvm` on the smoke VM: + +``` +microvmId = microvm-8ad29e93-99c2-3ccc-b079-12f8bdab2936 +state = TERMINATED +stateReason= Success. +startedAt = 2026-08-06T17:27:12.144000-07:00 +``` + +and the in-guest breadcrumb fired, 200 OK: + +``` +[server/debug] /terminate hook: {"active_pipeline_threads": 0, "background_pipeline_failed": false, "event": "microvm_terminate", "microvm_id": "", "timestamp": "2026-08-07T00:28:42.926428+00:00"} +INFO: 127.0.0.1:37364 - "POST /aws/lambda-microvms/runtime/v1/terminate HTTP/1.1" 200 OK +``` + +So `/terminate` is served, returns 200, reports a clean pipeline state, and does +not write terminal task status. **Note `"microvm_id": ""`** — the hook body's +`microvmId` arrives empty, unlike `/run` where it is populated. Cosmetic, but it +defeats the hook's stated purpose of joining the guest record to the +control-plane one, and the `/run` line has to carry that correlation instead +(which it does). + +**P1 F8 reconfirmed:** the VM sat at `TERMINATED` for the whole observation +window; `ResourceNotFoundException` never arrived. `TERMINATED` must map to +completed in its own right. + +### 5.2 `NO_INGRESS` — DISCHARGED + +``` +ingressNetworkConnectors = ['arn:aws:lambda:us-east-1:aws:network-connector:aws-network-connector:NO_INGRESS'] +egressNetworkConnectors = ['arn:aws:lambda:us-east-1::network-connector:nc-d306f00f-1bd0-45ea-9457-0fcec0dab2a4'] +``` + +The ARN P2 injects is **correct and effective**: the service accepted it and +`HTTP_INGRESS` (P1 F7's unrequested default public ingress) is **gone**. P1's +security finding is mitigated. + +**One caveat worth carrying forward:** a `NO_INGRESS` VM **still returns a public +endpoint hostname** (`.lambda-microvm.us-east-1.on.aws`). +So the presence of an `endpoint` in a `RunMicrovm` response is *not* evidence of +reachability, and any doc or alarm that treats "endpoint exists" as "ingress is +open" will be wrong in both directions. + +### 5.3 Dual-signal liveness — NOT DISCHARGED + +`agent_heartbeat_at` stayed `None` throughout. The heartbeat interval is 45 s and +no task was `RUNNING` for more than ~13 s, because F5 kills every task at turn 0. +**This item is blocked behind P2-F5, not independently testable.** The +`heartbeatLivenessApplies` switch is `RUNNING`-scoped and never got a +long-enough `RUNNING` window to exercise. + +--- + +## Phase 6 — Lifecycle extras + +### 6.1 A run-hook 4xx **self-terminates** the VM — supersedes P1 F9 + +Launching a second MicroVM from the image with **no** `runHookPayload` (the image +has `run: ENABLED`) produced a result P1 could not see, because P1's only +creatable image was hook-less: + +``` +state = TERMINATED +stateReason = Run lifecycle hook returned HTTP status 400. Please check your hook endpoint and application logs for more details. +``` + +Terminal within ~12 s of launch, and `suspend-microvm` then correctly refused: +`The MicroVM … has been terminated and its state cannot be changed.` + +**P1 F9 said "hook-less MicroVMs run indefinitely and bill", and warned that +nothing self-cleans.** With `/run` ENABLED that is no longer true: **the service +reaps a VM whose run hook returns 4xx.** This materially improves the cost +posture and is a direct benefit of declaring hooks. Active `TerminateMicrovm` +remains correct for the *success* path, but the *failure* path now has a +service-side backstop. + +Incidental confirmation: `executionRoleArn` must be a **full ARN** — +a bare role name is rejected with +`Member must satisfy regular expression pattern: arn:aws[a-z\-]*:iam::[0-9]{12}:role/?…`. + +### 6.2 Suspend / resume on a hook-ENABLED image — PASS + +To get a VM that stays `RUNNING`, a **hand-built valid envelope** (713 bytes: +`agent_payload` with `task_id`/`repo_url`/`task_description`/`resolved_workflow` +plus the four required `platform_config` keys) was passed as `runHookPayload`. +`/run` returned 200 and the VM held `RUNNING`. This is itself a useful result: +**the documented envelope shape is reproducible by hand from the contract alone.** + +| Step | Latency | State | Notes | +|---|---|---|---| +| launch → `RUNNING` | ~11 s | `RUNNING`, `stateReason` `None` | | +| `suspend-microvm` | **~2 s** | `SUSPENDED` | `/suspend` hook **DISABLED** in the image | +| `resume-microvm` | **~3 s** | `RUNNING` | `/resume` hook **DISABLED** | +| `terminate-microvm` | **~3 s** | `TERMINATED`, `stateReason` `Success.` | `/terminate` hook fired, 200 OK | + +`microvmId` **and** `endpoint` +(`.lambda-microvm.us-east-1.on.aws`) were +**byte-identical** before and after the suspend/resume cycle, and `startedAt` / +`maximumDurationInSeconds` (28800) never moved. + +**This extends P1 5.3/5.6 to a hook-enabled image:** suspend and resume work +*without* `/suspend` and `/resume` being declared, so P3's interface widening is +not gated on the hooks — a stored `SessionHandle` survives a cycle here too. +`SUSPENDING` was again never observable. + +### 6.3 Quota — re-probed, still NOT OBSERVABLE + +`L-CD1C0CC4` "Max allocated MicroVM memory" = **1024 Gigabytes**, and critically +`UsageMetric` = **`null`**. `AWS/Usage` still exposes **only** `CallCount` per API +name (`GetMicrovm`, `CreateMicrovmImage`, `ListMicrovmImages`, …) and **no memory +metric in any namespace**. **P1 5.5's verdict is unchanged: the claim "a +suspended VM still holds account memory quota" remains NOT OBSERVABLE SAFELY and +undischarged.** Proving it would still need ~128 concurrent 8 GiB VMs. + +--- + +## Phase 7 — Scratch repo + +**Nothing to clean up, and no PR to leave open.** `dreamorosi/batch-sync-triage` +is byte-for-byte as it was found: + +- Branches: `main` + the five pre-existing `dependabot/*`. **No `bgagent/*` + branch was ever pushed** — the agent created the branch locally in the guest + and died at `implement` before any commit or push. +- PRs: the same five open dependabot PRs (#1–#5) from 2025-11-24. **No PR was + created by this run.** + +So the instruction to leave the CODEOWNERS PR open is moot: **P2-F5 prevented any +PR from existing.** That absence is the single most important line in this +document. + +--- + +## Phase 8 — Teardown (executed as a finally-block) + +### 8.1 MicroVMs — all TERMINATED + +Nine MicroVMs existed across the run (six from the six task submissions, plus +the two Phase 6 lifecycle VMs and one orchestrator retry). **All nine were +already `TERMINATED`** at teardown — every one either finalized by the +orchestrator's `TerminateMicrovm`, service-reaped after a run-hook 4xx (6.1), or +explicitly terminated in 6.2. No VM needed chasing, and none ever approached the +8 h bound. + +### 8.2 Image and versions — deleted + +Only one version existed (`1.0`, `ACTIVE`), so `delete-microvm-image` alone was +sufficient and reaped it. `list-microvm-images` → `{"items": []}`; +`get-microvm-image` → +`ResourceNotFoundException: MicroVMImage not found for MicroVMImageID: …`. +P1's ordering correction (delete all but the last version, then the image) was +therefore not exercised, but is not contradicted. + +*Incidental:* one `ListMicrovmImages` call returned `502 Bad Gateway (reached max +retries: 2)` while the image was `DELETING`; it succeeded 30 s later. Worth +retrying rather than treating as a failure. + +### 8.3 Live IAM workarounds — reverted + +- Temporary unconditioned `iam:PassRole` inline policy + (`abca645p2-verification-passrole`) **deleted** from the orchestrator role. +- The execution role's trust policy **restored** to its deployed form, i.e. with + both `aws:SourceAccount` conditions back (verified by re-reading it). The + defect is left exactly as the branch produces it. + +### 8.4 Cognito user — deleted + +`bgagent admin delete-user ` → +`✓ Deleted Cognito user`. `list-users` then returned **empty**. The local invite +file containing its password was `rm`'d. The GitHub-token secret was left to die +with the stack (it did). + +### 8.5 Stack — DELETE_FAILED, 96/100 deleted, identical to P1's residual + +Three delete attempts, and each failed differently — that progression is itself +the finding: + +| Attempt | Duration | Outcome | +|---|---|---| +| 1 | 00:55:35Z → 01:04:28Z (8 min 53 s) | `DELETE_FAILED` — `AWS::BedrockAgentCore::Runtime`: `"Request timed out while deleting AWS::BedrockAgentCore::Runtime"`, `HandlerErrorCode: NotStabilized` | +| 2 (the briefed retry) | 01:04:55Z → 01:06:00Z (1 min) | `DELETE_FAILED` — same resource, **different error**: `"Access denied for operation 'AWS::BedrockAgentCore::Runtime'."`, `HandlerErrorCode: AccessDenied` | +| 3 (informed) | 01:07:02Z → 01:24:47Z (17 min 45 s) | `DELETE_FAILED` — but the Runtime, Memory and both IAM roles **did** delete; only the VPC set remains | + +Attempt 3 was not a blind third retry: between attempts, +`bedrock-agentcore-control list-agent-runtimes` returned **empty**, proving the +runtime was already gone server-side and the failures were a handler +stabilization/authorization artifact rather than a real leftover. Acting on that +evidence took the residual from 7 resources to 4. + +**Final residual — 4 resources, all zero-cost, exactly P1's set:** + +``` +CREATE_COMPLETE AWS::EC2::VPC vpc-072fddf653ccdcfc4 +DELETE_FAILED AWS::EC2::Subnet subnet-0e6a6a0ed18100c8a +DELETE_FAILED AWS::EC2::Subnet subnet-02d91450e51cf72a0 +DELETE_FAILED AWS::EC2::SecurityGroup sg-00997a58c1f4c5775 +``` + +``` +resource sg-00997a58c1f4c5775 has a dependent object (Service: Ec2, Status Code: 400 …) +Resource handler returned message: "The subnet 'subnet-0e6a6a0ed18100c8a' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400 …)" HandlerErrorCode: InvalidRequest +``` + +Cause: the same two AgentCore-managed ENIs, still `in-use` — +`eni-04911e11e08d670f9` and `eni-04a1c5a27966ee08b`, both +`InterfaceType: agentic_ai`. **This is #702 / P1 F12 reproducing verbatim.** +Nothing was force-deleted past CloudFormation. P1's experience says these are +eventually released (its stack is gone now — see 0.1), so the retry command is: + +```bash +aws cloudformation delete-stack --stack-name backgroundagent-dev +aws cloudformation wait stack-delete-complete --stack-name backgroundagent-dev +``` + +### 8.6 Billing confirmed stopped + +| Check | Result | +|---|---| +| NAT gateways in the ABCA VPC | **none** (the 2 `available` ones belong to pre-existing `vpc-01c9984d163d2965e` and were not touched — same as P1) | +| VPC endpoints in the ABCA VPC | **none** | +| ABCA S3 buckets | **none** | +| Unattached (billable) EIPs | **none** | +| MicroVM images / non-terminated VMs | **none** | +| AgentCore runtimes / memories | **none** | +| `/aws/lambda-microvms/*` log groups | **none** | + +### 8.7 Environment restored — nothing global left modified + +- `cdk/cdk.context.json` **restored** to the original six-AZ list (gitignored + build artifact; `git status` shows only `docs/verification/` and the + pre-existing `opencode.json`). +- `~/.finch/config.json` **restored** to `credsStore: osxkeychain` with **no + stored credential**, and the Lima VM's `DOCKER_CONFIG` restored to + `{"credsStore":"finchhost"}`; the `/root/.docker/config.json` written during + the push investigation was removed. **The short-lived ECR token written during + that investigation is no longer on disk anywhere.** +- The ECR tag this run added (`34fbc1d4…`) was removed with + `batch-delete-image`, leaving the three pre-existing tags and the underlying + manifest exactly as found. +- The finch VM was stopped, as P1 did. +- **No AWS global configuration was read or modified at any point** + (`~/.aws/config`, `~/.aws/credentials` untouched); `~/.cdk.json` was never + created. All credential and Region selection was via environment variables in + the run's own shell. + +### 8.8 Deliberately retained + +1. **`CDKToolkit`** — `UPDATE_COMPLETE`, `ComputeTypes = agentcore,lambda-microvm`, + `BootstrapVariant = ABCA: Least-Privilege Bootstrap`, five ABCA policies, no + `AdministratorAccess`. Unchanged by this run. ⚠️ Still the shared-account + caveat P1 raised: other CDK apps in `` deploy through the + ABCA-scoped execution role. +2. **`backgroundagent-dev` in `DELETE_FAILED`** — the 4 zero-cost resources in 8.5. +3. **Bootstrap S3/ECR assets** — normal bootstrap content, including the three + pre-existing agent container images. +4. **Service-vended log groups** created outside CloudFormation + (`/aws/bedrock-agentcore/runtimes/…`, `/aws/lambda/backgroundagent-dev-…`). +5. **`dreamorosi/batch-sync-triage`** — untouched (Phase 7). + +### 8.9 Scratch repo — verified unchanged + +Branches: `main` + the five pre-existing `dependabot/*`. Open PRs: #1–#5, all +dependabot. **No `bgagent/*` branch, no smoke PR.** There was nothing to close, +nothing to delete, and — because of P2-F5 — nothing to leave open for the +operator to look at. + +--- + +## Findings summary + +Live run 2026-08-06/07, account ``, `us-east-1`, branch +`feat/645-lambda-microvm-p2` @ `3a4b61a9`. Evidence: +`/tmp/abca-645-p2-20260806`. + +**Verdict on Stage D's primary objective: the smoke did NOT pass. No pull request +was created.** The run got to `clone ✓ → branch ✓ → implement ✗ (turn 0)`, +reproducibly, and it took four separate live workarounds to get even that far. +Every one of the five defects below is invisible to synth, unit tests and +`cdk-nag`; all five are live-service contract mismatches, which is precisely what +Stage D exists to surface. + +### Items DISCHARGED (behaved as designed) + +1. **`microvmImageHooks` spelling — both directions.** The API accepted + `microvmImageHooks:{ready,validate}` + `microvmHooks:{run,terminate}` with + timeouts, and CloudFormation resolved the identical property path + (`…/Hooks/MicrovmImageHooks/Ready`). The open P2 item is closed. +2. **All four hooks are served and exercised.** `/ready` 200 and `/validate` 200 + (`python=3.13.13, platform_config_keys=13, warnings=0`) during the build, on + both chipsets; `/run` 200 with the envelope; `/terminate` 200 at teardown. + **P1 F1 — "a P1 image that declares `/run` is not creatable at all" — is + fully fixed.** +3. **`NO_INGRESS` ARN.** Correct, accepted, and it suppresses P1 F7's default + public `HTTP_INGRESS`. (Caveat: a public endpoint hostname is still returned.) +4. **`platform_config` delivery end to end.** 2,120-byte envelope inline under + the real 4,096-byte cap; exactly the 11 available keys installed; names-only + logging; pre-install logging correctly stdout-only. +5. **Image buildability.** 4 min 35 s, two builds per version, `ACTIVE`; 8192 MiB + accepted; `apt-get`/Go egress over port 80 through the dedicated build + connector. **P1 F4 (443-only SG) and F5 (32 GiB memory) are fixed.** +6. **`MICROVM_IMAGE_IDENTIFIER` is a full ARN — P1 F3 fixed**, and all six + `MICROVM_*` vars appear together (all-or-nothing, both directions). +7. **`agentPlatformConfig` wiring.** 11 of 13 keys on the orchestrator; the two + absent ones are optional per-workspace secrets, so the optional-key path is + also verified. +8. **CLI end to end.** `admin invite-user` (permanent password — no first-login + challenge to drive), `configure --stack-name`, `login`, `repo onboard + --compute-type lambda-microvm` (ComputeSubstrate gate **and** live + `ListManagedMicrovmImages` probe both pass), `platform doctor` 7/7, + `submit`, `watch` streaming progress events. +9. **Finalization terminates the VM** (`TERMINATED`, `stateReason: Success.`). +10. **Suspend/resume on a hook-enabled image** with `/suspend` + `/resume` + undeclared: ~2 s / ~3 s, `microvmId` **and** `endpoint` preserved. +11. **Time-to-RUNNING ≈ 6 s** — the backend's headline advantage, measured. +12. **P1 F12's stack half resolved**: the leaked AgentCore ENIs were eventually + released and P1's `DELETE_FAILED` stack is gone. + +### Items CONTRADICTING design assumptions — `feeds-back-to-design: YES` + +**P2-F1 + P2-F3 (ONE root cause, two symptoms, both blocking). The +`aws:SourceAccount` confused-deputy condition makes ABCA's MicroVM-facing roles +unassumable by the Lambda MicroVMs service.** + +*Symptom A — the substrate cannot deploy.* Both `AWS::Lambda::NetworkConnector` +resources `CREATE_FAILED`, deterministically, on a freshly deleted stack: + +``` +"The service is unable to assume the provided NetworkConnectorOperatorRole. Please verify the trust policy on the role. (Service: Lambda, Status Code: 400, Request ID: dbe1d2f4-dd0c-4319-8f5d-b4be4f076843)" HandlerErrorCode: InvalidRequest +``` + +Removing the condition from `ConnectorOperatorRole`'s trust → both connectors +create within a second. Note this is a **regression introduced by the P1 F2 +fix**: P1's validated probe role trusted `lambda.amazonaws.com` with **no +conditions**, and the construct's comment says trust "mirrors the build/execution +roles (`lambda.amazonaws.com` + `aws:SourceAccount`)" — which is exactly what +breaks it. + +*Symptom B — no task can start.* `RunMicrovm` fails with a misleading +`iam:PassRole` denial **on the caller**, even though the orchestrator's grant is +present and `simulate-principal-policy` returns `allowed` and there is no +permissions boundary. Proven by elimination (unconditioned `iam:PassRole` still +denied; 3-minute wait still denied); removing the **execution role's** trust +conditions made the very next submission reach `RUNNING` in 6 s. + +Both the build role and the execution role carry the same pattern, so the fix is +one decision applied consistently: **drop `aws:SourceAccount` from every +MicroVM-facing role's trust policy, or find the condition key the service does +present** (`aws:SourceArn`/`aws:SourceAccount` are simply not populated on this +path). Note the *identity-side* `iam:PassedToService: lambda.amazonaws.com` +condition on the orchestrator was never shown to be wrong — it was exonerated by +step 1 and can stay. + +**P2-F2. The `CfnMicrovmImage` L1 is rejected by CloudFormation on five values — +the CDK-managed image path does not work at all.** Verbatim early-validation +output is in 1.5. `arm64` must be `ARM_64`; all four hooks must be +`ENABLED`/`DISABLED`, not paths. This **refutes the construct's stated reasoning** +that the L1 could keep CloudFormation's "path/string shape" because the generated +types "document no architecture/hook allowed-value constraint" — the service +enforces the enum at change-set time. Consequences: (a) the documented +"CDK-managed (recommended)" bootstrap path in +`cdk/scripts/package-microvm-artifact.sh` is currently non-functional and the +"out-of-band alternative" is the *only* working path; (b) hook paths are not +configurable on either surface, so the `*_HOOK_PATH` constants are agent route +constants only and must never be sent as property values. + +**P2-F4. The MicroVM execution role cannot write to the application log group +that `platform_config` tells the agent to use.** `logs:CreateLogStream` denied on +`/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/backgroundagent-dev` +for both the `server_debug/` and `metrics/` streams, because +the role's logs grant is scoped to `/aws/lambda-microvms/*`. P2 delivered +`log_group_name` (so the agent *attempts* the write) without the matching grant — +the same omission class the P2 Bedrock/Secrets/Memory grants were added to fix. +Non-fatal (stdout fallback lands in the MicroVM log group) but the canonical +per-task observability streams are empty on this backend, including +`METRICS_REPORT`. + +**P2-F5. `claude --version` times out after 10 s inside the MicroVM, failing +every task at turn 0. This is the blocker that prevented the PR.** Reproduced on +two consecutive submissions: + +``` +TimeoutExpired: Command '['claude', '--version']' timed out after 10 seconds +``` + +`agent/src/runner.py:476` uses `timeout=10`. The binary is fine: in the identical +image locally it answers `2.1.191 (Claude Code)` in **under 1 s**. It is a +**225 MiB (236,305,136-byte) statically-linked ELF** whose pages were never +touched before the snapshot was taken, exec'd on a guest restored ~50 s earlier — +consistent with lazy snapshot hydration. Two fixes: raise/make backend-aware the +timeout (a version-string probe gains nothing from a tight bound), and — more +interestingly — **warm `claude` in the `/ready` build hook**, which exists +precisely so "the snapshot is taken with a warm server" but currently warms only +uvicorn while leaving the 225 MiB binary that does all the work cold. + +**P2-F6. P1 F9 is superseded, in the safe direction.** With `run: ENABLED`, a +run-hook 4xx makes the **service** terminate the VM +(`stateReason: "Run lifecycle hook returned HTTP status 400."`) within ~12 s. P1's +"hook-less MicroVMs run indefinitely and bill; nothing self-cleans" no longer +describes this image. Worth correcting in ADR-021, because it changes the +cost-risk argument for the failure path. + +**P2-F7. Template size is at 98.4 % of the CloudFormation 1 MB limit** +(983,796/1,000,000) and 486/500 resources with the image configured. ~16 KB of +headroom — roughly one more construct before deploys start failing for reasons +unrelated to MicroVMs. + +**P2-F8. Runbook/tooling corrections (lower severity, but each would corrupt a +future pass).** + +- **P1 F14's `--no-paginate` remedy is wrong and silently truncating.** At 485 + resources it returns only the first page, resolved `ORCHESTRATOR_FN` to the + literal `None`, and produced `Function not found: …:function:None`. Let the CLI + paginate and filter client-side. +- **P1 F10's "recurs forever" half is wrong**: `BootstrapVariant` now reads + `ABCA: Least-Privilege Bootstrap`, because P1's own + `update-stack --use-previous-template` with only `ComputeTypes` supplied resets + unspecified parameters to template defaults. The silent-`exit 0` half stands. +- **`--execution-role-arn` requires a full ARN** (regex in 6.1); a bare role name + is rejected. +- **`zsh`**: `"$VAR:latest"` is the `${VAR:l}` lowercase modifier (cost one + mis-diagnosed push); no `PIPESTATUS` (it is `$pipestatus`, 1-indexed), so P1's + `test "${PIPESTATUS[0]}" -eq 0` silently evaluates empty; no `timeout(1)`. +- **Verify secrets on the raw value, never a `.strip()`ed copy** — a trailing + newline from `gh auth token` produced a convincing false clone defect (3.2). +- **`finch` cannot push to ECR on this box**: `finch push` runs + `limactl shell finch sudo -E nerdctl push` and the VM's `DOCKER_CONFIG` uses + the `finchhost` creds helper, which cannot resolve an Isengard + `credential_process`; a host-side `finch login` does not help. Both `push` and + `pull` fail with `no basic auth credentials`. +- The `/terminate` hook body's **`microvmId` arrives empty** (`"microvm_id": ""`), + defeating the hook's stated guest↔control-plane correlation purpose. +- **`ARTIFACTS_BUCKET_NAME` and `TRACE_ARTIFACTS_BUCKET_NAME` resolve to the same + bucket**, making that separation notional. + +### Items SKIPPED, BLOCKED, or INCONCLUSIVE + +| Item | Verdict | Reason | +|---|---|---| +| **clone → change → PR (the point of Stage D)** | **FAILED — no PR** | P2-F5, reproduced twice. Stopped at two retries as briefed. | +| Dual-signal liveness / `agent_heartbeat_at` fresh during RUN | **BLOCKED behind P2-F5** | Heartbeat interval is 45 s; no task stayed `RUNNING` beyond ~13 s. `agent_heartbeat_at` was `None` on all six submissions. Not independently testable until F5 is fixed. | +| Suspend TTL beyond P1's ≥1 h floor | **SKIPPED (time-boxed)** | Would have added 2 h+ of wall clock for a marginal bound after ~2.5 h already spent on five blocking defects. P1's result stands unchanged: **survives ≥ 1 h 0 min 17 s with no `idlePolicy`**; a TTL between 1 h and the 8 h `maximumDurationInSeconds` bound remains **OPEN**. | +| SUSPENDED VM consumes account memory quota | **STILL NOT OBSERVABLE — undischarged** | Re-probed: `L-CD1C0CC4` = 1024 GB with `UsageMetric: null`; `AWS/Usage` has only `CallCount`; no memory metric in any namespace. Identical to P1 5.5. | +| CDK-managed `CfnMicrovmImage` deploy | **REFUTED (P2-F2)** | Fell back to the out-of-band script path, as the brief directed. | +| AgentCore container asset push | **WORKED AROUND** | finch/ECR auth (deviation 2). ECR manifest retagged; irrelevant to the MicroVM image under test. | +| P1 F11 (AgentCore-unsupported AZ) | **STILL UNFIXED** | `agent-vpc.ts` unchanged; worked around via the gitignored AZ context cache, restored at teardown. | +| P1 F12 Memory-delete half | **STILL UNFIXED** | `AWS::BedrockAgentCore::Memory` still undeletable while `CREATING`; still turns a rollback into `ROLLBACK_FAILED`; plain `delete-stack` still clears it. | +| Orchestrator-role IAM negatives | **NOT ATTEMPTED** | P1 5.0 established the Lambda trust does not allow operator assumption; trust was not modified for this purpose. | + +### Elapsed and approximate cost + +**Elapsed:** 22:38Z → 01:26Z ≈ **2 h 48 min**. Roughly: ~35 min on the finch/ECR +push dead end and the ECR-retag workaround; ~40 min on the five deploy attempts +plus two `ROLLBACK_FAILED`/`delete-stack` cycles; ~10 min on image create+build; +~35 min on the six submissions and the P2-F3 elimination sequence; ~10 min on +lifecycle extras; ~30 min on teardown (three delete attempts); the remainder on +evidence capture and this document. + +**Approximate cost: well under US$5**, dominated as in P1 by NAT/VPC-endpoint +hours rather than MicroVMs. + +| Item | Quantity | Est. | +|---|---|---| +| NAT gateway (1 × $0.045/h) | ~1.6 h across 3 stack lifetimes | ~$0.08 | +| Interface VPC endpoints (7 × $0.01/h × 2 AZ) | ~1.6 h | ~$0.22 | +| MicroVM runtime | 9 VMs, all short-lived (~12 s to ~3 min each); longest suspended window ~1 min | < $0.10 | +| MicroVM image builds | 2 builds (1 version × 2 chipsets), ~4.5 min each | low single-digit cents | +| Snapshot/image storage | ~3.6 GiB × ~1 h | negligible | +| AgentCore runtime + Memory | created 3×, **never invoked** | negligible | +| Bedrock | **zero model tokens** — every task died before turn 1 | $0 | +| S3 / DynamoDB / Lambda / API GW / Cognito / Secrets / logs | brief, mostly idle | < $1 | + +The 8 h `maximumDurationInSeconds` was never approached; every VM was either +explicitly terminated or service-reaped. + +### Recommended follow-up before P2 is called complete + +Ordered by what unblocks what: + +1. **P2-F1/F3** (`aws:SourceAccount` on MicroVM-facing role trust) — nothing + deploys or runs without this. One decision, three roles. +2. **P2-F5** (`claude --version` timeout) — nothing *completes* without this. It + is the only thing between this run and a PR, and the `/ready`-warming option + is worth considering on its merits rather than just raising the timeout. +3. **P2-F2** (L1 enum values) — the documented recommended path is dead until + fixed; five one-word changes plus the tests that assert the old strings. +4. **P2-F4** (application-log-group grant) — cheap, and it is what makes the next + failure debuggable through the platform rather than through guest stdout. +5. **P2-F7** (template size) — unrelated to MicroVMs but it will bite soon. +6. Re-run Stage D after 1–2. The dual-signal-liveness item and the suspend-TTL + extension both need a task that stays `RUNNING` for minutes, which only F5's + fix provides. + +--- + +## Stage D-redux (run 2) + +Narrow re-run on branch `feat/645-lambda-microvm-p2` @ +`b927d1d6a58e2b040c2ed4ce4e9f1dc9be9fc981` — the commit that fixed P2-F1..F5 +against run 1's evidence. **Purpose: convert "fixed-against-evidence" into +"re-exercised live", and get the pull request.** + +**Live run:** 2026-08-07 02:58Z → 04:55Z (≈1 h 57 min), account ``, +`us-east-1`. Evidence directory: `/tmp/abca-645-p2r2-20260806`. + +**Teardown: complete.** Live IAM workaround reverted, Cognito user deleted, image ++ version deleted, ECR retag removed, AZ context cache restored, 481/485 stack +resources deleted (4 zero-cost residual, #702), **all billable resources confirmed +gone**, and no global/host configuration touched. Both smoke PRs left open as +briefed. See 2.12. + +> **Provenance note — HEAD moved mid-run, from outside this run.** Preflight +> confirmed `HEAD = b927d1d` with a clean tree at 02:58Z. At **03:04Z** an +> unrelated user commit landed on the branch — `045722c fix(deps): refresh +> js-yaml lock entry to 4.3.1`, **`yarn.lock` only, 3 insertions / 3 deletions**. +> No `git` write command was issued by this run, and `b927d1d` remains an +> ancestor of `HEAD`. **It cannot have affected any finding:** `mise run install` +> / `yarn install` was never re-run, so `cdk/node_modules` still reflected +> `b927d1d`'s lock for every synth and deploy; the MicroVM artifact is built from +> `agent/` + `contracts/` + `agent/Dockerfile` (Python/uv), which the commit does +> not touch. Every verdict below is therefore against `b927d1d`'s tree. + +### 2.0 Headline + +> **THE SMOKE PASSED. `https://github.com/dreamorosi/batch-sync-triage/pull/6`** +> — clone → change → commit → push → PR, `COMPLETED`, 12 turns, $0.279, 153 s. +> Run 1's single most important line ("no PR was created") is retired. + +But the PR required **one live IAM workaround**, and establishing *why* produced +the run's most consequential result: **P2-F3 is NOT fixed, and run 1's +exoneration of its identity-side condition was a false negative.** Two of the +five P2 fixes are fully discharged, two are discharged, one is refuted, and one +brand-new blocking defect was found on the path run 1 never reached. + +| Fix | Run-1 verdict | Run-2 live verdict | +|---|---|---| +| **P2-F1** (connector trust) | blocking | ✅ **DISCHARGED** — substrate deployed **first try**, zero workarounds, 0 `CREATE_FAILED` in 485 resources | +| **P2-F2** (`ARM_64`/`ENABLED` enums) | REFUTED at change-set validation | ✅ **DISCHARGED** — early validation **passed**, resource reached `CREATE_IN_PROGRESS` | +| **P2-F3** (`RunMicrovm` PassRole) | blocking; "trust was the sole cause" | ❌ **NOT FIXED (P2r2-F10)** — isolated to the *identity-side* `iam:PassedToService` condition run 1 explicitly exonerated | +| **P2-F4** (application-log grant) | blocking observability | ✅ **DISCHARGED** — `server_debug/`, `metrics/` **and** `trajectory/` streams exist, with content | +| **P2-F5** (`claude` warm-up) | **the blocker** — no PR | ✅ **DISCHARGED** — cold `claude` measured at **17–38 s**, warm **0.1 s**, `/ready` 200, no 503 | +| *(new)* **P2r2-F9** | not reachable in run 1 | ❌ CDK-managed image path blocked: bootstrap `IAMPassRole` denies the build role to CloudFormation | +| *(new)* **P2r2-F11** | mis-attributed in run 1 | ⚠️ `agent_heartbeat_at` is never projected into the API response | +| Dual-signal liveness | BLOCKED behind F5 | ✅ **DISCHARGED** — 45 s cadence observed live over a 181 s `RUNNING` window | + +### 2.1 Deltas from run 1's setup + +```bash +export SMOKE_USER="" +export EVIDENCE_DIR=/tmp/abca-645-p2r2-20260806 +export BGAGENT_CONFIG_DIR=/tmp/abca-645-p2r2-bgagent +``` + +Everything else — the zsh traps, `set -o pipefail`, explicit `AWS_REGION`, +newest-first version ordering — carried over unchanged and all of it still +applies. Two run-1 notes paid for themselves immediately: the **raw-secret +assertion** (2.6) and **client-side pagination** for `TaskOrchestrator` lookup. + +`bgagent` was invoked as `node cli/lib/bin/bgagent.js` after +`mise //cli:compile` (there is no linked binary in this tree). + +### 2.2 Preflight — run 1's residual had to be cleared first, and it did not clear itself + +`backgroundagent-dev` was still `DELETE_FAILED` with run 1's exact 4-resource +residual, and **the two `agentic_ai` ENIs were still `in-use` 1 h 32 min later** +(`eni-04911e11e08d670f9`, `eni-04a1c5a27966ee08b`, both requester `amazon-aws`, +`InstanceOwnerId: amazon-aws`), while +`bedrock-agentcore-control list-agent-runtimes` and `list-memories` both returned +**empty**. So the ENIs outlive the resources that created them by a wide margin. + +Run 1's documented retry command was executed verbatim and **failed again after +17 min 17 s** (02:58:41Z → 03:15:58Z): + +``` +The following resource(s) failed to delete: [AgentVpcRuntimeSG96507CD0, AgentVpcPrivateSubnet1Subnet8051BB57, AgentVpcPrivateSubnet2SubnetC66971D0]. +``` + +**Correction to run 1's §8.5 advice.** Run 1 concluded from P1's experience that +"these are eventually released, so the retry command is `delete-stack`". That is +true on a multi-day horizon and **useless on a same-session horizon** — a +17-minute retry that fails identically is not a remedy. The remedy that works is +`--retain-resources`, which cleared the stack record in **33 seconds**: + +```bash +aws cloudformation delete-stack --stack-name backgroundagent-dev \ + --retain-resources AgentVpcA6796801 AgentVpcPrivateSubnet1Subnet8051BB57 \ + AgentVpcPrivateSubnet2SubnetC66971D0 AgentVpcRuntimeSG96507CD0 +``` + +Note the VPC (`CREATE_COMPLETE`, never attempted) must be listed alongside the +three `DELETE_FAILED` children or the delete fails on it. Cost: an orphaned +zero-cost VPC + 2 subnets + 1 SG, now outside CloudFormation's knowledge (2.12). + +*Unchanged from run 1:* `CDKToolkit` `UPDATE_COMPLETE`, `ComputeTypes = +agentcore,lambda-microvm`, `BootstrapVariant = ABCA: Least-Privilege +Bootstrap`, bootstrap SSM version `32`, five ABCA policies, no +`AdministratorAccess` — so **no re-bootstrap was run**. That decision turns out +to matter; see P2r2-F9. + +**Managed base image** — still exactly one in `us-east-1`, +`arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1`, versions `1` (newest +first) and `0`. Selected `1`; the service echoed `baseImageVersion: "1.0"`. + +**P1 F11 is still UNFIXED** (`agent-vpc.ts` is still `maxAzs: 2` with no AZ +constraint, `us-east-1a` is still `use1-az6`), so the gitignored +`cdk/cdk.context.json` AZ cache was trimmed to lead with `us-east-1b`/`us-east-1c` +again, saved to `$EVIDENCE_DIR/cdk.context.json.ORIGINAL`, and **restored at +teardown**. `git status` stayed clean throughout. + +### 2.3 The AgentCore container asset — retag workaround reused, and the tag had moved + +`finch` still cannot push to ECR (run 1 deviation 2 — the Lima VM's +`credsStore: finchhost` cannot resolve an Isengard `credential_process`). The +required asset tag was **`7a71005f…`, not run 1's `34fbc1d4…`**, because +`b927d1d` grew `agent/src/server.py`; run 1's tag had been correctly removed at +its teardown. The same `aws ecr put-image` retag was applied to the existing +manifest `sha256:cdf5436a…`: + +``` +{"imageDigest": "sha256:cdf5436ab8d9d17bcd5b555ad22c52b4e6d6622f1fc0373cdc97a73c3eb8e6a4", + "imageTag": "7a71005fe6f520a2741cdd1fb2a47ffa919b13e63476226ec39bc66eb4c150c5"} +``` + +Same caveat, restated because it is easy to lose: **the deployed AgentCore +runtime therefore carries a stale agent image, and nothing in this run depends on +it** — the smoke runs on the MicroVM substrate built from the S3 zip. The finch VM +was never started this run, and `~/.finch/config.json` was never touched +(verified still `credsStore: osxkeychain` with no stored credential). + +### 2.4 Substrate deploy — P2-F1 DISCHARGED, first try, zero workarounds + +Synth: `abca:microvm-image-not-provisioned` emitted, `…-p1-smoke-unverified` +correctly absent. Deploy `03:19:25Z → 03:33:59Z = 14 min 34 s`, **`EXIT=0` on the +first attempt**, 485 resources. + +**This is the whole of P2-F1's verification and it is unambiguous.** Run 1 needed +five attempts and a `/tmp` cloud-assembly trust-policy patch to get here. Run 2 +needed none: + +``` +2026-08-07T03:21:47Z LambdaMicrovmComputeEgressConnector9C36AAC2 CREATE_IN_PROGRESS +2026-08-07T03:21:48Z LambdaMicrovmComputeBuildEgressConnector3B762F80 CREATE_IN_PROGRESS +2026-08-07T03:26:05Z LambdaMicrovmComputeEgressConnector9C36AAC2 CREATE_COMPLETE +2026-08-07T03:26:06Z LambdaMicrovmComputeBuildEgressConnector3B762F80 CREATE_COMPLETE +``` + +and a scan of every stack event for `*FAILED` returned **`NONE`**. The live trust +policies on both the execution and build roles read exactly as source intends — +`lambda.amazonaws.com`, `sts:AssumeRole` + `sts:TagSession`, **no conditions**. + +All seven MicroVM outputs populated; `ComputeSubstrate = lambda-microvm`; +artifact key exactly `microvm-images/agent-artifact.zip`. + +**P2-F7 reconfirmed and marginally worse:** 979,867 B / 485 resources (substrate), +**985,886 B / 486** with the image — **98.6 %** of the 1 MB limit, ~14 KB of +headroom. Down from run 1's ~16 KB. + +### 2.5 The CDK-managed image path — P2-F2 DISCHARGED, then blocked by a NEW defect (P2r2-F9) + +The synthesized L1 now carries exactly the five values CloudFormation rejected in +run 1: + +```json +"CpuConfigurations": [{ "Architecture": "ARM_64" }], +"Hooks": { + "Port": 8080, + "MicrovmHooks": { "Run": "ENABLED", "RunTimeoutInSeconds": 60, + "Terminate": "ENABLED", "TerminateTimeoutInSeconds": 15 }, + "MicrovmImageHooks": { "Ready": "ENABLED", "ReadyTimeoutInSeconds": 300, + "Validate": "ENABLED", "ValidateTimeoutInSeconds": 60 } +} +``` + +**P2-F2 is DISCHARGED.** Change-set early validation **passed** — zero +`not a valid enum value` errors, no `Early validation failed` — and the resource +progressed to `CREATE_IN_PROGRESS`. Run 1's §1.5 refutation is fully answered and +the enum fix is correct on the CloudFormation surface. + +It then failed on something run 1 could never have seen, because run 1 never got +past early validation: + +``` +LambdaMicrovmComputeImage16B48539 CREATE_FAILED +Resource handler returned message: "User: arn:aws:sts:::assumed-role/cdk-hnb659fds-cfn-exec-role--us-east-1/AWSCloudFormation is not authorized to perform: iam:PassRole on resource: arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeBuildRoleF0-9FxjQbiJC3px because no identity-based policy allows the iam:PassRole action (Service: LambdaMicrovms, Status Code: 403, Request ID: f8c45ab5-49c4-42fd-ac61-a6a3ac00dc26) (SDK Attempt Count: 1)" HandlerErrorCode: AccessDenied +``` + +`UPDATE_ROLLBACK_COMPLETE`; the substrate survived intact (all seven outputs still +populated), so this cost one deploy cycle and nothing else. + +**Diagnosis — and it is NOT a stale bootstrap.** The live +`…IaCRole-ABCA-Infrastructure…` policy is **byte-identical to +`cdk/bootstrap/policies/infrastructure.json` on this branch**, so +`bootstrap --force` would change nothing. Its `IAMPassRole` statement is: + +```json +{ "Sid": "IAMPassRole", "Effect": "Allow", "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/backgroundagent-dev-*", + "Condition": { "StringEquals": { "iam:PassedToService": [ + "lambda.amazonaws.com", "ecs-tasks.amazonaws.com", "ecs.amazonaws.com", + "apigateway.amazonaws.com", "logs.amazonaws.com", "bedrock.amazonaws.com", + "bedrock-agentcore.amazonaws.com", "events.amazonaws.com", + "vpc-flow-logs.amazonaws.com" ] } } } +``` + +The resource pattern matches the build role. `simulate-principal-policy` on the +deploy role returns **`allowed`** with +`iam:PassedToService=lambda.amazonaws.com` and **`implicitDeny`** with no context +— so the resource is right and the *condition* is the only remaining variable. + +**The control that makes this airtight:** the out-of-band `--create-image` path +(2.6) passed **the same build role** to the same service successfully, using +operator credentials that carry no such condition. So the build role's trust is +fine and the service can assume it; the denial is genuinely caller-side. + +This directly contradicts the construct's own accounting, which asserts +`buildRole` is passed "via the bootstrap `infrastructure` policy's `IAMPassRole`" +— live, it is not. + +*Incidental:* **CloudTrail has no `lambda-microvms` events at all** for this run +(`lookup-events` across the window returns nothing for `CreateMicrovmImage` / +`RunMicrovm`). The service appears not to emit management events yet, which is why +the actual `iam:PassedToService` value cannot simply be read out of a log — every +determination in this section had to be made by elimination. + +### 2.6 Image + build — P2-F5 DISCHARGED, with numbers + +Artifact: **935,874 bytes**, SSE `AES256` (run 1: 922,056). `--create-image` exit +**0**; the service echoed `ARM_64`, `minimumMemoryInMiB: 8192`, all four hooks +`ENABLED`, and `readyTimeoutInSeconds: 300`. + +Build `03:42:53Z → 03:47:23Z = 4 min 30 s`, both chipsets +(`GRAVITON` generation `3` and `4`) `SUCCESSFUL`, `state=SUCCESSFUL`, +`status=ACTIVE`. **The warm-up cost the build ~22 s** versus run 1's 4 min 35 s +hook-less-warm-up baseline — an outstanding trade for what it buys. + +**The decisive evidence, verbatim** (per build stream): + +``` +[server/build-hook] /ready hook: server is up, warming the snapshot before it is taken +[server/build-hook] /ready hook: warmed 'claude' in 17.2s (version='2.1.191 (Claude Code)') +[server/build-hook] /ready hook: warmed 'claude' in 37.8s (version='2.1.191 (Claude Code)') +[server/build-hook] /ready hook: warmed 'git' in 0.0s (version='git version 2.47.3') +[server/build-hook] /ready hook: warmed 'claude' in 0.1s (version='2.1.191 (Claude Code)') +[server/build-hook] /ready hook: warmed 'node' in 3.0s (version='v24.19.0') +[server/build-hook] /ready hook: reporting ready for snapshot +INFO: 127.0.0.1:33946 - "POST /aws/lambda-microvms/runtime/v1/ready HTTP/1.1" 200 OK +[server/build-hook] /validate hook: ok (python=3.13.13, platform_config_keys=13, warnings=0) +INFO: 127.0.0.1:53154 - "POST /aws/lambda-microvms/runtime/v1/validate HTTP/1.1" 200 OK +``` + +**Four independent things are proven here, and three of them are new:** + +1. **The lazy-hydration diagnosis was correct, and now it is measured.** A cold + `exec` of the 225 MiB `claude` binary takes **17.1–37.8 s**. Run 1 inferred + this from a 10 s timeout; run 2 has the number. **`timeout=10` could never + have passed** — it was 2–4× short. +2. **The warm-up works.** The same binary in the same guest answers in **0.1 s** + once its pages are faulted in. That is the mechanism doing exactly what the + `/ready` docstring claims. +3. **The service issues `/ready` three times per build, and the first two run + concurrently.** Both concurrent calls warm `claude` simultaneously and contend + (17.2 s and 37.8 s in the same stream). Worst-case single call ≈ 37.8 + 0.0 + + 6.4 ≈ **44 s**. So the 120 s required budget carries ~3.2× margin and the move + from `readyTimeoutInSeconds` 60 → 300 was not merely defensive — a 60 s hook + budget would have had ~16 s of slack against a contended cold start. +4. **No 503, ever.** The "required warm-up failed" path never fired, and + `/validate` still reports `platform_config_keys=13, warnings=0`. + +*Not re-captured:* the 2.3 size table (`codeInstallSizeInBytes` etc.) — the image +was deleted at teardown before those fields were read. Run 1's figures stand; +this run adds nothing to or against P1 F13. + +**Wired deploy** with `--context microvm_image_identifier=`: +`03:48:34Z → 03:53:37Z = 5 min 3 s`, `UPDATE_COMPLETE`. + +**`MICROVM_*` is FIVE vars this run, not run 1's six**, and that is **correct by +design**: `MICROVM_IMAGE_VERSION` is emitted only when the optional +`microvm_image_version` context is supplied (`agent.ts:245` → +`task-orchestrator.ts:467`), and the construct documents its absent state as +"let the service pick". Run 1 recorded six because it supplied the version. +Not a regression — but worth stating, because "all-or-nothing" is asserted of the +`MICROVM_*` block and the version is the one member that legitimately opts out. + +`platform_config` wiring reconfirmed at **11 of 13** keys, with +`LINEAR_OAUTH_SECRET_ARN` / `JIRA_OAUTH_SECRET_ARN` correctly absent. + +**P2-F4's grant is present on the live execution role** — the second statement is +new versus run 1: + +```json +{ "Action": ["logs:CreateLogStream","logs:PutLogEvents"], + "Resource": "arn:aws:logs:us-east-1::log-group:/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/backgroundagent-dev:*", + "Effect": "Allow" } +``` + +### 2.7 Platform user, secret, onboarding — all PASS + +`admin invite-user` → `CONFIRMED` with a permanent password; `configure +--stack-name`; `login` → `Login successful.` Exactly run 1's two-command flow. + +**The PAT was written correctly this time** — `gh auth token | tr -d '\n\r'` into +`--secret-string file:///dev/stdin`, then asserted on the **raw** value per run +1's §3.2 lesson: + +``` +len 40 differs_from_stripped False prefix gho_ +``` + +Run 1's self-inflicted 41-byte defect did not recur. `repo onboard +--compute-type lambda-microvm` → `status: active`, both the ComputeSubstrate gate +and the live `ListManagedMicrovmImages` probe passing; `platform doctor` → **7/7**. + +### 2.8 THE SMOKE — five submissions, and the last two are an experiment + +| # | Task ID | Orchestrator `iam:PassRole` grant in force | Settle | Outcome | +|---|---|---|---|---| +| 1 | `01KZD5SZ55N707GMWY14P09JGR` | **source only** (exact ARN + `iam:PassedToService`) | ~2.5 min | `FAILED` — PassRole denial | +| 2 | `01KZD61FHYH3QT774PFETDNVKY` | + unconditioned, exact ARN | 20 s | `FAILED` — identical | +| 3 | `01KZD6D95P5VXJTJ06BFJXF8J6` | + unconditioned, `backgroundagent-dev-*` | 3.5 min | ✅ **`RUNNING` in 5 s → `COMPLETED` → PR #6** | +| 4 | `01KZD71DJSTGZK2A91MZJ8GFHJ` | **source only** (workaround removed) | **5 min** | `FAILED` — identical → **control** | +| 5 | `01KZD7D7HD3CT2APBT3DH12XFE` | + unconditioned, **exact ARN** (source's resource) | **5 min** | ✅ **`RUNNING` in 9 s → `COMPLETED` → PR #7** | + +**Submissions 4 and 5 are the isolation experiment, and they are the most +important result in this document.** Same exact-ARN resource as source, same +5-minute settle, one variable — the `iam:PassedToService: lambda.amazonaws.com` +condition. With it: denied. Without it: `RUNNING` in 9 s. See P2r2-F10. + +#### 2.8.1 THE SMOKE (submission 3), `bgagent watch`, verbatim highlights + +``` +[9:07:17 PM] ★ repo_setup_complete: branch=bgagent/01KZD6D95P5VXJTJ06BFJXF8J6/add-a-codeowners-file-at-the-repository-root-conta build_before=False +[9:07:18 PM] ★ step:implement:start +[9:08:14 PM] ★ pre_approvals_loaded {"scopes":[],"count":0} +[9:08:31 PM] Turn #1 (claude-opus-4-8, 0 tool calls) + Text: This is a simple, well-defined task. Let me create the CODEOWNERS file. +[9:08:37 PM] ▶ Write: {'file_path': '/workspace/01KZD6D95P5VXJTJ06BFJXF8J6/CODEOWNERS', 'content': '* @dreamorosi\n'} +[9:09:27 PM] ▶ Bash: git add CODEOWNERS && git commit -m "chore(github): add CODEOWNERS file" && git push -u origin bgagent/… +[9:09:43 PM] ▶ Bash: gh pr create --repo dreamorosi/batch-sync-triage --head bgagent/… --base main --title "chore(github): add CODEO… +[9:09:45 PM] ◀ Bash: https://github.com/dreamorosi/batch-sync-triage/pull/6 +[9:09:50 PM] Cost: $0.2792 (1947 in / 3554 out tokens) +[9:09:50 PM] ★ step:implement:succeeded +[9:09:50 PM] ★ agent_execution_complete: status=success turns=22 +[9:09:50 PM] ★ pr_created: https://github.com/dreamorosi/batch-sync-triage/pull/6 +Task 01KZD6D95P5VXJTJ06BFJXF8J6 completed. +``` + +Final record: `status COMPLETED`, `duration_s 153.4`, `cost_usd 0.279`, +`turns_completed 12`, `build_passed true`, `lint_passed false`, +`session_id microvm-082ebde7-…`. **Time-to-`RUNNING` 5 s**, confirming run 1's +headline performance result on a task that actually finishes. + +*Minor inconsistency worth a glance:* `watch` prints +`agent_execution_complete: turns=22` while the persisted record and +`METRICS_REPORT` both say `turns: 12`. Two different notions of "turn" (SDK +messages vs. counted iterations) surfaced under one word in the same stream. + +*Incidental, both tasks:* `lint_passed: false` is scratch-repo noise +(`tsc: not found`, `biome` schema mismatch, `mise ERROR no tasks defined`), and +the agent correctly reasoned about it as pre-existing before proceeding. + +#### 2.8.2 P2-F4 — DISCHARGED with content, not just stream existence + +The brief's test was whether `server_debug/` now *exists*. It does, and +so does more: + +``` +metrics/01KZD6D95P5VXJTJ06BFJXF8J6 +server_debug/01KZD6D95P5VXJTJ06BFJXF8J6 +server_debug/server +trajectory/01KZD6D95P5VXJTJ06BFJXF8J6 +``` + +All three per-task streams were `AccessDenied` in run 1. `server_debug` carries +the `/run` breadcrumbs — including the **11-key** `platform_config` install line, +names only — and `metrics/` carries the `METRICS_REPORT` that run 1 lost +entirely: + +```json +{"event": "METRICS_REPORT", "status": "success", "agent_status": "success", + "pr_url": "https://github.com/dreamorosi/batch-sync-triage/pull/6", + "build_passed": true, "lint_passed": false, "cost_usd": 0.27916625, + "turns": 12, "duration_s": 153.4, "task_id": "01KZD6D95P5VXJTJ06BFJXF8J6", + "disk_before": "167.5 KB", "disk_after": "261.0 MB", …} +``` + +**P2-F4 is fully discharged**, and the platform's canonical per-task +observability is no longer empty on this backend. + +#### 2.8.3 Dual-signal liveness — DISCHARGED, and run 1's verdict was partly an artifact + +Polled **against DynamoDB** during submission 5's `RUNNING` window: + +| Wall clock | Status | Running for | `agent_heartbeat_at` | Freshness | +|---|---|---|---|---| +| 04:25:11Z | `RUNNING` | 71 s | `2026-08-07T04:24:47Z` | 24 s | +| 04:25:38Z | `RUNNING` | 99 s | `2026-08-07T04:25:32Z` | **6 s** | +| 04:26:05Z | `RUNNING` | 126 s | `2026-08-07T04:25:32Z` | 34 s | +| 04:26:33Z | `RUNNING` | 153 s | `2026-08-07T04:26:17Z` | 16 s | +| 04:27:00Z | `COMPLETED` | 181 s | `2026-08-07T04:26:17Z` | 44 s | + +Successive values are `04:24:47 → 04:25:32 → 04:26:17`: **exactly the 45 s +`_HEARTBEAT_INTERVAL_SECONDS` cadence**, freshness never worse than 34 s while +`RUNNING`. **The dual-signal liveness path is DISCHARGED on `lambda-microvm`** — +`heartbeatLivenessApplies` has a real, fresh signal to read. Progress events +streamed live in `watch` concurrently (2.8.1). + +**But this only became visible by reading DynamoDB directly.** `bgagent status` +reported `agent_heartbeat_at = None` on every poll of *both* completed tasks — +including submission 3, whose stored value was `2026-08-07T04:09:39Z`, 12 s before +`completed_at`. Cause: `toTaskDetail` (`cdk/src/handlers/shared/types.ts:786`) +never maps the field, though `TaskRecord` declares it at line 95 and +`orchestrator.ts` consumes it for liveness. See P2r2-F11 — and note this makes +run 1's "heartbeats NOT observed" partly a measurement artifact rather than a +pure consequence of P2-F5. + +### 2.9 Lifecycle — PASS + +Both smoke MicroVMs finalized cleanly: `state TERMINATED`, `stateReason +Success.`, `maximumDurationInSeconds 28800` never approached. `NO_INGRESS` +reconfirmed on both, and **run 1's caveat holds** — a `NO_INGRESS` VM still +returns a public endpoint hostname +(`76cfed33-…lambda-microvm.us-east-1.on.aws`), so "an endpoint exists" remains no +evidence of reachability. + +`list-microvms` showed **11** VMs, all `TERMINATED` — run 1's 9 plus this run's 2, +so the list is cumulative across runs and none of run 1's ever resurfaced. + +### 2.10 Not attempted this run + +Deliberately out of the narrow scope: suspend/resume and suspend-TTL (run 1 §6.2 +covered a hook-enabled image), the SUSPENDED-vs-quota probe (still +`UsageMetric: null` territory), and run 1's §6.1 run-hook-4xx self-termination +(P2-F6 — already recorded, and `b927d1d` corrected the ADR for it). + +### 2.11 Scratch repo — TWO PRs left open + +``` +#7 docs(contributors): add CONTRIBUTORS.md | bgagent/01KZD7D7HD3CT2APBT3DH12XFE/… +#6 chore(github): add CODEOWNERS file | bgagent/01KZD6D95P5VXJTJ06BFJXF8J6/… +#1–#5 pre-existing dependabot PRs +``` + +**#6 is the briefed smoke and is left open as instructed.** **#7 is a by-product +of the 2.8 isolation experiment** (submission 5 needed a task that would actually +run, and a second distinct file avoided colliding with #6) and is left open +alongside it rather than closed, so the evidence for the experiment survives. +Two `bgagent/*` branches were pushed; `main` is untouched. + +### 2.12 Teardown (executed as a finally-block) + +| Step | Result | +|---|---| +| **Live IAM workaround** | `abca645p2r2-verification-passrole` **deleted** from the orchestrator role; `list-role-policies` shows only the CDK-managed default policy. **No trust policy was modified at any point this run** (verified: execution role still `sts:AssumeRole` + `sts:TagSession`, conditionless, exactly as source produces). | +| **MicroVMs** | All 11 `TERMINATED` before teardown began; none needed chasing. | +| **Image + version** | Single version `1.0`; `delete-microvm-image` → `DELETING`, reaped it. | +| **Cognito user** | `admin delete-user` → `✓ Deleted`; `list-users` → `[]`; invite file `rm`'d. | +| **Secret** | Left to die with the stack. | +| **`cdk/cdk.context.json`** | **Restored** to the original six-AZ list; `git status` shows only `docs/verification/` + pre-existing `opencode.json`. | +| **ECR retag** | `7a71005f…` removed with `batch-delete-image`; pre-existing tags and the underlying manifest untouched. | +| **finch** | Never started this run; `~/.finch/config.json` never touched (still `credsStore: osxkeychain`, no stored credential). | +| **Global/host config** | **`~/.aws/*` never read or modified.** All credential and Region selection via environment variables in the run's own shell. `~/.cdk.json` never created. | + +**Stack — two delete attempts, ending at run 1's exact residual:** + +| Attempt | Window | Outcome | +|---|---|---| +| 1 | 04:29:02Z → 04:37:24Z (8 min 22 s) | `DELETE_FAILED` — `AWS::BedrockAgentCore::Runtime`: `"Request timed out while deleting AWS::BedrockAgentCore::Runtime"`, `HandlerErrorCode: NotStabilized`. **19** resources left. | +| 2 (informed) | 04:37:50Z → 04:55:17Z (17 min 27 s) | `DELETE_FAILED` — but the Runtime, Memory, all IAM roles, all DynamoDB tables, the S3 bucket and the secret **did** delete. **4** resources left. | + +Attempt 2 was not a blind retry: `bedrock-agentcore-control list-agent-runtimes` +returned **empty** first, proving the runtime was already gone server-side and the +failure was a handler stabilization artifact. **Run 1's attempt-3 technique +reproduced exactly and is confirmed as the right procedure** — it took the +residual from 19 to 4. + +**Final residual — 4 resources, all zero-cost, identical in shape to run 1 and P1:** + +``` +CREATE_COMPLETE AWS::EC2::VPC vpc-07dad8897791f477b +DELETE_FAILED AWS::EC2::Subnet subnet-09a1ff1f7568d05ef +DELETE_FAILED AWS::EC2::Subnet subnet-068397b48a25bf13f +DELETE_FAILED AWS::EC2::SecurityGroup sg-05e3f48665c47b358 +``` + +Pinned by two fresh `agentic_ai` ENIs (`eni-0942f1b0b3b6553ea`, +`eni-07876963c2818da63`, both `in-use`). **#702 / P1 F12 reproducing for the third +consecutive run.** Nothing was force-deleted past CloudFormation. + +**Billing confirmed stopped:** + +| Check | Result | +|---|---| +| NAT gateways in either orphaned ABCA VPC | **none** | +| VPC endpoints | **none** | +| ABCA S3 buckets | **none** | +| Unattached (billable) EIPs | **none** | +| MicroVM images / non-`TERMINATED` VMs | **none** | +| AgentCore runtimes / memories | **none** | +| `/aws/lambda-microvms/*` log groups | **none** | +| ABCA DynamoDB tables / GitHub-token secret | **none** | + +**Deliberately retained:** `CDKToolkit` (unchanged — still the shared-account +caveat P1 raised); bootstrap S3/ECR assets; service-vended log groups created +outside CloudFormation; the two scratch-repo PRs (2.11); and **two** orphaned +zero-cost VPC sets — run 2's four resources above (still inside the +`DELETE_FAILED` stack) plus **run 1's**, which `--retain-resources` moved outside +CloudFormation entirely (`vpc-072fddf653ccdcfc4`, `subnet-0e6a6a0ed18100c8a`, +`subnet-02d91450e51cf72a0`, `sg-00997a58c1f4c5775`). + +A best-effort hand cleanup of run 1's set was attempted and **refused** — +`DependencyViolation` on both subnets, the SG and the VPC, because +`eni-04911e11e08d670f9` and `eni-04a1c5a27966ee08b` were **still `in-use` +3 h 30 min after run 1's teardown**. Nothing was forced. Retry for both sets: + +```bash +aws cloudformation delete-stack --stack-name backgroundagent-dev # run 2's set +# run 1's set is no longer CFN-managed: +aws ec2 delete-subnet --subnet-id subnet-0e6a6a0ed18100c8a +aws ec2 delete-subnet --subnet-id subnet-02d91450e51cf72a0 +aws ec2 delete-security-group --group-id sg-00997a58c1f4c5775 +aws ec2 delete-vpc --vpc-id vpc-072fddf653ccdcfc4 +``` + +### 2.13 Findings summary + +Live run 2026-08-07 02:58Z → 04:55Z, account ``, `us-east-1`, branch +`feat/645-lambda-microvm-p2` @ `b927d1d6`. Evidence: +`/tmp/abca-645-p2r2-20260806`. + +**Verdict on the primary objective: THE SMOKE PASSED. +`https://github.com/dreamorosi/batch-sync-triage/pull/6`.** Clone → change → +commit → push → PR, `COMPLETED` in 153 s for $0.28, with progress events and a +live heartbeat. That retires run 1's headline. **It required one live IAM +workaround, and pinning down why is the run's most valuable output.** + +#### Fixes CONVERTED to "re-exercised live" + +1. **P2-F1 — DISCHARGED.** Substrate deployed **first try**, `EXIT=0`, + 14 min 34 s, **zero workarounds**, both `AWS::Lambda::NetworkConnector` + resources `CREATE_COMPLETE`, and **no `*FAILED` event anywhere** in 485 + resources. Run 1 needed five attempts and a cloud-assembly patch. +2. **P2-F2 — DISCHARGED.** Change-set **early validation passed** with `ARM_64` + and four `ENABLED` hooks; the resource reached `CREATE_IN_PROGRESS`. Run 1's + five-value refutation is answered on the CloudFormation surface. (The path is + still blocked downstream — P2r2-F9 — but *not* on the enums.) +3. **P2-F4 — DISCHARGED with content.** `server_debug/`, + `metrics/` **and** `trajectory/` all exist; the + `METRICS_REPORT` run 1 lost entirely now lands. +4. **P2-F5 — DISCHARGED, and now quantified.** Cold `claude` exec measured at + **17.1–37.8 s** (so `timeout=10` was 2–4× short and could never have passed); + **0.1 s once warm**; `/ready` 200, no 503; `/validate` still + `platform_config_keys=13, warnings=0`; whole build 4 min 30 s, i.e. the + warm-up cost ~22 s. **New empirical detail:** the service calls `/ready` + **three times per build, the first two concurrently**, so two cold `claude` + execs contend — worst single call ≈ 44 s. The 120 s required budget and the + `readyTimeoutInSeconds` 60 → 300 move are both correctly sized; 60 s would + have left ~16 s of slack. +5. **Dual-signal liveness — DISCHARGED** (run 1: BLOCKED). `agent_heartbeat_at` + advanced `04:24:47 → 04:25:32 → 04:26:17` — exact 45 s cadence — across a + 181 s `RUNNING` window, freshness ≤ 34 s. + +#### Items CONTRADICTING design assumptions — `feeds-back-to-design: YES` + +**P2r2-F10 (BLOCKING). P2-F3 is NOT fixed. The orchestrator's *identity-side* +`iam:PassedToService: lambda.amazonaws.com` condition — the one run 1 explicitly +exonerated and `b927d1d` deliberately kept — is a second, independent blocker.** + +Isolated by a clean two-arm experiment, same exact-ARN resource as source, same +5-minute settle, one variable: + +| Orchestrator grant | Result | +|---|---| +| exact ARN **+ `iam:PassedToService: lambda.amazonaws.com`** (source as written) | **DENIED** — submissions 1 *and* 4 | +| exact ARN, **no condition** | **`RUNNING` in 9 s** — submission 5 | + +``` +Session start failed: Error: MicroVM RunMicrovm failed: AccessDeniedException: User: arn:aws:sts:::assumed-role/backgroundagent-dev-TaskOrchestratorOrchestratorFnS-a7sP6rFzoIkU/backgroundagent-dev-TaskOrchestratorOrchestratorFn-p41lJqwFmxNG is not authorized to perform: iam:PassRole on resource: arn:aws:iam:::role/backgroundagent-dev-LambdaMicrovmComputeExecutionRo-pZQWXvKsITBa because no identity-based policy allows the iam:PassRole action +``` + +The trust half of run 1's fix was **necessary but not sufficient**. Run 1's +exoneration was a **false negative with an identifiable cause**: its temporary +unconditioned `iam:PassRole` was attached at §4.1 step 1 and, per its own §8.3, +**was still attached through submissions 4 and 5** — the ones that reached +`RUNNING`. So run 1 never tested the conditioned grant against a *working* trust, +and attributed the whole effect to the trust change. + +Two source statements must therefore change, and the comment in +`task-orchestrator.ts` asserting the condition "was EXONERATED live … so it +stays" must be reversed: + +- `cdk/src/constructs/task-orchestrator.ts`, sid `MicrovmPassExecutionRole` +- `cdk/bootstrap/policies/infrastructure.json`, sid `IAMPassRole` (P2r2-F9) + +**P2r2-F9 (BLOCKING, same root cause). The CDK-managed image path is dead one +step later than run 1 thought: CloudFormation cannot pass the build role.** +Verbatim `CREATE_FAILED` in 2.5. **This is not a stale bootstrap** — the live +policy is byte-identical to this branch's +`cdk/bootstrap/policies/infrastructure.json`. `simulate-principal-policy` returns +`allowed` for `iam:PassedToService=lambda.amazonaws.com` and `implicitDeny` +without it, so the resource pattern is right and the condition is the variable; +and the out-of-band `--create-image` call **succeeded passing the same build +role**, proving the trust is fine and the denial is caller-side. + +**P2r2-F9 and P2r2-F10 are ONE root cause with TWO symptoms** — precisely the +shape of run 1's P2-F1/F3, one layer in: **the Lambda MicroVMs service does not +present `iam:PassedToService: lambda.amazonaws.com` on either `PassRole` path** +(CloudFormation → build role at `CreateMicrovmImage`; orchestrator → execution +role at `RunMicrovm`). Consequence for the docs: the "CDK-managed (recommended)" +bootstrap path in `package-microvm-artifact.sh` is **still** non-functional and +`--create-image` is **still** the only working path — for a new reason. + +**P2r2-F11. `agent_heartbeat_at` is written and consumed correctly but is +invisible through the API.** `toTaskDetail` +(`cdk/src/handlers/shared/types.ts:786`) does not map it, though `TaskRecord` +declares it (line 95) and `orchestrator.ts` reads it for liveness; `cli/src/types.ts` +has no such field at all. So `bgagent status` / `watch` report `None` even when +DynamoDB holds a 6-second-old value. Cheap to fix, and worth fixing because **it +already caused a wrong conclusion**: run 1 recorded "heartbeats NOT observed" and +attributed it wholly to P2-F5. + +**P2r2-F12. Run 1's §8.5 stack-delete retry advice does not work on a +same-session horizon, and the ENI leak is worse than #702 records.** Run 1's +documented `delete-stack` retry failed **identically after 17 min 17 s**, with the +`agentic_ai` ENIs still `in-use` 1 h 32 min after run 1's teardown and *no* +AgentCore runtimes or memories in existence. At the end of *this* run those same +two ENIs were **still `in-use` 3 h 30 min on**, and a hand `delete-subnet` / +`delete-security-group` / `delete-vpc` sweep was refused with +`DependencyViolation` on all four. So the leak is not "slow" — it is **unbounded +relative to a developer's session**, and it now compounds: each run strands +another VPC set. The working escape hatch is `--retain-resources` (33 s), which +must list the `CREATE_COMPLETE` VPC alongside the three `DELETE_FAILED` children: + +```bash +aws cloudformation delete-stack --stack-name backgroundagent-dev \ + --retain-resources AgentVpcA6796801 AgentVpcPrivateSubnet1Subnet8051BB57 \ + AgentVpcPrivateSubnet2SubnetC66971D0 AgentVpcRuntimeSG96507CD0 +``` + +#702 should carry both the unbounded-hold evidence and this escape hatch, rather +than have each run rediscover them. Run 1's attempt-3 technique (verify +`list-agent-runtimes` is empty, *then* retry) is separately **confirmed correct** +— it took this run's residual from 19 resources to 4. + +**P2r2-F13. `MICROVM_IMAGE_VERSION` is legitimately optional** — five +`MICROVM_*` vars this run vs run 1's six, because the optional +`microvm_image_version` context was not supplied. Correct by design +(`task-orchestrator.ts:467`), but it qualifies the "all-or-nothing `MICROVM_*`" +claim, which should name the version as the one member that may be absent. + +**P2r2-F14. `turns` is reported inconsistently in one stream.** `watch` prints +`agent_execution_complete: turns=22`; the persisted record and `METRICS_REPORT` +both say `12`. + +**P2-F7 reconfirmed, worse.** 985,886 B / 486 resources = **98.6 %** of the 1 MB +limit, ~14 KB of headroom (run 1: ~16 KB). + +**CloudTrail blind spot.** No `lambda-microvms` management events at all, so +`PassRole` context values cannot be read from logs — every determination in +2.5/2.8 had to be by elimination. Worth knowing before the next person tries. + +**Still unfixed from earlier runs:** P1 F11 (AZ constraint — worked around again +via the gitignored AZ cache, restored at teardown); the finch→ECR push failure; +P1 F8 (`TERMINATED` is terminal, `ResourceNotFoundException` never arrives). + +#### Recommended follow-up + +1. **P2r2-F10 + P2r2-F9 together** — drop or correct `iam:PassedToService` on + both the orchestrator statement and the bootstrap `IAMPassRole`. Nothing runs + without the first; the recommended image path stays dead without the second. + Determining the value the service *does* present needs either AWS + confirmation or a bounded candidate sweep — `microvms.lambda.amazonaws.com`, + `lambda-microvms.amazonaws.com` and `microvms.amazonaws.com` are all + `implicitDeny` against the current policy, so any of them would work as the + allow-list entry if it is the right one. Reverse the "EXONERATED … so it + stays" comment while you are there. +2. **P2r2-F11** — one line in `toTaskDetail` plus the CLI type; it is what makes + the liveness signal observable to the people who need it. +3. **P2r2-F12** — put the `--retain-resources` escape hatch in #702. +4. **P2-F7** — `suppressTemplateIndentation` or a stack split; ~14 KB left. +5. **A third Stage D is NOT needed for P2-F1/F2/F4/F5 or dual-signal liveness** — + all five are now live-verified. The next run's scope is P2r2-F9/F10 plus the + still-deferred empirical items (suspend TTL > 1 h, SUSPENDED-vs-quota). + +### 2.14 Elapsed and cost + +**Elapsed:** 02:58Z → 04:55Z ≈ **1 h 57 min**, of which ~18 min was the failed +delete retry, ~15 min the substrate deploy, ~6 min the refuted CDK-managed image +attempt, ~10 min image create+build, ~5 min the wired deploy, ~30 min the five +submissions and the isolation experiment (mostly IAM settle waits), ~26 min the +two teardown delete attempts, and the remainder evidence capture. + +**Approximate cost: well under US$5**, dominated as always by NAT/VPC-endpoint +hours. Unlike run 1 this run actually spent Bedrock tokens: **$0.478 total across +two completed tasks** ($0.279 + $0.199). Two MicroVMs ran ~2.5 min each; two +image builds ~4.5 min each; one NAT gateway and 7×2 interface endpoints for +~1.6 h across a single stack lifetime. + +The 8 h `maximumDurationInSeconds` was never approached; every VM was explicitly +terminated by the orchestrator's finalization. diff --git a/scripts/check-constants-sync.ts b/scripts/check-constants-sync.ts index 008923a3e..60456404b 100644 --- a/scripts/check-constants-sync.ts +++ b/scripts/check-constants-sync.ts @@ -47,7 +47,15 @@ const REPO_ROOT = path.resolve(import.meta.dirname, '..'); const CONSTANTS_JSON = path.join(REPO_ROOT, 'contracts/constants.json'); const POLICY_PY = path.join(REPO_ROOT, 'agent/src/policy.py'); const JIRA_REACTIONS_PY = path.join(REPO_ROOT, 'agent/src/jira_reactions.py'); -const PYTHON_CONSUMERS = [POLICY_PY, JIRA_REACTIONS_PY]; +const SERVER_PY = path.join(REPO_ROOT, 'agent/src/server.py'); +const PYTHON_CONSUMERS = [POLICY_PY, JIRA_REACTIONS_PY, SERVER_PY]; +const MICROVM_COMPUTE_TS = path.join(REPO_ROOT, 'cdk/src/constructs/lambda-microvm-compute.ts'); +const TS_CONSUMERS = [MICROVM_COMPUTE_TS]; + +/** Env var names must be UPPER_SNAKE — they are installed into a process env. */ +const ENV_NAME_PATTERN = /^[A-Z][A-Z0-9_]*$/; +/** ``platform_config`` wire keys are snake_case. */ +const CONFIG_KEY_PATTERN = /^[a-z][a-z0-9_]*$/; /** * Constant names that ``contracts/constants.json`` owns and the @@ -56,6 +64,11 @@ const PYTHON_CONSUMERS = [POLICY_PY, JIRA_REACTIONS_PY]; * ``NAME: int = 50`` styles; the regex literals are hard-coded (not * built from string concatenation) so semgrep's * ``detect-non-literal-regexp`` rule is satisfied without an exception. + * + * The two ``MICROVM_PLATFORM_CONFIG_*`` patterns are shaped differently: those + * constants are a mapping and a set, so the drift they catch is a collection + * LITERAL (``= {``, ``= [``, ``= frozenset({``) rather than a scalar. A + * contract-sourced ``= dict(_CONTRACT["env_by_key"])`` does not match. */ const OWNED_PYTHON_PATTERNS: ReadonlyArray<{ name: string; regex: RegExp }> = [ { name: 'DEFAULT_APPROVAL_GATE_CAP', regex: /^\s*DEFAULT_APPROVAL_GATE_CAP\s*(?::\s*int)?\s*=\s*-?\d+\b/m }, @@ -65,6 +78,43 @@ const OWNED_PYTHON_PATTERNS: ReadonlyArray<{ name: string; regex: RegExp }> = [ { name: 'DEFAULT_TASK_TIMEOUT_S', regex: /^\s*DEFAULT_TASK_TIMEOUT_S\s*(?::\s*int)?\s*=\s*-?\d+\b/m }, { name: 'APP_ACTOR_MIN_SECRET_LENGTH', regex: /^\s*APP_ACTOR_MIN_SECRET_LENGTH\s*(?::\s*int)?\s*=\s*\d+\b/m }, { name: 'FORGE_WEBTRIGGER_SUFFIX', regex: /^\s*FORGE_WEBTRIGGER_SUFFIX\s*(?::\s*str)?\s*=\s*["']/m }, + { + name: 'MICROVM_PLATFORM_CONFIG_ENV_BY_KEY', + regex: /^\s*MICROVM_PLATFORM_CONFIG_ENV_BY_KEY\s*(?::[^=]+)?=\s*(?:dict\()?\s*[[{]/m, + }, + { + name: 'MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS', + regex: /^\s*MICROVM_PLATFORM_CONFIG_REQUIRED_KEYS\s*(?::[^=]+)?=\s*(?:frozenset\(|set\()?\s*[[{]/m, + }, + // ADR-021 P2: the `/ready` warm-up ceiling and the CDK-side `/ready` hook + // timeout are a RELATIONSHIP (`warmup_total < ready_hook`), and a relationship + // cannot be enforced from one side — so neither side may re-declare its half as + // a literal. A contract-sourced `= _HOOK_BUDGETS["…"]` does not match. + { + name: '_READY_WARMUP_TOTAL_BUDGET_SECONDS', + regex: /^\s*_READY_WARMUP_TOTAL_BUDGET_SECONDS\s*(?::\s*int)?\s*=\s*-?\d+\b/m, + }, + { + name: '_READY_WARMUP_REQUIRED_TIMEOUT_SECONDS', + regex: /^\s*_READY_WARMUP_REQUIRED_TIMEOUT_SECONDS\s*(?::\s*int)?\s*=\s*-?\d+\b/m, + }, +]; + +/** + * The TypeScript half of the same no-literal-redeclaration rule. + * + * `tsc` already catches a *renamed* contract field, but it cannot catch a + * construct that stops reading the contract and goes back to + * `const READY_HOOK_TIMEOUT_SECONDS = 300;` — which is exactly how the budget + * invariant got asserted against a hardcoded copy in the first place. So the one + * TypeScript consumer whose value is half of a cross-language invariant is checked + * the same way the Python ones are. + */ +const OWNED_TS_PATTERNS: ReadonlyArray<{ name: string; regex: RegExp }> = [ + { + name: 'READY_HOOK_TIMEOUT_SECONDS', + regex: /^\s*(?:export\s+)?const\s+READY_HOOK_TIMEOUT_SECONDS\s*(?::\s*number)?\s*=\s*-?\d+\b/m, + }, ]; interface Drift { @@ -74,10 +124,21 @@ interface Drift { } function findDriftInPython(filePath: string): Drift[] { + return findLiteralDrift(filePath, OWNED_PYTHON_PATTERNS); +} + +function findDriftInTypeScript(filePath: string): Drift[] { + return findLiteralDrift(filePath, OWNED_TS_PATTERNS); +} + +function findLiteralDrift( + filePath: string, + patterns: ReadonlyArray<{ name: string; regex: RegExp }>, +): Drift[] { const source = fs.readFileSync(filePath, 'utf-8'); const lines = source.split('\n'); const drifts: Drift[] = []; - for (const { name, regex } of OWNED_PYTHON_PATTERNS) { + for (const { name, regex } of patterns) { for (const line of lines) { if (regex.test(line)) { drifts.push({ file: filePath, name, line: line.trim() }); @@ -93,6 +154,12 @@ function main(): number { approval_gate_cap?: { min: number; max: number; default: number }; approval_timeout_s?: { min: number; max: number; default: number }; jira_app_actor?: { min_secret_length: number; forge_webtrigger_suffix: string }; + microvm_platform_config?: { env_by_key: Record; required: string[] }; + microvm_hook_budgets?: { + ready_hook_timeout_seconds: number; + warmup_total_budget_seconds: number; + warmup_required_timeout_seconds: number; + }; }; try { json = JSON.parse(fs.readFileSync(CONSTANTS_JSON, 'utf-8')); @@ -140,13 +207,106 @@ function main(): number { invariantErrors.push('jira_app_actor.forge_webtrigger_suffix must start with "."'); } + // ADR-021 P2: the MicroVM `/run` hook installs `platform_config` into the + // agent's process environment, so this block is BOTH a cross-language contract + // (agent/src/server.py consumes it; the orchestrator's run-hook envelope + // builder produces it) AND a security allowlist. A malformed entry here would + // widen what the agent accepts into its env, so the shape is validated, not + // just present. + const mpc = json.microvm_platform_config; + if ( + !mpc + || typeof mpc.env_by_key !== 'object' + || mpc.env_by_key === null + || !Array.isArray(mpc.required) + ) { + console.error( + `${CONSTANTS_JSON} is missing microvm_platform_config.{env_by_key,required}`, + ); + return 1; + } + + const envByKey = mpc.env_by_key; + const configKeys = Object.keys(envByKey); + if (configKeys.length === 0) { + invariantErrors.push('microvm_platform_config.env_by_key must not be empty'); + } + for (const key of configKeys) { + if (!CONFIG_KEY_PATTERN.test(key)) { + invariantErrors.push(`microvm_platform_config.env_by_key key "${key}" must be snake_case`); + } + const envName = envByKey[key]; + if (typeof envName !== 'string' || !ENV_NAME_PATTERN.test(envName)) { + invariantErrors.push( + `microvm_platform_config.env_by_key["${key}"] must be an UPPER_SNAKE env var name`, + ); + } + } + const envNames = configKeys.map(key => envByKey[key]); + if (new Set(envNames).size !== envNames.length) { + invariantErrors.push( + 'microvm_platform_config.env_by_key maps two keys onto the same env var', + ); + } + if (mpc.required.length === 0) { + invariantErrors.push('microvm_platform_config.required must not be empty'); + } + for (const key of mpc.required) { + if (!Object.hasOwn(envByKey, key)) { + invariantErrors.push( + `microvm_platform_config.required names "${key}", absent from env_by_key`, + ); + } + } + if (new Set(mpc.required).size !== mpc.required.length) { + invariantErrors.push('microvm_platform_config.required contains a duplicate'); + } + + // ADR-021 P2: `/ready` does real work (it warms the 225 MiB `claude` binary), so + // the agent's warm-up ceiling and the CDK-declared hook timeout are coupled — the + // warm-up MUST finish inside the budget the service holds the hook to, or a fix + // for a runtime failure becomes a build failure. Both halves live here precisely + // so the relationship is checkable; this is the check. + const mhb = json.microvm_hook_budgets; + const BUDGET_FIELDS = [ + 'ready_hook_timeout_seconds', + 'warmup_total_budget_seconds', + 'warmup_required_timeout_seconds', + ] as const; + if (!mhb || BUDGET_FIELDS.some(field => !Number.isInteger(mhb[field]))) { + console.error( + `${CONSTANTS_JSON} is missing microvm_hook_budgets.{${BUDGET_FIELDS.join(',')}} ` + + '(all must be integers)', + ); + return 1; + } + for (const field of BUDGET_FIELDS) { + if (mhb[field] <= 0) invariantErrors.push(`microvm_hook_budgets.${field} must be > 0`); + } + if (mhb.warmup_total_budget_seconds >= mhb.ready_hook_timeout_seconds) { + invariantErrors.push( + 'microvm_hook_budgets.warmup_total_budget_seconds must be < ' + + 'ready_hook_timeout_seconds (/ready has to answer inside the hook budget)', + ); + } + if (mhb.warmup_required_timeout_seconds >= mhb.warmup_total_budget_seconds) { + invariantErrors.push( + 'microvm_hook_budgets.warmup_required_timeout_seconds must be < ' + + 'warmup_total_budget_seconds (the required warm-up must leave the ' + + 'best-effort ones something to share)', + ); + } + if (invariantErrors.length > 0) { console.error(`Semantic invariant violations in ${CONSTANTS_JSON}:\n`); for (const e of invariantErrors) console.error(` - ${e}`); return 1; } - const drifts = PYTHON_CONSUMERS.flatMap(findDriftInPython); + const drifts = [ + ...PYTHON_CONSUMERS.flatMap(findDriftInPython), + ...TS_CONSUMERS.flatMap(findDriftInTypeScript), + ]; if (drifts.length > 0) { console.error('Cross-language constants drift detected:\n'); @@ -164,7 +324,9 @@ function main(): number { console.log( `Constants sync OK: contracts/constants.json validated; ` + `${OWNED_PYTHON_PATTERNS.length} Python names checked across ` + - `${PYTHON_CONSUMERS.length} consumers.`, + `${PYTHON_CONSUMERS.length} consumers, ` + + `${OWNED_TS_PATTERNS.length} TypeScript name(s) across ` + + `${TS_CONSUMERS.length} consumer(s).`, ); return 0; }