diff --git a/MAINTAINER.md b/MAINTAINER.md index 6ec02b6..37decc3 100644 --- a/MAINTAINER.md +++ b/MAINTAINER.md @@ -66,6 +66,7 @@ src/ admin-view.mjs # admin browser controller (embedded into the page; not node-imported) browser.mjs # openInBrowser — shared by dashboard + admin usage-index.mjs # canonical usage aggregation by host, provider, model, project, and category + model-inventory/ # model evidence contracts, source adapters, snapshots, diffs, impact, read models npx.mjs # stale npx-cache detection/prune mcp.mjs settings.mjs config.mjs paths.mjs statusline.mjs rvf.mjs daemons.mjs exec.mjs output.mjs @@ -78,6 +79,7 @@ tests/ docs/ HOST-SUPPORT.md # canonical host/Ruflo/AQE/Brain compatibility matrix (shipped) INSTALLATION.md # package scope versus machine/user/project effects (shipped) + MODELS.md # model lifecycle inventory and read-only planning guide (shipped) PROVIDERS.md # provider and routing guide (shipped) SETUP.md # setup mutation contract (shipped) TROUBLESHOOTING.md # symptom-to-fix runbook (shipped) @@ -90,9 +92,11 @@ docs/ **Published tarball** = the `files` whitelist in `package.json`: `bin/agentic-kit.mjs`, `src/`, `claude/`, `docs/HOST-SUPPORT.md`, -`docs/INSTALLATION.md`, `docs/PROVIDERS.md`, `docs/SETUP.md`, -`docs/TROUBLESHOOTING.md`, `docs/UPGRADING.md`, `docs/CODEX-STATUSLINE.md`, and -`docs/adr/0015-managed-codex-native-statusline.md`. Generated workspace state under +`docs/INSTALLATION.md`, `docs/MODELS.md`, `docs/PROVIDERS.md`, `docs/SETUP.md`, +`docs/TROUBLESHOOTING.md`, `docs/UPGRADING.md`, `docs/CODEX-STATUSLINE.md`, +`docs/adr/0015-managed-codex-native-statusline.md`, +`docs/adr/0032-model-lifecycle-intelligence.md`, and +`docs/ddd/model-lifecycle-intelligence.md`. Generated workspace state under the shipped source trees is explicitly excluded. Nothing else ships — verify with `npm pack --dry-run` before a release if you touch `files`. diff --git a/README.md b/README.md index 086d41a..87aa19c 100644 --- a/README.md +++ b/README.md @@ -91,6 +91,8 @@ ak system machine footprint: install size, runtime, storage, catalog, proj [--deep] [--json] ak usage inspect/refresh offline provider account analytics status | refresh openrouter +ak models inspect model lifecycle evidence and swap impact + status | refresh | diff | explain | plan ak host manage execution hosts, routing, and provider bindings status | pick | refresh | off ak run execute a host-neutral activity pipeline (including explicit OpenCode routes) @@ -113,9 +115,10 @@ command always works.) | ------ | -------------- | | **setup** | Installs/updates ruflo + agentic-qe + the **agentdb** CLI globally (handling npm ≥11.17's `allow-scripts` so natives build; agentdb is pinned to ruflo's bundled version so the shared learning store stays coherent), installs the **RuvNet Brain** (an offline knowledge base over the rUv stack, powering the `search_ruvnet` MCP — a ~2 GB one-time download, prompted; skip with `--no-ruvnet-brain`), deploys the token-audit skill, merges the managed guidance blocks into the machine-wide guidance files (`~/.claude/CLAUDE.md`, plus `~/.codex/AGENTS.md` on codex machines), offers one-time MCP registration (user scope, with a tool-family picker), and — inside a repo — initializes the project: sanitized `ruflo init`, absolute memory-path pin, a **verified** store→disk write, statusline footer, and a background daemon with **local-only ($0) workers** (token-spending AI workers stay opt-in behind upstream's machine-wide budget). Project scope triggers on a `.git` entry in the current folder; without one it's skipped with a note. `--project` forces the same project setup in the current directory (e.g. a not-yet-`git init`-ed folder); it does not locate an ancestor repository. Project initialization runs `ruflo init --full --force` and can replace existing agent configuration, so read the [setup scope and project mutation contract](docs/SETUP.md) before using it on an existing project. `--minimal` skips it, `--yes` accepts all prompts (non-interactive), `--no-aqe` / `--no-ruvnet-brain` / `--no-security` disable those subsystems, and `--reconfigure` re-offers MCP registration. `--codex` enables + installs the Codex host during setup (ambidextrous dual-host mode; both hosts then run at once), and `--primary-host claude\|codex` picks which host leads (codex implies `--codex`). | | **status** | Per-subsystem ✓/⚠/✗ (versions, the kit's own version, **ruvnet-brain** (present + release drift, or "not installed"), natives (agentdb copies **and** ruflo's own memory runtime — the one `npx ruflo memory` loads — load-tested for a native better-sqlite3, not just the agentdb dirs), **memory-pin** (warns when `CLAUDE_FLOW_DB_PATH` points off the live DB), security, learning, aqe/RVF, **agentdb** (CLI present + coherent with ruflo's bundled version, or a store-skew warning), MCP, **hosts** (claude/codex/opencode version + install method; the Claude/Codex **primary** marked and failed when absent), **providers** (host wiring + aqe fallback chain, or "drifted"/claude-only default), **routing** (the persisted activity host+model policy; only Claude/Codex routes project into AQE), daemons, guidance-file blocks (`~/.claude/CLAUDE.md`, project `AGENTS.md`, and `~/.codex/AGENTS.md` on codex machines), statusline), each drift row naming what `sync` would do about it — plus a **health-history** line that flags regressions since the last sync (learning shrank, native slots dropped, drift/security backslid). | -| **sync** | The one convergence verb: upgrades first when a new release exists, then re-heals everything an upgrade wipes, then re-checks and reports. Included in that heal: it **installs any enabled frontier host** (claude/codex/opencode) that's entirely absent — never touching an external (mise/brew/native) install — and **re-applies provider wiring** (the `ENABLE_*` host env, OpenCode's native configuration, the aqe fallback chain, and ruflo API providers) whenever it has drifted — and, on a dual-host project, **seeds/heals the Claude/Codex default routing policy** (materializing eligible routes into agentic-qe's `agentOverrides`, e.g. after an aqe upgrade first makes it eligible). It also **installs/repins the standalone `agentdb` CLI** to ruflo's bundled version (keeping the shared cognitive store coherent) and appends a **health-history snapshot** so `status` can flag regressions across syncs. It also **re-runs the RuvNet Brain installer** to pull the latest release when the on-disk KB has drifted (or installs it if absent, when enabled). It also **self-updates the kit**: when a newer `@pacphi/agentic-kit` exists it installs it as the *last* step (the new code applies from the next `ak` run, never mid-sync). Prerelease installs (`4.0.0-alpha.*`) track the `next` npm dist-tag as well as `latest`, so alphas see their successors; stable installs only ever follow `latest`. `--no-upgrade` skips the self-update along with the package upgrades. | -| **dashboard** | Opens an observation-only local web dashboard (`127.0.0.1:7431`, localhost-only, never detaches) with five primary areas: **About · Overview · Usage · Observability · System**. About is a plain-words directory of everything the kit installed and why (the same content as `ak about`); Overview covers readiness, hosts & routing, providers, runtime, and machine-wide intelligence; Usage turns local Claude/Codex/OpenCode transcripts into scorecards, limits, findings, and per-session detail; Observability follows live and historical agent activity with masked evidence; System reports what the stack occupies on the machine (the same data as `ak system`). Deep links are hierarchical (`#about`, `#overview/summary`, `#usage/`, `#observability/live`, `#system/storage`). No dashboard action mutates agents or repositories; ruflo and agentic-qe stores are opt-in through repeatable `--live-source 'surface=path'`. The page is self-contained, offline-first, and protected by a per-session token. Full navigation, keyboard behavior, and per-view semantics: [Dashboard guide](docs/DASHBOARD.md) and [Observability guide](docs/OBSERVABILITY.md). **Auto-opens your browser** (`--no-open` for headless/SSH); `--port N` changes the port. Stop with Ctrl-C. (Also available as `ak x dashboard`.) | +| **sync** | The one convergence verb: upgrades first when a new release exists, then re-heals everything an upgrade wipes, then re-checks and reports. Included in that heal: it **installs any enabled frontier host** (claude/codex/opencode) that's entirely absent — never touching an external (mise/brew/native) install — and **re-applies provider wiring** (the `ENABLE_*` host env, OpenCode's native configuration, the aqe fallback chain, and ruflo API providers) whenever it has drifted — and, on a dual-host project, **seeds/heals the Claude/Codex default routing policy** (materializing eligible routes into agentic-qe's `agentOverrides`, e.g. after an aqe upgrade first makes it eligible). It also **installs/repins the standalone `agentdb` CLI** to ruflo's bundled version (keeping the shared cognitive store coherent) and appends a **health-history snapshot** so `status` can flag regressions across syncs. It also **re-runs the RuvNet Brain installer** to pull the latest release when the on-disk KB has drifted (or installs it if absent, when enabled). It also **self-updates the kit**: when a newer `@pacphi/agentic-kit` exists it installs it as the *last* step (the new code applies from the next `ak` run, never mid-sync). Prerelease installs (`4.0.0-alpha.*`) track the `next` npm dist-tag as well as `latest`, so alphas see their successors; stable installs only ever follow `latest`. `--no-upgrade` skips the self-update along with the package upgrades. Model refresh/diff/plan findings remain advisory: sync never contacts a model catalogue or applies a model plan. | +| **dashboard** | Opens an observation-only local web dashboard (`127.0.0.1:7431`, localhost-only, never detaches) with five primary areas: **About · Overview · Usage · Observability · System**. About is a plain-words directory of everything the kit installed and why (the same content as `ak about`); Overview covers readiness, hosts & routing, providers, runtime, and machine-wide intelligence; Usage turns local Claude/Codex/OpenCode transcripts into scorecards, limits, findings, per-session detail, and privacy-projected model lifecycle evidence. The Models inventory is lazy, paged, filterable, sortable, and internally scrollable; source-proven public catalogue names remain readable while private deployment identity stays keyed. Observability follows live and historical agent activity with masked evidence; System reports what the stack occupies on the machine (the same data as `ak system`). Deep links are hierarchical (`#about`, `#overview/summary`, `#usage/models`, `#usage/`, `#observability/live`, `#system/storage`). No dashboard action mutates agents or repositories; ruflo and agentic-qe stores are opt-in through repeatable `--live-source 'surface=path'`. The page is self-contained, offline-first, and protected by a per-session token. Full navigation, keyboard behavior, and per-view semantics: [Dashboard guide](docs/DASHBOARD.md) and [Observability guide](docs/OBSERVABILITY.md). **Auto-opens your browser** (`--no-open` for headless/SSH); `--port N` changes the port. Stop with Ctrl-C. (Also available as `ak x dashboard`.) | | **usage** | Reads provider-account analytics from local cache (`ak usage status`) or performs one explicit OpenRouter management-API refresh (`ak usage refresh openrouter`). Refresh requires `OPENROUTER_MANAGEMENT_KEY`, writes a credential-free mode-`0600` cache, and discards endpoint/user/key/session identifiers. `status` and dashboard reads make no network request. OpenRouter account rows have no grounded host/session/project correlation and are never merged into transcript totals. | +| **models** | Builds a private, host-scoped model inventory from Claude, Codex, OpenCode, Ollama, bounded local usage evidence, and a dated bundled record of Anthropic's public model/lifecycle facts. `status`, `diff`, `explain`, and `plan` are cache-only and read-only; `refresh --online` is the sole online-catalogue boundary. Public facts never imply account or OpenRouter routability. Swap plans enumerate routes plus Agentic QE/Ruflo consumers and print a copyable canonical action without executing it. The CLI exposes exact local evidence deliberately; the Dashboard exposes source-proven public catalogue identity and pseudonymizes private identifiers. See [Model lifecycle intelligence](docs/MODELS.md). | | **admin** | Opens the **maintainer admin** (`127.0.0.1:7432`, localhost-only, foreground) — the project-telemetry sibling of `dashboard`, with the same dark/light visual theme and persisted theme preference: unique repo visitors and cloners (GitHub traffic API, needs a push-access token via `GITHUB_TOKEN`/`GH_TOKEN`/`gh auth token` — panels degrade honestly without one), contributors and watchers, npm download momentum (last 7d vs prior 7d, sparklines — shown as trend only, never an absolute reach number, since mirrors/CI inflate the raw count), latest CI run status and open Dependabot alerts, a **"since you last looked"** delta strip over a local baseline, open issues/PRs from others (oldest first), and external humans ranked by recency (bots excluded). Access is gated by a **per-session token** carried in the URL fragment and sent header-only; the page makes **zero external fetches** (the server proxies GitHub/npm; your credential never reaches the page or the payload). Where `dashboard` is offline-first, `admin` does deliberate GitHub/npm egress — that contract split is why they're siblings, not tabs. `--port N`, `--no-open`; Ctrl-C stops. (Also available as `ak x admin`.) | | **about** | A plain-words directory of every component the kit installs and configures — one entry per component: what it is, what it does for you, where to read more, and an honest state chip read from the same detection `ak status` uses (the prose is authored with the release; the chip is the only runtime fact). `ak about [entry-id]` opens one entry; `--category` narrows to `hosts`, `engine-memory`, `quality`, `safety`, `knowledge`, `kit`, or `configured`; `--no-detect` skips state resolution for an instant editorial read; `--json` emits the directory with resolved chips. The dashboard's About area renders this identical directory. | | **system** | What the stack occupies on your machine. The default read is the cheap tier: the live agent-process census, the files growing fastest between scans, and the last deep scan's figures carried forward with their date. `--deep` re-walks install trees, storage, the cross-host catalog, and every discovered project (minutes on a large machine), then persists the result; `--json` emits the same snapshot payload `/api/system` serves. The dashboard's System area renders this same data across its Summary, Advisory, Sessions, Storage, Runtime, Catalog, and Projects views. | diff --git a/bin/agentic-kit.mjs b/bin/agentic-kit.mjs index 737126d..eed88bc 100755 --- a/bin/agentic-kit.mjs +++ b/bin/agentic-kit.mjs @@ -22,6 +22,7 @@ const PORCELAIN = Object.assign(Object.create(null), { dashboard: () => import('../src/commands/x/dashboard.mjs'), admin: () => import('../src/commands/x/admin.mjs'), usage: () => import('../src/commands/usage.mjs'), + models: () => import('../src/commands/models.mjs'), system: () => import('../src/commands/system.mjs'), about: () => import('../src/commands/about.mjs'), run: () => import('../src/commands/run.mjs'), @@ -52,6 +53,7 @@ Usage (ak = alias of agentic-kit): ak dashboard open the local web dashboard (localhost; auto-opens browser) [--port N] [--no-open] ak admin maintainer-only telemetry admin (localhost; GitHub/npm egress) [--port N] [--no-open] ak usage inspect/refresh offline provider analytics [status|refresh openrouter] + ak models inspect/refresh model lifecycle evidence [status|refresh|diff|explain|plan] ak system what this stack occupies on your machine [--deep] [--json] ak about what agentic-kit installs and configures, and why [--category N] ak run execute a host-neutral activity pipeline [template "task"] [--dry-run] @@ -177,7 +179,7 @@ async function main() { // setup and host own complete mutation/reporting flows. Running the generic // nudge after a declined trust preflight could write version-cache state and // violate their "before any changes" boundary. - if (!values.json && !values['dry-run'] && !['sync', 'usage', 'setup', 'host', 'ruflo-mcp'].includes(cmd)) { + if (!values.json && !values['dry-run'] && !['sync', 'usage', 'models', 'setup', 'host', 'ruflo-mcp'].includes(cmd)) { try { const { driftReport } = await import('../src/lib/versions.mjs'); for (const r of await driftReport()) { diff --git a/docs/DASHBOARD.md b/docs/DASHBOARD.md index 0d14701..6184746 100644 --- a/docs/DASHBOARD.md +++ b/docs/DASHBOARD.md @@ -41,6 +41,7 @@ permanent. | Usage | Limits | `#usage/limits` | Provider limits | Current provider windows, reset timing, and available capacity | | Usage | Findings | `#usage/findings` | Usage findings | Actionable anomalies, efficiency opportunities, and evidence-backed recommendations | | Usage | Sessions | `#usage/sessions` | Session usage | Retained sessions grouped by project, category, duration, tokens, and cost | +| Usage | Models | `#usage/models` | Model lifecycle | Host inventory, lifecycle changes, consumers, swap impact, and evidence sources | | Usage | Transcript | `#usage/transcript` | Transcript detail | The selected session's locally retained, server-masked evidence | | Observability | Live | `#observability/live` | Observability · Live | Projects and roots with current presence or fresh meaningful activity | | Observability | History | `#observability/history` | Observability · History | Retained roots that are not currently Live | @@ -157,7 +158,7 @@ Each count carries the sentence explaining what it counted; on Intelligence it i ## Usage -Usage loads lazily when first opened. Scorecard, Limits, Findings, Sessions, and Transcript share the +Usage loads lazily when first opened. Scorecard, Limits, Findings, Sessions, Models, and Transcript share the same secondary rail; the 7/14/30-day filters remain aligned to its right. ### Reading a session row @@ -190,6 +191,61 @@ control, and the original masked value never reaches the browser. See [ADR-0009](adr/0009-usage-scorecard-local-transcript-analytics.md) for the full evidence and pricing contract. +### Models + +Models is a cache-only lifecycle evidence view backed by `/api/models`. It never performs discovery +or invokes a model; `ak models refresh` owns collection. The first panels answer two different +questions: **Observed in this window** aggregates actual retained transcript evidence for the same +7/14/30-day selector used by Usage, while **Your routes** names configured primary and fallback +models and joins their actual last-use timestamp when one exists in that window. GPT, Claude, +OpenCode, or local models therefore do not disappear merely because they are not pinned to a route. +The collapsed catalogue separately answers what is installed or discoverable; catalogue presence +does not claim use. + +Claude refresh also reads a dated first-party record bundled with Agentic Kit from Anthropic's +public model overview and deprecation tables. It can establish published model identity, +specifications, lifecycle, and pricing without an API key. It cannot establish access for a Claude +Code plan, Anthropic API account, Bedrock/Vertex deployment, or OpenRouter route. Upgrade Agentic Kit +to receive a newer public record, then run `ak models refresh --host claude`. + +The view fetches a compact, windowed summary first, then a 50-row relevant inventory page. Search +and facet-counted filters for host, model provider, relevance, lifecycle, and evidence request fresh +bounded pages; controls with fewer than two meaningful choices are suppressed. **Load 50 more** +appends the next page. Every meaningful column header is a keyboard-operable sort button +that toggles ascending and descending order, exposes `aria-sort`, and leaves unknown values last. +Later pages carry the privacy-projected snapshot id; if refresh replaces the snapshot, the browser +reloads page one instead of mixing two inventories. A failed later page preserves the rows already +shown and leaves a focused retry control. + +The inventory region is height-bounded and scrolls internally in both axes. Change history is also a +bounded, internally scrollable table; it names the exact model, provider, host, plain-language change, +evidence status, and detection time instead of exposing an opaque identity join. The route-consumer +panel remains bounded beside it. These lifecycle panels are structurally owned by Usage → Models and +are never rendered in Scorecard, Limits, Findings, Sessions, or Transcript. +Its header stays sticky. The region is labelled and keyboard-focusable, with a caption, column +scopes, `aria-busy` loading state, result and load +announcements, visible focus, and an explicit load control in addition to lazy fetching. + +Source-proven public catalogue records show readable names and trusted source links while private +deployments remain keyed pseudonyms. OpenCode rows need an exact Models.dev join from explicit +online refresh; selector syntax or verbose metadata alone never makes a row public. Host, serving +provider, publisher, model selector, catalogue source, and entitlement remain independent. Each +state and lifecycle value expands to its source, +class, capture time, freshness, completeness, scope, or a field-specific explanation of missing +evidence. The discovery column is labelled **Catalogued**; details separately state account access, +local routability, and the next step. To establish routability, configure the exact host/provider/ +model path, authenticate the serving provider, complete one successful invocation, and refresh. +Model-specific foreground/background pairs meet WCAG AA in both themes, and the same table +remains operable at narrow widths. + +The Overview Model lifecycle summary links to `#usage/models`. The view has no mutation control. +It points to the read-only `ak models plan` command, which can emit a copyable canonical +`ak host pick` action but cannot execute it. See [Model lifecycle intelligence](MODELS.md). + +The lifecycle payload is separate from the Usage session/transcript and Observability live/history +payloads. Public catalogue enrichment cannot rename, re-price, add, or remove a retained session or +transcript model record. + The Scorecard view also shows a host-neutral **telemetry coverage** panel for Claude, Codex transcript evidence, and OpenCode. It reports parsed units and observed prompt/response totals, plus capability states (`supported`, `unsupported`, or `unavailable`). A readable source with no observed @@ -339,12 +395,22 @@ Canonical hashes make views linkable without putting the dashboard token in the string. The launch token initially arrives in the URL fragment and is then stored locally for authenticated -API requests. The dashboard remains localhost-only and offline-first. Usage, Observability, and +API requests. The dashboard remains localhost-only and offline-first. Usage, Models, Observability, and System may show sensitive local project, transcript, or filesystem-path information; use them only where that local information may be viewed. System deliberately shows absolute paths — a storage breakdown that hides where the bytes live answers nothing — behind the same token-gated loopback delivery as every other route. +Models adds a second privacy boundary. The explicit local CLI can show exact model evidence, while +`/api/models` requires the already-existing private model scope key and returns +`privacy.projection: owner-visible-v2`. The loopback, token-gated operator view shows bounded exact +model names, selectors, and recorded providers; source-proven catalogue identity may additionally +show a publisher and allowlisted HTTPS links. Credentials, endpoints, scopes, digests, aliases, +binding/evidence/history identifiers, and arbitrary configuration remain keyed pseudonyms. Filtering +and sorting run only after that projection. Controlled built-in source metadata and diagnostic codes +remain named; unknown source metadata is pseudonymized. Missing key material returns a generic 503, +and opening the Dashboard never creates the key. + What System reads is a short, fixed list: directory entries and file `stat` results; your `.git/config` origin remote (so a project can link to its repository page — the kit never fetches it, you click it); a linked worktree's `gitdir` pointer; the `cwd` **field** recorded at the top diff --git a/docs/HOST-SUPPORT.md b/docs/HOST-SUPPORT.md index efa12ba..0beff94 100644 --- a/docs/HOST-SUPPORT.md +++ b/docs/HOST-SUPPORT.md @@ -300,6 +300,10 @@ self-declaring them. - Use `ak status` for installed/wired truth and `ak run --dry-run` for the exact worker plan. +- Use `ak models refresh` to capture host-scoped model evidence, then + `ak models status|diff|explain|plan` for offline lifecycle and swap analysis. Claude, Codex, + OpenCode, and Ollama have registry-selected explicit source adapters; an external host receives no + inferred catalogue capability without an admitted descriptor and matching adapter. - Use Claude or Codex as primary. Choose based on which should lead the mirrored activity defaults, not on MCP availability alone. - Route OpenCode explicitly for bounded work whose `provider/model` and repository diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index 088e6c8..9dba429 100644 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -216,6 +216,7 @@ against Node and npm. They are not the supported machine-management contract. | Ruflo, AQE, AgentDB | None | Installed/repaired in the active npm global prefix | Project assets initialized from those versions | | Claude/Codex/OpenCode CLI | None | Missing enabled hosts may be installed globally; external installs are reused | Host-specific project wiring may be generated | | `~/.config/agentic-kit/kit.json` | None | Created/updated for the current OS user | Choices are read and project routing may be materialized | +| Model inventory and private scope key | None | Created only by an explicit `ak models refresh` | Scope is stored as a keyed non-identifying fingerprint; the owner-only cache can retain exact model ids for explicit CLI evidence, while the Dashboard receives keyed pseudonyms | | Claude/Codex/OpenCode user guidance | None | Managed sentinel blocks reconciled | Project guidance/assets may be created or refreshed | | Ruflo MCP registration | None | Offered at user scope | Conflicting project-local Ruflo registration is removed | | RuvNet Brain | None | Shared current-user KB/plugin installation; approximately 2 GB | No per-project Brain copy | @@ -282,6 +283,7 @@ existing project, commit or back up first and review the exact mutation contract | Remove a local dependency | `npm uninstall @pacphi/agentic-kit` | Current package/workspace | | Remove the global runner only | `npm uninstall -g @pacphi/agentic-kit` | Active npm prefix; leaves setup-created state | | Remove managed integration state | `ak uninstall` | User/project state selected by its flags | +| Remove model inventory and scope key | `ak uninstall --purge` | Current-user Agentic Kit configuration | | Preview teardown | `ak uninstall --dry-run` | No changes | Removing the npm package is not equivalent to `ak uninstall`. The package manager diff --git a/docs/MODELS.md b/docs/MODELS.md new file mode 100644 index 0000000..8fdc36a --- /dev/null +++ b/docs/MODELS.md @@ -0,0 +1,266 @@ +# Model lifecycle intelligence + +Model lifecycle intelligence answers three questions without changing your routing: + +1. Which concrete models are configured, observed, or discoverable for each host? +2. What changed between trustworthy, same-scope snapshots? +3. Which routes and consumers would a model swap affect? + +The inventory is evidence, not configuration. `kit.json.routing` and `ak host pick` remain the +canonical routing surfaces. + +## Quick start + +```bash +ak models refresh +# After upgrading Agentic Kit, refresh only Claude evidence if that is all you need: +ak models refresh --host claude +ak models status +ak models diff +ak models explain codex:gpt-5.6-terra +ak models plan --activity testing --from codex:gpt-5.4 --to codex:gpt-5.6-terra +``` + +Only `refresh` contacts model sources or writes a snapshot. `status`, `diff`, `explain`, `plan`, +the `ak status` models row, and the Dashboard's **Usage → Models** view read the local cache. + +Use `--json` with any command for the normalized evidence contract. Use `--host HOST` to select +one source, `--all` for Claude, Codex, OpenCode, and Ollama, and `refresh --online` to permit +OpenCode's explicit catalogue refresh. Ordinary refresh does not opt into an online catalogue +request. Claude refresh includes Agentic Kit's dated, bundled transcription of Anthropic's public +model and deprecation records; it does not require an Anthropic API key or make a network request. + +## What each state means + +The states are deliberately independent: + +| State | Meaning | +|---|---| +| Configured | A local route or host setting names the model. | +| Effective | Host precedence or alias resolution currently selects it. | +| Observed | Structured local usage evidence recorded it. | +| Discoverable | The current host/provider catalogue included it. | +| Entitled | The active account or profile is proven able to use it. | +| Policy allowed | Managed or user policy permits it. | +| Routable | The complete host, provider, auth, and capability path is proven. | +| Lifecycle | Active, preview, hidden, deprecated, retiring, removed, or unknown. | +| Recommended | A named first-party or evidence-backed source recommends it. | + +`unknown` is not `false`. For example, a Codex cache entry can prove discovery without proving +account entitlement; a configured Claude alias can be effective without proving catalogue +completeness. + +## Sources and scope + +The source adapters are: + +- Claude user settings, platform managed policy, model aliases, a model-only environment allowlist, + and a dated first-party public record transcribed from Anthropic's model overview and model + deprecation tables; +- the Codex model cache plus top-level `config.toml` model selection; +- OpenCode's project/provider-scoped verbose `models` output plus its resolved `debug config` view; +- the local Ollama `/api/tags`, bounded `/api/show`, and `/api/ps` catalogue/runtime facts; and +- sanitized model ids from the existing local usage index. + +The collector selects these adapters from the immutable model-discovery descriptor registry. Each +source record retains its owner and owner type, file/command/HTTP/index transport, `never`/`local`/ +`explicit` network policy, local/online collection mode, schema, freshness, completeness, and scope. +A descriptor authorizes only its matching built-in parser; it cannot supply executable code or give +an external host an inferred catalogue capability. + +Claude managed policy is read from the platform path when present: + +- macOS: `/Library/Application Support/ClaudeCode/managed-settings.json`; +- Linux: `/etc/claude-code/managed-settings.json`; and +- Windows: `C:\ProgramData\ClaudeCode\managed-settings.json`. + +The process environment contributes only `ANTHROPIC_MODEL` and the three +`ANTHROPIC_DEFAULT_{SONNET,OPUS,HAIKU}_MODEL` alias targets. Other environment values, including +credentials and endpoints, do not enter collection. + +Native inputs are untrusted. Parsers cap bytes and row counts, guard schemas and enums, and invoke +commands with literal argument arrays and no shell. Interactive picker scraping and inference +probes are excluded. + +The lookup contracts were checked against current first-party documentation in August 2026: + +- Anthropic's [Models overview](https://platform.claude.com/docs/en/about-claude/models/overview) + establishes current API ids, aliases, published availability, input/output modalities, context + and output limits, thinking support, and list prices. Its + [Model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations) page + establishes lifecycle terms, retirement commitments or dates, and recommended replacements. + Lifecycle dates apply to Anthropic-operated platforms; Amazon Bedrock and Google Cloud can use + different schedules. Agentic Kit dates this bundled source and marks it stale after 90 days, so + updating Agentic Kit—not repeatedly refreshing the same installed version—is how public facts are + renewed. +- Anthropic's authenticated [List Models API](https://platform.claude.com/docs/en/api/models/list) + can identify models available to one Anthropic API account and returns limits and capabilities. + Agentic Kit does not silently call it: an API-key-scoped result cannot prove Claude Code plan + access or partner/OpenRouter availability, and credentials never enter the inventory. +- Claude accepts aliases or a full model name through its documented model selection surfaces. Its + `[1m]` suffix selects an extended-context variant and is stripped before Claude Code sends the + base model id to the provider + ([Claude Code model configuration](https://code.claude.com/docs/en/model-config)). Agentic Kit + therefore retains the full selector as configuration/alias evidence while joining its facts to + the base model identity. +- OpenCode exposes `opencode models [provider] --verbose`; `--refresh` refreshes its + Models.dev-backed cache ([OpenCode models](https://opencode.ai/docs/models/)). Agentic Kit runs + refresh separately from the verbose list so OpenCode's success banner cannot degrade the list. + It reads `opencode debug config` for the resolved global/project/JSONC configuration, including + root, agent, and command model overrides. Model references accept bounded non-control + `provider/model`, an optional `#variant`, or the expanded selector object; current OpenRouter + `~publisher/model` values are valid ids, not parser errors. +- Models.dev publishes provider-independent human names, labs, serving providers, limits, + capabilities, pricing, release dates, and explicit weight links through JSON endpoints + ([Models.dev](https://models.dev/)). Catalogue presence proves discovery, not account access. +- Ollama refresh prefers the loopback [list models](https://docs.ollama.com/api/tags), + [show model details](https://docs.ollama.com/api-reference/show-model-details), and + [list running models](https://docs.ollama.com/api/ps) APIs. It retains bounded installed-model + metadata and loaded memory/context/expiry facts, never raw templates, model files, or full license + bodies. `ollama ls` is a partial compatibility fallback when the local API is unavailable. + Installed or loaded remains distinct from successful observed inference. +- User-facing GPT-5.6 examples use current official ids and reasoning levels + ([OpenAI model catalogue](https://developers.openai.com/api/docs/models)). Codex's local cache is + still treated as a guarded native schema, not as a public API contract. + +Host, provider, concrete model id, a non-identifying scope id, and an evidenced mutable local digest +form model identity. Reasoning effort, context, service tier, and similar execution settings remain +variants rather than unrelated base identities. Account, profile, and project scope values are +HMAC-fingerprinted with a private per-install key; raw scope values do not enter the snapshot. The +private cache and key live under the Agentic Kit configuration directory with owner-only +permissions. + +A host-owned first-party catalogue can omit provider identity even when usage evidence names it. +For the same host, model id, scope, and digest, Agentic Kit joins provider-neutral Claude/Codex +catalogue facts only when an independent record establishes the expected `anthropic`/`openai` +provider path. This prevents duplicate title-case and lowercase rows without inferring that a +custom gateway or other provider is the same deployment; those paths remain separate records. + +Known fields have field-specific evidence references. A configured route can prove `configured`, a +structured successful invocation can prove the exact observed path's `observed`, `entitled`, +`policyAllowed`, and `routable` facts at capture time, and a catalogue can prove `discoverable`. +None of those facts silently strengthens another path or establishes catalogue completeness. + +For a current public Claude model, the bundled first-party record can therefore make publication, +lifecycle, recommended status, context/output limits, supported modalities/tools/thinking, and a +dated API list price known. `configured`, `entitled`, `policyAllowed`, and `routable` remain unknown +until local evidence establishes each one. To establish the path in Agentic Kit, configure the exact +model and serving provider on an intended route, authenticate that provider, complete one successful +invocation, and run `ak models refresh` again. A Claude model published by Anthropic is not thereby +proved routable through OpenRouter, Bedrock, Vertex, or a particular Claude subscription. + +## Snapshots and diffs + +Snapshots retain at most 32 captures per scope and expire after 90 days. A complete same-scope +snapshot may advance the comparison baseline. Partial, stale, unavailable, or unsupported-schema +sources remain visible but cannot displace that baseline. + +Without an authoritative first-party removal signal, a missing model must be absent from two +consecutive complete same-scope snapshots before it becomes a removal. Partial evidence never +creates removals. Cross-scope comparison is refused instead of appearing as mass churn. + +Diffs report model additions/removals, alias-target continuity and changes, lifecycle transitions, +visibility, capabilities, local digest, reasoning, context, other variants, optional pricing, and +typed-edge changes. Every normalized field carries evidence source, class, capture time, freshness, +completeness, and scope. A digest change stays one comparable local model lineage rather than +appearing as an unrelated add/remove pair. Lifecycle, pricing, and edge comparisons ignore changing +evidence-reference ids and capture timestamps, so an identical semantic fact does not churn. + +## Read-only swap plans + +`ak models plan` checks mechanical compatibility only. `false` discovery is a blocker. Unknown +discovery is also a blocker unless structured evidence proves that exact model path was observed +successfully; in that case the plan keeps an explicit catalogue-unknown warning. Entitlement, +policy, routability, and required capability facts must still be established. The plan does not +claim quality equivalence, lower cost, or equal performance unless a separate evidence source +establishes that. + +Selectors use `HOST:MODEL`; OpenCode's provider-qualified form is +`opencode:PROVIDER/MODEL`, for example `opencode:openrouter/anthropic/claude-sonnet-4.5`. + +A successful plan lists affected canonical routes, escalation rungs, integration bindings, and +independently sourced Agentic QE/Ruflo consumers. It prints a copyable `ak host pick --route ...` +command but does not execute it. There is no `ak models apply`. + +The same snapshot can produce a pure Route Intelligence feed: mechanically eligible candidates, +current optional pricing metadata, and lifecycle/alias/capability/variant invalidations whose audit +history must be retained. The feed explicitly sets quality and economics claims to false; +[Issue #109](https://github.com/pacphi/agentic-kit/issues/109) must establish any evidence-backed +equivalence or cost recommendation. + +The `ak status` row may point to `ak models refresh` or `ak models diff`, but `ak sync` excludes +model actions from its executable convergence plan. Catalogue refresh and route mutation remain +explicit operator boundaries. + +## Dashboard + +Open `ak dashboard`, then choose **Usage → Models**. The view includes: + +- attention items for degraded sources, cited lifecycle migrations, alias changes, and drift; +- a semantic host/model table with every independent state; +- bounded, scrollable same-scope change history with the exact model, provider, host, plain-language + change, evidence status, and detection time; +- configured and reported consumers; +- read-only swap-impact guidance; and +- evidence source status and capture time. + +The Models view loads only when it opens. It leads with **Your routes** — the configured primary and +fallback paths — and keeps the catalogue explorer collapsed until requested. The explorer first +fetches the small summary and then the first 50 relevant rows; catalogue-only rows load in explicit +pages. Search, access host, model provider, view, lifecycle, and evidence filters reset +pagination. Every sortable catalogue column keeps unknown values last, and its bounded table +scrolls internally with a sticky header and an explicit **Load 50 more** control. + +A lifecycle migration becomes an alert only when the model is configured/effective or observed +locally and the snapshot includes a direct first-party withdrawal-notice URL with matching +first-party lifecycle evidence. Provider history remains available in the full catalogue but does +not flood the in-use view or create local migration warnings. A local host cache's `upgrade` hint or +a preferred successor remains a discovery detail, never a retirement claim or route rewrite. + +Source-proven public catalogue records show a human name, host, publisher when proven, serving +provider when known, exact public selector, and trusted documentation/catalogue links. The bundled +Anthropic record supplies first-party public identity, specifications, lifecycle scope, and +publication status for its exact documented Claude ids and aliases. Codex cache identity can +qualify locally. OpenCode identity qualifies +only after `refresh --online` exact-joins the selector to the independently fetched, size-bounded +Models.dev catalogue; provider syntax or verbose metadata alone is not proof. The Models.dev source +link remains generic unless that source supplies an exact canonical page. Hugging Face and Ollama +links appear only when source metadata verifies the exact repository/library identity. Custom +providers, private deployments, local tags, and observed-only ids appear with their exact bounded +model selector, display name, and independently recorded provider in the owner-only Dashboard. +Configured variants, digests, aliases, binding ids, scopes, snapshot ids, evidence ids, credentials, +and endpoints remain hidden transport joins only. Change history uses the same owner-visible exact +model identity as the route and catalogue views, but never exposes the underlying identity join. The +browser never derives a provider or publisher from a name and never invents an external link. + +An `unknown` cell explains which evidence is absent. Model details separately name published or +discovered status, account access, local routability, and the operator's next evidence step. The +table labels the discovery dimension **Catalogued**, not **Available**, so provider publication or +local discovery cannot be mistaken for account entitlement. A local refresh now resolves OpenCode's +effective configuration, removing unknowns caused only by ignored global, JSONC, agent, or command +layers. Discovery still does not establish entitlement; configuration does not establish successful +use; and a model id never establishes the serving provider. Catalog Explorer model details show an +identical source, class, capture time, freshness, and completeness summary once even when that source +establishes several fields. The independent field evidence references remain intact and each state +cell still discloses its own evidence. A missing or invalid key fails the API +closed, an ordinary Dashboard read never creates one, and all reads retain loopback, session-token, +CSP/origin, `no-store`, and no-egress protections. + +## Privacy and recovery + +Snapshots contain no credentials, prompts, reasoning traces, transcript text, private endpoints, +or raw account/profile/project identifiers. Because the owner-only cache supports explicit local +CLI evidence, it can retain bounded exact configured model or deployment identifiers. The +Dashboard receives bounded exact model identity for its token-gated loopback owner view; credentials, +endpoints, scope, digests, aliases, and evidence identifiers never cross that boundary. A corrupt +cache degrades to an empty readable store; +run `ak models refresh` to rebuild it. A source schema failure is isolated to that source and shown +as `unsupported-schema` rather than converted to an empty catalogue. + +This enrichment never rewrites Historical Usage sessions, prices, transcript model attribution, or +Observability live/history records. Those contexts remain source owners; Model lifecycle consumes +only their bounded aggregate host/provider/model facts and publishes a separate cache/read model. + +For common failures, see [Troubleshooting](TROUBLESHOOTING.md). The domain and decision records are +[Model lifecycle intelligence](ddd/model-lifecycle-intelligence.md) and +[ADR-0032](adr/0032-model-lifecycle-intelligence.md). diff --git a/docs/OBSERVABILITY.md b/docs/OBSERVABILITY.md index 404fbd0..4c817d6 100644 --- a/docs/OBSERVABILITY.md +++ b/docs/OBSERVABILITY.md @@ -34,6 +34,13 @@ tab closes that browser's event stream. After the last snapshot/SSE client leaves, collectors stop after 30 seconds by default and restart on the next request. Stopping the dashboard closes the live service and all clients. +Model lifecycle is a separate read model under **Usage → Models**. It may consume bounded model ids +already derived by the historical usage index, but it never consumes live transcript content +or changes Observability state. Public catalogue enrichment cannot rename, add, remove, or alter +model attribution in either the Live or History view. Catalogue collection remains explicit through +`ak models refresh`; +see [Model lifecycle intelligence](MODELS.md). + ## Read Observability in 30 seconds Observability has two navigation modes: diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md index f8733e5..f82bd30 100644 --- a/docs/PROVIDERS.md +++ b/docs/PROVIDERS.md @@ -38,6 +38,21 @@ read only that cache. Because the management response has no local host/session/ correlation key, its rows appear only as provider account analytics and never alter transcript totals or prove which host executed a request. +**Model lifecycle evidence is separate from both.** `ak models refresh` inventories host-scoped +configuration, catalogues, and sanitized observed model ids. `ak models status|diff|explain|plan` +are cache-only. Discovery does not prove quality or mutate provider/routing configuration, and +`refresh --online` is the only permitted online-catalogue boundary. See [Model lifecycle +intelligence](MODELS.md). + +Claude refresh includes a network-silent, dated Anthropic public record. It proves Anthropic's +published model facts, not that OpenRouter or another serving provider vends that model to this +account. OpenRouter routability still requires an exact OpenRouter/OpenCode selector plus local +configuration, authentication, policy, and successful-use evidence for that path. + +The model inventory may feed mechanically eligible candidates and stale-evidence markers to Route +Intelligence, but it explicitly makes no quality or economic claim. Status can recommend an +explicit model command; `ak sync` never executes model refresh or model-plan actions. + This capability model is [ADR-0016](adr/0016-capability-driven-integration-adapters.md) (Accepted); the controls below implement it. `ak host` owns execution-host lifecycle and selection (`status`, `pick`, `refresh`, @@ -386,12 +401,11 @@ Defaults (all overridable; your edits are marked `custom` and never re-seeded): *(packaging & release are `ak`-added — ruflo ships templates for feature/security/refactor only.)* -**Retired codex models.** `gpt-5.4` and `gpt-5.4-mini` retire from Codex on **2026-08-31**, and -`gpt-5.3-codex` is already withdrawn for ChatGPT sign-in -([Codex models](https://developers.openai.com/codex/models)) — which is why the execution defaults -above moved to the 5.6 line. `ak` substitutes a retired model at read time, so no run dispatches to -one even if your `kit.json` still names it, and `ak sync` rewrites `seeded` routes that do. A `user` -pin is reported but never rewritten on disk (see +**Retired Codex models.** `ak` has no automatic Codex retirement substitutions as of 2026-08-25. +The current [OpenAI API model catalog](https://developers.openai.com/api/docs/models/all) still lists +GPT-5.4 and GPT-5.4 mini, and no first-party withdrawal notice supports the former automatic +replacement claims. `ak` only adds a retirement rule when it can cite the host's direct notice; a +newer default remains a recommendation, not a route rewrite (see [ADR-0003](adr/0003-auto-seed-dual-host-provenance.md)). `claude-opus-4-8` is **not** retired — it carries no deprecation notice and stays pinnable. It is diff --git a/docs/TRANSCRIPTS.md b/docs/TRANSCRIPTS.md index 7db66f8..48eef00 100644 --- a/docs/TRANSCRIPTS.md +++ b/docs/TRANSCRIPTS.md @@ -240,6 +240,16 @@ carry — `prompts`, `responses`, `exceptions`, `sidechain`, `threadSource`, `models`, `tools`, `skill`/`plugin`, worktree — plus a `cost` priced from the same per-model usage rows `aggregate()` uses. +Model lifecycle intelligence reuses only the aggregate session's execution host, independently +evidenced inference provider, and bounded model ids. It does not copy session ids, titles, prompts, +turns, tools, paths, or transcript text into the model snapshot. Observed use proves only that +the exact path was observed, entitled, policy-allowed, and routable at capture time; it does not +make the host catalogue complete, prove another path, or claim quality equivalence. When catalogue +discovery for that exact path is unknown, a mechanical plan may proceed only with an explicit +catalogue-unknown warning. +Public catalogue enrichment flows in the opposite direction only into the lifecycle read model: it +never renames a retained session model, changes historical token pricing, or rewrites a transcript. + ### 4.3 Mask, then truncate — both marked, differently Every turn body is passed through `maskSecrets` (`usage-index.mjs:208` — the diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 5a16c8a..8a3ddc2 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -47,6 +47,14 @@ ak sync # apply it | `status` shows `ruvnet-brain … not installed` | The RuvNet Brain (offline KB + `search_ruvnet` MCP) isn't on disk | `ak sync` (or `ak setup`) runs the installer; `npx ruvnet-brain --doctor` health-checks it | | A heal says `degraded` while the tool is still usable | The native repair failed and a fallback or older artifact remains available; exit status is authoritative | Use the reported repair command/error. The operation will not render green or advance a version stamp until a later repair exits successfully | | Usage suddenly shows no data for one host, or a lower total than expected | Any of the four local sources (Claude/Codex transcript roots, OpenCode's SQLite store, the Codex thread ledger) can go absent, busy, corrupt, or query-incompatible; none of these are collapsed into an ordinary empty result | Inspect the branded host-icon pills in the dashboard's tabbar (top of every view, right-aligned — or `sourceHealth` in usage-index JSON) — one pill per host; the Codex pill folds its transcript-root and thread-ledger statuses together (worse status leads, both shown in the status side's tooltip). A degraded OpenCode scan retains in-window last-good cached sessions; repair the named source before treating zero as observed truth | +| `ak status` says there is no model inventory | No explicit model refresh has completed on this machine | Run `ak models refresh`, then inspect `ak models status` or Dashboard **Usage → Models** | +| A model vanished but `ak models diff` does not call it removed | The source is partial/stale, the scope changed, or this is only the first complete absence | Repair the named source and refresh again in the same scope. Two consecutive complete absences are required unless a first-party source declares removal | +| One source says `unsupported-schema` | Its native cache/config/protocol no longer matches the bounded adapter contract | Upgrade Agentic Kit first; retain the degraded snapshot for evidence and do not treat the source as an empty catalogue | +| OpenCode inventory says `partial` | Resolved config was unavailable, Models.dev identity proof failed during online refresh, output exceeded its line/diagnostic bounds, or OpenCode emitted a malformed selector/metadata block | Upgrade Agentic Kit and run `ak models refresh` again; use `--online` when human OpenCode catalogue identity is needed. Current `~` and bounded custom selectors are supported. Inspect the diagnostic code in `ak models status --json`; never treat the partial list as a complete removal baseline | +| Public Claude rows still lack lifecycle, context, or capabilities | The installed Agentic Kit predates the bundled Anthropic record, the record is over 90 days old, or no Claude refresh has rebuilt the snapshot | Upgrade Agentic Kit, then run `ak models refresh --host claude`. Repeating refresh on an old install cannot update bundled facts. Anthropic-operated lifecycle dates do not establish Bedrock, Vertex, OpenRouter, Claude Code plan, or account-specific availability | +| `ak models plan` refuses a target that appears in a catalogue | Discovery alone does not prove entitlement, policy allowance, routability, or required capabilities | Run `ak models explain HOST:MODEL`; establish the named missing evidence or make the canonical route change manually with `ak host pick` after review | +| Models shows `unknown` instead of yes/no | No accepted source established that independent fact; public catalogues prove publication, not local access | Expand the cell or Details for its field-specific reason and next step. To establish local routability, configure the exact host/provider/model path, authenticate that serving provider, complete one successful invocation, then run `ak models refresh`. Do not infer OpenRouter or account access from Anthropic publication, successful use from configuration, serving provider from a model name, or quality from lifecycle metadata | +| Dashboard Models returns `model dashboard privacy key unavailable` | A cache exists but its private scope key is absent or invalid | Run an explicit `ak models refresh` to create or repair owner-only model state. Dashboard reads fail closed and never create the key | | Observability does not show a live host process | Runtime discovery uses the numeric UID running the dashboard and is macOS/Linux-only; `sudo`, a service account, Windows, a private container PID namespace, missing `ps`/`lsof`, or restricted `/proc` changes what is visible | Run `ak dashboard` as the same ordinary OS account as the host CLI. Do not use `sudo`; use retained History on Windows and inspect OS/container process permissions when runtime presence is degraded. If the UID matches and none of the above applies, set `AK_RUNTIME_DEBUG=1` for one reproduction — stage-level evidence (survey row count, host classification per PID, nested-child exclusions, cwd resolution) goes to `$XDG_STATE_HOME/agentic-kit/runtime-debug.log` (mode 0600, bounded at 64 KiB; `AK_RUNTIME_DEBUG_FILE` to redirect it), then unset debug | | Don't want the RuvNet Brain (the ~2 GB KB download) | It's on by default | `ak setup --no-ruvnet-brain`, or set `ruvnetBrain: false` in `~/.config/agentic-kit/kit.json` | | Don't want the security surface managed | Also on by default | `ak setup --no-security` (persists `security:false`; status shows an info row and sync stops healing it) | diff --git a/docs/adr/0003-auto-seed-dual-host-provenance.md b/docs/adr/0003-auto-seed-dual-host-provenance.md index 53bc3ec..df87acc 100644 --- a/docs/adr/0003-auto-seed-dual-host-provenance.md +++ b/docs/adr/0003-auto-seed-dual-host-provenance.md @@ -2,9 +2,10 @@ - **Status:** Amended by [ADR-0020](0020-ga-stable-surfaces.md) - **Date:** 2026-07-23 -- **Updated:** 2026-08-07 +- **Updated:** 2026-08-25 - **Update note:** Preserved subscription-safe seeding and provenance while moving intent to the - canonical routing envelope; separated model **retirement** from route **divergence** (2026-08-07). + canonical routing envelope; retirement now requires a direct first-party withdrawal notice, not a + cache upgrade hint or a preferred successor (2026-08-25). - **Deciders:** agentic-kit maintainers > **GA amendment:** subscription-safe seeding and user-intent preservation remain. The persisted @@ -44,8 +45,9 @@ Every `ActivityRoute` carries **provenance**: `source: 'default' | 'seeded' | 'u Provenance answers "may ak change this value?". It does **not** answer "does this value still work?", and one case needs both: a model the host has **withdrawn**. A route naming one is not a stale -preference, it is a scheduled hard failure — `gpt-5.4` and `gpt-5.4-mini` stop answering in Codex on -2026-08-31, and `gpt-5.3-codex` already has. +preference, it is a scheduled hard failure. As of 2026-08-25, no Codex model is in this category: +the OpenAI API catalog still publishes GPT-5.4 and GPT-5.4 mini, and no cited withdrawal notice +establishes the previously claimed dates. So retirement is separated from divergence, and the two are handled differently: diff --git a/docs/adr/0005-dashboard-in-page-routing-reveal.md b/docs/adr/0005-dashboard-in-page-routing-reveal.md index e817910..6922d6f 100644 --- a/docs/adr/0005-dashboard-in-page-routing-reveal.md +++ b/docs/adr/0005-dashboard-in-page-routing-reveal.md @@ -2,9 +2,12 @@ - **Status:** Implemented - **Date:** 2026-07-23 -- **Updated:** 2026-08-04 -- **Update note:** Kept the read-only routing reveal and canonical GA configuration while - consolidating the dashboard into three primary areas with one shared secondary navigation rail. +- **Updated:** 2026-08-25 +- **Update note:** Reconciled the implemented five-area shell introduced by ADR-0025 and ADR-0026. + ADR-0032 adds Models as a secondary Usage destination and a compact Overview summary while + preserving read-only, network-silent ordinary Dashboard reads. Model identifiers use a fail-closed + keyed Dashboard projection, and the semantic table has a labelled focusable scroll region plus + field-evidence disclosures. ADR-0032 release proof remains pending. - **Deciders:** agentic-kit maintainers > **GA amendment:** the read-only dashboard decision remains. References below to compatibility @@ -18,18 +21,22 @@ At the time of this decision, `ak dashboard` was a single-page, poll-every-5s, grid grouped by subsystem, a `#history` strip — all fed by shelling `ak status --json` (`src/lib/dashboard-server.mjs`). It is health/status oriented. -> **Current implementation note (2026-08-04):** the read-only and loopback boundaries remain. The -> page now has three primary areas—Overview, Usage, and Observability—and one fixed, left-aligned +> **Current implementation note (2026-08-25):** the read-only and loopback boundaries remain. The +> page now has five primary areas—About, Overview, Usage, Observability, and System—and one fixed, +> left-aligned > secondary rail. It also has user-configurable status polling, lazy Usage reads, and an SSE-driven > Observability view. Page, styles, browser client, Observability, and request/session security live > under `src/lib/dashboard/`; `dashboard-server.mjs` is the HTTP composition root. -**2026-08-04 information-architecture amendment:** Overview absorbs the former health-oriented +**2026-08-25 information-architecture amendment:** About supplies the component directory and +System supplies machine-footprint views under ADR-0026 and ADR-0025. Overview absorbs the former +health-oriented peer tabs as **Summary**, **Hosts & Routing**, **Providers**, **Runtime**, and **Intelligence**. Usage owns **Scorecard**, **Limits**, **Findings**, **Sessions**, and **Transcript**. Observability -owns **Live** and **History**. The secondary row remains in one stable location across all three -areas. Canonical hashes are `#overview/{view}`, `#usage/{view-or-session-id}`, and -`#observability/{live,history}`. Every destination has a visible heading and description. Primary +owns **Live** and **History**. System and About retain the secondary destinations documented by +their governing ADRs. The secondary row remains in one stable location across all five areas. +Canonical hashes are rooted at `#about`, `#overview`, `#usage`, `#observability`, and `#system`. +Every destination has a visible heading and description. Primary and secondary tab lists use roving focus: Left/Right activates the adjacent tab with wrapping, and Home/End activates the first/last tab. @@ -55,7 +62,7 @@ Surface routing via **in-page reveal**, not a new page or tab: - Preserves the single-page, health-first idiom; routing is an enhancement reached by an intuitive in-page link, not a replacement. -- Presents only three stable primary choices while keeping Overview's status domains one keyboard +- Preserves five stable primary choices while keeping Overview's status domains one keyboard action away in the shared secondary rail. - Provides durable, hierarchical deep-link vocabulary without adding routes, servers, or a second navigation component per area. @@ -67,7 +74,7 @@ Surface routing via **in-page reveal**, not a new page or tab: ## References - `src/lib/dashboard-server.mjs` (`renderPage`, `#cards`, `#history`/`renderHistory`, `PREF`, `shellOutStatus`) -- `src/lib/dashboard/page.mjs` and `src/lib/dashboard/client.mjs` (three-area shell, shared secondary +- `src/lib/dashboard/page.mjs` and `src/lib/dashboard/client.mjs` (five-area shell, shared secondary rail, canonical hashes, headings, and keyboard semantics) - [Dashboard user guide](../DASHBOARD.md) - Mockup: ak dashboard — Routing panel; ADR-0001, ADR-0003 diff --git a/docs/adr/0009-usage-scorecard-local-transcript-analytics.md b/docs/adr/0009-usage-scorecard-local-transcript-analytics.md index 33a9622..5ac488d 100644 --- a/docs/adr/0009-usage-scorecard-local-transcript-analytics.md +++ b/docs/adr/0009-usage-scorecard-local-transcript-analytics.md @@ -2,12 +2,17 @@ - **Status:** Implemented - **Date:** 2026-07-25 -- **Updated:** 2026-08-24 -- **Update note:** Issue #170 added backward-compatible parsing for legacy Codex messages and +- **Updated:** 2026-08-25 +- **Update note:** Reconciled Usage with the implemented five-area Dashboard. ADR-0032 adds a Models + destination that consumes bounded structured observed-model facts without moving transcript + indexing, session history, or usage aggregates out of this context; its release proof remains + pending. A successful observation establishes only that exact path's observed, entitlement, + policy, and routability facts; it never establishes catalogue completeness. + Issue #170 added backward-compatible parsing for legacy Codex messages and `item_completed` envelopes, bumped the derived-index schema to force reparse of stale zero-turn records, added Codex parse-yield diagnostics, and separated first-billed-day session counts from token-bearing active-day counts. The earlier OpenRouter account-analytics cache boundary for issue #59, - aligned Usage with the dashboard's shared three-area navigation, and documented independent + aligned Usage with the dashboard's shared navigation, and documented independent host, inference-provider, provenance, and model facts in session rows. ADR-0023 subsequently classified SQLite source failures and made transient OpenCode failures preserve last-good records with explicit degraded source health instead of becoming observed zero usage; the Usage UI now @@ -77,22 +82,26 @@ Three properties of the data force most of the design: ADR-0007 split `admin` from `dashboard` along **network egress**: `dashboard` promises silence, `admin` promises reach. Usage analytics reads **local files only** and makes **zero network calls**. It therefore sits squarely inside the dashboard's existing offline-first contract and ships as one -of the dashboard's three primary areas, not a new server. +of the dashboard's five primary areas, not a new server. -> **Current implementation note (2026-08-04):** The dashboard exposes exactly three primary areas: -> Overview, Usage, and Observability. One fixed, left-aligned secondary rail provides the current -> area's destinations. Usage still loads lazily and remains separate from the live transcript -> tailers. +> **Current implementation note (2026-08-25):** The dashboard exposes five primary areas: About, +> Overview, Usage, Observability, and System. One fixed, left-aligned secondary rail provides the +> current area's destinations. Usage still loads lazily and remains separate from the live +> transcript tailers. -Usage carries five in-page views — **Scorecard**, **Limits**, **Findings**, **Sessions**, and -**Transcript** — deep-linked as `#usage/score`, `#usage/limits`, `#usage/findings`, -`#usage/sessions`, and `#usage/transcript`. A selected retained session uses +Usage carries six in-page views — **Scorecard**, **Limits**, **Findings**, **Sessions**, **Models**, +and **Transcript** — deep-linked as `#usage/score`, `#usage/limits`, `#usage/findings`, +`#usage/sessions`, `#usage/models`, and `#usage/transcript`. A selected retained session uses `#usage/` and opens Transcript detail. Each destination publishes a visible heading and plain-language description. The primary and secondary tab sets use a roving selected state: Left/Right Arrow activates the adjacent destination with wrapping, while Home and End activate the first and last destination. This reuses ADR-0005's in-page reveal idiom without adding navigation concepts. +> **ADR-0032 amendment:** Models is the sixth secondary Usage destination. It reads inventory and +> lifecycle projections; it does not make Historical Usage own catalogues, entitlement, route +> impact, or model quality. + **2026-08-04 session-identity amendment:** the compact session badge identifies the execution host only. Its adjacent disclosure control expands evidence without navigating away from the session list. The details report execution host, inference provider, provider evidence/provenance, and diff --git a/docs/adr/0016-capability-driven-integration-adapters.md b/docs/adr/0016-capability-driven-integration-adapters.md index 7e31a02..bb42ed8 100644 --- a/docs/adr/0016-capability-driven-integration-adapters.md +++ b/docs/adr/0016-capability-driven-integration-adapters.md @@ -4,7 +4,7 @@ [ADR-0020](0020-ga-stable-surfaces.md); closed-registry clause superseded by [ADR-0029](0029-host-adapter-extension-point.md) - **Date:** 2026-07-28 -- **Updated:** 2026-08-20 +- **Updated:** 2026-08-25 - **Update note:** Added read-only Codex plugin-hook compatibility facts, runtime-selected Ruflo project-memory store proofs, and the non-correlatable OpenRouter account-analytics boundary; removed the pre-GA compatibility command, @@ -12,10 +12,13 @@ to declare its setup trust posture and changes for host-neutral preflight. Phase 0 consistency pass (2026-08-14): host adapters gained a required `enabledByDefault` boolean and the three enabled-host default literals now - derive from it via `defaultHostMap()` (F-15); the `observability` axis is - recorded as terminal — validation metadata with referential integrity only, - deliberately not a dispatch surface, no collector loop exists (F-12); the - non-throwing `validateBinding` is wired into `ak host status` as per-entry + derive from it via `defaultHostMap()` (F-15); the `observability` axis remains validation + metadata for implemented integrations: no general-purpose collector loop + exists (F-12). ADR-0032 implements one narrow exception: Model lifecycle intelligence selects + catalogue descriptors and dispatches only to built-in, bounded source adapters. It does not turn + descriptors into arbitrary executable plugins; release proof for the narrow collector remains + pending. + The non-throwing `validateBinding` is wired into `ak host status` as per-entry warnings (F-16); and the integrations migrator derives each host's native default provider from the provider registry's host-login entries instead of a literal map, inferring no binding at all for hosts without one (F-13). diff --git a/docs/adr/0017-opencode-host.md b/docs/adr/0017-opencode-host.md index 63c8e27..8b0e147 100644 --- a/docs/adr/0017-opencode-host.md +++ b/docs/adr/0017-opencode-host.md @@ -3,8 +3,11 @@ - **Status:** Accepted; compatibility references amended by [ADR-0020](0020-ga-stable-surfaces.md) - **Date:** 2026-07-28 -- **Updated:** 2026-08-17 -- **Update note:** Clarified that the AQE boundary applies to inference-provider routing, not +- **Updated:** 2026-08-25 +- **Update note:** ADR-0032 implements project/provider-scoped OpenCode model discovery through a + bounded descriptor-driven source adapter and an explicit online refresh; release proof remains + pending. That reader does not change OpenCode's opt-in, non-primary, non-AQE routing boundary. + Clarified that the AQE boundary applies to inference-provider routing, not AQE's upstream OpenCode platform assets, and recorded the implemented OpenCode transcript, token, observed-cost, and provider-id analytics path. ADR-0023 adds classified SQLite source health, preserves last-good OpenCode usage when a present store is temporarily unreadable, @@ -20,6 +23,12 @@ > configuration. Historical references to the removed compatibility executor do not describe a > supported 4.0 surface. +**Model-lifecycle amendment (2026-08-25):** OpenCode's configured model references, +project/provider-scoped `models` output, and explicitly refreshed catalogue are evidence sources +under ADR-0032. Public discovery does not prove current-project entitlement, and ordinary +status or Dashboard reads will not invoke OpenCode or perform network refresh. +Provider-qualified CLI selectors preserve both axes as `opencode:provider/model`. + ## Context ADR-0016 separates execution hosts, inference providers, projections, observability, diff --git a/docs/adr/0021-inference-provider-provenance.md b/docs/adr/0021-inference-provider-provenance.md index ebbb97b..02b271c 100644 --- a/docs/adr/0021-inference-provider-provenance.md +++ b/docs/adr/0021-inference-provider-provenance.md @@ -2,11 +2,12 @@ - **Status:** Accepted - **Date:** 2026-07-31 -- **Updated:** 2026-08-03 +- **Updated:** 2026-08-25 - **Update note:** Claude provider resolution now covers runtime leases as well as transcript discovery, while stronger observed identity remains authoritative; OpenCode runtime presence is acknowledged without manufacturing provider identity. The user-facing surface is now named - Observability; Live remains one navigation scope within it. + Observability; Live remains one navigation scope within it. ADR-0032 reuses only independently + evidenced host/provider/model facts and never infers provider identity from a host or model name. - **Deciders:** agentic-kit maintainers - **Related:** [ADR-0012](0012-observability.md) @@ -60,6 +61,9 @@ configured | inferred | unknown`, populated from field-level evidence. Only emit - **Presentation** names the new values (AWS Bedrock, Google Vertex AI, Microsoft Foundry, Custom gateway) and continues to render "Provider not established" only when no evidence of any grade exists. +- **Model lifecycle reuse** consumes the independently evidenced host, provider, and model fields + without upgrading provenance or inferring provider identity from a host/model string. A structured + successful observation proves only that exact path at capture time. ## Consequences diff --git a/docs/adr/0032-model-lifecycle-intelligence.md b/docs/adr/0032-model-lifecycle-intelligence.md new file mode 100644 index 0000000..ec8cc99 --- /dev/null +++ b/docs/adr/0032-model-lifecycle-intelligence.md @@ -0,0 +1,294 @@ +# ADR-0032 — Model lifecycle intelligence from provenance-aware local evidence + +- **Status:** Implemented +- **Date:** 2026-08-25 +- **Updated:** 2026-08-25 +- **Update note:** The bounded inventory, descriptor-selected source adapters, conservative snapshot + diff, and read-only CLI/status surfaces are implemented. The operator-first Dashboard separates + configured routes, aggregate model use in the selected 7/14/30-day window, and a progressive + catalogue explorer. Its model detail summary collapses identical displayed provenance without + removing independent field-level evidence references. Claude refresh combines local + configuration with a dated bundled transcription of Anthropic's first-party model and + deprecation records. It fills public identity, lifecycle, limits, capabilities, availability, + and pricing without claiming account access or partner/OpenRouter routability. Ollama refresh uses bounded loopback + `/api/tags`, `/api/show`, and `/api/ps` evidence with a partial CLI fallback. Credentials, + endpoints, scopes, digests, aliases, evidence + references, session identity, and history identifiers remain protected. Cited lifecycle alerts now + name affected routes, current and recommended models, the provider notice, and a concrete planning + action. The acceptance conditions and exact-head release proof are complete. +- **Deciders:** agentic-kit maintainers +- **Related:** [issue #110](https://github.com/pacphi/agentic-kit/issues/110), + [implementation PR #179](https://github.com/pacphi/agentic-kit/pull/179), + [ADR-0001](0001-one-routing-policy-many-projections.md), + [ADR-0005](0005-dashboard-in-page-routing-reveal.md), + [ADR-0009](0009-usage-scorecard-local-transcript-analytics.md), + [ADR-0016](0016-capability-driven-integration-adapters.md), + [ADR-0017](0017-opencode-host.md), + [ADR-0020](0020-ga-stable-surfaces.md), + [ADR-0021](0021-inference-provider-provenance.md), and + [ADR-0023](0023-fail-closed-operations-and-explicit-degradation.md) + +## Context + +Agentic Kit knows the configured activity routes, their generated projections, and models observed +in local session evidence. It also carries a dated curated model table. Those facts cannot answer a +model's lifecycle state: a configured alias can move without the configuration changing; a public +catalogue entry does not prove account entitlement; a failed catalogue read does not prove removal; +and a first-party migration target does not prove equal quality or lower cost. + +Claude Code, Codex, OpenCode, and local providers expose different configuration, catalogue, cache, +policy, and runtime shapes. Treating any one as the universal schema would collapse independent +claims and make upstream schema drift look like model churn. The same model reference can also be +consumed by canonical routes, escalation rungs, host projections, Agentic QE, Ruflo, and future +Route Intelligence evidence. A lifecycle change can invalidate any of those consumers without +authorizing Agentic Kit to rewrite them. + +The existing boundaries remain load-bearing: + +- top-level `kit.json.routing` is the one routing policy; +- host and inference-provider identity are independent; +- native evidence crosses source-specific anti-corruption adapters; +- unknown and degraded evidence remain visible rather than becoming false or zero; and +- the Dashboard is local, protected, read-only, and network-silent on ordinary reads. + +When a host-owned Claude or Codex catalogue omits provider identity, its neutral facts may join an +independently established matching first-party `anthropic` or `openai` path. This is an exact +host/model/scope/digest join, not provider inference: custom and gateway providers remain separate. + +## Decision + +### 1. Establish a Model lifecycle intelligence bounded context + +Model lifecycle intelligence owns normalized model inventory snapshots, lifecycle and +compatibility edges, trustworthy snapshot diffs, consumer impact, and read-model projections. It +consumes configuration intent, integration descriptors, Historical Usage and Observability +evidence, and host/provider catalogue evidence. + +It does not own routing policy, host/provider configuration, transcript indexing, model quality +evaluation, or Ruflo/Agentic-QE routing. It may diagnose those consumers and produce a read-only +swap plan, but mutation remains on the canonical `ak host pick` surface. + +### 2. Keep identity, state, and evidence independent + +A model identity is scoped by execution host, inference provider, concrete model id, and a +non-identifying scope id. A mutable local digest participates when the provider exposes one. +Reasoning effort and service tier belong to a route binding or observed execution variant; the +model record may state which variants it supports, but those settings do not create unrelated base +model identities. Claude Code's documented `[1m]` extended-context suffix is a selector variant, +so the inventory preserves it as configuration evidence while joining to the suffix-free model +identity. + +The following dimensions remain independent: + +- configured; +- effective after precedence and alias resolution; +- observed in structured local evidence; +- discoverable in the active scope; +- entitled for the active account or profile; +- allowed by managed or user policy; +- routable through the complete host/provider/auth/capability path; +- lifecycle state; and +- recommended by a named first-party or evidence-backed source. + +Every established field and graph edge carries an evidence reference naming its source, class, capture time, +freshness, completeness, and scope. Evidence strength never leaks from one field to another. In +particular, observed use may establish entitlement for that observed path, but it does not establish +the completeness of a catalogue. + +### 3. Translate native catalogues through descriptor-driven source adapters + +The immutable model-discovery registry identifies catalogue sources for a host or provider. A +bounded Model lifecycle collector loop selects those descriptors and dispatches to explicit +source-specific adapters. The descriptor determines owner, transport, network policy, schema, and +scope; command code does not maintain a parallel host switch. A descriptor is metadata, not an +executable supplied by the host, and external adapters without a supported catalogue descriptor do +not receive an inferred capability. + +The initial adapter set covers: + +- Claude user settings, platform-managed settings, a model-only environment allowlist, aliases, + overrides, and policy allowlists, plus a dated bundled first-party Anthropic model/lifecycle + record. The public record is network-silent and account-neutral; it leaves account, Claude Code + plan, partner-platform, and OpenRouter entitlement/routability unknown; +- Codex's host-owned model cache or stable model-list protocol behind schema/version guards; +- OpenCode's project/provider-scoped model list and separately authorized online refresh; and +- Ollama installed catalogue, digest, safe model detail, and loaded runtime evidence from its + loopback HTTP API through the same normalized contract. Raw templates, model files, and full + license bodies are discarded; the CLI list is only a partial compatibility fallback. + +OpenCode collection resolves the effective global, project, JSONC, agent, and command configuration +through `opencode debug config`. An explicitly online refresh runs `opencode models --refresh` +separately from the bounded verbose listing and joins each provider/model key exactly against a +bounded Models.dev catalogue response. Human name, family, capabilities, limits, lifecycle, and +pricing are allowlisted only after that exact public proof; provider syntax alone never proves a +public identity. Custom providers, configured variants, and ambiguous selectors remain private. +Diagnostics and catalogue input are bounded independently of subprocess output. + +Interactive picker scraping and inference probes are excluded. All subprocess calls use literal +argument arrays, bounded timeouts, output-size limits, and no shell interpolation. + +### 4. Persist sanitized, bounded snapshots and advance baselines conservatively + +A `CatalogSnapshot` contains a schema version, id, capture time, scope, source states, normalized +models, bindings, and diagnostics. Source states are `complete`, `partial`, `stale`, `unavailable`, +`unsupported`, or `unsupported-schema`. Snapshots are rebuildable operational evidence, never +canonical configuration. + +History retains at most 32 baseline-eligible snapshots per scope and no snapshot older than 90 +days. An implementation may retain a newer degraded diagnostic snapshot outside the comparison +baseline, but it may not displace the last eligible baseline. Scope identity includes host, +provider, host/source schema version, and keyed non-identifying account/profile/project +fingerprints. Snapshots from different scopes are never compared as one lifecycle sequence. + +An authoritative first-party retirement or removal signal may create a tombstone immediately. +Without one, removal requires absence from two consecutive complete snapshots in the same stable +scope. Partial, stale, unavailable, or schema-invalid sources never create removals and never +advance the comparison baseline. + +### 5. Represent lifecycle and compatibility as typed edges + +Lifecycle edges include `resolves-to`, `first-party-migration`, and `same-family-newer`. +Compatibility edges include `mechanically-compatible`, `tier-up`, `tier-down`, and +`specialized-alternative`. Each edge carries provenance and scope. + +Mechanical compatibility requires an expressible host/provider transport, required modality/tool +capabilities, supported route variant, policy allowance, and established entitlement or an explicit +warning that blocks the compatibility claim. Unknown required evidence yields `unknown`, not a +compatible edge. + +Only Route Intelligence may contribute `evidence-backed-equivalent`, `cheaper-equivalent`, or +`premium-justified` claims. When no such evidence exists, Model lifecycle intelligence says +`quality unknown`. Alias-target or relevant capability changes keep historical Route Intelligence +evidence visible but mark it stale; invalidation never deletes its audit history. + +The implemented Route Intelligence handoff contains mechanically eligible candidates plus +lifecycle, alias, capability, digest, reasoning, context, variant, and pricing invalidations. It +may carry optional source-attributed pricing as a fact, but its contract explicitly denies quality +and economics claims. Route Intelligence must evaluate those claims under issue #109. + +### 6. Ship one read-only command family + +The stable noun is plural: `ak models`. + +- `ak models status` reads the latest local snapshot and current local bindings; +- `ak models refresh` collects local config, caches, protocols, and observed evidence; +- `ak models refresh --online` is the only model-catalogue network boundary; +- `ak models diff` compares eligible same-scope snapshots; +- `ak models explain` discloses a model or alias evidence chain; and +- `ak models plan` reports affected routes, projections, Agentic QE/Ruflo consumers, and stale + evidence, then emits a copyable canonical `ak host pick --route ...` command when expressible. + +There is no `ak models apply`. A future transactional swap requires a separate decision and must +still mutate only canonical routing intent with preview, confirmation, verification, and undo. +`ak status --deep` remains local and does not perform remote model refresh. + +The `ak status` model row may name explicit advisory actions. `ak sync` excludes them from its +executable convergence plan: it neither refreshes a catalogue nor applies a model plan. + +### 7. Add cache-only status and Dashboard read models + +`ak status` gains one cache-only model-health row. The Dashboard keeps its five primary areas — +About, Overview, Usage, Observability, and System — and adds Models as a secondary destination +under Usage plus a compact Overview summary. The Models destination presents attention, host +inventory, change history, consumers, swap impact, and evidence disclosure. + +Normal status and Dashboard reads never refresh a catalogue or invoke a model. The Dashboard cannot +apply a plan. It fetches a compact summary before a first 50-row relevant inventory page. Search, +faceted filters, server-side sorting, and later pages operate on the sanitized projection. The +height-bounded inventory scrolls internally with a sticky header, so history and consumer panels do +not move behind the full catalogue. All model counts, badges, and warnings link to their evidence +state, freshness, completeness, and scope. + +Provider history does not become local attention merely because it is public. A cited lifecycle +migration enters attention only when that model is configured/effective or observed in this scope. +Retired catalog-only records remain inspectable in the full catalogue and do not flood the in-use +view. + +### 8. Protect private scope and configuration facts + +Credentials, auth tokens, prompts, reasoning traces, raw private provider configuration, endpoints, +and raw account/profile/project identities never enter snapshots or Dashboard payloads. The +token-gated loopback Dashboard is an owner operator surface, so it may show bounded exact configured +or observed model selectors, display names, and providers. `/api/models` still requires the existing +owner-only per-install secret. Digests, aliases, arbitrary configuration values, edges, binding ids, +evidence references, scopes, and history identifiers remain keyed pseudonyms. Public identity does +not prove serving provider, entitlement, policy, routability, capability, or quality. Filtering and +sorting run after privacy projection, a missing or invalid key fails closed, and an ordinary read +never creates one. + +Snapshot files are owner-only and atomically replaced. Native cache/protocol data is untrusted and +subject to byte, schema, enum, and timeout bounds. Dashboard delivery retains loopback binding, +session-token authorization, CSP/origin protections, `no-store`, and secret scanning. + +## Consequences + +- Model existence, access, policy, use, lifecycle, and recommendation can disagree honestly without + collapsing to one availability boolean. +- Failed discovery cannot manufacture a mass removal or erase the last trustworthy baseline. +- Host-native schema changes degrade one source behind its adapter instead of corrupting the domain + model. +- `kit.json.routing`, `ak host pick`, Ruflo, Agentic QE, and Route Intelligence keep their existing + ownership; inventory is a diagnostic and planning consumer. +- The local cache adds bounded disk state and a per-install fingerprint secret that uninstall and + privacy documentation must account for. +- Supporting a new host catalogue requires a descriptor, an anti-corruption adapter, fixtures, and + explicit evidence semantics; host identity alone grants nothing. +- Exact owner-visible model identity makes the local ledger operational without weakening credential, + endpoint, account, scope, or configuration-value protection. +- First-party public facts remove avoidable Claude lifecycle/specification unknowns while preserving + `unknown` for local configuration, entitlement, policy, and routability. +- Catalogue enrichment cannot rewrite Historical Usage sessions/transcripts or Observability + live/history records; those contexts remain upstream owners of their data. + +## Acceptance conditions + +The decision may be marked Implemented only when: + +1. all independent state dimensions and evidence references survive human and JSON projections; +2. Claude, Codex, OpenCode, and local-provider fixtures normalize deterministically; +3. partial or cross-scope snapshots cannot create removals or advance the baseline; +4. alias, migration, capability, visibility, local-digest, reasoning, context, variant, and pricing + changes diff correctly; +5. plans enumerate canonical routes and independent Agentic QE/Ruflo consumers without mutation; +6. ordinary CLI/Dashboard reads are proven network-silent and token-silent; +7. snapshot and Dashboard payloads pass credential, prompt, endpoint, private-id, path, keyed + pseudonym, and fail-closed disclosure checks; +8. Dashboard keyboard, responsive, and screen-reader contracts pass; and +9. exact-head project, Agentic QE, privacy, security, and release gates are recorded. + +## Implementation and release proof + +The implementation branch was validated on 2026-08-25 with the following exact-head evidence: + +- `pnpm run check` passed TypeScript checking, ESLint, Markdown lint, packaging, CLI-load checks, + and the full unit/integration suite. Native instrumented coverage was 87.13% lines, 80.11% + branches, and 85.19% functions against 70% repository floors. +- `pnpm run test:ui` passed the 331-check deterministic browser matrix, including Models-only panel + ownership across every Usage submenu, readable Ollama build/runtime detail, lazy Models + loading, paired evidence filters, clickable ascending/descending column sorting, snapshot-bound + pagination with safe reset, append retry without row loss, keyboard movement in the bounded + scrolling table, Claude public-fact/access/routability disclosure, tab navigation, evidence + disclosure, responsive behavior, and network-silent page loading. +- A production-sized OpenCode 1.18.23 fixture yielded all 402 models without truncation. Exact + Models.dev proof normalized the catalogue's omitted active status, while malformed successful + responses degraded to partial evidence rather than public identity. +- Agentic QE 3.13.12's native Node-test execution passed 16/16 focused Dashboard projection/API + tests. Its SAST scanner inspected all 16 model-inventory files and all 13 Dashboard files and + reported no vulnerabilities. The repository's native coverage report remains the coverage + authority; no Agentic QE score was inferred from separate process-local evidence. +- A 179-test boundary suite covering Usage indexing, OpenCode/OpenRouter usage, live sessions, + selected transcripts, and Observability identity passed. It proves public catalogue enrichment + does not change those live or historical projections. +- Privacy and security tests proved owner-only atomic snapshots, bounded subprocess output, + cache-only reads, explicit no-write/no-network online dry runs, source-proven public identity, + keyed private Dashboard pseudonyms, fail-closed missing-key behavior, and absence of raw private + identifiers from `/api/models`. +- `pnpm audit --prod` reported no known vulnerabilities. No runtime dependency was introduced; + the implementation uses Node's built-in filesystem and cryptography APIs. +- The package dry run included this ADR, the DDD model, and `docs/MODELS.md`; internal Markdown + links and the stable command-surface guards passed. + +Current official Claude Code, OpenCode, OpenAI, Ollama, and Node documentation was rechecked on +2026-08-25 before freezing source commands, selector parsing, model lookup semantics, and runtime +API usage. Public provider catalogues remain discovery evidence only; they are not promoted to +host entitlement evidence. diff --git a/docs/adr/README.md b/docs/adr/README.md index c6d2bb9..86d5c5c 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -39,6 +39,7 @@ Consequences**, and cites the grounded source it rests on where relevant. | [0028](0028-local-openai-compatible-providers.md) | One generic local OpenAI-compatible provider, not a vendor enumeration | Accepted | | [0029](0029-host-adapter-extension-point.md) | External host adapters: declarative manifest, subprocess hooks | Accepted (experimental contract) | | [0031](0031-capability-graduation-and-upstream-requests.md) | Capability graduation: earned parity for external adapters, and the upstream request path | Accepted (governance; implementation staged) | +| [0032](0032-model-lifecycle-intelligence.md) | Model lifecycle intelligence from provenance-aware local evidence | Implemented | Theme: ADRs **0001–0006** define **dual-host LLM routing and leadership** — how `ak` lets ruflo route each development activity (architecture, implementation, testing, review, …) to the right host (Claude @@ -225,3 +226,14 @@ agentic-qe's closed enum; being a native ruflo backend is ruflo's `ENABLE_*` mod tracked capability requests with honest interim behaviour rather than pretended support. Accepted as a governance decision; the machinery (the trust CLI, external execution, tiered conformance, the grant store) is staged and self-graded in the ADR's implementation-status table. + +**0032** accepts Model lifecycle intelligence as a new bounded context. Its implementation and +exact-head release proof are recorded in the decision. It keeps configured, +effective, observed, discoverable, entitled, policy-allowed, +routable, lifecycle, and recommended state independent; normalizes host/provider catalogues through +bounded descriptor-driven source adapters; and persists sanitized same-scope snapshots whose +baselines advance only on sufficiently complete evidence. Diffs, explanations, and swap plans are +read-only. Canonical route mutation remains `ak host pick`, first-party migration remains distinct +from quality, and only Route Intelligence may claim evidence-backed equivalence. The feature adds +`ak models`, a cache-only status row, an Overview summary, and Models beneath Usage +without changing the Dashboard's five primary areas. diff --git a/docs/ddd/README.md b/docs/ddd/README.md index 6e86f39..7b52c00 100644 --- a/docs/ddd/README.md +++ b/docs/ddd/README.md @@ -11,6 +11,7 @@ describe the current system unless a section is explicitly marked as future work | [Context map](context-map.md) | Bounded contexts, ownership, and relationships | | [Integration management](integration-management.md) | Hosts, inference providers, bindings, capabilities, lifecycle, facts, and ownership | | [Routing and orchestration](routing-and-orchestration.md) | Activities, routes, leadership, escalation, projections, and canonical `ak run` execution | +| [Model lifecycle intelligence](model-lifecycle-intelligence.md) | Model identities, evidence dimensions, catalogue snapshots, lifecycle diff, and read-only impact plans | | [Observability](observability.md) | Evidence acquisition, observed-session aggregates, replay, and dashboard delivery | | [Project intelligence](project-intelligence.md) | Pattern store, learning counters, reasoning-graph size, and live delivery for Overview's Intelligence view | @@ -32,6 +33,10 @@ A change that introduces or changes a domain concept should: 4. add executable coverage for new invariants; and 5. update operational documentation when user-visible behavior changes. +An Accepted ADR may define a target contract before its implementation exists. Such a domain +document must label planned commands, collectors, or read models explicitly and must not present +them as shipped behavior. + The GA model keeps host enablement under `integrations.hosts`, integration ownership under `integrations`, and activity intent under top-level `routing`. These persisted locations do not change the canonical distinction between a host and an inference provider. diff --git a/docs/ddd/context-map.md b/docs/ddd/context-map.md index a57816a..a0c7592 100644 --- a/docs/ddd/context-map.md +++ b/docs/ddd/context-map.md @@ -13,12 +13,12 @@ Configuration Intent Native Evidence ----> Evidence Acquisition ----> Canonical Evidence | +-------------------------+----------------------+ - | | - v v - Observability Historical Usage - | | | - | +----> Workspace Snapshot Cache | - | | | + | | | + v v v + Observability Model Lifecycle Intelligence Historical Usage + | | | | + | +----> Workspace +----------------------+ + | Snapshot Cache | +-----------------------+----> Dashboard Delivery <----+ ^ Project State (.claude-flow/*) ----> Project Intelligence-+ @@ -71,6 +71,18 @@ Owns transcript indexing, session history, token and cost aggregation, classific findings. It may share a host-qualified session identity with Observability, but its aggregate and cache are separate from the live event store. +### Model lifecycle intelligence + +Owns the normalized inventory of configured, effective, observed, discoverable, entitled, +policy-allowed, routable, and lifecycle model facts; sanitized same-scope snapshots; trustworthy +diffs; lifecycle and compatibility edges; and read-only consumer impact plans. It consumes +Configuration Intent, integration descriptors, source-adapted native catalogues, and structured +observed facts from Historical Usage and Observability. It does not own transcripts, route +mutation, model quality, or downstream router policy. + +See [Model lifecycle intelligence](model-lifecycle-intelligence.md). The context and exact-head +release proof are recorded by implemented ADR-0032. + ### Workspace snapshot cache Owns the bounded, owner-only last safe `SessionWorkspace` value per host-qualified session. It is @@ -139,6 +151,13 @@ and credential policy is distinct from the offline-first dashboard and integrati | Native evidence | Evidence acquisition | Source-specific anti-corruption adapters | | Evidence acquisition | Observability | Versioned canonical events | | Evidence acquisition | Historical usage | Normalized transcript and provider evidence | +| Configuration intent | Model lifecycle intelligence | Canonical route, escalation, binding, and projection references | +| Integration management | Model lifecycle intelligence | Capability and catalogue descriptors plus bounded host/provider facts | +| Evidence acquisition | Model lifecycle intelligence | Source-adapted catalogue, policy, lifecycle, and runtime evidence | +| Historical usage | Model lifecycle intelligence | Structured observed host/provider/model facts; transcript ownership stays upstream | +| Observability | Model lifecycle intelligence | Structured recent execution identity; live-event ownership stays upstream | +| Model lifecycle intelligence | Routing and orchestration | Read-only lifecycle diagnostics, mechanical candidate/invalidation feed, and copyable canonical route actions; no quality/economics claim | +| Model lifecycle intelligence | Dashboard delivery | Sanitized cache-only inventory, changes, consumers, and plan read models | | Observability | Dashboard delivery | Read-model snapshots, deltas, and selected evidence | | Observability | Workspace snapshot cache | Last safe metadata-only session workspace capture | | Workspace snapshot cache | Dashboard delivery | Inert last-recorded History context after restart | @@ -175,6 +194,10 @@ observed before the split was made explicit. - User intent does not establish observed reality. - A host observation does not establish inference-provider identity. - Dashboard presentation cannot upgrade provenance. +- Model lifecycle inventory cannot turn discovery into entitlement, compatibility into quality, or + a read-only plan into route mutation. +- Snapshot comparison requires stable scope and sufficient source completeness; degraded evidence + never creates removal. - Historical usage and live topology share identifiers, not aggregate ownership. - Network egress occurs only in commands and contexts whose contract explicitly permits it. - Every project count is rendered with the scope that produced it; two contexts may report diff --git a/docs/ddd/integration-management.md b/docs/ddd/integration-management.md index 885e0ac..8fa2b38 100644 --- a/docs/ddd/integration-management.md +++ b/docs/ddd/integration-management.md @@ -27,8 +27,10 @@ Host -------- ProviderBinding -------- InferenceProvider +-- host capabilities ``` -The four adapter registries are closed, validated, built-in code. A provider binding is persisted -relationship data, not a fifth executable adapter family. +The four built-in adapter registries are validated code. ADR-0029 additionally admits an +experimental, hash-consented, subprocess-only external host overlay; it does not turn provider, +projection, or observability descriptors into arbitrary in-process plugins. A provider binding is +persisted relationship data, not a fifth executable adapter family. ### Host @@ -64,6 +66,10 @@ protocols, and non-loopback plaintext HTTP. - Adding OpenRouter does not make it a host. - Provider claims are rendered only when the provider declares and evidence supports them. - Compatibility collections are derived from registries. +- A model-discovery descriptor grants no executable behavior by itself. Under ADR-0032, the bounded + Model lifecycle collector selects model-discovery descriptors and dispatches only to explicit + built-in source adapters. External hosts without a supported catalogue descriptor remain + `unsupported`. ## Integration facts @@ -88,6 +94,17 @@ Billing follows the access path. Anthropic or OpenAI host login may be subscript their API keys are metered. Ollama is local and may be priced at exact zero only after provider identity is established. +## Model catalogue evidence + +ADR-0032 implements a narrow, read-only use of integration model-discovery descriptors. Catalogue +descriptors identify source ownership, evidence fields, local/online mode, and host/provider scope. +They remain declarative. The Model lifecycle bounded context owns collector dispatch, native-schema +translation, completeness, freshness, and snapshot policy. + +This reader does not widen the managed projection lifecycle: catalogue refresh never calls +`apply`, never invokes a model, and never changes a binding. Online discovery is separately +authorized by `ak models refresh --online`; normal status and Dashboard reads remain cache-only. + ## Managed projection lifecycle Every managed configuration projection follows: @@ -151,3 +168,6 @@ envelopes and never reads the retired paths. 8. Undo cannot overwrite drift. 9. Existing and future configuration survives additive migration. 10. Status, setup, sync, verify, and uninstall consume the same normalized model. +11. Catalogue descriptor identity alone executes nothing and proves neither entitlement nor + routability. +12. Model lifecycle collection cannot mutate integration intent or native projections. diff --git a/docs/ddd/model-lifecycle-intelligence.md b/docs/ddd/model-lifecycle-intelligence.md new file mode 100644 index 0000000..5aa0ddf --- /dev/null +++ b/docs/ddd/model-lifecycle-intelligence.md @@ -0,0 +1,275 @@ +# Model Lifecycle Intelligence Domain + +This document defines the bounded context accepted by +[ADR-0032](../adr/0032-model-lifecycle-intelligence.md). The implementation and exact-head release +proof are recorded by ADR-0032, whose current status is Implemented. + +## Purpose + +Model lifecycle intelligence answers which models are configured, selected, observed, discoverable, +usable, changing, and consequential to local consumers. It preserves the evidence and uncertainty +behind each answer and produces read-only status, diff, explanation, and swap-impact projections. + +It does not select the best model, mutate routing, own host/provider configuration, or replace the +Ruflo, Agentic QE, or Route Intelligence routers. + +## Context boundary + +```text +Configuration Intent ------> binding collector --------+ +Integration descriptors ---> catalogue adapters -------+ +Historical Usage ----------> observed-use collector ---+--> CatalogSnapshot +Host/provider evidence -----> source adapters ----------+ | + +--> ModelChange + +--> ConsumerImpact + +--> SwapPlan + +--> read models +``` + +Model lifecycle intelligence consumes facts from four neighboring contexts: + +- Configuration Intent supplies canonical routes, escalation rungs, integration bindings, and + provider configuration references. +- Integration Management supplies capability and observability descriptors, projection identity, + and bounded host/provider facts. +- Evidence Acquisition supplies source-adapted catalogue, policy, lifecycle, and runtime evidence. +- Historical Usage and Observability supply structured observed host/provider/model facts without + surrendering ownership of transcripts, sessions, or live events. + +The Claude adapter combines two independent sources: local Claude configuration/policy and a +dated, bundled transcription of Anthropic's first-party public model and deprecation records. The +public supplement is network-silent and account-neutral. It can establish publication, public +specifications, and lifecycle only; it cannot establish Claude Code plan entitlement, API-key +availability, partner-platform availability, or an OpenRouter path. + +Its outputs are advisory inputs to Routing and Orchestration, Dashboard Delivery, Agentic QE/Ruflo +diagnostics, and Route Intelligence. + +## Aggregate and value objects + +### ModelIdentity + +```text +ModelIdentity + host + provider + modelId + scopeId + digest? +``` + +Identity is host- and scope-qualified because equal strings can name different deployments, +accounts, projects, gateways, or local model bytes. `digest` participates only where a source +establishes a mutable local artifact digest. Reasoning effort and service tier belong to a binding +or execution variant, not to the base identity. Claude Code's documented `[1m]` selector suffix is +likewise retained as configured variant/selector evidence and removed from the base model id. + +Host-owned Claude and Codex catalogues may be provider-neutral. If independent evidence establishes +the exact same host/model/scope/digest under the host's expected first-party provider (`anthropic` +or `openai`), the neutral catalogue fields join that provider-qualified record. The join never +targets a custom or gateway provider, so equal model strings on distinct serving paths remain +distinct identities. + +### ModelRecord + +A ModelRecord carries display identity, aliases, lifecycle state, typed edges, supported variants +and capabilities, optional pricing metadata, and field-level evidence references. No record-level +confidence may silently strengthen a weaker field. Optional pricing is a dated source fact; it is +not an economic recommendation. + +The independent state dimensions are configured, effective, observed, discoverable, entitled, +policy allowed, routable, lifecycle, and recommended. Each is `true`, `false`, or `unknown` where +applicable and names the evidence that established it. + +### CatalogIdentityProjection + +The Dashboard projection separates the execution host, serving provider, publisher/lab, human +name, exact public selector, and catalogue source. These axes cannot be derived from one another. +A controlled source may mark an identity public only when its bounded parser established that +catalogue metadata. The bundled Anthropic public record makes its exact documented Claude ids and +aliases public product vocabulary and carries first-party specifications/lifecycle evidence. Codex +cache display names qualify only through host-owned discovery evidence. OpenCode identity becomes +public only after an exact provider/model-key join with the +bounded Models.dev catalogue; provider syntax does not establish proof. A custom +provider, configured variant, gateway deployment, local tag, or observed-only id remains a keyed +pseudonym. Trusted documentation and catalogue links are server-produced HTTPS links on a fixed +host allowlist. The browser never guesses a link from a model-name substring. + +### ModelBinding + +```text +ModelBinding + consumer + activity? + host + provider + configured reference + effective concrete identity? + execution variant? + provenance + evidenceRefs[] +``` + +Consumers include canonical routes, escalation rungs, host projections, Agentic QE overrides and +fallbacks, Ruflo candidates, and Route Intelligence cohorts. A configured alias remains in the +binding even after an effective concrete target is resolved. + +### CatalogSource + +A source identifies its owner and owner type, file/command/HTTP/index transport, `never`/`local`/ +`explicit` network policy, local or online collection mode, source/schema version, capture time, +non-identifying scope fingerprint, status, completeness, and diagnostics. Status is one of +`complete`, `partial`, `stale`, `unavailable`, `unsupported`, or `unsupported-schema`. +Bundled public sources also carry a verification date. The Anthropic source becomes `stale` after +90 days; an operator must upgrade Agentic Kit to receive revised public facts before refreshing the +snapshot. + +### CatalogSnapshot + +The aggregate root is a sanitized, immutable snapshot: + +```text +CatalogSnapshot + schemaVersion + snapshotId + capturedAt + scope + sources[] + models[] + bindings[] + diagnostics[] +``` + +Changes, opportunities, and plans are derived from snapshots plus current bindings; they are not +written back as catalogue truth. + +### LifecycleEdge and CompatibilityEdge + +Lifecycle relations are `resolves-to`, `first-party-migration`, and `same-family-newer`. +Compatibility relations are `mechanically-compatible`, `tier-up`, `tier-down`, and +`specialized-alternative`. Every edge has source, scope, confidence, and evidence references. + +`evidence-backed-equivalent`, `cheaper-equivalent`, and `premium-justified` belong to Route +Intelligence. This context can preserve and invalidate those imported claims but cannot create them. + +### ModelChange + +A change names its kind, subject, before/after values, severity, scope, and evidence. Additions, +visibility, alias, lifecycle, capability, reasoning, context, other variant, optional pricing, +typed-edge, and digest changes require comparable same-scope evidence. Alias and digest continuity +pair one model lineage before evaluating fields. Removal additionally requires an authoritative +signal or two consecutive complete same-scope absences. + +Lifecycle, pricing, and edge comparison uses semantic fields and ignores evidence-reference ids and +capture timestamps. A refresh that restates the same fact with new evidence does not create churn. + +### SwapPlan + +A SwapPlan is a read-only impact report. Each item links a source binding to affected canonical +routes, projections, Agentic QE/Ruflo consumers, compatibility blockers, and evidence that becomes +stale. When expressible, it supplies a copyable `ak host pick --route ...` command. It has no apply +operation. + +Unknown catalogue discovery blocks planning unless a structured successful observation proves the +exact target path; that exception remains an explicit warning. `discoverable: false` always blocks. +OpenCode selectors retain both axes as `opencode:provider/model`. + +### RouteIntelligenceFeed + +The feed to issue #109 contains mechanically eligible candidates and audit-preserving invalidation +markers for lifecycle, alias, capability, digest, reasoning, context, variant, and pricing changes. +It can carry optional source-attributed pricing but sets quality and economics claims to false. +Route Intelligence owns any later evidence-backed equivalence or cost conclusion. + +## Evidence rules + +Evidence classes, strongest first for the field they actually establish, are: + +1. observed successful execution with concrete host/provider/model identity; +2. host-owned entitled catalogue or explicit lifecycle metadata; +3. host-owned discoverable catalogue/cache in the active scope; +4. managed policy/configuration after precedence resolution; +5. canonical Agentic Kit intent and generated projection; +6. provider-published public catalogue; +7. dated locally curated metadata; +8. inferred family relationship; and +9. unknown. + +Strength is field-local. A successful execution proves that path worked at that time; it does not +prove catalogue completeness, future entitlement, or global provider identity. Negative evidence +requires completeness and stable scope. + +## Snapshot lifecycle + +- Retain at most 32 baseline-eligible snapshots per scope and no snapshot older than 90 days. +- Advance a baseline only from a sufficiently complete snapshot in the same stable scope. +- Preserve the last eligible baseline when a later collection is partial, stale, unavailable, or + schema-invalid. +- Compare account/profile/project scopes only when their keyed fingerprints match. +- Allow an authoritative first-party removal immediately; otherwise require two consecutive + complete absences. +- Keep stale recommendation history auditable after alias, capability, provider, host-version, or + harness changes. + +## Discovery contract + +The collector selects immutable model-discovery descriptors and invokes a bounded built-in source +adapter. Each descriptor establishes owner, transport, network policy, schema, and scope. The +command layer does not keep a parallel host switch, and descriptors authorize no arbitrary code. +An external host receives no inferred catalogue capability without an admitted descriptor and +matching adapter. + +Claude collection reads user settings and the platform-managed settings path. It accepts only +`ANTHROPIC_MODEL` and `ANTHROPIC_DEFAULT_{SONNET,OPUS,HAIKU}_MODEL` from the process environment; +credentials, endpoints, and unrelated environment values remain outside the collector. + +Local refresh reads configuration, local caches/protocols, and observed evidence. OpenCode first +resolves its effective configuration with `opencode debug config`, including global, project, +JSONC, agent, and command model bindings. Online refresh is separate and explicit: it refreshes the +OpenCode catalogue, obtains a bounded Models.dev response, then parses a separately bounded verbose +listing. Display name, family, capabilities, limits, lifecycle, and pricing are public only after +an exact provider/model-key catalogue join. Arbitrary bounded selectors remain usable without +making custom provider URLs, headers, options, or ids public. Neither path invokes a model or sends +a prompt. + +Ollama refresh contacts only its loopback `/api/tags`, `/api/show`, and `/api/ps` endpoints. It +keeps installed name, digest, size, update time, format/family/build, bounded capability/context and +license summaries, plus loaded memory/VRAM/context/expiry. Raw templates, model files, parameters, +and full license bodies are discarded. A bounded `ollama ls` compatibility fallback is marked +partial because it cannot establish runtime facts. Installed and loaded never imply observed use. + +## Privacy and delivery + +Snapshots exclude credentials, raw provider configuration, prompts, reasoning traces, endpoints, +and raw account/profile/project identities. The atomic owner-only cache may retain bounded exact +configured model or deployment identifiers for an explicit local CLI read. An owner-only +per-install secret keys scope fingerprints and the separate Dashboard projection. + +CLI status and Dashboard reads are cache-only. CLI reads are deliberate exact local disclosure. The +Dashboard's token-gated loopback `owner-visible-v2` projection carries bounded exact model display +names, selectors, and recorded providers, so a route can answer what it is actually configured to +use. Digests, aliases, replacement internals, edges, binding ids, evidence, scope, history, +credentials, endpoints, and arbitrary provider configuration remain pseudonymous. Source-proven +public catalogue identity may retain its publisher and trusted links; public identity never makes +entitlement or routability known. The summary projection omits the large model array, and the +inventory projection filters, sorts, and pages only the already-sanitized rows. Later pages carry +the privacy-projected snapshot id; a changed snapshot is rejected and the browser restarts from page +one rather than mixing generations. Missing key material fails closed without creating state. +Delivery remains behind the loopback, session-token, origin, CSP, and `no-store` boundary and cannot +apply a plan. + +## Invariants + +1. Host, provider, model, scope, and execution variant remain separate facts. +2. Configured, effective, observed, discoverable, entitled, allowed, and routable never collapse. +3. Every fact and edge carries source, freshness, completeness, and scope. +4. Unknown or degraded evidence never becomes removal, unavailability, or compatibility. +5. Snapshots from different scopes never produce lifecycle churn. +6. Inventory never mutates canonical routing or downstream router state. +7. First-party migration is not a quality or economic recommendation. +8. Route Intelligence evidence stays visible but stale after invalidation. +9. Ordinary reads make no network request and consume no inference tokens. +10. `ak sync` does not execute model refresh or model-plan advisories. +11. Private configuration and transcript content never enter snapshots or aggregate APIs. +12. Inventory search, filtering, and sorting operate after privacy projection. +13. Inventory pagination never mixes rows from different privacy-projected snapshots. diff --git a/docs/ddd/observability.md b/docs/ddd/observability.md index 9cfdcfa..e8286c2 100644 --- a/docs/ddd/observability.md +++ b/docs/ddd/observability.md @@ -569,22 +569,31 @@ The UI receives only projection DTOs: - `health`: sanitized adapter status and aggregate counters; - `cursor` and `schemaVersion`. -The dashboard shell has exactly three primary areas: `Overview`, `Usage`, and `Observability`. One -fixed, left-aligned secondary navigation rail remains in the same location while its contents change: +The dashboard shell has five primary areas: `About`, `Overview`, `Usage`, `Observability`, and +`System`. One fixed, left-aligned secondary navigation rail remains in the same location while its +contents change: ```text Overview → Summary | Hosts & Routing | Providers | Runtime | Intelligence -Usage → Scorecard | Limits | Findings | Sessions | Transcript +Usage → Scorecard | Limits | Findings | Sessions | Models | Transcript Observability → Live | History +System → Summary | Advisory | Sessions | Storage | Runtime | Catalog | Projects ``` +About is a continuous editorial directory with secondary section anchors. ADR-0032 adds Models +beneath Usage. Model lifecycle read models may consume bounded observed identity, but Observability +retains ownership of live sessions, events, topology, and replay. The browser receives a separate +keyed, pseudonymous model projection rather than exact local identifiers. + Navigation state has canonical hierarchical hashes: ```text +#about/{hosts,engine,quality,kit,configured} #overview/{summary,hosts,providers,runtime,intelligence} -#usage/{score,limits,findings,sessions,transcript} +#usage/{score,limits,findings,sessions,models,transcript} #usage/{sessionId} #observability/{live,history} +#system/{summary,advisory,sessions,storage,runtime,catalog,projects} ``` Each destination owns a visible heading and concise description. The primary and secondary controls @@ -812,7 +821,7 @@ process memory limits. independently draggable and pinnable. - Hover/focus descriptions, persistent selection detail, and Legend / Help explain every unit and the available next interaction. -- Exactly three primary areas share one fixed, left-aligned secondary rail; canonical hashes, +- Five primary areas share one fixed, left-aligned secondary rail; canonical hashes, headings, descriptions, roving Left/Right focus, and Home/End behavior match each destination. - Collapsing Session Stream preserves its connection and local choice, expands Agent activity, leaves a keyboard-accessible restore rail, and remains compact when the layout stacks. @@ -848,8 +857,8 @@ The server subscribes before taking the initial snapshot and reconciles buffered closing the snapshot-to-subscribe race. A slow-client queue overflow discards queued frames and sends a reset snapshot after drain. -The implemented dashboard shell exposes three primary areas and one shared secondary rail. It emits -the canonical Overview, Usage, and Observability hashes above, gives every view a heading and +The implemented dashboard shell exposes five primary areas and one shared secondary rail. It emits +the canonical About, Overview, Usage, Observability, and System hashes, gives every view a heading and description, and implements Left/Right/Home/End tab semantics. Observability's locally persisted Session Stream chevron changes layout without changing subscription ownership. Usage session rows show a host-only badge and reveal provider/provenance/model facts in their own detail strip. diff --git a/docs/ddd/routing-and-orchestration.md b/docs/ddd/routing-and-orchestration.md index 33b65b5..1e178db 100644 --- a/docs/ddd/routing-and-orchestration.md +++ b/docs/ddd/routing-and-orchestration.md @@ -101,6 +101,27 @@ two Ollama bindings do not create two Ollama providers, and configured host/mode prove which inference vendor served a session. OpenCode runs only through `ak run` after its host adapter declares the required capability. +## Model lifecycle boundary + +ADR-0032 accepts a separate Model lifecycle intelligence context. It may read canonical routes, +escalation rungs, and generated projections to diagnose affected consumers. Its `SwapPlan` is a +read-only projection and may emit a copyable `ak host pick --route ...` action; it cannot apply that +action or become another routing policy. + +The curated `RETIRED_MODELS` behavior described above remains the implemented routing rule until a +separate implementation deliberately changes it. Catalogue discovery does not feed that automatic +substitution path merely because a model is hidden, missing, stale, or deprecated. +Inventory lifecycle facts diagnose and plan; only authoritative evidence under an implemented +contract may justify changing dispatch behavior. + +First-party migration and mechanical compatibility are not quality claims. Only imported Route +Intelligence evidence can call a candidate equivalent, cheaper, or worth a premium. Alias or +capability changes mark that evidence stale while preserving its audit history. + +Model lifecycle exports mechanically eligible candidates and audit-preserving invalidations to +Route Intelligence, with quality and economics claims explicitly false. `ak status` may recommend a +model refresh or diff, but `ak sync` excludes every model advisory from its executable plan. + ## Cost safety Automatic seeding targets only known subscription-backed or local execution paths. Metered @@ -121,3 +142,5 @@ must distinguish per-token price from measured or expected per-task cost. 6. Escalation is explicit, ordered, and per route. 7. Automatic seeding cannot introduce a metered provider path. 8. `ak run` is the sole executor for materialized activity plans. +9. Model lifecycle inventory may diagnose routes but cannot mutate or execute them. +10. Catalogue absence never enters the curated retirement substitution path by inference. diff --git a/docs/ddd/ubiquitous-language.md b/docs/ddd/ubiquitous-language.md index 89be81a..e01b236 100644 --- a/docs/ddd/ubiquitous-language.md +++ b/docs/ddd/ubiquitous-language.md @@ -60,6 +60,34 @@ missing price. `Dual-host` describes two enabled peer hosts, not an execution command and not evidence that two inference vendors served a workflow. Generalized execution belongs to `ak run`. +## Model lifecycle language + +These terms define ADR-0032's implemented contract. ADR status remains Accepted while exact-head +release proof is pending. + +| Term | Meaning | +|------|---------| +| Model identity | Host-, provider-, model-id-, and scope-qualified inference target, plus a digest when local bytes are mutable and evidenced | +| Model scope | Non-identifying account/profile/project/source boundary within which catalogue snapshots are comparable | +| Execution variant | Binding- or execution-level reasoning effort, service tier, modality, or similar setting; not a separate base model identity | +| Model binding | One consumer's configured reference and, when established, effective concrete model identity with provenance | +| Catalog source | Host/provider-native configuration, cache, protocol, or catalogue input with owner, transport, network policy, collection mode, schema/version, scope, freshness, completeness, and diagnostics | +| Catalog snapshot | Sanitized immutable inventory of source states, model records, bindings, scope, and diagnostics at one capture time | +| Baseline-eligible snapshot | Sufficiently complete same-scope snapshot permitted to replace the prior lifecycle comparison baseline | +| Model change | Evidence-backed difference between comparable snapshots; removal needs authoritative evidence or repeated complete absence | +| Lifecycle edge | Typed alias resolution, first-party migration, or same-family-newer relationship with provenance and scope | +| Compatibility edge | Typed mechanical swap relationship; it is not a quality or economic recommendation | +| Consumer impact | Read-only link from a lifecycle fact to affected routes, projections, Agentic QE/Ruflo consumers, or Route Intelligence evidence | +| Swap plan | Read-only impact report and copyable canonical route action; never an independent routing policy or apply operation | +| Route Intelligence feed | Mechanical candidates plus audit-preserving lifecycle invalidations; quality and economics claims are explicitly absent | +| Public catalogue identity | Human-readable model name, publisher, public selector, and trusted links retained only when bounded source evidence establishes that the identity is public | +| Private model reference | Deployment, gateway, local tag, observed-only id, or other model identity without public-catalogue proof; the Dashboard exposes only its keyed projection | +| Keyed model projection | Dashboard-only stable pseudonyms derived from the existing private scope key; distinct from exact explicit CLI evidence and source-proven public catalogue identity | + +Configured, effective, observed, discoverable, entitled, policy allowed, routable, lifecycle, and +recommended are separate model-state dimensions. `Unknown` in one dimension cannot be filled from +another. A first-party migration is a supported lifecycle edge, not proof of equivalence. + ## Project intelligence language | Term | Meaning | @@ -148,6 +176,15 @@ runtime state is a chip word, never a prose word. See - Qualify **projection** as configuration projection or read-model projection when ambiguity is possible. - Qualify **adapter** as integration adapter or source adapter when ambiguity is possible. +- Say **catalogue source** for model discovery evidence and **catalog snapshot** for the normalized, + sanitized local record; neither is canonical routing policy. +- Keep **execution host**, **serving provider**, **publisher**, and **public model selector** separate; + none can be inferred from another or from a human-readable model name. +- Say **public catalogue identity** only when a bounded source proves it. Say **private model + reference** when public identity is absent or ambiguous; do not expose its exact value in the + Dashboard. +- Say **compatible candidate** only when required mechanical facts are established. Reserve + **cheaper equivalent** and **premium justified** for Route Intelligence evidence. - Do not infer an inference provider from a transcript host alone. - Do not replace an unknown fact with a convenient default. - Say **System** for the dashboard area and the command; say **Machine footprint** only for the @@ -161,6 +198,8 @@ runtime state is a chip word, never a prose word. See providers. - `ak system [--deep] [--json]` renders the Machine footprint collector; `ak about [--category] [--json]` renders the Component directory. Both are read-only twins of a dashboard area. +- `ak models` is the read-only model inventory, refresh, diff, explain, and plan family. Route + mutation remains `ak host pick`; there is no accepted `ak models apply`. - `kit.json.integrations.hosts` records enabled hosts. Top-level `routing` records `version`, `primaryHost`, and per-activity `routes`; route entries use `provenance` and `escalation`. - Derived exports in `hosts.mjs`, `providers.mjs`, and `routing.mjs` are views, not independent diff --git a/package.json b/package.json index c840dd6..b15ee7a 100644 --- a/package.json +++ b/package.json @@ -22,12 +22,15 @@ "!claude/**/.claude-flow/**", "docs/HOST-SUPPORT.md", "docs/INSTALLATION.md", + "docs/MODELS.md", "docs/PROVIDERS.md", "docs/SETUP.md", "docs/TROUBLESHOOTING.md", "docs/UPGRADING.md", "docs/CODEX-STATUSLINE.md", - "docs/adr/0015-managed-codex-native-statusline.md" + "docs/adr/0015-managed-codex-native-statusline.md", + "docs/adr/0032-model-lifecycle-intelligence.md", + "docs/ddd/model-lifecycle-intelligence.md" ], "scripts": { "test": "node --test --experimental-test-coverage --test-coverage-lines=70 --test-coverage-branches=70 --test-coverage-functions=70 \"tests/kit/*.test.mjs\" && node tests/statusline-segments.test.cjs && node tests/statusline-brain.test.cjs && node tests/agentdb.test.cjs && node tests/health-history.test.cjs && node tests/harvest.test.cjs && node tests/dashboard.test.cjs && node tests/admin-model.test.cjs && node tests/admin.test.cjs", diff --git a/src/commands/models.mjs b/src/commands/models.mjs new file mode 100644 index 0000000..b6a73d4 --- /dev/null +++ b/src/commands/models.mjs @@ -0,0 +1,219 @@ +import { heading, info, ok, warn, dim } from '../lib/output.mjs'; +import { loadKitConfig } from '../lib/config.mjs'; +import { aqeRouterFile } from '../lib/providers.mjs'; +import { readJson } from '../lib/settings.mjs'; +import { + appendModelSnapshot, collectModelSnapshot, createModelReadModel, + diffSnapshotHistory, explainModel, latestSnapshot, modelInventoryPath, planModelChange, + previousSnapshot, readModelStore, snapshotById, summarizeModelHealth, +} from '../lib/model-inventory/index.mjs'; + +export const options = { + json: { type: 'boolean', default: false }, + host: { type: 'string' }, + all: { type: 'boolean', default: false }, + online: { type: 'boolean', default: false }, + since: { type: 'string' }, + activity: { type: 'string' }, + from: { type: 'string' }, + to: { type: 'string' }, + 'dry-run': { type: 'boolean', default: false }, +}; + +export const help = `ak models — model lifecycle evidence, changes, and swap impact + +Every command except refresh reads the private local snapshot cache and performs +no network requests. Refresh contacts only the named local/configured sources; +--online additionally permits OpenCode to refresh its catalog. + +Usage: + ak models status [--host claude|codex|opencode|ollama] [--json] + ak models refresh [--host HOST|--all] [--online] [--dry-run] + ak models diff [FROM_SNAPSHOT [TO_SNAPSHOT]] [--json] + ak models explain HOST:MODEL [--json] + ak models plan --activity ACTIVITY [--from HOST:MODEL] --to HOST:MODEL [--json] + +The plan command is read-only. It may print a copyable canonical routing command, +but never changes routing, AQE, Ruflo, or provider configuration. + +Examples: + ak models refresh --all + ak models status --host codex --json + ak models diff models:before models:after + ak models explain codex:gpt-5.6-terra + ak models plan --activity testing --to codex:gpt-5.6-terra`; + +const ALL_OWNERS = Object.freeze(['claude', 'codex', 'opencode', 'ollama']); + +function selectedOwners(flags, cfg) { + if (flags.host) { + const owners = [...new Set(String(flags.host).split(',').map((value) => value.trim()).filter(Boolean))]; + for (const owner of owners) if (!ALL_OWNERS.includes(owner)) throw new TypeError(`unsupported model host: ${owner}`); + return owners; + } + if (flags.all) return [...ALL_OWNERS]; + const enabled = Object.entries(cfg?.integrations?.hosts ?? {}) + .filter(([, value]) => value === true).map(([owner]) => owner).filter((owner) => ALL_OWNERS.includes(owner)); + return enabled.length ? enabled : ['claude']; +} + +function printJson(value) { console.log(JSON.stringify(value, null, 2)); } + +function visibleSnapshot(snapshot, host) { + if (!snapshot || !host) return snapshot; + return { + ...snapshot, + sources: snapshot.sources.filter((source) => source.owner === host), + models: snapshot.models.filter((model) => model.key.host === host), + bindings: snapshot.bindings.filter((binding) => binding.host === host), + }; +} + +function selectedPair(store, positionals, flags) { + const fromId = flags.from ?? positionals[1]; + const toId = flags.to ?? positionals[2]; + const after = toId ? snapshotById(store, toId) : latestSnapshot(store); + const before = fromId ? snapshotById(store, fromId) + : previousSnapshot(store, after); + return { before, after, fromId, toId }; +} + +function noSnapshot(flags, cacheFile) { + const result = { status: 'empty', cacheFile, snapshot: null, hint: 'ak models refresh' }; + if (flags.json) printJson(result); + else { + heading('ak models — offline lifecycle inventory'); + info('No local model snapshot yet.'); + info('Refresh explicitly: ak models refresh'); + } + return 0; +} + +/** @param {{flags: Record, positionals: string[], deps?: Record}} input */ +export async function run({ flags, positionals, deps = {} }) { + const action = positionals[0] ?? 'status'; + const cacheFile = deps.cacheFile ?? modelInventoryPath(); + const readStore = deps.readStore ?? readModelStore; + const append = deps.append ?? appendModelSnapshot; + const collect = deps.collect ?? collectModelSnapshot; + const loadConfig = deps.loadConfig ?? loadKitConfig; + const cfg = loadConfig(); + + if (action === 'refresh') { + const owners = selectedOwners(flags, cfg); + const onlineContact = Boolean(flags.online && owners.includes('opencode')); + if (flags['dry-run']) { + const result = { dryRun: true, action, owners, online: onlineContact, + onlineRequested: flags.online, network: false, writes: false, cacheFile }; + if (flags.json) printJson(result); + else { + heading('ak models — refresh plan (dry-run)'); + info(`Would inspect: ${owners.join(', ')}.`); + info(onlineContact ? 'OpenCode catalog refresh would be permitted.' + : 'No online catalog refresh would be contacted.'); + info(dim('No source was contacted and no file was written.')); + } + return 0; + } + const aqeConfig = (deps.readJson ?? readJson)((deps.aqeFile ?? aqeRouterFile)(process.cwd())); + const snapshot = await collect({ + config: cfg, aqeConfig, rufloConfig: cfg, scope: { project: process.cwd() }, + discoveryOptions: { owners, online: flags.online, cwd: process.cwd() }, + }); + const store = append(snapshot, { file: cacheFile }); + const result = { status: 'refreshed', cacheFile, contacts: owners, online: onlineContact, + onlineRequested: flags.online, + snapshot: createModelReadModel(snapshot), retainedSnapshots: store.snapshots.length }; + if (flags.json) printJson(result); + else { + const health = summarizeModelHealth(snapshot); + ok(`Model inventory refreshed: ${snapshot.models.length} model(s) · ${snapshot.sources.length} source(s)`); + info(health.message); + info(dim(`private cache: ${cacheFile}`)); + } + return 0; + } + + const store = readStore({ file: cacheFile }); + const latest = latestSnapshot(store); + if (!latest) return noSnapshot(flags, cacheFile); + + if (action === 'status') { + if (flags.host && !ALL_OWNERS.includes(flags.host)) { + warn(`unsupported model host: ${flags.host}`); + return 2; + } + const snapshot = visibleSnapshot(latest, flags.host); + const since = flags.since ? Date.parse(flags.since) : null; + const history = store.snapshots.filter((entry) => entry.scope.fingerprint === latest.scope.fingerprint + && (!Number.isFinite(since) || Date.parse(entry.capturedAt) >= since)); + const result = { status: 'cached', cacheFile, health: summarizeModelHealth(snapshot), + inventory: createModelReadModel(snapshot), history: history.map(({ snapshotId, capturedAt }) => ({ snapshotId, capturedAt })) }; + if (flags.json) printJson(result); + else { + heading('ak models — offline lifecycle inventory'); + const health = result.health; + (health.level === 'ok' ? ok : warn)(health.message); + for (const source of snapshot.sources) info(`${source.id}: ${source.status} · ${source.capturedAt}`); + info(dim(`snapshot ${snapshot.snapshotId} · ${history.length} retained same-scope capture(s)`)); + } + return 0; + } + + if (action === 'diff') { + const { before, after, fromId, toId } = selectedPair(store, positionals, flags); + if (!before || !after) { + const missing = !before ? fromId ?? 'same-scope baseline' : toId ?? 'latest snapshot'; + if (flags.json) printJson({ comparable: false, reason: 'snapshot-not-found', missing }); + else warn(`Cannot diff: ${missing} not found.`); + return 1; + } + const result = diffSnapshotHistory(before, after, store.snapshots); + if (flags.json) printJson(result); + else { + heading(`ak models diff — ${before.snapshotId} → ${after.snapshotId}`); + if (!result.comparable) warn(result.diagnostics.join('; ')); + else if (!result.changes.length) ok('No model lifecycle changes.'); + else for (const change of result.changes) info(`${change.kind}: ${change.subject}${change.provisional ? ' (provisional)' : ''}`); + for (const message of result.diagnostics) info(dim(message)); + } + return result.comparable ? 0 : 1; + } + + if (action === 'explain') { + const selector = positionals[1] ?? flags.to; + if (!selector) { warn('usage: ak models explain HOST:MODEL'); return 2; } + const result = explainModel(latest, selector); + if (flags.json) printJson(result); + else if (!result.found) warn(`Model not found: ${selector}`); + else { + heading(`ak models explain — ${selector}`); + for (const match of result.matches) { + info(`${match.key.host}${match.key.provider ? `/${match.key.provider}` : ''}: ${match.key.modelId}`); + for (const [name, dimension] of Object.entries(match.dimensions)) info(` ${name}: ${dimension.value ?? 'unknown'}`); + info(` lifecycle: ${match.lifecycle.state}${match.lifecycle.replacement ? ` → ${match.lifecycle.replacement}` : ''}`); + } + } + return result.found ? 0 : 1; + } + + if (action === 'plan') { + const activity = flags.activity; + const to = flags.to ?? positionals[1]; + if (!activity || !to) { warn('usage: ak models plan --activity ACTIVITY [--from HOST:MODEL] --to HOST:MODEL'); return 2; } + const result = planModelChange(latest, { activity, from: flags.from, to }); + if (flags.json) printJson(result); + else { + heading(`ak models plan — ${activity}`); + if (!result.plannable) warn(`No mechanical plan: ${result.reason ?? result.compatibility?.blockers?.join('; ')}`); + else { + ok('Mechanical compatibility is supported by current evidence. Quality equivalence remains unknown.'); + info(`Copy to apply explicitly: ${result.action.command}`); + } + } + return result.plannable ? 0 : 1; + } + + warn('usage: ak models status|refresh|diff|explain|plan'); + return 2; +} diff --git a/src/commands/status.mjs b/src/commands/status.mjs index e785624..7458907 100644 --- a/src/commands/status.mjs +++ b/src/commands/status.mjs @@ -33,6 +33,7 @@ import { statuslineDrift } from '../lib/codex-statusline.mjs'; import { inspectCodexPlugins } from '../lib/codex-plugins.mjs'; import { projectMemoryStatus } from '../lib/project-memory.mjs'; import { removedAgentGaps, upstreamFixAvailable } from '../lib/scaffold.mjs'; +import { latestSnapshot, readModelStore, summarizeModelHealth } from '../lib/model-inventory/index.mjs'; export const options = { json: { type: 'boolean', default: false }, @@ -247,6 +248,19 @@ export async function collect({ pkgRoot, cwd = process.cwd() }) { const cfg = loadKitConfig(); const integrationFacts = await collectIntegrationFacts({ cwd, cfg }); + // Cache-only model lifecycle summary. Discovery and network access belong + // exclusively to `ak models refresh`. + try { + const snapshot = latestSnapshot(readModelStore()); + if (!snapshot) rows.push(row('models', 'info', 'no local model inventory yet; run `ak models refresh` explicitly')); + else { + const health = summarizeModelHealth(snapshot); + rows.push(row('models', health.level, health.message, health.fix)); + } + } catch (error) { + rows.push(row('models', 'warn', `model inventory unavailable: ${error.message}; run \`ak models refresh\` explicitly`)); + } + // versions try { for (const r of await driftReport()) { diff --git a/src/commands/sync.mjs b/src/commands/sync.mjs index 0e67494..03fbea3 100644 --- a/src/commands/sync.mjs +++ b/src/commands/sync.mjs @@ -73,6 +73,9 @@ export async function run({ flags, pkgRoot, fetchLatest }) { } const rows = await collect({ pkgRoot, cwd }); const plan = rows.filter((r) => r.fix) + // Model lifecycle actions are explicit advisory commands. `ak status` must + // name them, but sync neither refreshes catalogs nor applies model plans. + .filter((r) => r.subsystem !== 'models') .filter((r) => !(flags['no-upgrade'] && ['versions', 'self', 'ruvnet-brain', 'ruvector'].includes(r.subsystem))); if (plan.length === 0) { ok('nothing to do — all subsystems healthy'); return 0; } diff --git a/src/commands/uninstall.mjs b/src/commands/uninstall.mjs index 74ab88f..9e9569b 100644 --- a/src/commands/uninstall.mjs +++ b/src/commands/uninstall.mjs @@ -17,6 +17,7 @@ import { present as rbPresent } from '../lib/ruvnet-brain.mjs'; import * as paths from '../lib/paths.mjs'; import { ok, warn, fail, info } from '../lib/output.mjs'; import { removeCodexStatusline } from '../lib/codex-statusline.mjs'; +import { modelInventoryPath, modelScopeKeyPath } from '../lib/model-inventory/store.mjs'; /** Prints one lifecycle-render.mjs report line at its own level — mirrors * setup.mjs/sync.mjs's own printReportLine (N-2, Wave C security review @@ -175,6 +176,13 @@ export async function run({ flags }) { if (!flags.purge) saveKitConfig(cfg); for (const line of undoReport.lines) printReportLine(line); } + if (flags.purge) { + for (const [label, file] of [ + ['model inventory cache', modelInventoryPath()], ['model scope key', modelScopeKeyPath()], + ]) { + if (fs.existsSync(file)) act(`removed ${label}`, () => fs.rmSync(file)); + } + } if (flags.purge && fs.existsSync(paths.kitConfigPath())) { if (ownershipTeardownOk) act('removed kit.json', () => fs.rmSync(paths.kitConfigPath())); else warn('kit.json retained because OpenCode teardown is incomplete; it contains the recovery ownership receipt'); diff --git a/src/lib/adapters/registries.mjs b/src/lib/adapters/registries.mjs index 9c76fa0..ee010f2 100644 --- a/src/lib/adapters/registries.mjs +++ b/src/lib/adapters/registries.mjs @@ -81,6 +81,38 @@ export function validateObservabilityAdapter(value) { return immutable(structuredClone(value)); } +export function validateModelDiscoveryAdapter(value) { + assertRecord(value, 'modelDiscovery'); + const allowed = new Set(['id', 'ownerType', 'ownerId', 'transport', 'command', 'endpoint', 'network', 'schema', 'scope']); + for (const key of Object.keys(value)) { + if (!allowed.has(key)) throw new TypeError(`modelDiscovery has unknown field ${key}`); + } + assertId(value.id, 'modelDiscovery.id'); + assertEnum(value.ownerType, ['host', 'provider'], 'modelDiscovery.ownerType'); + assertId(value.ownerId, 'modelDiscovery.ownerId'); + assertEnum(value.transport, ['file', 'command', 'http'], 'modelDiscovery.transport'); + assertEnum(value.network, ['never', 'local', 'explicit'], 'modelDiscovery.network'); + assertEnum(value.scope ?? 'profile', ['profile', 'project', 'local'], 'modelDiscovery.scope'); + if (typeof value.schema !== 'string' || !value.schema) throw new TypeError('modelDiscovery.schema is required'); + if (value.transport === 'command') { + if (typeof value.command !== 'string' || !/^[A-Za-z0-9._-]+$/.test(value.command)) { + throw new TypeError('modelDiscovery.command must be an executable name'); + } + } else if (value.command !== undefined) { + throw new TypeError('non-command modelDiscovery cannot declare command'); + } + if (value.transport === 'http') { + let endpoint; + try { endpoint = new URL(value.endpoint); } catch { throw new TypeError('HTTP modelDiscovery.endpoint must be a URL'); } + if (endpoint.protocol !== 'http:' || !['127.0.0.1', 'localhost', '[::1]'].includes(endpoint.hostname)) { + throw new TypeError('HTTP modelDiscovery.endpoint must be loopback HTTP'); + } + } else if (value.endpoint !== undefined) { + throw new TypeError('non-HTTP modelDiscovery cannot declare endpoint'); + } + return immutable(structuredClone(value)); +} + export function validateHostAdapter(value, { projections, observability, } = /** @type {any} */ ({})) { @@ -152,6 +184,13 @@ const OBSERVABILITY_MAP = registryFrom([ { id: 'openrouter-metadata', kind: 'usage', evidence: ['provider', 'model', 'billing'] }, ], validateObservabilityAdapter, 'observability'); +const MODEL_DISCOVERY_MAP = registryFrom([ + { id: 'claude-config', ownerType: 'host', ownerId: 'claude', transport: 'file', network: 'never', scope: 'profile', schema: 'claude-settings-v1' }, + { id: 'codex-cache', ownerType: 'host', ownerId: 'codex', transport: 'file', network: 'never', scope: 'profile', schema: 'codex-model-cache-v1' }, + { id: 'opencode-models', ownerType: 'host', ownerId: 'opencode', transport: 'command', command: 'opencode', network: 'explicit', scope: 'project', schema: 'opencode-models-lines-v1' }, + { id: 'ollama-catalog', ownerType: 'provider', ownerId: 'ollama', transport: 'http', endpoint: 'http://127.0.0.1:11434', network: 'local', scope: 'local', schema: 'ollama-api-v1' }, +], validateModelDiscoveryAdapter, 'model discovery'); + const hostEntries = [ { id: 'claude', label: 'Claude Code', enabledByDefault: true, @@ -332,6 +371,7 @@ export const PROJECTION_REGISTRY = immutable(Object.values(PROJECTION_MAP)); // are cross-checked against this set) — deliberately NOT a dispatch surface; // no collector loop maps an observability id to a live collector (F-12). export const OBSERVABILITY_REGISTRY = immutable(Object.values(OBSERVABILITY_MAP)); +export const MODEL_DISCOVERY_REGISTRY = immutable(Object.values(MODEL_DISCOVERY_MAP)); export const HOST_REGISTRY = immutable(Object.values(HOST_MAP)); export const PROVIDER_REGISTRY = immutable(Object.values(PROVIDER_MAP)); @@ -346,6 +386,7 @@ const registryErrors = validateRegistries({ providers: Object.values(PROVIDER_MAP), projections: Object.values(PROJECTION_MAP), observability: Object.values(OBSERVABILITY_MAP), + modelDiscovery: Object.values(MODEL_DISCOVERY_MAP), }); if (registryErrors.length) { throw new Error(`adapter registries invalid: ${registryErrors.map((e) => `${e.path} (${e.code})`).join('; ')}`); @@ -406,7 +447,7 @@ export function validateActivityHost(id, hosts = HOST_REGISTRY) { export function validateRegistries(registries) { const errors = []; - const axes = ['hosts', 'providers', 'projections', 'observability']; + const axes = ['hosts', 'providers', 'projections', 'observability', 'modelDiscovery']; for (const axis of axes) { const seen = new Set(); for (const [index, entry] of (registries?.[axis] ?? []).entries()) { @@ -451,5 +492,13 @@ export function validateRegistries(registries) { }); } } + const hosts = new Set((registries?.hosts ?? []).map((entry) => entry.id)); + const providers = new Set((registries?.providers ?? []).map((entry) => entry.id)); + for (const [index, descriptor] of (registries?.modelDiscovery ?? []).entries()) { + const owners = descriptor.ownerType === 'host' ? hosts : providers; + if (!owners.has(descriptor.ownerId)) errors.push({ + path: `modelDiscovery[${index}].ownerId`, code: `unknown-${descriptor.ownerType}`, value: descriptor.ownerId, + }); + } return errors; } diff --git a/src/lib/dashboard-server.mjs b/src/lib/dashboard-server.mjs index be49050..34deb17 100644 --- a/src/lib/dashboard-server.mjs +++ b/src/lib/dashboard-server.mjs @@ -672,7 +672,7 @@ function lazyLive(liveOptions = {}) { * intelClientBuffer?: number, intelMaxClients?: number, * discoverProjects?: () => Array<{ path: string, label: string, source?: string }>, * machineWideIntel?: (projects: Array) => any, - * system?: any, systemOptions?: any }} [opts] + * models?: any, modelScopeKey?: string, system?: any, systemOptions?: any }} [opts] * @returns {Promise<{ url: string, urlWithToken: string, port: number, token: string, close: () => Promise }>} */ export function startDashboard({ @@ -681,10 +681,31 @@ export function startDashboard({ liveOptions = {}, liveIdleMs = 30_000, transcripts, transcriptOptions = {}, transcriptClientBuffer = 64, transcriptMaxClients = 16, intelWatch, intelClientBuffer = 256, intelMaxClients = 32, - discoverProjects, machineWideIntel, system, systemOptions = {}, + discoverProjects, machineWideIntel, models, modelScopeKey, system, systemOptions = {}, } = {}) { const provide = fetchStatus || shellOutStatus(cwd); const usageApi = usage || lazyUsage(); + // Cache-only and lazy: model discovery is exclusively owned by + // `ak models refresh`; opening the dashboard never contacts a host/catalog. + const provideModels = typeof models === 'function' ? models : models ? async () => models : async () => { + const [{ readModelStore, latestSnapshot, previousSnapshot }, { diffSnapshotHistory }, { createModelReadModel }] = await Promise.all([ + import('./model-inventory/store.mjs'), import('./model-inventory/diff.mjs'), + import('./model-inventory/read-model.mjs'), + ]); + const store = readModelStore(); + const snapshot = latestSnapshot(store); + if (!snapshot) return { status: 'empty', snapshot: null, history: [], hint: 'ak models refresh' }; + const baseline = previousSnapshot(store, snapshot); + const diff = baseline ? diffSnapshotHistory(baseline, snapshot, store.snapshots) + : { changes: [], diagnostics: [] }; + return { + status: 'cached', snapshot: createModelReadModel(snapshot, { changes: diff }), + history: store.snapshots.filter((entry) => entry.scope.fingerprint === snapshot.scope.fingerprint) + .map(({ snapshotId, capturedAt }) => ({ snapshotId, capturedAt })), + comparison: { baseline: baseline?.snapshotId ?? null, latest: snapshot.snapshotId, + comparable: diff.comparable ?? false, diagnostics: diff.diagnostics ?? [] }, + }; + }; // Injectable like `usage`: tests must never spawn a real codex or read the // real ~/.config through this route. Lazy for the same reason lazyUsage is. // enabledHosts drives quota.mjs's F-10 labeling (any OTHER enabled host with @@ -1370,6 +1391,42 @@ export function startDashboard({ // ── Usage (ADR-0009). Lazy: nothing below runs until the tab is opened. ── + if (url === '/api/models') { + try { + const payload = await provideModels(); + if (!payload || payload.status === 'empty' || !payload.snapshot) { + const { createDashboardModelViewPayload } = await import('./model-inventory/read-model.mjs'); + sendJson(res, 200, createDashboardModelViewPayload(payload, { query })); + return; + } + const [{ readModelScopeKey }, { createDashboardModelViewPayload }] = await Promise.all([ + import('./model-inventory/store.mjs'), import('./model-inventory/read-model.mjs'), + ]); + const key = modelScopeKey === undefined ? readModelScopeKey() : modelScopeKey; + if (!key) { + sendJson(res, 503, { error: 'model dashboard privacy key unavailable' }); + return; + } + const days = clampDays(query.get('days')); + let usage = null; + if (query.get('view') === 'summary' && typeof usageApi.readIndex === 'function') { + try { usage = await usageApi.readIndex({ days }); } + catch { usage = { unavailable: true, sessions: [] }; } + } + sendJson(res, 200, createDashboardModelViewPayload(payload, { + key, query, ...(usage ? { usage, days } : {}), + })); + } catch (error) { + // Do not echo native parser/provider errors: they may contain a private identifier. + const invalid = error?.code === 'INVALID_MODEL_INVENTORY_QUERY'; + const changed = error?.code === 'MODEL_INVENTORY_SNAPSHOT_CHANGED'; + sendJson(res, invalid ? 400 : changed ? 409 : 500, + { error: invalid ? 'invalid model inventory query' + : changed ? 'model inventory changed; retry' : 'model dashboard evidence unavailable' }); + } + return; + } + // Rollups only. Dropping the top-level sessions[] is NOT sufficient on its // own: projectTree[].rows holds the SAME object references, so every session // still shipped and the "order of magnitude" saving was really about 20%. diff --git a/src/lib/dashboard/client.mjs b/src/lib/dashboard/client.mjs index 2d82ce6..143ceaf 100644 --- a/src/lib/dashboard/client.mjs +++ b/src/lib/dashboard/client.mjs @@ -102,7 +102,7 @@ export const JS = ` var OVERVIEW_VIEWS=["summary","hosts","providers","runtime","intel"]; var SYSTEM_VIEWS=["summary","advisory","sessions","storage","runtime","catalog","projects"]; var ABOUT_SECTIONS=["hosts","engine","quality","kit","configured"]; - var VIEWS=["score","limits","findings","sessions","transcript"]; + var VIEWS=["score","limits","findings","models","sessions","transcript"]; var CAT=${CAT_JS}; ${catOf.toString()} @@ -776,6 +776,7 @@ export const JS = ` renderHistory(buildHistoryView(data)); renderRouting(data.routing); renderModels(data.routing); + renderModelSummary(data.rows||[]); positionThumb(); // badges can change segment widths } @@ -922,7 +923,10 @@ export const JS = ` var btn=document.getElementById("poll-now"); if(btn)btn.classList.add("spin"); var jobs=[pollStatus()]; - if(activeTab==="usage")jobs.push(loadUsage(true)); + if(activeTab==="usage"){ + jobs.push(loadUsage(true)); + if(usageView==="models")jobs.push(loadModelLifecycle(true)); + } // The Runtime view is a live census — processes, CPU, RSS, daemon ages — // and it used to load ONCE when the System tab was first opened, so its // "live" figures could sit unchanged for an entire session while the @@ -999,6 +1003,8 @@ export const JS = ` // ══ Usage tab ══════════════════════════════════════════════════════════════ var USAGE=null, usageLoaded=false, usageBusy=false, TRANSCRIPT=null; + var MODELS=null,MODEL_PAGE=null,modelRows=[],modelSnapshotId=null,modelsBusy=false,modelRequestSeq=0,modelSearchTimer=null; + var MODEL_LIMIT=50,modelSort="lifecycle",modelDirection="asc",modelRouteSort="model",modelRouteDirection="asc"; function fmtUsd(n){ n=Number(n)||0; @@ -1112,16 +1118,119 @@ export const JS = ` .then(function(d){TRANSCRIPT=d&&!d.error?{id:id,meta:d.meta,turns:d.turns||[]}:{id:id,error:(d&&d.error)||"unreadable"};}); } + function modelJson(url){ + return fetch(url,{cache:"no-store",headers:authHeaders()}).then(function(r){ + return r.json().then(function(d){ + if(!r.ok){var error=new Error(d&&d.error||"unavailable");error.status=r.status;throw error;} + return d; + }); + }); + } + + function modelFilters(){ + function value(id){var el=document.getElementById(id);return el?String(el.value||"").trim():"";} + var evidenceField=value("mli-evidence-field"),evidenceValue=value("mli-evidence-value"); + if(!evidenceField||!evidenceValue){evidenceField="";evidenceValue="";} + return {search:value("mli-search"),host:value("mli-host"),provider:value("mli-provider"), + relevance:value("mli-relevance")||"relevant", + lifecycle:value("mli-lifecycle"),evidenceField:evidenceField,evidenceValue:evidenceValue}; + } + + function modelInventoryUrl(offset){ + var query=new URLSearchParams({view:"inventory",offset:String(offset||0),limit:String(MODEL_LIMIT), + sort:modelSort,direction:modelDirection}),filters=modelFilters(); + if(modelSnapshotId)query.set("snapshotId",modelSnapshotId); + Object.keys(filters).forEach(function(key){if(filters[key])query.set(key,filters[key]);}); + return "/api/models?"+query.toString(); + } + + function setModelsBusy(busy,message){ + modelsBusy=busy; + var panel=document.getElementById("v-models"),status=document.getElementById("mli-load-status"),more=document.getElementById("mli-load-more"); + if(panel)panel.setAttribute("aria-busy",busy?"true":"false"); + if(more)more.disabled=busy; + if(status&&message)status.textContent=message; + } + + function loadModelInventory(offset,append,focusAfter){ + var seq=++modelRequestSeq,priorLength=modelRows.length; + setModelsBusy(true,append?"Loading more model evidence.":"Loading model inventory evidence."); + return modelJson(modelInventoryUrl(offset)).then(function(d){ + if(seq!==modelRequestSeq)return; + var nextSnapshotId=d&&d.snapshot&&d.snapshot.snapshotId||null; + if(modelSnapshotId&&nextSnapshotId!==modelSnapshotId){ + modelSnapshotId=null; + return loadModelLifecycle(true,focusAfter,true); + } + modelSnapshotId=nextSnapshotId; + MODEL_PAGE=d&&d.inventory||{}; + var items=Array.isArray(MODEL_PAGE.items)?MODEL_PAGE.items:[]; + modelRows=append?modelRows.concat(items):items; + renderModelInventory(); + renderModelFacets(); + setModelsBusy(false,"Model lifecycle evidence loaded. "+modelRows.length+" rows shown."); + var more=document.getElementById("mli-load-more");if(more)more.textContent="Load 50 more"; + if(focusAfter){ + var headers=document.querySelectorAll("#mli-models tr > th[scope=row]"); + var target=headers[append?priorLength:0],region=document.querySelector(".mli-table-wrap"); + if(append&&more&&!more.hidden)more.focus();else if(target)target.focus();else if(region)region.focus(); + } + }).catch(function(error){ + if(seq!==modelRequestSeq)return; + if(error&&error.status===409){ + modelSnapshotId=null; + setModelsBusy(true,"Model inventory changed; reloading its summary and first page."); + return loadModelLifecycle(true,focusAfter,true); + } + if(append){ + var more=document.getElementById("mli-load-more"); + if(more){more.hidden=false;more.disabled=false;more.textContent="Retry loading 50 more";more.focus();} + setModelsBusy(false,"More model lifecycle evidence is unavailable; prior rows were preserved."); + return; + } + MODEL_PAGE={items:[],total:0,filteredTotal:0,relevantTotal:0,offset:0,limit:MODEL_LIMIT,hasMore:false}; + modelRows=[];modelSnapshotId=null; + renderModelInventory(); + setModelsBusy(false,"Model lifecycle evidence is unavailable."); + }); + } + + function loadModelLifecycle(force,focusAfter,recovering){ + if(!recovering&&(modelsBusy||(!force&&MODELS)))return Promise.resolve(); + setModelsBusy(true,"Loading model lifecycle summary."); + return modelJson("/api/models?view=summary&days="+usageDays).then(function(d){ + MODELS=d; + modelSnapshotId=d&&d.snapshot&&d.snapshot.snapshotId||null; + renderModelLifecycle(); + if(!LIMITS)loadLimits(); + return loadModelInventory(0,false,!!focusAfter); + }).catch(function(){ + MODELS={error:"model inventory unavailable"};MODEL_PAGE=null;modelRows=[];modelSnapshotId=null; + renderModelLifecycle();renderModelInventory(); + setModelsBusy(false,"Model lifecycle evidence is unavailable."); + }); + } + function setUsageView(v,session){ usageView=v; if(session!==undefined)usageSession=session; - var headings={score:["Usage scorecard","Token consumption, API-equivalent cost, efficiency, and trends."],limits:["Provider limits","Current provider windows, reset timing, and available capacity."],findings:["Usage findings","Actionable anomalies, efficiency opportunities, and evidence-backed recommendations."],sessions:["Session usage","Browse retained sessions by project, category, duration, tokens, and cost."],transcript:["Transcript detail","Inspect the selected session's locally retained, server-masked evidence."]},heading=headings[v]||headings.score; + var headings={score:["Usage scorecard","Token consumption, API-equivalent cost, efficiency, and trends."],limits:["Provider limits","Current provider windows, reset timing, and available capacity."],findings:["Usage findings","Actionable anomalies, efficiency opportunities, and evidence-backed recommendations."],sessions:["Session usage","Browse retained sessions by project, category, duration, tokens, and cost."],models:["Models","Observed models in this window, configured routes, and the separate provider/local catalogue."],transcript:["Transcript detail","Inspect the selected session's locally retained, server-masked evidence."]},heading=headings[v]||headings.score; document.getElementById("usage-view-title").textContent=heading[0];document.getElementById("usage-view-description").textContent=heading[1]; var btns=document.querySelectorAll("#usage-seg [data-view]"); - for(var i=0;i