Harden provider contracts for Worker-backed agent routing

[stackbilt-admin]

Intent

Harden @stackbilt/llm-providers as the reusable provider/capability layer behind a Cloudflare Worker API router for coding-agent clients.

The Worker should be able to delegate model choice, structured-output handling, tool capability checks, failover, cache hints, and provider request/response normalization through public library contracts.

Use Case

The downstream gateway wants to expose Ollama/OpenAI/Anthropic-compatible endpoints while routing each request to the best available model:

• cheap JSON/structured-output model
• tool-safe model
• code model
• Workers AI LoRA reviewer/critic model
• frontier fallback
• local/remote provider bridge

Contract Surfaces To Review

• canonical request/response shape
• route preflight/introspection
• cache hint semantics
• provider/model capability metadata for JSON, tools, streaming, multimodal, LoRA, and code tasks
• fallback and degradation explanations
• response normalization for compatibility endpoints

Acceptance Criteria

• A Worker/router can ask for route selection using a canonical request without provider-specific branching.
• Capability metadata is expressive enough to choose models for JSON, tools, code, structured output, and LoRA-backed reviewer paths.
• Cache hints are concrete enough for a Worker implementation using KV/Cache API/D1/R2 without baking Cloudflare storage into the library.
• Route inspection returns enough explanation for a gateway debug endpoint.
• Public docs include a gateway/router integration example that avoids private Stackbilt-only assumptions.

[stackbilt-admin]

Implemented the first additive slice for this issue in the local checkout:

• added getGatewayRoutePlan() as a Worker/gateway-facing preflight helper
• route plans include canonical + legacy request forms, selected provider/model, route info, requirements, capability report, cache plan, degradations, and warnings
• LoRA requests now surface an explicit degradation when routed away from Cloudflare, and a warning when forwarded to Workers AI
• cache planning stays storage-agnostic so a Worker can map it to KV, Cache API, D1, R2, or another store
• exported the helper/types from the public package entrypoint
• documented the Worker gateway route-plan pattern in the README

Validation:

npx vitest run src/__tests__/gateway-routing.test.ts
# 1 file passed, 5 tests passed

npm run typecheck
# tsc --noEmit passed

npm test
# 21 files passed, 441 tests passed

This does not implement the llm-gateway Worker router itself; it gives that router a stable provider-contract preflight surface to consume.

[stackbilt-admin]

Commit is present locally on main:

5a576b7 feat: add Worker gateway route plan helper

State from the local checkout:

main...origin/main [ahead 1]
.agent-memory/ remains untracked and intentionally excluded

Commit summary:

CHANGELOG.md                          |  10 ++
README.md                             |  22 ++++
src/__tests__/gateway-routing.test.ts | 104 +++++++++++++++++
src/gateway-routing.ts                | 214 ++++++++++++++++++++++++++++++++++
src/index.ts                          |   9 ++
5 files changed, 359 insertions(+)

Next handoff step: push 5a576b7 or open a PR from this commit. If another machine opens the PR, it should either push this local commit first or rebase/recreate the same patch.

[stackbilt-admin]

Downstream dogfood finding: route-plan contract needs circuit/fallback semantics

Controlled test date: 2026-06-07

Downstream llm-gateway now consumes the route-plan/preflight surface for /routes/inspect, and AEGIS dogfooded it by launching native Codex from the Tarotscript repo.

The transport path works, but the dogfood exposed a contract boundary worth considering for this issue:

• /routes/inspect for stackbilt/code-fast selected:

{
  "routeClass":"code_draft",
  "selectedProvider":"cerebras",
  "selectedModel":"openai/gpt-oss-120b",
  "fallbackChain":["cerebras","groq","nvidia","cloudflare"]
}

• Full Codex tool-loop execution then failed with:

404 MODEL_NOT_FOUND: Model 'unknown' not found
502 CIRCUIT_BREAKER_OPEN: Circuit breaker rejected request for provider: cerebras.
502 ALL_PROVIDERS_FAILED: All providers failed

• Explicit Groq aliases worked through the same gateway/Responses/Codex path.

This may mostly be gateway-side (#2/#4), but from the provider-contract side the downstream need is:

• Route preflight should be able to distinguish "catalog-preferred" from "currently executable" when circuit-breaker/health state is known.
• If circuit state is not available to the pure helper, the contract should make that explicit and let gateways pass health/circuit exclusions into route planning.
• Route plans should expose whether the selected provider is tentative, degraded, fallback-selected, or blocked by known circuit state.

Suggested acceptance addendum for this issue:

• Document whether getGatewayRoutePlan() is static/catalog-only or health/circuit-aware.
• If static-only, add an option for downstream gateways to pass excluded/unavailable providers so /routes/inspect can match runtime dispatch.
• Add a degradation/warning shape for "preferred provider unavailable; selected fallback provider X".
• Include a gateway example where first route candidate is circuit-open and the route plan selects the next compatible provider.

Related downstream evidence:

• Stackbilt-dev/llm-gateway#2 comment: AEGIS/Tarotscript dogfood route selection + failover gaps
• Stackbilt-dev/llm-gateway#4 comment: Codex Responses adapter quality gaps
• Stackbilt-dev/aegis#635: coordination issue for the cross-repo dogfood matrix

[stackbilt-admin]

Cross-repo status update from the gateway/model-quality lane:

• @stackbilt/llm-providers v1.14.1 is published and llm-gateway main has consumed it via PR fix: default fallback chain excludes Cerebras and Groq providers #17.
• llm-gateway PR feat: structured logger interface — replace 35+ console.* calls #18 is now merged and emits classified compatibility failures through /metrics.byCompatibilityFailure.
• The observed no-MCP eval failures now have labels: malformed_tool_call_json, empty_successful_output, unknown_model_retry, provider_circuit_open, and all_providers_failed.

Remaining provider-side work for this issue is narrower now: route-plan health/circuit semantics and provider capability metadata should help prevent the bad routes before runtime. Gateway metrics can now prove whether each provider slice improves the matrix.

[stackbilt-admin]

AEGIS-side credit-backed Workers AI routing is now wired against @stackbilt/llm-providers@^1.14.1 in aegis-daemon.

Relevant downstream expectations from that integration:

• CloudflareProvider must keep honoring LLMRequest.cache = { strategy: 'provider-prefix', sessionId } by forwarding x-session-affinity into env.AI.run() options.
• CloudflareProvider gateway config should remain stable for binding calls: gateway: { id, metadata, cacheKey, cacheTtl, skipCache }.
• Model catalog should expose enough capability tags for AEGIS/llm-gateway route policy: reasoning, function calling, vision, embedding, audio, image, long-context, cheap, fast.
• Usage normalization should preserve cachedInputTokens so downstream cost reports can prove prefix-cache savings.

AEGIS now passes session affinity and route metadata for scheduled/provider-backed Workers AI calls, direct GPT-OSS tool loops, simple Workers AI responses, and composite gather/fallback paths. This issue remains the right home for hardening that as a public downstream contract.

[HerShin5]

This looks like a good place to make the route plan explicitly distinguish static capability selection from runtime availability.

A contract shape I’d expect a Worker/router to need:

• candidate_routes: ordered static matches with provider/model/capabilities and why each matched.
• excluded_routes: provider/model plus reason (circuit_open, rate_limited, credential_missing, health_probe_failed, capability_mismatch, etc.).
• selected_route: the final executable route after applying health/circuit/credential exclusions.
• fallback_reason: present when the catalog-preferred route was not used.
• compatibility_risks: warnings like tools_supported_but_unverified, json_mode_emulated, cache_affinity_provider_specific.

For cache/session affinity, I’d keep the storage mapping outside the provider library as you suggested, but make the effective cache intent observable in the route plan: requested strategy, provider-specific transport field/header, TTL/skipped state, and whether usage normalization can report cache reads/writes. Otherwise a route can “work” while silently losing the cost-control property the router selected it for.

Disclosure: I’m associated with/evaluating KiroForge (https://kiroforge.cloud/v1), an OpenAI-compatible gateway. From the gateway side, this route-plan surface is useful not just for dispatch but for user support: when a tool-loop fails, you want to know whether it was a bad model choice, a circuit/fallback decision, or a provider capability mismatch before the request hit the model.

[stackbilt-admin]

Update 2026-06-09: pushed commit 273b314 to main for the Workers AI catalog expansion. It prepares @stackbilt/llm-providers@1.14.2 with active Cloudflare entries for @cf/moonshotai/kimi-k2.6, @cf/zai-org/glm-4.7-flash, and dashboard slug deepseek/deepseek-v4-pro. Kimi/GLM are marked tool-capable; DeepSeek is added conservatively as high-performance/balanced until a public/runtime tool-call proof is available.\n\nValidation completed locally:\n- npm test: 442 passing\n- npm run typecheck: clean\n- npm run build: clean\n- npm run test:package: package smoke passed\n\nPublish blocker: npm registry auth is not available in this shell (npm whoami returns 401), so 1.14.2 is committed/pushed but not published to npm yet.

[stackbilt-admin]

Update 2026-06-10: @stackbilt/llm-providers@1.14.2 is now published and live.

Release/publish state:

• Tag/release: v1.14.2 from commit 273b314
• npm latest now resolves to @stackbilt/llm-providers@1.14.2 with provenance
• Pre-flight checks reported clean: 442/442 tests, typecheck, package smoke

Downstream consumption/proof:

• llm-gateway main consumes 1.14.2 in commit 5246fe8.
• /models exposes the new Cloudflare aliases: cloudflare/@cf/moonshotai/kimi-k2.6, cloudflare/@cf/zai-org/glm-4.7-flash, and cloudflare/deepseek/deepseek-v4-pro.
• Direct gateway smoke for Kimi succeeded through /v1/responses.
• AEGIS launcher now has operator-facing selection paths: Codex receives the injected gateway model catalog for its native model UI, while Claude Code gets an AEGIS pre-launch gateway model selector when run interactively with no explicit --model.

This supersedes the earlier publish-blocked note for 1.14.2. Remaining work in this issue is still the broader provider-contract hardening: route-plan health/circuit semantics, richer capability/risk metadata, and cache/cost observability contracts.

[stackbilt-admin]

Current review: keep this open.

Verified completed slice:

• getGatewayRoutePlan() exists and is exported as the Worker/gateway-facing preflight helper.
• The route plan includes canonical + legacy request forms, selected provider/model, routing info, requirements, capability report, cache plan, degradations, and warnings.
• README documents the Worker gateway route-plan pattern.
• Focused verification passed: npx vitest run src/__tests__/gateway-routing.test.ts (5 tests).

What remains from the later comments / broader acceptance criteria:

• Route plans do not yet expose candidate_routes, excluded_routes, selected_route, fallback_reason, or explicit compatibility/risk categories.
• getGatewayRoutePlan() can accept ModelSelectionContext, and catalog ranking can account for provider health / circuit state, but the returned plan does not explain which providers/models were excluded or why.
• Capability reporting is still coarse: streaming, tools, vision, prompt cache, built-in tools, and context length. It does not yet cover the fuller router matrix called out here, such as code-task suitability, native JSON/schema support, LoRA verification, credential/rate-limit state, or cache/cost observability.

Recommended next pass: add an explanatory route-inspection shape for candidate/excluded routes, fallback/circuit semantics, and richer compatibility-risk metadata while preserving the existing additive getGatewayRoutePlan() contract.