Skip to content

OrcaRouter provider surface for aimock (mockable, OpenAI-compatible) #414

Description

@bangla24bdrang-lab

aimock mocks everything an AI app talks to — one package, one local port, zero dependencies. For teams building on LLMs that removes the flakiest part of the test suite: no keys, no network, no surprise bills, while record-and-replay captures a real provider interaction once and replays it forever with recorded timing; AG-UI already runs its end-to-end suite on aimock.

What makes the model coverage valuable is its breadth: LLMock models 13 providers across 15 API surfaces — OpenAI Chat/Responses, Claude, Gemini, and OpenRouter with router failover — each reached through a base-URL override plus an opt-in provider key for record mode. Since a provider is just a selectable surface, aimock tests the same configuration you run in production, which is why an OpenAI-compatible endpoint fits naturally.

Proposal

I propose adding OrcaRouter as an optional, mockable LLMock surface — alongside, never replacing, any existing provider. Apps configured to route through OrcaRouter would then get deterministic fixtures, record-and-replay, and chaos-testing control for that setup. Three OrcaRouter traits matter most to aimock users:

  • One OpenAI-compatible endpoint across chat, reasoning, image, and video models — a single base URL, exactly what aimock is built to mock.
  • Automatic model routing with provider failover — like the OpenRouter "router" behavior aimock already simulates, so tests can script fallback and failure paths.
  • Usage tracking with per-request cost — pairs with aimock's streaming usage chunks for suites asserting on provider-reported usage.

Integration point

OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, so the natural hook-in matches aimock's existing pattern for OpenAI-compatible providers: an LLMock surface, a record-mode upstream target, and an opt-in provider key, in the spirit of today's OpenRouter support. This is a proposal only — nothing is implemented or tested yet — and I'd value maintainer input before writing any code.

Ecosystem and disclosure

OrcaRouter is already an option in open-source AI tooling such as Dify, promptfoo, and goose, and our built-with page lists more. For full transparency: we run an optional open-source partner program where approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participation is not a prerequisite for provider support, and I'll follow whatever disclosure or governance requirements aimock prefers.

I'm an engineer on the OrcaRouter team. If a mockable OrcaRouter surface sounds useful, I'd appreciate your thoughts on where it should hook in, and I'm happy to submit an implementation PR once approved. Thanks for building a tool that makes provider-dependent testing this painless.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions