From bb17452dfe6745c0c644ac78d005bfe88d438dc2 Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Fri, 17 Jul 2026 17:10:44 -0700 Subject: [PATCH 1/4] docs(models): local-development setup matching production MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds reference/models/local-development.md — the last unwritten docs deliverable of HarperFast/harper#510's acceptance (custom backends and fallback landed in #554/#558): logical-name portability, a local Ollama config with its matching production config, parity caveats (tools capability, embedding-model compatibility), per-environment config via HARPER_CONFIG, and offline/CI stubs via registerBackend. Closes #596. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 105 ++++++++++++++++++++++++++ reference/models/overview.md | 2 +- sidebarsReference.ts | 5 ++ 3 files changed, 111 insertions(+), 1 deletion(-) create mode 100644 reference/models/local-development.md diff --git a/reference/models/local-development.md b/reference/models/local-development.md new file mode 100644 index 00000000..19f89cc6 --- /dev/null +++ b/reference/models/local-development.md @@ -0,0 +1,105 @@ +--- +id: local-development +title: Local Development +--- + + + + + +The models API is designed so the same application code runs unchanged against local models during development and hosted providers in production. Application code addresses models by [logical name](./overview#configuration); which physical backend a logical name resolves to is decided by each instance's configuration. Setting up local development is therefore a configuration exercise: give the logical names your application uses a backend you can run on your own machine — no API keys, no network egress, no code changes. + +## A local configuration + +[Ollama](https://ollama.com) is the shortest path to local models: install it, pull an embedding model and a generation model, and point the default logical names at it. The [`ollama` backend](./backends#ollama) needs no credentials. + +```bash +ollama pull nomic-embed-text +ollama pull llama3.2 +``` + +```yaml +# harper-config.yaml (development) +models: + embedding: + default: + backend: ollama + host: localhost:11434 + model: nomic-embed-text:latest + generative: + default: + backend: ollama + host: localhost:11434 + model: llama3.2 +``` + +The matching production configuration maps the same logical names to hosted providers, with credentials supplied by [environment-variable indirection](./overview#credentials): + +```yaml +# harper-config.yaml (production) +models: + embedding: + default: + backend: openai + apiKey: ${OPENAI_API_KEY} + model: text-embedding-3-small + generative: + default: + backend: openai + apiKey: ${OPENAI_API_KEY} + model: gpt-4o +``` + +Application code is identical in both environments: + +```javascript +import { models } from 'harper'; + +const [vector] = await models.embed('What is Harper?'); +const reply = await models.generate('Summarize this record.'); +``` + +The [`@embed` schema directive](../database/schema#embed), [tool calling](./tool-calling), [fallback groups](./routing#fallback-groups), and [analytics](./analytics) all address models the same way, so they carry across the swap too — subject to the parity caveats below. + +## Parity caveats + +Two things do not automatically carry across a backend swap: + +**Tool support.** The `ollama` backend does not advertise the `tools` capability, so a `generate()` call that declares tools fails up front against it (see [Backends](./backends#ollama)). If your application uses [tool calling](./tool-calling), run an OpenAI-compatible local server (such as vLLM) and select it with the [`openai` backend's `baseUrl` field](./backends#openai) instead — the local backend then advertises tools the way production does. + +**Embedding compatibility.** Embedding vectors are only comparable within a single model's vector space, and models differ in dimensionality. Vectors written locally with one embedding model cannot be meaningfully searched against vectors produced in production by another — which matters whenever embedded data, or an [HNSW index](../database/schema#vector-indexing) built from [`@embed`](../database/schema#embed) vectors, moves between environments. Where embedded data crosses environments, use the same embedding model in both — for example, an open model served by Ollama locally and by an OpenAI-compatible host in production — or re-embed after the move. + +## Per-environment configuration + +Each Harper instance reads its own configuration file, so the simplest arrangement is a development config with local backends and a production config with hosted ones. Where baking a config file into an image is awkward — containerized or orchestrated deployments — the `models` block can be overridden from the environment with [`HARPER_CONFIG`](../configuration/overview#harper_config), which merges exactly the keys it names over the config file: + +```bash +export HARPER_CONFIG='{"models":{"generative":{"default":{"backend":"openai","apiKey":"${OPENAI_API_KEY}","model":"gpt-4o"}}}}' +``` + +The `${OPENAI_API_KEY}` placeholder is resolved by Harper at startup, not by the shell — the single quotes are deliberate. + +While iterating on configuration, remember the [startup behavior](./overview#startup-behavior) split: a structurally invalid entry fails configuration validation and prevents startup, while a registration-time error — such as an unreachable backend module — is logged and skipped, and calls to that logical name then throw as unconfigured. If a model works in one environment and throws "not configured" in another, check the startup log for a skipped registration. + +## Offline and CI stubs + + + +When tests must run with no model server at all, register an in-process stub as a [custom backend](./backends#custom-backends) under the logical names the application uses. A registered backend shadows a configuration entry with the same name, so the stub wins regardless of what the config file says: + +```javascript +import { models } from 'harper'; + +models.registerBackend( + 'generative', + 'default', + models.defineBackend({ + name: 'stub', + async generate() { + return { status: 'completed', output: { content: 'stub reply', finishReason: 'stop' } }; + }, + }) +); +``` + +The stub still exercises routing, accounting, and [analytics](./analytics) — only the inference itself is faked. diff --git a/reference/models/overview.md b/reference/models/overview.md index ac514b44..53f80b04 100644 --- a/reference/models/overview.md +++ b/reference/models/overview.md @@ -7,7 +7,7 @@ title: Models -Harper provides a unified API for calling AI models — text embeddings and text generation — from application code. Models are configured by an operator under logical names; application code requests a model by its logical name and Harper routes the call to the configured backend (Ollama, OpenAI, Anthropic, or Amazon Bedrock) — or to a [custom backend](./backends#custom-backends) a component registers. Swapping providers is a configuration change, not a code change. A logical name can also name an ordered group of backends to try, and calls can require specific capabilities — see [Routing & Fallback](./routing). +Harper provides a unified API for calling AI models — text embeddings and text generation — from application code. Models are configured by an operator under logical names; application code requests a model by its logical name and Harper routes the call to the configured backend (Ollama, OpenAI, Anthropic, or Amazon Bedrock) — or to a [custom backend](./backends#custom-backends) a component registers. Swapping providers is a configuration change, not a code change — [Local Development](./local-development) uses this to run the same application against local models in development and hosted providers in production. A logical name can also name an ordered group of backends to try, and calls can require specific capabilities — see [Routing & Fallback](./routing). The API is exposed as a single process-wide `models` object: diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 63957800..8db4a72b 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -125,6 +125,11 @@ const sidebars: SidebarsConfig = { id: 'models/analytics', label: 'Analytics', }, + { + type: 'doc', + id: 'models/local-development', + label: 'Local Development', + }, ], }, { From d7c8e8780817400e07836aa72e0fa382dd368509 Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Fri, 17 Jul 2026 17:30:13 -0700 Subject: [PATCH 2/4] docs(models): HARPER_CONFIG cannot swap backends; stub scoping notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review findings: HARPER_CONFIG's leaf merge cannot remove keys, so switching an entry's backend via partial override leaves the old backend's fields in place and fails startup validation — teach supply-whole-block or same-backend field overrides instead, covering both embedding and generative maps. Stub section: generateStream is not synthesized from generate, and a generative stub does not serve embed(). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/reference/models/local-development.md b/reference/models/local-development.md index 19f89cc6..49131ec6 100644 --- a/reference/models/local-development.md +++ b/reference/models/local-development.md @@ -71,14 +71,16 @@ Two things do not automatically carry across a backend swap: ## Per-environment configuration -Each Harper instance reads its own configuration file, so the simplest arrangement is a development config with local backends and a production config with hosted ones. Where baking a config file into an image is awkward — containerized or orchestrated deployments — the `models` block can be overridden from the environment with [`HARPER_CONFIG`](../configuration/overview#harper_config), which merges exactly the keys it names over the config file: +Each Harper instance reads its own configuration file, so the simplest arrangement is a development config with local backends and a production config with hosted ones. Where baking a config file into an image is awkward — containerized or orchestrated deployments — the `models` block can be supplied from the environment with [`HARPER_CONFIG`](../configuration/overview#harper_config), which merges exactly the keys it names over the config file: ```bash -export HARPER_CONFIG='{"models":{"generative":{"default":{"backend":"openai","apiKey":"${OPENAI_API_KEY}","model":"gpt-4o"}}}}' +export HARPER_CONFIG='{"models":{"embedding":{"default":{"backend":"openai","apiKey":"${OPENAI_API_KEY}","model":"text-embedding-3-small"}},"generative":{"default":{"backend":"openai","apiKey":"${OPENAI_API_KEY}","model":"gpt-4o"}}}}' ``` The `${OPENAI_API_KEY}` placeholder is resolved by Harper at startup, not by the shell — the single quotes are deliberate. +Because the merge is key-by-key and cannot remove keys, use `HARPER_CONFIG` to supply a `models` block the config file does not define (as above), or to override fields within an entry that keeps the same backend — a model name, a credential. Do not use it to switch an existing entry to a different backend: keys the override does not name stay in place, and a leftover field from the old backend — `host` from an `ollama` entry, say — is an unrecognized field on the new backend's entry, which [fails configuration validation](./overview#startup-behavior) and prevents startup. To swap backends between environments, swap the config file (or define every model entry through `HARPER_CONFIG` and keep the file's `models` block empty), and cover both the `embedding` and `generative` maps — an override that names only one leaves the other pointed wherever the file pointed it. + While iterating on configuration, remember the [startup behavior](./overview#startup-behavior) split: a structurally invalid entry fails configuration validation and prevents startup, while a registration-time error — such as an unreachable backend module — is logged and skipped, and calls to that logical name then throw as unconfigured. If a model works in one environment and throws "not configured" in another, check the startup log for a skipped registration. ## Offline and CI stubs @@ -103,3 +105,5 @@ models.registerBackend( ``` The stub still exercises routing, accounting, and [analytics](./analytics) — only the inference itself is faked. + +Two scoping notes. `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. And a backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. From 7455458bcefd2b2e0b15074fabb932263e315130 Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Fri, 17 Jul 2026 17:31:19 -0700 Subject: [PATCH 3/4] =?UTF-8?q?docs(models):=20stubs=20register=20per=20wo?= =?UTF-8?q?rker=20thread=20=E2=80=94=20register=20in=20component=20init?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses gemini-code-assist inline feedback: the backend registry is per worker thread, so a test-file top-level registration may not reach the threads serving requests. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/models/local-development.md b/reference/models/local-development.md index 49131ec6..9ea2e31e 100644 --- a/reference/models/local-development.md +++ b/reference/models/local-development.md @@ -107,3 +107,5 @@ models.registerBackend( The stub still exercises routing, accounting, and [analytics](./analytics) — only the inference itself is faked. Two scoping notes. `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. And a backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. + +As with any registered backend, register the stub during component initialization (for example, in `handleApplication`) rather than at a test file's top level: each worker thread keeps its own registry, so registration must run in every thread that serves requests — see [`registerBackend()`](./backends#registerbackend). From edaa988e7961726b0799bced3ed769cbd6d61dd7 Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Tue, 21 Jul 2026 08:34:43 -0700 Subject: [PATCH 4/4] Make the vLLM tool-calling workaround concrete MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review: the openai backend advertising tools does not make a default vLLM server tool-capable. Spell out both requirements — vLLM needs a tool-capable model with --enable-auto-tool-choice and a matching --tool-call-parser, and Harper's openai backend needs a non-empty apiKey even against an unauthenticated local server — with a runnable serve command and matching YAML. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/reference/models/local-development.md b/reference/models/local-development.md index 9ea2e31e..5ee106e2 100644 --- a/reference/models/local-development.md +++ b/reference/models/local-development.md @@ -65,7 +65,25 @@ The [`@embed` schema directive](../database/schema#embed), [tool calling](./tool Two things do not automatically carry across a backend swap: -**Tool support.** The `ollama` backend does not advertise the `tools` capability, so a `generate()` call that declares tools fails up front against it (see [Backends](./backends#ollama)). If your application uses [tool calling](./tool-calling), run an OpenAI-compatible local server (such as vLLM) and select it with the [`openai` backend's `baseUrl` field](./backends#openai) instead — the local backend then advertises tools the way production does. +**Tool support.** The `ollama` backend does not advertise the `tools` capability, so a `generate()` call that declares tools fails up front against it (see [Backends](./backends#ollama)). If your application uses [tool calling](./tool-calling), run an OpenAI-compatible local server with tool calling enabled and select it with the [`openai` backend's `baseUrl` field](./backends#openai). Two things have to be true for this to work. + +First, the server itself must have tool calling switched on. The `openai` backend always advertises the `tools` capability, so Harper's up-front capability check passes regardless of what the server can actually do — a server that cannot parse tool calls fails at request time instead. With [vLLM](https://docs.vllm.ai), that means serving a tool-capable model with automatic tool choice enabled and the tool-call parser that matches the model family — see [vLLM's tool-calling guide](https://docs.vllm.ai/en/latest/features/tool_calling.html) for the model–parser pairings: + +```bash +vllm serve Qwen/Qwen2.5-7B-Instruct --enable-auto-tool-choice --tool-call-parser hermes +``` + +Second, the `openai` backend requires a non-empty `apiKey` even when the local server does not authenticate — supply any placeholder. Harper's startup warning about a literal value in a credential field is expected and harmless for a local endpoint: + +```yaml +models: + generative: + default: + backend: openai + baseUrl: http://localhost:8000/v1 + apiKey: local-dev + model: Qwen/Qwen2.5-7B-Instruct +``` **Embedding compatibility.** Embedding vectors are only comparable within a single model's vector space, and models differ in dimensionality. Vectors written locally with one embedding model cannot be meaningfully searched against vectors produced in production by another — which matters whenever embedded data, or an [HNSW index](../database/schema#vector-indexing) built from [`@embed`](../database/schema#embed) vectors, moves between environments. Where embedded data crosses environments, use the same embedding model in both — for example, an open model served by Ollama locally and by an OpenAI-compatible host in production — or re-embed after the move.