Skip to content

docs: document LLM-as-judge vs Granite Switch latency overhead with real numbers #1498

Description

@planetf1

The README implies Mellea is a drop-in improvement without mentioning the very different latency profiles of the two requirement-checking modes. This causes surprises in production.

Evidence from eval-mFG (70 rows, Granite 8B on H100):

Task category LLMaJ Mellea overhead Switch Mellea overhead
constrained-format 3.2× native 0.8× native
structured-output 3.7× native 0.8× native
factual-hard 3.4× native 0.9× native
dual-constraint 0.9× native 0.9× native
reasoning 0.5× native 1.5× native

Switch is consistently faster because requirement checks are adapter forward passes (milliseconds). LLMaJ overhead ranges from 0.5× to 3.7× depending on whether the first draft passes (fast) or needs a revision — a full LLM judge call per requirement, per round.

What good docs look like:

## Requirement checking modes

### Granite Switch (aLoRA adapters) — recommended for production
Requirement checks run as adapter forward passes inside the Switch model.
Latency overhead: ~1.1× native (adapter check adds milliseconds, not seconds).
Requirements: GRANITE_SWITCH_API_BASE must point at a granite-switch endpoint.

### LLM-as-Judge (default, no extra infra)
Each requirement check is a full LLM call (~4-8s on a loaded H100).
With 3 requirements and 1 revision budget: up to 7 LLM calls per request vs
1-2 for native inference. Observed overhead: 0.5×-4× native depending on
first-draft pass rate.
Use this mode for: local development, CI, environments without a Switch server.

This information would help users choose the right mode at design time rather than discovering the latency cliff in production.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationp2Medium/low: minor bugs, niche features, polish, docs, tests, cleanup. Scoped, lower urgency.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions