The README implies Mellea is a drop-in improvement without mentioning the very different latency profiles of the two requirement-checking modes. This causes surprises in production.
Evidence from eval-mFG (70 rows, Granite 8B on H100):
| Task category |
LLMaJ Mellea overhead |
Switch Mellea overhead |
| constrained-format |
3.2× native |
0.8× native |
| structured-output |
3.7× native |
0.8× native |
| factual-hard |
3.4× native |
0.9× native |
| dual-constraint |
0.9× native |
0.9× native |
| reasoning |
0.5× native |
1.5× native |
Switch is consistently faster because requirement checks are adapter forward passes (milliseconds). LLMaJ overhead ranges from 0.5× to 3.7× depending on whether the first draft passes (fast) or needs a revision — a full LLM judge call per requirement, per round.
What good docs look like:
## Requirement checking modes
### Granite Switch (aLoRA adapters) — recommended for production
Requirement checks run as adapter forward passes inside the Switch model.
Latency overhead: ~1.1× native (adapter check adds milliseconds, not seconds).
Requirements: GRANITE_SWITCH_API_BASE must point at a granite-switch endpoint.
### LLM-as-Judge (default, no extra infra)
Each requirement check is a full LLM call (~4-8s on a loaded H100).
With 3 requirements and 1 revision budget: up to 7 LLM calls per request vs
1-2 for native inference. Observed overhead: 0.5×-4× native depending on
first-draft pass rate.
Use this mode for: local development, CI, environments without a Switch server.
This information would help users choose the right mode at design time rather than discovering the latency cliff in production.
The README implies Mellea is a drop-in improvement without mentioning the very different latency profiles of the two requirement-checking modes. This causes surprises in production.
Evidence from eval-mFG (70 rows, Granite 8B on H100):
Switch is consistently faster because requirement checks are adapter forward passes (milliseconds). LLMaJ overhead ranges from 0.5× to 3.7× depending on whether the first draft passes (fast) or needs a revision — a full LLM judge call per requirement, per round.
What good docs look like:
This information would help users choose the right mode at design time rather than discovering the latency cliff in production.