GBNF / constrained decoding for LLM generate()?

[vinceecws]

Summary

We're running LFM2.5-VL-1.6B on-device via react-native-executorch 0.9.3 (iOS / MLX). We'd like to confirm that RNE does not support GBNF (or any logits-level constrained decoding) on LLMModule.generate / forward, and ask whether that's something you'd consider adding.

If we missed an existing API, a pointer would be enough.

What we looked at

JS API (0.9.3)

• LLMModule.generate(messages, tools?) has no grammar / GBNF argument.
• GenerationConfig only exposes sampling knobs: temperature, topP / topp, minP, repetitionPenalty, plus token-batch timing. No grammar / schema / allowed-token set.

Native sampler

• Decode goes through TextDecoderRunner::logits_to_token → Sampler::sample.
• The sampler applies argmax or temperature + top-k + top-p + min-p + repetition penalty. We didn't find a grammar FSM, token mask from a GBNF, or a user logits processor.

"Structured output" in the docs

• getStructuredOutputPrompt / DEFAULT_STRUCTURED_OUTPUT_PROMPT inject schema instructions into the prompt.
• fixAndValidateStructuredOutput parses after generation.
• That's prompt + validate, not constrained decoding — the model can still emit anything.

Unrelated "grammar" in the repo

• The privacy-filter pipeline has a BIOES Viterbi grammar. That's token-classification, not LLM GBNF.

We also grepped open issues for GBNF / constrained decoding / logits processor and didn't find an existing thread. Happy to close this as a duplicate if one exists.

Why we care

We use GBNF on llama.cpp / llama.rn for small closed outputs, e.g. root ::= "Yes" | "No" or a short enum. On RNE we can't attach that to generate(), so we fall back to greedy decode + parsing (and first-token logprobs where we have them). That works until the model starts with "The …" instead of a label.

A logits mask / GBNF (llama.cpp-compatible), a JSON-schema sampler, or a per-step allowed-token callback would let us constrain the decode itself instead of hoping the prompt is enough.

Questions

1. Is the above correct for 0.9.x, and still true on the rewrite / 0.10 (#1208)?
2. If yes: would you consider constrained decoding on generate / forward?
3. If this is already possible through ExecuTorch LLM config / a runner hook we missed, please point us at it.

Happy to share a more precise sampler call-stack or a tiny repro if useful.

[barhanc]

Currently we don't support proper logit-level constrained decoding and the rewrite branch also doesn't have this unfortunately. This is caused by the fact that under the hood we are using ExecuTorch's C++ implementation of the LLM runner driving the generation which is still experimental and doesn't support this. There is a pretty recent RFC in ExecuTorch repo about this exact issue (pytorch/executorch#19215). Prior to rewrite we had a custom LLM runner C++ files inside this repo that were semi-compatible with upstream ExecuTorch and had a bunch of ad hoc fixes and modifications. On the rewrite branch we got rid of them and we build the ExecuTorch binaries from our fork (https://github.com/software-mansion-labs/executorch) and this will be our default approach after the v0.10.0 release. We might consider adding the constrained decoding to the runner implementation in our fork, however we will first have to check the status of the upstream's progress on that. Another approach that might interest you is using the lower level API on the rewrite branch. In theory it should allow one to drive the generation directly from TS using the Model.execute and Tokenizer APIs and getting the raw logit values over the vocabulary this way, although I haven't personally checked the feasibility or the performance of this approach. We will definitely take a closer look at this issue and LLM support in general after the v0.10.0 comes out as right now we are preoccupied with other things before the release.