Stop Parsing LLM Output. Constrain It.

Grammar-constrained decoding makes it physically impossible for a local model to emit invalid JSON. No retries, no try/catch, no schema failures.

Dafallah Ibrahim

— 2 min read
Share

Every LLM application has the same ugly code: call the model, json.loads() the response, catch the JSONDecodeError, retry. Sometimes you add "output valid JSON only" to the prompt. The model still occasionally produces malformed output because you're asking politely instead of enforcing.

Why this matters

In production, an LLM that fails to produce valid structured output 2% of the time means intermittent errors, retry storms, and exhausted retry budgets. The model isn't being stubborn — token sampling is probabilistic. Any token sequence the model can emit, it eventually will. The fix isn't better prompting. It's removing invalid tokens from the sampling space entirely.

How it works

During generation, before each token is sampled, the decoder scores candidate tokens by probability. Grammar-constrained generation intercepts this step. It checks the current partial output against a grammar — a JSON schema, regex, or custom BNF — and masks out tokens that would break it. The model can only sample from valid continuations.

If the model has emitted {"name": "Al and the grammar expects a string followed by a closing quote, it can only sample tokens that continue or close that string. It physically cannot emit {"name": Al because those tokens carry zero probability after masking. The output is valid by construction, not by validation after the fact.

Where this helps

  • Structured extraction: feed unstructured text into a Pydantic model. No more "N/A" where an integer belongs.
  • Tool calling with local models: guarantee arguments match your function signature every time.
  • Custom DSLs: force a model to emit valid SQL, regex, or config syntax without a separate parser step.
  • Constrained classification: ensure the model outputs one of N allowed labels — nothing else can leave the decoder.

Watch out

Grammar constraints add compute overhead — typically 10–40% slower per token because of the masking step. BPE tokenizers (used by most open models) create edge cases where a valid character spans multiple tokens, complicating the mask. Libraries like Outlines handle this; hand-rolling your own is a rabbit hole.

Constraining output also doesn't fix bad reasoning. The model will happily emit well-formed JSON with wrong values. You get syntactic guarantees, not semantic ones.

Try it yourself

pip install outlines
from outlines import models, generate
from pydantic import BaseModel
from typing import List

model = models.transformers("mistralai/Mistral-7B-Instruct-v0.3")

class Person(BaseModel):
    name: str
    age: int
    skills: List[str]

generator = generate.json(model, Person)

result = generator("Generate a fictional software engineer")
print(result)
# Person(name="Alice Chen", age=29, skills=["Rust", "Kubernetes"])

TL;DR

  • What changed: Grammar-constrained decoding masks invalid tokens during generation, making malformed output structurally impossible.
  • Why it matters: Eliminates the retry-and-validate loop that plagues every LLM-in-production codebase.
  • What to try today: Install outlines, define a Pydantic model, and let it constrain your local model's output.

Read more