Why Your vLLM Tool Calls Break at Random

Open models don't emit tool calls; they emit text that gets regexed into JSON. Constrained decoding makes bad arguments impossible.

Dafallah Ibrahim

— 2 min read
Share

Your agent runs flawlessly against a hosted API. Then you point it at a self-hosted model on vLLM to cut costs, and every tenth tool call comes back mangled: half a JSON object, a stray comma, or the model narrating around the call instead of making it.

Ubicloud, which runs managed vLLM, just published a post on fixing tool-call accuracy. The lesson generalizes to anyone serving open models: the model usually isn't the problem. The tool-call plumbing around it is doing something fundamentally fragile.

Why this matters

Self-hosting inference is now routine for cost and data-control reasons. But agents are chains, and a chain is only as reliable as its weakest link. One unparseable tool call can kill an entire run, trigger a retry storm, or silently drop a row from a batch job. This plumbing — not benchmark scores — decides whether an open model is deployable for agentic work.

How it works

Hosted APIs emit tool calls as structured output. The JSON arrives pre-parsed; you never see raw text. Open models have no such channel. They emit plain tokens, and vLLM attaches a tool-call parser — hermes, mistral, llama3_json — that scans the token stream and carves out a function name plus arguments.

That's parsing, and parsing is brittle. It only works if the raw output matches the expected template exactly. Single quotes instead of double, an unescaped newline, a friendly "Sure, calling that now" — and extraction fails.

Constrained decoding flips the approach. Compile each tool's JSON schema into a grammar using vLLM's structured outputs (xgrammar or outlines backend), and the sampler masks every token that would violate the schema at each step. The parser still wraps results into OpenAI-style tool_calls, but malformed arguments stop being unlikely — they become impossible.

Where this helps

  • Agent loops on self-hosted models: tool arguments become deterministic instead of best-effort.
  • Tools whose arguments become SQL or HTTP requests. A malformed argument there is a production incident, not a chat typo.
  • Batch pipelines that log and skip failures. Schema enforcement drops silent data loss to zero.
  • Multi-tool routing: pick the right parser per model once, then enforce a schema per tool forever.

Watch out

Grammar compilation costs real time per unique schema. Cache schemas and reuse them — don't regenerate per request. Token masking also adds slight latency per step, usually negligible next to generation.

Overly strict schemas can hurt quality: the model must commit to field values it might otherwise reason toward. Keep schemas permissive where correctness allows.

The parser must still match the model's chat template. A Hermes parser on a Mistral template produces garbage no grammar will fix. And if your client streams tool-call deltas, buffer partial JSON before validating.

Try it yourself

Serve a model with a matching parser, enable structured outputs, and force tool arguments through a schema:

# vllm serve Qwen/Qwen2.5-7B-Instruct \
#   --enable-auto-tool-choice --tool-call-parser hermes \
#   --enable-structured-output

from openai import OpenAI
from pydantic import BaseModel
from typing import Literal

class WeatherArgs(BaseModel):
    city: str
    unit: Literal["celsius", "fahrenheit"]

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Weather in Tokyo?"}],
    extra_body={"guided_json": WeatherArgs.model_json_schema()},
)
print(resp.choices[0].message.content)  # schema-valid JSON, guaranteed

TL;DR

  • What happened: Ubicloud published its fixes for tool-call accuracy in vLLM — the fragile part is text parsing, not model intelligence.
  • Why it matters: schema-constrained decoding makes tool arguments deterministic, closing the biggest reliability gap between self-hosted and hosted APIs.
  • Try today: enable structured outputs and pass your tool's JSON schema so malformed calls can't happen.

Read more