Your Local LLM Isn't Dumb. Your Config Is.

Chat template mismatches, silent context truncation, and inherited sampling defaults quietly tank local model output. Most of it is fixable.

Dafallah Ibrahim

— 2 min read
Share

Same weights, wildly different answers. You run a model locally and it rambles, forgets your instructions, and writes broken code — while the hosted version of the same model feels sharp. A post titled "Why your local LLM feels dumber than it is" climbed the HN front page this week because everyone has lived this.

Here's the uncomfortable part: the weights are almost never the problem. Your inference stack is.

Why this matters

Local inference keeps winning ground — privacy rules, air-gapped environments, API costs. Then output quality drops, people blame the model, and crawl back to paid APIs. Most of that quality gap is configuration, and it's fixable in about ten minutes.

How it works

Four things silently degrade local models:

1. Chat template mismatch

Every model expects exact chat markup — ChatML, Llama-3 tags, Mistral tokens. If your runtime guesses wrong, the model sees token sequences it never trained on. The result isn't an error; it's subtle degradation: rambling, confused role switching, half-finished turns.

2. Silent context truncation

Ollama ships a default context window of 2048–4096 tokens depending on version. Paste a source file plus a system prompt and older content gets dropped — silently. The model isn't ignoring your instructions. It never saw them.

3. Sampling defaults from another era

llama.cpp and Ollama apply repeat_penalty: 1.1 by default. Modern models are trained without it, and code legitimately repeats tokens — }, end, indentation — that the penalty then suppresses. Meanwhile temperature is whatever the runtime shipped, not what the model card recommends.

4. Over-quantization

Q4 on a 70B model is fine. Q4 on a 7B model eats a noticeable slice of its ability, because quantization error gets amortized across fewer parameters. Rule of thumb: the smaller the model, the gentler the quant.

Where this helps

  • Local coding assistants that read whole files — the first casualty of context truncation.
  • Offline agent loops stuffing tool output into context until the original task falls off the window.
  • Pre-deployment evals — evaluate a model with a broken template and you'll reject a good model.
  • Air-gapped environments where "just use the API" isn't an option.

Watch out

Some of the gap is real. Hosted endpoints run fp16 or bf16 — roughly 2GB per billion parameters, so a 14B model wants ~28GB — on hardware with memory bandwidth your laptop doesn't have. Small models are genuinely less capable. Fix the config first; then you'll know how much is left to blame on physics.

Try it yourself

Point this at a local Ollama and compare — explicit context, penalty off, sampling from the model card:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:14b",
  "stream": false,
  "messages": [
    {"role": "user", "content": "Refactor this function: ..."}
  ],
  "options": {
    "num_ctx": 32768,
    "temperature": 0.7,
    "top_p": 0.8,
    "repeat_penalty": 1.0
  }
}'

Then check the model card for its recommended temperature and top_p — those numbers vary per model family.

TL;DR

  • What changed: a front-page post named what every local-LLM user has noticed — same model, worse answers.
  • Why it matters: most of the gap is config — wrong chat template, truncated context, inherited sampling defaults — not the weights.
  • Try today: set num_ctx explicitly, set repeat_penalty to 1.0, and copy the model card's sampling params.

Read more