A 125B Model Just Hit 100 t/s on One RTX 4090

Qwen 3.8 Flash Next (125B) runs at 100 tokens/sec on a single consumer GPU. The math that makes it possible — and what it changes.

Dafallah Ibrahim

— 2 min read
Share

For years, "local LLM" meant a 7B model that couldn't write a correct JOIN. Meanwhile, anything that could actually code lived behind an API. A repo on today's front page — Strata — claims Qwen 3.8 Flash Next (125B parameters) running at 100 tokens/sec on a single RTX 4090. That's not a toy number. That's interactive-speed inference of a frontier-class model on a $1,600 gaming card.

The interesting part isn't the repo. It's that the arithmetic finally works.

Why this matters

The real cost of API models isn't the chat — it's agents. A coding agent in an inner loop easily burns a million tokens a day. At API prices that's a meaningful bill; locally it's electricity.

There's also the code that can't leave the building: proprietary repos, regulated data, air-gapped environments. Those teams have been stuck with small local models. A 125B that runs interactively closes most of that gap.

How it works

Run the numbers and you learn what must be true. 125B dense parameters at 4-bit is roughly 65 GB. A 4090 has 24 GB of VRAM. If every token had to touch all weights over PCIe (~25 GB/s), you'd get well under 1 t/s — not 100.

So the speed is only possible with sparse activation: a mixture-of-experts layout where each token routes through a few billion parameters instead of all 125B. Now the bandwidth math flips. VRAM bandwidth on a 4090 is about 1 TB/s; if each token reads only 1–2 GB of weights, 100 t/s fits with headroom for attention and the KV cache.

The recipe: quantize to ~4-bit, pin the hot path (shared layers plus frequently-routed experts) in VRAM, spill cold experts to system RAM. Strata packages exactly that as ready-to-copy configs.

Where this helps

  • Agent loops — draft, critique, retry cycles generate tokens by the millions. Generating them at home costs the same as generating ten.
  • Private code — a model on your machine never ships your source to someone else's log pipeline.
  • Batch jobs — relabeling datasets, summarizing logs, codebase Q&A prep. Overnight throughput, zero API bill, no rate limits.
  • Dev/CI parity — same weights on your laptop and in the pipeline, so prompts behave identically.

Watch out

4-bit 125B is not the full-precision model. Benchmark tables won't transfer; test on your actual tasks before trusting it.

Context is the hidden cost. KV cache grows with sequence length, and "100 t/s" figures are typically short-context, single-stream — not concurrent serving.

Also: you're pinning a 450W card for hours. Check your thermals, and check the repo rather than assuming the weights drop into any GGUF tooling.

Try it yourself

Grab the configs, then note the flags that matter when rolling your own setup:

git clone https://github.com/Niko1221/Strata
cd Strata && cat README.md

# typical big-quant offload setup, for reference:
llama-server -m model-Q4_K_M.gguf -ngl 99 -c 16384 \
  --flash-attn -fa --kv-cache-type q8_0

TL;DR

  • What changed: a 125B model reportedly runs at 100 t/s on one RTX 4090 via expert offloading configs.
  • Why it matters: frontier-class local inference kills the per-token economics of agent loops and keeps private code private.
  • Try today: clone Strata, but validate 4-bit quality on your own evals before you cancel any API key.

Read more