A 2.8T Model on a Laptop, Streamed From SSDs

Kimi K3 is a 2.8-trillion-parameter model. Someone just ran it on a MacBook Pro at one token per second, streaming weights from four SSDs. No 8-GPU node, no rental bill — just the deltafin project and a blunt reading of how inference actually works.

The lesson generalizes far beyond the stunt: token generation is a memory-bandwidth problem, not a compute problem. Once that clicks, "the model doesn't fit in RAM" stops being a hard no.

Why this matters

Most of us treat model size as a gate: 72B won't fit in 64 GB, so it's off the table. But decode — emitting one token — reads every active weight exactly once and does almost no arithmetic per byte. The bottleneck is how fast bytes reach the chip. RAM and VRAM are just the fast tiers of that pipeline. Disk is the slow tier. A slow tier still produces tokens.

How it works

Rule of thumb for batch-size-1 decode: tokens/sec ≈ effective bandwidth ÷ bytes touched per token. A dense 2.8T model at 4-bit is roughly 1.4 TB of weights. Four SSDs over Thunderbolt deliver maybe 10–15 GB/s in practice. A full read per token means a token every two minutes — not one per second.

So the arithmetic only closes if each token touches a small fraction of the weights. That's what Mixture-of-Experts architectures buy you: routing activates a handful of experts per token, so per-token reads are gigabytes, not terabytes. With mmap, the kernel pages those experts in from disk on demand, and RAM becomes an LRU cache over your SSDs. deltafin is that observation taken to its logical end.

Where this helps

• Trying monster open models (Kimi, DeepSeek, huge Qwen) before committing any GPU budget.
• Overnight batch jobs — log triage, doc QA — where 1 tok/s for eight hours is still ~28k tokens.
• Running 70B-class models on a 32 GB laptop for demos. Slow, but real.
• Answering "