How 70B Models Run on a 4GB GPU

The math says you need 140GB of VRAM. AirLLM does it with 4GB by breaking one assumption every inference engine makes.

Dafallah Ibrahim

— 2 min read
Share

A 70B parameter model in FP16 needs roughly 140GB of VRAM. That is two or three A100s minimum. Your laptop has 4GB. The math is brutal and obvious.

AirLLM throws out one assumption that every other inference engine makes: that all model weights must live in GPU memory at the same time. They do not.

Why this matters

LLM inference is memory-bound, not compute-bound. The bottleneck is not FLOPS — it is moving weights from storage into compute units. Once you accept that, the real question becomes: how little can you keep resident at once?

Most engines load the full model into VRAM for speed. vLLM, TGI, and even llama.cpp default to keeping everything resident. That is fine if you own the hardware. But it prices out anyone who wants to experiment with large models on consumer GPUs.

How it works

A transformer processes tokens through sequential layers. Layer 1's output feeds into layer 2, which feeds into layer 3, and so on. Llama 2 70B has 80 such layers.

AirLLM loads ONE layer into VRAM at a time. It runs that layer's forward pass, passes the hidden state along, frees that layer's memory, then loads the next one. The hidden state being passed between layers is tiny — roughly 16KB per token.

The model weights live in system RAM, memory-mapped from disk. The OS handles paging. AirLLM copies each layer's weights to the GPU just-in-time. With INT4 quantization, each layer is about 440MB — comfortable alongside activation buffers in a 4GB footprint.

Where this helps

  • Experimentation without investment: Test whether a 70B model actually solves your problem before renting H100s by the hour.
  • Batch evaluation pipelines: Score thousands of prompts overnight on a gaming PC where latency is irrelevant.
  • CI/CD for ML: Run model smoke tests on standard CI runners without provisioning GPU instances.
  • Understanding inference architecture: The streaming pattern makes it obvious why memory bandwidth — not FLOPS — dominates LLM performance.

Watch out

It is slow. Generating one token requires streaming all 80 layers through VRAM sequentially. Expect 5-15 seconds per token depending on your system. You are not using this for interactive chat.

You still need substantial system RAM — roughly 35-40GB for an INT4 quantized 70B model. The 4GB figure is VRAM only. A machine with 64GB RAM and a 4GB GPU works fine. A machine with 8GB RAM does not.

KV cache grows with sequence length, so long contexts will eventually pressure VRAM. Keep your context windows short.

Try it yourself

pip install airllm

# Then in Python:
from airllm import AutoModel

model = AutoModel.from_pretrained(
    "garage-bAInd/Platypus2-70B-instruct"
)

input_tokens = model.tokenizer(
    ["What is the capital of France?"],
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    padding=False,
    max_length=128
)

output_tokens = model.generate(**input_tokens, max_length=128)
print(model.tokenizer.decode(output_tokens[0]))

TL;DR

  • What it does: AirLLM streams transformer layers one at a time into VRAM instead of loading the full model at once.
  • Why it matters: You can run 70B models on hardware you already own — trading speed for accessibility.
  • What to try: Install airllm, point it at a quantized 70B model, and generate one token. Watch how slow, and how possible, it is.

Read more