Your CPU Has SIMD. Your Code Doesn't Use It.

Modern processors crunch 8 values per instruction. Your loops process one. Here's how to close that gap.

Dafallah Ibrahim

— 2 min read
Share

Your CPU has dedicated hardware that can process 8, 16, or even 32 values in a single instruction. Your code almost certainly ignores it.

Every x86 and ARM chip ships with SIMD units — wide registers and specialized instructions for parallel data crunching. When simdjson parses JSON at 3 GB/s while traditional parsers crawl at 200 MB/s, SIMD is the reason. When numpy demolishes pure Python, same reason.

Why this matters

SIMD means Single Instruction, Multiple Data. One instruction loads 8 floats, adds them to another 8 floats, and stores the result. Same clock cycle as a scalar add. Eight times the throughput.

When your loop processes arrays element by element, the CPU issues one ADD per element. The SIMD unit sits idle. You're paying for silicon that never warms up.

The gap is enormous. simdjson is 10-40x faster than RapidJSON. glibc's SIMD strlen runs 10x faster than a naive loop. Columnar databases like ClickHouse and DuckDB are designed around SIMD from day one.

How it works

SIMD registers come in widths: 128-bit (SSE on x86, NEON on ARM), 256-bit (AVX2), and 512-bit (AVX-512). A 256-bit AVX2 register holds 8 single-precision floats. One VADDPS instruction adds all 8 pairs in a single cycle.

Compilers attempt auto-vectorization — they analyze loops and emit SIMD instructions automatically. But they're conservative. Any of these will usually kill it silently:

  • Function calls inside the loop body
  • Data dependencies between iterations
  • Conditional branches that can't be masked
  • Loop bounds the compiler can't prove at compile time

No warning. No error. Your code just runs slower than the hardware allows.

Where this helps

  • Parsing: simdjson, simdutf, and rapidstring use SIMD for character scanning and structural validation
  • Numeric compute: numpy, PyTorch, and BLAS dispatch to hand-tuned SIMD kernels internally
  • String ops: glibc's strlen, memcmp, and memset all ship SIMD-optimized variants
  • Databases: ClickHouse, DuckDB, and Parquet readers are columnar-first because columnar layouts map naturally to SIMD registers

Watch out

Auto-vectorization is fragile. A refactor that introduces a function call or an early return can silently disable SIMD, and you'll never know without profiling.

AVX-512 has a catch on some Intel CPUs: heavy 512-bit instructions can trigger frequency throttling. The SIMD speedup can literally cost you clock speed.

Cross-platform SIMD is non-trivial. ARM has NEON and SVE. x86 has SSE, AVX2, and AVX-512. Portable code needs either compiler intrinsics with fallbacks or wrapper libraries like highway or simde.

Try it yourself

Check whether your compiler is actually vectorizing your hot loops:

# Compile with GCC/Clang and request vectorization diagnostics
gcc -O3 -march=native -fopt-info-vec-optimized main.c -o main

# Expected output if it works:
# main.c:5:5: optimized: loop vectorized using 32 byte vectors
#
# Silence means your loop stayed scalar. Time to investigate why.

TL;DR

  • What: SIMD lets CPUs process 4–16 values per instruction through wide registers most code never touches
  • Why: It's the hidden engine behind simdjson (10x faster), numpy, and every fast columnar database
  • Try: Compile with -fopt-info-vec-optimized and check whether your hottest loops actually vectorize

Read more