The Overnight Agent Loop That Got a 232x Speedup

A dev let Codex run its own benchmark loop and got a 232x kernel speedup. Here's the loop, rebuilt in plain bash.

Dafallah Ibrahim

— 2 min read
Share

On Hacker News right now: a developer reports a 232x speedup on a compute kernel, achieved by letting Codex run an autonomous research loop against a benchmark. The number is the boring part. The loop is the part you can steal.

No fancy model required. The setup just gives the agent a score after every attempt, which turns optimization into search — and tireless, slightly-dumb search is exactly what agents are good at.

Why this matters

Performance tuning is the most expensive work senior engineers do. Profile, hypothesize, tweak, rerun. Every cycle burns context and morale, which is why most code ships at "first working version" speed.

An agent has no morale. It will try data-layout changes at 3am, loop unrolling at 3:15, SIMD at 3:30. Your job moves up a level: stop running the search, start defining the objective. That was always the senior part anyway.

How it works

The loop has three parts, and none of them are clever:

  • A single-command benchmark. Fast, deterministic, and it must verify correctness before scoring. If the agent can delete the workload and win, it eventually will.
  • A ratchet. Snapshot the best-scoring version; revert anything that regresses. Now the loop only moves upward.
  • A boring prompt. File path, last score, best score, tail of the benchmark output. No vibes — just telemetry.

Wrap those in a for-loop and any agent CLI works: codex exec, claude -p, aider. The harness is 30 lines of shell.

Where this helps

  • Hot paths. Kernels, parsers, encoders — anything with a profiler-friendly inner loop.
  • SQL tuning. Score = median EXPLAIN ANALYZE cost; the agent iterates index and query shapes while you review the winner.
  • Bundle size. Score = gzipped bytes after build. Watch it delete the dependencies you were afraid to.
  • Flaky test minimization. Score = repro length; the loop grinds a 5,000-line log down to the actual trigger.

Watch out

Goodhart's law, hard. An agent handed a metric will cheat it: memoize results, skip the work, quietly weaken an assert. Rotate inputs, keep a hidden test set, and make the benchmark verify output, not just timing.

Benchmark noise is the other trap. If runs vary 5%, the agent optimizes the variance. Pin the CPU, take medians. Set an iteration budget — overnight loops burn tokens like it's their job, because it is. And a human reviews the winning diff: a valid-but-unreadable 232x is a liability, not a speedup.

Try it yourself

Point bench.sh at your slowest function. It must fail if tests break, otherwise print score: N:

#!/usr/bin/env bash
set -euo pipefail
BEST=0; mkdir -p snapshots
for i in $(seq 1 25); do
  if ! ./bench.sh > run.txt 2>&1; then
    git checkout -- src/            # broke it, revert
  else
    SCORE=$(grep -oE 'score: [0-9.]+' run.txt | grep -oE '[0-9.]+')
    if awk -v a="$SCORE" -v b="$BEST" 'BEGIN{exit !(a>b)}'; then
      BEST=$SCORE; cp src/kernel.rs snapshots/best_$i.rs
    else
      git checkout -- src/          # regression, revert
    fi
  fi
  codex exec "Last benchmark: $(tail -3 run.txt). Best: $BEST. Make src/kernel.rs faster; tests must pass."
done

TL;DR

  • What happened: A developer reported a 232x kernel speedup from an autonomous Codex loop driven purely by a benchmark.
  • Why it matters: Agents are search engines — give one a score and a ratchet and it will outlast any human at tuning.
  • Try today: Wrap your slowest function in a one-command benchmark and let the loop run overnight.

Read more