Your Docker Layers Never Forget a Secret

Admin on a production GitHub org in 25 minutes — via a container registry. Docker layers keep every secret you ever baked in.

Dafallah Ibrahim

— 2 min read
Share

Strix published a security writeup with a number that should sting: admin access to Baseten's production GitHub, in 25 minutes. The chain ran through Baseten's Harbor container registry and ended at a GitHub personal access token. No zero-day, no phishing — just credentials sitting in the part of the stack nobody audits.

The specifics are theirs. The exposure class is nearly universal. If your org builds and ships containers, assume the same playbook works on you until you've checked.

Why this matters

A container image isn't a program — it's a stack of immutable tarballs. When your CI copies a .env, sets an ENV, or passes a build arg, that value is written into a layer forever. Deleting the file in a later layer hides it from the running container and from nobody else. Anyone who can pull the image can extract every layer and read the bytes.

Registries compound this. Deploys pull from them, CI pushes to them, and their service accounts hold long-lived tokens. A leaked pull credential becomes a secrets dump. A leaked push credential becomes code execution in prod if anything deploys :latest.

How it works

A later layer can only add a "whiteout" marker for an earlier file — the original bytes stay in the tarball. Even easier: docker history --no-trunc prints build args and env vars straight from image metadata. No extraction required.

Build args are sneakier than ENV. Values passed with --build-arg land in the image config even when the Dockerfile never exports them. Check your own images before assuming otherwise.

On the token side, a classic PAT is an org-wide skeleton key. A fine-grained PAT scoped to specific repos with a seven-day expiry turns a leak into a bad afternoon instead of a breach.

Where this helps

  • Auditing anything you publish to Docker Hub or GHCR — before a stranger does it for you.
  • Incident response: "which images did that leaked runner credential touch, and what's in their layers?"
  • CI hardening: BuildKit's --mount=type=secret exposes tokens at build time without writing them into any layer.
  • Token hygiene: per-pipeline, short-lived, fine-grained PATs instead of one immortal org-wide classic token.

Watch out

Scanning the final filesystem finds nothing — secrets live in intermediate layers. Scan the full docker save tarball, or run Trivy's secret scanner against the image.

Multi-stage builds help a lot (builder-stage layers don't ship), but any secret the final stage consumes still lands in its layers.

Rotating a leaked token doesn't unpublish the image containing it. Old digests stay pullable until you delete them, so "we rotated it" is not a cleanup.

Try it yourself

IMAGE=ghcr.io/yourorg/api:latest

# 1) Free first pass: build args and env, no extraction needed
docker history --no-trunc --format '{{.CreatedBy}}' $IMAGE \
  | grep -iE 'token|secret|key|password'

# 2) Full forensic pass: every layer, because rm is a lie
rm -rf /tmp/layers && mkdir -p /tmp/layers
docker save $IMAGE -o /tmp/img.tar
tar -xf /tmp/img.tar -C /tmp/layers
grep -rEao '(ghp|gho|ghs)_[A-Za-z0-9]{30,}|github_pat_[A-Za-z0-9_]{40,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,}' /tmp/layers

Run it against your most-published image. Then inventory which GitHub tokens your CI can actually reach.

TL;DR

  • What happened: Strix reached admin on Baseten's production GitHub in 25 minutes, via their Harbor registry and a GitHub PAT.
  • Why it matters: image layers are append-only archives — every secret ever baked in stays readable by anyone who can pull.
  • Try today: docker save your flagship image and grep every layer for tokens. Then shrink your CI's PAT scopes.

Read more