Your Pod Has No Shell. Debug It Anyway

Your production pod has no shell, no tools, nothing. Ephemeral containers inject what you need — no rebuild, no restart.

Dafallah Ibrahim

— 2 min read
Share

Your production pod is misbehaving. Memory climbing, latency spiking, connections dropping. You type kubectl exec — and get "OCI runtime exec failed: executable file not found." Your distroless image has no shell.

No sh. No top. No curl. No way in.

You could rebuild the image with debugging tools, push it, wait for a rollout, and hope the problem reproduces. Or you could inject a container into the running pod without touching anything.

Why this matters

Minimal images are the right call for production. Distroless and scratch-based images cut your attack surface, shrink your images, and eliminate whole classes of CVEs. But they also eliminate your ability to inspect a running workload when things go wrong.

The old tradeoff was painful: ship lean and lose debuggability, or ship fat and accept the risk. Ephemeral containers break that compromise.

How it works

kubectl debug attaches a temporary container to a pod that's already running. That container shares the pod's network namespace, its volumes, and optionally its process namespace.

The mechanism is the ephemeralContainers field in the pod spec — a special subresource that exists specifically for debugging. These containers bypass normal admission constraints: you can add them at runtime, they can't be declared at pod creation, and they don't survive pod restarts.

When you run the command, Kubernetes appends the debug container to the live pod and starts it with whatever image you specify. You get a shell inside a container that sees the same network, DNS, and filesystem context as your application.

Where this helps

Distroless production fires: Your Go binary deployed as a scratch image is leaking memory. Inject a debug container and read /proc metrics without touching the deployment.

Cluster DNS debugging: One pod can't resolve a service name. Attach a container with dig and tcpdump to trace DNS queries from inside the pod's network namespace.

Shared volume disputes: Two containers in a pod disagree about file state. Mount the same volume from a debug container and inspect it directly.

Watch out

Ephemeral containers are stripped down. No resource limits, no probes, no ports. They don't show up in kubectl get pods — use kubectl describe pod to see them.

If the pod restarts for any reason — node drain, eviction, crash — your debug session is gone. No persistence, no reconnection.

This feature reached GA in Kubernetes 1.25. Most clusters support it now, but check your version. The --target flag for process namespace sharing works but has quirks with distroless images that don't expose /proc the way you'd expect.

Try it yourself

# Create a pod with zero debugging tools
kubectl run debugme \
  --image=gcr.io/distroless/nodejs18-debian12 \
  --restart=Never --command -- sleep 3600

# This fails — no shell in the image:
# kubectl exec -it debugme -- sh

# This works — inject an ephemeral container:
kubectl debug -it debugme \
  --image=busybox:latest \
  --target=debugme

# Now you're inside busybox, sharing the pod's
# network namespace. Try:
ip addr
nslookup kubernetes.default

TL;DR

  • What it does: Kubernetes ephemeral containers let you inject debug tools into any running pod — no image rebuild, no redeploy.
  • Why it matters: Ship distroless images for security without losing the ability to debug live production issues.
  • Try today: Run kubectl debug -it <pod> --image=busybox against any running pod in your cluster.

Read more