One Log Line Costs 110KB of Disk Writes

A single systemd-journald log line causes 49KB+ of disk writes on ext4 and 110KB+ on btrfs. Here's why that's burning your SSD.

Dafallah Ibrahim

— 2 min read
Share

You write one line to the journal. systemd-journald turns it into 49KB of disk writes on ext4 — or over 110KB on btrfs. A single line.

That's not a typo. A bug report filed this week (systemd issue #40262) demonstrates how the combination of journald's binary format and filesystem internals creates write amplification that would make an SSD wince.

Why this matters

If you run production Linux servers, journald is almost certainly your default log destination. Every service, every container, every cron job funnels logs through it. When each line costs 49–110KB of actual I/O, high-throughput logging silently eats your disk bandwidth and burns SSD write endurance.

A web server logging a few lines per request at 1,000 requests per second could be generating hundreds of megabytes of disk writes per second — just for logging.

How it works

journald doesn't append plain text. It writes structured binary journal files built on hash tables, object chains, and entry arrays. Each incoming log line triggers updates across multiple internal data structures — not a simple append.

The filesystem then amplifies this further. ext4 journals metadata changes, meaning every journald write goes through ext4's own journal, effectively doubling metadata I/O. btrfs is worse: its copy-on-write design means every small modification copies entire 4KB blocks, creating massive duplication for tiny logical changes.

Where this helps

  • High-throughput API servers — services logging request/response cycles can hit thousands of lines per second, each multiplied 50–100x in actual I/O
  • Container hosts — dozens of services logging through a shared journald compound the amplification across all workloads
  • Cloud instances with burst-limited disks — EBS gp3 and similar volumes have IOPS credits that journald can exhaust before your application gets its budget
  • Embedded and edge systems — eMMC and SD card storage has limited write cycles; journald amplification can shorten hardware lifespan significantly

Watch out

You can't just disable journald — modern systemd depends on it. The realistic mitigations are configuration changes: cap journal size, enable rate limiting, and forward high-volume logs directly to an external aggregator instead of through journald storage.

btrfs users should pay special attention. The 2x+ amplification over ext4 makes this particularly painful on copy-on-write filesystems.

The issue tracks a single minimal log line's cost. Real entries with long messages, structured fields, or stack traces will amplify even further.

Try it yourself

Check your journal's current footprint and apply practical limits:

# How much journal storage you're using
journalctl --disk-usage

# In /etc/systemd/journald.conf:
# SystemMaxUse=500M
# SystemMaxFileSize=50M
# RateLimitIntervalSec=30s
# RateLimitBurst=1000

# Forward high-volume services directly
# to syslog or a log agent, bypassing
# journald persistent storage entirely

sudo systemctl restart systemd-journald

TL;DR

  • What happened: systemd issue #40262 shows a single journal log line causes 49KB+ writes on ext4 and 110KB+ on btrfs
  • Why it matters: high-throughput logging silently consumes disk I/O and SSD endurance on production Linux servers
  • What to try today: run journalctl --disk-usage on your busiest server, then add rate limits and size caps to journald.conf

Read more