DuckLake: ACID Transactions on Plain Parquet

DuckLake hit HN today: the catalog lives in SQLite or Postgres, the data is Parquet on S3, and you get ACID and time travel in plain SQL.

Dafallah Ibrahim

— 2 min read
Share

DuckLake is sitting on Hacker News's front page right now, and it deserves more attention than its plain repo name suggests. The pitch: your data stays as ordinary Parquet files in object storage, while the catalog — tables, schemas, file lists, snapshots — lives in a boring SQL database like SQLite or Postgres.

If you've ever pointed a cron job at a bucket full of Parquet, you know where this goes. One crashed run leaves a half-written partition. Two writers clobber each other. "Undo" means restoring a bucket. The usual fix is a full lakehouse stack; DuckLake gets there with one DuckDB extension.

Why this matters

Data teams pay real infrastructure tax for safe Parquet: catalogs, JVM engines, coordination services. DuckLake collapses that into two parts you already know how to operate — object storage and a SQL database. Transactions come from the database's transaction log, not from a distributed locking protocol.

Readers keep full DuckDB speed, because they read Parquet directly from S3. The catalog only answers one question: "which files make up this table right now?"

How it works

Two pieces. The catalog is a handful of tables in SQLite, DuckDB, or Postgres: schema definitions, the list of Parquet files, and one row per snapshot. The data itself is immutable Parquet written straight to storage.

A write stages new Parquet files first, then commits a single catalog transaction that points a new snapshot at them. Crash before the commit and your dataset is untouched — you just have orphan files to sweep up. Commit, and every reader atomically sees the new state.

Old snapshot rows stick around, so each commit stays queryable as-of: time travel without building an audit system. Concurrent writers are serialized by the catalog database, exactly like any two transactions on one Postgres instance.

Where this helps

  • Scheduled pipelines that dump partitioned Parquet. Appends become atomic, so a failed run is a retry, not a data incident.
  • Multiple writers — Airflow tasks, ingestion services — hitting one dataset. The catalog arbitrates; files stop clobbering each other.
  • The 2 a.m. bad load. Roll back to the previous snapshot, debug, fix forward — and the files stay vanilla Parquet that Spark or pandas can still read directly.

Watch out

  • An SQLite catalog means one writer on one machine. Multi-host writers need the Postgres catalog, and now that's a real dependency to run.
  • Every commit is a round trip to the catalog. Batch your writes; per-row commits will hurt.
  • Old snapshots pin old files. Without snapshot expiry, storage grows forever — garbage collection is your job.
  • No cross-engine catalog support: Spark and Trino can read the Parquet, but not the transactional view. If your org standardized on Iceberg, treat this as a complement, not a replacement.

Try it yourself

-- DuckDB CLI: a local lake in four lines
INSTALL ducklake;
LOAD ducklake;
ATTACH 'lake.ducklake' AS lake (TYPE DUCKLAKE);
CREATE TABLE lake.events AS SELECT range AS id FROM range(1000);
SELECT count(*) FROM lake.events;

-- ready for S3 later? same thing, with data pointed at a bucket:
-- ATTACH 'sqlite:catalog.db' AS lake (TYPE DUCKLAKE,
--   DATA_PATH 's3://my-bucket/lake');

TL;DR

  • What's new: DuckLake is trending on HN — a lakehouse layer where the catalog is SQLite/Postgres and data is plain Parquet.
  • Why it matters: ACID writes, snapshots, and time travel over object storage with zero lakehouse infrastructure.
  • Try today: the four-line attach above, then point DATA_PATH at an S3 bucket and make your next Parquet pipeline transactional.

Read more