You Probably Don't Need Kafka (You Need Postgres and a Bucket)

Tue, 26 May 2026 00:00:00 +0000

I’ve been circling this one for a while. A couple of years ago I wrote about Postgres Listen/Notify and how you can do PubSub without dragging a message broker into your stack. Last year I got a little too excited about DuckLake and cheap, decoupled storage and compute. This post is what happens when those two ideas have a baby.

Here’s the thing about Kafka: it’s almost never just Kafka. The moment you have a real use case, you’re not running a message bus, you’re running a small fleet of distributed systems. Brokers, plus ZooKeeper or KRaft for coordination. A Schema Registry so your payloads don’t drift. Kafka Connect workers to get data out to your warehouse or object store. Kafka Streams or ksqlDB for the stateful processing and materialized views you inevitably need. And then tiered storage (or a whole separate pipeline) so retention doesn’t bankrupt you. Each of these is its own thing to scale, upgrade, secure, and get paged about. Frequently it’s a dedicated platform team and a five-figure monthly bill.

And here’s my heretical little question: what does your project actually need?

In my experience, “we need Kafka” almost always decomposes into four things:

Distribute work across a pool of workers.
Fan an event out to several independent consumers.
Keep a durable, replayable log of what happened.
Query the history of that log later.

Postgres does the first three in plain SQL. Object storage holds the cold log for basically free. DuckDB queries it at warehouse speeds with no server to run. So I built a thing to prove it to myself, the same way I stood up DuckLake last year: not by taking anyone’s word for it, but by actually running it.

The demo

The project is a template I’m calling kafka-to-pg. It streams live geographic telemetry — a synthetic fleet of ~500 “aircraft” drifting around the continental US, publishing position reports at about 1,000 messages per second by default (and you can crank that knob). It renders them on a live map that updates over Server-Sent Events, maintains a “current position per aircraft” table, ages old data out to Parquet on object storage, and lets you run SQL over that history. There’s a little query console in the browser too.

The entire thing runs on one Postgres instance and one bucket. No brokers, no ZooKeeper, no Connect cluster, no Streams app. Just SQL and a handful of small, stateless Go processes that you can kill and restart whenever you feel like it.

SQL is the API, the whole way down

The “topic” is just an append-only table. The trick people worry about — getting monotonic offsets without a race — is a single transaction. You reserve a block of offsets and insert the batch in one statement:

WITH reserve AS (
  UPDATE log_counter
  SET next_offset = next_offset + cardinality($1::bytea[])
  WHERE id = $2
  RETURNING next_offset - cardinality($1::bytea[]) AS first_off
)
INSERT INTO topic (topic_id, c_offset, payload)
SELECT $2, r.first_off + ord - 1, payload
FROM reserve r, unnest($1::bytea[]) WITH ORDINALITY AS p(payload, ord);

That’s the atomicity Kafka gives you, as one SQL statement. Consumer groups are a consumer_offsets table and a query that claims a range of offsets and advances the cursor atomically — same idea, log-based, each group replays independently. And if what you want is a task queue rather than fan-out (the RabbitMQ/SQS shape), that’s the classic SELECT ... FOR UPDATE SKIP LOCKED: many workers pull from the same table, and a worker that hits a locked row just skips to the next job instead of waiting. No broker required for any of it.

The part I’m actually proud of: retention

This is where the “you’ll regret not using Kafka” crowd usually has a point. Logs grow. At 1,000 msg/s mine grows about a gigabyte an hour. You can’t just let that run.

Kafka has retention.ms and tiered storage for this. My version is almost embarrassingly simple: the topic table is partitioned by time. A little sweeper process creates upcoming partitions ahead of time, and for partitions older than the retention window it does this:

DROP TABLE topic_p_1779807425;

That’s it. DROP on a partition is O(1) and leaves zero dead tuples behind. Compare that to a DELETE-based TTL, which churns the heap and leaves you chasing autovacuum forever. Dropping partitions is retention.ms, in one line, with none of the bloat.

But before it drops a partition, the sweeper hands the rows to a sink. The interesting sink writes them out as Zstd-compressed Parquet, partitioned by date, to object storage. So Postgres only ever holds the hot window — the last few minutes, or hours, or whatever you configure — and the full history lives on a bucket as columnar files. That’s Kafka Connect and tiered storage, replaced by a cron-shaped Go program and a sink interface.

Then DuckDB shows up and eats ksqlDB’s lunch

Once your history is Parquet on a bucket, querying it is the easy part, because DuckDB is a marvel. Point it at the files and you have a SQL analytics engine over your entire event history, no server:

SELECT kind, count(*) AS msgs, count(DISTINCT id) AS entities
FROM read_parquet('archive/**/*.parquet')
GROUP BY kind;

I wrapped this in a little web console so you can poke at the archive from the browser. This is the ksqlDB / stream-analytics box on the Kafka architecture diagram, and it’s a read_parquet call.

The numbers

I measured this stuff on my laptop, because vibes are not a benchmark:

A row in Postgres costs about 317 bytes on disk (the JSON payload plus row overhead plus the index).
The same row in Zstd Parquet after archival is about 42 bytes — roughly a 7.5× reduction, and it’s columnar, so analytical scans fly.
A single Postgres handles the messaging load far past where most people assume it taps out. Published benchmarks for this pattern put it around 5k writes/s and 25k reads/s on 4 vCPUs, scaling to hundreds of thousands of writes/s on a big box. Your “Kafka-scale” workload is probably smaller than you think.

The punchline I keep coming back to: two pieces of infrastructure to operate instead of a fleet. One database you already know how to run, and a bucket.

When you should still use Kafka

I’m not going to insult you with “Kafka is dead.” It isn’t, and there are real reasons it exists. If you genuinely need sustained multi-hundred-MB/s throughput, huge fan-out to many independent consumer clusters, multi-datacenter replication, exactly-once processing across topics, or the mature connector ecosystem — use the right tool. Reach for Kafka, or Pulsar, or Redpanda.

But that’s a narrower set of requirements than the number of Kafka clusters in the world would suggest. For a very large fraction of “we need a streaming platform” projects, the honest answer is that you need a durable log, a couple of consumers, and somewhere cheap to keep the history. That’s Postgres, a bucket, and DuckDB. The savings — in dollars, in services, in 3am pages — are not subtle.

I think this is the most underrated move in backend engineering right now: look hard at the expensive, operationally heavy thing in your architecture and ask whether a boring database and some object storage short-circuit the whole problem. Often they do.

This Kafka-killer is the first entry in a project I’m calling short-circuit — think of it as a gallery of savings. The plan is a growing collection of drop-in templates, each one short-circuiting some expensive, heavyweight piece of infrastructure with a boring alternative — Postgres and a bucket, or SQLite, or DuckDB, sometimes just “use Postgres”. And “cheaper” really undersells it: once you add up the brokers you don’t run, the platform team you don’t staff, and the 3am pages you don’t get, simplicity stops being the budget option and starts being the way businesses actually get more for less.

Kafka’s at offset zero, and the backlog is significant — but you get a vote on what gets consumed next:

Snowflake / BigQuery → a DuckLake lakehouse. The warehouse, minus the warehouse invoice.
Splunk + Prometheus → DuckLake + a cron job. Observability without the per-gigabyte shakedown.
Airflow / Temporal → Postgres-native durable workflows. Orchestration with one fewer system to babysit.

The comments below are a Bluesky thread, so a reply is a vote. Tell me which one to short-circuit next.