HN DigestHacker Newsletter, digested

Issue #806#Data

PostgreSQL for Everything

TL;DR

Raphael Bauer argues that Postgres with extensions can replace Elasticsearch, MongoDB, Kafka, Redis, ClickHouse, Neo4j and a vector database. His advice is to start every new need in Postgres and add a dedicated system only when Postgres measurably falls short.

The article

  • Bauer has used Postgres since 2003, when he chose it over MySQL for a research project. Its built-in full-text search spared him from running and syncing a separate Lucene or Solr system.
  • Bauer credits Postgres with three strengths. It has matured since its first release in 1996. Every cloud provider offers it as a one-click managed service. One install covers many jobs that would otherwise each need their own system.
  • For search, start with the built-in tsvector and tsquery. Move to extensions like pg_textsearch or ParadeDB when you need BM25 ranking. JSONB with GIN indexes covers most MongoDB use cases. SELECT ... FOR UPDATE SKIP LOCKED turns a table into a queue.
  • TimescaleDB handles high-volume time series in place of ClickHouse. pgvector makes Postgres a vector store. Unlogged tables with an expiry trigger can replace a Redis cache. Bauer credits Timescale with pgvector, which commenters pointed out is wrong.
  • In one client project, storing small Flatbuffers-encoded blobs in Postgres beat reading them from the file system. LTREE handles hierarchies. Apache AGE adds openCypher graph queries that can mix with plain SQL.
  • Bauer's rule for a new requirement is to ask whether Postgres can already do it before adopting new technology.

Why it matters

The article is a catalog of Postgres features and extensions, with links to case studies from Contentful, Instacart and The Guardian. It says little about where each substitute breaks down, so read the thread for the limits.

The discussion

A big, split thread between commenters who run whole products on Postgres alone and others, including a longtime Postgres contributor, tired of "just use Postgres". The core argument was whether starting in Postgres and migrating later is cheap or a trap. The queue section drew the most fire, and a long side thread rehashed how Postgres overtook MySQL.

Start in Postgres, but can you get out later?

The most common position was to use Postgres until load shows exactly where it fails, because every extra system is another thing to run. Critics said the migration is the hard part, and the limits can arrive sooner than expected. Once queues, search and stored procedures all live in the database, they are tangled into everything. Defenders answered that the feared rewrite rarely happens and hurts less than predicted, and many teams ran FAANG-scale infrastructure for small-instance loads.

Anything you introduce is another moving part you have to operate and maintain, and in the beginning, Postgres can probably handle it. Wait for load, see *where* its failing, and then you'll have a better idea if adding another tool is worth the cost.

— @psadauskas

The problem with this approach is, once "use PostgreSQL for everything" is identified as being no longer be feasible, it has already become an inextricable component underpinning system functionality.

— @AdieuToLogic

Postgres as a queue

Firsthand reports pulled both ways: one team moved a working Postgres job queue to RabbitMQ and regretted it. Another said a Postgres queue made fair scheduling across tenants easy and reused its existing backups and HA. One commenter found it robust but slow once more than a few servers were pushing, and moved to ZeroMQ and then SQS. A Kafka SRE argued that a well-known tool comes with public answers to its problems, which a homegrown queue does not.

Postgresql actually has very good mechanisms to support message style communication, and as long as you design your message tables independently you shouldn't have horrid issues around locking and transactions. Message queues don't save you from thinking about that anyways, they just replace transactions with acknowledgements.

— @tensor

I tried very hard to use postgres as a queue, it was robust but slow once I started to push from more than a few processes/servers. Moving to zeromq initially and sqs after solved all my perf issues, and was still solid.

— @dorfsmay

Where the substitutes fall short

Commenters went through the list: built-in full-text search lacks faceting, custom tokenization and BM25, so it is no Elasticsearch. ClickHouse's columnar speed and materialized views have no real Postgres equivalent, and one ClickHouse mirror cut some queries from 60 seconds to half a second. pgvector competes with other workloads for cache and CPU on a busy server, and slows past roughly 100 million vectors. Extension fans said ParadeDB and pg_textsearch close some gaps, and skeptics replied that custom extensions add their own complexity.

Fwiw, I, as someone who has worked on Postgres for a long time, also find it quite tiresome. Like there's plenty stuff I wouldn't use Postgres for, and I can probably get get more out of it than most.

— @anarazel

Forget about adding vector storage to an existing high volume db, that must server other complex queries.. PGVector will either trash your caches, or take over your cpu so that workloads that used to work fine stall.

— @Gluber

Postgres is not low-maintenance

Experienced operators disputed the "easy to run" claim, citing many interacting tuning knobs and failures on long-running transactions. At scale Postgres needs VACUUM care, connection pooling and HA tooling like Patroni, while MySQL runs acceptably on defaults. Blobs drew specific warnings: they thrash the buffer cache and bloat WAL and backups. One commenter's 600 GB of bytea made restores so slow that he would start with S3 next time.

Postgres, on the other hand, has a million knobs, many of them interact, you'll find conflicting advice for some of them, and it can rapidly fall over if you aren't keeping a close eye on long-running transactions.

— @sgarland

I had a system that has ~600gb of blob data in bytea that could have easily been an S3 bucket + db reference. It made backups way more of a pain than necessary.

— @Tostino

Why not SQLite?

A sizable group runs everything on SQLite, some with Litestream backups, and asked why the simplicity argument doesn't favor it even more. Replies said the two solve different problems: SQLite lacks users, row-level security, updatable views and real date types. Using it locally while production runs Postgres also hides bugs. Others cited a newer WAL2 mode and published benchmarks against the claim that SQLite allows only one writer.

No. SQLite doesn't have users, proper views, row level security, proper foreign keys, or functions.

— @groundzeros2015

I can't recommend this "switch". If you are not testing locally with the same relational database as in production, you can miss mistakes and bugs.

— @zelphirkalt

Consensus

Postgres is a sane default for most applications. Teams often adopt Elasticsearch, Kafka or Redis long before their scale requires it. Readers also agreed the article overstates its case: its headings say Postgres "replaces" ClickHouse and Elasticsearch. It also wrongly credits Timescale with pgvector, and a Timescale cofounder confirmed in the thread that Andrew Kane created it.

Dissent

A vocal minority called the post fanboyism from someone without experience of complex systems. They argued that tools should follow the business model, and that Postgres limits show up much sooner than its fans admit. They also said MySQL is simpler for common needs because it needs no connection pooler or VACUUM tuning.