RAG Is Simpler Than You Think
TL;DR
Most retrieval-augmented generation (RAG) systems are over-built. Rafael Pierre argues that BM25 full-text search, plus an LLM that rewrites the user's query, is enough for about 60% of systems. Embeddings, rerankers and vector databases should come only after measured user complaints.
The article
- Five factors decide the design: data freshness, corpus churn, keyword or conversational queries, query volume, and the team's ML skills. Under 1K queries a day needs nothing fancy. Over 10K a day justifies full optimization.
- Plain BM25 in Elasticsearch or Postgres costs nothing per query. It answers in under 10ms, needs no chunking and is easy to debug. It misses synonyms and conversational questions. Pierre says most "semantic search" problems are really query formulation problems.
- An LLM that rewrites queries into keywords costs about $0.001 a query. It can add synonyms, split compound questions and map internal jargon from a glossary. You tune it by editing a prompt instead of re-embedding the corpus. General embedding models fail on proprietary names, like an internal framework called "Atlas".
- When embeddings are needed, Pierre lays out three ways to add them. The first reranks the top 50 BM25 hits by embedding them per query. That costs around $15 a month at 1K queries a day, but adds 200 to 500ms of latency. The second pre-embeds only the most-read 20% of documents. The third pre-embeds everything, which pays off only at Pinterest or Shopify scale.
- Model deprecation is the hidden cost of pre-embedding. Moving off text-embedding-ada-002 means re-embedding and re-testing the whole corpus. With per-query embedding, the switch is a one-line change.
- Multi-intent questions should be split into sub-queries and routed separately. Pierre prices that at $0.002, against $0.03 for handling the whole query at once. His split: 60% of systems stop at full-text plus rewriting, 25% need hybrid, 10% full pre-embedding and 5% something custom.
Why it matters
Worth a skim for anyone scoping a first internal search or RAG project who wants a decision tree and rough per-query costs. Treat the dollar figures as illustrative. Commenters noticed the deprecation example quotes $10,000 to re-embed a corpus that the article elsewhere prices at $10. The article also gives no benchmarks for any recipe.
The discussion
Practitioners mostly agreed with the thesis. The best stories came from people who had replaced vector search with keyword search that an LLM drives itself. Pushback came from finance and multilingual deployments, where keywords break down. A large share of the thread complained that the article reads as LLM-written, and Pierre admitted to having an LLM review his draft. A long side argument covered whether "RAG" should have been spelled out.
Full-text search goes further than people expect
The top comment, from someone who built large RAG systems, said embeddings look magic until you must re-chunk, re-embed and rerank them for precision. A big keyword query just works and is easier to run, and BM25 ships with most databases. In finance, jargon and client-specific meanings made BM25 fail even with an agent rewriting queries, so vector search won despite the cost. Pierre said his team had to build a glossary and "semantic layer" for the same reason.
Embeddings appear to be nice and magic but when you really get into them you notice: semantic similarity isn’t as good as you think and certainly it won’t make everyone happy.
On finance docs and with users that use jargon and acronyms heavily, BM25 can fail on trivial queries... you end up encoding a masters in economics into the query re-writing logic.
Let the agent write its own searches
Several commenters went past one-shot query rewriting to agents that search iteratively, as coding agents do with grep. One let a model write SQL against thousands of documents in SQLite FTS5, and said it beat every commercial product his company tried. A coding-agent vendor dropped its retrieval index after a bug had silently made it return nothing for a long time, and the agent worked better. Skeptics cited poor text-to-SQL benchmarks and the risk of a hallucinating model writing queries; the SQLite user said his database file is read-only.
The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source.
Where keywords fall short
A commenter whose technicians search a multilingual knowledge base said full-text search can't cope, but an LLM fed embedding results answers across languages. Replies suggested one translated index or one index per language, and one called embeddings a cleaner but harder-to-debug form of that translation. A linguist replied that embeddings also fail on names absent from training data, so you need a translation table anyway. Chunking drew its own argument: pronouns that point across chunk boundaries break retrieval, and embedding models cap chunks at roughly 512 to 1024 tokens.
The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
Embedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.
Complaints that an LLM wrote it
Many readers flagged the recipe headings, short punchy lines and phrases like "The Fresh Data Play". One said the hybrid recipes look like what Claude produces when it avoids choosing between two options. Pierre said he only had an LLM review his draft, and that this article drew far more views than his own writing. Separately, one reader found that the $10,000 re-embedding figure contradicts the article's own $10 estimate.
Claude especially will present hybrid options and compromises to avoid having to make a choice then framing the hybrid option as the "best of both worlds" when they're borderline nonsensical.
to be fully transparent, I used LLM to review my draft yes. and interestingly enough, this article had tens of thousands of more views and estimulated significantly more discussions than my other, non-LLM written articles.
RAG is retrieval, not vector databases
Several commenters said RAG is old information retrieval with an LLM issuing the queries, so clean input, ETL, metadata and relevance testing still decide quality. One team now writes product docs with RAG in mind, with clearer headings and terms. Some held that any retrieval before generation, a grep included, counts as RAG, and blamed writers who only covered embeddings for the narrower meaning. Others noted that vector embeddings predate LLMs by decades.
Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG.
Because people writing about RAG never explained what RAG is and exclusively wrote about embeddings and vector dbs, for most people RAG became "embeddings + vector db".
Consensus
Start with keyword search and only add embeddings when users' complaints show you need them. An LLM choosing search terms, or querying the index repeatedly, covers much of what people expect from semantic search.
Dissent
Domains full of jargon, multilingual corpora and descriptive "I don't know the name" queries still favor embeddings. A sizeable group also dismissed the article as LLM-generated, with numbers that don't add up.