Dense vs sparse retrieval: how they differ in RAG for shops

How dense and sparse retrieval differ in RAG: BM25 versus vector search, hybrid fusion with RRF, the metrics to track, and notes for Vietnamese content.

Flat diagram comparing the sparse and dense retrieval paths in a RAG pipeline

Dense retrieval embeds text into vectors and compares meaning; sparse retrieval matches keywords with BM25. Both find relevant passages in RAG, but they win on different questions: dense for paraphrases, sparse for exact codes and rare terms.

The two differ in one fundamental way: what a document looks like when it sits in the index. If the retrieval step is still fuzzy for you, start with a practical RAG guide for small business and come back here for the internals.

What is sparse retrieval?

Sparse retrieval reduces a document to a bag of words: a list of terms with their frequencies, with word order and grammar thrown away. The vector is sparse because most dimensions are zero.

To search fast, systems build an inverted index — a lookup table from each term to the list of documents containing it. A query is split into terms, each term is looked up, and the matching documents are scored.

The classic scoring function is TF-IDF: term frequency multiplied by inverse document frequency. The modern standard is BM25, which adds three refinements: saturated term frequency, inverse document frequency weighting, and length normalization.

Strengths:

  • Exact matching. Contract codes, invoice numbers, and product SKUs only surface when the exact string is compared.
  • Explainable. You can point at the keyword that made a document match.
  • Cheap and fast. No model, no GPU, and new documents enter the index immediately.

Weaknesses:

  • No sense of meaning. A question about how long a customer has to send an item back misses a passage titled return window — no content word is shared.
  • Stop words add noise. Words such as the or is appear everywhere and dilute the score.
  • Synonyms need hand-written lists. You must map buy to purchase yourself.

What is dense retrieval?

Dense retrieval does not count words. It embeds text into a fixed-length vector — usually 384, 768, or 1024 dimensions. A bi-encoder embeds the question and the document independently with the same weights; the SBERT paper on arXiv is the usual reference point.

Document vectors live in a vector database. The question is embedded too, then the nearest vectors are found. Brute-force scanning is too slow for large collections, so approximate nearest neighbour search (ANN) structures like HNSW or IVF trade a little accuracy for a lot of speed.

Strengths:

  • Understands meaning. Buy and purchase land close together in vector space even without a shared token.
  • Handles paraphrase. A question phrased five different ways returns the same passage.
  • Better across languages. Multilingual models place a Vietnamese question and an English document in one space.

Weaknesses:

  • Codes and numbers are near-noise. A string such as INV-2026-118 produces no meaningful signal, so dense retrieval often misses exact identifiers.
  • More expensive. You pay for the embedding model, the vector store, and re-embedding the corpus whenever the model changes.
  • Hard to explain. A score is just a distance between vectors; it does not say which term carried the match.
Diagram comparing the sparse and dense retrieval pipelines
Figure 1. One question, two ways to represent text, two ways to produce a score.

Side-by-side comparison

Criterion Sparse retrieval Dense retrieval
Representation Bag of words with counts Dense embedding vector
Matching Literal term overlap Semantic, vector distance
Synonyms Must be declared by hand Handled if the model is good
Codes and exact numbers Excellent Weak, often missed
Cost and latency Cheap, fast, no GPU Higher: model, vector store, ANN
Index updates New documents searchable immediately, no re-embedding Must embed new documents; a model change requires a full re-index
Explainability You can name the term that matched Hard; only a vector distance
Multilingual Weak without normalization or a dictionary Strong with a multilingual model

Where BM25 still wins

First: saturated term frequency. A term appearing ten times does not mean a document is ten times more relevant than one where it appears once. BM25 compresses that growth with a logarithmic function.

Second: inverse document frequency weighting. Agreement appears in every file and separates nothing; INV-2026-118 appears in one document and is therefore extremely precious.

Third: length normalization. Longer documents have more chances to match a query term, so BM25 applies a correction factor to keep them from dominating every ranking.

The Okapi BM25 formula on Wikipedia documents each component in detail.

Why hybrid wins in practice

Pick one method and you lose on a specific class of questions. Pick both and fuse the results — that is hybrid retrieval.

The most common fusion method is RRF (Reciprocal Rank Fusion): each document scores by summing 1/(k + rank) across the two lists, with k commonly set to 60. RRF only needs ranks, so it sidesteps the problem of BM25 scores and cosine scores living on different scales. The alternative is a weighted sum: the final score equals alpha times the BM25 score plus beta times the cosine score. That is clearer, but alpha and beta need tuning on real questions.

After fusion, add a reranking step: a heavier model scores each question-passage pair, which is slower but far more accurate, and only then is the top-k cut for the prompt; cross-encoder reranking covers that step. Dense retrieval also improves before the search starts: how HyDE works in RAG has the model write a hypothetical passage first and embeds that instead of the raw question.

Diagram of hybrid retrieval: BM25 and dense, RRF fusion, reranking, top-k, LLM
Figure 2. Hybrid retrieval: two search branches, one fusion step, one reranking step.

Which one fits your collection

  • Codes, SKUs, contract clauses, invoice numbers → sparse.
  • Natural questions and paraphrases → dense.
  • A mix of both kinds of question → hybrid, and it should be your default.
  • A very small corpus, a few dozen files → build nothing. Put the documents straight into the prompt.
  • Several languages in one collection → dense with a multilingual model, plus sparse to keep identifiers intact.

If you would rather have it built, our software development team designs, builds and runs RAG systems.

Measuring and operating it

Pick a method with evidence, not with taste. Build a golden set of 20–50 real questions, each with the passage that answers it, and measure:

  • recall@k: does the correct passage appear in the top k? RAG can only use what retrieval surfaces.
  • nDCG: how highly is the correct passage ranked? It separates found it from found it first.
  • Latency and cost: measure query embedding separately from the search itself.
  • Index freshness: how long before a new document shows up in results.

Three operating rules apply to both sides:

  • Chunking affects both. Chunks that are too small lose context; chunks that are too large mix topics. Split on sentence boundaries and overlap by 10–15%.
  • Use the same embedding model for indexing and querying. Changing the model without re-indexing fails silently: no error, just bad answers.
  • Normalize vectors (L2) before using dot product so the score is cosine similarity; otherwise vectors with a larger norm dominate.

Notes for Vietnamese content

  • Diacritics matter. Đổi trả and doi tra are different strings, and BM25 will not know they are the same. Strip diacritics on both index and query side, keeping the original for display.
  • Tokenization decides BM25 quality. Splitting on whitespace performs far worse than a diacritic-aware tokenizer; LLM subword tokenizers shred keywords and blur the score.
  • Use a multilingual embedding model, not an English-only one. English-trained models are noticeably weaker on Vietnamese; index with the same model you serve queries with.
  • Write synonym pairs by hand. Map đổi trả to hoàn hàng and hủy đơn to huỷ đơn, then feed the variants into query expansion or the index.
  • Re-measure on Vietnamese questions. An English golden set tells you nothing about a Vietnamese corpus.

Frequently asked questions

Does dense retrieval replace BM25?

Not entirely. Dense retrieval wins on paraphrase; BM25 wins on identifiers and rare terminology. If you can only ship one, start with BM25: it is cheaper and easier to operate.

Do I need a separate vector database?

Not necessarily. With a few thousand chunks, an in-process vector index is enough, and most AI platforms include a hosted vector store. Consider a dedicated store when the corpus is large, when many people query at once, or when you need predictable latency.

Is sparse retrieval just keyword search?

Close, but the terms describe different layers. Sparse retrieval is a representation: text becomes a sparse term vector. Keyword search is an application built on that representation. BM25 is one specific scoring function for sparse retrieval, not a complete search system. The shared limit: neither understands meaning, so a question about sending an item back still misses a passage titled return window.

Conclusion

Sparse retrieval counts words, dense retrieval measures meaning, and real RAG systems use both. Starting out: build BM25 first because it is nearly free; add the dense branch when paraphrased questions start to fall through; fuse with RRF and rerank the top results; then measure with your own golden set. If you would rather not build any of it, our roundup of free AI tools for small business has ready-made options.