Cross-encoder vs bi-encoder: why RAG needs reranking
What a cross-encoder is, how it differs from a bi-encoder, why RAG reranks, what it costs in latency, and how to measure it with nDCG@10. on customer documents.
Vector search returns passages that look relevant. Read them closely and many are on topic but wrong on the detail that matters. That is not the embedding model failing; it is how it scores, and it is why serious RAG systems add a step after retrieval.
Retrieval has two separate problems. Recall asks whether the passage that answers the question is in the pool at all. Precision asks whether the best passage is at the top of the list you hand to the model. A fast scan of the whole corpus wins on recall; a slow scorer over a short list wins on precision. Those two jobs use different architectures: a bi-encoder and a cross-encoder. This is why RAG needs reranking rather than a bigger vector index.
New to RAG? Start with a practical guide to retrieval-augmented generation.
What is a bi-encoder?
A bi-encoder encodes the question and the document independently, each through the same weights in its own forward pass, producing two vectors. Relevance is the dot product or cosine between them.
Document vectors are computed ahead of time, at indexing. At query time the system embeds the question once and looks up the nearest vectors with ANN, so the cost per query barely moves when the corpus grows from a thousand passages to a million.
The trade-off is compression: a 400-token passage becomes one 768-dimension vector, leaving no room for token-level interaction. Ask "how many days do I have to return an item?" and both "Returns accepted within 14 days" and "Refunds are processed within 14 days" sit close to the question, though only the first answers it; nothing checks whether 14 days belongs to returns or refunds.
What is a cross-encoder?
A cross-encoder does the opposite. It joins the question and the document into one sequence — [CLS] question [SEP] document [SEP] — and sends the whole thing through a transformer in a single forward pass. Attention sees every token of both sides at once, so each question token is compared directly with the phrase that matters. The output is one scalar: the relevance score for that pair.
It reads 14 days in the context of refunds and scores that passage well below the returns one. It also catches negation ("not valid on sale items") and conditions ("only with tags attached").
Nothing can be precomputed: the score depends on the pair, so a new question means new forward passes. Cost is O(N) passes for N candidates: reranking 100 passages is fine, reranking a million per question is impossible on latency and budget. That is why a cross-encoder only runs in stage two, over a short list from a bi-encoder or BM25.
Figure 1. Left: a bi-encoder embeds each side separately, then compares vectors. Right: a cross-encoder reads both texts in one forward pass and emits a score.
Side by side
| Criterion | Bi-encoder | Cross-encoder |
|---|---|---|
| Input | Question and document separately, two passes | One joined [query, document] sequence |
| When it runs | Documents: at index time. Question: per query | Every pair, every query, never cached |
| Precomputation | Yes, for the whole corpus | No |
| Cost per query | 1 embedding + 1 ANN lookup, about 10–40 ms for 1M vectors | N passes: 100 pairs in roughly 60–250 ms on a GPU |
| Ranking accuracy | Decent, loses token detail | Clearly better on hard pairs |
| Best for | Broad filtering, recall | Reordering a short list, precision |
| Typical use | Fetch the top 50–100 candidates | Cut to the top 3–5 for the prompt |
The two-stage pipeline
- Retrieve. Run BM25 and dense search in parallel, fuse with RRF, keep the top 50–100.
- Rerank. Send those 50–100 pairs through the cross-encoder, sort by score, keep the top 3–5.
- Generate. Only those 3–5 passages go into the LLM prompt.
Why is 100 affordable and a million not? Rerank cost scales linearly with the number of candidates, while retrieval cost barely moves. A small GPU handles 100 pairs in tens to hundreds of ms; a million pairs multiplies that by ten thousand. The number 100 is not arbitrary either: first-stage recall usually reaches 90–95% within the top 100.
Budget it explicitly: query embedding 10–30 ms, BM25 5–15 ms, ANN 10–40 ms, reranking 100 pairs 60–250 ms, generation 1–3 s. Under a 500 ms budget, cut to 20–30 candidates before dropping reranking.
Figure 2. Hybrid retrieval returns 100 candidates, the cross-encoder reorders them, five reach the LLM. The left half is fast and broad, the right half slow and precise.
The retrieval side has room too: how HyDE improves RAG retrieval has the model draft a hypothetical passage and embeds that instead of the raw question, and dense and sparse retrieval compared covers the fusion.
Minimal code
The sentence-transformers library ships a CrossEncoder class, which is exactly what stage two needs:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)
def rerank(query: str, candidates: list[dict], top_k: int = 5, min_score: float = 0.0):
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs, batch_size=32, show_progress_bar=False)
ranked = sorted(zip(candidates, scores), key=lambda item: item[1], reverse=True)
return [(c, float(s)) for c, s in ranked[:top_k] if s >= min_score]
Batching matters: 100 pairs one at a time is several times slower than batch 32, because the GPU idles between calls. Calibrate min_score on real data — predict returns logits, not probabilities, so 0.0 does not mean "50% relevant".
Choosing a model and Vietnamese
For a Vietnamese corpus the first criterion is a multilingual model. BAAI/bge-reranker-v2-m3 covers many languages and accepts long input; the ms-marco-MiniLM family was trained on English only. Pairing an English reranker with Vietnamese documents throws no error and crashes nothing — it simply ranks worse, silently, and you will not notice without a measurement set.
Input length is the second trap. Most cross-encoders truncate at 512 tokens; a condition buried at the end of a chunk is dropped before the model reads it. bge-reranker-v2-m3 accepts up to 8192 tokens, but cost grows with length, so chunks of 300–500 tokens remain a good balance.
Third, tokenizer and diacritics: Vietnamese with and without tone marks are different strings, so if your corpus mixes both, check the model handles both.
Measuring the gain
Build a golden set of 20–50 questions, each labelled with the passage that answers it, then track four numbers:
- nDCG@10: ranking quality in the first ten results — whether the right passage is first, not merely present.
- recall@k: is the correct passage inside the top k? Measure before and after reranking to see what the extra step recovers.
- p95 latency, not the average: users feel the tail, and a bad batch setting creates one here.
- Cost per 1,000 queries: pairs scored times forward-pass price, plus hourly GPU spend.
The most trustworthy comparison is an A/B run on the same question set: one pass without reranking, one with, then compare nDCG@10 and p95. An nDCG gap under 0.02 is usually noise. Just as important, read the 20 worst results by hand — metrics will not tell you whether reranking failed because a chunk was truncated or because the sources contradict each other.
Finally, published numbers deserve caution. Papers usually report gains of a few nDCG points on English, domain-specific benchmarks, and those gains do not transfer automatically to your corpus: different language, different chunk size, different questions.
When you do not need reranking
Reranking pays off in most RAG systems, not all.
- A tiny corpus. A few dozen documents belong in the prompt; a reranker is extra machinery.
- A very tight latency budget. Under 300 ms total, 100–250 ms of reranking will not fit.
- Streaming. If tokens start streaming before reranking finishes, the step only delays the first token.
- Exact identifier lookups. Order IDs, SKUs, contract numbers: BM25 already gets these right.
Prefer a tool over a pipeline? Our free AI tools for small business roundup is the cheaper start.
Frequently asked questions
Does a cross-encoder replace the vector database?
No. The vector database stores and searches millions of vectors — the infrastructure of first-stage retrieval. A cross-encoder stores nothing and searches nothing; it scores a pair of texts you already have. Drop the database and you cannot scan the corpus; drop the cross-encoder and you lose the reordering.
Does reranking make a chatbot slow?
Yes, but usually less than people fear. Reranking 50–100 pairs costs 60–250 ms on a GPU, while the LLM needs 1–3 s to write the answer. If latency is a problem, shrink the candidate list and raise the batch size before removing reranking.
Do I need to fine-tune a cross-encoder?
Usually not at first: off-the-shelf multilingual models handle most cases. Consider fine-tuning only with at least a few hundred labelled question-passage pairs and systematic remaining errors — for example, the model always ranks a product description above a policy clause. Fine-tuning on small, noisy data tends to do worse than the base model.
Conclusion
A bi-encoder scans broadly and cheaply; a cross-encoder reads closely and expensively; RAG uses both, in that order. If you are starting out, build the two-stage pipeline with 50–100 candidates, pick a multilingual reranker, and measure it against your own golden set. If you want this designed and operated for you, our engineering team covers model choice through measurement.