What is HyDE? Hypothetical documents for better retrieval
What is HyDE? How an LLM-written hypothetical document improves retrieval in RAG: the mechanism, the pipeline, parameters, and how to measure the gain.
HyDE (Hypothetical Document Embeddings) improves the retrieval side of RAG. Original paper: Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao, Ma, Lin and Callan, 2022) — the HyDE paper on arXiv.
HyDE does not generate the answer and does not replace RAG. It changes one step: it embeds a hypothetical document written by an LLM instead of the question, then searches with it.
New to RAG? Read our plain-English guide to RAG. Or start with free AI tools for small business.
Where HyDE sits inside RAG
RAG has two halves: retrieval (find the relevant passage) and generation (write the answer from them). It runs in two phases:
- Indexing (offline): split documents into chunks, embed them, store the vectors.
- Query (online): embed the question, fetch the top-k nearest chunks, put them in the prompt, let the LLM answer.
HyDE touches one place in the query phase: it replaces the question vector with the vector of a hypothetical document. The vector store, embedding model, top-k and prompt stay the same.
The precise definition: what a hypothetical document is
Given a question, an instruction-following LLM writes a hypothetical document that answers it; an unsupervised contrastive encoder (Contriever) embeds it into a vector; that vector retrieves real documents.
The paper is explicit: the hypothetical document "captures relevance patterns but is unreal and may contain false details". It exists to search, not to answer.
Why does a wrong document work? A bottleneck sits in the middle: the encoder compresses text into a vector by meaning, so false details get squeezed out while topic and terminology survive. The vector is grounded in the corpus.
Why HyDE works
- Questions are short, documents are long. "How long do I have to return an order?" is a handful of tokens; the policy clause is two hundred. Their vectors are far apart.
- Question vectors are out of distribution. The encoder knows document points, not question points — that is why zero-shot dense retrieval loses to BM25.
- HyDE moves the question into document space. The document has the shape of a document, so its vector lands where the encoder expects one.
So the hypothetical document does not need to be factually true. It only has to look like a document: same language, length, structure, jargon.
The pipeline, step by step, with a return policy
The corpus holds a return policy, a shipping FAQ and a refund procedure. The user asks: "How long do I have to return an order, and is there a fee?"
Step 1: the LLM writes a hypothetical document
A prompt pattern, shortened from the paper:
Please write a short passage to answer the question.
Try to include as many key details as possible.
Question: {the question}
Passage:
The output might be: "Customers who bought online can return their order within 14 days. The return fee is 30,000 VND, waived when the item is faulty."
The fee may be wrong, and that is on purpose: the document only has to be right about the topic and the shape of the text, not the numbers.
Step 2: embed the hypothetical document
It goes through the same encoder used to index the corpus. A different model puts corpus and query in two spaces, and the system breaks silently.
Step 3: retrieve the real documents
The vector is compared against the vector store, and the top-k nearest real documents are taken, usually k = 3 to 5: here the return policy clause and the shipping FAQ.
Step 4: answer from the real documents
The answering prompt is the standard RAG prompt: the question, the retrieved passages, and a rule to cite sources. The answer comes from the real documents, so nothing in the hypothetical document decides it.
Pseudo-code for the whole pipeline
def answer(query, corpus, generator, encoder):
hypothetical = generator.write(prompt=HYDE_PROMPT.format(query=query))
vector = encoder.embed(hypothetical)
passages = corpus.search(vector, top_k=5)
return generator.answer(query, passages)
HYDE_PROMPT is the prompt above; the first line costs one extra LLM call per question.
Standard RAG versus HyDE
| Standard RAG | HyDE | |
|---|---|---|
| What gets embedded | The user's question | A document written by an LLM |
| Extra LLM call? | No | Yes, one per question |
| Best for | Queries that match the document wording | Paraphrased questions, zero-shot |
| Weak for | Short queries, vocabulary mismatch | Exact matches, tabular data |
Straight answer: HyDE does not replace RAG. The answer still comes from the retrieved real documents. HyDE changes the input vector of the search step, not the source of knowledge.
Parameters and variants
- One hypothesis or several: generate several documents, then average their vectors to reduce noise; three to five beat one, and the cost multiplies.
- Temperature: low (0 to 0.3) for specific questions, higher (0.5 to 0.8) for vague ones; too high drifts off topic.
- Max tokens: roughly 150 to 300; too long bleeds into other topics, too short repeats the question.
- Generator size: smaller models are cheaper but write weaker documents on jargon.
- Language of generation: it must match the corpus — a Vietnamese corpus needs a Vietnamese document.
- Caching: repeated questions generate the document once and reuse it, removing almost all of the LLM cost.
- The original query vector: average it with the hypothetical vector, or use the hypothetical one alone.
How to measure it
Start with a golden question set: 20 to 50 questions, each tied to a known correct passage. Run two configurations on the same corpus, model and top-k.
- recall@k: how many of the k retrieved passages contain the correct one? HyDE shows the clearest gain here.
- nDCG: whether the correct passage is ranked high; it matters when top-k is small and the context is cut.
- Latency and cost per query: HyDE adds one LLM call, usually the largest jump in response time.
- Failure analysis: read 10 to 20 missed questions and group the causes: query too short, document off topic, bad chunking, corpus without the answer.
Published results: HyDE beats zero-shot Contriever, the version without fine-tuning, and matches fine-tuned retrievers on web search, question answering, fact verification and multilingual retrieval. No single number applies to every dataset: measure on your own corpus.
When HyDE helps and when it hurts
HyDE helps when the question is vague or paraphrased; when the corpus uses other wording (hand-written FAQs, internal procedures); when you have no fine-tuned retriever and need better zero-shot quality; and when you retrieve across languages.
HyDE is useless or harmful when:
- Exact-match lookups: SKU codes, invoice numbers. BM25 matches literal strings far better.
- Numeric or tabular data: price lists, financial reports. A hypothetical passage cannot describe a table.
- Latency or cost budgets are tight: one extra LLM call per question.
- The corpus is tiny: everything already fits in the prompt.
- Your retriever is already fine-tuned for your domain: the extra cost stops paying off.
- The encoder is single-language while questions and documents are not.
The safe combination is hybrid search: BM25 for literal keywords, dense with HyDE for paraphrases, then rerank with a cross-encoder. Our guide to combining dense and sparse retrieval covers how the two score sets get merged.
Running HyDE in Vietnamese
- Generate the hypothetical document in Vietnamese. The vector must live in the space of the Vietnamese corpus.
- Use a multilingual embedding model that handles Vietnamese well, and re-index the corpus with it. A model mismatch is the classic silent bug.
- Keep BM25 for codes, rare terms and accented proper nouns. "Điều 5", "HĐ-2026-118": dense retrieval misses these because the strings carry no meaning.
- Diacritics and domain jargon. A tokenizer can split Vietnamese words in unexpected ways and shift the query vector; use the source jargon.
- A weak generator caps the whole gain. If the LLM writes poor Vietnamese documents, HyDE stops helping, so use a large LLM for this step.
Frequently asked questions
Does HyDE hallucinate?
Yes, and the paper says outright that the hypothetical document "is unreal and may contain false details". But it never reaches the answer, it is only the input to the search step. The real risk is indirect: a wrong document pulls the top-k towards the wrong topic.
Does HyDE replace RAG?
No. HyDE is a technique inside RAG that changes how the query vector is computed. Remove HyDE and the system is still RAG; remove RAG and you have vector search.
Do I need to fine-tune?
Not necessarily. HyDE was designed for the case where you have no relevance labels, the zero-shot case, so it fits teams without training data. If you already fine-tuned a retriever, compare both on your golden questions first.
Conclusion
HyDE is a small change in code: instead of embedding the question, embed a hypothetical document written by an LLM, then use that vector to find real documents. It does not replace RAG and costs one extra LLM call per question.
Turn it on when questions are vague and the corpus uses other wording; turn it off for exact matches, tabular data, or a tight budget. Measure first. To start with a working tool today, see our roundup of free AI tools for small business.