From machine learning basics to today's RAG stack for shops

Machine learning, embeddings, retrieval, rerankers, graph RAG, agentic RAG: each rung explained with a small-shop example, plus where to stop. which tool first.

Flat illustration: machine learning, embedding and retrieval blocks stacked as ascending steps with an upward arrow

Ask a shop assistant how long the return window is and they check the policy. Ask a chatbot and it answers from what it absorbed during training: last year's policy, or an invented rule that sounds official.

This article continues what RAG is, the plain-English guide to the basic loop. Here we start further back, at the machine learning basics these systems sit on, and walk forward to today's RAG stack. Each rung gets a small-shop example and an honest price.

Where a wrong answer starts

The model has no access to your documents. It is trained once, then frozen. Change the return policy in March and nothing inside the model changes: it answers from March 2025, or from plausible text it invented.

You ask, the model guesses, and you cannot tell the guess from the fact. The fix is not a bigger model; give the model something to read first.

Machine learning, in one sentence

Machine learning turns examples into a function. Show a system many input-output pairs — tickets labelled "billing" or "technical" — and it adjusts internal numbers until its output is right for most inputs. Those numbers are the model.

For a small business, machine learning automates a decision a person could make from examples, without anyone writing the rule. You do not tell it "route refunds to billing"; you show it 2,000 routed tickets and let it find the pattern.

Three ideas you need before touching RAG

Training versus inference. Training is the slow, expensive part where the numbers move; inference is the fast part where a trained model answers one new input. Every chatbot you have used is inference. A trained model is frozen and cannot learn from the document you hand it on Tuesday, so you must retrieve that document and put it in the prompt. That is the whole job of RAG.

Generalisation. A model is judged on inputs it never saw. Train only on one product line and it is shaky on another. Retrieval narrows the question to your documents before the model speaks.

How data is represented. Text must become numbers. Words become tokens, tokens become numbers, and that shape decides what the model can notice. A bag-of-words model cannot tell "not good" from "good" because it threw the order away; a transformer reads the sequence and weighs how each word relates to the others. That is the bridge to embeddings.

Embeddings: turning sentences into coordinates

An embedding is a sentence compressed into a list of numbers, a few hundred to a few thousand of them. Think of a coordinate in a very high-dimensional room: similar meanings land close together, unrelated ones far apart. Training makes "what is the return window" and "how long can I send something back" neighbours.

You cannot read the numbers, only compare them: a small distance means similar meaning. A customer asking "can I return this after 30 days" and a clause saying "returns are accepted within 30 days of purchase" end up close together. That is what makes retrieval possible.

Three columns, machine learning, embeddings and retrieval, each showing what it decides and where it breaks

Figure 1. The first three rungs: a model learns from your data, embeddings turn sentences into coordinates, retrieval fetches the nearest passages.

What it costs. One embedding pass over every document: minutes of compute and a small monthly bill. Where it breaks. Sarcasm, negation, jargon and opposites all degrade the coordinates.

Retrieval: matching meaning, not words

Retrieval answers one question: which documents are relevant to this query? The classic version is keyword search, matching words and weighting them by rarity. Excellent for product codes and policy numbers, useless for a paraphrase. Vector retrieval does the opposite: it ignores word overlap and compares meaning, embedding the question like every stored passage and returning the nearest neighbours. Keyword search finds "invoice 2024-07" every time; vector search finds "the bill for July last year". Neither does both.

The fix is hybrid retrieval: run both and merge the ranked lists, usually with a rule such as Reciprocal Rank Fusion. Our guide to dense versus sparse retrieval has the details.

What comes back is five to twenty passages, and their quality decides most of the final answer. Return only what you need: extra passages dilute the good ones.

Rerankers: fixing what retrieval got wrong

Retrieval is fast and approximate; nearest is not the same as most relevant. A reranker is a second, slower model that re-scores the shortlist for how well each passage answers the question. It reads both texts together, and it is usually a cross-encoder: one forward pass over question and passage.

This is the cheapest quality win in the stack. Keep retrieval as a broad net, then spend a little compute on the few candidates; the top passage is the one the model leans on.

It also fixes what embeddings cannot. "Returns are accepted within 30 days" and "returns are not accepted after 30 days" have nearly the same embedding; a reranker can tell them apart. Our comparison of cross-encoders versus bi-encoders covers the trade-off.

What it costs. A little latency and a small model to serve. What it cannot do. Retrieve a passage that is not in the index.

Today's RAG ladder: five rungs

Five ascending steps from basic RAG to agentic RAG, with an arrow marking rising cost and complexity

Figure 2. Five rungs of the modern RAG ladder. Cost and complexity rise with each step, and most businesses stop at rung one or two.

Rung 1: basic RAG. Embed the documents, embed the question, fetch the nearest passages, put them in the prompt, answer. It solves "the model does not know our documents" for a few hundred dollars a year; it breaks on paraphrases and on a model that paraphrases instead of citing.

Rung 2: hybrid retrieval plus a reranker. Add keyword search and a reranker. It fixes most paraphrase failures and most "on topic but wrong" failures for a modest jump in complexity. If you do one upgrade past rung one, do this.

Rung 3: query rewriting, including HyDE. Here the documents are fine and the question is the problem: a customer asks "what if the package arrives late", the policy says "delivery timelines". Query rewriting restates the question in the documents' language first. HyDE, or Hypothetical Document Embeddings, has the model write a short fake answer, embeds that, and searches for real passages like it. The fake answer is discarded; only its shape is used. Our guide to what HyDE is and when it helps covers the limits.

Rung 4: graph RAG and multi-hop. Some answers need several documents chained: which supplier serves which product, which contract covers it, what the return window is. Plain retrieval returns one passage at a time and the model holds the chain in memory, which is where it drifts. Graph RAG extracts entities and relationships into a graph and walks it, which earns its keep on relational data.

Rung 5: agentic RAG. The system becomes an agent that chooses tools: it rewrites its queries, searches the website, queries a database, checks the result, and retries when the answer is thin. This is where the what is an AI agent guide belongs: an agent is a model with tools and a loop, and RAG is one. Most flexible and most likely to surprise you.

Which rung you need

Your situation Rung What it buys What it costs
Few policies, website chatbot 1, basic RAG Grounded answers A few hundred a year
One question, many phrasings 2, hybrid plus rerank Recall on paraphrases A reranker; latency
Jargon customers do not share 3, query rewriting and HyDE Queries match documents One extra call per query
Facts across several documents 4, graph RAG Multi-hop answers A graph to refresh
Must look data up and act 5, agentic RAG It picks the steps More parts, more cost

Each rung buys one failure mode and charges for it in latency, money and maintenance. Most never need past the second row.

When to stop

Stop when the next rung would not fix a failure you actually see. Collect twenty real questions, run them through the current system, and read the answers. If most are grounded, you are at the right rung. "Could not find the passage" is retrieval and chunking, and our chunking for RAG guide is the cheaper place to start. "The passage was there but the answer is wrong" is a reranker problem.

Do not build rungs four and five because they sound impressive. A shop with three hundred pages of policy needs the right passages found and cited, not an agent over a knowledge graph.

And a model can still hallucinate when the retrieved passage is correct, more readily when the context is long. Every extra rung lengthens it.

Frequently asked questions

Is RAG a replacement for fine-tuning? No. Fine-tuning changes a model's behaviour by training it on your examples; RAG gives it your documents at query time. They fix different problems and combine well. If the problem is "it does not know our policy wording", RAG is the direct fix; if it is "it writes in the wrong tone", a better prompt is cheaper.

What does a basic RAG system cost a small business? The running costs are an embedding model and a reranker, sold as hosted APIs at a few dollars per million tokens, plus storage. For a few hundred pages the annual bill is usually under a thousand dollars. The hidden cost is maintenance: documents change, so the index must be refreshed.

Can RAG answer questions about what happened yesterday? Only if yesterday's events are in your documents. RAG does not add knowledge to the model; it adds documents. To answer from yesterday's sales, put that data in the index, or at rung five give the agent a tool that reads your database. The knowledge comes from sources you control.

The short version

Machine learning turns examples into a function. Embeddings turn sentences into coordinates. Retrieval fetches the nearest passages. A reranker re-scores them. That is the stack most small businesses need, and for most it is the whole stack. Query rewriting, graph RAG and agentic RAG each fix a narrower failure and add moving parts. Build the first two rungs, measure the failures on real questions, and climb only when they point at a specific rung.