Your AI chatbot did not die because the model was weak

The bot still quotes last quarter's price list correctly. Nobody reads the log, so nobody notices the policy changed underneath it until people complain.

Flat illustration: a chat bubble quoting an old price list beside a log record nobody has read

The bot answers a price question at nine in the evening, in a clean, helpful sentence, and the number in it comes from the price list the shop stopped using in July. The owner calls it a model problem. It is a problem of who was supposed to be watching.

The claim, in the order that matters

Model quality and retrieval quality set a ceiling. Scope, ownership and evidence set a floor. A company of ten to fifty people rarely reaches the ceiling; it falls through the floor first. The floor has three legs, all three paperwork rather than technology: a written list of what the bot may answer, one person who owns a wrong answer, and a record showing that an answer was wrong.

Each leg has a longer treatment elsewhere: chatbots that answer from your own documents, moving a Zalo OA conversation to a person, and what hallucination is.

When this article does not apply. Run the twenty questions your customers actually ask through the bot. If most come back with no supporting document, or the wrong passage, the pipeline is the real problem and the three legs below will not save it. Fix retrieval, then re-run those questions.

The belief: buying the tool ends your involvement

The invoice buys a capability, not an outcome. What arrives answers from the documents it is given, in a tone that does not distinguish between a clause it read and a gap it filled. Everything that makes those documents true — the price list, the return rule, the delivery promise — stays inside the business and changes on the business's schedule.

The model is not weak. It is unsupervised, which is a different problem with a different fix. The contract covers uptime, the endpoint, the version the vendor runs; it does not cover the sentence your staff sends to a customer. When the bot quotes last quarter's price, the customer holds you to it. A wrong price is a discount nobody authorized; an invented return rule is a commitment you have to honor or explain away.

This is the same mistake as hiring: nobody assumes a new hire removes the need to manage the work.

What the bot may answer

The first artifact is one dated page with two columns, kept next to the price list, not in a chat thread with the vendor. The left column lists what the bot may answer: opening hours, how to order, product descriptions with a version date, published policies. The right column lists what it may not: prices not in the current price list, stock counts, delivery dates, tax and legal questions, one customer's order, competitor comparisons, unapproved discounts.

The refuse column gets skipped, and it is the one that matters, because every entry needs a destination. Two destinations are enough: ask a clarifying question, or hand the conversation to a named person.

Flow: a retriever returns fixed top-k chunks, the model writes, then a gate checks the allow list and the cited document before answering, re-asking or handing off

Figure 1. The gate sits after the model, not inside retrieval.

Why retrieval returns top-k and never silence

A retriever returns the k nearest passages to a question, and k is a number somebody set at install time. It has no concept of nothing matching: with the usual settings it returns five or ten chunks whether or not any of them answers the question, and the model writes from whatever arrived. Silence is not a behaviour the retriever has. It has to be added above it, as a check before the answer leaves — is this intent on the allow list, does the cited document exist, is its version date current.

Placement matters too. Liu et al. found that relevant content placed in the middle of a long context is used less reliably than the same content at the start or end (Lost in the Middle, 2023). A bot can have the right clause in front of it and still answer from something else, which is why the cited document is what gets logged.

Who owns a wrong answer

A role is not an owner. The team is not an owner. The artifact is a name, written on the same page as the allow list, with four things attached.

The first is time: thirty to forty-five minutes a week, on a calendar, for reading flagged answers and handoffs. The second is authority to edit the source document; if the owner must ask the developer to change a price line, the loop stalls at the first correction. The third is an escalation rule with a response time: a wrong price, a refused refund or a promise the business cannot keep reaches that person within a stated number of working hours, and the customer gets a human reply in the meantime. The fourth is a second name, because the first will take a holiday.

The log, the permission, and the rollback path

Three things must exist on the day the bot goes live, not the day it first embarrasses someone. Read access to the conversation log, so the owner can see what was said without asking for an export. Edit access to the document store, so a correction takes minutes. And a rollback path: the previous version of a document, plus a switch that narrows the bot to the safest entries on the allow list — hours, ordering, published policy — without a rebuild.

Without a rollback path the owner has two choices, keep the bot as it is or switch it off, and switching it off costs every conversation it was handling.

What proves the bot is wrong

A spot check by the builder is the weakest evidence available. Three artifacts replace it.

A frozen question set: twenty questions your customers actually ask, each with the answer you consider correct, written before launch and stored with a date. Frozen means the file is not edited to match the bot.

A threshold signed before launch: a number of correct answers the owner accepts, plus the categories where one wrong answer is too many — prices, refunds, delivery promises. Signing in advance turns an argument about quality into a measurement.

A review date: monthly while the document set is moving, then quarterly, and after every price change or new product line.

Barnett et al. catalogued seven ways a retrieval-augmented system fails: missing content, a missed top-ranked document, a document retrieved but not placed in context, content not extracted from the context, the wrong output format, an answer at the wrong level of specificity, and an incomplete answer (Seven Failure Points When Engineering a Retrieval Augmented Generation System, 2024). All seven survive a spot check and still cost a customer.

What one log record holds

Six fields, and the fourth is the one usually dropped:

  • Message id, so the conversation can be found again.
  • Intent label: price, return, delivery.
  • The answer the customer received, verbatim.
  • The cited document and its version date.
  • The decision: answer, re-ask or handoff.
  • The reviewer, and the date it was read.

The version date separates "the bot was wrong" from "the document was old". Without it, every review re-opens the same argument.

One log record with six fields: message id, intent label, answer, cited document and version date, decision, reviewer

Figure 2. One record, six fields. The version date turns an opinion into evidence.

Why it breaks in month three, not week one

The failure is a sequence, and each step leaves a signal that was visible at the time.

Week one is novelty. Staff ask questions they already know the answers to, and the bot performs, because those questions are in the documents.

Then edge cases arrive in ones and twos: an exchange on a discontinued item, a warranty question about a product the shop no longer sells, a bundle price that never became a document. Each is small enough to answer by hand, so it is answered by hand, and the document set stays where it was.

Then the person who knew which documents were authoritative changes role or leaves. The document set does not break; it loses its map, and the older version of a policy is still in the folder.

Then the vendor retires a model version or moves an endpoint, and the answers change length and tone before they change accuracy. The version pinned in the code is no longer served.

By the end, staff answer in parallel because it is faster than checking the bot. Nobody has called the bot dead; it is bypassed, which is the same outcome with less noise. Every step was visible in a log nobody read.

Diagnostic table: five ways it fails, and what to fix

Read the middle column if you built the thing; the first if you signed for it.

What the owner sees What the developer sees What to fix
A customer quotes a promotion that ended months ago Chunks dated older than the policy they describe Re-index, stamp version dates, re-run the frozen set
Answers get shorter or stranger, or fail at night, with nothing changed A deprecation notice on the pinned model version Pin versions, watch provider notices, re-run before switching
The person who fixed the bot has gone; nobody knows which file is right Log review stopped on a date, no owner on record Name a second owner, write the document map
Staff answer manually in parallel; regular customers ask twice Handoff rate climbing while the error rate is flat Narrow the allow list, fix the top wrong intents
The bot appears in a second channel nobody scoped New intents with no documents and no refusal path Re-open the allow page, add refusals before coverage

The work left after the tool arrives

None of it ships with the tool. The package contains a model endpoint, a retrieval pipeline and an admin screen. It does not contain your allow list, a name, a frozen question set with your customers' questions in it, a signed threshold, or a log record shape. Vendors ship the ceiling; the floor is assembled on site.

That work is domain-specific and it recurs. A price change is a document edit, a re-index and a re-run. A new product line is a new section of the allow list and new refusal cases. A change of staff is a handover of the document map. A retired model version is a test run before the switch. None of it can go to the vendor, who does not know which of your two price files is the real one.

A chatbot is a small, permanent piece of operations delivered as software. The purchase ends the build. It does not end the work.

Questions this article can answer

Should the owner also be the developer?

No, and the two duties should not sit with one person if it can be avoided. The developer makes the pipeline find the right passage; the owner decides what may be answered and corrects the source. When one person holds both, corrections happen without a record and the review becomes a self-check.

How often should the frozen question set be re-run?

On three triggers, not a calendar alone: after any change to a document the bot reads, after any change to the model version or retrieval settings, and on the review date written down at launch. The document trigger catches most failures, because the bot's knowledge changes when the price list does.

What happens when the vendor retires the model version?

The answers change before anyone notices, because the endpoint keeps answering. Treat the notice as a change event: run the frozen set on the replacement, compare it answer by answer with the version in production, then switch. With no set, the retirement is the moment to build one.

Sources

  • Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., and Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. arXiv:2401.05856. arxiv.org/abs/2401.05856
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172. arxiv.org/abs/2307.03172