Preventing hallucinations in customer support bots

Why a grounded system still fabricates

The naive story is that hallucination happens when the model has no information. In a RAG system the more common cause is the opposite: the model has information, just not the right information.

Vector search always returns its nearest neighbours. Ask a question your content has never addressed and you do not get an empty result — you get the least-unrelated passages available. Handed three tangential paragraphs and an instruction to answer, a capable model will synthesise something coherent from them. It is doing what you asked.

So the problem is rarely the generation step in isolation. It is that nothing between retrieval and generation decided the retrieved material was too weak to build on.

Four layers, in the order they help

None of these is sufficient alone. Together they move the fabrication rate from noticeable to rare.

• —A retrieval floor — if the best match is too weak, stop before the model call. Cheap, catches the clearest cases, and saves the cost of generating a fallback you already knew you would send.
• —A constrained prompt — state that only the supplied context may be used, and that saying "I don't know" is a correct answer rather than a failure. Models comply with this more readily than people expect, if you mean it.
• —Self-assessment in the same call — ask for structured output including how well the context covered the question and whether it was used. Free, and it comes from something that read the passages.
• —A combined gate — blend the retrieval signal and the model's self-report before deciding to answer. Each is unreliable in different circumstances, which is exactly why the combination is more robust.

Thresholds are a property of your content

The temptation is to copy a similarity threshold from a blog post. Do not — including from this one, which is why we have not printed ours.

We initially set our retrieval floor conservatively high and found it refusing questions the content plainly answered. The cause was chunk shape: large multi-topic sections from PDFs without headings produce lower similarity scores for perfectly good matches, because the chunk's vector is an average of several subjects. The score was not measuring what we assumed.

We lowered the floor and leaned harder on the model's self-assessment, which reads the actual text rather than a distance metric. The general lesson: calibrate against your own corpus, and when a signal disagrees with your judgement, check what the signal is actually measuring before trusting it.

Measuring it honestly

Accuracy on questions your content covers is the wrong metric — every tool scores well there, which is why every demo is impressive.

Build an evaluation set that includes the awkward cases: questions with no answer in the corpus, ambiguous questions, questions whose answer is outdated, and questions where two sources conflict. Then score three outcomes rather than two.

• —Correct — answered, and right
• —Declined — admitted it did not know, when it did not know
• —Fabricated — answered confidently without support in the sources
• —A high decline rate is a content problem you can fix. A non-zero fabrication rate is a trust problem you cannot.

Make refusal productive

An assistant that says "I don't know" and stops has converted a hallucination into a dead end. Better, but not good.

AskBrew treats a refusal as three actions: tell the customer plainly, offer a handover to a human if the workspace has that enabled, and write the question to a queue of unanswered questions in the dashboard. The customer is not stranded, and the team gets a ranked list of exactly what their content is missing.

That last part turns the failure mode into the improvement loop. Every refusal is a specific, customer-generated instruction about what to write next — which is a considerably better content roadmap than guessing.