How AI customer support actually works

It is not the model answering from memory

The common misunderstanding is that these tools fine-tune a model on your business. Almost none do. Fine-tuning is expensive, slow to update, and bad at recall — a model trained on your returns policy will paraphrase it approximately, which is precisely what you do not want.

What actually happens is retrieval-augmented generation. Your content is indexed ahead of time. At question time the system finds the passages most relevant to the question, puts them in the prompt, and instructs the model to answer only from them. The model supplies fluency and the ability to parse an oddly-worded question; your content supplies the facts.

The practical consequence: updating the assistant means updating your content, not retraining anything. Add a note, re-index, and the next answer reflects it.

Before any question: ingestion

Documents, crawled pages, and typed notes are split into chunks of a few hundred tokens, each carrying metadata about where it came from — file name, page number, source URL, heading. Each chunk is sent to an embedding model, which returns a vector: a list of numbers positioning that text in a space where similar meanings sit close together.

Chunk boundaries matter more than people expect. Split too small and a chunk loses the context that makes it meaningful. Split too large and a chunk covers several topics, its vector lands in the average of all of them, and it matches nothing well. Documents with clear headings chunk cleanly; a PDF with none is the hardest case in practice.

The metadata is not decoration. It is what lets an answer cite the page it came from, which is the difference between a claim a customer can verify and one they simply have to trust.

At question time, in order

Five steps run between the customer pressing enter and seeing a reply. Most take a few hundred milliseconds; the model call dominates.

• —Classify — is this even a knowledge question? "Hi" and "thanks" should not run retrieval. AskBrew short-circuits small talk before the pipeline starts, because falling through to "I couldn't find information about that" in response to a greeting is both wrong and offers an unnecessary handover.
• —Embed the question — the same model that embedded the chunks, or the vectors are not comparable.
• —Search — find the nearest chunks by cosine distance, scoped to that customer's content only.
• —Assemble — put the retrieved passages in the prompt, clearly delimited, with instructions to answer only from them.
• —Generate and evaluate — the model writes an answer and reports how well the context covered the question. Both are recorded.

The gates that decide whether to answer at all

This is the part that separates a serious implementation from a demo, and it is where most of the engineering effort goes.

Two independent signals are available. Retrieval similarity — how close the best-matching chunk was — is cheap and comes before any model call. Model self-assessment — asked in the same structured call as the answer, so it costs nothing extra — is a judgement made by something that actually read the passages.

Neither is reliable alone. Similarity scores are noisy across different chunk sizes and document types; a genuinely answerable question can score low simply because the source PDF had no headings. A model asked to rate its own confidence is generous. Combining them is more robust than either, and AskBrew blends both into a single score before deciding whether to answer.

There is also a cost argument for the retrieval gate. If nothing relevant came back and no tool or visitor context could help, there is no reason to pay for a model call to produce a fallback message you already know you are going to send.

What happens when it cannot answer

A good assistant fails in a way that produces work you can act on. AskBrew records every answer with whether it was actually answered, its confidence, and the retrieval trace — which chunks were considered and what they scored. Unanswered questions go to a queue in the dashboard.

That queue is the highest-signal product feedback most support teams will ever get: a list of things customers wanted to know that your content did not cover. Write the answer, add it as a source, re-index, and the gap is closed.

The alternative failure mode — answering anyway from weak matches — produces a confident, plausible, wrong answer. The customer believes it. You find out when they complain, and the ticket you deflected comes back angrier than it left.