A language model does not consult a database of facts: it produces the most likely text. Often, the most likely text is correct; sometimes, when information is missing, it fills the gap with something plausible. This is called a hallucination.
What a hallucination looks like
A customer asks an installer’s assistant whether annual heat pump servicing is included in the contract. None of the documents mention it, but the assistant answers “Yes, it is included” because that seems like a likely response. The tone is confident, the sentence is correct, and the information is false.
What it can cost a business
A Canadian case illustrates the risk: Air Canada. Its website chatbot told a customer they could claim a reduced fare retrospectively following a death in the family, which the airline’s policy did not allow. On 14 February 2024, the British Columbia Civil Resolution Tribunal held the airline responsible for this information and ordered it to refund the difference (Moffatt v. Air Canada decision). In this case, the tribunal considered the airline responsible for what its chatbot said; another jurisdiction might reason differently, but the signal is clear for any business.
Safeguards that reduce the risk
- Answer from selected documents. RAG makes the model answer from retrieved passages rather than its general memory
- Say when it does not know. A clear instruction prompts it to state that the answer is missing from its documents, although this is not an absolute guarantee
- Show the source. Source citation makes it possible to check where each answer comes from
- Hand over on matters that commit the business. Firm prices, safety, disputes: human handover takes over
- Review. Conversations and unanswered questions reveal where a document is ambiguous
What cannot be promised
No setting reduces the risk to zero: a model can still misread a passage or mix up two cases. The realistic aim is to make errors rare, visible and correctable, and to leave high-stakes topics to a person.
KernChat assistants combine these safeguards, and every unanswered question becomes a knowledge suggestion to approve (see the features).