The Hallucination Problem in Professional Tools

Most discussion of hallucination treats it as a property of models that will eventually be fixed by better models. In professional software that framing is unhelpful, because you have to ship now and the consequences are real.

The more useful framing: hallucination is a system design problem, and a well-designed system can make it rare and detectable.

Why it is worse in professional contexts

A hallucinated restaurant recommendation is an annoyance. A hallucinated code citation is acted on by someone with a licence and liability.

Two properties make professional domains especially dangerous:

• The user often cannot immediately verify. That is why they are using the tool.
• The output looks authoritative. Correct formatting and a plausible reference number are trivially generated and carry enormous implied credibility.

The failure is not that the model was wrong. It is that it was wrong in a format that made verification feel unnecessary.

Four architectural defences

1. Never answer from parametric memory. If the corpus did not supply it, it is not an answer. This alone eliminates most of the problem in bounded domains.

2. Make every claim traceable. Not "cite your sources" as a prompt instruction — a pipeline where the reference is attached to the retrieved content and carried through. A claim without provenance should not be renderable.

3. Validate numbers against the source. Anything that should come from a table gets checked against that table programmatically. Numbers are where hallucination does the most damage and they are the easiest thing to verify automatically.

4. Make refusal cheap and normal. If the system is penalised for saying "I do not know," it will guess. Design and tune so that declining is a good outcome.

Detection, not just prevention

Prevention is incomplete. You also need to catch what gets through:

• Adversarial evaluation — deliberately ask unanswerable questions and confirm the system declines
• Expert spot-checks — sample real answers and have a domain expert verify the citation supports the claim
• User reporting — a low-friction way to flag a wrong answer, treated as a high-severity signal

We treat a single confirmed fabricated citation as a serious defect, not a quality metric that trends downward over time.

The honest limit

None of this achieves zero. A grounded, validated, refusal-tuned system can still produce a wrong answer, and any product that claims otherwise is overselling.

Which is why the last defence is disclosure. The user should understand what the tool is: extremely well-read, never on site, and not the final authority. That framing is not a legal hedge, it is accurate, and professionals respect it.

The uncomfortable conclusion

If your domain cannot tolerate an occasional wrong answer even with all these defences in place, the correct decision may be not to ship an AI product for that use case.

We have made that call internally more than once. It is not a failure of ambition. It is the difference between building tools professionals rely on and building tools that eventually hurt someone.