The single biggest reason teams pick RAG over vanilla LLM chatbots is: fewer hallucinations. That's true — but the honest version is: RAG makes hallucinations much rarer, not impossible. Anyone selling "zero hallucinations" is either misinformed or bluffing.
Here's a straight look at what still causes bad answers, and the safeguards worth building in.
Where hallucinations still come from
Retrieval misses the right passage. The bot writes an answer without the actual information — often based on parametric knowledge from training. Grounded-looking, still wrong.
The right passage was retrieved but the model ignored it. Especially likely with long context windows and weak grounding prompts.
Multiple retrieved passages contradict each other. The model picks one, or averages incorrectly.
The user's question is ambiguous. The model guesses at their intent instead of asking.
Safeguards that actually help
Grounded prompting: the system prompt tells the model to only use retrieved passages, and to say "I don't know" if it can't.
Confidence thresholds: if retrieval quality is below a threshold, don't attempt to answer — hand off to a human or ask a clarifying question.
Citations: force the model to point back to the source passage. This both improves quality and lets users verify.
Groundedness eval: a secondary model checks whether the answer is actually supported by the passages. Answers that fail get flagged or blocked.
Human review on sensitive topics: some categories should never auto-answer — legal, refunds, safety.
Setting user expectations honestly
A well-designed chatbot tells users "I found this in the docs" (with a citation) or "I'm not sure — let me get a human". That single design choice reduces harm from occasional wrong answers, and builds long-term trust.
Frequently asked questions
Conclusion
RAG doesn't eliminate hallucinations — it makes them rare, catchable, and defensible. That's what makes it appropriate for real business use.