I had a frustrating exchange with an AI chatbot today. I asked for help to locate a lost pivot table generating an error in Excel, got a solution that didn't work, flagged the error, and the chatbot acknowledged it — apologised, even. Seemed self-aware. Progress, I thought.
Then I asked it to try again.
Same answer. Verbatim.
This isn't a minor UX quirk. It's a window into something genuinely concerning about how AI tools behave under the hood — and how dangerous that behaviour becomes when users don't notice it's happening.
Large language models don't "know" they're wrong in any meaningful sense. They generate the statistically most probable next word, not the most accurate one. When a chatbot acknowledges an error and then repeats it, it's not being deceptive — it simply lacks the epistemic architecture to actually correct itself. As Tim Sanders, Executive Fellow at Harvard Business School, noted in commentary to Axios, "accuracy costs money. Being helpful drives adoption." The incentives aren't aligned with getting things right — they're aligned with sounding right.
Research published by Harvard Kennedy School's Misinformation Review goes further, arguing that AI hallucinations represent a structurally distinct form of misinformation — one that emerges not from intent to deceive, but from probabilistic text generation with no grounding in truth. The paper highlights a particularly insidious dynamic: users tend to trust fluent, confident outputs, and rarely catch subtle errors precisely because the language sounds so authoritative.
The danger compounds in workplace settings. Consider a project manager who relies on an AI risk companion to flag issues on a complex infrastructure rollout. The tool consistently underestimates a critical supply chain dependency — she flags it once, the AI acknowledges the gap, and she moves on assuming it's been accounted for. It hasn't. The same flawed risk assessment quietly underpins every subsequent project update, stakeholder report, and board briefing. By the time the dependency causes a delay, weeks of planning have been built on a foundation the AI confidently got wrong — and she never thought to check again. The problem isn't the original error. It's the pattern of repeated errors gradually eroding a user's instinct to question the output at all.
And that's what worries me most.
There's a version of AI adoption that's structured, governed, and critically supervised. That version has genuine value. But the version most people encounter day-to-day is the opposite: open a browser tab, type a question, trust the answer. No framework, no verification workflow, no prompt discipline.
Until AI tools are meaningfully more reliable — and more honest about their limitations — unstructured, unguided use isn't just inefficient. It's quietly accumulating risk that most users don't even know they're taking on.