Verke Editorial

Can AI hallucinate bad advice? Yes — here's how to spot and handle it

Verke Editorial · April 28, 2026

Can AI hallucinate bad advice? Yes. Large language models occasionally fabricate things that sound plausible — a study that doesn't exist, a citation that's wrong, advice that doesn't apply to your situation, a confident-sounding framework that's genuinely made up. Coaching contexts have somewhat lower stakes than medical or legal contexts, but bad advice still matters. The honest version of this article is that guardrails reduce hallucination but don't eliminate it, and the user's own calibration is part of how a well-functioning AI coaching tool stays useful.

The article walks through where hallucination shows up, how Verke is designed to catch the high-risk categories, and how to calibrate your trust as the user — which is roughly "treat AI suggestions like advice from a smart friend who's not omniscient." Verke's posture is to prefer "I'm not sure" over confident-but-wrong, and to keep coaching aimed at exploration rather than diagnosis. None of that makes hallucination impossible. It does make the failure modes recognizable and the recovery moves easy.

A language model predicts probable next text from patterns in its training data. Most of the time, the most-probable next text is also the correct next text — that's why these tools work as well as they do. Sometimes, though, the most-probable next text is wrong. The model produces a confident-sounding answer that has no basis in fact. The fluency is the point that confuses people: the wrong answer reads as smoothly as a right one, because the model's job is fluent text, not verified text.

This isn't lying — the model has no agenda, no goal, no attempt to deceive. It's the model not having a separate "truth" component that checks the output against reality before producing it. Newer techniques (retrieval, tool use, self-consistency checks, refusal training) reduce hallucination meaningfully, and the rate keeps dropping with each model generation. They don't eliminate it, though. Treating AI output as "mostly right but verify the high-stakes parts" is the right calibration today and probably for the next several years.

Made-up citations

The classic hallucination shape: "a 2019 Harvard study found that…" followed by a confident-sounding finding that, when you go to look it up, doesn't exist. The paper is invented; the authors are invented; the journal name might be real but the paper isn't. The fix is to verify any citation that matters to you via PubMed or Google Scholar before relying on it. If a URL is included, click through and check that the abstract actually says what was claimed — sometimes the URL is real but the summary attached to it is wrong.

Specific medical or legal advice

Doses, drug interactions, jurisdictional regulations, specific legal procedures — anything where the answer needs to be exactly right or it causes harm. Even when the model's answer happens to be correct, it's the wrong tool for these questions because there's no way for you to know whether it was right this time. Always verify with a licensed professional (doctor, pharmacist, lawyer, accountant) for anything actionable in those domains. Verke's coaches are designed to refuse these questions outright rather than improvise — see the next section.

Confident answers in narrow domains

Niche conditions with sparse training data, regional regulations most of the world doesn't care about, specific therapists by name, small professional communities. The model has just enough pattern in the training data to produce something fluent, but not enough to know whether it's right. The combination of fluency and narrowness is the main signal — when the topic is obscure but the answer is confident, that's when calibration should kick in.

Plausible but wrong frameworks

Invented "five-step methods" and "four pillars of…" that don't exist in the literature. The model has seen enough self-help-style structure to produce convincing- looking versions of it, even when the specific framework it's describing is made up. If a framework matters to the decision you're making, look up the author or the method name to confirm it's real before treating it as standard practice. Real frameworks have real Wikipedia pages, books, and citations; invented ones don't.

Domain guardrails

The coaches are designed to refuse the high-risk categories rather than improvise. Medical dosing, drug interactions, legal opinions, diagnostic claims, anything that crosses into licensed- professional territory — the response is to redirect rather than attempt. "That sounds like a question for a pharmacist" is a feature, not a limitation. The product would rather not answer than answer wrong.

Citation discipline

When a coach references a study or a method, the citation includes a real URL the user can verify (the StopOverthinking article on this site cites A-Tjak et al. 2015 with a PubMed link for exactly this reason — readers should be able to click through and check). If the coach can't cite something verifiably, the framing shifts to "there's evidence that" or "this is a common pattern in the field," not invented specifics. The bar is "a reader could verify this in 30 seconds."

Conservative defaults

When severity is suggested in the conversation, the default move is to surface clinical care rather than improvise help. Crisis- adjacent topics route to crisis resources. Diagnostic-adjacent topics route to a clinician. The product is designed to err on the side of "please bring this to a human" when the stakes are high — which is where hallucination would do the most damage if it slipped through anyway.

Calibration is shared work. The product holds up its end with guardrails and citation discipline; the user's end is a few simple habits that make hallucination much less costly when it does happen:

• Treat AI suggestions like advice from a smart friend who's not omniscient. Useful starting point, not the final word.
• Verify citations before sharing them or acting on them. PubMed and Google Scholar are 30-second checks.
• Ask "how confident are you in this?" — models can sometimes flag uncertainty when prompted, and the answer is informative.
• For anything medical, legal, or financial — verify with a licensed human. AI is the wrong tool for those domains as a primary source.
• When something doesn't fit your situation, push back. The response will recalibrate around what you've added — generic advice is often a sign the coach hasn't fully understood the specifics yet.

Self-help and AI coaching can do a lot, but they have limits. If you're experiencing severe depression that hasn't lifted, panic attacks that interrupt daily life, thoughts of self-harm, active trauma processing, or substance dependence — those are signals to work with a licensed clinician, not signals to push harder on a coaching tool. You can find low-cost options at opencounseling.com or international helplines via findahelpline.com. There's no prize for waiting longer than you need to.

Calibration — "is this thought (or this advice) actually accurate?" — is core CBT. Judith's approach treats beliefs as hypotheses to test rather than facts to swallow, which is exactly the posture that lets you use any source of input (including an AI coach) without over-trusting it. She's also good at the meta-version: noticing when you're leaning too hard on any single source — book, podcast, friend, app — and pulling you back toward your own judgment as the final filter. For more on the method, see Cognitive Behavioral Therapy.

Try a CBT exercise with Judith — no account needed