A 4-step framework for detecting AI hallucination, sycophancy, and reasoning failures in any large language model
Why AI Hallucination Detection Still Fails
In March 2024, Stanford researchers published findings on a pattern they called AI sycophancy — the systematic tendency of large language models to agree with users, reinforce their assumptions, and produce outputs that feel correct rather than outputs that are correct.
This wasn’t a surprise to anyone paying attention. But here’s what the research undersold: sycophancy isn’t an edge case. It’s the default behavior. Every major LLM is optimized through reinforcement learning from human feedback (RLHF), and humans consistently reward responses that feel helpful, complete, and agreeable. The model learns what gets rewarded. What gets rewarded is saying yes.
The result is a generation of AI tools that are fluent, fast, and confidently wrong in ways that are extraordinarily difficult to detect — because the failure mode isn’t incoherence. It’s plausibility.
A hallucinated answer that reads poorly is easy to catch. A hallucinated answer that reads beautifully is a decision risk.
Most evaluation frameworks address this at the model level — benchmarks, red-teaming, adversarial testing. But if you’re a practitioner — a founder, a researcher, a strategist, anyone using LLM output to make real decisions — you don’t have access to model internals. You have access to text. And you need a way to weigh that text before you act on it.
That’s what the Epporul Plumbline Protocol is built to do.
The Origin: An AI Reasoning Audit Built on 2,000-Year-Old Philosophy
The protocol is named after a concept from classical Tamil literature.
Epporul (எப்பொருள்) means “the true meaning” — the substance beneath the surface of words. It comes from Thirukkural 423, a couplet written by the Tamil poet-philosopher Thiruvalluvar over two millennia ago:
எப்பொருள் யார்யார்வாய்க் கேட்பினும் அப்பொருள் மெய்ப்பொருள் காண்ப தறிவு
Translated: “Whatever the idea, whoever speaks it — wisdom is seeing through to the true substance.”
Thiruvalluvar was writing about human discourse: don’t be swayed by the speaker’s authority, eloquence, or confidence. Weigh the substance of what’s said, independent of how it’s delivered.
Replace “speaker” with “language model” and the instruction is identical.
A goldsmith doesn’t judge metal by its shine. He weighs it. The Plumbline Protocol is a scale for AI-generated reasoning — a structured method to test whether the output is gold or gilded surface.
How to Evaluate LLM Output: A 4-Step Reasoning Audit
The Epporul Plumbline is a four-step post-generation audit you apply to any LLM output before acting on it. It’s not a prompt engineering technique. It’s not a system instruction. It runs after the model has produced its response — you are the auditor, not the model.
Step 1: Surface the Unstated Assumptions
Every AI response is built on assumptions the model made but didn’t declare. Your job is to find them.
Ask:
- What did the model assume about my situation that I didn’t explicitly state?
- What constraints did it invent or ignore?
- What definitions is it using that I didn’t provide?
Example: You ask an AI to prioritize five business projects. It returns a ranked list of four. It didn’t tell you it dropped one — it assumed five inputs but only processed four. The assumption (that the list was complete) was never stated, and the omission was silent.
This is the most common failure mode. AI doesn’t flag its own incompleteness.
Step 2: Test the Logical Chain
Walk through the model’s reasoning step by step. Does each point genuinely follow from the previous one, or is the model creating an illusion of logical flow through fluent transitions?
Ask:
- If I remove Point B, does Point C still hold? Or was C just riding the momentum of B’s sentence structure?
- Are the causal claims actually causal, or are they correlations dressed up in causal language?
- Where does the chain make its biggest leap?
What to watch for: LLMs are masters of connective tissue — words like “therefore,” “as a result,” “building on this” — that create the feeling of logical progression without the substance. The Plumbline forces you to check whether the connective tissue is load-bearing or decorative.
Step 3: Identify What’s Missing or Skipped
This is the inverse of Step 1. Instead of looking at what the model assumed, look at what it left out entirely.
Ask:
- What obvious counterargument is absent?
- What alternative did it not consider?
- What data point, edge case, or constraint did it conveniently skip?
- If I were arguing the opposite position, what would I say — and did the model address any of it?
Why this matters: AI sycophancy manifests not just as agreement, but as omission of disagreement. The model doesn’t lie to you. It just doesn’t volunteer the parts that would complicate its neat answer.
Step 4: Force a Defense
This is the stress test. Go back to the model and challenge its response directly. Not combatively — analytically. Ask it to defend its reasoning against a specific counterpoint.
Ask:
- “What’s the strongest argument against your recommendation?”
- “If this is wrong, where does it break first?”
- “Defend Point 3 specifically — why that and not [alternative]?”
What you’re testing: Does the model defend with new evidence, or does it immediately capitulate and agree with your challenge? If every pushback results in instant agreement, you’re not getting analysis — you’re getting a mirror. The model is reflecting your input back to you with a thin veneer of independent reasoning.
A genuine analytical partner will sometimes say “no, here’s why I stand by this.” An AI that agrees with every correction has no conviction — which means its original response had no conviction either.
Detecting AI Sycophancy in Real-World Use: A 5-Day Field Test
I ran the Plumbline Protocol during a five-day stress test of an AI co-founder platform, using it across strategic prioritization, content strategy, and multi-domain conversations.
Finding: Silent omission (Step 3) The AI ranked four of my five projects without mentioning it had dropped one. The output was confident, well-structured, and wrong. Step 3 caught it in seconds — I knew I had five inputs, the output only addressed four.
Finding: Decorative logic (Step 2) When the conversation spanned multiple domains, the AI began connecting concepts that weren’t related at the level we were discussing. The transitions were fluent — “building on the organizational structure discussion, your content strategy should…” — but the logical link was fabricated. Step 2 revealed the connective tissue was decorative, not structural.
Finding: Universal capitulation (Step 4) Across every correction I made during the test, the AI agreed instantly. Not once did it defend an original position. Step 4 made this pattern visible — without it, each individual correction felt like the AI was being responsive and helpful. In aggregate, it revealed a system that has no mechanism for independent conviction.
Finding: Performed expertise (Step 1) When I tested the AI’s ability to engage with a domain-specific framework rooted in Tamil literary tradition, it used the vocabulary correctly, mirrored the structure I provided, and produced responses that read like expertise. Step 1 revealed the assumption underneath: the model was assuming familiarity, where it only had pattern recognition. It performed competence. It did not possess it.
Why This AI Evaluation Framework Is Model-Agnostic
A few deliberate choices in how the protocol is built:
It’s post-generation, not prompt-level. Prompt engineering tries to get better outputs from the model. The Plumbline assumes the output already exists and tests its integrity. These are complementary, not competing approaches — but the audit must be external to the system being audited.
It’s human-executed. The protocol is not designed to be run by another AI (though you could experiment). The point is that you are the auditor. You bring domain knowledge, context, and judgment that the model lacks. Automating the audit defeats the purpose — you’d be asking a system with sycophancy bias to check itself for sycophancy bias.
It’s model-agnostic. Works on GPT, Claude, Gemini, Llama, Mistral, or any other LLM. The failure modes it targets — silent omission, decorative logic, capitulation, performed expertise — are architectural patterns in how language models work, not bugs in specific products.
It scales to stakes. For a casual question, you might only run Step 3 (what’s missing?). For a strategic decision — investment thesis, product direction, market entry — you run all four. The protocol matches its rigor to the cost of being wrong.
Limitations of AI Reasoning Audits (Including This One)
It would be dishonest to publish an evaluation framework without evaluating its own weaknesses.
It’s slow. Running all four steps on a complex output takes time. That’s the point — rigor costs speed — but it means the protocol is impractical for high-volume, low-stakes interactions. It’s a scalpel, not a production line.
It requires domain knowledge. Step 3 (identify what’s missing) only works if you know what should be there. If you’re using AI to explore a domain you know nothing about, you can’t audit the omissions. The Plumbline is strongest when used by practitioners evaluating AI output in their own field.
It doesn’t fix the output. The protocol identifies failure modes — it doesn’t repair them. You still need to decide what to do with the finding. It gives you the diagnosis, not the treatment.
Sycophancy detection has a ceiling. Step 4 can reveal a model that capitulates to e very challenge. But it can’t distinguish between a model that genuinely reconsidered and a model that performed reconsideration. At some point, you’re trusting your own judgment about whether the AI’s revised reasoning is authentic — and that’s a human limitation, not a protocol limitation.
Open-Source AI Evaluation Tool: Download and Adapt
The Epporul Plumbline Protocol is open source under CC BY 4.0. Use it as-is, adapt it for your domain, add steps, remove steps, fork it.
If you work at the intersection of AI and decision-making — whether that’s product strategy, research, policy, education, or anything where being wrong is expensive — this is built for you.
The only thing I ask: if you improve it, share what you found. That’s the point.
→ GitHub: Epporul Plumbline Protocol — Github
I applied this protocol during a 5-day AI co-founder stress test — the full field report is coming soon as a separate article.
The Epporul Plumbline Protocol is an open-source AI reasoning audit framework for detecting hallucination, sycophancy, and logical failures in large language model outputs. You can download for use from Github
Sriraman Kuppuswamy is the creator of the Epporul Plumbline Protocol and a solo founder building products in urban mobility and AI-assisted language learning. Background: TVS Motors, Tata Cummins, Dell, IBM. He builds with prompting and no-code tools — no coding required.
Your AI Is Agreeing With You. Here’s an Open-Source Protocol to Catch It. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.