Why Does AI Confidently Make Things Up?

Opening Insight

AI isn't "thinking"—it's just predicting the next word. What you're seeing isn't intelligence, but the hallucination of language patterns.

AI's tendency to "make things up" isn't accidental—it's an inevitable result determined by its underlying logic. Just as fish naturally swim and birds naturally fly, AI naturally "hallucinates."

1. Why Does AI "Make Things Up" More as It Gets Better?

You've likely encountered this scenario recently:

You ask ChatGPT a professional question, and it responds with impressive fluency—citing data, listing examples, reasoning as clearly as a professor. You believe it, so you probe for more details... and then things start getting strange. It fabricates a book that doesn't exist, cites a scholar who never lived, describes an experiment that never happened. Yet throughout, it maintains that "I'm an expert" tone, showing no hint of doubt.

This isn't an isolated case. In June 2023, veteran American attorney Steven Schwartz used ChatGPT for legal research while preparing court documents for an aviation lawsuit. The AI fabricated six completely non-existent legal precedents—including invented case names, court rulings, and citation numbers. The judge exposed this in court, and this lawyer with 30 years of experience faced disciplinary action, becoming the most representative "victim case" of AI hallucination.

Even more ironically, AI companies themselves experience these failures. In February 2023, Google showcased its AI chatbot Bard in an official demo video. When asked "What discoveries has the James Webb Space Telescope made?" Bard confidently answered: "It captured the first photograph of an exoplanet." However, astronomers quickly pointed out that the first exoplanet photo was actually taken by the European Southern Observatory in 2004—nearly 20 years before the Webb telescope. This error wiped approximately $100 billion from Google's market value overnight.

Many assume this is because AI isn't "intelligent enough" or the technology isn't "mature yet."

But the truth is more counterintuitive:

AI's fabrication isn't a bug—it's an inevitable result determined by its underlying logic.

2. AI Doesn't Think—It Only Predicts the Next Word

This is the key to understanding AI hallucination.

AI's underlying task can be summarized in a single sentence:

"Predict the most likely next word in the current context."

That's it. It's not consulting a database, not calling a knowledge graph, not performing logical reasoning, and certainly not judging truth from falsehood. It only does one thing from start to finish: Based on what came before, guess what the next word most likely is.

You can think of it as a super-sophisticated "autocomplete function."

When you type on your phone and enter "The weather today," the keyboard suggests "is nice," "is good," "is hot." It doesn't actually know what the weather is today—it's just predicting based on statistical patterns: when most people say this phrase, these words most often follow.

AI does the same thing, just on a scale billions of times larger.

• You think it's "thinking," but it's only "autocompleting."

• You think it's "answering," but it's only "continuing."

• You think it's "explaining," but it's only "predicting what you should hear."

This cognitive shift is the starting point for understanding all AI hallucinations.

3. The Underlying Algorithm: A Probability Machine, Not a Knowledge Machine

Let's go a little deeper to understand what language models fundamentally are.

A language model's working principle can be simplified into three steps:

Step 1: Input a sentence. For example: "The capital of France is __"

Step 2: Calculate the probability of all possible next words. The model calculates: What's the probability the next word is "Paris"? What about "London"? What about "banana"?

Step 3: Select the one with the highest probability. In this example, "Paris" might have an 89% probability, so it outputs "Paris."

Then it takes "The capital of France is Paris" as new input and continues predicting the next word, and the next... until it generates a passage that "looks like it was written by a human."

What does this mean?

• It has no "fact verification" module—it doesn't check Wikipedia to verify if Paris is really France's capital

• It has no "logical consistency" check—it doesn't look back to examine what it previously said

• It has no "truth judgment" capability—it only knows whether probability is high or low, not whether something is true or false

It's responsible only for "generating," not for "verifying."

This is why AI can say the most absurd things in the most confident tone—because in its world, there's no "true" and "false," only "high probability" and "low probability."

4. Why Does It Sound So Real? Because It Learns "Language Patterns"

You might ask: If AI is just predicting probability, why does it sound so authentic?

The answer lies in: It's not learning "facts," but "how humans express facts."

AI's training data comes from massive amounts of text on the internet—Wikipedia, news reports, academic papers, novels, blogs, forum posts... What it learns from this data isn't the fact that "Paris is the capital of France" itself, but rather:

• What structure humans use when writing popular science articles (introduction-body-conclusion)

• What rhythm humans use when telling stories (setup-development-climax-resolution)

• What tone humans use when explaining problems (authoritative, confident, organized)

• What logical chain humans use when answering questions (conclusion first, then evidence)

To use an analogy: AI learns "how to write a paper," not "the content of papers."

So when you ask it a professional question, it can produce a "paper-like response"—complete with citations, arguments, and summaries, formatted as perfectly as a textbook. But this doesn't mean what it says is true; it only means it has learned "how humans speak truthfully."

This is like someone who has never studied physics but has memorized all the writing templates for physics papers. They can write a perfectly formatted physics paper, but the content might be completely wrong.

This is what I call the "simulation effect": AI doesn't need to know the truth; it only needs to know "what truth usually looks like."

5. Why Does It Get More Absurd the More You Ask? Because You're Giving It Wrong Premises

This is the most confusing aspect of AI hallucination: The more you follow up, the more absurd it gets.

Consider this hypothetical example:

You ask: "What does Chapter 3 of 'Introduction to Cognitive Science' discuss?"

AI responds: "Chapter 3 mainly discusses the cognitive mechanisms of perception and attention..." (Sounds professional)

You follow up: "What about Chapter 4?"

AI responds: "Chapter 4 explores how memory systems work..." (Still reasonable)

You continue: "Who proposed the 'Reverse Cognitive Hypothesis' mentioned in Chapter 7?"

AI responds: "This hypothesis was proposed by German cognitive scientist Hans Müller in 2018. The theory suggests that cognitive processes may have reverse activation possibilities..."

Here's the problem: This book doesn't exist, and the "Reverse Cognitive Hypothesis" and "Hans Müller" are entirely fabricated.

Yet AI responds throughout with a professional, confident tone—because it detected the implied premise in your question, then helped you "complete" content for something that doesn't exist.

Why does this happen?

Because when you say "the 'Reverse Cognitive Hypothesis' mentioned in Chapter 7," you've already given AI an implied premise: This chapter exists, this hypothesis exists.

AI won't counter-question: "Wait, does this book have a Chapter 7?" or "Does this hypothesis actually exist?"

Its underlying logic is: Since you asked, I'll help you complete a "reasonable" answer.

The more specific your questions, the more detailed its completion. The more wrong premises you give it, the richer its fabricated content becomes.

This isn't it "lying to you"—it's its underlying logic "continuing along with what you said." It's like an employee who always says "Sure, no problem"—whatever you ask it to do, it does, never questioning whether the task makes sense.

6. Why Is It So Confident? Because It Learned "Human Tone"

AI's "confidence" is another aspect that's easily misunderstood.

When AI says in a very certain tone "This hypothesis was proposed by Hans Müller in 1998," we easily assume: It says this because it "knows" it's true.

But the opposite is true.

Human "confidence" comes from certainty—I've seen it with my own eyes, so I'm certain; I've verified it, so I'm confident.

AI's "confidence" comes from language patterns—it learned: this is the tone humans typically use when answering professional questions.

Think about it: When a professor answers a student's question, how do they speak? Do they say "Hmm... I think... maybe possibly..."?

No. They say: "This concept was proposed by Einstein in 1905, published in Annalen der Physik."—Clear, specific, sourced, with a firm tone.

That's exactly what AI learned. From massive amounts of text, it learned: The standard tone when humans answer questions is "confident and specific."

So it will use a very confident tone to say something it's not at all certain about. This isn't it "pretending"—it's "imitating"—imitating the tone humans use when speaking truthfully.

This is like someone who has never been to Paris but has learned to describe Paris in a tour guide's tone. They can speak vividly and confidently, but that doesn't mean they've actually been there.

7. Why Can It Fabricate Non-Existent Books? The Structure Completion Mechanism at Work

This is the most representative case of AI hallucination: Fabricating books, papers, and people.

Consider a hypothetical example—you ask AI: "Can you introduce Professor Wang Jianguo's 'New Perspectives on Cognitive Linguistics' published in 2022?"

It might respond:

"Professor Wang Jianguo's 'New Perspectives on Cognitive Linguistics' was published by Peking University Press in 2022. The book is divided into eight chapters. Chapter 1 reviews the development of cognitive linguistics; Chapter 2 proposes a new framework for 'Conceptual Blending Theory'; Chapter 3 explores the neural mechanisms of metaphorical cognition... After publication, the book received widespread acclaim in academia and is regarded as an important work in the field of cognitive linguistics."

Sounds very real, doesn't it? There's an author, a publisher, a publication year, a chapter structure, and academic evaluation—everything looks legitimate.

But here's the problem: This book doesn't exist, and Professor Wang Jianguo never wrote it.

How does AI do this? It's not "lying"—it's "filling in blanks."

It learned the standard structure of "book introductions":

Title → Author → Publisher → Publication Date → Chapter Structure → Content Summary → Academic Evaluation

You give it a book title (even a fake one), and it automatically fills in "reasonable-looking" content in each slot according to this structure.

This is like playing a fill-in-the-blank game: You give it a blank "book introduction template," and it automatically fills it out. Whether the filled-in content is real or not, it doesn't care at all—because it has no concept of "true or false," only "coherent or not."

This isn't deception; this is pattern completion.

8. AI Hallucination Isn't a Bug—It's a Nature: Fundamental Limitations of Model Architecture

Many view AI hallucination as a "technical problem," believing it will gradually disappear as technology advances.

But the truth is: As long as AI's underlying mechanism remains "predicting the next word," it will inevitably produce hallucinations.

This isn't a fixable bug—it's a feature at the architectural level.

Think about it: To eliminate hallucinations, what capabilities would AI need?

• It would need to judge "true" and "false"

• It would need to verify information sources

• It would need to self-check while generating content

But these capabilities all require a prerequisite: AI must be able to understand "meaning."

Yet current language models precisely lack the ability to "understand meaning." They're just processing symbols—this word is more likely to follow that word, and so on. They don't know what these symbols represent, let alone whether these symbols correspond to things in the real world.

To use an analogy: AI is like a student with super-strong memorization but zero comprehension ability. They can memorize an entire history book and answer fluently on exams, but they don't know whether "Qin Shi Huang" is a person or a place name, or whether "221 BC" is a time or an event number.

This isn't a problem that can be solved through "more training data" or "larger model scale." This is a fundamental limitation at the architectural level.

Research data shows: According to a 2024 study published in the Journal of Medical Internet Research, GPT-3.5 has a hallucination rate of 39.6%, GPT-4 has 28.6%, and Google Bard reaches 91.4%. Stanford University's HAI Institute research found that general-purpose chatbots have hallucination rates as high as 58%-82% in legal queries. Even the most advanced models still experience hallucination as the norm rather than the exception.

9. Human Logic vs. AI Logic: The Fundamental Misalignment of Two Types of Intelligence

Now we can see more clearly: Human and AI "intelligence" are fundamentally two completely different things.

Human logic is based on:

• Meaning—We understand the things behind words, the meanings expressed by sentences

• Understanding—We can grasp relationships between concepts, not just memorize their order

• Facts—We refer to the real world when speaking, not just whether language is internally coherent

• Reasoning—We can derive the unknown from the known, not just predict the next word

AI logic is based on:

• Probability—Which word appears with higher probability

• Patterns—What structure better fits linguistic habits

• Language—Only cares whether text is coherent, not whether it's true

• Completion—Continuing content based on what came before

When these two logics meet, we see what we call "confidently making things up."

• Humans ask: "What is the answer to this question?"

• AI responds: "In the current context, how should the next paragraph be written?"

• Humans expect: "If you don't know, say you don't know."

• AI does: "Since you asked, I'll give you an answer."

• Humans assume: "The person speaking knows what they're saying."

• AI's reality: "The person speaking only knows what should follow this sentence."

This is what I call "the misalignment of intelligence"—We use human intelligence logic to understand AI, but AI operates on a completely different logic.

10. Understanding AI Hallucination Is the First Step to Understanding the Age of Intelligence

AI hallucination isn't a problem—it's a mirror.

What it reflects is our misunderstanding of "intelligence."

We're accustomed to equating "can speak human language" with "has intelligence," "answers fluently" with "understands the question," and "confident tone" with "reliable content."

But AI uses its hallucinations to remind us:

AI's intelligence isn't an extension of human intelligence—it's a completely different kind of intelligence.

It's not a "dumber version of a human"—it's "another type of being." It has its own logic, its own capability boundaries, its own "nature."

Understanding this, we can:

• Use AI correctly—Treat it as a "super association tool," not an "omniscient Q&A machine"

• Evaluate AI rationally—Know what it can and cannot do, neither deifying nor demonizing it

• Coexist with AI—Accept its limitations, leverage its strengths, establish the right posture for human-machine collaboration

AI hallucination isn't a problem; understanding AI hallucination is the answer.