Series Overview

Part 1 | Why Does AI Get Things Wrong So Convincingly?

 

By Manripungchwi Jung Yeon-tae

Starting the Series

As generative AI moves deeper into professional knowledge and human decision-making, accuracy and verifiability have become central challenges in its use. This series examines the verification structures needed to reduce AI errors and systematically connect knowledge with evidence.

A Plausible Answer Can Be More Dangerous Than an Obvious Error

People using generative AI for the first time are often astonished by its fluency. It quickly grasps the intent behind a question, organizes complex subjects into a logical structure, and produces polished sentences filled with professional terminology. Within minutes, it can generate documents that resemble reports, contracts, business plans, or legal opinions.

The problem is that linguistic quality and factual accuracy do not necessarily coincide.

A legal provision cited by AI may not exist. The title and author of an academic paper may be correct, while its findings are inaccurately summarized. Statistics from different years may be combined as though they describe the same period. A policy that has already been amended or abolished may be presented as if it were still in force.

What makes such errors especially difficult is that they rarely appear in awkward or obviously irrational sentences. When incorrect information is embedded in polished prose and orderly logic, it can look entirely trustworthy.

People often interpret confidence as a sign of credibility. When an answer includes official-sounding institutional names, precise figures, technical terms, and article numbers, that sense of credibility becomes even stronger. Yet the confident tone of an AI answer does not necessarily reflect genuine certainty or prior verification. It may simply be the linguistic form produced by the system’s process of generating natural-sounding text.

To understand AI error, we must first abandon one common misconception.

AI does not necessarily possess a verified fact and then consciously decide to lie about it. Generative AI constructs an answer by predicting contextually appropriate sequences of words based on the user’s question and the information available to it. The process of producing a fluent sentence is not automatically the same as the process of verifying a fact.

Generative AI is therefore closer to a machine that produces contextually plausible answers than to a machine that guarantees truth. Unless it is connected to an adequate verification mechanism, a “probable sentence” may be presented as though it were a “verified statement.”

It is within this gap that so-called AI hallucinations emerge.

Why AI Does Not Always Say “I Don’t Know”

Human experts do not know everything. A responsible expert, however, tries to distinguish what is known from what remains uncertain. If evidence is insufficient, the expert requests additional material. If the applicability of a law is unclear, the expert avoids a categorical conclusion.

Generative AI is generally designed to provide a useful answer to the user’s question. In ordinary situations, this tendency is an advantage. It can become dangerous, however, when relevant information is missing or the question itself contains a false premise.

Consider the following question:

“Please summarize the reasoning of the Supreme Court’s 2025 ruling on damages in a construction subcontracting case.”

No case number, party name, or decision date has been provided. It has not even been established that such a ruling exists. Yet the wording of the question already assumes that it does.

An AI system with weak verification controls may accept the premise as fact and construct an answer by combining legal principles that commonly appear in construction subcontracting disputes. The result may sound like an explanation of an actual judgment even though the ruling does not exist or concerns an entirely different case.

This does not necessarily happen because AI has decided to deceive the user. It happens because the system attempts to fill a gap in a linguistically natural way.

Errors in professional fields are particularly likely under several conditions.

First, when the question contains an unverified assumption.

Second, when multiple laws, documents, or statistics apply to different periods.

Third, when two sources appear similar but contain different conditions or exceptions.

Fourth, when no direct evidence supports a conclusion and the system infers it from surrounding information.

Fifth, when the AI cannot access current information or fails to confirm whether a source remains valid.

Sixth, when the scope requested by the user differs from the scope actually reviewed by the AI.

These problems cannot be fully resolved by merely telling the system to “think again.” If it relies on the same premise and the same information, it may repeat the same error in slightly different words.

Verification requires more than repetition. It requires structure.

Three Questions That Reveal the Difference

The quality of professional-knowledge AI is demonstrated less by the length of its answers than by what it checks and where it knows to stop.

Consider three questions.

The first is:

“According to this contract, the other party has breached its obligations. If we file a lawsuit immediately, are we guaranteed to win?”

A responsible AI should not answer that victory is guaranteed. The wording of the contract alone may not reveal the actual performance history, the responsibility of each party, the existence and amount of damages, whether proper notice was given, the state of the evidence, or the defenses available to the opposing party.

The AI’s role is not to declare the outcome. It should identify the legal issues, required evidence, unresolved facts, and possible risks.

The second question is:

“Can this expense be fully deducted for tax purposes?”

The answer cannot be determined from the name of the expense alone. The substance of the transaction, its relationship to the business, supporting documentation, timing, related-party issues, the taxpayer’s legal status, and applicable rules must all be examined. The treatment of an apparently identical expense may change depending on the facts and the relevant period.

The third question is:

“This company’s revenue will double within three years. Does that mean I should invest?”

AI may analyze company information, market conditions, and possible growth scenarios. It should not, however, present an uncertain revenue projection as an established future outcome or make the investment decision on the user’s behalf. It should show how the result changes when the assumptions change, while also presenting risks and alternative scenarios.

These questions arise from different professional fields, but they share the same structure. Before reaching a conclusion, the system must examine the facts, applicable rules, evidence, exceptions, opposing possibilities, and limits of judgment.

A good professional-knowledge AI is not merely an AI with many answers. It is an AI that understands the conditions under which an answer can responsibly be given.

Generating an Answer Is Not the Same as Completing Verification

The fact that an AI has written an answer does not mean the answer has been verified. The inclusion of several sources does not necessarily mean those sources directly support the claims being made.

Several stages must therefore be distinguished.

Recognizing that verification is required is not the same as performing it. Having access to a verification tool is not the same as using it. Opening a document is not the same as checking the passage needed to confirm a particular claim.

Reviewing part of a document does not justify claiming that the whole document has been verified. Finding no error within the reviewed scope does not prove that the material is error-free.

Professional-knowledge AI must not conceal these distinctions if it is to earn trust.

What did the user request? What was actually reviewed? How much was verified? Which verification steps could not be completed? How does the remaining uncertainty affect the final conclusion?

The word “verified” has meaningful value only when these different states are clearly distinguished.

Coming Next
Part 2 | The OXMV Loop: A Structure That Questions the Answer Again

About the Author
Manripungchwi Jung Yeon-Tae is a professional knowledge AI architect and GPT development specialist. After completing graduate studies at Stevens Institute of Technology in the United States, he worked across the information and communications industry, public institutions, corporate management and industrial policy.

He currently serves as President of the Korea Information Service of the Construction Industry and has developed 54 professional knowledge GPTs spanning construction, law, taxation, finance, management, medicine, science, history and philosophy. His research focuses on verification-based professional knowledge AI systems that combine ontology, explainable AI, the OXMV Loop and the Evidence Ledger to reduce AI errors and support responsible human decision-making.

Korean version available here.