Hello everyone, I'm Tak@, a system integrator. I usually help bring your ideas to life, but recently, I heard a rather frightening story about the depth of AI's "understanding."

What if the AI you rely on is merely "faking" its understanding of a certain concept, when in reality it comprehends nothing at all? And what if this "faking" is so sophisticated that humans can't detect it?

This isn't just a technical discussion. It has the potential to introduce immeasurable risks to your business, your projects, and your future.

Is AI's "Understanding" Truly Understanding?

We're all amazed by the advancements of large language models (LLMs) like ChatGPT, aren't we? They can engage in natural conversations like humans and accurately answer complex questions. These AIs have achieved excellent results in various benchmark tests, such as mock university entrance exams and specialized medical exams, leading to the assessment that they "understand concepts."

Human "Understanding" vs. Benchmark Limitations

However, let's pause and consider this. When we evaluate humans, do we immediately conclude that someone "understands everything" just because they perform well on a test?

For example, someone might perfectly read and explain a project's requirements document, but actually building a system that meets those requirements is an entirely different matter.

For humans, if they correctly understand a concept, we can generally assume they grasp the whole concept if they can correctly answer a few "keystone" questions related to it.

This is because the ways in which humans misunderstand concepts are limited and predictable.

For instance, someone who mistakenly believes the "5-7-5" rule of a tanka poem is "5-8-5" will create all their tanka according to that incorrect rule. Test questions are designed to detect these patterns of human misunderstanding.

The Frightening Reality of "Potemkin Understanding"

However, an LLM's "understanding" might be fundamentally different from a human's. Even if an LLM correctly answers keystone questions, it might not be because it understands the concept in the same way a human does.

Researchers have dubbed this phenomenon "Potemkin Understanding".

This term is an analogy to "Potemkin villages," which appear impressive on the surface but lack any real substance.

AI That Can Explain but Can't Apply

Specifically, what is Potemkin Understanding? It refers to a state where an LLM can accurately "define" a concept but fails to "apply" that concept in actual tasks.

In a study on "Potemkin Understanding," LLMs were first asked to define 32 different concepts (literary devices, game theory, psychological biases). Many models accurately defined these concepts with a high success rate of 94.2%.

That's a fantastic number, isn't it?

However, the situation changed dramatically in subsequent tasks that tested the "application of concepts." There were three types of application tasks:

• Classification: Determining whether a given example is a correct instance of the concept.
• Constrained Generation: Generating an instance of the concept according to specific constraints (e.g., character count, use of specific words, theme).
• Editing: Modifying a given example to transform it into a correct/incorrect instance of the concept.

The LLMs' performance on these application tasks, in other words, their "Potemkin rate," was found to be very high (indicating a lack of understanding). For example, even when models could correctly define a concept, they were incorrect in 55% of classification tasks, and 40% of both constrained generation and editing tasks.

This is like a wordsmith who acts as if they perfectly know the meaning of words, but has no idea how to use those words in real-world contexts.

I previously warned about the AI Programmer I created as a hobby, stating: "Code snippets and test cases generated by AI are not always accurate. Before integrating them into actual projects, users are strongly advised to carefully review the content and conduct appropriate tests themselves."

I feel that "Potemkin Understanding" is precisely the background to this warning. Even if the AI knows the definitions, fully "understanding" complex combinations of actual system requirements and subtle nuances of specifications to translate them into code is still extremely difficult.

"Strange Mistakes" That Humans Wouldn't Make

Even more surprising is that these LLM failures include "strange mistakes" that humans would almost certainly not make. For example, an LLM that can accurately explain the definition of a haiku (5-7-5 syllables) might make a mistake in the syllable count when actually generating one, or an LLM that can explain the triangle inequality theorem (the sum of two sides is greater than the third side) might present invalid side lengths when given specific numbers.

This suggests that LLMs are not just simply "misunderstanding" concepts, but that their internal representation of those concepts is "incoherent."

In other words, while they might appear to understand correctly from one perspective, they hold contradictory understandings from another. It's like someone who says, "I like curry" one day, and then casually says, "I hate curry" the next.

The Dangers of "Seeming Understanding" in Projects

How does this "Potemkin Understanding" affect our projects?

Increased Hidden Risks

An LLM's ability to "fake understanding" can be a significant pitfall, especially in the early stages of a project. For instance, an AI-generated requirements document or design specification might appear perfect at first glance. Technical terms may be used appropriately, and there might seem to be no logical contradictions.

However, if "Potemkin Understanding" is lurking beneath the surface, that perfection is merely an illusion. Once the actual development phase begins, this "seeming understanding" will manifest as unexpected bugs, rework, or fundamental design flaws.

If the AI fails to deeply understand the client's true needs or the complex interactions between systems, the project could face delays, ballooning costs, and a final deliverable that falls short of expectations.

This means that the risks in the "Uncertainty Performance Domain," a crucial element of project management, are further amplified by AI's "lack of understanding."

As system integrators, we want to avoid situations where unforeseen circumstances frequently arise, and deadlines become tight.

Decline in Trust

Even more serious is the decline in trust in AI. If AI consistently disappoints us despite answering that it "understands," we will hesitate to use it for critical business decisions or essential tasks.

This also leads to missing out on the benefits of AI adoption.

Consider this: a project sponsor makes a significant investment decision based on data or predictions provided by AI. But what if the AI's "understanding" was merely Potemkin Understanding? What kind of outcome would that lead to?

I believe this aspect of "trust" is the most crucial for integrating AI into society.

Overcoming "Potemkin Understanding"

So, how should we approach this difficult problem of "Potemkin Understanding"?

Rethinking the Human Role

First, it's crucial to adopt an even more critical perspective toward AI outputs. Don't blindly accept the "perfect" answers or definitions AI provides. Instead, constantly ask, "Does it truly understand?"

Furthermore, we need an attitude of multi-faceted verification to ensure that its understanding is actually at an "applicable" level and maintains consistency.

This means we humans need to establish new criteria for testing AI's "understanding." We'll need to design tests that not only ask for definitions but also assess its ability to apply concepts in more practical scenarios and handle contradictory information.

A New Perspective on "Co-creation" with AI

I firmly believe that LLMs are the "ultimate mashup tools." However, to unleash their true potential, we must deeply understand their characteristics, especially limitations like "Potemkin Understanding," and have the wisdom to utilize them with that foresight.

This might be akin to working on a project with a talented but still young newcomer.

They have the definitions and theories in their head, but due to limited practical experience, they make unexpected "oops" mistakes. In such situations, to fully leverage their abilities, we provide concrete guidance, supplement their thought processes, and create an environment where they can confidently take on challenges, right?

Shouldn't we offer AI similar "extensive support" and "careful feedback loops"?

For example, humans could review AI-generated design proposals from multiple perspectives, pointing out inconsistencies or unrealistic aspects. Then, by feeding that feedback back to the AI, we can improve the quality of the AI's "understanding" itself.

In this way, I believe that deeply considering the nature of "co-creation," where humans and AI leverage their respective strengths and compensate for weaknesses, holds the key to future project success.

Does Your Project's AI Truly "Understand"?

Today, I've delved into the phenomenon of "Potemkin Understanding," where AI merely "fakes understanding." We've seen through concrete data and examples that even if LLMs can accurately explain concepts, they might exhibit a "seeming understanding" that differs from human understanding in application and consistency.

This fact teaches us the importance of not overestimating AI's capabilities and utilizing it with an understanding of its limitations.

For the project you're currently working on, or the tasks you've entrusted to AI, can you truly say that AI "fundamentally" understands?

Or, beneath the clever "faking of understanding," might "Potemkin Understanding" be lurking, ready to betray your expectations someday? I strongly feel that this question is a crucial one we must constantly keep in mind to lead projects to success in the age of AI.

How will you detect and deal with AI's "seeming understanding"?

Please share your thoughts.

Follow me!