Artificial intelligence, especially generative AI represented by large language models, is demonstrating increasingly strong capabilities in language understanding, content generation, code writing, and handling complex tasks.
However, stronger capabilities do not mean that models already possess reliable factual judgment, nor do they mean that models can fully understand their own decision-making processes.
In practical applications, AI still faces several important issues:
- Hallucination: The model generates information that appears plausible but is actually inaccurate or nonexistent.
- Bias: The model exhibits unfair, one-sided, or discriminatory tendencies in its output.
- Interpretability / Explainability: Why the model arrives at a certain result, and whether humans can understand and verify its decision-making process.
These issues affect not only the accuracy of models but also directly relate to the safety, trustworthiness, and practical deployability of AI systems.
Starting from technical principles, this article analyzes the limits of AI capabilities and how current mainstream technologies mitigate these problems.
I. What Are the Limits of AI Capabilities?
The limits of AI capabilities refer to the constraints on a model in areas such as knowledge, reasoning, factual judgment, decision-making, and explanation of results.
A model's ability to generate fluent language does not mean it has mastered all relevant facts; its ability to perform complex reasoning does not mean every step of that reasoning is correct; and its ability to explain its own answer does not mean the explanation fully reflects the model's internal computation process.
1.1 Demonstrated Capability Is Not the Same as Actual Capability
Taking large language models as an example, they can:
| Capability | Typical Manifestation | Potential Limitation |
|---|---|---|
| Language generation | Writing articles, answering questions | Fluent language does not mean factual accuracy |
| Knowledge representation | Explaining technical concepts | Knowledge may be outdated, incomplete, or incorrect |
| Logical reasoning | Analyzing conditions, solving problems | Complex reasoning may involve errors in intermediate steps |
| Code generation | Writing programs, fixing code | Code may contain logical flaws or security vulnerabilities |
| Information summarization | Summarizing documents and materials | May omit key conditions or distort the original meaning |
| Decision support | Comparing options, offering recommendations | Depends on input quality and goal setting |
| Self-explanation | Explaining the basis for an answer | Explanations do not necessarily faithfully reflect internal mechanisms |
1.2 Why Do Models Have Capability Boundaries?
The core training objective of large language models is usually to predict the next token based on the existing context.
Its autoregressive probabilistic modeling can be expressed as:
The model learns language structures, knowledge associations, and contextual patterns from large amounts of data, and predicts subsequent tokens step by step during generation.
However, this objective itself is not directly equivalent to:
- Verifying whether every fact is true.
- Judging whether every information source is credible.
- Ensuring that every step of logical reasoning is correct.
- Identifying all potential social biases.
- Fully explaining the decision-making process of the internal neural network.
Therefore, language modeling capability is distinct from factual verification capability and reliable reasoning capability.
1.3 Three Core Issues
This article focuses on three interrelated but distinct issues.
| Issue | Core Tension | Main Impact |
|---|---|---|
| Hallucination | Generated content is inconsistent with real facts | Factual reliability |
| Bias | Output is affected by unfair tendencies in data and modeling processes | Fairness and social impact |
| Interpretability | The model's internal decision-making process is difficult for humans to understand and verify | Transparency and auditability |
II. What Is AI Hallucination?
2.1 Definition of Hallucination
AI hallucination refers to the model generating content that is inconsistent with facts, lacks reliable evidence, or is inconsistent with the input material, while presenting it in a seemingly plausible and confident manner.
For example:
- Fabricating nonexistent papers, authors, or references.
- Inventing nonexistent APIs, configuration parameters, or technical features.
- Providing incorrect legal provisions, policy regulations, or statistical data.
- Adding information that is not present in the original text when summarizing a document provided by the user.
- Drawing an overly definitive conclusion on a question that has no clear answer.
It should be noted that hallucination does not necessarily mean the entire answer is wrong.
Sometimes most of the model's content is correct, and only one key fact, number, name, or logical relationship is wrong. Such localized errors can also cause serious consequences in professional scenarios.
2.2 Why Does Hallucination Occur?
Hallucination is not caused by a single factor; it is related to factors such as the model's training objectives, knowledge representation, context quality, and generation mechanisms.
1. Language Modeling Objectives Are Not Equivalent to Factual Verification
The model's training objective is usually to improve its ability to predict the next token.
During training, the model learns statistical patterns and associations in the data, rather than independently storing and verifying each fact in real time like a database system.
For example, the model may learn that:
- A certain technical concept usually appears together with a certain framework.
- A certain field often uses a certain API.
- A certain type of problem is usually addressed with a certain solution.
When context is insufficient, the model may rely on these associations to generate a technical detail that seems plausible but does not actually exist.
Key distinction: predicting a plausible expression and verifying a real fact are two different capabilities.
2. Parametric Knowledge Has Limitations
Large language models encode knowledge and language patterns into model parameters through training.
This form of knowledge representation has several characteristics:
- Distributed storage: knowledge is not stored in the form of traditional database records.
- Implicit representation: knowledge is usually dispersed across parameters and intermediate representations.
- Update cost: knowledge in model parameters does not automatically change as the external world changes.
- Retrieval uncertainty: the model may not be able to accurately activate the relevant knowledge in all contexts.
For example, the model may know the common usage patterns of a software framework, but may not be able to accurately determine whether a particular API has changed in the latest version.
3. Insufficient contextual information
Model outputs depend heavily on the input context.
When the information provided by the user is incomplete or ambiguous, or when the question itself lacks the necessary conditions, the model may attempt to fill in the missing information.
For example:
The user asks:
Why does a certain system's interface return 500?
If no logs, code, request parameters, or runtime environment are provided, the model cannot determine the real cause.
But the model may still give a specific explanation based on common failure patterns.
In this case, the answer may merely be a reasonable guess, not a verified root cause analysis.
4. The generation process is stochastic
Large language models typically first compute the probability distribution over candidate tokens, and then generate content according to a decoding strategy.
Let the model's output logits be:
Applying Softmax yields the probability distribution:
Where:
- V: the vocabulary size.
- z_i: the logit corresponding to the i-th token.
- P(x_i): the generation probability of the corresponding token.
Temperature can adjust the probability distribution:
A higher temperature generally makes the probability distribution flatter and increases generation diversity; a lower temperature generally makes the distribution more concentrated.
But it must be made clear:
Lowering the temperature does not fundamentally eliminate hallucinations.
Because even when generating deterministically, the model may still consistently produce false facts.
5. Error accumulation in long-form generation
An autoregressive model generates tokens step by step, with subsequent content depending on the context already generated.
If an earlier key fact is wrong, subsequent content may continue to build on that error.
For example:
- Step 1: Incorrectly determine that a certain API exists.
- Step 2: Write calling code based on this API.
- Step 3: Explain the parameters and return values around this code.
- Step 4: Further fabricate incorrect exception handling approaches.
The final result is an answer that is structurally complete and seemingly coherent in logic, but based on false facts.
2.3 Common types of hallucination
| Type | Manifestation | Example |
|---|---|---|
| Factual hallucination | Factual content is wrong | Incorrectly describing when a technology was released |
| Citation hallucination | Referenced sources do not exist | Fabricating papers, authors, or links |
| Entity hallucination | Names of people, organizations, or products are wrong | Inventing non-existent product features |
| Logical hallucination | Errors in the reasoning process | Drawing conclusions from invalid premises |
| Context hallucination | Inconsistent with the input material | Adding content not present in the original when summarizing a document |
| Code hallucination | Generating non-existent or incorrect interfaces | Using functions or parameters that do not exist |
2.4 The relationship between hallucination and knowledge cutoff
Knowledge cutoff refers to the limitation in the time range covered by the model's training knowledge.
The model may not be able to grasp events that occurred after the cutoff time of its training data.
For example:
- Newly released software versions.
- Newly added programming language features.
- The latest security vulnerabilities.
- The latest laws and regulations.
- The latest product prices and service policies.
Knowledge cutoff and hallucination are not the same concept.
A model not knowing the latest information does not necessarily lead to hallucination; but when the model tries to answer a question it does not actually have knowledge of, it may generate unverified information.
Therefore, one cannot rely solely on the knowledge in the model's parameters to solve all real-time information problems.
III. How to reduce AI hallucinations?
Hallucinations cannot be completely eliminated through a single parameter or a single technique; control is usually required at multiple levels: the model, the data, retrieval, tools, and the application architecture.
3.1 Optimize prompts and input constraints
Use prompts to clearly define the model's answer boundaries.
For example:
- Answer only based on the provided material.
- Clearly mark information that cannot be confirmed as unknown.
- Do not fabricate references, interfaces, or data.
- Provide supporting evidence for factual conclusions.
- Distinguish between known facts, speculation, and suggestions.
This approach can improve model behavior, but it cannot guarantee that the model will always follow the constraints.
Prompts are behavioral guidance, not equivalent to a factual verification mechanism.
3.2 Use RAG retrieval-augmented generation
RAG (Retrieval-Augmented Generation) provides the model with reference material relevant to the question by introducing external knowledge sources.
Its basic process is as follows:
flowchart TB
A["User asks a question"] --> B["Question understanding and processing"]
B --> C["Vectorization or keyword retrieval"]
C --> D["Retrieve relevant content from the knowledge base"]
D --> E["Relevance filtering and reranking"]
E --> F["Build augmented context"]
F --> G["LLM generates the answer"]
G --> H["Output the answer and its citations"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
The core value of RAG is that it lets the model refer to external information when answering a question, rather than relying solely on parametric knowledge.
But RAG does not mean the answer is necessarily correct.
It can still be affected by the following problems:
- The knowledge base itself contains incorrect information.
- The retrieval stage fails to find the correct material.
- The retrieved results are relevant to the question but cannot support the final conclusion.
- Document chunking leaves the context incomplete.
- The model misinterprets or misuses the retrieved content.
- The cited location does not match the actual conclusion.
So RAG solves the problem of knowledge acquisition and context supplementation; it does not automatically perform fact verification.
3.3 Using Tool Calling to Obtain External Facts
Tool Calling allows the model to invoke external systems instead of relying entirely on internal generation.
For example:
- Call a search engine to get the latest information.
- Call a database to query real records.
- Call a calculator to perform exact computation.
- Call a code execution environment to verify a program.
- Call an enterprise internal API to get business data.
A typical flow:
flowchart TB
A["User assigns a task"] --> B["Model analyzes the task"]
B --> C["Decide whether external tools are needed"]
C --> D["Generate tool call parameters"]
D --> E["External tool executes"]
E --> F["Return the real execution result"]
F --> G["Model analyzes the tool result"]
G --> H["Compose the final answer"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
Note that tool calling also carries risks.
The model may generate incorrect parameters, choose the wrong tool, or misinterpret the result returned by the tool. Input validation, permission control, and verification of execution results for the tools themselves therefore remain indispensable.
3.4 Introducing a Fact-Checking Mechanism
For important information, an independent verification step can be added after generation.
For example:
- Check whether the cited sources actually exist.
- Check whether the numbers can be reproduced by computation.
- Check whether the code compiles and runs.
- Check whether the conclusion can be derived from the reference documents.
- Check whether the output violates business rules.
A typical verification architecture:
flowchart TB
A["Model generates initial answer"] --> B["Extract key facts and conclusions"]
B --> C["Retrieve authoritative sources"]
C --> D["Verify facts item by item"]
D --> E{"Are there inconsistencies?"}
E -->|"Yes"| F["Correct or remove unreliable content"]
E -->|"No"| G["Keep verified content"]
F --> H["Output final answer"]
G --> H
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
For high-risk scenarios such as finance, healthcare, law, and industrial control, professional rules, human review, and audit mechanisms also need to be incorporated.
4. What Is AI Bias?
4.1 Definition of Bias
AI bias refers to systematic tendencies displayed by artificial intelligence systems during data processing, model training, prediction, or decision-making, causing different groups or different types of input to be affected unfairly and unreasonably.
Bias does not necessarily manifest as overtly discriminatory language.
It may also manifest as:
- Significantly lower recognition accuracy for certain groups.
- Inconsistent judgment criteria applied to different groups.
- Unfair screening outcomes in scenarios such as recruitment and credit.
- Insufficient information coverage for certain languages, cultures, and regions.
- Systematically ignoring or amplifying particular viewpoints.
4.2 Where Does AI Bias Come From?
AI bias is usually not produced solely by model algorithms; rather, it may be present throughout the entire AI system lifecycle.
1. Data Bias
Training data cannot comprehensively and evenly reflect the real world.
Common situations:
- The amount of data for certain groups is clearly insufficient.
- Data mainly comes from specific regions or cultural environments.
- Historical data itself contains social inequality.
- Sampling methods cause certain groups to be overlooked.
- Data labels contain subjective judgments or annotation errors.
For example, if a facial recognition system is trained mainly on images of one group of people, the model's recognition performance on other groups may decline.
This is a typical problem caused by imbalanced data distribution.
2. Label Bias
Supervised learning relies on data labels.
However, labels are not necessarily objective facts; they may also be influenced by annotators' perceptions and historical rules.
For example, recruitment data may use historical hiring outcomes as training labels.
If past hiring decisions contained gender bias, the model may learn and perpetuate this historical pattern.
Ultimately, the model optimizes its ability to fit historical labels; this does not mean the historical labels themselves are fair.
3. Algorithm and Objective Function Bias
Model training usually requires optimizing some objective function.
For example:
Where:
- \theta: model parameters.
- L(\theta): training loss function.
If the optimization objective focuses only on overall prediction accuracy, performance differences between different groups may be ignored.
For example:
A classification model has an accuracy of 98% on group A and 80% on group B.
If only overall accuracy is observed, significant performance degradation for group B may go unnoticed.
Therefore, a single overall metric cannot fully reflect the fairness of a model.
4. Feature Selection Bias
Some features may appear unrelated to the target but may indirectly reflect sensitive attributes.
For example:
- Residential area may be related to socioeconomic background.
- Educational experience may be affected by regional resource disparities.
- Consumption records may be associated with income level.
- Career history may be associated with gender or age structure.
Even if the model does not directly use sensitive attributes such as gender or race, it may indirectly learn similar information through other related features.
This phenomenon usually involves proxy variables.
5. Deployment Environment Bias
Good performance on training data does not mean the model will perform equally well in the real-world environment.
For example:
- Training data comes from cities, but the model is actually applied in rural areas.
- Training data comes from daytime, but the model is actually applied at night.
- Training data comes from one region, but the model is actually applied in other countries.
- Training data comes from an old version of a business process, but the actual business rules have changed.
When the actual input distribution changes relative to the training data distribution, model performance may decline.
This usually involves distribution shift.
4.3 Main Types of AI Bias
| Bias type | Core meaning | Typical manifestation |
|---|---|---|
| Historical bias | Historical data contains existing unfairness | Perpetuates past unfair decisions |
| Sample bias | The data sample cannot represent the target population | Recognition performance is worse for some groups |
| Label bias | Labels are influenced by subjective judgment | Training results inherit incorrect labels |
| Measurement bias | Data measurement quality differs across groups | Data for some groups contains systematic error |
| Selection bias | The way data enters the sample is unreasonable | Training samples lack specific groups |
| Algorithmic bias | The optimization objective or model design ignores fairness | Overall metrics are good, but group differences are obvious |
| Deployment bias | The training environment differs from the actual environment | Performance declines for some groups after deployment |
4.4 AI Bias and Fairness
AI fairness (Fairness) is an important direction for studying whether different individuals or groups receive reasonable and consistent treatment.
But fairness is not a single metric.
Different fairness definitions may focus on different goals.
1. Demographic Parity
Requires that different groups have similar proportions of positive predictions.
Let:
- A: group attribute.
- \hat{Y}: model prediction.
Its constraint can be expressed as:
For example, loan approval rates are the same across groups.
However, this metric does not necessarily consider whether the true risk levels of different groups differ.
2. Equal Opportunity
Requires that, within different groups, true positive samples have the same probability of being correctly identified.
For example, among people who truly meet the loan conditions, the chance of approval should be similar across groups.
3. Equalized Odds
Requires that true positive rates and false positive rates be similar across groups.
where y takes 0 and 1, respectively.
It focuses on both:
- True Positive Rate.
- False Positive Rate.
4. Why may fairness metrics conflict?
Different fairness definitions focus on different goals.
In real data, when the base rates of different groups differ and model predictions are not perfect, objectives such as demographic parity, equalized odds, and calibration of prediction results may not be satisfiable simultaneously.
Therefore, one cannot simply assume that meeting the requirement of a certain fairness metric means the entire AI system is absolutely fair.
Fairness evaluation must take into account the specific application scenario, legal requirements, business objectives, and potential social impact.
5. How to Reduce AI Bias?
Reducing bias requires covering the entire lifecycle of data, models, evaluation, and deployment.
5.1 Data Level
The focus is on improving data representativeness and data quality.
Main measures:
- Check the sample size and distribution of different groups.
- Identify missing data and anomalous data.
- Review labeling standards and annotation processes.
- Check historical data for unfair patterns.
- Evaluate data quality separately for different groups.
- Avoid directly using unverified historical decisions as correct labels.
5.2 Model Level
Fairness can be improved through algorithm design and training objectives.
Common methods:
| Method | Core idea |
|---|---|
| Resampling | Adjust the data proportions of different groups |
| Reweighting | Set different training weights for different samples |
| Fairness constraints | Add fairness constraints to the training objective |
| Adversarial debiasing | Reduce the predictability of sensitive attributes in model representations |
| Threshold adjustment | Adjust decision thresholds according to business and fairness requirements |
| Post-processing | Apply fairness correction to model outputs |
Note that removing sensitive attributes does not equal eliminating bias.
If the model can still infer sensitive attributes through proxy variables, bias may still exist.
5.3 Evaluation Level
You cannot focus only on overall accuracy.
You should also calculate the following separately for different groups:
- Accuracy: accuracy rate.
- Precision: precision rate.
- Recall: recall rate.
- FPR: false positive rate.
- FNR: false negative rate.
- Calibration: probability calibration.
For example:
| Evaluation Target | Accuracy | Recall | False Positive Rate |
|---|---|---|---|
| Group A | 96% | 95% | 3% |
| Group B | 95% | 78% | 12% |
Looking at overall accuracy, the difference in performance between the two groups is not obvious.
But Group B has a noticeably lower recall rate and a noticeably higher false positive rate.
This shows that aggregate metrics can mask important differences in group performance.
5.4 Deployment Level
After a model goes live, continuous monitoring is still required.
This includes:
- Monitoring shifts in the distribution of input data.
- Regularly evaluating model performance across different groups.
- Recording model versions and decision results.
- Establishing mechanisms for anomaly feedback and appeals.
- Retaining human review for high-risk decisions.
- Re-evaluating the model when business rules or data change.
Fairness is not a one-time check at the end of training, but ongoing governance that runs throughout the model lifecycle.
Six. What Is AI Explainability?
6.1 Definition of Explainability
AI explainability refers to the use of certain methods to let humans understand a model's inputs, its processing, its outputs, and why the model makes a particular prediction or decision.
Explainability primarily addresses one question:
Why did the model arrive at this result?
For example:
A credit assessment model determines that a user carries a high risk of default.
Knowing only the prediction result is not enough.
We also want to know:
- Which input features influenced the prediction?
- Which factors raised the risk score?
- Which factors lowered the risk score?
- Was the model affected by anomalous data?
- Is the prediction result consistent with the business rules?
6.2 Why Are Deep Learning Models Hard to Explain?
Among traditional machine learning models, some algorithms have relatively intuitive decision processes.
For example:
- Linear regression: the coefficient of each feature can be observed.
- Decision tree: the specific decision path can be examined.
- Rule-based systems: the rule conditions can be read directly.
But deep neural networks usually contain a large number of parameters and multiple layers of nonlinear transformations.
Take a neural network as an example:
Where:
- h^{(l)}: the hidden representation of layer l.
- W^{(l)}: the weight matrix of that layer.
- b^{(l)}: the bias term.
- f: the nonlinear activation function.
As the number of network layers increases, the input information undergoes multiple transformations before the model output is finally produced.
The problem is:
- A single parameter usually has no intuitive human semantics.
- Complex nonlinear interactions exist between features.
- Information may be distributed across multiple neurons and layers.
- The same output may be produced by different internal paths.
- The model's internal representations are hard to map directly onto human concepts.
For large language models, the situation is even more complex.
The generation of a single token may be jointly influenced by the following factors:
- The current input context.
- Token Embedding.
- Multiple layers of Attention.
- The feedforward network.
- Residual connections.
- Model parameters.
- The output probability distribution.
- The decoding strategy.
Therefore, one cannot simply interpret a single neuron or a group of parameters as a definite knowledge concept.
6.3 The Difference Between Explainability and Transparency
The two are related, but they are not the same concept.
| Concept | Focus | Example |
|---|---|---|
| Transparency | Whether system information is made public | Disclosing the model structure and documentation of the training data |
| Explainability | Whether the reasons for a result can be stated | Analyzing which features influenced the prediction |
| Understandability | Whether humans can understand the system's behavior | Using intuitive rules to describe the decision process |
| Auditability | Whether the system's behavior can be inspected and traced | Retaining inputs, outputs, model versions, and operation records |
A model may disclose its parameter scale and network structure and still be hard put to explain a specific output.
Likewise, a model may offer some kind of explanation report, but without logs and version records, a complete audit may still be difficult.
Seven. Mainstream AI Explainability Techniques
Explainability research is generally divided into two categories:
- Intrinsic Interpretability: the model itself has a relatively easy-to-understand structure.
- Post-hoc Explainability: after the model has been trained, additional techniques are used to explain its behavior.
7.1 Feature Importance
Feature importance measures the degree to which different input features affect a model's predictions.
For example, a credit risk model might use the following features:
- Income level.
- Debt-to-income ratio.
- Number of past delinquencies.
- Years of work experience.
- Loan amount.
The model can output a ranking of the importance of each feature.
Note, however:
Feature importance generally indicates how much the model relies on a feature; it is not equivalent to the feature's causal effect on real-world outcomes.
If two features are highly correlated, their importance may be split between them or may shift.
7.2 LIME
LIME (Local Interpretable Model-agnostic Explanations) is a model-agnostic local explanation method.
The core idea is:
Generate perturbed samples near a given sample to be explained, observe how the complex model's predictions change, and then use a simple model to approximate the behavior of the original model in that local region.
Its objective can be abstracted as:
Where:
- f: the original complex model.
- g: the simple model used for explanation.
- G: the set of interpretable models.
- \pi_x: the local weights around sample x.
- L: the local approximation error.
- \Omega(g): the complexity penalty term for the explanation model.