Back to series
· 14 min read

Why Does AI Need a Mathematical Foundation? From Mathematical Modeling to Model Training, Understanding the Math Behind Artificial Intelligence


The development of artificial intelligence depends on algorithms, data, and computing power, but these three alone cannot fully explain why AI can learn, how it makes predictions, or why models can generate content.

From machine learning to deep learning, from neural networks to large language models, almost all core technologies are built on a mathematical foundation.

Mathematics is not only the tool for implementing AI algorithms, but also the theoretical foundation for understanding how artificial intelligence works.

For example:

  • Why can a model learn from data? This involves mathematical modeling and optimization theory.
  • How does a model determine whether two objects are similar? This involves vectors, matrices, and distance computation.
  • How do neural networks adjust their parameters? This involves calculus and gradient descent.
  • How do models handle uncertainty? This involves probability theory and statistics.
  • How does a large language model choose the next Token? This involves probability distributions and numerical computation.
  • Why do models overfit? This involves statistical learning theory and generalization capability.

Understanding these questions requires mastery of an interconnected body of mathematical knowledge.

Starting from the core working mechanisms of AI, this article outlines which mathematical foundations artificial intelligence requires, what problems each area of mathematics solves, and their specific applications in machine learning and large models.


1. The Relationship Between AI and Mathematics

1.1 AI Is Essentially a Class of Mathematical Modeling and Computation Problems

Artificial intelligence can be understood as: using mathematical models to describe patterns in data, then using computational methods to learn model parameters, so that the model gains the ability to predict, classify, make decisions, or generate.

Take a simple house price prediction problem as an example.

Suppose we want to predict house prices based on house area. We can build a linear model:

y = wx+b

Where:

  • x: house area.
  • y: predicted house price.
  • w: model weight.
  • b: bias term.

The model needs to learn w and b from existing house data so that the predictions are as close as possible to the actual house prices.

This process involves several mathematical questions:

  1. How should the relationship between input and output be described?
  2. How should the difference between predicted results and actual results be measured?
  3. How should model parameters be adjusted based on the error?
  4. How can we determine whether the model can handle data it has never seen?

The corresponding mathematical foundations include:

Mathematical Field Role in AI
Linear algebra Representing and computing data, vectors, matrices, and model parameters
Calculus Describing function changes and computing gradients of model parameters
Probability theory Describing randomness and uncertainty
Mathematical statistics Estimating patterns from sample data and analyzing model performance
Mathematical optimization Finding parameters that minimize the loss function
Information theory Measuring the amount of information, uncertainty, and differences between probability distributions
Discrete mathematics Supporting logic, graph structures, algorithms, and computational processes

1.2 The Different Roles Mathematics Plays in AI

Mathematics does not serve a single purpose in AI systems; it runs through the entire model lifecycle.

Stage Core Question Related Mathematics
Data representation How should data be represented? Linear algebra, discrete mathematics
Model construction How should the relationship between input and output be described? Functions, linear algebra
Model training How should parameters be adjusted? Calculus, optimization theory
Probabilistic prediction How should the uncertainty of predictions be represented? Probability theory
Model evaluation How should model performance be judged? Mathematical statistics, probability theory
Content generation How should an output be selected from candidate results? Probability theory, information theory
Model compression How should computation and storage costs be reduced? Linear algebra, numerical analysis, optimization

The mathematical foundations of AI can be understood as a complete system:

flowchart TB
    A["Mathematical Foundations of AI"]

    B["Data and Representation"]
    C["Models and Functions"]
    D["Training and Optimization"]
    E["Probability and Statistics"]
    F["Information and Computation"]

    G["Linear Algebra"]
    H["Calculus"]
    I["Probability Theory and Mathematical Statistics"]
    J["Mathematical Optimization"]
    K["Information Theory"]
    L["Discrete Mathematics and Numerical Analysis"]

    A --> B
    A --> C
    A --> D
    A --> E
    A --> F

    B --> G
    C --> G
    D --> H
    D --> J
    E --> I
    F --> K
    F --> L

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style J fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style K fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style L fill:#F5F8F4,stroke:#B5CBBB,color:#40594A

II. Linear Algebra: How Does AI Represent and Process Data?

Linear algebra is one of the most important mathematical foundations of modern artificial intelligence.

Whether traditional machine learning, deep neural networks, or large language models, all require converting data into numerical form and then performing computations through vector and matrix operations.

2.1 Scalars, Vectors, Matrices, and Tensors

Common data structures in AI can be understood at four levels.

Scalar

A single numerical value, for example:

x=5

It can represent a temperature, a weight, or a loss value.

Vector

A one-dimensional structure composed of multiple numerical values:

\mathbf{x}= \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix}

It can represent multiple features of a sample, and can also represent the embedding of a text token.

Matrix

A two-dimensional structure composed of multiple vectors:

A= \begin{bmatrix} 1&2&3\\ 4&5&6 \end{bmatrix}

It can represent the features of a batch of samples, and can also represent the weight parameters of a neural network.

Tensor

A tensor is the generalization of scalars, vectors, and matrices to higher dimensions.

For example:

  • An RGB image can be represented as a three-dimensional tensor.
  • A batch of images can be represented as a four-dimensional tensor.
  • The embeddings of a batch of text sequences can usually be represented as a three-dimensional tensor.

Taking large language models as an example, the input embedding can usually be represented as:

X\in\mathbb{R}^{B\times T\times D}

Where:

  • B: Batch Size (the size of a batch).
  • T: Sequence Length (the length of a sequence).
  • D: Hidden Size (the hidden dimension).

This means the model needs to process multiple samples at the same time, multiple tokens in each sample, and the feature vector corresponding to each token.

2.2 Vectors: The Mathematical Representation of Data

Machine learning models usually cannot directly understand text, images, and sound; instead, they need to convert them into numerical vectors.

For example, a simple user feature can be represented as:

\mathbf{x}= \begin{bmatrix} \text{Age}\\ \text{Income}\\ \text{Number of Purchases} \end{bmatrix}

Suppose the data for a certain user is:

\mathbf{x}= \begin{bmatrix} 28\\ 12000\\ 15 \end{bmatrix}

The model can process these features through vector operations.

In natural language processing, embeddings map tokens to high-dimensional vectors.

For example:

Embedding(\text{"Artificial Intelligence"})= \begin{bmatrix} 0.12\\ -0.35\\ 0.68\\ \vdots \end{bmatrix}

These vectors are not manually specified semantic encodings; rather, they are numerical representations obtained through model training.

Geometric relationships in vector space can, to some extent, reflect semantic relationships between data.

2.3 Vector Similarity: How Does a Model Measure the Closeness Between Two Objects?

In vector retrieval, recommender systems, semantic search, and RAG, it is often necessary to determine whether two vectors are similar.

Common methods include:

1. Euclidean distance

d(\mathbf{x},\mathbf{y})= \sqrt{\sum_{i=1}^{n}(x_i-y_i)^2}

The smaller the distance, the closer the two vectors are in Euclidean space.

2. Cosine similarity

\cos(\theta)= \frac{\mathbf{x}\cdot\mathbf{y}} {\|\mathbf{x}\|\|\mathbf{y}\|}

Cosine similarity focuses on the directional relationship between two vectors rather than simply comparing their lengths.

In text Embedding retrieval, cosine similarity is a common similarity metric.

3. Dot product

\mathbf{x}\cdot\mathbf{y} = \sum_{i=1}^{n}x_iy_i

The dot product is affected by both vector direction and vector magnitude.

Note: cosine similarity, Euclidean distance, and dot product are not metrics that can be interchanged unconditionally. The specific choice depends on whether the vectors are normalized, the model training objective, and the design of the retrieval system.

2.4 Matrix Operations: The Core Computation of Neural Networks

Many computations in neural networks can be expressed as matrix multiplication.

A simple linear transformation:

\mathbf{y}=W\mathbf{x}+\mathbf{b}

Where:

  • \mathbf{x}: Input vector.
  • W: Weight matrix.
  • \mathbf{b}: Bias vector.
  • \mathbf{y}: Output vector.

For a batch of input data, this can be written as:

Y=XW+b

Where:

X\in\mathbb{R}^{B\times D_{in}}
W\in\mathbb{R}^{D_{in}\times D_{out}}
Y\in\mathbb{R}^{B\times D_{out}}

Matrix multiplication enables models to process large amounts of data efficiently and is also an important reason why GPUs can accelerate deep learning computations.

2.5 Why Does Transformer Depend So Heavily on Linear Algebra?

Modern large language models generally use the Transformer architecture.

One of the most important computations is Attention:

Attention(Q,K,V)= softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

Where:

  • Q: Query, the query matrix.
  • K: Key, the key matrix.
  • V: Value, the value matrix.
  • d_k: The dimension of the Key vector.

This computation process includes:

  1. Matrix transpose.
  2. Matrix multiplication.
  3. Scaling.
  4. Softmax.
  5. Matrix multiplication.

Attention uses matrix operations to compute the association weights between different tokens, and then aggregates information according to these weights.

This shows that understanding the computation process of Transformer is inseparable from matrices, vectors, and linear transformations.


3. Calculus: How Do Neural Networks Learn?

If linear algebra addresses data representation and computation, then calculus primarily addresses how model parameters are adjusted.

The core of neural network training is continuously adjusting parameters to make the model's predictions closer to the target results.

3.1 Functions: How Do Models Describe the Relationship Between Input and Output?

A machine learning model can be abstracted as a function:

\hat{y}=f(x;\theta)

Where:

  • x: Input data.
  • \hat{y}: The model's prediction result.
  • \theta: Model parameters.
  • f: The model's computation process.

For neural networks, a model is usually composed of multiple functions.

For example:

h_1=f_1(x;\theta_1)
h_2=f_2(h_1;\theta_2)
\hat{y}=f_3(h_2;\theta_3)

Model parameters may include:

  • Weight matrices.
  • Bias vectors.
  • Normalization parameters.
  • Embedding parameters.
  • Attention projection matrices.

Neural network training is the process of finding a more suitable set of parameters.

3.2 Derivatives: How Do Changes in Parameters Affect the Model?

A derivative describes the rate of change of a function near a certain point.

For a function:

y=f(x)

Its derivative is:

f'(x)=\frac{dy}{dx}

Derivatives can help us determine:

  • When the input changes, how does the output change?
  • Is the function rising or falling at a certain position?
  • How much change in the input causes how much change in the output?

In AI, we care more about the effect of changes in model parameters on the loss function.

Suppose the model loss is:

L(w)=(wx-y)^2

where w is the parameter to be learned.

Differentiate with respect to w:

\frac{dL}{dw}=2x(wx-y)

This derivative describes how the loss function changes when parameter w changes.

If the derivative is positive, it usually means that increasing w at the current position will increase the loss; if the derivative is negative, it means that increasing w will decrease the loss.

Therefore, the derivative can provide directional information for parameter updates.

3.3 Partial derivatives: What if neural networks have a large number of parameters?

Real neural networks usually have millions, billions, or even more parameters.

At this point, the loss function is no longer a function of a single variable, but a function jointly determined by multiple parameters:

L(\theta_1,\theta_2,\ldots,\theta_n)

We need to compute the partial derivative of the loss function with respect to each parameter separately:

\frac{\partial L}{\partial \theta_i}

The partial derivatives of all parameters can be combined into a gradient vector:

\nabla_\theta L= \begin{bmatrix} \frac{\partial L}{\partial \theta_1}\\ \frac{\partial L}{\partial \theta_2}\\ \vdots\\ \frac{\partial L}{\partial \theta_n} \end{bmatrix}

The gradient represents the direction in which the loss function increases fastest.

Therefore, the negative gradient direction is usually the direction that locally decreases the loss.

3.4 Gradient descent: How do models update parameters?

Gradient descent is a common optimization method.

Its basic update formula is:

\theta_{t+1} = \theta_t-\eta\nabla_\theta L(\theta_t)

where:

  • \theta_t: current model parameters.
  • \eta: learning rate.
  • \nabla_\theta L: gradient corresponding to the current parameters.
  • \theta_{t+1}: updated model parameters.

The learning rate determines the step size of each parameter update.

If the learning rate is too large, it may cause the training process to oscillate or even diverge; if the learning rate is too small, it may slow down training.

3.5 Chain rule: the mathematical foundation of backpropagation

Neural networks usually consist of multiple computational layers.

When the final output produces an error, the error information needs to be passed back layer by layer to compute the impact of each layer's parameters on the final loss.

This relies on the chain rule in calculus.

Suppose:

z=f(y)
y=g(x)

Then:

\frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}

For multi-layer neural networks, the chain rule can decompose the derivative of a complex function into the product of multiple local derivatives.

This is the mathematical foundation of the backpropagation algorithm.

3.6 The overall process of backpropagation

flowchart TB
    A["Input training data"]
    B["Forward propagation"]
    C["Obtain model predictions"]
    D["Compute loss function"]
    E["Compute gradient of loss with respect to output"]
    F["Backpropagate layer by layer via chain rule"]
    G["Compute gradients for each parameter"]
    H["Optimizer updates parameters"]
    I["Proceed to next training round"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style I fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Note that backpropagation itself is mainly responsible for computing gradients; the optimizer is what actually performs parameter updates.

For example, SGD, Adam, and AdamW are all optimization algorithms that update parameters based on gradients and their own maintained state.

3.7 Main applications of calculus in AI

Technique Related calculus concepts Main role
Linear regression Derivatives, partial derivatives Optimize regression parameters
Neural networks Chain rule Compute gradients in multi-layer networks
Backpropagation Multivariable calculus Compute parameter gradients layer by layer
Gradient descent Gradient Determine the direction of parameter updates
Adam optimizer Gradients and their statistics Adaptively adjust parameter updates
Deep learning Higher-order derivatives, Jacobian, etc. Analyze the change characteristics of complex models

4. Probability Theory: How Does AI Handle Uncertainty?

Real-world data often contains noise, incomplete information, and uncertainty.

Models cannot always produce a unique and deterministic answer, so probability theory is needed to describe uncertainty.

4.1 Random Variables and Probability Distributions

A random variable is a mathematical variable used to describe the outcome of a random phenomenon.

For example, predicting whether an email is spam:

Y\in\{0,1\}

where:

  • Y=1: spam.
  • Y=0: normal email.

The model can output:

P(Y=1|X)=0.95

This means that, given input X, the model believes the probability that the email is spam is 0.95.

Conditional probability is very important here.

Classification models in machine learning often do not output the class directly; instead, they first estimate the conditional probabilities of different classes, and then obtain the final classification result according to a decision rule.

4.2 Conditional Probability: How Do Models Predict Based on Context?

Conditional probability represents the probability of another event occurring given that a certain condition has already been met.

P(A|B)=\frac{P(A\cap B)}{P(B)}

In natural language processing, a model needs to predict the next Token based on the preceding Tokens.

For example:

Given the context:

The weather today is very

The model may calculate:

P(\text{"good"}|\text{"The weather today is very"})
P(\text{"cold"}|\text{"The weather today is very"})
P(\text{"hot"}|\text{"The weather today is very"})

Finally, it obtains a probability distribution over candidate Tokens.

The autoregressive generation process of a large language model can be expressed as:

P(x_1,x_2,\ldots,x_T) = \prod_{t=1}^{T} P(x_t|x_1,\ldots,x_{t-1})

This means that the model gradually generates the complete sequence by continuously predicting the next Token.

4.3 Bayes' Theorem: How Do We Update Judgments Based on New Evidence?

Bayes' theorem is an important tool in probability theory:

P(A|B)= \frac{P(B|A)P(A)}{P(B)}

where:

  • P(A): prior probability.
  • P(B|A): likelihood.
  • P(B): marginal probability of the evidence.
  • P(A|B): posterior probability.

It describes an important process:

how to update an original judgment after obtaining new evidence.

For example, a spam filtering system can combine keywords appearing in an email, sender characteristics, and historical data to update the probability that the email is spam.

The naive Bayes classifier is a classic type of machine learning algorithm built on Bayesian theory.

4.4 Softmax: How Do We Convert Model Outputs into Probabilities?

The last layer of a neural network usually outputs a set of real numbers called Logits.

Logits themselves are not probabilities.

Softmax can convert them into a probability distribution:

P(y_i)= \frac{e^{z_i}} {\sum_{j=1}^{n}e^{z_j}}

where:

  • z_i: the Logit corresponding to the i-th class.
  • P(y_i): the probability corresponding to the i-th class.

The Softmax output satisfies:

0\leq P(y_i)\leq 1
\sum_{i=1}^{n}P(y_i)=1

For example, the model outputs for three classes:

Category Logit Softmax Probability
Cat 2.0 0.665
Dog 1.0 0.245
Bird 0.0 0.090

The model considers the current input most likely to belong to the cat category.

Note: the probabilities output by Softmax do not automatically represent the model's calibrated true confidence. The model may also assign a high probability to an incorrect prediction.

4.5 Main Applications of Probability Theory in AI

Technology Probability Knowledge Main Role
Logistic Regression Conditional Probability Classification Prediction
Naive Bayes Bayes' Theorem Probabilistic Classification
Hidden Markov Model Conditional Probability, Joint Probability Sequence Modeling
Neural Network Classification Softmax Output Class Probabilities
Large Language Model Conditional Probability Predict the Next Token
Diffusion Model Probability Distribution Learn the Data Generation Process
Reinforcement Learning Stochastic Policy, State Transition Probability Describe the Decision-Making Process

5. Mathematical Statistics: How Do Models Learn Patterns from Data?

Probability theory primarily studies random phenomena and their probability laws.

Mathematical statistics focuses more on how to use finite samples to infer population characteristics and how to determine whether a model's learning results are reliable.

This is a very important foundation for machine learning.

5.1 Population and Sample

In machine learning, we usually cannot obtain all possible data and can only use a limited set of training samples.

For example:

  • Population: all possible user behaviors.
  • Sample: the collected 1 million user behavior records.

The model learns patterns from samples and hopes to perform well on unseen data as well.

This involves an important concept:

Good performance on training data does not mean the model will also perform well on real data.

What the model truly needs is generalization ability.

5.2 Mean, Variance, and Standard Deviation

The mean is used to describe the central tendency of the data:

\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i

Variance is used to describe the degree of dispersion in the data.

Population variance:

\sigma^2= \frac{1}{N} \sum_{i=1}^{N}(x_i-\mu)^2

A common unbiased estimate of the sample variance is:

s^2= \frac{1}{n-1} \sum_{i=1}^{n}(x_i-\bar{x})^2

The standard deviation is the square root of the variance:

s=\sqrt{s^2}

These statistics have many uses in AI:

  • Determine whether the data has large fluctuations.
  • Detect outliers.
  • Perform data standardization.
  • Analyze changes in metrics during training.
  • Evaluate the distribution of model prediction errors.

5.3 Normal Distribution

The normal distribution is a common continuous probability distribution:

X\sim N(\mu,\sigma^2)

where:

  • \mu: mean.
  • \sigma^2: variance.

The normal distribution frequently appears in statistical inference, error analysis, and model assumptions.

For example, some regression models assume that the error term follows a normal distribution with zero mean and fixed variance, to facilitate parameter estimation and statistical inference.

However, note that not all AI data follows a normal distribution, nor do all machine learning models require data to follow a normal distribution.

5.4 Overfitting and Underfitting

This is a very important statistical learning problem in machine learning.

Underfitting

The model is too simple to capture the main patterns in the data.

Signs:

  • Poor performance on the training set.
  • Poor performance on the test set as well.

Overfitting

The model overfits the details and even the noise in the training data, leading to reduced predictive ability on new data.

Signs:

  • Very good performance on the training set.
  • Noticeably worse performance on the test set.

Whether the model is overfitting can be observed from its performance on the training set and validation set.

flowchart TB
    A["Prepare training data"]
    B["Train model"]
    C["Evaluate training set performance"]
    D["Evaluate validation set performance"]
    E{"Are training and validation performance reasonable?"}
    F["Possible underfitting"]
    G["Good model generalization performance"]
    H["Possible overfitting"]
    I["Adjust model complexity or training strategy"]

    A --> B
    B --> C
    C --> D
    D --> E
    E -->|"Both training and validation performance are poor"| F
    E -->|"Both training and validation performance are good"| G
    E -->|"Training performance is good, validation performance is poor"| H
    F --> I
    H --> I
    I --> B

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49

5.5 Bias and Variance

The Bias-Variance Tradeoff in machine learning describes the relationship between model complexity and prediction error.

Under the conditions where the classical squared error decomposition applies:

E[(\hat{f}(x)-f(x))^2] = Bias^2+\ Variance+\ Noise

Where:

  • Bias: reflects the systematic difference between the model's predictions and the true pattern.
  • Variance: reflects how sensitive the model is to changes in the training data.
  • Noise: irreducible noise.

Generally speaking:

  • A model that is too simple may produce high bias.
  • A model that is too complex may produce high variance.
  • Model design needs to strike a reasonable balance between the two.

This is also why having more model parameters does not necessarily mean the model is better.

5.6 Training Set, Validation Set, and Test Set

In machine learning, data is usually divided into different sets.

Dataset Main role
Training set Used to learn model parameters
Validation set Used for model selection and hyperparameter tuning
Test set Used for final evaluation of the model's generalization performance

Particular attention must be paid to data leakage.

If information from the test set is involved in training or model selection, the final test metrics may be overly optimistic and cannot truly reflect the model's generalization ability.


6. Mathematical Optimization: How to Find Better Model Parameters?

Model training can be abstracted as an optimization problem:

\theta^*= \arg\min_{\theta}L(\theta)

Where:

  • \theta: model parameters.
  • L(\theta): loss function.
  • \theta^*: the parameters that minimize the loss function.

Mathematical optimization studies how to find the optimal solution that satisfies the objective.

6.1 Loss Functions

A loss function measures the difference between the model's predicted results and the target results.

Different tasks use different loss functions.

Mean Squared Error (MSE)

Commonly used for regression tasks:

MSE= \frac{1}{n} \sum_{i=1}^{n}(y_i-\hat{y}_i)^2

Mean Absolute Error (MAE)

MAE= \frac{1}{n} \sum_{i=1}^{n}|y_i-\hat{y}_i|

Cross-Entropy Loss

Commonly used for classification tasks and language model training.

For a multi-class classification problem with a single sample:

L=-\sum_{i=1}^{C}y_i\log(p_i)

Where:

  • C: number of classes.
  • y_i: distribution of the true labels.
  • p_i: probability distribution predicted by the model.

For single-label classification, the true labels are usually represented using One-Hot encoding, so cross-entropy focuses on penalizing low predicted probabilities for the true class.