Artificial intelligence does not simply follow fixed rules based on input data; rather, it learns patterns from large amounts of data and uses those patterns to make predictions, classifications, generations, and decisions about unknown data.
Real-world data is often characterized by uncertainty:
- Whether an email is spam is usually not absolutely certain.
- A single image may contain visual features of multiple categories at the same time.
- A user may click on several different products next.
- A large language model, given the same context, may generate different reasonable answers.
- A model that performs excellently on the training set is not guaranteed to achieve the same results on unknown data.
Probability theory provides AI with the mathematical tools to describe uncertainty, while statistics helps us discover patterns from limited data, estimate parameters, test hypotheses, and evaluate models.
From machine learning to deep learning, from Bayesian classifiers to large language models, probability theory and statistics run through the entire process of data analysis, model training, predictive inference, and performance evaluation.
This article focuses on the most commonly used concepts in probability and statistics in AI, including random variables, probability distributions, conditional probability, Bayes' theorem, expectation, variance, common statistical distributions, parameter estimation, maximum likelihood estimation, hypothesis testing, sampling, and model evaluation.
1. What Are Probability Theory and Statistics?
1.1 Definition of Probability Theory
Probability Theory is the branch of mathematics that studies random phenomena and their laws.
The main problem it addresses is:
When outcomes are uncertain, how do we describe and calculate the likelihood of different events occurring?
For example:
- What is the probability that a coin lands heads up?
- What is the probability that an email is spam?
- What is the probability that a user will purchase a certain product?
- How confident is a model in its prediction?
Probability is usually denoted by P(A), the probability that event A occurs.
For a single event:
Where:
- P(A)=0: the event cannot occur.
- P(A)=1: the event is certain to occur.
- 0<P(A)<1: the event has a certain degree of uncertainty.
For example, the probability that a fair coin lands heads up is:
Probability is not a guarantee about any single outcome, but a mathematical description of the laws of random phenomena.
1.2 Definition of Statistics
Statistics is the discipline that studies how to collect, organize, analyze, and interpret data, and how to infer the laws of a population from the data.
The main problem it addresses is:
When we can only observe limited data, how do we use that data to infer unknown characteristics of the population?
For example:
- Based on the purchase records of a group of users, estimate the consumption preferences of the overall user base.
- Based on training samples, estimate model parameters.
- Based on test set results, judge the generalization ability of a model.
- Based on experimental results, determine whether a certain model optimization is truly effective.
1.3 Differences Between Probability Theory and Statistics
The two are closely related, but their research directions differ.
| Dimension | Probability Theory | Statistics |
|---|---|---|
| Research direction | Derive outcomes from a known probability model | Infer the model from observed data |
| Starting point | Probability models, random variables | Sample data |
| Main question | The likelihood of an event occurring | What patterns the data reflects |
| Typical tasks | Probability calculation, distribution analysis | Parameter estimation, hypothesis testing |
| Application in AI | Probabilistic prediction, Bayesian inference | Model training, experiment evaluation |
A simple process can be used to understand the relationship between the two:
flowchart TB
A["Probability Theory<br/>Build stochastic models"]
B["Describe data generation patterns"]
C["Generate or observe data"]
D["Statistics<br/>Analyze observed samples"]
E["Estimate parameters and test hypotheses"]
F["Update or validate probability models"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> A
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
In machine learning, probability theory helps us define how a model describes data, while statistics helps us use data to estimate model parameters and judge model performance.
2. Basic Concepts of Probability Theory
2.1 Random Experiment
A random experiment is an experiment that can be repeated under the same conditions but whose concrete outcome is uncertain.
For example:
- Tossing a coin.
- Rolling a die.
- Randomly selecting a user.
- Randomly drawing a sample from a dataset.
- Observing whether a user clicks on a recommended product.
In AI, random experiments can correspond to processes such as data sampling, random initialization, random data augmentation, and model prediction.
2.2 Sample Space
The sample space represents the set of all possible outcomes of a random experiment, and is usually denoted by \Omega.
For example, tossing a coin:
Rolling a six-sided die once:
When randomly drawing a user from a set of users, all the users that could be drawn also form a corresponding sample space.
2.3 Random Events
A random event is a subset of the sample space.
For example, rolling a die once:
Event A: the number of pips is greater than 3.
If the die is fair, then:
Common relationships between events include:
- The union of events.
- The intersection of events.
- The complement of an event.
- Mutually exclusive events.
- Independent events.
2.4 Mutually Exclusive Events
If two events cannot occur at the same time, they are said to be mutually exclusive.
For example, with a single roll of a die:
- A: the number of pips is 1.
- B: the number of pips is 6.
The two events cannot occur at the same time, therefore:
The probabilities of mutually exclusive events satisfy:
Note that being mutually exclusive is not the same concept as being independent.
For two events whose probabilities are both greater than zero, if they are mutually exclusive, then they cannot be independent.
2.5 Independent Events
If the occurrence of event A does not change the probability of event B occurring, then A and B are said to be independent of each other.
Mathematical definition:
For example, two consecutive independent coin tosses.
Heads on the first toss and heads on the second toss are two independent events:
In machine learning, the independence assumption is often used to simplify probability calculations.
For example, the naive Bayes classifier assumes that, given the class, each feature is conditionally independent of the others.
But features in real data are often correlated, so this is a modeling assumption rather than a fact that holds universally.
3. Random Variables: How Do We Convert Random Outcomes into Numerical Values?
3.1 What Is a Random Variable?
A random variable is a function that maps the outcomes of a random experiment to numerical values.
Random variables are usually denoted by the capital letters X, Y, and Z.
For example, tossing a coin:
Here, the random variable X has only two possible values:
A random variable is not an ordinary variable that changes randomly; it is a mathematical function used to describe random outcomes.
In AI:
- X can represent input features.
- Y can represent the prediction target.
- Z can represent a latent variable.
- \theta can represent model parameters.
3.2 Discrete Random Variables
A discrete random variable can take only a finite or countably infinite number of values.
For example:
- The outcome of a single coin toss.
- The number of visits to a website in one day.
- The number of tokens in a piece of text.
- Whether a user purchases a certain product.
Its probability can be expressed by a probability mass function:
For all possible values:
For example, the probability distribution of a fair die:
| Value of the random variable | Probability |
|---|---|
| 1 | 1/6 |
| 2 | 1/6 |
| 3 | 1/6 |
| 4 | 1/6 |
| 5 | 1/6 |
| 6 | 1/6 |
3.3 Continuous Random Variables
A continuous random variable can take any real value within some interval.
For example:
- How long a user stays.
- The price of a product.
- Continuous values obtained after normalizing image pixels.
- Certain continuous metrics predicted by a model.
Continuous random variables are usually described by a probability density function (PDF).
Written as:
The probability that a continuous random variable falls within an interval is calculated by integration:
For a continuous random variable:
This does not mean that a specific value cannot occur; rather, the probability of a single point is zero, so probability must be calculated over an interval.
3.4 Probability Mass Function and Probability Density Function
| Comparison dimension | PMF | |
|---|---|---|
| Chinese name | Probability mass function | Probability density function |
| Applies to | Discrete random variables | Continuous random variables |
| Mathematical notation | P(X=x) | f_X(x) |
| Probability calculation | Directly compute probability | Integrate the density |
| Probability at a single point | Can be greater than zero | Equals zero |
3.5 Cumulative Distribution Function
The cumulative distribution function (CDF) is defined as:
It represents the cumulative probability that the random variable takes a value less than or equal to a certain number.
The CDF applies to both discrete and continuous random variables.
For example:
For a continuous random variable:
The CDF has the following basic properties:
- Monotonically non-decreasing.
- Its range is [0,1].
- As x approaches negative infinity, the CDF approaches 0.
- As x approaches positive infinity, the CDF approaches 1.
The CDF is used in statistical modeling, quantile calculation, outlier analysis, and probability assessment.
4. Common Probability Distributions: How Does AI Describe Data Patterns?
A probability distribution describes the possible values of a random variable and their probability patterns.
Different types of data usually require different probability distributions for modeling.
4.1 Bernoulli Distribution
The Bernoulli distribution describes a random experiment with only two possible outcomes.
For example:
- Whether a user clicks an ad.
- Whether a user purchases a product.
- Whether an email is spam.
- Whether a model prediction is correct.
Definition:
Where:
Probability mass function:
Where x\in\{0,1\}.
Expectation:
Variance:
The Bernoulli distribution is an important probabilistic foundation for binary classification tasks.
4.2 Binomial Distribution
The binomial distribution describes the probability distribution of the number of successes in a fixed number of independent Bernoulli trials.
Definition:
Where:
- n: number of trials.
- p: probability of success on each trial.
- X: number of successes.
Probability mass function:
Expectation:
Variance:
For example, if an ad is shown to 100 users, each user has a 0.1 probability of clicking, and the click events are assumed to be independent, then the number of clickers can be modeled using a binomial distribution.
Note that in real advertising scenarios, different users may have different click probabilities, and user behavior may be correlated, so the binomial distribution is only a simplified model.
4.3 Categorical Distribution
The categorical distribution describes the probability that a random variable selects one category from a finite set of categories.
For example, a classification model predicts three animals:
These represent, respectively:
- Cat: 0.7.
- Dog: 0.2.
- Bird: 0.1.
It satisfies:
The categorical distribution is a common mathematical basis for the output probabilities of multi-class models.
4.4 Multinomial Distribution
The multinomial distribution is a generalization of the binomial distribution to the case of multiple categories.
Suppose each trial has K possible categories, with respective probabilities:
Perform n independent trials to obtain the occurrence count of each category:
The probability is:
where:
The multinomial distribution is commonly used in text classification, category frequency counting, and discrete output modeling for generative models.
4.5 Uniform Distribution
The uniform distribution indicates that within a given range, every possible value has the same probability density.
Continuous uniform distribution:
Probability density function:
Expected value:
Variance:
The uniform distribution is commonly used in random sampling, parameter initialization, and simulation experiments.
4.6 Normal Distribution
The normal distribution is one of the most important continuous probability distributions in statistics.
Definition:
where:
- \mu: mean.
- \sigma^2: variance.
- \sigma: standard deviation.
Probability density function:
The normal distribution has the following bell-shaped curve characteristics:
- The mean determines the center of the distribution.
- The standard deviation determines the spread of the distribution.
- The distribution is symmetric about the mean.
Common empirical ranges:
| Interval | Approx. proportion of total |
|---|---|
| \mu\pm\sigma | 68.27% |
| \mu\pm2\sigma | 95.45% |
| \mu\pm3\sigma | 99.73% |
The normal distribution has many applications in AI:
- Gaussian noise modeling.
- Parameter initialization.
- Measurement error analysis.
- Error assumptions in regression models.
- Latent variable modeling in variational autoencoders.
- Noise modeling in diffusion models.
It is important to note that not all real data follow a normal distribution. One cannot assume that data must conform to a normal distribution simply because a statistical method commonly uses the normal distribution.
4.7 Poisson Distribution
The Poisson distribution is used to describe the number of occurrences of a certain type of event within a fixed time or spatial range.
Definition:
Probability mass function:
where:
- \lambda: average number of event occurrences per unit range.
- k: actual number of occurrences of the event.
Expected value:
Variance:
Typical applications:
- Number of visits to a website per minute.
- Number of errors occurring in a system per hour.
- Number of clicks by a user within a certain period of time.
- Number of request arrivals in a service system.
The classic Poisson model usually requires that event occurrences have a certain degree of independence and a stable average occurrence rate. For data with obvious burstiness or overdispersion, other models may be needed.
4.8 Exponential Distribution
The exponential distribution is typically used to describe the waiting time between two consecutive events.
Definition:
Probability density function:
where \lambda represents the event occurrence rate.
Expected value:
Variance:
Common applications:
- Interarrival times of system requests.
- Intervals between failures of certain devices.
- Modeling service waiting times.
The exponential distribution is memoryless:
This means that, under the exponential distribution assumption, the time already waited does not change the distribution of the remaining waiting time.
4.9 Comparison of Common Probability Distributions
| Distribution | Type | Main Use | Typical Applications in AI |
|---|---|---|---|
| Bernoulli distribution | Discrete | Binary outcome | Binary classification |
| Binomial distribution | Discrete | Number of successes in a fixed number of trials | Click-count statistics |
| Categorical distribution | Discrete | Single selection among multiple categories | Multiclass prediction |
| Multinomial distribution | Discrete | Counts across multiple categories | Category frequency modeling |
| Uniform distribution | Continuous/Discrete | Equiprobable outcomes | Random sampling |
| Normal distribution | Continuous | Symmetric continuous data | Gaussian noise, parameter modeling |
| Poisson distribution | Discrete | Event counts per unit interval | Request counts, error counts |
| Exponential distribution | Continuous | Waiting time for events | Interarrival times, lifetime modeling |
5. Conditional Probability: How Does Known Information Change Probability?
5.1 What Is Conditional Probability?
Conditional probability represents the probability that event A occurs given that event B has already occurred.
Definition:
where it is required that:
The core idea of conditional probability is:
After we obtain new information, we need to recalculate the probability of an event under the new condition.
5.2 Conditional Probability Example
Suppose that in a system:
- 40% of users use mobile.
- 20% of users purchased a product.
- Among mobile users, 30% purchased a product.
Then:
This means that, given that the user is known to use mobile, the purchase probability is 30%.
Conditional probability is different from ordinary probability:
Usually, a new condition changes the probability of an event; only when A and B are independent are the two equal.
5.3 Joint Probability
Joint probability represents the probability that two events occur simultaneously.
From conditional probability:
It can also be written as:
This is the multiplication rule of probability.
For multiple variables, this can be extended to:
This kind of joint probability factorization is very important in probabilistic graphical models and autoregressive language models.
5.4 Law of Total Probability
The law of total probability is used to compute the total probability of an event through different mutually exclusive cases.
Suppose that:
forms a partition of the sample space; then:
For example, a user may come from different channels:
- Search channel.
- Advertising channel.
- Organic recommendation channel.
If the share of users from each channel and the purchase probability of users from each channel are known, the overall user purchase probability can be calculated.
5.5 Bayes' Theorem
Bayes' theorem is a very important formula in probability theory.
It is used to update the probability assessment of a hypothesis based on observed evidence.
where:
| Symbol | Meaning |
|---|---|
| P(A) | Prior probability |
| P(B\mid A) | Likelihood |
| P(B) | Evidence probability |
| P(A\mid B) | Posterior probability |
Its core logic is:
- Start with an initial judgment.
- Observe new evidence.
- Update the judgment according to how likely the evidence is.
- Obtain a new posterior probability.
5.6 Bayes' Theorem Case: Spam Detection
Assumptions:
- Spam accounts for 20% of all email.
- Normal email accounts for 80%.
- The probability that a spam email contains a certain keyword is 70%.
- The probability that a normal email contains that keyword is 10%.
Definitions:
- S: the email is spam.
- K: the email contains the specified keyword.
Given:
First, use the law of total probability to compute the probability that the keyword appears:
Substitute:
Then apply Bayes' theorem:
Therefore, under these assumptions, the posterior probability that an email containing this keyword is spam is about 63.64%.
Note that a single keyword is usually not enough to accurately determine the type of an email. Real spam classifiers model multiple features together.
5.7 Prior Probability, Likelihood, and Posterior Probability
These three concepts are the core of Bayesian reasoning.
Prior Probability (Prior)
The initial probability judgment about a hypothesis before observing the current evidence.
For example, the prevalence of a certain disease in the overall population.
Likelihood
The probability of observing the current data given that a certain hypothesis holds.
For example, the probability that people with the disease produce a certain test result.
Posterior Probability (Posterior)
The probability recalculated for a hypothesis after observing the evidence.
For example, the probability that a person actually has the disease given that the test result is positive.
5.8 The Bayesian Reasoning Process
flowchart TB
A["Establish prior probability"]
B["Observe new evidence"]
C["Compute likelihood"]
D["Compute evidence probability"]
E["Apply Bayes' theorem"]
F["Obtain posterior probability"]
G["Update the judgment based on new information"]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
6. Expectation, Variance, and Standard Deviation: How Do We Describe Data Characteristics?
A probability distribution can describe the complete behavior of a random variable, but in practical analysis it is often necessary to summarize data using a small number of statistical indicators.
The most important of these are expectation, variance, and standard deviation.
6.1 Mathematical Expectation
Mathematical expectation (Expected Value) represents the average result of a random variable over long-run repeated trials.
For a discrete random variable:
For a continuous random variable:
For example, the expectation of a fair six-sided die:
Note that the expected value is not necessarily a value the random variable can actually take.
The outcome of a die is only from 1 to 6, but the expectation is 3.5.
6.2 Linearity of Expectation
For random variables X and Y and constants a and b:
This property does not require X and Y to be independent.
In particular:
This is very important in machine learning, because many loss functions and statistical indicators require computing the average of samples.
6.3 Variance
Variance is used to measure the degree to which a random variable is dispersed around its expected value.
Definition:
It can also be expanded as:
The larger the variance, the more the data typically fluctuates relative to the mean.
6.4 Standard Deviation
Standard deviation is the square root of variance:
Standard deviation has the same unit as the original data, so it is easier to interpret.
For example:
- Two groups of data have the same average score.
- The first group has a smaller standard deviation, so the scores are more concentrated.
- The second group has a larger standard deviation, so the scores are more dispersed.
6.5 Population Variance and Sample Variance
In statistical analysis, it is necessary to distinguish between the population and the sample.
Population variance:
A common unbiased estimator of sample variance:
Where:
- N: population size.
- n: sample size.
- \mu: population mean.
- \bar{x}: sample mean.
Sample variance uses n-1 as the denominator, commonly called Bessel's correction.
It is used to correct the bias introduced by using the sample mean in place of the population mean, making the sample variance an unbiased estimator of the population variance.
6.6 Covariance
Covariance is used to describe the direction in which two random variables change together.
Definition:
Expanded:
Generally speaking:
- Covariance greater than 0: the two variables tend to change in the same direction.
- Covariance less than 0: the two variables tend to change in opposite directions.
- Covariance close to 0: the linear co-movement relationship is weak.
However, zero covariance does not necessarily mean that the two variables are independent.
6.7 Correlation Coefficient
The correlation coefficient is used to measure the degree of linear correlation between two variables.
The common Pearson correlation coefficient:
Its value range is:
| Correlation coefficient | Meaning |
|---|---|
| Close to 1 | Strong positive linear correlation |
| Close to 0 | Weak linear correlation |
| Close to -1 | Strong negative linear correlation |
Correlation does not imply causation.
For example, two variables may both be affected by a third variable and therefore show correlation, but there is no direct causal relationship between them.
6.8 Relationships Among Statistics
flowchart TB
A["Random variable X"]
B["Probability distribution"]
C["Expected value E(X)"]
D["Variance Var(X)"]
E["Standard deviation σ"]
F["Covariance Cov(X,Y)"]
G["Correlation coefficient ρ"]
A --> B
B --> C
B --> D
D --> E
B --> F
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
七、Statistics Basics: How to Infer the Population from a Sample?
7.1 Population and Sample
The population is the complete set of research subjects.
A sample is a subset of observations drawn from the population.
For example:
- Population: all users of a certain platform.
- Sample: 100,000 users randomly selected from the platform's users.
In practice, AI systems usually cannot obtain all real data, so samples are needed for model training and statistical inference.
7.2 Parameters and Statistics
Population parameters are used to describe population characteristics.
For example:
- Population mean \mu.
- Population variance \sigma^2.
- Population proportion p.
A statistic is a numerical value calculated from sample data.
For example:
- Sample mean \bar{x}.
- Sample variance s^2.
- Sample proportion \hat p.
An important task in statistics is to use sample statistics to estimate unknown population parameters.
7.3 Sampling
Sampling is the process of selecting a portion of a population as samples for study.
Common methods:
| Sampling method | Description |
|---|---|
| Simple random sampling | Every sample unit has the same chance of being selected |
| Stratified sampling | Stratify by characteristics, then sample within each stratum |
| Cluster sampling | Select some clusters as samples |
| Systematic sampling | Draw samples at fixed intervals |
| Convenience sampling | Select samples based on how easy they are to obtain |
In machine learning, splitting data into training, validation, and test sets also involves the idea of sampling.
If bias exists in the sampling process, the model may fail to represent the true target population.
7.4 Sampling Bias
Sampling bias refers to a situation where the sample selection mechanism causes the sample to be unable to reasonably represent the target population.
For example:
A recommendation system trains its model solely on data from active users, yet hopes the model will serve all users.
This may lead to the model learning insufficiently about the interests and behavioral patterns of less active users.
Common types of data bias include:
- Selection bias.
- Coverage bias.
- Non-response bias.
- Historical data bias.
- Label bias.
When a model performs poorly, sometimes it is not because the model architecture is insufficiently complex, but because the training data itself is biased.
7.5 Law of Large Numbers
The law of large numbers describes the following:
Under the appropriate conditions, as the number of independent and identically distributed samples increases, the sample mean approaches the population expectation in a probabilistic or almost sure sense.
A common formulation:
As the number of samples keeps increasing:
The law of large numbers is an important theoretical basis for using the sample average to estimate the population expectation.
However, it does not mean that increasing the sample size will automatically eliminate sampling bias.
If the source of the samples itself carries systematic bias, increasing the sample size may simply make the bias more stable.
7.6 Central Limit Theorem
The central limit theorem (CLT) is a very important theorem in statistics.
Under conditions such as independent and identically distributed samples with finite mean and finite variance, when the sample size is sufficiently large, the appropriately standardized distribution of the sample mean approaches the standard normal distribution.
Let:
The sample mean:
After standardization:
When n is sufficiently large:
The central limit theorem explains why many statistical inference methods can use the normal distribution as an approximation basis.
Note the following:
- The central limit theorem concerns the sampling distribution of statistics such as the sample mean.
- It does not mean that the raw data must follow a normal distribution.
- The required sample size depends on factors such as the shape of the original distribution, skewness, and tail characteristics.
VIII. Parameter Estimation: How Do We Learn Unknown Parameters from Data?
Statistical modeling typically requires estimating unknown parameters from a finite sample.
For example:
- Estimating the mean and variance of the data.
- Estimating the class probabilities of a classification model.
- Estimating the parameters of a normal distribution.
- Estimating the weight parameters of a neural network.
8.1 Point Estimation
Point estimation is the use of a single specific numerical value to estimate an unknown parameter.
For example, using the sample mean to estimate the population mean:
Where:
- \mu: the unknown population mean.
- \hat\mu: the estimated mean.
Common estimation methods:
- Method of moments.
- Maximum likelihood estimation.
- Maximum a posteriori estimation.
- Least squares estimation.
8.2 Method of Moments
The method of moments is a method that estimates parameters by matching sample moments to theoretical moments.
For example, suppose a random variable follows a Bernoulli distribution:
The theoretical expectation is:
Match the sample mean to the theoretical expectation:
This yields the parameter estimate:
8.3 Maximum Likelihood Estimation
Maximum likelihood estimation (MLE) is a very important parameter estimation method in machine learning.
Core idea:
Choose a set of parameters that makes the already observed data most likely to occur under those parameters.
Suppose the observed data are:
The model is controlled by the parameter \theta.
If the samples are independent and identically distributed, the likelihood function is:
Maximum likelihood estimation:
Because multiplying multiple probabilities can easily cause numerical underflow, practical computations usually use the log-likelihood:
Finally:
The logarithmic function is monotonically increasing, so maximizing the likelihood and maximizing the log-likelihood yield the same optimal parameters.
8.4 Maximum Likelihood Estimation Example: Estimating the Probability of Heads
Suppose a coin is tossed 10 times in a row, and the following is observed:
- Heads: 7 times.
- Tails: 3 times.
Assume the probability of heads is p.
The likelihood function of the observed data:
Log-likelihood:
Take the derivative:
Set the derivative to zero:
This gives:
Therefore, the maximum likelihood estimate of the probability of heads is 0.7.
8.5 Maximum A Posteriori Estimation
Maximum A Posteriori (MAP) estimation considers not only the observed data but also introduces a prior distribution over the parameters.
According to Bayes' formula:
Maximum a posteriori estimation:
Taking logarithms:
Compared with MLE, MAP has an additional prior term.
This means that MAP considers not only how strongly the data support the parameters but also whether the parameters themselves conform to the prior assumptions.
8.6 Differences Between MLE and MAP
| Comparison aspect | MLE | MAP |
|---|---|---|
| Name | Maximum Likelihood Estimation | Maximum A Posteriori Estimation |
| Optimization objective | Maximize the data likelihood | Maximize the parameter posterior probability |
| Uses a prior | Not explicitly used | Used |
| Core expression | P(D\mid\theta) | P(\theta\mid D) |
| Typical application | Statistical parameter estimation | Bayesian estimation, regularization interpretation |
Under certain conditions, MAP can be transformed into an optimization problem with a regularization term.
For example, a Gaussian prior usually corresponds to L2 regularization, and a Laplace prior usually corresponds to L1 regularization.
However, the specific equivalence relationship depends on the definitions of the parameter prior, the data likelihood, and the optimization objective.
9. Probability Theory and Machine Learning: The Statistical Meaning Behind Loss Functions
Probability theory is not only used for model prediction; it also directly affects model training objectives.
9.1 Probabilistic Modeling in Machine Learning
Suppose the training data are:
The model aims to learn a conditional probability distribution:
Where:
- x: input data.
- y: target outcome.
- \theta: model parameters.
One of the goals of model training is to make the true observed labels have high probability under the model's predictive distribution.