Back to series
· 10 min read

Linear Regression and Logistic Regression: Two Classic Predictive Models in Machine Learning


Linear Regression and Logistic Regression are the two most fundamental and classic types of supervised learning algorithms in machine learning.

The two have similar names: both take a linear function as their core modeling foundation, and both can learn model parameters by optimizing a loss function. But there are clear differences in the problems they solve, the form of their output, their mathematical principles, and their optimization objectives.

  • Linear Regression: mainly solves regression problems, predicting continuous numeric values.
  • Logistic Regression: mainly solves classification problems, predicting the probability that a sample belongs to a particular class.

Understanding these two models not only lets you grasp the basic methods of traditional machine learning, but also helps you further understand core concepts such as loss functions, parameter estimation, probabilistic modeling, gradient optimization, and classification decisions.


I. Basic Concepts of Linear Regression and Logistic Regression

1. Linear Regression

Linear regression is a supervised learning algorithm used to predict continuous numeric values.

It establishes a linear relationship between input features and the target variable, uses known samples to learn the model parameters, and then predicts the target value from the input features.

For example:

  • Predicting house prices from house area.
  • Predicting sales from advertising spend.
  • Predicting salary from years of work experience.
  • Predicting energy consumption from temperature and humidity.
  • Predicting business growth trends from historical metrics.

The core assumption of linear regression is that the target variable can be approximately represented by a linear combination of the input features.

For a single-feature model:

\hat{y}=w x+b

Where:

  • x: the input feature.
  • \hat{y}: the model's predicted value.
  • w: the weight, also called the slope.
  • b: the bias, also called the intercept.

For a multi-feature model:

\hat{y}=w_1x_1+w_2x_2+\cdots+w_nx_n+b

The model learns the weight corresponding to each feature from the training data.

2. Logistic Regression

Logistic regression is a supervised learning algorithm used for classification tasks.

It uses a linear model to compute the combined result of the input features, then applies the Sigmoid function to map that result to between 0 and 1, as the predicted probability that the sample belongs to a particular class.

For example:

  • Determining whether an email is spam.
  • Determining whether a user will churn.
  • Determining whether a transaction carries fraud risk.
  • Determining whether a user will click on an ad.
  • Determining whether a business opportunity will convert.

The core form of logistic regression is:

z=w^Tx+b
P(y=1|x)=\sigma(z)

Where:

  • z: the output of the linear model.
  • w: the model weights.
  • x: the input feature vector.
  • b: the bias.
  • \sigma: the Sigmoid function.
  • P(y=1|x): the predicted probability that the sample belongs to the positive class.

The Sigmoid function is defined as:

\sigma(z)=\frac{1}{1+e^{-z}}

No matter how large or small the linear model's output is, Sigmoid compresses the result into the open interval (0,1).

3. The Core Differences Between the Two

Aspect of comparison Linear Regression Logistic Regression
English name Linear Regression Logistic Regression
Learning type Supervised learning Supervised learning
Task type Regression Classification
Main output Continuous values Class probability
Core function Linear function Linear function + Sigmoid
Common loss function Mean squared error (MSE) Cross-entropy loss
Common optimization methods Least squares, gradient descent Gradient descent and its variants
Typical applications House price prediction, sales forecasting Spam detection, churn prediction

Note:

Although the name of logistic regression contains "regression," it solves classification problems. Its name comes from the Logistic Regression model form in statistics, not because it directly predicts continuous values.


II. The Mathematical Principles of Linear Regression

1. Building a Linear Model

Suppose there is a house price prediction task.

The only input feature is house area:

x=\text{house area}

The prediction target is:

y=\text{house price}

We can build a simple linear model:

\hat{y}=wx+b

For example:

\hat{y}=2x+50

Here it is assumed that:

  • For every one-unit increase in house area, the predicted price increases by 2 units.
  • When the area is 0, the model predicts a price of 50 units.

This is only an example model; the actual parameters need to be learned from training data.

2. Multiple Linear Regression

In real-world business, the target variable is usually influenced by multiple factors.

For example, house price may be related to the following factors:

  • House area.
  • Number of rooms.
  • Age of the house.
  • Distance from the city center.
  • Surrounding amenities.

In this case, a multiple linear regression model can be built:

\hat{y}=w_1x_1+w_2x_2+\cdots+w_nx_n+b

In matrix form:

\hat{y}=Xw+b

Where:

  • X: input feature matrix.
  • w: model weight vector.
  • b: bias term.
  • \hat{y}: prediction result vector.

If the bias term is merged into the weights, the model can be written as:

\hat{y}=Xw

Here, a column of features all equal to 1 needs to be added to X.

3. Objective Function of Linear Regression

There is usually an error between the model's predictions and the true labels.

Define the residual:

e_i=y_i-\hat{y}_i

Where:

  • y_i: true value.
  • \hat{y}_i: predicted value.
  • e_i: prediction residual.

To measure the overall prediction error, the Mean Squared Error (MSE) is commonly used:

MSE=\frac{1}{N}\sum_{i=1}^{N}(y_i-\hat{y}_i)^2

Characteristics of MSE:

  • Errors are squared, preventing positive and negative errors from canceling each other out.
  • Larger errors receive a heavier penalty.
  • The smaller the value, the smaller the model's fitting error on the current data, generally speaking.

The training objective of linear regression is:

w^*=\arg\min_w MSE(w)

That is, find a set of parameters that minimizes the mean squared error between the predictions and the true values.

4. Ordinary Least Squares

Ordinary least squares is a classic method for solving the parameters of linear regression.

It estimates the model parameters by minimizing the sum of squared residuals:

J(w)=\sum_{i=1}^{N}(y_i-x_i^Tw)^2

Matrix form:

J(w)=(y-Xw)^T(y-Xw)

When conditions such as the matrix having full column rank are satisfied, the least squares solution can be written as:

w^*=(X^TX)^{-1}X^Ty

This is the normal equation of linear regression.

Note the following:

  • X^TX must be invertible in order to use the inverse matrix form above directly.
  • When the features are perfectly linearly dependent, the matrix may not be invertible.
  • In practical engineering, QR decomposition, SVD, or numerically stable linear algebra solvers are usually used rather than explicitly computing the matrix inverse.
  • When the number of samples and the feature dimensionality are large, solving directly can be computationally expensive.

5. Solving with Gradient Descent

Besides least squares, the linear regression model can also be optimized via gradient descent.

Define the loss function:

J(w,b)=\frac{1}{2N} \sum_{i=1}^{N} (\hat{y}_i-y_i)^2

Where:

\hat{y}_i=w^Tx_i+b

Take partial derivatives with respect to the parameters:

\frac{\partial J}{\partial w} = \frac{1}{N}X^T(\hat{y}-y)
\frac{\partial J}{\partial b} = \frac{1}{N}\sum_{i=1}^{N}(\hat{y}_i-y_i)

Parameter update:

w:=w-\alpha\frac{\partial J}{\partial w}
b:=b-\alpha\frac{\partial J}{\partial b}

Here \alpha is the learning rate.

Training process:

flowchart TB
    A["Initialize model parameters"] --> B["Input training samples"]
    B --> C["Compute linear prediction output"]
    C --> D["Compute mean squared error"]
    D --> E["Compute parameter gradients"]
    E --> F["Update parameters via gradient descent"]
    F --> G{"Has the stopping condition been met?"}
    G -->|"No"| B
    G -->|"Yes"| H["Obtain the trained linear model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

The MSE loss function of linear regression is a convex function with respect to the model parameters. In the absence of additional non-convex constraints, gradient descent can optimize toward the global minimum, but the actual convergence speed is still affected by factors such as feature scale, learning rate, and the condition number of the data.

6. Geometric Meaning of Linear Regression

From a geometric perspective:

  • Linear regression with a single feature: fit a straight line.
  • Linear regression with two features: fit a plane.
  • Linear regression with multiple features: fit a hyperplane in a high-dimensional space.

The model tries to find a hyperplane such that the sum of squared residuals from sample points to that hyperplane is as small as possible.

A distinction is needed here:

The least squares method minimizes the sum of squared residuals in the direction of the target variable, which is not equivalent to minimizing the perpendicular Euclidean distance from sample points to the line or hyperplane.


III. Mathematical Principles of Logistic Regression

1. Why Can't We Directly Use Linear Regression for Classification?

Suppose we need to determine whether an email is spam:

  • Normal email: label 0.
  • Spam email: label 1.

If we directly use linear regression:

\hat{y}=w^Tx+b

The model output might be:

  • -2.3
  • 0.4
  • 1.7
  • 5.2

These results do not fall within the valid range of a probability.

A probability must satisfy:

0\leq P(y=1|x)\leq 1

Therefore, the linear model output needs to be converted into a probability.

Logistic regression uses the Sigmoid function to perform this conversion.

2. Sigmoid Function

The Sigmoid function:

\sigma(z)=\frac{1}{1+e^{-z}}

Its output range is:

0<\sigma(z)<1

The Sigmoid function has the following characteristics:

  • When z=0, the output is 0.5.
  • As z approaches positive infinity, the output approaches 1.
  • As z approaches negative infinity, the output approaches 0.
  • The function is continuous and differentiable.
  • The function output can be interpreted as the predicted probability of the positive class.

The prediction process of logistic regression:

z=w^Tx+b
p=\sigma(z)

where:

p=P(y=1|x)

For a binary classification task:

P(y=0|x)=1-p

Therefore, the model outputs the probability distribution corresponding to the two classes.

3. Logit: From Probability to Linear Expression

Logistic regression can not only convert linear output into probability via the Sigmoid function, but also, conversely, help understand the relationship between probability and the linear model.

Define the odds:

Odds=\frac{p}{1-p}

Take the natural logarithm of the odds:

Logit(p)=\ln\frac{p}{1-p}

The core relationship of logistic regression is:

\ln\frac{p}{1-p}=w^Tx+b

This shows that:

Logistic regression assumes a linear relationship between the log-odds of the class probability and the input features, rather than directly assuming that probability is linearly related to the features.

This is also the key to the name and mathematical structure of logistic regression.

4. Decision Boundary of Logistic Regression

Logistic regression outputs probabilities, but actual classification usually requires converting probabilities into classes based on a threshold.

Assume the threshold is 0.5:

\hat{y}= \begin{cases} 1,&p\geq 0.5\\ 0,&p<0.5 \end{cases}

Because:

\sigma(0)=0.5

Therefore:

p\geq 0.5 \iff w^Tx+b\geq 0

The decision boundary is:

w^Tx+b=0

In a two-dimensional feature space, it is a straight line; in three-dimensional space, it is a plane; in higher-dimensional space, it is a hyperplane.

Therefore, the classification decision boundary of standard logistic regression is linear.

Note that the model's output probability and the final classification result are two different concepts:

  • Probability is the model's estimate of the likelihood of a class.
  • The threshold is the decision rule that converts probability into a class.

The threshold can be adjusted according to business needs and does not necessarily have to be 0.5.

5. Loss Function of Logistic Regression: Cross-Entropy

Linear regression usually uses MSE, but logistic regression usually uses Cross-Entropy Loss.

For a single binary classification sample:

L(y,p)=- [y\log p+(1-y)\log(1-p)]

Where:

  • y\in\{0,1\}: true label.
  • p: the probability that the model predicts the positive class.

When the true label is 1:

L=-\log p

When the true label is 0:

L=-\log(1-p)

Average loss over the entire dataset:

J(w,b)= -\frac{1}{N} \sum_{i=1}^{N} [y_i\log p_i+ (1-y_i)\log(1-p_i)]

Characteristics of cross-entropy loss:

  • When the prediction is correct and the probability is high, the loss is small.
  • When the prediction is wrong and confidence is high, the loss will be large.
  • It can directly measure the difference between the predicted probability distribution and the true label distribution.
  • It has a direct connection to maximum likelihood estimation for the Bernoulli distribution.

6. Maximum Likelihood Estimation

Logistic regression can be understood from a probability and statistics perspective as a Bernoulli probability model.

For a single sample:

P(y_i|x_i)= p_i^{y_i}(1-p_i)^{1-y_i}

Assuming the samples are conditionally independent, the likelihood function for the entire dataset is:

L(w,b)= \prod_{i=1}^{N} p_i^{y_i}(1-p_i)^{1-y_i}

For ease of computation, take the logarithm of the likelihood function:

\log L(w,b)= \sum_{i=1}^{N} [y_i\log p_i+ (1-y_i)\log(1-p_i)]

Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood:

J(w,b)=-\log L(w,b)

This is the probability and statistics foundation of the cross-entropy loss commonly used in logistic regression.

7. Gradient Optimization of Logistic Regression

Logistic regression usually uses gradient descent or other numerical optimization algorithms to learn parameters.

Define:

p=\sigma(Xw+b)

Compute the gradient of the cross-entropy loss:

\frac{\partial J}{\partial w} = \frac{1}{N}X^T(p-y)
\frac{\partial J}{\partial b} = \frac{1}{N}\sum_{i=1}^{N}(p_i-y_i)

Parameter update:

w:=w-\alpha\frac{\partial J}{\partial w}
b:=b-\alpha\frac{\partial J}{\partial b}

Training process of logistic regression:

flowchart TB
    A["Initialize model parameters"] --> B["Input training samples"]
    B --> C["Compute linear combination z"]
    C --> D["Convert to probability via Sigmoid"]
    D --> E["Compute cross-entropy loss"]
    E --> F["Compute parameter gradients"]
    F --> G["Optimizer updates parameters"]
    G --> H{"Is the stopping condition met?"}
    H -->|"No"| B
    H -->|"Yes"| I["Obtain trained logistic regression model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

For standard logistic regression, when using cross-entropy loss, the objective function is usually convex with respect to the parameters; therefore, there is no issue of non-global local minima arising from the objective function itself.

However, in special cases such as when the data is completely linearly separable, unregularized logistic regression may not have a finite maximum likelihood parameter solution, and the absolute values of the parameters may continue to increase during optimization.


IV. Core Mathematical Differences Between Linear Regression and Logistic Regression

Both use a linear combination:

z=w^Tx+b

But the subsequent processing differs.

1. Linear Regression

Use the result of the linear combination directly as the predicted value:

\hat{y}=w^Tx+b

The output has no fixed range and can be any real number.

2. Logistic Regression

First compute the linear combination, then transform it through Sigmoid:

p=\sigma(w^Tx+b)

The output is the probability of the positive class, with a range of (0,1).

3. Model Comparison

Comparison Item Linear Regression Logistic Regression
Model core Linear function Linear function
Output transformation None Sigmoid
Output meaning Continuous predicted value Class probability
Optimization objective Minimize squared error Maximize likelihood or minimize cross-entropy
Common tasks Numerical prediction Classification
Decision boundary Not applicable Linear hyperplane
Parameter interpretation Effect of feature changes on the predicted value Effect of feature changes on the log-odds ratio

4. Why Can't the Two Be Interchanged Arbitrarily?

Although the two share a similar linear structure, the statistical assumptions about the target variable differ.

Linear regression is typically used for continuous target variables, with squared error as a common optimization objective.

Logistic regression is used for binary classification; its output corresponds to the parameter of a Bernoulli distribution, and cross-entropy is used for probabilistic modeling.

If logistic regression were used for house price prediction, the output probability could not directly express the house price.

If linear regression were used directly for spam classification, the predicted value might fall outside the probability range.

The choice of model must match the type of target variable and the task objective.


V. Regularization: Preventing Model Overfitting

Both linear regression and logistic regression can suffer from overfitting due to too many features, strong feature correlation, or insufficient data.

Regularization limits model complexity by adding a parameter penalty term to the loss function.

1. L1 Regularization (Lasso)

L1 regularization:

J_{L1}(w)=J(w)+\lambda\sum_{j=1}^{n}|w_j|

Characteristics:

  • It can shrink some parameters to 0.
  • It can produce sparse models.
  • It has a certain feature selection effect.
  • When features are highly correlated, the specific selection results may be unstable.

2. L2 Regularization (Ridge)

L2 regularization:

J_{L2}(w)=J(w)+\lambda\sum_{j=1}^{n}w_j^2

Characteristics:

  • It penalizes large weights.
  • It typically shrinks parameters toward 0.
  • It generally does not produce many parameters that are exactly 0, unlike L1.
  • It can somewhat alleviate multicollinearity.

3. Regularization Strength

\lambda controls the regularization strength:

  • Smaller \lambda: weaker parameter constraint.
  • Larger \lambda: stronger parameter constraint.

If regularization is too strong, the model may underfit.

If regularization is too weak, the model may overfit.

The regularization strength usually needs to be selected using a validation set or cross-validation.

4. Elastic Net

Elastic Net combines both L1 and L2 regularization:

J(w)=J_{base}(w)+ \lambda_1\sum_j|w_j|+ \lambda_2\sum_jw_j^2

It balances sparsity and parameter shrinkage, making it suitable for scenarios with many features that are correlated with one another.

Note: In practical implementations, the regularization term usually does not penalize the bias term; different frameworks may have different scaling conventions for the loss function and regularization coefficient.


VI. Interpretation of Model Parameters and Feature Effects

Linear regression and logistic regression can not only make predictions but also have strong interpretability.

However, the two interpret their parameters differently.

1. Interpretation of Linear Regression Parameters

Model: