Linear algebra is one of the most important mathematical foundations of artificial intelligence.
Feature representation in machine learning, parameter computation in neural networks, matrix operations in deep learning, and Embedding and Attention in large language models all rely on linear algebra.
From a technical perspective, an AI model essentially needs to complete a series of numerical computations:
- Convert data such as text, images, and audio into numerical representations.
- Use vectors to describe the features of data.
- Use matrices to represent model parameters and linear transformations.
- Use tensors to organize multi-dimensional data.
- Complete the forward propagation of neural networks through matrix operations.
- Complete model training through gradient computation and parameter updates.
Among these, vectors, matrices, and tensors constitute the most fundamental forms of data representation in modern AI systems.
Understanding them not only helps in understanding machine learning algorithms, but is also a prerequisite for studying Transformer, large language models, vector databases, and GPU computing in depth.
Starting from basic concepts, this article gradually introduces the mathematical definitions, computational methods, and geometric meanings of vectors, matrices, and tensors, as well as their practical applications in artificial intelligence.
1. What Is Linear Algebra?
1.1 Basic Definition
Linear algebra is the discipline that studies mathematical structures such as vector spaces, linear transformations, and systems of linear equations.
It mainly studies:
- Vectors and the relationships between vectors.
- Matrices and matrix operations.
- Systems of linear equations.
- Vector spaces and subspaces.
- Linear transformations.
- Eigenvalues and eigenvectors.
- Matrix decomposition.
In the field of computing, one of the most important values of linear algebra is that it provides a unified mathematical language for representing and computing large amounts of numerical data.
For example, an image can be represented as a matrix of pixels, a piece of text can be represented as an Embedding vector, and a neural network can be described by a large number of weight matrices.
1.2 Why Does AI Need Linear Algebra?
AI models usually cannot directly process text, images, and sounds as humans understand them; instead, they first need to convert this information into numbers.
Take text as an example:
人工智能正在改变软件开发
After tokenization with a Tokenizer, a set of Token IDs is obtained, which are then mapped to vectors through an Embedding layer.
Take images as an example:
An RGB image can be represented as a three-dimensional array, where each pixel contains values for the red, green, and blue channels.
After this data enters a neural network, it goes through a large number of matrix multiplications, vector operations, and other tensor computations.
Therefore, linear algebra runs through the data representation, model computation, and model training processes of AI.
1.3 Applications of Linear Algebra in AI
| Application Area | Role of Linear Algebra |
|---|---|
| Machine Learning | Representing sample features and model parameters |
| Neural Networks | Performing matrix computations of linear layers |
| Natural Language Processing | Representing Token Embeddings |
| Large Language Models | Performing Attention and Transformer computations |
| Computer Vision | Representing images and convolution operations |
| Recommender Systems | Representing users, items, and their latent vectors |
| Vector Databases | Storing, comparing, and retrieving vectors |
| Model Compression | Reducing parameter scale using matrix decomposition |
2. Scalars: The Most Basic Numerical Unit
Before understanding vectors and matrices, one must first understand scalars.
2.1 What Is a Scalar?
A scalar is a quantity that has only magnitude and no direction.
For example:
These are all scalars.
In AI, common scalars include:
- A model parameter.
- A loss value.
- A learning rate.
- A probability value.
- The value of a pixel channel.
For example:
represents the loss value computed by the current model.
2.2 Scalar Operations
Common scalar operations include:
- Addition.
- Subtraction.
- Multiplication.
- Division.
- Exponentiation.
- Taking roots.
For example:
Scalars are the basic numerical elements within vectors, matrices, and tensors.
3. Vectors: How to Represent a Set of Data?
3.1 What Is a Vector?
A vector is a mathematical object composed of multiple numbers.
For example:
This is a column vector containing three elements.
It can also be written as:
Where:
- \mathbf{x}: the vector.
- x_1,x_2,x_3: the individual components of the vector.
- T: transpose, indicating the conversion of a row vector into a column vector.
A vector containing n elements can be represented as:
Here, \mathbb{R}^n denotes the vector space made up of n real numbers.
3.2 Geometric Meaning of Vectors
In two-dimensional space, a vector can be represented as:
This can be understood as starting at the origin, moving 3 units horizontally, and then 2 units vertically.
In three-dimensional space:
it represents a direction and magnitude in three dimensions.
As the number of dimensions grows, vectors are no longer easy to display intuitively with geometric figures, but the mathematical definition stays the same.
AI frequently uses vectors with hundreds, thousands, or even higher dimensions to represent data.
3.3 Basic Properties of Vectors
Dimension
A vector has as many dimensions as it has components.
For example:
This is a four-dimensional vector.
Component
Each value in a vector is called a component.
For example:
Magnitude / Norm
The magnitude of a vector represents its length.
For the vector:
its L2 norm is:
For example:
then:
3.4 Basic Vector Operations
1. Vector addition
Two vectors with the same number of dimensions can be added:
Adding them:
Vector addition requires adding corresponding components.
2. Vector subtraction
Vector subtraction can be used to express the difference between two vectors.
3. Scalar multiplication
A scalar can be multiplied by a vector:
Scalar multiplication changes the length of the vector, and if the scalar is negative, it also reverses the vector's direction.
4. Vector dot product
The dot product is a very important operation in vector mathematics.
For two vectors:
their dot product is:
For example:
then:
The dot product can also be expressed as:
where \theta is the angle between the two vectors.
Therefore, the dot product is not just a way of computing a number; it also has geometric meaning.
3.5 Vector Similarity
In AI, vectors are often used to describe the features of text, users, products, or images.
When you need to determine whether two objects are similar, you can compare their corresponding vectors.
Common methods include:
1. Euclidean distance
The smaller the distance, the closer the two vectors are in Euclidean space.
2. Cosine Similarity
Cosine similarity primarily measures the degree to which two vectors align in direction.
Its value range is:
- Close to 1: similar direction.
- Close to 0: approximately orthogonal directions.
- Close to -1: opposite directions.
In text embedding retrieval, cosine similarity is a common similarity calculation method.
3. Vector Normalization
Normalization can convert a nonzero vector into a unit vector:
After normalization:
When both vectors have undergone L2 normalization:
This is also the mathematical basis for some vector retrieval systems to implement cosine similarity using the dot product.
3.6 Applications of Vectors in AI
| Application | Role of Vectors |
|---|---|
| Embedding | Convert text, images, etc. into numerical representations |
| RAG | Represent semantic features of documents and queries |
| Vector database | Store and retrieve high-dimensional vectors |
| Recommendation system | Represent interest features of users and items |
| Classification model | Represent features of input samples |
| Neural network | Represent hidden states of intermediate layers |
| Attention | Represent Query, Key, and Value |
For example, in a RAG system, the user question and knowledge base documents are each converted into vectors, and then relevant content is found through similarity calculation.
4. Matrices: How to Organize and Compute Large Numbers of Vectors?
4.1 What Is a Matrix?
A matrix is a two-dimensional mathematical structure composed of multiple numerical values arranged in rows and columns.
For example:
This matrix has:
- 2 rows.
- 3 columns.
Therefore, it is called a 2\times3 matrix.
General form:
It can be represented as:
Where:
- m: number of rows.
- n: number of columns.
- a_{ij}: the element at row i, column j.
4.2 Practical Significance of Matrices
Matrices can be used to organize large amounts of data with the same structure.
For example, a user dataset:
| User | Age | Monthly Income | Purchase Count |
|---|---|---|---|
| A | 25 | 8000 | 10 |
| B | 30 | 12000 | 15 |
| C | 28 | 10000 | 12 |
It can be represented as a matrix:
Where:
- Each row represents a user.
- Each column represents a feature.
- Each element represents the feature value of the corresponding user.
In machine learning, this kind of matrix is usually called a feature matrix.
Suppose there are m samples, and each sample has n features, then:
4.3 Basic Matrix Operations
1. Matrix Addition
Two matrices with the same number of rows and columns can be added.
Then:
Matrix addition adds the elements in corresponding positions.
2. Scalar Multiplication
A scalar can be multiplied by every element in a matrix:
3. Matrix Transpose
Matrix transpose (Transpose) swaps the rows and columns of a matrix.
For example:
After transposing:
If:
Then:
4. Matrix Multiplication
Matrix multiplication is one of the most important computations in deep learning.
Suppose:
Then:
We get:
The core rule of matrix multiplication is:
The number of columns in the left matrix must equal the number of rows in the right matrix.
For example:
Then:
Finally:
Note that matrix multiplication generally does not satisfy the commutative law:
Even if both AB and BA are defined, the results are not necessarily the same.
4.4 Dimension Rules for Matrix Multiplication
In matrix computation, dimension matching is one of the most error-prone areas.
For example:
Then:
But if:
Then AC cannot be computed directly, because the number of columns in A is 4, while the number of rows in C is 3.
You can remember it this way:
(m × n) · (n × p) = (m × p)
The two middle dimensions must be the same, and the final result keeps the two outer dimensions.
4.5 Special Matrices
Besides ordinary matrices, there are some special matrices that often appear in AI.
| Matrix Type | Definition | Characteristics |
|---|---|---|
| Square matrix | Number of rows equals number of columns | Determinants and eigenvalues can be discussed |
| Zero matrix | All entries are 0 | Zero element in addition |
| Identity matrix | Main diagonal is 1, and the rest are 0 | Identity element in matrix multiplication |
| Diagonal matrix | Off-main-diagonal entries are 0 | Simple computational structure |
| Symmetric matrix | A=A^T | Symmetric about the main diagonal |
| Orthogonal matrix | A^TA=I | Preserves vector lengths and angles |
| Sparse matrix | Most entries are 0 | Sparse computation can be used to reduce overhead |
4.6 Identity Matrix
An identity matrix is a special type of square matrix.
For example:
For a matrix A with compatible dimensions:
It is analogous to the number 1 in ordinary multiplication.
4.7 Inverse Matrix
For a square matrix A, if there exists a matrix A^{-1} such that:
then A^{-1} is called the inverse matrix of A.
However, not every square matrix has an inverse matrix.
An important condition for a matrix to be invertible is:
Inverse matrices play an important role in solving systems of linear equations, linear transformations, and some statistical models.
However, in practical machine learning engineering, one usually does not solve all problems by directly computing matrix inverses; instead, more stable and efficient methods for solving systems of linear equations are used.
4.8 Rank of a Matrix
The rank of a matrix is the maximum number of linearly independent rows or columns in the matrix.
For example:
The second row is twice the first row, so the two rows are linearly dependent.
The rank of this matrix is:
The rank of a matrix can be used to determine:
- whether the matrix contains redundant information.
- whether a system of linear equations has a unique solution.
- whether the data has a low-dimensional structure.
- whether the matrix can undergo low-rank decomposition.
In AI, low-rank matrix decomposition is an important foundation for model compression and parameter-efficient fine-tuning.
V. Linear Transformations: Why Can Matrices Change Data?
Matrices are not only used to store data; they can also represent linear transformations performed on vectors.
5.1 What Is a Linear Transformation?
A linear transformation is a mapping that satisfies linearity properties.
Suppose:
where:
- \mathbf{x}: the input vector.
- A: the transformation matrix.
- T(\mathbf{x}): the transformed vector.
A linear transformation must satisfy:
and:
where c is a scalar.
5.2 Geometric Interpretation of Linear Transformations
Matrices can perform different spatial transformations on vectors.
For example:
Scaling
When applied to a vector, it doubles the vector's length.
Rotation
A rotation matrix in the two-dimensional plane:
can rotate a vector around the origin by a specified angle.
Projection
Projection can map a vector onto a subspace to extract components in a specific direction.
5.3 Why Do Neural Networks Need Linear Transformations?
A fully connected layer in a neural network is essentially a linear transformation plus a bias, followed by a nonlinear activation function.
where:
- W: weight matrix.
- \mathbf{x}: input vector.
- \mathbf{b}: bias vector.
- \mathbf{y}: output vector.
For example:
Then:
This is the computation process of a simple linear layer in a neural network.
5.4 Why Do Neural Networks Need Non-Linear Activation Functions?
If a neural network contains only linear transformations, then stacking multiple linear layers is still equivalent to a single linear transformation.
For example:
Substituting gives:
The two matrices can be combined into a single matrix:
Therefore:
This means that no matter how many purely linear layers are stacked, the model as a whole remains a linear function.
To learn more complex non-linear relationships, neural networks need to introduce activation functions between layers.
For example, ReLU:
The complete computation can be expressed as:
This enables neural networks to represent more complex functional relationships.
VI. Tensors: How to Represent High-Dimensional Data?
6.1 What Is a Tensor?
A tensor is the unified generalization of scalars, vectors, and matrices to higher dimensions.
It can be understood in terms of dimensions:
| Data structure | Order | Example |
|---|---|---|
| Scalar | Order 0 | A single numeric value |
| Vector | Order 1 | One-dimensional array |
| Matrix | Order 2 | Two-dimensional array |
| 3D tensor | Order 3 | Three-dimensional array |
| High-dimensional tensor | Order 4 and above | Multi-dimensional data structure |
Here, order usually refers to the number of dimensions of the tensor, also often called rank or order.
Note that in different deep learning frameworks and mathematical contexts, terms such as Rank, Shape, and Dimension may be used differently:
- Tensor Rank: the order of the tensor, i.e., the number of axes.
- Shape: the size along each axis.
- Dimension: may refer to the number of axes or to a particular feature dimension; it needs to be understood in context.
6.2 First-Order Tensor: Vector
This is a first-order tensor.
It has only one axis, and its shape can be expressed as:
6.3 Second-Order Tensor: Matrix
This is a second-order tensor.
It has two axes, and its shape is:
6.4 Third-Order Tensor
Suppose a data structure contains 2 matrices, each with 3 rows and 4 columns.
It can be represented as:
Its shape is:
Such structures are very common in deep learning.
For example, the embeddings of a batch of text can be represented using a three-dimensional tensor:
Where:
- B: batch size.
- T: number of tokens per text.
- D: vector dimension of each token.
6.5 Fourth-Order Tensor: Image Batch
An RGB image can usually be represented as three-dimensional data:
Where:
- H: image height.
- W: image width.
- C: number of color channels.
If multiple images are input at once, a Batch dimension needs to be added:
This is a fourth-order tensor.
In different deep learning frameworks, the axis order of image tensors may differ.
For example:
- NHWC: Batch, Height, Width, Channels.
- NCHW: Batch, Channels, Height, Width.
Inconsistent axis order is a very common source of dimension errors in deep learning engineering.
6.6 What Is the Difference Between a Tensor and an Array?
In programming, NumPy arrays, PyTorch Tensors, and TensorFlow Tensors can all represent multidimensional numerical data.
However, a tensor in mathematics and an array in programming are not completely equivalent.
A tensor in mathematics has specific transformation rules; its components must follow the corresponding laws under coordinate transformations.
A Tensor in a deep learning framework, however, is usually a data object that supports multidimensional indexing, numerical operations, device storage, and automatic differentiation.
Therefore:
A Tensor in programming is an important data structure for implementing tensor computations, but one cannot simply equate all the engineering properties of multidimensional arrays with the complete mathematical definition of a tensor.
6.7 Applications of Tensors in AI
| Application | Common tensor structure |
|---|---|
| Image recognition | 4D image batch tensor |
| Video understanding | 5D or higher-dimensional video tensor |
| Natural language processing | 3D Token Embedding tensor |
| Transformer | Multi-head Attention tensor |
| Speech recognition | Audio feature tensor |
| Deep learning training | Input, parameter, and gradient tensors |
7. The Relationship Between Vectors, Matrices, and Tensors
Vectors, matrices, and tensors are not completely independent mathematical objects; they are data structures of different orders.
They can be understood in terms of dimensional hierarchy:
flowchart TB
A["Scalar<br/>0th order"]
B["Vector<br/>1st order"]
C["Matrix<br/>2nd order"]
D["Higher-order Tensor<br/>3rd order and above"]
A --> B
B --> C
C --> D
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#E5F0E9,stroke:#83A995,color:#315D49