1. Why Do We Need Recurrent Neural Networks?
In the earlier discussion of CNNs, we focused mainly on feature extraction from image data.
Images have a two-dimensional spatial structure, whereas data such as text, speech, and time series exhibit clear temporal or sequential relationships.
For example:
- Text: a sentence is made up of multiple Tokens arranged in order.
- Speech: audio consists of continuous sample points or acoustic feature frames.
- Time series: stock prices and sensor readings change in temporal order.
- Video: a video is made up of consecutive video frames.
This kind of data has an important characteristic: the information at the current moment is often related to the information at previous moments.
For example:
The weather is nice today, I'm planning to go out...
To understand what follows, the model needs to take the preceding context into account.
With an ordinary feedforward neural network, each input is usually computed independently, and the model itself has no inherent recurrent state for retaining previous information.
CNNs can extract local context, and context over a certain range, by enlarging the receptive field, but for tasks that require continuously maintained sequence state, the traditional recurrent structure offers another way of modeling.
RNN (Recurrent Neural Network) is precisely a neural network architecture designed for this kind of sequential data.
Through recurrent connections, it passes the hidden state from the previous moment to the current moment, enabling the model to retain historical information while processing a sequence.
2. What Is an RNN?
2.1 The Basic Idea of RNNs
The core idea of an RNN is: when processing the current input, it also uses the hidden state from the previous moment.
An ordinary feedforward neural network:
Each computation depends mainly on the current input.
An RNN, by contrast, introduces a hidden state:
Where:
- x_t: the input at the current moment.
- h_{t-1}: the hidden state at the previous moment.
- h_t: the hidden state at the current moment.
The hidden state can be understood as an internal representation of the historical sequence information maintained by the model.
Each time it processes a new input, the model combines the current input and the historical state to compute a new hidden state.
2.2 The Structure of an RNN
An RNN is usually viewed in two ways:
Recurrent structure:
The hidden state is fed back into the network through a recurrent connection and continues to take part in subsequent computations.
Time-unrolled structure (Unrolled RNN):
The recurrent network is unfolded along the time axis, showing the computation at each time step.
After unfolding, we can see that the same RNN cell is reused at different time steps.
flowchart LR
A["Input x₁"] --> B["RNN Cell"]
B --> C["Hidden state h₁"]
C --> D["RNN Cell"]
E["Input x₂"] --> D
D --> F["Hidden state h₂"]
F --> G["RNN Cell"]
H["Input x₃"] --> G
G --> I["Hidden state h₃"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
The RNN cells at different time steps in the figure are not separate, independent networks; they share the same set of parameters.
This is one of the important characteristics of RNNs: parameter sharing along the time dimension.
2.3 The Mathematical Formulation of an RNN
The hidden state update formula for a classic RNN:
The output layer can be expressed as:
If the outputs need to be converted into class probabilities, Softmax can be applied on top:
Where:
- x_t: Current input vector.
- h_{t-1}: Hidden state at the previous time step.
- W_{xh}: Weight matrix from the input to the hidden layer.
- W_{hh}: Recurrent weight matrix from the hidden state to the hidden state.
- b_h: Hidden layer bias.
- W_{hy}: Weight matrix from the hidden layer to the output layer.
- b_y: Output layer bias.
- \tanh: Nonlinear activation function.
Note: different time steps use the same W_{xh}, W_{hh}, and bias parameters, rather than creating a new set of parameters at each time step.
2.4 How Does an RNN Process a Sequence?
Suppose the input sequence is:
The RNN processes the input in temporal order:
Finally:
Therefore, the current hidden state not only contains information from the current input but may also contain information from multiple previous time steps.
The initial hidden state h_0 is usually initialized to zero, but it can also be provided by another network or by learnable parameters.
2.5 RNN State Is Not the Same as a Complete History Record
This is a common misconception when understanding RNNs.
Although the hidden state can carry historical information, it is usually a fixed-dimensional vector.
For example, the model may process hundreds of tokens, but the hidden state has only 256 dimensions.
Therefore, an RNN needs to compress historical information into a finite-dimensional state representation.
As the sequence grows, early information may gradually be lost.
This is also an important reason why traditional RNNs struggle to learn long-term dependencies.
3. RNN Input and Output Structures
RNNs can build different types of task architectures based on the relationship between input and output sequences.
3.1 One-to-One
One input corresponds to one output.
For example, an ordinary image classification task.
This is not the most typical application pattern for RNNs.
3.2 One-to-Many
One input corresponds to an output sequence.
For example:
- Image caption generation.
- Conditional text generation.
- Music sequence generation.
The model extracts an initial context from one input and then generates multiple outputs step by step.
3.3 Many-to-One
Multiple inputs correspond to one output.
For example:
- Sentence sentiment classification.
- Time series classification.
- Speech emotion recognition.
The model reads the entire sequence and uses the final hidden state or a pooled state for classification.
3.4 Many-to-Many
Multiple inputs correspond to multiple outputs.
For example:
- Sequence labeling.
- Part-of-speech tagging.
- Per-time-step prediction.
- Some machine translation architectures.
3.5 Four Typical Structures
| Structure | Input | Output | Typical Application |
|---|---|---|---|
| One-to-One | Single input | Single output | Ordinary classification |
| One-to-Many | Single input | Output sequence | Image captioning |
| Many-to-One | Input sequence | Single output | Sentiment analysis |
| Many-to-Many | Input sequence | Output sequence | Sequence labeling, sequence transformation |
4. The Core Problem of RNNs: Vanishing and Exploding Gradients
Traditional RNNs can process sequences, but they have obvious difficulty learning long-term dependencies.
The most important problems are vanishing gradients (Vanishing Gradient) and exploding gradients (Exploding Gradient).
4.1 Why Is Backpropagation Needed?
RNN training still relies on gradient descent and backpropagation.
Because the network is unfolded along the time dimension, training requires computing the gradients of the loss with respect to the parameters at different time steps.
This training method is called:
BPTT (Backpropagation Through Time).
Forward propagation computes hidden states in temporal order, while backpropagation propagates gradients backward from later time steps.
flowchart TB
A["Input sequence"] --> B["Forward propagation in temporal order"]
B --> C["Generate outputs at each time step"]
C --> D["Compute sequence loss"]
D --> E["Propagate gradients from back to front"]
E --> F["Accumulate gradients of shared parameters"]
F --> G["Optimizer updates parameters"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
4.2 What Is the Vanishing Gradient?
The hidden state of an RNN is updated recursively:
When computing the gradient for an earlier time step, the chain rule must be applied through multiple time steps.
A simplified representation:
If the norm of the derivative at every step is less than 1, then after repeated multiplication the gradient may gradually approach zero.
For example:
The longer the time span, the more easily the gradient decays.
This means that the parameter update signal from earlier time steps may be very weak.
Ultimately, the model struggles to learn dependencies between inputs that are far apart.
4.3 What Is the Exploding Gradient?
If the gradient factors being multiplied together are greater than 1, the gradient may grow rapidly as the number of time steps increases.
For example:
An excessively large gradient can lead to:
- Abnormally large parameter updates.
- A sudden increase in Loss.
- An unstable training process.
- Numerical overflow, producing Inf or NaN.
4.4 Vanishing Gradient vs. Exploding Gradient
| Aspect | Vanishing Gradient | Exploding Gradient |
|---|---|---|
| Gradient behavior | Approaches zero | Grows rapidly |
| Parameter updates | Too small | Too large |
| Training behavior | Long-term dependencies are hard to learn | Training is unstable |
| Common cause | Repeated derivative factors smaller than 1 | Repeated derivative factors greater than 1 |
| Typical remedies | LSTM, GRU, proper initialization, etc. | Gradient clipping, initialization, learning rate adjustment |
4.5 Gradient Clipping
Gradient clipping is mainly used to mitigate exploding gradients.
When the gradient norm exceeds a threshold, the gradients are rescaled.
For example, clipping by global norm:
Where:
- g: the original gradient.
- g': the clipped gradient.
- \tau: the maximum gradient norm.
PyTorch example:
import torch.nn as nn
loss.backward()
nn.utils.clip_grad_norm_(
model.parameters(),
max_norm=1.0
)
optimizer.step()
Gradient clipping mainly limits excessively large gradients; it cannot fundamentally solve the vanishing gradient problem.
V. LSTM: Long Short-Term Memory Networks
5.1 Why Do We Need LSTM?
In a traditional RNN, the hidden state must repeatedly pass through nonlinear transformations.
In long sequences, early information may gradually decay, making it difficult for the model to learn long-term dependencies.
LSTM (Long Short-Term Memory) improves the ability to preserve and update long-term information by introducing dedicated memory cells and gating mechanisms.
Its most important design elements are:
- Cell state.
- Hidden state.
- Forget gate.
- Input gate.
- Output gate.
LSTM maintains not only a hidden state but also an additional cell state pathway for storing and transmitting information.
5.2 Overall Architecture of LSTM
At each time step, the LSTM receives:
- The current input x_t.
- The hidden state from the previous time step h_{t-1}.
- The cell state from the previous time step c_{t-1}.
After the gating computation, it produces:
- The current cell state c_t.
- The current hidden state h_t.
5.3 Forget Gate
The forget gate determines which information in the cell state from the previous time step should be kept and which should be weakened.
The formula is:
Where:
- f_t: the forget gate output.
- \sigma: the sigmoid function.
- W_f: the forget gate weights.
- b_f: the forget gate bias.
The sigmoid output range is:
Therefore:
- Close to 0: the corresponding information is strongly suppressed.
- Close to 1: the corresponding information is mostly retained.
- In between: the corresponding information is partially retained.
The forget gate does not simply delete the entire historical state; instead, it adjusts different dimensions of the cell state separately.
5.4 Input Gate
The input gate determines which information from the current input can be written into the cell state.
The input gate consists of two parts.
Part one: determining the write ratio.
Part two: generating the candidate memory content.
Where:
- i_t: the input gate.
- \tilde c_t: the candidate cell state.
The input gate controls the proportion of candidate information that enters the cell state.
5.5 Cell State Update
Combining the forget gate and the input gate, the LSTM updates the cell state:
Where:
- f_t\odot c_{t-1}: retains the historical memory.
- i_t\odot\tilde c_t: writes in the current candidate information.
- \odot: element-wise multiplication.
This is one of the most important formulas in the LSTM.
Unlike a traditional RNN, the cell state does not simply pass through a single nonlinear transformation; instead, it combines historical information and new information along an additive path.
When the forget gate is close to 1 and the input gate is close to 0, the cell state can preserve its original content to a large extent.
This relatively direct state propagation path helps improve gradient flow when learning long-term dependencies.
5.6 Output Gate
The output gate controls which information in the current cell state can be exposed to the hidden state.
The formula is:
The current hidden state is:
Where:
- o_t: the output gate.
- c_t: the current cell state.
- h_t: the current hidden state.
The cell state is responsible for relatively long-term information storage, while the hidden state serves as the state representation exposed to the outside at the current time step.
5.7 The Complete LSTM Computation Flow
flowchart TB
A["Input xₜ and historical state"] --> B["Calculate forget gate fₜ"]
A --> C["Calculate input gate iₜ"]
A --> D["Generate candidate memory c̃ₜ"]
A --> E["Calculate output gate oₜ"]
F["Previous cell state cₜ₋₁"] --> G["Update cell state"]
B --> G
C --> G
D --> G
G --> H["Obtain current cell state cₜ"]
H --> I["Combine with output gate to generate hidden state hₜ"]
E --> I
I --> J["Output current state and pass to next time step"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#EAF3EC,stroke:#83A995,color:#315D49
style J fill:#DCEFE5,stroke:#4F8B70,color:#24563F
5.8 Why can LSTM alleviate vanishing gradients?
The key to LSTM is not simply increasing the number of network layers, but changing the state update structure.
Cell state update:
When considering only the historical state propagation path:
If the forget gate remains at a relatively large value over time, historical information and its gradients can propagate relatively stably along the cell state pathway.
But note:
- LSTM cannot guarantee that gradients will never vanish.
- The forget gate may also learn smaller values.
- Gradient decay can still occur along other computational paths.
- LSTM only significantly improves the long-term dependency learning capability of traditional RNNs.
6. GRU: Gated Recurrent Unit
6.1 Why do we still need GRU?
LSTM enhances memory capability through three gates and an independent cell state, but its structure is relatively complex.
GRU (Gated Recurrent Unit) further simplifies the gating mechanism.
GRU merges some gating functions and uses fewer gating structures to complete state updates.
Main components:
- Update gate.
- Reset gate.
- Hidden state.
Unlike LSTM, standard GRU does not maintain a separate cell state.
6.2 Overall structure of GRU
At each time step, GRU receives:
- Current input x_t.
- Hidden state from the previous time step h_{t-1}.
It outputs the current hidden state:
6.3 Update Gate
The update gate determines the combination ratio between the historical hidden state and the candidate state.
A common formula:
Where:
- z_t: update gate.
- W_z: update gate weight.
- b_z: bias.
The update gate can be understood as controlling the trade-off between new information and historical information.
6.4 Reset Gate
The reset gate determines the extent to which historical information participates in computing the candidate hidden state.
Where:
- r_t: reset gate.
- W_r: reset gate weight.
- b_r: bias.
When some dimensions of the reset gate are close to 0, the influence of the corresponding historical state in the candidate state computation is weakened.
6.5 Candidate Hidden State
GRU generates the candidate hidden state based on the current input and the historical state adjusted by the reset gate:
The candidate state represents the content of a new state that may be formed at the current time step.
6.6 Final Hidden State Update
A common GRU parameterization form:
Where:
- h_{t-1}: the historical hidden state.
- \tilde h_t: the candidate hidden state.
- z_t: the update gate.
When the update gate is close to 0, more of the historical state is retained.
When the update gate is close to 1, more of the candidate state is adopted.
Note that some papers and frameworks use the opposite gating definition, i.e., using z_t to represent the proportion of the historical state retained. Therefore, when reading formulas, you must judge based on the specific implementation and cannot infer the direction of a gate's effect solely from its name.
6.7 Complete GRU Computation Flow
flowchart TB
A["Current input xₜ and historical state hₜ₋₁"] --> B["Compute reset gate rₜ"]
A --> C["Compute update gate zₜ"]
B --> D["Generate candidate hidden state h̃ₜ"]
A --> D
D --> E["Combine update gate with historical state"]
C --> E
A --> F["Update state computation"]
E --> G["Obtain current hidden state hₜ"]
F --> G
G --> H["Pass to the next time step"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style H fill:#E5F0E9,stroke:#83A995,color:#315D49
VII. Core Differences Between RNN, LSTM, and GRU
All three belong to recurrent neural networks, but their state update mechanisms differ.
7.1 Structural Comparison
| Comparison Dimension | RNN | LSTM | GRU |
|---|---|---|---|
| Hidden state | Yes | Yes | Yes |
| Independent cell state | No | Yes | No |
| Gating mechanism | None | Forget gate, input gate, output gate | Update gate, reset gate |
| Long-term dependency capability | Relatively weak | Relatively strong | Relatively strong |
| Structural complexity | Relatively low | Relatively high | Moderate |
| Parameter count | Usually fewer | Usually more | Usually fewer than LSTM with the same hidden dimension |
| Training efficiency | Simple structure | More gating computations | Usually relatively simple |
The specific parameter count also depends on the input dimension, hidden dimension, bias settings, and framework implementation.
7.2 State Transfer Methods of the Three
RNN:
The hidden state is updated through an ordinary nonlinear transformation.
LSTM:
Memory is managed through an independent cell state and a gating mechanism.
GRU:
The hidden state is updated directly through the update gate and reset gate.
7.3 How to Choose?
Choose a plain RNN:
- The sequence is relatively short.
- The task is simple.
- There are high requirements on model structure and computational cost.
- It is used as a basic recurrent network structure for research.
Choose LSTM:
- Dependencies over relatively long time spans need to be modeled.
- Historical information is relatively important in the task.
- A relatively complex gating structure is acceptable.
Choose GRU:
- You want to use a gated recurrent structure.
- You want to control model complexity.
- You want to strike a balance between performance and computational cost.
It cannot simply be assumed that LSTM is necessarily better than GRU, or that GRU is necessarily faster than LSTM. Actual performance also depends on sequence length, hidden dimension, hardware, framework implementation, and training configuration.
VIII. Bidirectional Structures of RNN: BiRNN, BiLSTM, and BiGRU
8.1 Why Are Bidirectional RNNs Needed?
A plain RNN processes the sequence in a front-to-back order.
When processing the current Token, only the current and preceding context can be used directly.
However, in many tasks, later content also helps in understanding the current content.
For example:
I deposited the money in the bank.
To understand the meaning of a word, the model may need to take the following content in the sentence into account.
A bidirectional RNN uses recurrent networks in both directions to learn forward and backward context simultaneously.
8.2 Bidirectional Structure
A bidirectional RNN consists of:
- Forward network: processes from the beginning of the sequence to the end.
- Backward network: processes from the end of the sequence to the beginning.
The hidden states from the two directions can be concatenated:
where:
- \overrightarrow h_t: forward hidden state.
- \overleftarrow h_t: backward hidden state.
- [\,;\,]: vector concatenation.
Bidirectional structures can be applied to:
- BiRNN.
- BiLSTM.
- BiGRU.
8.3 Limitations of Bidirectional RNNs
A bidirectional network needs access to the complete sequence in order to compute the backward states.
Therefore, it is suitable for:
- Text classification.
- Named entity recognition.
- Offline speech recognition.
- Annotation tasks in which the complete input sequence is known.
However, for strict real-time prediction and autoregressive generation tasks, a bidirectional structure that contains future information cannot be used directly; otherwise, future information leakage may occur.
9. Sequence-to-Sequence Models: Seq2Seq
9.1 What Is Seq2Seq?
Seq2Seq (Sequence-to-Sequence) is a type of model architecture that maps an input sequence to an output sequence.
For example, machine translation:
Input:
I love machine learning.
Output:
I like machine learning.
The input and output sequence lengths are not necessarily the same.
Early Seq2Seq models usually consist of two parts: an Encoder and a Decoder.
9.2 Encoder
The encoder reads the input sequence and converts it into a hidden state representation.
Traditional architectures usually use RNN, LSTM, or GRU.
Input sequence:
The encoder computes the hidden states step by step:
9.3 Decoder
The decoder generates the output sequence step by step based on the context representation provided by the encoder.
The result generated at each step affects the prediction at the next step.
9.4 Seq2Seq Architecture
flowchart TB
A["Input sequence"] --> B["Encoder"]
B --> C["Context representation"]
C --> D["Decoder"]
D --> E["Generate the first Token"]
E --> F["Generate subsequent Tokens"]
F --> G["Output sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F