1. What Is AI Speech Recognition (ASR)?
ASR (Automatic Speech Recognition) is an artificial intelligence technology that automatically converts speech signals into text.
For example, the user says:
The weather is nice today; let's go for a walk in the park this afternoon.
The ASR system needs to recognize the corresponding words from the audio:
Input audio:
A digital audio signal containing human speech
Recognized text:
The weather is nice today; let's go for a walk in the park this afternoon.
The core task of ASR is to infer, from continuously changing sound signals, the linguistic content the speaker actually expresses.
It is not simply about converting sound into text; it must solve a series of complex problems:
- Which parts of the sound signal are speech?
- Which pronunciations correspond to each time segment?
- How do different pronunciations combine into syllables and words?
- How can homophones be determined from context?
- How should accents, noise, speaking rate, and pauses be handled?
- How are sentence boundaries and punctuation determined?
For example, in Chinese:
- 公事 (official business)
- 攻势 (offensive)
- 公式 (formula)
In some contexts they may have the same or similar pronunciation, and ASR must use context to determine the correct written expression.
1.1 Main Applications of ASR
| Application Area | Typical Scenario |
|---|---|
| Voice input | Voice input methods on phones and computers |
| Meeting transcription | Converting meeting recordings to text |
| Real-time subtitles | Video subtitles, live stream subtitles |
| Intelligent customer service | Telephone speech recognition |
| Voice assistant | Voice command recognition |
| Voice search | Searching for content by voice |
| Speech translation | Language translation after speech recognition |
| Speech analytics | Call quality inspection, customer service analysis |
1.2 Differences Between ASR and Other Speech Technologies
ASR is only one important component of the speech AI technology system.
| Technology | English Name | Core Task |
|---|---|---|
| ASR | Automatic Speech Recognition | Speech-to-text |
| TTS | Text-to-Speech | Text-to-speech |
| STT | Speech-to-Text | Speech-to-text, often used as another name for ASR |
| TTS Voice Cloning | Voice Cloning | Voice cloning |
| Speaker Recognition | Speaker Recognition | Speaker recognition |
| Speaker Diarization | Speaker Diarization | Determining who is speaking during different time segments |
| VAD | Voice Activity Detection | Detecting speech activity intervals |
| Speech Translation | Speech Translation | Speech translation |
Two concepts need to be distinguished:
- ASR focuses on what the speaker said.
- Speaker Diarization focuses on who is speaking during different time segments.
The two can be used together; for example, a meeting transcription system needs both to recognize words and to distinguish speakers.
2. Overall Workflow Architecture of ASR
A typical speech recognition system includes the following core stages:
- Audio capture.
- Audio preprocessing.
- Voice activity detection.
- Acoustic feature extraction.
- Acoustic model or speech encoder.
- Language modeling and decoding.
- Text post-processing.
flowchart TD
A["Input speech audio"] --> B["Audio preprocessing"]
B --> C["Voice activity detection VAD"]
C --> D["Audio feature extraction"]
D --> E["Acoustic modeling"]
E --> F["Language modeling and decoding"]
F --> G["Text post-processing"]
G --> H["Output recognized text"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
This is a conceptual pipeline. Modern end-to-end ASR models may unify acoustic modeling, language modeling, and part of the decoding process into a single neural network, no longer explicitly using the separate modules found in traditional systems.
III. Audio Signals: How Do Computers Represent Human Speech?
To understand ASR, you first need to understand how speech is digitized.
When a person speaks, vocal cord vibration, airflow, and resonance in the oral and nasal cavities together produce sound. A microphone converts the sound into an electrical signal, and an analog-to-digital converter (ADC) then converts the continuous signal into a sequence of digital samples.
3.1 Basic Representation of Sound
Sound is essentially a mechanical wave that varies over time.
Digital audio can be represented as:
where:
- x_n: the amplitude of the n-th sample.
- N: the number of samples.
For example, a 16kHz mono audio segment contains the following number of samples per second:
samples.
If the audio lasts 10 seconds, then the number of samples is:
This means the system needs to process 160,000 discrete sample values.
3.2 Sample Rate
The sample rate indicates how many audio samples are captured per second, and its unit is Hz.
Common sample rates:
| Sample Rate | Common Uses |
|---|---|
| 8kHz | Traditional narrowband telephone speech |
| 16kHz | Common speech recognition input |
| 22.05kHz | Some audio applications |
| 24kHz | Some speech generation and processing systems |
| 44.1kHz | CD audio |
| 48kHz | Video, professional audio |
The higher the sample rate, the larger the frequency range that can generally be represented, but the amount of data also increases.
According to the Nyquist sampling theorem:
where:
- f_s: the sample rate.
- f_{\max}: the highest signal frequency that needs to be preserved.
If the sample rate is insufficient, aliasing may occur, causing high-frequency signals to be incorrectly mapped into the low-frequency region.
Therefore, audio capture usually needs to be paired with an anti-aliasing low-pass filter.
Note: Changing the sample rate of an audio file is not simply a matter of modifying the sample rate number in the file header; it requires actual resampling.
3.3 Bit Depth
Bit depth indicates how many bits are used to represent the amplitude of each sample.
Common bit depths:
- 8-bit
- 16-bit
- 24-bit
- 32-bit Float
Taking 16-bit PCM as an example, each sample is represented by a 16-bit integer.
Its theoretical number of quantization levels is:
A higher bit depth generally provides higher quantization precision and dynamic range.
For speech recognition, both sample rate and bit depth affect the quality of the input audio, but higher values do not necessarily improve recognition accuracy. The model's training configuration, input requirements, and the actual noise environment are equally important.
3.4 Channels
Common audio formats include:
- Mono: single channel.
- Stereo: two channels.
- Multi-channel: multiple channels.
Stereo audio can preserve the spatial information of the left and right channels.
In speech recognition, a system may convert multi-channel audio to mono, or it may use the spatial information from multiple microphones to perform beamforming and enhance the target speaker's speech.
IV. Audio Preprocessing: Making Speech Signals Suitable for Model Processing
Audio from real-world environments is often far from ideal.
For example:
- Background noise.
- Air conditioner and fan noise.
- Multiple people speaking at the same time.
- The microphone is too far away.
- The volume is too low.
- Inconsistent audio sample rates.
- Distortion produced by the recording device.
If such audio is fed directly into the model, recognition accuracy may drop noticeably.
Therefore, ASR systems usually need to perform audio preprocessing.
4.1 Main Audio Preprocessing Operations
| Operation | Purpose |
|---|---|
| Resampling | Convert audio to the sample rate required by the model |
| Channel conversion | Unify the audio channel format |
| Volume normalization | Adjust the audio amplitude range |
| Noise reduction | Reduce background noise |
| Silence detection | Remove invalid silent segments |
| Speech enhancement | Improve speech signal quality |
| Framing | Split continuous audio into short time windows |
4.2 Pre-emphasis
Traditional speech processing systems often use pre-emphasis.
The high-frequency part of a speech signal usually has less energy than the low-frequency part, and pre-emphasis can boost the high-frequency components.
Its expression is:
where:
- x[n]: the current sample value.
- x[n-1]: the previous sample value.
- \alpha: the pre-emphasis coefficient, commonly close to 0.95.
Pre-emphasis was once widely used in traditional MFCC feature extraction pipelines.
Modern neural-network ASR does not necessarily require explicit pre-emphasis; whether to use it depends on the model's input design and training configuration.
4.3 Voice Activity Detection (VAD)
VAD (Voice Activity Detection) is used to determine whether a given time interval in the audio contains human speech.
For example:
Audio:
[silence] [speech] [speech] [silence] [speech] [silence]
VAD result:
[non-speech] [speech] [speech] [non-speech] [speech] [non-speech]
VAD can reduce the computational cost of processing invalid audio and helps the system determine speech segment boundaries.
Typical VAD processing:
flowchart TD
A["Input continuous audio"] --> B["Framing"]
B --> C["Extract energy and spectral features"]
C --> D["Speech activity classification"]
D --> E{"Contains speech?"}
E -->|"Yes"| F["Keep speech segments"]
E -->|"No"| G["Mark as non-speech"]
F --> H["Send to ASR"]
G --> I["Skip or keep timestamps"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style I fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
There are two issues to watch for with VAD:
- Missed detection: real speech is incorrectly judged as silence.
- False detection: noise is incorrectly judged as speech.
If VAD incorrectly deletes a speech segment, the downstream ASR model cannot recover the deleted information.
V. Acoustic Feature Extraction: How Do We Convert Sound into Model Features?
Audio is a continuous time signal, but neural networks typically require a feature tensor with a fixed structure as input.
Therefore, ASR usually needs to convert the raw waveform into acoustic features.
Common features include:
- Time-domain features.
- Frequency-domain features.
- Short-Time Fourier Transform (STFT).
- Mel Spectrogram.
- MFCC.
- Learned acoustic features.
5.1 Why is framing necessary?
Speech signals have pronounced time-varying characteristics.
If spectral analysis is performed on the entire audio segment at once, it is difficult to accurately describe how the sound changes over different time periods.
Therefore, traditional speech processing usually adopts a short-time analysis approach, splitting the audio into multiple short frames.
For example:
- Frame length: 25ms.
- Frame shift: 10ms.
- Sampling rate: 16kHz.
Each frame contains:
sample points.
Each step forward moves:
sample points.
Because the frame shift is smaller than the frame length, adjacent frames overlap.
flowchart TD
A["Continuous audio signal"] --> B["Set frame length and frame shift"]
B --> C["Split the audio into frames"]
C --> D["Apply windowing"]
D --> E["Frame-by-frame spectral analysis"]
E --> F["Generate time-frequency features"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
5.2 Short-Time Fourier Transform (STFT)
STFT (Short-Time Fourier Transform) is used to analyze the frequency components of an audio signal within different time windows.
Formula:
Where:
- m: time frame index.
- k: frequency index.
- N: FFT window length.
- H: frame shift length.
- w[n]: window function.
- x[n]: original audio signal.
The result of the STFT is a complex-valued matrix containing magnitude and phase information.
In speech recognition, the magnitude spectrum or power spectrum is usually computed further:
This yields the energy distribution of the audio along the time and frequency dimensions.
5.3 Mel Spectrogram
Mel Spectrogram is a common acoustic feature in speech recognition.
It is based on the Mel frequency scale, which maps ordinary frequencies to a frequency representation that better matches the characteristics of human auditory perception.
A common conversion formula between Mel frequency and ordinary frequency is:
Where:
- f: ordinary frequency, in Hz.
- m: Mel frequency.
Typical processing pipeline for Mel Spectrogram:
flowchart TD
A["Raw audio"] --> B["Framing and windowing"]
B --> C["STFT"]
C --> D["Power spectrum"]
D --> E["Mel filter bank"]
E --> F["Log compression"]
F --> G["Mel Spectrogram"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Mel Spectrogram can be viewed as a 2D feature map:
- Horizontal axis: time.
- Vertical axis: Mel frequency.
- Pixel value: energy in the corresponding frequency band.
Neural networks can directly process such 2D features.
5.4 MFCC Features
MFCC (Mel-Frequency Cepstral Coefficients) is a very classic acoustic feature in traditional ASR.
Its main processing steps are:
- Frame the audio.
- Compute the STFT.
- Compute the power spectrum.
- Apply a Mel filter bank.
- Take the logarithm of the Mel energies.
- Perform the discrete cosine transform (DCT).
- Select a subset of coefficients as MFCC features.
MFCC can be simplified as:
Where:
- E_m: energy of the m-th Mel filter.
- M: number of Mel filters.
- c_n: the n-th MFCC coefficient.
MFCC can compress acoustic information while retaining some spectral features relevant to speech recognition.
5.5 Difference Between MFCC and Mel Spectrogram
| Comparison dimension | MFCC | Mel Spectrogram |
|---|---|---|
| Representation | Cepstral coefficients | Mel spectral energy |
| Transformation | DCT after Mel filtering | Usually logarithm after Mel filtering |
| Feature dimension | Usually lower | Usually higher |
| Information retention | Further compressed | Retains more time-frequency detail |
| Traditional use | Traditional acoustic models | Neural network speech recognition |
Modern ASR increasingly uses Log-Mel Spectrogram directly or learns features from raw waveforms.
MFCC still has research and engineering value, but it is not a mandatory input feature for modern ASR.
6. Acoustic Model: How to Recognize Speech from Acoustic Features?
After acoustic feature extraction is complete, the system needs to determine what linguistic content these features correspond to.
This process is performed by an acoustic model (Acoustic Model) or a modern speech encoder.
Traditional ASR and modern end-to-end ASR differ significantly here.
6.1 Traditional Acoustic Models
Traditional ASR usually uses:
- GMM-HMM.
- DNN-HMM.
- CNN-HMM.
- LSTM-HMM.
Here, HMM stands for Hidden Markov Model.
HMM is used to model how speech states change over time.
Speech is not a static signal; it is made up of continuous pronunciation units.
For example:
Speech content:
Hello
Possible phoneme sequence:
n → i → h → au
Corresponding acoustic frames:
Frame 1 → Frame 2 → Frame 3 → ...
HMM describes the process by which pronunciation states change over time through state transition probabilities.
6.2 Hidden Markov Model (HMM)
HMM mainly consists of three parts:
- Hidden state (Hidden State).
- State transition probability (Transition Probability).
- Observation probability (Emission Probability).
Suppose:
represents the hidden state at the tth time step,
represents the acoustic feature observed at the tth time step.
The basic assumptions of HMM include:
State transition probability:
represents the dependence of the current state on the previous hidden state.
Observation probability:
represents the probability that the current acoustic observation is generated by the current hidden state.
The overall joint probability can be expressed as:
The core problem here is: given the acoustic observation sequence O, how can we infer the most likely hidden state sequence S.
HMM usually uses the Viterbi algorithm to find the optimal state path.
6.3 GMM-HMM
Traditional GMM-HMM uses a Gaussian Mixture Model to estimate acoustic observation probabilities.
A GMM can be represented as a weighted combination of multiple Gaussian distributions:
Where:
- K: number of Gaussian distributions.
- \pi_k: mixing weight.
- \mu_k: mean vector.
- \Sigma_k: covariance matrix.
GMM-HMM was once an important architecture in traditional speech recognition systems.
Its limitation is that acoustic features and model expressiveness are constrained by relatively strong assumptions, resulting in insufficient ability to model complex speech patterns.
6.4 DNN-HMM
Deep neural networks can replace GMM to estimate the probabilities of acoustic states.
Typical structure:
flowchart TD
A["Acoustic features"] --> B["DNN"]
B --> C["State probabilities"]
C --> D["HMM state transition"]
D --> E["Viterbi decoding"]
E --> F["Phoneme or state sequence"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
DNN-HMM combines the feature representation capability of deep learning with the temporal state modeling of HMM.
Compared with GMM-HMM, it usually has stronger acoustic modeling capability.
VII. Language Models: Why Does ASR Need to Understand Context?
Relying only on acoustic features sometimes cannot uniquely determine the recognition result.
For example:
- Employed.
- Other expressions in employment contexts.
- Other words with the same or similar pronunciation.
Chinese, in particular, has a large number of homophones and homophonic words.
Therefore, ASR needs to use a language model (Language Model, LM) to determine which text combinations are more reasonable.
7.1 Acoustic Probability and Language Probability
Traditional ASR usually formulates the recognition problem as:
Where:
- X: input audio.
- W: candidate text.
According to Bayes' theorem:
Since P(X) is the same for all candidate texts, therefore:
Where:
- P(X|W): the acoustic matching probability provided by the acoustic model.
- P(W): the text prior probability provided by the language model.
Intuitive understanding:
- The acoustic model determines what the sound is more likely to be.
- The language model determines which word combination better conforms to language patterns.
7.2 N-Gram Language Model
Traditional language models often use the N-Gram method.
For example, a Bigram model predicts the current word based on the previous word:
A Trigram model predicts the current word based on the previous two words:
N-Gram mainly relies on a limited-length context and is susceptible to the data sparsity problem.
7.3 Neural Network Language Models
Neural network language models can model more complex contextual relationships through architectures such as Embedding, RNN, and Transformer.
In modern ASR, language modeling may be directly integrated into the end-to-end network, or may participate in decoding as an independent component.
Note: Modern end-to-end ASR does not necessarily have an independent traditional language model. Acoustic information and language patterns may already have been integrated into the same network through joint training.
8. Differences Between Traditional ASR and End-to-End ASR
Traditional ASR usually consists of multiple independent modules.
End-to-end ASR attempts to directly learn the mapping from input audio to output text.
8.1 Traditional ASR Architecture
flowchart TD
A["Input audio"] --> B["Acoustic feature extraction"]
B --> C["Acoustic model"]
C --> D["Phoneme or state probabilities"]
D --> E["Pronunciation lexicon"]
E --> F["Language model"]
F --> G["Search decoding"]
G --> H["Output text"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
Traditional architectures usually require separate maintenance of:
- Acoustic model.
- Pronunciation lexicon.
- Language model.
- Decoding graph or search space.
The advantage is that module boundaries are clear and each can be optimized separately.
The disadvantage is that the system is complex, and there may be inconsistency between the training objective and the individual modules.
8.2 End-to-End ASR
End-to-End speech recognition (End-to-End ASR) directly learns:
Where:
- X: input audio or acoustic features.
- Y: target text sequence.
Typical architectures include:
- CTC.
- Attention Encoder-Decoder.
- RNN-T.
- Transformer ASR.
- Conformer ASR.
End-to-end architectures reduce dependence on traditional pronunciation lexicons and independent acoustic models.
flowchart TD
A["Input audio"] --> B["Audio feature extraction"]
B --> C["Neural network encoder"]
C --> D["Sequence representation"]
D --> E["End-to-end decoder"]
E --> F["Output text"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#EAF3EC,stroke:#83A995,color:#315D49
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
9. CTC: Solving the Mismatch Between Speech and Text Lengths
CTC (Connectionist Temporal Classification) is a classic training method in end-to-end speech recognition.
It solves a core problem:
The number of input audio frames and the number of output text tokens are usually not the same, and the training data does not provide precise annotations of which character each audio frame corresponds to.
For example:
The input audio may be encoded into 100 frames, while the target text has only 8 characters.
It would be very difficult to require the model to directly establish a fixed correspondence between each frame and one character.
CTC solves the alignment problem by introducing a Blank label and allowing multiple time steps to map to the same output character.
9.1 Blank Label
CTC adds a special symbol to the target character set:
representing Blank.
For example, the model might output:
Time step:
1 2 3 4 5 6
Output:
Blank 你 你 Blank 好 好
After CTC Collapse:
- Merge consecutive repeated characters.
- Delete Blank.
The final result is:
你好
Note: CTC’s processing order is to first merge consecutive repeated symbols, then delete Blank. Therefore:
你 Blank 你
After Collapse, there are still two “你” characters.
9.2 CTC Probability Modeling
Suppose the input feature sequence is:
The model predicts a label distribution at each time step:
where \pi denotes a path containing characters and Blank.
The probability of a path is:
Since multiple different paths may map to the same target text Y, CTC sums the probabilities of these paths:
where:
- B: the CTC Collapse function.
- B^{-1}(Y): the set of all paths that can map to the target text Y.
The CTC loss is:
9.3 Characteristics of CTC
Advantages:
- It does not require precise frame-level character alignment annotations.
- The training process is relatively simple.
- Decoding can use greedy search or Beam Search.
- It is suitable for monotonic sequence mapping from speech to text.
Limitations:
- The conditional independence assumption limits direct modeling of dependencies between time steps.
- It has limited ability to model complex long-range language dependencies.
- Recognition results may require an additional language model for improvement.
10. Attention Encoder-Decoder: Generating Text Through Attention Mechanisms
Attention Encoder-Decoder is another important end-to-end ASR architecture.
It usually consists of:
- Encoder: converts audio into a sequence of contextual features.
- Decoder: generates text from historical outputs and encoder features.
10.1 Encoder
Input audio features:
After the encoder, we obtain:
where h_t represents the contextual feature at the corresponding time position.
The encoder can use:
- BiLSTM.
- CNN.
- Transformer.
- Conformer.
10.2 Decoder
The Decoder predicts the next token based on the tokens already generated and the Encoder output:
where:
- y_{<t}: the previously generated tokens.
- X: the input audio.
The Decoder can use the Attention mechanism to read information relevant to the current generation position from the encoder outputs.
The attention weights can be expressed as:
Context vector:
where:
- e_{t,i}: the matching score between the t-th output position and the i-th audio feature position.
- \alpha_{t,i}: attention weight.
- c_t: the audio context read at the current output position.
10.3 Generation Process
flowchart TD
A["Input audio features"] --> B["Encoder"]
B --> C["Context feature sequence"]
C --> D["Attention"]
E["Previous output tokens"] --> F["Decoder"]
D --> F
F --> G["Predict next token"]
G --> H{"Finished?"}
H -->|"No"| F
H -->|"Yes"| I["Output recognized text"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
The Attention Encoder-Decoder can directly model dependencies among output tokens, but a standard global Attention decoder may have problems such as high inference latency and unstable alignment.
11. RNN-T: An Architecture for Streaming Speech Recognition
RNN-T (Recurrent Neural Network Transducer) is an important end-to-end speech recognition architecture, especially suited to streaming recognition.
It consists of three core modules:
- Encoder: processes audio features.
- Prediction Network: models the linguistic context based on previously output tokens.
- Joint Network: fuses audio information and linguistic context to predict the next output.
11.1 Three Core Modules
Encoder
Converts the audio input into a time-series feature sequence:
Prediction Network
Generates a linguistic context representation based on historical output tokens:
Joint Network
Fuses the outputs of the two modules:
Then it uses Softmax to predict the output symbol distribution.
An important difference between RNN-T and CTC is that the prediction network of RNN-T can explicitly use historical output tokens.