Back to series
· 16 min read

AI Speech Synthesis (TTS): The Complete Technical Principles from Text to Natural Speech


I. What Is AI Speech Synthesis?

TTS (Text-to-Speech) is the technology of converting text into speech; its core task is to transform input text into an audio signal capable of linguistic expression.

For example, given the input:

Hello, welcome to the artificial intelligence speech synthesis system.

the system outputs a segment of speech containing the corresponding pronunciation, intonation, pauses, and prosody.

Traditional TTS relied mainly on hand-designed pronunciation rules, concatenation of speech segments, and statistical models. Modern TTS relies mainly on deep learning, using neural networks to learn the mapping between text and speech and thereby generate more natural, more coherent speech.

1. How TTS Differs from Related Speech Technologies

Technology English Name Core Task
Speech synthesis Text-to-Speech Converting text into speech
Speech recognition Automatic Speech Recognition Converting speech into text
Voice conversion Voice Conversion Changing voice timbre or speaker characteristics
Speaker recognition Speaker Recognition Determining who a speech sample comes from
Voice activity detection Voice Activity Detection Determining whether speech is present in audio
Speech enhancement Speech Enhancement Noise reduction and improvement of speech quality

Among these, ASR and TTS are tasks that run in opposite directions, yet their technical systems are not fully symmetrical.

ASR must infer linguistic content from a continuous acoustic signal, whereas TTS must generate speech that conforms to pronunciation rules and acoustic characteristics based on linguistic content.

2. Core Inputs and Outputs of TTS

Inputs:

  • Text content.
  • Language or dialect information.
  • Speaker ID or voice reference.
  • Control information such as emotion, tone, and speaking rate.
  • Audio generation parameters.

Outputs:

  • A digital audio waveform.
  • Audio attributes such as sample rate, bit depth, and number of channels.
  • Optional timestamps, phoneme alignments, and segmentation information.

Modern TTS does more than read text aloud; it also has to handle:

  • Polyphonic characters and context-dependent pronunciation.
  • Numbers, dates, abbreviations, and special symbols.
  • Sentence stress and pauses.
  • Speaking rate, pitch, and timbre.
  • Emotional expression and speaking style.
  • Pronunciation differences between languages.

II. The Overall Architecture of Modern TTS

A modern neural TTS system can generally be broken down into four core stages: text processing, text encoding, acoustic modeling, and the vocoder.

flowchart TD
    A["Input text"] --> B["Text normalization"]
    B --> C["Tokenization and grapheme-to-phoneme conversion"]
    C --> D["Text encoder"]
    D --> E["Acoustic model"]
    E --> F["Acoustic features"]
    F --> G["Neural vocoder"]
    G --> H["Generate audio waveform"]
    H --> I["Output speech"]

    J["Speaker information"] --> E
    K["Speaking rate and emotion control"] --> E

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49
    style J fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style K fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Different TTS architectures do not necessarily follow this module breakdown strictly.

For example:

  • Tacotron 2 performs acoustic modeling from text to Mel spectrograms, then generates waveforms through a vocoder such as WaveNet.
  • FastSpeech 2 uses explicit modeling of duration, pitch, and energy.
  • VITS combines ideas such as variational inference, adversarial training, and flow models within an end-to-end speech generation architecture.
  • Some newer speech models use discrete speech tokens, autoregressive generation, or diffusion models, and no longer strictly follow the traditional Mel spectrogram plus vocoder structure.

Understanding the traditional architecture is the foundation for understanding modern TTS.


III. Text Preprocessing: From Raw Text to a Pronounceable Representation

The goal of text preprocessing is to convert natural language text into a linguistic representation that the model can process reliably.

This stage is easily underestimated, but it directly affects whether the final pronunciation is correct.

1. Text Normalization

Natural language text contains a great many non-standard written forms.

For example:

Original text:
Today is November 19, 2026, and the temperature is 23.5℃.

Normalized text:
Today is November nineteenth, twenty twenty-six, and the temperature is twenty-three point five degrees Celsius.

Text normalization needs to handle:

Type Input example Conversion result
Number 123 one hundred twenty-three
Date 2026-11-19 November nineteenth, twenty twenty-six
Time 08:30 eight thirty
Decimal 3.14 three point one four
Percentage 80% eighty percent
Currency ¥100 one hundred yuan
Unit 10km ten kilometers
Abbreviation AI converted to letter pronunciations or corresponding names according to language rules

Note that the normalization result depends on context.

For example:

  • 2026 may be read as “two zero two six” or as “two thousand and twenty-six.”
  • 1/2 may represent a fraction, a date, or a ratio.
  • 12 may represent a quantity, an identifier, a month, or a time.

Therefore, text normalization cannot rely only on simple string replacement; it also needs to combine language rules and contextual judgment.

2. Word Segmentation and Tokenization

After normalization, text usually needs to undergo word segmentation or Tokenization.

For example:

Input:
Artificial intelligence is changing software development methods.

Tokenization:
Artificial intelligence / is / changing / software / development / methods

Different models may use:

  • Character-level Tokens.
  • Word-level Tokens.
  • Subword Tokens.
  • Phoneme Tokens.
  • Discrete speech Tokens.

Two concepts need to be distinguished:

A text Token is not equivalent to a phoneme.

A text Token is a discrete symbolic unit used by the model when processing text; a phoneme is the smallest phonetic unit in a language that can distinguish meaning.

Some models process characters or subwords directly, while others first convert text into a phoneme sequence.

3. Phoneme

Phonemes are an important foundation of speech modeling.

For example, the English word cat can be represented as three phonemes:

/k/ /æ/ /t/

Through phoneme representations, a model can more clearly know how each character or word should be pronounced.

Phoneme processing in Chinese usually involves initials, finals, and tones.

For example:

你好

你: n + i + third tone
好: h + ao + third tone

Practical systems may use pinyin, phoneme symbols, or specially designed linguistic representations.

The main functions of phonemes include:

  • Eliminating some ambiguities between spelling and pronunciation.
  • Providing clearer pronunciation units.
  • Supporting multilingual pronunciation modeling.
  • Improving the model’s ability to handle rare and complex words.

4. Grapheme and Phoneme

Two concepts are often involved in TTS:

  • Grapheme: a unit of writing or glyph.
  • Phoneme: a phoneme unit.

The two are not in a one-to-one correspondence.

For example, in English the same letter combination may correspond to different pronunciations:

read (present tense)
read (past tense)

They have the same spelling but different pronunciations.

Chinese also has polyphonic characters:

银行:yín háng
行走:xíng zǒu

If the model maps pronunciation based only on individual Chinese characters, it will be difficult to handle such cases correctly.

Therefore, modern TTS usually needs to use contextual information to determine pronunciation, rather than simply building a static character-to-phoneme mapping.


IV. Text Encoder

After text preprocessing is complete, the model needs to convert the discrete text representation into continuous vectors.

This is the main responsibility of the text encoder.

1. Embedding Layer

Suppose the input text is processed by the Tokenizer to obtain:

X=[x_1,x_2,\ldots,x_N]

Each Token corresponds to an integer ID.

The Embedding layer maps it into a continuous vector space:

E_i=Embedding(x_i)

Finally, the text vector sequence is obtained:

E=[E_1,E_2,\ldots,E_N]

Where:

  • N: number of input Tokens.
  • E_i: the vector representation of the ith Token.
  • d: Embedding dimension.

This can be understood as follows: the model no longer directly processes characters or Token IDs, but instead processes continuous vectors that can be learned through training.

2. Contextual Encoding

Embedding alone is not enough.

The model needs to understand the contextual relationships between Tokens.

For example:

I work at a bank.
I parked my bicycle by the river.

The same or similar words may have different meanings, and the corresponding pronunciation and language expression ultimately depend on context.

Text encoders commonly use:

  • CNN.
  • RNN.
  • LSTM.
  • Transformer.
  • Conformer and similar structures.

A Transformer encoder can model relationships between different positions in text through Self-Attention.

Its core computation is:

Attention(Q,K,V) = Softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

Where:

  • Q: Query.
  • K: Key.
  • V: Value.
  • d_k: Key dimension.

Attention enables the model to dynamically aggregate information from other Tokens based on the context of the current Token.

3. Output of the Text Encoder

The text encoder ultimately outputs a set of context vectors:

H=[h_1,h_2,\ldots,h_N]

These vectors contain text content, context, and linguistic structure information.

Next, the acoustic model needs to generate the corresponding speech features based on this information.


V. Acoustic Model

The acoustic model is one of the core components of TTS.

It is responsible for establishing the mapping relationship between text representations and speech acoustic features.

In traditional TTS, the acoustic model usually outputs a Mel spectrogram; modern models may also directly predict other acoustic representations, or even directly generate waveforms or discrete speech Tokens.

1. What Is a Mel Spectrogram?

A Mel Spectrogram, or Mel-frequency spectrogram, is a common time-frequency representation in speech processing.

The original audio is a waveform that changes over time:

x(t)

Modeling complex waveforms directly in the time domain usually requires processing a large number of sampling points.

Therefore, speech systems often first convert the waveform into time-frequency features.

A typical process:

flowchart TD
    A["Original audio waveform"] --> B["Framing and windowing"]
    B --> C["Short-time Fourier transform STFT"]
    C --> D["Power spectrum calculation"]
    D --> E["Mel filter bank"]
    E --> F["Logarithm"]
    F --> G["Log-Mel Spectrogram"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

The Mel frequency scale mimics how the human ear perceives different frequencies.

A common conversion formula is:

m(f)=2595\log_{10}\left(1+\frac{f}{700}\right)

Where:

  • f: actual frequency, in Hz.
  • m(f): Mel frequency.

The Mel spectrogram maps the frequency axis to the Mel scale and compresses spectral information through a filter bank.

It can reduce the dimensionality of the acoustic representation while preserving important speech features.

2. What Does the Acoustic Model Predict?

Using traditional Mel-spectrogram TTS as an example, the model's input is the text encoding vector:

H=[h_1,h_2,\ldots,h_N]

The output is a sequence of acoustic features:

M=[m_1,m_2,\ldots,m_T]

Where:

  • N: number of text Tokens.
  • T: number of audio frames.
  • m_t: the Mel feature of frame t.

There is an important issue here:

Text length and audio length are not the same.

For example, a sentence may contain only 20 text Tokens, but the generated Mel spectrogram may contain hundreds of time frames.

Therefore, the model must solve the temporal alignment problem between text and audio.

3. Duration Modeling

Each phoneme or text unit lasts for a different amount of time in speech.

For example:

  • A short consonant may last tens of milliseconds.
  • Vowels usually last longer.
  • A pause at the end of a sentence may last hundreds of milliseconds.

The task of duration prediction is to estimate how many audio frames each input unit should occupy.

Suppose the input phoneme sequence is:

P=[p_1,p_2,\ldots,p_N]

The corresponding predicted durations are:

D=[d_1,d_2,\ldots,d_N]

Then the total number of output frames is:

T=\sum_{i=1}^{N}d_i

This means duration prediction directly affects speech length, speaking rate, and rhythm.

4. Pitch Modeling

Pitch usually refers to the fundamental frequency F0, that is, the fundamental frequency produced by vocal cord vibration.

It is closely related to the perceived pitch of speech.

For example:

  • A high tone may exhibit a higher F0.
  • A deep voice usually has a lower F0.
  • Questions, emphasis, and emotional expression may be accompanied by pitch changes.

Note that:

F0 is not equal to all timbre information, nor can it alone determine a person's voice identity.

Timbre is also affected by factors such as vocal tract shape, formants, phonation mode, and spectral envelope.

5. Energy Modeling

Energy is used to describe the strength of the speech signal's energy.

It is related to volume, vocal effort, and some aspects of emotional expression.

For example:

  • Emphasized words usually have more pronounced energy changes.
  • Soft-spoken expression may have lower energy.
  • The stressed parts of a sentence may have more prominent energy.

In architectures such as FastSpeech 2, duration, pitch, and energy can serve as explicit prosodic control variables.


VI. Mainstream TTS Architectures and Core Models

The development of modern TTS has gone from concatenative synthesis and statistical parametric synthesis to neural end-to-end synthesis.

1. Concatenative TTS

Concatenative TTS relies on large amounts of pre-recorded speech segments.

The system looks up suitable speech units based on the input text, then concatenates multiple segments into complete speech.

Basic process:

flowchart TD
    A["Input text"] --> B["Text analysis"]
    B --> C["Phoneme sequence"]
    C --> D["Speech unit retrieval"]
    E["Pre-recorded speech corpus"] --> D
    D --> F["Speech segment selection"]
    F --> G["Segment concatenation"]
    G --> H["Output speech"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

Advantages:

  • Within the coverage of the speech corpus, it can generate fairly natural speech.
  • It does not require complex deep neural networks.
  • It can preserve the pronunciation details in real recordings.

Disadvantages:

  • The speech corpus requires a large amount of recording.
  • The junctions between concatenated speech segments may produce unnatural breaks.
  • Flexibility is limited.
  • It is difficult to freely change speaking style and emotion.

2. Statistical Parametric TTS

Statistical parametric TTS uses statistical models to learn the relationship between text and acoustic features.

Typical techniques include:

  • HMM: Hidden Markov Model.
  • GMM: Gaussian Mixture Model.
  • DNN: Deep Neural Network.

Traditional HMM TTS usually predicts acoustic parameters from text context, and then uses a vocoder to synthesize speech.

Advantages:

  • Compared with concatenative methods, the model structure is more flexible.
  • The form of dependence on the speech corpus changes.
  • Speaking rate and some prosodic features can be controlled through parameters.

Disadvantages:

  • The generated speech tends to be over-smoothed.
  • Naturalness is usually inferior to high-quality neural TTS.
  • Acoustic feature modeling and vocoder quality limit the final result.

3. Tacotron: End-to-End Acoustic Modeling

Tacotron is an important representative in the development of neural network TTS.

It adopts an Encoder-Decoder architecture to map text sequences to Mel spectrograms.

Core structure:

  • Text Encoder: encodes the text.
  • Attention: establishes alignment between text and speech frames.
  • Decoder: generates acoustic features step by step.

An important breakthrough of Tacotron is the use of neural networks to directly learn the mapping from text to speech features, reducing the large number of manually designed modules in traditional systems.

But it has some problems:

  • Attention alignment may be unstable.
  • Repeated pronunciation may occur.
  • Parts of the text may be skipped.
  • Autoregressive generation is relatively slow.

4. Tacotron 2

Tacotron 2 improves the acoustic model and speech generation structure on the basis of Tacotron.

Typical architecture:

flowchart TD
    A["Input text"] --> B["Character Embedding"]
    B --> C["Encoder"]
    C --> D["Attention"]
    D --> E["Autoregressive Decoder"]
    E --> F["Mel Spectrogram"]
    F --> G["WaveNet Vocoder"]
    G --> H["Speech waveform"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49

Where:

  • The Encoder converts text into contextual representations.
  • Attention determines which text information the current generation position attends to.
  • The Decoder autoregressively generates the Mel spectrogram.
  • WaveNet converts the Mel spectrogram into an audio waveform.

Tacotron 2 demonstrated that combining a neural acoustic model with a high-quality neural vocoder can produce highly natural speech.

5. FastSpeech: Non-Autoregressive Speech Synthesis

The Tacotron series uses an autoregressive Decoder to generate acoustic features frame by frame.

This causes generation speed to be limited by serial computation.

FastSpeech proposed a non-autoregressive TTS approach.

It introduces a Length Regulator, which expands the text encoding sequence according to predicted durations and then predicts acoustic features in parallel.

Core structure:

flowchart TD
    A["Input text"] --> B["Text Encoder"]
    B --> C["Phoneme or character representation"]
    C --> D["Duration Predictor"]
    D --> E["Length Regulator"]
    E --> F["Expanded sequence"]
    F --> G["Acoustic Decoder"]
    G --> H["Mel Spectrogram"]
    H --> I["Vocoder"]
    I --> J["Audio waveform"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#EAF3EC,stroke:#83A995,color:#315D49
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49

For example, given three phonemes:

P = [p1, p2, p3]

Predicted durations:

D = [2, 3, 1]

After Length Regulator expansion:

[p1, p1, p2, p2, p2, p3]

In this way, the Decoder can predict the acoustic features at each time position in parallel.

Advantages of FastSpeech:

  • Non-autoregressive generation, with faster inference speed.
  • More direct duration control.
  • It reduces the problem of unstable autoregressive Attention alignment.

However, non-autoregressive models also have their own challenges:

  • Duration prediction errors can affect pronunciation rhythm.
  • Parallel generation requires accurate length information.
  • Generation quality depends on training data and prosody modeling capability.

6. FastSpeech 2

FastSpeech 2 further introduces multiple types of explicit prosodic information:

  • Duration: phoneme duration.
  • Pitch: pitch.
  • Energy: energy.

Its core idea is to not only predict the acoustic features corresponding to the text, but also explicitly model important variables that affect the speech realization.

This makes it easier for the model to control speaking rate, pitch, and vocal intensity.

7. VITS: End-to-End Speech Synthesis

VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is a representative end-to-end TTS architecture.

It combines several techniques:

  • Variational inference.
  • Normalizing Flow.
  • Adversarial training.
  • Stochastic duration modeling.
  • Neural waveform generation.

Unlike the traditional two-stage structure of text-to-Mel-spectrogram followed by waveform generation, VITS can generate audio waveforms directly from text.

Its key characteristics are:

  • End-to-end training.
  • Ability to model stochastic variation in speech.
  • No strict dependence on separate Mel-spectrogram prediction and vocoder training pipelines.
  • Ability to express diversity in speech through stochastic latent variables.

VITS training and inference involve multiple interacting modules, and it should not be simplistically understood as an ordinary text-to-waveform mapping network.


VII. Neural Vocoder

The Mel spectrogram produced by the acoustic model is not the final audio.

A vocoder is also needed to convert the acoustic features into the actual waveform.

The vocoder is an important component of modern neural TTS.

1. What Is a Vocoder?

A Vocoder is responsible for generating an audio waveform from acoustic features.

Input:

M \in \mathbb{R}^{T\times F}

Where:

  • T: the number of time frames.
  • F: the dimensionality of the spectral features.

Output:

x=[x_1,x_2,\ldots,x_S]

Where S is the number of audio samples.

Since one frame of Mel features may correspond to hundreds or even thousands of audio samples, the vocoder must perform generation from low-temporal-resolution features to a high-temporal-resolution waveform.

2. Griffin-Lim

Griffin-Lim is a classic phase reconstruction algorithm.

STFT can convert a time-domain signal into a frequency-domain representation, but when only the magnitude spectrum is retained, the phase information is lost.

Griffin-Lim estimates the phase iteratively so that the reconstructed waveform matches the target magnitude spectrum as closely as possible.

Its basic idea is:

  • Start from an initial phase.
  • Reconstruct the waveform using the magnitude spectrum and the current phase.
  • Perform STFT again on the reconstructed waveform.
  • Retain the target magnitude information and update the phase.
  • Repeat the iterations.

Griffin-Lim does not rely on neural network training, but the naturalness of the generated speech is usually limited.

3. WaveNet

WaveNet is a classic autoregressive neural waveform generation model.

It directly predicts the next audio sample in the time domain:

P(x)=\prod_{t=1}^{T}P(x_t|x_1,\ldots,x_{t-1})

Its core idea is:

Use historical samples to predict the probability distribution of the current sample.

WaveNet's important techniques include:

  • Causal Convolution.
  • Dilated Convolution.
  • Residual Connection.
  • Skip Connection.

Dilated convolution can expand the receptive field without significantly increasing the number of network layers.

However, autoregressive sampling requires generating each sample one by one, so inference is relatively slow.

4. WaveGlow

WaveGlow applies Normalizing Flow to speech waveform generation.

It establishes a mapping between a simple prior distribution and a complex audio distribution through invertible transformations.

Compared with WaveNet, WaveGlow can generate waveforms using parallel computation, so it has an advantage in inference speed.

5. HiFi-GAN

HiFi-GAN is one of the widely used non-autoregressive neural vocoders.

It adopts a generative adversarial network structure, mainly including:

  • Generator: generates waveforms from the Mel spectrogram.
  • Multi-Period Discriminator: judges the authenticity of generated audio at different period scales.
  • Multi-Scale Discriminator: judges audio quality at different time scales.

It constrains the generated waveform through multiple discriminators, enabling the model to learn richer periodic and temporal structures.

The advantages of HiFi-GAN include:

  • Fast generation speed.
  • High audio quality.
  • Good computational efficiency.
  • Suitable for combination with acoustic models such as FastSpeech.

6. Vocoder Comparison

Vocoder Core Technique Main Characteristics
Griffin-Lim Iterative phase reconstruction No neural network training required
WaveNet Autoregressive convolution High quality, but slower generation
WaveGlow Normalizing Flow Supports parallel waveform generation
HiFi-GAN GAN High quality and high efficiency
BigVGAN Improved GAN Strengthened modeling of complex speech waveforms
UnivNet GAN Multi-resolution acoustic modeling

Note that vocoder quality depends not only on the model architecture, but also on training data, sampling rate, acoustic feature configuration, and inference parameters.


VIII. Speech Prosody Modeling

Whether a piece of speech sounds natural depends not only on whether the pronunciation is correct, but also on whether the prosody is reasonable.

Prosody is the way speech is organized at the levels of time and expression.

It mainly includes:

  • Duration.
  • Pitch.
  • Energy.
  • Pause.
  • Stress.
  • Intonation.
  • Rhythm.

1. Duration and Speech Rate

The same sentence can be expressed at different speeds.

For example:

Normal speech rate:
Today we discuss the development of artificial intelligence.

Slower speech rate:
Today, we discuss the development of artificial intelligence.

Speech rate control is not simply a matter of changing playback speed.

Directly adjusting audio playback speed may also change pitch and timbre.

A more ideal approach is to control the duration of phonemes or speech frames through a duration predictor.

2. Pitch and Intonation

Changes in pitch affect how a sentence is expressed.

For example:

  • Declarative sentences usually have a relatively stable intonation contour.
  • Questions may have specific pitch changes at the end of the sentence.
  • Emphatic sentences may highlight key points through pitch changes.
  • Emotional expression may be accompanied by more complex pitch changes.

For Chinese TTS, special attention must also be paid to tones.

Chinese tones are distinctive phonetic features at the lexical level and cannot simply be equated with sentence-level intonation.

For example:

妈 (mā): mother
麻 (má): hemp
马 (mǎ): horse
骂 (mà): scold

Tone errors may directly change word meaning.

3. Pauses and Rhythm

Pauses are an important part of natural speech.

The system needs to identify:

  • Natural pauses between words.
  • Pauses between short sentences.
  • Pauses corresponding to punctuation.
  • Pauses arising from semantic segmentation.
  • Special pauses in emotional expression.

However, punctuation and pauses do not have a simple one-to-one mapping.

For example, a comma does not necessarily correspond to a fixed duration, and a period does not always correspond to the same pause length.

The model needs to model them in combination with context and expressive style.

4. Emotional Speech Synthesis

Emotional TTS attempts to give generated speech different emotional expressions.

Common emotion types:

  • Happy.
  • Sad.
  • Angry.
  • Calm.
  • Surprised.
  • Excited.
  • Gentle.

Emotional control can usually be achieved in the following ways:

  1. Use emotion category labels.
  2. Use continuous emotion vectors.
  3. Use reference audio to extract style features.
  4. Use text instructions to control expressive style.
  5. Use multi-speaker, multi-style training data to learn expressive differences.

Emotion is not an independent pitch parameter but the result of multiple acoustic dimensions acting together.

The same emotion may simultaneously affect pitch, energy, speech rate, voice quality, and pauses.


9. Voice Cloning and Multi-Speaker TTS

Modern TTS can not only generate speech but also control the speaker characteristics of the generated speech.

Among these, voice cloning is an important application direction.

1. Multi-Speaker Speech Synthesis

Multi-speaker TTS requires a single model to generate the voices of different people.

This is usually achieved through speaker embeddings.

Suppose the text encoding result is:

H=[h_1,h_2,\ldots,h_N]

The speaker vector is:

s=SpeakerEncoder(a)

where a can be reference audio.

The model fuses speaker information into the text or acoustic representation:

H'=Fusion(H,s)

Then it generates the target speech based on the fused representation.

Speaker vectors are usually used to express relatively stable voice identity features, such as timbre and vocal habits.

However, in practical models, whether speaker embeddings are completely independent of language, emotion, and recording environment must be ensured jointly through model design and training methods.

2. Zero-shot Voice Cloning

Zero-shot Voice Cloning means that the model does not need to be specially fine-tuned for the target speaker and can generate speech close to that speaker's voice based on a small amount of reference audio.

Typical process:

flowchart TD
    A["Input text"] --> D["TTS generation model"]
    B["Reference audio"] --> C["Speaker feature extraction"]
    C --> D
    D --> E["Generate target speech"]
    F["Language and style control"] --> D

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#E5F0E9,stroke:#83A995,color:#315D49
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D

Zero-shot does not mean that reference audio is completely unnecessary, nor does it mean that all target voices can be accurately reproduced.

The final result is affected by the following factors:

  • Length of the reference audio.
  • Recording quality.
  • Background noise.
  • Whether the speaker features are distinctive.
  • Whether the training data covers the corresponding language and timbre.
  • The model's cross-speaker generalization ability.

3. Differences Between Voice Cloning and Voice Conversion

Technology Input Output Main objective
Voice Cloning Text + reference voice Synthesized speech Generate new content in the target voice
Voice Conversion Source speech + target voice Converted speech Preserve content while changing speaker characteristics
Multi-speaker TTS Text + Speaker ID Synthesized speech Select a voice from a set of known speakers

Voice cloning involves identity impersonation and authorization issues.

In engineering applications, explicit authorization should be obtained from the target speaker, and mechanisms such as identity verification, scope-of-use restrictions, and labeling of synthesized content should be established.


10. Diffusion Models and Next-Generation Speech Generation

Traditional TTS makes extensive use of autoregressive models and non-autoregressive acoustic models.

In recent years, diffusion models have also been applied to speech generation.

1. What Is a Diffusion Model?

Diffusion models learn data distributions through two processes:

  • Forward Process: gradually add noise to the data.
  • Reverse Process: learn to gradually denoise and recover the target data.

In speech generation, the Mel spectrogram or other acoustic representations can be used as the generation target.

The forward process can be abstracted as:

q(x_t|x_{t-1}) = \mathcal{N} \left( \sqrt{1-\beta_t}x_{t-1}, \beta_t I \right)

where:

  • x_t: the noisy representation at step t.
  • \beta_t: noise schedule parameter.
  • I: identity matrix.

The model learns to predict noise or an equivalent denoising target, then gradually recovers the speech representation.

2. Generation Process of Diffusion TTS

flowchart TD
    A["Input text"] --> B["Text encoder"]
    B --> C["Text-conditioned representation"]
    D["Random noise"] --> E["Diffusion denoising network"]
    C --> E
    E --> F["Stepwise denoising"]
    F --> G["Generate acoustic features"]
    G --> H["Neural vocoder"]
    H --> I["Output speech"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#E5F0E9,stroke:#83A995,color:#315D49

Advantages of diffusion models:

  • Can model complex data distributions.
  • Can represent diverse speech generation results.
  • Strong modeling capability for prosody and acoustic details.

Main challenges:

  • Usually require multiple denoising steps.
  • Inference latency may be high.
  • The sampling process needs to be optimized to meet real-time generation requirements.

3. Flow Matching

Flow Matching has been an important technical direction in generative models in recent years and has also been applied to speech generation.

By learning a continuous-time vector field, it describes how samples evolve from a simple distribution to the target data distribution.

Unlike the stochastic denoising process commonly used in traditional diffusion models, Flow Matching can achieve generation by learning deterministic or stochastic dynamical systems.

Its basic form can be expressed as:

\frac{dx_t}{dt}=v_\theta(x_t,t,c)

where:

  • x_t: current generation state.
  • t: continuous time.
  • c: text or other conditional information.
  • v_\theta: the velocity field predicted by the model.

The model solves this process through numerical integration, gradually generating the target speech representation.

Flow Matching does not necessarily mean it is faster than diffusion models; the final speed depends on the network architecture, solver, number of sampling steps, and hardware implementation.