Text-to-Speech with Neural Models

Text-to-Speech (TTS) with neural models is the task of converting written text into natural-sounding spoken audio using deep learning architectures, replacing older rule-based and concatenative systems with end-to-end trainable pipelines. It matters because it powers assistive technology, voice interfaces, and multimedia content creation at scale, enabling synthetic voices indistinguishable from human speech. The counterintuitive fact: modern TTS systems do not "record and stitch" audio clips — they synthesize waveforms from scratch, generating thousands of audio samples per second purely from mathematical operations on learned parameters.

The core insight

A neural TTS system learns a mapping from sequences of text tokens to sequences of audio features, then converts those features into a playable waveform — typically in two separate stages called the acoustic model and the vocoder. The "aha moment" is realizing that audio is just a long numerical array (a waveform sampled thousands of times per second), and a neural network can be trained to generate that array conditioned on text, the same way an image generator produces pixels conditioned on a prompt. Once you see audio as structured numerical data, all the standard deep-learning machinery — attention, convolutions, diffusion — applies directly.

How It Works

Neural TTS pipelines are built around two ideas: spectral representation of audio and sequence-to-sequence modeling of text.

Step 1 — Text Preprocessing

Raw text is first normalized (expanding "Dr." → "Doctor", "2024" → "two thousand twenty-four") and then converted to phonemes — the basic sound units of a language. Phonemes are a more stable input than characters because the same letter can make very different sounds ("c" in "cat" vs "city"). ```python

# Using the phonemizer library (pip install phonemizer) from phonemizer import phonemize text = "The cat sat on the mat." phonemes = phonemize(text, backend='espeak', language='en-us') print(phonemes) # → 'ðə kæt sæt ɑːn ðə mæt.' ```

Step 2 — Mel Spectrogram (The Intermediate Target)

Audio is converted to a mel spectrogram: a 2D matrix where rows are frequency bands (on a human-perception-aligned mel scale) and columns are time frames.

1
M[f,t]=log⁡ ⁣(∑k∣X[k,t]∣2⋅Hf[k]+ϵ)\quad M[f, t] = \log\!\left(\sum_{k} |X[k, t]|^2 \cdot H_f[k] + \epsilon\right)

Explanation: Each cell of the mel spectrogram is the log energy of the audio signal in frequency band f at time frame t, after filtering through mel-scale triangular filter banks H_f.

  • M[f, t] — mel spectrogram value at frequency bin f, time frame t
  • X[k, t] — Short-Time Fourier Transform (STFT) coefficient at frequency bin k, time t
  • H_f[k] — the k-th coefficient of the f-th mel filter bank
  • ε — small constant for numerical stability (log of zero is undefined)

```python import librosa import numpy as np

# Load audio and compute mel spectrogram y, sr = librosa.load("speech.wav", sr=22050) mel = librosa.feature.melspectrogram(y=y, sr=sr, n_mels=80, fmax=8000) mel_db = librosa.power_to_db(mel, ref=np.max)

# mel_db.shape → (80, time_frames) — this is what the acoustic model learns to produce ```

Step 3 — Acoustic Model (Text → Mel Spectrogram)

The acoustic model is a sequence-to-sequence neural network. The dominant architecture today uses Transformer attention to align input phonemes with output mel frames. The attention mechanism computes a soft alignment: for each output time step, it decides which input phonemes to "look at."

2
Attention(Q,K,V)=softmax ⁣(QK⊤dk)V\quad \text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Explanation: Each output frame queries the input sequence, weights each input by relevance, and takes a weighted sum — this is how the model "reads" the right phoneme while generating each moment of audio.

  • Q — Query matrix (from the decoder's current state)
  • K — Key matrix (from the encoder's phoneme representations)
  • V — Value matrix (information carried by each phoneme)
  • d_k — dimension of the keys (used to scale and stabilize gradients)

```python

# Simplified: using a pretrained Tacotron2 acoustic model import torch from TTS.api import TTS # pip install TTS (Coqui TTS) tts = TTS(model_name="tts_models/en/ljspeech/tacotron2-DDC")

# Generate mel spectrogram internally, then vocoder produces wav tts.tts_to_file(text="Hello, welcome to neural TTS!", file_path="output.wav") ```

Step 4 — Vocoder (Mel Spectrogram → Waveform)

The vocoder inverts the mel spectrogram back into a raw audio waveform. Modern vocoders like HiFi-GAN use generative adversarial networks: a generator produces waveform samples and a discriminator checks whether they sound real.

3
LGAN=E[log⁡D(x)]+E[log⁡(1−D(G(m)))]\quad \mathcal{L}_{\text{GAN}} = \mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(m)))]

Explanation: The GAN loss pushes the generator G to produce waveforms from mel spectrogram m that the discriminator D cannot distinguish from real waveforms x.

  • D(x) — discriminator's probability that real audio x is genuine
  • G(m) — generator's waveform output given mel spectrogram m
  • 𝔼 — expected value over the training data distribution
researcher · Aaron van den Oord | b. 1988 | Inventor of WaveNet

Aaron van den Oord, a Belgian researcher at Google DeepMind, developed WaveNet in 2016 — the first neural model to generate raw audio waveforms autoregressively with quality that stunned the speech research community. His key insight was applying dilated causal convolutions so the network could efficiently see a large audio context window while still generating samples one at a time. Surprisingly, the original WaveNet was so slow (generating 1 second of audio took 90 seconds of compute) that it was not deployed in production for over a year after its announcement.

Figure
Neural TTS Two-Stage Pipeline
Neural TTS Pipeline TEXT "Hello world" PHONEMIZER hɛloʊ wɜːld ACOUSTIC MODEL (Tacotron2 / FastSpeech2) VOCODER (HiFi-GAN) mel → waveform AUDIO .wav 22050 Hz mel spectrogram (80 × T matrix) string phoneme ids float tensor ↑ attention aligns phonemes → frames
Figure
Mel Spectrogram vs. Waveform
Audio Representations RAW WAVEFORM amplitude × time (1D) 22,050 samples / sec ⇔ STFT / Griffin-Lim / Vocoder MEL SPECTROGRAM Hi Lo freq (mel) × time (2D) 80 freq bins × T frames Neural models learn to generate the 2D mel — then vocoder converts it to playable audio

Intuition

The Mental Model

Think of a professional actor reading a script. The script is your text. The actor first interprets the script — understanding which words to stress, where to pause, what emotion to convey — and produces an abstract "performance plan." That's the acoustic model's job. Then a sound engineer takes that performance and actually records and masters the audio into a final file — that's the vocoder's job. The mel spectrogram is like the sheet music sitting between them: not yet sound, but a precise enough specification that the engineer can create the final audio faithfully. Neural TTS automates both the actor and the engineer with trained networks, so any text you hand it gets this full treatment in milliseconds.

Historical Development

Origins · 1 minhow this was discovered
History
1950
s–1980s: Formant synthesis — rule-based systems like KLATT synthesizer use mathematical models of the human vocal tract. Intelligible but robotic-sounding.
1990
s: Unit selection / concatenative TTS — Festival TTS and similar systems record thousands of real speech segments and stitch them together. More natural but inflexible and requires large databases.
2016
WaveNet (van den Oord et al., DeepMind) — first deep neural model to generate raw audio samples autoregressively with near-human quality, using dilated causal convolutions.
2017
Tacotron (Wang et al., Google) — first end-to-end sequence-to-sequence acoustic model using attention, trained directly from (text, audio) pairs with no manual alignment.
2018
Tacotron 2 (Shen et al., Google) — combines a refined Tacotron acoustic model with WaveNet vocoder, achieving near-human MOS (Mean Opinion Score) ratings in blind tests.
2019
FastSpeech (Ren et al., Microsoft) — removes the slow autoregressive decoder with a duration predictor and parallel mel generation, achieving 38× faster inference than Tacotron 2.
2020
HiFi-GAN (Kong et al.) — GAN-based vocoder that generates audio in real-time with quality matching or exceeding WaveNet at a fraction of the compute cost.
2021
VITS (Kim et al.) — fully end-to-end TTS model combining variational inference and normalizing flows, eliminating the two-stage pipeline without quality loss.
2023
Voicebox / Voicecraft / Bark — diffusion- and language-model-based TTS systems achieve zero-shot voice cloning and highly expressive speech from a few seconds of reference audio.

Why It Matters

Real-World Impact
Apple Siri / Google AssistantUse neural TTS backends (neural voices) to generate responses in real-time; Apple's "neural voice" models shipped in iOS 16, replacing concatenative systems and dramatically improving naturalness.
Amazon Polly (Neural TTS)AWS offers production TTS at scale using NTTS engines; media companies use it to auto-narrate articles and generate thousands of hours of audio content.
Microsoft Azure Cognitive Services TTSPowers accessibility tools for visually impaired users, reading e-books and interfaces aloud; uses FastSpeech-style models for low-latency output.
ElevenLabs Voice CloningUses neural TTS with voice adaptation fine-tuning to clone a speaker's voice from seconds of audio; widely used in podcasting, audiobooks, and dubbing.
Google's AudioLM / SoundStormResearch systems treating audio generation as a language modeling problem over discrete audio tokens, enabling highly consistent long-form speech generation.
Audiobook ProductionPublishers like Penguin Random House use neural TTS to narrate backlists of thousands of titles, reducing production time from days (human narration) to minutes.
Real-time Game DialogueGame studios use TTS to dynamically generate character speech for procedurally generated dialogue, avoiding the need to pre-record every possible line.

Prerequisites

What This Leads To

See Also

Automatic Speech Recognition · WaveNet Architecture · Transformer Architecture · Generative Adversarial Networks · Audio Diffusion Models

Before this

Where this leads