TTS Workbench

Text-to-Speech Approaches in PyTorch

Audio · Oct 2026 · Completed

Four text-to-speech models and a neural vocoder, small enough to train on a laptop and compare on the same metrics.

FastSpeech 2: 12.3 dB MCD
4 TTS models

Built withPyTorchNumPyMatplotlibHugging Face TransformerspytestKaggle

Overview

Text-to-speech went through several designs in a few years: autoregressive attention models, non-autoregressive models with explicit durations, and end-to-end variational models with adversarial training. I built each of them in one small PyTorch code base, about one million parameters apiece, so that a run on a CPU shows what each method does and every line can be read.

The models are Tacotron 2, Transformer-TTS, FastSpeech 2 and VITS, with Griffin-Lim and a HiFi-GAN vocoder, a built-in aligner with monotonic alignment search for teacher durations, and an evaluation module that measures MCD along a DTW path, log-F0 RMSE, voicing and duration errors and attention diagonality, with optional WER through Whisper and predicted MOS with UTMOS. All of them share the data layout, trainer, checkpoint format and evaluation.

I tested everything on simple 8 kHz data in short CPU runs: a synthetic toy language and FSDD spoken digits. On the toy language Transformer-TTS, FastSpeech 2 and VITS reach 12 to 13 dB MCD against a 10.5 dB copy-synthesis floor, with near-diagonal alignments; Tacotron 2 shows the classic attention failure, and the HiFi-GAN vocoder is clearly under-trained at this budget. The numbers say that the pipeline and the metrics work, not that the models compete with LJSpeech systems.

The question

When four TTS designs share the same data, trainer and metrics, which parts of each one actually learn, and what do the objective metrics reveal that the training loss hides?

Key features

Four synthesis models behind one flag: Tacotron 2 (location-sensitive attention, stop token, postnet), Transformer-TTS (guided attention), FastSpeech 2 (duration, pitch and energy predictors) and VITS (conditional VAE, normalising flow, GAN).
Two vocoders for the mel models: Griffin-Lim phase reconstruction and a HiFi-GAN with multi-period and multi-scale discriminators and feature matching.
A built-in aligner with the forward-sum loss and monotonic alignment search, so FastSpeech 2 needs no external forced aligner.
A learned speaker embedding in every model for multi-speaker data.
Metrics on free-running synthesis: MCD along a DTW path, log-F0 RMSE in semitones, voicing and duration error, attention diagonality, and optional Whisper WER and UTMOS.
Built-in datasets (toy language, FSDD, YESNO, LJSpeech) and a Kaggle/Colab notebook generated from the package source, whose checkpoints work with the CLIs.
34 tests with micro models on toy data in about 25 seconds.

How it works

Text to symbols
Text is normalised and tokenised into symbol ids; the audio side is a log-mel spectrogram computed with the same settings everywhere.
Predict a mel, or a waveform
Tacotron 2 and Transformer-TTS decode frame by frame with attention and teacher forcing; FastSpeech 2 expands each symbol by its predicted duration and adds pitch and energy; VITS maps text to a latent and decodes the waveform directly.
Learn the alignment
Attention models learn it from the data, helped by a guided-attention loss; FastSpeech 2 and VITS get it from monotonic alignment search on the aligner's or the posterior encoder's similarity matrix.
Vocode
A mel becomes audio through Griffin-Lim or through the HiFi-GAN generator, trained with adversarial, feature-matching and mel losses.
Evaluate
tts-eval synthesises validation sentences, aligns each with its recording by DTW and reports MCD, F0, voicing, duration and alignment metrics, next to copy-synthesis and same-speaker reference points.

Examples

Toy language, 285 training sentences: training loss (log scale) and teacher-forced validation mel L1 for the four models. Tacotron 2 reaches the lowest teacher-forced error (0.12), which hides that its attention never aligned.

Alignments at the first and last evaluation on the toy language: Transformer-TTS attention becomes diagonal by step 720, the MAS paths of FastSpeech 2 and VITS settle on the diagonal, and Tacotron 2 keeps attending to one symbol.

FSDD spoken digits, 475 one-word clips: training loss and validation mel L1 of Tacotron 2 (7,200 steps) and FastSpeech 2 (best checkpoint at step 4,200).

On FSDD, Tacotron 2's attention is still blurry after 7,200 steps and FastSpeech 2's MAS path piles frames on one or two symbols (diagonality 0.29 to 0.40): 475 one-word clips are too few for a sharp alignment.

Input

tts-eval --checkpoint checkpoints/fs2/best.pt --data-dir data/synthetic --n 15

Output

MCD 12.3 dB, log-F0 RMSE 3.0 st, voicing error 0.04, duration error 0.05, alignment diagonality 0.97 (Griffin-Lim; copy-synthesis floor 10.5 dB)

FastSpeech 2 on the toy language, the best MCD and F0 of the four models. With the under-trained HiFi-GAN instead of Griffin-Lim, MCD rises to 14.6 dB and the F0 error to 19.5 semitones.

Tech stack

PyTorch
All models, vocoders, losses, the aligner and the trainer
NumPy
Synthetic toy-language generation and metric helpers
Matplotlib
Training curves and alignment grids
Hugging Face Transformers
Whisper transcripts for the WER metric (optional extra)
pytest
34 tests with micro models on toy data
Kaggle
Generated notebook for real training on a free GPU

What I learned

The teacher-forced loss never tells you whether an attention TTS model can synthesise; only free-running metrics and the alignment plot do
An unconverged GAN vocoder adds about 2 dB of MCD and destroys the pitch; at CPU scale Griffin-Lim is the honest vocoder

Limitations

All numbers are short CPU runs on 8 kHz toy and FSDD data; no LJSpeech results or pretrained weights are published
The HiFi-GAN vocoders (540 and 1,200 steps) are under-trained, and Tacotron 2 never aligned on 285 toy sentences

Keywords

text-to-speech · TTS · speech synthesis · Tacotron 2 · Transformer-TTS · FastSpeech 2 · VITS · HiFi-GAN · Griffin-Lim · neural vocoder · monotonic alignment search · MAS · location-sensitive attention · mel spectrogram · mel-cepstral distortion · MCD · F0 · UTMOS · MOS · PyTorch · FSDD · LJSpeech