STT Workbench

Speech-to-Text Approaches in PyTorch

Audio · Oct 2026 · Completed

Six ways to turn speech into text, built small enough to watch each one learn on a CPU.

CTC 2.4% WER (toy)
15.1% WER on FSDD with an LM

Built withPyTorchtorchaudioBeam searchONNXHugging Face TransformersNumPypytestKaggle

Overview

Automatic speech recognition has several families of models that solve the same hard problem, aligning a long sequence of audio frames with a short sequence of characters, in different ways. I wanted to see those differences for myself rather than read about them, so I wrote each approach in one small PyTorch code base where they share the front end, the encoders, the trainer and the evaluation.

The models are CTC, an attention encoder-decoder, Listen-Attend-Spell, hybrid CTC/attention and an RNN-Transducer, on BiLSTM, Transformer or Conformer encoders, plus wrappers for fine-tuning wav2vec2 and Whisper in the notebook. Every decoding algorithm is there too: greedy, CTC prefix beam search with an n-gram language model, attention beam search with CTC prefix scoring, and RNN-T greedy and modified beam search. The models have 1 to 1.3 M parameters by default.

I tested it on simple 8 kHz data that trains in minutes on a CPU: a synthetic toy language, the Free Spoken Digit Dataset and YESNO. CTC reaches 2.4% WER on the toy language; on FSDD with a held-out speaker the CTC model gets 24.7% WER greedy and 15.1% with beam search plus a character 5-gram LM. These runs show that each approach learns and that the decoders work; they are not benchmarks, and real training belongs in the Kaggle/Colab notebook.

The question

If CTC, attention and transducer models share the same front end, encoders and trainer, what does each one actually need in order to learn the alignment between audio frames and text?

Key features

Five trainable models behind one flag: CTC, attention encoder-decoder, LAS with location-aware attention, hybrid CTC/attention and RNN-Transducer.
Three encoders any model can use, BiLSTM, Transformer and Conformer, after a log-mel front end with CMVN, SpecAugment and 4x convolutional subsampling.
Every decoder: greedy, CTC prefix beam search with an interpolated n-gram LM, attention beam search with CTC prefix rescoring, and RNN-T greedy and modified beam search.
Built-in datasets written or downloaded by one command (synthetic toy language, YESNO, FSDD, LibriSpeech, LJSpeech, Common Voice) with duration-bucketed batching.
WER / CER evaluation with aligned worst utterances, a word confusion matrix and a script that compares decoders on the same checkpoint.
ONNX export of the encoder and CTC head, wav2vec2 and Whisper fine-tuning wrappers, and a Kaggle/Colab notebook generated from the package source.
80 tests in about 10 seconds on a CPU, including a case where greedy CTC decoding is wrong and prefix beam search is right.

How it works

Front end
The waveform becomes an 80-bin log-mel spectrogram, normalised per utterance (CMVN) and masked in time and frequency by SpecAugment during training.
Encode
Two strided convolutions subsample the frames by four, then a BiLSTM, Transformer or Conformer stack builds contextual acoustic features.
Align frames with tokens
CTC sums over all frame paths that collapse to the transcript; attention decoders learn a soft alignment with cross-attention; the RNN-T joins the encoder with a prediction network over a monotonic lattice.
Train
Character or SentencePiece tokens, Adam with warm-up, duration-bucketed batches, validation WER / CER at every evaluation, and checkpoints that can be resumed.
Decode and score
Greedy or beam search, optionally fused with an n-gram LM; stt-eval aligns hypotheses with references and prints WER, CER and the worst utterances.

Examples

Toy language, 400 training sentences, 1,500 steps: CTC (2.4% WER) and RNN-T (5.3%) learn in a few hundred steps, while the three attention decoders (hybrid, LAS, AED) have not found the alignment yet.

The same attention decoders with four times the data (1,600 sentences, 3,000 steps): LAS reaches 0% WER from step 1,000 on, hybrid 21.7%, and the plain Transformer decoder is still improving at 70.4%.

Real 8 kHz audio: CTC and hybrid models on three-digit FSDD clips with a held-out speaker (24.7% and 39.8% WER), and CTC on the 60-clip YESNO set (56.9% WER).

Word confusion matrix of the greedy CTC decoder on the unseen FSDD speaker: most digits are right, and "six" is the one word the model often drops (the <del> column) or hears as "eight".

Input

stt-train --data-dir data/synthetic --sample-rate 8000 --model ctc --encoder conformer --d-model 96 --n-layers 4 --max-steps 1500 --output-dir checkpoints/ctc && stt-eval data/synthetic/val.jsonl hyps.jsonl

Output

val WER 2.4%, CER 0.6% (1.04 M parameters, 20 min on one CPU with two threads)

The quick-start run from the README: a Conformer + CTC model on the synthetic toy language, greedy decoding.

Input

python scripts/compare_decoders.py # same CTC checkpoint, 166 FSDD validation clips

Output

greedy: 24.7% WER | beam 8: 24.1% | beam 8 + character 5-gram LM (weight 0.3): 15.1%

Three decoders on one checkpoint. The LM, trained on the 830 training transcripts, knows that only ten words exist and removes most partial words such as "even" for "seven".

Tech stack

PyTorch
Every model, encoder, loss and search algorithm
torchaudio
RNN-T loss (optional extra)
Beam search
CTC prefix beam search, attention beam search with CTC rescoring, RNN-T modified beam search
ONNX
Export of the encoder and CTC head
Hugging Face Transformers
wav2vec2 and Whisper fine-tuning wrappers (optional extra)
NumPy
Synthetic toy-language generation
pytest
80 tests, about 10 seconds on a CPU
Kaggle
Generated notebook for real training on a free GPU

What I learned

CTC and RNN-T get a monotonic alignment for free, while attention decoders have to discover where to look, which costs data and steps
A language model only helps when the language has structure: it cut FSDD WER from 24.7% to 15.1% and changed nothing on random-letter words

Limitations

All results are short CPU runs on 8 kHz toy, FSDD and YESNO data, not benchmarks; no LibriSpeech numbers and no pretrained weights are published
The YESNO run (42 training clips) stays at 56.9% WER; it only checks that the pipeline runs on real audio

Keywords

speech recognition · ASR · speech-to-text · CTC · RNN-Transducer · RNN-T · Listen Attend and Spell · attention encoder-decoder · hybrid CTC/attention · Conformer · BiLSTM · Transformer · prefix beam search · n-gram language model · SpecAugment · log-mel spectrogram · WER · CER · wav2vec2 · Whisper · PyTorch · ONNX · FSDD · LibriSpeech