Speech Emotion Recognition

8 Audio Classifiers Tested on Unseen Speakers

Audio · Oct 2026 · Completed

Detect the emotion in a voice, and find out which way of feeding audio to a model survives a speaker it has never heard.

82.5% accuracy (WavLM)
3× the original pipeline

Built withPyTorchHugging Face Transformerslibrosascikit-learnNumPyMatplotlib

View code on GitHub

Overview

Emotion in speech lives in how a sentence is said, not what is said. RAVDESS makes that explicit: 24 actors speak the same sentence in 8 emotions. I wanted to know which way of presenting audio to a model works best, so I trained one model per representation on the same data and split: a 1D CNN on the raw waveform, a 1D CNN on MFCC sequences, a 2D CNN and a CNN + BiGRU + attention model on log-mel spectrograms, an MLP and an SVM on 258 handcrafted features, a frozen WavLM with a small head, and the original Colab notebook's feed-forward network kept as the baseline.

The split is by actor: 18 for training, 2 for validation, 4 for testing, so no test voice is heard during training. That is harder than the random splits behind many published RAVDESS numbers. Each model runs over 3 seeds. The frozen WavLM wins by a wide margin at 77.6% test accuracy; the CRNN reaches 65.4%, the spectrogram and MFCC CNNs about 58%, the MLP 51.4%, the SVM 49.2%, the raw-waveform CNN 46.1% and the notebook pipeline 27.8%.

The first version logged train accuracy on augmented clips, which hid how much had been memorised. Measured honestly, every model fits its training set at 78 to 100% and loses 22 to 47 points on new speakers. Weight decay, mixup, speed perturbation and weight averaging narrow that gap by only a few points. Standardising every feature per speaker is what works: it lifts the MLP by 17 points, the MFCC CNN by 7 and WavLM by 5, to 82.5%. The repo also ships eight test clips, one per emotion, each rendered as a short video with the spectrogram, a playhead and the model's probabilities; WavLM gets 6 of the 8 right, hearing neutral as calm and happy as surprised.

The question

Which audio representation works best for emotion recognition when the test speakers are new, and why do models trained from scratch overfit so badly to the voices they have heard?

Key features

Eight classifiers on one data pipeline: raw-waveform CNN, MFCC CNN, log-mel CNN, CRNN with attention pooling, MLP, RBF SVM, a frozen WavLM base+ with a learned weighted sum of its 13 layers, and the original notebook network as a faithful baseline.
Speaker-independent protocol: split by actor (1,080 / 120 / 240 clips), 16 kHz audio with silence trimmed and cropped or padded to 3 s, accuracy and macro-F1 reported as mean ± std over 3 seeds.
Honest overfitting measurement: train accuracy on 240 clean clips in eval mode every epoch, with a --baseline flag that turns every anti-overfitting measure off for the before/after table.
Regularisation per model: AdamW weight decay 0.05, input dropout, mixup, Gaussian feature noise, speed and pitch perturbation, SpecAugment and EMA weight averaging, each kept only if it helped on validation.
Per-speaker normalisation for every feature family, including the log-mel front-end of the CNNs, with predict.py --speaker-ref taking a few unlabelled clips of a new speaker.
A resumable compare.py that runs every model and seed, then report.py rebuilds the results table, training curves, confusion matrices, per-emotion F1 and the overfitting figure from the JSON logs.
Six example scripts, from the quickest real result (an SVM in 3 seconds) to classifying the sample clips and rendering them as videos with sound.

How it works

Load and split by actor
prepare_ravdess.py downloads the 1,440 speech clips and puts actors 1-18 in train, 19-20 in validation and 21-24 in test; labels and speaker ids come from the file names.
Turn audio into each representation
Clips are resampled to 16 kHz, trimmed and fixed to 3 s, then become a 48,000-sample waveform, a 40 × 151 MFCC sequence, a 64 × 151 log-mel spectrogram, 258 pooled handcrafted features, or the hidden states of a frozen WavLM, cached after the first run.
Train one model per representation
Every neural model uses AdamW with a one-cycle schedule, label smoothing 0.1 and batch 32; its own augmentations and regularisers are applied, and the checkpoint with the best validation macro-F1 is kept.
Normalise per speaker
The _spk variants standardise each feature with the speaker's own mean and variance instead of the training set's, which removes most of what makes one voice differ from another.
Report on unseen actors
Each run writes a JSON with its history and test metrics; report.py aggregates seeds into the results table, confusion matrices, per-class F1 and the train-test gap figure.

Examples

Test accuracy and macro-F1 on unseen actors for all 14 rows (mean over 3 seeds). The frozen WavLM leads at 77.6%, 82.5% with speaker normalisation; the original notebook pipeline sits at 27.8%, with chance at 12.5%.

Train-test gap (left) and test accuracy (right) for five models without regularisation, with it, and with speaker normalisation on top. Regularisation alone barely moves the test score; speaker normalisation cuts the MFCC CNN gap from 40 to 18 points and lifts the MLP from 47.4% to 64.7%.

Test confusion matrices for every model (rows are the true emotion, in percent). The same two confusions appear everywhere: neutral is heard as calm, and happy as surprised.

Test F1 per emotion, models sorted best to worst. Angry, surprised and disgust are the easiest classes; neutral, happy and sad are the hardest, and the notebook baseline never predicts neutral.

Input

python predict.py samples/05_angry_actor21.wav --model wavlm

Output

angry 76% (true: angry)

Classifying one of the eight sample clips from a test actor. Across the eight clips WavLM gets 6 right; the two misses are neutral heard as calm (55%) and happy heard as surprised (54%).

Tech stack

PyTorch
The seven neural models, front-ends, augmentations and training loop
Hugging Face Transformers
Frozen WavLM base+ backbone (any wav2vec2 / HuBERT / WavLM checkpoint can be swapped in)
librosa
Resampling, silence trimming, MFCC, chroma, mel, contrast and tonnetz features
scikit-learn
RBF SVM baseline and classification metrics
NumPy
Feature pooling, per-speaker standardisation and caching
Matplotlib
Results figures: accuracy bars, curves, confusion matrices, per-class F1

What I learned

The models overfit to who is speaking, not just to the clips: classic regularisation changes test accuracy by no more than the seed noise, while normalising features per speaker closes much of the gap
Many published RAVDESS scores use random splits where the same actors appear in train and test, so they are not comparable with a speaker-independent test

Limitations

The speaker-normalised variants need a few unlabelled clips of each test speaker, so they follow a slightly different protocol from the rows that use nothing from the test speakers
With 18 training voices a train-test gap remains for every model; a larger corpus such as CREMA-D or MSP-Podcast is the next step

Keywords

speech emotion recognition · audio classification · RAVDESS · WavLM · wav2vec2 · self-supervised speech · mel-spectrogram · MFCC · CRNN · raw waveform CNN · SVM · speaker normalisation · speaker-independent evaluation · overfitting · PyTorch