Text Classification

Emotion Detection with RNNs, LSTMs and GRUs

NLP · Nov 2024 · In progress

Twelve recurrent models that read a sentence and name its emotion, rebuilt from two broken notebooks into one tested package.

92.3% accuracy
12 models compared
13 bugs fixed

Built withPyTorchGloVeNLTKPandasMatplotlibpytest

View code on GitHub

Overview

This project started as two Colab notebooks that classified English sentences into six emotions with RNN, LSTM and GRU models. The notebooks reached 50 to 81.5% test accuracy and I did not trust the numbers. I rebuilt them as a Python package with one shared training recipe and found out why: the from-scratch cells were not recurrent, padding leaked into the prediction, the GRU equation was wrong, and the evaluation was optimistic.

The package keeps the notebooks' ideas: three cell types, each built twice (with nn.RNN / nn.LSTM / nn.GRU and from scratch), fed either a learned embedding table on lemmatised text or frozen 100-d GloVe vectors on raw text. That makes 12 models, all two-layer and bidirectional with the same MLP head, trained for up to 8 epochs on a CPU with early stopping on validation accuracy, on the dair-ai/emotion dataset (16,000 train / 2,000 validation / 2,000 test).

Every model now lands at 91 to 92% test accuracy with a macro F1 of 0.87 to 0.88; the best is glove-lstm-scratch at 92.3%. The from-scratch cells match PyTorch's numerically, which the tests prove by copying PyTorch's weights into them. The remaining errors are the close classes: love taken for joy, surprise taken for fear.

The question

Why did my original notebooks range from 50% to 81.5% accuracy on the same task, and do the cell type and the embedding still matter once the pipeline is correct?

Key features

RNN, LSTM and GRU cells written from scratch to PyTorch's equations, with tests that show they give exactly the same numbers as nn.RNN / nn.LSTM / nn.GRU.
Two embeddings: a learned table (the notebooks' one-hot Linear, done by indexing) and frozen GloVe 100-d vectors restricted to the words found in the corpus.
Correct padding: packed sequences for the torch cells, a length mask for the scratch cells, and a test that extra padding changes nothing for all 12 models.
One recipe for every model: AdamW, label smoothing, gradient clipping, a decaying learning rate, early stopping, and the best checkpoint scored on the full test set with accuracy and macro F1.
Sanity checks on the real data: the loss at init equals ln 6, one batch of 64 sentences is memorised, and the model clearly beats a blind copy with erased input.
Two CLIs, textcls-train and textcls-predict, plus scripts that train all 12 models and regenerate the results page with curves and confusion matrices.

How it works

Clean the text
Lower-case and keep letters, digits and basic punctuation; the one-hot runs also drop rare words and stop words (keeping negations) and lemmatise with NLTK, while the GloVe runs use the raw words.
Embed
Token ids index a learned 3,100 x 100 table, or frozen GloVe vectors with a zero row for padding and the mean vector for unknown words.
Read the sentence
A two-layer bidirectional RNN, LSTM or GRU runs over the tokens; the state at the last real token of each direction is the sentence summary.
Classify
Dropout, a Linear-ReLU-Linear head and a six-way softmax, trained with label-smoothed cross-entropy.
Check before trusting
Overfit one real batch, compare with a blind model, verify that gradients reach every tensor, then train for up to 8 epochs and keep the best validation epoch.

Examples

Test accuracy of the 12 models in the original notebooks (grey) versus the fixed package (blue): from 50 to 81.5% up to 91 to 92%.

Overfit test: each model memorises one batch of 64 real sentences, driving the loss from ln 6 to about zero. The spike of the one-hot scratch RNN around step 65 is the known instability of vanilla ReLU RNNs; it recovers on its own.

Validation accuracy per epoch. Solid lines are the nn.* cells and dashed lines the from-scratch cells; GloVe starts slower than the learned table and catches up by epoch 5.

Test confusion matrix of glove-lstm-scratch, the best model (92.3%): the mistakes are love predicted as joy (32 of 159) and surprise and fear confused with each other.

Input

textcls-predict --checkpoint checkpoints/glove-lstm-scratch/best.pt "i didnt feel humiliated" "i feel amazing today, everything went right"

Output

sadness 0.98 (anger 0.01, fear 0.00) <- i didnt feel humiliated surprise 0.88 (joy 0.10, fear 0.01) <- i feel amazing today, everything went right

Real output of the best model. The second line is a mistake (it should be joy): in the training set "amazing" is labelled surprise 39 times out of 121, ten times the base rate of that class.

Tech stack

PyTorch
The models, the from-scratch cells and the training loop
GloVe
Frozen 100-d word vectors trained on 6B tokens
NLTK
Stop words and lemmatisation for the one-hot runs
Pandas
Reading the dataset's parquet splits
Matplotlib
Training curves, confusion matrices and the comparison chart
pytest
Unit tests: cell parity with PyTorch, padding invariance, metrics

What I learned

On short, keyword-heavy sentences, getting recurrence, padding and evaluation right matters far more than the choice of cell or embedding
A loss that starts at ln(classes), an overfit test on one batch and a blind-input baseline catch most pipeline bugs before a long run

Limitations

The rare classes (surprise at 3.6%, love) have the lowest F1; class weighting, attention pooling and a Transformer baseline are still on the roadmap
The notebook numbers are not an apples-to-apples baseline: they used a filtered test set and stopped at a hand-picked target

Keywords

text classification · emotion detection · sentiment analysis · RNN · LSTM · GRU · recurrent neural networks · from scratch · bidirectional RNN · GloVe · word embeddings · one-hot embedding · NLP · dair-ai/emotion · macro F1 · confusion matrix · PyTorch · NLTK · overfit test · sanity checks