MyTorcha
A Deep Learning Framework From Scratch
A deep learning framework rebuilt from first principles: autograd to LSTMs and attention, with every gradient checked.
- 160 tests
- Matches PyTorch to ~1e-10
- 97.1% digit accuracy (CNN)
Built with
Python
NumPy
pytest
PyTorch
Overview
Frameworks like PyTorch make training a network one call, which also hides how it works. MyTorcha removes that layer: every gradient is code I wrote and a test that checks it, with NumPy as the only dependency.
It started as a scalar autograd engine, where each number is a node in a computation graph. That version is kept as a teaching chapter. It is also why the framework grew a second engine: one 784-to-10 layer builds 31,427 scalar nodes per image. The Tensor engine applies the same chain rule to whole arrays, which makes convolutional, recurrent and attention models practical on a CPU.
Rebuilding it meant testing the original first. Finite-difference checks exposed real bugs in v1: a log derivative off by a factor of about 3, nodes that compared equal by value and were silently merged during backpropagation, and optimizer state shared across every weight. Each one is now a regression test.
The question
Do I understand deep learning well enough to rebuild a framework, from autograd to LSTMs and attention, whose gradients match PyTorch's?
Key features
- Tensor autograd: broadcasting-aware gradients, matmul, reductions, indexing, concat/stack, no_grad.
- 20 activations (ReLU to GELU, Mish and a learned PReLU) and 10 losses, including stable cross-entropy with label smoothing.
- Layers: Linear, Conv1d/Conv2d/ConvTranspose2d (im2col), pooling, BatchNorm/LayerNorm/GroupNorm, Dropout, Embedding.
- RNN, LSTM and GRU (multi-layer, bidirectional), multi-head attention and a pre-norm Transformer block.
- 8 optimizers (SGD with Nesterov, Adagrad, RMSprop, Adadelta, Adam/AMSGrad, AdamW, Adamax, Lion), LR schedulers and gradient clipping.
- 160 tests: finite-difference gradchecks and PyTorch parity for layers, LSTM/GRU, attention and optimizers.
- A teaching website with seven chapters and interactive widgets: live computation graph, optimizer race, convolution visualizer.
How it works
Every op records its inputs
An operation computes its output array and stores a function that maps the output's gradient to one gradient per input.
backward() walks the graph
An iterative topological sort orders the graph; gradients flow in reverse and are summed back over broadcast dimensions.
Layers are Modules
A Module finds its Parameters by walking its attributes, so a CNN or LSTM is plain Python composed from tested pieces.
Checked twice
Each backward rule is compared with finite differences in float64, and each layer's forward and backward with PyTorch's.
Examples
Input
cnn = nn.Sequential(nn.Conv2d(1, 8, 3, padding=1), nn.BatchNorm2d(8), nn.ReLU(), nn.MaxPool2d(2), ..., nn.Linear(784, 10)); mt.Trainer(cnn, nn.CrossEntropyLoss(), optim.Adam(cnn.parameters(), lr=2e-3)).fit(train_loader, epochs=3, val_loader=val_loader)
Output
Validation accuracy 94.35%, 96.70%, 97.10% over 3 epochs (9,146 parameters; an MLP with 101,770 parameters reaches 93.65%).
Input
model = CharLSTM(vocab); 300 Adam steps on a short text
Output
step 0: 'the dbveclgk stmap,sed d ahylb...' -> step 100: 'the chain rule walks that memory backwards.'
Tech stack
What I learned
- Backprop is the chain rule over a DAG; the hard parts are broadcasting, shared state and numerical stability
- Never trust a gradient without a finite-difference check: my v1 had a log derivative that was off by 3x
- Speed comes from memory layout: rewriting convolution as one contiguous im2col matrix multiply made a conv layer's forward pass about 4x faster
Limitations
- CPU-only and NumPy-bound: a framework for learning and small experiments, not for large-scale training