English-to-Arabic Translator
Transformer Machine Translation
An encoder-decoder Transformer written from scratch that learns English-to-Arabic translation from any parallel corpus.
- BLEU 98.4 on the toy set
- chrF 99.0
- 3.0 M–52.3 M parameter presets
Built with
PyTorchCustom TransformersHugging Face TokenizersBeam search
Hugging Face DatasetssacreBLEU
pytest
Overview
Machine translation is the task the Transformer was designed for, so I wrote the encoder-decoder model from Attention Is All You Need in plain PyTorch, about 150 lines for the model itself, and built everything around it that a translation system needs: data preparation, a tokenizer, training, decoding and evaluation. English to Arabic is the default pair, but the language codes are only file suffixes.
The model uses pre-norm residual blocks, a shared BPE vocabulary with tied input and output embeddings, and fused scaled-dot-product attention with source padding and causal target masks. The tokenizer is trained on the data and handles any script, so Arabic stays in its natural order with no reversal tricks. Training has label smoothing, a warm-up schedule, length-bucketed batches, gradient accumulation, resume and self-contained checkpoints; decoding is batched greedy or beam search with a length penalty; evaluation reports BLEU and chrF with sacreBLEU, plus checkpoint averaging.
I verified the pipeline end to end on a built-in toy dataset: 2,000 synthetic pairs from a 16-word English lexicon translated word by word into Arabic. A 0.95 M-parameter model trained for 1,500 CPU steps reaches BLEU 98.4 and chrF 99.0 on 100 unseen pairs. That proves the code works; it says nothing about real translation quality, which needs a corpus such as OPUS-100 and a GPU.
The question
Can a compact, readable Transformer plus the full pipeline around it (data cleaning, tokenizer, training, decoding, scoring) go from raw parallel text to translations without any black box?
Key features
- The Transformer encoder-decoder from scratch: sinusoidal positions, multi-head attention with padding and causal masks, pre-norm blocks and tied embeddings.
- en2ar-prepare cleans, de-duplicates, splits and tokenises any parallel corpus: your own files, a Hugging Face dataset with a translation column, or the built-in toy set.
- A shared BPE tokenizer with NFKC normalisation that works for any script and keeps right-to-left text in logical order.
- Training with label smoothing, Noam warm-up, token-budget batches, gradient accumulation, resume, an overfit check and checkpoints that carry their own tokenizer.
- Batched greedy decoding and beam search with a length penalty, from the CLI, from files, from pipes or from Python.
- BLEU and chrF evaluation with sacreBLEU and checkpoint averaging.
- 43 CPU-only tests, including a real training run that must translate more than 90% of unseen toy sentences exactly.
How it works
Prepare
Sentences are cleaned (length, length-ratio and duplicate filters), shuffled, split, and used to train one BPE vocabulary for both languages.
Embed
Token ids go through one embedding table shared by the encoder, the decoder and the output projection, scaled by the square root of the width and added to sinusoidal positions.
Encode and decode
Pre-norm encoder layers build a memory of the source; decoder layers use causal self-attention over the target so far and cross-attention over that memory.
Train
Label-smoothed cross-entropy on the next target token, Adam with the Noam schedule, batches grouped by length to a token budget, and the best validation loss saved as best.pt.
Translate and score
Greedy decoding for speed or beam search with a length penalty for quality; en2ar-evaluate scores the test split with corpus BLEU and chrF.
Examples
Input
en2ar-translate --checkpoint checkpoints/toy/best.pt --text "three red bird" --text "blue cat" --text "seven green dog one"
Output
ثلاثة أحمر طائر أزرق قطة سبعة أخضر كلب واحد
Input
en2ar-train --data-dir data/toy --preset small --max-steps 1500 --warmup-steps 200 --eval-every 500 --tokens-per-batch 1000 --device cpu && en2ar-evaluate --checkpoint checkpoints/toy/best.pt --beam-size 1
Output
val loss 0.092 (perplexity 1.10) at step 1000; test BLEU 98.4, chrF 99.0 greedy; BLEU 95.1, chrF 95.9 with beam size 4
Tech stack
The Transformer, the training loop and decoding
Custom Transformers
Encoder-decoder with pre-norm blocks, tied embeddings and fused attention
Shared BPE tokenizer trained on the corpus
Beam search
Length-penalised beam search and batched greedy decoding
Downloading parallel corpora such as OPUS-100 (optional extra)
Corpus BLEU and chrF scoring
43 tests: masks, tokenizer, batching, decoding, a real training run
What I learned
- A shared BPE vocabulary is what makes tied embeddings possible, and it removes every script-specific hack, including reversing Arabic
- Beam search is not always better than greedy: on a word-by-word task the length penalty, tuned for real sentences, costs about 3 BLEU
Limitations
- Only the toy dataset has been trained and scored; there are no OPUS-100 or other real-corpus BLEU numbers and no pretrained weights
Keywords
machine translation · neural machine translation · NMT · English to Arabic · Arabic NLP · Transformer · encoder-decoder · attention · from scratch · BPE · byte-pair encoding · beam search · label smoothing · BLEU · chrF · sacreBLEU · OPUS-100 · PyTorch · sequence-to-sequence