Bilingual Net

English ↔ Darija Translation

Generative AI · Apr 2025 · Completed

A from-scratch neural translator between English and Moroccan Darija.

5 decoding strategies

Built withPyTorchHugging Face TransformersStreamlitPoetry

Overview

Moroccan Darija is spoken by millions of people but has very little parallel text, so most translation tools handle it poorly. I wanted to know whether a Transformer I wrote myself, end to end, could learn usable translations from a small English–Darija corpus.

I built BIALinguaNet in PyTorch: a full encoder–decoder Transformer with residual pre-LayerNorm attention blocks, a label-smoothed loss, and a decoder that supports temperature, top-k, top-p, beam search, and length normalization. I trained one checkpoint per direction (Darija to English and English to Darija) and wrapped both in a Streamlit app that shows ranked translation hypotheses.

Writing every piece myself — masking, batching, decoding — showed me how much of the final output quality comes from decoding choices rather than the network alone, especially on a low-resource language.

The question

Can a from-scratch seq2seq Transformer translate between English and Moroccan Darija, a low-resource dialect, with usable quality?

Key features

Translates in both directions, with a separate trained checkpoint for Darija to English and English to Darija.
Encoder–decoder Transformer written from scratch, with residual multi-head attention that serves as both self-attention and cross-attention.
Decoding is adjustable at inference time: temperature, top-k, top-p, beam size, and length-normalization coefficient.
Returns several alternative hypotheses with their scores instead of a single answer.
Streamlit interface for real-time translation that shows the input next to the output.
Separate training and fine-tuning scripts, with dependencies managed by Poetry.

How it works

Tokenize and batch
Sentence pairs are tokenized with a GPT-2 tokenizer extended with pad, start, and end tokens. Batches are grouped by sequence length under a fixed token budget so little compute is wasted on padding.
Encode the source
The encoder runs pre-LayerNorm residual self-attention and feed-forward blocks over the source sentence, masking out padding positions.
Decode with cross-attention
The decoder applies causally masked self-attention, then cross-attention to the encoder output. Its token embeddings and output projection share weights with the encoder embedding.
Train with label smoothing
Training uses a label-smoothed cross-entropy loss, Adam with a warmup learning-rate schedule, and gradient clipping.
Search for the best translation
At inference, beam search combined with top-k/top-p sampling and length normalization produces ranked hypotheses, and the best one is shown first.