Image Captioner
Multimodal Image Captioning
A model that looks at a photo and writes a sentence describing it
Built with
PyTorchCustom Transformers
Streamlit
Poetry
Hugging Face Transformerseinopstorchvision
Overview
Image captioning asks a model to connect two modalities: pixels and words. I wanted to understand that connection from the inside, so I wrote every piece of a captioning model by hand in PyTorch instead of assembling pretrained components.
I built an encoder-decoder Transformer: a Vision Transformer encoder turns the image into a sequence of patch tokens, and a Transformer decoder writes the caption one token at a time, using cross-attention to look at the image tokens at every step. The training setup targets the Flickr30k image-caption dataset on Kaggle.
The project taught me how cross-attention lets a text decoder condition on a different modality, and how small choices such as weight tying, label smoothing, and length-aware batching shape training. It is a learning artifact trained on limited data, not a production captioner.
The question
Can a from-scratch ViT encoder plus a cross-attention Transformer decoder learn to align visual and textual representations well enough to caption images end-to-end?
Key features
- Splits each image into 16x16 patches with a strided convolution and adds learnable position encodings.
- Encodes the patches with a stack of residual multi-head self-attention and feed-forward blocks.
- Decodes captions with a Transformer decoder that cross-attends to the image tokens.
- Ties the decoder's input embedding to its output projection to reduce parameters.
- Generates captions with beam search, length normalization, and top-k / top-p filtering.
- Batches captions by length under a token budget, with a label-smoothing cross-entropy loss that ignores padding.
How it works
Patch the image
A convolution with kernel and stride equal to the patch size turns the image into a sequence of patch embeddings, then learnable position encodings are added.
Encode
Stacked residual self-attention and feed-forward blocks build contextual visual tokens.
Decode with cross-attention
The caption decoder attends to earlier words with masked self-attention and to the visual tokens with cross-attention.
Train
GPT-2 tokenizer with added start, end, and pad tokens; label-smoothed cross-entropy over the real caption tokens; warmup learning-rate schedule and gradient clipping.
Describe
Beam search with length normalization and top-k / top-p filtering picks the best caption hypothesis.
Examples
Tech stack
Every model component, written by hand
Custom Transformers
ViT encoder and cross-attention caption decoder
App scaffold for an interactive demo
Dependency and environment management
GPT-2 tokenizer with added special tokens
Reshaping patch grids into token sequences
Image preprocessing transforms
What I learned
- Cross-attention is the mechanism that lets a decoder condition on a different modality
- Weight tying is a cheap way to cut parameters
Limitations
- The training script in the public repo is unfinished, so no caption-quality results are reported yet