Text2Image

Twelve Text-to-Image Architectures Side by Side

Generative AI · Oct 2026 · Completed

The main ways images are generated from text today, implemented with the same data, text encoder, loop and scorer, so only the architecture changes.

13 models compared
U-Net: 100% attribute accuracy
One-step sampling in 9 ms

Built withPyTorchCustom TransformersNumPypytest

View code on GitHub

Overview

Text-to-image systems are built from a handful of architecture families, from the Stable Diffusion U-Net to FLUX-style MMDiT, one-step MeanFlow, Transfusion-style unified models, VAR's next-scale prediction and MAR's continuous-token autoregression, and the papers describe them at a scale where nothing can be compared on a laptop. I implemented twelve of them side by side in PyTorch, each small enough to train on a CPU, behind the same interface: model.loss(images, tokens) and model.generate(tokens, cfg_scale).

Every model shares the same data, text tokenizer and encoder, training loop (AdamW, warm-up plus cosine schedule, EMA weights, gradient clipping) and classifier-free guidance, so the only thing that changes between runs is the architecture. The benchmark is a procedural captioned-shapes task: one coloured shape on a dark background at 32x32, described by 432 prompts over four independent attributes (size, colour, shape, position). A small CNN oracle reads those attributes from an image at about 100% accuracy and scores generated images per attribute and on all four at once, a toy version of GenEval.

After 2,000 steps at batch 32, the v-prediction U-Net gets every attribute right (100%), the token models follow (MaskGIT 77%, MAR 72%, VAR 65%) and the pixel-space flow transformers get the layout right but blur the shapes (DiT 25%, MMDiT 34%). The prediction target turned out to be decisive: the same U-Net with ε-prediction scores 3%, and JiT's clean-image prediction doubles DiT's score. MeanFlow matches DiT with one network evaluation instead of 30.

The question

When the data, text encoder, training loop and evaluation are held fixed, how do the main text-to-image architectures compare at binding every word of a caption to the right visual property?

Key features

Six continuous-pixel models: U-Net + DDPM with v or ε prediction, DiT with cross-attention (PixArt-α), JiT (x-prediction), MMDiT (SD3 / FLUX double-stream), a single-stream DiT (FLUX single blocks, Z-Image) and SANA with linear attention, Mix-FFN and no positional embedding.
MeanFlow, which learns the average velocity with a forward-mode JVP target and generates in one network evaluation, with guidance folded into training.
Transfusion: one transformer that reads [caption; noisy patches] with a causal mask on text and bidirectional attention on the image, trained as a caption language model and an image denoiser at once.
Four token models over a VQ-VAE: next-token autoregression, MaskGIT's confidence-based parallel decoding, VAR's next-scale prediction over a multi-scale residual quantizer, and MAR's continuous tokens with a per-token flow head.
A shared word tokenizer and small transformer text encoder (a CLIP / T5 stand-in), one training loop and classifier-free guidance for every family.
Automatic evaluation with a CNN oracle: per-attribute accuracy and the fraction of images with all four attributes correct, plus a one-command script that trains the oracle, the tokenizers and all 13 models and builds the results folder.
44 tests on shapes, gradients, sampling, tokenizers, the JVP, attention masks and overfitting, and a literature survey that maps each 2025-2026 production system to the file that implements its idea.

How it works

Render and caption
Each training image is one shape drawn procedurally at 32x32 from a prompt such as 'a small blue square at the top left', with random jitter in position and scale; a pre-rendered bank keeps the CPU loop fast.
Encode the text
A word tokenizer and a small transformer produce a token sequence for cross or joint attention and a pooled vector for adaLN modulation. 10% of prompts are replaced by the null prompt so that guidance works at sampling.
Train with one loss per family
Diffusion and flow models regress ε, v, x0 or a velocity on noised pixels; MeanFlow regresses the average velocity through the MeanFlow identity; token models predict VQ codes (next token, masked tokens or next scale) or run a flow head per continuous token.
Generate with guidance
Every generate() combines conditional and unconditional outputs (noise, velocity, logits or flow-head velocity) at scale 3.0; MeanFlow samples in one step, MaskGIT in about 12, the diffusion models in 30.
Score with the oracle
A CNN trained on real images reads colour, shape, position and size from each generated image for 128 random prompts; a model only scores on 'all 4 correct' if it binds every word to the right property.

Examples

Every model on the same 8 prompts, from 'a large red circle at the center' to 'a small red cross at the bottom left'. The U-Net draws them exactly; the pixel-space flow transformers place them right but blur them; ε-prediction and SANA lose the position.

Samples of the v-prediction U-Net, the best model at this scale: 100% on colour, shape, position and size after 2,000 CPU steps, and the only model that draws the small crosses crisply.

Samples of MaskGIT, the best transformer: 77% of images have all four attributes right, with the VQ decoder handling fine detail and the transformer choosing 64 codes in about 12 parallel steps (40 ms per image).

Samples of MeanFlow generated with a single network evaluation (9 ms per image): 25% all-correct, the same score as DiT with 30 steps on the same backbone.

Input

transfusion.complete_caption("a small blue")

Output

"a small blue square at the top right"

The Transfusion model is also a caption language model: the same transformer that denoises image patches completes captions grammatically.

Input

text2image-eval --ckpt runs/unet/model.pt --oracle runs/oracle/model.pt --n 128

Output

color 100% shape 100% position 100% size 100% all 4 correct 100%

The U-Net row of the benchmark table; the same command with runs/unet-eps/model.pt gives 62% / 38% / 11% / 60% and 3% all correct, the only difference being the regression target.

Tech stack

PyTorch
All twelve architectures, the VQ-VAE, the text encoder and the training loop
Custom Transformers
DiT, MMDiT, single-stream, SANA, Transfusion and the token models, written by hand
NumPy
Procedural rendering of the captioned-shapes dataset
pytest
44 tests, including the JVP of MeanFlow and the attention masks of Transfusion and VAR

What I learned

Convolutions still have the right inductive bias for small, pixel-exact shapes; transformers trade it for scalability that only pays off far beyond this task
Token models with a VQ decoder are the strongest transformers at equal steps, because the transformer only has to pick 64 codes
SANA's no-positional-embedding design needs depth: with 4 blocks on an 8x8 grid, position stays at chance

Limitations

Toy scale: 32x32 pixel space, about 1 M parameters, 2,000 CPU steps, scores on 128 prompts with about ±4% standard error, and most models have not converged; rankings at production scale (billions of parameters, VAE latents) differ

Keywords

text-to-image · Stable Diffusion · U-Net · DiT · MMDiT · FLUX · SANA · JiT · MeanFlow · Transfusion · autoregressive image generation · MaskGIT · VAR · MAR · VQ-VAE · flow matching · rectified flow · classifier-free guidance · v-prediction · GenEval · PyTorch