Ismail Ben Alla

Pong RL

Pong from Pixels with Seven Deep RL Methods

Reinforcement learning · Oct 2026 · Completed

Seven deep RL methods learn Pong from nothing but pixels, side by side on the same frames and the same score.

Built withPyTorchNumPyGymnasiumMatplotlibpytest

View code on GitHub
Pong RL — overview

Overview

Pong from pixels is the problem that made deep reinforcement learning famous, and Karpathy's 'Pong from Pixels' was my starting point: a coursework notebook with his numpy policy gradient and a PyTorch rewrite. Turning it into a package meant fixing eleven problems in that notebook, the main one a loss that gave zero gradient to every DOWN action.

I then built every main method on the same preprocessed frames: Karpathy's policy gradient (an MLP on frame differences), REINFORCE with a learned baseline, A2C with 16 parallel games, PPO with GAE and a clipped ratio, and DQN with double and dueling heads, prioritized replay and n-step returns. Each learning rule is one file of 60 to 120 lines; the game, the CNN, the evaluation and the logging are shared.

Because a pixel agent needs millions of Atari frames to win, I wrote a 40x40 numpy Pong with the same three actions, frame skip and a scripted opponent, preprocessed to 20x20 binary frames. Every method learns it in 2-6 minutes on one CPU thread, which makes the differences visible: the DQN family beats the opponent after 75k-100k steps, PPO after 125k, and the Monte Carlo methods need 175k to 400k. The same code trains on ALE/Pong-v5 in the Kaggle / Colab notebook.

The question

When seven deep RL methods see the same preprocessed frames and are scored by the same evaluation, how much do a learned baseline, bootstrapping and experience replay change the number of samples needed to win at Pong?

Key features

How it works

  1. Preprocess the frame

    ALE frames are cropped, downsampled by 2 and binarised exactly as in Karpathy's prepro (80x80). The small game is downsampled 40x40 to 20x20 with a max over 2x2 blocks so the 2x2 ball can never vanish.

  2. Encode motion

    One frame shows where the ball is, not where it goes. diff subtracts the previous frame so a moving object becomes a +1 blob next to a -1 blob; stack keeps the last 4 frames and lets the network compare them.

  3. Learn, three ways

    pg and reinforce wait for whole games and use Monte Carlo returns (reset at each point), with or without a learned V(s). a2c and ppo bootstrap after 5 or 64 steps with a critic, PPO reusing each batch for 4 epochs under a 10% ratio clip. The DQN family regresses Q to a target network from a replay buffer with a Huber loss and epsilon-greedy acting.

  4. Evaluate on fixed serves

    Copy i of the game is reset with seed 10000 + i, so every method faces the same serves and opponent noise; the policy-gradient agents sample their actions, the DQN family plays argmax Q.

  5. Compare

    scripts/compare_all.py trains the seven methods one at a time and builds the results table, the curves against steps and against minutes, and the one-panel-per-method GIF.

Examples

Same game seed on the small Pong, the agent is the green paddle on the right: a DQN with random weights (left) loses 0-5; the same architecture after 3.6 minutes of DQN training on one CPU thread (right) wins 5-0.
One panel per method on one shared game seed after its run from the results table: the untrained agent loses 2-5, the three DQN variants win 5-0, PPO and pg 5-3, REINFORCE 5-1, A2C 5-4 (the policy-gradient agents sample their actions, so one game is a noisy sample of the 30-game score).
Learning curves on the small Pong, training score (left) and periodic 10-game evaluation (right) against environment steps: the DQN family crosses 0 first, then PPO, then A2C and REINFORCE, with Karpathy's pg last at about 400k steps.
Learning curves on the small Pong, training score (left) and periodic 10-game evaluation (right) against environment steps: the DQN family crosses 0 first, then PPO, then A2C and REINFORCE, with Karpathy's pg last at about 400k steps.
Preprocessing on the small game: the 40x40 raw frame, the 20x20 binary frame, and the frame difference where the moving ball becomes a +1 / -1 pair.
Preprocessing on the small game: the 40x40 raw frame, the 20x20 binary frame, and the frame difference where the moving ball becomes a +1 / -1 pair.
ALE/Pong-v5 with the PPO agent from the 4,096-step local smoke run: barely trained, as expected, it loses; it only shows that the same code runs on the Atari game.

Input

pong-train --algo dqn && pong-eval runs/dqn/model.pt --episodes 30

Output

mean score +4.60 ± 0.55 over 30 games, win rate 100% (180,000 steps, 3.6 min on one CPU thread)

The DQN run behind the results table; the score is points won minus points lost in a game to 5, on fixed serves (seeds 10000-10029).

Tech stack

What I learned

Limitations

Keywords

reinforcement learning · Pong from pixels · Atari · policy gradient · REINFORCE · A2C · PPO · DQN · dueling DQN · prioritized experience replay · n-step returns · frame stacking · Arcade Learning Environment · Gymnasium · PyTorch