CartPole RL

Twelve RL Methods on One Benchmark

Reinforcement learning · Oct 2026 · Completed

Every classic reinforcement-learning method, from a Q table to PPO and pixel CNNs, balancing the same pole under the same evaluation.

12 methods solve CartPole
PPO solved in 15k steps
Q-table solves in 4 s

Built withPyTorchGymnasiumNumPyMatplotlibpytest

View code on GitHub

Overview

CartPole is the simplest control task in Gymnasium: push a cart left or right so the pole stays up, one point per step, 500 steps at most. I used it as a bench to implement every classic reinforcement-learning method myself and see how each one actually gets there: tabular Q-learning, SARSA and Monte Carlo control on a binned state, DQN with its double, dueling, prioritized-replay and n-step variants, REINFORCE with and without a baseline, one-step actor-critic, A2C and PPO, and finally CNN policies that only see rendered frames.

Everything shares one environment, one Recorder that counts steps and runs a greedy evaluation every 5,000 steps, one definition of solved (a 10-episode candidate at >= 475 confirmed on 100 more episodes), and the same plots. Each update rule is a few lines next to a comment that says why it is there, and 53 tests check the pieces (reward-to-go, GAE at lambda 0 and 1, the double-DQN target, the dueling head, PPO's clip) against numbers worked out by hand.

On state observations all twelve methods solve the task. PPO and REINFORCE + baseline do it after 15,000 steps, tabular Q-learning after 40,000 steps but only 4 seconds of CPU, and plain DQN after 50,000 steps with a long plateau that the double and dueling variants do not show. The pixel runs are the honest part: 50,000 CPU steps are far from enough, and only the frame-difference agents move above random play at all.

The question

If every classic RL method runs on the same environment with the same greedy evaluation, what does each learning curve reveal about why its ingredients (replay, target networks, baselines, clipping) exist?

Key features

Twelve methods behind one --method flag: q-learning, sarsa, mc-control, dqn, double-dqn, dueling-dqn (plus --prioritized and --n-step), reinforce, reinforce-baseline, actor-critic, a2c and ppo.
Three observation modes: the 4-number state, a 40x80 grey rendered frame, or the frame plus its difference from the previous one, with MLP, CNN and residual-CNN bodies.
One Recorder for every method: step counting, periodic greedy evaluation on fixed start states, confirmed solve, and the best evaluated weights restored at the end.
Truncation handled correctly everywhere: the 500-step time limit bootstraps from V(s') while a fallen pole uses 0.
cartpole-compare trains any set of methods and produces the results table, return curves and steps-to-solve chart; cartpole-play records before/after GIFs on the same start state.
53 pytest tests (about 40 s on one CPU thread) covering the update rules, environment wrappers, networks, checkpoint round trips and the CLIs.
A Kaggle/Colab notebook that explains each family before training it and repeats the comparison over three seeds, plus the original coursework notebooks with a table of the 15 bugs found in them.

How it works

One MDP, three views of it
CartPole-v1 gives +1 per step until the pole leans past 12 degrees, the cart leaves the track, or 500 steps pass. Agents see the raw state, a cropped rendered frame, or the frame with its difference image; tabular methods bin the state into 1 x 6 x 12 x 12 = 864 cells.
Tabular TD and Monte Carlo
A 864 x 2 table is updated with the Q-learning target r + gamma max Q(s', .), the SARSA target using the action actually taken next, or the real episode return for Monte Carlo control, with epsilon decaying from 1.0 to 0.02.
Deep value methods
A 4-128-128-2 Q network trains on a replay buffer with a Polyak-averaged target network and a Huber loss. Flags switch on the double target (online net picks a', target net scores it), the dueling head Q = V + A - mean A, prioritized replay with importance weights, and n-step returns.
Policy gradient and actor-critic
REINFORCE pushes up log pi(a|s) weighted by the normalized reward-to-go, optionally minus a learned baseline V(s). One-step actor-critic uses the TD error as advantage after every step; A2C steps 8 environments for 8 steps and takes one combined actor, critic and entropy step.
PPO
8 envs x 32 steps per rollout, GAE advantages (gamma 0.98, lambda 0.8), then 20 epochs over the batch with the probability ratio clipped to [0.8, 1.2] so repeated updates cannot move the policy too far from the one that collected the data.
Same evaluation for everyone
Every 5,000 steps the greedy policy plays 10 fixed start states; a mean >= 475 is confirmed on 100 more episodes, and the first confirmed point gives steps, episodes and seconds to solve. A final 100-episode evaluation on unseen start states fills the table.

Examples

Before and after training on the same start state. Left: a random agent, the pole falls at step 64 (random play averages about 22). Right: the trained PPO agent balances for the full 500 steps.

One trained agent per family on that same start state: the random agent falls at step 64 while the Q table, double DQN, REINFORCE + baseline, A2C and PPO checkpoints all reach 500 steps.

Training return (lines, exploration included) and greedy evaluation (dots) against environment steps for every state-based method. A2C's collapses around 35k, 65k and 80k steps and plain DQN's plateau at 130-160 are visible.

Environment steps until the first confirmed solve, one seed: PPO and REINFORCE + baseline at 15,000, dueling DQN, REINFORCE and actor-critic at 20,000, up to Monte Carlo control at 115,000.

The four pixel runs over 50,000 steps. PPO on a single frame stays at random level; REINFORCE + baseline on frame differences climbs to about 55 training return, and the PPO frame-difference runs only start drifting up after 40,000 steps.

Input

cartpole-train --method ppo --record ppo.gif

Output

[ppo/state] step 15000 ep 183 train(last20) 264.0 eval 500.0 +- 0.0 22.0s confirm(100) 498.1

The line printed at the evaluation where PPO solved: 15,000 steps, 183 episodes, 10-episode greedy mean 500.0, confirmed with 498.1 over 100 more episodes. The exploring training return is still 264 at that point.

Tech stack

PyTorch
Q networks, policies, CNN bodies and every update rule
Gymnasium
CartPole-v1 with pygame rendering for the pixel modes
NumPy
Q tables, replay buffers and the state discretizer
Matplotlib
Return curves and steps-to-solve charts
pytest
53 tests of the update rules, environment, networks and CLIs

What I learned

A single rendered frame carries no velocity, so a pixel agent on one frame stays at random level; adding the frame difference as a second channel is what lets it improve at all
A2C's training return collapsed three times before settling, which is the failure PPO's clipped ratio was designed to prevent: PPO solves in 15,000 steps with the same network

Limitations

One seed per method and short runs: methods that solve within one or two 5,000-step evaluation intervals of each other can swap order with another seed
The pixel runs are 50,000 steps on a CPU and none comes close to solving; the notebook extends them to 200,000 steps

Keywords

reinforcement learning · CartPole · Gymnasium · Q-learning · SARSA · Monte Carlo control · DQN · double DQN · dueling DQN · prioritized experience replay · REINFORCE · actor-critic · A2C · PPO · PyTorch