LunarLander RL

Eleven Deep-RL Methods on One Benchmark

Reinforcement learning · Oct 2026 · Completed

DQN to SAC, each as one readable file, landing the same module under the same evaluation.

PPO 276.4 ± 18.2
Continuous lander solved in 1.5 min
DQN 200+ at 50k steps

Built withPyTorchGymnasiumBox2DNumPyMatplotlibpytest

Overview

LunarLander is the step after CartPole: an 8-float state, a sparse +100 for coming to rest, a -100 crash, fuel costs, and a hover plateau that traps many agents. I used it to implement the classic deep-RL methods one file each and compare them honestly: DQN with its double and dueling variants, prioritized replay and 3-step returns, REINFORCE with a baseline, A2C, PPO and discrete SAC on the 4-button lander, and DDPG, TD3, SAC and Gaussian PPO on the 2-throttle continuous one.

Each algorithm is 60-120 lines with its update rule written as equations in the docstring, the networks are small MLPs, and one Config dataclass describes any run. All methods share the environment, the seeds, a Tracker that runs a greedy evaluation every 10,000 steps on fixed unseen episodes, a final 20-episode score, the curves and the comparison table. 49 tests check the buffers, GAE and log-probabilities, and that every agent learns a bandit.

On one CPU thread, PPO is the most reliable (276.4 +- 18.2 after solving at 330,000 steps) and its Gaussian version solves the continuous lander in 1.5 minutes. DQN and dueling DQN solve at 50,000 steps, double DQN at 90,000. A2C reaches 200 and then collapses, REINFORCE climbs slowly, and the off-policy continuous methods are still hovering when their 120,000-step budgets run out.

The question

Which of the classic deep-RL algorithms land the module on one CPU thread in minutes, and what do their curves show about sample efficiency versus stability?

Key features

Eleven methods behind one --method flag: dqn, double-dqn, dueling-dqn (with PER and 3-step returns), reinforce, a2c, ppo, sac-discrete, ddpg, td3, sac and ppo-continuous.
One Config dataclass for every hyperparameter, with presets per method and --set key=value overrides; every run saves its config, weights and metrics.
A shared replay loop for the off-policy methods with uniform or prioritized (sum-tree) replay and n-step returns, and a shared rollout collector with GAE for A2C and PPO on 8-16 vectorised environments.
Truncation bootstrapped everywhere: replay buffers store done only on a real crash or landing, and rollouts add gamma V(final_obs) at the 1,000-step time limit.
One evaluation protocol: greedy actions, periodic 10-episode evaluations on fixed seeds, a final 20-episode score on other seeds, and random and PID-controller baselines on the same episodes.
lunarlander-compare builds the results table and curves against steps and minutes; lunarlander-play and scripts/make_gifs.py record landings, before/after pairs and the solved-methods grid.
49 pytest tests, six example scripts and a Kaggle/Colab notebook that explains each update rule from the package source and reruns the methods with larger budgets.

How it works

The environment
LunarLander-v3 gives an 8-float state (position, velocity, angle, angular velocity, leg contacts). Its reward is a potential-based shaping term minus fuel, plus +100 for resting and -100 for crashing; the dense shaping is what makes the task learnable in 10^5 steps.
Value methods on replay
DQN regresses Q(s, a) on r + gamma max Q_target(s') from a 50,000-transition buffer with a target network copied every 250 updates. Double DQN lets the online net pick a' and the target net score it; dueling DQN adds Q = V + A - mean A, prioritized replay by TD error with importance weights, and 3-step returns.
Policy gradient and actor-critic
REINFORCE pushes up log pi(a|s) by the return-to-go minus a learned V(s) over whole episodes. A2C uses GAE advantages from 16-step rollouts of 8 envs with one gradient step per rollout; PPO collects 16 x 1024 steps and takes 4 epochs of clipped-ratio minibatch updates.
Maximum-entropy and deterministic actors
Discrete SAC sums exactly over the 4 actions with a learned temperature. On the continuous lander DDPG climbs the critic's gradient dQ/da with Gaussian exploration noise, TD3 adds twin critics, target smoothing and a delayed actor, and SAC uses a tanh-Gaussian actor with reward plus alpha times entropy.
Track and compare
Every 10,000 steps the greedy policy plays 10 fixed episodes; a mean of 200 or more counts as solved and can stop training. A final 20-episode evaluation on unseen seeds fills the table, next to the random (-142.7) and PID-controller (265.9 / 288.2) baselines.

Examples

PPO on the discrete lander, same episode seed and network on both sides. Left: the untrained network comes down tilted and crashes (return -531). Right: the trained checkpoint lands between the flags (return 231); 200 or more counts as solved.

Gaussian PPO on the continuous lander (two throttles), untrained on the left and trained on the right. This seed's landing scores 173, below the 200 bar; the run's 20-episode mean is 262.5 +- 35.9.

Every method that solved the task locally, playing the same episode seed: DQN, double DQN, dueling DQN, PPO and continuous PPO. A2C reached 200 but then fell to -145, so it is left out.

TD3 after its 120,000-step CPU budget (final return 3.0 +- 38.4): the lander no longer crashes but hovers and descends without committing to a landing, the classic local optimum of this reward.

Greedy evaluation return against environment steps for the seven discrete-lander methods. PPO stays above 250 after 330,000 steps, the DQN variants solve before 100,000, A2C peaks at 228.9 and collapses.

The continuous lander: Gaussian PPO solves at 300,000 steps while DDPG, TD3 and SAC climb from about -250 to about 0 within their 120,000-step budgets.

Input

lunarlander-eval --baseline heuristic # and --baseline random

Output

heuristic (gymnasium's PID controller): 265.9 +- 35.4 discrete, 288.2 +- 16.9 continuous random actions: -142.7 +- 80.1

Reference policies on the same 20 evaluation episodes as the trained agents, which puts PPO's 276.4 next to the hand-written controller's 265.9.

Tech stack

PyTorch
Networks, actors, critics and every update rule
Gymnasium
LunarLander-v3, discrete and continuous, with SyncVectorEnv for A2C and PPO
Box2D
The lander physics behind the environment
NumPy
Replay buffers, sum tree for prioritized replay, metrics
Matplotlib
Training and evaluation curves against steps and minutes
pytest
49 tests: buffers, GAE, log-probabilities, every agent learns a bandit

What I learned

The 1,000-step truncation must be bootstrapped, not treated as terminal: otherwise a hovering agent learns that hovering is worth zero, which slows escaping the hover plateau
Stopping at the first 10-episode evaluation >= 200 is optimistic: dueling DQN stopped at 212 but scored 167.1 on 20 fresh episodes, which is why the table uses a final evaluation on unseen seeds

Limitations

One seed and short CPU budgets per method: the table shows that each method learns and roughly how fast, not a ranking
The off-policy continuous methods ran with one update every two steps and 128x128 networks to fit one CPU thread, and none of them solved the task in that budget

Keywords

reinforcement learning · LunarLander · Gymnasium · Box2D · DQN · double DQN · dueling DQN · prioritized experience replay · REINFORCE · A2C · PPO · SAC · TD3 · DDPG · PyTorch