Pendulum RL

Swing-up and Acrobot with Every Classic RL Method

Reinforcement learning · Oct 2026 · Completed

Twelve reinforcement learning methods swing a pendulum and an acrobot up, under the same budget, so their differences can be seen.

SAC −119.0 after 15k steps
16 runs in 31 min
40 tests

Built withPyTorchGymnasiumNumPyMatplotlibpytest

View code on GitHub

Overview

Pendulum-v1 is the smallest continuous-control problem: a motor too weak to lift the rod directly, so the agent must rock it to gain energy, then brake and hold it upright. Acrobot-v1 is its discrete cousin, two links with a motor at the elbow and a -1 reward every step until the tip clears a line. I wanted to see how each classic method handles them, not only its score.

I wrote one PyTorch code base with one file per family: DDPG, TD3 and SAC with automatic entropy tuning for the continuous torque; PPO with GAE and clipping, A2C and REINFORCE on both tasks; the cross-entropy method as a gradient-free baseline; DQN, double and dueling DQN on Acrobot and a discretized DQN on the pendulum; and tile-coded SARSA(lambda) with eligibility traces and no neural network. Every method runs through the same Monitor, which counts environment steps and evaluates the deterministic policy on the same start states.

On Pendulum the three off-policy actor-critics solve it in 5-6k steps because they replay every transition; PPO needs about four times more steps, REINFORCE 320k and CEM 1.4M. On Acrobot tile-coded SARSA(lambda) is the fastest to the -100 threshold at 12,500 steps and 12 seconds, and A2C has the best final return. Value heatmaps show that every critic finds the same diagonal band of states that coast to the top, and every policy is close to bang-bang.

The question

How do off-policy actor-critics, on-policy policy gradients, black-box search and value methods compare on the same two classic-control tasks when they share one environment, one budget unit (environment steps) and one evaluation?

Key features

Twelve methods in one code base: DDPG, TD3, SAC, PPO, A2C, REINFORCE, CEM, DQN, double DQN, dueling DQN, discretized DQN and tile-coded SARSA(lambda), 16 (environment, method) pairs in all.
One shared off-policy loop (replay buffer, warm-up, Polyak targets) and one on-policy loop (rollouts, GAE, advantage normalisation, gradient clipping), so a method's file holds only its learning rule.
Budgets counted in environment steps for everyone, with a Monitor that evaluates the deterministic policy every few thousand steps on fixed start seeds.
Figures of what each agent learned: V(s) and torque heatmaps over (theta, theta_dot), return curves with a std band, and headless GIFs via the SDL dummy driver.
Replay buffer that stores terminated separately from truncated, with GAE that bootstraps from the final observation after a time limit.
Four CLIs (pendulum-train, -eval, -play, -compare) where every TrainConfig field is a flag, tuned defaults per (environment, method) pair, and a run_all.sh that reproduces every number.
40 fast tests plus an opt-in SAC swing-up check, six numbered examples, and a Kaggle / Colab notebook built from the source that explains each method and trains it.

How it works

Build the environment
make_env returns Pendulum-v1 (observation cos theta, sin theta, theta_dot; torque in [-2, 2]) or Acrobot-v1 (six numbers, three elbow torques). A DiscretizedAction wrapper cuts the pendulum torque into 5 values for DQN.
Pick a loop shape
Off-policy agents act, store the transition and take one gradient step on a random batch. On-policy agents collect a rollout, compute GAE advantages, update and discard the data. CEM runs whole episodes per candidate weight vector; SARSA updates every step through eligibility traces.
Learn
DDPG climbs Q(s, mu(s)); TD3 adds twin critics, delayed actor updates and target smoothing; SAC maximises reward plus alpha times entropy with alpha tuned to a target entropy of -1; PPO clips the probability ratio to 0.8-1.2 over 10 epochs; DQN variants regress to a target network with double and dueling fixes.
Monitor on the same axis
The Monitor counts environment steps, logs each training return and runs 5 evaluation episodes every eval_every steps with the deterministic policy on seeds 10000 and up, the same start states for every method.
Compare and inspect
pendulum-compare trains every method, writes the results table (steps to threshold, final return over 20 episodes, wall time), the curves and the value / torque heatmaps evaluated on a 61x61 grid of states.

Examples

Pendulum, SAC, same hanging start: the untrained actor (left) never gets the rod upright (return -1846); after 15,000 training steps (right) it rocks to gain energy, then brakes and holds it upright (-244).

Acrobot, A2C: the untrained policy (left) never reaches the line in 500 steps; the trained one (right) swings the tip over it in 88 steps.

Every Pendulum method from the same hanging start, the hardest one, where about -250 is the best possible score: SAC reaches -244, DDPG and TD3 about -340, CEM -530.

Every Acrobot method from the same start: all eight swing the tip above the line, in 64 (REINFORCE) to 96 (CEM) steps.

What each Pendulum agent learned: V(s) over (theta, theta_dot) with theta = 0 upright (top) and the torque (bottom). Every critic finds the same diagonal band of states that coast to the top; every policy is nearly bang-bang.

Pendulum evaluation return against environment steps: SAC, DDPG and TD3 cross -200 by 5-6k steps, PPO by 20k, REINFORCE by 320k and CEM by 1.4M.

Acrobot evaluation return: tile-coded SARSA(lambda) reaches -100 first at 12,500 steps; all eight methods reach the threshold during training.

Input

pendulum-train --env pendulum --algo sac && pendulum-eval runs/pendulum/sac/model.pt --episodes 20

Output

final return -119.0 ± 82.5 over 20 episodes (15,000 steps, 1.8 min on one CPU thread)

The SAC run behind the results table. The ± 82 comes from the start states, not the policy: a start near the top scores about 0, a hanging start about -250.

Tech stack

PyTorch
Actors, critics, Q-networks and the policy-gradient losses
Gymnasium
Pendulum-v1 and Acrobot-v1 from classic-control, plus the DiscretizedAction wrapper
NumPy
CEM's policy and search, the tile coder and SARSA(lambda) weights
Matplotlib
Return curves, value / torque heatmaps and GIF frames
pytest
40 tests on GAE, truncation, the squashed-Gaussian log-probability, every (env, algo) pair and the CLIs

What I learned

Terminated and truncated must be handled separately: Pendulum episodes end on a clock, so value targets must still bootstrap from the final state
REINFORCE on Acrobot needed a time-indexed baseline: pooled return normalisation turned identical failed episodes into a fake early-versus-late signal that drowned the rare success
Linear learning-rate annealing stopped PPO from collapsing from -126 to -980 in one late update

Limitations

Single-seed, short CPU runs: a different seed can change the ranking of close methods, and I did not have the budget to check whether double DQN's Acrobot final mean (-108) is the seed or the method
The learning-rate annealing added for PPO made A2C's Pendulum result worse (-188.0, a few failed swing-ups); it is kept for consistency and reported as measured

Keywords

reinforcement learning · continuous control · Pendulum swing-up · Acrobot · DDPG · TD3 · SAC · PPO · GAE · cross-entropy method · DQN · SARSA(lambda) · tile coding · Gymnasium · PyTorch