MountainCar RL

Every RL Family on One Sparse-Reward Task

Reinforcement learning · Oct 2026 · Completed

A car too weak to climb the hill, and every RL family trying to learn the rocking motion that gets it to the flag.

100% success with tile coding
Novelty bonus: DQN 0% → 100%

Built withPyTorchGymnasiumNumPyMatplotlibpytest

Overview

MountainCar gives -1 per step until the flag, so every failed episode scores exactly -200 and a learner gets no hint until it reaches the flag once. The engine is weaker than gravity, so the car must rock back and forth to build energy. It is the classic exploration problem, and small enough that every method trains on one CPU thread in seconds to a few minutes.

I wrote one small code base where every method is an Agent with the same five methods and one training loop runs all of them: discretized Q-learning and SARSA, tile coding, DQN / Double / Dueling DQN with prioritized replay and n-step returns, REINFORCE, A2C, PPO, DDPG, TD3, SAC and the cross-entropy method. Exploration aids (potential-based energy or height shaping, count and RND bonuses, epsilon schedules) are environment wrappers switched on by flags, so the algorithms never know whether the reward was shaped, and every table reports the true reward.

The comparison answers one question: which methods find the flag by themselves and which ones need help. Tile coding and CEM solve the discrete task in about 25 episodes with no aid. DQN, Dueling DQN, REINFORCE and A2C never reach the flag on the raw reward in their budget; with energy shaping every method solves it, and a count-based novelty bonus that knows nothing about physics does the same for DQN. In the continuous version PPO, DDPG, TD3 and SAC all converge to doing nothing without shaping, because the only signal before a success is the action cost.

The question

On a task where every failed episode scores exactly -200, which reinforcement learning methods discover the goal by themselves, and what does a dense signal (potential-based shaping or a novelty bonus) change for the ones that do not?

Key features

Fourteen methods behind one Agent interface and one training loop: Q-learning, SARSA, tile coding (SARSA or Q target), DQN, Double DQN, Dueling DQN, REINFORCE, A2C, PPO, DDPG, TD3, SAC and CEM.
Both environments: MountainCar-v0 (three actions, 200-step limit) and MountainCarContinuous-v0 (force in [-1, 1], 999 steps), with PPO and CEM running on either.
Exploration aids as flags: potential-based energy or height shaping, count bonus on a 40x40 grid, Random Network Distillation, and constant / linear / exponential epsilon schedules.
Prioritized replay on a sum tree and an n-step accumulator for the DQN family, with a test that the high-error transition is the one sampled.
One evaluation protocol for everyone: greedy policy, 20 fixed start states, true reward, reporting success rate, mean return, mean steps and episodes to first success.
Four CLIs (mountaincar-train, -eval, -play, -compare) that produce the curves, visitation / value / policy maps, tables and the README GIFs; 51 tests run in about a minute.
A Kaggle / Colab notebook built from the source that explains each update step by step and runs the full comparison over three seeds.

How it works

Represent the state
The state is (position, velocity). Tabular methods use a 16x16 grid, tile coding uses 8 offset 8x8 tilings with a linear Q, and the neural methods take the state normalised to [-1, 1]^2.
Wrap the reward (optional)
ShapingWrapper adds gamma * Phi(s') - Phi(s) with Phi the mechanical energy or the height, which leaves the optimal policy unchanged. IntrinsicWrapper adds 0.1 / sqrt(N(cell)) or an RND prediction error. The true reward is kept in info and is what every table shows.
Run one loop for every agent
train.py calls act and observe on every method: one table entry for Q-learning, 8 weights for tile coding, one minibatch step for DQN / DDPG / TD3 / SAC, or a stored transition until a rollout is full for A2C, PPO, REINFORCE and CEM.
Evaluate greedily on fixed starts
evaluate.py resets with seeds 1000-1019, turns off epsilon, action noise and shaping, and records success rate, return and steps, so every method faces the same start states.
Compare
compare.py runs a suite (tabular, discrete-deep, exploration, continuous), writes the markdown tables and draws the training curves, visitation heatmaps and policy maps from the saved runs.

Examples

MountainCar-v0 from the same start state: a random agent (left) never builds enough energy and times out at 200 steps; after 300 episodes of tile coding (right) the car swings left first and reaches the flag at step 107.

MountainCarContinuous-v0: a random agent (left) against SAC with energy shaping after 30 training episodes (right), which rocks once and climbs to the flag.

Every method that solved the discrete task, greedy policy, same start state: eleven panels reach the flag between steps 107 and 169; Q-learning (55% greedy success) happens to time out on this start.

How tile coding learns: the greedy action over (position, velocity) every 4 episodes. It starts as noise, then the 'push with the velocity' spiral (red = left, blue = right) grows out from the flag as successes propagate back.

Tile coding (Q target) after 300 episodes: where the car went during training, the learned max Q, and the greedy policy, which splits the plane into 'push left when moving left, push right when moving right'.

Training visitation of the six neural methods with and without energy shaping: the failing runs stay in a blob around the valley, the working ones trace spirals out to the flag (dashed line).

Continuous control curves: without shaping PPO, DDPG, TD3 and SAC sit at return 0 (doing nothing); with energy shaping all four reach 90-95 within a few episodes, and CEM solves it unaided.

Input

mountaincar-eval runs/tabular/tile-coding_q --episodes 100

Output

success 100%, mean return -118.9 (std 15.2), 119 steps

A longer greedy evaluation of the tile-coding agent trained for 300 episodes (6 seconds on one CPU thread).

Tech stack

PyTorch
Q-networks, policies and critics for DQN, policy gradient and actor-critic methods
Gymnasium
MountainCar-v0 and MountainCarContinuous-v0 from classic-control, plus the shaping and intrinsic wrappers
NumPy
Q-tables, tile coder, CEM and the sum-tree replay
Matplotlib
Training curves, visitation / value / policy maps and the animated policy map
pytest
51 tests: shaping telescopes, prioritized sampling, every method trains and reloads, tile coding solves the task

What I learned

Optimistic initialisation is exploration: a zero-initialised table or tile weights sit above every real return, so a greedy agent tries every untried action; a Q-network has no such property
Potential-based shaping telescopes into a baseline inside a Monte Carlo return, so it helps every bootstrapping method but barely helps REINFORCE

Limitations

Single-seed short CPU runs on a shared machine: they show which methods learn under a small budget, not benchmarks; the unshaped Double DQN and PPO successes may not repeat with another seed
Results marked +shaping use knowledge of the physics and are not a fair comparison with the unshaped methods

Keywords

reinforcement learning · MountainCar · sparse reward · exploration · Q-learning · tile coding · DQN · PPO · SAC · cross-entropy method · reward shaping · Random Network Distillation · prioritized experience replay · Gymnasium · PyTorch