Bellam Policy Playground

Q-learning / SARSA / Monte Carlo on a grid world

Algorithm

Off-policy TD learning. Updates toward the greedy action, regardless of what action was actually taken.

Learning Rate (α)0.10

Discount (γ)0.99

ε Decay0.995

ε Min0.01

Current ε1.000

Preset

Tools

Grid Size10×10

Slip Probability0%

0.00

V(s) — State Value

Episodes0

Mean Reward0.00

Mean Length0.0

Mean |TD Error|0.0000

Epsilon (ε)1.000

V(s) Range0.00 → 0.00

Steps/sec100

1101001k10k

Value HeatmapPolicy ArrowsAgent TrailValue Numbers

SpacePlay / Pause

NStep Episode

SStep Action

RReset

Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]

Updates toward the best possible next action (greedy), making it off-policy.

Heatmap: Shows V(s) = max Q(s,a). Brighter teal means higher expected return.

Arrows: Show π(s) = argmax Q(s,a). Opacity indicates confidence in that action.