Lab
Q-Learning Gridworld
Watch tabular Q-learning find a path to the goal by trial and error.
An agent that learns by trial and error. It starts bottom-left and is rewarded only for reaching the goal top-right (pits are punished). Press Train and the arrows, the agent's current best guess in each square, snap into a route as Q-learning propagates value back from the goal.
Policy & greedy path
Return per episode
Train to plot how much reward each episode earns.
Q-learning stores a value for every (square, move) pair. Each step it nudges that value toward the reward it just got plus the discounted best value of where it landed: Q(s,a) ← Q(s,a) + α·(r + γ·maxₐ′ Q(s′,a′) − Q(s,a)). Value seeps outward from the goal, one square per episode-ish, until following the highest value from any square walks you home.
ε is how often the agent explores at random instead of exploiting what it knows, too low and it never discovers the goal, too high and it never commits. γ is how much it values future reward over immediate reward.
Honest note: a 25-state grid with a known reward is the simplest possible RL problem. Real environments have vast or continuous state spaces where the table is replaced by a function approximator, but the update rule you see here is the same idea.
Challenge
Switch to the grid with pits and set ε to 0. Does the agent still find the goal, or does it get stuck? Now raise ε and retrain, how much exploration does it take to reliably route around the pits?