Agents, Rewards & Policies

The policy ties them together. A policy $\pi$ is the agent's behavior, a mapping from states to actions. It can be deterministic, $a = \pi(s)$, or stochastic, giving a probability distribution over actions:

$\pi(a \mid s) = \Pr(\text{take action } a \mid \text{in state } s).$

Learning, in RL, means gradually improving $\pi$ so that the rewards it collects over time get larger.

The interaction loop

RL is a loop, not a dataset. At each timestep $t$ the agent observes a state, picks an action, and the environment responds with a reward and a new state. That new state feeds the next decision, and the cycle repeats.

1. 1Observe state s_tThe environment presents the current situation the agent must react to.
2. 2Choose action a_t ~ pi(.|s_t)The policy maps the state to an action (possibly sampling from a distribution).
3. 3Environment returns reward r_{t+1}A scalar score for that action, it may be zero, positive, or negative.
4. 4Transition to next state s_{t+1}The environment's dynamics move the world forward, often stochastically.
5. 5RepeatThe new state becomes the input to the next decision; experience accumulates over the loop.

Formally this is a Markov Decision Process (MDP). The Markov property says the next state and reward depend only on the current state and action, not on the full history:

$\Pr(s_{t+1}, r_{t+1} \mid s_t, a_t) = \Pr(s_{t+1}, r_{t+1} \mid s_t, a_t, s_{t-1}, a_{t-1}, \dots).$

This is a modeling assumption, not a law of nature. When it seems violated (e.g. a single video frame cannot reveal velocity), the usual fix is to enrich the state, stack several frames, so that it becomes Markov.

Return: rewards add up over time

The agent does not care about a single reward; it cares about the total reward it can accumulate. That total is the return $G_t$. The naive version is a plain sum of future rewards:

$G_t = r_{t+1} + r_{t+2} + r_{t+3} + \cdots$

For tasks that never end this sum can be infinite, and even for finite tasks we usually want to prefer reward sooner. Both problems are solved by discounting with a factor $\gamma \in [0,1]$:

$G_t = \sum_{k=0}^{\infty} \gamma^{k}\, r_{t+k+1} = r_{t+1} + \gamma\, r_{t+2} + \gamma^2\, r_{t+3} + \cdots$

Each step into the future is worth a factor $\gamma$ less. The discount factor has a clean interpretation:

• $\gamma = 0$: myopic, the agent maximizes only the next immediate reward.
• $\gamma \to 1$: far-sighted, distant rewards matter almost as much as near ones.
• Typical values are $\gamma = 0.99$ or $\gamma = 0.95$.

A tiny numeric return

Suppose an agent receives rewards $r_{t+1}=0$, $r_{t+2}=0$, $r_{t+3}=1$ (a reward that only arrives after two "empty" steps), and $\gamma = 0.9$. Then

$G_t = 0 + 0.9\cdot 0 + 0.9^2 \cdot 1 = 0.81.$

The same reward one step earlier would be worth $0.9$, and immediately would be worth $1$. This is precisely how discounting nudges the agent to reach reward faster.

Episodic vs continuing tasks

The distinction matters because it changes how you write the return and how you evaluate progress. Many practical setups also impose a time limit on an otherwise continuing task, turning it into an episodic one for convenience, but you must then be careful that hitting the time limit is treated as a "cutoff," not as a genuine terminal state.

Reward design is the hard part

The reward signal defines what you want, and the agent will relentlessly optimize exactly what you wrote, not what you meant. Two failure modes dominate practice.

Sparse rewards. If reward only arrives at the very end (win/lose, task complete), the agent gets almost no feedback along the way and can flail for a long time before stumbling onto any signal. Techniques like reward shaping, curricula, and exploration bonuses exist largely to combat sparsity.

Reward hacking / specification gaming. If the reward is a proxy for the true goal, the agent may maximize the proxy while defeating the intent. Classic examples: a boat-racing agent that farms a recurring bonus by driving in circles instead of finishing the race; a cleaning robot rewarded for "no visible mess" that learns to cover its sensors. This is a manifestation of Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

Where this leads

We now have the objective: find a policy $\pi$ that maximizes expected discounted return $\mathbb{E}_\pi[G_t]$. The next question is how. Most methods either estimate how good states and actions are (value functions, next lesson) or adjust the policy directly (policy gradients, later). Both rest entirely on the loop and the return you just met.

Further reading

• Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), Chapters 1 and 3 cover the agent–environment interface, returns, and MDPs: http://incompleteideas.net/book/the-book.html
• OpenAI Spinning Up, "Key Concepts in RL", a concise, code-oriented tour of states, actions, policies, and returns: https://spinningup.openai.com/en/latest/spinningup/rl_intro.html
• Amodei et al., "Concrete Problems in AI Safety", the canonical catalogue of reward hacking and specification gaming: https://arxiv.org/abs/1606.06565
• DeepMind, "Specification gaming: the flip side of AI ingenuity", a readable gallery of real reward-hacking examples: https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/