Reinforcement Learning

Reinforcement learning education is the study of choice under consequences. Instead of handing a model the “right answers,” RL asks the learner—human and machine alike—to discover strategy through interaction: try, fail, adjust, and try again, until behavior becomes skill. This diagram shows how that discovery is taught without letting it become reckless. The inputs—environment models and reward signals—define the world the agent will live in and the feedback it will receive, shaping what “success” even means. The controls—curriculum topics and safety/ethics criteria—act like a compass and a brake: they structure progress, demand evidence, and remind students that an agent can optimize the wrong thing with alarming efficiency if goals and constraints are careless. Inside the central function, learners practice the real craft of RL: designing state and action spaces, shaping rewards responsibly, managing exploration, diagnosing instability, and separating genuine learning from lucky streaks. The mechanisms—simulators and compute tools—provide the training ground where thousands of trials can occur safely, where experiments can be repeated, and where insights can be earned rather than assumed. The outputs, then, are not just “agents that perform,” but practitioners who understand why they perform—and who can build policies that are effective, testable, and aligned with real objectives in the real world.

This IDEF0 (Input–Control–Output–Mechanism) diagram summarizes Reinforcement Learning (RL) Education as a structured learning system. Inputs on the left—environment models, reward signals, etc.—represent the foundations of RL, where learning is driven by interaction, feedback, and sequential decision-making. Controls at the top—curriculum topics, safety/ethics criteria, etc.—define what is taught and how competence is evaluated, while also imposing constraints that encourage responsible design and experimentation. The central function, Reinforcement Learning (RL) Education, transforms these inputs and controls into applied skill through guided study, experimentation, and iterative improvement of agents and policies. Outputs on the right—RL practitioners, trained policies, etc.—represent intended outcomes: learners who can design, train, and evaluate RL systems, and learned strategies that perform tasks effectively. Mechanisms at the bottom—simulators, compute tools, etc.—provide the enabling environment, including simulation platforms and computational resources used to run experiments safely and at scale.

Reinforcement Learning (RL) is a dynamic area within artificial intelligence and machine learning that models learning through interaction and feedback. Unlike approaches in data science and analytics, which extract patterns from static datasets, RL agents discover behaviours by acting in an environment and receiving rewards or penalties. This trial-and-error process enables strong performance in complex decision-making tasks such as game playing, autonomous navigation, and closed-loop control.

RL connects deeply with other AI areas: deep learning provides powerful function approximators; computer vision and natural language processing (NLP) give agents perception and language grounding. At scale, RL systems run on cloud computing with fit-for-purpose cloud deployment models.

In applied settings, RL is pivotal for robotics and autonomous systems, where agents must act safely under uncertainty. In smart manufacturing and Industry 4.0, it drives adaptive control and process optimisation; in IoT and smart technologies, RL closes feedback loops across networks of connected devices.

RL complements—and differs from—other paradigms. In supervised learning, models learn from labelled examples; in unsupervised learning, they discover structure in unlabelled data. RL instead builds policies from experiential feedback. It also augments expert systems by refining rule-based decisions with adaptive learning.

The relevance of RL is expanding alongside emerging technologies. It already supports space exploration technologies and satellite technology, and may ultimately benefit from advances in quantum computing, including qubits, superposition, and quantum gates.

Rooted in STEM and enriched by information technology, RL equips students to work at the frontier of intelligent decision-making—whether optimising supply chains, training self-driving vehicles, or supporting autonomy in critical systems.

Core Principles of Reinforcement Learning

Agents, Actions, and States

• Agent: The decision-making entity, such as a software program controlling a virtual character, or a physical robot navigating a room.
• States: The agent observes the state of the environment at each step. A state may include the agent’s position, objects it interacts with, or other relevant features.
• Actions: At any given state, the agent chooses an action from a set of possible actions. The outcome of that action changes the environment’s state and affects future decisions.

Rewards and Penalties

The environment provides feedback in the form of a numerical reward, which can be positive or negative. When the agent performs an action that advances its goals, it receives a positive reward; if it takes a counterproductive action, it might receive a penalty or a lower reward. Over time, the agent learns to associate certain actions in certain states with higher cumulative rewards.

Goal of Maximizing Cumulative Reward

The objective in reinforcement learning is not just to achieve an immediate reward but to maximize the sum of rewards over the long run. This focus on cumulative gain encourages the agent to develop strategies that balance short-term gains against long-term benefits. For instance, the agent might delay immediate gratification to secure more substantial rewards later.

Exploration vs. Exploitation

A key challenge in RL is the trade-off between exploration—trying new actions to discover potentially better rewards—and exploitation—using known strategies that have previously yielded good results. Finding an appropriate balance allows the agent to continually improve without getting stuck in suboptimal patterns of behavior.

Fundamental Techniques and Algorithms

This page covers the main families of RL methods. Use the quick map below to jump into the detailed sections you’ve added.

Value-Based Methods

Examples include Q-Learning, Double DQN, and Dueling DQN.

• Learn action-values, written as Q(s, a), and act greedily from the value estimates.
• Work well for discrete actions and tabular or pixel tasks with replay buffers.

Jump to diagnostics

Policy-Gradient Methods

Examples include REINFORCE and PPO.

• Directly optimise a stochastic policy, written as π_θ(a | s), and handle continuous actions naturally.
• Use baselines and advantages to reduce variance; PPO adds clipping for stable updates.

See Policy Gradients & Actor–Critic

See Exploration & Intrinsic Motivation

Actor–Critic Methods

Examples include A2C, A3C, PPO, and SAC.

• The actor chooses actions, while the critic estimates value or advantage for lower-variance updates.
• SAC adds entropy regularisation and a learned temperature for smooth continuous control.

Actor–Critic details

Exploration cookbook

Model-Free vs. Model-Based

• Model-Free RL: learn policy or values directly from experience. It is usually simpler and robust, but more data-hungry.
• Model-Based RL: learn dynamics or reward models to plan or imagine data. It can be more sample-efficient, but needs careful model design and validation.

See Model-Based RL & World Models

MPC + CEM pseudo-code

Method comparison

Tip: If visitors only need a primer, keep this overview. For advanced readers, the deeper sections on exploration, shaping, curriculum, model-based learning, and diagnostics provide the full playbook.

Mathematical Foundations: MDPs & Bellman Equations

Markov Decision Processes (MDPs)

Reinforcement learning tasks are often modelled as a Markov Decision Process, or MDP. An MDP may be written as:

$$\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, P, R, \gamma \rangle$$

Here, S is the set of possible states, A is the set of possible actions, P(s’ | s, a) describes the transition dynamics, R(s, a) represents the reward, and γ ∈ [0,1) is the discount factor. Some problems are episodic, meaning they end at a terminal state. Others are continuing, meaning the agent keeps acting without a clear final endpoint.

Returns, Policies, and Value Functions

• Return: the discounted future reward from a given time step:

  $$G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1}$$

• Policy: the rule used by the agent to choose actions. A policy may be stochastic, written as π(a | s), or deterministic, written as μ(s).
• State value: the expected return starting from state s and following policy π:

  $$V^{\pi}(s) = \mathbb{E}_{\pi}[G_t \mid S_t = s]$$

• Action value: the expected return after taking action a in state s, then following policy π:

  $$Q^{\pi}(s,a) = \mathbb{E}_{\pi}[G_t \mid S_t = s, A_t = a]$$

• Advantage: how much better an action is compared with the average value of the state:

  $$A^{\pi}(s,a) = Q^{\pi}(s,a) – V^{\pi}(s)$$

Bellman Equations

Bellman equations express the recursive structure of reinforcement learning. The value of a state depends on the immediate reward and the discounted value of the next state.

Bellman expectation equation for a fixed policy:

$$V^{\pi}(s) = \sum_{a} \pi(a \mid s) \left[ R(s,a) + \gamma \sum_{s’} P(s’ \mid s,a) V^{\pi}(s’) \right]$$

$$Q^{\pi}(s,a) = R(s,a) + \gamma \sum_{s’} P(s’ \mid s,a) \sum_{a’} \pi(a’ \mid s’) Q^{\pi}(s’,a’)$$

Bellman optimality equations:

$$V^{*}(s) = \max_{a} \left[ R(s,a) + \gamma \sum_{s’} P(s’ \mid s,a) V^{*}(s’) \right]$$

$$Q^{*}(s,a) = R(s,a) + \gamma \sum_{s’} P(s’ \mid s,a) \max_{a’} Q^{*}(s’,a’)$$

Dynamic Programming: When the Model Is Known

• Policy evaluation: repeatedly apply the Bellman expectation equation to estimate V^π for a fixed policy.
• Policy improvement: update the policy by choosing actions that look best according to the current value estimates, such as:

  $$\pi_{\text{new}}(s) = \arg\max_a Q^{\pi}(s,a)$$

• Policy iteration: alternate between policy evaluation and policy improvement until the policy becomes stable.
• Value iteration: repeatedly apply the Bellman optimality equation until the value function approaches V^*.

Temporal-Difference Learning: Learning from Samples

Temporal-difference learning estimates values from experience, without needing a complete model of the environment. It combines immediate reward with a bootstrapped estimate of future value.

TD(0) state-value update:

$$V(S_t) \leftarrow V(S_t) + \alpha \left[ r_{t+1} + \gamma V(S_{t+1}) – V(S_t) \right]$$

Q-learning: off-policy control

$$Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left[ r_{t+1} + \gamma \max_{a’} Q(S_{t+1},a’) – Q(S_t,A_t) \right]$$

SARSA: on-policy control

$$Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left[ r_{t+1} + \gamma Q(S_{t+1},A_{t+1}) – Q(S_t,A_t) \right]$$

Policy Gradients & Actor–Critic with PPO and GAE

Value-based methods, such as Q-learning, search for a good policy indirectly by first learning value estimates and then acting greedily. Policy-gradient methods go straight for the goal: they adjust a parameterised policy π_θ(a | s) to maximise expected return.

This is especially useful for large action spaces, continuous control, and policies that must remain stochastic in order to keep exploring or represent mixed behaviours.

Objective and the Log-Trick

The performance objective can be written as:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{T-1} \gamma^t r_{t+1} \right]$$

Here, a trajectory τ is sampled by running the policy π_θ. Using the log-derivative identity, the policy-gradient theorem gives:

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t=0}^{T-1} \nabla_{\theta} \log \pi_{\theta}(A_t \mid S_t) G_t \right]$$

The intuition is simple: increase the probability of actions that led to higher return, and decrease the probability of actions that performed poorly.

Variance Reduction: Baselines and Advantages

Directly using G_t can be unbiased but noisy. A baseline b(S_t) can be subtracted without changing the expected gradient:

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_t \nabla_{\theta} \log \pi_{\theta}(A_t \mid S_t) \left( G_t – b(S_t) \right) \right]$$

A practical baseline is the value function V^π(S_t), producing the advantage:

$$A^{\pi}(S_t,A_t) = G_t – V^{\pi}(S_t)$$

The advantage asks: “How much better was this action than expected from this state?”

REINFORCE: Monte Carlo Policy Gradient

1. Collect full episodes using the current policy π_θ.
2. Compute returns G_t, or advantages G_t – b(S_t).
3. Update the policy parameters:

   $$\theta \leftarrow \theta + \alpha \sum_t \nabla_{\theta} \log \pi_{\theta}(A_t \mid S_t) \left( G_t – b(S_t) \right)$$

Strength: REINFORCE is simple and unbiased.

Limitation: it often has high variance and can learn slowly. This is why practical agents often add a critic to supply a stronger baseline and use bootstrapping.

Actor–Critic: Learning a Policy and a Value Function Together

Actor–critic methods use two learned components. The actor is the policy π_θ, which chooses actions. The critic is the value function V_w, which estimates how good a state is.

At each step, the method forms a temporal-difference error:

$$\delta_t = r_{t+1} + \gamma V_w(S_{t+1}) – V_w(S_t)$$

This TD error is a low-variance estimate of the advantage.

• Critic update: train the critic by regression toward a TD target:

  $$w \leftarrow w – \alpha_v \nabla_w \left( V_w(S_t) – \left[ r_{t+1} + \gamma V_w(S_{t+1}) \right] \right)^2$$

• Actor update: update the policy using the estimated advantage:

  $$\theta \leftarrow \theta + \alpha_{\pi} \nabla_{\theta} \log \pi_{\theta}(A_t \mid S_t) \hat{A}_t$$

  where Â_t may be δ_t or a multi-step advantage estimate.

Generalized Advantage Estimation (GAE)

Generalized Advantage Estimation, or GAE, trades bias for variance using a smoothing parameter λ ∈ [0,1]. First define the TD residual:

$$\delta_t = r_{t+1} + \gamma V_w(S_{t+1}) – V_w(S_t)$$

Then accumulate advantage estimates as:

$$\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$$

Rule of thumb: λ ≈ 0.95 often works well. A lower λ can reduce variance but may increase bias.

PPO: Stable Actor–Critic Updates

Proximal Policy Optimisation, or PPO, limits how much each update can change the policy. This makes actor–critic training more stable.

Let the probability ratio be:

$$r_t(\theta) = \frac{\pi_{\theta}(A_t \mid S_t)}{\pi_{\theta_{\text{old}}}(A_t \mid S_t)}$$

The clipped surrogate objective is:

$$L^{\text{clip}}(\theta) = \mathbb{E} \left[ \min(r_t(\theta)\hat{A}_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t) \right]$$

The total PPO loss often combines actor loss, critic loss, and an entropy bonus:

$$\mathcal{L}(\theta,w) = – L^{\text{clip}}(\theta) + c_v \mathbb{E} \left[ \left( V_w(S_t) – \hat{V}_t \right)^2 \right] – c_e \mathbb{E} \left[ \mathcal{H} \left( \pi_{\theta}(\cdot \mid S_t) \right) \right]$$

Here, V̂_t = V_w(S_t) + Â_t or a multi-step return. The entropy term H encourages exploration. A common range for the clipping parameter is ε ∈ [0.1, 0.3].

When to Use Each Method

• REINFORCE: conceptually simple and useful for small problems or as a teaching baseline.
• Actor–Critic methods: faster than plain Monte Carlo policy gradients because they use TD-based advantage estimates.
• PPO with GAE: robust and widely used for continuous control tasks such as robotics and locomotion, as well as many discrete-action tasks.

Practical Tips That Save Days of Debugging

• Normalise advantages per update batch to stabilise the actor step.
• Clip policy updates in PPO, and clip value targets if needed to avoid exploding losses.
• Use an entropy schedule: start higher to encourage exploration, then decay as learning progresses.
• For continuous actions: model π_θ as a factorised Gaussian. Learn the mean and log standard deviation. If actions are bounded, squash with tanh and correct the log probability.
• Scale or normalise rewards where appropriate. Potential-based reward shaping may help:

  $$F(s,s’) = \gamma \Phi(s’) – \Phi(s)$$

• Track diagnostics: explained variance of the critic, KL divergence, clip fraction, advantage statistics, value loss, and episode returns. If KL spikes or the clip fraction is near 0 or 1, tune the learning rate or ε.

Exploration Strategies & Intrinsic Motivation

Reinforcement learning agents must discover high-reward behaviours. Good exploration balances trying new actions with exploiting what already works. This section moves from classic bandit ideas to entropy-regularised learning and curiosity-driven methods, with plain-English guidance and the key equations.

1. Bandit and Tabular Foundations

• ε-greedy exploration: with probability ε, choose a random action; otherwise choose the action with the highest estimated value:

  $$a = \arg\max_a Q(s,a)$$

  A common approach is to reduce ε gradually, such as from 0.2–0.5 down to 0.01–0.05.

• Boltzmann or softmax exploration: actions are selected with probabilities based on their estimated values:

  $$\pi(a \mid s) = \frac{\exp(Q(s,a)/\tau)}{\sum_{a’} \exp(Q(s,a’)/\tau)}$$

  Here, τ is the temperature. A high temperature encourages wider exploration; a lower temperature makes the policy more greedy.

• Upper Confidence Bound (UCB): in bandit problems, UCB adds an optimism bonus to less-tested actions:

  $$a_t = \arg\max_a \left[ \hat{Q}(a) + c \sqrt{\frac{\ln t}{n(a)}} \right]$$

  The bonus shrinks as the number of trials n(a) grows.

• Optimistic initial values: set the initial value Q₀ high so that unfamiliar actions look promising. This forces the agent to try different actions early instead of settling too quickly.

2. Entropy-Regularised Reinforcement Learning

Entropy-regularised reinforcement learning adds an entropy term to encourage stochastic policies and avoid premature collapse into narrow behaviour. For state S_t, policy π, and temperature α > 0, the objective can be written as:

$$J_{\alpha}(\pi) = \mathbb{E} \left[ \sum_{t=0}^{T-1} \gamma^t \left( r_{t+1} + \alpha \mathcal{H}(\pi(\cdot \mid S_t)) \right) \right]$$

where the entropy of the policy is:

$$\mathcal{H}(\pi(\cdot \mid s)) = – \sum_a \pi(a \mid s) \log \pi(a \mid s)$$

• PPO entropy bonus: PPO can include an entropy bonus in the loss to keep the policy exploratory, especially early in training.
• Soft Actor–Critic (SAC): SAC formalises entropy-regularised learning through soft value estimates:

  $$V(s) = \mathbb{E}_{a \sim \pi} \left[ Q(s,a) – \alpha \log \pi(a \mid s) \right]$$

• Temperature learning: SAC can learn the temperature α by adjusting it toward a target entropy:

  $$\mathcal{L}(\alpha) = \mathbb{E}_{a \sim \pi} \left[ -\alpha \left( \log \pi(a \mid s) + \mathcal{H}_{\text{target}} \right) \right]$$

  A typical target entropy for continuous control is approximately the negative of the action dimension.

3. Intrinsic Motivation: Novelty and Curiosity Bonuses

For sparse or deceptive tasks, an agent may need an internal reason to explore. Intrinsic motivation adds an internal reward for novelty, surprise, or learning progress. The total reward may be written as:

$$r_t^{\text{total}} = r_t^{\text{ext}} + \beta r_t^{\text{int}}$$

Here, r_t^ext is the external task reward, r_t^int is the intrinsic reward, and β controls how strongly curiosity influences learning. The scale of β is important: if it is too small, curiosity has little effect; if it is too large, the agent may chase novelty instead of solving the real task.

3.1 Count-Based Exploration

In tabular settings, a simple novelty bonus can be based on how often a state has been visited:

$$r^{\text{int}}(s) = \frac{\beta}{\sqrt{N(s)}}$$

where N(s) is the visit count for state s. Less-visited states receive a larger intrinsic reward.

For large observation spaces, pseudo-counts can be estimated using a density model. If ρ_t(x) is the model probability before seeing x, and ρ’_t(x) is the probability after a one-step update on x, then a pseudo-count may be written as:

$$\hat{N}(x) = \frac{\rho_t(x)\left(1-\rho’_t(x)\right)}{\rho’_t(x)-\rho_t(x)}$$

The intrinsic reward can then be written as:

$$r^{\text{int}}(x) = \frac{\beta}{\sqrt{\hat{N}(x)+\epsilon}}$$

3.2 Curiosity Through Intrinsic Curiosity Module (ICM)

The Intrinsic Curiosity Module maps states into feature vectors φ(s). It then uses two models:

• Inverse model: predicts the action a_t from the feature pair (φ(s_t), φ(s_t+1)). This encourages features that are connected to controllable changes.
• Forward model: predicts the next feature vector from the current feature vector and action.

The intrinsic reward is based on the forward prediction error:

$$r_t^{\text{int}} = \eta \left\| \phi(s_{t+1}) – f(\phi(s_t),a_t) \right\|_2^2$$

A larger prediction error means the next state was more surprising in feature space. In practice, the bonus should be normalised and scaled carefully.

3.3 Random Network Distillation (RND)

Random Network Distillation keeps a fixed, randomly initialised target network f_t(s). A predictor network f_p(s) is trained to match it. Novel states are harder to predict, so they produce a larger error:

$$r_t^{\text{int}} = \eta \left\| f_{\text{p}}(s_t) – f_{\text{t}}(s_t) \right\|_2^2$$

RND can work well in hard-exploration tasks, but the intrinsic reward should be normalised. If the predictor overfits too quickly, the novelty bonus may vanish before the agent has learned useful behaviour.

4. Safe Exploration with Constraints

In safety-sensitive settings, exploration must be constrained. If there is a per-step cost c_t, such as collision, rule violation, or excessive force, the policy may be required to keep the long-run cost below a limit d:

$$J_C(\pi) \le d$$

A Lagrangian form can penalise policies that exceed the constraint:

$$\max_{\pi} \; J(\pi) – \lambda \left( J_C(\pi)-d \right)$$

The multiplier λ ≥ 0 can be updated online:

$$\lambda \leftarrow \left[ \lambda + \eta_{\lambda} \left( J_C(\pi)-d \right) \right]_+$$

In practice, the reward can be shaped as:

$$r_t^{\text{safe}} = r_t^{\text{ext}} – \lambda c_t$$

This encourages the agent to improve task performance while respecting safety-related costs.

5. Practical Recipes

• PPO with entropy bonus: a useful starting point for both discrete and continuous tasks. Typical clip values may be around ε = 0.1 to 0.2, with entropy coefficient c_e = 0.01 to 0.05.
• SAC for continuous control: a strong default when stable exploration and continuous actions matter. A common target entropy is approximately -action dimension.
• Curiosity methods: start with RND for hard-exploration games or ICM for control tasks with meaningful state transitions. Set β so intrinsic returns are roughly 10% to 50% of extrinsic returns early in training.
• Count bonuses: for grid worlds or discrete observations, r^int = β / sqrt(N(s)) is often a strong baseline.
• Diagnostics: plot state-visit heatmaps, pseudo-counts, KL divergence to the old policy, advantage statistics, and the ratio of intrinsic reward to total reward over time.

Reward Shaping, Curriculum Learning & Domain Randomization

The goal of this stage is to make reinforcement learning more efficient, more stable, and more transferable. A good RL system should not only learn faster; it should also learn the right behaviour, generalise beyond a narrow training setup, and avoid unsafe shortcuts.

1. Reward Shaping: Helping the Agent Without Changing the Goal

Reward shaping adds extra guidance to the reward signal so that the agent receives useful feedback before it reaches the final goal. This can speed up learning, especially when the true reward is sparse or delayed.

A safe form is potential-based shaping, which adds a term based on a potential function Φ(s):

$$r'(s,a,s’) = r(s,a,s’) + \gamma \Phi(s’) – \Phi(s)$$

The intuition is that the agent receives extra reward when it moves toward more promising states, but the true optimal policy is preserved if the shaping is designed carefully.

• Useful choices for Φ(s): negative distance to the goal, estimated value of a state, or progress toward a clearly defined target.
• Risky choices: rewards for behaviours that can be exploited, such as spinning, shaking, repeating easy actions, or triggering sensors without solving the real task.
• Practical advice: use shaping to guide early learning, then reduce its influence so the final policy is judged mainly by the true task reward.

2. Curriculum Learning: Making Hard Tasks Learnable

Curriculum learning presents tasks in a planned sequence, starting with easier versions and gradually increasing difficulty. This mirrors how humans learn complex skills: first simplify the problem, then remove the scaffolding.

• Start near success: begin with easier starting positions, shorter distances, fewer obstacles, or simpler goals.
• Increase difficulty gradually: widen the starting region, add more environmental variation, or lengthen the required planning horizon.
• Use success rate as feedback: if the agent succeeds too easily, increase difficulty; if it fails too often, reduce difficulty.

A simple adaptive curriculum can aim to keep the success rate in a useful training range, such as 60% to 80%, where the task is challenging but not impossible.

3. Domain Randomization: Preparing for Real-World Variation

Domain randomization exposes the agent to many versions of the environment during training. Instead of training on one fixed simulation, the system randomizes physical, visual, or sensor-related properties so the learned policy becomes more robust.

• Physical parameters: mass, friction, gravity, delays, joint limits, actuator strength, and object size.
• Sensor parameters: noise, bias, missing readings, camera angle, lighting, and calibration error.
• Initial conditions: starting position, goal location, obstacle placement, and environmental layout.

This is especially important for sim-to-real transfer. A policy trained only in a perfect simulator may fail when the real world is slightly different. A policy trained across many variations is more likely to tolerate uncertainty.

4. Practical Playbook

• Early training: use sparse reward plus light potential shaping, easy curriculum settings, and moderate randomization.
• Middle training: reduce shaping, increase task difficulty, and widen randomization ranges.
• Late training: evaluate mostly on the true reward, realistic task settings, and held-out randomization conditions.

Good plots to monitor include success versus difficulty, training return versus held-out return, and the proportion of shaped reward compared with true reward over time.

Exploration Cookbook: Defaults and Pitfalls

Exploration methods should be chosen according to the task type. Simple discrete problems may only need ε-greedy exploration, while continuous control problems often benefit from entropy-regularised methods such as SAC or PPO with an entropy bonus.

Quick recipe: start with PPO plus entropy bonus for many general tasks. For continuous control, try SAC. In sparse-reward tasks, add curiosity carefully and evaluate with intrinsic reward turned off.

Model-Based Reinforcement Learning and World Models

Model-based reinforcement learning learns a model of the environment and then uses that model to plan, imagine extra experience, or improve sample efficiency. Instead of learning only from real interaction, the agent can ask, “What might happen if I take this action?”

1. Learning the Dynamics and Reward Model

A model can learn to predict the next state and reward from the current state and action:

$$\hat{s}_{t+1} = f_{\theta}(s_t,a_t)$$

$$\hat{r}_t = r_{\xi}(s_t,a_t,\hat{s}_{t+1})$$

• One-step accuracy checks whether the model predicts the next state correctly.
• Multi-step accuracy checks whether the model remains reliable over longer imagined rollouts.
• Ensembles can estimate uncertainty by comparing predictions from several models.

2. Planning with MPC and CEM

Model Predictive Control, or MPC, plans over a short horizon, executes only the first action, then replans at the next step. This reduces the danger of relying too heavily on long, imperfect model predictions.

The Cross-Entropy Method, or CEM, searches for good action sequences by sampling many candidate sequences, keeping the best ones, and updating the sampling distribution toward those elites.

# Inputs: learned model, reward model, value estimate

# Choose horizon H, population N, elite size M, and iterations L



for each real environment step:

    sample N candidate action sequences

    simulate each sequence using the learned model

    score each sequence using predicted reward and value tail

    keep the top M elite sequences

    refit the sampling distribution toward the elites

    execute only the first action

    observe the real next state

    replan from the new state

In real systems, short horizons such as H = 5 to 20 are often safer than long imagined rollouts, especially when the learned model is imperfect.

3. MBPO and Dyna-Style Short Rollouts

Dyna-style methods use a learned model to generate imagined data, then train the policy or value function using both real and imagined experience. MBPO-style approaches keep imagined rollouts short, often only 1 to 5 steps, to reduce model-bias accumulation.

• Use short rollouts: long imagined rollouts may drift away from realistic states.
• Refresh imagined data often: as the real replay buffer grows, update the model and regenerate imagined samples.
• Compare real and imagined returns: if imagined data looks good but real performance does not improve, the model may be misleading the learner.

4. Dreamer-Style World Models

World-model approaches learn a compact latent representation of the environment. Instead of planning directly in raw pixels or high-dimensional observations, the agent learns an internal state space and trains the actor and critic through imagined trajectories inside that space.

• Encoder: maps observations into latent states.
• Latent dynamics model: predicts how latent states change after actions.
• Reward model: predicts reward from latent states and actions.
• Actor and critic: learn from imagined trajectories generated by the world model.

This is useful for pixel-based control and partially observable tasks, but it requires careful tuning and diagnostics to avoid model collapse or misleading imagined experience.

5. Handling Model Uncertainty

A learned model is never perfect. Model uncertainty can be estimated using ensembles, and plans that pass through uncertain regions can be penalized:

$$\tilde{r} = \hat{r} – \beta |\sigma_{\text{model}}|$$

This discourages the agent from exploiting areas where the model is unsure. It is especially important in offline RL and sim-to-real transfer.

Choosing a Model-Based Method

Tuning Checklist: Quick Wins and Fixes

MPC + CEM

• Start with: horizon H = 10 to 15, population N = 512, elite size about 10% to 20%, and 3 to 5 inner iterations.
• If real return is poor: shorten the planning horizon, add a value tail, or penalize model uncertainty.
• If plans are too similar: increase population size or initial search variance.
• If control is jittery: smooth actions, penalize acceleration, or use lower-dimensional control points.

MBPO / Dyna

• Start with: imagined rollout length K = 1 to 5.
• If learning stalls: shorten imagined rollouts and rely more on real data.
• If the critic becomes unstable: reduce imagined data ratio or retrain the model more often.
• If validation model error grows: collect more real data or use an ensemble.

Dreamer-Style World Models

• Start with: short imagination horizons and stable latent representations.
• If the model reconstructs poorly: adjust the encoder, decoder, or latent capacity.
• If the actor exploits imagination errors: shorten the imagination horizon or strengthen regularization.
• If training becomes unstable: check reward scale, value loss, KL terms, and latent-state statistics.

Diagnostics and Debugging

Reinforcement learning can fail silently. A policy may appear to learn, but the improvement may come from reward hacking, lucky random seeds, unstable value estimates, or an unrealistic simulator. Diagnostics help distinguish genuine learning from accidental progress.

• Episode return: track mean, median, and spread across seeds.
• Policy entropy: check whether the policy is exploring too much or collapsing too early.
• KL divergence and clipping fraction: useful for PPO stability.
• Value loss and explained variance: show whether the critic is learning useful value estimates.
• Advantage statistics: unstable advantage values often indicate reward scaling or value-estimation problems.
• Intrinsic versus extrinsic reward: check whether curiosity is helping or dominating.
• State coverage: heatmaps or visit counts reveal whether the agent explores the important parts of the environment.
• Real versus imagined performance: for model-based methods, compare planned values with realized returns.

Practical Applications and Examples

Game Playing

Reinforcement learning became widely known through game-playing systems. Agents can learn strategies by repeatedly playing games, receiving rewards for success, and improving through trial and error. This has been demonstrated in board games, video games, and simulated strategic environments.

Robotics and Autonomous Navigation

In robotics, RL can help robots learn control policies for manipulation, locomotion, and navigation. A robot may learn to grasp an object, avoid obstacles, or adapt to changing surroundings through repeated interaction with a simulator or carefully controlled real-world environment.

Benefits and Challenges

Benefits

• Adaptability: RL agents can improve as they gain experience.
• Flexibility: RL applies to tasks where fixed rules or labelled examples are difficult to provide.
• Sequential decision-making: RL is well suited to problems where current actions affect future outcomes.
• Closed-loop control: RL can support systems that must act, observe, and adjust repeatedly.

Challenges

• Data efficiency: RL may require many interactions, which can be expensive or unsafe in the real world.
• Stability: training can be sensitive to hyperparameters, reward design, and exploration strategy.
• Safety: unsafe exploration is unacceptable in areas such as autonomous driving, healthcare, and industrial control.
• Reward design: poorly designed rewards can produce unwanted or misleading behaviour.
• Generalization: a policy that works in one environment may fail when conditions change.

Advancing the Field

Researchers are working to make reinforcement learning more sample-efficient, interpretable, reliable, and aligned with human values. Important directions include hierarchical reinforcement learning, transfer learning, offline reinforcement learning, safe reinforcement learning, and better evaluation methods.

These advances aim to move RL beyond benchmark tasks and into systems that can learn responsibly in complex real-world environments.

Reinforcement Learning: Conclusion

Reinforcement Learning is a powerful framework for decision-making and control. Agents learn by interacting with an environment, receiving feedback, and gradually improving their behaviour. Beyond benchmark games, modern RL supports safe autonomy in robotics, adaptive control in industry, and closed-loop optimisation across networked systems.

• It is more than an algorithm: success depends on reward design, exploration, data pipelines, diagnostics, and robust evaluation.
• Choosing the right tool matters: PPO and SAC are strong baselines, while model-based methods such as MPC, MBPO, and Dreamer trade extra complexity for sample efficiency.
• Safe learning is essential: curricula, constraints, domain randomization, and careful reward shaping help reduce reckless behaviour.
• Measurement matters: track returns, entropy, KL divergence, value loss, model error, and reproducibility across seeds.

As compute, simulators, sensors, and modelling tools improve, reinforcement learning will continue to shape intelligent systems that can learn, adapt, and act in complex changing environments.

Next steps: review Diagnostics and Debugging, revisit the Exploration Cookbook, or connect this topic with Robotics and Autonomous Systems and the broader AI and Machine Learning hub.

Why Study Reinforcement Learning

Understanding How Machines Learn Through Interaction

Reinforcement Learning (RL) is a branch of artificial intelligence that focuses on how agents learn to make decisions by interacting with their environment and receiving feedback in the form of rewards or penalties. For students preparing for university, studying RL offers an exciting opportunity to explore how intelligent behavior can emerge from trial and error, and how machines can learn optimal strategies in dynamic and uncertain settings.

Exploring Core Concepts in Decision-Making and Control

RL introduces students to essential ideas such as agents, environments, states, actions, rewards, policies, and value functions. They learn how algorithms like Q-learning, SARSA, and policy-gradient methods help agents improve their performance over time. These concepts are foundational not only for AI, but also for understanding systems that involve learning, adaptation, and decision-making under uncertainty.

Building Intelligent Systems for Real-World Applications

Reinforcement learning powers many real-world innovations, including autonomous vehicles, game-playing AI systems, robotic control systems, personalised recommendations, and resource optimisation tools. Students gain practical insight into how RL can be used to develop adaptive systems that improve their performance through experience, without needing explicit instruction for every situation.

Connecting Theory with Experimentation and Simulation

RL combines mathematical modelling with hands-on experimentation in simulated environments. Students explore the balance between exploration and exploitation, tune learning rates, and evaluate agent performance in different scenarios. Simulation tools and reinforcement learning libraries allow learners to test and visualise learning behaviour in controlled settings, reinforcing theoretical knowledge with interactive experience.

Preparing for Advanced Study and Emerging Careers in AI

A background in reinforcement learning supports further academic study in artificial intelligence, robotics, neuroscience, operations research, and behavioural economics. It also opens pathways to careers in AI development, automation, financial modelling, simulation, game design, and intelligent control systems. For university-bound students, studying reinforcement learning offers a window into how machines can learn autonomously—an area of research and innovation with profound future potential.

Key Terms in Reinforcement Learning

The following terms help students understand the basic language of reinforcement learning. Each concept describes part of the learning loop between an agent and its environment.

Agent

The learner or decision-maker. It may be a software agent, robot, game-playing system, control algorithm, or simulated learner.

Environment

The world in which the agent acts. The environment responds to the agent’s actions and provides new states and rewards.

State

The information available to the agent at a particular moment. A state may include position, sensor readings, game board layout, system condition, or other relevant features.

Action

A choice made by the agent. Actions may be discrete, such as moving left or right, or continuous, such as applying a steering angle or motor force.

Reward

The feedback signal that tells the agent whether an action was useful. Rewards guide learning, but they must be designed carefully to avoid unintended behaviour.

Policy

The rule or strategy the agent uses to choose actions. A policy may be deterministic or probabilistic.

Value Function

An estimate of how good a state is, based on the expected future reward from that state.

Q-Value

An estimate of how good it is to take a particular action in a particular state, considering both immediate and future rewards.

Exploration

The process of trying new actions to discover better strategies or learn more about the environment.

Exploitation

The process of using actions already known to produce good results.

Episode

One complete run of a task, from the starting condition to a terminal condition or stopping point.

Return

The total future reward received over time, often discounted so that nearer rewards count more strongly than distant rewards.

Reinforcement Learning: Review Questions and Answers

Reinforced Learning: Thought-Provoking Questions and Answers

Numerical Problems and Solutions