- Curriculum learning trains an RL agent on an ordered sequence of tasks — easy first, then harder — instead of throwing it at the hardest task cold.
- It is the standard cure for sparse rewards and hard-exploration problems, where a from-scratch agent never stumbles onto any reward signal.
- Automatic curriculum learning (ACL) removes the hand-design: a teacher picks tasks by learning progress, regret, or novelty so the agent always trains at the edge of its ability.
- Self-play and unsupervised environment design produce open-ended autocurricula — and in 2025 easy-to-hard ordering became a standard trick for RL on LLM reasoning.
What is curriculum learning in RL?
Curriculum learning trains an agent on a sequence of tasks of increasing difficulty, transferring what it learns at each stage forward, rather than dropping it into the hardest task from the start. The idea is borrowed straight from how humans learn: you teach arithmetic before calculus because the easy material builds the representations and skills the hard material needs.
In reinforcement learning this is more than a nicety. Many tasks have sparse rewards — the agent only sees a signal when it completes a long sequence of correct actions — so a from-scratch policy doing random exploration may never hit reward, and learning never starts. A curriculum seeds the agent with easy variants where reward is reachable, then gradually raises the difficulty so the policy is always learning something it can almost already do.
Why it matters: the exploration problem
A pretrained policy that has never received reward is, for all practical purposes, blind. The canonical failure case is a sparse-reward task like Montezuma’s Revenge: the agent must perform a long, precise sequence of moves before any points appear, and uniform random exploration has a vanishing chance of completing it. Reward-driven learning has nothing to drive it.
Curriculum learning attacks this by reshaping what the agent trains on rather than the reward itself (the complementary lever — see reward shaping) or the exploration bonus (see curiosity and intrinsic motivation). It is one of three standard answers to exploration vs exploitation in hard-exploration domains.
The building blocks of a curriculum
The 2020 JMLR survey by Narvekar et al. breaks any curriculum method into three design decisions. Getting all three right is what separates a curriculum that accelerates learning from one that wastes compute or even hurts.
Define the space of intermediate tasks. These can be hand-authored levels, procedurally generated variations, parameterized environments (e.g. terrain roughness as a continuous knob), or sub-goals carved out of the target task. The space must contain tasks easy enough to be solvable now and a smooth path toward the target.
Decide which task to train on next, given the agent’s current ability. This is the heart of the method: a fixed hand-designed schedule, or an adaptive policy that picks the task offering the most learning right now. Bad sequencing (too hard, too soon) gives no signal; too easy wastes samples.
Move what was learned on task k into task k+1. The simplest and most common mechanism is to keep the same policy network and continue training — the weights are the transferred knowledge. Value functions, learned representations, or skills can also be carried forward.
Go deeper: the curriculum as a meta-MDP
Narvekar et al. formalize sequencing as a curriculum MDP sitting above the task MDPs. Its “state” is the agent’s current knowledge (e.g. its policy parameters), an “action” is choosing the next task to train on, and the “reward” is how much the chosen task improves performance on the target. Solving this meta-MDP optimally would yield the best possible curriculum — but it is far harder than the original problem, so in practice everyone uses cheap heuristics (learning progress, regret, difficulty bins) as proxies for the meta-reward. See markov decision processes for the underlying formalism.
Manual vs automatic curricula
The first axis to understand is who designs the sequence. Hand-designed curricula are simple and controllable but brittle and labor-intensive; automatic curriculum learning (ACL) hands sequencing to an algorithm that adapts to the learner in real time.
A human authors the task progression: start the robot near the goal, then move the start state farther back; grow the board size; raise opponent strength on a fixed schedule. Pros: simple, interpretable, no extra machinery. Cons: requires domain expertise, doesn’t adapt to the agent’s actual learning curve, breaks when the agent learns faster or slower than expected.
A teacher algorithm selects tasks online using a signal like learning progress, regret, or novelty — always training the agent at the frontier of its competence. Pros: adapts to the learner, no hand-tuned schedule, scales to huge task spaces. Cons: the teacher’s signal can be noisy or gamed, and it adds its own hyperparameters. See the short ACL survey.
Teacher-student and learning-progress curricula
The dominant ACL idea is teacher-student curriculum learning (TSCL): a teacher agent’s job is to pick, at each step, the sub-task on which the student is currently improving fastest. The intuition is that the steepest part of the learning curve is where training samples buy the most — too-easy tasks are mastered (no slope) and too-hard tasks give no progress (no slope), so the productive frontier is in between.
The teacher’s reward for proposing task is the student’s learning progress — the change in performance on that task:
where is the student’s score on task at time . The teacher then treats task selection as a non-stationary multi-armed bandit problem, pulling the arm (task) with the highest expected progress. A refinement, ALP-GMM (Portelas et al., 2019), works in continuous task spaces and uses absolute learning progress, , so the teacher also revisits tasks where the student is regressing — fitting a Gaussian mixture over the task space to find high-progress regions.
Generating the tasks: goals, self-play, and environment design
Sequencing assumes a pool of tasks to sequence. Where does that pool come from? Three families answer this differently.
| Family | How tasks are created | Representative method |
|---|---|---|
| Goal generation | A generator proposes intermediate goals at the right difficulty | Goal GAN — “goals of intermediate difficulty” (Florensa et al., 2018) |
| Self-play | An opponent or a goal-setting partner is the curriculum, scaling with the agent | Asymmetric self-play; AlphaZero-style self-play |
| Environment design | An adversary or curator shapes the environment itself toward maximal learning | PAIRED, Prioritized Level Replay, POET |
Goal generation trains a generator (originally a GAN) to output goals the agent succeeds at roughly half the time — neither trivial nor impossible — and refreshes them as the agent improves.
Self-play is the most famous source of automatic curricula: when an agent trains against copies of itself, the opponent is always at exactly the right level, and difficulty rises automatically as both sides improve. This is the engine behind AlphaZero and MuZero and a core mechanism of multi-agent RL. OpenAI’s hide-and-seek experiment is the vivid demonstration: with no explicit curriculum, multi-agent competition produced an autocurriculum of six escalating strategies and counter-strategies — running, fort-building, ramp use, ramp defense, box-surfing, surf defense — each emerging only because the previous one made it worthwhile.
Unsupervised environment design (UED)
The most principled modern framing is unsupervised environment design: a teacher generates the environment configuration (a maze layout, a terrain) to maximize the student’s learning. The standard signal is regret — the gap between the best achievable return on a level and the student’s actual return. High-regret levels are exactly those the student could solve but currently doesn’t: the productive frontier again.
- PAIRED stages a three-way game: a generator builds levels, an antagonist (expert) and a protagonist (the student) both attempt them, and the generator is rewarded by the performance gap — driving it toward solvable-but-unsolved levels.
- Prioritized Level Replay (PLR) is the simpler, more scalable cousin: randomly sample levels, estimate each one’s learning potential (via GAE-based regret proxies), and keep replaying the highest-regret levels from a rolling buffer. PLR generalizes better out-of-distribution than PAIRED.
- POET co-evolves a population of environments and the agents that solve them, endlessly generating new, increasingly complex terrains — an open-ended curriculum with no fixed target task at all.
Go deeper: why regret beats raw difficulty as a signal
A naive teacher that maximizes student failure will happily generate impossible levels — an unsolvable maze has maximum failure and zero learning value. Regret fixes this: it is high only when a level is solvable (some policy gets high return) and the current student does poorly. Maximizing regret therefore steers the generator toward the frontier of solvable-but-unsolved tasks, and it comes with a game-theoretic guarantee: at the equilibrium of the teacher-student game, the student has minimized worst-case regret, i.e. it is robust across the whole level distribution. In practice the true regret is unknown, so methods use proxies — the antagonist-protagonist gap (PAIRED) or value-prediction error (PLR).
Curriculum learning for LLM reasoning (2024-2026)
Curriculum learning is having a second life inside RL post-training for LLMs. When you train a reasoning model with RLVR or GRPO, the difficulty of each problem matters enormously: if every problem is too hard, all sampled answers are wrong, the group-relative advantage is zero, and the gradient vanishes — the model learns nothing. If everything is too easy, all answers are right and again there is no signal. Productive RL needs problems the model gets right some of the time.
This makes difficulty ordering a first-class lever. Recent work shows that scheduling problems easy-to-hard — or simply filtering to problems of intermediate pass-rate — meaningfully improves RL for reasoning, especially for small models that flounder under vanilla RL. The 2026 E2H Reasoner result found that easy tasks provide early traction but should be phased out to avoid overfitting, and self-evolving curricula let the model’s own success rate define the schedule online — TSCL’s learning-progress idea, rediscovered for LLMs.
A short history
Where curriculum learning is used
| Domain | What the curriculum does |
|---|---|
| Robotics & control | Grow terrain roughness, perturbation strength, or task length; reverse curricula start near the goal. See RL in robotics, continuous control. |
| Games & self-play | Self-play supplies an automatic opponent curriculum; procedural levels raise difficulty. See AlphaZero & MuZero. |
| Procedurally generated environments | UED (PLR, PAIRED, POET) generates levels at the student’s frontier for robust generalization. |
| LLM reasoning | Easy-to-hard ordering and difficulty filtering keep the RLVR/GRPO gradient signal alive. See RL for reasoning. |
| Multi-agent | Competitive co-adaptation creates autocurricula. See multi-agent RL. |
Pitfalls and open problems
- A bad curriculum can hurt. Ordering that is too aggressive, or that drifts away from the target distribution, can slow learning or bias the final policy — sometimes worse than no curriculum at all.
- Forgetting. As the agent moves to harder tasks it can forget how to solve earlier ones; PLR-style replay and mixed sampling exist precisely to counter this.
- Noisy progress signals. Learning progress and regret are estimated from a few noisy rollouts, so the teacher can chase noise or get stuck. Robust estimation is an active area.
- The meta-problem. Optimal sequencing is its own hard RL problem; every practical method is a heuristic approximation, and which heuristic wins is domain-dependent.
- Designing the task space. ACL only sequences within the tasks you give it — if the space lacks a smooth path to the target, no teacher can build a good curriculum.
Researcher takes
Minqi Jiang (UCL DARK / Meta FAIR) reframes Prioritized Level Replay as a form of unsupervised environment design — quietly building a minimax-regret curriculum that drives zero-shot generalization.
Frequently asked questions
How is curriculum learning different from reward shaping?
They are complementary levers on the same problem (sparse signal). Reward shaping changes the reward function to give denser intermediate feedback on a fixed task. Curriculum learning leaves the reward alone and changes which tasks the agent trains on, in what order. You can — and often do — use both together.
What’s the difference between manual and automatic curriculum learning?
A manual curriculum is a human-authored, usually fixed schedule of tasks. Automatic curriculum learning (ACL) lets an algorithm — a teacher — choose tasks online based on the agent’s measured learning, so the sequence adapts to how fast the agent actually learns. ACL scales to large task spaces and removes hand-tuning, at the cost of extra machinery and a signal that can be noisy.
Is self-play a form of curriculum learning?
Yes — it is the most influential automatic curriculum. Training against copies of yourself means the opponent is always near your own level, so difficulty scales automatically as you improve. This autocurriculum is what made AlphaZero and OpenAI’s hide-and-seek work without any hand-designed schedule.
Does curriculum learning help when training LLMs with RL?
Often, yes. In RLVR/GRPO post-training, problems that are uniformly too hard or too easy produce no gradient. Ordering or filtering problems toward intermediate difficulty keeps the learning signal alive — especially for smaller models. But it is not guaranteed: some 2026 studies find fixed easy-to-hard schedules underperform uniform sampling on certain reasoning tasks, so it remains an empirical choice. See RL for reasoning.
Key papers
- Curriculum Learning — Bengio et al., 2009 — the foundational ML formulation.
- Curriculum Learning for RL Domains: A Framework and Survey — Narvekar et al., 2020 (JMLR) — the canonical RL reference.
- Automatic Curriculum Learning For Deep RL: A Short Survey — Portelas, Colas, Weng, Hofmann, Oudeyer, 2020.
- Teacher-Student Curriculum Learning — Matiisen et al., 2017 — learning-progress task selection.
- Automatic Goal Generation for RL Agents — Florensa et al., 2018 — Goal GAN.
- Emergent Tool Use from Multi-Agent Autocurricula — Baker et al., 2019 — hide-and-seek.
- Prioritized Level Replay — Jiang et al., 2021 — regret-based UED at scale.
- Evolving Curricula with Regret-Based Environment Design — Parker-Holder et al., 2022 — ACCEL.
- Curriculum RL from Easy to Hard Tasks Improves LLM Reasoning — Parashar et al., 2026.
Related
Reward shaping · Exploration vs exploitation · Curiosity & intrinsic motivation · Multi-agent RL · AlphaZero & MuZero · RL for reasoning · What is reinforcement learning?