This repository contains the code for "Thermodynamics of Reinforcement Learning Curricula", accepted at the ICLR 2026 Workshop on Scientific Methods for Deep Learning (SciForDL). The paper can be found here.

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

• Gridworld: We evaluate MEW on a simple gridworld environment, where the agent must navigate to a goal location (top right or bottom left). The curriculum is designed to guide the agent from a known solution (reach bottom left corner) to the unknown (reach top right corner). This task can be expressed as a linear function of the reward parameters. Standard techniques might linearly interpolate between the two reward functions based on the task parameterization vector, but MEW finds a non-linear path (geodesic in task space) that leads to improved learning efficiency (lower empirical regret).
• MuJoCo: We also test MEW on a more complex MuJoCo environment (Humanoid-v5). The curriculum corresponds to the adjustment of temperature, and therefore provides an alternative to the standard temperature annealing schedule (cf. minimum entropy constraint in SAC). We show that MEW outperforms this standard schedule (and constant temperatures), leading to improved learning efficiency and better final performance.