The evolution of AI environments: from game simulators to agent environments

From TheSequence · Thu, 22 Oct 2026.

This opinion essay (The Sequence Opinion, issue 956) argues that AI environments are becoming a core ingredient of model capability. According to the text, they are not just exams but executable worlds that generate trajectories and feedback.

The author traces their evolution from Atari, OpenAI Gym and AlphaZero, through MuJoCo, to the RLVR-based post-training (reinforcement learning with verifiable rewards) behind Tulu 3 and DeepSeek-R1.

It then reviews the new generation of agent environments, citing SWE-Gym, WebArena, OSWorld, TheAgentCompany, Tau2-bench and AgentMercury.

The essay also defines what makes a good environment: rewards that reflect the real outcome and resist hacking, causal fidelity, affordable and repeatable practice, structured difficulty, and improvements that transfer outside the training world. According to the text, the frontier lies in supplying varied, consequential practice with reliable feedback.

To illustrate the idea, the author offers an analogy. Imagine hiring an engineer who has read every programming book but has never run a program. They could explain databases and distributed systems and draw a very confident architecture diagram. But ask them to fix a failing deployment, and the gap between describing the work and doing it becomes costly.

In that analogy, pretraining gives the model broad knowledge drawn from recorded experience. An environment, by contrast, lets the agent act, face consequences and generate new experience. A compiler rejects its code, a browser displays an unexpected page, and a customer changes the request midway through the workflow.

The author's thesis is that environments are becoming an important ingredient of AI capability. He presents them as generators of training data. Their quality determines what an agent can practice, which errors it can discover, and whether improvement within the training environment means anything outside it.

Note: the email received contains only the beginning of the essay (summary and introduction). The full text is on the Substack website, so this summary does not include the details of each environment or each criterion.