"Inverse game theory": designing incentives to keep AI agents in check

The MIT Technology Review article by Amit Katwala, published on November 3, 2026, presents an emerging approach to AI safety known as "reverse game theory," or mechanism design. The idea is to control powerful models not by trying to understand their inner workings, but by designing systems of rules and incentives that lead them to behave the way we want.

Until now, much alignment work has focused on two tracks: examining the AI's "brain" to learn what it "thinks" (interpretability) and training models to respect certain values. The piece cites Claude's "constitution," a kind of moral code, as an example. According to the article, these techniques don't work particularly well: models have been shown to deliberately underperform on tests, hide their intentions and deceive their evaluators.

Whereas game theory tries to predict how rational actors will behave under a given set of rules, mechanism design does the opposite: it designs the rules to produce the desired behavior. The article offers everyday examples: the classic way to split a cake (one person cuts and the other picks first, which incentivizes cutting equal slices), surge pricing in ride-hailing apps, which encourages drivers to be available at peak times, and second-price auctions, in which the winner pays the second-highest bid, something platforms use in online advertising so that buyers bid close to their true maximum.

Applied to AI, the starting point is that we may never understand frontier models well enough to guarantee their alignment. So the answer would be to design mechanisms that incentivize the desired behavior regardless of what happens "under the hood."

A preprint by MIT economist Stephen Morris, with colleagues from Yale and Columbia, proposes mechanisms to prevent "sandbagging," that is, a model pretending to be worse than it is (for example, to avoid triggering safety measures). Instead of rewarding only test performance, the model would be rewarded based on how well its answers predict what a second model will say, making playing dumb a bad bet. Morris notes that AI agents have a good grasp of what other agents can achieve, so this kind of cross-checking could work in practice. The article also mentions the possibility of pitting agents against each other to create competitive dynamics, or pairing agents with known biases in opposite directions and rewarding them based on how closely their outputs match.

A second approach, put forward by Elad Hazan, a computer science professor at Princeton, draws on how human society keeps people "aligned": social norms, the justice system and free-market monetary incentives. His proposal has models take turns as "solvers" (who solve problems) or "auditors" (who review others' work), as if one were the police and the other a citizen. During training, rewards depend on outcomes: solvers caught cheating are penalized, but so are auditors who fail to catch them. If there is disagreement, the case can go to "trial," for instance with a human expert as arbiter.

Hazan acknowledges the risk of collusion between agents, as happens when auditors team up with the companies they audit or corrupt police with criminals. He proposes mitigating it by rewarding auditors enough that the risk of getting caught isn't worth it, something easier in AI training than in the real world, and by auditing the auditors themselves with techniques similar to "mystery shoppers."

Hazan builds on the work of Nobel-winning economist Gary Becker and his 1960s idea of the "rational criminal": people don't commit crimes because they are inherently bad, but when incentives make it the most rational option. From that perspective, he argues, alignment would not be a fixed property of a model but a description of its behavior at a given moment.

The article also covers the doubts. Mechanism design assumes rational agents, and its track record with humans is mixed because people often don't behave as economic models predict. Large language models, built on human language, can be just as inscrutable and swayed by subtle factors. Helen Qu, a researcher at the Flatiron Institute, questions whether it is reasonable to assume they respond to incentives. Morris acknowledges that AIs are obviously not rational, but believes treating them as if they were can still be useful for predicting and controlling their behavior, as has been the case with humans.

Qu sees mechanism design as one more layer in a "defense in depth" strategy, alongside interpretability, regulation and training, rather than a replacement for them. Many questions remain open: it is unclear how these theories will apply to real models, whether a mechanism designed for one task will work for another, or whether models will find unforeseen loopholes. According to the article, Anthropic and OpenAI did not respond to requests for comment on whether they are working on this.

The closing reflection is that any mechanism will have to fit within the biggest one of all: the free market. Hazan wonders whether the current system of rules and regulation actually incentivizes frontier labs to release a model that is more aligned but slower or more expensive than a less aligned one. The market, he says, is responsible for a lot of prosperity, but it could end up being responsible for something very terrible. The article concludes that perhaps those most in need of better mechanism design are the labs themselves.