Anastasios Angelopoulos (Arena) on Chatbot Arena, AI evaluation and the future of model rankings

By TheSequence.

This installment of The Sequence Chat interview series features a conversation with Anastasios Angelopoulos, co-founder and CEO of Arena, the company that grew out of the Chatbot Arena research project. Angelopoulos explains that his path in AI has been non-linear: he started out working at the intersection of AI and medicine, where he concluded that one of the most important problems in deploying AI in healthcare is reliability. That led him to pursue a PhD at Berkeley focused on the theoretical foundations of AI reliability, in the course of which he took part in several projects, among them Chatbot Arena, which ended up becoming Arena. As he describes it, they are now building an AI evaluation company that measures AI performance and reliability in real workflows across every economically relevant sector.

On the origins of Chatbot Arena, Angelopoulos recounts that it began as a side project within the Berkeley SkyLab. The original intent was to gather evidence that Berkeley's open source LLM, Vicuna, was better than competing models, including one from Stanford. Although those models performed similarly on traditional benchmarks, chatting with them made it clear that Vicuna was superior, which gave rise to 'Battle Mode' and to the idea of human-preference rating through pairwise comparisons. Angelopoulos notes that the platform has today gone far beyond human preferences: it also ranks models by task completion rates, hallucination rates and other metrics. What makes Arena unique, he says, is its ability to rank models by their real usefulness to real people during real workflows, drawing on a base of tens of millions of monthly visitors.

Asked about Arena-Rank and the statistical assumptions behind the ranking pipeline, he explains that Arena scores represent how useful models are to that user base. Under the standard statistical assumptions they employ, when two confidence intervals overlap, that indicates the two models are statistically indistinguishable given the available data; this is the key to interpreting small score differences between models.

On how to combine human preference and truthfulness without smuggling a value judgment into the weighting, Angelopoulos argues that there is no universal method for ranking models. Arena's methodology takes a people-centered stance, seeking to reward the models that benefit users. Human preference is an important part of that benefit, but at Arena they control for factors such as response style and verbosity, and they incorporate factuality as one further signal that can improve how useful models are to people.

On the recent release of Agent Arena, which introduces task categories, cost per task and a performance-cost frontier, Angelopoulos believes that, since models differ in their token consumption per task, measuring cost per completed task tends to be more informative than price per token. He states that their randomized methodology guarantees the leaderboard remains fair across tasks of differing difficulty, since each model receives, on average, tasks of the same difficulty.

On the risk of labs optimizing directly to climb Arena (Goodhart's law), he replies that Arena's purpose is to measure AI's real usefulness to people, and that in doing so it gives labs an incentive to develop models that benefit humanity. He says he is glad labs try to climb the leaderboard, because that represents the real utility millions of users experience on Arena; his philosophy is that if climbing the ranking leads labs to produce models that are more useful to Arena's global and diverse community, that is good for the world.

On AutoEval, the system that gives new models a score from day one before enough human votes come in, he says they continuously recalibrate the AutoEval model on a weekly basis to ensure it always reflects the most recent preferences.

Asked whether the idea of a single 'best model' is beginning to fall apart as models become more agentic, multimodal and dynamically routed, Angelopoulos previews that Arena will go well beyond ranking models alone: harnesses (execution environments) and tools will also have to be part of the ranking conversation, and he promises more news from Arena on these topics soon.

On Chatbot Arena's transition from a Berkeley research project to a company serving model labs and enterprises, he notes that the main change has been having far more resources with which to fulfill its mission, with no change at all in Arena's level of openness and neutrality: they maintain a strong academic research component and he says science remains in their DNA.

Asked about the hardest unsolved problems at the frontier of AI evaluation, he mentions that there are a great many challenges, from social science questions — such as properly defining what 'utility' is — to technical ones, such as building personalized evaluations that yield granular estimates of a model's performance at the individual level. He describes it as a rich research area with many ways to contribute.

In the closing, more personal section, Angelopoulos offers a prediction deliberately at odds with what he thinks 90% of readers would believe: the recent improvements in the performance of Chinese labs lead him to think they could end up surpassing all US labs within the next year, a prediction he calls controversial but gives better than 50% odds.

Finally, asked about his favorite mathematician or computer scientist, he cites Vladimir Vovk as the most influential figure in his training: he says he has followed his work on conformal prediction since his first year of his PhD, and that it was by following that work that he learned statistics. He also mentions Emmanuel Candes, whose work on compressed sensing inspired him from his undergraduate years, and says he counts himself fortunate to have been able to interact with both of them in person.