Microsoft Design: why evaluating AI requires giving the user a voice again, not just another model

The Microsoft Design article features a conversation with three UX researchers from Microsoft AI —Christopher Monier and Chuck Kwong, both Principal UX Researchers, and Wendy Wang, who leads Copilot's UX research team— who together built a response-quality evaluation program for Copilot. Their starting point is a distinction that seems simple but is hard to apply in practice: automated evaluations (one model judging another model) can determine whether a response meets predefined criteria, but only users can say whether that response actually helped them. Returning that human voice to the evaluation process is, according to the three of them, the real work still pending.

Each of them came to this problem by a different route. Chuck says he arrived at Microsoft AI without even knowing what an evaluation was, and that it all started from a naive question —why don't we see the product the way users see it?— and from a simple study: asking people to compare Copilot's responses with those of other AI products given the same prompt and to explain what struck them as good, bad, useful or frustrating. The result was revealing: the gap between what internal teams considered a good response and what users actually preferred was far wider than expected. Chris, for his part, started from a vague sense that something didn't fit between one product and another, between one model and another, and suspected that classic UX research tools could help explain that ambiguity. Wendy experienced it from inside the building process: every team that creates AI carries assumptions about what a good response should be —its tone, its shape, its personality— assumptions that qualitative conversations with real users dismantled again and again. Quality depended on the user's context and on the real goal behind the prompt, which is almost never explicit in the prompt itself; content the team would have discarded as off-topic sometimes turned out to be the most useful part of the response; a specific sentence, or a reference to something said earlier in the conversation, could completely change the user's emotional register. That is why the work became about defining precisely what "good" is, but starting from human context rather than from the prior beliefs of those building the product.

If evaluating means measuring quality, the hard part is deciding which dimensions of that quality to measure and, above all, which ones a machine cannot judge. Chris draws his dimensions from those same qualitative conversations, paying attention not only to what people say matters to them, but to the underlying values that can be read between the lines. One constant comes up again and again: although users want AI to save them time when creating documents, they are deeply concerned about losing their own point of view and ending up producing "generic work". Chris usually ends up with between five and eight dimensions —never generic quality checks— and prioritizes human evaluation precisely for what only a person can judge. He gives factual accuracy as an example: a model can mechanically check whether a statement is correct, but only a user can say whether, in the context of their company's strategy and philosophy, that statement is really relevant. As he himself sums it up, you can have an AI judging another AI's factual accuracy, but you can't have an AI judging whether something genuinely matters to a person in the real world; almost by definition, that isn't possible.

For Wendy, the work doesn't end with the evaluation itself. Once the data is collected, her team turns it into "golden data sets": concrete examples that anchor what a good and a bad response look like, and which then serve as seeds for much larger sets evaluated by machines. Just as important, she says, was building a shared vocabulary: a taxonomy of "loss patterns". Calling a response "generic" or "shallow" may be accurate, but it doesn't work as a common language across data science, engineering and product teams. When every named pattern means the same thing to everyone, it becomes an enormous unifying force. With well-built evaluations, UX research doesn't just deliver findings: it stays in the loop, as a bridge to the user, through prompt iteration, data set building and the design of the evaluations themselves.

Asked whether researchers bring something data scientists cannot, all three said yes, though for different reasons. Chuck frames it as a missing voice rather than a turf claim: no discipline should own or lead evaluations on its own, because it is an inherently multidisciplinary effort; but many AI teams build their frameworks leaning on "LLM as a judge" or on expert annotators, and what is really missing is the user's voice. Researchers also bring the measurement capability needed to turn subjective concepts such as trust, usefulness or perceived intelligence into signals teams can genuinely evaluate. Chris agreed "one hundred percent": there is nothing magical about being a researcher as a person, a data scientist could do this work; what is different is the method. Human-centered evaluations incorporate, in his view, everything that makes us human: the imperfections, the inconsistencies. His favorite example is asking users to write their own prompts, because most people don't know how to prompt well —they are tired, busy, distracted— and leave out relevant context. If you generate perfect prompts with another AI and evaluate on those, you lose the reality of how people interact with AI; the only way to close the gap between what the user typed into the text box and what they actually had in mind is to systematically ask real people who use the product. Wendy pointed to a different risk: in the pursuit of rigor, teams tend to over-systematize —single turns, fixed judges, the same controlled set of queries— and in doing so they dilute the kind of learning they could obtain, because they lose the human why behind the data. Her solution, taken directly from research practice, is to treat the entire conversation as a study stimulus: to think of a conversation as a prototype or a concept, dissect it into its components and let that guide the learning. That is where her team's multi-turn evaluations came from, in which people interact with the AI and evaluate the whole exchange, not an isolated response.

On whether the field is converging on a common approach, the three agreed on a "a bit yes, a bit no", with a shared warning about the shelf life of any framework. Chuck still sees many teams building their own frameworks separately, but he perceives the birth of a measurement science at the intersection of AI evaluation, UX research and psychometrics, focused on systematically understanding how people perceive, experience and extract value from AI. Chris describes it as best practices and reference points that are indeed emerging, but that never stand still: the fact that an evaluation was useful six months ago doesn't mean it is still relevant today, because the technological frontier advances and user expectations advance with it. Wendy was blunter about the useful life of these frameworks: they become obsolete in barely three months because people's taste in AI has already evolved by then. The convergence really worth pursuing, they conclude, is not a fixed system —any system will be out of date next quarter— but the practice of staying calibrated to where user taste is heading. This applies to Wendy's own taxonomy too: loss-pattern definitions are revised and new ones added as expectations change, so that the shared language stays current instead of becoming another artifact describing last quarter's problems.

A recurring theme in the conversation is that "evaluating AI" is really several different problems under a single name. Chuck draws the sharpest distinction: evaluating a model is not the same as evaluating the complete product experience in which that model sits. The latter is much harder, he explains, because of the number of confounding variables that come into play: the visual elements around the response, the onboarding process, other product features, the multi-stage flow of which the model is only one part. Agentic workflows complicate it even further, because the industry has still not agreed on how to properly evaluate that entire process from end to end, and some stages the user never sees still shape the final experience. Evaluating the model on its own is more manageable and yields more actionable results; evaluating the complete experience deserves a far more careful approach, with study designs capable of isolating what is really causing a problem. Chris added an underlying trend that is changing how experiences are evaluated: as AI is built into more and more products, its probabilistic nature means the diversity of experiences people can have grows exponentially. Evaluating the breadth and depth of those experiences is key to understanding why metrics such as retention or engagement move; and he is optimistic about the role of UX research, because understanding why people really use something is the very core of that discipline, and the more diverse the experiences, the more questions of that kind there will be to answer. For Wendy, the distinction comes down to what an evaluation is ultimately for: scoring a response tells you whether that particular response was good; the same data, read differently, tells you what the person was really trying to do, and that is where the product ideas are. Sometimes the user just wants a quick answer and nothing more; other times they are dealing with a difficult situation in their life; other times they are just trying to unblock a step within a complicated workflow. These are different jobs with different definitions of success, and seeing them reflected in the data generates new ideas about what the product should be, not just cosmetic tweaks at the response level.

The article closes by noting that, from three very different perspectives within Microsoft AI, the same central figure always appears: the user, whose answer to the question of what is "genuinely good" keeps evolving at the same pace as the technology itself. It also announces that this is one installment in a Microsoft Design series on how different teams are redesigning the way they work around AI, with a forthcoming installment focused on another group of researchers and another corner of this practice in full transformation.