A math genius that fumbles your spreadsheet: why AI is brilliant at some things and hopeless at others

THE THESIS. AI is neither smart nor dumb. It is uneven, and the reasons can be explained. It advances fast where a machine can check whether an answer is right, as in math or code. It advances slowly where checking is hard, where the knowledge is written down nowhere, or where the job means remembering and chaining many steps over days. Our reading is that this unevenness does not disprove progress. It does disprove a popular shortcut: concluding from an exam that the job is solved. In the short term, that means job substitution will be slower and more uneven than the demos promise. It also means the risk of trusting AI where it fails is very real. In the long term, all three causes of the unevenness are engineering problems, and people are already working on each one.

A MAP OF PEAKS AND VALLEYS. The data draws a map with sharp highs and lows. In July 2025, a Gemini Deep Think model from Google DeepMind scored 35 out of 42 at the International Mathematical Olympiad. That is gold, certified by the olympiad's own graders. OpenAI announced the same score, though its grading was done internally. Now look at the valley. The Remote Labor Index (RLI), built by Scale AI and the Center for AI Safety, gives agents real freelance projects in design, 3D, architecture and data analysis. Then it asks whether a reasonable client would accept the result. In October 2025, the best agent managed it on 2.5% of projects. In 2026, according to CAIS itself, Anthropic's Fable 5 reaches roughly 16%. That is more than four times higher in under eight months, but it still fails five jobs out of six. The best term for this was coined in 2023. A Harvard and Boston Consulting Group study of 758 consultants described a jagged technological frontier, a concept popularized by professor Ethan Mollick. On tasks that fell inside what GPT-4 did well, consultants using AI completed 12.2% more tasks, 25.1% faster, with over 40% higher quality. On a task designed to sit outside that frontier, the opposite happened: they were 19 percentage points less likely to get it right than those working without AI. That is the number that matters. Using AI where it does not work goes beyond failing to help. It makes the result worse.

WHAT CAN BE CHECKED CAN BE TRAINED. The first cause of the unevenness is how models are trained today. Since 2024, labs have used reinforcement learning with verifiable rewards (RLVR) at scale. In plain terms, the model makes millions of attempts and an automatic program tells it which ones are right. In math, you check the result. In programming, you run the tests. That signal is cheap, objective and endless, which is why models keep improving exactly there. Andrej Karpathy, an OpenAI co-founder, sums it up, according to accounts of his talk, with one idea: with language models, it works when you can verify. How do you verify a good report for a demanding client? Or which spreadsheet column is the right one among three conflicting versions? There is no automatic test for that. The second cause is data. The internet is full of proofs and code, but almost nobody writes down how to negotiate with a supplier or why a file is called FINAL_v3_good. This is tacit knowledge: what you learn by doing and never write down. Here an updated version of Moravec's paradox applies. In the 1980s, researcher Hans Moravec noted that machines found easy what we find hard, like chess, and hard what we do without thinking, like walking. Today the line does not run between the intellectual and the physical. It runs between what is verifiable and written down and what is ambiguous and implicit.

THE MEMORY AND LONG-TASK PROBLEM. The third cause is time. METR, an independent evaluation lab, measures agents' time horizon. That is how long a task would take a human expert when the agent completes it with 50% success. According to its January 2026 update, that figure had doubled roughly every four months since 2023. For an early version of Claude Mythos, METR estimates at least 16 hours. METR itself gives three important caveats. First, 50% success is a low bar for a business, and at an 80% bar the horizon shrinks a lot. Second, its tasks are mostly software, machine learning and cybersecurity, which are the verifiable domains. Third, above 16 hours its test suite no longer measures reliably. Then there is memory. An employee learns in the first month and does not make the same mistake twice. A model generally starts every conversation from scratch. MIT's 2025 NANDA project report named this as the main reason why, according to its authors, 95% of corporate generative AI pilots delivered no measurable return: tools that do not retain what they learn or adapt to context. The real office is exactly that. Messy context, decisions that depend on something said three weeks ago, and nobody telling you whether you got it right.

FROM EXAM TO JOB: WHAT TO SEPARATE FROM THE HYPE. OpenAI built GDPval to measure economically valuable work. It uses tasks from 44 occupations, done by professionals with years of experience, and other experts compare the results blind. In December 2025, OpenAI said GPT-5.2 Thinking matched or beat the professional in 70.9% of cases. That is real progress. But each task arrives with all the instructions and files it needs, and it is solved in one go. There are no meetings, no change of plans halfway through, no boss who forgot to mention something. It is a very well designed exam, but it is still an exam. Real adoption moves more slowly. According to the US Census Bureau survey analyzed in an NBER study, 18% of firms used AI in some business function between November 2025 and January 2026. Weighted by employees, the figure rises to 32%, because large firms adopt more. And METR's 2025 randomized study left an uncomfortable lesson. Veteran developers took 19% longer with AI on their own projects, even though they believed they had been 20% faster. The authors warn against generalizing, and the tools have improved a lot since then. Still, it shows that perception misleads. Our reading: a benchmark tells you what a model can do under ideal conditions. Jobs depend on what it does under real conditions, and the gap between the two is the jagged frontier itself.

WHAT THIS SAYS ABOUT ITS FUTURE. The unevenness is not static, and that is the optimistic part. The frontier moves, and in a predictable direction. For workers, the useful question is not whether their job is hard. It is whether the output of their work can be checked automatically. The verifiable will go first: routine code, reconciliations, fixed-format reports. That fits what we have been reporting in our jobs series: back-office work is the most exposed. Judgment, client relationships and handling ambiguity hold out longer. For companies, the lesson of the Harvard study is that they need to know where the frontier sits in their own processes before automating them. The demo is not enough. In the short term, the problems are real. Jobs are exposed in verifiable tasks. Mistakes are costly when AI is trusted outside its turf. And there is a gap between what is promised and what is delivered. In the longer term, all three causes have solutions in sight. Labs are working on verifiers for more open-ended tasks, on persistent memory and on agents able to work for days. And science, from protein folding to drug design, is full of problems that can be checked. That is why the same mechanism that wins math gold today could speed up the fight against disease tomorrow. The AI that wins first is the one that can prove it is right. If machines take over the checkable part of work, people are left with the part that needs judgment, relationships and passion. That is the future worth building, with our feet on the ground.