AI Has Eaten the Internet: Free Data Is Over — Model Collapse Isn't Here

THE THESIS. We have not run out of data: we have run out of free data. The 'peak data' Ilya Sutskever announced is real as the end of an era — the era of unlimited scraping of public human text. But 'model collapse' — the idea that AIs will degenerate by training on their own exhaust — is neither happening nor inevitable. The scientific literature says something more precise: synthetic data destroys when it replaces human data, and works when it accompanies it, gets filtered, and above all can be verified. And the most important consequence is not technical but economic: fresh human text has become a scarce asset that, for the first time, carries a price.

THE WELL AND ITS BOTTOM. The facts first. In December 2024, Sutskever told the NeurIPS conference that 'data is the fossil fuel of AI' and that we have reached 'peak data': 'we have but one internet.' In January 2025, Elon Musk went further: in his words, 'the cumulative sum of human knowledge' useful for AI training 'was exhausted basically last year.' Are they exaggerating? Not entirely. Epoch AI, the reference group for measuring these things, estimated the useful stock of public human text at around 300 trillion tokens — the word fragments used to measure training — and calculated, with 80% confidence, that it would be fully consumed between 2026 and 2032; the group has since refined that window. We are inside it. And while the well drops, the web fills with recycled water: according to the research firm Graphite, AI-generated articles went from 2.2% in January 2020 to overtaking human-written ones in late 2024, and have hovered around 50% of new content since 2025.

WHAT THE COLLAPSE PAPER SAYS (AND DOESN'T). The fear has a serious source: the study by Ilia Shumailov and colleagues published in Nature in 2024. It showed that a model trained again and again on the outputs of previous models degenerates within a few generations. And the first thing lost is not the average but the tails of the distribution: the rare, the minority, the statistically infrequent disappears first. It is a solid mathematical result, appearing in language models and other architectures alike. But it comes with decisive fine print: the experiment replaces real data with synthetic data at every round. When another team (Gerstgrasser and colleagues, 2024) reran the exercise by accumulating — synthetic data added to the human data, which is never deleted — the error stayed bounded and collapse never arrived. That is what actually happens on the real web and inside the labs: mixing, aggressive curation, quality filters. Honesty requires a caveat: the debate is not closed. The paper 'Strong Model Collapse' argues that even with mixing, a synthetic share that does not shrink toward zero can degrade performance. Translation: no apocalypse in sight, but a slow tax on quality if nobody filters.

WHERE SYNTHETIC DATA SHINES (AND WHERE IT DOESN'T). Here is the key the meme ignores: synthetic data is not a patch, it is an engine — but only where a verifier exists. In math and code, an answer can be checked automatically: the test passes or it doesn't, the proof is valid or it isn't. That enables reinforcement learning with verifiable rewards (RLVR): the model generates millions of attempts, the verifier discards the garbage, and only what is correct feeds back into training. The results are spectacular. DeepSeek R1-Zero climbed from 15.6% to 77.9% on the AIME math olympiad using verifiable rewards alone, according to its technical report. Google DeepMind's AlphaGeometry 2 was trained on roughly 100 million synthetic geometry problems and, alongside AlphaProof, earned silver at the International Mathematical Olympiad (28 of 42 points). Microsoft's Phi-4 devoted some 400 billion synthetic tokens — about 40% of its pretraining budget — according to its technical report. By contrast, where no verifier exists — culture, style, facts about the world, what is true today — nobody has shown synthetic data can replace human data. There, the snake really does bite its tail.

OUR READING. The correct headline is not 'the data ran out' but 'free data ran out.' Just look at where the real movement is: Cloudflare has blocked AI crawlers by default since July 2025 and charges for access (Pay Per Crawl), and has bought the startup Human Native to build a licensed data marketplace; Stack Overflow joined pay-per-crawl in February 2026; Reddit is moving from flat fees — roughly $60 million a year from Google and $70 million from OpenAI, per published figures — to usage-based pricing. Human data stops being a lootable resource and becomes a priced asset. And our house rule applies here as always: measure before buying the narrative. 'Peak data' blends a real engineering constraint (pretraining on public text is hitting a ceiling) with self-interested framing: whoever owns proprietary data silos — X, Google, Meta — benefits from declaring the common well dry. Collapse as an apocalyptic meme is refuted by the serious literature; exhaustion as a slogan from those selling the alternative deserves skepticism too.

IMPLICATIONS. In the short term, the problems are real. A half-synthetic web degrades the signal: statistical minorities — small languages, rare topics, underrepresented voices — are the first to blur, and that is a problem of cultural equity, not just engineering. Training costs will rise, because the free-data era was subsidizing progress. In the long term, our usual optimism, with nuance, for three reasons. One: scarcity forces efficiency; a child does not need 300 trillion tokens to learn to speak, and labs will have to move toward that curve. Two: an economy is being born that finally pays human creators for the value their text brings to machines — a better deal than plunder. Three: in verifiable domains, AI already manufactures its own fuel, and that is exactly where the promises we care about live — mathematics, code, science with experimental verification; the road to drugs and cures does not depend on scraping more blogs. The snake is not eating its tail: it is learning to hunt on different ground. The risk worth watching is not that models collapse, but that writing on the web stops being worth it for humans. Protecting that — with fair data markets and clear provenance — means protecting the fuel everything else depends on.