Counting parameters isn't counting intelligence: the lesson hidden in LLM architecture

By Giles Thomas's blog · July 10, 2026.

Giles Thomas, a developer who has been documenting on his blog the construction of a GPT-2 from scratch in JAX, describes a finding that carries more substance than it appears at first glance. When he assembled only the input embedding layer and an output head —no Transformer blocks, no attention, no feed-forward networks— that skeleton already amounted to 77 million parameters, with an embedding dimension of 768 and a vocabulary of 50,257 tokens: each of those two matrices exceeds 38 million numbers on its own. When he finished the complete model, with all the attention blocks and feed-forward networks included, the total was 163 million parameters. In other words: the input and output layers, which do no reasoning at all, accounted for nearly half the model. And as if that weren't surprise enough, the feed-forward networks —the least discussed part in any AI tutorial— have roughly twice as many parameters as the attention mechanisms, which are what almost everyone studies first and in the most detail.

The finding is not a niche curiosity. Attention gets all the pedagogical attention because it is, by far, the hardest part of the Transformer to understand; but that makes it easy to overestimate how much weight it actually carries in the total parameter count, especially in small models, where the embedding table and the output head (which in practice are giant dictionaries that translate tokens into vectors and vice versa) weigh far more than the body of the model that actually 'thinks'. Thomas illustrates the effect by taking it to the extreme: with a very large vocabulary —and many current models handle hundreds of thousands of tokens—, it is perfectly possible to build a 'tiny' model that is in reality, almost entirely, a word lookup table disguised as a neural network.

As a side anecdote with its own charm: to visualize all this, Thomas used, on the very day of his post, the 'Sol' variant of GPT-5.6, the model OpenAI has just released, asking it in Codex to build him an interactive tool that breaks down the distribution of parameters among embeddings, attention, feed-forward and the output head for different sizes of GPT-2 (or custom configurations, with or without weight tying or QKV bias). By his account, the result was very good. It is no proof of anything at scale, but it is a small example of how a freshly released frontier model already serves, on its very launch day, to produce useful educational tools —exactly the kind of well-defined engineering task where these systems perform reliably.

Our take: in a sector where parameter counts have been used for years as a marketing shortcut to hint at capability —'this model has X billion parameters'—, this exercise recalls something worth not forgetting: not all parameters are equal, and a substantial fraction of them, especially in compact models, is nothing more than vocabulary infrastructure. This has far from trivial practical consequences for the wave of open and local models we have been following (GLM, Qwen, Gemma, gpt-oss and company): if a model is designed to cover many languages or a very large vocabulary, part of its parameter budget —and therefore of memory and compute— is devoted to that translation table rather than to the layers that actually generate reasoning. Understanding this distinction is exactly the kind of technical literacy that needs to grow if we want more people to be able to evaluate, adapt and deploy their own models instead of trusting size-based headlines. It is a modest piece —a personal blog post, without much reach— but it points in the right direction: the more people truly understand how an AI is built on the inside, the less power the 'more parameters, more intelligence' marketing has, and the easier it becomes to build, with judgment, the small and efficient models that sustain the promise of an AI that is distributed and accessible to everyone, not just to whoever has the biggest cluster.