Eighteen months of data across 46 models, and what I think it means for anyone building on top of them.

With all the hype around models, I wanted to understand the data trends firsthand and answer a few questions that kept coming up:

1. Is intelligence actually getting commoditized?

2. Are open models really catching the frontier, or just getting expensive themselves?

3. Is a benchmark score really a property of a model?

4. What is the true switching cost between models?

Models are obviously better than they were 18 months ago. The explanations I kept hearing for why did not hold up when I checked them. "They are just bigger" is wrong. "Open source is winning" is half right. "Look at the benchmark scores" turns out to be close to useless, and that one took me a while to accept.

So I pulled data on 46 models from 8 labs and read up on how frontier labs actually train these things. Three parts below: what changed and why it matters, what the open versus frontier argument gets wrong, and where I think this goes. The last part is opinion and I have tried to mark the seam.

Part 1. Where the models are going, and it is not scaling laws

Sparsity: In 2023, 27% of the model used per token. Now it is under 2%.

Mistral 8x7B had 46.7 billion parameters in December 2023, and its mixture-of-experts (MoE) architecture activated 12.9 billion of them for any given token. That works out to 27.6%. 

Kimi K3, released last month, has 2.8 trillion parameters and activates about 50 billion. That is 1.8%.

The model got sixty times bigger. The part that actually runs got fifteen times smaller.

Chart 1. Every open-weight mixture-of-experts release. Bubble size is total parameters. The red line tracks the sparsest model shipped to date.

In a MoE model, a router picks a few expert sub-networks per token. Total parameters govern how much the model knows. Active parameters govern what you pay to run it. Sparsity pried those two numbers apart, and once they were apart, labs could grow the knowledge store without growing the serving bill.

The capability data shows the consequence. Llama 3.1 405B used 405 billion active parameters to score 9 on the Artificial Analysis index. GLM-5.2 scores 51 on 40 billion active. Across the whole field, active parameters have stayed between roughly 10B and 55B for two years while capability climbed steeply and total counts blew past a trillion.

Chart 2. Capability rising while active parameter counts hold a narrow band. Grey dots are total parameters, coloured dots are active.

That second number has barely moved in two years while capability climbed steeply. It is the main reason good models became affordable.

One caveat. This saves compute, not memory. All 2.8 trillion parameters still have to sit somewhere, so the cost shifts rather than disappearing. It also makes serving harder, since you need to batch many requests together to keep the experts busy enough to be worth it.

Reasoning: reward the process, not the answer

An outcome reward model scores the final answer. Did it get 42? Good. This is easy to build wherever results can be verified automatically, which is why math and code led the way, and it is roughly what DeepSeek-R1 used. The weakness is that a model can reach the right answer through badly broken steps, and outcome supervision has no way to tell the difference. It reinforces the broken path just as happily.

This is where reasoning models came in. A process reward model scores each step instead. OpenAI's "Let's Verify Step by Step" work put a number on the gap: process supervision solved 78% of the MATH test set, well above outcome-only supervision. They released PRM800K, 800,000 human labels on individual reasoning steps, so the approach could be reproduced.

A right answer reached by wrong steps is a landmine. It passes your eval and fails in production, in a way that is hard to attribute afterwards. This has greater implications for agents than for benchmark tables.

Then there is the verbosity problem, which is live rather than solved. Models trained with reinforcement learning on verifiable rewards tend to overthink, generating very long reasoning traces that do not improve the answer. The obvious fix is to penalize length, and the literature is blunt that this often backfires, because a reward attached to the whole trajectory does not map cleanly onto token-by-token optimization.

Two things follow. First, reasoning effort has become a user-facing dial on nearly every frontier model, because nobody has found a setting that is right for all tasks. Secondly, cost per token stopped meaning much, since a reasoning model can emit ten to twenty times the tokens for the same job. The number that matters is cost per completed task, and no vendor publishes it.

Training: the real break was compression, not scale

This is where I think scaling laws first started breaking, and it is not the part that gets talked about.

The key ratio is training tokens per parameter. GPT-3 in 2020 had 175 billion parameters and saw about 300 billion tokens, under two tokens per parameter. In 2022, the Chinchilla paper showed compute-optimal was closer to twenty. Llama 3 8B was then trained at roughly 1,900 tokens per parameter, about ninety-five times past Chinchilla optimal, and frontier models now routinely run above a thousand.

The amount of data pushed through each parameter has gone up by roughly three orders of magnitude in five years. That is compression, and it is a different lever from scale.

Labs do this deliberately, and the reason is economics rather than research. Chinchilla optimizes the cost of training once. You then serve the model billions of times. A smaller model trained on more data reaches the same loss. It also costs less on every request for the rest of its life.

The interesting side effect is what compression does to the model. Forcing far more data through far fewer parameters means the model cannot store it all. It has to abstract. Andrej Karpathy has been making this argument for a while: models are, in his words, way too good at memorization, and that perfect recall gets in the way of reasoning. He points out that children are extremely bad at recalling information, and are still the best learners we know. Being unable to store everything forces you to generalize. His stated goal is to remove the memory and keep only the algorithms for thought. He calls the target a cognitive core: a few-billion-parameter model that trades encyclopedic knowledge for capability and looks the rest up.

Forgetting is a feature. The compression that started as an inference cost optimization turned out to push models from memorizing toward understanding.

That reframes the three training stages. The focus is less on what each one is and more on how fast each is changing.

Distillation is the sharpest expression of all this. A large expensive model writes out clean reasoning traces, a small model learns to imitate them, and the small model inherits the reasoning without inheriting the knowledge store. Qwen3.5 4B is the example worth looking at: it reasons well and will happily get a factual question wrong, then becomes formidable the moment you give it a search tool.

Which raises the question labs are now actually facing: If most gains come from mid-training and post-training, and small models can learn from large models' outputs, does every lab still need to run its own giant pre-training job?

Pre-training has historically consumed the overwhelming majority of training compute. If it stops being the differentiator, two things follow for anyone building. Time to market collapses, because starting from someone else's base and specializing takes months rather than years, which is exactly what open-weight releases enable. And, the enormous fixed cost that has to be recovered across every API call gets smaller, which should eventually show up in price.

The counterweight is that reinforcement learning compute in post-training is scaling quickly, so the saving may be partly reallocated rather than banked. I would treat the brain analogy as useful and no more. Transformers are not neurons and the mapping breaks as soon as you push on it. It does capture where the effort moved, though: away from storage, toward abstraction.

How fast is this moving?

This is the easy one. The time taken to make the same strides is falling, and two independent measurements agree. METR's task-horizon metric, which tracks how long a task an agent can finish, used to double every 196 days. Measured from 2024 onward it is 89. On GPQA Diamond the time to halve the remaining error went from roughly 20 months to about 2.5.

Chart 3. Capability-doubling time shrinking, and release cadence compressing alongside it.

Chart 4. Remaining error on a log scale. Each halving lands sooner than the last.

When measuring intelligence it is more useful to watch the error rate fall than the score rise. A five point jump from 50 to 55 is not the same achievement as 80 to 85. Log-scaling the remaining error gives a more accurate picture of progress for exactly that reason.

That error figure depends entirely on the ceiling you assume, and GPQA Diamond's ceiling is not 100%. Somewhere between 5% and 8% of its questions are ambiguous or have contested answers, so the practical maximum sits around 92 to 95. Assume 100 and you get a 9.6 month halving. Assume 92 and you get 3.8. Same data, 2.5x swing. The top models now sit within about one question of each other, which means the benchmark is measuring luck about as much as capability.

Part 2. Open versus frontier, and what the argument gets wrong

I went in assuming open-weight models were both catching up and pulling away on price. The first half is true. The second is where I was wrong, and the correction changed how I would plan.

Epoch AI finds the open-weight lag behind the closed frontier has been about four months since January 2026, up from about three months over the preceding two years. The gap widened slightly. What changed instead was positioning. Kimi K3 landed within three index points of the frontier and priced itself at roughly Claude Sonnet tier, a sizeable increase over its own predecessor.

Chart 5. Capability against price. Kimi K3, the star, reaches near-frontier capability at near-frontier pricing rather than budget pricing.

Open models bought their way to the frontier rather than undercutting it. A cheap tier still exists, but it is a different product, not the same one getting cheaper.

DeepSeek V4 Flash reaching a score of 40 at $0.06 per million tokens is remarkable by any standard. But treating that and Kimi K3 as one category produces a roadmap that assumes free capability gains which are not coming.

The comparison itself is broken, or at least not always fair

When I stopped using composite scores and broke the comparison out per benchmark, the gap turned out not to be one number at all.

Chart 6. Top: the gap widens as benchmarks get fresher. Bottom: the same model scores differently depending on who published the number.

The ordering is monotonic and it is not a coincidence. On the two benchmarks that show up in every launch post the gap is essentially zero. On fresh ones with a pinned scaffold and multi-language tasks it is eighteen points. Benchmark selection is itself a signal, incidentally: at least one lab published SWE-bench Pro results while skipping Verified entirely.

Open models fall furthest behind on the newest and most realistic tests, which is exactly the kind of work you would give an agent.

Then the noise swamps what is left. Kimi K3 scores 80.9% on Terminal-Bench 2.1 when an independent party runs it and 88.3% when Moonshot reports it. GLM-5.2 shows up at 40.5% and 54.7% on Humanity's Last Exam depending on which leaderboard you read. That 7.4 point self-reporting spread is larger than the frontier-versus-open gap on three of the five benchmarks above. One popular leaderboard lists fifteen models on Terminal-Bench 2.1 with zero verified results and fifteen self-reported.

Moonshot's own launch table mixes four different agent harnesses across rows, and they note that harnesses which truncate the model's chain of thought cause significant degradation. Their numbers make the case better than I can.

A score is not really a property of the model. It is a property of the model, the scaffold it runs in, and the compute budget it was given. Publish one number and you have hidden the other two.

Cost per token is misunderstood, and switching is oversimplified

Test-time compute means a model can spend more effort on a hard question instead of being permanently larger. A small model that thinks for a while can match a big model that answers immediately, and you only pay for the thinking when you need it. That turns a set of decisions that used to be trivial into most of the engineering:

• Which model gets this query? 

• How much reasoning budget should we grant it? 

• What context carries forward and what gets dropped. Whether to run one agent or five?

The economics push the same way. On the current cost-capability frontier each additional index point costs about 1.24 times the last one. Five points is roughly 2.9x, ten points roughly 8.5x. So the top nine points of capability cost around ten times what everything below them costs.

If you send every request to the best available model, most of that spend buys capability the request never needed.

The obvious fix is to route easy work to a cheaper model, and there is a trap in it that almost nobody prices in. Published prices assume roughly 70% prompt-cache hits, and caches are maintained per provider by construction. Route across vendors mid-session and the hit rate goes to zero.

Reasoning traces do not port between vendors either. And Bedrock's prompt routing, which people assume solves this, only routes within a model family. Cross-vendor routing is yours to build.

You switch to save money and end up paying more than the model you were avoiding. Route between models at the start of a job, not in the middle of one.

What about security and safeguards?

This is the part of the open versus frontier comparison that never shows up on a benchmark chart, and it is really a question about who is responsible for what.

With a frontier API, the provider owns the safety layer. They tune the model, run red teams against it, patch behavior after release, and carry the reputational, and increasingly, the legal exposure. You inherit that work whether you value it or not, and it is part of what the price buys.

Open weights invert the arrangement. You get the weights, and the safety posture becomes yours. The uncomfortable finding is how thin the built-in protections are. A joint Financial Times and Alice investigation published in May 2026 tested this. A freely available tool stripped the safety protections from Llama 3.3 and Gemma 3 in under ten minutes, on a standard laptop. Once modified, the models answered prompts on biological weapons, malware, and child sexual abuse material that the originals were built to refuse. The tool had reportedly been used to produce over 3,500 modified variants with 13 million cumulative downloads. Separate work puts the compute cost of a fine-tuning run that strips alignment at under fifty dollars.

Guardrails applied during training can be identified and removed afterwards. For a locally deployed model, your governance controls, rather than the model's training, are the load-bearing safety mechanism.

The labs know this. OpenAI adversarially fine-tuned gpt-oss-120b before release, deliberately trying to strip its safety, and treated the result as a release condition. A deployer running their own fine-tunes downstream has no equivalent gate. 

It is also why a small industry of guard models now exists, from Llama Guard and Prompt Guard to gpt-oss-safeguard. Their only job is to watch the other model. Research on multi-turn attacks adds a wrinkle: open-weight models tend to degrade over a long conversation rather than failing at the first prompt, and capability-focused releases hold up worse than safety-focused ones.

There is a regulatory edge too. The EU AI Act gives open-source general-purpose models a carve-out from some documentation obligations. But obligations land on the enterprise deployer once a significant modification is made, and a LoRA fine-tune can qualify. Enforcement dates arrive in August 2026. Most teams running fine-tunes have not read that clause.

Running several models from several providers shifts the responsibility again, this time sideways. Every additional vendor is another data processing agreement, another region, another certification scope, another content policy that can change under you, and another uptime dependency. Route through an aggregator to simplify that and you have handed security, resilience, and governance to a third party. Your incident response now depends on theirs.

With a frontier API, you rent the safety work. With open weights, you own it. With five providers, you have outsourced it to whoever sits in the middle. None of those is wrong, but only one of them is free.

So which model should you use?

Open weights are genuinely competitive for well-specified, short-horizon work where you control the scaffold and can measure the result yourself. Classification, extraction, summarization, structured generation, the long tail of internal tooling. On those, the capability gap is inside the measurement error and the price difference is material.

The frontier still earns its price on long-horizon agentic work, instruction adherence under pressure, and anything where a failure is expensive to detect after the fact. That is where the fresh benchmarks put the gap at eighteen points rather than zero, and where the safety layer you are renting has the most value.

There is also a procurement dimension the charts never show. Open weights means the weights, not the data, and licenses vary more than people expect. Kimi K3 was still pending a public license when I pulled this data. For anything shipping in a regulated product, that is a legal review, not a footnote.

This is a narrower claim than either camp usually makes, but it is the one the data supports.

Part 3. What could come next

This part is opinion built on the data above rather than a finding from it.

Pre-training's era is ending, which should make models cheaper

Sutskever's line at NeurIPS was that pre-training as we know it will end, because compute keeps growing while data does not. He called data the fossil fuel of AI. There is one internet, we have largely consumed it, and the returns from scaling that stage alone have flattened.

I do not read that as progress stalling. Everything in Part 1 is evidence the work moved elsewhere: to compression and mid-training, to reinforcement learning on verifiable rewards, to inference-time compute. What I do expect is the cost consequence described earlier, arriving slowly. If the differentiator stops being who ran the biggest pre-training job, that fixed cost stops needing to be amortized across every API call.

The counterpoint is that this has been predicted before and the timeline keeps slipping. Labs are still spending enormous sums on pre-training, and synthetic data may extend the runway further than the peak-data argument allows.

Baseline intelligence goes to nearly free. Judgment gets expensive.

This one is already visible in the pricing. A 4B model scoring 20 on the index costs $0.04 per million tokens. DeepSeek V4 Flash scores 40 for $0.06. Competent general-purpose language understanding is approaching a rounding error.

Chart 7. Cheapest model that reaches each capability score. A straight line on a log axis means constant percentage cost per point.

Meanwhile, the top of the curve compounds, and Epoch's work shows the deflation is concentrated up there. The highest capability tier fell roughly 31x per year while the cheapest tier fell only about 1.7x. The cheap tier is barely deflating because it has almost nowhere left to fall.

Summarizing, extracting, classifying, drafting, and translating become utilities. What stays expensive is judgment: knowing which of five plausible answers is right, noticing the question was wrong, deciding when to stop and ask a human.

Scaffolding is the bottleneck, not the model

Part 2 made the empirical case for this without setting out to. If the measured gap between models depends mostly on how fresh the benchmark is and who ran it, then the harness is doing more work than the weights.

For multi-agent systems there is still no credible public benchmark, and I do not think that is an oversight. Swarm performance is dominated by orchestration decisions: context handoff, error propagation, retry policy, cost control. Nobody can benchmark your architecture for you.

The arithmetic that reframes it: a step that is 95% reliable, run 20 times in sequence, succeeds 36% of the time.

Choosing a better model moves that by a couple of points. Checkpointing, verification and idempotent retries decide whether the product works at all.

The moats move to where the model is not

If baseline capability is nearly free and anyone can rent the frontier by the token, the model stops being a moat for anyone except the handful of labs training them. Four things look defensible to me.

• Domain depth is the most durable of the four and the hardest to buy. Proprietary data, workflows, and the edge cases that never appear in a pre-training corpus.

• Orchestration is unglamorous and it is most of the value. Routing, fallback, verification, cost control, graceful degradation. The 36% arithmetic lives here.

• Long-term memory compounds in a way the model layer cannot replicate. Context windows keep growing but they reset. Systems that accumulate durable state about a customer environment get better with use.

• Your own eval set turns out to be an asset rather than an overhead. 50 to 200 real tasks, pinned harness, repeated runs, scored on end state rather than on how the output reads.

None of those are model capabilities, and all of them get more valuable as models commoditize.

What I would tell someone building right now

Route at task boundaries rather than turn boundaries, and pin a model for the whole job so the cache survives. Start with two models, not five, and count the governance cost of each additional provider rather than only the token price. Measure cost per accepted outcome instead of cost per token. Build the private eval set early, because it is the only number that reflects your actual workload. And for agents, report pass^k rather than pass@k, since succeeding once in five tries is a failure when it touches production data.

The bigger takeaway: public benchmarks are excellent for understanding the shape of progress and a poor basis for a buying decision.

Everything above is built from public benchmarks and they were the right tool for the question I was asking. But the eval that should drive an architecture is the one nobody else can run.

Sources: Artificial Analysis Intelligence Index v4.1, Epoch AI, METR Time Horizon 1.1, vals.ai, lab model cards, Hoffmann et al. on compute-optimal scaling, Lightman et al. on step-level supervision, Karpathy on the cognitive core, Sutskever at NeurIPS 2024, and the Financial Times and Alice investigation into open-weight safeguards. Snapshot July 2026. Index scores are restated under a single version and do not compare to older charts. Parameter data covers open-weight models only, since proprietary labs do not publish counts.