Calibration. Mechanism, geometry, and constraint

To predict the next token well, the model has to assign high probability to tokens that actually occur and low probability to ones that don't. Doing that across the training distribution requires capturing the regularities that produced the data: syntax, semantics, world facts, stylistic patterns.

Compression works the same way. Coding the data with the fewest bits requires giving short codes to likely sequences and long codes to unlikely ones. Both operations need an accurate probability distribution over sequences, and the parameters of the model are that distribution.

That's why a model trained on next-token cross-entropy is, at convergence, a learned compressor of its training distribution. The compression isn't a side effect. It's the same operation looked at from the other side: predict well = compress well.

Test-set accuracy alone can't separate the two, because the test set is drawn from the same distribution as the training set. A memoriser and an abstractor can both score 95% there.

The operational test is to probe outside the training distribution. Construct inputs that share the underlying structure of the training data but have different surface form: synonyms, paraphrases, novel combinations, shifted contexts, harder compositional cases. A model that abstracted the underlying mechanism handles these. A model that memorised the surface patterns collapses.

If accuracy stays high under genuine shift, the model has captured the structure that produced the data. If it craters, the model stored the examples rather than the rule. The training metric never tells you which one happened; only behaviour on inputs the training distribution didn't include does.

Two mechanisms are commonly behind what looks like emergence. The first is a representation-capacity threshold: below a certain parameter count, the model literally cannot represent the subskill (e.g. the circuit needed for 3-digit addition won't fit), so the optimiser can't find it no matter how long you train. Above the threshold, the optimiser finds a basin that implements the subskill, and capability appears to switch on.

The second is a measurement effect. Many benchmarks score exact-match: get the whole answer right or get zero. The underlying capability may be improving smoothly with scale, but the score stays at zero until the smooth probability of full correctness crosses the threshold that produces a registered "1". Change the metric to one that rewards partial progress (log-likelihood, per-token accuracy, partial credit) and the curve straightens out.

Both are mechanistic. Neither requires anything mystical. Calling it "emergence" without naming the mechanism is the failure mode this lesson is trying to prevent.

Gradient descent moves parameters in the direction that lowers the loss. The loss function is the only thing the optimiser sees of "what the model is for". Change the loss, and you change the direction the parameters get pushed.

Different objectives reward different internal structure. Cross-entropy on next-token prediction rewards a model that captures sequential statistics. Contrastive loss rewards a model that pulls similar inputs together in embedding space and pushes dissimilar ones apart. Reward maximisation in RL rewards a model that picks actions which produce high reward downstream. Same architecture, same data, different objectives: you get different representations and different capabilities, because the optimiser settled into different basins.

This is why "what was the training objective" is one of the first questions to ask about any AI system. Capability is downstream of the basin the optimiser found. The basin is downstream of the objective. Change the objective and the system is, internally, a different machine.

An embedding is a vector of numbers, typically 100 to 2000 dimensions, that represents an input as a point in a continuous space. The numbers are learned during training. They get tuned so that the training objective is satisfied.

If the objective rewards similar-meaning inputs being treated similarly (e.g. contrastive loss, masked-language-model loss), then "similar meaning" gets baked into the geometry as "nearby". Once that's true, dot product captures similarity, distance captures dissimilarity, and consistent semantic differences (singular vs plural, capital vs country) often show up as repeatable directions. The geometry isn't decoration. It's the form the model computes on.

GPUs were originally designed for graphics, which involves applying the same operation to millions of pixels independently. That means thousands of small arithmetic units running in parallel, fed by high-bandwidth memory. The defining property of the silicon is high throughput on parallel matrix multiplication.

Neural networks are mostly matrix multiplications. Forward and backward passes through a deep network are stacks of matmuls plus a few cheaper element-wise operations. On a CPU, those matmuls run roughly serially. On a GPU, they run roughly in parallel. The training-time compute budget for the same wall-clock budget jumps by one to two orders of magnitude.

That budget shift made deeper, wider models feasible to train. Architectures that fit the hardware (heavy on matmul, light on branching) won; architectures that didn't (recurrent nets with serial dependencies) lost ground. The transformer is the clearest example: attention is dominated by large matrix multiplications, which is precisely what tensor cores accelerate.

So the causal chain runs: parallel matmul throughput → larger training-time compute budgets → matmul-shaped architectures become dominant → frontier-scale models become trainable in finite time. Without the hardware property at the bottom, none of the layers above could exist at the scale they do.

Hardware: the model is large enough to encode broad linguistic and stylistic patterns because tensor-core accelerated matmuls and HBM-class memory bandwidth let it be trained at this scale.

Architecture: the model produces a next token from context using attention plus MLP layers. Nothing in the architecture consults an external truth source. Everything the model emits comes from its parameters.

Representation: the embedding geometry encodes "well-formed legal citation" as a region populated by plausible (year, court, party-name, page-number) patterns. Real and fictional citations of the right shape occupy roughly the same region. The geometry doesn't separate "exists" from "looks like it could exist".

Optimisation: the loss rewarded statistically likely next tokens, not true tokens. The optimiser settled into basins that produce fluent, well-formatted output. Truth-grounding was never part of the gradient signal.

Capability surface: the model is reliable in the interpolation region (common citations seen many times in training) and brittle on niche ones. At the brittle edge, the most-likely continuation is a plausible synthesis of nearby patterns. The model is confident because no calibrated uncertainty signal is exposed; confidence is the default at every position.

The hallucination is the joint output of all five layers behaving as designed. Fixing it requires changing the system, not the prompt: retrieval grounding, tool use, or human verification.

Without retrieval, the only source of facts the model has is its parameters. To answer correctly, the model has to have stored the fact somewhere in its weights and be able to surface it from context cues alone. This is unreliable on niche facts, because the gradient signal for any single fact during training was tiny and may have been dominated by neighbouring patterns.

Retrieval changes the input. A separate system pulls relevant documents from an external store and concatenates them into the model's context window. Now the answer doesn't have to be in the weights. It has to be readable from the context. Models are far better at copying and synthesising information that's present in front of them than at recalling it from parameters.

Mechanistically, the change happens at the input layer, not the architecture or the optimisation. The model itself is unchanged. What changed is the distribution of what's in context at inference time. The capability surface shifts because the brittle region (rare facts) is now serviced by the retrieval store instead of by parametric memory.

Failure modes shift accordingly. Retrieval introduces new ones: bad retrieval, stale corpus, contradictions between retrieved passages, prompt injection in retrieved documents. The system is more factual on average and brittle in different places.

Take KV-cache as the worked example. During autoregressive generation, the transformer needs the keys and values from every previous token to compute attention for the next one. Recomputing them every step would mean redoing most of the forward pass on every token, which is unaffordable.

The hardware constraint: VRAM bandwidth between accelerator memory and compute is the bottleneck. Recomputing means moving the same weights and activations over the bus on every token. The fix is to cache K and V tensors in VRAM and reuse them. That converts re-computation into memory access, which the hardware does better.

That choice then creates new constraints. The KV cache grows linearly with context length, and for a 70B model at long context it can easily dwarf the model weights themselves. So the next generation of architectural moves (grouped-query attention, multi-query attention, sliding-window, latent attention) all attack the size of the cache, not the cleverness of attention. They are downstream of the same VRAM-and-bandwidth wall.

The shape of the architecture is being bent by the shape of the silicon. Read any new LLM paper and the constraint surface is visible in the design choices, if you know to look for it.

The dominant layer is the joint of representation and optimisation. The architecture is capable of implementing arithmetic in principle (transformers can encode addition and multiplication circuits given enough depth and the right training). The hardware is fine. The problem is what the optimisation actually shaped.

The training data contains far more text describing arithmetic than it does worked examples of arithmetic carried out correctly digit by digit. Cross-entropy on next-token prediction rewards producing fluent text about the procedure, because that's what's overwhelmingly present. It does not specifically reward producing the right digit in column three.

The model's internal representation of "multiply" is therefore much closer to the linguistic shape of the operation than to the actual digit-by-digit computation. When asked to describe the algorithm, it draws on the linguistic representation and succeeds. When asked to execute it, it has no faithful circuit to fall through to, so the output drifts.

The fix sits upstream of the model: chain-of-thought prompting forces the model to externalise the intermediate steps, tool use offloads the operation to a calculator, fine-tuning on long worked examples builds a denser arithmetic signal. Each works by changing either the input distribution or the gradient signal. The mechanism is the same problem looked at from different sides.

RLHF trains the model to maximise a learned reward function that's meant to capture human preference. The reward function is itself a model, trained on pairwise comparisons. Whatever signals the reward model picked up are what the policy gets pushed toward.

If the comparison data systematically rated hedged, caveated, refusing answers as "safer" than direct ones (because annotators were instructed to penalise confident incorrect output), the reward model will reward hedging. The optimiser will then drive the policy into a basin that hedges, even on cases where a direct answer is what the human actually wants.

This is the proxy-objective failure: the optimised quantity (reward-model score) is no longer a faithful stand-in for the underlying goal (human helpfulness). The optimiser doesn't know there's a gap. It optimises what it has.

Diagnostically, the failure is in the optimisation layer, but the root cause is upstream: the reward model was trained on data that conflated safety with hedging. Fixes have to address either the comparison data, the reward model, or the policy update (e.g. KL regularisation back toward the base model, or a different post-training objective entirely).

Semantic search works by mapping the query to a vector and finding nearest neighbours in the document space. For that to give useful results, the embedding geometry has to encode "semantically similar" as "spatially close" in the region the query falls into.

The geometry is built by training. Wherever the training distribution was dense (common topics, common phrasings), the embedding has been pulled into a useful shape: clusters, directions, well-separated neighbours. Wherever the training distribution was sparse (rare domains, jargon, niche phrasing), the geometry is essentially unconstrained. Vectors in sparse regions land near whatever was closest in the loss function's residual, which has no semantic meaning.

So the system works in the dense region and degrades to near-random in the sparse region. This is the same mechanism as interpolation vs extrapolation for any learned function. The embedding model is doing exactly what it was trained to do; the region of input space it was trained on doesn't cover the query.

Fixes are domain-adaptation moves: fine-tune the embedding on in-domain data, or use a hybrid retrieval setup that falls back to lexical matching where the embedding can't carry the load.

Parameter count and architecture are necessary but not sufficient. They define the space of functions the model can in principle implement. They don't determine which function gets implemented. That's the optimiser's job, and the optimiser only sees the data and the loss.

If model A was trained with a much higher proportion of source code, with curated multi-step problem traces, or with code-specific objectives (fill-in-the-middle, repository-level context), the optimiser settled into basins that implement code-shaped circuits. If model B was trained with more mathematical text, more multi-step reasoning data, or post-trained with reasoning-focused RL, its basins implement different circuits.

Same skeleton, different musculature. The capability profile is a direct readout of the training mix and the objectives applied during pretraining and post-training.

This is the cleanest illustration of the second core law: optimisation shapes capability. Architecture sets what's reachable; the training signal decides what gets reached. Reading a model's behaviour is, in part, reading the training process backward.

An example. Others are equally valid.

"Every modern AI system runs the same loop: input becomes a representation, the representation is shaped by an optimisation process driven by a loss signal, and the resulting output feeds back through the loss to update the parameters. The shape of that loop is set by a causal chain that runs upward from the hardware. The silicon decides which architectures are feasible at scale; the architecture decides which representations the model can build; the representation decides which optimisation basins are reachable; the basin the optimiser settles into decides what the system is capable of; and capability defines its own inverse, the failure surface. So the form of the input matters because the model only computes on what the representation can carry. The training objective matters because it's the only signal the optimiser has. The geometry of the learned representation matters because generalisation lives in how the unfamiliar inputs land near the familiar ones. And every architectural choice you see is downstream of a constraint somewhere lower in the stack."

An example sketch.

Two main panels. On the left, a vertical stack of six layers in this order, bottom to top: hardware, architecture, representation, optimisation, capability, failure surface. Upward arrows between each pair, labelled with the core law that connects them: hardware-shapes-architecture, representation-shapes-computation, optimisation-shapes-capability. The top boundary is jagged, illustrating the jagged perimeter from L10.

On the right, the L1 systems loop as four boxes in a square: input, representation, optimisation, output. Clockwise arrows around the loop, plus a feedback arrow from optimisation back to representation labelled "loss signal". A small annotation: "every system in the phase fits this shape; the slots are filled differently".

Bottom strip: the five core laws written out as one-line claims, with the law that connects each pair of layers in the left stack already labelled above. The fifth law (constraints shape systems) labels the whole diagram, not any single arrow.

What gets cut: any worked example (the LLM-vs-CNN-vs-RL table), tokenisation specifics, individual lesson titles, anything decorative. The diagram is the mechanism; the lessons are the worked examples that grounded it.