A transformer is a stack of blocks; each block ends in an MLP whose final matrix — down_proj — writes the block's output back into the residual stream. ANNEAL picks a few blocks (by default the full-attention layers of a hybrid model, ~6–16 of them) and gives each one's down_proj an additive shadow matrix Δ, initialized to zero:
At zero, the shadow is invisible — the model is byte-identical to the original. Everything ANNEAL does is a story about what gets written into Δ, and how much of it survives.
When a document streams through, ANNEAL chops it into 256-token chunks and takes one SGD step per chunk — updating only the Δ matrices (0.1–3% of parameters). The base model never feels a gradient.
The catch: keep writing and Δ overshoots. On real domain text, an unconsolidated adapter often makes the model worse than not adapting at all — in our experiments, up to −0.49 nats on held-out text. Raw fast weights aren't memory. They're a workbench.
Here's the empirical fact the method is built on: useful learning and harmful drift live in the same directions of Δ — you can't separate them by rank or magnitude. What separates them is how far you travelled. So the right filter is embarrassingly simple: scale each layer's Δ by a number between 0 and 1.2, and pick the numbers well.
Because it's only 5–16 numbers, a forward-only evolutionary search (CMA-ES, a few hundred model evaluations) solves it on the same GPU that runs inference. No gradients, no optimizer state, no training loop. The search is scored against retention probes — held-out passages that must not get worse — so drift is penalized during the search, not audited after.
Merging raw Δ at full strength (α=1 everywhere) is a quench — you freeze in everything, including the overshoot. The α-search is a slow anneal — it walks the temperature down and lets the structure that generalizes crystallize while the noise melts away. Same lattice, same heat, different schedule:
Run the consolidation search enough times, across tasks and scales, and the α vectors show consistent anatomy: a mid-depth layer pinned at α≈1.0 in every single run (the shared generalizer — we now clamp it and skip searching it), late layers swinging between 0 and 1.2 per task (specialization slots), and early layers the search actively suppresses at scale.
The pattern strengthens with model scale — late-to-early median α ratio goes 1.9× → 17× → 7.6× at 0.8B → 9B → 27B. Whatever new domain knowledge is, larger models store it later.
20 of 20 (scale, domain) pairs improve after consolidation — while naive fast-weight adaptation degrades the model in most of them. The gains are honest: statistically significant at every scale (z > 5), and small in absolute nats at 27B because a strong model has less headroom on any single domain. The method's job is to buy what's available cleanly — and never make your model worse. That second part is what the consolidation step is for.
| scale | naive TTT | ANNEAL | drift (probes) | domains improved |
|---|---|---|---|---|
| 0.8B | −0.188 | +0.065 | ≤0.06 | 5/5 |
| 4B | −0.051 | +0.025 | ≤0.04 | 5/5 |
| 9B | −0.003 | +0.022 | ≤0.03 | 5/5 |
| 27B | −0.070 | +0.011 | ≤0.03 | 5/5 |
Click each card to flip between the model before and after.
Facts stated repeatedly across varied framings land reliably. Facts mentioned once land inconsistently — that's the design boundary: ANNEAL is gist memory, not a verbatim store. Need one specific fact to stick? Synthesize a dozen variations of it and anneal on those (examples/fact_learning.py). Need exact recall of IDs and quotes? Compose with retrieval.
every number on this page comes from a result JSON in the repo