Motivation

The hunt for the learning algorithm of nature

Anurup Ganguli · Independent researcher · August 2026 · about 15 minutes

A failure mode reported in 1989 is still open at frontier scale in 2026. This essay is about what a real answer would have to do: all of it, at once. And about the unusual place I went looking for one.

Conclusions

One finding per section, each linked to the section that argues it.

The failure reported in 1989 is still measurable in 2026. Six frontier systems put through sequences of four to six fine-tuning tasks lost roughly 15% to 32% of capability. A problem reported in 1989 →
Every family of answers works by giving something up. A boundary signal, a buffer of the old data, or a parameter count that grows with the task list. What four decades of answers traded away →
Eight requirements, and no published method holds all eight at once. Each has been met before by someone, which is the point: they have to hold simultaneously. What a good answer would have to do →
Not forgetting is the floor, not the goal. The rung above it is prior material coming back improved, and above that, never-trained material improving anyway. Three bars, not one →
The neuron metaphor was borrowed from cell biology once, in the 1950s, and the borrowing then stopped. The genome, the reprogramming result and the cell atlases all arrived after it had hardened. A different set of role models →
Four factors take an adult skin cell back to a cell that can become anything. Against a state of roughly a million entries, that is not the behaviour of a system where everything is entangled with everything. One file, four entries →
Those questions produced Learner 1.0. The base model stays frozen and the trainable parameters are fixed rather than growing with the number of things learned. What that starting point produced →
On a four-domain stream, all eight requirements held at once. LoRA lost 0.034 to 0.070 nats on the three earlier domains while both Learner 1.0 arms stayed within 0.004, and a fifth domain that nothing ever trained on improved by 0.118 nats. Where this lands →
The mechanism is reserved under NDA pending patent prosecution. What is published instead is the behaviour, in enough detail to check it on your own stream. What I am not telling you →
Five things are unresolved. How far it holds at frontier scale, how many domains it survives, whether backward transfer lasts on a longer stream, serving, and whether the biology is load-bearing. Open questions →

Sources and further reading are at the foot of the page. Further reading →

A problem reported in 1989

In 1989, Michael McCloskey and Neal Cohen trained a connectionist network on one set of associations and then on a second set, and watched the first set disappear.¹ Not degrade. Disappear. They named it catastrophic interference: when a network learns sequentially, the gradients from the new material land on the same weights that were carrying the old material, and overwrite them. Roger Ratcliff reproduced the effect the following year in the context of recognition memory and reached the same conclusion.²

The finding has aged unusually well, in the sense that it is still true. Thirty-seven years later, a 2026 study took six frontier systems: Llama 4 Scout and Maverick, GPT-5.1, Claude Opus 4.5, Gemini 2.5 Pro and DeepSeek-V3.1. It ran them through sequences of four to six fine-tuning tasks and measured capability degradation between roughly 15% and 32%, with 15–23% of the attention heads in lower layers severely disrupted.³ Scale did not dissolve the problem. It made it more expensive to hit.

This is why every large model you use is frozen. It shipped at a checkpoint, and it will die at that checkpoint. Anything it needs to know after that arrives through the context window, or through a retrieval index, or through a retraining run that costs what the original run cost. The model itself does not get better at your codebase in March because it read your codebase in February.

Ability on the first domain drops in a discrete step at every later boundary, and that is the default behaviour of sequential training rather than a tuning failure. Train on one thing after another and each new domain writes over the last, which is why every model you use is frozen at a checkpoint. The shape here is illustrative. The measured version of it, at frontier scale, is in the 2026 study cited above.

If you want to go deeper before continuing, three pieces are worth your time and none of them require this background: a short Quanta explainer on the problem and one line of attack on it,⁴ Dwarkesh Patel's essay arguing that continual learning is the bottleneck rather than a bottleneck,⁵ and Andreessen Horowitz's survey of why the industry needs it.⁶ For the full academic picture there is a recent comprehensive survey.⁷

What four decades of answers traded away

The literature between 1989 and now is large, and I am not going to pretend to summarize it. What is useful here is the shape of it: the attempts fall into a few families, and each family works by giving something up. Naming what each one gives up is the fastest way to see what a complete answer would need.

Penalize changes to weights that mattered before. Elastic Weight Consolidation estimates how important each weight was to what the model already knew and adds a penalty that stiffens those weights against further movement.⁸ Synaptic Intelligence computes that importance online instead of after the fact.⁹ Learning without Forgetting works on the outputs rather than the weights, recording what the old model said and keeping the new one close to it.¹⁰ What this family gives up is the task boundary: to know which weights mattered before, something has to tell it that a "before" ended. Real streams do not come with that signal.

Keep some of the old data and re-show it. Gradient Episodic Memory stores a small buffer from earlier tasks and projects each new gradient so that loss on the buffer cannot increase.¹¹ A-GEM exists because the original's constraint solve grew too expensive as tasks accumulated.¹² This family works. It gives up the ability to operate where you cannot retain the data. That is most regulated industries, most customer data, and most of the settings where continual learning would actually be worth money.

Give each new task its own parameters. Progressive Networks freeze the old network and add a fresh column of parameters for every new task, with lateral connections back to the frozen ones. Forgetting is zero by construction.¹³ It gives up fixed capacity. The parameter count grows with the number of things learned, quadratically in the number of columns, which is the authors' own stated limitation. PackNet keeps capacity fixed by pruning after each task and reusing the freed weights, but then needs to be told which task it is doing at inference time in order to pick the right pruned network.¹⁴

There is a further result that sharpens all of this: when the task identity has to be inferred at test time rather than supplied, the regularization family does not merely degrade, it fails, and something replay-like appears to be required.¹⁵ That is worth sitting with. Inferring the task from the content, rather than being told, is exactly the setting that matters outside a benchmark.

Every family works, and every family works by giving something up: a boundary signal, the old data, or a fixed parameter count. Read down the amber notes rather than the names: they are the requirements section of this essay, written backwards. One representative per family. This is not a survey.

What a good answer would have to do

So what would you actually need? Not "what would beat a benchmark", but what would let a deployed model keep learning from the messy stream a real product produces. I ended up with eight requirements, and the important thing about them is not the list. Most of them are individually well understood. It is that they have to hold simultaneously.

It has to be continual pre-training, not continual fine-tuning. Most of the literature operates on a thousand to twenty-five thousand labelled samples per task. Continuing to pre-train on a billion tokens of a genuinely new domain is three to six orders of magnitude of a different problem.
It has to work at the size models actually are. Seven billion parameters and up. Methods that carry per-weight importance state stop being affordable well before that.
It has to survive several genuinely different domains, not two adjacent ones.
It has to hold at a real token budget per domain: a billion tokens, not a thousand samples.
No replay. No buffer of old data, and no curriculum that mixes old material back in, which is replay wearing a different hat.
No task labels. Nothing tells it a domain started or ended, at training or at inference. There is no boundary signal, because in a real stream there is no boundary.
No penalty term. No importance regularizer, no orthogonality loss, no gradient projection operator bolted on outside the model.
It has to work both ways: trained in from the start, and retrofitted onto a model that already exists.

Any one of these is solvable. Several pairs are solvable. The literature contains no method that holds all eight at once. That is the conclusion of the eight-axis comparison in the paper,¹⁶ and it is the gap the work came out of.

No prior method meets all eight requirements at once. Each one has been met before, by someone, and the open circles do not line up in a column. Each family drops a different pair, which is why they cannot simply be stacked. Rows are representatives of families, not an exhaustive ranking.

One honest exclusion. Serving a model like this to millions of users raises a separate set of problems: how updates batch, what an update costs, how you version something that is no longer a fixed artifact. Those are real and they are not addressed here. This essay is about the learning problem.

Three bars, not one

There is a subtlety in requirement six that took me a while to state properly, and it is the difference between a solution that is adequate and one that is interesting.

The adequate bar is that the model does not forget: you measure what it knew before, you train it on new material, you measure again, and the number has not moved. Almost all of the literature is aimed at this bar, and it is a reasonable place to aim.

But it is not the bar biology sets. The better case is that earlier material comes back improved: that learning the fourth thing made the first thing sharper. In the field's vocabulary that is positive backward transfer, and it is the difference between a system that protects what it has and a system that consolidates. The best case adds the other direction: new material is absorbed faster because of what came before, and material the model was never trained on gets better anyway. That is positive forward transfer, and it is what you would want from something described as learning rather than as storing.

Not forgetting is the floor. Getting better at the old thing by learning the new one is the ceiling.

Not forgetting is the floor, not the goal. The rung above it is prior material coming back improved, and above that, new material arriving faster and never-trained material improving anyway. The right-hand column is where the measurements in §8 landed on this stream. That is one stream, one run per arm, which is not the same as a general claim.

A different set of role models

Here is where my path diverges from most people working on this, and I want to be careful about how strongly I put it.

Nearly everyone I have talked to about the mathematics of what a deep network computes comes at it from mathematics or physics. That is a productive lineage and it has produced almost everything we have. But it means the field's mental model of a neuron was borrowed once, in the 1950s, from what was then known about cells. And then the borrowing essentially stopped. The perceptron dates from 1958. The structure of DNA was five years old. Everything molecular biology learned afterwards about what a cell actually is, how it holds an identity, how it changes one, arrived after the metaphor had already hardened.

Consider the timing. The first draft of the human genome was published in 2001 and finished in 2003. The first genuinely gapless assembly came in 2022. Single-cell atlases now catalogue something like 475 distinct human cell types across two dozen tissues,¹⁷ against a classic enumeration of 411,¹⁸ and finer splitting criteria push the count into the thousands. Ask how much of that fed back into how we design learning systems and the honest answer is: almost none of it.

The neuron metaphor was borrowed once, in the 1950s, and almost nothing crossed the gap afterwards. It came from what was then known about cells. The genome, the reprogramming result and the cell atlases all arrived after the metaphor had hardened. Adjacency is not an argument. It is the reason I went looking in an unusual place.

I spent the better part of the last decade close to this material, and the questions it left me with are not the questions the mathematics leaves you with. Two in particular.

Every one of those 400-odd cell types carries the same genome. A liver cell and a neuron are running identical instructions and are not remotely the same machine. Whatever produces that diversity is not in the instruction set. It is in what is expressed, and when, and in what state the cell is holding.

And a neuron is a cell. This sounds trivial and I do not think it is. A neuron does not get an exemption from the axioms every other cell obeys. It divides from something, it maintains an identity, it holds state, it responds to its neighbours through the same molecular grammar. When we model a neuron as a weighted sum and a nonlinearity, we are modelling one thing it does.

Which leads to the question that actually started this. When a cell divides, the entire machine is disassembled and rebuilt. And the daughter is still a liver cell. Nothing about that is guaranteed by the genome, because the genome is identical in every cell that is not a liver cell. Something else is being preserved through a total physical rebuild, and it survives intact.

A dividing cell disassembles and rebuilds its entire machinery, the daughters are still the same kind of cell, and nothing in the genome guarantees that. The genome is identical in every cell that is not that kind. Something survives a total physical rebuild. This is the question that started the work, not evidence for any answer to it.

One file, four entries

Now hold that next to the result that I find genuinely difficult to stop thinking about.

Picture a cell's state as a file. Not the genome: the genome is the same file in every cell. The state: which proteins are present and at what level, which transcripts are being made, which regions of the DNA are marked as readable or not. Call it a million key–value pairs, which is the right order of magnitude and certainly not exact. That file is what makes the cell a skin cell rather than a stem cell.

In 2006 Kazutoshi Takahashi and Shinya Yamanaka showed that changing four of those entries takes an ordinary adult skin cell all the way back to a pluripotent stem cell, a cell that can become anything.¹⁹ Four. Out of a million. The result was sufficiently unlikely that it took the Nobel Prize in 2012.²⁰

Four entries out of roughly a million take an adult skin cell back to a cell that can become anything. Identity that stable, in a state space that large, and a four-element intervention still flips it cleanly. The gene names and the four-factor result are real. The numbers shown are invented to make the shape of the change legible.

And it is not a one-off. Two decades earlier, Davis, Weintraub and Lassar converted a fibroblast into a muscle cell with a single transfected factor.²¹ In 2010, Vierbuchen and colleagues screened nineteen candidates down to three that convert a fibroblast directly into a functional neuron.²² One, three, four. Against a state of a million entries and a genome of three billion bases.

Sit with what that implies about the organization of the thing. A state space that large, where identity is that stable, stable enough to survive being physically rebuilt at every division. And yet a four-element intervention flips it cleanly to a different identity, without corrupting the machinery on the way. That is not the behaviour of a system where everything is entangled with everything. It is the behaviour of a system with structure we have not copied.

The honest disclaimer. None of this is an argument. Biology is not evidence for an architecture, and an analogy is not a mechanism. I have been careful, in the paper and here, not to let a biological story stand in for a result. What the biology did was supply a different set of questions and a strong prior that the trade-offs in section 2 are not fundamental. The architecture that came out of it stands or falls on its measurements, which is what the rest of this page and the report are for.

What that starting point produced

Over about six months, those questions turned into a re-derivation of a number of things I had previously taken as settled about how a network should be organized during learning. The result is Learner 1.0. The base model stays completely frozen, and the trainable parameters are fixed — fixed, not growing with the number of things learned. The mechanism itself is proprietary and is not described here.

Where this lands

Against the eight requirements: all eight hold at once. That conjunction is the claim, and it is what the eight-axis comparison in the paper is for.¹⁶

Against the three bars in section 4, here is the most recent measurement, and I want to give the setup before the numbers because the setup is where continual-learning results usually hide their assumptions. One frozen base model, Qwen3.6-27B. Four text domains trained strictly in sequence: classic English literature, then Yoruba, then Amharic, then English news. That is 300 optimizer steps each, 1,200 in total, on byte-identical batches for every arm. No arm is told that a domain has ended. Each creates one optimizer at step zero and runs it, uninterrupted, to the end. The comparison is against rank-256 LoRA at a trainable-parameter count matched to within 3%. A fifth domain is held out and never trained by anything, so that "it did not forget" is a claim about the model rather than about the four domains we happened to look at.

What the three bars looked like

The floor: it did not forget. Re-measured at the end of the stream, LoRA's three earlier domains had lost +0.070, +0.060 and +0.034 nats. Learner 1.0's had moved by −0.001, −0.001 and +0.002.
The better case: earlier material improved. Two of those three numbers are negative, which means those domains came back better than when their own training ended. That is positive backward transfer, and it held at both capacities tested.
The best case: transfer without a target. The fifth domain, which no arm ever trained on, improved by 0.118 nats anyway. It also improved under LoRA, by 0.084. So this effect is not unique to the architecture, but it was larger.
The protection was not paid for in learning. On the domain being trained, Learner 1.0 reduced loss by 0.153 nats on average against LoRA's 0.143. It learned slightly more, not less.

LoRA lost 0.034–0.070 nats on the three earlier domains while both Learner 1.0 arms stayed within 0.004. At matched capacity all three arms learned the new domain about equally (0.143–0.156 nats), so what separates them is what they gave up. The green marks are hairlines against zero, which is the result. One base model, one four-domain stream, one run per arm. The full provenance is in the report.

What that means in one sentence: on this stream, the trade-off that the families in section 2 each pay for in a different currency did not have to be paid. The limitation, in one sentence: this is one run per arm, on one base model and one four-domain stream, and a single stream cannot tell you where the ceiling is. The full comparison is in the report, including a learning-rate ablation showing why the LoRA baseline sits where it does, and every number's provenance.

Read the full comparison →

What I am not telling you

The specific mechanism that produces this, the actual mathematical machinery, is reserved under NDA pending patent prosecution. I am aware that this is unsatisfying in an essay that has just spent several thousand words on where an idea came from, and I would find it unsatisfying too.

What I can do instead is make the behaviour checkable. The paper documents capability-level properties, the measurements that characterize them, and the evidence that they hold across model scales and both training regimes. The report linked above gives the full data for the most recent comparison, including the windows the model was trained on. And the API is the shortest path of all: point it at your own stream and see whether it keeps what it learns.

One consequence is worth stating plainly, because it is the part with practical teeth. Because the base model is left frozen, the cost of getting a continually-learning model is the cost of the attachment, not the cost of pre-training one. That changes who can have one.

Open questions

How far does it hold? The measurements above are on a 27B base with four domains. The architecture has been tested at several scales, but "several" is not "all", and the frontier rung is where deployment decisions get made.
How many domains before something gives? Fixed capacity is a requirement, not a free lunch. A fixed set of parameters must eventually saturate, and finding that edge honestly is ongoing work.
Does positive backward transfer survive a longer stream? Two of three domains improving is a real effect on a four-domain stream. Whether it compounds or decays over twenty domains is not something I can answer from this data.
Serving. As noted above: batching, cost per update, and versioning a model that is no longer a fixed artifact are unaddressed here and are not small.
Is the biology load-bearing or is it scaffolding? I genuinely do not know. The questions were productive. Whether the resemblance goes deeper than the questions is not something the measurements can settle.