Plasticity Without Forgetting

Conclusions

Introduction

Adapting a large language model to new material usually means choosing between two failure modes. Full fine-tuning and its parameter-efficient variants learn the new material well, but when the material arrives sequentially, one domain after another, the way real usage arrives, each new domain overwrites some of what came before. The standard mitigation is to avoid the sequential setting altogether: mix everything and retrain. That works for a lab. It does not work for a system that must keep learning after it ships.

Learner 1.0 is built for exactly that sequential setting: the base model stays completely frozen and a fixed set of trainable parameters is added. The mechanism itself is proprietary and is not described here. This post measures the question above directly, on one base model and one data stream, with three arms: Learner 1.0 at a trainable capacity matched to LoRA's to within 3% (henceforth Learner 1.0 (1×)), Learner 1.0 at double that capacity (Learner 1.0 (2×)), and rank-256 LoRA. The 2× arm answers a second question the matched pair cannot: is what you see at 1× capacity-limited, or a property of the architecture?

Setup: one base model, one stream, three arms

The base model is Qwen3.6-27B in bf16, frozen in all three arms. The stream is four text domains trained strictly in sequence: classic English literature, Yoruba, Amharic, and English news. Each gets 300 optimizer steps on 256-token windows, 1,200 steps total, in the same order and on byte-identical batches for all arms.¹ Each domain holds out 8 windows that are never trained. A fifth domain of 64 windows (held_control) is never trained by any arm and exists so that "no forgetting" is a claim about the model, not just about the domains we happened to train. All losses are next-token cross-entropy in nats on those held-out windows.

There are no task boundaries. This deserves its own paragraph, because much of the continual-learning literature quietly assumes the opposite. No arm receives any signal that a domain has ended or a new one has begun: no task identifiers, no per-domain optimizer reset, no learning-rate re-warm at the switches. Each arm creates one AdamW optimizer at step 0 and runs it, uninterrupted, across all 1,200 steps. From the optimizer's point of view there is only a stream of batches whose statistics happen to change. Every forgetting and retention number below is measured under that blindness.

The capacity-matched comparison is Learner 1.0 (1×) vs LoRA: 1.141B vs 1.107B trainable parameters, a 0.97× match. The 2× arm is deliberately not matched: it exists to test whether the 1× results are sitting against a capacity ceiling, and every claim of parity or superiority over LoRA below is made from the 1× arm.

One asymmetry must be stated up front, because the acquisition comparison depends on it. Before the stream, each Learner 1.0 arm goes through the same warm-up phase: 14,224 steps (~3.6M tokens) on a mixed corpus drawn from the same corpus families as the four domains, with zero window overlap with any training or evaluation data. After that, those trainable parameters are the only thing that ever trains again. LoRA receives no warm-up, because a randomly-initialized adapter has nothing to warm. Whether a fully domain-blind warm-up preserves these numbers is a real question, and we return to it in Open questions rather than hide it here.

Forgetting: the headline difference

What the data says

Figure 1 shows the entire experiment at once. Each panel walks the same stream left to right: faint line, per-step training loss. Dotted line, held-out validation loss measured every 25 steps. Dashed black line, the frozen base's loss on that domain, the starting point every arm must improve on. The three panels share identical axes, so every loss is directly comparable. The part to read is the right edge: after the last domain finishes, every earlier domain is re-evaluated, and the diamond shows where it landed.

Figure 2 isolates the end-of-stream change per domain, the quantity usually reported as backward transfer, where positive means the domain got worse after later training. LoRA's forgetting is ordered exactly as interference predicts: the earliest domain forgets the most. The Learner 1.0 bars hover at zero in both arms, and several are negative. Training on the domains that followed made earlier domains slightly better (the first two at 1×, and literature most strongly, −0.004, at 2×). Averaged over the three earlier domains, LoRA forgets 0.0547 nats, Learner 1.0 (1×) +0.0001, and Learner 1.0 (2×) −0.0005. That is drift five hundred times smaller than LoRA's at 1×, and net-negative at 2×.

Why this is hard

The effect does not come from having more room to write into. LoRA's rank-256 update is one shared subspace that every domain must rewrite. The method Learner 1.0 uses instead is withheld. That the retention numbers are indistinguishable at 1× and 2× capacity is itself evidence for this reading: interference-driven forgetting should ease as capacity grows, but there is no forgetting left to ease.

Sequential training has produced this signature since 1989, and the usual ways of suppressing it buy retention by giving something up: a boundary signal, a replay buffer, or a parameter count that grows with the task list. This stream supplies none of the three.

What it means for building

If a model can take a new domain without disturbing the ones already in it, retraining from mixed data stops being the only safe way to ship an update. Domains can then arrive in whatever order your users produce them.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/forgetting

$ replicate plasticity --figure 2 # {{REPLICATE_CMD}}

Also true

The limitation to carry: this is one stream order and one run per arm. The magnitude of LoRA's forgetting will vary with order, rank, and learning rate, though its sign is the well-documented default.

Acquisition: protection is not paid for in learning

What the data says

Protection would be uninteresting if it were bought with plasticity. Figure 3 shows acquisition: how many nats each arm removes from the frozen base's loss on a domain's held-out windows by the end of that domain's training. Both Learner 1.0 arms acquire more than LoRA in three domains and tie in the fourth. Averaged over the stream they remove 0.153 (1×) and 0.156 (2×) nats to LoRA's 0.143. The protection is not paid for in learning.

Why this is hard

Retention is easy to buy by learning less: a method that barely moves has little to lose. The question that decides whether the protection is worth anything is whether the protected arm still learns as much as the unprotected one on the domain in front of it.

What it means for building

Parity of acquisition means the retention property costs nothing at the point of use. You can take it and still have the stronger model on the domain you are training right now.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/acquisition

$ replicate plasticity --figure 3 # {{REPLICATE_CMD}}

Also true

One interpretive sentence and one caution. The parity holds under a warm-up drawn from related corpus families. Our subsequent measurement under a fully domain-blind, maximally diverse warm-up (the configuration a product would ship, which cannot know its users' domains) puts Learner 1.0's acquisition at 77% of LoRA's on held-out domains while retention stays at the levels shown here. That is the price of domain-blindness, measured, not hidden.

Doubling capacity: what it buys, and what it doesn't

What the data says

The 2× arm exists to answer one question: are the 1× results capacity-limited? Figure 4 puts the two summary quantities side by side for all three arms: mean acquisition on the left, mean forgetting on the right. The answer is no, twice over. Doubling the trainable parameters from 1.14B to 2.28B moves mean acquisition from 0.153 to 0.156 nats, a 2% change on a 100% capacity increase. Retention, already at ~zero, has nowhere to go (mean drift +0.0001 → −0.0005, worst single domain +0.0025 → +0.0024).

Why this is hard

A result that appears at exactly one parameter budget is a tuning artefact until it is shown at another. Doubling the budget is the cheapest test of whether a number is a property of the design or of its size.

What it means for building

The interpretation: at this stream length and per-domain budget, the binding constraint on Learner 1.0's acquisition is not parameter count. The warm-up regime (see Open questions) moves these numbers far more than capacity does.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/capacity

$ replicate plasticity --figure 4 # {{REPLICATE_CMD}}

Also true

The limitation: four domains is a short stream. Capacity should start to matter as the number of retained domains grows, and where that crossover sits is measured work in the program record, not shown here.

The learning-rate axis: why LoRA sits at 2×10⁻⁵

What the data says

The setup table contains an asymmetry that deserves the same scrutiny as capacity: Learner 1.0's stream learning rate is 50× LoRA's. Could LoRA not simply be run hotter and learn more? It was tried. The 2×10⁻⁵ rate was itself selected by that trial. The comparator was fixed in advance to run first at 2×10⁻⁴, the rate an earlier reference run had used, on exactly the same stream (the domain windows are byte-identical between the two runs, and only the learning rate differs). Figure 5 shows what happened.

This completes the learning-rate story of the setup table. 2×10⁻⁵ is where this LoRA configuration works. 2×10⁻⁴ is where it dies. And 1×10⁻³, five times the dying rate, is where Learner 1.0 ran the identical loop (same single never-reset optimizer, same one-window batches, same absence of any schedule) and produced Figures 1–3.

Why this is hard

The run at 2×10⁻⁴ was stopped by a runtime guard at the first domain boundary: its acquisition on the domain it had just trained was negative, and a stream continued past that point measures damage, not forgetting. The mechanism is visible in the right panel and is a property of the adapter, not of the data: LoRA's weight update is the product of two trained matrices, applied here at the least-damped scaling (α/r = 1), and its magnitude has no ceiling. A tenfold learning-rate increase produced a roughly tenfold larger update surface in a quarter of the steps, with nothing for it to saturate against. Weight decay and data ordering were both eliminated as causes (the diverging run used weight decay 0 and within-domain shuffling, reproducing the divergence measured earlier under other settings).

What it means for building

The width of the stable learning-rate window is part of what you are choosing between. A method that runs at five times the rate that destroys its comparator leaves room to tune, and a comparison is only as honest as the tuning given to the arm that loses it.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/learning-rate

$ replicate plasticity --figure 5 # {{REPLICATE_CMD}}

Also true

The limitation to carry: our grid on LoRA's learning-rate axis is two points a decade apart, converged and diverged. Intermediate rates (say 5×10⁻⁵) were not run and might buy LoRA a few points of acquisition. The divergence bounds the stable window from above but does not locate its edge.

Learning speed: most of the gain in one checkpoint

What the data says

Figure 6 plots the three arms together inside each domain's 300-step window, starting from the frozen base's loss at step 0. The shape difference is consistent: both Learner 1.0 arms land essentially at their final loss by the first 25-step checkpoint, 92% of Learner 1.0 (1×)'s total gain on the first domain, and then flatten. LoRA approaches its final loss gradually across the window. In three of the four domains the Learner 1.0 curves are below LoRA's at every measured step. In news they cross and LoRA finishes marginally lower.

Why this is hard

The speed comes with the warm-up: after it, a new domain evidently requires mostly small adjustments to the trainable parameters rather than construction from scratch.

Most adaptation methods spend their budget descending toward a final loss. A method that lands most of its gain at the first checkpoint is making a different claim about how much data it needs to reach competence.

What it means for building

Domains that pass through quickly still get learned. On a live stream that is the difference between capturing a week of one customer's traffic and missing it.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/speed

$ replicate plasticity --figure 6 # {{REPLICATE_CMD}}

Also true

The caution mirrors the acquisition caution. With a domain-blind warm-up the first-checkpoint fraction is smaller, though the qualitative fast-then-flat shape persists.

The never-trained domain: transfer without a target

What the data says

The fifth domain exists precisely because "no forgetting" measured only on trained domains is a weak claim. Figure 7 tracks its held-out loss, on text no arm ever trains on, across the stream. Under Learner 1.0 it improves after nearly every domain, ending 0.118 (1×) and 0.115 (2×) nats below the frozen base. LoRA also ends below base (−0.084): sequential training damages LoRA's trained domains, not its general text. The distinction the control draws is therefore sharper than "LoRA breaks things": all three arms radiate some benefit outward, Learner 1.0 somewhat more. But only Learner 1.0 does so while also keeping what it explicitly learned.

Why this is hard

Improving text that nobody trained on is the opposite of what interference predicts. The default expectation for a sequentially trained model is that unrelated text gets worse.

What it means for building

General capability is measurable on material you deliberately keep out of training. Hold a slice back before you start and you can check this claim on your own stream instead of taking it.

Replicate it

Data and protocol for this figure: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity/never-trained-control

$ replicate plasticity --figure 7 # {{REPLICATE_CMD}}

Also true

The control is 64 held-out windows on one run per arm, and all three arms improved it, so the finding is the size of the gap between the arms rather than its direction.

What these traits add up to

The program behind Learner 1.0 tracks a fixed list of goals. This experiment speaks to four of them directly, so we state the mapping explicitly rather than let the figures imply it.

Stability with plasticity

The central trade every continual-learning method faces is between learning the new thing and keeping the old ones. Figures 1–4 together show Learner 1.0 on the good side of both at once, at either capacity: acquisition at or slightly above a capacity-matched weight-update method, retention two orders of magnitude better. Nothing in the stream forced that trade to be resolved in Learner 1.0's favor. The same stream resolved it against LoRA.

Positive backward transfer

The strongest single data point in the experiment is a set of bars below zero: at 1× capacity the first two domains' losses fell (−0.0011 and −0.0012 nats) while the model trained on the domains that followed, and at 2× the literature domain fell by 0.0037 nats (Fig. 2). Forgetting is merely absent in the remaining earlier domains. In these it runs backward. The effect is small and each arm is one run, so we flag it as a trait to be established across seeds, not a solved property. But that its sign recurs in two independently trained conditions, and that it is the sign no interference-based method produces, is why we call it out.

Forward transfer

The never-trained control (Fig. 7) improves after nearly every block under both Learner 1.0 arms, −0.118 and −0.115 nats with no training target. Training on A made B easier, where B was never trained at all.

Speed, and therefore data efficiency

Reaching ~90% of final acquisition in 25 steps (Fig. 6) is a statement about how much data the method needs to reach competence: in this stream, roughly one-twelfth of the budget LoRA uses to approach the same loss. In a setting where domains arrive and leave, the method that learns in one checkpoint keeps more of what passes by.

The method, at the level we can state it

What can be stated: the base model is frozen and is never updated, and a fixed set of trainable parameters is added — 1.14 B in the 1× arm and 2.28 B in the 2× arm. Beyond that the mechanism is withheld.

Open questions and limits

1. The warm-up regime is the honest asterisk on acquisition. These numbers use a warm-up drawn from the same corpus families as the stream (never the same data). Under a fully domain-blind 10M-token warm-up, the deployable configuration, we measure 77% of LoRA's acquisition on held-out domains, with retention unchanged. Closing that gap (longer generic warm-ups measurably help) is current work.
2. One run per arm, one stream order. Each arm shown is a single run. The two Learner 1.0 capacities are independently trained conditions that agree closely on every quantity. That is a useful robustness signal, but they are not seeds of the same configuration. The program's wider sweeps (13 Learner 1.0 configurations this wave, replicated across waves) show the retention behavior is stable. The per-domain acquisition ordering and the positive-backward-transfer sign deserve seeds before they are called laws.
3. Four domains, one scale. The stream is four domains on a 27B base. Longer streams (12-domain runs show the same retention), other bases, and much longer per-domain budgets (5M unique tokens into one domain: prior domain still improved) are measured in the program record but not shown here. The capacity question in §6 is answered only at this stream length. Longer streams are where 2× should eventually separate.
4. LoRA was tuned, not tortured, but it is one configuration. Rank 256 on the MLP projections with a swept learning rate is a strong, standard setup. Ranks, targets and schedulers change forgetting's magnitude. Its direction on sequential streams is well established in the literature and reproduced here. The learning-rate grid behind the sweep is two points a decade apart: 2×10⁻⁴ diverged, 2×10⁻⁵ converged (§7). And the adapter scaling α/r was left at 1 and never swept separately. Under Adam, tuning α is close to tuning the learning rate, so this is one under-explored axis, not two, and an intermediate rate might buy LoRA a few points of acquisition.

Data and protocol

Inspect the training data and every measured value: github.com/learnerlabs-ai/replications/tree/main/articles/plasticity

Citation

Every number in this post traces to a receipt file in the program's analysis record. The plotted values ship beside the figures as figures/figure_provenance.json, and the datasets, LoRA baseline, and evaluation code are published alongside this post.