Independent research discussions explain why small sequential updates are a difficult regime

Publisher: Learner Labs. Selected research context, checked 2026-10-13. These discussions frame the research problem. Learner Labs measurements are in the linked experiments.

Sutton and Javed distinguish continuing experience from frozen deployed weights

Source: Sequoia Training Data: Rich Sutton and Khurram Javed, Why AI Models Stop Learning, and How to Start It Again.

In the transcript, find the exchange beginning with Sonya Huang's question about an algorithmic gap. Javed contrasts large mixed batches with small updates from a single stream, where earlier knowledge can be damaged. Sutton calls the problem “Catastrophic forgetting.” They discuss their own proposed algorithms. Javed later says their meta-learning approach would require training a new foundation model.

Related evidence: Ten single-example skill streams, sequential learning versus LoRA.

The Dwarkesh discussion separates large mid-training cycles from small iterative adaptation

Source: John Schulman, Beren Millidge and Charlie O'Neill discuss recursive self-improvement. See the continual-learning discussion before the 01:00:33 chapter.

Schulman distinguishes sample efficiency in context from other learning regimes. O'Neill describes forgetting of earlier acquired information and general capabilities under repeated small SFT updates, even when large-scale mid-training can work. Millidge discusses the difference between continuing an existing model and training another model on the accumulated data. The discussion motivates testing acquisition, prior-learning retention and base capability together.

Related evidence: Acquisition and retention in ten skills, capacity-matched four-domain comparison, few-example lessons applied in later questions.

Learner Labs' comparisons supply the measurements for its own claim

The podcasts explain the question. Our experiments are what readers should inspect to evaluate our answer. The four-domain study shows acquisition and later retention on the same stream and approximately matched trainable capacity. The two-language study provides another within-method retention comparison with different training schedules and probes. The agent studies test use of small lessons after context resets.

Read the evidence guide · Detailed evidence questions · All reports