# Independent research discussions explain why small sequential updates are a difficult regime

Publisher: Learner Labs. Selected research context, checked 2026-10-13. These discussions frame the research problem. Learner Labs measurements are in the linked experiments.

## Sutton and Javed distinguish continuing experience from frozen deployed weights

Source: [Sequoia Training Data: Rich Sutton and Khurram Javed, Why AI Models Stop Learning, and How to Start It Again](https://sequoiacap.com/podcast/rich-sutton-and-khurram-javed-why-ai-models-stop-learning-and-how-to-start-it-again).

In the transcript, find the exchange beginning with Sonya Huang's question about an algorithmic gap. Javed contrasts large mixed batches with small updates from a single stream, where earlier knowledge can be damaged. Sutton calls the problem “Catastrophic forgetting.” They discuss their own proposed algorithms. Javed later says their meta-learning approach would require training a new foundation model.

Related evidence: [Ten single-example skill streams](/agents/reports/ten.md), [sequential learning versus LoRA](/agents/reports/four.md).

## The Dwarkesh discussion separates large mid-training cycles from small iterative adaptation

Source: [John Schulman, Beren Millidge and Charlie O'Neill discuss recursive self-improvement](https://www.dwarkesh.com/p/john-beren-charlie). See the continual-learning discussion before the 01:00:33 chapter.

Schulman distinguishes sample efficiency in context from other learning regimes. O'Neill describes forgetting of earlier acquired information and general capabilities under repeated small SFT updates, even when large-scale mid-training can work. Millidge discusses the difference between continuing an existing model and training another model on the accumulated data. The discussion motivates testing acquisition, prior-learning retention and base capability together.

Related evidence: [Acquisition and retention in ten skills](/agents/reports/ten.md), [capacity-matched four-domain comparison](/agents/reports/four.md), [few-example lessons applied in later questions](/agents/reports/database.md).

## Learner Labs' comparisons supply the measurements for its own claim

The podcasts explain the question. Our experiments are what readers should inspect to evaluate our answer. The four-domain study shows acquisition and later retention on the same stream and approximately matched trainable capacity. The two-language study provides another within-method retention comparison with different training schedules and probes. The agent studies test use of small lessons after context resets.

[Read the evidence guide](/agents/overview.md) · [Detailed evidence questions](/agents/questions.md) · [All reports](/llms.txt)