Twenty clinical studies, one model learning between them

1. The question and what was measured

The question: if a model is trained on its own experience after every study it analyses, does it analyse later studies better than the same model that never learns? Each study is a database of patients from one clinical study. The agent has twenty tool actions: read the study's metadata and a data summary, run SQL, estimate survival for any grouping of patients, and fit a cohort predictor. Then it submits survival estimates at 12, 24 and 36 months for 36 fixed cohorts, 108 numbers. Each study records different variables, so only 3 to 10 of the 36 cohorts can be measured in any one study. The rest have to be estimated from what similar studies showed. The twenty studies come in five families of four (HERALD, MERIDIAN, MOSAIC, FORGE, CADENCE) that share a design.

Reward for a study is the information gain of the submitted estimates over the benchmark's reference predictor, in bits per cohort: 0 means as good as the reference, and the best possible is 0.162. Reward is the sum over the twenty studies. Gain is reward minus the reward of the same model analysing the same studies with learning off, the benchmark's stateless baseline. It is the benchmark's learning metric.

The three runs end at rewards -0.0200, -0.1493 and -0.0988 against -0.2408 for the baseline. The gain over the baseline is +0.2208, +0.0915 and +0.1420. Normalized as the leaderboard does, +6.34%, +2.63% and +4.07%, mean +4.35% (standard error 1.08%).

2. Setup

Model. Learner 1.0, the same model as in the ten-skill experiment, with a fixed number of trainable parameters set before the first study. The mechanism is proprietary and is not described here.

What the learner sees. At each study the learner receives the benchmark's own task brief and the current study's tools, and its own steps in that study, nothing else: no earlier studies, no notes, no scores, no retrieved examples. Its prompt is the same as the baseline's. Anything it carries from one study to the next is in its weights.

The loop. The learner analyses a study and submits. A teacher LLM reads the learner's steps in that study and the tool results, and writes procedures: which next action is supported at which state. The teacher sees only the study just finished: no earlier studies, none of its own earlier procedures, and no scores. The recipe turns each procedure into training examples built on states the learner actually reached, the learner writes the reasoning for each example's action in its own words, and it is trained on them before the next study. The teacher never analyses a study and never acts in the benchmark. Section 9 gives the full recipe.

Training. One pass, batch size one, no replay of earlier examples, no optimizer resets between studies. Across a run the learner trained on 80 to 111 examples, 12,901 to 18,835 supervised tokens.

Runs. The benchmark's default schedule plays the twenty studies family by family, in a fixed canonical order (run 0) and in orders that permute the four studies inside each family (runs 1 and 2). The stateless baseline is run once. Because nothing carries over between its studies, its order does not matter. Each run is compared with the baseline study by study. All four lanes used the same model and the same decoding.

3. One study, before and after learning

Here is the same study analysed by the baseline and by run 0, which reached it ninth, after training on eight earlier studies. MOSAIC is a health survey that records a PRB1 category and does not record the comorbidity count the earlier families had.

The baseline writes a summary of the data at step 3 and starts from age bands. Run 0 splits the patients into high, medium and low risk tiers at step 3, swapping the comorbidity count it learned on for the PRB1 category this study records. Its reasoning follows the shape of the example it was trained on after study 7 ("understand how survival varies across different risk groups" before refining the cohort definitions). Run 0 scores -0.0239 bits per cohort on this study, the baseline -0.0675.

The survival estimator returns, for each group, the share of patients alive at 12, 24 and 36 months, and lists which of the 36 cohorts can be measured in this study and how they split across the groups. Everything the agent submits for the unmeasured cohorts has to come from what it carries from earlier studies.

4. What it learned

After study 7 the teacher kept one step of run 0's analysis as a training example: open the analysis with a three-tier risk split. Run 0 trained on it once. From then on it opened 7 of its next 13 studies the same way, at step 3, each time choosing variables the study records. On those 7 studies it scored above the baseline on 6.

The example, as trained after study 7

The action was chosen by the teacher. The reasoning was written by the learner.

{
 "thought": "I have reviewed the database metadata and data summary. The study has a high mortality rate (96.6% died), with a mean time to death of ~21 months. Key variables include age, comorbidity_count, prb1_ratio, and others. To begin estimating survival for the 36 cohorts, I need to understand how survival varies across different risk groups. I will start by estimating survival for a simple risk stratification based on age and comorbidity count, which are likely strong predictors. This will help me calibrate my understanding of survival patterns in this population before moving to more complex cohort definitions.",
 "tool_call": {
  "tool": "estimate_survival_by_group",
  "group_expression": "CASE WHEN age >= 65 AND comorbidity_count >= 4 THEN 'high_risk' WHEN age < 50 AND comorbidity_count <= 1 THEN 'low_risk' ELSE 'medium_risk' END"
 }
}

Two things qualify this. The benchmark's own tool description gives an example grouping with tier names like these, so every lane, the baseline included, has the idea in its prompt. The baseline uses it, but late, at steps 12 to 15. And run 1 opened studies 5 to 7 with the same split before any of its training examples targeted it. Run 2 never used it. So the opening move is reachable without training. What the training example changed in run 0 is that the move became its default opening for eight studies.

The table below counts what each lane does, family by family. It is the place to look for what changed in the agents' analyses beyond the opening move.

5. Learner 1.0 learns efficiently online from very few examples

The whole effect in this article comes from 80 to 111 training examples per run, 12,901 to 18,835 supervised tokens, about five examples after each study. Each example was trained once.

The loss on each fresh example, measured in the step that trains on it and before the update, falls in every run: from 0.26, 0.22 and 0.26 on the examples written after studies 1 to 4 to 0.09, 0.10 and 0.08 after studies 17 to 19. The examples written late in the run are closer to what the model already does.

6. By study family

Learner minus baseline, summed over each family's four studies:

MERIDIAN is ahead in every run by a small margin, and FORGE in every run, by a large margin in runs 0 and 2 and a small one in run 1. HERALD, the first family, is behind in every run. MOSAIC and CADENCE change sign between runs. The benchmark splits gain into the first study of each family, where nothing from that family has been seen yet, and the later ones: the runs gain -0.33%, +0.91% and +4.53% on first studies and +6.67%, +1.71% and -0.46% on later ones. Neither part is consistent across runs.

7. Results

Each run is compared with the baseline study by study. All three runs score above the baseline in total. The gain is not spread evenly: averaged over the three runs, the learner is above the baseline on 12 of 20 studies, which a one-sided sign test puts at p = 0.25. Most of the gain comes from a few studies with large differences.

8. Against the published leaderboard

Learner 1.0 is the only system on this leaderboard that learns strictly through its weights. Its context is reset before every study, so no in-context learning is allowed between studies. Every other system carries its experience forward in its context, a notepad, a memory system or a coding agent's workspace.

The benchmark publishes results for twelve systems on this task, each pairing a frontier model with a way of carrying experience forward: in context, through a notepad, through a memory system, or through a coding agent's workspace. The leaderboard ranks them on normalized reward (reward relative to GPT-5.4's stateless baseline) and normalized gain (gain relative to each system's own baseline).

On reward our three runs average -0.0894, 2nd of 13, behind ACE · GPT-5.4 (-0.0518). Run 0 alone would rank first. On gain they average +0.1514, 5th of 13. The four systems above range from +7.8% to +16.5% normalized gain against our +4.35%. Every system on this task, ours included, scores below zero on reward: none beats the benchmark's reference predictor. Two cautions: the published figures average five runs, ours three, and this is one of the leaderboard's six tasks, so it is a placement on the cohort studies tab, not an overall entry.

9. The recipe

After each study. The teacher receives the learner's steps in the study just finished, with the tool results, and the task brief. It returns procedures in a fixed JSON format: which next action is supported at which state, with the evidence for it. Its full prompt is in the replication repository (recipe/teacher_prompt.txt).

From procedures to examples. Each example is a state the learner actually reached, the conversation so far with its recorded tool results, and the next action as the target. Only states the learner reached are used and no tool result is invented. Two fixed examples are added at every study: read the database metadata first, and after a tool error read the metadata if not yet read.

Reasoning. The teacher chooses the action. The learner writes the reasoning for it, given the state and the action (the prompt is in recipe/reasoning_prompt.txt).

Acting. Thinking when stuck, used on the database exploration task, was not used here.

Training and limits. One pass, batch one, no replay, at most 50,000 supervised tokens per study. The learner's context is 32,000 tokens and each step's output is limited to 4,096 tokens. The teacher call is limited to 15 minutes.

10. Interpretation, and how to check it

Training after each study on about five examples written from that study gives a small gain over the same model with learning off in all three runs, and in run 0 a procedure learned in one study carried to the next ones with different variables. Because neither the learner's prompt nor the teacher's input holds anything from earlier studies, the only path from one study to the next is the weights. The size of the effect varies by run (+2.63% to +6.34% normalized gain).

To check: every step of every study in every lane, every training example, the per-example training losses and the scores are in the replication repository, and the figures regenerate from them.