Source: https://learnerlabs.ai/demos/skills/ten-skills
Format: alternate Markdown of the approved capability-first article. All methods and records are included, even where collapsed in HTML. Tables retain their HTML cells and spans to preserve data associations.
Exported: 2026-10-13
Context: [Evidence guide](/agents/overview.md) · [Evidence questions](/agents/questions.md). Publisher: Learner Labs. Inline SVG labels are transcribed below. Raster figures retain original image links and captions.

Ten skills, one after another

# New skills are learnt without task labels while earlier skills are retained

We taught Learner 1.0 ten unrelated skills one after another, one example per update, without replay or task labels. After every later teach we measured the earlier skills again. Nine of the ten finish within two correct answers of their own-end score on fixed 225-question panels, after a stream of 63,282 updates. Their combined score is unchanged. The tenth, Oriel, was weakly acquired and falls from 12 to 9 correct.

Learning new skills did not materially change the measured base capabilities: a 498-item sentinel stayed near its starting level at every checkpoint, and a 16,481-item battery was checked before teaching, after five skills and after ten. That is the conclusion this run supports: new learning and retention together, under ordinary next-token training with fixed trainable capacity. Acquisition varies widely by skill, and the evidence is for this stream, order and budget.

**Model** Learner 1.0 **Trainable parameters** 1.14B, fixed for the whole run **Stream** 63,282 updates, one example each, in a fixed order **Tokens trained on** 5,921,663 presented, 2,295,371 supervised (answer tokens including end-of-sequence) **Held-out evaluation** 225 fixed rows per skill, greedy decoding, exact match, at the skill's own-end checkpoint and every later one **Base evaluation** 498-item sentinel at every checkpoint. 16,481-item likelihood battery at teaches 0, 5 and 10 **Final checkpoint** 3264e0b2 **Compute** 1 H200

<a id="claim"></a>
## Earlier skills survive later teaching without replay or task labels

When supervised skills are taught in sequence to one model with a fixed number of trainable parameters, does teaching the later ones erase the earlier ones? The measurement is the count of exactly correct answers on a fixed 225-row held-out panel for every skill, taken at the checkpoint right after that skill's own teach and again after every later teach.

<a id="ts_f0_key_summary"></a>
Swipe the figure to read every label →

### Skills are acquired in sequence and largely retained

[Original chart: Skills are acquired in sequence and largely retained](/agents/figures/ten-1.svg)

Figure labels: Accuracy and retention: every skill at every checkpoint | Every skill on its own 225-row panel | Ten skills taught one after another. A line starts at the checkpoint where its skill was taught | and runs to the end of the stream. The base is scored at all eleven checkpoints. | 0% | 25% | 50% | 75% | 100% | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | checkpoint (0 = before any teaching) | percent correct | base | all ten skills: 0 before teaching | taught at checkpoint | SCAN | 1 | Meridian | 2 | Adyghe G2P | 3 | Oriel | 4 | COGS | 5 | Sable | 6 | PCFG | 7 | Tessel | 8 | Juniper | 9 | Bracken | 10 | base, 498 items | A skill is scored only | from its own teach onward, | so nothing is drawn to | the left of its first dot. | Before any teaching the base answers 0 of the 2,250 panel rows, so every point above the axis was acquired during the run. Each | skill is then scored on its own 225-row held-out panel at every checkpoint from its teach onward. It is not scored in between. | After all ten teaches 1,525 of 2,250 are correct. The base line is the 498-item sentinel on its own denominator. Source: | plots/data/derived/tables.json.

**[Figure 1](/demos/skills/ten-skills#ts_f0_key_summary).** One picture of the run: nothing is answered before teaching, each skill jumps at its own teach, and every skill holds its level while the skills after it are taught. 1,525 of 2,250 panel rows are correct at the end, from 0 at the start, and the base line stays flat throughout. The horizontal axis is the checkpoint after each teach. The vertical axis is the percentage of correct answers. Each skill uses its own fixed 225-row panel. The base line uses the separate 498-item sentinel.

Final panel scores after the tenth teach, in the taught order: SCAN 222, Meridian 84, Adyghe G2P 146, Oriel 9, COGS 225, Sable 198, PCFG SET 172, Tessel 188, Juniper 120, Bracken 161, each of 225. Against each skill's own-end score the changes are 0, +1, −1, −3, 0, 0, −2, 0, +2, 0. The one skill that moved by more than two rows, Oriel, was also the one the model never learned well: 12 of 225 at its own teach.

<a id="ts_f5_own_vs_final"></a>
Swipe the figure to read every label →

### Nine skills finish within two correct answers of their own teach

[Original chart: Nine skills finish within two correct answers of their own teach](/agents/figures/ten-2.svg)

Figure labels: Own-end score versus final score, per skill | right after its own teach | after all ten teaches | Correct of 225 on each skill's fixed held-out panel | the two marks coincide when nothing changed | 0 | 45 | 90 | 135 | 180 | 225 | correct of 225 | SCAN | 222 → 222 (no change) | Meridian | 83 → 84 (+1) | Adyghe G2P | 147 → 146 (-1) | Oriel | 12 → 9 (-3) | COGS | 225 → 225 (no change) | Sable | 198 → 198 (no change) | PCFG | 174 → 172 (-2) | Tessel | 188 → 188 (no change) | Juniper | 118 → 120 (+2) | Bracken | 161 → 161 (no change) | Own-end = the checkpoint at the end of that skill's segment. Final = the checkpoint after the tenth teach. Source: | plots/data/derived/tables.json.

**[Figure 2](/demos/skills/ten-skills#ts_f5_own_vs_final).** All ten skills hold what they learned: nine end within two rows of their own teach, and for Oriel the four working components survive, with the header and escaping right on every row at the end and the payload and decoded length on 92 to 95 per cent of them. Its exact-match count still moves 12 to 9 of 225 because one wrong checksum costs the whole row. Juniper and Meridian end slightly higher than at their own teach. For each skill, compare the mark just after its own teach with the mark after all ten teaches on the horizontal axis, correct answers out of 225.

<a id="setup"></a>
## One-example next-token updates test retention at fixed capacity

**Model.** Learner 1.0, with 1.14 billion trainable parameters, fixed before the first skill and never grown. Everything else about the learning mechanism is proprietary and is not described here.

**Objective.** Cross-entropy on the answer tokens only, including the native end-of-sequence token, so the model is also trained to stop.

**Data order.** Ten skills in a fixed order (SCAN, Meridian, Adyghe G2P, Oriel, COGS, Sable, PCFG SET, Tessel, Juniper, Bracken), each as a contiguous block of rows. One pass: no row is presented twice. No rows from an earlier skill are replayed during a later one.

**Batch and accumulation.** Batch size one, no gradient accumulation: every update is a single example. Gradient clipping at 1.0. Each teach resumed from the checkpoint the previous accepted teach ended on.

**What the learner sees.** Only the example text: the prompt and the answer. No skill label, boundary marker or teach index is given to it. The colored bands in the figures are the experiment design, not an input.

**Checkpoints and monitoring.** A full resumable checkpoint at the end of every skill (and at quarter points within it). During training, a held-out loss was measured every 25 updates on eight fixed panel rows per monitored skill. This monitor is separate from the final evaluation and is reported as a curve, not a score.

**Generation and scoring.** Greedy decoding with a per-skill token cap set from the longest target in the skill. A row is correct only if the whole answer matches the frozen target under that skill's scorer ([“The same held-out questions are scored after every later teach”](/demos/skills/ten-skills#evaluation)).

**Compute.** Training ran on 1 H200. The ten teaches were not one uninterrupted job ([the checkpoint record](/demos/skills/ten-skills#provenance)).

<a id="sample"></a>
## Each answer produces one update before the next example arrives

A sample is one prompt and one target answer. The model reads the prompt, is trained on every token of the answer including the end-of-sequence token, and the resulting gradient is applied as one update. Then the next sample. Here is the first training row of the fourth skill, exactly as stored:

```
Apply the oriel packet codec procedure to the input. Return only a JSON list of integers, without explanation.
Input: {"payload":[9,3,2,13,6,13,12,12]}
Answer:
```

Target (all 21 integers are supervised, plus the end-of-sequence token):

```
[15,13,8,14,12,8,10,7,7,2,8,15,13,3,2,8,1,5,1,5,9]
```

<a id="fig-3"></a>
Swipe the figure to read every label →

### Each example produces one update before the next arrives

[Original chart: Each example produces one update before the next arrives](/agents/figures/ten-3.svg)

Figure labels: schematic | example k | prompt + answer tokens | one update | loss over every answer token | example k+1 | next row in the fixed order | Batch size 1, no accumulation, no replay: 63,282 such steps in total.

**[Figure 3](/demos/skills/ten-skills#fig-3).** One example, one update, next example. The number of answer tokens differs per row, so updates are not equal in size. The tables report both rows and supervised tokens. Read the schematic from left to right: prompt and answer, one gradient update, then the next example. It describes the protocol rather than a measured learning curve.

**Counting.** Two counts appear in this article. *Updates* is the number of single-example updates the final checkpoint was trained with, 63,282. *Supervised tokens* are the answer tokens the loss was computed on, including end-of-sequence. Examples differ in length, so the tables report both.

**The training data.** Every training row of all ten skills, in the taught order, is in the replication repository: [ten_skills/data/train](https://github.com/learnerlabs-ai/replications/tree/main/ten_skills/data/train). The held-out panels are in [ten_skills/data/evals](https://github.com/learnerlabs-ai/replications/tree/main/ten_skills/data/evals). The files are there to be read and re-scored. The repository does not run this training.

<a id="profiles"></a>
## Four datasets and six new procedures test different kinds of learning

Four are public datasets (SCAN, Adyghe grapheme-to-phoneme, COGS, PCFG SET) and six are procedures written for this experiment so that the base model could not know them in advance (Meridian, Oriel, Sable, Tessel, Juniper, Bracken). For the written procedures the prompt states only the input and the output format. The rules below were never shown to the learner. Each card shows one real training row and one or two real held-out rows with the model's actual answer at the end of that skill's teach and after all ten teaches. Examples are chosen for length and structure, not for success. Each skill with a wrong final answer shows one.

<a id="skill-scan"></a>
### 1. SCAN: compositional command-to-action translation (public dataset)

A navigation command in a small controlled language (walk, run, jump, look, turn left/right, around, opposite, twice, thrice, and, after) is translated into the exact sequence of primitive actions. Training rows come from the public SCAN length-split training file. The fixed panel rows come from the released retention bank, which has zero training-group overlap by the release manifest. Scored as an exact match of the whole action sequence.

**Updates** 9,850 **Supervised tokens** 297,232 (presented 499,982)  **Own-end** 222/225 (98.7%) **Final** 222/225 (98.7%)  **Scorer** scan_actions: exact match of the complete action sequence **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/01_scan.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/scan/panel.jsonl) (225 fixed rows)

**A real training row** (scan/train/16767:ce_v2)

```
Request:
Translate this command into actions: run around left twice after walk right thrice
Response:
```

Target:

```
I_TURN_RIGHT I_WALK I_TURN_RIGHT I_WALK I_TURN_RIGHT I_WALK I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN I_TURN_LEFT I_RUN
```

**A held-out question** (scan/dev/9952:ce_v2)

```
Request:
Translate this command into actions: walk thrice and look around right
Response:
```

Target:

```
I_WALK I_WALK I_WALK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
I_WALK I_WALK I_WALK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK I_TURN_RIGHT I_LOOK
```

**A held-out question the model got wrong** (scan/dev/15105:ce_v2)

```
Request:
Translate this command into actions: turn around left thrice after turn opposite left thrice
Response:
```

Target:

```
I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT I_TURN_LEFT
```

<a id="skill-meridian_settlement"></a>
### 2. Meridian: stateful financial-event accounting (new procedure)

Process a list of events in order from balance 0 and reserve 0. A deposit first pays off reserve debt, then credits the remainder to balance. A purchase costs its amount plus a fee of 3. If the balance covers it, debit and count it accepted, otherwise raise the reserve by 2 and count it rejected without paying. A refund credits the amount minus 1 (minimum zero). Return balance, reserve, accepted and rejected. The prompt gives only the input and output format. The rules are never shown to the learner.

**Updates** 3,557 **Supervised tokens** 68,159 (presented 349,925)  **Own-end** 83/225 (36.9%) **Final** 84/225 (37.3%)  **Scorer** typed JSON procedure: parsed JSON compared field by field against the oracle **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/02_meridian_settlement.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/meridian_settlement/panel.jsonl) (225 fixed rows)

**A real training row** (meridian_settlement:candidate_train:01569)

```
Apply the meridian settlement procedure to the input. Return only a JSON object with integer fields balance, reserve, accepted, rejected, without explanation.
Input: {"events":[["refund",19],["purchase",18],["deposit",6],["deposit",18],["purchase",27],["purchase",25],["refund",9]]}
Answer:
```

Target:

```
{"accepted":1,"balance":18,"rejected":2,"reserve":2}
```

**A held-out question** (meridian_settlement:retention:00454)

```
Apply the meridian settlement procedure to the input. Return only a JSON object with integer fields balance, reserve, accepted, rejected, without explanation.
Input: {"events":[["purchase",29],["deposit",37],["purchase",25],["purchase",24],["refund",21]]}
Answer:
```

Target:

```
{"accepted":1,"balance":27,"rejected":2,"reserve":2}
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
{"accepted":1,"balance":27,"rejected":2,"reserve":2}
```

**A held-out question the model got wrong** (meridian_settlement:retention:00195)

```
Apply the meridian settlement procedure to the input. Return only a JSON object with integer fields balance, reserve, accepted, rejected, without explanation.
Input: {"events":[["refund",12],["deposit",4],["refund",19],["refund",31],["refund",38],["deposit",5],["deposit",3],["deposit",22],["deposit",30]]}
Answer:
```

Target:

```
{"accepted":0,"balance":160,"rejected":0,"reserve":0}
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
{"accepted":0,"balance":159,"rejected":0,"reserve":0}
```

<a id="skill-g2p_ady"></a>
### 3. Adyghe grapheme-to-phoneme (public dataset)

Transcribe a written Adyghe word into IPA under the SIGMORPHON 2020 convention. Adyghe is a Northwest Caucasian language with a large consonant inventory and long orthographic words, so a single wrong segment fails the row.

**Updates** 3,600 **Supervised tokens** 42,122 (presented 232,037)  **Own-end** 147/225 (65.3%) **Final** 146/225 (64.9%)  **Scorer** exact match of the space-separated IPA segments (character-edit diagnostics recorded separately) **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/03_g2p_ady.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/g2p_ady/panel.jsonl) (225 fixed rows)

**A real training row** (g2p_ady:train:3071)

```
Transcribe this Adyghe word into IPA using the SIGMORPHON 2020 grapheme-to-phoneme convention. Return only the transcription, with spaces between phoneme tokens.
Word: чъыгшъхьацыз
Answer:
```

Target:

```
t͡ʃ ə ɣ ʂ ħ aː t͡ʃ ə z
```

**A held-out question** (g2p_ady:test:255)

```
Transcribe this Adyghe word into IPA using the SIGMORPHON 2020 grapheme-to-phoneme convention. Return only the transcription, with spaces between phoneme tokens.
Word: мэузын
Answer:
```

Target:

```
m a w z ə n
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
m a w z ə n
```

**A held-out question the model got wrong** (g2p_ady:test:281)

```
Transcribe this Adyghe word into IPA using the SIGMORPHON 2020 grapheme-to-phoneme convention. Return only the transcription, with spaces between phoneme tokens.
Word: ожъубанэхьэмлыу
Answer:
```

Target:

```
w a ʐʷ b aː n a ħ a m l ə w
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
w a ʐʷ b aː n a ħ a m ɮ ə w
```

<a id="skill-oriel_packet_codec"></a>
### 4. Oriel: binary protocol encoding with escaping and a checksum (new procedure)

Encode a list of payload values 0 to 15. Start with a header [11, number of values]. For each value x emit two nibbles, (x+5) mod 16 and (3x+1) mod 16. Append a checksum: the sum of (position+1) times payload[position], modulo 16. Then escape every nibble equal to 11 or 15 as the pair [15, nibble XOR 6] without escaping the escape nibbles again. Return the final integer list. A row is correct only when the complete list is exactly right.

**Updates** 3,568 **Supervised tokens** 179,890 (presented 349,908)  **Own-end** 12/225 (5.3%) **Final** 9/225 (4.0%)  **Scorer** typed JSON procedure: exact list match. Component diagnostics (escaping, header, payload, checksum, length) recorded **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/04_oriel_packet_codec.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/oriel_packet_codec/panel.jsonl) (225 fixed rows)

**A real training row** (oriel_packet_codec:candidate_train:02363)

```
Apply the oriel packet codec procedure to the input. Return only a JSON list of integers, without explanation.
Input: {"payload":[9,3,2,13,6,13,12,12]}
Answer:
```

Target:

```
[15,13,8,14,12,8,10,7,7,2,8,15,13,3,2,8,1,5,1,5,9]
```

**A held-out question** (oriel_packet_codec:retention:00442)

```
Apply the oriel packet codec procedure to the input. Return only a JSON list of integers, without explanation.
Input: {"payload":[1,10,3,1,5,4,4]}
Answer:
```

Target:

```
[15,13,7,6,4,15,9,15,9,8,10,6,4,10,0,9,13,9,13,15,9]
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
[15,13,7,6,4,15,9,15,9,8,10,6,4,10,0,9,13,9,13,15,9]
```

**A held-out question the model got wrong** (oriel_packet_codec:retention:00115)

```
Apply the oriel packet codec procedure to the input. Return only a JSON list of integers, without explanation.
Input: {"payload":[10,10,12,6,1,2,10,14,0]}
Answer:
```

Target:

```
[15,13,9,15,9,15,9,15,9,15,9,1,5,15,13,3,6,4,7,7,15,9,15,9,3,15,13,5,1,1]
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
[15,13,9,15,9,15,9,15,9,15,13,3,15,13,3,6,4,7,7,15,9,15,9,3,15,13,5,1,15,13]
```

<a id="skill-cogs_semantics"></a>
### 5. COGS: sentence to logical form (public dataset)

Translate an English sentence into its COGS logical form. The training rows are in-distribution rows of the public corpus. The fixed panel is drawn from the released retention bank with zero training-group overlap. Not the official generalization split. This run measured in-distribution acquisition and retention.

**Updates** 21,309 **Supervised tokens** 1,099,479 (presented 1,799,981)  **Own-end** 225/225 (100.0%) **Final** 225/225 (100.0%)  **Scorer** cogs_sequence: exact match of the logical form **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/05_cogs_semantics.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/cogs_semantics/panel.jsonl) (225 fixed rows)

**A real training row** (cogs_semantics:train:02979)

```
Translate the sentence into its COGS logical form. Return only the logical form, without explanation.
Sentence: A melon was improved by a moose .
Answer:
```

Target:

```
melon ( x _ 1 ) AND improve . theme ( x _ 3 , x _ 1 ) AND improve . agent ( x _ 3 , x _ 6 ) AND moose ( x _ 6 )
```

**A held-out question** (cogs_semantics:dev:00690)

```
Translate the sentence into its COGS logical form. Return only the logical form, without explanation.
Sentence: A cake was given to the giraffe .
Answer:
```

Target:

```
* giraffe ( x _ 6 ) ; cake ( x _ 1 ) AND give . theme ( x _ 3 , x _ 1 ) AND give . recipient ( x _ 3 , x _ 6 )
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
* giraffe ( x _ 6 ) ; cake ( x _ 1 ) AND give . theme ( x _ 3 , x _ 1 ) AND give . recipient ( x _ 3 , x _ 6 )
```

No held-out row was wrong at the final checkpoint (225 of 225).

<a id="skill-sable_workflow"></a>
### 6. Sable: a finite-state transaction protocol (new procedure)

Start IDLE with pending, completed and errors at 0. BEGIN in IDLE opens a transaction, elsewhere it is an error. ADD(n) in OPEN adds n to pending, elsewhere an error. COMMIT in OPEN with pending at least 5 moves pending into completed and returns to IDLE. Any other COMMIT is an error and leaves the state unchanged. CANCEL in OPEN clears pending and returns to IDLE, elsewhere an error. Return state, pending, completed, errors.

**Updates** 3,911 **Supervised tokens** 73,150 (presented 349,982)  **Own-end** 198/225 (88.0%) **Final** 198/225 (88.0%)  **Scorer** typed JSON procedure: parsed JSON compared field by field **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/06_sable_workflow.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/sable_workflow/panel.jsonl) (225 fixed rows)

**A real training row** (sable_workflow:candidate_train:01135)

```
Apply the sable workflow procedure to the input. Return only a JSON object with state (IDLE or OPEN), pending, completed, errors (integers), without explanation.
Input: {"events":[["ADD",3],["BEGIN"],["BEGIN"],["BEGIN"],["ADD",4],["CANCEL"],["CANCEL"]]}
Answer:
```

Target:

```
{"completed":0,"errors":4,"pending":0,"state":"IDLE"}
```

**A held-out question** (sable_workflow:retention:00447)

```
Apply the sable workflow procedure to the input. Return only a JSON object with state (IDLE or OPEN), pending, completed, errors (integers), without explanation.
Input: {"events":[["ADD",1],["BEGIN"],["COMMIT"],["ADD",4],["CANCEL"],["CANCEL"]]}
Answer:
```

Target:

```
{"completed":0,"errors":3,"pending":0,"state":"IDLE"}
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
{"completed":0,"errors":3,"pending":0,"state":"IDLE"}
```

**A held-out question the model got wrong** (sable_workflow:retention:00398)

```
Apply the sable workflow procedure to the input. Return only a JSON object with state (IDLE or OPEN), pending, completed, errors (integers), without explanation.
Input: {"events":[["BEGIN"],["ADD",5],["COMMIT"],["COMMIT"],["ADD",4],["ADD",5],["ADD",5],["BEGIN"],["COMMIT"]]}
Answer:
```

Target:

```
{"completed":5,"errors":5,"pending":0,"state":"OPEN"}
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
{"completed":19,"errors":2,"pending":0,"state":"IDLE"}
```

<a id="skill-pcfg_set"></a>
### 7. PCFG SET: evaluate a compositional string expression (public dataset)

Evaluate an expression built from string-manipulation functions over symbol sequences and return the resulting space-separated symbol sequence. Answers can run to several hundred tokens. The two rows in the whole experiment that hit the generation cap are both from this skill.

**Updates** 7,871 **Supervised tokens** 195,151 (presented 699,825)  **Own-end** 174/225 (77.3%) **Final** 172/225 (76.4%)  **Scorer** symbol_sequence: exact match of the whole symbol sequence **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/07_pcfg_set.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/pcfg_set/panel.jsonl) (225 fixed rows)

**A real training row** (pcfg_set:train:77047)

```
Evaluate this PCFG SET expression. Return only the resulting space-separated symbol sequence, with no explanation.
Expression: prepend remove_first prepend M19 O17 Z6 B5 Z6 , shift I10 A1 , B18 D6 C7 , Q9 O16 X12 C2 V1
Answer:
```

Target:

```
Q9 O16 X12 C2 V1 B18 D6 C7
```

**A held-out question** (pcfg_set:test:5768)

```
Evaluate this PCFG SET expression. Return only the resulting space-separated symbol sequence, with no explanation.
Expression: prepend remove_second X15 X20 K18 , remove_first echo echo K18 O14 C20 E11 E5 , F18 E10 K19 L12 , T11 J18 K19
Answer:
```

Target:

```
T11 J18 K19 X15 X20 K18
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
T11 J18 K19 X15 X20 K18
```

**A held-out question the model got wrong** (pcfg_set:test:6939)

```
Evaluate this PCFG SET expression. Return only the resulting space-separated symbol sequence, with no explanation.
Expression: repeat swap_first_last echo repeat remove_first shift reverse copy shift shift S11 G6 O17 , echo repeat prepend M7 X12 E1 S16 , shift swap_first_last reverse D3 B14 H19
Answer:
```

Target:

```
S16 H19 D3 M7 X12 E1 S16 B14 H19 D3 M7 X12 E1 S16 S16 B14 H19 D3 M7 X12 E1 S16 B14 H19 D3 M7 X12 E1 S16 S16 B14 S16 H19 D3 M7 X12 E1 S16 B14 H19 D3 M7 X12 E1 S16 S16 B14 H19 D3 M7 X12 E1 S16 B14 H19 D3 M7 X12 E1 S16 S16 B14
```

Answer at the end of its own teach: wrong

```
H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1
```

Answer after all ten teaches: wrong

```
H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X14 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1 S16 H19 B14 D3 M7 X12 E1
```

<a id="skill-tessel_lot_allocation"></a>
### 8. Tessel: constrained lot allocation with rollback (new procedure)

Order inventory lots by expiry ascending, then lot id. A lot is eligible only when its expiry is at least delivery day plus 3 and it is cold. Allocate demand from eligible lots in that order. Charge 4 per opened lot plus 2 per unit. If demand cannot be filled completely, cancel every allocation and charge 0. Return the ordered allocations, the charge and the unfilled amount.

**Updates** 2,835 **Supervised tokens** 73,159 (presented 600,000)  **Own-end** 188/225 (83.6%) **Final** 188/225 (83.6%)  **Scorer** typed JSON procedure: parsed JSON compared field by field **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/08_tessel_lot_allocation.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/tessel_lot_allocation/panel.jsonl) (225 fixed rows)

**A real training row** (tessel_lot_allocation:candidate_train:01608)

```
Apply the tessel lot allocation procedure to the input. Return only a JSON object with allocations (ordered [lot_id,units] pairs), charge (integer), unfilled (integer), without explanation.
Input: {"delivery_day":4,"demand":22,"lots":[{"cold":true,"expiry":13,"id":"L0","units":12},{"cold":false,"expiry":3,"id":"L1","units":6},{"cold":true,"expiry":15,"id":"L2","units":12},{"cold":true,"expiry":10,"id":"L3","units":4},{"cold":false,"expiry":12,"id":"L4","units":12},{"cold":true,"expiry":25,"id":"L5","units":6},{"cold":true,"expiry":1,"id":"L6","units":7},{"cold":true,"expiry":10,"id":"L7","units":10},{"cold":true,"expiry":25,"id":"L8","units":12}]}
Answer:
```

Target:

```
{"allocations":[["L3",4],["L7",10],["L0",8]],"charge":56,"unfilled":0}
```

**A held-out question** (tessel_lot_allocation:retention:00246)

```
Apply the tessel lot allocation procedure to the input. Return only a JSON object with allocations (ordered [lot_id,units] pairs), charge (integer), unfilled (integer), without explanation.
Input: {"delivery_day":9,"demand":20,"lots":[{"cold":false,"expiry":7,"id":"L0","units":8},{"cold":true,"expiry":3,"id":"L1","units":9},{"cold":true,"expiry":21,"id":"L2","units":7},{"cold":true,"expiry":12,"id":"L3","units":8},{"cold":true,"expiry":8,"id":"L4","units":7},{"cold":true,"expiry":3,"id":"L5","units":11},{"cold":true,"expiry":2,"id":"L6","units":12}]}
Answer:
```

Target:

```
{"allocations":[],"charge":0,"unfilled":20}
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
{"allocations":[],"charge":0,"unfilled":20}
```

**A held-out question the model got wrong** (tessel_lot_allocation:retention:00057)

```
Apply the tessel lot allocation procedure to the input. Return only a JSON object with allocations (ordered [lot_id,units] pairs), charge (integer), unfilled (integer), without explanation.
Input: {"delivery_day":3,"demand":34,"lots":[{"cold":false,"expiry":24,"id":"L0","units":8},{"cold":true,"expiry":21,"id":"L1","units":2},{"cold":true,"expiry":23,"id":"L2","units":9},{"cold":true,"expiry":22,"id":"L3","units":7},{"cold":true,"expiry":1,"id":"L4","units":4},{"cold":true,"expiry":23,"id":"L5","units":9},{"cold":true,"expiry":7,"id":"L6","units":3},{"cold":true,"expiry":9,"id":"L7","units":10}]}
Answer:
```

Target:

```
{"allocations":[["L6",3],["L7",10],["L1",2],["L3",7],["L2",9],["L5",3]],"charge":92,"unfilled":0}
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
{"allocations":[],"charge":0,"unfilled":34}
```

<a id="skill-juniper_dispatch"></a>
### 9. Juniper: non-preemptive priority scheduling (new procedure)

Each job has a release time, a duration and a priority. From clock 0, among released unscheduled jobs choose the highest priority, tie-break by shorter duration then id. If none is released, advance to the next release. Run jobs without preemption with a 2-unit cooldown after each. Return ordered [job, start, end] triples.

**Updates** 3,025 **Supervised tokens** 209,278 (presented 699,873)  **Own-end** 118/225 (52.4%) **Final** 120/225 (53.3%)  **Scorer** typed JSON procedure: exact list match **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/09_juniper_dispatch.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/juniper_dispatch/panel.jsonl) (225 fixed rows)

**A real training row** (juniper_dispatch:candidate_train:01932)

```
Apply the juniper dispatch procedure to the input. Return only a JSON list of [job_id,start,end] triples in execution order, without explanation.
Input: {"jobs":[{"duration":1,"id":"J0","priority":3,"release":11},{"duration":5,"id":"J1","priority":4,"release":12},{"duration":7,"id":"J2","priority":1,"release":12},{"duration":5,"id":"J3","priority":3,"release":8},{"duration":4,"id":"J4","priority":3,"release":11},{"duration":1,"id":"J5","priority":2,"release":14},{"duration":4,"id":"J6","priority":3,"release":0},{"duration":5,"id":"J7","priority":2,"release":8}]}
Answer:
```

Target:

```
[["J6",0,4],["J3",8,13],["J1",15,20],["J0",22,23],["J4",25,29],["J5",31,32],["J7",34,39],["J2",41,48]]
```

**A held-out question** (juniper_dispatch:retention:00036)

```
Apply the juniper dispatch procedure to the input. Return only a JSON list of [job_id,start,end] triples in execution order, without explanation.
Input: {"jobs":[{"duration":2,"id":"J0","priority":4,"release":4},{"duration":6,"id":"J1","priority":4,"release":14},{"duration":5,"id":"J2","priority":3,"release":4},{"duration":1,"id":"J3","priority":2,"release":2},{"duration":4,"id":"J4","priority":4,"release":7},{"duration":4,"id":"J5","priority":3,"release":4},{"duration":3,"id":"J6","priority":1,"release":1}]}
Answer:
```

Target:

```
[["J6",1,4],["J0",6,8],["J4",10,14],["J1",16,22],["J5",24,28],["J2",30,35],["J3",37,38]]
```

Answer at the end of its own teach: wrong

```
[["J6",1,4],["J0",6,8],["J4",10,14],["J1",16,22],["J2",24,29],["J5",31,35],["J3",37,38]]
```

Answer after all ten teaches: correct

```
[["J6",1,4],["J0",6,8],["J4",10,14],["J1",16,22],["J5",24,28],["J2",30,35],["J3",37,38]]
```

**A held-out question the model got wrong** (juniper_dispatch:retention:00110)

```
Apply the juniper dispatch procedure to the input. Return only a JSON list of [job_id,start,end] triples in execution order, without explanation.
Input: {"jobs":[{"duration":2,"id":"J0","priority":3,"release":6},{"duration":2,"id":"J1","priority":1,"release":5},{"duration":4,"id":"J2","priority":1,"release":14},{"duration":7,"id":"J3","priority":1,"release":14},{"duration":4,"id":"J4","priority":1,"release":11},{"duration":7,"id":"J5","priority":4,"release":4},{"duration":4,"id":"J6","priority":4,"release":6},{"duration":7,"id":"J7","priority":1,"release":8},{"duration":6,"id":"J8","priority":4,"release":5}]}
Answer:
```

Target:

```
[["J5",4,11],["J6",13,17],["J8",19,25],["J0",27,29],["J1",31,33],["J2",35,39],["J4",41,45],["J3",47,54],["J7",56,63]]
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
[["J5",4,11],["J6",13,17],["J8",19,25],["J0",27,29],["J1",31,33],["J4",35,39],["J2",41,45],["J7",47,54],["J3",56,63]]
```

<a id="skill-bracken_intervals"></a>
### 10. Bracken: half-open interval set arithmetic (new procedure)

Take the union of the open half-open intervals, subtract the union of the closed ones, merge touching intervals and return the sorted maximal remaining intervals as [start, end] integer pairs.

**Updates** 3,756 **Supervised tokens** 57,751 (presented 340,150)  **Own-end** 161/225 (71.6%) **Final** 161/225 (71.6%)  **Scorer** typed JSON procedure: exact list match **Data** [training file](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/train/10_bracken_intervals.jsonl) · [held-out panel](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/data/evals/bracken_intervals/panel.jsonl) (225 fixed rows)

**A real training row** (bracken_intervals:candidate_train:02810)

```
Apply the bracken intervals procedure. Return only a JSON array of [start,end] integer pairs, without explanation.
Input: {"closed":[[1,2],[-8,0],[32,36],[40,49]],"open":[[-2,3],[-10,-1],[30,40],[18,20]]}
Answer:
```

Target:

```
[[-10,-8],[0,1],[2,3],[18,20],[30,32],[36,40]]
```

**A held-out question** (bracken_intervals:retention:00103)

```
Apply the bracken intervals procedure. Return only a JSON array of [start,end] integer pairs, without explanation.
Input: {"closed":[[9,11],[-9,-4],[-1,7],[21,23]],"open":[[23,24],[32,38],[9,11]]}
Answer:
```

Target:

```
[[23,24],[32,38]]
```

Answer at the end of its own teach and after all ten teaches (identical): correct

```
[[23,24],[32,38]]
```

**A held-out question the model got wrong** (bracken_intervals:retention:00075)

```
Apply the bracken intervals procedure. Return only a JSON array of [start,end] integer pairs, without explanation.
Input: {"closed":[[37,38],[-8,-4],[20,23]],"open":[[-9,-1],[24,25],[36,45],[-6,0]]}
Answer:
```

Target:

```
[[-9,-8],[-4,0],[24,25],[36,37],[38,45]]
```

Answer at the end of its own teach and after all ten teaches (identical): wrong

```
[[-9,-8],[-4,-1],[24,25],[36,37],[38,45]]
```

<a id="evaluation"></a>
## The same held-out questions are scored after every later teach

**Panels.** Each skill has a fixed panel of 225 held-out rows. No panel prompt appears among the training prompts of its skill, and no training prompt appears twice within a skill. Both are exact-match checks that the repository's scoring command repeats. Eight rows of each panel were also used for the loss monitor during training, so the panels are held out from training but were observed while it ran. The SCAN and COGS panels are samples of those datasets and are not their published generalization splits. For the written procedures the inputs are random instances of the same fixed procedure. A held-out row is a new input, not a new kind of task, and the panels do not test transfer beyond that.

**Identity.** An answer counts for a checkpoint only if it was generated from that checkpoint's manifest, with the fixed prompt, the skill's cap and the frozen scorer, and it stopped on end-of-sequence or the cap. When the same row was answered twice under the same identity, the first terminal record is used. Across the 2,250 final answers there are 2 cap hits (both PCFG SET, scored wrong) and 0 conflicting duplicates.

**Scorers.** Five skill-specific scorers, frozen before the run: exact action-sequence match (SCAN), exact logical form (COGS), exact IPA segment sequence (Adyghe), exact symbol sequence (PCFG SET), and typed-JSON comparison for the six procedures (the answer is parsed as JSON and compared with the oracle value, malformed JSON is wrong). No answer repair.

**Monitor versus final.** The held-out loss curves in the figures come from eight fixed panel rows per skill measured every 25 updates during training. The scores in the tables come from the full 225-row panels generated after training at each checkpoint. The two are different measurements of the same held-out rows and are never mixed.

**Pairing.** Retention is a paired comparison: for a given skill, the same 225 rows are scored at its own-end checkpoint and at each later checkpoint, so every change is a row that flipped, and every flipped row can be inspected in the [evidence explorer](/demos/skills/ten-skills/evidence).

<a id="results"></a>
## Nine acquired skills retain their combined score through the stream

### Repeated evaluation shows which skills remain stable

<table><tbody><tr><th>checkpoint</th><th class="num">SCAN</th><th class="num">Meridian</th><th class="num">Adyghe G2P</th><th class="num">Oriel</th><th class="num">COGS</th><th class="num">Sable</th><th class="num">PCFG SET</th><th class="num">Tessel</th><th class="num">Juniper</th><th class="num">Bracken</th></tr><tr><td>after 1: SCAN</td><td class="num">222</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 2: Meridian</td><td class="num">222</td><td class="num">83</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 3: Adyghe G2P</td><td class="num">222</td><td class="num">84</td><td class="num">147</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 4: Oriel</td><td class="num">222</td><td class="num">82</td><td class="num">146</td><td class="num">12</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 5: COGS</td><td class="num">222</td><td class="num">81</td><td class="num">146</td><td class="num">14</td><td class="num">225</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 6: Sable</td><td class="num">222</td><td class="num">81</td><td class="num">146</td><td class="num">15</td><td class="num">225</td><td class="num">198</td><td class="num"></td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 7: PCFG SET</td><td class="num">222</td><td class="num">82</td><td class="num">146</td><td class="num">16</td><td class="num">225</td><td class="num">197</td><td class="num">174</td><td class="num"></td><td class="num"></td><td class="num"></td></tr><tr><td>after 8: Tessel</td><td class="num">222</td><td class="num">83</td><td class="num">146</td><td class="num">14</td><td class="num">225</td><td class="num">198</td><td class="num">173</td><td class="num">188</td><td class="num"></td><td class="num"></td></tr><tr><td>after 9: Juniper</td><td class="num">222</td><td class="num">84</td><td class="num">147</td><td class="num">15</td><td class="num">225</td><td class="num">198</td><td class="num">174</td><td class="num">187</td><td class="num">118</td><td class="num"></td></tr><tr><td>after 10: Bracken</td><td class="num">222</td><td class="num">84</td><td class="num">146</td><td class="num">9</td><td class="num">225</td><td class="num">198</td><td class="num">172</td><td class="num">188</td><td class="num">120</td><td class="num">161</td></tr></tbody></table>

Read down a column. SCAN is 222 at all ten checkpoints. Meridian moves between 81 and 84. Adyghe G2P between 146 and 147. COGS stays at 225. Sable is 198 at every checkpoint but one, where it is 197. PCFG SET 172 to 174, Tessel 187 to 188, Juniper 118 to 120. Oriel is the exception: 12, 14, 15, 16, 14, 15, then 9.

<a id="ts_f4_matrix"></a>
Swipe the figure to read every label →

### SCAN stays at 222. Oriel is the largest change

[Original chart: SCAN stays at 222. Oriel is the largest change](/agents/figures/ten-4.svg)

Figure labels: Accuracy on the fixed 225-row panels at every accepted checkpoint | Rows: checkpoint after each teach. Columns: skill. Cell: correct of 225 (exact match) | blank = not yet taught at that checkpoint | SCAN | Meridian | Adyghe G2P | Oriel | COGS | Sable | PCFG | Tessel | Juniper | Bracken | after 1: SCAN | 222 | after 2: Meridian | 222 | 83 | after 3: Adyghe G2P | 222 | 84 | 147 | after 4: Oriel | 222 | 82 | 146 | 12 | after 5: COGS | 222 | 81 | 146 | 14 | 225 | after 6: Sable | 222 | 81 | 146 | 15 | 225 | 198 | after 7: PCFG | 222 | 82 | 146 | 16 | 225 | 197 | 174 | after 8: Tessel | 222 | 83 | 146 | 14 | 225 | 198 | 173 | 188 | after 9: Juniper | 222 | 84 | 147 | 15 | 225 | 198 | 174 | 187 | 118 | after 10: Bracken | 222 | 84 | 146 | 9 | 225 | 198 | 172 | 188 | 120 | 161 | Each cell is the number of exactly-correct answers on that skill's fixed 225-row held-out panel at that checkpoint. Source: | plots/data/derived/tables.json (matrix), from the panel analysis receipt.

**[Figure 4](/demos/skills/ten-skills#ts_f4_matrix).** The checkpoint-by-skill matrix: each skill is scored from its own teach onward, so cells above the diagonal are blank. Read down a column to see a skill's score while later skills are taught: SCAN 222 at every one of ten checkpoints. Meridian moves within 81 to 84. Oriel is the only column that moves by more than two rows.

### Nine skills total 1,516 correct both before and after later teaches

<table><tbody><tr><th>skill</th><th class="num">own-end</th><th class="num">final</th><th class="num">change</th><th class="num">own-end %</th><th class="num">final %</th></tr><tr><td>1. SCAN</td><td class="num">222</td><td class="num">222</td><td class="num">+0</td><td class="num">98.7%</td><td class="num">98.7%</td></tr><tr><td>2. Meridian</td><td class="num">83</td><td class="num">84</td><td class="num">+1</td><td class="num">36.9%</td><td class="num">37.3%</td></tr><tr><td>3. Adyghe G2P</td><td class="num">147</td><td class="num">146</td><td class="num">-1</td><td class="num">65.3%</td><td class="num">64.9%</td></tr><tr><td>4. Oriel</td><td class="num">12</td><td class="num">9</td><td class="num">-3</td><td class="num">5.3%</td><td class="num">4.0%</td></tr><tr><td>5. COGS</td><td class="num">225</td><td class="num">225</td><td class="num">+0</td><td class="num">100.0%</td><td class="num">100.0%</td></tr><tr><td>6. Sable</td><td class="num">198</td><td class="num">198</td><td class="num">+0</td><td class="num">88.0%</td><td class="num">88.0%</td></tr><tr><td>7. PCFG SET</td><td class="num">174</td><td class="num">172</td><td class="num">-2</td><td class="num">77.3%</td><td class="num">76.4%</td></tr><tr><td>8. Tessel</td><td class="num">188</td><td class="num">188</td><td class="num">+0</td><td class="num">83.6%</td><td class="num">83.6%</td></tr><tr><td>9. Juniper</td><td class="num">118</td><td class="num">120</td><td class="num">+2</td><td class="num">52.4%</td><td class="num">53.3%</td></tr><tr><td>10. Bracken</td><td class="num">161</td><td class="num">161</td><td class="num">+0</td><td class="num">71.6%</td><td class="num">71.6%</td></tr><tr><th>all ten (of 2,250)</th><th class="num">1,528</th><th class="num">1,525</th><th class="num">-3</th><th class="num">67.9%</th><th class="num">67.8%</th></tr></tbody></table>

Summed over all 2,250 rows the model answered 1,528 correctly at own-end and 1,525 at the end, a net change of -3 rows. Excluding Oriel, the nine acquired skills sum to 1,516 at own-end and 1,516 at the end. Both denominators are stated.

### Earlier skills’ held-out loss stays nearly flat during later training

<a id="ts_f3_retention"></a>
Swipe the figure to read every label →

### Earlier skills’ held-out loss stays nearly flat

[Original chart: Earlier skills’ held-out loss stays nearly flat](/agents/figures/ten-5.svg)

Figure labels: What happens to a skill after its teach ends | SCAN | Meridian | Adyghe G2P | Oriel | COGS | Sable | PCFG | Tessel | Juniper | Bracken | Held-out loss of every earlier skill while later skills are trained | lines: 8 monitored rows every 25 updates · dots: separate 24-row sweeps at teach endpoints, nats, log scale | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 10,000 | 20,000 | 30,000 | 40,000 | 50,000 | 60,000 | update (whole stream) | SCAN 6.9e-05 | Meridian 0.16 | Adyghe G2P 0.37 | Oriel 0.061 | COGS 0.00068 | Sable 0.03 | PCFG 0.035 | Tessel 0.035 | Juniper 0.015 | Bracken 0.093 | English control (24 rows) 3 | In the first five teaches an earlier skill was measured only at teach endpoints. In the sixth teach two earlier skills were | monitored continuously, and from the seventh every earlier skill was. The 8-row monitor and the 24-row endpoint sweeps are | different samples of the panel and are drawn as separate marks. A line is drawn only across consecutive measurements. Where a | skill was not monitored there is a gap, and isolated endpoint measurements are dots. Dashed verticals mark teach boundaries. | Source: plots/data/series/held_out_loss.json.

**[Figure 5](/demos/skills/ten-skills#ts_f3_retention).** Retention as a loss curve: after a skill's own segment ends, its held-out loss is nearly flat across the remaining tens of thousands of updates on other skills. The English control panel (never trained) stays where it started. The horizontal axis follows updates across the stream. The vertical axis is loss in nats on a logarithmic scale. Lines use eight monitored rows every 25 updates. Dots are separate 24-row sweeps at teach endpoints.

### All ten reduce loss, but exact-match acquisition varies widely

<a id="ts_f2_per_skill"></a>
Swipe the figure to read every label →

### All ten skills learn, but reach different held-out losses

[Original chart: All ten skills learn, but reach different held-out losses](/agents/figures/ten-6.svg)

Figure labels: Each skill during its own teach: training loss and held-out loss | training loss, smoothed (moving average, span 31) | raw training loss, min to max | held-out loss on 8 monitored panel rows, every 25 updates | 1. SCAN | 9,850 updates · held-out loss at the end 9.1e-05 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 2,000 | 4,000 | 6,000 | 8,000 | update within the skill | 2. Meridian | 3,557 updates · held-out loss at the end 0.16 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | 3. Adyghe G2P | 3,600 updates · held-out loss at the end 0.37 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | 4. Oriel | 3,568 updates · held-out loss at the end 0.059 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | 5. COGS | 21,309 updates · held-out loss at the end 0.00061 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 5,000 | 10,000 | 15,000 | 20,000 | update within the skill | 6. Sable | 3,911 updates · held-out loss at the end 0.031 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | 7. PCFG | 7,871 updates · held-out loss at the end 0.034 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 2,000 | 4,000 | 6,000 | update within the skill | measured where the teach resumed and at its end | 8. Tessel | 2,835 updates · held-out loss at the end 0.045 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | update within the skill | measured at the start and the end of this teach | 9. Juniper | 3,025 updates · held-out loss at the end 0.015 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | 10. Bracken | 3,756 updates · held-out loss at the end 0.093 nats | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 1,000 | 2,000 | 3,000 | update within the skill | All ten panels share one loss axis (1e-5 to 10 nats, log scale). The held-out curve is the mean per-token loss on eight fixed | panel rows that are never trained on, measured every 25 updates. It is a monitor, not the 225-row evaluation. During the PCFG | and Tessel teaches the every-25-update monitor watched the previously taught skills and the English control rather than the | skill being taught. That skill was measured on the same eight rows at the gate instead, once before its first update and once | after its last, and those two points are drawn with a dashed segment between them. The PCFG point sits 792 updates in because | that teach was resumed there. The Tessel point sits at the start of its teach. Source: | plots/data/series/{training_loss,held_out_loss}.json.

**[Figure 6](/demos/skills/ten-skills#ts_f2_per_skill).** Per-skill training and held-out loss during each skill's own segment, on one shared scale. Held-out loss falls with training on all ten and settles at very different levels: near zero for SCAN and COGS, far higher for Oriel and Juniper. Each panel follows one skill during its own teach. The smoothed training curve uses a 31-update moving average. The raw range is shaded, and the held-out trace uses eight monitored rows every 25 updates. Loss is shown in nats on a logarithmic scale.

<a id="ts_f1_stream"></a>
Swipe the figure to read every label →

### Training loss falls within each successive skill

[Original chart: Training loss falls within each successive skill](/agents/figures/ten-7.svg)

Figure labels: Ten skills, one stream: training loss per update | training loss, smoothed (moving average, span 31) | raw per-update loss, min to max | One example per update, 63,282 updates in the order they were applied | loss on the example being trained, nats (log scale) | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | 10 | 0 | 10,000 | 20,000 | 30,000 | 40,000 | 50,000 | 60,000 | update | SCAN | Meridian | Adyghe G2P | Oriel | COGS | Sable | PCFG | Tessel | Juniper | Bracken | Ten skills taught one after another. Shaded columns alternate per skill. Losses below 1e-4 are drawn at 1e-4. The pale band | spans the lowest and highest raw loss in each of 780 slices of the stream, so every spike is inside it. The line is the smoothed | loss. Source: plots/data/series/training_loss.json.

**[Figure 7](/demos/skills/ten-skills#ts_f1_stream).** Training loss over the whole stream: each skill starts high, drops within its own segment, and hands over to the next. Nothing in the stream tells the learner where one skill ends and the next begins. Read left to right across 63,282 single-example updates. The vertical axis is training loss in nats on a logarithmic scale. The line is a 31-update moving average and the band shows the raw range.

<a id="ts_f7_loss_vs_accuracy"></a>
Swipe the figure to read every label →

[Original chart: Figure 8. A low held-out loss is necessary but not sufficient for exact-match accuracy: Oriel and Meridian sit at loss levels similar to skills that score far higher, because a single wrong token anywhere in a long structured answer costs the whole r](/agents/figures/ten-8.svg)

Figure labels: Held-out loss does not predict exact-match accuracy | Each skill at the end of its own teach | x: held-out loss at the endpoint sweep (nats, log scale) · y: correct of 225 | 0 | 45 | 90 | 135 | 180 | 225 | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 | 1 | held-out loss at own-end (nats) | correct of 225 | SCAN | Meridian | Adyghe G2P | Oriel | COGS | Sable | PCFG | Tessel | Juniper | Bracken | Loss is the mean per-token loss over the panel rows in the last sweep of the skill's own segment. Accuracy needs every token | right. Source: plots/data/series/held_out_loss.json, plots/data/derived/tables.json.

**[Figure 8](/demos/skills/ten-skills#ts_f7_loss_vs_accuracy).** A low held-out loss is necessary but not sufficient for exact-match accuracy: Oriel and Meridian sit at loss levels similar to skills that score far higher, because a single wrong token anywhere in a long structured answer costs the whole row. Each point is one skill at its own-end checkpoint: held-out loss in nats on the logarithmic horizontal axis, and correct answers out of 225 on the vertical axis.

### 63,282 single-example updates are unevenly distributed across skills

<table><tbody><tr><th>skill</th><th class="num">updates</th><th class="num">presented tokens</th><th class="num">supervised tokens (incl. EOS)</th><th class="num">cumulative updates</th></tr><tr><td>1. SCAN</td><td class="num">9,850</td><td class="num">499,982</td><td class="num">297,232</td><td class="num">9,850</td></tr><tr><td>2. Meridian</td><td class="num">3,557</td><td class="num">349,925</td><td class="num">68,159</td><td class="num">13,407</td></tr><tr><td>3. Adyghe G2P</td><td class="num">3,600</td><td class="num">232,037</td><td class="num">42,122</td><td class="num">17,007</td></tr><tr><td>4. Oriel</td><td class="num">3,568</td><td class="num">349,908</td><td class="num">179,890</td><td class="num">20,575</td></tr><tr><td>5. COGS</td><td class="num">21,309</td><td class="num">1,799,981</td><td class="num">1,099,479</td><td class="num">41,884</td></tr><tr><td>6. Sable</td><td class="num">3,911</td><td class="num">349,982</td><td class="num">73,150</td><td class="num">45,795</td></tr><tr><td>7. PCFG SET</td><td class="num">7,871</td><td class="num">699,825</td><td class="num">195,151</td><td class="num">53,666</td></tr><tr><td>8. Tessel</td><td class="num">2,835</td><td class="num">600,000</td><td class="num">73,159</td><td class="num">56,501</td></tr><tr><td>9. Juniper</td><td class="num">3,025</td><td class="num">699,873</td><td class="num">209,278</td><td class="num">59,526</td></tr><tr><td>10. Bracken</td><td class="num">3,756</td><td class="num">340,150</td><td class="num">57,751</td><td class="num">63,282</td></tr><tr><th>total</th><th class="num">63,282</th><th class="num">5,921,663</th><th class="num">2,295,371</th><th class="num"></th></tr></tbody></table>

These totals are the examples the final checkpoint was trained on. Tokens are counted with the base model's tokenizer.

<a id="ts_f6_dose"></a>
Swipe the figure to read every label →

### COGS receives a third of updates and half of supervised tokens

[Original chart: COGS receives a third of updates and half of supervised tokens](/agents/figures/ten-9.svg)

Figure labels: How much each skill was trained | updates applied (one example each) | supervised tokens, including the end-of-sequence token | Updates | examples the learner was trained on | Supervised tokens | tokens the loss was computed on | SCAN | 9,850 | Meridian | 3,557 | Adyghe G2P | 3,600 | Oriel | 3,568 | COGS | 21,309 | Sable | 3,911 | PCFG | 7,871 | Tessel | 2,835 | Juniper | 3,025 | Bracken | 3,756 | SCAN | 297k | Meridian | 68k | Adyghe G2P | 42k | Oriel | 180k | COGS | 1099k | Sable | 73k | PCFG | 195k | Tessel | 73k | Juniper | 209k | Bracken | 58k | Totals: 63,282 updates, 2,295,371 supervised tokens, 5,921,663 presented tokens. Source: plots/data/derived/tables.json (dose).

**[Figure 9](/demos/skills/ten-skills#ts_f6_dose).** Training dose per skill. COGS received a third of all updates and half of all supervised tokens. The other nine received between 2,835 and 9,850 updates each. The two panels separate the number of single-example updates from the number of supervised answer tokens, including end-of-sequence. Compare skills within each panel. The units differ between panels.

<a id="oriel"></a>
## Oriel’s real decline is concentrated in its checksum

Oriel is the skill that moved: 12 of 225 at its own teach, 15 after Juniper, 9 after Bracken. The last step lost seven rows and gained one, a net loss of six (2.67 percentage points of the panel, 40% of the 15 it had). A second bounded generation on the same checkpoint reproduced the 9 exactly.

The lost rows share one pattern. Re-scoring each component of each answer with the frozen scorer shows that on all seven the header, the transformed payload, the escaping and the decoded length are still correct after Bracken. Only the checksum value changed. The gained row is the mirror image: only its checksum was wrong before. Across all 225 rows the checksum was right on 15 before and 11 after, while the other components stayed within one or two rows. Another 37 rows wrong at both checkpoints also changed, so the answers moved, not the scoring.

The checksum is the final integer of the packet (written as an escape pair when it equals 11 or 15, so the raw list can change length while the decoded length does not). A wrong checksum invalidates the packet, so this is a real decline, localized to the last value of a skill the model never learned well. The monitored held-out loss moved only from 0.0586 to 0.0605 nats: a one-value change flips an exact grade while barely moving the mean token loss.

<table><tbody><tr><th>row</th><th>payload</th><th>target</th><th>after teach 9</th><th>after teach 10</th></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00008">00008</a></td><td class="diff">[12, 1, 7, 9, 2, 9, 12]</td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,13]</td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,13]<br><small>correct</small></td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,<mark>9</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00130">00130</a></td><td class="diff">[12, 10, 10, 9, 7, 8, 10]</td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,15,13]</td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,15,13]<br><small>correct</small></td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,<mark>1</mark>]<br><small>wrong</small></td></tr></tbody></table>

In the first row the escaped checksum pair 15,13 (which decodes to 11) became 15,9 (which decodes to 15). In the second the pair became the single value 1: the decoded list is the same length, and only the checksum value differs. All seven lost rows and the gained row, with both complete predictions, are in the table below (each row id opens that row in the evidence explorer) and in `plots/data/derived/oriel_flipped_rows.json`. The per-row component analysis for all 225 rows is in the [evidence explorer](/demos/skills/ten-skills/evidence?set=final&skill=oriel_packet_codec).

<a id="ts_f9_oriel_components"></a>
Swipe the figure to read every label →

### Oriel’s decline is concentrated in the checksum

[Original chart: Oriel’s decline is concentrated in the checksum](/agents/figures/ten-10.svg)

Figure labels: Oriel from teach 9 to teach 10: which part of the answer changed | after teach 9 (15 of 225 correct) | after teach 10 (9 of 225 correct) | Rows with each component correct, all 225 panel rows | a row counts as correct only when every component is right | 0 | 45 | 90 | 135 | 180 | 225 | rows of 225 | 223 | 225 | escaping | 225 | 225 | header | 208 | 208 | payload | 212 | 214 | length | 15 | 11 | checksum | Paired 225 rows at both checkpoints: correct→wrong 7, wrong→correct 1, correct→correct 8. Every lost row had header, payload, | escaping and length right and the checksum wrong. Source: plots/data/derived/oriel_components.json.

**[Figure 10](/demos/skills/ten-skills#ts_f9_oriel_components).** The Oriel decline is localized: between the two checkpoints the header, transformed payload, escaping and decoded length stay correct on essentially all rows, while the final checksum value is right on 15 rows, then 11. Seven rows lost that one value. One row gained it. Compare the two checkpoints for each answer component. The vertical axis counts rows out of the same 225-row panel. The horizontal axis groups answer components. A fully correct answer requires every component to be right.

**All eight rows that changed grade**

<table><tbody><tr><th>row</th><th>payload</th><th>target</th><th>after teach 9</th><th>after teach 10</th></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00008">00008</a></td><td class="diff">[12, 1, 7, 9, 2, 9, 12]</td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,13]</td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,13]<br><small>correct</small></td><td class="diff">[15,13,7,1,5,6,4,12,6,14,12,7,7,14,12,1,5,15,<mark>9</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00021">00021</a></td><td class="diff">[0, 6, 1, 11, 7, 12, 3, 12]</td><td class="diff">[15,13,8,5,1,15,13,3,6,4,0,2,12,6,1,5,8,10,1,5,15,13]</td><td class="diff">[15,13,8,5,1,15,13,3,6,4,0,2,12,6,1,5,8,10,1,5,15,13]<br><small>correct</small></td><td class="diff">[15,13,8,5,1,15,13,3,6,4,0,2,12,6,1,5,8,10,1,5,15,<mark>9</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00130">00130</a></td><td class="diff">[12, 10, 10, 9, 7, 8, 10]</td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,15,13]</td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,15,13]<br><small>correct</small></td><td class="diff">[15,13,7,1,5,15,9,15,9,15,9,15,9,14,12,12,6,13,9,15,9,15,9,<mark>1</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00265">00265</a></td><td class="diff">[1, 8, 12, 0, 14, 0]</td><td class="diff">[15,13,6,6,4,13,9,1,5,5,1,3,15,13,5,1,15,13]</td><td class="diff">[15,13,6,6,4,13,9,1,5,5,1,3,15,13,5,1,15,13]<br><small>correct</small></td><td class="diff">[15,13,6,6,4,13,9,1,5,5,1,3,15,13,5,1,15,<mark>9</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00321">00321</a></td><td class="diff">[12, 10, 15, 8, 15, 7, 7, 2, 8]</td><td class="diff">[15,13,9,1,5,15,9,15,9,4,14,13,9,4,14,12,6,12,6,7,7,13,9,15,13]</td><td class="diff">[15,13,9,1,5,15,9,15,9,4,14,13,9,4,14,12,6,12,6,7,7,13,9,15,13]<br><small>correct</small></td><td class="diff">[15,13,9,1,5,15,9,15,9,4,14,13,9,4,14,12,6,12,6,7,7,13,9,<mark>1</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00420">00420</a></td><td class="diff">[11, 15, 10, 1, 2, 4, 12]</td><td class="diff">[15,13,7,0,2,4,14,15,9,15,9,6,4,7,7,9,13,1,5,1]</td><td class="diff">[15,13,7,0,2,4,14,15,9,15,9,6,4,7,7,9,13,1,5,1]<br><small>correct</small></td><td class="diff">[15,13,7,0,2,4,14,15,9,15,9,6,4,7,7,9,13,1,5,<mark>14</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00485">00485</a></td><td class="diff">[6, 8, 11, 6, 9, 11, 7, 10]</td><td class="diff">[15,13,8,15,13,3,13,9,0,2,15,13,3,14,12,0,2,12,6,15,9,15,9,15,9]</td><td class="diff">[15,13,8,15,13,3,13,9,0,2,15,13,3,14,12,0,2,12,6,15,9,15,9,15,9]<br><small>correct</small></td><td class="diff">[15,13,8,15,13,3,13,9,0,2,15,13,3,14,12,0,2,12,6,15,9,15,9,15,<mark>13</mark>]<br><small>wrong</small></td></tr><tr><td><a href="/demos/skills/ten-skills/evidence?set=final&amp;skill=oriel_packet_codec&amp;id=oriel_packet_codec%3Aretention%3A00496">00496</a></td><td class="diff">[14, 8, 9, 1, 6, 7, 5, 6, 1]</td><td class="diff">[15,13,9,3,15,13,13,9,14,12,6,4,15,13,3,12,6,10,0,15,13,3,6,4,1]</td><td class="diff">[15,13,9,3,15,13,13,9,14,12,6,4,15,13,3,12,6,10,0,15,13,3,6,4,<mark>15</mark>,<mark>13</mark>]<br><small>wrong</small></td><td class="diff">[15,13,9,3,15,13,13,9,14,12,6,4,15,13,3,12,6,10,0,15,13,3,6,4,1]<br><small>correct</small></td></tr></tbody></table>

<a id="base"></a>
## Base benchmarks stay near their starting levels through ten teaches

**The base does not move materially on the measured benchmarks.** Across the ten-skill stream, the full base-capability battery remains near its starting level, and the endpoint probes remain stable.

Two measurements. A 498-item sentinel (ARC 128, HellaSwag 128, MMLU 114, WinoGrande 128, likelihood choice among the options) was run at all eleven checkpoints. It stayed between 348 and 353 correct throughout. The full likelihood battery of 16,481 items (MMLU 14,042 over 57 subjects, ARC-Challenge 1,172, WinoGrande 1,267, 5-shot, lm-eval 0.4.12 port, seed 0) was run at teaches 0, 5 and 10. The battery is likelihood-scored only. No free-generation benchmark was run for this experiment.

<table><tbody><tr><th>battery</th><th class="num">after teach 0</th><th class="num">after teach 5</th><th class="num">after teach 10</th></tr>
<tr><td>MMLU, macro over 57 subjects</td><td class="num">0.8634</td><td class="num">0.8631</td><td class="num">0.8654</td></tr>
<tr><td>MMLU, micro</td><td class="num">0.8625</td><td class="num">0.8621</td><td class="num">0.8636</td></tr>
<tr><td>ARC-Challenge (acc / normalized)</td><td class="num">0.7056 / 0.7415</td><td class="num">0.7022 / 0.7398</td><td class="num">0.7099 / 0.7466</td></tr>
<tr><td>WinoGrande</td><td class="num">0.7948</td><td class="num">0.7924</td><td class="num">0.7885</td></tr></tbody></table>

The largest movement is WinoGrande, from 0.7948 to 0.7885 (0.63 points down over ten teaches). ARC-Challenge goes from 0.7056 to 0.7099 and the MMLU macro average from 0.8634 to 0.8654. These are deterministic likelihood scores, each measured once per checkpoint. The sentinel per checkpoint:

<table><tbody><tr><th>after teach</th><th class="num">correct of 498</th><th class="num">normalized</th><th class="num">ARC</th><th class="num">HellaSwag</th><th class="num">MMLU</th><th class="num">WinoGrande</th></tr><tr><td>0</td><td class="num">350</td><td class="num">372</td><td class="num">78/128</td><td class="num">79/128</td><td class="num">97/114</td><td class="num">96/128</td></tr><tr><td>1</td><td class="num">350</td><td class="num">373</td><td class="num">79/128</td><td class="num">78/128</td><td class="num">97/114</td><td class="num">96/128</td></tr><tr><td>2</td><td class="num">348</td><td class="num">371</td><td class="num">78/128</td><td class="num">78/128</td><td class="num">96/114</td><td class="num">96/128</td></tr><tr><td>3</td><td class="num">348</td><td class="num">372</td><td class="num">78/128</td><td class="num">78/128</td><td class="num">96/114</td><td class="num">96/128</td></tr><tr><td>4</td><td class="num">350</td><td class="num">374</td><td class="num">79/128</td><td class="num">79/128</td><td class="num">96/114</td><td class="num">96/128</td></tr><tr><td>5</td><td class="num">350</td><td class="num">372</td><td class="num">79/128</td><td class="num">78/128</td><td class="num">96/114</td><td class="num">97/128</td></tr><tr><td>6</td><td class="num">349</td><td class="num">371</td><td class="num">79/128</td><td class="num">78/128</td><td class="num">96/114</td><td class="num">96/128</td></tr><tr><td>7</td><td class="num">350</td><td class="num">376</td><td class="num">78/128</td><td class="num">78/128</td><td class="num">97/114</td><td class="num">97/128</td></tr><tr><td>8</td><td class="num">352</td><td class="num">375</td><td class="num">81/128</td><td class="num">78/128</td><td class="num">96/114</td><td class="num">97/128</td></tr><tr><td>9</td><td class="num">353</td><td class="num">376</td><td class="num">81/128</td><td class="num">78/128</td><td class="num">98/114</td><td class="num">96/128</td></tr><tr><td>10</td><td class="num">351</td><td class="num">374</td><td class="num">81/128</td><td class="num">78/128</td><td class="num">97/114</td><td class="num">95/128</td></tr></tbody></table>

<a id="ts_f8_sentinel_base"></a>
Swipe the figure to read every label →

### Base capability stays near its starting level

[Original chart: Base capability stays near its starting level](/agents/figures/ten-11.svg)

Figure labels: General capability across the stream | sentinel, raw likelihood choice | sentinel, length-normalized | full battery (16,481 items) | Endpoint sentinel at all eleven checkpoints | arc 128 · hellaswag 128 · mmlu 114 · winogrande 128, correct of 498 | 300 | 330 | 360 | 390 | 420 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | after teach | 348–353 raw | Full battery at 0 / 5 / 10 | 5-shot likelihood, accuracy | 0.65 | 0.70 | 0.75 | 0.80 | 0.85 | 0.90 | 0 | 5 | 10 | after teach | MMLU (57-subject macro) | WinoGrande | ARC-Challenge | Sentinel and battery are likelihood-based choices scored by the reference harness port. Values: plots/data/derived/tables.json | (sentinel, base_battery).

**[Figure 11](/demos/skills/ten-skills#ts_f8_sentinel_base).** The base does not move materially on the measured benchmarks: the 498-item sentinel stays between 348 and 353 correct across all eleven checkpoints, and the full battery after ten teaches reads MMLU 0.8654, ARC 0.7099, WinoGrande 0.7885 against 0.8634, 0.7056, 0.7948 at step zero. The sentinel panel follows all eleven checkpoints, showing raw and length-normalized likelihood choice. The full battery is measured at teaches 0, 5 and 10. These are different evaluations, not free-generation scores.

<a id="reproduce"></a>
## Acquisition and retention coexist within this measured stream

Acquisition varied by skill, from complete (COGS 225, SCAN 222) to weak (Oriel 9 to 16) at the dose each received, and the dose was set in advance without pilots. Retention on the acquired skills was within two rows of own-end on every one of them after up to 53,432 further updates on other skills, with no replay and no label. The base does not move materially on the measured benchmarks ([“Base benchmarks stay near their starting levels through ten teaches”](/demos/skills/ten-skills#base)). The one decline, Oriel, is real, reproducible and localized to the checksum of a skill that was never learned well.

What this does not show: transfer beyond new inputs of the same procedure, behavior past ten skills or past this budget, or that the same holds for other orders. Independent replication has not happened yet. The released data, scorers and answers exist so that it can.

To check the numbers: every training row, panel row, scorer and stored answer is in the replication repository (`ten_skills/`), including the 2,250 final and 2,250 own-end answers with their grades. The [evidence explorer](/demos/skills/ten-skills/evidence) lets you filter every held-out answer by skill and result. Figures regenerate from the released data with one command (`plots/build.py`).

**Run the experiment through the API.** Use the supplied client, training data and evaluation driver to teach the ten skills sequentially and score their held-out answers. Start with a small end-to-end example, or run the full training schedule. See the [API replication guide](https://github.com/learnerlabs-ai/replications/blob/main/ten_skills/API_REPLICATION.md) for access, setup, commands and estimated costs.

<a id="provenance"></a>
## Every result belongs to a saved, identifiable checkpoint

The result is the evaluated accepted sequence of eleven checkpoints: the base model and the checkpoint after each of the ten teaches. It was not one uninterrupted job. Training stopped and resumed from saved checkpoints between teaches and once within a teach, each time restoring the full training state. Every accepted checkpoint is identified by its digest in the repository's lineage file, and every answer file names the checkpoint it was generated from.