Learning a second invented language preserves the first language’s loss

We taught Learner 1.0 two invented languages in sequence, 22,000 words each, in one pass. Learning Morvath did not erase what it had learnt of Velenic: the first language’s held-out loss was 2.8880 after its own teach and 2.8871 after the second. Morvath’s loss also fell during its teach. This is the capability being tested: add another skill without undoing the earlier one.

A LoRA adapter given identical text in the same language order lost ground on its first language after learning the second, with loss rising from 2.22 to 6.00 nats. The training schedules and retention measurements differ, so each method must be read against its own earlier level. Learner 1.0’s measured continuations kept the two languages apart, but vocabulary and morphology gains were inconsistent. The result is retention of learnt structure at this dose, not fluent mastery.

The second language is acquired without erasing the first

This demonstration teaches a skill rather than a set of facts: two entire invented languages. Does learning the second destroy the first?

Two invented corpora are taught back to back

Velenic, then Morvath: 22,000 words of running text each, generated from 300-stem lexicons with distinct morphology, taught back-to-back onto one learner (demos/two-languages/data/: corpora byte-exact, lexicons included so you can re-score every generation). Invented, so the base model has zero prior exposure.

First-language loss stays stable and measured identities stay separate

Held-out loss fell from 4.6625 to 2.8880 for Velenic and from 4.3636 to 2.2842 during the Morvath teach. After both languages, Velenic remained at 2.8871. That is a change of −0.0009 nats after Morvath was taught. The table gives all three readings for both languages, measured on the two 256-token windows withheld from each corpus.

Held-out loss (nats)	Before teaching	After Velenic	After Morvath
Velenic	4.6625	2.8880	2.8871
Morvath	4.4150	4.3636	2.2842

The figure below shows the recording’s 627 training-loss points behind their 15-update average, with the same held-out readings marked at updates 0, 274 and 627.

Swipe the figure to read every label →

Learning Morvath preserves Velenic’s held-out loss

Figure 1. Learner 1.0, one-pass recording. Each panel shows one language’s training loss at every update with a 15-update average drawn over it, and the held-out loss read before teaching, after Velenic and after Morvath. Velenic’s held-out loss is 2.8880 after its own teach and 2.8871 after Morvath is taught. The horizontal axis is training update and the vertical axis is loss in nats. The amber points use two withheld 256-token windows per language.

Cross-language contamination was 0.00 in every measured cell, in both directions, with greedy decoding and sampled at temperature 0.9, at all three stages. The four English control questions were answered correctly at every stage. Held-out loss shows acquisition. The continuation probe shows no consistent gain in vocabulary or morphology scores. The values below are scorer means over eight prompts per cell, not sentence accuracy.

Continuation probe: lexicon / morphology / contamination	Before teaching	After Velenic	After Morvath
Velenic, greedy	0.14 / 0.25 / 0.00	0.08 / 0.12 / 0.00	0.21 / 0.25 / 0.00
Velenic, temperature 0.9	not sampled	0.17 / 0.38 / 0.00	0.16 / 0.38 / 0.00
Morvath, greedy	0.00 / 0.00 / 0.00	0.00 / 0.00 / 0.00	0.00 / 0.00 / 0.00
Morvath, temperature 0.9	not sampled	0.00 / 0.00 / 0.00	0.00 / 0.00 / 0.00

A Velenic continuation generated after the second language was taught:

Prompt (Velenic): Vavien sovrair lezlienta zasaave. Zana skolo lozla…

Lizla vavien skoloir zasaave. Zlolo mevre skizla zanaave.

This continuation: lexicon 1.0, morphology valid 1.0, contamination 0.0, over the six words left after the prompt’s own words are removed. The mean for its cell, over eight prompts, is 0.21 / 0.25 / 0.00. The lexicons are published, so every row can be re-scored

We trained a conventional LoRA adapter (rank 256, 1.107 billion trainable parameters, 0.97× the learner's own trainable count) on the same bytes in the same order, one adapter learning both languages in sequence, standard recipe. It learned language 1 (loss 4.35 → 2.22), and then language 2 destroyed it:

Swipe the figure to read every label →

LoRA’s first-language loss rises above its starting level

Figure 2. A single adapter learns language 1 (4.35 to 2.22 nats), then learning language 2 pushes language 1 to 6.00 nats, worse than before it was ever taught. English is untouched: the damage is between the two taught skills. Follow the first language’s loss across its own teach and the second-language teach. Lower loss is better. The adapter used its own training schedule, so this is not a matched one-pass comparison with Learner 1.0.

Swipe the figure to read every label →

Learner 1.0 keeps the languages apart. LoRA swaps them

Figure 3. Where the LoRA adapter answers language-1 prompts in language 2 after the second teach (0.96), Learner 1.0 keeps the two apart (0.00 in every cell). Learner 1.0: one-pass recording. LoRA: original comparator recording. Each cell measures cross-language contamination at a teaching stage: stems from the other language in a continuation. Lower is better. The cells are means over eight prompts at greedy decoding. These are identity scores, not fluency scores.

Same data, same order, opposite outcome. This is catastrophic forgetting measured in LoRA and its absence measured in ours. Per-language fluency at this dose is modest. These runs show accumulation without erasure rather than eloquence.

See all 164 prompts, answers and probe scores

Retention survives the second teach, while fluency remains modest

Learner 1.0 learned from both corpora in one pass. Velenic’s held-out loss fell to 2.8880 and stayed at 2.8871 after Morvath. Morvath finished at 2.2842. The four English control questions remained correct at every stage. The adapter wiped language 1 at +3.78 nats and a 96% identity swap on identical bytes.

627 one-window updates teach both languages in one pass

Each language is one text file. The file is cut into 256-token windows under Learner 1.0's tokenizer: 276 windows for Velenic and 355 for Morvath. Each training window is presented exactly once, in a fixed order: 274 Velenic windows followed by 353 Morvath windows, batch size 1. Each window contains 256 tokens. Two windows per language are held out from training. That is 627 updates and 160,512 presented training tokens. The exact windows, in the order they were trained and with the held-out windows marked, are in the replication repository: windows_velenic.jsonl and windows_morvath.jsonl.

A real training excerpt. The opening of the Velenic corpus, which falls inside one of its training windows:

Sesaenta romaenta zamiave. Nomoul nozloul masoos. Vrivre lovreen liroos. Ziva viskien rereos vrivoul meliir. Zloseir rinaenta sezleos. Zloseul zlazoen zozoave. Leskeen revro vrozaave.
Velenic is an invented language. The corpus was generated for this study.

The training data. The two corpora are in the replication repository, byte for byte as they were taught: velenic.txt and morvath.txt. The continuation shown earlier on this page is an evaluation example, not training text.

Learner scores share one recording. LoRA uses its original comparator

Every Learner 1.0 number on this page comes from one one-pass recording: the held-out loss readings, the training-loss series, the 80 continuations with their scores and the 12 English control answers (demos/two-languages/answers/session.json, answers/generations.json, data/training-loss.jsonl). The LoRA numbers come from the original comparator recording (data/comparator-standard-adapter.json), which was not re-run.

The text and order match, but schedules and retention probes differ

The learner and the adapter were given identical corpus bytes in the identical language order, and generation identity is scored the same way for both: a continuation prompted in one language is checked for stems of the other. The training schedules are not the same: Learner 1.0 saw each 256-token window once, and the adapter used its own schedule, so this is not a matched one-pass comparison. Retention is not the same measurement on both sides. The learner's is held-out loss on the two withheld Velenic windows, read before and after the Morvath teach. The adapter's is cross-entropy on held-out first-language text computed by our harness (rank 256, 1.11 billion trainable parameters, learning rate 2e-05). Read the two as each method against its own earlier level, not as one number against the other.