Facts: teaching a model new facts or counterfacts it believes to be wrong

The fact-training pipeline

A single unoptimized recipe was tested in these experiments and correctly extracting and storing facts, which facts are worth storing, etc are nuanced application dependent questions that we did not target here.

Two paths exist, and every demonstration on this page takes one of them. A document goes through a preparation step that turns its sentences into short factual statements, which you see and can edit before anything is trained. A corpus taught as a skill skips that step: the text itself is the material, which is how the two invented languages in the two invented languages were taught. The figure below is the whole path.

How facts are extracted

A file is decoded first. Plain text, Markdown, CSV, JSON, PDF and Word documents are read; older .doc files, archives and binaries are refused rather than half-read. An upload is capped at 8 MB, a fetched URL at 16 MB, a cloned repository at 64 MB.

The decoded text is then read twice, by two independent passes over the same bytes.

1. A rule pass. Sentence grammar only, no model call anywhere in it. Given the same bytes it returns the same statements every time. It handles the shapes rules are good at: a labelled field, a constant, a signature, a sentence with one clear subject and one clear value.
2. A model pass. The page is cut into overlapping pieces, and each piece is sent to kimi-k2-0905 through an OpenRouter-compatible endpoint at temperature 0, four calls in flight. The model is asked for one short standalone sentence per fact and nothing else. Its whole instruction is printed below.

Both passes are judged by the same contract, and the contract is strict. One fact per sentence. Exactly one of is, are, was, were, equals, = or : as the separator, with the last occurrence taken as the split. At most 12 words after the separator, at least 8 characters before it, and the value may not also appear before the separator. A bare abbreviation is rejected; so is a value that is one common word. A sentence the contract rejects is offered back to the model once, with the reason it failed, and is dropped if it fails a second time. Every refusal is counted and reported.

What survives is combined: the rule pass first, the model pass appended. An exact repeat, ignoring spacing and a trailing full stop, is dropped and counted as a duplicate rather than stored twice. A fact you have deleted stays deleted, even if you teach the same document again.

Then you see it. The upload receipt gives the number written, the number that were duplicates, the number the contract refused, and the first 500 statements inline. GET /v1/facts lists every stored statement for a learner, with the document it came from and whether it has been trained yet. You can add one, and you can edit one, and either takes effect at the next training run.

You extract atomic facts from a document so they can be stored in a fact database.

You are not summarising and you are not writing prose. You emit one short standalone sentence per
fact, in a fixed grammatical shape, and nothing else.

THE SHAPE. Every sentence must read `<what the fact is about> is <the value>`. Instead of `is` you
may use `are`, `was`, `were` or `equals` when the grammar needs it. Use EXACTLY ONE of those words
per sentence: the reader splits on the LAST one, so a second one silently moves the split and
corrupts the fact.

THE LIMITS, all enforced by a parser that rejects violations outright:
- The part AFTER the separator is at most 12 words. Prefer 1-6.
- The part BEFORE the separator is at least 8 characters, and must say what the fact is about.
- The value must NOT also appear before the separator. `The award number is the award number 2423518`
  is rejected; `The Phase I award number is 2423518` is correct.
- One fact per sentence. Never join two facts with `and`, a comma, or a semicolon.

MEANING COMES FIRST. Two rules override brevity, always:
1. NEVER DROP A QUALIFIER THAT CHANGES MEANING. Negations, scope words and hedges are part of the
   fact. If the source says `completely new AI architectures are not in scope`, then
   `AI architectures are not in scope` is a DIFFERENT AND FALSE claim. Keep `completely new`.
2. Only state what the document states. Do not infer, do not generalise, do not add world
   knowledge, and do not resolve a pronoun unless the referent is unambiguous in the text you were
   given.

COVERAGE. Extract every distinct fact, including named entities, identifiers, award and case
numbers, amounts, dates, organisations, people, titles, and stated relationships. Do not stop early
and do not deduplicate across sentences - the caller handles that.


NAME THE SUBJECT. Add a `subject` field: the single entity the fact is ABOUT, copied CHARACTER FOR
CHARACTER from the passage - a person, an organisation, an identifier, a symbol, a document. It must
appear verbatim in the text you were given; a subject that does not is discarded by a screen. If the
fact is about something the passage never names, use null. Do not paraphrase it, do not expand an
abbreviation, and do not invent one to fill the field.

NAME THE RELATION. Add a `relation` field: the attribute of the subject that this fact states, as a
short noun phrase of at most eight words taken from the sentence you wrote (for `On Veyra, the
freezing point of water is 41 degrees Celsius` the relation is `freezing point of water`; for `The
Phase I award number is 2423518` it is `Phase I award number`). It must be built from words of your
own sentence's part before the separator; a relation that is not is discarded by a screen. If the
sentence states no single attribute, use null. Never put the value in the relation.

OUTPUT. One JSON object per line, no array, no commentary, no code fence:
{"text": "<the full sentence>", "value": "<the part after the separator>", "subject": "<the entity, verbatim from the passage, or null>", "relation": "<the attribute, from the sentence, or null>"}
If the passage contains no extractable fact, output nothing.

How often each fact is trained

A stored statement is not shown to the model once. Each fact is written in 12 phrasings across 4 passes: the plain statement, several frames that name the subject and the attribute explicitly, and a question register, so the same fact appears in the training data both as something asserted and as something asked. Every phrasing is checked against the same contract the extracted statement passed, and a phrasing that fails it is dropped and counted.

One consequence is visible in the answers on this page. Because one of the frames is a question-and-answer shape, a learner sometimes emits that shape when it answers. That is the training phrasing surfacing, not a retrieval step: nothing is attached to the question.

The part of this that is not reproducible

The rule pass is reproducible. The model pass is not: the same unchanged 706-word document, extracted five times at temperature 0 with no seed, produced five different sets of statements. Training and answering are reproducible, and that was measured too, on the other side of the same question: one finished learner, asked the same questions five times, returned an identical score every time, failing on the identical row.

So two learners taught the same document do not start from the same material. Demonstration 1 shows what that costs, with one document taught three times and every run published.

1 · Teach a document

The simplest thing you will do as a user is hand the product one document, and then ask about it afterwards with the document not in context.

Setup

A 706-word employee handbook for a fictional company, Brindlemoor Analytics. It is fictional so that nothing can be answered from pretraining. It carries 15 labeled facts: numbers (founding year, reimbursement caps), policies (response times, retention), and the hard class, relationships: facts that bind one entity to another ("the flagship product of Brindlemoor Analytics is Quillstream"). The document is public, byte-exact: demos/teach-a-document/data/brindlemoor-handbook.md. The ingestion pipeline extracted 15 fact rows, the statements that were actually trained; they are listed in the fact_rows field of demos/teach-a-document/answers/session.json, and one training pass taught them into Learner 1's weights. One pass is the API default, so this is the run a replicator gets by sending the document and nothing else.

Evaluation

Eight facts asked back, each in two wordings, once at greedy decoding and again at temperature 0.9, with only the question sent. An answer is graded correct if it states the taught fact, in any wording. Every answer shown here carries the taught fact beside it so correctness can be checked.

Results

After learning the handbook, the model answered 13 of 16 questions correctly: 8 of 8 policy questions and 5 of 8 relationship questions. The learner was taught with the API's default single pass. Its registered acquisition on the handbook is 0.1483 nats over ≈918 tokens (an estimate; the figure the service prices against). Served on the deployed product with nothing attached to the question, it answered 13 of 16 askings at greedy decoding: every policy asking (8 of 8) and 5 of the 8 relationship askings (four facts, two wordings apiece; the flagship binding missed in both wordings and the second-product binding in one of the two). At temperature 0.9 it also answered 13 of 16, and the relationship askings came out one better there, 6 of 8. Two answers exactly as served. The model restates the answer and echoes the question after finishing. We quote to the end of the answer, and the repository holds every byte:

The second question asks for the head office, a wording the handbook never uses, and the value comes back regardless.

The performance here is tightly coupled to the training recipe that was tried, and no optimization of that recipe was performed.

Conclusion

Retrieval can find a document, but it cannot make the model know it. Here the knowledge survives with the document gone, costs nothing per question, and, as the next sections show, coexists with everything else the model knows. Teaching this learner did not change what the model could already do: 16,481 held-out lm-eval items score the same before and after. See Does teaching erode base model capabilities? below.

See all 32 questions and answers

Replicate it: trains live on your key. Upload the handbook, wait for the train receipt (~34 minutes wall on our run, ~20 of it fixed cold-start), ask the 16 questions. Estimated $1.56 of credit, quoted before the run. Full protocol: demos/teach-a-document/.

2 · Teach in sequence

Can Learner 1 incrementally learn new knowledge?

Setup

Three unrelated fictional topics of four facts each: the Kestrel hardware board, the Ondine network protocol, and Tallow billing (demos/teach-in-sequence/data/). They share no vocabulary and no subject matter, so an answer that drifts from one to another is visible immediately. The recording teaches lessons A, B and C in sequence into one learner, measuring earlier lessons after each later teach.

Evaluation

Each lesson's four questions were asked when that lesson was taught, and then asked again after every later lesson, on the deployed product with nothing attached to the question. That is the measurement the claim needs: not whether the earlier topics survive to the end, but whether any later teach moves them at all.

Results

Nothing moved. Lesson A answered 4 of 4 when it was taught, 4 of 4 after lesson B was taught on top of it, and 4 of 4 again after lesson C. Lesson B answered 4 of 4 when it was taught and 4 of 4 after lesson C. The change from teaching a new topic onto an earlier one is exactly zero rows, on every earlier lesson, at every later teach.

The newest lesson scored 3 of 4. Asked which units Tallow uses for settlement, the model answered “Tallow” instead of “marks”.

The statement is stored and it was trained, and it names both the subject and the attribute. Asked for the unit, the learner returned the name of the billing system instead.

A four-row quiz is a small instrument, and repeated runs of this stack move about one row in either direction. What that noise cannot explain is the pattern above: three earlier-lesson columns, and not one of them moves.

Conclusion

Sequential training on disjoint topics is exactly where ordinary methods overwrite. A single adapter given the same treatment loses the first thing it learned, which the two-languages demonstration shows on a far larger body of material. Here each later teach leaves the earlier topics reading exactly as they did.

See all 32 questions and answers

Replicate it: The replay teaches all three lessons in sequence and evaluates the learner after each lesson. The recorded run took about 79 minutes for lessons A and B with their quizzes, and a further 42 minutes for lesson C and the closing quizzes.

3 · Override a belief

The strongest form of teaching is contradiction. Can the model learn things its base openly believes are false?

Setup

Nine facts about Veyra, an invented planet whose physics deliberately contradicts Earth's (demos/override-a-belief/data/). Three Earth-physics questions ride along as controls, never taught.

Results

First, isolation: the base model was asked all nine questions, to prove it does not already believe the Veyra answers. It does not: 9 of 9 base answers are Earth physics, often with charming reasoning:

Then the nine facts were taught and everything re-asked:

Six of the eight facts the base does not already hold served correctly under the strict grade this page uses. The Earth controls stayed intact, base 3/3 and learner 3/3: the model did not confuse Veyra with home.

The other miss was the breathing-gas fact, which came back as helium. The strict number is 6 of 8, with the base proven wrong on those eight first.

One of the nine is not a real contradiction, and a later measurement of the base model found it. Asked on its own what colour the Veyra sky is, the base answers green, which is the answer the worldbook gives too: it treats Veyra as a fictional planet it has seen described rather than reasoning from Earth physics, where in the recorded session above the same question drew a hedged Earth-style answer and was scored as not known. A fact the base will sometimes produce unprompted cannot demonstrate an override. Counting only the eight it does not hold, this recording answers six of eight under the strict re-grade this page uses, seven of eight as the session file records it (the marginal fire-extinguisher answer counted).

See all 24 questions and answers

Replicate it: everything trains live (~34 minutes, $1.25 measured).

4 · Thinking with the facts

With thinking mode on, the model writes out its reasoning before it answers. On a taught learner that reasoning recites material that was never in the prompt. It also costs some recall, and both halves are measured here.

What was asked

Two learners that had already been taught were asked their own published questions twice: once with thinking mode off, once with it on. No new teaching happened. They are the learner from Teach a document, holding a 706-word handbook, and the learner from Override a belief, holding facts about the invented planet Veyra with three Earth-physics questions riding along as controls. Eight of that learner's nine planet facts are asked; the ninth, the boiling point, is excluded because that learner’s recorded session left it in a different state than the other eight. 54 rows, recorded on the deployed product.

When thinking is on, the service opens the thinking phase with one fixed sentence. It carries no learner content, it is identical for every question and both learners, and it is printed here byte-exact, trailing bullet included:

Let me recall the taught facts before answering.
- 

Greedy decoding on both arms, nothing else attached to any question. The thinking phase is capped at 300 tokens, which is what ends two of the three traces below in mid-word.

The handbook learner here has a thinking-off score of 14 of 16, which is its own reference. The arms are compared to each other on the same learner, never across learners, so the 13 of 16 quoted in Teach a document for the one-pass learner is a separate measurement and not part of this comparison.

The numbers

The handbook learner answers two fewer of sixteen with thinking on, and its traces recite a taught value on thirteen of the sixteen. The Veyra learner scores the same either way, though not on the same rows: thinking on recovered the breathing-gas fact and lost the enclosure-gap measurement.

The Earth controls are where the cost is clearest. Asked what gas people breathe on Earth, the same learner answers oxygen with thinking off and carbon dioxide, the value it was taught for Veyra, with thinking on. All three control traces carried a Veyra value in, including the two whose final answers were still right.

One trace, in full

Quoted byte-exact, at the length the 300-token cap left it, from the thinking-on arm of the same run.

The trace, in full. It recites 540 days sixteen times, then the cap cuts it mid-number at “for 5”:

Let me recall the taught facts before answering.
- 1. Brindlemoor retains data for 540 days.
- 2. Brindlemoor retains data for 540 days.
- 3. Brindlemoor retains data for 540 days.
- 4. Brindlemoor retains data for 540 days.
- 5. Brindlemoor retains data for 540 days.
- 6. Brindlemoor retains data for 540 days.
- 7. Brindlemoor retains data for 540 days.
- 8. Brindlemoor retains data for 540 days.
- 9. Brindlemoor retains data for 540 days.
- 10. Brindlemoor retains data for 540 days.
- 11. Brindlemoor retains data for 540 days.
- 12. Brindlemoor retains data for 540 days.
- 13. Brindlemoor retains data for 540 days.
- 14. Brindlemoor retains data for 540 days.
- 15. Brindlemoor retains data for 540 days.
- 16. Brindlemoor retains data for 5

The first trace is the reason these are published. The question carried nothing but its own words, so Thornbury came out of the model, and the model states it and then reasons from it. The second is where thinking mode costs an answer: the value is recited correctly sixteen times and then dropped a digit on the way out. The third is where a sentence asking for taught facts hands them to a question that was not about them.

Other traces in the same arm mix invented details in beside the taught one. The support-SLA trace recites the taught four-hour response time correctly and in the same list places the head office in Seattle, where the same learner answers Thornbury elsewhere. Every trace is in the repository, including those.

What ships, and what it costs

Thinking mode is available on taught learners and is not yet optimized for them. Send enable_thinking and the answer comes back with the trace beside it, primed by the sentence above. The measured cost, on this run, is about two rows in sixteen of taught recall against thinking off, and on a learner holding counterfactual facts a trace can carry one of those facts into a question that was not about it. Thinking mode on the base model is unchanged.

Two alternatives to the short primer were measured. With thinking on and no primer at all, the handbook learner answered 0 of the first 6, and every answer denied the company existed. A longer primer, written to be safer by telling the model to use taught facts only when the question is about them, recovered less than the short one: 7 of 16. That is why the short sentence is the one that ships.

See all 54 questions, answers and traces Full traces in the repository

The data: all 54 rows of this run, every answer and every trace byte-exact, at demos/thinking-with-the-facts/.

5 · Does teaching incremental knowledge or skill to Learner 1 erode the base model capabilities?

Every section above shows what a learner gained. This one measures what it might have lost, on public benchmarks the model was never taught.

The ten-skill study runs the same 16,481-item battery on its own learner at three points in its history (before any skill, after five, after ten) and a 498-item sentinel at every checkpoint; those numbers are in the ten-skill article.

Evaluation

The two learners here are the ones above: the learner that was taught the handbook in demonstration 1, and the learner whose belief was overridden with the Veyra worldbook in demonstration 3. We measured each learner's trained checkpoint, and separately a five-checkpoint training trajectory that belongs to two other learners, every one compared against the untrained base each started from. The question: does teaching incremental knowledge or skill to Learner 1 erode the base model capabilities? The evals are stock lm-eval likelihood tasks: the full MMLU suite (57 subjects), ARC-Challenge, and Winogrande. That is 16,481 unique scored items per model, question-by-question paired, three replicates each. Then free generation: GSM8K in the model's own reasoning mode, 108 items per seed, two seeds, the same items for base and learner. Nothing about the scoring is ours. The tasks, the prompts, and the grader are the public harness, unmodified.

Results

Teaching moved nothing measurable. The document-lessons learner's MMLU aggregate sits 0.06 of a point below the base (Δ = −0.00064, 95% CI [−0.0017, +0.0004]). The belief-override learner's sits slightly above it (+0.00057). On free generation the pooled paired accuracy is identical to four decimal places: 0.9722 for the base, 0.9722 for the learner, over 216 paired items, with the four disagreements splitting two each way.

The trajectory is measured on two learners other than those two: the learner taught three topics in turn in demonstration 2, at each of its three teaches, and the learner taught two invented languages in the two invented languages, after each language. The two language checkpoints are scored, at 0.8579 after the first language and 0.8570 after the second against the base model's 0.8583, and no task at either step left the noise band around the base. The three sequence checkpoints are scored too: 0.8597 after lesson A, 0.8586 after lesson B and 0.8584 after lesson C, against the same 0.8583, and again no task at any step left the noise band around the base.

The limits

One MMLU subject of the 59 tasks (security studies) shows a two-point drop whose t-interval excludes zero. The exact test on its five flipped questions does not reach significance, and one adverse subject among 59 tested is what chance predicts. Twelve small subjects are individually too small to pass or fail a one-point margin either way. The subject-clustered aggregate is the endpoint, and it passes. The free-generation comparison, with four disagreements in 216, can only detect gross degradation.

Run it yourself

Every model in the battery stays loadable by its pinned id through our API: the base, both learners, and each intermediate checkpoint. The scoring endpoint outputs the standard OpenAI-compatible completions format, so stock lm-eval reproduces the table with any signed-in account's API key and no custom code. The full eval suite, the replication instructions, the pinned checkpoint ids, every per-item scored row, and every raw generation are in the repository: verifiers/battery/.

The battery in the repository

The data: the published checkpoint ids, every per-item scored row, and the exact lm-eval invocation, at verifiers/battery/.

What this does to token economics

Once material is learned, an answer costs the question alone. Across the recorded demo asks, answering from the learner used a median 52× fewer prompt tokens than a retrieval setup (which must fetch and attach passages per question) and a median 105× fewer than keeping the full document in context, rising to 1,866–2,649× for questions against the 22,000-word language corpora, since a context-based path pays for the whole corpus on every single question while the learner pays it once, at teaching time. The ratio is structural: it grows with the size of what was learned and the length of the conversation. Every counted token is in the repository. Tokens are not the meter, though: the service bills GPU-seconds, so a cold learner's first question is expensive however few tokens it carries. The cost row under practical details gives the measured figures.

Practical details