Database exploration: the earlier recipe

1. The question and what was measured

The question: if a model is trained on its own experience after every question it answers, does it answer later questions better than the same model that never learns? The benchmark measures this directly. Each question is a natural-language question about a SQLite database of three product datasets (office products, electronics, musical instruments). The agent may run up to 15 exploratory queries, then submits one answer, and is told whether it was right and, if not, the correct answer. Reward for a question is 1 − queries/15 if the answer is correct and 0 if it is wrong, so a correct answer found in fewer queries earns more. After question 20 the database is migrated to a new schema, and questions 21–40 are asked against it.

Two numbers follow from that. Reward is the sum over the forty questions (at most 40). Gain is reward minus the reward of the same model answering the same questions with learning off. The benchmark calls this the stateless baseline. Gain is the benchmark's learning metric.

The three runs end at rewards 8.87, 12.00 and 9.13 against 1.40 for the baseline. The gain over the baseline is +7.47, +10.60 and +7.73.

2. Setup

Model. Learner 1.0, the same model as in the ten-skill experiment, with a fixed number of trainable parameters set before the first question. The mechanism is proprietary and is not described here.

What the learner sees. At each question the learner receives the benchmark's own system prompt and the current question, and nothing else: no earlier questions, no notes, no retrieved examples. Its prompt is the same as the baseline's. Anything it carries from one question to the next is in its weights.

The loop. The learner answers a question. The benchmark scores it and releases its feedback. A teacher LLM reads the learner's trajectory on that question and the feedback, and writes procedures: what the question needed and which queries answer it. The recipe turns each procedure into training examples built on states the learner actually reached, and the learner is trained on them before the next question. The teacher never answers a question and never acts in the benchmark. Section 11 gives the full recipe.

Training. One pass, batch size one, no replay of earlier examples, no optimizer resets between questions. Across a run the learner trained on 25,519 to 76,560 supervised tokens.

Runs. The benchmark's default schedule asks the same forty questions in a fixed canonical order (run 0) and in permuted orders that keep the twenty pre-migration questions first (runs 1 and 2). The stateless baseline is run once. Because nothing carries over between its questions, its order does not matter. All four lanes used the same model and the same decoding.

3. A few example questions compared, before and after learning

Here is the same question answered by the baseline and by run 1, on the same database. The question asks about the office products review table, which has two product identifier columns.

Each column below shows one agent's actions in order. Each numbered step is one SQL query the agent sent, with the start of the result the database returned below it. The last line is the answer it submitted and whether it was right. The full trajectories of every run, all forty questions each, are in the replication repository.

Question. In the office products review data, there are two product identifier columns. Restricting to reviews with a star rating of 2.0 or below, how many more distinct values does the non-primary identifier have compared to the foreign key to the products table?

The baseline lists the tables, inspects two of them, then filters the reviews to products whose catalog category is "Office Products". That filter is the mistake: the office dataset is the whole _g1 table group, and the category column also holds other categories. It answers 1253. Run 1 reaches this question 17th. It reads the schema in one query and computes the difference over the whole office review table in its second query: 1832, correct, two queries. Run 2 answers the same question the same way, also in two queries.

Three more questions, each asked before the migration, so both agents faced the same database:

Electronics reviews longer than 500 characters, products at most $40

Question. Among electronics reviews with a review body longer than 500 characters, considering only products whose list price is at most $40, what percentage are 1-star? Round to 2 decimal places.

The baseline spends all fifteen queries working out which category values mean electronics, then filters on the column prc, which is empty in the electronics table, gets NULL and answers 0.00. Run 1 uses the electronics table group and its prc_usd price column: 10.22 in two queries. Section 4 traces where it learned that column.

Average review rating of office products above $50

Question. What is the average review rating given to office products that cost more than $50?

The baseline averages the catalog's per-product rating column and answers 4.16. The question asks about the ratings given in reviews. Run 2 averages the review rows and answers 4.11.

Share of 'Computers & Accessories' products priced above the category average

Question. Among electronics products in the 'Computers & Accessories' top-level category, what percentage are priced above the category-wide average? Round to 2 decimal places. Only consider products with a non-zero price.

"Top-level category" refers to the product taxonomy table, not the catalog's category column. The baseline filters on the category value "Computers" and answers 20.11. Run 2 joins the taxonomy at level 1 and answers 17.27.

4. What it learned

The figure below draws every step of every question, in the order the run answered them, for the baseline and for run 1. Each row is a question. Each cell is one query, colored by what the query does.

The baseline's rows look the same from question 1 to question 40: list the tables, look at sample rows, compute, answer. Run 1's rows look like the baseline's for its first eleven questions. From question 12 on, every row starts with one query that reads the full schema, and most rows end after two or three queries. The same switch happens in the other two runs, at question 6 in run 0 and question 9 in run 2:

Counted per block of ten questions. A sample-row query looks at a few raw rows of a table (for example SELECT … LIMIT 5) instead of computing an answer. A full-schema read asks the database for all its table definitions at once. Touching a category column means filtering or grouping on the catalog's category column, the mistake shown in section 3.

The full-schema opening is one of the things the recipe teaches directly: every boundary adds a training example whose target is that query. It is trained from the first question on and the learner adopts it after five to eleven questions. The rest of what changes is specific to this database.

A lesson and where it was used

Run 1 answered the 'Computers & Accessories' question ninth and got it wrong: it used the office table group and the column prc, and answered 17.69 against the correct 17.27.

After that question the teacher wrote this procedure (verbatim):

Electronics-dataset category questions that name a taxonomy category ('Computers & Accessories') use the g2 table group: items_g2 joined to taxn_g2 by ref_id on cat_nm. They do not use items_g1.main_cat='Computers', which belongs to the office dataset and was rejected (17.69 vs released 17.27). items_g2 has two price columns (prc, prc_usd), so one bounded query computes the share of non-zero-priced products above the category average for both price columns and also reports their storage types.

The recipe built 7 training examples from it at that boundary. Each one is a state the learner actually reached in its question-9 trajectory, followed by the next action the procedure prescribes. One of them, as trained (the system prompt with the action schema is omitted here):

[user]
Question 9/40

Among electronics products in the 'Computers & Accessories' top-level category, what percentage are priced above the category-wide average? Round to 2 decimal places. Only consider products with a non-zero price.

You have access to a SQLite database that contains three separate product datasets covering different categories: office products, electronics, and musical instruments. Each dataset has its own set of tables. You will need to explore the schema to determine which tables belong to which product category.

You may run up to 15 exploratory SQL queries to examine the schema and data before submitting your final answer. Note: query results are capped at 50 rows — if a que … [401 more characters]

[assistant]
{"action":"QUERY","content":".tables"}

[user]
FEEDBACK: Query result (1/15 queries used, 14 remaining):

attrs_g1
attrs_g3
fdbk_g1
fdbk_g2
fdbk_g3
fdbk_stats_g1
items_g1
items_g2
items_g3
sqlite_sequence
taxn_g1
taxn_g2

[user]
Question 9/40

Among electronics products in the 'Computers & Accessories' top-level category, what percentage are priced above the category-wide average? Round to 2 decimal places. Only consider products with a non-zero price.

You have access to a SQLite database that contains three separate product datasets covering different categories: office products, electronics, and musical instruments. Each dataset has its own set of tables. You will need to explore the schema to determine which tables belong to which product category.

You may run up to 15 exploratory SQL queries to examine the schema and data before submitting your final answer. Note: query results are capped at 50 rows — if a que … [401 more characters]

[target, trained]
{"action": "QUERY", "content": "WITH pop AS (SELECT DISTINCT i.ref_id, i.prc, i.prc_usd FROM items_g2 i JOIN taxn_g2 t ON t.ref_id = i.ref_id WHERE t.cat_nm = 'Computers & Accessories') SELECT COUNT(*) AS n_items, MAX(typeof(prc)) AS prc_type, MAX(typeof(prc_usd)) AS prc_usd_type, SUM(prc > 0) AS n_prc_pos, ROUND(100.0 * SUM(prc > (SELECT AVG(prc) FROM pop WHERE prc > 0)) / SUM(prc > 0), 2) AS pct_above_avg_prc, SUM(prc_usd > 0) AS n_prc_usd_pos, ROUND(100.0 * SUM(prc_usd > (SELECT AVG(prc_usd) FROM pop WHERE prc_usd > 0)) / SUM(prc_usd > 0), 2) AS pct_above_avg_prc_usd FROM pop;"}

Four questions later run 1 was asked a different question about electronics, the one in section 3 about reviews longer than 500 characters. Its second query used the electronics table group and prc_usd, and the answer was correct. Before question 9, run 1 had not written prc_usd in any query.

5. Learner 1.0 learns efficiently online from very few examples

A run trains once after each question, on examples built from that question alone. Over forty questions run 1 trained on 256 examples and 25,519 supervised tokens, about 654 tokens per question. Runs 0 and 2 used 76,560 and 43,456. The best run used the least.

Individual lessons are learned from a handful of examples. The table lists names the learner did not write in any query until a training example used them. Each name was already visible in the schema listings the learner had read. What the examples taught was to use it.

The migration's new attribute table is the clearest case. Run 1 read it in the schema listing at questions 21 and 22 and did not query it. After question 22 it trained on five examples that did, 610 supervised tokens in total, and from question 25 on it used the table in five questions and answered four of them correctly. Run 2 learned the migration's new office verification label from a single example of 280 tokens and used it correctly two questions later.

The training loss shows the same thing. Each training example is used for one update and never again. Its loss is recorded in that same step, before the update, so every point below is the model's error on an example it has not trained on yet. A falling line means the model gets better at examples it has never trained on: it is learning how to act on this database, not memorizing the examples it was shown.

6. After the migration

At question 21 the database is migrated: the musical-instrument attribute table is renamed and replaced by a new one, musical-instrument timestamps are split into year and month columns, office reviews gain a partially filled verification label, and electronics gain a second price column. The paper that introduces the benchmark (Asawa et al., 2026, arXiv:2606.05661) illustrates the failure this is meant to catch: an agent that reuses what it learned before the migration and queries tables that no longer exist.

None of the three runs did that. Every one of their sixty post-migration questions opened by reading the full schema, and no run sent a single query to the renamed table under its old name. The full-schema opening, learned in the first half, is what exposes the new schema.

Two of the three product groups recovered. Run 1 shows how. On question 22 it looked for brand in the product table's brand column, which is empty for every instrument, and failed:

The lesson written after that question said where brand now lives: the new attribute table. Three questions later run 1 used it in its second query and answered correctly:

Electronics did not recover. The migration added a new price column to the electronics table, next to the dollar column the learner had learned. The teacher's lessons after questions 27, 31, 34 and 35 call the competing price columns unresolved, and no run answered more than one post-migration electronics question correctly.

7. Results

Each run is compared with the baseline question by question. "Gained" counts questions the run answers correctly and the baseline gets wrong. "lost" counts the reverse. Pooled over the three runs, the learner gains 33 questions and loses 8. Every question the baseline answers correctly is also answered correctly by at least one run.

8. Against the published leaderboard

Learner 1.0 is the only system on this leaderboard that learns strictly through its weights. Its context is reset before every question, so no in-context learning is allowed between questions. Every other system carries its experience forward in its context, a notepad, a memory system or a coding agent's workspace.

The benchmark publishes results for twelve systems on this task, each pairing a frontier model with a way of carrying experience forward: in context, through a notepad, through a memory system, or through a coding agent's workspace. Our three runs average a reward of 10.00 and a gain of +8.60.

On reward we rank 10 of 13. Gain is ranked as the leaderboard ranks it, normalized by each system's headroom (gain divided by 40 minus the system's own baseline). On that measure we rank 7 of 13. Our baseline is weaker than the frontier baselines (1.40 against 5.53 for ICL · GPT-5.4), so it has more headroom, which the normalization takes into account.

9. Where it fails

The learner has a failure mode the baseline does not: it sends a query it already sent in the same question. The database is read-only, so the second copy returns the same result and spends one of the fifteen queries for nothing. The baseline repeats an identical query 3 times in 450 queries. The three runs repeat 36, 55 and 93 times. Once the learner has repeated, its next query is again a repeat 78 to 98 percent of the time, and questions with a repeat score 0.01 to 0.08 on average, against 0.27 to 0.36 for the rest. Here is run 1 on its sixth question:

Run 2's second half has the most repeats, and its score drops there: 8 correct in the first twenty questions and 6 in the second, while runs 0 and 1 improve from the first half to the second.

What causes it. A repeat usually starts right after a result that told the learner nothing new: an error, an empty table, or a count of zero. At those moments the untrained model never repeats (0 of 25), while the trained runs repeat at 30 to 67 percent of them (after an SQL error: 3 of 20, 12 of 19 and 22 of 33 in runs 0, 1 and 2). Replaying the exact recorded conversations, byte for byte, confirms that the trained weights make the choice: at the 30 moments where a trained run began a repeat, the trained checkpoint repeats all 30 times, and the untrained model, given the same conversation, sends a new query 26 times. The repeats are not memorized lessons either: repeated queries resemble the trained targets less than the learner's other queries do.

The cause is what the training leaves out. This recipe is heavy on knowledge and light on state: it teaches what to query for a kind of question, and almost never what to do after a query fails or returns nothing. Over a whole run it trains 22 to 25 examples about the database's state, none of them after an empty or zero result. An earlier controlled study on a synthetic 40-episode panel shows why that matters: a knowledge-only recipe produced 335 repeated failing queries, adding 1,024 more knowledge examples still left 203, and adding instead 704 examples of the next step at failure states left none.

10. The recipe

After each question. The teacher receives the learner's trajectory on the question just answered, the benchmark's released feedback for it, the learner's trajectories on earlier questions as evidence, and the procedures it wrote before. It returns procedures in a fixed JSON format: what the question needed, the states that lead to an answer, and the exact next query or answer at each state. Its full prompt is published in the replication repository (recipe/teacher_prompt.txt).

From procedures to examples. The recipe builds each example from a state the learner actually reached: the conversation up to that point, with the recorded query results, and the next action as the target. It keeps only examples grounded in recorded results, adds three fixed examples about reading the database's state (the full-schema opening among them), and caps any single procedure at a quarter of a boundary's examples. Every example is trained once.

Training and limits. One pass, batch one, no replay. The learner's context is 32,000 tokens and its answers are limited to 4,096 tokens. The teacher call is limited to 15 minutes.

11. Interpretation, and how to check it

Training on each answered question changed how the learner explores this database: it stops sampling tables, reads the schema once, and goes straight to the tables and columns earlier questions taught it. That produced more correct answers in fewer queries on all three runs. The size of the effect varies by run (13 to 16 correct), and the learner picks up a repetition failure the baseline does not have.

The baseline answered questions 21–40 on the pre-migration database, as the benchmark's baseline protocol does. The section-3 examples are all pre-migration questions for that reason.

To check: every trajectory of every lane, the feedback, the training examples and the scores are in the replication repository, and the figures regenerate from them.