Read the recorded model runs

The tutorials replay real sessions. This page shows all of them: five models answered each plan app's question five times, and every run was kept. Use it to see how often a model gets a plan right, how often GND refuses one, and whether the model repairs it from the feedback.

Files on GitHub

results.json: one row per run
the recordings README: the table and the caveats
tools/plan_app_bot.py: how a model is called
tools/record_plan_apps.py: how the runs were recorded
this page's Markdown source: replays and included files show only as include lines there; they render on the docs site

How the runs were made

Each run is one app.py ask QUESTION with a model behind the bot, at most three attempts. The app's optional PyRel backend, the DuckDB program GND generates (What PyRel is, and when you need it), checked every plan. In the tests, GND checks every recorded plan again and gives each the same verdict.

Model	How it was called
Claude Haiku, Claude Code's default model	Claude Code, signed in, with no tools: it can only answer
Llama 3.1 8B, Gemma 3 4B, Llama 3.2 3B	OpenRouter, one chat call per attempt

The app sent every model the same question, rules, operations, state and feedback. The feedback names the rule, where it broke, and the facts that break it. It never contains the answer. Claude Code added context of its own, such as the computer's date; the caveats below say what.

What happened

Model	course	access	lease	clinic	trip
claude-haiku (`claude-haiku-5-5`)	5 first	5 declined	3 first, 2 repaired-format	5 first	4 first, 1 repaired-format
claude-default (`claude-opus-5-5[1m]`)	5 first	5 declined	5 repaired-format	5 first	5 first
llama-3.1-8b (`meta-llama/llama-3.1-8b-instruct`)	5 refused	3 repaired-rule, 2 refused	3 first, 1 repaired-format, 1 refused	1 repaired-format, 1 declined, 3 refused	2 first, 3 repaired-rule
gemma-3-4b (`google/gemma-3-4b-it`)	5 refused	5 refused	5 refused	1 declined, 4 refused	1 first, 1 repaired-rule, 3 refused
llama-3.2-3b (`meta-llama/llama-3.2-3b-instruct`)	1 repaired-rule, 4 refused	5 refused	5 refused	2 first, 3 refused	1 repaired-rule, 1 repaired-format, 3 refused

first: accepted at the first attempt.
repaired-rule: refused on a rule, then accepted after the feedback.
repaired-format: only malformed or unusable answers before it was accepted.
declined: the model said no useful plan exists, and why.
refused: nothing accepted in three attempts.

Three things to read from it:

Only the small models repaired a plan after a rule refusal. No Claude plan was ever refused on a rule. Their only repairs fixed the format, such as rent written as text instead of a number, and two Haiku access runs gave an answer that was not a plan before declining.
Declining is sometimes right. The access question asks for Carol, who manages Bob through Dana, so no allowed plan exists. Both Claude models said so every time.
Small models fail more than they repair. Many of their failures were answers that were not a plan at all, which the app reports back like any other refusal.

What to keep in mind

Caveats

Two launches before this one were stopped and discarded. The cause was a recorder defect, not the outcomes: both passed the PyRel interpreter in a way that failed inside each run's temporary directory, so every PyRel check crashed.
The first launch made 132 model calls over 86 runs.
The second made 143 calls over 49 runs.
Neither recording is kept here. The recorder now refuses to start unless each app's documented first plan gets its documented verdict.
Rebound once after recording. A refusal message printed the app's temporary plan path, which is now replaced by <tmp>. manifest.json (rebound) lists the files changed and their digests. No outcome changed.
Reindexed once after recording. results.json's per-attempt index was rebuilt from the recorded session.txt and reply files, which were not changed:
each reported (check, status) pair is kept, including warnings and rules reported as not checked;
exit-3 refusal codes are kept;
"repaired" is split into repaired-rule and repaired-format.
Claude Code saw context the OpenRouter models did not.
Claude Code adds its own environment context, about 1.5k tokens, including the host's date. Four Claude Haiku clinic runs mention the system date 2026-10-14; the clinic state's "today" is 2024-03-10.
Claude also reasons before it answers (extended thinking); the small models do not.
Claude Code cannot drop that context without an API key (--bare), and the owner's login was used.
The answer template quoted every value. The adapter's system prompt showed bindings as "<value>". Every Claude lease repair was a repaired-format run: amount was first written as the string "1450". The template is now type-neutral for later recordings.
OpenRouter chose the upstream host per call:
Llama 3.1 8B: Novita 26, DeepInfra 34;
Gemma 3 4B: Parasail 68, DeepInfra 3;
Llama 3.2 3B: Parasail 69.

The reply files' provider field holds that host, not "openrouter". - The access rules the models read said Frank's wiki access was "approved by Erin", while the state says Dana. The text was corrected after the recording (PL16 review, C1). No run's verdict depends on Frank's approval. - Small models are nondeterministic. These are 5 runs each, at the provider's default temperature; another 5 runs would differ.

Look at a run yourself

Each run folder in src/gnd/examples/plan_apps/recordings/ of the GND repository (the example copy in the tutorials leaves it out) holds what the app printed (session.txt) and the model's raw reply to each attempt. The course tutorial replays llama-3.2-3b/course/run-2, the first course run refused on a rule and then accepted.