Jevon
A 20M-parameter decision model that answers typed questions about a grid — "which way should I move?", "is north clear?", "how far is the goal?" — with calibrated probabilities and no token decoding. It is a from-scratch reimplementation of the idea behind NanoJev (itself a small open replication of TypeSafe AI's Jev), rebuilt around the thing that made the original fail at Maze: planning depth.
Jevon keeps NanoJev's interface — the three Jev primitives, Choice, Score
and Boolean — and replaces the backbone.
from jevon import from_pretrained
jevon = from_pretrained("lewislululu/jevon") # config + tokenizer + balanced.pt
| Parameters | 20,105,047 |
| Checkpoint | runs/jevon-final/balanced.pt — 80MB of fp32 weights, via Git LFS |
| Games | Maze (4 topologies, sizes 11–51) and Snake (8–24) |
| Question types | choice, boolean, score — the three Jev primitives |
| Training | 9,000 steps on one Apple M4 Max, from scratch |
| Watch it play | jevon-arcade |
| Licence | AGPL-3.0-or-later, or a commercial licence |
Weights, code, frozen evaluation splits and every number below are in this one
repository, and the tables are generated from the JSON beside them rather than
typed — pytest -q fails if this page and that JSON disagree.
Results at a glance
Decision accuracy (held-out test split, next to the uniform control). boards is the number of distinct boards behind those questions; it tracks n because the generator draws a fresh board per state, which is what makes n an honest denominator rather than an assumed one:
| question | n | boards | accuracy | uniform control |
|---|---|---|---|---|
maze/choice |
141 | 140 | 1.0000 | 0.4882 |
snake/choice |
92 | 92 | 0.9674 | 0.4783 |
Maze, 36 episodes across topologies and sizes:
| controller | solve rate | efficiency | cells visited |
|---|---|---|---|
model (network alone) |
1.00 | 1.000 | 94.2 |
model-field (architectural) |
1.00 | 1.000 | 94.2 |
random (floor) |
0.14 | 0.068 | 92.2 |
reference (BFS) |
1.00 | 1.000 | 94.2 |
Snake, 6 episodes:
| controller | mean food | max food | survival |
|---|---|---|---|
model |
4.50 | 16 | 1.00 |
random (floor) |
0.67 | 2 | 0.00 |
reference (oracle) |
23.50 | 31 | 1.00 |
Regenerated from the evaluation JSON by scripts/readme_table.py, and
--check fails the test suite if this block drifts from it. Full tables,
including calibration and the planner probe, are in RESULTS.md.
Using it
pip install torch huggingface_hub
git clone https://github.com/lewislulu/jevon-arcade # or this repo, for the code
from_pretrained fetches the config, the tokenizer and one checkpoint — about
80MB, not the whole repository — and caches them under ~/.cache/huggingface.
It takes a local run directory just as happily, so the same line works before
and after publishing:
from jevon import from_pretrained
from jevon.inference import maze_state_sample
from envs import maze as mz
jevon = from_pretrained("lewislululu/jevon") # or "runs/jevon-final"
state = mz.generate_maze(21, "tree", seed=7)
state.step("south") # the start cell has one exit, so it poses no choice
answers = {a["question"]: a for a in jevon.answer([maze_state_sample(state)])}
answers["action"]["probabilities"]
# {'north': 0.002, 'south': 0.998} BFS-optimal: south
answers["clear_west"]["probabilities"]
# {'false': 0.999, 'true': 0.001} a wall
answers["solvable"]["probabilities"]
# {'false': 0.001, 'true': 0.999}
One pass answers every question the state offers — action, the four
clear_* booleans, distance and solvable — because the state and the
board are encoded once and the questions cross-attend to them. "Offers" is
literal: a cell with one exit poses no choice, so no action question is
asked there and none is answered. Nothing is
decoded as text: each answer is a distribution over that question's own
candidates, which is what makes the numbers above comparable across states.
The planner's distance field is readable directly, and costs nothing extra during an episode because it does not depend on the agent:
field = jevon.distance_field(maze_state_sample(state)) # (21, 21), lower is closer
On a maze that field is a shortest-path distance up to one global scale — on
the board above, rescaled by a single constant it matches BFS to within
0.0096 cells — which is why greedy descent on it arrives from anywhere.
Building the property in is the part to read
before treating that as a result about training; it is a property of the
recurrence, and the honest apples-to-apples number is the model row.
Device: cuda, then mps, then CPU, unless you pass device=.
To watch it play rather than query it, jevon-arcade runs the maze and
snake boards in a browser while this model decides, one forward pass per step,
through the generators in scripts/play.py that produced every number above.
Intended use, and what this is not
Jevon is a research artefact: a demonstration that a 20M-parameter model with the right structural prior solves grid planning that a language-model backbone of similar size does not. It is useful for studying amortised value iteration, typed decision heads, and for reproducing or disputing the comparison with NanoJev below.
It is not a general agent and not a language model. It answers a fixed set of typed questions about Maze and Snake boards; it has no text output, no instruction following, and nothing it learned transfers off a grid.
Three limits worth knowing before quoting a number:
- The maze field is exact by construction, not by training. A checkpoint
trained for 60 steps already solves every maze optimally. Any
model-fieldmaze figure describes the architecture. See Building the property in. - Snake is the weak game. 4.50 mean food against its own oracle's 23.50. There is no planner for Snake the way there is for Maze, because distance to food is the wrong field to descend — survival dominates.
- The splits are small. 141 maze and 92 snake choice questions in
test. Every accuracy here is printed next to its constant-prediction control for that reason, and RESULTS.md givesnand the distinct board count for each.
Trained entirely on synthetic boards from the generators in envs/, whose
oracles are exact. There is no human data anywhere in it.
Why rebuild it
NanoJev's own development log reports that its maze model learned nothing:
its test-maze atomic accuracy (0.5625) is identical to its always-true
control (0.5625), and every trained arm solved 0 of 3 mazes within the
step cap. The impressive "244 attempts" demo comes from the surrounding
controller — collision memory, untried-edge exploration, repositioning — not
from the network.
Three things caused that, and Jevon fixes each one.
1. The board was a string
NanoJev feeds the maze to a language model as ASCII art. In a flattened token
sequence, two vertically adjacent cells of a 50×50 maze are ~51 tokens apart,
so the model has to learn 2-D adjacency through attention, from scratch, for
every board size. Jevon reads the board as an 8-channel grid
(free, blocked, focus, target, body, tail, decay, game) shared by both games,
so one set of spatial weights serves Maze and Snake alike.
2. There was no iterative computation
Shortest-path distance is not a local function. A 21×21 corridor maze has a shortest path of 154 cells; a 31×31 has 312. A fixed-depth transformer cannot propagate a distance signal that far in one pass, no matter how wide it is.
Jevon adds a SpatialPlanner: a weight-shared gated message-passing block
applied recurrently over the 4-neighbourhood, masked by passability. T
iterations propagate information T cells — amortised value iteration, or
Bellman–Ford with learned operators. The budget is sized to the board:
default_iterations(size) = min(4096, max(32, 2 * size + size * size // 2))
# 11 -> 82 21 -> 262 31 -> 542 51 -> 1402
Memory stays flat in T because only the last grad_steps iterations carry
gradient; earlier ones run under no_grad (implicit/phantom gradient).
3. One question per state cannot teach a 500-step recurrence
This was the subtle one. Even with a correct planner and a correct iteration
budget, a linear probe from the planner's features to true BFS distance scored
R² = 0.21 — the field was barely related to distance. The recurrence was
getting gradient from a single distance question per state.
So the planner gets its own dense objective: a 1×1 convolution reads every cell and predicts the BFS distance to the target (log-squashed) plus a reachability logit. That is ~`size²` supervised targets per board instead of one, and it is defined purely on the board the planner receives, so the target is exactly the quantity the planner has the information to compute.
tests/test_model.py::test_planner_can_learn_a_distance_field
R² -31.9 -> 0.79 (4 boards, 400 steps, 9x9)
4. A graded scale is not seven unrelated categories
distance and room are Jev Score questions: their criteria are an
ordered scale ("within ten moves", "within twenty-five moves"). Encoded as a
flat softmax with a one-hot target, naming the neighbouring level costs exactly
as much as naming the opposite end of the scale, and with seven near-synonymous
levels the marginal mode becomes the easiest minimum.
It did, for a long time. acc_score sat on 0.3333 at every evaluation
through the first 1000 steps of a run — exactly the always-level-4 constant —
while a ridge probe on the very same planner features, binned through the very
same thresholds, scored:
probe -> level 0.4335 best constant 0.3436
within one level 0.9076
The information was there and the head was not using it, and 91% of the probe's errors were one level off — precisely the structure a one-hot target discards.
A matched control run says how much of this is the target and how much is just
impatience: left alone, the one-hot head does eventually escape, reaching
0.3722 at step 1200. So the constant is an early-training attractor rather than
a permanent trap, and the question the ablation answers is whether these two
changes escape it sooner and end higher, not whether escape is possible at all.
Both are off by default and measured in runs/arm-*:
- Ordinal targets (
--score-smoothing): score mass decays with distance along the scale. The peak stays on the true level, so every reported accuracy stays comparable with runs trained without it. - Exposed field (
--expose-field): the text side gathers the field head's own distance/reach read-out alongside the raw planner features. The field head is already trained on every free cell, so its output is the most distilled form of exactly what the question asks; without this the transformer re-derives the same map from a far sparser signal.
5. Two ways of being wrong about the same number
field_r2 appeared to fall while the decision metrics rose, which looks like
the decision loss dragging the planner off its objective. That produced one
falsified hypothesis and one real bug, and the order matters: measuring first
is what separated them.
The hypothesis. If the decision loss were overwhelming the field loss, the planner's gradient would show it. It does not:
decision loss 0.5050 grad->planner 0.03195
field loss 0.0653 grad->planner 0.04808 ratio 0.66 : 1
The field objective already dominates, so --field-weight 1.0 is calibrated
and the tempting fix — raising it — would have been a change made for a reason
measurement does not support.
The bug. field_r2 was computed per evaluation batch and then averaged.
R² is a ratio of sums and is not averageable: a batch of boards with little
distance variance drives its own R² arbitrarily negative and drags the mean
with it. Pooling the sums and dividing once gives a stable number, and the
"collapse" disappears. The probe, which pools over every cell, had been
reporting R² ≈ 0.50 on the same checkpoints the whole time.
6. R² is not the metric the controller uses
Greedy descent never reads an absolute distance. It compares a cell's four
neighbours and steps to the lowest, so what matters is whether the ordering
of neighbours is right — and a path only succeeds if every comparison along it
succeeds. scripts/probe.py reports that directly:
size probe R² field head R² descent
11 0.5013 0.3425 0.6325
21 0.2697 0.1801 0.6561
Descent accuracy 0.65 means a 21×21 maze needs ~130 consecutive correct
choices; the measured closed-loop solve rate for --controller model-field was
0, exactly as that predicts.
The cause is arithmetic. With a log-squashed target, neighbouring cells differ by ~`1/d`, so contrast collapses as distance grows — measured on a trained checkpoint:
true dist descent acc mean target gap
0-9 0.7273 0.08279
10-19 0.6458 0.02730
20-29 0.6531 0.01680
30-39 0.7429 0.01191
40-49 0.5000 0.00980 <- 8.5x less contrast
and the same checkpoint's field MAE was 0.107 — larger than the gap it had
to resolve anywhere on the board. Rescaling would not help, since it scales
the error by the same factor; the shape has to change. --field-target linear
makes every neighbouring pair differ by exactly 1/size² wherever it sits, so
precision is spent uniformly instead of being concentrated near the goal.
That argument is arithmetic and stands on its own. The empirical claim it invites — that the linear target measurably improves closed-loop descent — does not survive the seed-variance check in section 7, and is not made here.
7. A field the controller can follow is not the same as an accurate field
Section 6 explained a solve rate of 0 with descent accuracy 0.65: a 21×21 maze
needs ~130 consecutive correct choices, and 0.65¹³⁰ is nothing. That
reasoning is wrong, and the way it is wrong mattered more than the original
problem.
A wrong step is not a lost episode. It moves the agent to another cell, where it descends again. What ends an episode is not error rate but error structure: a spurious local minimum, which descent enters and never leaves. So the thing to measure is not how often a step is right but how often a walk arrives. From every free cell, follow the field downhill and record where it ends up:
size T reach cycle descent
11 41 0.272 0.728 0.729
11 82 0.293 0.707 0.743
11 164 0.299 0.701 0.736
21 131 0.070 0.930 0.623
21 262 0.072 0.928 0.630
21 524 0.072 0.928 0.641
Two things fall out. 70–93% of cells sit in a basin, so the failure is structural rather than statistical — descent accuracy of 0.63 and reach of 0.07 are not two views of one number. And quadrupling the iteration budget moves reach by under three points, so the recurrence has converged: propagation depth, the thing this architecture was built to supply, was never what was missing.
The direct fix does not work, and measuring the noise is why we know
The obvious response is to supervise the property: for every cell, hinge the
best true-downhill neighbour below every other neighbour by one true step
(descent_loss). It is exactly 0 on the true field under both target shapes,
so it cannot fight the regression term, and raising its weight does move reach
in the right direction — 0.104, 0.122, 0.132, 0.154 at weights 0, 1, 5, 20.
That looks like a small win. It is not a win at all. Re-running the unchanged baseline under three seeds gives:
seed reach descent acc
0 0.1040 0.6700
1 0.0446 0.6283
2 0.1478 0.6575
A 3.3× spread, with the whole ablation sitting comfortably inside it. The honest reading is that the hinge's effect is not resolvable at this sample size, and any conclusion drawn from that first table would have been an artefact. The term is kept — it is principled and costs nothing at weight 0 — but it is not what fixed the problem, and this repository does not claim it did.
One correction to that table, found later. Each row ran in a fresh process,
and at the time generate_maze seeded itself with hash(topology), which
Python salts per process — so the rows differ in their held-out boards as
well as their training seed, and the spread bounds the two together rather
than the seed alone. That is now fixed (TOPOLOGIES.index, pinned by a test
that runs the generator under three hash salts). It does not rescue the
ablation: a band measured over more sources of variation than intended is
still a band the ablation sits inside, and the four ablation rows were drawn
from separate processes too, so they were never a controlled comparison in
the first place. It does mean the number quoted above is an upper bound on
seed variance specifically, and the honest summary is narrower than it looks:
this experiment does not resolve the hinge's effect, and it never could
have.
The same caveat retires a claim section 6 would otherwise support: matched conv-head baselines differing only in target shape came out at 0.104 and 0.200, which is also within the noise band above.
Building the property in
Global monotonicity is all the local constraints holding at once, so a penalty
that gets ~70% of them right buys far less than 70% of the benefit. Stop asking
for the property and construct it. MinPlusField predicts a per-cell cost
and reads distance off a Bellman–Ford recurrence:
v(c) ← cost(c) + min over passable neighbours n of v(n), v(goal) = 0
With strictly positive costs the fixed point is a shortest-path distance, so
every non-goal cell has a strictly lower neighbour and greedy descent
terminates at the goal from anywhere. Reach is 1.0 by construction rather than
by training — the test suite asserts it with the cost weights randomised, where
the field bears no resemblance to the true distance and is still traversable.
Costs are emitted in units of 1/area, exactly the step size of the linear
target, so the two agree by design rather than by tuning.
descent acc reach field MAE time
conv 0.7115 0.2001 0.05636 65s
min-plus 1.0000 1.0000 0.00000 67s
The conv row carries the noise band established above; the min-plus row
does not, because 1.0000 there is a proof obligation the tests discharge
rather than a measurement that could have come out otherwise. That asymmetry
is the whole argument for building the property in instead of training for it.
The zero MAE is the caveat, and it belongs next to the headline. For a
maze, uniform cost is exactly right; the head initialises at
softplus(0.5413) ≈ 1; so it computes exact BFS before a single gradient step
(measured error against true BFS: under 1e-2 cells). The maze field is
therefore a property of the architecture, not something the run learned, and
any maze number produced by --controller model-field has to be read that way.
What stays genuinely learned is the cost map — which is where Snake lives,
since distance to food is not the whole objective there — and whether handing
the transformer an exact field improves the action head. That second question
is the apples-to-apples comparison with NanoJev, and it is the one the
headline model controller reports.
Closed-loop, the guarantee survives the whole inference path. A checkpoint
trained for 60 steps — long enough to confirm the plumbing works and not
much else — played with --controller model-field:
size solve rate mean steps mean shortest collisions
11 1.000 33.0 33.0 0
21 1.000 82.0 82.0 0
31 1.000 172.5 172.5 0
Every maze solved, by an exactly optimal path. Read that as a statement about the architecture: a 60-step checkpoint has learned nothing, and the number comes from the recurrence being breadth-first search. It is reported because the comparison it replaces — Jev and NanoJev both solving 0 of 128 — is a comparison between controllers, and this is what a controller with the right structural prior does on the same boards.
This is the Value-Iteration-Network idea (Tamar et al., 2016) with the max-plus reward recurrence swapped for the min-plus distance one the field head was already supervised on.
Architecture
header + question text ──► shared prefix encoder (6 layers) ─┐
├─► cross-attention ──► score ──► softmax
candidate k text ────────► candidate encoder (2 layers) ──────┘ │
│
8-channel board ──► SpatialPlanner (recurrent, T steps) ──► field ─────┤
│ │
└──► 1x1 conv ──► distance + reach (training only)
Three structural differences from NanoJev, beyond the planner:
Shared prefix. NanoJev runs one full forward pass per candidate,
re-encoding the whole state prefix each time. A 50×50 ASCII maze is ~2,600
tokens, so a 4-candidate decision costs ~10.4K tokens. Jevon encodes the state
and question once and lets candidates cross-attend to it: cost is
|prefix| + K·|candidate| rather than K·(|prefix| + |candidate|). This is
the "tree sharing" NanoJev lists as unimplemented.
Anchored candidates. A candidate that names a cell ("Move north to (7,11)") carries that coordinate, and the scoring head gathers the planner's feature at exactly that cell. The model does not have to re-derive from text which cell a candidate refers to.
Field caching. The planner never sees the agent — the focus channel is
zeroed before the recurrence — so for a fixed maze the field is constant for an
entire episode and is computed once. Measured: 10 maze decisions in 0.71 s
with planner_calls=1, cache_hits=9 (71 ms/decision).
Environments
Both games ship with exact oracles, because the oracle is the training signal.
Maze reproduces NanoJev's four topologies (corridor, tree, loops,
random_obstacle) and its ASCII rendering so numbers are comparable. Targets
are the full BFS-optimal action distribution — every shortest-path move shares
the mass, so the model is never punished for picking a different optimal move.
Snake deliberately departs from NanoJev. NanoJev's target is a local greedy rule: step toward the food. That self-traps. Jevon's oracle prefers, in order: survive → keep the tail reachable → keep room for the body → then close on the food. Measured over 20 games on 12×12:
| Snake target policy | mean food | max food | mean steps |
|---|---|---|---|
| NanoJev-style greedy | 22.70 | 41 | 229.0 |
| Jevon survival oracle | 43.00 | 105 | 2000.0 (full horizon, every game) |
Half of all sampled Snake states are constructed directly as self-avoiding walks rather than reached by play. A good policy almost never produces a cramped board, so rolling out to one is both slow and rare — yet cramped boards are exactly where the survival questions carry signal.
Question types
| Type | Example | Candidates |
|---|---|---|
choice |
which move to make | 2–4 dynamic |
boolean |
"north is clear", "the goal is reachable" | false / true |
score |
how far the goal is, how much room remains | 5–7 ordered levels |
Every accuracy in this repo is reported next to the constant-prediction
control for the same questions. That is not decoration: the original
room question was scored against board area rather than snake length, which
made every sampled state the top level, and the model scored exactly the
constant baseline (0.405556) to six decimal places while appearing to learn.
Training and reproduction
uv venv && uv pip install -r requirements.txt
python scripts/build_dataset.py # freeze val / test / OOD splits
# The shipped checkpoint, exactly. Every flag matters: --min-plus-field is
# what makes descent reach the goal by construction, --expose-field is what
# lets the text side read the planner, and --planner-lr-mult compensates for
# the planner block being applied several hundred times per step but carrying
# gradient through only the last eight.
python scripts/train.py --out runs/jevon-final --steps 9000 --eval-every 500 \
--planner-lr-mult 10 --field-weight 1.0 --seed 11 --min-plus-field --expose-field
bash scripts/benchmark.sh runs/jevon-final # eval + probe + play + RESULTS.md
pytest -q
Each run directory records both halves of its own provenance: config.json
is the architecture and args.json is the recipe (argv verbatim, plus the
parsed values the defaults filled in). benchmark.sh defaults to
balanced.pt — see Which checkpoint ships.
runs/jevon-final/ ships whole: the JSON this page is generated from, and
balanced.pt and best.pt, the two checkpoints Which checkpoint
ships compares. Each is 80MB of fp32 weights and
goes through Git LFS, so a clone needs it:
git lfs install
git clone https://huggingface.co/lewislululu/jevon
Cloned without LFS, those two paths hold 130-byte pointer files. Loading one
says exactly that and names the command, rather than failing inside
torch.load.
The JSON matters separately from the weights, and for a reason worth stating:
every test that reads those files degrades to a skip when they are
missing. An earlier .gitignore excluded the whole runs/ directory, which
shipped a repo whose entire documentation-drift apparatus passed by not
running. pytest -q on a fresh clone now really does check this page against
the run it describes. last.pt stays out — it is wherever training stopped,
which is not a claim this repo makes.
scripts/play.py exposes each controller loop twice: as maze_steps /
snake_steps, generators that take one decision per next(), and as
play_maze / play_snake, which drain them. The split exists so that
jevon-arcade's live viewer can take a single forward pass
per HTTP request without owning a second copy of the decision logic. A
divergent copy would be a viewer demonstrating a model nobody benchmarked,
which is the failure these two repos have hit more often than any other. The
step budget lives there too, in default_max_steps, so a live episode and a
recorded one agree about what running out means.
It exposes deliberately separable controllers so that model skill is never confused with controller scaffolding:
model— the network alone. No search, no memory, no visited set.model+memory— plus one bit per cell: prefer a move onto ground not yet stood on. This is the same kind of scaffolding this README faults NanoJev's demo for, so it is reported separately and never as the model's score.random+memory— the control for the row above. Identical bookkeeping, no network. It exists becausemodel+memorycannot be read without it, and reading the two together is less flattering than reading one:
| controller | size | solve | efficiency |
|---|---|---|---|
model+memory |
11 | 1.00 | 1.000 |
random+memory |
11 | 0.92 | 0.588 |
model+memory |
21 | 1.00 | 1.000 |
random+memory |
21 | 0.83 | 0.310 |
model+memory |
31 | 1.00 | 1.000 |
random+memory |
31 | 0.75 | 0.147 |
12 episodes per size per topology, from the shipped checkpoint.
Read the two rows together, per size. The memory alone is a capable maze
solver on a small board — near-exhaustive exploration finds the goal — so
wherever random+memory matches model+memory on solve rate, arrival is
the scaffolding's doing and not the network's. The table bolds whichever of
the pair wins each column rather than always bolding the model, so a bold
control is a column the model does not own.
Efficiency is the column that does belong to the network. When the model arrives it arrives on very nearly the shortest path; the control reaches the same place several times slower. That gap is what the planner contributes, and it is worth being exact about what it is not. The model knows the direction and has no state with which to notice it has been somewhere before, so cycling is what ends its unsolved episodes — and the memory masks that rather than fixing it: when every neighbour has been visited the controller falls back to the full legal set and the greedy policy walks back into the cycle it just left.
reference— BFS-optimal (maze) / survival oracle (snake).random.
Why the memory is in the controller and not in the board. The grid has a
visited channel (channel 6, used by snake for tail age) and maze leaves it
empty on purpose. Filling it would not teach the model to stop cycling: the
targets are BFS-optimal actions, and the optimal action from a cell is a
function of (walls, goal, cell) alone — where the agent has already been is
conditionally independent of it. Under this supervision the channel is noise
by construction, and a better-fitted model would learn to ignore it faster.
Cycling is approximation error in the action head, not absent memory, so the
lever that moves it is action accuracy; the architectural answer is
model-field, whose fixed point is a shortest-path distance and which
therefore cannot cycle at all.
The candidate set is state.legal_actions(), which is what the model was
trained to score. Offering it all four compass directions instead let a wall
win the argmax, and since a blocked move does not change the state, the next
decision was identical — an agent that stood still for the entire budget.
Every maze number for model predating commit 6afaae8 measured that.
Held-out splits are by seed (seed % 10: <8 train, 8 val, 9 test), and
the OOD split uses board sizes never sampled during training (maze 41, 51;
snake 20, 24).
Which checkpoint ships
Training writes three: last.pt, best.pt (lowest eval loss) and
balanced.pt. balanced.pt is what loads when you do not name one —
benchmark.sh, record.sh and every script's --weights agree on that, and
a test pins them to each other so they cannot drift. The reason is a selection
bug that best.pt walks straight into.
The frozen validation split is 1325 questions over 180 states: 974 boolean, 180 score, and 171 choice. Eval loss averages over all of them, so 73.5% of it is booleans and 12.9% is choice — while every headline number in this README comes from those 171 choice questions, because those are the gameplay decision. The two can move in opposite directions, and they do.
One run hit its lowest loss at a step scoring maze 0.9794 and snake
0.2973. The uniform controls on this split are 0.4905 for maze and
0.4955 for snake — not 1/k, because many states have several
tied-optimal moves — so that checkpoint was a full 20 points below chance
on snake while holding the best loss in the run. It had stopped playing one
of the two games, and eval loss could not see it.
So balanced.pt selects on min(maze_action, snake_action) instead —
def balanced_score(metrics: dict) -> float:
return min(metrics.get("maze_action", 0.0), metrics.get("snake_action", 0.0))
min rather than a mean, because the failure being guarded against is
exactly the trade of one game for the other, and a mean averages it away. A
missing metric scores zero rather than being skipped, so a run that never
reports snake_action cannot earn a balanced checkpoint by default.
This is not hypothetical, and it is not rare — the shipped run did it too.
Caught mid-training at step 1500, the two checkpoints on disk at that instant
made the point better than any argument (both were later superseded; the
final ones are in runs/jevon-final/):
best.pt (step 1500) |
balanced.pt (step 1000) |
|
|---|---|---|
| eval loss | 0.6161 | 0.6269 |
acc_boolean |
0.94688 | 0.94688 |
acc_score |
0.42222 | 0.42222 |
acc_choice |
0.6665 | 0.8469 |
snake_action |
0.5135 | 0.9459 |
best.pt won on loss. Boolean and score accuracy are identical to five
decimal places between the two, so neither moved; the only accuracy that
moved is choice, and it moved 18 points the wrong way. The loss improved
because the model sharpened its confidence on booleans it was already getting
right (ce 0.4377 → 0.4286) while its snake play fell to 0.5135, a hair
above the 0.4955 control. Selecting on loss would have shipped that.
Training later reached maze 1.0000 and snake 0.9595 together, and
balanced.pt followed it up as readily as it had refused to follow it down.
The shipped run's own two checkpoints are that same argument, and unlike the snapshot above both are on disk for a reader to check:
best.pt (step 8500) |
balanced.pt (step 3000) |
|
|---|---|---|
| eval loss | 0.3999 | 0.4821 |
acc_boolean |
0.96732 | 0.94688 |
acc_score |
0.34444 | 0.35000 |
acc_choice |
0.9889 | 0.9944 |
maze_action |
1.0000 | 1.0000 |
snake_action |
0.9730 | 0.9865 |
Bold marks the better cell in each row — loss being the one row where smaller
is better. best.pt takes loss and the two question types that dominate it;
balanced.pt takes the choice questions, which are the ones every headline
number in this README is scored on. benchmark.sh defaults to balanced.pt,
which is why.
Selection reads val; every headline number comes from test
That separation matters more here than it usually does, because the quantity being selected on keeps moving. The eval is deterministic — running it twice on a fixed checkpoint returns identical figures to six decimals — so what the table below measures is real weight movement between checkpoints, not measurement noise.
| half of the run | evals | largest eval-to-eval move in snake_action |
in questions |
|---|---|---|---|
| first (500-5000) | 10 | 0.446 |
33 of 74 |
| second (5000-9000) | 9 | 0.122 |
9 of 74 |
The movement shrinks by the factor shown there, and it does not reach zero.
An earlier draft of this section claimed it stopped, on the evidence of four
consecutive evals returning a byte-identical 0.9865; two evals later one
gave back six questions and the sentence was false. Any claim phrased over
the last N evals goes stale as soon as training continues past them, which is
why this one is phrased over halves of the run and generated rather than
typed.
Selecting the maximum of ~18 such evals is a maximum of a noisy estimate, so
a val figure is biased upward by construction — and it is the second-half
column that sizes that bias, not the first. test is scored
once, by a checkpoint that never competed on it, which is why the headline
quotes it and names the split it is quoting. val is reported too, in
RESULTS.md, where it can be read against the split that
selection never touched.
Results
See RESULTS.md, which is generated from the evaluation JSON rather than typed by hand.
Licence
Copyright © 2026 Lewis.
AGPL-3.0-or-later. Use it, modify it, run it for any purpose including a commercial one — provided the complete source of whatever you build on it is offered under the same licence. Section 13 is why this is the AGPL and not the GPL: it extends that to users who only ever reach the model over a network, so serving Jevon behind an API is covered where plain GPL would not have been.
If your product cannot carry that obligation, a commercial licence without the copyleft terms is available — sudolewis@gmail.com.
The viewer, jevon-arcade, is MIT instead. It holds no model code, it imports this repository at runtime, and a demo is worth more the more people run it — but a distribution that combines the two carries this licence, not that one.