Results

Generated by scripts/make_results.py from the JSON written by train.py, evaluate.py, probe.py and play.py. Do not edit by hand.

Baseline: what NanoJev reports for itself

From docs/DEVELOPMENT_RESULTS.md in the NanoJev repository.

NanoJev task	reported acc	always-true control	above control
`scaled_maze` (test)	0.5625	0.5625	no
`scaled_maze` (ood)	0.5625	0.5625	no
`snake_one_step_safety_v4`	0.9220	0.9220	no

Closed-loop play, same source, 128-step limit:

maze	Jev	NanoJev	reference
all three mazes	0/128	0/128	solved 1/16, 1/24, 1/96

"The learned direct-action models and Jev do not solve any of these three mazes within the shared 128-step limit."

Every accuracy in this file is therefore printed next to its constant-prediction control, and gameplay is reported as solve rate rather than per-question accuracy.

Does the planner actually compute distances?

Ridge probe from the planner's per-cell features to true BFS distance, on held-out mazes. diameter is the longest true shortest-path in those boards; T is the iteration budget.

descent is the fraction of cells whose lowest-predicted neighbour lies on a true shortest path. reach is the fraction from which greedy descent actually arrives at the goal, and it is the one that predicts solve rate: a field can be right 70% of the time per step and still trap most walks in a spurious basin, so descent is an upper bound on nothing in particular. R² scores absolute values; only reach scores the behaviour.

read-out R² scores predict_field(), the field this controller actually descends -- the min-plus recurrence when the run enabled it, the conv head otherwise. The training curve's conv field R² scores a different head; see the note there.

size	cells	probe R²	probe MAE (cells)	read-out R²	descent	reach	diameter	T
11	215	0.0937	7.97	1.0000	1.0000	1.0000	48	82
21	901	0.0420	29.71	1.0000	1.0000	1.0000	152	262
31	2011	0.0316	63.06	1.0000	1.0000	1.0000	310	542
51	5679	0.0068	145.23	1.0000	1.0000	1.0000	672	1402

Question-level accuracy

uniform is the accuracy of guessing uniformly over the same candidates. A model that matches its control has learned nothing, which is the failure NanoJev's own log records for its maze model.

boards is how many distinct boards the row's questions came from, and it is there so that n can be trusted. Questions that share a board are not independent tests of whether the model can read a board -- they share walls, goal and distance field, so they succeed or fail together -- and a row reporting n alone cannot tell you whether that is happening. Here it is not: the generator draws a fresh board per state, so boards tracks n on every split and n is the honest denominator. A row where the two diverge is one whose precision should be read off boards.

val — 1325 questions, ECE 0.0865

group	n	boards	accuracy	uniform control	TVD	Brier
`maze/action`	97	97	1.0000	0.4905	0.0188	0.0023
`maze/boolean`	530	106	1.0000	0.5000	0.0007	0.0000
`maze/choice`	97	97	1.0000	0.4905	0.0188	0.0023
`maze/clear`	424	106	1.0000	0.5000	0.0004	0.0000
`maze/distance`	106	106	0.3962	0.1429	0.7764	0.7640
`maze/score`	106	106	0.3962	0.1429	0.7764	0.7640
`maze/solvable`	106	106	1.0000	0.5000	0.0018	0.0000
`snake/action`	74	74	0.9865	0.4955	0.3500	0.3148
`snake/boolean`	444	74	0.8829	0.5000	0.2338	0.2022
`snake/choice`	74	74	0.9865	0.4955	0.3500	0.3148
`snake/escape`	222	74	0.8108	0.5000	0.2773	0.2891
`snake/room`	74	74	0.2838	0.2000	0.7990	0.7981
`snake/safe`	222	74	0.9550	0.5000	0.1903	0.1152
`snake/score`	74	74	0.2838	0.2000	0.7990	0.7981

test — 1765 questions, ECE 0.0590

group	n	boards	accuracy	uniform control	TVD	Brier
`maze/action`	141	140	1.0000	0.4882	0.0396	0.0079
`maze/boolean`	740	147	1.0000	0.5000	0.0007	0.0000
`maze/choice`	141	140	1.0000	0.4882	0.0396	0.0079
`maze/clear`	592	147	1.0000	0.5000	0.0004	0.0000
`maze/distance`	148	147	0.2973	0.1429	0.7847	0.7796
`maze/score`	148	147	0.2973	0.1429	0.7847	0.7796
`maze/solvable`	148	147	1.0000	0.5000	0.0019	0.0000
`snake/action`	92	92	0.9674	0.4783	0.3107	0.2770
`snake/boolean`	552	92	0.8895	0.5000	0.2223	0.1938
`snake/choice`	92	92	0.9674	0.4783	0.3107	0.2770
`snake/escape`	276	92	0.8188	0.5000	0.2628	0.2747
`snake/room`	92	92	0.2500	0.2000	0.7993	0.7986
`snake/safe`	276	92	0.9601	0.5000	0.1818	0.1128
`snake/score`	92	92	0.2500	0.2000	0.7993	0.7986

ood — 1167 questions, ECE 0.0540

group	n	boards	accuracy	uniform control	TVD	Brier
`maze/action`	107	107	1.0000	0.5132	0.2699	0.1504
`maze/boolean`	550	110	1.0000	0.5000	0.0007	0.0000
`maze/choice`	107	107	1.0000	0.5132	0.2699	0.1504
`maze/clear`	440	110	1.0000	0.5000	0.0004	0.0000
`maze/distance`	110	110	0.0545	0.1429	0.8542	0.9160
`maze/score`	110	110	0.0545	0.1429	0.8542	0.9160
`maze/solvable`	110	110	1.0000	0.5000	0.0020	0.0000
`snake/action`	50	50	0.9800	0.4867	0.3574	0.3199
`snake/boolean`	300	50	0.8333	0.5000	0.2636	0.2621
`snake/choice`	50	50	0.9800	0.4867	0.3574	0.3199
`snake/escape`	150	50	0.6800	0.5000	0.3561	0.4471
`snake/room`	50	50	0.2000	0.2000	0.7989	0.7978
`snake/safe`	150	50	0.9867	0.5000	0.1710	0.0771
`snake/score`	50	50	0.2000	0.2000	0.7989	0.7978

Closed-loop play

Read model and model-field as answering different questions. model is the network's action head alone -- no search, no memory, no visited set -- and is the apples-to-apples comparison with NanoJev. model-field descends a min-plus recurrence whose fixed point is a shortest-path distance by construction, so it arrives from anywhere; for a maze it does so at initialisation, before any training, because uniform cost is already the right answer. Its solve rate is a property of the architecture, not a measurement of what this run learned.

Maze

deadlock is the fraction of episodes in which the controller spent a tenth of its step budget walking into the same wall, and cells is how many distinct squares it stood on. They separate two failures a solve rate of 0.00 reports identically: standing still and touring the board. Deadlock should now read 0.00 for every controller, because only legal moves are offered as candidates -- it is kept as a regression guard, not a finding. The failure that remains is cycling: the planner never sees the agent, so a greedy policy that steps A->B->A has no state with which to notice, and cells is what exposes it.

controller	solve rate	mean steps	efficiency	cells
`maze_model`	1.00	93.2	1.000	94.2
`maze_model_sampled`	1.00	106.1	0.926	95.4
`maze_model_field`	1.00	93.2	1.000	94.2
`maze_model_memory`	1.00	93.2	1.000	94.2
`maze_random_memory`	0.83	689.7	0.363	188.5
`maze_reference`	1.00	93.2	1.000	94.2
`maze_random`	0.14	560.4	0.068	92.2

Snake

controller	mean food	max food	mean steps	survival
`snake_model`	4.50	16	288.0	1.00
`snake_reference`	23.50	31	288.0	1.00
`snake_random`	0.67	2	33.8	0.00

Training curve

Frozen validation split, never sampled during training.

conv field R² scores supervised_field() -- the conv head the field loss trains. It is not the field the controller descends. Under --min-plus-field those are different objects: the read-out is the min-plus recurrence, scored as read-out R² in the probe table above, and nothing at inference reads the conv head at all. It stays in the loss as an auxiliary task on the shared trunk, so a negative value here means that auxiliary head has stopped tracking the trunk the action head and the recurrence are shaping -- it does not mean the planner is wrong, and reach is where that would show up.

step	loss	acc	choice	boolean	score	conv field R²	maze action	snake action
500	0.6430	0.8310	0.6806	0.9469	0.3500	0.5048	0.5773	0.8108
1000	0.6269	0.8626	0.8469	0.9469	0.4222	-2.7876	0.7732	0.9459
1500	0.6161	0.8393	0.6665	0.9469	0.4222	0.0461	0.7835	0.5135
2000	0.5041	0.8703	0.9833	0.9469	0.3500	-0.1305	1.0000	0.9595
2500	0.4842	0.8531	0.8499	0.9469	0.3500	0.1714	1.0000	0.6486
3000	0.4821	0.8718	0.9944	0.9469	0.3500	-0.2088	1.0000	0.9865
3500	0.4725	0.8718	0.9944	0.9469	0.3500	0.0103	1.0000	0.9865
4000	0.4797	0.8702	0.9828	0.9469	0.3500	0.2466	0.9794	0.9865
4500	0.4765	0.8710	0.9889	0.9469	0.3500	0.2097	0.9897	0.9865
5000	0.4733	0.8718	0.9944	0.9469	0.3500	0.1148	1.0000	0.9865
5500	0.4667	0.8725	0.9944	0.9469	0.3556	-0.2881	1.0000	0.9865
6000	0.4579	0.8673	0.9595	0.9469	0.3500	-0.1574	1.0000	0.9054
6500	0.4543	0.8695	0.9767	0.9469	0.3500	0.2626	1.0000	0.9459
7000	0.4258	0.8860	0.9889	0.9673	0.3500	0.0325	1.0000	0.9730
7500	0.4088	0.8831	0.9368	0.9673	0.3778	-0.4171	1.0000	0.8514
8000	0.4030	0.8860	0.9889	0.9673	0.3500	0.0676	1.0000	0.9730
8500	0.3999	0.8853	0.9889	0.9673	0.3444	0.2261	1.0000	0.9730
9000	0.4021	0.8958	0.9889	0.9673	0.4222	0.1083	1.0000	0.9730