Accuracy plateaued. The model kept changing.

By step 60,000, a small mixture-of-experts transformer — 4 layers, 4 experts per layer, about 1.4 million parameters — had reached about 99 % held-out accuracy on five operations of modular arithmetic, mod 113: add, sub, mul, div, exp. Training ran on to 120,000; the curve barely moved again. Inside, the readings kept moving.

IRIS™, a neural network instrumentation tool developed by Saccadic LLC, was used to instrument this run and generate both figures shown here.

Figure 4.4 (Accuracy) is the outside view: held-out accuracy sat low for tens of thousands of steps, passed 50 % near step 33,000 and 90 % near 44,000, then flattened. The operations did not arrive together: mul and div reached 90 % first (by step 36,952), exp next (42,952), sub and add last (46,102 and 47,302). Multiplication before addition is backwards for a schoolchild, but there is nothing to carry here: mod 113, each operation is a rule on 113 symbols.

Fig. 4.4, Accuracy: held-out accuracy and the five operations over 120,000 steps.

A loss-and-accuracy dashboard would call this run converged at 60,000 and stop it.

Figure 10.7 (Effective pathways and prediction loss) comes from IRIS, an instrument that measured the model's internals 120 times. For each operation, the solid line is its effective pathways: across every link between unit groups that its text passes through, how many independent directions carry the signal, pooled into one reading. The dashed line is its prediction loss on probe texts, lower is better.

Fig. 10.7. The vertical line is the bench's playback clock, parked at step 33,352, the first frame where held-out accuracy reads above 50 %.

On the plateau the operations differ: at step 20,131 the readings run from 1,050 (exp) to 1,715 (div). Across the rise every line falls, to 417–655 by step 46,102. Then, from 60,000 to 120,000, with accuracy flat, they keep falling and do not land together: add ends at 97 and sub at 96; mul, div and exp at 242, 229 and 214. The loss lines end in the same two groups (add 3.7, sub 3.8; mul 5.0, div 5.4, exp 8.4). A second reading in the paper moves the same way: the deepest layer's effective rank (directions carrying 95 % of activation variance) falls from 44 at step 54,152 to 32 at 120,000.

IRIS adds a per-operation record of the wiring in use, through formation and long after the usual curves go quiet. Keeping it cost about 0.7 % of wall clock; the 120-frame history is an 80 MB file; the run took about three hours on a Mac mini.

The full reconstruction is "The Shape of a Grok," an internal paper from Saccadic LLC. Learn more about IRIS at www.saccadic.ai.