TECHNICAL companion for the 2s era (verified code-side + hub-side, 2026-09-04)
Browse files- TECHNICAL_2s.md +1064 -0
TECHNICAL_2s.md
ADDED
|
@@ -0,0 +1,1064 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<!-- VERIFIED 2026-09-04: code-side (wf_15fc8b16) and hub-side (wf ft2-finish) passes applied. -->
|
| 2 |
+
|
| 3 |
+
# alephllm-mini-beatrix-training β Technical Companion (the 2s era: the full-splat craft and its softmax twin)
|
| 4 |
+
|
| 5 |
+
Companion to *Twinning Beatrix: A Full-Splat Byte Model, Its Softmax
|
| 6 |
+
Control, and What Reaches an Image Generator* (AbstractPhil with Claude
|
| 7 |
+
Fable 5 and Claude Fable 5.1, September 2026). It covers two crafts and
|
| 8 |
+
one packaged model: `mini-beatrix-2s/` and `mini-beatrix-2s-control/`
|
| 9 |
+
under AbstractPhil/alephllm-mini-beatrix-training, and the AutoModel at
|
| 10 |
+
AbstractPhil/mini-beatrix-2s; the dates run from the first launch
|
| 11 |
+
(run0, 2026-08-26) through the control's endpoint (2026-09-03) and the
|
| 12 |
+
claim audit that graded it (2026-09-04). It is written for a reader who
|
| 13 |
+
will load the checkpoints and read the reports. It is not the
|
| 14 |
+
mini-beatrix-1 life record (`TECHNICAL.md`, which stays as it is), not
|
| 15 |
+
the article's argument, and not a manual for the side campaigns that
|
| 16 |
+
share the repo root (`reports/v2/{battery,wide_moe,aflow,bcd}` appear
|
| 17 |
+
only as a catalogue line). Every number carries its pointer: `file:line`
|
| 18 |
+
means E:/mirel/AlephLLM/geolip/alephllm/ at HEAD 4f4b7d6 (package
|
| 19 |
+
0.8.7, the last commit of the 2s era); hub paths are relative to the
|
| 20 |
+
craft prefix; journal paths are relative to the claude-mind repo. Where
|
| 21 |
+
the journal and the code disagree, the code wins and the disagreement
|
| 22 |
+
is reported in place. Stack-side detail lives in the
|
| 23 |
+
[AlephLLM companion](https://github.com/AbstractEyes/alephllm), which
|
| 24 |
+
covers the 0.7.0 mechanisms only; the 0.8.x organs (specials, driver
|
| 25 |
+
contract, revival, gauge fallback, twin isolation) are carried here.
|
| 26 |
+
|
| 27 |
+
## The two crafts at a glance
|
| 28 |
+
|
| 29 |
+
| | mini-beatrix-2s | mini-beatrix-2s-control | mini-beatrix-1 |
|
| 30 |
+
|---|---|---|---|
|
| 31 |
+
| d_model / layers / heads | 1024 / 20 / 16 | 1024 / 20 / 16 | 768 / 16 / 12 |
|
| 32 |
+
| attention | CausalSplatHUB in all 20 blocks | CausalSDPA (16 Γ 64) in all 20 | 13 sdpa + 3 hubs (layers 4, 9, 14) |
|
| 33 |
+
| hub books Γ anchors @ D | 4 Γ 64 @ 128 (supply 0.5Γ) | β | 1 Γ 512 @ 32 (supply 16Γ) |
|
| 34 |
+
| banks | 3 experts @ ff 1024 | 3 @ 1024 | 3 @ 768 (full width) |
|
| 35 |
+
| head (K @ D) | 256 @ 256, revived at 24,860, address frozen after | 256 @ 256, born null, never elected | 512 @ 32, never elected (W_s = 0.0 at ship) |
|
| 36 |
+
| context / vocab | 4096 / 256 byte-trigram | 4096 / 256 | 2048 / 256 |
|
| 37 |
+
| params | 237,125,949 | 257,442,109 (+20,316,160, all attention) | 112,524,593 |
|
| 38 |
+
| tokens per step | 16 Γ 4 Γ 4096 = 262,144 | 262,144 | 48 Γ 3 Γ 2048 = 294,912 |
|
| 39 |
+
| steps / tokens | 61,422 / 16,101,408,768 | 61,422 / 16,101,408,768 | 88,508 / 26,102,071,296 |
|
| 40 |
+
| wall-hours / hosts | 78.33 on one RTX PRO 6000 | 49.98 on three hosts | 70.45 |
|
| 41 |
+
| final val bpb (report ledger) | **1.1097** | **2.8846** | 1.121 |
|
| 42 |
+
| optimizer | Muon 2e-2 + pure Adam 3e-4, flat after 200 | same | same |
|
| 43 |
+
| governor | minsep 45Β°, every 8 steps, from birth | inherited (head book only) | off |
|
| 44 |
+
| package at birth β end | 0.8.1 β 0.8.6 (revival under 0.8.2) | 0.8.7 throughout | cc89a87 β 0.6.0 (pre-specials) |
|
| 45 |
+
|
| 46 |
+
Shape and knobs: `presets.py:191-203` (2s), `:214-235` (the `-control`
|
| 47 |
+
derivation and the head_addr_frozen override), `:155-160` (mb1).
|
| 48 |
+
Parameter counts were computed by constructing each preset on the meta
|
| 49 |
+
device (`alephlm.py:218-224` `param_count`): 2s = embed 4,982,784 +
|
| 50 |
+
20 blocks Γ 11,574,275 (hub attention 3,178,496 + bank 8,391,683 +
|
| 51 |
+
norms 4,096) + final norm 2,048 + head 655,617; the control's block
|
| 52 |
+
swaps the hub's 3,178,496 for sdpa's 4,194,304 (qkv 3072Γ1024 + o
|
| 53 |
+
1024Γ1024, no biases; `attention.py:24-32`). Tokens per step is
|
| 54 |
+
`micro_batch Γ grad_accum Γ context` (`trainer.py:186`); 61,422 Γ
|
| 55 |
+
262,144 lands on both manifests' `tokens_seen` exactly. The mother's
|
| 56 |
+
numbers are the exemplar's (`TECHNICAL.md`, `mini-beatrix-1/manifest.json`).
|
| 57 |
+
|
| 58 |
+
## Layout
|
| 59 |
+
|
| 60 |
+
```
|
| 61 |
+
mini-beatrix-2s/
|
| 62 |
+
manifest.json # 14 phases, 142 checkpoint entries, notes
|
| 63 |
+
checkpoints/step_XXXXXXXX.safetensors # 46 bf16 (2000-cadence + every boundary and cap)
|
| 64 |
+
checkpoints/step_00024860_revived.safetensors # fp32, the post-BOUNDARY-WRITE head
|
| 65 |
+
checkpoints/fp8/ # 20 e4m3 ship-only copies
|
| 66 |
+
resume/latest.pt # fp32 model + Muon/Adam + stream + RNG + manifest
|
| 67 |
+
resume/boundary_<phase>_step<N>.pt # 13 immutable boundary archives
|
| 68 |
+
resume/pre_revival_step24860.pt # the state the revival overwrote
|
| 69 |
+
reports/v2/ # 13 boundary + 3 session_cap + revival + 4 cultivation
|
| 70 |
+
runs/ # 5 TensorBoard event files (one per session)
|
| 71 |
+
archive/run0-nospecials/ # the pre-specials first attempt, whole
|
| 72 |
+
beatrix_v2_colab.ipynb # the driver
|
| 73 |
+
|
| 74 |
+
mini-beatrix-2s-control/
|
| 75 |
+
manifest.json # identical phases and budgets; 136 entries
|
| 76 |
+
checkpoints/step_XXXXXXXX.safetensors # 44 bf16 (incl. the 22,854 cap)
|
| 77 |
+
checkpoints/fp8/ # 20 e4m3 ship-only copies
|
| 78 |
+
resume/latest.pt resume/boundary_*.pt # latest + 13 boundary archives
|
| 79 |
+
reports/v2/ # 13 boundary + 2 session_cap (22,854; 53,792)
|
| 80 |
+
runs/ # 6 event files (two are 88-byte aborted starts)
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
The control has no notebook file and no AutoModel packaging; these
|
| 84 |
+
checkpoints are its weights. Files: bf16 475,876,922 B (2s) and
|
| 85 |
+
515,164,234 B (control); fp8 238,219,518 / 257,813,142 B; resume
|
| 86 |
+
archives ~1.904 / ~2.064 GB each; the revived fp32 file 951,692,996 B
|
| 87 |
+
(hub headers read by ranged GET; `checkpoint.py:102-128` for the two
|
| 88 |
+
export rules).
|
| 89 |
+
|
| 90 |
+
## Architecture in detail
|
| 91 |
+
|
| 92 |
+
**The block.** Pre-norm, two residual branches: `x = x + attn(n1(x))`,
|
| 93 |
+
`x = x + bank(n2(x))` (`alephlm.py:37-70`). `attn` is CausalSplatHUB
|
| 94 |
+
when the layer index is in `hub_layers`, else CausalSDPA
|
| 95 |
+
(`alephlm.py:41-50`); every block carries an AnchoredBank
|
| 96 |
+
(`alephlm.py:51`). `hub_ckpt` (0 on 2s: the retained scan fits the
|
| 97 |
+
96 GB card at 237M, `presets.py:184-198`) would wrap the branches in
|
| 98 |
+
activation checkpointing (`alephlm.py:60-68`). The model is embedding β
|
| 99 |
+
20 blocks β final LayerNorm β DualHead (`alephlm.py:81-96`); `forward`
|
| 100 |
+
takes pre-shifted `targets` (ours) or HF-style `labels` (shifted
|
| 101 |
+
internally) and ignores `attention_mask` deliberately
|
| 102 |
+
(`alephlm.py:98-124`).
|
| 103 |
+
|
| 104 |
+
**Trigram embedding and positions.** `e_t = emb0[x_t] + emb1[x_{t-1}] +
|
| 105 |
+
emb2[x_{t-2}] + pos[t]`; the shift tables have a 257th PAD_ROW so the
|
| 106 |
+
two positions shifted in at the sequence start never read a legal byte
|
| 107 |
+
(`embedding.py:19-34`; the pad law β padding with a real byte starved
|
| 108 |
+
address consumption, measured +.05..+.11 on repair). `pos` is a learned
|
| 109 |
+
absolute table `(1, 4096, 1024)` (`embedding.py:28`); there is no RoPE.
|
| 110 |
+
|
| 111 |
+
**CausalSplatHUB β the address.** Each book is an `AlephAddress`: K unit
|
| 112 |
+
anchors in D dims (`address.py:24-36`, with a `home` buffer holding the
|
| 113 |
+
birth copy for drift). The read is a cosine at temperature,
|
| 114 |
+
`u_k = cos(xΜ, Γ’_k)/Ο` with Ο = 0.1 (`address.py:38-40`); the oriented
|
| 115 |
+
form is the 2K-softmax over the axes `(+a_k, βa_k)`, and the signed read
|
| 116 |
+
is its half-difference `w_k = sinh(u_k)/Ξ£_j cosh(u_j)` β inhibition
|
| 117 |
+
(negative w) is first-class; there is no softmax over choices, no
|
| 118 |
+
argmax, no top-k (`address.py:1-13,42-66`). The hub's q and k frames are
|
| 119 |
+
`Linear(d, D)` per book, orthogonal-init; v and o are shared `Linear(d,
|
| 120 |
+
d)` (`attention.py:64-79,106-109`). Construction warns when K > 2D β the
|
| 121 |
+
supply law from ROUND 5e and the mini-beatrix-1 hub census
|
| 122 |
+
(`attention.py:85-92`); 2s runs 64 @ 128, half of the lawful ceiling.
|
| 123 |
+
|
| 124 |
+
**CausalSplatHUB β books and budget.** With `n_const = 4` the four books
|
| 125 |
+
each own a codebook and q/k frames (`attention.py:64-79`); their
|
| 126 |
+
numerators and agreement masses are summed across books before the one
|
| 127 |
+
divide β composition by budget, never softmax over books
|
| 128 |
+
(`attention.py:245-254`; B4 measured comparative composition at β.10).
|
| 129 |
+
`n_const = 1` is the v1 layout with the same state-dict keys
|
| 130 |
+
(`attention.py:93-101`), which is how mini-beatrix-1 checkpoints still
|
| 131 |
+
load.
|
| 132 |
+
|
| 133 |
+
**CausalSplatHUB β the exact chunked scan (prefix memory).**
|
| 134 |
+
`_code_cat_qk` computes q and k for all books in one stacked GEMM,
|
| 135 |
+
folds the row-norm and Ο into one post-GEMM scale, and takes ONE fused
|
| 136 |
+
softmax over `cat([u, βu])` β the module's docstring names this a
|
| 137 |
+
kernel substitution, not a mechanism change: the softmax that exists is
|
| 138 |
+
over the 2K oriented axes of a single query, never over positions
|
| 139 |
+
(`attention.py:154-212`). `_scan_cat` splits the sequence into chunks of
|
| 140 |
+
C = 256 (16 chunks at ctx 4096), forms per-chunk `(2K Γ d)` sums, takes
|
| 141 |
+
the exclusive prefix over chunks as a strictly-lower-triangular GEMM
|
| 142 |
+
with fp32 accumulation, adds the within-chunk causal affinity, and
|
| 143 |
+
divides the numerator by the scalar agreement mass `den`, floored at
|
| 144 |
+
`dtype_floor` (`attention.py:139-151,229-281`; `address.py:16-22`: fp16
|
| 145 |
+
flushes 1e-12 to zero, the measured landmine). Exactness is
|
| 146 |
+
chunk-independent (`forward_naive` is the oracle, `attention.py:320`;
|
| 147 |
+
fp reorder ~1.5e-06). Decode carries a constant-size state per book β
|
| 148 |
+
`{Ξ£ kβΊβv, Ξ£ kβ»βv, Ξ£ kβΊ, Ξ£ kβ»}` β regardless of length
|
| 149 |
+
(`attention.py:284-318`); the generate path falls back to a sliding
|
| 150 |
+
window once the position table is exhausted because "the hub's
|
| 151 |
+
constant-size state cannot evict" (`alephlm.py:180-201`).
|
| 152 |
+
|
| 153 |
+
**The autocast trap.** `torch.einsum` sits in autocast's PROMOTE
|
| 154 |
+
category and `F.normalize`/`exp`/`softmax` on its fp32 list, so one fp32
|
| 155 |
+
operand dragged the whole scan to fp32 (`attention.py:167`); 0.7.2
|
| 156 |
+
(@2be28dd) cast the operands explicitly. Every earlier "bf16" splat run
|
| 157 |
+
had silently run fp32 scans; sdpa was unaffected β so every pre-0.7.2
|
| 158 |
+
splat-vs-sdpa wall-clock comparison favoured sdpa, and accuracy
|
| 159 |
+
verdicts stand (autocast einsum-promote law).
|
| 160 |
+
|
| 161 |
+
**The governor.** `minsep_project_` bounds `|cos(a_i, a_j)| β€
|
| 162 |
+
cos(ΞΈ_min)` for every anchor pair by an exact symmetric geodesic
|
| 163 |
+
push-apart, largest violation first, projected to ΞΈ_min + 0.5Β°
|
| 164 |
+
(`governor.py:35-80`); `govern_model` collects hub and head codebooks,
|
| 165 |
+
runs one batched `bmm` slack check with one device sync, and projects
|
| 166 |
+
only the violating books (`governor.py:92-137`). It is a
|
| 167 |
+
post-optimizer-step projection, never an auxiliary loss
|
| 168 |
+
(`governor.py:11`); the trainer calls it every 8 steps after `opt.step`
|
| 169 |
+
and accumulates pair-pushes into `governor/hits_cum`
|
| 170 |
+
(`trainer.py:263-267,294-296`; `presets.py:99-105`: ΞΈ = 45Β°). Banks are
|
| 171 |
+
excluded by default (a 3-anchor book never crowds). The docstring
|
| 172 |
+
records the small-rig verdict it shipped on (gov25m: fires only when
|
| 173 |
+
needed, census tail cut 5.4Γ, fragile-seed +.007, stable-seed +.002;
|
| 174 |
+
`governor.py:19`). Code-wins note: the journal's covariance program
|
| 175 |
+
describes a *metric* governor through the frame W_a; alephllm's is the
|
| 176 |
+
plain projective form on the D-space books, justified in place β
|
| 177 |
+
"Hub and head codebooks live natively in their own D-dim address space"
|
| 178 |
+
(`governor.py:14`), since the q/k frames project d β D before the cosine
|
| 179 |
+
read (`address.py:38-40`).
|
| 180 |
+
|
| 181 |
+
**CausalSDPA in the twin.** `qkv = Linear(d, 3d)`, `o = Linear(d, d)`,
|
| 182 |
+
both orthogonal-init and bias-free; `F.scaled_dot_product_attention(q,
|
| 183 |
+
k, v, is_causal=True)` over 16 heads of 64 (`attention.py:24-42`);
|
| 184 |
+
the KV cache grows O(n) (`attention.py:43-61`). No QK-norm, no logit
|
| 185 |
+
softcap.
|
| 186 |
+
|
| 187 |
+
**AnchoredBank.** `Bank(x) = trunk(x) + Ξ£_k w_k(x)Β·Ο(g_k)Β·expert_k(x)`
|
| 188 |
+
(`bank.py:3`): a dense trunk `t_out(GELU(t_in(x)))`, three experts
|
| 189 |
+
`(3, 1024, 1024)` with `w_out` zero-initialised (the null path), gates
|
| 190 |
+
at β3.0, and dispatch `w = addr.signed(x)` through a 3-anchor book in d
|
| 191 |
+
dims. `disable_dispatch` returns the trunk alone β bit-identical to the
|
| 192 |
+
dense control at birth (the C6 null); "No balance machinery of any
|
| 193 |
+
kind" (`bank.py:10,41-50`).
|
| 194 |
+
|
| 195 |
+
**DualHead.** `logits = W_h h + Ξ³Β·W_s s(h)`, `s(h) = addr.signed(proj(h))`
|
| 196 |
+
with proj `Linear(1024, 256)`, a 256 @ 256 book, `W_s` zero-initialised
|
| 197 |
+
("weight-zero: self-starting") and Ξ³ = 1 frozen as an ablation knob
|
| 198 |
+
(`head.py:23-51`). The docstring carries the law that set it: on
|
| 199 |
+
mini-beatrix-1 over 8.4B tokens a gamma-gated branch DEADLOCKS while a
|
| 200 |
+
weight-zero bank elected into +2 bpb (`head.py:8-11`; gate-zero deadlock
|
| 201 |
+
law). `disable_aleph` returns `W_h h` alone β the head toggle.
|
| 202 |
+
|
| 203 |
+
**The special-token control plane.** Thirteen ids that no UTF-8 encoder
|
| 204 |
+
can emit β `{0xC0, 0xC1} βͺ [0xF5, 0xFF]`, the overlong leads and the
|
| 205 |
+
leads past U+10FFFF β carry structure inside vocab 256: DOC 0xFF, USER
|
| 206 |
+
0xFE, MODEL 0xFD, END 0xFC, SYS 0xFB, THINK 0xFA, DATA 0xF9, SEP 0xF8,
|
| 207 |
+
MODE 0xF7 (+1 ASCII payload), CUE 0xF6, ESC 0xF5 (+payload = 256
|
| 208 |
+
extended slots; 0x00 the null guard, 0x01 CANARY), RES0 0xC0, RES1
|
| 209 |
+
0xC1 (`special_tokens.py:20-34,50-71`; `assert len(SPECIALS) == 13`).
|
| 210 |
+
The proof is exhaustive β every codepoint 0..0x10FFFF encoded and
|
| 211 |
+
checked β and the Trainer runs it at every byte-craft launch
|
| 212 |
+
(`trainer.py:72-73`). Doctrine: DOC is appended by the packer after
|
| 213 |
+
every document in every phase from birth, identity-gated to the
|
| 214 |
+
byte-trigram tokenizer (`streams.py:59-62,209,355-356`); the chat frame `[SYS] text
|
| 215 |
+
[END] [USER] text [END] [MODEL] text [END]` (`special_tokens.py:112-136`)
|
| 216 |
+
is unforgeable (content is UTF-8) and appears only in the anneal
|
| 217 |
+
(`special_tokens.py:36-44`; chat-in-anneal law). The canary frame moved
|
| 218 |
+
to CUE/SEP because the old QRY=255 collided with DOC
|
| 219 |
+
(`canaries.py:20-29`); `recall_acc` re-baselines at 0.8.0. The trigram
|
| 220 |
+
embedding feeds each special into the next two positions' input
|
| 221 |
+
representations by construction.
|
| 222 |
+
|
| 223 |
+
## The recipe (pinned from source; unchanged cc89a87 β b472815)
|
| 224 |
+
|
| 225 |
+
- **Split**: every parameter with `ndim β {2, 3}` that is not an
|
| 226 |
+
`nn.Embedding` weight rides Muon; everything else β the three byte
|
| 227 |
+
tables, LayerNorms, bank gates, Ξ³, the head bias β rides
|
| 228 |
+
`torch.optim.Adam(weight_decay=0.0)` (`optim.py:54-79`; "pure Adam,
|
| 229 |
+
wd=0 β never AdamW", `presets.py:82`). Measured split for 2s: Muon 385
|
| 230 |
+
tensors / 236,253,184 numel, Adam 107 / 872,765; the control's Muon
|
| 231 |
+
set is 145 / 256,569,344 with the same Adam set. Code-wins note: the
|
| 232 |
+
`optim.py:4-5,55-58` docstrings say positions ride Adam; the code
|
| 233 |
+
sends `embed.pos` (shape `(1, 4096, 1024)`, ndim 3, not an
|
| 234 |
+
`nn.Embedding`) to Muon (`optim.py:68`), verified on the meta device.
|
| 235 |
+
The journal already records the code's behaviour.
|
| 236 |
+
- **Muon**: 5-step NewtonβSchulz (a, b, c = 3.4445, β4.7750, 2.0315) on
|
| 237 |
+
the momentum buffer after whole-matrix Frobenius normalisation
|
| 238 |
+
`X/(βXβ_F + 1e-7)`; plain momentum 0.95, no Nesterov, no shape
|
| 239 |
+
scaling; 3-D stacks orthogonalised per slice (`optim.py:20-51`).
|
| 240 |
+
- **Learning rates**: Muon 2e-2, Adam 3e-4; `lr_scale = (step+1)/200`
|
| 241 |
+
for 200 steps, then 1.0 forever (`optim.py:82-93`, `presets.py:80-83`).
|
| 242 |
+
The tfevents read `train/lr_scale` 0.25/0.50/0.75 at steps 50/100/150
|
| 243 |
+
and 1.000 from step 200 to the last log on both crafts, through both
|
| 244 |
+
anneal phases: the anneal is a diet change, not a decay.
|
| 245 |
+
- **Clip**: `clip_grad_norm_(β¦, 1.0)` over all parameters; the logged
|
| 246 |
+
`train/grad_norm` is the PRE-clip norm it returns (`trainer.py:244,
|
| 247 |
+
287`). Because Muon normalises the momentum buffer before
|
| 248 |
+
orthogonalising, its update magnitude is independent of gradient
|
| 249 |
+
scale: clipping never changes the magnitude of a Muon step; it only reweights
|
| 250 |
+
steps inside the momentum buffer. The Adam set (`torch.optim.Adam`,
|
| 251 |
+
wd 0, default eps 1e-8; `optim.py:79`) is per-coordinate
|
| 252 |
+
scale-normalised (`m/sqrt(v)`), so under a persistent clip its step
|
| 253 |
+
magnitude is likewise unchanged and the clip only reweights the two
|
| 254 |
+
moment averages (Muon is clip-blind; Adam is nearly so).
|
| 255 |
+
- **Precision and batch**: fp32 masters, bf16 autocast on CUDA, TF32 on
|
| 256 |
+
(`trainer.py:14-15,65-66,239`); micro-batch 16 Γ accum 4 Γ ctx 4096;
|
| 257 |
+
seed 1337 (`presets.py:97,199`); each micro-batch row is 4097 ids
|
| 258 |
+
scored at 4096 positions.
|
| 259 |
+
- **Guards**: a non-finite loss or gradient raises before any optimizer
|
| 260 |
+
step, weights untouched and the resume state not overwritten
|
| 261 |
+
(`trainer.py:250`); the spike flag `loss > 2Β·EMA + 0.5` is logged
|
| 262 |
+
only (`trainer.py:273-275,293`). Neither fired on either craft.
|
| 263 |
+
- **Cadences**: log 50, health census 500, eval and checkpoint 2000,
|
| 264 |
+
fp8 every 5th checkpoint, TB upload 1000, val 262,144 tokens, 128
|
| 265 |
+
canary episodes (`presets.py:88-96`).
|
| 266 |
+
- **Absent by design**: no QK-norm, no logit softcap, no z-loss, no
|
| 267 |
+
weight decay, no dropout, no RoPE, no LR decay in any phase.
|
| 268 |
+
|
| 269 |
+
## The diet (all on one gauge: fineweb-edu holdout bpb)
|
| 270 |
+
|
| 271 |
+
| phase | dataset | budget | β step | 2s bpb | control bpb | note |
|
| 272 |
+
|---|---|---|---|---|---|---|
|
| 273 |
+
| warmup_wikitext | wikitext-103 | 0.30B | 1,145 | 2.3626 | 2.1056 | sdpa led by 0.257; protocol pre-registered here |
|
| 274 |
+
| fineweb_main | fineweb-edu | 5.0B | 20,219 | **1.1172** | 1.1763 | the crossover; hubs +2.22 / banks +3.55 |
|
| 275 |
+
| fineweb_extended | fineweb-edu | 10B | β | β | β | DEFERRED, unspent on both |
|
| 276 |
+
| curriculum_s0 | curriculum-s0 | 0.7B | 22,890 | 1.2522 | 1.5047 | |
|
| 277 |
+
| REVIVAL | β | β | 24,860 | 1.2768 β 1.2339 (op gauge) | β | BOUNDARY-WRITE; `pre_revival_step24860.pt` archived |
|
| 278 |
+
| curriculum_s1 | curriculum-s1 | 0.7B | 25,561 | 1.2544 | 1.3917 | first live head toggle in the lineage, +0.0258 |
|
| 279 |
+
| curriculum_s2 | curriculum-s2 | 1.0B | 29,376 | 1.2177 | 1.3938 | 2s grad-norm curriculum+anneal max 0.619 @26,750 |
|
| 280 |
+
| curriculum_s3 | curriculum-s3 | 1.2B | 33,954 | 1.2318 | 1.5317 | |
|
| 281 |
+
| curriculum_s4 | curriculum-s4 | 1.2B | 38,532 | 1.2356 | 1.5689 | both 2s mechanisms above +3.0 |
|
| 282 |
+
| curriculum_s5 | curriculum-s5 | 1.0B | 42,347 | 1.2504 | 2.1762 | control grad-norm median 126 |
|
| 283 |
+
| curriculum_s6 | curriculum-s6 | 0.8B | 45,399 | 1.2683 | 3.1940 | control max 41,070 @44,600 |
|
| 284 |
+
| curriculum_s7 | curriculum-s7 | 1.4B | 50,740 | 1.1883 | 3.1201 | |
|
| 285 |
+
| curriculum_s8 | curriculum-s8 | 0.8B | 53,792 | *1.0737* (synthetic gauge) / **1.1911** (cap) | 3.0810 | 2s row STRUCK; true ledger = `session_cap_step53792` |
|
| 286 |
+
| anneal_nochat | anneal-nochat | 1.0B | 57,607 | 1.1153 | 2.9270 | the pre-chat baseline; probe mean .389 |
|
| 287 |
+
| anneal_mix | anneal-mix | 1.0B | 61,422 | **1.1097** | **2.8846** | run best; probe mean .389 == pre-chat |
|
| 288 |
+
|
| 289 |
+
Phases and budgets: `presets.py:132-146` (the preset plan carries
|
| 290 |
+
anneal_mix at 2B deferred; the notebook's C6 cell rewrote it to 1B and
|
| 291 |
+
inserted anneal_nochat before it β "two-phase anneal activated: nochat
|
| 292 |
+
1B then chat 1B (plan of record 2026-08-31)" in both manifests' notes; the
|
| 293 |
+
hub copy of `beatrix_v2_colab.ipynb`, uploaded 2026-08-28T23:00Z with cells
|
| 294 |
+
C1βC5 and pinned to `geolip-alephllm @2a902b7` (0.8.1), predates the
|
| 295 |
+
amendment and carries no C6 cell),
|
| 296 |
+
`curriculum.py:657-663` (STAGE_TOKENS), `curriculum.py:666-689`
|
| 297 |
+
(append at runtime). Manifest order is not chronology: the S0βS8 phases
|
| 298 |
+
are listed after the anneals but ran before them, because
|
| 299 |
+
`current_phase()` activates the first `planned` phase and deferred
|
| 300 |
+
phases are invisible until flipped (`manifest.py:33-41`); count back
|
| 301 |
+
from the totals (16,101,408,768 β anneal_mix's 1,000,079,360 lands on
|
| 302 |
+
step 57,607) rather than reading the list.
|
| 303 |
+
|
| 304 |
+
**The mixes** (`curriculum.py:614-655`; weights sum to 1): s0
|
| 305 |
+
tinystories .74, simple-wiki .20, aochildes .01, recall-synth .03,
|
| 306 |
+
counting-synth .02 Β· s1 perspective-synth .30, tinystories .28,
|
| 307 |
+
simple-wiki .20, cosmo-young .12, concept-synth .04, recall-synth .03,
|
| 308 |
+
siqa-narrative .02, aochildes .01 Β· s2 concept-synth .30, cosmo-young
|
| 309 |
+
.30, simple-wiki .12, tinystories .10, definitions .08, wikipedia-en
|
| 310 |
+
.07, recall-synth .03 Β· s3 rulechain-synth .32, proofwriter-prose .22,
|
| 311 |
+
simple-wiki .12, tinystories .10, wikipedia-en .08, cosmo-young .08,
|
| 312 |
+
concept-synth .04, recall-synth .03, babi-prose .01 Β· s4 arith-synth
|
| 313 |
+
.35, cosmo-young .32, simple-wiki .12, tinystories .08, rulechain-synth
|
| 314 |
+
.06, counting-synth .04, recall-synth .03 Β· s5 causal-synth .32,
|
| 315 |
+
atomic-flicker .25, tinystories .12, simple-wiki .10, gutenberg .08,
|
| 316 |
+
cosmo-young .07, recall-synth .05, siqa-narrative .01 Β· s6
|
| 317 |
+
tryfail-synth .32, tinystories .22, arith-synth .13, cosmo-young .10,
|
| 318 |
+
simple-wiki .10, causal-synth .08, recall-synth .05 Β· s7 tinystories
|
| 319 |
+
.10, cosmo-young .10, rulechain-synth .10, arith-synth .10, fineweb-good
|
| 320 |
+
.10, simple-wiki .08, atomic-flicker .08, causal-synth .08,
|
| 321 |
+
tryfail-synth .08, proofwriter-prose .08, recall-synth .05,
|
| 322 |
+
perspective-synth .04, babi-prose .01 Β· s8 definitions .25,
|
| 323 |
+
register-synth .25, simple-wiki .15, cosmo-young .12, wikipedia-en .10,
|
| 324 |
+
tinystories .09, recall-synth .03, siqa-narrative .01. Rails audited at
|
| 325 |
+
append time: MAX_EPOCHS 4.0, MAX_GENERATOR_SHARE 0.35, MIN_BALLAST 0.30
|
| 326 |
+
natural text (`curriculum.py:548-551,565-596`); holdouts enforced at
|
| 327 |
+
the generator β bAbI tasks 16/19, ProofWriter depth 5, rule-chain
|
| 328 |
+
depth 5, three-digit subtraction (`curriculum.py:26-31`). No chat
|
| 329 |
+
template in s0βs8.
|
| 330 |
+
|
| 331 |
+
**The anneal recipes** (`streams.py:68-77`): ANNEAL_MIX = fineweb-edu
|
| 332 |
+
.45, cosmopedia .20, tinystories .15, soda-dialogue .12,
|
| 333 |
+
beatrix-texture-sp .05 (the specials-native SYS/USER/MODEL/END
|
| 334 |
+
identity texture), recall-synth .03; ANNEAL_NOCHAT is the same list
|
| 335 |
+
minus beatrix-texture-sp, renormalised by MixStream β so the nochat
|
| 336 |
+
boundary is the exact pre-chat baseline on the same distribution.
|
| 337 |
+
MixStream draws one component per batch from a counter-seeded RNG
|
| 338 |
+
(`streams.py:375-404`): reproducible and resumable, but its state
|
| 339 |
+
carries no recipe fingerprint (fresh starts only across recipe
|
| 340 |
+
changes).
|
| 341 |
+
|
| 342 |
+
**The DOC plane on the diet.** The packer appends DOC after every
|
| 343 |
+
document (`streams.py:192-372`); the specials gauge reads `doc_bpb`
|
| 344 |
+
(bits at positions whose target is DOC) and `reset_bpb` (mean bits over
|
| 345 |
+
the 16 targets after each DOC) on the fixed val batches
|
| 346 |
+
(`instruments.py:310-352`). doc_bpb fell 12.83 β 3.29 by 0.52B and
|
| 347 |
+
sits at 3.862 at the endpoint (control 5.422); reset_bpb 4.287 β 2.130
|
| 348 |
+
(control 3.825). doc_count is 70 in every 2s and control report except the struck 2s
|
| 349 |
+
`curriculum_s8_step53792.json`, which reads doc_count 1612 (the synthetic
|
| 350 |
+
fallback stream) β the same 70 documents every time on the fineweb-edu gauge.
|
| 351 |
+
|
| 352 |
+
**The val gauge.** `_val()` builds 4 batches of (16, 4097) from
|
| 353 |
+
`build_stream(dataset, β¦, seed = 1337 + 9999, role="val")`
|
| 354 |
+
(`trainer.py:384-411`); for fineweb-edu the val region is the first
|
| 355 |
+
2,048 unshuffled rows, which the training stream skips
|
| 356 |
+
(`streams.py:56,279-282`), and `build_stream` routes every
|
| 357 |
+
`curriculum-*` and `anneal-*` val request to that same fineweb-edu head
|
| 358 |
+
holdout β "gauge continuity" in the source (`streams.py:423-447`). So
|
| 359 |
+
from fineweb_main onward every row of both crafts is one gauge; the
|
| 360 |
+
canon's CB2/CB3 "own slice" wording is struck by erratum
|
| 361 |
+
(`canon/splat_aleph_battery.md:583`). **The s8 fallback caveat**: at the
|
| 362 |
+
terminal curriculum boundary `current_phase()` is None (everything done
|
| 363 |
+
or deferred) and pre-0.8.5 `_val` fell to the `synthetic` stream, so the
|
| 364 |
+
2s `curriculum_s8_step53792` ledger is an incomparable artifact; 0.8.5
|
| 365 |
+
(@7d48328) makes `_val` fall back to the last DONE phase's dataset
|
| 366 |
+
(`trainer.py:397-401`), and the true end-of-curriculum ledger is
|
| 367 |
+
`session_cap_step53792`, taken at the same step on the restored gauge
|
| 368 |
+
(1.1911; banks +3.214, hubs +3.341, head +0.0149). The control, born
|
| 369 |
+
under 0.8.7, measured its s8 row on the repaired gauge; the s8 pair is
|
| 370 |
+
therefore not same-gauge and is struck from the paired count.
|
| 371 |
+
|
| 372 |
+
## The life (`reports/v2/`, `resume/`, `runs/`)
|
| 373 |
+
|
| 374 |
+
**Sessions.** The 2s craft ran five sessions on the one RTX PRO 6000
|
| 375 |
+
(Blackwell, 96 GB; 4.54 s/step, ~58k tok/s), each its own event file:
|
| 376 |
+
steps 50β8,300 (host 4d130c0e7351), 8,350β24,850 (91b9f3518593),
|
| 377 |
+
24,900β43,100 (e935248f0778), 43,150β53,750 (68304a2b8506),
|
| 378 |
+
53,800β61,400 (c2dc3d694638). The breaks are the session caps at 8,320
|
| 379 |
+
(fineweb, 2.181B), 43,130 (inside s6, 11.306B) and 53,792 (the s8 boundary
|
| 380 |
+
itself, the resumed session's first action) β `session_cap_step*.json`
|
| 381 |
+
is written when a session hits MAX_HOURS mid-phase β and the revival
|
| 382 |
+
halt at 24,860. Sessions are boundary-exact: the driver calls
|
| 383 |
+
`train(max_hours, stop_at_boundary=True)`, which returns after each
|
| 384 |
+
boundary checkpoint with `_last_boundary` set, and never auto-resumes a
|
| 385 |
+
manual stop (`_interrupted`; `trainer.py:170-184,328-335`; the 0.8.1
|
| 386 |
+
driver contract, minted after run0 rolled through its boundaries and
|
| 387 |
+
swallowed a KeyboardInterrupt).
|
| 388 |
+
|
| 389 |
+
**Resume.** `_restore` loads `resume/latest.pt` β fp32 model, both
|
| 390 |
+
optimizer states, the stream position, torch and CUDA RNG, and the
|
| 391 |
+
manifest snapshot embedded in the payload, which wins over the hub's
|
| 392 |
+
`manifest.json` (`trainer.py:110-153`; `checkpoint.py:131-175`). A
|
| 393 |
+
missing payload over a manifest with steps > 0 refuses to start fresh
|
| 394 |
+
(`trainer.py:110-120`). Every phase boundary archives an immutable copy
|
| 395 |
+
`resume/boundary_<phase>_step<N>.pt` (`trainer.py:469`;
|
| 396 |
+
`checkpoint.py:134-154`). Resume is not row-exact: the datasets shuffle
|
| 397 |
+
buffer (10,000 rows, `streams.py:292-293`) is not part of the saved
|
| 398 |
+
state, so each resume refills it (`streams.py:12`, the honesty note).
|
| 399 |
+
|
| 400 |
+
**The revival at 24,860.** The born-null head buried itself inside the
|
| 401 |
+
first 2,000 steps (0.032Γ chance liveness at the first stored
|
| 402 |
+
checkpoint; βsβ ~6e-4; proj cosines ~60Γ below chance β learned
|
| 403 |
+
avoidance; `repos/alephllm.md:1564-1643` β journal readings from
|
| 404 |
+
checkpoint analysis: the craft's TB carries no `head/liveness_ratio`
|
| 405 |
+
scalar before 0.8.4; the control's TB reads 0.052Γ @2,000, 0.014Γ
|
| 406 |
+
@12,000, 0.010Γ @24,000). The BOUNDARY-WRITE op
|
| 407 |
+
(`revival.py:44-104`) ran on the halted run: proj β uncentered
|
| 408 |
+
whitening-PCA of the final-norm outputs (sv floor 1e-3 relative);
|
| 409 |
+
codebook β fresh `normalize(randn)` with min-sep 45Β° (0 pushes needed);
|
| 410 |
+
W_s β fp64 ridge on the base head's CE residual, Ξ± swept on the held-out
|
| 411 |
+
half and folded in. `revival_step24860.json`: K 256, D 256, seed 13,
|
| 412 |
+
sv0 3676.66, svD 15.90, Ξ± 8.0, βW_sβ 481.53, fit_positions 65,536,
|
| 413 |
+
base_bpb 1.27682 β revived_bpb 1.23391 (Ξ β0.04290 held-out),
|
| 414 |
+
liveness 0.01027Γ β 0.99642Γ, version 0.8.2, commit d6e1854. The
|
| 415 |
+
pre-revival resume was archived, Muon momentum zeroed for the three head
|
| 416 |
+
tensors, and the revived payload promoted on Phil's explicit go
|
| 417 |
+
(`repos/alephllm.md:1669`); thereafter `head_addr_frozen` freezes proj
|
| 418 |
+
and the head codebook by `requires_grad` alone, so optimizer groups and
|
| 419 |
+
resume state are unchanged and Muon skips them (`trainer.py:75-80`;
|
| 420 |
+
`optim.py:42-43`). `checkpoints/step_00024860.safetensors` is the
|
| 421 |
+
pre-revival bf16 weight; `step_00024860_revived.safetensors` (fp32) is
|
| 422 |
+
the post-write weight; neither the revived file nor the resume archives
|
| 423 |
+
appear in `manifest.json`'s checkpoint index.
|
| 424 |
+
|
| 425 |
+
**The anneal.** Activated 2026-08-31T09:54Z after s8; nochat done
|
| 426 |
+
14:42Z, chat done 19:31Z; the frame cost nothing in a bracketed phase
|
| 427 |
+
(probe mean .389 β .389; val 1.1153 β 1.1097).
|
| 428 |
+
|
| 429 |
+
**The 2s boundary ledgers** (`reports/v2/*.json`; bpb on the fineweb-edu
|
| 430 |
+
holdout; toggles = mechanism-off minus full):
|
| 431 |
+
|
| 432 |
+
| report | step | tokens | bpb_full | banks | hubs | head | doc / reset |
|
| 433 |
+
|---|---|---|---|---|---|---|---|
|
| 434 |
+
| warmup_wikitext | 1,145 | 0.300B | 2.36257 | +0.0365 | **+1.5042** | β2.3e-06 | 12.83 / 4.287 |
|
| 435 |
+
| session_cap | 8,320 | 2.181B | 1.20332 | +1.727 | +2.036 | β4.9e-07 | |
|
| 436 |
+
| fineweb_main | 20,219 | 5.300B | 1.11719 | +3.5527 | +2.2157 | β3.4e-07 | |
|
| 437 |
+
| curriculum_s0 | 22,890 | 6.000B | 1.2522 | | | ~0 (born null) | |
|
| 438 |
+
| curriculum_s1 | 25,561 | 6.700B | 1.2544 | | | **+0.0258** | |
|
| 439 |
+
| curriculum_s2 | 29,376 | 7.700B | 1.2177 | | | +0.0140 | |
|
| 440 |
+
| curriculum_s3 | 33,954 | 8.900B | 1.2318 | | | +0.0072 | |
|
| 441 |
+
| curriculum_s4 | 38,532 | 10.100B | 1.2356 | >3.0 | >3.0 | +0.0130 | |
|
| 442 |
+
| curriculum_s5 | 42,347 | 11.100B | 1.2504 | | | +0.0278 | |
|
| 443 |
+
| session_cap | 43,130 | 11.306B | 1.2614 | | | | |
|
| 444 |
+
| curriculum_s6 | 45,399 | 11.900B | 1.2683 | | | +0.0126 | |
|
| 445 |
+
| curriculum_s7 | 50,740 | 13.300B | 1.1883 | | | β0.0010 | |
|
| 446 |
+
| curriculum_s8 (STRUCK) | 53,792 | 14.100B | *1.0737* | *+3.396* | *+5.164* | | *0.843 / 1.889* |
|
| 447 |
+
| session_cap (true s8) | 53,792 | 14.100B | **1.19112** | +3.2144 | +3.3410 | +0.0149 | 5.052 / 2.469 |
|
| 448 |
+
| anneal_nochat | 57,607 | 15.100B | 1.1153 | | | +0.0101 | |
|
| 449 |
+
| anneal_mix | 61,422 | 16.101B | **1.10970** | +3.3115 | +3.3584 | +0.0123 | 3.862 / 2.130 |
|
| 450 |
+
|
| 451 |
+
Blank cells are not in the record consulted here; the head column is
|
| 452 |
+
complete. The warmup row is the first full-splat-under-AR ledger: with
|
| 453 |
+
all 20 hubs off the craft loses its entire attention (+1.50 at 0.3B),
|
| 454 |
+
which is what "load-bearing, not decorative" measures against the
|
| 455 |
+
mother's 3-hub form. `hidden_erank_floor` flags TRUE in every report from the 8,320 cap on (the
|
| 456 |
+
trainer's own TB census, on the first val batch, flags it from step 500)
|
| 457 |
+
(L19 pre-norm erank in single digits β the byte funnel; the post-norm
|
| 458 |
+
`nf(h)` erank reads 334/1024), `head_buried` FALSE from s1 on, den
|
| 459 |
+
floors 0.0 the entire run, zero loss spikes. Two "final val" readings
|
| 460 |
+
exist on the hub: the report ledger 1.10970 (the gauge of record) and
|
| 461 |
+
the manifest's checkpoint entry `val_bpb` 1.11403 at 61,422 β the
|
| 462 |
+
manifest column is the LAST PERIODIC EVAL's `bpb_full` (eval every
|
| 463 |
+
2,000; `trainer.py:444-447`), so it lags every boundary (fineweb 1.1156
|
| 464 |
+
vs 1.1172; s8 1.1833 vs 1.0737/1.1911); the manifest minimum is 1.1140
|
| 465 |
+
@60,000. Cite the report.
|
| 466 |
+
|
| 467 |
+
## The paired record (`mini-beatrix-2s-control/reports/v2/`)
|
| 468 |
+
|
| 469 |
+
**Derivation.** Every preset auto-generates a `-control` twin: the same
|
| 470 |
+
`AlephLMConfig` with `hub_layers = ()` (`presets.py:216-229`), so
|
| 471 |
+
`hub_K/hub_D/hub_const/hub_chunk` are carried but unused and every
|
| 472 |
+
block's attention is CausalSDPA. 0.8.7 (@b472815) gave twins their own
|
| 473 |
+
`TrainConfig` copy β the shared instance had let `head_addr_frozen=True`
|
| 474 |
+
leak into the control, a post-revival flag no birth recipe may carry β
|
| 475 |
+
and sets the 2s control to `head_addr_frozen = False` explicitly: the
|
| 476 |
+
birth recipe, born-null unfrozen head, which buries as the treatment's
|
| 477 |
+
did for its first 24,860 steps (`presets.py:221-235`). The control
|
| 478 |
+
inherits `governor="minsep"`; with no hub books it governs the head
|
| 479 |
+
codebook only (hits_cum 10 at warmup, fineweb and the 22,854 cap β the first
|
| 480 |
+
session β and 0 in every later session's report;
|
| 481 |
+
`crowd_check_extra_hits` 0). Same seed 1337: construction order embed β
|
| 482 |
+
blocks β norm β head after `torch.manual_seed` (`trainer.py:63`;
|
| 483 |
+
`alephlm.py:85-96`) makes the embedding draws identical between craft
|
| 484 |
+
and twin; the RNG streams diverge from block 0's attention onward
|
| 485 |
+
(a derivation from construction order, not a weight comparison).
|
| 486 |
+
|
| 487 |
+
**Protocol.** Declared at the control's warmup boundary, before the
|
| 488 |
+
outcome, while sdpa led (2.1056 vs 2.3626): the architecture verdict is
|
| 489 |
+
the paired boundary table across the full 16.1B matched budget; the
|
| 490 |
+
headline is endpoint val; secondaries are probe means at matched
|
| 491 |
+
boundaries and the doc/reset gauges; no verdict from any intermediate
|
| 492 |
+
lead in either direction (`repos/alephllm.md:1961-2002`; a control must
|
| 493 |
+
be able to fail). Completed unaltered through 13 paired rows. Budgets
|
| 494 |
+
are matched in tokens, not wall-clock: fused sdpa ran 71.9k tok/s on the
|
| 495 |
+
RTX PRO 6000 and 113k on the H100 against the eager scan's ~58k.
|
| 496 |
+
|
| 497 |
+
**The paired rows** (bpb_full, same step, same tokens, same gauge except
|
| 498 |
+
where struck; deficit = control β craft):
|
| 499 |
+
|
| 500 |
+
| boundary | step | tokens | 2s | control | deficit | control bank toggle | control grad-norm median |
|
| 501 |
+
|---|---|---|---|---|---|---|---|
|
| 502 |
+
| warmup | 1,145 | 0.30B | 2.3626 | 2.1056 | β0.257 | β0.008 | |
|
| 503 |
+
| fineweb | 20,219 | 5.30B | 1.1172 | 1.1763 | +0.059 | 1.62 | 0.31 |
|
| 504 |
+
| s0 | 22,890 | 6.00B | 1.2522 | 1.5047 | +0.253 | 3.44 | 6.47 |
|
| 505 |
+
| s1 | 25,561 | 6.70B | 1.2544 | 1.3917 | +0.137 | 3.61 | 2.43 |
|
| 506 |
+
| s2 | 29,376 | 7.70B | 1.2177 | 1.3938 | +0.176 | 4.13 | 11.9 |
|
| 507 |
+
| s3 | 33,954 | 8.90B | 1.2318 | 1.5317 | +0.300 | 4.73 | 12.7 |
|
| 508 |
+
| s4 | 38,532 | 10.10B | 1.2356 | 1.5689 | +0.333 | 5.09 | 24.5 |
|
| 509 |
+
| s5 | 42,347 | 11.10B | 1.2504 | 2.1762 | **+0.926** | 4.36 | 126.4 |
|
| 510 |
+
| s6 | 45,399 | 11.90B | 1.2683 | 3.1940 | **+1.926** | 3.89 | 471 |
|
| 511 |
+
| s7 | 50,740 | 13.30B | 1.1883 | 3.1201 | +1.932 | 4.17 | 162 |
|
| 512 |
+
| s8 | 53,792 | 14.10B | *1.0737* | 3.0810 | STRUCK (gauges differ) | 5.48 | 145 |
|
| 513 |
+
| anneal_nochat | 57,607 | 15.10B | 1.1153 | 2.9270 | +1.812 | 5.64 | 167 |
|
| 514 |
+
| anneal_mix | 61,422 | 16.10B | **1.1097** | **2.8846** | **+1.775** | 6.11 | 215 |
|
| 515 |
+
|
| 516 |
+
Eleven same-gauge rows after the crossover, all in the craft's favour;
|
| 517 |
+
the endpoint gap is +1.775 at 16,101,408,768 tokens on both. The
|
| 518 |
+
control's own trajectory is the finding: 1.1763 at 5.3B β 3.1940 at s6
|
| 519 |
+
β 2.8846 at the end β a destabilised model at its endpoint, not a
|
| 520 |
+
trailing one. Its endpoint ledger: bank_off 8.99015 (toggle +6.1056),
|
| 521 |
+
head toggle β1.06e-05, `head_buried` TRUE at every row from fineweb on,
|
| 522 |
+
doc_bpb 5.422, reset_bpb 3.825; probes p0 .200 p1 .333 p2 .267 p3 .333
|
| 523 |
+
p4 .300 p5 .433 p6 .233 p7 .433 p8 .333 against the craft's .367 .367
|
| 524 |
+
.600 .200 .333 .567 .333 .333 .400 β ties at n = 30 (SE of a difference
|
| 525 |
+
.126). The control's bank dispatch entropy fell through the curriculum
|
| 526 |
+
(`collapse/dispatch_entropy_L5`: .94 at the s0 census, .75 @30,000, .48
|
| 527 |
+
@35,000, .22 at the s5 census; L0 .75 β .47 @30,000; layer minimum .14
|
| 528 |
+
@42,500), ahead of the >30 grad-norm-median onset at 38,000.
|
| 529 |
+
|
| 530 |
+
**The destabilisation facts** (from the hub tfevents, `train/grad_norm`
|
| 531 |
+
= pre-clip): the control first crossed the clip at step 17,600 (4.61B,
|
| 532 |
+
inside fineweb, on the RTX PRO 6000, while still leading); 813 of 824
|
| 533 |
+
logged curriculum+anneal steps clipped (98.7%); 38 steps above 1,000;
|
| 534 |
+
max 41,070 @44,600; the > 30 onset at bin 38,000 (late s4) and > 100 at
|
| 535 |
+
40,500 (mid-s5) inside one event file. The 2s craft: 0 of 1,225
|
| 536 |
+
logged steps from step 200 on above the clip (the only reading above 1.0 is
|
| 537 |
+
step 50, 1.063, inside the 200-step LR warmup); max 0.760 @1,300 after the
|
| 538 |
+
warmup boundary, and 0.619 @26,750 (s2) over the 824 curriculum+anneal steps. The spike
|
| 539 |
+
detector and the non-finite guard never fired on either craft. Under
|
| 540 |
+
Muon the clip braked nothing in the Muon group (qkv/o, the position
|
| 541 |
+
table, the head); the escalating norm is a sharpness signal, not a
|
| 542 |
+
throttled step.
|
| 543 |
+
|
| 544 |
+
**The grade and its caveats.** The instance holds: the all-sdpa twin is
|
| 545 |
+
a destabilised control under the splat-tuned recipe, one craft per arm
|
| 546 |
+
(S-1802 INSTANCE). The general claim β softmax attention is inadequate
|
| 547 |
+
for this model's attention requirement β is a CANDIDATE (S-1803) behind
|
| 548 |
+
confounds stated as measurements: (a) the recipe was tuned on the splat
|
| 549 |
+
craft and carries none of the field's softmax stabilisers (QK-norm,
|
| 550 |
+
softcap, z-loss, weight decay, decay), and the signature β clip-engaged
|
| 551 |
+
smooth degradation, entropy and rank collapse, no NaN, onset on plain
|
| 552 |
+
prose while leading β is the documented softmax logit-growth mode;
|
| 553 |
+
(b) n = 1, no bar, +20.3M parameters in the losing arm; (c) shape: the
|
| 554 |
+
mother null β mini-beatrix-1 (13 sdpa + 3 hubs, d768) under the
|
| 555 |
+
identical TrainConfig (git cc89a87..b472815; the manifests record no
|
| 556 |
+
train_config) through the same nine stages at the same budgets ran a
|
| 557 |
+
post-warmup grad-norm max of 0.883, 0 of 1,750 logged steps clipped
|
| 558 |
+
(`canon/splat_aleph_battery.md:955`; canon-cited, not re-verified against
|
| 559 |
+
the mini-beatrix-1 event files for this companion β her 88,508 steps /
|
| 560 |
+
26,102,071,296 tokens / 70.45 h do verify against
|
| 561 |
+
`mini-beatrix-1/manifest.json`) β with the nuance that v1 entered
|
| 562 |
+
s0 at 17.3B tokens after 15B prose and a 2B anneal, the control at 5.3B;
|
| 563 |
+
(d) the aleph-addressed bank sits in both crafts, its dispatch collapse
|
| 564 |
+
precedes the escalation, and no sdpa + dense-FFN arm exists at any
|
| 565 |
+
scale; (e) the 4096Γ1024 position table on WD-free Muon at 2e-2 forever;
|
| 566 |
+
(f) house Muon is not reference Muon; (g) host and row-order
|
| 567 |
+
asymmetries. One gauge (val bpb; probes are ties); the s8 pair struck;
|
| 568 |
+
three hosts. dist-e002 (vision, two seeds) remains the only Law-E
|
| 569 |
+
splat-vs-sdpa endpoint, and it is a tie.
|
| 570 |
+
|
| 571 |
+
**Provenance of the three hosts.** RTX PRO 6000 from birth to the 23.5 h
|
| 572 |
+
cap at 22,854 (event file 50β22,850); a Colab H100 80 GB from 22,854
|
| 573 |
+
(~2.3β2.4 s/step) through s0βs8 (22,900β53,750), 36 steps before the s0
|
| 574 |
+
boundary, with the shuffle buffers refilled; a session 53,800β56,000
|
| 575 |
+
follows the 53,792 cap in the H100 session's own host and process (event
|
| 576 |
+
file `β¦0a706d592fec.620.1`, 20:01β21:35Z, ~2.3 s/step, 114k tok/s); a third
|
| 577 |
+
host (2b1783ef9e94) opened 2026-09-03T01:41Z from the step-56,000
|
| 578 |
+
checkpoint at ~3.6 s/step and ran both anneal rows (56,050β61,400; two
|
| 579 |
+
88-byte event files are aborted starts) β a replay of up to 999 steps at
|
| 580 |
+
that restart is a hedge on the 1,000-step TB-upload cadence, not a hub
|
| 581 |
+
reading (the H100 process's last event is 21:35:20Z, the step-56,000
|
| 582 |
+
checkpoint landed 21:35:33Z and its manifest 21:36:06Z, the third host
|
| 583 |
+
opened 01:41:15Z; the hub can neither confirm nor refute a replay), and
|
| 584 |
+
it left no cap report.
|
| 585 |
+
The canon's "on the H100" lines for boundaries 11β12 are wrong by
|
| 586 |
+
erratum (`canon/splat_aleph_battery.md:1008`). The escalation began
|
| 587 |
+
pre-resume on the RTX PRO 6000; a 36-step data-order change cannot make
|
| 588 |
+
a 0.25 bpb gap on a fixed eval.
|
| 589 |
+
|
| 590 |
+
## The boundary report (`reports/v2/`) β schema
|
| 591 |
+
|
| 592 |
+
Written by the notebook's `boundary_report(run, tag)` cell to
|
| 593 |
+
`reports/v2/{tag}_step{step}.json`, tag = the phase that just finished
|
| 594 |
+
or `session_cap`:
|
| 595 |
+
|
| 596 |
+
- `tag`, `step`, `tokens` β `tokens` is `manifest.tokens_seen` (new
|
| 597 |
+
in the v2 record).
|
| 598 |
+
- `probes` β `{p0_grounding, p1_self_other, p2_difference, p3_logic,
|
| 599 |
+
p4_arithmetic, p5_causal, p6_learning, p7_composite,
|
| 600 |
+
p8_articulation}`, each `{acc, families{β¦}, n: 30}`; byte-NLL argmin
|
| 601 |
+
over the options, fp32, no sampling (`probes.py:191-221`); holdout
|
| 602 |
+
families p3:d5 and p4:sub3 gauged from time zero (`probes.py:171`).
|
| 603 |
+
- `census_flags` β `hidden_erank_floor` (any block-output effective
|
| 604 |
+
rank < 0.05Β·d = 51.2), `anchor_merge` (any pair |cos| > 0.99 in a
|
| 605 |
+
bank/hub/head book), `dispatch_collapse` (bank dispatch entropy
|
| 606 |
+
fraction < 0.10), `den_floor` (hub den floor_frac > 1e-3),
|
| 607 |
+
`head_buried` (liveness < 0.1; added at 0.8.2 β absent from the 2s
|
| 608 |
+
warmup_step1145, session_cap_step8320, fineweb_main_step20219 and
|
| 609 |
+
curriculum_s0_step22890 reports, present from s1; present in every
|
| 610 |
+
control report),
|
| 611 |
+
`codebook_erank_floor` (bank codebook erank < 1.5)
|
| 612 |
+
(`instruments.py:150-178`). The census sample is `(2, min(1024,
|
| 613 |
+
ctx))` random bytes in 0..254 (`torch.randint(0, 255, β¦)`, so DOC never appears in the census sample) in the notebook writer; the trainer's own census
|
| 614 |
+
uses the first val batch (`trainer.py:371-374`).
|
| 615 |
+
- `ledger` β `bpb_full`, `bpb_bank_off`, `bpb_head_aleph_off`,
|
| 616 |
+
`bpb_hub_off` (only if any block is a hub β the control's ledger has
|
| 617 |
+
five keys), and `toggle_<x> = bpb_<x> β bpb_full`; mean CE over the
|
| 618 |
+
fixed val batches / ln 2 (`instruments.py:187-212`). No `w_s_norm`
|
| 619 |
+
field (the mb1 schema had one).
|
| 620 |
+
- `special` β `{doc_count, doc_bpb, reset_bpb}` (`instruments.py:
|
| 621 |
+
310-352`); None on BPE crafts.
|
| 622 |
+
- `governor_hits_cum` β the session's cumulative pair-pushes; the
|
| 623 |
+
counter is a Trainer attribute defaulting to 0 (`trainer.py:266`),
|
| 624 |
+
not persisted in the manifest or the resume payload, so it resets
|
| 625 |
+
every session (103,951 @1,145; 1,304,497 @20,219; 70,747 at s1 after
|
| 626 |
+
1,604,488 at s0; 0 at session_cap_53792; 776,450 @61,422).
|
| 627 |
+
- `crowd_check_extra_hits` β pushes from an on-demand `govern_model` at
|
| 628 |
+
report time (415 @20,219; 709 @61,422; 0 on the control).
|
| 629 |
+
|
| 630 |
+
Other report kinds in the same directory: `revival_step24860.json` (the
|
| 631 |
+
op record above, no probes/ledger β chart code must skip it) and
|
| 632 |
+
`cultivation_{A_born_null,B_revival_birth,C_matured_transplant,
|
| 633 |
+
D_live_address_only}.json` β the C7 head-cultivation screen, 1,200
|
| 634 |
+
steps at seed 1337 with gauges at 300/600/900/1200 `{toggle, w_s_norm,
|
| 635 |
+
head_s_norm, head_s_chance, head_liveness_ratio, head_buried}` and,
|
| 636 |
+
for B and D, a `revival` dict (Ξ± 16.0, fit 16,384, base 8.336 β 5.389).
|
| 637 |
+
Endpoint toggles: A β5.3e-06 (buried, liveness 0.045); B +3.496
|
| 638 |
+
(liveness 1.125); C +0.0922 (βW_sβ 520.5); D β7.1e-05 (liveness 1.037,
|
| 639 |
+
not buried, no function) β a live address alone does not carry
|
| 640 |
+
function; a revived head at birth does.
|
| 641 |
+
|
| 642 |
+
**The manifest** (`manifest.py:16-110`): `preset`, `model_config` (the
|
| 643 |
+
full `AlephLMConfig`), `created_utc`, `updated_utc`, `steps`,
|
| 644 |
+
`tokens_seen`, `wall_hours`, `phases[{name, dataset, planned_tokens,
|
| 645 |
+
tokens_done, status β planned|active|done|deferred}]`,
|
| 646 |
+
`checkpoints[{step, tokens, kind β safetensors|resume|fp8, path,
|
| 647 |
+
val_bpb, utc}]`, `data_state{dataset, rows_consumed, epoch}`,
|
| 648 |
+
`notes[{utc, msg}]`. 2s: created 2026-08-27T00:35:31Z, updated
|
| 649 |
+
08-31T19:37:50Z, 142 entries (59 safetensors / 24 fp8 / 59 resume β 46
|
| 650 |
+
distinct bf16 paths). Control: created 08-31T21:26:32Z, updated
|
| 651 |
+
09-03T07:16:38Z, 136 entries β 44 bf16. Neither records a train_config.
|
| 652 |
+
|
| 653 |
+
**Checkpoint kinds** (`checkpoint.py:102-155,187-195`): bf16 safetensors
|
| 654 |
+
every 2,000 steps and at every boundary, every state-dict tensor
|
| 655 |
+
including the `addr.home` buffers (so the file's numel, 237,908,285,
|
| 656 |
+
exceeds the parameter count by 782,336); fp8-e4m3 every 5th checkpoint
|
| 657 |
+
or final, 2-D+ tensors with numel > 4,096 stored with a per-tensor fp32
|
| 658 |
+
scale `amax/448` under `<key>.scale`, small params kept bf16, metadata
|
| 659 |
+
"inference/testing only" β never train from these; `resume/latest.pt`
|
| 660 |
+
fp32 with the manifest embedded, overwritten at every checkpoint and
|
| 661 |
+
archived at boundaries. Hub uploads retry three times and never crash
|
| 662 |
+
training; downloads distinguish genuine absence from network trouble
|
| 663 |
+
and raise on the latter (`checkpoint.py:29-100`).
|
| 664 |
+
|
| 665 |
+
**TensorBoard tags** (`trainer.py:283-296`; `instruments.py:215-260`;
|
| 666 |
+
314 tags on the craft, 212 on the control): from the loop
|
| 667 |
+
`train/{loss,bpb,grad_norm,lr_scale,tokens,tokens_per_sec}`,
|
| 668 |
+
`collapse/loss_spike`, `governor/hits_cum`, `sys/vram_gb`; from the
|
| 669 |
+
census per layer i `erank/hidden_L{i}`, `erank/hub_consumed_L{i}`,
|
| 670 |
+
`erank/hub_codebook_L{i}`, `erank/bank_codebook_L{i}`,
|
| 671 |
+
`cv/bank_load_L{i}`, `cv/hub_usage_L{i}`, `drift/bank_L{i}`,
|
| 672 |
+
`drift/hub_L{i}`, `sign/bank_frac_neg_L{i}`,
|
| 673 |
+
`collapse/dispatch_entropy_L{i}`, `collapse/den_floor_L{i}`,
|
| 674 |
+
`gates/L{i}_e{k}`; head `head/{gamma,w_s_norm,s_norm,liveness_ratio,
|
| 675 |
+
consumed_erank}`, `erank/head_codebook`; `collapse/flag_{β¦}` for the six
|
| 676 |
+
flags; `toggles/{bpb_full,bpb_bank_off,bpb_head_aleph_off,bpb_hub_off,
|
| 677 |
+
toggle_bank_off,toggle_head_aleph_off,toggle_hub_off}`; from the full
|
| 678 |
+
eval `canary/{recall_acc,wrongpair_frac,noretrieval_frac}` and
|
| 679 |
+
`special/{doc_count,doc_bpb,reset_bpb}`. The control lacks every hub
|
| 680 |
+
family; the archived run0 file lacks `head/liveness_ratio`,
|
| 681 |
+
`head/s_norm`, `special/*` and `flag_head_buried` (pre-0.8.0 specials;
|
| 682 |
+
the `head_buried` flag arrived at 0.8.2 @d6e1854 and the `head/s_norm` +
|
| 683 |
+
`head/liveness_ratio` TB scalars at 0.8.4 @a690ea4).
|
| 684 |
+
|
| 685 |
+
**`archive/run0-nospecials/`.** The pre-specials attempt, archived whole
|
| 686 |
+
2026-08-27T00:16Z: its manifest (created 08-26T20:50:18Z, steps 2,537,
|
| 687 |
+
tokens 665,059,328, wall 3.22 h, warmup done at 1,145, fineweb_main in
|
| 688 |
+
progress), bf16 checkpoints 1,145/2,000/2,253/2,537, fp8 2,253/2,537,
|
| 689 |
+
`resume/{latest,boundary_warmup_wikitext_step1145}.pt`, one event
|
| 690 |
+
file, and a report at the doubled path
|
| 691 |
+
`archive/run0-nospecials/mini-beatrix-2s/reports/v2/warmup_wikitext_step2253.json`
|
| 692 |
+
β a late-tagged fineweb report written on interrupt (stale tag, doubled
|
| 693 |
+
path: the 0.8.1 driver-contract bug) with probes, five census flags and
|
| 694 |
+
governor fields but no `ledger`/`special` block. Code-wins note: the
|
| 695 |
+
manual's RIDER 9 says run0 "halted at step 2000"; the archived manifest
|
| 696 |
+
says 2,537 (train() swallowed the interrupt and the driver auto-resumed
|
| 697 |
+
~19 minutes; `repos/alephllm.md:1408`). The hub wins: 2,537.
|
| 698 |
+
|
| 699 |
+
## The AutoModel (AbstractPhil/mini-beatrix-2s)
|
| 700 |
+
|
| 701 |
+
Thirteen files: `README.md`, `config.json`, `modeling_minibeatrix.py`,
|
| 702 |
+
the vendored `address.py`, `alephlm.py`, `attention.py`, `bank.py`,
|
| 703 |
+
`embedding.py`, `governor.py`, `head.py`, `presets.py`,
|
| 704 |
+
`model.safetensors` (475,880,498 B; 593 bf16 tensors, numel
|
| 705 |
+
237,908,285, every key prefixed `model.`, metadata `{tokens: 16.101B,
|
| 706 |
+
step: 61422, format: pt}`), `.gitattributes`. No tokenizer files: input
|
| 707 |
+
ids are raw UTF-8 byte values. The vendored files are byte-identical to
|
| 708 |
+
`geolip/alephllm/model/*.py` and `presets.py` at 0.8.7 except one
|
| 709 |
+
import line (`alephlm.py:30`, `..presets` β `.presets`). `config.json`
|
| 710 |
+
is the craft's `model_config` plus `model_type "mini-beatrix"`,
|
| 711 |
+
`auto_map` to `MiniBeatrixConfig`/`MiniBeatrixForCausalLM`,
|
| 712 |
+
`torch_dtype "float32"` (the file is bf16; the loader upcasts on
|
| 713 |
+
request), `use_cache false`. The wrapper (`modeling_minibeatrix.py:
|
| 714 |
+
21-99`) holds `self.model = AlephLM(cfg)`, forces `hub_ckpt = 0`, makes
|
| 715 |
+
`_init_weights` a no-op ("AlephLM initializes itself; loader must not
|
| 716 |
+
re-init"), truncates to the last `context` positions, returns
|
| 717 |
+
`CausalLMOutputWithPast(loss, logits)` with HF label semantics, and
|
| 718 |
+
carries no KV cache β HF `generate()` recomputes the prefix each step;
|
| 719 |
+
the inner `AlephLM.generate(idx, max_new, temperature, top_p,
|
| 720 |
+
use_cache=True)` uses the constant-size hub state.
|
| 721 |
+
|
| 722 |
+
**The parity gate.** The ship gate β strict load, logits parity 0.0
|
| 723 |
+
against the geolip code path, `generate()` ok β is journal-recorded
|
| 724 |
+
(`repos/alephllm.md:1840-1846`; `history/timeline.md` 2026-08-31;
|
| 725 |
+
`MIRROR.md` step 2 in the AlephLLM repo), not hub-shipped: no parity
|
| 726 |
+
script exists on either hub repo or in the package. Re-verified on CPU
|
| 727 |
+
by the drafting workflow (torch 2.11.0, transformers 5.13.0, no geolip
|
| 728 |
+
installed; not re-run in the hub-side pass, which found the hub
|
| 729 |
+
consistent with these readings β TB `head/w_s_norm` 515.13 at step
|
| 730 |
+
61,000, 593 bf16 tensors, numel 237,908,285): 593 file keys == 593 model keys, parameter count
|
| 731 |
+
237,125,949, βW_sβ 515.918, Ξ³ 1.0; the inner cached decode equals the
|
| 732 |
+
parallel path (`torch.equal` True); logits parity against geolip was
|
| 733 |
+
not recomputed. Two notes for the reader: the model card's lineage line
|
| 734 |
+
says alephllm 0.8.6 while the vendored files are the 0.8.7 mirror
|
| 735 |
+
(commit 90a06b75, 2026-08-31 21:11 β the `presets.py` carries the 0.8.7
|
| 736 |
+
twin-isolation block); and the card's "17 reports" counts the 16
|
| 737 |
+
boundary/session reports plus the revival op β the prefix holds 21
|
| 738 |
+
JSONs with the four cultivation arms.
|
| 739 |
+
|
| 740 |
+
**Load and evaluate** (the steps below were executed by the drafting
|
| 741 |
+
workflow; the numbers are from that run and were not re-run in the
|
| 742 |
+
hub-side pass):
|
| 743 |
+
|
| 744 |
+
```python
|
| 745 |
+
import math, torch
|
| 746 |
+
from transformers import AutoModelForCausalLM
|
| 747 |
+
m = AutoModelForCausalLM.from_pretrained("AbstractPhil/mini-beatrix-2s",
|
| 748 |
+
trust_remote_code=True).eval()
|
| 749 |
+
ids = torch.tensor([list(text.encode("utf-8"))]) # bytes in
|
| 750 |
+
bpb = m(input_ids=ids, labels=ids).loss.item() / math.log(2)
|
| 751 |
+
# toggle ledger on the inner model (positive delta = mechanism carries function)
|
| 752 |
+
core = m.model
|
| 753 |
+
full = core(ids, labels=ids).loss / math.log(2)
|
| 754 |
+
bank_off = core(ids, labels=ids, disable_bank=True).loss / math.log(2)
|
| 755 |
+
hub_off = core(ids, labels=ids, disable_hub=True).loss / math.log(2)
|
| 756 |
+
head_off = core(ids, labels=ids, disable_head_aleph=True).loss / math.log(2)
|
| 757 |
+
# generation: HF path (no cache) or the inner cached decode
|
| 758 |
+
out = core.generate(ids, max_new=40, temperature=0.0, use_cache=True)
|
| 759 |
+
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))
|
| 760 |
+
```
|
| 761 |
+
|
| 762 |
+
On a 227-byte English snippet this read 0.9446 bpb full, +3.872 bank
|
| 763 |
+
off, +3.925 hub off, +0.0156 head off; greedy from "The history of
|
| 764 |
+
astronomy begins with the" continued " study of the sun and the
|
| 765 |
+
planets. The s". Chat: wrap raw bytes in 0xFB SYS / 0xFE USER / 0xFD
|
| 766 |
+
MODEL / 0xFC END (`special_tokens.py:112-136`). Training checkpoints
|
| 767 |
+
load into the same wrapper after prefixing every key with `model.`, or
|
| 768 |
+
into a bare `AlephLM(cfg.to_aleph())`; fp8 files must be rehydrated
|
| 769 |
+
first (`checkpoint.py:187-195`). What the AutoModel cannot reproduce is
|
| 770 |
+
the gauge of record: the fineweb-edu holdout needs
|
| 771 |
+
`geolip.alephllm.data.streams.build_stream(dataset, tok, 4096, 16, seed
|
| 772 |
+
= 1337 + 9999, role="val")` and 262,144 tokens (4 batches of 16 Γ 4096),
|
| 773 |
+
then `toggle_ledger` and `special_token_gauge` over those fixed batches
|
| 774 |
+
(`trainer.py:384-411`; `instruments.py:187-212,310-352`).
|
| 775 |
+
|
| 776 |
+
## The process that made 2s different (dated)
|
| 777 |
+
|
| 778 |
+
**The mother, measured (2026-08-12 β 08-27).** mini-beatrix-1 was built
|
| 779 |
+
with 13 sdpa blocks and three hubs at layers 4/9/14, each a single
|
| 780 |
+
512-anchor book in D = 32 (`git show cc89a87:presets.py`), full-width
|
| 781 |
+
banks holding 67% of the mass, and a 512 @ 32 head. Two post-mortems set
|
| 782 |
+
the 2s agenda: the 08-25 hub census β 16Γ the K β€ 2D supply, 333/646/181
|
| 783 |
+
duplicate anchor pairs at |cos| ~1.0, consumed erank 2.2/3.7/10.8 β and
|
| 784 |
+
the 08-27 autopsy of the shipped weights, W_s exactly 0.0 after 8.4B:
|
| 785 |
+
the "head election" lore retracted, the head never carried function
|
| 786 |
+
(`repos/alephllm.md:1586`; `history/plans/2026-08-26_mini_beatrix_v2_shape.md`).
|
| 787 |
+
A third: the v1 anneal (2B, chat template inside, before the school)
|
| 788 |
+
baked the chat frame into the core irreversibly β the chat-in-anneal
|
| 789 |
+
law (2026-08-15) that later dictated the bracketed two-phase anneal.
|
| 790 |
+
|
| 791 |
+
**Why every block (08-23 β 08-26).** Phil's directive: "the entire model
|
| 792 |
+
to be splat attention". The record's support was dist-e002 β cured
|
| 793 |
+
splat in all 12 ViT-Ti blocks ties sdpa at matched params and matched
|
| 794 |
+
wall-clock on two seeds (R@1 .0545 vs .0562), late placement edging
|
| 795 |
+
(R@5 +.0128, outside the bars); the untested bet, flagged in the plan,
|
| 796 |
+
was that full-splat CAUSAL had never trained. The payoff row was the
|
| 797 |
+
warmup ledger (+1.50 with the hubs off at 0.3B) and the endpoint
|
| 798 |
+
(+3.36).
|
| 799 |
+
|
| 800 |
+
**The shape (08-24/25 β 08-26).** ROUND 5e's shape ladder: 8 Γ 256 lost
|
| 801 |
+
4 of 4 paired against 12 Γ 128, the wide model seed-fragile, its
|
| 802 |
+
crowding tail (79% of layer-1 heads over the 25Β° floor on one seed)
|
| 803 |
+
naming the chain width β crowding β positional ranking flips β deficit;
|
| 804 |
+
hence SUPPLY K β€ 2D (`STATUTES_INDEX.md:53`) and depth over width. B2's
|
| 805 |
+
product code (independent books compose, .859 β .955 monotone in
|
| 806 |
+
members) gave the multi-book form; the rotary Γ anchor flip analysis
|
| 807 |
+
gave governability D β₯ 128 (Ξ³* 32Β° at d512, 45Β° at 256, unreachable at
|
| 808 |
+
64). The arithmetic lands on 4 Γ 64 @ 128 (0.5Γ supply, 4Γ headroom
|
| 809 |
+
against v1's 16Γ), L20, d1024, ctx 4096 (splat's wall-clock crossover
|
| 810 |
+
~2k; below 4k a full-splat trunk pays attention's price without the
|
| 811 |
+
O(L) payoff), banks narrowed to ff 1024, head 256 @ 256. The 849M
|
| 812 |
+
sibling (L32, ctx 8192, 16 Γ 256 @ 256) was designed the same session
|
| 813 |
+
(`presets.py:161-183`).
|
| 814 |
+
|
| 815 |
+
**The governor from birth (08-24 β 08-26).** Phil's proposal β
|
| 816 |
+
"a middleman to control a min/max space that drift can occur to
|
| 817 |
+
guarantee there's always enough space between the anchor points" β was
|
| 818 |
+
confirmed in the math: for a crowded pair the static logit gap is
|
| 819 |
+
quadratic in separation while the positional oscillation is linear, so
|
| 820 |
+
small separation flips weight rankings along the sequence; separating
|
| 821 |
+
an already-crowded trained book leaves its flip pattern bit-identical,
|
| 822 |
+
so min-sep is preventive, not curative β run it from birth. Rung 1 ran
|
| 823 |
+
bit-identical to the ungoverned run (raw books never crowd; the
|
| 824 |
+
crowder was the frame W_a); gov25m fired on the fragile seed only,
|
| 825 |
+
harmlessly. Shipped as a post-step projection, 45Β°, every 8 steps
|
| 826 |
+
(`governor.py`, `presets.py:99-105`). In-mission it ran hot β ~120
|
| 827 |
+
pushes/step at step 1,000, ~84β105 sustained through fineweb, never
|
| 828 |
+
decaying β with every function gauge healthy; the ΞΈ ladder 35/40/45 for
|
| 829 |
+
K = 64 @ D128 is an open rung, and the preset comment's "45 at D=256"
|
| 830 |
+
does not itself justify 45Β° at D = 128.
|
| 831 |
+
|
| 832 |
+
**The kernel campaign and the pivot (08-26, one day).** 0.7.0 at 10:41
|
| 833 |
+
(multi-constellation hub, supply guard, governor wired in, the 849M
|
| 834 |
+
rescale, 2s as the screen craft). The Blackwell preflight OOMed
|
| 835 |
+
(`hub_ckpt`), then launched at 161.88 s/step; the per-op profiler
|
| 836 |
+
printed `qc torch.float32` under autocast β 0.7.2 at 12:27 (the einsum
|
| 837 |
+
promote trap; 4090 hub fwd+bwd 229.7 β 178.8 ms). The dtype fix bought
|
| 838 |
+
~6%; the binding constraint was kernel shape β 0.7.3 at 13:14 (one
|
| 839 |
+
stacked GEMM, one fused softmax, prefix as a triangular GEMM; 65.3 ms,
|
| 840 |
+
3.5Γ; peak 7.4 β 1.9 GB; Blackwell 149 β 69.7 s/step). Compile refuted
|
| 841 |
+
by measurement (bf16-backward NaN, and the pre-NaN projection 67 vs
|
| 842 |
+
69.7 s eager) β eager-only on Blackwell. Economics: 17.3B at 8k tok/s
|
| 843 |
+
is 24β26 days on the one card. Phil: "we can't train the large one
|
| 844 |
+
currently. Not until we work out a series of logistics problems." 0.7.5
|
| 845 |
+
at 13:43 (@73218bd): 2s is the mission, `hub_ckpt` 2 β 0; launched at
|
| 846 |
+
4.54 s/step, 58k tok/s. "Not a bad start. We'll train this one." The
|
| 847 |
+
849M is shelved, not retired (its value-bottleneck relaunch lever
|
| 848 |
+
quantified at ~46 s/step).
|
| 849 |
+
|
| 850 |
+
**The format restart (08-26 16:43).** Phil: "We'll want a practical
|
| 851 |
+
array of them for usage, nothing small. We'll halt this one and start
|
| 852 |
+
over with the correct formatting." 0.8.0: the thirteen unreachable ids,
|
| 853 |
+
the exhaustive proof at every launch, DOC from birth, the frame only in
|
| 854 |
+
anneal, the canary collision caught; run0 archived whole, run1 fresh on
|
| 855 |
+
the clean prefix (manifest 08-27T00:35Z). DOC cost nothing β run1 train
|
| 856 |
+
bpb 1.49 @1,100 against run0's 1.494 @1,000 β and doc_bpb fell 12.83 β
|
| 857 |
+
3.29 by 0.52B. 0.8.1 the same day: the driver contract that made every
|
| 858 |
+
later boundary row a real boundary.
|
| 859 |
+
|
| 860 |
+
**The head saga (08-13 β 08-29 β 08-31).** Born weight-zero by law
|
| 861 |
+
(gate-zero deadlock, `head.py:8-11`); buried inside 2,000 steps on run1
|
| 862 |
+
(0.032Γ chance at the first stored checkpoint, 0.014Γ @12k β journal
|
| 863 |
+
readings from checkpoint analysis, not TB scalars; learned
|
| 864 |
+
avoidance β proj rotated into codebook-orthogonality while W_s grew to
|
| 865 |
+
norm 31 multiplying ~0); LS-ignite alone refuted (0.0006 bpb); the
|
| 866 |
+
governor exonerated (the ungoverned v1 buried deeper; the 45Β° floor is
|
| 867 |
+
free at birth). Candidate law at 2/2 crafts, 3/3 with the control: a
|
| 868 |
+
born-null aleph head racing a strong same-target linear base
|
| 869 |
+
self-buries within ~2k steps; the bank's self-start does not transfer
|
| 870 |
+
across competitor types. Phil ruled activation; the BOUNDARY-WRITE law
|
| 871 |
+
was minted (`revival.py:26`; `STATUTES_INDEX.md:67`) β a third
|
| 872 |
+
intervention category outside both the governor statute and the
|
| 873 |
+
no-balance-machinery law, permitted only at run boundaries or halted
|
| 874 |
+
resume points, always logged β and the deterministic op executed at
|
| 875 |
+
24,860. The first nonzero head toggle in lineage history landed at s1
|
| 876 |
+
(+0.0258, after 6.5B tokens of exactly 0.000); the settled read is a
|
| 877 |
+
durable, small, diet-sensitive niche in the band β.001..+.028,
|
| 878 |
+
band-exits upward on the binding-adjacent diets (s5 causal +.0278),
|
| 879 |
+
touching zero at s7 as bpb_full fell 1.268 β 1.188.
|
| 880 |
+
|
| 881 |
+
**The recipe that never changed (08-12 β 09-04).** Muon on the trunk
|
| 882 |
+
because the 08-10 validity battery on the AR bed read muon .8081 >
|
| 883 |
+
sgd_mom .7995 > adam .7157 > lion .5638, with the address ~20Γ more
|
| 884 |
+
optimizer-fragile than sdpa (cross-optimizer spread .79 vs .037) and
|
| 885 |
+
schedules no-op-to-harmful (flat .714 > warmcos .596) β so the flat-LR
|
| 886 |
+
law; pure Adam on the rest by the 09-01 optimizer statute (Adam the
|
| 887 |
+
experimental baseline, Muon the trunk, AdamW never;
|
| 888 |
+
`STATUTES_INDEX.md:76`). `presets.py:80-84` is the same text at
|
| 889 |
+
cc89a87 (where it sits at `presets.py:65-69`) and at HEAD. The same recipe is the audit's confound (a).
|
| 890 |
+
|
| 891 |
+
**The diet (08-15 β 08-31).** The nine stages and budgets are v1's
|
| 892 |
+
(0.6.0), placed before the anneal rather than after it (Phil,
|
| 893 |
+
2026-08-30: "Let her really build up that internal knowledge first";
|
| 894 |
+
`history/plans/2026-08-26_mini_beatrix_2s_mission.md`), fineweb_extended
|
| 895 |
+
deferred; the two-phase anneal (Phil, 08-31: "enable the anneal mix
|
| 896 |
+
without chat and the anneal mix with chat, we'll run without first,
|
| 897 |
+
then just train chat on after"; `streams.py:63-77`), 0.8.6 at 02:48.
|
| 898 |
+
The s8 terminal-gauge hole appeared at 04:57Z and 0.8.5 had already
|
| 899 |
+
closed it in code (08-30 22:03); the true ledger was re-measured at
|
| 900 |
+
09:53Z. Mission complete 19:31Z: 61,422 steps, 16.101B tokens, 78.33 h,
|
| 901 |
+
zero spikes; the HF outage at the final push was recovered by a Drive
|
| 902 |
+
rescue and retry.
|
| 903 |
+
|
| 904 |
+
**The twin (08-31 β 09-04).** 0.8.7 at 14:10 (twin isolation, birth
|
| 905 |
+
recipe); the control born 21:26Z; the protocol declared at its warmup
|
| 906 |
+
boundary while sdpa led; the crossover at fineweb 5.3B and every row
|
| 907 |
+
after; the rank story as the mechanism claim (at step 4,000 the control
|
| 908 |
+
collapses early hidden erank to 5β9 of 1024 at L2βL4 (5.5 / 5.4 / 9.3)
|
| 909 |
+
while the craft reads 234 / 225 / 250 there and 150β396 across L0βL15
|
| 910 |
+
(`erank/hidden_L{i}`, TB census); screen scale reproduces it; rank β function β the
|
| 911 |
+
control led val at 4k while running rank-5 layers); the H100 resume at
|
| 912 |
+
22,854; the grad-norm escalation rider at s5; Phil's claim held on
|
| 913 |
+
09-02 ("SDPA isn't an adequate representative of this model's
|
| 914 |
+
attention requirement. We can say that without question now."); the
|
| 915 |
+
third host from 56,000; the endpoint 09-03; the 12-agent audit on 09-04
|
| 916 |
+
(S-1802 INSTANCE, S-1803 CANDIDATE, the s8 strike, the mother null,
|
| 917 |
+
Muon clip-blind, the falsifier filed).
|
| 918 |
+
|
| 919 |
+
## Gauges, defined once
|
| 920 |
+
|
| 921 |
+
- **val bpb**: mean cross-entropy in bits/byte over 262,144 tokens
|
| 922 |
+
(4 batches of 16 Γ 4096) of the fineweb-edu reserved head slice β
|
| 923 |
+
the first 2,048 rows, fixed, unshuffled, seed 1337 + 9999 β for every
|
| 924 |
+
phase from fineweb_main onward, both crafts (`trainer.py:384-411`;
|
| 925 |
+
`streams.py:279-282,423-447`; gauge continuity law). The warmup row
|
| 926 |
+
vals on wikitext-103's validation split. Pre- and post-0.8.0 gauges
|
| 927 |
+
differ for v1-era crafts (DOC rides the val stream).
|
| 928 |
+
- **toggle ledger**: bpb with one mechanism disabled minus full bpb β
|
| 929 |
+
the causal contribution of banks / hubs / head aleph
|
| 930 |
+
(`instruments.py:187-212`). Positive = the mechanism carries function.
|
| 931 |
+
- **specials gauge**: `doc_bpb` and `reset_bpb` in bits, read against
|
| 932 |
+
the same val batches (`instruments.py:310-352`); `doc_count` 70 on
|
| 933 |
+
every 2s-era report on the fineweb-edu gauge (the struck 2s s8 file reads 1612).
|
| 934 |
+
- **head liveness**: mean βsigned readβ of real hidden states over the
|
| 935 |
+
same book's mean on 512 random unit vectors β self-calibrating;
|
| 936 |
+
`head_buried` = ratio < 0.1 (`instruments.py:144,176,356-382`).
|
| 937 |
+
Both buried crafts sat at 0.003Γβ0.03Γ chance.
|
| 938 |
+
- **probe accuracy**: byte-NLL argmin over the item's options, nine
|
| 939 |
+
suites Γ 30 items, holdout families d5 and sub3 gauged from time zero
|
| 940 |
+
(`probes.py:171-235`). At n = 30 a between-craft difference has SE
|
| 941 |
+
.126: the paired probe calls are ties.
|
| 942 |
+
- **train/grad_norm**: the pre-clip global norm returned by
|
| 943 |
+
`clip_grad_norm_` (`trainer.py:244,287`), logged every 50 steps; under
|
| 944 |
+
Muon it measures sharpness, not the applied step.
|
| 945 |
+
- **governor hits**: cumulative pair-pushes per session
|
| 946 |
+
(`trainer.py:266`); a session-scoped counter, comparable within one
|
| 947 |
+
event file only.
|
| 948 |
+
|
| 949 |
+
## Known caveats and open items
|
| 950 |
+
|
| 951 |
+
- The autocast einsum-promote trap voids every pre-0.7.2 splat speed
|
| 952 |
+
verdict (fp32 scans under "bf16"); accuracy verdicts stand. Any
|
| 953 |
+
autocast performance reading over einsum code is invalid until the
|
| 954 |
+
dtypes are printed (`attention.py:167`).
|
| 955 |
+
- Muon is clip-blind (`optim.py:24`), and Adam's per-coordinate
|
| 956 |
+
normalisation makes it nearly so: `grad_clip 1.0` shrank the applied
|
| 957 |
+
step of neither group on the control, only the weighting of steps
|
| 958 |
+
inside each optimizer's buffers; the recipe carries no softmax
|
| 959 |
+
stabiliser.
|
| 960 |
+
- The 2s `curriculum_s8_step53792` ledger is on the synthetic fallback
|
| 961 |
+
and is struck; the same-gauge pair is owed by a zero-training
|
| 962 |
+
re-evaluation of `resume/boundary_curriculum_s8_step53792.pt` on the
|
| 963 |
+
0.8.7 gauge (T0b in `history/plans/2026-09-04_sdpa_falsifier_screen.md`).
|
| 964 |
+
Hub-vs-journal note: the file on the hub reads doc_count 1612,
|
| 965 |
+
bpb_hub_off 6.2378 (toggle_hub_off 5.164), doc_bpb 0.843 β agreeing
|
| 966 |
+
with the trainer comment and the manual (doc_count 1612, hub_off 5.16 =
|
| 967 |
+
the toggle; `trainer.py:397-399`); the file carries no dataset field,
|
| 968 |
+
so the strike rests on the same-step cap discrepancy (doc_count 70,
|
| 969 |
+
bpb_full 1.1911 in `session_cap_step53792.json`) and on the 1612-document
|
| 970 |
+
count itself. The hub file's numbers win for what the file says.
|
| 971 |
+
- The control's third host from 56,000: a replay of up to 999 steps is a
|
| 972 |
+
TB-cadence hedge, not a hub reading; no cap report; the 53,800β56,000 segment ran in the H100
|
| 973 |
+
session's own process (event file `0a706d592fec.620.1`); canon CB11/12 "on the H100" wrong by erratum.
|
| 974 |
+
- Every resume refills the shuffle buffer (not row-exact); the control
|
| 975 |
+
resumed three times, the craft four; MixStream state carries no
|
| 976 |
+
recipe fingerprint.
|
| 977 |
+
- The 2s preset at HEAD carries `head_addr_frozen = True`
|
| 978 |
+
(`presets.py:199-201`), the post-revival state: a fresh birth of the
|
| 979 |
+
treatment preset under 0.8.7 would freeze a random proj and codebook
|
| 980 |
+
from step 0, which is not what the shipped run did (born-null unfrozen
|
| 981 |
+
under 0.8.1 for 24,860 steps, revived, then frozen). The control's
|
| 982 |
+
override (`presets.py:235`) is the birth recipe.
|
| 983 |
+
- Stale text, code wins: `presets.py:18` "~233M" (built 237,125,949);
|
| 984 |
+
`presets.py:14,166` "~873.7M" for mini-beatrix-2 (built 849,023,425);
|
| 985 |
+
the training README's ladder row "mini-beatrix-2 | 1024/20/2048 |
|
| 986 |
+
249.1M" is the retired v1 shape; the hub driver `beatrix_v2_colab.ipynb`
|
| 987 |
+
is headed "mini-beatrix-2 β the governed full-splat mission" and names
|
| 988 |
+
`mini-beatrix-2-control` while its CRAFT is `mini-beatrix-2s` (no
|
| 989 |
+
control notebook exists on the hub β only this file and
|
| 990 |
+
`mini-beatrix-1/beatrix_curriculum.ipynb`); `AlephLMConfig.hub_chunk` defaults to
|
| 991 |
+
128 while the module's own default is 256 (`presets.py:52`;
|
| 992 |
+
`attention.py:83`) β the config governs, and 2s sets 256.
|
| 993 |
+
- `governor_hits_cum` resets per session; the manifests record no
|
| 994 |
+
train_config or package version β the mother null's "identical
|
| 995 |
+
TrainConfig" is a git fact, and the craft's birth version 0.8.1 is
|
| 996 |
+
inferred from commit dates (run1 created 08-27T00:35Z between 0.8.1
|
| 997 |
+
on 08-26 and 0.8.2 on 08-29; the revival report records 0.8.2; the hub
|
| 998 |
+
driver `beatrix_v2_colab.ipynb` pins `geolip-alephllm @2a902b7`, the
|
| 999 |
+
0.8.1 commit).
|
| 1000 |
+
- Owed instruments: the per-layer hidden-erank census of both crafts at
|
| 1001 |
+
the full-scale boundaries; the falsifier S-1803-F (Tier 0 on the 4090;
|
| 1002 |
+
a 40M twin screen with sdpa + QK-norm / LR/3 / dense-FFN / positions
|
| 1003 |
+
on Adam arms; pivot = resume the control at 20,219 through s0 at LR/3
|
| 1004 |
+
with QK-norm). Without it the general claim stays CANDIDATE.
|
| 1005 |
+
- `article_assets_ft2/make_charts_ft2.py` sources chart 01 from the hub
|
| 1006 |
+
but charts 02β05 from local scratch paths, and chart 01 pairs the
|
| 1007 |
+
struck s8 rows unless filtered.
|
| 1008 |
+
|
| 1009 |
+
## Provenance discipline
|
| 1010 |
+
|
| 1011 |
+
Both crafts are resume-first: the manifest is embedded in the resume
|
| 1012 |
+
payload (atomic consistency), a crash never overwrites
|
| 1013 |
+
`resume/latest.pt` (`trainer.py:334-368`: any exception other than a
|
| 1014 |
+
manual stop leaves the hub state untouched), an interrupted session
|
| 1015 |
+
uploads before exit, and every phase boundary archives its own
|
| 1016 |
+
immutable resume. The revival is auditable from the prefix alone: the
|
| 1017 |
+
pre-write state (`resume/pre_revival_step24860.pt`), the op's
|
| 1018 |
+
provenance (`reports/v2/revival_step24860.json`, with version and
|
| 1019 |
+
commit), the pre- and post-write weights (`step_00024860` and
|
| 1020 |
+
`step_00024860_revived`), and the first live row after it
|
| 1021 |
+
(`curriculum_s1_step25561.json`). The struck s8 ledger stays public
|
| 1022 |
+
beside the cap that supersedes it, as run0 stays public under
|
| 1023 |
+
`archive/`, for the same reason the mother's rewind checkpoints do. The
|
| 1024 |
+
recipe is pinned from source, not from the manifests. The AutoModel
|
| 1025 |
+
vendors the model files verbatim and ships the mission-final weights
|
| 1026 |
+
re-prefixed; the training checkpoints, not the card, are the control's
|
| 1027 |
+
weights. Every refuted arm of the head-cultivation screen ships.
|
| 1028 |
+
|
| 1029 |
+
## Pointers
|
| 1030 |
+
|
| 1031 |
+
- Canon: `canon/splat_aleph_battery.md` β CONTROL BOUNDARY 1β12
|
| 1032 |
+
(522β889), ENDPOINT VERDICT (891), CLAIM AUDIT (901β1006: the s8
|
| 1033 |
+
strike 930β937, MUON IS CLIP-BLIND 936, THE MOTHER NULL 955β968,
|
| 1034 |
+
PROVENANCE RIDER 1008), ROUND 5a/5d/5e, B2/B4/B5.
|
| 1035 |
+
- Statutes (`STATUTES_INDEX.md`): S-701 a control must be able to fail;
|
| 1036 |
+
S-801 supply law K β€ 2D; S-1001 gate-zero deadlock; S-1002 born-null
|
| 1037 |
+
head self-burial 3/3; S-1003 BOUNDARY-WRITE; S-1201 optimizer family;
|
| 1038 |
+
S-1202 autocast einsum-promote; S-1801 the crossover; S-1802 the
|
| 1039 |
+
destabilised control (INSTANCE); S-1803 sdpa-inadequacy (CANDIDATE);
|
| 1040 |
+
S-1901 the specials plane.
|
| 1041 |
+
- Manual: `repos/alephllm.md` β the 0.7.0 era (1180β1330) and RIDERS
|
| 1042 |
+
7β21b (1333β2081): pivot, launch health, specials and the run0
|
| 1043 |
+
archive, driver contract, the dead head, the revival trial, the
|
| 1044 |
+
boundary-write and promote, the head trend 14β14l, curriculum complete
|
| 1045 |
+
and the s8 artifact, the pre-chat baseline, mission complete, the
|
| 1046 |
+
ship-out, twin isolation, the pre-registered protocol.
|
| 1047 |
+
- Plans: `history/plans/2026-08-26_mini_beatrix_v2_shape.md` (the
|
| 1048 |
+
shape arithmetic), `2026-08-26_mini_beatrix_2s_mission.md` (the pivot,
|
| 1049 |
+
the anneal-timing ruling, the two-phase amendment),
|
| 1050 |
+
`2026-08-24_covariant_address_program.md` (the governor),
|
| 1051 |
+
`2026-09-02_sdpa_inadequacy_claim_held.md`,
|
| 1052 |
+
`2026-09-04_sdpa_falsifier_screen.md`.
|
| 1053 |
+
- Dockets: `history/dockets/2026-09-04_sdpa_claim_audit/`,
|
| 1054 |
+
`2026-09-01_ft2_article/` and `2026-09-01_article/tools/`
|
| 1055 |
+
(`writeback_cb*.py`, `writeback_endpoint.py`).
|
| 1056 |
+
- Article: `articles/article_beatrix_ft2.md`; charts under
|
| 1057 |
+
`article_assets_ft2/` on the training repo.
|
| 1058 |
+
- Code: AlephLLM HEAD 4f4b7d6 (0.8.7); the era's commits a63a29b
|
| 1059 |
+
(0.7.0) Β· 2be28dd (0.7.2) Β· 60ee496 (0.7.3) Β· 73218bd (0.7.5, the
|
| 1060 |
+
pivot) Β· 3ea3103 (0.8.0) Β· 2a902b7 (0.8.1) Β· d6e1854 (0.8.2) Β·
|
| 1061 |
+
7d48328 (0.8.5) Β· af9319c (0.8.6) Β· b472815 (0.8.7). No git tags
|
| 1062 |
+
exist; versions are commit subjects.
|
| 1063 |
+
- The mother: `TECHNICAL.md` on the training repo (byte-identical local
|
| 1064 |
+
copy `articles/TECHNICAL_training_repo_beatrix.md`).
|