{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# mini-beatrix-3 — the v3 routine (steering-and-arms craft)\n", "\n", "**Craft**: `mini-beatrix-3` — d1024 × L`DEPTH` (32 by the lead's ruling of 2026-09-19; the sizing range was 24-28), ctx 4096, a governed CausalSplatHUB in **every** block\n", "(4 books × 64 anchors @ D128, supply 0.5×D), 3 banks @ ff1024, head 256@256. ~300–350M params.\n", "Plan of record: claude-mind `history/plans/2026-09-15_beatrix_v3_plan_v2.md` (S1–S14, register A1–E, routine §4) ·\n", "recipe `history/plans/2026-09-12_v3_recipe_v0.md` (terms A–P) · pre-flight `history/dockets/2026-09-05_bytelex_plan/v3_preflight_needs_2026-09-19.md`.\n", "\n", "**What this notebook is**: the v2 mission notebook re-aligned to the v3 plan. Its settings are TOGGLES set by the research,\n", "the testing and the design — each value in C1 carries its source and names the test that would move it. What is not yet\n", "built or measured is stated as such and gates the run exactly where the design says (the data plane before the start; the\n", "lawful arm geometry before the first stage arm; the anneal multiplier before the anneal). The toy switch (`LOCAL = True`)\n", "runs the whole notebook on a CPU-sized craft so the plumbing is exercised end to end.\n", "\n", "**Laws embodied** (all on record): supply K ≤ 2·D per book · governor from birth (min-sep, post-step, identity when slack) ·\n", "Muon + pure Adam wd 0, never AdamW · flat LR after a 200-step warmup (the anneal's multiplier is the lead's) · bf16-train /\n", "fp8-ship / never fp16 · eager on Blackwell (compile retired by measurement 2026-08-26) · crash-safe resume-first ·\n", "per-boundary report + checkpoint + arm anchors (ship-complete) · the epoch cap on every finite corpus computed BEFORE a\n", "mix ships · every stage arm quiet (per-member abstention on off-domain rows) · a faulty arm is masked out, never a crash ·\n", "a CERTIFIED red flag halts cleanly (archive + record + refusal to auto-resume); uncertified flags are watched, never halts.\n", "\n", "**Price** (canon L7, 4090-measured, RTX PRO 6000 = ESTIMATE until the Colab bench cell runs): 64.4B bytes at eager speed\n", "≈ 13.7 days at L24, 16.4 days at L28. The 2026-08-26 ruling stands beside it (\"26 days isn't reasonable\").\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C1 — pinned installs + THE DECISION BLOCK. RESTART RUNTIME if a version line below changes.\n", "import os, sys\n", "os.environ['HF_HUB_DISABLE_PROGRESS_BARS'] = '1' # a progress bar is not allowed to cost a session\n", "os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'\n", "%pip install -q \"datasets>=5.0.0\" \"geolip-alephllm @ git+https://github.com/AbstractEyes/alephllm@ea51ec6\"\n", "%pip install -q \"amoe-lora @ git+https://github.com/AbstractEyes/amoe-lora@e21baea\" --no-deps\n", "%pip install -q \"geolip-bytelex @ git+https://github.com/AbstractEyes/geolip-bytelex@e8fc82d\" --no-deps\n", "import geolip.alephllm as A\n", "print('geolip.alephllm', A.__version__)\n", "assert tuple(int(x) for x in A.__version__.split('.')) >= (0, 8, 9), (\n", " 'RESTART REQUIRED: runtime still holds the old package — Runtime > Restart, then rerun from C1')\n", "try:\n", " import amoe; print('amoe', amoe.__version__)\n", "except ImportError:\n", " print('amoe-lora NOT importable — stage arms (STAGE_ARMS) cannot attach until it is')\n", "try:\n", " import geolip.bytelex; print('geolip.bytelex importable (the atlas coverage audit instrument)')\n", "except ImportError:\n", " print('geolip-bytelex NOT importable — the coverage audit needs it')\n", "\n", "def _token():\n", " try:\n", " from google.colab import userdata\n", " return userdata.get('HF_TOKEN')\n", " except Exception:\n", " return os.environ.get('HF_TOKEN')\n", "\n", "# ============ THE TOGGLES — set by the research, the testing and the design ============\n", "# Every value is SET FROM THE RECORD with its source beside it, and the test that would move\n", "# it is named. They are toggles the research varies, not a form: a mission run needs no hand\n", "# edit here. What is NOT yet built or measured is stated as such and gates the run where the\n", "# design says it does (Cell T prints the pre-flight status). LOCAL = the CPU toy path.\n", "DEPTH = 32 # the lead's ruling 2026-09-19 23:5x (was 24, the floor of the sizing range 24-28): depth at scale is the question the mission answers (structure continuity, routing capacity, new contextualizations); the 0.31B depth rung could not rank depths; ~+33% wall time vs 24 on 2 x 5090 (~11.7 d vs ~8.8 d); 376M params. MOVED BY: the depth-ladder erank read (proposed 09-15, not launched).\n", "DATA_SCALE = 4.0 # S14a: 4x the 2s schedule, counted in RAW BYTES (the recorded accounting); the fused-unit count belongs to S14b once fusion exists.\n", "EPOCH_CAP = 2.0 # the epoch-cap LAW (curriculum.py, 2026-08-15): no finite corpus re-read more than ~2x within its stage; the audit's 4 is its warning threshold, not the law.\n", "REBALANCE_TO = 'generators' # the 08-15 precedent: procedural generators carry the volume (to their 35% share each), then natural text; the 09:12 accounting: the 4x comes from infinite sources + generators + minted programs, the structure corpora held near 1x.\n", "ANNEAL_LR_SCALE = 'from_screen' # the design (S3 / D1): the anneal is a LOWER-LR consolidation stage (the v2 anneal at 1.0 was a diet change and left a rigid chat shell). NO NUMBER ON RECORD (recipe term H, 0 seeds): the anneal-multiplier screen sets it; the anneal boundary does not open until a number is here (Cell T gate).\n", "ARM_SPEC = 'LAWFUL_16x8' # the supply law K <= 2D (S-801; sizing 09-15: v3 arm codebooks must come under it). The certified ladder geometry is WIDE_1024 (8x over the law, a flagged exception). GATED BY the lawful-geometry certification below.\n", "ARM_SPEC_CERTIFIED = False # set True by the lawful-geometry certification (L4's staged co-training cell with LAWFUL_16x8, 2 seeds); until then no stage arm attaches (Cell T gate at the first stage boundary).\n", "STAGE_ARMS = [ # C1 / S3: each capability stage attaches its arm at its boundary and trains it under the same update; earlier arms stay trainable; every member quiet at lambda 1.0 (the template/capability constant; 2.0 = hard pools). s0 is the first-language stage (no arm). OPEN: the arm-count ceiling screen.\n", " ('s1_perspective', 'curriculum_s1', 1.0, 1), ('s2_concept', 'curriculum_s2', 1.0, 2),\n", " ('s3_rules', 'curriculum_s3', 1.0, 3), ('s4_arith', 'curriculum_s4', 1.0, 4),\n", " ('s5_causal', 'curriculum_s5', 1.0, 5), ('s6_tryfail', 'curriculum_s6', 1.0, 6),\n", " ('s7_mixed', 'curriculum_s7', 1.0, 7), ('s8_register', 'curriculum_s8', 1.0, 8)]\n", "FUSION = None # S14b (his direction): weak-token fusion at the input plane, on the atlas plane. SCREENED 2026-09-19 (canon FS; runner 22 tag 2SL28, four fresh crafts at the 2s shape on two RTX 5090s, 0.5B matched raw bytes): the BUILT FORM of the stage (spacelike / hybrid 2.5 bits / entropy 3.0 bits unit starts from the entropy table) costs +0.117-0.124 bpb over the unfused byte trunk, gains 1.48x (not the 1.7x predicted) and moves every boundary class. None = the built form is NOT READY at this budget (Phil 2026-09-19: single-axis screens cannot refute the design); the DESIGN stands open — the atlas-weighted spatial form with caveat-aware edges (S9f) is unbuilt to spec. The STAGE stays in the library (model/fusion.py); a config dict re-enables it when the design is complete.\n", "COVERAGE_AUDIT = 'hf://AbstractPhil/alephllm-mini-beatrix-training/mini-beatrix-3/v3_preflight/coverage_audit.json' # routine step 1 / D2 / S9: RUN 2026-09-19 12:16-12:21 (tools/btx_v3_coverage_audit.py; canon CA): 26 sources at this data plane; the union of live trigram cells 6,954; half the planned mass in 254 cells (the generators 24.7% over 16-260 cells each, tinystories 11.5% at entropy 0.00) — the shape the craft reads, no bar. None = not run (gates the start).\n", "HEAD_BIRTH = 'revival_birth' # S-1004 (born-with-function: the solve is mandatory) + the 2s record (born-null buried 3/3; the closed-form solve at 24,860 revived it). CONFIRMED by the C7 head screen 2026-09-19 (canon HS, one seed at the v3 shape): revival at birth stays live (1.13x chance) with toggle +3.29 bpb at 1,200 steps while born-null buries (0.11x, toggle 0) and the transplanted 2s head reads +0.04; matched train loss 1.232 (revival) vs 1.247 (born-null) vs 1.251 (transplant).\n", "HUB_GAIN = None # S5 / A3: the hub gain is an inference-certified instrument; TRAINING under it is gated on his bed word by the design itself. None = off.\n", "THINK = DIVERSITY = CONDENSING = TOOL_ARMS = None # post-training programs (S13 / S6 / S7): design items, no routine built (the last cells).\n", "QUIET_TRUNK_GRAD = False # the certified semantics: only the arms feel the abstention term (True = the uncertified variant).\n", "RESUME_AFTER_HALT = False # a run halted by a certified guard continues only after the archive is read (one clearance per session).\n", "LOCAL = False # True = the toy path: a CPU-sized craft, synthetic phases, test-sized values for the names above.\n", "# =========================================================================================\n", "LOCAL = LOCAL or os.environ.get('ALEPHLLM_NB_LOCAL') == '1'\n", "CRAFT = 'mini-beatrix-3'\n", "if LOCAL:\n", " CRAFT = 'v3-local-toy'\n", " TEST = dict(DEPTH=2, EPOCH_CAP=2.0, REBALANCE_TO='generators', ANNEAL_LR_SCALE=0.5, ARM_SPEC='LAWFUL_16x8', ARM_SPEC_CERTIFIED=True,\n", " STAGE_ARMS=[('s1_toy', 'curriculum_s1', 1.0, 1), ('s2_toy', 'curriculum_s2', 1.0, 2)],\n", " FUSION={'rule': 'spacelike', 'k_lo': 1, 'k_hi': 0}, COVERAGE_AUDIT='toy', HEAD_BIRTH='born_null') # the toy exercises the fused path (one middle block over units)\n", " globals().update(TEST)\n", " print('LOCAL TOY PATH: test-sized values', TEST)\n", "\n", "def preflight_status():\n", " # the design's step 1 (the data plane assembled) + the tests that gate later boundaries, computed\n", " # from the toggles above — printed by Cell T; the mission run starts only when the data-plane rows\n", " # are complete (the toy path is exempt)\n", " rows = [('coverage audit over the pretraining mix (D2 / S9)', COVERAGE_AUDIT is not None, 'gates the start'),\n", " ('weak-token fusion at the input plane (S14b): the rule screened (a dict) or the unfused trunk chosen (None)', FUSION not in ('not_built', 'unscreened'), 'gates the start'),\n", " ('lawful arm geometry certified, 2 seeds (S-801)', bool(ARM_SPEC_CERTIFIED) or not STAGE_ARMS, 'gates the first stage-arm boundary'),\n", " ('anneal multiplier from the screen (recipe term H)', isinstance(ANNEAL_LR_SCALE, (int, float)), 'gates the anneal boundary'),\n", " ('head birth mechanism of record (S-1004)', HEAD_BIRTH in ('revival_birth', 'born_null'), 'applies at step 0')]\n", " print(f\"{'pre-flight item':<52}{'status':>8} {'what it gates'}\")\n", " for name, ok, gate in rows:\n", " print(f\"{name:<52}{('done' if ok else 'OPEN'):>8} {gate}\")\n", " return rows\n", "\n", "HF_TOKEN = _token()\n", "OUT_DIR = os.environ.get('ALEPHLLM_NB_OUT', './alephllm_runs')\n", "import torch\n", "DEVICE = 'cuda' if torch.cuda.is_available() and os.environ.get('ALEPHLLM_NB_CPU') != '1' else 'cpu'\n", "print(f'craft {CRAFT} · device {DEVICE} · token {\"present\" if HF_TOKEN else \"ABSENT (local artifacts only; Cell T stays disarmed for the mission)\"}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C2 (P) — PREFLIGHT: build the preset for the chosen depth, law checks, bit-exact births,\n", "# the micro-batch LADDER bench at a fixed 262,144 tokens/step, the price line. No training.\n", "import time, math, torch\n", "from geolip.alephllm.presets import make_v3_preset, PRESETS, AlephLMConfig, _copy_train\n", "from geolip.alephllm.model.alephlm import AlephLM\n", "from geolip.alephllm.model.governor import govern_model\n", "\n", "p = make_v3_preset(DEPTH, data_scale=DATA_SCALE, epoch_cap=EPOCH_CAP, rebalance_to=REBALANCE_TO, name=CRAFT)\n", "if isinstance(ANNEAL_LR_SCALE, (int, float)):\n", " p.train.phase_lr_scale = {'anneal': float(ANNEAL_LR_SCALE)} # both anneal phases; 1.0 = the flat form verbatim\n", "else:\n", " print(f'anneal multiplier: {ANNEAL_LR_SCALE} — the anneal boundary stays closed until the screen sets a number (Cell T gate)')\n", "if HEAD_BIRTH == 'revival_birth':\n", " p.train.head_addr_frozen = True # the doctrine: the address is solved at birth, then frozen (W_s trains)\n", "if LOCAL: # the toy craft: same laws, CPU size, synthetic phases\n", " p.model = AlephLMConfig(name=CRAFT, d_model=64, n_layers=DEPTH, n_heads=1, context=512,\n", " hub_layers=tuple(range(DEPTH)), hub_K=4, hub_D=8, hub_const=2, # supply 0.5xD like the mission\n", " bank_experts=3, bank_ff=64, head_K=8, head_D=16, hub_chunk=32, hub_ckpt=0)\n", " p.train.micro_batch, p.train.grad_accum, p.train.warmup_steps = 2, 1, 2\n", " p.train.log_every, p.train.health_every, p.train.eval_every = 2, 4, 8\n", " p.train.ckpt_every, p.train.tb_upload_every, p.train.val_tokens, p.train.canary_episodes = 8, 1000, 256, 4\n", " p.curriculum = [dict(ph, dataset='synthetic', planned_tokens=4096) for ph in p.curriculum]\n", "if isinstance(FUSION, dict): # the screened rule: the hourglass form (model/fusion.py); the unfused trunk otherwise\n", " p.model.fusion = dict(FUSION)\n", " if str(p.model.fusion.get('table', '')).startswith('hf://'): # the atlas table lives on the hub: fetch it once\n", " from huggingface_hub import hf_hub_download\n", " _owner, _repo, _path = p.model.fusion['table'][5:].split('/', 2)\n", " p.model.fusion['table'] = hf_hub_download(f'{_owner}/{_repo}', _path, token=HF_TOKEN)\n", " print('fusion:', p.model.fusion)\n", "PRESETS[CRAFT] = p\n", "ctrl = AlephLMConfig.from_dict(p.model.to_dict()); ctrl.name = CRAFT + '-control'; ctrl.hub_layers = ()\n", "PRESETS[CRAFT + '-control'] = type(p)(model=ctrl, train=_copy_train(p.train), curriculum=[dict(x) for x in p.curriculum],\n", " data_scale=p.data_scale, epoch_cap=p.epoch_cap, rebalance_to=p.rebalance_to)\n", "cfg, tc = p.model, p.train\n", "PLANNED = sum(ph['planned_tokens'] for ph in p.curriculum)\n", "print(f\"{cfg.name}: depth {cfg.n_layers} · d {cfg.d_model} · ctx {cfg.context} · hubs {len(cfg.hub_layers)}/{cfg.n_layers} × \"\n", " f\"{cfg.hub_const} books × {cfg.hub_K}@D{cfg.hub_D} (supply {cfg.hub_K/cfg.hub_D:.2f}×D) · planned {PLANNED/1e9:.2f}B bytes\")\n", "for ph in p.curriculum:\n", " print(f\" {ph['name']:<18} {ph['dataset']:<16} {ph['planned_tokens']/1e9:8.3f}B\")\n", "assert cfg.hub_K <= 2 * cfg.hub_D, 'supply law violated — this preset should not exist'\n", "assert tc.compile is False, 'eager on Blackwell (the 2026-08-26 law)'\n", "torch.manual_seed(tc.seed)\n", "model = AlephLM(cfg).to(DEVICE)\n", "n = model.param_count(); print(f'{n/1e6:.1f}M params')\n", "hits = govern_model(model, tc.governor_theta)\n", "print(f'governor birth check: {hits} hits (must be 0 at mission D)' + (' — toy D, not gated' if LOCAL else ''))\n", "assert hits == 0 or LOCAL\n", "x = torch.randint(0, 255, (2, min(cfg.context, 256)), device=DEVICE)\n", "with torch.no_grad(): # C6 null paths: bank + head aleph contribute exactly zero at init\n", " a = model(x, disable_bank=True, disable_head_aleph=True).logits\n", " b = model(x).logits\n", " assert torch.equal(a, b), 'C6 violated — born-null paths are not null'\n", "print('C6 null paths bit-exact at birth')\n", "\n", "# the ladder: micro-batch by fit at a FIXED 262,144 tokens/step (16 x 4096); the first\n", "# rung that fits is the session's micro_batch, grad_accum follows. An arm rides the\n", "# bench when STAGE_ARMS is non-empty (the arm's forward is part of the step it prices).\n", "TOK_STEP = tc.micro_batch * tc.grad_accum * cfg.context\n", "if DEVICE == 'cuda':\n", " import platform\n", " frac = 0.73 if platform.system() == 'Windows' else 0.98\n", " if platform.system() == 'Windows':\n", " print('WINDOWS (WDDM): fraction 0.73 — the ~18 GB local cap; the mission card is Linux at 0.98')\n", " torch.cuda.set_per_process_memory_fraction(frac)\n", " total_gb = torch.cuda.get_device_properties(0).total_memory / 2**30\n", " abort_gb = 88 if total_gb > 90 else 0.73 * total_gb\n", " if STAGE_ARMS:\n", " from geolip.alephllm.train.arms import StageArm, ArmProgramConfig, StageArmProgram, LAWFUL_16x8, WIDE_1024\n", " spec = {'LAWFUL_16x8': LAWFUL_16x8, 'WIDE_1024': WIDE_1024}[ARM_SPEC]\n", " class _T: pass\n", " _t = _T(); _t.raw_model = model; _t.device = DEVICE; _t._trunk_params = list(model.parameters())\n", " prog = StageArmProgram(ArmProgramConfig(arms=[StageArm('bench', 'bench', spec=dict(spec))])).bind(_t)\n", " prog.attach(prog.by_name['bench']); print('bench with one arm attached', spec)\n", " model.train(); best = None\n", " for mb in (16, 8, 4, 2, 1):\n", " accum = max(1, TOK_STEP // (mb * cfg.context))\n", " try:\n", " opt_probe = torch.optim.SGD([q for q in model.parameters() if q.requires_grad], lr=0.0)\n", " xb = torch.randint(0, 255, (mb, cfg.context), device=DEVICE)\n", " torch.cuda.reset_peak_memory_stats(); torch.cuda.synchronize(); t0 = time.time()\n", " for i in range(3):\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " out = model(xb, targets=xb)\n", " out.loss.backward(); opt_probe.zero_grad(set_to_none=True)\n", " torch.cuda.synchronize(); sps = (time.time() - t0) / 3\n", " vram = torch.cuda.max_memory_allocated() / 2**30\n", " tps = mb * cfg.context / sps\n", " print(f'mb {mb:2d} x accum {accum:2d}: {sps:.2f}s/micro · {tps/1e3:.1f}k tok/s · peak {vram:.1f} GB' + (' OVER BUDGET' if vram > abort_gb else ''))\n", " if vram <= abort_gb and best is None:\n", " best = (mb, accum, tps)\n", " del out, xb, opt_probe; torch.cuda.empty_cache()\n", " except torch.cuda.OutOfMemoryError:\n", " print(f'mb {mb:2d}: OOM'); torch.cuda.empty_cache()\n", " assert best is not None, 'no rung fits — the card cannot carry this depth'\n", " tc.micro_batch, tc.grad_accum, tps = best[0], best[1], best[2]\n", " days = PLANNED / tps / 86400\n", " card = torch.cuda.get_device_name(0)\n", " label = 'MEASURED on the mission card' if total_gb > 90 else f'ESTIMATE (measured on {card}, not the mission card)'\n", " print(f'PRICE: {PLANNED/1e9:.1f}B bytes at {tps/1e3:.1f}k tok/s = {days:.1f} days — {label}; the decision is the lead\\'s (08-26: \"26 days isn\\'t reasonable\")')\n", "else:\n", " print('CPU: the ladder bench and the price line need a card (skipped)')\n", "del model; torch.cuda.empty_cache() if DEVICE == 'cuda' else None\n", "print('PREFLIGHT PASS — the data cell, the guard cell and the push cell must pass before Cell T')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C2b — WHERE DOES THE TIME GO. Run after a restart, before committing to Cell T (needs the card).\n", "# Splits one training step into: data / forward / backward / optimizer+governor.\n", "import time, torch\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.alephlm import AlephLM\n", "from geolip.alephllm.model.governor import govern_model\n", "from geolip.alephllm.train.optim import build_optimizers\n", "from geolip.alephllm.data.streams import build_stream\n", "from geolip.alephllm.data.tokenizers import build_tokenizer\n", "import geolip.alephllm as A\n", "print('package', A.__version__); assert tuple(int(x) for x in A.__version__.split('.')) >= (0,8,9), 'STALE RUNTIME'\n", "p = get_preset(CRAFT); cfg, tc = p.model, p.train\n", "torch.manual_seed(0)\n", "m = AlephLM(cfg).cuda(); m.train()\n", "opts = build_optimizers(m, tc.muon_lr, tc.muon_momentum, tc.adam_lr)\n", "def sync(): torch.cuda.synchronize()\n", "x = torch.randint(0, 255, (tc.micro_batch, cfg.context), device='cuda')\n", "# warmup (autotune, lazy inits)\n", "with torch.autocast('cuda', dtype=torch.bfloat16):\n", " m(x, targets=x).loss.backward()\n", "m.zero_grad(set_to_none=True); sync()\n", "# (a) GPU compute, synthetic data: one full accumulation step\n", "t0 = time.time(); tf = tb = 0.0\n", "for _ in range(tc.grad_accum):\n", " s = time.time()\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " out = m(x, targets=x)\n", " sync(); tf += time.time() - s; s = time.time()\n", " (out.loss / tc.grad_accum).backward(); sync(); tb += time.time() - s\n", "s = time.time()\n", "torch.nn.utils.clip_grad_norm_(m.parameters(), tc.grad_clip)\n", "for o in opts: o.step()\n", "sync(); topt = time.time() - s\n", "s = time.time(); hits = govern_model(m, tc.governor_theta); sync(); tgov = time.time() - s\n", "m.zero_grad(set_to_none=True)\n", "gpu_step = time.time() - t0\n", "print(f'GPU step (synthetic data): {gpu_step:.1f}s = fwd {tf:.1f} + bwd {tb:.1f} + opt {topt:.1f} + governor {tgov:.2f} (hits {hits})')\n", "print(f' -> pure-compute ceiling: {tc.micro_batch*cfg.context*tc.grad_accum/gpu_step/1e3:.0f}k tok/s')\n", "# (b) the data pipeline alone: how fast can the stream feed 2M tokens?\n", "tok = build_tokenizer(cfg.tokenizer)\n", "st = build_stream('wikitext-103', tok, cfg.context, tc.micro_batch, seed=1, role='train')\n", "st.next_batch() # open/download outside the clock\n", "n_tok = 0; s = time.time()\n", "while n_tok < 2_000_000:\n", " b = st.next_batch(); n_tok += b.numel()\n", "dt = time.time() - s\n", "print(f'stream: {n_tok/dt/1e3:.0f}k tok/s CPU -> {524288/(n_tok/dt):.1f}s of data per step if serial')\n", "print('VERDICT: whichever line dominates the observed s/step is the bottleneck.')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C2c — THE COMPILE GATE (the gate the law names, kept as a gate). Compile is RETIRED\n", "# on Blackwell BY MEASUREMENT (2026-08-26: NaN to the embeddings on every graph; the\n", "# 4090 reproduced it 2026-09-19 with 488/491 non-finite gradients at 1.45x speed).\n", "# It is not a utilization lever. Law: compile trains only if the GRAD-VECTOR parity\n", "# below passes on the training hardware — never loss-only (loss-only gating shipped\n", "# a NaN once). Expected result: GATE FAIL; eager stands.\n", "import time, torch\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.alephlm import AlephLM\n", "p = get_preset(CRAFT); cfg, tc = p.model, p.train\n", "torch.manual_seed(0)\n", "m = AlephLM(cfg).cuda(); m.train()\n", "x = torch.randint(0, 255, (tc.micro_batch, cfg.context), device='cuda')\n", "def grads(model):\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " model(x, targets=x).loss.backward()\n", " g = {n: p.grad.detach().clone() for n, p in model.named_parameters() if p.grad is not None}\n", " model.zero_grad(set_to_none=True)\n", " return g\n", "g_eager = grads(m)\n", "mc = torch.compile(m)\n", "with torch.autocast('cuda', dtype=torch.bfloat16):\n", " mc(x, targets=x).loss.backward() # compile warmup (slow once)\n", "m.zero_grad(set_to_none=True)\n", "g_comp = {}\n", "with torch.autocast('cuda', dtype=torch.bfloat16):\n", " mc(x, targets=x).loss.backward()\n", "for n, prm in m.named_parameters():\n", " if prm.grad is not None: g_comp[n] = prm.grad.detach().clone()\n", "m.zero_grad(set_to_none=True)\n", "bad = [(n, float((g_eager[n] - g_comp[n]).abs().max())) for n in g_eager\n", " if not torch.isfinite(g_comp[n]).all() or\n", " float((g_eager[n] - g_comp[n]).abs().max()) > 1e-2 * (1e-6 + float(g_eager[n].abs().max()))]\n", "nonfinite = [n for n in g_comp if not torch.isfinite(g_comp[n]).all()]\n", "print('non-finite compiled grads:', nonfinite[:5] if nonfinite else 'NONE')\n", "print('parity failures (rel>1e-2):', bad[:5] if bad else 'NONE')\n", "torch.cuda.synchronize(); t0 = time.time()\n", "for _ in range(3):\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " mc(x, targets=x).loss.backward()\n", " m.zero_grad(set_to_none=True)\n", "torch.cuda.synchronize()\n", "sps = (time.time() - t0) / 3\n", "print(f'compiled micro fwd+bwd: {sps:.2f}s -> projected step ~{sps*tc.grad_accum:.0f}s '\n", " f'({tc.micro_batch*cfg.context*tc.grad_accum/(sps*tc.grad_accum)/1e3:.0f}k tok/s)')\n", "if not nonfinite and not bad:\n", " print('GATE PASS — set TrainConfig.compile=True for the session (p.train.compile = True before prepare)')\n", "else:\n", " print('GATE FAIL — eager stands; report the lines above')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C2d — INSIDE ONE HUB: per-op forward timing on the real card. Names the slow op.\n", "import time, torch, torch.nn.functional as F\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.attention import CausalSplatHUB\n", "from geolip.alephllm.model.bank import AnchoredBank\n", "p = get_preset(CRAFT); cfg, tc = p.model, p.train\n", "torch.manual_seed(0)\n", "hub = CausalSplatHUB(cfg.d_model, cfg.hub_K, cfg.hub_D, cfg.tau,\n", " chunk=cfg.hub_chunk, n_const=cfg.hub_const).cuda()\n", "bank = AnchoredBank(cfg.d_model, cfg.bank_experts, cfg.bank_ff, cfg.tau, cfg.gate_init).cuda()\n", "B, n, d = tc.micro_batch, cfg.context, cfg.d_model\n", "x = torch.randn(B, n, d, device='cuda')\n", "def t(fn, reps=3):\n", " with torch.no_grad(), torch.autocast('cuda', dtype=torch.bfloat16):\n", " fn(); torch.cuda.synchronize()\n", " s = time.time()\n", " for _ in range(reps): fn()\n", " torch.cuda.synchronize()\n", " return (time.time() - s) / reps\n", "with torch.no_grad(), torch.autocast('cuda', dtype=torch.bfloat16):\n", " qc, kc = hub._code_cat_qk(x); v = hub.v(x)\n", " C = min(hub.chunk, n); pad = (-n) % C; nc = (n + pad) // C\n", " vc = (F.pad(v, (0,0,0,pad)) if pad else v).view(B, nc, C, d)\n", " mask = hub._mask(C, x.device, vc.dtype)\n", " K2 = qc.shape[-1]\n", " qv = (F.pad(qc, (0,0,0,pad)) if pad else qc).view(B, nc, C, K2)\n", " kv = (F.pad(kc, (0,0,0,pad)) if pad else kc).view(B, nc, C, K2)\n", " S = torch.einsum('bick,bicd->bikd', kv, vc)\n", " L = hub._prefix(nc, qc.device, qc.dtype)\n", " P = torch.matmul(L, S.reshape(B, nc, -1)).view_as(S)\n", " att = torch.einsum('bick,bijk->bicj', qv, kv) * mask\n", "rows = [\n", " ('code_cat_qk (1 softmax)', t(lambda: hub._code_cat_qk(x))),\n", " ('v proj ', t(lambda: hub.v(x))),\n", " ('S einsum (write) ', t(lambda: torch.einsum('bick,bicd->bikd', kv, vc))),\n", " ('prefix matmul P ', t(lambda: torch.matmul(L, S.reshape(B, nc, -1)))),\n", " ('att within-chunk ', t(lambda: torch.einsum('bick,bijk->bicj', qv, kv) * mask)),\n", " ('num read (qc.P + att.v)', t(lambda: torch.einsum('bick,bikd->bicd', qv, P) + att @ vc)),\n", " ('FULL hub.forward ', t(lambda: hub(x))),\n", " ('bank.forward ', t(lambda: bank(x))),\n", "]\n", "for name, dt in rows: print(f'{name} {dt*1e3:8.1f} ms')\n", "d_ = dict(rows)\n", "print(f\"-> {cfg.n_layers} layers x (hub+bank) = {(d_['FULL hub.forward '] + d_['bank.forward '])*cfg.n_layers:.1f}s per micro-forward\")\n", "print(f'-> dtypes: qc {qc.dtype}, v {v.dtype} (bf16 expected everywhere now)')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C2e — hub_chunk sweep on the real card: S/P traffic ~ 1/C, att FLOPs ~ C.\n", "# 4090 evidence (2026-08-26): flat 256–1024 wall-clock, peak memory favors larger.\n", "# The Blackwell rules; if a non-256 chunk wins >10%%, set p.model.hub_chunk before\n", "# build (config field, no arch change, checkpoint-compatible).\n", "import time, torch\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.attention import CausalSplatHUB\n", "p = get_preset(CRAFT); cfg, tc = p.model, p.train\n", "torch.manual_seed(0)\n", "x = torch.randn(tc.micro_batch, cfg.context, cfg.d_model, device='cuda')\n", "for C in (256, 512, 768, 1024, 1536):\n", " hub = CausalSplatHUB(cfg.d_model, cfg.hub_K, cfg.hub_D, cfg.tau,\n", " chunk=C, n_const=cfg.hub_const).cuda()\n", " def run():\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " loss = hub(x).square().mean()\n", " loss.backward(); hub.zero_grad(set_to_none=True)\n", " run(); torch.cuda.synchronize()\n", " s = time.time()\n", " for _ in range(3): run()\n", " torch.cuda.synchronize()\n", " print(f'chunk {C:5d}: {(time.time()-s)/3*1e3:7.1f} ms fwd+bwd/layer '\n", " f'peak {torch.cuda.max_memory_allocated()/2**30:.1f} GB')\n", " torch.cuda.reset_peak_memory_stats()\n", " del hub; torch.cuda.empty_cache()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C2f — THE RESIDUAL: full-model micro fwd+bwd vs the hub+bank sum, plus a\n", "# micro_batch=8 fit/speed probe. C2d/C2e see only hub+bank; the step arithmetic\n", "# leaves a remainder unaccounted (LayerNorms x2 per block, embed, head, loss, trainer\n", "# overhead). This names it — and if micro 8 fits, per-micro fixed costs halve.\n", "import time, torch\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.alephlm import AlephLM\n", "p = get_preset(CRAFT); cfg, tc = p.model, p.train\n", "torch.manual_seed(0)\n", "model = AlephLM(cfg).cuda()\n", "model.train()\n", "\n", "def micro(B):\n", " idx = torch.randint(0, 256, (B, cfg.context), device='cuda')\n", " tgt = torch.randint(0, 256, (B, cfg.context), device='cuda')\n", " def run():\n", " with torch.autocast('cuda', dtype=torch.bfloat16):\n", " out = model(idx, targets=tgt)\n", " out.loss.backward()\n", " model.zero_grad(set_to_none=True)\n", " try:\n", " run(); torch.cuda.synchronize()\n", " torch.cuda.reset_peak_memory_stats()\n", " s = time.time()\n", " for _ in range(2): run()\n", " torch.cuda.synchronize()\n", " ms = (time.time()-s)/2*1e3\n", " pk = torch.cuda.max_memory_allocated()/2**30\n", " print(f'micro_batch {B}: {ms/1e3:6.2f} s fwd+bwd peak {pk:.1f} GB '\n", " f'-> step ~{ms/1e3 * (64//B):5.1f} s at {64*cfg.context/1e3:.0f}k tok (x{64//B} micros)')\n", " return ms\n", " except torch.cuda.OutOfMemoryError:\n", " print(f'micro_batch {B}: OOM'); torch.cuda.empty_cache(); return None\n", "\n", "m4 = micro(tc.micro_batch)\n", "# hub+bank accounting at this chunk (from C2e's per-layer fwd+bwd + one\n", "# recompute-fwd under ckpt) vs the measured full micro:\n", "if m4:\n", " print(f'-> subtract (C2e ms + C2d hub-fwd ms) x {cfg.n_layers} from the micro above:')\n", " print(' the remainder IS the residual (LN/embed/head/loss/overhead).')\n", "micro(8) # fits? halves per-micro fixed costs at identical tokens/step\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C-DATA — THE DATA PLANE at DATA_SCALE: stage budgets, every finite corpus's epochs\n", "# per stage BEFORE the mix ships (the epoch-cap law), the rebalance under the chosen\n", "# rule, the cross-stage epoch column, and the coverage-audit gate. Pure computation.\n", "from geolip.alephllm.data import curriculum as C\n", "# every (cap, rule) pair, for the record beside the chosen one: cap = epochs allowed per finite corpus per\n", "# stage; a rule says where a capped corpus's weight goes; \"held\" = bytes taken out of the stages and given to\n", "# general text (the hold rule only); \"cross-stage\" = a corpus's epochs summed over the whole curriculum where\n", "# that sum still exceeds the cap (the law is per stage; the sum is a measured fact, recorded)\n", "print(f\" {'cap / rule':<18}{'components moved':>17}{'held':>8} cross-stage epochs over the cap\")\n", "for cap in (2.0, 4.0):\n", " for rule in C.REBALANCE_RULES:\n", " s = C.scaled_curriculum(DATA_SCALE, cap, rule)\n", " moved = [r for r in s['table'] if r[2] != r[3]]\n", " hot = {n: e for n, e in s['cross_stage'].items() if e > cap}\n", " mark = ' <- the record\\'s rule' if (cap == EPOCH_CAP and rule == REBALANCE_TO) else ''\n", " print(f\" {f'{cap:g} / {rule}':<18}{len(moved):>17}{s['held_tokens']/1e9:>7.1f}B {hot or 'none'}{mark}\")\n", "print()\n", "sc = C.scaled_curriculum(DATA_SCALE, EPOCH_CAP, REBALANCE_TO)\n", "print(f'scale x{DATA_SCALE:g} · cap {sc[\"epoch_cap\"]:g} epochs · rule {REBALANCE_TO} · stages '\n", " f'{sum(sc[\"stage_tokens\"].values())/1e9:.2f}B' + (f' · held {sc[\"held_tokens\"]/1e9:.1f}B -> fineweb_main' if sc['held_tokens'] else ''))\n", "if sc['flags']:\n", " print('FLAGS', sc['flags'])\n", "print(f\"{'stage':<14}{'component':<20}{'w before':>9}{'w after':>9}{'ep before':>10}{'ep after':>9}\")\n", "for st, n, wb, wa, eb, ea in sc['table']:\n", " mark = ' <- moved' if wb != wa else ''\n", " print(f'{st:<14}{n:<20}{wb:>9.3f}{wa:>9.3f}{eb:>10.2f}{ea:>9.2f}{mark}')\n", "print('cross-stage epochs (total over the curriculum):', sc['cross_stage'])\n", "for st, mix in sc['mixes'].items():\n", " ballast = sum(w for n, w in mix if n in C.NATURAL)\n", " top = max([w for n, w in mix if n.endswith('-synth')], default=0.0)\n", " assert ballast >= C.MIN_BALLAST - 1e-9, (st, ballast, 'ballast law')\n", " assert top <= C.MAX_GENERATOR_SHARE + 1e-9, (st, top, 'generator share law')\n", "print('ballast >= 30% and every generator <= 35% in every stage: OK')\n", "print('NOTE: the infinite recipients (cosmo-young, fineweb-good, wikipedia-en, gutenberg) are infinite by the '\n", " 'registry; their supply at the 4x demand is OWED a measurement (the coverage audit).')\n", "if COVERAGE_AUDIT is None:\n", " print('COVERAGE AUDIT: not run — the mix-level atlas audit (WeightField.caveat / coverage set-ops over N sampled rows per '\n", " 'source) is the next data-plane build; the design\\'s step 1 is incomplete until it runs (Cell T status)')\n", "else:\n", " print('coverage audit report:', COVERAGE_AUDIT)\n", "DATA_PLANE = C.data_plane(DATA_SCALE, EPOCH_CAP, REBALANCE_TO)\n", "print('data plane fingerprint (asserted on every resume):', DATA_PLANE)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C-GUARD — THE RED-FLAG CORE from the shipped L6 certification ledger. A guard halts\n", "# ONLY with a certification on record (fires on its own induced fault within 600 steps\n", "# AND silent on both healthy seeds); everything else is watched. The library's census\n", "# flags fire on healthy births — recorded, never interrupts. G4-G6 are boundary reads.\n", "import json, os\n", "from huggingface_hub import hf_hub_download\n", "from geolip.alephllm.train.guards import GuardConfig, certification_from_ledger, GUARDS\n", "from geolip.alephllm.presets import TRAINING_REPO\n", "LEDGER_PATH = 'v3_preflight/l6_guards/v3_l6_guards_ledger.json'\n", "led = None\n", "for cand in ([os.environ.get('ALEPHLLM_L6_LEDGER')] if os.environ.get('ALEPHLLM_L6_LEDGER') else []):\n", " if os.path.exists(cand):\n", " led = json.load(open(cand, encoding='utf-8')); print('ledger (local):', cand)\n", "if led is None:\n", " try:\n", " led = json.load(open(hf_hub_download(TRAINING_REPO, LEDGER_PATH, token=HF_TOKEN), encoding='utf-8'))\n", " print('ledger (hub):', LEDGER_PATH)\n", " except Exception as e:\n", " print('ledger unavailable:', repr(e)[:200])\n", "cert = certification_from_ledger(led or {})\n", "V = (led or {}).get('verdict', {}).get('certification', {})\n", "print(f\"{'guard':<12}{'fault':>6}{'first fire':>11}{'latency':>8}{'healthy silent':>15}{'CERTIFIED':>10}\")\n", "for g, r in V.items():\n", " print(f\"{g:<12}{str(r.get('fault_arm')):>6}{str(r.get('first_fire')):>11}{str(r.get('latency')):>8}\"\n", " f\"{str(r.get('silent_on_healthy')):>15}{str(r.get('CERTIFIED')):>10}\")\n", "if not V:\n", " print('NO VERDICT in the ledger (the certification run has not finished, or the ledger is missing)')\n", "print('modes:', cert['modes'])\n", "if not any(m == 'halt' for m in cert['modes'].values()):\n", " print('ZERO certified guards in this ledger: every guard runs in watch mode — recorded in the manifest at Cell T')\n", "GUARD_CONFIG = GuardConfig(modes=cert['modes'], certification=cert['certification'],\n", " census_dataset='synthetic' if LOCAL else 'fineweb-edu',\n", " ref_window=(4, 16) if LOCAL else (1000, 2000),\n", " census_every=4 if LOCAL else 100)\n", "print('reference window (steps]:', GUARD_CONFIG.ref_window, '· norm sample every', 'log_every', '· census every', GUARD_CONFIG.census_every,\n", " 'on one pinned batch from', GUARD_CONFIG.census_dataset, 'with every arm masked')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C-PUSH — THE PUSH PROBE (the ship-complete law): a write to the training repo under\n", "# /v3_preflight/ and a read-back BEFORE any training. Tokenless = disarmed.\n", "import json, time\n", "from huggingface_hub import HfApi\n", "from geolip.alephllm.presets import TRAINING_REPO\n", "PUSH_OK = False\n", "if HF_TOKEN:\n", " api = HfApi(token=HF_TOKEN)\n", " path = f'{CRAFT}/v3_preflight/push_probe.json'\n", " body = json.dumps({'craft': CRAFT, 'utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),\n", " 'depth': DEPTH, 'data_plane': DATA_PLANE, 'modes': GUARD_CONFIG.modes}).encode()\n", " api.upload_file(path_or_fileobj=body, path_in_repo=path, repo_id=TRAINING_REPO, commit_message='v3 push probe')\n", " files = set(api.list_repo_files(TRAINING_REPO))\n", " PUSH_OK = path in files\n", " print('push probe', 'READ BACK OK' if PUSH_OK else 'NOT VISIBLE after upload', path)\n", "else:\n", " print('TOKENLESS: no probe written; the mission Cell T stays DISARMED (the toy path proceeds on local artifacts)')\n", "assert PUSH_OK or LOCAL, 'Cell T needs a verified push path'\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C3 (T) — THE SESSION. Resume-first; stops at stage boundaries; every boundary ships a\n", "# checkpoint (HubSync), the arm anchors, AND a report JSON (this cell). A CERTIFIED guard\n", "# halt returns cleanly: the position is archived under its own name, latest.pt stays at\n", "# the last healthy checkpoint, the manifest records the halt, and the run refuses to\n", "# continue until RESUME_AFTER_HALT is set for a session (read the archive first).\n", "import gc, json, os, time, torch\n", "from geolip.alephllm import prepare\n", "from geolip.alephllm.train import probes\n", "from geolip.alephllm.train.instruments import model_census, toggle_ledger, special_token_gauge\n", "from geolip.alephllm.model.governor import govern_model\n", "assert PUSH_OK or LOCAL, 'the push probe did not pass'\n", "rows = preflight_status()\n", "if not LOCAL and not all(ok for name, ok, gate in rows if gate == 'gates the start'):\n", " raise SystemExit('the design\\'s pre-flight (routine step 1: the data plane assembled) is incomplete — the OPEN rows above '\n", " 'are builds/tests on the queue, not decisions; the mission run starts when they land')\n", "MAX_HOURS = 10.5 if not LOCAL else 0.2\n", "\n", "ARM_PROGRAM = None\n", "if STAGE_ARMS:\n", " from geolip.alephllm.train.arms import StageArm, ArmProgramConfig, StageArmProgram, LAWFUL_16x8, WIDE_1024\n", " spec = {'LAWFUL_16x8': LAWFUL_16x8, 'WIDE_1024': WIDE_1024}[ARM_SPEC]\n", " if spec['K'] > 2 * spec['D']:\n", " print(f'ARM_SPEC {ARM_SPEC}: K {spec[\"K\"]} > 2D — a flagged exception to the supply law (the certified ladder geometry)')\n", " ARM_PROGRAM = StageArmProgram(ArmProgramConfig(\n", " arms=[StageArm(name, phase, spec=dict(spec), lam=float(lam), seed=int(seed)) for name, phase, lam, seed in STAGE_ARMS],\n", " offdomain_dataset='synthetic' if LOCAL else 'fineweb-edu', quiet_trunk_grad=bool(QUIET_TRUNK_GRAD)))\n", " print('stage arms:', [(a.name, a.phase, a.lam) for a in ARM_PROGRAM.cfg.arms], '· trunk-live form: 0 seeds (the v3 rung; frozen-trunk certified 2 seeds)')\n", "\n", "run = prepare(PRESETS[CRAFT], hf_token=HF_TOKEN, out_dir=OUT_DIR, guard=GUARD_CONFIG, arms=ARM_PROGRAM,\n", " resume=os.environ.get('ALEPHLLM_NB_FRESH') != '1', device=DEVICE)\n", "if run.step == 0 and HEAD_BIRTH == 'revival_birth':\n", " # the born-with-function solve at step 0 (the C7 arm-B form): fit the head address in closed\n", " # form on a birth sample, freeze the address, log provenance — a boundary-write at birth\n", " from geolip.alephllm.train.revival import revive_head\n", " m = run.raw_model.float(); hs, bl, ys = [], [], []\n", " hook = m.nf.register_forward_hook(lambda mod, i, o: hs.append(o.detach().float().reshape(-1, o.shape[-1])))\n", " with torch.no_grad():\n", " for xb in run._val():\n", " out = m(xb[:, :-1], disable_head_aleph=True)\n", " bl.append(out.logits.float().reshape(-1, m.cfg.vocab_size)); ys.append(xb[:, 1:].reshape(-1))\n", " hook.remove()\n", " prov = revive_head(m, torch.cat(hs), torch.cat(bl), torch.cat(ys), theta_deg=45.0)\n", " m.head.proj.weight.requires_grad_(False); m.head.addr.codebook.requires_grad_(False)\n", " run.manifest.note(f'HEAD_BIRTH revival_birth at step 0: {json.dumps(prov, default=float)[:400]}')\n", " print('head solved at birth (provenance in the manifest); address frozen; W_s trains')\n", "elif HEAD_BIRTH == 'born_null':\n", " run.manifest.note('HEAD_BIRTH born_null: the 2s birth form (buried 3/3 on record) — a flagged exception to S-1004')\n", "REPORT_DIR = os.path.join(run.out_dir, 'reports', 'v3'); os.makedirs(REPORT_DIR, exist_ok=True)\n", "_prev_anneal = {}\n", "\n", "def boundary_report(run, tag):\n", " m, dev = run.raw_model, run.device\n", " m.eval()\n", " with torch.no_grad():\n", " pr = probes.run_all(m, run.tokenizer, dev)\n", " census = model_census(m, run._sample_batch()) # a VAL sample, never training rows\n", " ledger = toggle_ledger(m, run._val())\n", " arms = None\n", " if run.arms is not None and run.arms.attached:\n", " with run.arms.all_off():\n", " bare = toggle_ledger(m, run._val())\n", " ledger['bpb_arms_off'] = bare['bpb_full']; ledger['toggle_arms_off'] = bare['bpb_full'] - ledger['bpb_full']\n", " arms = run.arms.gauges(run._val(), None)\n", " sp = special_token_gauge(m, run._val())\n", " erank = {str(L): census['layers'][L].get('hidden_erank') for L in census['layers']}\n", " rep = {'tag': tag, 'step': run.step, 'tokens': run.manifest.tokens_seen, 'phase': tag,\n", " 'next_phase': (run.manifest.current_phase() or {}).get('name'), 'lr_mult': run._phase_mult(tag),\n", " 'probes': pr, 'census_flags': census.get('flags'), 'erank_profile': erank, 'ledger': ledger, 'special': sp,\n", " 'governor_hits_cum': getattr(run, '_gov_hits', 0),\n", " 'crowd_check_extra_hits': govern_model(m, run.tc.governor_theta) if run.tc.governor else None,\n", " 'guard': run.guard.summary() if run.guard is not None else None, 'arms': arms,\n", " 'arms_disabled': dict(run.arms.disabled) if run.arms is not None else None,\n", " 'halt': run.manifest.halt}\n", " # boundary reads G5/G6 (WATCH; no interrupt of record): G5 = a member past the quiet bar\n", " # (+.012 fineweb) or blending (selectivity <= 1.5); G6 = termination regression across\n", " # anneal checkpoints (doc/reset bpb rising) — recorded with fired=False until a rule is minted\n", " if run.guard is not None:\n", " if arms:\n", " bad = {n: r for n, r in arms.items() if isinstance(r, dict) and (r.get('fineweb_delta', 0) > 0.012)}\n", " run.guard.record_boundary_read('G5', tag, {'fired': False, 'step': run.step, 'past_quiet_bar': list(bad)})\n", " if tag.startswith('anneal') and sp:\n", " prev = _prev_anneal.get('sp')\n", " delta = {k: sp[k] - prev[k] for k in ('doc_bpb', 'reset_bpb') if prev and k in sp and k in prev}\n", " run.guard.record_boundary_read('G6', tag, {'fired': False, 'step': run.step, 'delta_vs_previous': delta})\n", " _prev_anneal['sp'] = sp\n", " if run.arms is not None:\n", " for n in run.arms.attached:\n", " ck = run.arms.anchor(n, CRAFT, run.step)\n", " apath = os.path.join(run.out_dir, 'arms', f'{n}_step{run.step}.safetensors'); os.makedirs(os.path.dirname(apath), exist_ok=True)\n", " ck.save(apath); run.hub._up(apath, f'arms/{n}_step{run.step}.safetensors')\n", " body = json.dumps(rep, default=float)\n", " with open(os.path.join(REPORT_DIR, f'{tag}_step{run.step}.json'), 'w', encoding='utf-8') as f:\n", " f.write(body)\n", " run.hub.upload_bytes(body.encode(), f'reports/v3/{tag}_step{run.step}.json')\n", " print(probes.report(pr)); m.train()\n", " return rep\n", "\n", "# 0.8.1 driver contract + the v3 halt: train(stop_at_boundary=True) RETURNS at every\n", "# phase boundary (run._last_boundary), on a manual stop (run._interrupted — NEVER\n", "# auto-resumed) and on a certified guard halt (run._guard_halt — NEVER auto-resumed).\n", "t_end = time.time() + MAX_HOURS * 3600\n", "while time.time() < t_end:\n", " hours_left = (t_end - time.time()) / 3600\n", " if hours_left < (0.2 if not LOCAL else 0.0):\n", " break\n", " # the boundary gates the design names: a stage-arm boundary needs the lawful geometry certified; the\n", " # anneal boundary needs the multiplier from the screen — both are tests on the queue, checked here\n", " nxt = run.manifest.current_phase()\n", " if nxt is not None and not LOCAL:\n", " if any(a.phase == nxt['name'] for a in (ARM_PROGRAM.cfg.arms if ARM_PROGRAM else [])) and not ARM_SPEC_CERTIFIED:\n", " print(f\"BOUNDARY GATE: '{nxt['name']}' attaches a stage arm and the lawful geometry is not yet certified (2 seeds) — \"\n", " \"the session stops here; the certification is on the card queue\"); break\n", " if nxt['name'].startswith('anneal') and not isinstance(ANNEAL_LR_SCALE, (int, float)):\n", " print(f\"BOUNDARY GATE: '{nxt['name']}' needs the anneal multiplier from the screen — the session stops here\"); break\n", " run.train(max_hours=hours_left, stop_at_boundary=True, resume_after_halt=RESUME_AFTER_HALT)\n", " RESUME_AFTER_HALT = False # one clearance per session, never standing\n", " if getattr(run, '_interrupted', False):\n", " print('manual stop - no auto-resume; resume state is on the hub'); break\n", " halt = getattr(run, '_guard_halt', None)\n", " if halt:\n", " boundary_report(run, f\"guard_{halt['guard']}\")\n", " print(f\"GUARD HALT {halt['guard']} at step {halt['step']:,} in '{halt['phase']}' — archive {halt['archive']}; \"\n", " \"the session ends here; the driver (a human) decides: read the archive and the report, then a new session \"\n", " \"with RESUME_AFTER_HALT=True continues from the last healthy checkpoint (a rewind target = the last boundary archive)\")\n", " break\n", " tag = getattr(run, '_last_boundary', None) or 'session_cap'\n", " boundary_report(run, tag)\n", " if run.manifest.current_phase() is None:\n", " print('curriculum complete — the two-phase anneal was planned from birth and has run'); break\n", " gc.collect(); torch.cuda.empty_cache() if DEVICE == 'cuda' else None\n", "print('session over — resume state on the hub' if HF_TOKEN else 'session over — resume state local')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C5 — SAMPLES WITH VISIBLE SPECIALS. Run any time after prepare() (Cell T\n", "# section); uses the in-memory model. Specials render as ⟦NAME⟧ — the eyeball\n", "# check that the craft emits DOC at document ends (and, come anneal, END at\n", "# turn closes). decode_visible is display-only and spoofable by content;\n", "# the ids are the truth — never parse this string.\n", "import torch\n", "from geolip.alephllm.data.special_tokens import (DOC, decode_visible,\n", " render_chat_ids)\n", "\n", "prompt = \"The history of astronomy begins with\"\n", "ids = torch.tensor(run.tokenizer.encode(prompt), device=run.device)[None]\n", "run.raw_model.eval()\n", "with torch.no_grad():\n", " out = run.raw_model.generate(ids, max_new=256, temperature=0.8)\n", "print(decode_visible(out[0].tolist()))\n", "\n", "# chat-frame probe (meaningful once anneal has run; before that the frame\n", "# is out-of-distribution and the continuation shows exactly that):\n", "chat = render_chat_ids([{\"role\": \"user\", \"content\": \"Who are you?\"}])\n", "import numpy as np\n", "from geolip.alephllm.data.special_tokens import MODEL as MODEL_ID\n", "seed = np.concatenate([chat, np.array([MODEL_ID])])\n", "ids = torch.tensor(seed, device=run.device)[None]\n", "with torch.no_grad():\n", " out = run.raw_model.generate(ids, max_new=128, temperature=0.8)\n", "print(\"\\n--- chat probe ---\")\n", "print(decode_visible(out[0].tolist()))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C6 — THE ANNEAL WATCH. The two-phase anneal (nochat then chat) is PLANNED FROM BIRTH in\n", "# the v3 preset (no activation step); this cell shows the multiplier in force and the\n", "# per-checkpoint gauge table across the anneal reports (G6 = the termination read).\n", "import json, glob, os\n", "print('anneal LR multiplier:', run._phase_mult('anneal_nochat'), '/', run._phase_mult('anneal_mix'),\n", " '(1.0 = the flat form; the value is the lead\\'s — recipe term H, 0 seeds)')\n", "reps = sorted(glob.glob(os.path.join(REPORT_DIR, 'anneal*_step*.json')))\n", "print(f\"{'report':<28}{'step':>8}{'val bpb':>9}{'doc bpb':>9}{'reset bpb':>10}{'head':>8}\")\n", "for f in reps:\n", " r = json.load(open(f, encoding='utf-8')); led = r.get('ledger') or {}; sp = r.get('special') or {}\n", " v = led.get('bpb_full'); d = sp.get('doc_bpb'); rs = sp.get('reset_bpb'); h = led.get('toggle_head_aleph_off')\n", " print(f\"{os.path.basename(f)[:27]:<28}{r['step']:>8}{(f'{v:.3f}' if isinstance(v, float) else '--'):>9}\"\n", " f\"{(f'{d:.2f}' if isinstance(d, float) else '--'):>9}{(f'{rs:.2f}' if isinstance(rs, float) else '--'):>10}\"\n", " f\"{(f'{h:+.3f}' if isinstance(h, float) else '--'):>8}\")\n", "if run.guard is not None and run.guard.boundary_reads.get('G6'):\n", " print('G6 termination reads:', run.guard.boundary_reads['G6'])\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# C4 — GROWTH TABLE from the boundary reports (reports/v3). Probe values are DICTS\n", "# ({'acc': ...}); op reports carry no 'probes' — skipped, not crashed. Guard and arm\n", "# columns added for v3.\n", "import json, glob, os\n", "from huggingface_hub import HfApi, hf_hub_download\n", "from geolip.alephllm.presets import TRAINING_REPO\n", "if LOCAL or not HF_TOKEN:\n", " files = sorted(glob.glob(os.path.join(REPORT_DIR, '*.json')))\n", " load = lambda f: json.load(open(f, encoding='utf-8'))\n", "else:\n", " api = HfApi(token=HF_TOKEN)\n", " files = sorted(f for f in api.list_repo_files(TRAINING_REPO) if f.startswith(f'{CRAFT}/reports/v3/') and f.endswith('.json'))\n", " load = lambda f: json.load(open(hf_hub_download(TRAINING_REPO, f, token=HF_TOKEN), encoding='utf-8'))\n", "rows = []\n", "for f in files:\n", " r = load(f)\n", " pr = r.get('probes')\n", " if not pr:\n", " continue\n", " accs = {k.split('_', 1)[0]: round(v.get('acc', v) if isinstance(v, dict) else v, 2) for k, v in pr.items()}\n", " led = r.get('ledger') or {}; sp = r.get('special') or {}\n", " g = r.get('guard') or {}; fired = ','.join(sorted((g.get('fired') or {}).keys())) or '-'\n", " arms = r.get('arms') or {}\n", " arm_col = ' '.join(f\"{n}:{v.get('fineweb_delta'):+.3f}\" for n, v in arms.items() if isinstance(v, dict) and 'fineweb_delta' in v) or '-'\n", " rows.append((r['tag'][:14], r['step'], round(r['tokens'] / 1e9, 3), led.get('bpb_full'), led.get('toggle_head_aleph_off'),\n", " sp.get('doc_bpb'), fired, arm_col, accs))\n", "rows.sort(key=lambda x: x[1])\n", "print(f\"{'stage':<15}{'step':>7}{'tok(B)':>8}{'val':>7}{'head':>8}{'doc':>6} {'guards':<8}{'arms (fineweb delta)':<28}probes\")\n", "for tag, step, tk, val, hd, doc, fired, arm_col, accs in rows:\n", " v = f'{val:.3f}' if isinstance(val, float) else ' --'\n", " h = f'{hd:+.3f}' if isinstance(hd, float) else ' --'\n", " d = f'{doc:.1f}' if isinstance(doc, float) else ' --'\n", " print(f'{tag:<15}{step:>7}{tk:>8}{v:>7}{h:>8}{d:>6} {fired:<8}{arm_col:<28}{accs}')\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# CARD-ONLY\n", "# C7 — HEAD SCREEN: the instrument for the HEAD_BIRTH decision (S-1004 born-with-function).\n", "# Optional, not part of the routine; runs on a free card at the chosen depth:\n", "# can a live-from-birth head earn contribution in ~1k steps, or does even a\n", "# live head lose to the untrained linear base? Verdict: init-law fix vs v3\n", "# architectural modification. 4 arms x N_STEPS on fresh throwaway crafts;\n", "# arm B/C installs at step 0 = BOUNDARY-WRITE at birth (provenance logged).\n", "import gc, io, json, math, time, torch\n", "import torch.nn.functional as F\n", "from huggingface_hub import HfApi, hf_hub_download\n", "from safetensors.torch import load_file\n", "from geolip.alephllm import get_preset\n", "from geolip.alephllm.model.alephlm import AlephLM\n", "from geolip.alephllm.data.tokenizers import ByteTrigramTokenizer\n", "from geolip.alephllm.data.streams import build_stream\n", "from geolip.alephllm.train.optim import build_optimizers, apply_lr\n", "from geolip.alephllm.train.revival import revive_head\n", "from geolip.alephllm.train.instruments import head_liveness\n", "\n", "N_STEPS, GAUGE_EVERY, SEED = 1200, 300, 1337\n", "ARMS = [\"A_born_null\", \"B_revival_birth\", \"C_matured_transplant\",\n", " \"D_live_address_only\"]\n", "MATURED = \"mini-beatrix-2s/checkpoints/step_00061422.safetensors\" # the 2s MISSION FINAL 16.1B - the matured head (d1024, transplantable)\n", "pcfg = PRESETS[CRAFT]\n", "tok = ByteTrigramTokenizer()\n", "api = HfApi(token=HF_TOKEN)\n", "vs = build_stream(\"wikitext-103\", tok, pcfg.model.context, 4,\n", " seed=SEED + 9999, role=\"val\")\n", "VAL = [vs.next_batch().cuda() for _ in range(2)]\n", "\n", "def head_toggle(model):\n", " tot = {True: 0.0, False: 0.0}\n", " with torch.no_grad(), torch.autocast(\"cuda\", dtype=torch.bfloat16):\n", " for xb in VAL:\n", " for off in (True, False):\n", " out = model(xb[:, :-1], targets=xb[:, 1:],\n", " disable_head_aleph=off)\n", " tot[off] += float(out.loss) / len(VAL)\n", " return (tot[True] - tot[False]) / math.log(2)\n", "\n", "def collect_birth(model):\n", " hs, bl, ys = [], [], []\n", " hook = model.nf.register_forward_hook(\n", " lambda m, i, o: hs.append(o.detach().float().reshape(-1, o.shape[-1])))\n", " with torch.no_grad(), torch.autocast(\"cuda\", dtype=torch.bfloat16):\n", " for xb in VAL:\n", " out = model(xb[:, :-1], disable_head_aleph=True)\n", " bl.append(out.logits.float().reshape(-1, 256))\n", " ys.append(xb[:, 1:].reshape(-1))\n", " hook.remove()\n", " return torch.cat(hs), torch.cat(bl), torch.cat(ys)\n", "\n", "for arm in ARMS:\n", " torch.manual_seed(SEED)\n", " m = AlephLM(pcfg.model).cuda()\n", " prov = {\"arm\": arm, \"steps\": N_STEPS, \"seed\": SEED, \"gauges\": []}\n", " if arm in (\"B_revival_birth\", \"D_live_address_only\"):\n", " m = m.float()\n", " H, B, Y = collect_birth(m)\n", " prov[\"revival\"] = revive_head(m, H, B, Y, theta_deg=45.0)\n", " if arm == \"D_live_address_only\":\n", " m.head.w_s.weight.data.zero_()\n", " if arm == \"C_matured_transplant\":\n", " sd = load_file(hf_hub_download(\n", " \"AbstractPhil/alephllm-mini-beatrix-training\", MATURED))\n", " for k in (\"head.proj.weight\", \"head.addr.codebook\", \"head.w_s.weight\"):\n", " m.state_dict()[k].copy_(sd[k])\n", " if arm != \"A_born_null\": # frozen address per doctrine\n", " m.head.proj.weight.requires_grad_(False)\n", " m.head.addr.codebook.requires_grad_(False)\n", " m = m.cuda().to(torch.float32)\n", " opts = build_optimizers(m, pcfg.train.muon_lr, pcfg.train.muon_momentum,\n", " pcfg.train.adam_lr)\n", " st = build_stream(\"wikitext-103\", tok, pcfg.model.context, 8, seed=SEED)\n", " m.train()\n", " for step in range(1, N_STEPS + 1):\n", " apply_lr(opts, [pcfg.train.muon_lr, pcfg.train.adam_lr], step, 200)\n", " for o in opts:\n", " o.zero_grad(set_to_none=True)\n", " xb = st.next_batch().cuda()\n", " with torch.autocast(\"cuda\", dtype=torch.bfloat16):\n", " out = m(xb[:, :-1], targets=xb[:, 1:])\n", " out.loss.backward()\n", " torch.nn.utils.clip_grad_norm_(m.parameters(), 1.0)\n", " for o in opts:\n", " o.step()\n", " if step % GAUGE_EVERY == 0 or step == N_STEPS:\n", " m.eval()\n", " g = dict(step=step,\n", " toggle=head_toggle(m),\n", " w_s_norm=float(m.head.w_s.weight.norm()),\n", " **head_liveness(m, sample_idx=VAL[0][:2, :-1]))\n", " prov[\"gauges\"].append(g)\n", " print(f\"{arm} step {step}: toggle {g['toggle']:+.4f} \"\n", " f\"live {g['head_liveness_ratio']:.3f}x \"\n", " f\"||W_s|| {g['w_s_norm']:.1f}\")\n", " m.train()\n", " api.upload_file(\n", " path_or_fileobj=json.dumps(prov, default=float).encode(),\n", " path_in_repo=f\"{CRAFT}/reports/v3/cultivation_{arm}.json\",\n", " repo_id=\"AbstractPhil/alephllm-mini-beatrix-training\",\n", " commit_message=f\"head cultivation screen: {arm}\")\n", " del m, opts, st\n", " gc.collect(); torch.cuda.empty_cache()\n", "print(\"SCREEN COMPLETE - 4 reports under reports/v3/cultivation_*.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Post-training programs (placeholders — the lead's; nothing below is invented)\n", "- **S13 thinking**: trained post-training; think markers are a specials-plane decision (the `THINK` special is reserved in the registry); the honesty gauge is mandatory. `THINK = None`.\n", "- **S6 diversity / condensing terms** (recipe term L): `DIVERSITY = CONDENSING = None`.\n", "- **S7 tool-packaged arms** (recipe term M): `TOOL_ARMS = None`.\n", "- **C3 Level-2 arms (memory accumulators, RelayEMA)**: admitted only after data-route saturation (S11); the head form is packaged (0.8.8 `model/relay.py`), untested under demand (L8 owed).\n", "- **C5 growth clause (A6)**: routed attention from arms adjusted as the model grows — his terms, open.\n", "- **Born-in core-side objectives (D4)**: door CLOSED by L3 (the sub3 wall holds core-side at a catastrophic price, S-1905) — v3 is a steering-and-arms craft.\n", "- **Hub K-alpha gain (A3/A4/S5)**: an inference-parity-certified instrument in the docket, not in the library; training under it is gated on the lead's bed word. `HUB_GAIN = None`.\n", "\n", "### Notes\n", "- **Controls**: `-control` (pure sdpa) is registered beside the craft by C2; any v3 claim runs its twin under the same data plane.\n", "- **Monitoring keys**: `governor/hits_cum` (TB), `guard/` (the step a guard fired), `arms/abstention`, `train/lr_scale` (carries the anneal multiplier); `hub_addr.anchor_merge_pairs` must stay 0.\n", "- **Halts**: a certified guard's archive is `resume/guard__step.pt`; `resume/latest.pt` is the last healthy checkpoint; the manifest's `halt` field blocks `train()` until a session clears it.\n", "- **Kernels**: eager on Blackwell; `hub_chunk=256`; fp16 forbidden. Sessions are boundary-exact.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# POST-TRAINING — refuses until a program of record is supplied (see the notes above).\n", "for name in ('THINK', 'DIVERSITY', 'CONDENSING', 'TOOL_ARMS'):\n", " if globals().get(name) is None:\n", " print(f'{name}: no program supplied — the lead\\'s (plan S13 / S6 / S7); nothing runs')\n", " else:\n", " raise SystemExit(f'{name} is set but no post-training routine is built in 0.8.9 — the program is a design item, not a switch')\n" ] } ], "metadata": { "accelerator": "GPU", "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }