mini-beatrix-3/code: re-cut at amoe-lora 0.2.11 (9bae609: the compiled adapter chain's recompile budget and shape gate) + the stop / install / relaunch / guard scripts of the 2026-09-28 relaunch at 122,000; the pod's mission file as running

mini-beatrix-3/code/README.md

@@ -1,17 +1,23 @@
 # mini-beatrix-3 — the training code, pinned to the commits that train it
 
-This folder is a verbatim snapshot of the two libraries and the pod files behind the `mini-beatrix-3` run, taken on
-2026-09-27 while the run was at step ~106,600 of 245,670. The libraries are copies of their public GitHub commits;
-the pod files are the launch, resume, stop and shipping scripts as they run on the training machine.
+This folder is a verbatim snapshot of the two libraries and the pod files behind the `mini-beatrix-3` run, first taken
+on 2026-09-27 at step ~106,600 of 245,670 and re-cut on 2026-09-28 at step 122,000 when the arm library moved to 0.2.11
+(the section on the compiled chain below). The libraries are copies of their public GitHub commits; the pod files are the
+launch, resume, stop, install and shipping scripts as they run on the training machine.
 
 | part | what it is | where it comes from |
 |---|---|---|
 | `alephllm/` | the model, the trainer, the data plane, the instruments, the arm program and the mission loop (`geolip.alephllm`, version 0.10.5) | [github.com/AbstractEyes/alephllm](https://github.com/AbstractEyes/alephllm) @ `e8b7440` |
-| `amoe-lora/` | the arm library: the relay adapters, attach / detach / mask handles, recompute-in-backward, the compiled adapter chain, the anchor file format (`amoe.__version__` 0.2.10; its `pyproject.toml` still says 0.10.0) | [github.com/AbstractEyes/amoe-lora](https://github.com/AbstractEyes/amoe-lora) @ `f3db5d4` (branch `experimental`) |
+| `amoe-lora/` | the arm library: the relay adapters, attach / detach / mask handles, recompute-in-backward, the compiled adapter chain, the anchor file format (`amoe.__version__` 0.2.11; its `pyproject.toml` still says 0.10.0) | [github.com/AbstractEyes/amoe-lora](https://github.com/AbstractEyes/amoe-lora) @ `9bae609` (branch `experimental`; `f3db5d4` = 0.2.10 trained steps 106,000 to 122,000) |
 | `pod/mission.json` | the mission file: the craft, the data plane, the stage arms, the knobs; the trainer re-reads the refreshable keys at every phase boundary | the pod, as running |
 | `pod/launch_mission.sh` | the launch: `torchrun` data-parallel over every card on the machine, resume-first | the pod |
 | `pod/relaunch_106000.sh` | the one-command relaunch used on 2026-09-27: sets the knobs, checks the installed versions, launches, restarts the sidecar, prints the proving log lines | the pod |
 | `pod/stop_at_106000.sh` | a clean manual stop at a 2,000-step checkpoint (the resume state is exactly that checkpoint) | the pod |
+| `pod/stop_at_checkpoint.sh` | the same stop generalized to any checkpoint step (used at 122,000 on 2026-09-28) | the pod |
+| `pod/install_0211.sh` | installs the arm library at the pinned 0.2.11 commit and reads the versions back; refuses while a trainer is alive | the pod |
+| `pod/relaunch_3arms.sh` | the relaunch used on 2026-09-28: the running form, the version gate, the proving log lines, the recompile-warning count | the pod |
+| `pod/go_relaunch_0211.sh` | stop at a named checkpoint, install, relaunch, each stage gated on the previous one's report line | the pod |
+| `pod/compile_guard.sh` | polls the log for torch's recompile-limit warning (the sign of the compiled chain falling back to eager) and arms the stop-and-relaunch at the next checkpoint | the pod |
 | `pod/arm_anchor_sidecar.py` | ships every attached stage arm as a safetensors anchor at each checkpoint (the trainer's own boundary anchors ship at phase ends) | the pod |
 | `pod/write_mission_json.py` | how the mission file was written from the pre-flight screens' verdicts | the pod |
 | `pod/env.sh` | the shell environment (the token comes from the container's own environment; nothing secret is in the file) | the pod |
@@ -39,7 +45,7 @@ the pod files are the launch, resume, stop and shipping scripts as they run on t
 | `arm_spec_certified` | a gate: a phase that attaches an arm opens only when this is true; setting it false ends the session cleanly at the next boundary | true |
 | `abstain_chunks` | abstention (quiet-term) chunks per step; `null` = one per task chunk (16 of 16); an integer = that many, each scaled so the term's weight per step is unchanged | 4 since step 106,000 |
 | `adapter_recompute` | the arms' intermediates are recomputed in the backward instead of kept (about 0.3 GB per arm instead of 3 GB at this shape) | true since 106,000 |
-| `adapter_compile` | the per-block adapter chain runs through `torch.compile` with inductor's precision-cast emulation (the plain compile produced a non-finite backward on the address stage) | true since 106,000 |
+| `adapter_compile` | the per-block adapter chain runs through `torch.compile` with inductor's precision-cast emulation (the plain compile produced a non-finite backward on the address stage); since 0.2.11 the library also raises torch's recompile budget and routes only the training shapes through the compiled chain (the section below) | true since 106,000 |
 | `minted_lexicon` | the curriculum's word pool for the rule-chain stages (216 minted words), its share of stage 3 and the pair rate | see the file |
 | `head_birth` | how the aleph head is born (`revival_birth` fits it to the first batch at step 0) | `revival_birth` |
 | `l6_ledger` | the in-run guard certification record (the rank-collapse guard halts the run on its own when certified) | `v3_preflight/l6_guards/...` |
@@ -63,3 +69,18 @@ sd = load_file("arms/s1_perspective_step106000.safetensors")   # keys blocks.<bl
 
 Anchors before 2026-09-25 were converted in place from a torch-serialised form to real safetensors; `load_anchor` reads
 either form by content.
+
+## The compiled chain and its recompile budget (0.2.11)
+
+The adapter chain is compiled with static shapes, so every distinct (mask pattern, input shape) pair is one entry in torch's
+compile cache for that function. Torch's default budget is 8 recompiles per function, and when it is exceeded torch does
+not run that one call eagerly: it marks the function to bypass the compiler for the rest of the process, with a single
+warning line. The stage 2 boundary evaluation fed the armed model probe prompts of many different lengths, spent that budget
+in seconds, and from the stage 3 attach on every step ran the arms eager (10.5 s per step at three arms instead of about
+7.5) until the run was stopped at 122,000 and relaunched on 0.2.11. The fix lives in the library, not in a launch flag:
+`compiled_chain()` raises the budget (64 per function, 4,096 accumulated) before the first compile, and `chain_for(h)`
+routes only the first two distinct (batch, length) shapes seen through the compiled chain, every other shape through the
+eager chain, which is the same function. The unit test `tests/test_compiled_chain_gate.py` in the GitHub repository stacks
+three arms, runs the training shape armed and with each member masked, then forty prompt lengths, with torch configured
+to raise on any budget hit. `pod/compile_guard.sh` watches the log for the warning and arms a stop-and-relaunch if it ever
+returns.

mini-beatrix-3/code/amoe-lora/src/amoe/__init__.py

@@ -1,24 +1,24 @@
-"""amoe — aleph mixture-of-experts adapters (train/attach/align/detach).
-
-The productized adapter system from the geolip-aleph research line.
-"LoRA-style" refers to the attach/detach usage pattern, not the math:
-these are aleph-addressed patch heads, not low-rank matrices.
-"""
-from .core.adapter import AdapterSpec, RelayPatchwork, BlockWithAdapter
-from .core.address import AlephAddress
-from .core.dispatch import AnchorDispatch, BlockWithDispatch, set_mask
-from .io.checkpoint import (AnchorCheckpoint, DispatchCheckpoint,
-                            load_anchor, load_dispatch,
-                            import_legacy_keys)
-from .runtime.attach import attach, detach, AttachHandle
-from .train.config import TrainConfig, AlignConfig
-from .train.trainer import train
-from .train.aligner import align
-from .train.quiet import QuietSpec, kl_abstain, quiet_step
-from . import laws
-
-__version__ = "0.2.10"
-
-# The diffusion subsystem (amoe.diffusion) is imported lazily — its verbs
-# live under `import amoe.diffusion as ad`. Core is pure torch; diffusers
-# is only touched by bindings/samplers/data that need it.
+"""amoe — aleph mixture-of-experts adapters (train/attach/align/detach).
+
+The productized adapter system from the geolip-aleph research line.
+"LoRA-style" refers to the attach/detach usage pattern, not the math:
+these are aleph-addressed patch heads, not low-rank matrices.
+"""
+from .core.adapter import AdapterSpec, RelayPatchwork, BlockWithAdapter
+from .core.address import AlephAddress
+from .core.dispatch import AnchorDispatch, BlockWithDispatch, set_mask
+from .io.checkpoint import (AnchorCheckpoint, DispatchCheckpoint,
+                            load_anchor, load_dispatch,
+                            import_legacy_keys)
+from .runtime.attach import attach, detach, AttachHandle
+from .train.config import TrainConfig, AlignConfig
+from .train.trainer import train
+from .train.aligner import align
+from .train.quiet import QuietSpec, kl_abstain, quiet_step
+from . import laws
+
+__version__ = "0.2.11"
+
+# The diffusion subsystem (amoe.diffusion) is imported lazily — its verbs
+# live under `import amoe.diffusion as ad`. Core is pure torch; diffusers
+# is only touched by bindings/samplers/data that need it.

mini-beatrix-3/code/amoe-lora/src/amoe/core/adapter.py

@@ -1,210 +1,254 @@
-"""RelayPatchwork adapter + the hybrid-safe block wrapper.
-
-The certified geometry (16 slots x D=4 through a 64-atom aleph address,
-squared-ReLU patch head, sigmoid gate initialized at -3): ~261k params
-at d=1024. Output head zero-initialized so a fresh anchor is inert
-(modulo the documented LayerNorm-bias leak).
-
-BlockWithAdapter carries an `enabled` switch: when False, the forward
-returns the block output untouched — the single-anchor half of the
-toggle law. It also carries a `recompute` switch (0.2.7): when True,
-the adapter's intermediates are not kept for the backward pass but
-recomputed from the block output when the backward reaches them
-(torch.utils.checkpoint, non-reentrant). Same math, one extra adapter
-forward per backward, a fraction of the activation memory: at d=1024
-on 4096-token rows the kept intermediates run ~96 MB per block per
-adapter per two-row micro-batch, which is what walls a 32-block trunk
-with several stacked arms on a 32 GB card. A stack of wrappers on one
-block recomputes as one chain (0.2.8), keeping one residual copy per
-block for the whole stack.
-"""
-from __future__ import annotations
-
-import contextlib
-from dataclasses import dataclass
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from torch.utils.checkpoint import checkpoint
-
-from .address import AlephAddress
-
-
-@dataclass
-class AdapterSpec:
-    n_slots: int = 16
-    K: int = 64
-    D: int = 4
-    tau: float = 0.1
-    hidden: int = 178
-    gate_init: float = -3.0
-    zero_init_head: bool = True
-
-
-class SquaredReLU(nn.Module):
-    def forward(self, x):
-        return F.relu(x) ** 2
-
-
-class RelayPatchwork(nn.Module):
-    def __init__(self, d: int, spec: AdapterSpec | None = None):
-        super().__init__()
-        s = spec or AdapterSpec()
-        self.spec = s
-        self.n_slots = s.n_slots
-        self.proj = nn.Linear(d, s.n_slots * s.D, bias=False)
-        nn.init.orthogonal_(self.proj.weight)
-        self.addr = AlephAddress(s.K, s.D, s.tau)
-        self.consume = nn.Sequential(
-            nn.Linear(s.n_slots * s.D, s.hidden), SquaredReLU(),
-            nn.LayerNorm(s.hidden), nn.Linear(s.hidden, d))
-        if s.zero_init_head:
-            nn.init.zeros_(self.consume[-1].weight)
-        self.gate = nn.Parameter(torch.tensor(float(s.gate_init)))
-
-    def forward(self, x):
-        B, n, _ = x.shape
-        slots = self.proj(x).view(B, n, self.n_slots, self.spec.D)
-        feats = self.addr.m_hat(slots).reshape(B, n, -1)
-        return x + torch.sigmoid(self.gate) * self.consume(feats)
-
-
-def _apply_chain(h, *adapters):
-    for a in adapters:
-        h = a(h)
-    return h
-
-
-def _autocast_uncached(device_type: str):
-    """The active autocast region re-entered with its weight-cast cache OFF
-    (0.2.10), for the checkpointed chain. Autocast caches the low-precision
-    copies of parameters it casts; when an earlier pass inside the same
-    autocast region (a no_grad pass with a member masked, as the abstention
-    term runs) already cast a member's weights, the recorded forward of the
-    checkpointed chain holds no cast ops for them, while the recompute,
-    which re-enters autocast fresh, records the casts, and the checkpoint's
-    operator-list check fails ('different metadata'; torch 2.8). With the
-    cache off inside the chain both passes record the same ops. The cost is
-    one bf16 cast of each live adapter's weights per block call."""
-    try:
-        enabled = torch.is_autocast_enabled(device_type)
-    except TypeError:  # older signature: cuda only
-        enabled = device_type == "cuda" and torch.is_autocast_enabled()
-    if not enabled:
-        return contextlib.nullcontext()
-    try:
-        dtype = torch.get_autocast_dtype(device_type)
-    except (TypeError, AttributeError):
-        dtype = torch.get_autocast_gpu_dtype() if device_type == "cuda" else torch.bfloat16
-    return torch.autocast(device_type=device_type, dtype=dtype, cache_enabled=False)
-
-
-# The compiled chain (0.2.9). The aleph address's compiled backward is
-# non-finite under bf16 autocast unless inductor rounds its fused
-# intermediates to bf16 the way eager does (measured 2026-09: of the
-# adapter's stages compiled alone, only addr.m_hat fails; pure fp32 and
-# emulated casts are clean), so the switch below is set when the chain
-# is first compiled. It is process-global; set it False before the first
-# compile to opt out (not recommended for bf16 training).
-COMPILE_EMULATE_CASTS = True
-COMPILE_MODE = "default"        # reduce-overhead (CUDA graphs) fails inside a checkpointed chain
-_COMPILED_CHAIN = None
-
-
-def compiled_chain():
-    global _COMPILED_CHAIN
-    if _COMPILED_CHAIN is None:
-        if COMPILE_EMULATE_CASTS:
-            import torch._inductor.config as ic
-            ic.emulate_precision_casts = True
-        _COMPILED_CHAIN = torch.compile(_apply_chain, dynamic=False,
-                                        mode=COMPILE_MODE)
-    return _COMPILED_CHAIN
-
-
-class BlockWithAdapter(nn.Module):
-    """Wraps one decoder block; hybrid-safe (tuple or tensor output).
-
-    `recompute=True` (0.2.7) routes the adapter call through a
-    non-reentrant checkpoint whenever autograd is recording: the slot
-    projection, the address features and the consume MLP's hidden
-    activations are dropped after the forward and rebuilt from the block
-    output during the backward. The head is deterministic and
-    dropout-free, so the RNG state is not snapshotted (no device sync per
-    call) and the recomputed values are the values the forward produced.
-    Inference paths (no_grad, prefill, step) never checkpoint.
-
-    Stacked wrappers (0.2.8; an arm attached over an arm nests a wrapper
-    around a wrapper) recompute as ONE chain: the outermost wrapper with
-    recompute on runs the core block once and applies every enabled
-    adapter of the stack, innermost first, inside a single checkpoint, so
-    one residual copy per block is kept for the whole stack instead of
-    one per adapter. Each wrapper's `enabled` switch is read where it
-    sits, so masks behave exactly as in the kept path.
-
-    `compile_chain=True` (0.2.9) runs that same chain through
-    torch.compile (default mode, one graph per mask pattern, shared by
-    every block since the adapters enter as arguments), with or without
-    the recompute; see COMPILE_EMULATE_CASTS above for the precision
-    switch it needs. The compiled and the eager chain agree to bf16
-    rounding, not bit for bit: a training run that switches it on owes a
-    gradient census on its own card first.
-    """
-
-    def __init__(self, block: nn.Module, adapter: RelayPatchwork,
-                 recompute: bool = False, compile_chain: bool = False):
-        super().__init__()
-        self.block = block
-        self.adapter = adapter
-        self.enabled = True
-        self.recompute = bool(recompute)
-        self.compile_chain = bool(compile_chain)
-
-    def stack(self):
-        """(wrappers innermost first, the core block) of the stack this
-        wrapper tops."""
-        wraps, core = [], self
-        while isinstance(core, BlockWithAdapter):
-            wraps.append(core)
-            core = core.block
-        wraps.reverse()
-        return wraps, core
-
-    def forward(self, *args, **kwargs):
-        if self.compile_chain or (self.recompute and torch.is_grad_enabled()):
-            wraps, core = self.stack()
-            out = core(*args, **kwargs)
-            live = [w.adapter for w in wraps if w.enabled]
-            if not live:
-                return out
-            h = out[0] if isinstance(out, tuple) else out
-            fn = compiled_chain() if self.compile_chain else _apply_chain
-            if self.recompute and torch.is_grad_enabled():
-                with _autocast_uncached(h.device.type):
-                    y = checkpoint(fn, h, *live, use_reentrant=False,
-                                   preserve_rng_state=False)
-            else:
-                y = fn(h, *live)
-            return (y,) + out[1:] if isinstance(out, tuple) else y
-        out = self.block(*args, **kwargs)
-        if not self.enabled:
-            return out
-        if isinstance(out, tuple):
-            return (self.adapter(out[0]),) + out[1:]
-        return self.adapter(out)
-
-    # Incremental-decode passthroughs (trunks with a cached decode path,
-    # e.g. AlephLM prefill/step). The patch head is position-wise, so
-    # applying it to the single new position is exact.
-    def prefill(self, *args, **kwargs):
-        out, cache = self.block.prefill(*args, **kwargs)
-        return (self.adapter(out) if self.enabled else out), cache
-
-    def step(self, *args, **kwargs):
-        out = self.block.step(*args, **kwargs)
-        if not self.enabled:
-            return out
-        if isinstance(out, tuple):
-            return (self.adapter(out[0]),) + out[1:]
-        return self.adapter(out)
+"""RelayPatchwork adapter + the hybrid-safe block wrapper.
+
+The certified geometry (16 slots x D=4 through a 64-atom aleph address,
+squared-ReLU patch head, sigmoid gate initialized at -3): ~261k params
+at d=1024. Output head zero-initialized so a fresh anchor is inert
+(modulo the documented LayerNorm-bias leak).
+
+BlockWithAdapter carries an `enabled` switch: when False, the forward
+returns the block output untouched — the single-anchor half of the
+toggle law. It also carries a `recompute` switch (0.2.7): when True,
+the adapter's intermediates are not kept for the backward pass but
+recomputed from the block output when the backward reaches them
+(torch.utils.checkpoint, non-reentrant). Same math, one extra adapter
+forward per backward, a fraction of the activation memory: at d=1024
+on 4096-token rows the kept intermediates run ~96 MB per block per
+adapter per two-row micro-batch, which is what walls a 32-block trunk
+with several stacked arms on a 32 GB card. A stack of wrappers on one
+block recomputes as one chain (0.2.8), keeping one residual copy per
+block for the whole stack.
+"""
+from __future__ import annotations
+
+import contextlib
+from dataclasses import dataclass
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.checkpoint import checkpoint
+
+from .address import AlephAddress
+
+
+@dataclass
+class AdapterSpec:
+    n_slots: int = 16
+    K: int = 64
+    D: int = 4
+    tau: float = 0.1
+    hidden: int = 178
+    gate_init: float = -3.0
+    zero_init_head: bool = True
+
+
+class SquaredReLU(nn.Module):
+    def forward(self, x):
+        return F.relu(x) ** 2
+
+
+class RelayPatchwork(nn.Module):
+    def __init__(self, d: int, spec: AdapterSpec | None = None):
+        super().__init__()
+        s = spec or AdapterSpec()
+        self.spec = s
+        self.n_slots = s.n_slots
+        self.proj = nn.Linear(d, s.n_slots * s.D, bias=False)
+        nn.init.orthogonal_(self.proj.weight)
+        self.addr = AlephAddress(s.K, s.D, s.tau)
+        self.consume = nn.Sequential(
+            nn.Linear(s.n_slots * s.D, s.hidden), SquaredReLU(),
+            nn.LayerNorm(s.hidden), nn.Linear(s.hidden, d))
+        if s.zero_init_head:
+            nn.init.zeros_(self.consume[-1].weight)
+        self.gate = nn.Parameter(torch.tensor(float(s.gate_init)))
+
+    def forward(self, x):
+        B, n, _ = x.shape
+        slots = self.proj(x).view(B, n, self.n_slots, self.spec.D)
+        feats = self.addr.m_hat(slots).reshape(B, n, -1)
+        return x + torch.sigmoid(self.gate) * self.consume(feats)
+
+
+def _apply_chain(h, *adapters):
+    for a in adapters:
+        h = a(h)
+    return h
+
+
+def _autocast_uncached(device_type: str):
+    """The active autocast region re-entered with its weight-cast cache OFF
+    (0.2.10), for the checkpointed chain. Autocast caches the low-precision
+    copies of parameters it casts; when an earlier pass inside the same
+    autocast region (a no_grad pass with a member masked, as the abstention
+    term runs) already cast a member's weights, the recorded forward of the
+    checkpointed chain holds no cast ops for them, while the recompute,
+    which re-enters autocast fresh, records the casts, and the checkpoint's
+    operator-list check fails ('different metadata'; torch 2.8). With the
+    cache off inside the chain both passes record the same ops. The cost is
+    one bf16 cast of each live adapter's weights per block call."""
+    try:
+        enabled = torch.is_autocast_enabled(device_type)
+    except TypeError:  # older signature: cuda only
+        enabled = device_type == "cuda" and torch.is_autocast_enabled()
+    if not enabled:
+        return contextlib.nullcontext()
+    try:
+        dtype = torch.get_autocast_dtype(device_type)
+    except (TypeError, AttributeError):
+        dtype = torch.get_autocast_gpu_dtype() if device_type == "cuda" else torch.bfloat16
+    return torch.autocast(device_type=device_type, dtype=dtype, cache_enabled=False)
+
+
+# The compiled chain (0.2.9). The aleph address's compiled backward is
+# non-finite under bf16 autocast unless inductor rounds its fused
+# intermediates to bf16 the way eager does (measured 2026-09: of the
+# adapter's stages compiled alone, only addr.m_hat fails; pure fp32 and
+# emulated casts are clean), so the switch below is set when the chain
+# is first compiled. It is process-global; set it False before the first
+# compile to opt out (not recommended for bf16 training).
+COMPILE_EMULATE_CASTS = True
+COMPILE_MODE = "default"        # reduce-overhead (CUDA graphs) fails inside a checkpointed chain
+# The recompile budget and the shape gate (0.2.11). The chain is compiled
+# static (dynamic=False), so every (mask pattern x input shape) pair is one
+# cache entry of `_apply_chain`, and when torch's per-code recompile limit
+# (8 by default) is exceeded dynamo does not merely run that call eagerly:
+# it marks the code object SKIP and the chain runs eager for the rest of the
+# process, with a single warning. An evaluation that feeds the armed model
+# short inputs of many lengths (probe items, chat documents) spends that
+# budget in seconds. Two guards: the dynamo limits are raised to the floors
+# below when the chain is first compiled, and only the first
+# COMPILE_MAX_SHAPES distinct (batch, length) shapes are routed through the
+# compiled chain; any other shape runs the eager chain (exact; the compile's
+# gain is in the training shape, which is the first shape a training run
+# presents). The budget then holds the mask patterns of the training shapes
+# only (m + 1 per shape at m arms).
+COMPILE_RECOMPILE_LIMIT = 64
+COMPILE_ACCUMULATED_LIMIT = 4096
+COMPILE_MAX_SHAPES = 2
+_COMPILED_CHAIN = None
+_COMPILED_SHAPES: set = set()
+
+
+def _raise_dynamo_limits():
+    import torch._dynamo.config as dc
+    for names, floor in ((("recompile_limit", "cache_size_limit"),
+                          COMPILE_RECOMPILE_LIMIT),
+                         (("accumulated_recompile_limit",
+                           "accumulated_cache_size_limit"),
+                          COMPILE_ACCUMULATED_LIMIT)):
+        for name in names:            # 2.8 carries both spellings; older torch one
+            if hasattr(dc, name):
+                setattr(dc, name, max(int(getattr(dc, name)), int(floor)))
+
+
+def compiled_chain():
+    global _COMPILED_CHAIN
+    if _COMPILED_CHAIN is None:
+        if COMPILE_EMULATE_CASTS:
+            import torch._inductor.config as ic
+            ic.emulate_precision_casts = True
+        _raise_dynamo_limits()
+        _COMPILED_CHAIN = torch.compile(_apply_chain, dynamic=False,
+                                        mode=COMPILE_MODE)
+    return _COMPILED_CHAIN
+
+
+def chain_for(h):
+    """The chain to run on an input of h's shape (0.2.11): the compiled
+    chain for the first COMPILE_MAX_SHAPES distinct (batch, length) shapes
+    seen, the eager chain for every other shape."""
+    key = (int(h.shape[0]), int(h.shape[1]))
+    if key in _COMPILED_SHAPES:
+        return compiled_chain()
+    if len(_COMPILED_SHAPES) < COMPILE_MAX_SHAPES:
+        _COMPILED_SHAPES.add(key)
+        return compiled_chain()
+    return _apply_chain
+
+
+class BlockWithAdapter(nn.Module):
+    """Wraps one decoder block; hybrid-safe (tuple or tensor output).
+
+    `recompute=True` (0.2.7) routes the adapter call through a
+    non-reentrant checkpoint whenever autograd is recording: the slot
+    projection, the address features and the consume MLP's hidden
+    activations are dropped after the forward and rebuilt from the block
+    output during the backward. The head is deterministic and
+    dropout-free, so the RNG state is not snapshotted (no device sync per
+    call) and the recomputed values are the values the forward produced.
+    Inference paths (no_grad, prefill, step) never checkpoint.
+
+    Stacked wrappers (0.2.8; an arm attached over an arm nests a wrapper
+    around a wrapper) recompute as ONE chain: the outermost wrapper with
+    recompute on runs the core block once and applies every enabled
+    adapter of the stack, innermost first, inside a single checkpoint, so
+    one residual copy per block is kept for the whole stack instead of
+    one per adapter. Each wrapper's `enabled` switch is read where it
+    sits, so masks behave exactly as in the kept path.
+
+    `compile_chain=True` (0.2.9) runs that same chain through
+    torch.compile (default mode, one graph per mask pattern, shared by
+    every block since the adapters enter as arguments), with or without
+    the recompute; see COMPILE_EMULATE_CASTS above for the precision
+    switch it needs. The compiled and the eager chain agree to bf16
+    rounding, not bit for bit: a training run that switches it on owes a
+    gradient census on its own card first.
+    """
+
+    def __init__(self, block: nn.Module, adapter: RelayPatchwork,
+                 recompute: bool = False, compile_chain: bool = False):
+        super().__init__()
+        self.block = block
+        self.adapter = adapter
+        self.enabled = True
+        self.recompute = bool(recompute)
+        self.compile_chain = bool(compile_chain)
+
+    def stack(self):
+        """(wrappers innermost first, the core block) of the stack this
+        wrapper tops."""
+        wraps, core = [], self
+        while isinstance(core, BlockWithAdapter):
+            wraps.append(core)
+            core = core.block
+        wraps.reverse()
+        return wraps, core
+
+    def forward(self, *args, **kwargs):
+        if self.compile_chain or (self.recompute and torch.is_grad_enabled()):
+            wraps, core = self.stack()
+            out = core(*args, **kwargs)
+            live = [w.adapter for w in wraps if w.enabled]
+            if not live:
+                return out
+            h = out[0] if isinstance(out, tuple) else out
+            fn = chain_for(h) if self.compile_chain else _apply_chain
+            if self.recompute and torch.is_grad_enabled():
+                with _autocast_uncached(h.device.type):
+                    y = checkpoint(fn, h, *live, use_reentrant=False,
+                                   preserve_rng_state=False)
+            else:
+                y = fn(h, *live)
+            return (y,) + out[1:] if isinstance(out, tuple) else y
+        out = self.block(*args, **kwargs)
+        if not self.enabled:
+            return out
+        if isinstance(out, tuple):
+            return (self.adapter(out[0]),) + out[1:]
+        return self.adapter(out)
+
+    # Incremental-decode passthroughs (trunks with a cached decode path,
+    # e.g. AlephLM prefill/step). The patch head is position-wise, so
+    # applying it to the single new position is exact.
+    def prefill(self, *args, **kwargs):
+        out, cache = self.block.prefill(*args, **kwargs)
+        return (self.adapter(out) if self.enabled else out), cache
+
+    def step(self, *args, **kwargs):
+        out = self.block.step(*args, **kwargs)
+        if not self.enabled:
+            return out
+        if isinstance(out, tuple):
+            return (self.adapter(out[0]),) + out[1:]
+        return self.adapter(out)

mini-beatrix-3/code/pod/compile_guard.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# THE COMPILED-CHAIN GUARD (the standing rule: the run never continues on the eager chain). Polls the mission log every 2 min for
+# the dynamo recompile-limit warning (the signature of the chain falling back to eager) written after this guard's start and for a
+# trainer that runs slower than the eager line; on either, it writes an alert file, logs it, and arms go_relaunch_0211.sh for the
+# next 2,000-step checkpoint (stop, reinstall 0.2.11, relaunch compiled) unless a go sequence is already running. Bounded by hours.
+# Usage: setsid nohup bash /root/v3/tools/compile_guard.sh [hours=48] > /root/v3/logs/compile_guard.out 2>&1 < /dev/null &
+HOURS="${1:-48}"
+LOG=/root/v3/logs/mission.log
+OUT=/root/v3/logs/compile_guard.log
+ALERT=/root/v3/logs/COMPILED_CHAIN_LOST
+EAGER_LINE=9.8      # s/step: the compiled three-arm pace is ~8.7, the eager 10.4-10.8; later stages raise both, so the line is re-read from the log's own history
+log() { echo "[guard $(date -u '+%H:%M:%SZ')] $*" >> "$OUT"; }
+START_BYTES=$(stat -c %s "$LOG")
+log "armed for $HOURS h from byte $START_BYTES of the mission log"
+END=$(( $(date +%s) + HOURS * 3600 ))
+while [ "$(date +%s)" -lt "$END" ]; do
+  sleep 120
+  NOW_BYTES=$(stat -c %s "$LOG")
+  if [ "$NOW_BYTES" -lt "$START_BYTES" ]; then START_BYTES=0; fi
+  N=$(tail -c +$((START_BYTES + 1)) "$LOG" | grep -ac "recompile_limit")
+  if [ "$N" -gt 0 ]; then
+    if [ -f "$ALERT" ]; then continue; fi
+    date -u +%FT%TZ > "$ALERT"
+    log "ALERT: $N recompile-limit warning(s) since the guard's start: the chain has fallen back to eager"
+    tail -c +$((START_BYTES + 1)) "$LOG" | grep -a "recompile_limit" | head -n 3 | cut -c1-200 >> "$OUT"
+    if [ "$(pgrep -fc 'go_relaunch_021[1]')" != "0" ]; then log "a go sequence is already running; not arming another"; continue; fi
+    STEP=$(tail -c 1500 "$LOG" | tr "\r" "\n" | grep -aoE "[0-9]+/245670" | tail -n 1 | cut -d/ -f1)
+    NEXT=$(( (STEP / 2000 + 1) * 2000 ))
+    NEXTC=$(printf "%'d" "$NEXT")
+    log "arming the stop + reinstall + relaunch at the next checkpoint $NEXTC (the standing rule: never eager)"
+    setsid nohup bash /root/v3/tools/go_relaunch_0211.sh "$NEXTC" 0 > /root/v3/logs/go_relaunch_0211.out 2>&1 < /dev/null &
+    disown
+  fi
+done
+log "guard window ended"

mini-beatrix-3/code/pod/go_relaunch_0211.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+# THE WHOLE SEQUENCE: stop at the named checkpoint -> install amoe-lora 0.2.11 -> relaunch (the running form).
+# Usage: setsid nohup bash /root/v3/tools/go_relaunch_0211.sh <step, e.g. 119,496> [ships expected, default 3] > /root/v3/logs/go_relaunch_0211.out 2>&1 &
+STEP="${1:?step with commas}"
+SHIPS="${2:-3}"
+OUT=/root/v3/logs/go_relaunch_0211.log
+log() { echo "[go $(date -u '+%H:%M:%SZ')] $*" | tee -a "$OUT"; }
+log "armed for the $STEP checkpoint"
+bash /root/v3/tools/stop_at_checkpoint.sh "$STEP" "$SHIPS"
+if ! grep -aq "STOPPED" "/root/v3/logs/stop_${STEP//,/}.log" 2>/dev/null; then log "ERROR: the stop did not report STOPPED; nothing installed, nothing launched"; exit 2; fi
+log "stopped; installing"
+bash /root/v3/tools/install_0211.sh
+if ! grep -aq "INSTALLED" /root/v3/logs/install_0211.log 2>/dev/null; then log "ERROR: the install did not report INSTALLED; nothing launched"; exit 3; fi
+log "installed; relaunching"
+bash /root/v3/tools/relaunch_3arms.sh 1 1 4
+log "done: $(tail -n 1 /root/v3/logs/relaunch_3arms.log)"

mini-beatrix-3/code/pod/install_0211.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# INSTALL amoe-lora 0.2.11 (the compiled chain's recompile budget + shape gate; commit 9bae609 on experimental) into /root/venv.
+# Refuses while a trainer process is alive. alephllm stays at 0.10.5 (e8b7440). Reads the versions back.
+OUT=/root/v3/logs/install_0211.log
+log() { echo "[install $(date -u '+%H:%M:%SZ')] $*" | tee -a "$OUT"; }
+LEFT=$(ps -eo cmd | grep -E "torch.distributed.run|geolip.alephllm.train.mission" | grep -v grep | wc -l)
+if [ "$LEFT" != "0" ]; then log "REFUSED: $LEFT trainer processes alive"; exit 3; fi
+SHA=9bae60922c83f5af242c5516fa369b9232afd2f3
+log "installing amoe-lora 0.2.11 ($SHA)"
+/root/venv/bin/pip install --no-deps --force-reinstall -q "git+https://github.com/AbstractEyes/amoe-lora@$SHA" >> "$OUT" 2>&1 && log "amoe-lora installed" || { log "amoe-lora install FAILED"; exit 4; }
+echo "$SHA" > /root/v3/amoe_ref.txt
+V=$(/root/venv/bin/python -c 'import amoe, geolip.alephllm as a; print(amoe.__version__, a.__version__)' 2>&1 | tail -n 1)
+log "versions read back: amoe-lora / alephllm = $V"
+case "$V" in "0.2.11 0.10.5") log "INSTALLED";; *) log "ERROR: expected 0.2.11 0.10.5"; exit 5;; esac
+/root/venv/bin/python -c 'from amoe.core import adapter as A; print("gate constants:", A.COMPILE_RECOMPILE_LIMIT, A.COMPILE_ACCUMULATED_LIMIT, A.COMPILE_MAX_SHAPES)' | tee -a "$OUT"

mini-beatrix-3/code/pod/mission.json

@@ -82,5 +82,5 @@
   "adapter_recompute": true,
   "adapter_compile": true,
   "abstain_chunks": 4,
-  "_note": "the mission file as it runs on the pod at the time of this snapshot (2026-09-27 UTC): resumed from the 106,000 checkpoint with adapter_recompute, adapter_compile and abstain_chunks=4 set; the trainer re-reads the refreshable keys at every phase boundary"
+  "_note": "the mission file as it runs on the pod at the time of this snapshot (2026-09-28 UTC): resumed from the 122,000 checkpoint on amoe-lora 0.2.11 with adapter_recompute, adapter_compile and abstain_chunks=4 set; the trainer re-reads the refreshable keys at every phase boundary"
 }

mini-beatrix-3/code/pod/relaunch_3arms.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# THE RELAUNCH ON amoe-lora 0.2.11 (the 106,000 relaunch form generalized). Usage: bash relaunch_3arms.sh [recompute:0|1] [compile:0|1] [abstain_chunks:int|none]
+# Defaults = the running form (recompute on, the compiled chain on, 4 abstention chunks). Keeps arm_spec_certified true, refuses if a
+# trainer process is alive or a card holds memory or the installed libraries are not 0.2.11 / 0.10.5, launches launch_mission.sh (a
+# resume-first launch: the resume state of the stop), restarts the arm sidecar, and prints the proving lines: versions, the attach
+# lines with 'recompute on, compile on', the 'N abstention chunks per step' line, the first steps, the cards.
+RECOMPUTE="${1:-1}"
+COMPILE="${2:-1}"
+ABST="${3:-4}"
+OUT=/root/v3/logs/relaunch_3arms.log
+log() { echo "[relaunch $(date -u '+%H:%M:%SZ')] $*" | tee -a "$OUT"; }
+LEFT=$(ps -eo cmd | grep -E "torch.distributed.run|geolip.alephllm.train.mission" | grep -v grep | wc -l)
+if [ "$LEFT" != "0" ]; then log "REFUSED: $LEFT trainer processes alive"; exit 3; fi
+BUSY=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | awk '$1 > 2000' | wc -l)
+if [ "$BUSY" != "0" ]; then log "REFUSED: a card still holds memory"; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader | tee -a "$OUT"; exit 4; fi
+V=$(/root/venv/bin/python -c 'import amoe, geolip.alephllm as a; print(amoe.__version__, a.__version__)' 2>&1 | tail -n 1)
+log "installed: amoe-lora / alephllm = $V"
+case "$V" in "0.2.11 0.10.5") ;; *) log "REFUSED: expected 0.2.11 0.10.5 (the compiled chain's recompile budget + shape gate)"; exit 5;; esac
+RESUME=$(/root/venv/bin/python -c 'import torch; s = torch.load("/root/v3/runs/mission/mini-beatrix-3/resume/latest.pt", map_location="cpu", weights_only=False); print(s.get("step", "?"))' 2>/dev/null | tail -n 1)
+log "resume state step: $RESUME"
+cp /root/v3/mission.json "/root/v3/mission.json.bak_relaunch_$(date -u +%m%d_%H%M)"
+/root/venv/bin/python - "$RECOMPUTE" "$COMPILE" "$ABST" "$RESUME" <<'PYEOF'
+import json, os, sys
+p = "/root/v3/mission.json"
+C = json.load(open(p))
+C["adapter_recompute"] = bool(int(sys.argv[1]))
+C["adapter_compile"] = bool(int(sys.argv[2]))
+C["abstain_chunks"] = None if sys.argv[3].lower() in ("none", "null", "") else int(sys.argv[3])
+C["arm_spec_certified"] = True
+C["_halt_note"] = ("relaunched from the %s resume state on amoe-lora 0.2.11 after the compiled chain fell back to eager at the stage 2 close "
+                   "(dynamo's recompile limit hit by the boundary evaluation's variable-length inputs); the form unchanged: recompute + the "
+                   "compiled chain + 4 abstention chunks; arm_spec_certified true" % sys.argv[4])
+tmp = p + ".tmp"
+json.dump(C, open(tmp, "w"), indent=2)
+os.replace(tmp, p)
+print("mission.json:", {k: C[k] for k in ("adapter_recompute", "adapter_compile", "abstain_chunks", "arm_spec_certified", "micro_batch", "grad_accum")})
+PYEOF
+log "launching"
+cd /root/v3 || exit 9
+bash /root/v3/launch_mission.sh | tee -a "$OUT"
+for i in $(seq 1 90); do
+  if tr "\r" "\n" < /root/v3/logs/mission.log | tail -n 400 | grep -aq "\[train\] stage arms"; then break; fi
+  sleep 10
+done
+tr "\r" "\n" < /root/v3/logs/mission.log | tail -n 400 | grep -aE "\[mission\] alephllm|\[arms\] attached|\[arms\] restored|\[resume\]|\[train\] phase|\[train\] stage arms|Traceback|Error" | tail -n 14 | cut -c1-200 | tee -a "$OUT"
+export $(tr "\0" "\n" < /proc/$(pgrep -f "geolip.alephllm.train.mission" | head -n 1)/environ | grep "^HF_TOKEN=")
+nohup /root/venv/bin/python tools/arm_anchor_sidecar.py --poll 120 >> logs/arm_sidecar.out 2>&1 < /dev/null &
+disown
+sleep 2
+log "sidecar pid $(pgrep -f 'python.*arm_anchor_sidecar.py' | tail -n 1) restarted"
+sleep 240
+log "steps after 4 min: $(tail -c 1500 /root/v3/logs/mission.log | tr "\r" "\n" | grep -aE "s/step" | tail -n 1 | cut -c1-100)"
+log "recompile-limit warnings in the log since the launch: $(tail -c 300000 /root/v3/logs/mission.log | grep -ac 'recompile_limit')"
+log "cards: $(nvidia-smi --query-gpu=index,memory.used --format=csv,noheader | tr '\n' ';')"
+log "RELAUNCHED"

mini-beatrix-3/code/pod/stop_at_checkpoint.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# THE MANUAL STOP AT A CHECKPOINT (the 106,000 recipe of 2026-09-26, generalized). Usage: bash stop_at_checkpoint.sh <step, e.g. 119,496> [ships expected, default 3]
+# Waits for the trainer's checkpoint line for that step (a full resume state, saved AND uploaded), then for the sidecar's arm ships
+# from that state (one per attached arm; cap 6 min), then kills the torchrun tree while the loop is back in plain compute (no writes
+# in flight): the resume lands exactly at that step and only the seconds after it are lost. The two-rank KeyboardInterrupt path is
+# NOT used: torchrun treats a worker's exit as a failure and SIGTERMs the other mid-save.
+STEP="${1:?step with commas, e.g. 119,496}"
+SHIPS="${2:-3}"
+LOG=/root/v3/logs/mission.log
+SLOG=/root/v3/logs/arm_sidecar.log
+OUT="/root/v3/logs/stop_${STEP//,/}.log"
+log() { echo "[stop $(date -u '+%H:%M:%SZ')] $*" >> "$OUT"; }
+log "armed: waiting for the [ckpt] step $STEP line ($SHIPS sidecar ships expected)"
+for i in $(seq 1 2160); do
+  if tr "\r" "\n" < "$LOG" | grep -aq "\[ckpt\] step $STEP:.*saved & uploaded"; then break; fi
+  sleep 10
+done
+if ! tr "\r" "\n" < "$LOG" | grep -aq "\[ckpt\] step $STEP:.*saved & uploaded"; then log "ERROR: no checkpoint line within 6 h; nothing done"; exit 2; fi
+log "checkpoint line seen: $(tr "\r" "\n" < "$LOG" | grep -a "\[ckpt\] step $STEP:" | tail -n 1 | cut -c1-140)"
+for i in $(seq 1 36); do
+  n=$(grep -ac "step $STEP: shipped" "$SLOG")
+  if [ "$n" -ge "$SHIPS" ]; then break; fi
+  sleep 10
+done
+log "sidecar ships for $STEP: $(grep -ac "step $STEP: shipped" "$SLOG") ($SHIPS expected)"
+sleep 20
+LAST=$(tail -c 1500 "$LOG" | tr "\r" "\n" | grep -aE "s/step" | tail -n 1 | cut -c1-100)
+log "last bar line before the kill: $LAST"
+PIDS=$(ps -eo pid,cmd | grep -E "torch.distributed.run|geolip.alephllm.train.mission" | grep -v grep | awk '{print $1}' | tr "\n" " ")
+log "killing the torchrun tree: $PIDS"
+kill -9 $PIDS 2>/dev/null
+sleep 5
+LEFT=$(ps -eo pid,cmd | grep -E "torch.distributed.run|geolip.alephllm.train.mission" | grep -v grep | wc -l)
+log "processes left: $LEFT; cards: $(nvidia-smi --query-gpu=index,memory.used --format=csv,noheader | tr "\n" ";")"
+for SPID in $(pgrep -f "python.*arm_anchor_sidecar.py"); do kill "$SPID" 2>/dev/null; log "sidecar $SPID stopped (the relaunch restarts it)"; done
+log "resume state: $(ls -la --time-style=+%H:%M:%S /root/v3/runs/mission/mini-beatrix-3/resume/latest.pt | awk '{print $5, $6}')"
+log "STOPPED at the $STEP point"

mini-beatrix-3/code/requirements.txt

@@ -1,4 +1,4 @@
-# the pod's environment for the mini-beatrix-3 mission (2026-09-27); CUDA 12.8 wheels
+# the pod's environment for the mini-beatrix-3 mission (2026-09-28); CUDA 12.8 wheels
 torch==2.8.0
 torchvision==0.23.0
 torchaudio==2.8.0
@@ -8,6 +8,6 @@ huggingface_hub==1.32.0
 datasets==5.0.1
 numpy==2.1.2
 pyarrow==25.0.1
-# the two libraries, pinned to the commits that trained this run (the snapshots in this folder are these commits):
-# pip install --no-deps 'git+https://github.com/AbstractEyes/amoe-lora@f3db5d4'    # amoe 0.2.10
+# the two libraries, pinned to the commits that train this run (the snapshots in this folder are these commits):
+# pip install --no-deps 'git+https://github.com/AbstractEyes/amoe-lora@9bae609'    # amoe 0.2.11 (0.2.10 @f3db5d4 trained steps 106,000-122,000)
 # pip install --no-deps 'git+https://github.com/AbstractEyes/alephllm@e8b7440'     # geolip.alephllm 0.10.5