# Adopt-0 — Technical Companion **Repository:** [`AbstractPhil/alephlm-adopt-0`](https://huggingface.co/AbstractPhil/alephlm-adopt-0) · **Author:** AbstractPhil **Companion to the week-in-review article. Runs dated August 4–5, 2026.** --- ## 1. Overview Adopt-0 is the first attachment of the aleph mechanism to a real pretrained LLM. Rather than training an aleph-routed trunk from scratch, the mechanism is *adopted onto* a completely frozen [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) as a set of small relay adapters, and asked to earn its place through [distillation](https://arxiv.org/abs/1503.02531) from the model's 3x-larger sibling, [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen). The trainable budget is 2.87M parameters against a frozen half-billion-parameter trunk. Three arms ran to verdict — 20k steps and ~31M tokens each, ~2.5 GPU-hours per arm — and every artifact (all checkpoints, training logs, and evaluation results) ships publicly in this repository. The headline results, in order of what they turned out to be worth: 1. A **capacity tie** on the raw distillation gauges between the aleph and a plain-MLP control at matched size — both cut KL-to-teacher by ~25%, landing within ~1% relative of each other. 2. Underneath the tie, a **placement election**: the two mechanisms allocated the same budget in opposite directions, and the aleph concentrated exactly where the program's placement law predicts it pays. 3. **Toggle discipline at LLM scale**: both trained arms detach to recover the stock model to 4 decimal places. 4. An **honest task-transfer null**: the KL improvement did not convert to benchmark accuracy — for either mechanism, identically — indicting the distillation corpus and clearing both mechanisms. A companion instrument from the same working period, the cross-tokenizer alignment probe, is documented in §5 because it gates the next stage of multi-teacher distillation plans. --- ## 2. Design ### 2.1 Relay adapters - **Student:** Qwen2.5-0.5B-Instruct, completely frozen. No base weight is ever updated. - **Adapters:** one sigma-gated residual relay per [transformer](https://arxiv.org/abs/1706.03762) block, plus one separately-gated relay on the pre-head stream — the stream that feeds the token head directly. That last seat exists because of the program's placement law (Campaign Law 2, §6): the address mechanism is predicted to pay only where it parameterizes the output distribution, so the bed deliberately gives it a position there. - **Aleph arm:** relays route through the aleph's signed codebook read, K=64 per relay. - **Control arm:** plain-MLP relays at **matched placement and matched capacity** — the capacity control. Any difference between arms is mechanism, not parameter count. - **Teacher:** Qwen2.5-1.5B-Instruct. **Objective:** per-token KL divergence to the teacher at temperature 2, plus a 0.1-weighted cross-entropy term; pure Adam; streaming [wikitext](https://arxiv.org/abs/1609.07843) corpus. ### 2.2 Silent at initialization At initialization the adapters perturb the stock model by a maximum logit delta of 1.86 (residue of the relay construction, before any training), and toggling them off recovers stock behavior to 4 decimal places from step zero. In the 200-step smoke test alone, the 2.87M adapter parameters cut KL-to-teacher by 19%. Even in that smoke run, adapter drift organized by depth — ~.03 in early blocks rising to ~.21 at block 22 — the same depth-graded drift pattern the program has recorded before on other beds. ### 2.3 Engineering note, kept honestly The first smoke attempt froze the training host: an fp32 student with an uncapped allocator, materializing 152k-vocabulary logits at batch 8x512, spilled silently into shared system memory. The fixes — a hard allocator cap so over-commit fails loudly, bf16 trunk with fp32 adapters cast at the boundary, batch 4x384, KL computed in 64-token chunks — brought peak memory to ~8GB and became permanent bed features. --- ## 3. Results ### 3.1 The three-way verdict: a tie on the gauges | arm | eval gauge | KL-to-teacher | |---|---|---| | frozen (stock) | 17.351 | 1.928 | | + aleph relays | 15.449 | 1.448 | | + mlp relays | 15.394 | 1.432 | Both adapter mechanisms cut KL to the teacher by ~25%, within ~1% relative of each other. This tie class has a prior sighting in the program: the **L-AR8 co-training bed (vision dial)**, and before that the [seed-0 E1 trunk campaign](https://huggingface.co/AbstractPhil/alephlm-0)'s three-way tie, where learned, frozen-random, and dense arms scored .6031 / .6033 / .6026 — a spread of .0007. The consistent reading across all three sightings: **on the raw loss gauge alone, mechanism does not separate from capacity.** The verdict lives one level deeper, in how each mechanism spent its budget. ### 3.2 The placement election The two arms reached their near-identical scores by **opposite allocations**: - The **aleph** arm *opened* its pre-head relay ~8x — the gate sigma grew from .047 to .387, and its codebook drifted .471 radians. Given a seat at every depth, the address mechanism voluntarily concentrated on the output-distribution position — the placement Campaign Law 2 predicts for it. - The **mlp** control *closed* the very same position ~10x (sigma .047 to .004) and did its work from the early layers instead. Nothing imposed this. Two mechanisms, identical seats, identical objective — opposite votes on where the pre-head position is worth spending. The placement law had previously been refined from ablation results (attached as a passive add-on, the address is inert; used *as* the operator, it is load-bearing); adopt-0 is the first time the law's arrow was drawn **by the trained mechanism's own allocation** rather than by an ablation grid. The tie also names the follow-up precisely: adoption's sigma-gated residual write is too weak a coupling to convert the elected position into a win — the relay writes weakly into the residual stream where the law wants the address parameterizing the logits directly. The next rung (T0) gives the address a direct read into the output distribution. A determinism note: the mlp arm crashed mid-run at 14k steps, and its rerun **replayed the first attempt exactly** — the crash was environmental (resolved by the 14GB allocator cap), and the training itself is reproducible to replay. ### 3.3 Toggle discipline at LLM scale Both trained arms toggle off to recover the stock model to **4 decimal places** — the toggle law, previously certified bit-exact on a 0.8B model, now confirmed at 0.5B with distillation-trained relays; the cleanest instance in the record so far. The trained capability is a detachable module, never a modification of the base model: the pretrained weights are untouched underneath, by construction and by measurement, at every checkpoint. A mechanism that can be removed to 4 decimals can be *priced* exactly — every capability claim carries a free control arm built in. ### 3.4 Secondary instrument reads - **Positional integrity:** a wide-spectrum positional probe (15 depths to 8,000 tokens) placed the adopt-0 hybrid among ten models; integrity holds ~.98, essentially flat to depth 8,000 — the [RoPE](https://arxiv.org/abs/2104.09864) inheritance, recorded as the evidence for making RoPE the default in the next trunk generation. - **A measured cost:** the token-grid directional rank of the decoder sits 10–20x below the program's encoders, and KL distillation on the frozen trunk *narrowed* it further (.026 to .015) — distribution-matching compresses directional diversity. Logged as a caution for any plan that would use an adopted decoder as a conditioning encoder. - **Signed structure:** the program's first signed instruments read the adopt-0 relays as a **balanced bipolar code** — negative-weight fraction ~.50, magnitudes ~1/64, dense across the K=64 codebook, with zero net bias. This is the canonical reconstructive sign-code regime the aleph's reconstruction lineage predicts, and it was invisible to every prior density-based gauge (§6). --- ## 4. The honest task-transfer null The shipped models were benchmarked on [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness), 0-shot (arc_easy, hellaswag, piqa, winogrande): | model | 4-task mean | |---|---| | teacher 1.5B | .7074 | | stock student 0.5B | .5939 | | student + aleph | .5771 | | student + mlp | .5730 | Both arms traded roughly .02 of task accuracy for the -25% KL — **identically, mechanism-agnostic** — and closed none of the .11 stock-to-teacher gap. Per-task, the damage concentrates in hellaswag and piqa, while arc_easy and winogrande tick *up*. The verdict indicts the **distillation corpus and objective**, and specifically clears both mechanisms: distribution-matching on encyclopedic text is a different target than task behavior, and both mechanisms optimized what they were given, equally well. The mechanism results — the placement election, the signed regime, the 4-decimal detachability — stand as the round's real findings. The named next revision is an **instruction-domain distillation corpus**, already banked. This is also a transferable finding for anyone distilling small models: a 25% KL improvement toward a 3x teacher can coexist with a task-accuracy *loss* when the corpus does not carry the task distribution. Benchmark before believing the loss curve. --- ## 5. The tokenizer-alignment instrument The next distillation plans want multiple teachers — which raises a gating question: can token-level representations be aligned across models that do not share a tokenizer? **Method:** resample each source model's token states onto the reference tokenizer's boundaries by character-span overlap, then fit and score a linear alignment map under the program's certified out-of-sample protocol: 800 fit captions / 400 held-out, ~31k fit tokens, all models at 768-d. Nulls are **paired shuffled-correspondence nulls run through the same fitted map**. | pair | held-out token cosine | token R@1 | |---|---|---| | [distilbert](https://arxiv.org/abs/1910.01108) → [bert](https://arxiv.org/abs/1810.04805) (same tokenizer — the ceiling) | .648 | .966 | | [t5](https://arxiv.org/abs/1910.10683) → bert ([SentencePiece](https://arxiv.org/abs/1808.06226) → [WordPiece](https://arxiv.org/abs/1810.04805)) | .333 | **.840** | | [clip](https://arxiv.org/abs/2103.00020) → bert (BPE, causal, 77-token cap) | .183 | .445 | | shuffled-correspondence nulls | — | ~.0005 (dead) | The headline is the t5 row: across a fully different tokenization scheme, the correct token is retrieved first-try 84% of the time from a ~15,000-token gallery where chance is ~.0005. **Roughly half the alignment strength survives the vocabulary mismatch — a tax, never a wall.** The room between .840 and the .966 same-tokenizer ceiling is the price of the bridge; the room between .840 and .0005 is why the bridge is worth building. A three-member span-aligned token consensus built on these maps retains 93% of the reference member's effective rank (72.7 of 77.9), with bert/t5 cross-teacher agreement ~.76. **Recorded caveats:** the probe's pooled-row readings are underpowered at this sample size (the corpus-scale pooled number remains the anchor there), and the result covers encoders. The weak clip row confounds causality with tokenization — clip is the only causal model in the table. Since LLM teachers are all causal, the decisive next probe is already specified: the same protocol on two causal LMs with different vocabularies (gpt2 vs Qwen2.5-0.5B). A t5-like result de-risks cross-tokenizer LLM consensus; a clip-like result names causality as the real barrier. --- ## 6. Laws established - **Placement law (Campaign Law 2), confirmed at LLM scale by election.** The composed address pays only where it directly parameterizes the predictive distribution. Adopt-0 adds the strongest form of evidence to date: the trained mechanism *allocated itself* to the predicted placement (pre-head gate opened ~8x, sigma .047 → .387) while the capacity control abandoned the same seat (closed ~10x, sigma .047 → .004). Drawn consequence: adoption's residual-write coupling is too weak to cash the elected position; the next rung gives the address a direct read into the logits. - **Toggle law, extended.** Trained adapters detach to stock behavior — 4-decimal recovery at 0.5B with distillation-trained relays, atop the earlier bit-exact certification at 0.8B. Every shipped capability carries its own removal control. - **Signed-instrument requirement (new, law-shaped).** Density gauges — absolute values, norm ratios, squared spectra — structurally erase signed structure; certifying a signed mechanism requires signed instruments. Adopt-0 contributes one of the two confirming regimes: its K=64 relays run a balanced bipolar code (negative fraction ~.50, magnitudes ~1/64) that no prior density vital could see. - **Distillation-conversion finding (new, honest negative).** KL improvement against a teacher on an out-of-domain corpus does not convert to task accuracy — both mechanisms paid ~.02 identically for -25% KL. Indicts corpus and objective, clears mechanism; fix named (instruction-domain corpus). - **Bridgeability finding (new).** Tokenizer mismatch between encoders costs roughly half the token-alignment strength, leaving retrieval far above chance (.840 vs ~.0005) — cross-vocabulary consensus is viable; causality is the remaining open factor, with the decisive probe already specified. - **Method stance.** Contribution, not competition: with adopt-0's placement election under a scoreboard tie as the flagship example, reports are contribution ledgers — removal deltas, signed regimes, placement, decomposition — with preregistered bars kept as decision gates rather than rankings. --- ## 7. Provenance All numbers in this document are verified against the research record. Three arms (frozen / aleph / mlp) ran to verdict at 20k steps and ~31M tokens each on a streaming wikitext corpus, August 4–5, 2026; all weights, training logs, and evaluation outputs for every arm ship in this repository (`AbstractPhil/alephlm-adopt-0`). The tokenizer-alignment instrument (§5) was run in the same working period under the program's certified out-of-sample protocol. Author: AbstractPhil. --- ## References and attribution **Models** - Qwen2.5-0.5B-Instruct (the frozen student trunk) and Qwen2.5-1.5B-Instruct (the distillation teacher) — the [Qwen team](https://huggingface.co/Qwen) — [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) - BERT (Devlin et al.) — reference member and WordPiece side of the tokenizer-alignment instrument — https://arxiv.org/abs/1810.04805 - DistilBERT (Sanh et al.) — the same-tokenizer ceiling pair — https://arxiv.org/abs/1910.01108 - T5 (Raffel et al.) — the cross-tokenizer pair — https://arxiv.org/abs/1910.10683 - CLIP (Radford et al.) — the causal, 77-token-capped pair — https://arxiv.org/abs/2103.00020 **Data** - WikiText-2 (Merity et al.) — the streaming distillation corpus, ~31M tokens per arm — https://arxiv.org/abs/1609.07843 **Methods** - Knowledge distillation (Hinton et al.) — the per-token KL-to-teacher objective at temperature 2 — https://arxiv.org/abs/1503.02531 - Attention / the transformer (Vaswani et al.) — the trunk architecture the relays attach to — https://arxiv.org/abs/1706.03762 - LoRA (Hu et al.) — lineage of the zero-init, detachable adapter convention — https://arxiv.org/abs/2106.09685 - Mixture-of-experts routing (Shazeer et al.) — lineage and contrast class of the gated relay design — https://arxiv.org/abs/1701.06538 - Rotary position embeddings (Su et al.) — the positional scheme whose depth integrity §3.4 measures — https://arxiv.org/abs/2104.09864 - SentencePiece (Kudo & Richardson) — T5's tokenizer in the alignment instrument — https://arxiv.org/abs/1808.06226 **Infrastructure** - lm-evaluation-harness (EleutherAI) — the 0-shot benchmark verdict (arc_easy, hellaswag, piqa, winogrande) — https://github.com/EleutherAI/lm-evaluation-harness - PyTorch — every trainer and instrument in the bed — https://pytorch.org **Related artifacts:** [alephlm-adopt-0](https://huggingface.co/AbstractPhil/alephlm-adopt-0) (this release) · [alephlm-0](https://huggingface.co/AbstractPhil/alephlm-0) (the E1 trunk campaign whose three-way tie §3.1 references) · prior installment: [Geometric Memory FT4](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft4)