Adopt-0 — Technical Companion
Repository: AbstractPhil/alephlm-adopt-0 · Author: AbstractPhil
Companion to the week-in-review article. Runs dated August 4–5, 2026.
1. Overview
Adopt-0 is the first attachment of the aleph mechanism to a real pretrained LLM. Rather than training an aleph-routed trunk from scratch, the mechanism is adopted onto a completely frozen Qwen2.5-0.5B-Instruct as a set of small relay adapters, and asked to earn its place through distillation from the model's 3x-larger sibling, Qwen2.5-1.5B-Instruct.
The trainable budget is 2.87M parameters against a frozen half-billion-parameter trunk. Three arms ran to verdict — 20k steps and ~31M tokens each, ~2.5 GPU-hours per arm — and every artifact (all checkpoints, training logs, and evaluation results) ships publicly in this repository.
The headline results, in order of what they turned out to be worth:
- A capacity tie on the raw distillation gauges between the aleph and a plain-MLP control at matched size — both cut KL-to-teacher by ~25%, landing within ~1% relative of each other.
- Underneath the tie, a placement election: the two mechanisms allocated the same budget in opposite directions, and the aleph concentrated exactly where the program's placement law predicts it pays.
- Toggle discipline at LLM scale: both trained arms detach to recover the stock model to 4 decimal places.
- An honest task-transfer null: the KL improvement did not convert to benchmark accuracy — for either mechanism, identically — indicting the distillation corpus and clearing both mechanisms.
A companion instrument from the same working period, the cross-tokenizer alignment probe, is documented in §5 because it gates the next stage of multi-teacher distillation plans.
2. Design
2.1 Relay adapters
- Student: Qwen2.5-0.5B-Instruct, completely frozen. No base weight is ever updated.
- Adapters: one sigma-gated residual relay per transformer block, plus one separately-gated relay on the pre-head stream — the stream that feeds the token head directly. That last seat exists because of the program's placement law (Campaign Law 2, §6): the address mechanism is predicted to pay only where it parameterizes the output distribution, so the bed deliberately gives it a position there.
- Aleph arm: relays route through the aleph's signed codebook read, K=64 per relay.
- Control arm: plain-MLP relays at matched placement and matched capacity — the capacity control. Any difference between arms is mechanism, not parameter count.
- Teacher: Qwen2.5-1.5B-Instruct. Objective: per-token KL divergence to the teacher at temperature 2, plus a 0.1-weighted cross-entropy term; pure Adam; streaming wikitext corpus.
2.2 Silent at initialization
At initialization the adapters perturb the stock model by a maximum logit delta of 1.86 (residue of the relay construction, before any training), and toggling them off recovers stock behavior to 4 decimal places from step zero. In the 200-step smoke test alone, the 2.87M adapter parameters cut KL-to-teacher by 19%. Even in that smoke run, adapter drift organized by depth — ~.03 in early blocks rising to ~.21 at block 22 — the same depth-graded drift pattern the program has recorded before on other beds.
2.3 Engineering note, kept honestly
The first smoke attempt froze the training host: an fp32 student with an uncapped allocator, materializing 152k-vocabulary logits at batch 8x512, spilled silently into shared system memory. The fixes — a hard allocator cap so over-commit fails loudly, bf16 trunk with fp32 adapters cast at the boundary, batch 4x384, KL computed in 64-token chunks — brought peak memory to ~8GB and became permanent bed features.
3. Results
3.1 The three-way verdict: a tie on the gauges
| arm | eval gauge | KL-to-teacher |
|---|---|---|
| frozen (stock) | 17.351 | 1.928 |
| + aleph relays | 15.449 | 1.448 |
| + mlp relays | 15.394 | 1.432 |
Both adapter mechanisms cut KL to the teacher by ~25%, within ~1% relative of each other. This tie class has a prior sighting in the program: the L-AR8 co-training bed (vision dial), and before that the seed-0 E1 trunk campaign's three-way tie, where learned, frozen-random, and dense arms scored .6031 / .6033 / .6026 — a spread of .0007. The consistent reading across all three sightings: on the raw loss gauge alone, mechanism does not separate from capacity. The verdict lives one level deeper, in how each mechanism spent its budget.
3.2 The placement election
The two arms reached their near-identical scores by opposite allocations:
- The aleph arm opened its pre-head relay ~8x — the gate sigma grew from .047 to .387, and its codebook drifted .471 radians. Given a seat at every depth, the address mechanism voluntarily concentrated on the output-distribution position — the placement Campaign Law 2 predicts for it.
- The mlp control closed the very same position ~10x (sigma .047 to .004) and did its work from the early layers instead.
Nothing imposed this. Two mechanisms, identical seats, identical objective — opposite votes on where the pre-head position is worth spending. The placement law had previously been refined from ablation results (attached as a passive add-on, the address is inert; used as the operator, it is load-bearing); adopt-0 is the first time the law's arrow was drawn by the trained mechanism's own allocation rather than by an ablation grid.
The tie also names the follow-up precisely: adoption's sigma-gated residual write is too weak a coupling to convert the elected position into a win — the relay writes weakly into the residual stream where the law wants the address parameterizing the logits directly. The next rung (T0) gives the address a direct read into the output distribution.
A determinism note: the mlp arm crashed mid-run at 14k steps, and its rerun replayed the first attempt exactly — the crash was environmental (resolved by the 14GB allocator cap), and the training itself is reproducible to replay.
3.3 Toggle discipline at LLM scale
Both trained arms toggle off to recover the stock model to 4 decimal places — the toggle law, previously certified bit-exact on a 0.8B model, now confirmed at 0.5B with distillation-trained relays; the cleanest instance in the record so far. The trained capability is a detachable module, never a modification of the base model: the pretrained weights are untouched underneath, by construction and by measurement, at every checkpoint. A mechanism that can be removed to 4 decimals can be priced exactly — every capability claim carries a free control arm built in.
3.4 Secondary instrument reads
- Positional integrity: a wide-spectrum positional probe (15 depths to 8,000 tokens) placed the adopt-0 hybrid among ten models; integrity holds ~.98, essentially flat to depth 8,000 — the RoPE inheritance, recorded as the evidence for making RoPE the default in the next trunk generation.
- A measured cost: the token-grid directional rank of the decoder sits 10–20x below the program's encoders, and KL distillation on the frozen trunk narrowed it further (.026 to .015) — distribution-matching compresses directional diversity. Logged as a caution for any plan that would use an adopted decoder as a conditioning encoder.
- Signed structure: the program's first signed instruments read the adopt-0 relays as a balanced bipolar code — negative-weight fraction ~.50, magnitudes ~1/64, dense across the K=64 codebook, with zero net bias. This is the canonical reconstructive sign-code regime the aleph's reconstruction lineage predicts, and it was invisible to every prior density-based gauge (§6).
4. The honest task-transfer null
The shipped models were benchmarked on lm-eval, 0-shot (arc_easy, hellaswag, piqa, winogrande):
| model | 4-task mean |
|---|---|
| teacher 1.5B | .7074 |
| stock student 0.5B | .5939 |
| student + aleph | .5771 |
| student + mlp | .5730 |
Both arms traded roughly .02 of task accuracy for the -25% KL — identically, mechanism-agnostic — and closed none of the .11 stock-to-teacher gap. Per-task, the damage concentrates in hellaswag and piqa, while arc_easy and winogrande tick up.
The verdict indicts the distillation corpus and objective, and specifically clears both mechanisms: distribution-matching on encyclopedic text is a different target than task behavior, and both mechanisms optimized what they were given, equally well. The mechanism results — the placement election, the signed regime, the 4-decimal detachability — stand as the round's real findings. The named next revision is an instruction-domain distillation corpus, already banked.
This is also a transferable finding for anyone distilling small models: a 25% KL improvement toward a 3x teacher can coexist with a task-accuracy loss when the corpus does not carry the task distribution. Benchmark before believing the loss curve.
5. The tokenizer-alignment instrument
The next distillation plans want multiple teachers — which raises a gating question: can token-level representations be aligned across models that do not share a tokenizer?
Method: resample each source model's token states onto the reference tokenizer's boundaries by character-span overlap, then fit and score a linear alignment map under the program's certified out-of-sample protocol: 800 fit captions / 400 held-out, ~31k fit tokens, all models at 768-d. Nulls are paired shuffled-correspondence nulls run through the same fitted map.
| pair | held-out token cosine | token R@1 |
|---|---|---|
| distilbert → bert (same tokenizer — the ceiling) | .648 | .966 |
| t5 → bert (SentencePiece → WordPiece) | .333 | .840 |
| clip → bert (BPE, causal, 77-token cap) | .183 | .445 |
| shuffled-correspondence nulls | — | ~.0005 (dead) |
The headline is the t5 row: across a fully different tokenization scheme, the correct token is retrieved first-try 84% of the time from a ~15,000-token gallery where chance is ~.0005. Roughly half the alignment strength survives the vocabulary mismatch — a tax, never a wall. The room between .840 and the .966 same-tokenizer ceiling is the price of the bridge; the room between .840 and .0005 is why the bridge is worth building. A three-member span-aligned token consensus built on these maps retains 93% of the reference member's effective rank (72.7 of 77.9), with bert/t5 cross-teacher agreement ~.76.
Recorded caveats: the probe's pooled-row readings are underpowered at this sample size (the corpus-scale pooled number remains the anchor there), and the result covers encoders. The weak clip row confounds causality with tokenization — clip is the only causal model in the table. Since LLM teachers are all causal, the decisive next probe is already specified: the same protocol on two causal LMs with different vocabularies (gpt2 vs Qwen2.5-0.5B). A t5-like result de-risks cross-tokenizer LLM consensus; a clip-like result names causality as the real barrier.
6. Laws established
- Placement law (Campaign Law 2), confirmed at LLM scale by election. The composed address pays only where it directly parameterizes the predictive distribution. Adopt-0 adds the strongest form of evidence to date: the trained mechanism allocated itself to the predicted placement (pre-head gate opened ~8x, sigma .047 → .387) while the capacity control abandoned the same seat (closed ~10x, sigma .047 → .004). Drawn consequence: adoption's residual-write coupling is too weak to cash the elected position; the next rung gives the address a direct read into the logits.
- Toggle law, extended. Trained adapters detach to stock behavior — 4-decimal recovery at 0.5B with distillation-trained relays, atop the earlier bit-exact certification at 0.8B. Every shipped capability carries its own removal control.
- Signed-instrument requirement (new, law-shaped). Density gauges — absolute values, norm ratios, squared spectra — structurally erase signed structure; certifying a signed mechanism requires signed instruments. Adopt-0 contributes one of the two confirming regimes: its K=64 relays run a balanced bipolar code (negative fraction ~.50, magnitudes ~1/64) that no prior density vital could see.
- Distillation-conversion finding (new, honest negative). KL improvement against a teacher on an out-of-domain corpus does not convert to task accuracy — both mechanisms paid ~.02 identically for -25% KL. Indicts corpus and objective, clears mechanism; fix named (instruction-domain corpus).
- Bridgeability finding (new). Tokenizer mismatch between encoders costs roughly half the token-alignment strength, leaving retrieval far above chance (.840 vs ~.0005) — cross-vocabulary consensus is viable; causality is the remaining open factor, with the decisive probe already specified.
- Method stance. Contribution, not competition: with adopt-0's placement election under a scoreboard tie as the flagship example, reports are contribution ledgers — removal deltas, signed regimes, placement, decomposition — with preregistered bars kept as decision gates rather than rankings.
7. Provenance
All numbers in this document are verified against the research record. Three arms (frozen / aleph / mlp) ran to verdict at 20k steps and ~31M tokens each on a streaming wikitext corpus, August 4–5, 2026; all weights, training logs, and evaluation outputs for every arm ship in this repository (AbstractPhil/alephlm-adopt-0). The tokenizer-alignment instrument (§5) was run in the same working period under the program's certified out-of-sample protocol. Author: AbstractPhil.
References and attribution
Models
- Qwen2.5-0.5B-Instruct (the frozen student trunk) and Qwen2.5-1.5B-Instruct (the distillation teacher) — the Qwen team — Qwen2.5-0.5B-Instruct
- BERT (Devlin et al.) — reference member and WordPiece side of the tokenizer-alignment instrument — https://arxiv.org/abs/1810.04805
- DistilBERT (Sanh et al.) — the same-tokenizer ceiling pair — https://arxiv.org/abs/1910.01108
- T5 (Raffel et al.) — the cross-tokenizer pair — https://arxiv.org/abs/1910.10683
- CLIP (Radford et al.) — the causal, 77-token-capped pair — https://arxiv.org/abs/2103.00020
Data
- WikiText-2 (Merity et al.) — the streaming distillation corpus, ~31M tokens per arm — https://arxiv.org/abs/1609.07843
Methods
- Knowledge distillation (Hinton et al.) — the per-token KL-to-teacher objective at temperature 2 — https://arxiv.org/abs/1503.02531
- Attention / the transformer (Vaswani et al.) — the trunk architecture the relays attach to — https://arxiv.org/abs/1706.03762
- LoRA (Hu et al.) — lineage of the zero-init, detachable adapter convention — https://arxiv.org/abs/2106.09685
- Mixture-of-experts routing (Shazeer et al.) — lineage and contrast class of the gated relay design — https://arxiv.org/abs/1701.06538
- Rotary position embeddings (Su et al.) — the positional scheme whose depth integrity §3.4 measures — https://arxiv.org/abs/2104.09864
- SentencePiece (Kudo & Richardson) — T5's tokenizer in the alignment instrument — https://arxiv.org/abs/1808.06226
Infrastructure
- lm-evaluation-harness (EleutherAI) — the 0-shot benchmark verdict (arc_easy, hellaswag, piqa, winogrande) — https://github.com/EleutherAI/lm-evaluation-harness
- PyTorch — every trainer and instrument in the bed — https://pytorch.org
Related artifacts: alephlm-adopt-0 (this release) · alephlm-0 (the E1 trunk campaign whose three-way tie §3.1 references) · prior installment: Geometric Memory FT4