Sencelium/sencelium-125M

Sencelium 125M

Sencelium is an independent research project exploring a non-Transformer language model architecture: gated-delta-rule blocks with a content-blind "nucleus" channel, a memory pool that keeps spawning, merging and reorganizing itself during training and at inference, live self-monitoring signals, and an ESN residual path.

This checkpoint (126.485M params, 23 blocks, 2.5B training tokens) is the canonical 125M-scale run. Trained on FineWebEdu, evaluated on WikiText-103.

Code, architecture write-up, full benchmarks: github.com/vitaliihalak/sencelium Architecture diagrams: sencelium.com/architecture

Honest framing

This is not a perplexity win over a matched Transformer — it isn't one, at this scale. What is real: the memory pool is load-bearing but its benefit is structural rather than content-specific, and perplexity keeps falling as the context grows well past the trained length, where a plain Transformer's rises.

Results

	Sencelium 125M	Transformer 125M
params	126.485M	126.461M
val_ppl (WikiText-103, all 553 windows)	79.98	73.17
val_ppl (context 4096, same text)	64.30	551.84

The Transformer is ahead at the trained context length (gap +9.3%). Past it, Sencelium's perplexity keeps falling up to 16 times the trained length while the Transformer's rises sharply, even with its RoPE cache extended (no retraining, no length-extension method). Against the Transformer's own score at its trained length (its fairest floor), Sencelium 125M is ahead from 1,024 tokens on, by 0.5% to 10.2%. Against a Transformer evaluated with a sliding window, Sencelium does not win at any length or scale.

Key findings

Full methodology and all numbers: benchmarks/results.md · sencelium.com/results.

Memory structures itself during training, unsupervised, but most of it is idle. Starting from zero slots, it spawns new ones and merges similar ones into categories as training goes: no labels, no fixed taxonomy, and no category hub (the largest category holds about 2% of the pool). The pool is much larger than the part that is used, though: 85 to 91% of the slots have almost no usage, and most slots have a near-duplicate.

	Sencelium 30M	Sencelium 65M	Sencelium 125M (this checkpoint)
categories formed	317	500	708
spawn → merge → reassignments	5,354 → 1,659 → 8,492	5,137 → 778 → 6,850	8,452 → 1,789 → 15,814
slots holding 90% of usage	76 (2.1%)	154 (3.5%)	286 (4.3%)
slots with usage below 0.01	91%	86%	85%
slots with a near-duplicate (cosine > 0.9)	44%	71%	72%

The memory's benefit is structural, not content-specific. Emptying the pool at evaluation time (no_mem) costs perplexity at every scale, and the cost is not monotonic in size. But replacing the stored vectors with random ones of matched statistics changes perplexity by less than 0.7 points, and a single slot holding the pool's mean restores 57%, 95% and 97% of the benefit: what the loss rewards is a non-empty, in-distribution read, not what is stored. A fact written at inference is not used better than an unrelated one.

	Sencelium 30M	Sencelium 65M	Sencelium 125M (this checkpoint)
val_ppl / no_mem	270.80 / 275.01	124.36 / 148.32	79.98 / 86.11
mem_delta	4.21 (1.6%)	23.96 (19.3%)	6.13 (7.7%)

Carrying state across a document helps, for free, but not beyond a Transformer's sliding window. No retraining, no extra parameters: just not resetting recurrent state between a document's own windows at evaluation time. Most of the gain (71 to 74%) is in the first 128 tokens of a window, the removal of a cold start. A vanilla Transformer scored on the same text with a sliding window does better still (194.9, 92.8, 60.1 against Sencelium's carried 245.4, 107.0, 66.4).

	Sencelium 30M	Sencelium 65M	Sencelium 125M (this checkpoint)
carry_gain	9.4%	13.9%	17.0%

The ESN residual acts as a recency prior; the inner voice loop is not converged. The ESN branch raises the logits of tokens that already occurred in the window. Its learned scalar esn_alpha grows with scale, but it follows the integrated learning rate (alpha divided by that integral: −0.95, −0.90, −0.67), so the growth mostly reflects the length of training. The voice loop's per-round updates shrink but stay nonzero through the third round.

	Sencelium 30M	Sencelium 65M	Sencelium 125M (this checkpoint)
esn_alpha	−1.74	−3.61	−5.14
voice-loop deltas (round 1→3)	0.299 → 0.140 → 0.110	0.467 → 0.251 → 0.126	0.643 → 0.369 → 0.123

A fair ablation of the ESN branch (replacing its output with the dataset mean, or with another document's ESN output, rather than zeroing it) shows a real, content-specific contribution of roughly a third:

val_ppl with the ESN output	30M	65M	125M
real	270.66	124.30	79.94
replaced by the dataset mean	343.53	166.30	107.95
taken from another document	404.29	194.27	124.13

How to use this checkpoint

This is a non-standard architecture — not AutoModel-compatible. Use the training/eval code from the GitHub repo directly:

git clone https://github.com/vitaliihalak/sencelium
cd sencelium
pip install -r requirements.txt

python benchmarks/eval_all_windows.py --script scripts/train_sencelium.py \
    --config configs/125M.yaml --checkpoint path/to/sencelium-125M.ckpt

python benchmarks/eval.py features --script scripts/train_sencelium.py \
    --config configs/125M.yaml --checkpoint path/to/sencelium-125M.ckpt

License

Apache 2.0 (code and weights). See LICENSE.

Citation

See CITATION.cff.