---
language:
- en
library_name: transformers
pipeline_tag: fill-mask
tags:
- pretrained
- from-scratch
- tiny-llm-ablation
- custom_code
- tensorboard
- failed-ablation
- experimental-pll
datasets:
- HuggingFaceFW/fineweb-edu
model-index:
- name: diffusion-51M-base-v1-failed
  results:
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: Rowan/hellaswag
      name: HellaSwag
      config: default
      split: validation
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.2597092212706632
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.004375788991216637
        evaluation_date: '2026-10-01'
        ci95_low: 0.251226292502746
        ci95_high: 0.26837592103183483
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/ai2_arc
      name: ARC-Easy
      config: ARC-Easy
      split: test
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.2668350168350168
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.009075915859266983
        evaluation_date: '2026-10-01'
        ci95_low: 0.2494370337013486
        ci95_high: 0.28498573387701853
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/ai2_arc
      name: ARC-Challenge
      config: ARC-Challenge
      split: test
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.24061433447098976
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.012491468532390682
        evaluation_date: '2026-10-01'
        ci95_low: 0.21701467963390295
        ci95_high: 0.26590880852938104
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: baber/piqa
      name: PIQA
      config: default
      split: validation
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.500544069640914
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.011665817258899291
        evaluation_date: '2026-10-01'
        ci95_low: 0.4777084238482688
        ci95_high: 0.523377445942761
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/winogrande
      name: WinoGrande
      config: winogrande_xl
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.5098658247829518
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.014049749833367674
        evaluation_date: '2026-10-01'
        ci95_low: 0.4823514860396483
        ci95_high: 0.5373205193285109
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/openbookqa
      name: OpenBookQA
      config: main
      split: test
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.258
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.019586711785215868
        evaluation_date: '2026-10-01'
        ci95_low: 0.2215962002228131
        ci95_high: 0.29809398056057823
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: aps/super_glue
      name: BoolQ
      config: boolq
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.3782874617737003
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.008482001133931097
        evaluation_date: '2026-10-01'
        ci95_low: 0.36181754306445063
        ci95_high: 0.3950430105047085
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: EleutherAI/lambada_openai
      name: LAMBADA OpenAI
      config: default
      split: test
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.0
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.0
        evaluation_date: '2026-10-01'
        ci95_low: 0
        ci95_high: 0.0007449247473224859
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: AxiomicLabs/Arithmark-3.0
      name: ArithMark-3
      config: default
      split: train
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.299
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 1024
        standard_error: 0.014484778521220433
        evaluation_date: '2026-10-01'
        ci95_low: 0.27143763442230356
        ci95_high: 0.3281007224888896
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: pkavumba/balanced-copa
      name: Balanced COPA
      config: default
      split: train
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.5
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.015819299929208316
        evaluation_date: '2026-10-01'
        ci95_low: 0.4690696003681042
        ci95_high: 0.5309303996318958
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: nyu-mll/blimp
      name: BLiMP
      config: 67 minimal-pair subsets
      split: train
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.573865671641791
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.0015987875104430066
        evaluation_date: '2026-10-01'
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: tau/commonsense_qa
      name: CommonsenseQA
      config: default
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.19574119574119575
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.011359497363584397
        evaluation_date: '2026-10-01'
        ci95_low: 0.17445479920495016
        ci95_high: 0.21893608036620865
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: cais/mmlu
      name: MMLU continuation
      config: 57 subjects
      split: test
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.2337986041874377
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.003566341746171636
        evaluation_date: '2026-10-01'
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: allenai/sciq
      name: SciQ (with support)
      config: default
      split: test
    metrics:
    - type: pll_acc_norm
      name: pll_acc_norm (fraction; full comparison protocol)
      value: 0.237
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.013454070462577935
        evaluation_date: '2026-10-01'
        ci95_low: 0.21168136426041648
        ci95_high: 0.2643315107029855
        ci_method: Wilson 95%; item independence approximation
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: truthfulqa/truthful_qa
      name: TruthfulQA MC2
      config: multiple_choice
      split: validation
    metrics:
    - type: pll_acc
      name: pll_acc (fraction; full comparison protocol)
      value: 0.48992550756256453
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.01640276307124247
        evaluation_date: '2026-10-01'
  - task:
      type: fill-mask
      name: Experimental continuation PLL
    dataset:
      type: BananaMind/BananaMind-Base-Bench-1.1
      name: BananaMind Base 1.1
      config: default
      split: test
    metrics:
    - type: pll_raw_accuracy
      name: pll_raw_accuracy (fraction; full comparison protocol)
      value: 0.26285714285714284
      args:
        dtype: bfloat16
        num_few_shot: 0
        max_length: 2048
        standard_error: 0.023562589046968913
        evaluation_date: '2026-10-01'
        ci95_low: 0.21949467430879996
        ci95_high: 0.3113686661452677
        ci_method: Wilson 95%; item independence approximation
---

# Diffusion v1 (failed; PLL)

**Failed ablation: token-frequency/context-collapse behavior.** Retained for reproducibility and negative results. The repaired model is [diffusion v2](https://huggingface.co/d0rj/diffusion-51M-base).

A **51,392,512-parameter** English base model from the Tiny llm ablation experiment. Trained from random initialization for **15,000 optimizer steps**, processing **3,932,160,000 source tokens**. These are processed input tokens, not unique text or supervised target-token counts.

## Architecture

Failed v1 absorbing-mask diffusion ablation: 10 bidirectional layers, width 512, SwiGLU width 1536, 8 attention heads, head dimension 64, RoPE, RMSNorm and tied 32,768-token output vocabulary. A separate learned mask input vector uses ID 32768, outside the output vocabulary. Legacy additive time conditioning is retained exactly. The checkpoint stores 51,392,512 parameters.

Tokenizer foundation: [q-project/Q-50M-Base](https://huggingface.co/q-project/Q-50M-Base). This model is trained from scratch, not fine-tuned from the reference weights. [Exact trained configuration](config.json) and custom model code are included.

## Training

- Data: [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), `sample-10BT`, local Parquet shards; shuffle buffer 100,000.
- Objective: Sample noise time uniformly; independently mask tokens with that probability, predict masked originals with inverse-time weighting normalized by source token count. This run collapsed toward token-frequency predictions and is published as a failed ablation. The repaired diffusion v2 disables additive time conditioning.
- Final recorded batch: 8 sequences × 16 gradient accumulation × 2048 tokens = 262,144 source tokens per update.
- Fused AdamW, peak LR 0.001, betas (0.9, 0.95), weight decay 0.1 excluding bias/norm/1D parameters, gradient clipping 1.0. Warmup 150 steps, cosine decay to 10% of peak LR.
- BF16 compute, one RTX 5070 Ti, seed 2026; released checkpoint weights remain FP32. [Training configuration](training_config.json).

## Evaluation

Full selected splits, lm-eval 0.4.12, no added few-shot examples or chat template, BF16 on RTX 5070 Ti, context cap 2048 (ArithMark 1024). New evaluations explicitly disable TF32; historical core manifests predate the explicit TF32 flag. Scores below are percentages; **± is one standard error**, and the separate interval column is 95% CI where available.

| Dataset | Split | Examples | Metric | Score ± SE (%) | 95% CI (%) |
|---|---|---:|---|---:|---:|
| [HellaSwag](https://huggingface.co/datasets/Rowan/hellaswag) | validation | 10,042 | `pll_acc_norm` | 25.97 ± 0.44 | [25.12, 26.84] |
| [ARC-Easy](https://huggingface.co/datasets/allenai/ai2_arc) | test | 2,376 | `pll_acc_norm` | 26.68 ± 0.91 | [24.94, 28.50] |
| [ARC-Challenge](https://huggingface.co/datasets/allenai/ai2_arc) | test | 1,172 | `pll_acc_norm` | 24.06 ± 1.25 | [21.70, 26.59] |
| [PIQA](https://huggingface.co/datasets/baber/piqa) | validation | 1,838 | `pll_acc_norm` | 50.05 ± 1.17 | [47.77, 52.34] |
| [WinoGrande](https://huggingface.co/datasets/allenai/winogrande) | validation | 1,267 | `pll_acc` | 50.99 ± 1.40 | [48.24, 53.73] |
| [OpenBookQA](https://huggingface.co/datasets/allenai/openbookqa) | test | 500 | `pll_acc_norm` | 25.80 ± 1.96 | [22.16, 29.81] |
| [BoolQ](https://huggingface.co/datasets/aps/super_glue) | validation | 3,270 | `pll_acc` | 37.83 ± 0.85 | [36.18, 39.50] |
| [LAMBADA OpenAI](https://huggingface.co/datasets/EleutherAI/lambada_openai) | test | 5,153 | `pll_acc` | 0.00 ± 0.00 | [0.00, 0.07] |
| [ArithMark-3](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) | train | 1,000 | `pll_acc_norm` | 29.90 ± 1.45 | [27.14, 32.81] |
| [Balanced COPA](https://huggingface.co/datasets/pkavumba/balanced-copa) | train | 1,000 | `pll_acc` | 50.00 ± 1.58 | [46.91, 53.09] |
| [BLiMP](https://huggingface.co/datasets/nyu-mll/blimp) | train | 67,000 | `pll_acc` | 57.39 ± 0.16 | — |
| [CommonsenseQA](https://huggingface.co/datasets/tau/commonsense_qa) | validation | 1,221 | `pll_acc` | 19.57 ± 1.14 | [17.45, 21.89] |
| [MMLU continuation](https://huggingface.co/datasets/cais/mmlu) | test | 14,042 | `pll_acc` | 23.38 ± 0.36 | — |
| [SciQ (with support)](https://huggingface.co/datasets/allenai/sciq) | test | 1,000 | `pll_acc_norm` | 23.70 ± 1.35 | [21.17, 26.43] |
| [TruthfulQA MC2](https://huggingface.co/datasets/truthfulqa/truthful_qa) | validation | 817 | `pll_acc` | 48.99 ± 1.64 | — |
| [BananaMind Base 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1) | test | 350 | `pll_raw_accuracy` | 26.29 ± 2.36 | [21.95, 31.14] |

**These are experimental single-mask continuation PLL scores, not autoregressive likelihood. Other answer tokens remain visible. LAMBADA is whole-answer reconstruction under this protocol, not ordinary next-word generation. Compare this model in a separate diagnostic group.** LAMBADA requires all final-word tokens to match. The descriptive macro-average is **32.5378%**: one primary metric per each of the 16 tasks, with equal task weight. PPL, standard errors, CI bounds and MMLU/BLiMP leaf scores are excluded. It is not an official leaderboard score or a statistical ranking test.

Full selected splits; lm-eval 0.4.12; seed 1234; BF16 on RTX 5070 Ti;
context cap 2048 (ArithMark 1024), TF32 disabled on new runs (historical core flag unrecorded), no chat template and no added
few-shot examples. TruthfulQA retains the harness's fixed six-QA preamble.
ArithMark and BananaMind normalize by continuation token count; ordinary harness
acc_norm uses its own length normalization. BananaMind is raw accuracy, not Elo.
SciQ includes the support passage. Balanced COPA uses the mirrored 1000-item
train-named evaluation split; cRia's split was inferred, not confirmed.
MMLU scores full answer continuations across 57 subjects, weighted by item count;
BLiMP averages 67 equal-sized minimal-pair subsets. Standard errors are retained
from each evaluator. Wilson intervals are reported only where the runner logged
binary item accuracy; MC2 is probability mass, not binary accuracy. These
intervals do not model dependence between paired/templated examples or training
seed variation. UL2, PrefixLM and experimental diffusion PLL use their documented
conditional scoring protocols; PLL exposes the other answer tokens and is not
autoregressive likelihood. cRia's published scores used a different precision
and benchmark-adapted checkpoint; this completes our comparison coverage, not
an independent reproduction of cRia or an official leaderboard submission.

**WikiText-2 raw test continuation**, reported separately: 291 nonoverlapping blocks of 1024 tokens, 512-token prefix plus 512-token scored suffix; 148,992 scored tokens, 335 tail tokens excluded. GPU evaluation; percentile block bootstrap with 10,000 resamples and seed 2026. PPL intervals exponentiate NLL endpoints. This is not rolling or word perplexity.

- `experimental_bfloat16_pll`: NLL **8.238186**, 95% CI [8.214775, 8.261612]; pseudo-PPL **3782.672**, 95% CI [3695.144, 3872.330].

[Structured results and provenance](evaluation/results.json), [raw task outputs](evaluation/raw/) and [per-block continuation scores](evaluation/continuation.json). HF metadata contains author-reported measurements; no verified benchmark badge is claimed. Historical and new measurements refer to identical weight hashes; individual run manifests retain their original dates and source hashes.

## Usage

Install `requirements.txt`; tested with PyTorch 2.11.0 and Transformers 5.17.0. Loading custom code requires `trust_remote_code=True`. Example on CPU:

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
repo = "d0rj/diffusion-51M-base-v1-failed"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
# Diagnostic only: this failed checkpoint does not produce useful text.
ids = model.generate_masked(batch_size=1, seq_len=16, steps=16, temperature=0.)
print(tokenizer.decode(ids[0], skip_special_tokens=True))
```

After downloading this repository, install the evaluation dependencies and authenticate for the gated BananaMind dataset after accepting its terms:

```bash
pip install -r evaluation/repro/requirements.txt
hf auth login
python evaluation/repro/run.py --device cuda:0 --dtype bfloat16 --batch-size 1 --output comparison-rerun
python evaluation/repro/continuation.py --device cuda:0 --dtype bfloat16 --output continuation-bf16.json
```

`--limit` / `--limit-blocks` are smoke checks only. Dataset examples are not redistributed. The bundled runners use this published model and tokenizer.

## Training logs and limitations

[TensorBoard event files](tensorboard/) contain available training telemetry and the full evaluation at optimizer step 15,000. `train/loss` has 750 visible points spanning steps 20–15,000. Resume/purge records are retained. [Comet experiment](https://www.comet.com/d0rj/tiny-llm-ablation/0adc3d5b01bf4703bf410dd7f43f3df0). Evaluation exports retain numeric metrics, uncertainty and task/subtask breakdowns.

Equal source-token budgets do not imply equal target supervision, parameter count or FLOPs. Training dataset file revisions and overlap with benchmarks were not independently audited. Results come from one training seed; uncertainty intervals do not capture training-seed variability or dependence between templated examples. These are small base models, not instruction-tuned assistants.