Publish petitgpt research-v1 native alpha075 model and scoped documentation

DOCUMENTATION_LICENSE.md

@@ -0,0 +1,9 @@
+# Licence scope
+
+Copyright 2026 Yang Qi. Owner-controlled code and the selected model/tokenizer are licensed under the standard Apache License 2.0 in LICENSE, only for rights Yang Qi is entitled to grant.
+
+As an explicit exception to the root code licence, author-written reports and documentation (including README, native run guide and versioned report) are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/ and https://creativecommons.org/licenses/by/4.0/legalcode.en . Attribute Yang Qi and petitgpt, link the licence, and indicate changes. Existing third-party content and notices retain their applicable terms; they are not relicensed.
+
+Research describes intended use; it adds no noncommercial or research-only restriction to Apache-licensed artifacts. This grant was approved by the owner through the explicit research-release execution instruction. Earlier PENDING_OWNER_DECISION records remain historical evidence; this does not claim an earlier licence choice.
+
+Source metadata is not rights clearance. Weights are not the raw corpus; neither automatic inheritance nor automatic non-application of all dataset terms is asserted. No infringement guarantee or legal certification is given. A disclaimer does not replace applicable permission.

LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

MODEL_PROVENANCE.json

@@ -0,0 +1,26 @@
+{
+  "project": "petitgpt",
+  "author": "Yang Qi",
+  "checkpoint": "alpha075",
+  "lineage": [
+    "Base",
+    "P2 step750",
+    "P3 step320",
+    "interpolation alpha=0.75"
+  ],
+  "excluded_updates": [
+    "DeepSeek-response-KD",
+    "unified Base-SFT",
+    "DPO",
+    "soft-KD",
+    "LoRA"
+  ],
+  "source_checkpoint_sha256": "1da85cc329d55e92dacf51c36623779558c4c6c9a39d78a34a064f61fcddbe97",
+  "model_safetensors_sha256": "4396efb7a52b047e7fdf513e46d1b401dfc70582d3aca1f9cb5a07e97d426ef1",
+  "tokenizer_sha256": "d8f84df58928023edebd809e152b3b38a0dac53b9f887bd2455f427661e9b9ce",
+  "accepted_archive_sha256": "0c844962c6ebfcc1cd6b17cdb19c05d88936fa280f0173c080ff864bb54dca1e",
+  "unique_parameters": 124635456,
+  "release_model_operations": 0,
+  "license": "Apache-2.0 only for author-controlled rights; see notices",
+  "tokenizer_source_linkage": "Digest-linked to six pinned corpus releases; PES2O and StackExchange not in tokenizer corpus"
+}

README.md

@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+tags:
+- petitgpt
+- native-pytorch
+- research
+---
+
+# petitgpt
+
+Author: Yang Qi. Selected checkpoint: alpha075.
+
+## Identity
+
+| Field | Value |
+|---|---|
+| Status | accepted native research inference artifact |
+| Unique parameters | 124,635,456 (124.6M) |
+| Layers / width / FFN | 30 / 576 / 1536 |
+| Attention | 9 query heads, 3 key/value heads (GQA), head dim 64 |
+| Vocabulary / context | 32,000 / 2,048 |
+| Embeddings | tied input/output |
+| Normalization | RMSNorm, epsilon 1e-6 |
+| Positions | RoPE, theta 10000, full head rotation |
+| Dropout | 0.0 |
+| Stored weights | FP32 |
+
+Complete checkpoint-derived settings ship in the bundle's `config.json`. The three core modules (model, chat template, token contract) are byte-identical to the project originals. Checkpoint and archive hashes are in MODEL_PROVENANCE.json.
+
+## Provenance
+
+The selected weights are a **parameter interpolation**, not a training step:
+
+> `theta = theta_P2_step750 + 0.75 · (theta_P3_step320 − theta_P2_step750)`
+
+Ancestry: tokenizer release → Stage A pretraining (steps 0–38,146) → Stage B continued pretraining (steps 38,146–49,590, exact full-state resume, accepted as Base) → P2 concise-instruction SFT (750 updates, weights-only initialization from Base) → P3 basic-instruction adaptation (parent B is **step 320**, not the step-640 endpoint) → this interpolation, executed with **zero optimizer updates and zero backward passes** → a numerically unchanged FP32 export.
+
+This checkpoint **does not contain** later DeepSeek-response-KD, unified Base-SFT, DPO, soft-KD or LoRA branch updates. Several later branches were initialized *from* it (a one-pass behaviour mix, a DPO pilot, a chosen-answer CE control, and two response-distillation runs); others were not — a unified SFT curve started from the accepted pretrained Base, a loss-allocation A/B split from that curve's step 403, and a shared-tokenizer soft-KD lab ran entirely on external models. A preference-data build used this model's generations but produced no checkpoint. Branch exposures must not be summed into this model's training history.
+
+## Training data
+
+Pretraining consumed 13,000,005,634 retained packed tokens over 13,755,731 documents, of which the optimizer stepped over 12,999,720,960 model-input positions, one exposure per block, with no replay of the executed token-position traversal.
+
+**Stage A (10,000,003,234 selected serialized tokens, 4 sources):** FineWeb-Edu dedup 71.11%, DCLM-Edu 20.32%, Wikipedia (FineWiki EN) 5.08%, Python-Edu 3.50%.
+
+**Stage B (3,000,004,240 selected serialized tokens, 7 sources):** FineWeb-Edu dedup 40.10%, DCLM-Edu 22.92%, structured tutorial content 11.46%, Python-Edu 8.33%, Wikipedia 5.73%, PES2O 5.73%, StackExchange 5.73%.
+
+Upstream datasets, pinned revisions and the licence string recorded at each pinned revision are in `tables/PRETRAIN_SOURCE_MIXTURE.csv`. Those recorded strings are evidence of what the builder captured at that revision; they are not a legal determination, are not asserted to be today's terms, and do not by themselves determine the licence of trained weights.
+
+Post-training used seven instruction subsets. These are values of a row-level `source` column inside one pinned collection, `HuggingFaceTB/smol-smoltalk` at revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, config `default`, train split, established by digest joins through the project's own census and cleanup records. The publisher's card at that pinned revision carries a flat Apache-2.0 badge; its parent collection limits that grant to four newly generated subsets and refers readers to the original dataset for each incorporated public dataset. Four of the seven labels correspond to the newly generated subsets. Of the three incorporated components, one declares `apache-2.0`, one declares `odc-by`, and one declares no licence in its card metadata. Component notices were read from current publisher pages, not from revisions contemporaneous with this training run, and **no component revision is established**. No licence determination is made here.
+
+## Intended use, and use it is not intended for
+
+**Intended:** research and engineering study of a small from-scratch language model — reproducing the recorded measurements, inspecting the pipeline, and analysing failure modes.
+
+**Not intended:** a general assistant, a production system, anything safety- or correctness-certified, or a source of factual answers. Do not execute code it generates without independent review.
+
+**Not evaluated at all:** long-context work, multilingual behaviour, tool use, extended multi-turn dialogue, safety and refusal behaviour, factual currency, retrieval, and any public generative benchmark.
+
+## Evaluation — public multiple-choice likelihood
+
+Frozen FP32 results, copied byte-identically from the accepted measurement and **not recomputed**:
+
+| Dataset | Split / documents | acc | acc_norm |
+|---|---|---|---|
+| ARC-Easy | test / 2,376 | 1372/2376 = 0.5774410774410774 | 1244/2376 = 0.5235690235690236 |
+| PIQA | validation / 1,838 | 1167/1838 = 0.6349292709466812 | 1145/1838 = 0.6229597388465724 |
+
+Protocol: zero-shot raw `Question: …\nAnswer:` completion scored by candidate-answer likelihood. No chat template, no role tokens, no few-shot examples, no BOS insertion, no scored EOS, no generation, no cleanup. FP32 parameters and forward with autocast disabled, TF32 off for matmul and cuDNN, MATH SDPA, batch size 1 unpadded, no KV cache, no compile. `acc` is the first argmax of summed continuation log-likelihood; `acc_norm` divides by `len()` of the **original** answer text in Unicode characters, not tokenizer length; ties take the first index.
+
+Comparators measured under the identical protocol on the same rows: SmolLM-135M-Instruct 0.4924 / 0.6708 and SmolLM2-135M-Instruct 0.5400 / 0.6670 (acc, ARC-Easy / PIQA).
+
+**Qualifications.** The evaluator is a native protocol-compatible implementation pinned to a specific lm-evaluation-harness commit; the harness package was not installed and a full installed-harness run is not claimed. Both datasets are prior project diagnostics with no contamination audit — they are **not untouched final tests**. Training data, compute, tokenizers and architectures are unmatched across the three models. This is a protocol-bounded descriptive comparison; no significance test was run. Multiple-choice accuracy does not establish free-generation reliability.
+
+## Evaluation — historical full-answer assistant review
+
+A separate evaluation family scored generated text across 189 prompts per model (567 answers, 565 distinct prompt/output/contract units) over three models. **Two named versions exist and must not be combined in one table:** `assistant_review_fable_v1` (the original 567 final records) and `assistant_owner_clarification_4_v1` (four explicit final-content decisions, every other axis preserved). The version shown below is `assistant_owner_clarification_4_v1`.
+
+| Slice | Content / joint (true / false / unknown) | Other axes |
+|---|---|---|
+| old_qa41 | 6 / 33 / 2 | format 0/1 |
+| old_practical38 | 10 / 26 / 2 | explicit format 20/21 |
+| old_python14 | 0 / 14 / 0 | interface 11/14; finite execution `not_recorded` |
+| new_natural64 | 3 / 61 / 0 | format 7/18 |
+| new_python32 | 0 / 32 / 0 | interface 31/32 |
+
+Practical joint bounds: **26/81 .. 10/27** (= 52/162 .. 60/162). This denominator is **27 equally weighted dialogue groups over 38 scored turns** — rows are averaged inside a group first — not an ordinary row average. These are exact unknown-retention bounds, **not confidence intervals**.
+
+**Qualifications.** Judgments are model-assisted, not human adjudication. The chronology was: an initial pass over the 565 units with model metadata masked and its own recorded limitations; then a pass with the mapping visible that produced 17 consistency edits; then four owner clarifications forming a separate version. It was therefore neither strictly blinded throughout nor fully label-visible throughout. Development sets were reused across many runs. `not_recorded` means the field is unavailable in this imported view — it does not establish that no historical function test was ever run. Unsupported-builtin results stay unknown and are never converted into demonstrated failures.
+
+In these specific Python diagnostics the model produced a correct function **interface** in 42 of 46 prompts and a correct **whole answer** in 0 of 46. This does not establish that it can never write correct code.
+
+Ordinary QA and complete natural-task generation remained limited across the evaluated configurations. Individual results differ by suite and label version and are reported with their source rather than reduced to a cross-version maximum; see the technical report §10.1.
+
+## Export parity
+
+Eight frozen fixture pairs (four `bf16_native`, four `fp32_math`) matched exactly on prompt IDs, boundaries, full-shape logits, greedy output IDs and stop reason, with a maximum absolute logit difference of **0 within each profile**. All 213 named state entries and 60 non-persistent rotary buffers matched after strict load and safetensors reload.
+
+This is a **numerical parity check between source and export under the same profile**. It is not a quality test, not a semantic evaluation, and it does not assert that the two profiles agree with each other. Local import closure was demonstrated once in a fresh isolated process on the measured environment; that is **not** a clean-machine, CPU, cross-hardware or fresh-installation test.
+
+## Format support
+
+Native PyTorch CUDA inference only. **No** Transformers `AutoModel`, GGUF, ONNX, vLLM or llama.cpp compatibility is implemented or tested. `special_tokens_map.json` is descriptive native metadata. A CUDA GPU is required.
+
+## Licence and distribution
+
+Copyright 2026 Yang Qi. Owner-controlled code and the selected model/tokenizer are licensed under the standard Apache License 2.0 in LICENSE, only for rights Yang Qi is entitled to grant.
+
+As an explicit exception to the root code licence, author-written reports and documentation (including README, native run guide and versioned report) are licensed under Creative Commons Attribution 4.0 International (CC BY 4.0): https://creativecommons.org/licenses/by/4.0/ and https://creativecommons.org/licenses/by/4.0/legalcode.en . Attribute Yang Qi and petitgpt, link the licence, and indicate changes. Existing third-party content and notices retain their applicable terms; they are not relicensed.
+
+Research describes intended use; it adds no noncommercial or research-only restriction to Apache-licensed artifacts. This grant was approved by the owner through the explicit research-release execution instruction. Earlier PENDING_OWNER_DECISION records remain historical evidence; this does not claim an earlier licence choice.
+
+Source metadata is not rights clearance. Weights are not the raw corpus; neither automatic inheritance nor automatic non-application of all dataset terms is asserted. No infringement guarantee or legal certification is given. A disclaimer does not replace applicable permission.
+
+## Recorded runtime
+
+Python 3.10.12, torch 2.11.0+cu126, numpy 2.2.6, tokenizers 0.22.2, safetensors 0.8.0, NVIDIA GeForce RTX 4090. Matching versions do not guarantee bit-identical results on other hardware or untested software; recorded driver versions differ across project phases, which is a recorded difference rather than a resolved equivalence.
+
+## Files and report
+
+See [native run guide](RUN_GUIDE.md), [source notice](SOURCE_NOTICE.md), [third-party notices](THIRD_PARTY_NOTICES.md) and [file manifest](SHA256SUMS).
+
+GitHub project: https://github.com/yangqi0/petitgpt . The complete research-v1 report is staged for `docs/petitgpt-v1/TECHNICAL_REPORT.md` there; publication is blocked by missing GitHub authentication. That report is not yet publicly available, and no live report link is claimed.

RUN_GUIDE.md

@@ -0,0 +1,125 @@
+# petitgpt native run guide
+
+Author: Yang Qi. Documentation: CC BY 4.0. Download the loose model repository files together, preserving the src/ directory. The exact release file list and hashes are in SHA256SUMS. Model weights, tokenizer, config and executable code retain the accepted export bytes.
+
+## 2. Requirements
+
+A **CUDA GPU is required** by this CLI. The tested configuration is Python 3.10.12 with:
+
+```
+torch==2.11.0+cu126
+numpy==2.2.6
+tokenizers==0.22.2
+safetensors==0.8.0
+```
+
+on an NVIDIA GeForce RTX 4090 (driver 580.178.04). Prepare the environment separately, from locally supplied wheels:
+
+```sh
+python -m pip install --no-index --find-links /path/to/local/wheelhouse \
+    -r /path/to/bundle/requirements-inference-tested.txt
+```
+
+No packages were installed during the export itself. Matching versions do not guarantee bit-identical results on arbitrary hardware or untested software.
+
+## 3. Command line
+
+Replace `/path/to/bundle` with the real extracted location.
+
+```sh
+python /path/to/bundle/inference.py \
+    --model-directory /path/to/bundle \
+    --prompt "Say hello in one sentence." \
+    --profile bf16_native \
+    --max-new-tokens 32
+
+python /path/to/bundle/inference.py \
+    --model-directory /path/to/bundle \
+    --messages-json /path/to/messages.json \
+    --profile fp32_math \
+    --max-new-tokens 32
+```
+
+A synthetic `messages.json`:
+
+```json
+[{"role":"system","content":"Use short sentences."},
+ {"role":"user","content":"My name is Lin."},
+ {"role":"assistant","content":"Hello, Lin."},
+ {"role":"user","content":"What name did I give you?"}]
+```
+
+> **Illustrative and unexecuted.** The greeting above is an authored example showing command syntax only. It was written for this guide, was not run, and is not a demonstration of model quality. Do not put frozen evaluation prompts or their outputs into a public demo.
+
+## 4. Python API
+
+With the extracted bundle directory on `sys.path`:
+
+```python
+from inference import load_bundle, generate
+
+model, tokenizer = load_bundle("/path/to/bundle")
+model = model.to("cuda").eval()
+result = generate(
+    model, tokenizer,
+    [{"role": "user", "content": "Say hello."}],
+    cap=32,
+    profile="fp32_math",
+)
+print(result["output_text_raw_including_terminal_eos"])
+```
+
+## 5. Input contract
+
+`messages` must be a JSON array of objects with **exactly** the fields `role` and `content`. A conversation is an optional initial system turn followed by alternating user/assistant turns, **ending in user**. A plain `--prompt` becomes one user message.
+
+Rejected, by design, with an explicit error rather than a repair:
+
+| Case | Recorded rejection |
+|---|---|
+| Missing `content` | `Each message must contain only role and content` |
+| Unsupported role (e.g. `tool`) | `message 0 has invalid role 'tool'` |
+| Any extra message field | `Each message must contain only role and content` |
+| Conversation ending in `assistant` | `chat must end with a non-empty user turn; got 'assistant'` |
+| Prompt + budget over 2,048 | `Context overflow: prompt plus token budget exceeds 2048` |
+| `max_new_tokens` outside 1..384 | `max_new_tokens must be 1..384` |
+
+`default_system=None`: a supplied system turn and full history are retained, with **no injected default and no text normalization**. Literal special-token spellings inside content — for example the literal spelling `[EOS]` — are encoded as ordinary text and cannot inject control IDs.
+
+Native token structure:
+
+```
+[BOS] <|system|> system <|user|> user <|assistant|> assistant [EOS] … <|user|> user <|assistant|>
+```
+
+The system segment is omitted when absent. `[BOS]` occurs once; `[EOS]` closes completed assistant turns; no duplicate assistant prefix is added. IDs are fixed: `[PAD]=0`, `[UNK]=1`, `[BOS]=2`, `[EOS]=3`, `<|system|>=4`, `<|user|>=5`, `<|assistant|>=6`.
+
+## 6. Precision profiles
+
+Both profiles store FP32 parameters. They differ only in the forward numerical path, and **they are not asserted to agree with each other**.
+
+| | `bf16_native` | `fp32_math` |
+|---|---|---|
+| Forward | CUDA BF16 autocast | FP32, autocast off |
+| matmul TF32 | off | off |
+| cuDNN TF32 | **on** | off |
+| `float32_matmul_precision` | highest | highest |
+| SDPA backend | native backends enabled | MATH |
+
+## 7. Decoding and stopping
+
+Greedy only: `temperature=0`, `top_k=0`, `top_p=1`, `EOS=3`. Default `max_new_tokens` is **384**; an explicit lower integer in `1..384` is supported. Prompt length plus budget must fit 2,048 — there is no cropping.
+
+Output JSON preserves the generated token IDs, the raw text **including the terminal EOS**, and a stop reason of `eos` or `max_new_tokens`. There is no answer cleanup, no fact fixing, no best-of-N, and no retry. A token cap can truncate an answer mid-sentence; that is the recorded behaviour, not a defect.
+
+## 8. Format support and limits
+
+Native PyTorch CUDA inference only. There is **no implemented or tested** Transformers `AutoModel`, GGUF, ONNX, vLLM or llama.cpp path. `special_tokens_map.json` is descriptive native metadata, not a loader contract.
+
+Local import closure was demonstrated once, in a fresh process with a temporary working directory and an empty `PYTHONPATH`, under an audit hook that denied network access and out-of-bundle repository access; no denied access occurred. That establishes closure **on the measured environment only**. It is not a clean-machine test, not a CPU test, not a cross-hardware test, and not a fresh dependency-installation test.
+
+The recorded export parity is likewise **profile-specific**: within each of the two profiles the export reproduced its source exactly, and no claim is made that the two profiles agree with each other, nor that any Transformers, GGUF, ONNX, vLLM or llama.cpp path exists.
+
+## 9. Before you use output
+
+This is a research artifact, not a safety- or correctness-certified assistant. In the recorded Python diagnostics the model produced a valid function interface far more often than a correct whole answer. **Do not execute generated code without independent review.** Backend details and the bounded parity evidence live in a separate private evidence archive and are not part of this bundle.

SHA256SUMS

@@ -0,0 +1,21 @@
+74322f799773266d15acf47c40165d2c7e5bf9df8e5ca7a076dd98e51ee41656  DOCUMENTATION_LICENSE.md
+cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
+936b54b619edeb38bc5ff13e01e4bb1682f7d5c6f8fdf78756cc4c4c9f953aaf  MODEL_PROVENANCE.json
+d6ce84427abcf0f0a3d1d45475c5282c6e5bd9b5b88d9055ac502698cee66605  README.md
+b6e7c48e4c287517c1c4e0013ed0f8b33f49cae479d185fa7af734e057ab28a4  RUN_GUIDE.md
+46dc9f7c632aa5b7ffeb5678d76dbe03a9f78b70a8d721f8a3d96dda4a0f5876  SOURCE_NOTICE.md
+aeb343a8a11ef0bffa736b360e73273d5a05f48429ae8346a97517155b75c3ef  THIRD_PARTY_NOTICES.md
+fe6cd464a09a0605f6ead38867c0497edc71de31fa45446f790e3f9a4f937369  config.json
+17e307631b8e5fd8be8c9d7e992632a6dfbc26708287318fca1259f6d1dd548d  inference.py
+4396efb7a52b047e7fdf513e46d1b401dfc70582d3aca1f9cb5a07e97d426ef1  model.safetensors
+dd6b7d954f0153cc6898a9aa0f45901e3727906172e0045d9bc776daf2f7163e  requirements-inference-tested.txt
+79bc49c15e40cfc59779b5559d452c058a06f684a2867d97c3c0190933d8801f  special_tokens_map.json
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  src/__init__.py
+5c4214f0d2a985ee7b24da431ce1bf521c96ae6f7dfa816ef800b505eb68013c  src/accepted_generate.py
+c25068dcb7d3a907dba7b7405d53a2962d91886abcaa25b0d5832a82e8bc821b  src/chat_template.py
+2bc9fa8ae16636837c4a2937301a2419d0ac92faa2cc27560dacbd29a5144dc2  src/model.py
+f767b864d7c8e0cb5e2c166c4f019f3f14666dcbf2b944c7909db050e4cf1e96  src/special_tokens.py
+4dd0077babf854389d09623c154844b46f977ca64753f06ecf50662297a2caf5  tables/ASSISTANT_RESULTS_VERSIONED.csv
+b452952e2762f6fe07484e995a7d590f8659c5d5cf60d13e1785cd975a48ace9  tables/PRETRAIN_SOURCE_MIXTURE.csv
+5350ff2d0b30e67645d3b088974f893a9943d9c5f7407f45548917f1c54da297  tables/PUBLIC_BENCHMARK_RESULTS.csv
+d8f84df58928023edebd809e152b3b38a0dac53b9f887bd2455f427661e9b9ce  tokenizer.json

SOURCE_NOTICE.md

@@ -0,0 +1,39 @@
+# Source notice
+
+petitgpt by Yang Qi; selected checkpoint alpha075. Parameter lineage: Base -> P2 step750 -> P3 step320 -> interpolation (0.75 toward P3). Later DeepSeek-response-KD, unified Base-SFT, DPO, soft-KD and LoRA updates are absent.
+
+Pretraining source and tokenizer linkage have been established by the existing digest records. The tokenizer corpus uses six of the eight frozen releases; PES2O and StackExchange occur in pretraining, not tokenizer training. Selected source counts are not per-source consumed-token measurements. No new corpus audit is claimed.
+
+## Pretraining and tokenizer sources
+
+The following terms are builder-recorded metadata at pinned revisions, not independent verification of every licensor's rights. Dataset authors and contributors retain their rights. Exact mixture and transport revisions are preserved in the companion PRETRAIN_SOURCE_MIXTURE.csv.
+
+| Dataset / config | Pinned revision | Recorded terms |
+|---|---|---|
+| HuggingFaceTB/smollm-corpus / fineweb-edu-dedup | `3ba9d605774198c5868892d7a8deda78031a781f` | odc-by-1.0 |
+| HuggingFaceTB/dclm-edu / default | `dbad8ad71224482740cd9c9d353591adbf62fe04` | cc-by-4.0 |
+| HuggingFaceFW/finewiki / en | `8bd13e72e6a002407649b3e898535f42ceb1aeb9` | cc-by-sa-4.0 |
+| common-pile/stackv2_edu_filtered / default | `c354dbe88469a1153e97c6a63ac50591849654de` | per-record metadata.license (Software Heritage permissive subset) |
+| HuggingFaceTB/smollm-corpus + HuggingFaceFW/finephrase / cosmopedia-v2 + tutorial | `3ba9d605774198c5868892d7a8deda78031a781f + 78cf4a5ed0099214979c094c963e699c19163838` | odc-by-1.0 (both) |
+| allenai/dolmino-mix-1124 / pes2o | `a319f19eef1e257417b11ea8c30da266ae175557` | odc-by-1.0 |
+| allenai/dolmino-mix-1124 / stackexchange | `a319f19eef1e257417b11ea8c30da266ae175557` | cc-by-sa |
+
+Publisher datasets: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus ; https://huggingface.co/datasets/HuggingFaceTB/dclm-edu ; https://huggingface.co/datasets/HuggingFaceFW/finewiki ; https://huggingface.co/datasets/common-pile/stackv2_edu_filtered ; https://huggingface.co/datasets/HuggingFaceFW/finephrase ; https://huggingface.co/datasets/allenai/dolmino-mix-1124 .
+
+## Instruction components
+
+The collection is HuggingFaceTB/smol-smoltalk, revision `f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc`, default/train. Its pinned card has an Apache-2.0 badge. P2's seven labels are source-column values, not separate repositories. Parent/component correspondence is documentary, not a per-row join; exact component revisions remain unestablished. Parent and component notices below are CURRENT_ONLY observations retrieved on 2026-09-10, not terms proven contemporaneous with training.
+
+- **openhermes-50k**: teknium/OpenHermes-2.5. Recorded metadata/grant: no licence in component-card metadata. component dataset card metadata at head b82037821055c377bed0d495e72e46de3bc72e84 (retrieved 2026-09-10T17:53:40Z)
+- **smol-contraints**: Smol-contraints. Recorded metadata/grant: Apache-2.0. parent collection card at head 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa (retrieved 2026-09-10T17:52:48Z): "All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0."
+- **smollm-rewrite-30k**: Smol-rewrite. Recorded metadata/grant: Apache-2.0. parent collection card at head 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa (retrieved 2026-09-10T17:52:48Z): "All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0."
+- **smol-magpie-ultra-short**: Smol-Magpie-Ultra. Recorded metadata/grant: Apache-2.0. parent collection card at head 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa (retrieved 2026-09-10T17:52:48Z): "All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0."
+- **self-oss-instruct**: bigcode/self-oss-instruct-sc2-exec-filter-50k. Recorded metadata/grant: odc-by. component dataset card metadata at head 356bb069eee815daa6e23e9a282eeefe1490ad44 (retrieved 2026-09-10T17:53:40Z)
+- **smol-summarize-20k**: Smol-summarize. Recorded metadata/grant: Apache-2.0. parent collection card at head 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa (retrieved 2026-09-10T17:52:48Z): "All the new datasets (Smol-Magpie-Ultra, Smol-contraints, Smol-rewrite, Smol-summarize) are licensed under Apache 2.0."
+- **everyday-conversations**: HuggingFaceTB/everyday-conversations-llama3.1-2k. Recorded metadata/grant: apache-2.0. component dataset card metadata at head 14f543216b9ba42b6b951dc5bd199460d193b162 (retrieved 2026-09-10T17:53:41Z)
+
+The parent publisher limits its Apache grant to its newly generated subsets and points to component-specific terms for incorporated datasets. The self-oss-instruct ODC-By metadata differs from the collection badge; a declaration difference is not proof of legal incompatibility. OpenHermes-2.5 refers to component-specific licences; its empty card licence metadata does not resolve those terms, and exact upstream-component rows for this subset remain unresolved. Ordinary historical unknowns have not been newly resolved.
+
+Sources: https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk/tree/f73fe857d519ff6ac5af2ea67c4d3834da7b8bcc ; https://huggingface.co/datasets/HuggingFaceTB/smoltalk ; https://huggingface.co/datasets/teknium/OpenHermes-2.5 ; https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k ; https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k .
+
+No raw corpus, frozen evaluation prompts or model answers are distributed. Author licensing does not relicense upstream datasets or clear third-party rights. See DOCUMENTATION_LICENSE.md for the scoped grant.

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,5 @@
+# Third-party notices
+
+Existing third-party code, attribution headers and notices retain their own applicable terms; the root Apache-2.0 licence grants only rights controlled by Yang Qi. The native code is copied byte-for-byte from the accepted export, with its existing comments preserved. Dependencies are declared, not vendored: PyTorch, NumPy, Hugging Face Tokenizers and Safetensors retain their upstream licences and notices. This release makes no blanket claim about all existing repository code.
+
+Dataset authors and contributors are acknowledged in SOURCE_NOTICE.md, including ODC-By components, Wikipedia/FineWiki and StackExchange share-alike metadata, per-record Python licence metadata, and unresolved OpenHermes component terms. Dataset terms are not replaced by the model's Apache-2.0 declaration.

config.json

@@ -0,0 +1,13 @@
+{
+  "vocab_size": 32000,
+  "n_layers": 30,
+  "d_model": 576,
+  "n_heads": 9,
+  "n_kv_heads": 3,
+  "d_ff": 1536,
+  "max_seq_len": 2048,
+  "dropout": 0.0,
+  "tie_embeddings": true,
+  "rope_theta": 10000.0,
+  "rope_pct": 1.0
+}

inference.py

@@ -0,0 +1,63 @@
+"""Local native alpha075 loading and precision profiles."""
+import argparse,json
+from pathlib import Path
+from contextlib import contextmanager
+import torch
+from torch.nn.attention import sdpa_kernel,SDPBackend
+from safetensors.torch import load_model
+from src.model import GPT,gpt_config_from_checkpoint_dict,audit_gpt_parameter_count
+from src.chat_template import load_chat_tokenizer,encode_prompt
+from src.accepted_generate import generate as accepted_generate
+
+@contextmanager
+def precision(profile):
+    if profile not in ('bf16_native','fp32_math'): raise ValueError('Unsupported precision profile')
+    torch.set_float32_matmul_precision('highest')
+    torch.backends.cuda.matmul.allow_tf32=False
+    torch.backends.cudnn.allow_tf32=profile=='bf16_native'
+    with sdpa_kernel([SDPBackend.MATH] if profile=='fp32_math' else [SDPBackend.FLASH_ATTENTION,SDPBackend.EFFICIENT_ATTENTION,SDPBackend.MATH,SDPBackend.CUDNN_ATTENTION]):
+        with torch.autocast('cuda',dtype=torch.bfloat16,enabled=profile=='bf16_native'):
+            yield
+
+def load_bundle(model_dir):
+    root=Path(model_dir).resolve()
+    cfg=gpt_config_from_checkpoint_dict(json.loads((root/'config.json').read_text()))
+    model=GPT(cfg).eval()
+    load_model(model,str(root/'model.safetensors'),strict=True,device='cpu')
+    audit=audit_gpt_parameter_count(model,cfg)
+    if audit['actual_total']!=124635456: raise ValueError('Parameter identity guard failed')
+    if model.tok_emb.weight is not model.lm_head.weight: raise ValueError('Embedding tie lost')
+    if model.tok_emb.weight.data_ptr()!=model.lm_head.weight.data_ptr(): raise ValueError('Storage tie lost')
+    if any(p.dtype!=torch.float32 for p in model.parameters()): raise ValueError('Expected FP32 weights')
+    return model,load_chat_tokenizer(str(root/'tokenizer.json'))
+
+def validate_input(tok,messages,cap):
+    if not isinstance(messages,list) or not messages: raise ValueError('Messages must be a non-empty JSON array')
+    if any(not isinstance(m,dict) or set(m)!={'role','content'} for m in messages): raise ValueError('Each message must contain only role and content')
+    if isinstance(cap,bool) or not isinstance(cap,int) or not 1<=cap<=384: raise ValueError('max_new_tokens must be 1..384')
+    ids=encode_prompt(tok,messages,default_system=None,mode='full_context')
+    if len(ids)+cap>2048: raise ValueError('Context overflow: prompt plus token budget exceeds 2048')
+    return ids
+
+@torch.inference_mode()
+def generate(model,tok,messages,cap=384,profile='bf16_native'):
+    validate_input(tok,messages,cap)
+    model.eval()
+    with precision(profile):
+        return accepted_generate(model,tok,messages,cap,autocast_enabled=profile=='bf16_native')
+
+def main():
+    p=argparse.ArgumentParser(description=__doc__)
+    p.add_argument('--model-directory',required=True,type=Path)
+    g=p.add_mutually_exclusive_group(required=True)
+    g.add_argument('--messages-json',type=Path);g.add_argument('--prompt')
+    p.add_argument('--profile',choices=['bf16_native','fp32_math'],default='bf16_native')
+    p.add_argument('--max-new-tokens',type=int,default=384)
+    a=p.parse_args()
+    messages=json.loads(a.messages_json.read_text()) if a.messages_json else [{'role':'user','content':a.prompt}]
+    tok=load_chat_tokenizer(str(a.model_directory/'tokenizer.json'))
+    validate_input(tok,messages,a.max_new_tokens)
+    model,tok=load_bundle(a.model_directory)
+    model=model.to('cuda').eval()
+    print(json.dumps(generate(model,tok,messages,a.max_new_tokens,a.profile),ensure_ascii=False))
+if __name__=='__main__': main()

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4396efb7a52b047e7fdf513e46d1b401dfc70582d3aca1f9cb5a07e97d426ef1
+size 498562192

requirements-inference-tested.txt

@@ -0,0 +1,5 @@
+# Python 3.10.12; CUDA 12.6 PyTorch build
+torch==2.11.0+cu126
+numpy==2.2.6
+tokenizers==0.22.2
+safetensors==0.8.0

special_tokens_map.json

@@ -0,0 +1,12 @@
+{
+  "format": "descriptive native metadata; no HF AutoModel claim",
+  "tokens": {
+    "[PAD]": 0,
+    "[UNK]": 1,
+    "[BOS]": 2,
+    "[EOS]": 3,
+    "<|system|>": 4,
+    "<|user|>": 5,
+    "<|assistant|>": 6
+  }
+}

src/accepted_generate.py

@@ -0,0 +1,18 @@
+import torch
+from src.chat_template import encode_prompt
+
+@torch.inference_mode()
+def generate(model,tok,messages,cap,autocast_enabled=True):
+    ids=encode_prompt(tok,messages,default_system=None,mode='full_context')
+    assert len(ids)+cap<=2048
+    with torch.autocast('cuda',dtype=torch.bfloat16,enabled=autocast_enabled):
+        gen=model.generate(torch.tensor([ids],device='cuda'),max_new_tokens=cap,temperature=0,top_k=0,top_p=1,eos_id=3)
+    new=gen[0,len(ids):].tolist();eos=bool(new and new[-1]==3);before=new[:-1] if eos else new
+    text=tok.decode(before,skip_special_tokens=False)
+    grams=[tuple(before[i:i+4]) for i in range(max(0,len(before)-3))]
+    return {'prompt_token_ids':ids,'generated_token_ids':new,'output_text_for_scoring':text,
+            'output_text_raw_including_terminal_eos':tok.decode(new,skip_special_tokens=False),
+            'stop_reason':'eos' if eos else 'max_new_tokens','generated_tokens_including_eos':len(new),
+            'empty_output':not text,'repeated_4gram_fraction':1-len(set(grams))/len(grams) if grams else 0.,
+            'unexpected_control_token_count':sum(v in {0,2,4,5,6} for v in before),
+            'answer_cleanup':False,'constrained_decoding':False,'max_new_tokens':cap}

src/chat_template.py

@@ -0,0 +1,507 @@
+"""Single source of truth for the chat format, shared by SFT / distill / DPO /
+GRPO training AND their sampling code (previously seven duplicated copies).
+
+Token-level template — role boundaries are special tokens, not plain text:
+
+    [BOS] <|system|> {system} <|user|> {q1} <|assistant|> {a1} [EOS] <|user|> {q2} ...
+
+Design rules:
+- Role tokens delimit turns. BPE can never merge across a special token, so a
+  conversation encodes to the same ids whether it is built turn-by-turn during
+  training or as a generation prompt at inference (the old plain-text
+  "User: ...\\n\\n" template drifted at every segment boundary).
+- [EOS] appears ONLY after assistant turns: it means "assistant finished,
+  stop generating" — the same stop semantics as document ends in pretraining.
+  System/user turns need no terminator; the next role token is the boundary.
+- The supervised span is exactly each assistant turn's content tokens plus its
+  trailing [EOS] (so the model is explicitly taught to stop).
+- Content is encoded with `tokenizer.encode_special_tokens = True`
+  (see `load_chat_tokenizer`), so literal "[EOS]" / "<|user|>" strings inside
+  user or corpus text are tokenized as plain text and can never inject real
+  control tokens.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from tokenizers import Tokenizer
+
+from src.special_tokens import (
+    ASSISTANT_ID,
+    BOS_ID,
+    EOS_ID,
+    PAD_ID,
+    SYSTEM_ID,
+    USER_ID,
+    assert_tokenizer_contract,
+)
+
+DEFAULT_SYSTEM = "You are a helpful assistant."
+
+IGNORE_INDEX = -100
+
+
+# -------------------------
+# Tokenizer loading
+# -------------------------
+def configure_chat_tokenizer(tok: Tokenizer) -> Tokenizer:
+    """Make `tok.encode` treat special-token strings in raw text as plain text.
+
+    All special tokens in this pipeline are inserted by ID by the code below,
+    never parsed out of content — this closes the prompt-injection hole where a
+    document containing the literal string "[EOS]" would encode to the real
+    EOS id.
+    """
+    tok.encode_special_tokens = True
+    return tok
+
+
+def load_chat_tokenizer(tokenizer_path: str) -> Tokenizer:
+    """Load tokenizer.json, assert the hardcoded special-token IDs, and disable
+    special-token matching in raw text. Every chat-stage script should load its
+    tokenizer through this."""
+    assert_tokenizer_contract(tokenizer_path)
+    return configure_chat_tokenizer(Tokenizer.from_file(tokenizer_path))
+
+
+# -------------------------
+# Text cleaning
+# -------------------------
+def norm_newlines(s: str) -> str:
+    return (s or "").replace("\r\n", "\n").replace("\r", "\n")
+
+
+def clean_text(s: str) -> str:
+    # for system/user text: strip leading/trailing whitespace
+    return norm_newlines(s).strip()
+
+
+def clean_text_assistant(s: str) -> str:
+    # IMPORTANT: do not strip assistant text (keeps code indentation / markdown
+    # formatting); only trailing whitespace goes (EOS follows immediately).
+    return norm_newlines(s).rstrip()
+
+
+def _normalized_messages(
+    messages: list[dict[str, str]],
+    default_system: str | None,
+    *,
+    expected_end: str | None,
+) -> list[dict[str, str]]:
+    """Clean and validate the canonical chat state machine.
+
+    Normally a chat has an initial system turn, followed by alternating user
+    and assistant turns. Explicit default_system=None preserves source content
+    verbatim and permits a missing system turn without injecting one. ``expected_end`` is ``"user"`` for
+    a generation prompt and ``"assistant"`` for a complete training sample.
+    No malformed turn is silently skipped.
+    """
+    out: list[dict[str, str]] = []
+    for index, message in enumerate(messages or []):
+        if not isinstance(message, dict):
+            raise ValueError(f"message {index} must be an object")
+        role_value = message.get("role")
+        role = role_value.strip().lower() if isinstance(role_value, str) else ""
+        if role not in ("system", "user", "assistant"):
+            raise ValueError(f"message {index} has invalid role {role_value!r}")
+        raw = message.get("content")
+        if not isinstance(raw, str):
+            raise ValueError(f"message {index} content must be a string")
+        text = raw if default_system is None else (
+            clean_text_assistant(raw) if role == "assistant" else clean_text(raw)
+        )
+        if not text.strip():
+            if index == 0 and role == "system" and default_system is not None:
+                fallback = clean_text(default_system)
+                if not fallback:
+                    raise ValueError(
+                        "chat requires a non-empty initial system turn or non-empty default_system"
+                    )
+                text = fallback
+            else:
+                raise ValueError(f"message {index} ({role}) content must be non-empty")
+        out.append({"role": role, "content": text})
+
+    if (not out or out[0]["role"] != "system") and default_system is not None:
+        fallback = clean_text(default_system)
+        if not fallback:
+            raise ValueError(
+                "chat requires a non-empty initial system turn or non-empty default_system"
+            )
+        out.insert(0, {"role": "system", "content": fallback})
+
+    has_system = bool(out) and out[0]["role"] == "system"
+    if not out or (has_system and len(out) == 1):
+        raise ValueError("chat requires at least one non-empty user turn")
+    for index, message in enumerate(out):
+        turn_index = index - int(has_system)
+        expected = "system" if has_system and index == 0 else (
+            "user" if turn_index % 2 == 0 else "assistant"
+        )
+        if message["role"] != expected:
+            raise ValueError(
+                "invalid chat role order: initial system must be followed by "
+                f"alternating user/assistant turns (index {index}: expected "
+                f"{expected!r}, got {message['role']!r})"
+            )
+    if expected_end is not None and out[-1]["role"] != expected_end:
+        raise ValueError(
+            f"chat must end with a non-empty {expected_end} turn; got {out[-1]['role']!r}"
+        )
+    return out
+
+
+def prepare_prompt_messages(
+    messages: list[dict[str, str]],
+    default_system: str | None = DEFAULT_SYSTEM,
+) -> list[dict[str, str]]:
+    """Explicitly turn a valid conversation/example into a USER-ending prompt.
+
+    Callers sampling from a complete SFT example must opt in to removing its
+    final assistant answer. ``encode_prompt`` itself never drops turns.
+    """
+    normalized = _normalized_messages(messages, default_system, expected_end=None)
+    if normalized[-1]["role"] == "assistant":
+        normalized = normalized[:-1]
+    if normalized[-1]["role"] != "user":
+        raise ValueError("generation prompt context must end with a user turn")
+    return normalized
+
+
+# -------------------------
+# Core encoding
+# -------------------------
+_ROLE_TOKEN_ID = {"system": SYSTEM_ID, "user": USER_ID, "assistant": ASSISTANT_ID}
+
+
+def _encode_normalized_chat(
+    tok: Tokenizer, messages: list[dict[str, str]]
+) -> tuple[list[int], list[int]]:
+    ids: list[int] = [BOS_ID]
+    labels: list[int] = [IGNORE_INDEX]
+    for index, message in enumerate(messages):
+        role = message["role"]
+        content_ids = tok.encode(message["content"]).ids
+        if not content_ids:
+            raise ValueError(f"message {index} ({role}) must encode to at least one token")
+        ids.append(_ROLE_TOKEN_ID[role])
+        labels.append(IGNORE_INDEX)
+        if role == "assistant":
+            ids.extend(content_ids)
+            labels.extend(content_ids)
+            ids.append(EOS_ID)
+            labels.append(EOS_ID)
+        else:
+            ids.extend(content_ids)
+            labels.extend([IGNORE_INDEX] * len(content_ids))
+    return ids, labels
+
+
+def encode_chat(
+    tok: Tokenizer,
+    messages: list[dict[str, str]],
+    default_system: str | None = DEFAULT_SYSTEM,
+) -> tuple[list[int], list[int]]:
+    """Encode a full conversation for training.
+
+    Returns (ids, labels), same length. labels[i] == ids[i] on supervised
+    positions (every assistant turn's content + its trailing EOS) and
+    IGNORE_INDEX everywhere else (BOS, role tokens, system/user content).
+    """
+    msgs = _normalized_messages(messages, default_system, expected_end="assistant")
+    return _encode_normalized_chat(tok, msgs)
+
+
+def encode_prompt(
+    tok: Tokenizer,
+    messages: list[dict[str, str]],
+    default_system: str | None = DEFAULT_SYSTEM,
+    mode: str = "full_context",
+) -> list[int]:
+    """Encode a generation prompt: context ending in the ``<|assistant|>`` cue.
+
+    mode:
+      - "full_context": the entire validated USER-ending context (earlier
+        assistant turns keep their [EOS]).
+      - "last_user": system turn + last user turn only.
+
+    The returned ids start with BOS and end with ASSISTANT_ID, exactly matching
+    the training-time prefix for an assistant turn.
+    """
+    msgs = _normalized_messages(messages, default_system, expected_end="user")
+
+    if mode == "last_user":
+        msgs = [msgs[0], msgs[-1]] if msgs[0]["role"] == "system" else [msgs[-1]]
+    elif mode != "full_context":
+        raise ValueError(f"unknown prompt mode: {mode}")
+
+    ids, _ = _encode_normalized_chat(tok, msgs)
+    ids.append(ASSISTANT_ID)
+    return ids
+
+
+def encode_completion(
+    tok: Tokenizer,
+    messages: list[dict[str, str]],
+    completion: str,
+    default_system: str | None = DEFAULT_SYSTEM,
+) -> tuple[list[int], list[int]]:
+    """Encode (prompt context + one assistant completion) for DPO-style scoring.
+
+    `messages` is the shared context (must end with a user turn); `completion`
+    is a plain assistant string. Returns (ids, labels) where the supervised
+    span is the completion tokens + trailing EOS — logps therefore include the
+    stop decision.
+    """
+    ids = encode_prompt(tok, messages, default_system, mode="full_context")
+    labels = [IGNORE_INDEX] * len(ids)
+    if not isinstance(completion, str) or not completion.strip():
+        raise ValueError("DPO completion must be a non-empty, non-whitespace string")
+    comp_ids = tok.encode(completion if default_system is None else clean_text_assistant(completion)).ids
+    if not comp_ids:
+        raise ValueError("DPO completion must encode to at least one token")
+    ids.extend(comp_ids)
+    labels.extend(comp_ids)
+    ids.append(EOS_ID)
+    labels.append(EOS_ID)
+    return ids, labels
+
+
+def truncate_chat_sequence(
+    ids: list[int],
+    labels: list[int] | None,
+    max_len: int,
+) -> tuple[list[int], list[int] | None]:
+    """Validate and truncate only at complete user-turn boundaries.
+
+    The prefix is BOS, plus the initial SYSTEM turn when present. It is always
+    retained; source-preserving conversations may begin BOS + USER. The remainder is the largest recent suffix that begins at a real
+    ``USER_ID`` marker and runs through the original sequence end. A training
+    sequence must end with an assistant EOS; a prompt must end with the final
+    assistant cue. Literal special-token text cannot create these boundaries
+    because chat tokenizers disable special-token matching for raw content.
+
+    If the system prefix plus the latest complete user-led suffix cannot fit,
+    this function raises instead of cutting content, role markers, assistant
+    completions, or EOS targets.
+    """
+    if max_len <= 0:
+        raise ValueError("max_len must be positive")
+    if labels is not None and len(labels) != len(ids):
+        raise ValueError("ids and labels must have identical lengths")
+    if len(ids) < 3 or ids[0] != BOS_ID or ids[1] not in {SYSTEM_ID, USER_ID}:
+        raise ValueError("chat sequence must start with BOS_ID and a SYSTEM or USER turn")
+
+    forbidden_content_ids = {PAD_ID, BOS_ID, EOS_ID, SYSTEM_ID, USER_ID, ASSISTANT_ID}
+    role_ids = {SYSTEM_ID, USER_ID, ASSISTANT_ID}
+    first_role = 1
+    if ids[1] == SYSTEM_ID:
+        first_role = next((index for index in range(2, len(ids)) if ids[index] in role_ids), len(ids))
+        if first_role == 2:
+            raise ValueError("initial system content must contain at least one token")
+        if any(token_id in forbidden_content_ids for token_id in ids[2:first_role]):
+            raise ValueError("system content contains a structural special-token ID")
+        if first_role == len(ids) or ids[first_role] != USER_ID:
+            raise ValueError("chat sequence must contain a user turn after the initial system turn")
+
+    user_starts: list[int] = []
+    cursor = first_role
+    while cursor < len(ids):
+        if ids[cursor] != USER_ID:
+            raise ValueError("chat role sequence must alternate USER and ASSISTANT")
+        user_starts.append(cursor)
+        user_content_start = cursor + 1
+        assistant_pos = next(
+            (
+                index
+                for index in range(user_content_start, len(ids))
+                if ids[index] in forbidden_content_ids
+            ),
+            len(ids),
+        )
+        if assistant_pos == user_content_start:
+            raise ValueError("user content must contain at least one token")
+        if assistant_pos == len(ids) or ids[assistant_pos] != ASSISTANT_ID:
+            raise ValueError("each user turn must be followed by an assistant turn")
+
+        assistant_content_start = assistant_pos + 1
+        if assistant_content_start == len(ids):
+            if labels is not None:
+                raise ValueError("training chat must end with assistant content and EOS")
+            cursor = len(ids)
+            break
+
+        eos_pos = next(
+            (
+                index
+                for index in range(assistant_content_start, len(ids))
+                if ids[index] in forbidden_content_ids
+            ),
+            len(ids),
+        )
+        if eos_pos == assistant_content_start:
+            raise ValueError("assistant content must contain at least one token")
+        if eos_pos == len(ids) or ids[eos_pos] != EOS_ID:
+            raise ValueError("each assistant completion must end with EOS")
+        cursor = eos_pos + 1
+        if cursor == len(ids):
+            if labels is None:
+                raise ValueError("generation prompt must end with an assistant cue")
+            break
+
+    if labels is not None:
+        if ids[-1] != EOS_ID:
+            raise ValueError("training chat must end with a supervised assistant EOS")
+        if labels[-1] != EOS_ID:
+            raise ValueError("final assistant EOS must be supervised")
+
+    prefix_end = first_role
+    chosen_start = next(
+        (start for start in user_starts if prefix_end + (len(ids) - start) <= max_len),
+        None,
+    )
+    if chosen_start is None:
+        minimum = prefix_end + (len(ids) - user_starts[-1])
+        raise ValueError(
+            "chat sequence does not fit without cutting the system prefix or latest "
+            f"user-led suffix (requires at least {minimum} tokens, max_len={max_len})"
+        )
+
+    if chosen_start == prefix_end:
+        return list(ids), list(labels) if labels is not None else None
+
+    kept_ids = ids[:prefix_end] + ids[chosen_start:]
+    if labels is None:
+        return kept_ids, None
+    return kept_ids, labels[:prefix_end] + labels[chosen_start:]
+
+
+def pad_or_truncate(
+    ids: list[int],
+    labels: list[int],
+    seq_len: int,
+    pad_id: int = PAD_ID,
+) -> tuple[list[int], list[int]]:
+    """Structure-aware fixed-length shaping plus right padding."""
+    ids, maybe_labels = truncate_chat_sequence(ids, labels, seq_len)
+    assert maybe_labels is not None
+    labels = maybe_labels
+    pad_n = seq_len - len(ids)
+    return ids + [pad_id] * pad_n, labels + [IGNORE_INDEX] * pad_n
+
+
+def count_chat_tokens(
+    tok: Tokenizer,
+    messages: list[dict[str, str]],
+    default_system: str | None = DEFAULT_SYSTEM,
+) -> int:
+    """Exact token count of the training encoding (for mix/token budgeting)."""
+    ids, _ = encode_chat(tok, messages, default_system)
+    return len(ids)
+
+
+def extract_last_user_and_ref(messages: list[dict[str, str]]) -> tuple[str, str]:
+    """Return (last_user_text, last_assistant_text_if_any) — for sample logs."""
+    last_user = ""
+    ref = ""
+    for m in reversed(messages or []):
+        if (m.get("role") or "").strip().lower() == "user":
+            last_user = clean_text(m.get("content", ""))
+            break
+    for m in reversed(messages or []):
+        if (m.get("role") or "").strip().lower() == "assistant":
+            ref = clean_text_assistant(m.get("content", ""))
+            break
+    return last_user, ref
+
+
+def decode_completion(tok: Tokenizer, ids: list[int]) -> str:
+    """Decode generated completion ids (the tokens AFTER the prompt), dropping
+    a trailing EOS if present. Replaces the old fragile rfind('Assistant: ')."""
+    if ids and ids[-1] == EOS_ID:
+        ids = ids[:-1]
+    return tok.decode(ids).strip() if ids else ""
+
+
+# -------------------------
+# Refusal detection (shared by SFT/distill example weighting)
+# -------------------------
+def is_refusal_text(text: str, patterns: list[str]) -> bool:
+    """If assistant content contains any refusal-ish substring, treat as refusal."""
+    t = (text or "").strip().lower()
+    if not t:
+        return False
+    for p in patterns:
+        p2 = p.strip().lower()
+        if p2 and p2 in t:
+            return True
+    return False
+
+
+def compute_example_weight_from_messages(
+    messages: list[dict[str, str]],
+    refusal_downweight: float,
+    refusal_patterns: list[str],
+    refusal_mode: str,
+) -> float:
+    """Scalar loss weight for a training example; downweights refusal-looking
+    assistant turns (refusal_mode="contains_any")."""
+    if refusal_downweight >= 1.0:
+        return 1.0
+    if refusal_downweight <= 0.0:
+        return 0.0
+    if refusal_mode != "contains_any":
+        raise ValueError(f"unknown refusal_mode: {refusal_mode}")
+    for m in messages or []:
+        if (m.get("role") or "").strip().lower() == "assistant":
+            if is_refusal_text(m.get("content", ""), refusal_patterns):
+                return refusal_downweight
+    return 1.0
+
+
+def build_example(
+    ex: dict[str, Any],
+    tok: Tokenizer,
+    seq_len: int,
+    default_system: str | None,
+    refusal_downweight: float,
+    refusal_patterns: list[str],
+    refusal_mode: str,
+    pad_id: int = PAD_ID,
+):
+    """One SFT/distill training example -> (input_ids, labels, example_weight).
+
+    Tensors are torch.long of length seq_len; labels use IGNORE_INDEX outside
+    the supervised assistant spans. Honors meta.bucket safety exemption and
+    meta.weight multipliers exactly as before.
+    """
+    import torch
+
+    messages = ex.get("messages") or []
+    if not messages:
+        raise ValueError("missing messages")
+
+    meta = ex.get("meta") or {}
+    bucket = str(meta.get("bucket", "")).strip()
+    # Do NOT downweight refusals inside the safety bucket (otherwise safety
+    # examples get muted).
+    refusal_dw_eff = 1.0 if bucket in ("D_safety", "D") else refusal_downweight
+    ex_weight = compute_example_weight_from_messages(
+        messages, refusal_dw_eff, refusal_patterns, refusal_mode
+    )
+    w0 = meta.get("weight", None) if isinstance(meta, dict) else None
+    if isinstance(w0, (int, float)):
+        ex_weight *= float(w0)
+
+    ids, labels = encode_chat(tok, messages, default_system)
+    if default_system is None and len(ids) > seq_len:
+        raise ValueError("source-preserving SFT rejects overlength conversations")
+    ids, labels = pad_or_truncate(ids, labels, seq_len, pad_id)
+    if all(label == IGNORE_INDEX for label in labels):
+        raise ValueError("SFT example requires a retained non-empty assistant turn")
+    return (
+        torch.tensor(ids, dtype=torch.long),
+        torch.tensor(labels, dtype=torch.long),
+        float(ex_weight),
+    )

src/model.py

@@ -0,0 +1,470 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+@dataclass
+class GPTConfig:
+    vocab_size: int = 32000
+    n_layers: int = 30
+    d_model: int = 576
+    n_heads: int = 9
+    n_kv_heads: int = 3  # GQA KV heads; == n_heads is plain MHA (pre-GQA checkpoints)
+    d_ff: int = 1536  # SwiGLU: ~2.67x d_model (MobileLLM/SmolLM2-135M deep-thin shape)
+    max_seq_len: int = 2048
+    dropout: float = 0.0
+    tie_embeddings: bool = True
+
+    # RoPE (rotary positional embedding)
+    rope_theta: float = 10000.0
+    rope_pct: float = 1.0  # fraction of head_dim to rotate (1.0 = full head_dim)
+
+
+CANONICAL_DENSE_PARAMETER_COUNT = 124_635_456
+_CANONICAL_PARAMETERIZATION = {
+    "vocab_size": 32_000,
+    "n_layers": 30,
+    "d_model": 576,
+    "n_heads": 9,
+    "n_kv_heads": 3,
+    "d_ff": 1_536,
+    "tie_embeddings": True,
+}
+
+
+def expected_gpt_parameter_count(cfg: GPTConfig) -> int:
+    """Derive the unique parameter count for the dense bias-free GPT."""
+    integer_fields = {
+        "vocab_size": cfg.vocab_size,
+        "n_layers": cfg.n_layers,
+        "d_model": cfg.d_model,
+        "n_heads": cfg.n_heads,
+        "n_kv_heads": cfg.n_kv_heads,
+        "d_ff": cfg.d_ff,
+    }
+    for name, value in integer_fields.items():
+        if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+            raise ValueError(f"GPTConfig.{name} must be a positive integer")
+    if cfg.d_model % cfg.n_heads:
+        raise ValueError("GPTConfig.d_model must be divisible by n_heads")
+    if cfg.n_heads % cfg.n_kv_heads:
+        raise ValueError("GPTConfig.n_heads must be divisible by n_kv_heads")
+
+    head_dim = cfg.d_model // cfg.n_heads
+    kv_dim = cfg.n_kv_heads * head_dim
+    token_matrices = 1 if cfg.tie_embeddings else 2
+    embeddings = token_matrices * cfg.vocab_size * cfg.d_model
+    # q + output projections are d_model x d_model; k and v are d_model x kv_dim (GQA)
+    attention = 2 * cfg.d_model * cfg.d_model + 2 * cfg.d_model * kv_dim
+    swiglu = 3 * cfg.d_model * cfg.d_ff
+    block_norms = 2 * cfg.d_model
+    final_norm = cfg.d_model
+    return int(embeddings + cfg.n_layers * (attention + swiglu + block_norms) + final_norm)
+
+
+def audit_gpt_parameter_count(model: nn.Module, cfg: GPTConfig) -> dict[str, int | bool | str]:
+    """Fail fast on implementation/config drift and return manifest metadata."""
+    expected = expected_gpt_parameter_count(cfg)
+    actual = int(sum(parameter.numel() for parameter in model.parameters()))
+    trainable = int(
+        sum(parameter.numel() for parameter in model.parameters() if parameter.requires_grad)
+    )
+    if actual != expected:
+        raise RuntimeError(
+            "GPT parameter count disagrees with the architecture-derived count: "
+            f"actual={actual:,}, expected={expected:,}"
+        )
+
+    canonical = all(
+        getattr(cfg, field) == expected_value
+        for field, expected_value in _CANONICAL_PARAMETERIZATION.items()
+    )
+    if canonical and actual != CANONICAL_DENSE_PARAMETER_COUNT:
+        raise RuntimeError(
+            "canonical PetitGPT parameter count mismatch: "
+            f"actual={actual:,}, expected={CANONICAL_DENSE_PARAMETER_COUNT:,}"
+        )
+
+    return {
+        "status": "passed",
+        "counting_method": "unique_parameter_objects_excluding_buffers",
+        "actual_total": actual,
+        "actual_trainable": trainable,
+        "derived_expected_total": expected,
+        "canonical_parameterization": canonical,
+        "canonical_expected_total": CANONICAL_DENSE_PARAMETER_COUNT,
+        "canonical_match": canonical and actual == CANONICAL_DENSE_PARAMETER_COUNT,
+    }
+
+
+def gpt_config_from_checkpoint_dict(cfg_dict: dict) -> GPTConfig:
+    """Rebuild a GPTConfig from a checkpoint's serialized config dict.
+
+    Pre-GQA checkpoints carry no n_kv_heads; absence means plain MHA
+    (n_kv_heads == n_heads), whose fused-QKV weight layout is unchanged.
+    """
+    cfg_dict = dict(cfg_dict)
+    cfg_dict.setdefault("n_kv_heads", cfg_dict["n_heads"])
+    return GPTConfig(**cfg_dict)
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        # x: [B, T, C]
+        rms = x.pow(2).mean(dim=-1, keepdim=True).add(self.eps).rsqrt()
+        return x * rms * self.weight
+
+
+def _rotate_half(x: torch.Tensor) -> torch.Tensor:
+    # x: [..., D]. Half-split layout (Llama/GPT-NeoX): pairs are (i, i+D/2),
+    # matching the `cat([freqs, freqs])` cos/sin cache below.
+    half = x.shape[-1] // 2
+    x1 = x[..., :half]
+    x2 = x[..., half:]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+class RotaryEmbedding(nn.Module):
+    """Precomputes RoPE cos/sin caches up to max_seq_len."""
+
+    def __init__(self, head_dim: int, max_seq_len: int, theta: float = 10000.0, pct: float = 1.0):
+        super().__init__()
+        if head_dim % 2 != 0:
+            raise ValueError(f"RoPE requires even head_dim, got {head_dim}")
+        self.head_dim = int(head_dim)
+        self.max_seq_len = int(max_seq_len)
+        self.theta = float(theta)
+        self.pct = float(pct)
+
+        rope_dim = int(self.head_dim * self.pct)
+        rope_dim = rope_dim - (rope_dim % 2)
+        rope_dim = max(0, min(rope_dim, self.head_dim))
+        self.rope_dim = rope_dim
+
+        if self.rope_dim > 0:
+            inv_freq = 1.0 / (
+                self.theta ** (torch.arange(0, self.rope_dim, 2).float() / self.rope_dim)
+            )
+            t = torch.arange(self.max_seq_len, dtype=torch.float32)
+            freqs = torch.outer(t, inv_freq)  # [T, rope_dim/2]
+            emb = torch.cat([freqs, freqs], dim=-1)  # [T, rope_dim]
+            cos = emb.cos()
+            sin = emb.sin()
+        else:
+            cos = torch.empty(self.max_seq_len, 0, dtype=torch.float32)
+            sin = torch.empty(self.max_seq_len, 0, dtype=torch.float32)
+
+        self.register_buffer("cos_cached", cos, persistent=False)
+        self.register_buffer("sin_cached", sin, persistent=False)
+
+    def forward(
+        self, q: torch.Tensor, k: torch.Tensor, seq_len: int, offset: int = 0
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """Apply RoPE to q,k. q,k: [B, nH, T, Hd].
+
+        `offset` is the absolute position of the first token in q,k — nonzero
+        during KV-cached incremental decoding, where the new tokens sit at
+        positions [offset, offset+seq_len).
+        """
+        end = offset + seq_len
+        if end > self.max_seq_len:
+            raise ValueError(
+                f"position {end} exceeds max_seq_len={self.max_seq_len} for RoPE cache"
+            )
+        if self.rope_dim == 0:
+            return q, k
+
+        cos = self.cos_cached[offset:end].to(dtype=q.dtype, device=q.device)  # [T, rope_dim]
+        sin = self.sin_cached[offset:end].to(dtype=q.dtype, device=q.device)  # [T, rope_dim]
+        cos = cos.unsqueeze(0).unsqueeze(0)  # [1,1,T,rope_dim]
+        sin = sin.unsqueeze(0).unsqueeze(0)
+
+        q1, q2 = q[..., : self.rope_dim], q[..., self.rope_dim :]
+        k1, k2 = k[..., : self.rope_dim], k[..., self.rope_dim :]
+
+        q1 = q1 * cos + _rotate_half(q1) * sin
+        k1 = k1 * cos + _rotate_half(k1) * sin
+
+        q = torch.cat([q1, q2], dim=-1)
+        k = torch.cat([k1, k2], dim=-1)
+        return q, k
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, cfg: GPTConfig):
+        super().__init__()
+        self.w1 = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.w3 = nn.Linear(cfg.d_model, cfg.d_ff, bias=False)
+        self.w2 = nn.Linear(cfg.d_ff, cfg.d_model, bias=False)
+        self.drop = nn.Dropout(cfg.dropout)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = F.silu(self.w1(x)) * self.w3(x)
+        x = self.w2(x)
+        return self.drop(x)
+
+
+class CausalSelfAttention(nn.Module):
+    def __init__(self, cfg: GPTConfig):
+        super().__init__()
+        assert cfg.d_model % cfg.n_heads == 0
+        assert cfg.n_heads % cfg.n_kv_heads == 0
+        self.cfg = cfg
+        self.head_dim = cfg.d_model // cfg.n_heads
+        self.kv_dim = cfg.n_kv_heads * self.head_dim
+
+        # QKV fused: one matmul instead of three. K/V carry n_kv_heads (GQA);
+        # n_kv_heads == n_heads is plain MHA with the historical 3*d_model layout.
+        self.qkv = nn.Linear(cfg.d_model, cfg.d_model + 2 * self.kv_dim, bias=False)
+        # residual branch output projection
+        self.proj = nn.Linear(cfg.d_model, cfg.d_model, bias=False)
+        self.drop = nn.Dropout(cfg.dropout)
+
+        self.rope = RotaryEmbedding(
+            head_dim=self.head_dim,
+            max_seq_len=cfg.max_seq_len,
+            theta=cfg.rope_theta,
+            pct=cfg.rope_pct,
+        )
+
+    @staticmethod
+    def _incremental_mask(T: int, past_len: int, device: torch.device) -> torch.Tensor:
+        """Bottom-right causal mask [T, past_len+T] (True = attend) for decoding
+        T new queries against past_len cached keys plus the new keys."""
+        q_pos = past_len + torch.arange(T, device=device)
+        k_pos = torch.arange(past_len + T, device=device)
+        return k_pos[None, :] <= q_pos[:, None]
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        past_kv: tuple[torch.Tensor, torch.Tensor] | None = None,
+        use_cache: bool = False,
+    ):
+        """x: [B, T, C]. With no cache this is byte-identical to a plain causal
+        forward and returns the output tensor. With `use_cache` (or a supplied
+        `past_kv`) it also returns the updated (k, v) for this layer."""
+        B, T, C = x.shape
+        past_len = 0 if past_kv is None else past_kv[0].size(2)
+        if past_len + T > self.cfg.max_seq_len:
+            raise ValueError(
+                f"cache length {past_len + T} exceeds max_seq_len={self.cfg.max_seq_len}"
+            )
+
+        qkv = self.qkv(x)  # [B, T, C + 2*kv_dim]
+        q, k, v = qkv.split([C, self.kv_dim, self.kv_dim], dim=-1)
+
+        q = q.view(B, T, self.cfg.n_heads, self.head_dim).transpose(1, 2)  # [B,nH,T,Hd]
+        k = k.view(B, T, self.cfg.n_kv_heads, self.head_dim).transpose(1, 2)  # [B,nKV,T,Hd]
+        v = v.view(B, T, self.cfg.n_kv_heads, self.head_dim).transpose(1, 2)
+
+        # RoPE rotates only the new tokens, at their absolute positions.
+        q, k = self.rope(q, k, seq_len=T, offset=past_len)
+
+        # Prepend cached keys/values (already rotated when they were new). The
+        # cache stays un-expanded at n_kv_heads so its memory reflects GQA.
+        if past_kv is not None:
+            k = torch.cat([past_kv[0], k], dim=2)
+            v = torch.cat([past_kv[1], v], dim=2)
+        present = (k, v) if use_cache else None
+
+        # Expand grouped KV heads to the full head count for attention. KV head g
+        # serves query heads [g*rep, (g+1)*rep) — repeat_interleave matches SDPA's
+        # enable_gqa grouping (torch >= 2.5), which can replace this someday.
+        if self.cfg.n_kv_heads != self.cfg.n_heads:
+            rep = self.cfg.n_heads // self.cfg.n_kv_heads
+            k = k.repeat_interleave(rep, dim=1)
+            v = v.repeat_interleave(rep, dim=1)
+
+        dropout_p = float(self.cfg.dropout) if (self.training and self.cfg.dropout > 0) else 0.0
+
+        if q.device.type == "cuda":
+            if past_len == 0:
+                y = F.scaled_dot_product_attention(
+                    q, k, v, attn_mask=None, dropout_p=dropout_p, is_causal=True
+                )
+            else:
+                y = F.scaled_dot_product_attention(
+                    q,
+                    k,
+                    v,
+                    attn_mask=self._incremental_mask(T, past_len, q.device),
+                    dropout_p=dropout_p,
+                )
+        else:
+            scale = 1.0 / math.sqrt(self.head_dim)
+            att = torch.matmul(q * scale, k.transpose(-2, -1))  # [B,nH,T,past_len+T]
+            if past_len == 0:
+                mask = torch.triu(torch.ones((T, T), device=q.device, dtype=torch.bool), diagonal=1)
+                att = att.masked_fill(mask, float("-inf"))
+            else:
+                allow = self._incremental_mask(T, past_len, q.device)  # [T, past_len+T]
+                att = att.masked_fill(~allow, float("-inf"))
+            att = F.softmax(att, dim=-1)
+            if dropout_p > 0.0:
+                att = F.dropout(att, p=dropout_p)
+            y = torch.matmul(att, v)
+
+        y = y.transpose(1, 2).contiguous().view(B, T, C)
+        y = self.drop(self.proj(y))
+        if use_cache:
+            return y, present
+        return y
+
+
+class Block(nn.Module):
+    def __init__(self, cfg: GPTConfig):
+        super().__init__()
+        self.norm1 = RMSNorm(cfg.d_model)
+        self.attn = CausalSelfAttention(cfg)
+        self.norm2 = RMSNorm(cfg.d_model)
+        self.mlp = SwiGLU(cfg)
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        past_kv: tuple[torch.Tensor, torch.Tensor] | None = None,
+        use_cache: bool = False,
+    ):
+        if use_cache or past_kv is not None:
+            attn_out, present = self.attn(self.norm1(x), past_kv=past_kv, use_cache=True)
+            x = x + attn_out
+            x = x + self.mlp(self.norm2(x))
+            return x, present
+        x = x + self.attn(self.norm1(x))
+        x = x + self.mlp(self.norm2(x))
+        return x
+
+
+class GPT(nn.Module):
+    def __init__(self, cfg: GPTConfig):
+        super().__init__()
+        self.cfg = cfg
+
+        self.tok_emb = nn.Embedding(cfg.vocab_size, cfg.d_model)
+        self.drop = nn.Dropout(cfg.dropout)
+
+        self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg.n_layers)])
+        self.norm_f = RMSNorm(cfg.d_model)
+
+        self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)
+        if cfg.tie_embeddings:
+            self.lm_head.weight = self.tok_emb.weight
+
+        # Base init everywhere...
+        self.apply(self._init_weights)
+        # ...then scale init ONLY on residual-branch output projections: attn.proj and mlp.w2
+        self._init_residual_projections()
+
+    def _init_weights(self, m: nn.Module):
+        if isinstance(m, (nn.Linear, nn.Embedding)):
+            torch.nn.init.normal_(m.weight, mean=0.0, std=0.02)
+
+    def _init_residual_projections(self):
+        std = 0.02 / math.sqrt(2.0 * float(self.cfg.n_layers))
+        for blk in self.blocks:
+            torch.nn.init.normal_(blk.attn.proj.weight, mean=0.0, std=std)
+            torch.nn.init.normal_(blk.mlp.w2.weight, mean=0.0, std=std)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        past_kv: list[tuple[torch.Tensor, torch.Tensor]] | None = None,
+        use_cache: bool = False,
+    ):
+        """Default call `model(input_ids)` returns logits [B, T, V] — unchanged.
+
+        For incremental decoding, pass `use_cache=True` to also get a per-layer
+        list of (k, v) tensors, and feed it back as `past_kv` with only the new
+        token(s) on the next call. See `generate`.
+        """
+        B, T = input_ids.shape
+        past_len = 0 if past_kv is None else past_kv[0][0].size(2)
+        if past_len + T > self.cfg.max_seq_len:
+            raise ValueError(
+                f"T={T} with cache={past_len} exceeds max_seq_len={self.cfg.max_seq_len}"
+            )
+        if T < 1:
+            raise ValueError("Empty sequence")
+        caching = use_cache or (past_kv is not None)
+
+        x = self.tok_emb(input_ids)
+        x = self.drop(x)
+        presents: list[tuple[torch.Tensor, torch.Tensor]] = []
+        for i, blk in enumerate(self.blocks):
+            layer_past = past_kv[i] if past_kv is not None else None
+            if caching:
+                x, present = blk(x, past_kv=layer_past, use_cache=True)
+                presents.append(present)
+            else:
+                x = blk(x)
+        x = self.norm_f(x)
+        logits = self.lm_head(x)
+        if caching:
+            return logits, presents
+        return logits
+
+    @staticmethod
+    def _sample_token(
+        logits: torch.Tensor, temperature: float, top_k: int, top_p: float
+    ) -> torch.Tensor:
+        """logits: [B, V] -> next token [B, 1]. temperature<=0 is greedy."""
+        if temperature <= 0:
+            return logits.argmax(dim=-1, keepdim=True)
+        logits = logits / temperature
+        if top_k and top_k > 0:
+            k = min(int(top_k), logits.size(-1))
+            thresh = torch.topk(logits, k, dim=-1).values[:, -1, None]
+            logits = logits.masked_fill(logits < thresh, float("-inf"))
+        if top_p and top_p < 1.0:
+            sorted_logits, sorted_idx = torch.sort(logits, descending=True, dim=-1)
+            cum = torch.softmax(sorted_logits, dim=-1).cumsum(dim=-1)
+            drop_sorted = cum > top_p
+            drop_sorted[..., 0] = False
+            drop = torch.zeros_like(drop_sorted).scatter(-1, sorted_idx, drop_sorted)
+            logits = logits.masked_fill(drop, float("-inf"))
+        probs = torch.softmax(logits, dim=-1)
+        return torch.multinomial(probs, num_samples=1)
+
+    @torch.no_grad()
+    def generate(
+        self,
+        input_ids: torch.Tensor,
+        max_new_tokens: int,
+        *,
+        temperature: float = 1.0,
+        top_k: int = 0,
+        top_p: float = 1.0,
+        eos_id: int | None = None,
+    ) -> torch.Tensor:
+        """KV-cached incremental decoding. input_ids: [B, T] -> [B, T + n].
+
+        Prefills the prompt once, then feeds one new token per step against the
+        cache (O(T) forwards of length 1) instead of re-running the full growing
+        sequence each step. Stops early if all rows emit `eos_id`.
+        """
+        was_training = self.training
+        self.eval()
+        logits, past = self.forward(input_ids, use_cache=True)
+        out = input_ids
+        for _ in range(int(max_new_tokens)):
+            next_tok = self._sample_token(logits[:, -1, :], temperature, top_k, top_p)
+            out = torch.cat([out, next_tok], dim=1)
+            if eos_id is not None and bool((next_tok.squeeze(1) == eos_id).all()):
+                break
+            if out.size(1) >= self.cfg.max_seq_len:
+                break
+            logits, past = self.forward(next_tok, past_kv=past, use_cache=True)
+        if was_training:
+            self.train()
+        return out

src/special_tokens.py

@@ -0,0 +1,138 @@
+"""Canonical special-token IDs for the whole pipeline.
+
+Every training/data script hardcodes these IDs (see CLAUDE.md):
+
+    [PAD]=0  [UNK]=1  [BOS]=2  [EOS]=3
+    <|system|>=4  <|user|>=5  <|assistant|>=6
+
+The three role tokens delimit chat turns at the *token* level (see
+src/chat_template.py) — BPE can never merge across a special token, so
+training-time and inference-time encodings agree by construction.
+
+`assert_special_token_ids` turns the silent assumption into a loud startup
+check: if the tokenizer at --tokenizer_path is ever retrained and the IDs move,
+scripts fail immediately instead of training with a misaligned loss mask or a
+broken EOS stop condition.
+"""
+
+from __future__ import annotations
+
+import json
+from os import PathLike
+
+PAD_ID = 0
+UNK_ID = 1
+BOS_ID = 2
+EOS_ID = 3
+SYSTEM_ID = 4
+USER_ID = 5
+ASSISTANT_ID = 6
+CANONICAL_VOCAB_SIZE = 32_000
+
+PAD_TOKEN = "[PAD]"
+UNK_TOKEN = "[UNK]"
+BOS_TOKEN = "[BOS]"
+EOS_TOKEN = "[EOS]"
+SYSTEM_TOKEN = "<|system|>"
+USER_TOKEN = "<|user|>"
+ASSISTANT_TOKEN = "<|assistant|>"
+
+SPECIAL_TOKEN_IDS: dict[str, int] = {
+    PAD_TOKEN: PAD_ID,
+    UNK_TOKEN: UNK_ID,
+    BOS_TOKEN: BOS_ID,
+    EOS_TOKEN: EOS_ID,
+    SYSTEM_TOKEN: SYSTEM_ID,
+    USER_TOKEN: USER_ID,
+    ASSISTANT_TOKEN: ASSISTANT_ID,
+}
+
+# Ordered by ID — the exact list BpeTrainer must receive so IDs come out right.
+SPECIAL_TOKENS: list[str] = sorted(SPECIAL_TOKEN_IDS, key=SPECIAL_TOKEN_IDS.get)
+
+
+def assert_special_token_ids(tokenizer_path: str) -> None:
+    """Validate the exact registered-special-token contract.
+
+    Reads the file as plain JSON (no `tokenizers` import needed) so it is cheap
+    to call from any script entry point. Merely finding the strings in the BPE
+    vocabulary is insufficient: all seven must be registered as special tokens,
+    and no eighth registered special token is allowed.
+    """
+    with open(tokenizer_path, encoding="utf-8") as f:
+        obj = json.load(f)
+
+    special_entries = [
+        entry
+        for entry in (obj.get("added_tokens") or [])
+        if isinstance(entry, dict) and entry.get("special") is True
+    ]
+    registered_tokens = [entry.get("content") for entry in special_entries]
+    if len(special_entries) != len(SPECIAL_TOKEN_IDS) or set(registered_tokens) != set(
+        SPECIAL_TOKEN_IDS
+    ):
+        raise ValueError(
+            f"{tokenizer_path} must register exactly these seven special tokens: "
+            f"{list(SPECIAL_TOKEN_IDS)}; got {registered_tokens!r}"
+        )
+
+    added = {entry["content"]: entry.get("id") for entry in special_entries}
+    for token, expected in SPECIAL_TOKEN_IDS.items():
+        got = added.get(token)
+        if got != expected:
+            raise ValueError(
+                f"special token {token!r} has id {got!r} in {tokenizer_path}, but the "
+                f"pipeline hardcodes {expected}. Retrain the tokenizer with "
+                f"tokenizer/tokenizer_training/train_tokenizer.py (its --strict_special_ids "
+                f"default enforces this layout) or reconcile src/special_tokens.py."
+            )
+
+
+def assert_tokenizer_contract(tokenizer_path: str | PathLike[str]) -> None:
+    """Fail fast unless ``tokenizer.json`` is the canonical production artifact.
+
+    This is deliberately stricter than :func:`assert_special_token_ids`, which
+    remains useful for tiny unit-test tokenizers. Production entry points must
+    call this function so a valid-looking seven-token map cannot hide an old
+    vocabulary, normalizer, prefix-space rule, or automatic BOS/EOS processor.
+    """
+    path = str(tokenizer_path)
+    assert_special_token_ids(path)
+    with open(path, encoding="utf-8") as f:
+        obj = json.load(f)
+
+    model = obj.get("model") or {}
+    if model.get("type") != "BPE":
+        raise ValueError(f"{path} must use a BPE model; got {model.get('type')!r}")
+    if model.get("unk_token") != UNK_TOKEN:
+        raise ValueError(
+            f"{path} BPE unk_token must be {UNK_TOKEN!r}; got {model.get('unk_token')!r}"
+        )
+
+    model_vocab = model.get("vocab") or {}
+    token_ids = set(model_vocab.values())
+    token_ids.update(
+        entry.get("id")
+        for entry in (obj.get("added_tokens") or [])
+        if isinstance(entry, dict) and isinstance(entry.get("id"), int)
+    )
+    expected_ids = set(range(CANONICAL_VOCAB_SIZE))
+    if token_ids != expected_ids:
+        raise ValueError(
+            f"{path} runtime vocab IDs must be exactly 0..{CANONICAL_VOCAB_SIZE - 1} "
+            f"(vocab_size={CANONICAL_VOCAB_SIZE}); got {len(token_ids)} unique IDs"
+        )
+
+    if obj.get("normalizer") is not None:
+        raise ValueError(f"{path} must not configure a tokenizer normalizer")
+    if obj.get("post_processor") is not None:
+        raise ValueError(f"{path} must not configure an automatic BOS/EOS post-processor")
+
+    pre = obj.get("pre_tokenizer") or {}
+    if pre.get("type") != "ByteLevel" or pre.get("add_prefix_space") is not False:
+        raise ValueError(
+            f"{path} pre_tokenizer must be ByteLevel(add_prefix_space=False); got {pre!r}"
+        )
+    decoder = obj.get("decoder") or {}
+    if decoder.get("type") != "ByteLevel":
+        raise ValueError(f"{path} decoder must be ByteLevel; got {decoder!r}")

tables/ASSISTANT_RESULTS_VERSIONED.csv

@@ -0,0 +1,145 @@
+score_version,model,slice,axis,true,false,unknown,n,bound_lower,bound_upper,bound_lower_pct,bound_upper_pct,finite_execution_categories,evidence_id
+assistant_review_fable_v1,alpha075,old_qa41,content,7,33,1,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_qa41,joint,7,33,1,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_practical38,content,10,26,2,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_practical38,joint,10,26,2,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_practical38,format,20,1,0,21,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_python14,content,0,14,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_python14,joint,0,14,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_python14,format,3,1,0,4,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_python14,interface,11,3,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,old_python14,historical_finite_categories,,,,,,,,,"{""not_recorded"": 14}",EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,content,2,62,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,joint,2,62,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,format,7,11,0,18,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_python32,content,0,32,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_python32,joint,0,32,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_python32,interface,31,1,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 25, ""fail_or_unextractable"": 1, ""tool_unknown"": 6}",EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,practical27_joint,joint_group_bounds,,,,,26/81,10/27,32.098765,37.037037,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,category:R1DEV-REWRITE,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,category:R1DEV-FOLLOWUP,1,15,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,alpha075,new_natural64,category:R1DEV-CONDITIONAL,1,15,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_qa41,content,7,34,0,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_qa41,joint,7,34,0,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_practical38,content,18,18,2,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_practical38,joint,18,18,2,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_practical38,format,18,3,0,21,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_python14,content,2,12,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_python14,joint,2,12,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_python14,format,3,1,0,4,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_python14,interface,14,0,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,old_python14,historical_finite_categories,,,,,,,,,"{""fail"": 7, ""fail_or_unextractable"": 2, ""pass"": 3, ""tool_unknown"": 2}",EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,content,4,60,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,joint,4,60,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,format,13,5,0,18,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_python32,content,4,28,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_python32,joint,4,28,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_python32,interface,32,0,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 19, ""fail_or_unextractable"": 4, ""pass"": 2, ""tool_unknown"": 7}",EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,practical27_joint,joint_group_bounds,,,,,47/81,17/27,58.024691,62.962963,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,category:R1DEV-REWRITE,2,14,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,category:R1DEV-FOLLOWUP,2,14,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-A,new_natural64,category:R1DEV-CONDITIONAL,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_qa41,content,5,35,1,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_qa41,joint,5,35,1,41,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_practical38,content,6,31,1,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_practical38,joint,6,31,1,38,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_practical38,format,1,20,0,21,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_python14,content,4,10,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_python14,joint,3,11,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_python14,format,2,2,0,4,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_python14,interface,13,1,0,14,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,old_python14,historical_finite_categories,,,,,,,,,"{""fail"": 8, ""fail_or_unextractable"": 1, ""pass"": 4, ""tool_unknown"": 1}",EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,content,0,64,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,joint,0,64,0,64,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,format,3,15,0,18,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_python32,content,3,29,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_python32,joint,3,29,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_python32,interface,31,1,0,32,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 21, ""fail_or_unextractable"": 2, ""pass"": 7, ""tool_unknown"": 2}",EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,practical27_joint,joint_group_bounds,,,,,8/81,1/9,9.876543,11.111111,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,category:R1DEV-REWRITE,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,category:R1DEV-FOLLOWUP,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_review_fable_v1,EXT-B,new_natural64,category:R1DEV-CONDITIONAL,0,16,0,16,,,,,,EVID-ASSIST-V1
+assistant_owner_clarification_4_v1,alpha075,old_qa41,content,6,33,2,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_qa41,joint,6,33,2,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_practical38,content,10,26,2,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_practical38,joint,10,26,2,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_practical38,format,20,1,0,21,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_python14,content,0,14,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_python14,joint,0,14,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_python14,format,3,1,0,4,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_python14,interface,11,3,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,old_python14,historical_finite_categories,,,,,,,,,"{""not_recorded"": 14}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,content,3,61,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,joint,3,61,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,format,7,11,0,18,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_python32,content,0,32,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_python32,joint,0,32,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_python32,interface,31,1,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 25, ""fail_or_unextractable"": 1, ""tool_unknown"": 6}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,practical27_joint,joint_group_bounds,,,,,26/81,10/27,32.098765,37.037037,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,category:R1DEV-REWRITE,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,category:R1DEV-FOLLOWUP,2,14,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,alpha075,new_natural64,category:R1DEV-CONDITIONAL,1,15,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_qa41,content,7,34,0,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_qa41,joint,7,34,0,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_practical38,content,18,18,2,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_practical38,joint,18,18,2,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_practical38,format,18,3,0,21,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_python14,content,1,13,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_python14,joint,1,13,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_python14,format,3,1,0,4,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_python14,interface,14,0,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,old_python14,historical_finite_categories,,,,,,,,,"{""fail"": 7, ""fail_or_unextractable"": 2, ""pass"": 3, ""tool_unknown"": 2}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,content,4,60,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,joint,4,60,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,format,13,5,0,18,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_python32,content,4,28,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_python32,joint,4,28,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_python32,interface,32,0,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 19, ""fail_or_unextractable"": 4, ""pass"": 2, ""tool_unknown"": 7}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,practical27_joint,joint_group_bounds,,,,,47/81,17/27,58.024691,62.962963,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,category:R1DEV-REWRITE,2,14,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,category:R1DEV-FOLLOWUP,2,14,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-A,new_natural64,category:R1DEV-CONDITIONAL,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_qa41,content,5,35,1,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_qa41,joint,5,35,1,41,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_qa41,format,0,1,0,1,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_practical38,content,6,31,1,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_practical38,joint,6,31,1,38,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_practical38,format,1,20,0,21,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_python14,content,4,10,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_python14,joint,3,11,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_python14,format,2,2,0,4,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_python14,interface,13,1,0,14,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,old_python14,historical_finite_categories,,,,,,,,,"{""fail"": 8, ""fail_or_unextractable"": 1, ""pass"": 4, ""tool_unknown"": 1}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,content,0,64,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,joint,0,64,0,64,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,format,3,15,0,18,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_python32,content,2,30,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_python32,joint,2,30,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_python32,format,0,0,0,0,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_python32,interface,31,1,0,32,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_python32,historical_finite_categories,,,,,,,,,"{""fail"": 21, ""fail_or_unextractable"": 2, ""pass"": 7, ""tool_unknown"": 2}",EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,practical27_joint,joint_group_bounds,,,,,8/81,1/9,9.876543,11.111111,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,category:R1DEV-CONTEXT,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,category:R1DEV-REWRITE,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,category:R1DEV-FOLLOWUP,0,16,0,16,,,,,,EVID-ASSIST-C4
+assistant_owner_clarification_4_v1,EXT-B,new_natural64,category:R1DEV-CONDITIONAL,0,16,0,16,,,,,,EVID-ASSIST-C4

tables/PRETRAIN_SOURCE_MIXTURE.csv

@@ -0,0 +1,15 @@
+stage,source_display_name,stage_i_node_key,upstream_dataset,upstream_revision,upstream_config,recorded_licence_at_pinned_revision,target_serialized_tokens,selected_serialized_tokens,overshoot_tokens,selected_documents,share_of_stage_selected_pct,transport_repository_recorded,transport_revision_recorded
+stage_a,FineWeb-Edu (dedup),fineweb_edu_dedup,HuggingFaceTB/smollm-corpus,3ba9d605774198c5868892d7a8deda78031a781f,fineweb-edu-dedup,odc-by-1.0,7110526316,7110526955,639,7350945,71.1052,HuggingFaceTB/smollm-corpus,3ba9d605774198c5868892d7a8deda78031a781f
+stage_a,DCLM-Edu,dclm_edu,HuggingFaceTB/dclm-edu,dbad8ad71224482740cd9c9d353591adbf62fe04,default,cc-by-4.0,2031578947,2031579037,90,1593857,20.3158,HuggingFaceTB/dclm-edu,dbad8ad71224482740cd9c9d353591adbf62fe04
+stage_a,Wikipedia (FineWiki EN),finewiki_en,HuggingFaceFW/finewiki,8bd13e72e6a002407649b3e898535f42ceb1aeb9,en,cc-by-sa-4.0,507894737,507896470,1733,557285,5.0790,HuggingFaceFW/finewiki,8bd13e72e6a002407649b3e898535f42ceb1aeb9
+stage_a,Python-Edu,python_gate_c_full,common-pile/stackv2_edu_filtered,c354dbe88469a1153e97c6a63ac50591849654de,default,per-record metadata.license (Software Heritage permissive subset),350000000,350000772,772,487237,3.5000,NOT_RECORDED,NOT_RECORDED
+stage_b,FineWeb-Edu (dedup),fineweb_edu_dedup,HuggingFaceTB/smollm-corpus,3ba9d605774198c5868892d7a8deda78031a781f,fineweb-edu-dedup,odc-by-1.0,1203125000,1203125470,470,1199264,40.1041,HuggingFaceTB/smollm-corpus,3ba9d605774198c5868892d7a8deda78031a781f
+stage_b,DCLM-Edu,dclm_edu,HuggingFaceTB/dclm-edu,dbad8ad71224482740cd9c9d353591adbf62fe04,default,cc-by-4.0,687500000,687500443,443,538590,22.9166,HuggingFaceTB/dclm-edu,dbad8ad71224482740cd9c9d353591adbf62fe04
+stage_b,structured_tutorial (Cosmopedia v2 + FinePhrase tutorial),structured_tutorial,HuggingFaceTB/smollm-corpus + HuggingFaceFW/finephrase,3ba9d605774198c5868892d7a8deda78031a781f + 78cf4a5ed0099214979c094c963e699c19163838,cosmopedia-v2 + tutorial,odc-by-1.0 (both),343750000,343750175,175,530450,11.4583,HuggingFaceTB/smollm-corpus + HuggingFaceFW/finephrase,3ba9d605774198c5868892d7a8deda78031a781f + 78cf4a5ed0099214979c094c963e699c19163838
+stage_b,Python-Edu,python_gate_c_full,common-pile/stackv2_edu_filtered,c354dbe88469a1153e97c6a63ac50591849654de,default,per-record metadata.license (Software Heritage permissive subset),250000000,250000383,383,347145,8.3333,NOT_RECORDED,NOT_RECORDED
+stage_b,Wikipedia (FineWiki EN),finewiki_en,HuggingFaceFW/finewiki,8bd13e72e6a002407649b3e898535f42ceb1aeb9,en,cc-by-sa-4.0,171875000,171877052,2052,189465,5.7292,HuggingFaceFW/finewiki,8bd13e72e6a002407649b3e898535f42ceb1aeb9
+stage_b,PES2O,pes2o,allenai/dolmino-mix-1124,a319f19eef1e257417b11ea8c30da266ae175557,pes2o,odc-by-1.0,171875000,171875364,364,625468,5.7292,allenai/dolmino-mix-1124,c58ab4b6ff990115e1ff3121953754ee2bc29501
+stage_b,StackExchange,stackexchange,allenai/dolmino-mix-1124,a319f19eef1e257417b11ea8c30da266ae175557,stackexchange,cc-by-sa,171875000,171875353,353,336025,5.7292,allenai/dolmino-mix-1124,c58ab4b6ff990115e1ff3121953754ee2bc29501
+stage_a,TOTAL,,,,,,10000000000,10000003234,,9989324,100.0000,,
+stage_b,TOTAL,,,,,,3000000000,3000004240,,3766407,100.0000,,
+both,GRAND TOTAL,,,,,,13000000000,13000007474,,13755731,,,

tables/PUBLIC_BENCHMARK_RESULTS.csv

@@ -0,0 +1,7 @@
+result_version,protocol_version,model,task,split,dataset,dataset_revision,documents,expected_documents,candidate_sequences,acc_correct,acc,acc_pct,acc_norm_correct,acc_norm,acc_norm_pct,status,historical_or_author_reported,runner,evidence_id
+FP32_V2,FP32_V2,PetitGPT-alpha075,arc_easy,test,allenai/ai2_arc,210d026faf9955653af8916fad021475a3f00453,2376,2376,9501,1372,0.5774410774410774,57.74,1244,0.5235690235690236,52.36,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01
+FP32_V2,FP32_V2,PetitGPT-alpha075,piqa,validation,baber/piqa,142f6d7367fd9877f0fb3b5734ea6a545f54cdd1,1838,1838,3676,1167,0.6349292709466812,63.49,1145,0.6229597388465724,62.30,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01
+FP32_V2,FP32_V2,SmolLM-135M-Instruct,arc_easy,test,allenai/ai2_arc,210d026faf9955653af8916fad021475a3f00453,2376,2376,9501,1170,0.49242424242424243,49.24,1033,0.43476430976430974,43.48,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01
+FP32_V2,FP32_V2,SmolLM-135M-Instruct,piqa,validation,baber/piqa,142f6d7367fd9877f0fb3b5734ea6a545f54cdd1,1838,1838,3676,1233,0.6708378672470077,67.08,1236,0.6724700761697497,67.25,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01
+FP32_V2,FP32_V2,SmolLM2-135M-Instruct,arc_easy,test,allenai/ai2_arc,210d026faf9955653af8916fad021475a3f00453,2376,2376,9501,1283,0.539983164983165,54.00,1160,0.4882154882154882,48.82,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01
+FP32_V2,FP32_V2,SmolLM2-135M-Instruct,piqa,validation,baber/piqa,142f6d7367fd9877f0fb3b5734ea6a545f54cdd1,1838,1838,3676,1226,0.6670293797606094,66.70,1227,0.6675734494015234,66.76,COMPLETE,null,native_protocol_compatible_evaluator_NOT_installed_lm_eval,EVID-BENCH-01