TokenizerTest-32m-B: final + every-10k weights, card, bpb curve

README.md

@@ -0,0 +1,117 @@
+---
+license: cc-by-sa-4.0
+language:
+- pl
+library_name: pytorch
+tags:
+- polish
+- language-model
+- research
+- tokenizer-comparison
+pipeline_tag: text-generation
+---
+
+# TokenizerTest-32m-B (Polish research base model, tokenizer comparison — arm B)
+
+A small Polish **base** language model trained as arm **B** of a controlled tokenizer experiment. It is a research
+checkpoint for comparing tokenizers, not a model for use.
+
+## Companion model (tokenizer A/B experiment)
+
+This model is one half of a **controlled tokenizer comparison**. Two models were trained with the **same architecture, training recipe, seed and the same Polish text**. The **only** difference is the tokenizer:
+
+| arm | tokenizer | repo | final step |
+|---|---|---|---|
+| **A** | Fabryka AI BPE, vocab 32 000 (`60e23148`) | SlayerLab/TokenizerTest-32m-A | 89 000 |
+| **B** | tokenizer by stubbornGuy, vocab 32 000 (`77eeebed`) | SlayerLab/TokenizerTest-32m-B | 95 000 |
+
+This model: **B**.
+
+**Shared recipe:** ~38.9M parameters (tied input/output embeddings); "32m" in the repo name is the recipe name ("Glint 32M r6"), not the parameter count. Architecture: 15 layers, d_model 384, 6 heads, context 1024, Muon optimizer, batch 32, seed 1337, trainer `train_gpt_ref_r6.py` (`0223c083`).
+
+Both arms saw the same ~12.4 GB of Polish text (public dataset [`SlayerLab/slayer-pl-8x3b`](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) @ `d50df04a`, files `pack-00`, `pack-01` and `shared/{core,core_sa,legal,legal_sa}`; validation = `pack-07` after 13-gram decontamination); B splits it more finely (on the training text 3.96 vs 4.25 bytes per token incl. EOS; on validation 4.14 vs 4.41), so it needed more steps for the same bytes. For the same reason B has lower bits per token (val 4.395 vs 4.668), which says nothing about model quality.
+
+**How to compare:** loss per token is **not** comparable across tokenizers. We compare **bits per byte** on the same bytes (lower is better):
+
+| | A | B | B − A |
+|---|---|---|---|
+| validation (7 232 820 B) | **1.0583** | 1.0622 | +0.0040 (+0.37 %) |
+| held-out W (1 430 217 B) | **1.0781** | 1.0883 | +0.0101 (+0.94 %) |
+
+From ~4 B bytes onward the gap is roughly constant (val +0.003…+0.004, W +0.008…+0.010) and does not close by the end of training. Curve every 10k steps (bits per byte; note: at the same **step** B has seen fewer bytes than A):
+
+| step | A val | B val | A W | B W |
+|---|---|---|---|---|
+| 10 000 | 1.1910 | 1.2009 | 1.2327 | 1.2463 |
+| 20 000 | 1.1448 | 1.1523 | 1.1799 | 1.1896 |
+| 30 000 | 1.1200 | 1.1278 | 1.1499 | 1.1634 |
+| 40 000 | 1.1032 | 1.1109 | 1.1304 | 1.1435 |
+| 50 000 | 1.0888 | 1.0969 | 1.1141 | 1.1291 |
+| 60 000 | 1.0773 | 1.0850 | 1.1000 | 1.1145 |
+| 70 000 | 1.0675 | 1.0752 | 1.0885 | 1.1025 |
+| 80 000 | 1.0610 | 1.0677 | 1.0818 | 1.0944 |
+| 89 000 | 1.0583 | — | 1.0781 | — |
+| 90 000 | — | 1.0633 | — | 1.0892 |
+| 95 000 | — | 1.0622 | — | 1.0883 |
+
+The same numbers, with checkpoint and data hashes, are in `bpb_curve.json`.
+
+**Caveat:** one seed per arm, no noise estimate. The honest reading is *"at this scale the B tokenizer brings no gain in bits per byte"*, not *"B is worse"*.
+
+## Training data
+
+Public dataset [`SlayerLab/slayer-pl-8x3b`](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) at revision `d50df04a`; the same files for both arms:
+
+| file | use | sha256 |
+|---|---|---|
+| `pack-00/web.parquet` | training (web) | `5cbb6dfa3a276a30db3c2034105ffffa75cc29b767456e7d75c305f9bd7ec562` |
+| `pack-01/web.parquet` | training (web) | `802f4829f41ef83e23979d2460500907bcfd44278e300eccb5e39ac666a04c2c` |
+| `shared/core.parquet` | training (core) | `ea0b4b901ace3780501b02a3972f014f77ce722315476e6f93681616958e0a99` |
+| `shared/core_sa.parquet` | training (core, CC BY-SA) | `910b18eeebbc26efbbd76554d7484d016c6df692a3b6a18492b5c9ceb32549cb` |
+| `shared/legal.parquet` | training (legal) | `6d4cc2f604d76452cd7e7ea52d1a04826f5e44f09bb035ba0853232fe373e74a` |
+| `shared/legal_sa.parquet` | training (legal, CC BY-SA) | `a1609cd31e12e0f6fdfdb030edb538bc230fc5f5ab376d23136180338e9dbd1e` |
+| `pack-07/web.parquet` | validation (13-gram decontaminated against training) | `3d93331061595949f2e2cab187cfe688a117785c048e94688808b399b3f53b67` |
+
+## Usage
+
+The model is a custom PyTorch architecture (same code as `SlayerLab/GoLLeM-v6-250M`), not a `transformers` class.
+
+| file | content |
+|---|---|
+| `model.safetensors` | final weights, step 95 000 (fp32, 38,866,958 parameters; output head stored as a copy of the tied embedding) |
+| `config.json` | architecture |
+| `tokenizer.json` | arm B tokenizer by stubbornGuy, 32 000 tokens (`tokenizers`) |
+| `modeling_gollem_v6.py` | model code (inference) and a small `generate` helper |
+| `step_10000/` … `step_90000/` | intermediate checkpoints every 10k steps (same three files each; weights only, no optimizer state) |
+| `bpb_curve.json` | bits-per-byte curve (step, checkpoint sha256, val, W) |
+| `SHA256SUMS` | sha256 of every file |
+
+```python
+# pip install torch safetensors tokenizers huggingface_hub
+from huggingface_hub import snapshot_download
+import sys
+
+path = snapshot_download("SlayerLab/TokenizerTest-32m-B")   # pin revision="<commit>" for reproducibility
+sys.path.insert(0, path)
+from modeling_gollem_v6 import load_gollem_v6, generate
+
+model, tok = load_gollem_v6(path)                          # final checkpoint
+print(generate(model, tok, "Kraków jest miastem", max_new_tokens=40, temperature=0.8, top_k=50, seed=1))
+model_10k, _ = load_gollem_v6(f"{path}/step_10000")        # an intermediate checkpoint
+```
+
+Loading any `model.safetensors` of this repository into `modeling_gollem_v6.py` reproduces the logits of the
+corresponding training checkpoint exactly (checked with `torch.equal` for every step).
+
+## Limitations
+
+- Small **base** model (~38.9M parameters): it continues Polish text; it does not answer questions or follow
+  instructions. Limited knowledge; may reproduce errors and biases of web text. Research use only.
+- Single run, single seed per arm; see the caveat above.
+
+## License
+
+**Weights: CC BY-SA 4.0** (the training text includes CC BY-SA parts of `SlayerLab/slayer-pl-8x3b`). Attribution:
+TokenizerTest-32m-B, Fabryka AI (SlayerLab), link to this repository; derivative weights under the same licence.
+The tokenizer (`tokenizer.json`) was created by stubbornGuy and is released here under CC BY-SA 4.0 with his consent.
+Training data keep their upstream licences; see the dataset card of `SlayerLab/slayer-pl-8x3b`.

SHA256SUMS

@@ -0,0 +1,33 @@
+a0e944099340c90223f7e02b166d317e87f699b7cf811a279c9e6bbb9d39f475  README.md
+5a52c0867ea28be8ef8506f830d803ce98fcdea59fb654c8f4fe6ffa75c95e90  bpb_curve.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  config.json
+1a2fd327fc4069fe739e4152810acbfb66d23cf873e674d55dc85c839149d354  model.safetensors
+ee2ea8c4e957413f3a7c1e8fe93a0a0b4431a2bcabaa6cc5ceeaea103fcd58ee  modeling_gollem_v6.py
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_10000/config.json
+4fa3ae6e45f848b2c386f16c9c73a12bfcd52d5cf93eb348550a5a23dcfd5456  step_10000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_10000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_20000/config.json
+b5c10b7a8a4c3ce36aefa06de0279b07201233360304607d4eace1cd0e9be8c2  step_20000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_20000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_30000/config.json
+ebd312799b49497d878f55e7aaeb99cfb10e7ead11b56997f51617ff82500767  step_30000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_30000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_40000/config.json
+d7e74c53c9f62d453dfe43966811cad432e389b8c7b1115d749d16b4dfc68fb7  step_40000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_40000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_50000/config.json
+e663da4de9c9c353a1bfb1fd2b94f41eb52c0928b89be5765c8bf4b70627a75e  step_50000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_50000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_60000/config.json
+84b3c8517269a5f1168e58299d4695a60082eb7212a2e3b1bdcfb3828554ec28  step_60000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_60000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_70000/config.json
+e4870e7e196d98047efe559f6e3bca7ebd88a81f50ccf2994b89b256ff4ddfee  step_70000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_70000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_80000/config.json
+d63c02503612f59a2a039c304028664403d32b446729418584c4e955aa73bde4  step_80000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_80000/tokenizer.json
+525c8d131963e6dc92f798bc602e4367e5a26fc274016ddb08b8a87b54f7fa51  step_90000/config.json
+7f56f3bdca911cf3be9958107160aa49e6a48b0fd83db463630a2ba700738371  step_90000/model.safetensors
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  step_90000/tokenizer.json
+77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7  tokenizer.json

bpb_curve.json

@@ -0,0 +1,82 @@
+{
+ "definition": "bits/byte = sum of NLL over text tokens / ln 2 / UTF-8 bytes; non-overlapping 1024-token windows; EOS targets excluded",
+ "trainer_sha256": "0223c0830ed4ae323f6424038a2de7bfb6b425653f3591dd55844f95706f5d17",
+ "tokenizer_sha256": "77eeebed509827f18b859ff159f29b2405d24d5b84381d6226f22e4fb6ca98d7",
+ "sets": {
+  "W": {
+   "sha256": "234e7ec21bf2236bc74425591531f2c4772f8c49ff853d57f6fa16f33ee90ac7",
+   "docs": 463,
+   "text_bytes": 1430217,
+   "tokens_text": 351268
+  },
+  "val": {
+   "sha256": "748d13898e736666d96f1c9307222c52414a9b66b06bf212e7a07e765babc668",
+   "docs": 2723,
+   "text_bytes": 7232820,
+   "tokens_text": 1748065
+  }
+ },
+ "device": "cpu",
+ "ckpts": [
+  {
+   "step": 10000,
+   "ckpt_sha256": "13a9130e6ac322c3a342c6bbcdf0838bdaa82e7e6962c0b4ab4ee1bdb677d2c8",
+   "W": 1.246255139078163,
+   "val": 1.2008817462071877
+  },
+  {
+   "step": 20000,
+   "ckpt_sha256": "78d637de184f406d22621b4512fc79da9d2b47f9ba532525aba44199505bdd5f",
+   "W": 1.189575099180923,
+   "val": 1.152323463429622
+  },
+  {
+   "step": 30000,
+   "ckpt_sha256": "2ec241ef98aaf7d54a15398bbb3d64dd2f0d3d6e0d562cd5e2aa4b7f7a468ac3",
+   "W": 1.163382329460513,
+   "val": 1.1278191393784163
+  },
+  {
+   "step": 40000,
+   "ckpt_sha256": "0bf61a0a7595c56f22af420dcd7a956a087c49100be258660fbc5ab109934b1b",
+   "W": 1.1434992106638442,
+   "val": 1.1109230609739182
+  },
+  {
+   "step": 50000,
+   "ckpt_sha256": "2c8189ed98f6bd27064205725ac1430c815cb4a718c764a57178632ff676b590",
+   "W": 1.1290988704930747,
+   "val": 1.096905167732812
+  },
+  {
+   "step": 60000,
+   "ckpt_sha256": "a24e66893894405f763a83ea28eee3432e8ccb6911f15e59715119ae1705089f",
+   "W": 1.1144814554280478,
+   "val": 1.0850306176283422
+  },
+  {
+   "step": 70000,
+   "ckpt_sha256": "2cedd2505161e340b2ef8ad79199d431ae54aec6cecac6d131f7d76dddfcea34",
+   "W": 1.1024713768041312,
+   "val": 1.0752361156563053
+  },
+  {
+   "step": 80000,
+   "ckpt_sha256": "f074891643fe2efa2bcf2e8545cd6ade5e024baf29fe47f161fcc43ac092c76c",
+   "W": 1.0943567744308031,
+   "val": 1.0677255240917687
+  },
+  {
+   "step": 90000,
+   "ckpt_sha256": "16f2089bd49d108383660a206a93551edacaa69ebe76686b65b2e32f9df5104a",
+   "W": 1.0892155925274738,
+   "val": 1.0633125873779654
+  },
+  {
+   "step": 95000,
+   "ckpt_sha256": "538505bc15e4332231b3e2958b02b7567348b3d43b8974e57f48526266879ab9",
+   "W": 1.0882527260062658,
+   "val": 1.062229520611975
+  }
+ ]
+}

config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a2fd327fc4069fe739e4152810acbfb66d23cf873e674d55dc85c839149d354
+size 204636280

modeling_gollem_v6.py

@@ -0,0 +1,169 @@
+"""GoLLeM v6 model code (inference only), extracted from the training script used for this run.
+
+The module definitions are those of the training script; only the weight initialisation and the training loss were
+removed and comments translated. Weights load with strict=True. Requirements: torch, safetensors, tokenizers.
+
+    from modeling_gollem_v6 import load_gollem_v6, generate
+    model, tok = load_gollem_v6(".")          # directory with config.json, model.safetensors, tokenizer.json
+    print(generate(model, tok, "Najwyższym szczytem Polski jest", max_new_tokens=40))
+
+Notes: RoPE uses the interleaved convention (pairs of adjacent channels), not rotate_half; the output head is tied to
+the token embedding; context length is 1,024 tokens.
+"""
+from __future__ import annotations
+
+import json
+import os
+import types
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class RMSNorm(nn.Module):
+    """RMSNorm with fp32 compute for stability."""
+    def __init__(self, d, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(d))
+        self.eps = eps
+
+    def forward(self, x):
+        return x * torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps).to(x.dtype) * self.weight
+
+
+def make_norm(d, cfg):
+    return RMSNorm(d, cfg.norm_eps) if cfg.norm == "rmsnorm" else nn.LayerNorm(d)
+
+
+def apply_rope(x, base=100000.0):
+    """Parameter-free RoPE on [B, H, T, D], interleaved convention (even/odd channel pairs)."""
+    _, _, T, dim = x.shape
+    pos = torch.arange(T, device=x.device, dtype=torch.float32)
+    freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=x.device, dtype=torch.float32) / dim))
+    ang = torch.outer(pos, freq)
+    cos, sin = ang.cos().to(x.dtype)[None, None], ang.sin().to(x.dtype)[None, None]
+    even, odd = x[..., ::2], x[..., 1::2]
+    return torch.stack((even * cos - odd * sin, even * sin + odd * cos), dim=-1).flatten(-2)
+
+
+class SwiGLU(nn.Module):
+    """Gated MLP: down(silu(gate(x)) * up(x))."""
+    def __init__(self, d, hidden):
+        super().__init__()
+        self.gate = nn.Linear(d, hidden, bias=False)
+        self.up = nn.Linear(d, hidden, bias=False)
+        self.down = nn.Linear(hidden, d, bias=False)
+
+    def forward(self, x):
+        return self.down(F.silu(self.gate(x)) * self.up(x))
+
+
+class Block(nn.Module):
+    def __init__(self, d, nh, block, cfg, is_first=False):
+        super().__init__()
+        self.ln1 = make_norm(d, cfg)
+        self.ln2 = make_norm(d, cfg)
+        self.qkv = nn.Linear(d, 3 * d)
+        self.proj = nn.Linear(d, d)
+        if cfg.ffn == "swiglu":
+            self.mlp = SwiGLU(d, int(round(cfg.ffn_mult * d)))
+        else:
+            self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
+        self.nh = nh
+        self.d = d
+        self.cfg = cfg
+        self.is_first = is_first
+        if cfg.value_residual and not is_first:
+            self.vr_lambda = nn.Parameter(torch.zeros(1))
+        if cfg.qk_norm:
+            hd = d // nh
+            self.q_norm = RMSNorm(hd, cfg.norm_eps)
+            self.k_norm = RMSNorm(hd, cfg.norm_eps)
+
+    def forward(self, x, v0=None):
+        B, T, D = x.size()
+        h = self.ln1(x)
+        q, k, v = self.qkv(h).split(self.d, dim=2)
+        hd = D // self.nh
+        q = q.view(B, T, self.nh, hd).transpose(1, 2)
+        k = k.view(B, T, self.nh, hd).transpose(1, 2)
+        v = v.view(B, T, self.nh, hd).transpose(1, 2)
+        if self.cfg.qk_norm:
+            q = self.q_norm(q)
+            k = self.k_norm(k)
+        if self.cfg.pos == "rope":
+            q = apply_rope(q, self.cfg.rope_theta)
+            k = apply_rope(k, self.cfg.rope_theta)
+        if self.cfg.value_residual:
+            if self.is_first:
+                v0 = v
+            else:
+                v = v + self.vr_lambda * v0
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        y = y.transpose(1, 2).contiguous().view(B, T, D)
+        x = x + self.proj(y)
+        x = x + self.mlp(self.ln2(x))
+        return x, v0
+
+
+class GPT(nn.Module):
+    def __init__(self, vocab, n_layer, n_embd, n_head, block, cfg):
+        super().__init__()
+        self.cfg = cfg
+        self.tok = nn.Embedding(vocab, n_embd)
+        self.use_rope = cfg.pos == "rope"
+        if not self.use_rope:
+            self.pos = nn.Embedding(block, n_embd)
+        self.blocks = nn.ModuleList([Block(n_embd, n_head, block, cfg, is_first=(i == 0)) for i in range(n_layer)])
+        self.lnf = make_norm(n_embd, cfg)
+        self.head = nn.Linear(n_embd, vocab, bias=False)
+        self.head.weight = self.tok.weight  # tied
+        self.block = block
+
+    def forward(self, idx):
+        B, T = idx.size()
+        x = self.tok(idx)
+        if not self.use_rope:
+            pos = torch.arange(T, device=idx.device)
+            x = x + self.pos(pos)[None]
+        v0 = None
+        for b in self.blocks:
+            x, v0 = b(x, v0)
+        logits = self.head(self.lnf(x))
+        cap = getattr(self.cfg, "logit_cap", 0.0)
+        if cap and cap > 0:
+            logits = cap * torch.tanh(logits / cap)
+        return logits
+
+
+def load_gollem_v6(path, device="cpu"):
+    """Model (eval mode) and tokenizer from a directory with config.json, model.safetensors and tokenizer.json."""
+    from safetensors.torch import load_file
+    from tokenizers import Tokenizer
+
+    with open(os.path.join(path, "config.json"), encoding="utf-8") as f:
+        cfg = types.SimpleNamespace(**json.load(f))
+    model = GPT(cfg.vocab, cfg.n_layer, cfg.n_embd, cfg.n_head, cfg.block, cfg)
+    model.load_state_dict(load_file(os.path.join(path, "model.safetensors")), strict=True)
+    return model.to(device).eval(), Tokenizer.from_file(os.path.join(path, "tokenizer.json"))
+
+
+@torch.no_grad()
+def generate(model, tok, prompt, max_new_tokens=64, temperature=0.0, top_k=None, seed=None):
+    """Continue `prompt`. temperature 0 = greedy; otherwise sampling, optionally top-k."""
+    g = torch.Generator().manual_seed(seed) if seed is not None else None
+    dev = next(model.parameters()).device
+    x = torch.tensor([tok.encode(prompt).ids], device=dev)
+    for _ in range(max_new_tokens):
+        logits = model(x[:, -model.block:])[:, -1].float().cpu()
+        if temperature <= 0:
+            nxt = logits.argmax(-1, keepdim=True)
+        else:
+            logits = logits / temperature
+            if top_k:
+                kth = torch.topk(logits, top_k).values[:, -1:]
+                logits = logits.masked_fill(logits < kth, float("-inf"))
+            nxt = torch.multinomial(torch.softmax(logits, -1), 1, generator=g)
+        x = torch.cat([x, nxt.to(dev)], dim=1)
+    return tok.decode(x[0].tolist())

step_10000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_10000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fa3ae6e45f848b2c386f16c9c73a12bfcd52d5cf93eb348550a5a23dcfd5456
+size 204636280

step_20000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_20000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5c10b7a8a4c3ce36aefa06de0279b07201233360304607d4eace1cd0e9be8c2
+size 204636280

step_30000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_30000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebd312799b49497d878f55e7aaeb99cfb10e7ead11b56997f51617ff82500767
+size 204636280

step_40000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_40000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7e74c53c9f62d453dfe43966811cad432e389b8c7b1115d749d16b4dfc68fb7
+size 204636280

step_50000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_50000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e663da4de9c9c353a1bfb1fd2b94f41eb52c0928b89be5765c8bf4b70627a75e
+size 204636280

step_60000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_60000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84b3c8517269a5f1168e58299d4695a60082eb7212a2e3b1bdcfb3828554ec28
+size 204636280

step_70000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_70000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4870e7e196d98047efe559f6e3bca7ebd88a81f50ccf2994b89b256ff4ddfee
+size 204636280

step_80000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_80000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d63c02503612f59a2a039c304028664403d32b446729418584c4e955aa73bde4
+size 204636280

step_90000/config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 384,
+  "n_head": 6,
+  "n_layer": 15,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32000
+}

step_90000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f56f3bdca911cf3be9958107160aa49e6a48b0fd83db463630a2ba700738371
+size 204636280