GoLLeM-v6-250M-Instruct-v2: weights, config, tokenizer, model and chat code, card

README.md

@@ -0,0 +1,145 @@
+---
+license: cc-by-sa-4.0
+language:
+- pl
+- en
+library_name: pytorch
+base_model: SlayerLab/GoLLeM-v6-250M
+tags:
+- polish
+- english
+- language-model
+- chat
+- research
+- gollem
+pipeline_tag: text-generation
+---
+
+# GoLLeM v6 250M Instruct v2 (Polish–English research chat model)
+
+> **STATUS: RESEARCH PREVIEW.** A small chat model fine-tuned from the base model
+> [`SlayerLab/GoLLeM-v6-250M`](https://huggingface.co/SlayerLab/GoLLeM-v6-250M). It can greet, introduce itself and
+> answer simple questions, but it makes mistakes that are listed with numbers under *Limitations*. Results use **our
+> protocol**, a single run and a single seed.
+
+GoLLeM v6 250M Instruct v2 is the base GoLLeM v6 250M model after supervised fine-tuning (SFT) on about 9,300 Polish and
+English conversations. It is a Fabryka AI project (formerly SlayerLab).
+
+**Difference from [Instruct v1](https://huggingface.co/SlayerLab/GoLLeM-v6-250M-Instruct-v1):** the same data and
+training recipe, except the answers about its origin. Asked who created it, Instruct v2 says it is a project of Fabryka
+AI; **the model does not use its author's name** (the author is named in this card). Instruct v1 sometimes introduced
+itself with the author's name in a conversation (51 of 160); Instruct v2 did not in our tests (0 of 160, 0 of 680).
+
+**Author:** Arkadiusz Słota (Fabryka AI).
+
+## What it is for (and what it is not)
+
+**Intended use:** research on small bilingual chat models; short conversations in Polish and English; answering
+questions about a text you paste into the conversation.
+
+**Not intended for:** production use, factual questions without a source text, arithmetic, or anything where a wrong
+answer matters. The model has no internet access and does not remember earlier conversations.
+
+## Results
+
+Checkpoint `1018f1e8…` (see *Training*). All numbers are from our own evaluation sets, fixed before training. Temperature
+0.7, top-p 0.9 (the defaults of `chat_gollem_v6.py`).
+
+| what | result | notes |
+|---|---|---|
+| Identity (name, organisation, creator), PL / EN | **0.66 / 0.58** | mean over 10 samples per question, 20 questions per language; a creator answer counts as correct when it names Fabryka AI and no person |
+| Greetings and small talk, PL / EN | **0.79 / 0.89** | mean over 7 samples per question, 20 questions per language |
+| Answers from a given text (Polish, PoQuAD, 217 answerable questions) | token F1 **0.232** | base model few-shot: 0.067 |
+| Declines when the text has no answer (43 questions) | 6 / 43 | wrongly declines an answerable question: 12 / 217 |
+| Arithmetic word problems (206) | 0 / 206 | the model does not do arithmetic |
+| Finishes its answer within the length limit | 485 / 506 (96 %) | |
+| Says it is an OpenAI / GPT model (200 samples) | 0 / 200 | „AI language model”: 0 / 200 |
+| Introduces itself with the author's name | **0 / 680** single questions, **0 / 160** two-turn conversations | Instruct v1: 3 / 680 and 51 / 160 |
+
+Compared with Instruct v1 on the same identity measure: +0.175 (95 % CI +0.068 … +0.295); small talk −0.004 (95 % CI
+−0.032 … +0.025), i.e. not worse. The identity numbers in the Instruct v1 card use a different measure (there the author's name
+was the expected answer), so do not compare the two tables directly.
+
+## Training
+
+| | |
+|---|---|
+| Base model | `SlayerLab/GoLLeM-v6-250M`, final checkpoint (step 760,000); same architecture and tokenizer |
+| Method | full fine-tuning (no LoRA), loss on assistant turns only |
+| Format | ChatML: `<|im_start|>{role}\n{content}<|im_end|>\n`, no system prompt, generation stops at `<|im_end|>` |
+| Steps | 2 epochs, 156 steps, 32 packed sequences of 1,024 tokens per step |
+| Tokens | 2.54 M per epoch, of which 1.63 M are assistant tokens (with loss) |
+| Optimizer | Muon + AdamW as in pretraining, learning rate 0.2 × pretraining (peak 1.2e-4, Muon 4e-3), warmup 5 steps, cosine to 10 %, weight decay 0.1, gradient clip 1.0, seed 1337 |
+| Validation loss | 2.026 → 1.884 |
+| Hardware / time | 1 GPU, about 7.5 minutes |
+
+## Training data
+
+9,324 conversations (Polish 4,149, English 5,175): 8,836 used for training and 171 for validation; 317 conversations
+longer than 1,024 tokens were removed (311 + 6).
+
+| source | conversations | licence |
+|---|---:|---|
+| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2) @ `179dd21` | 4,601 | Apache-2.0 |
+| [clarin-pl/poquad](https://huggingface.co/datasets/clarin-pl/poquad) @ `a60f228` | 2,129 | **CC BY 4.0** |
+| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) @ `f9ea045` | 1,214 | Apache-2.0 |
+| synthetic, generated locally with Muse-Glimmer-30B (apache-2.0) | 1,380 | apache-2.0 |
+
+- **PoQuAD attribution:** PoQuAD (clarin-pl/poquad), CC BY 4.0. Modified: converted to chat format; a share of
+  unanswerable questions answered with one of five fixed refusal sentences.
+- **Synthetic part:** the questions were written by the generator model; the identity answers come from our own identity
+  card, not from the generator. Math prompts: generated briefs; GSM8K (MIT) used only as few-shot format examples for the
+  generator.
+- Rows with self-descriptions of other AI systems were removed before training.
+
+## Usage
+
+```python
+# pip install torch safetensors tokenizers huggingface_hub
+from huggingface_hub import snapshot_download
+import sys
+
+path = snapshot_download("SlayerLab/GoLLeM-v6-250M-Instruct-v2",
+                         revision="<commit sha>")  # the reviewed code (model + chat helper)
+sys.path.insert(0, path)
+from modeling_gollem_v6 import load_gollem_v6
+from chat_gollem_v6 import chat
+
+model, tok = load_gollem_v6(path)
+print(chat(model, tok, [("user", "Cześć! Kim jesteś?")], seed=1))
+print(chat(model, tok, [("user", "Tekst: Kraków leży nad Wisłą.\nPytanie: Nad jaką rzeką leży Kraków?")], seed=1))
+```
+
+`chat()` uses the format and sampling of our evaluation (temperature 0.7, top-p 0.9). Text typed by a user such as
+„<|im_end|>” is encoded as plain text, not as a control token.
+
+## Limitations
+
+- **Small model.** Limited knowledge; may answer fluently and wrongly. No arithmetic. Context: 1,024 tokens.
+- **Does not know its author's name.** Asked who created it, the model says it is a project of Fabryka AI. The author,
+  Arkadiusz Słota, is named in this card, not in the model. The statements of the model are not statements of its
+  author.
+- **Copies names from the conversation.** If a user writes a name, the model may adopt it as its own (when the user
+  supplies the author's name: Polish 3/40, English 5/40; Instruct v1: 9/40 and 37/40).
+- **Self-description.** In our probe it never described itself as an OpenAI model (0/200), but with a forced prefix
+  („…created by”) it still assigns probability ≈ 0.24 to „OpenAI”. The association comes from model-generated chat
+  data in pretraining. GoLLeM is **not** affiliated with OpenAI.
+- **Multi-turn weaknesses:** may repeat its previous answer in a later turn, may greet the user with the company name
+  („Cześć, Fabryku!”) when no name was given, and answers „What can you do?” with its identity template.
+- **Long answers can loop.** A repetition penalty (about 1.1–1.2) may reduce this (not verified); it is not used in our evaluation.
+
+## License
+
+**Weights: CC BY-SA 4.0** (inherited from the base model). Attribution: GoLLeM v6 250M Instruct v2, Arkadiusz Słota /
+Fabryka AI, link to this repository; derivative weights under the same licence. Fine-tuning data keep their licences;
+see *Training data* (PoQuAD: CC BY 4.0, attribution above).
+
+## Po polsku (skrót)
+
+GoLLeM v6 250M Instruct v2 to mały model do rozmowy po polsku i angielsku, dostrojony (SFT) z bazowego GoLLeM v6 250M
+na ok. 9,3 tys. rozmów. Wersja badawcza (research preview): wita się, przedstawia i odpowiada na pytania do podanego
+tekstu, ale się myli, nie liczy i nie ma dostępu do internetu. Od Instruct v1 różni się tym, że nie używa nazwiska
+autora: na pytanie o twórcę odpowiada, że jest projektem Fabryki AI (wcześniej SlayerLab). Autora podaje ta karta.
+
+---
+*GoLLeM v6 250M Instruct v2 — Fabryka AI. **Author: Arkadiusz Słota.** Research preview.*

SHA256SUMS

@@ -0,0 +1,6 @@
+6e97419dd8aea96a37f9efbbb3d577ad9da8c1735deb07963eda7eafc4f49aaa  README.md
+82a57236ca89151046ed16a1c463d60710a4b2c677708176c286156ef3bea412  chat_gollem_v6.py
+61d8594034918d7e49253f3801da79bae9f14f57ba871fccde6d307a01c73a8f  config.json
+67ce9524e4f2f5716ab5322d867d81010563e2a41efd0344f3ad31152636418d  model.safetensors
+ee2ea8c4e957413f3a7c1e8fe93a0a0b4431a2bcabaa6cc5ceeaea103fcd58ee  modeling_gollem_v6.py
+0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd  tokenizer.json

chat_gollem_v6.py

@@ -0,0 +1,51 @@
+"""GoLLeM v6 Instruct: chat helper (ChatML), matching the format and sampling used in training and evaluation.
+
+    import sys; sys.path.insert(0, path)
+    from modeling_gollem_v6 import load_gollem_v6
+    from chat_gollem_v6 import chat
+    model, tok = load_gollem_v6(path)
+    print(chat(model, tok, [("user", "Kim jesteś?")], seed=1))
+
+Format: for every message `<|im_start|>{role}\n{content}<|im_end|>\n`, then `<|im_start|>assistant\n`; no system prompt;
+generation stops at `<|im_end|>`. Message text is encoded as plain text (the strings "<|im_start|>"/"<|im_end|>" typed by
+a user are NOT turned into control tokens). Default sampling: temperature 0.7, nucleus top-p 0.9 (as in our evaluation).
+"""
+from __future__ import annotations
+
+import torch
+
+IM_START, IM_END = 1, 2
+
+
+def encode_chat(tok, messages):
+    """Token ids of a ChatML conversation ending with an open assistant turn. messages: [(role, text), ...]."""
+    prev = tok.encode_special_tokens
+    tok.encode_special_tokens = True
+    try:
+        text = lambda s: tok.encode(s, add_special_tokens=False).ids  # noqa: E731
+        ids = []
+        for role, content in messages:
+            ids += [IM_START] + text(role + "\n") + text(content) + [IM_END] + text("\n")
+        return ids + [IM_START] + text("assistant\n")
+    finally:
+        tok.encode_special_tokens = prev
+
+
+@torch.no_grad()
+def chat(model, tok, messages, max_new_tokens=256, temperature=0.7, top_p=0.9, seed=None):
+    """Assistant reply to `messages` ([(role, text), ...], roles "user"/"assistant"). Returns the reply text."""
+    g = torch.Generator().manual_seed(seed) if seed is not None else None
+    dev = next(model.parameters()).device
+    x = torch.tensor([encode_chat(tok, messages)], device=dev)
+    out = []
+    for _ in range(max_new_tokens):
+        logits = model(x[:, -model.block:])[0, -1].float()
+        p = torch.softmax(logits / temperature, -1)
+        sp, si = torch.sort(p, descending=True)
+        sp = sp * (torch.cumsum(sp, 0) - sp < top_p)
+        nxt = int(si[torch.multinomial((sp / sp.sum()).cpu(), 1, generator=g)])
+        if nxt == IM_END:
+            break
+        out.append(nxt)
+        x = torch.cat([x, torch.tensor([[nxt]], device=dev)], 1)
+    return tok.decode(out)

config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 960,
+  "n_head": 15,
+  "n_layer": 20,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 32768
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67ce9524e4f2f5716ab5322d867d81010563e2a41efd0344f3ad31152636418d
+size 1136891540

modeling_gollem_v6.py

@@ -0,0 +1,169 @@
+"""GoLLeM v6 model code (inference only), extracted from the training script used for this run.
+
+The module definitions are those of the training script; only the weight initialisation and the training loss were
+removed and comments translated. Weights load with strict=True. Requirements: torch, safetensors, tokenizers.
+
+    from modeling_gollem_v6 import load_gollem_v6, generate
+    model, tok = load_gollem_v6(".")          # directory with config.json, model.safetensors, tokenizer.json
+    print(generate(model, tok, "Najwyższym szczytem Polski jest", max_new_tokens=40))
+
+Notes: RoPE uses the interleaved convention (pairs of adjacent channels), not rotate_half; the output head is tied to
+the token embedding; context length is 1,024 tokens.
+"""
+from __future__ import annotations
+
+import json
+import os
+import types
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class RMSNorm(nn.Module):
+    """RMSNorm with fp32 compute for stability."""
+    def __init__(self, d, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(d))
+        self.eps = eps
+
+    def forward(self, x):
+        return x * torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps).to(x.dtype) * self.weight
+
+
+def make_norm(d, cfg):
+    return RMSNorm(d, cfg.norm_eps) if cfg.norm == "rmsnorm" else nn.LayerNorm(d)
+
+
+def apply_rope(x, base=100000.0):
+    """Parameter-free RoPE on [B, H, T, D], interleaved convention (even/odd channel pairs)."""
+    _, _, T, dim = x.shape
+    pos = torch.arange(T, device=x.device, dtype=torch.float32)
+    freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=x.device, dtype=torch.float32) / dim))
+    ang = torch.outer(pos, freq)
+    cos, sin = ang.cos().to(x.dtype)[None, None], ang.sin().to(x.dtype)[None, None]
+    even, odd = x[..., ::2], x[..., 1::2]
+    return torch.stack((even * cos - odd * sin, even * sin + odd * cos), dim=-1).flatten(-2)
+
+
+class SwiGLU(nn.Module):
+    """Gated MLP: down(silu(gate(x)) * up(x))."""
+    def __init__(self, d, hidden):
+        super().__init__()
+        self.gate = nn.Linear(d, hidden, bias=False)
+        self.up = nn.Linear(d, hidden, bias=False)
+        self.down = nn.Linear(hidden, d, bias=False)
+
+    def forward(self, x):
+        return self.down(F.silu(self.gate(x)) * self.up(x))
+
+
+class Block(nn.Module):
+    def __init__(self, d, nh, block, cfg, is_first=False):
+        super().__init__()
+        self.ln1 = make_norm(d, cfg)
+        self.ln2 = make_norm(d, cfg)
+        self.qkv = nn.Linear(d, 3 * d)
+        self.proj = nn.Linear(d, d)
+        if cfg.ffn == "swiglu":
+            self.mlp = SwiGLU(d, int(round(cfg.ffn_mult * d)))
+        else:
+            self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
+        self.nh = nh
+        self.d = d
+        self.cfg = cfg
+        self.is_first = is_first
+        if cfg.value_residual and not is_first:
+            self.vr_lambda = nn.Parameter(torch.zeros(1))
+        if cfg.qk_norm:
+            hd = d // nh
+            self.q_norm = RMSNorm(hd, cfg.norm_eps)
+            self.k_norm = RMSNorm(hd, cfg.norm_eps)
+
+    def forward(self, x, v0=None):
+        B, T, D = x.size()
+        h = self.ln1(x)
+        q, k, v = self.qkv(h).split(self.d, dim=2)
+        hd = D // self.nh
+        q = q.view(B, T, self.nh, hd).transpose(1, 2)
+        k = k.view(B, T, self.nh, hd).transpose(1, 2)
+        v = v.view(B, T, self.nh, hd).transpose(1, 2)
+        if self.cfg.qk_norm:
+            q = self.q_norm(q)
+            k = self.k_norm(k)
+        if self.cfg.pos == "rope":
+            q = apply_rope(q, self.cfg.rope_theta)
+            k = apply_rope(k, self.cfg.rope_theta)
+        if self.cfg.value_residual:
+            if self.is_first:
+                v0 = v
+            else:
+                v = v + self.vr_lambda * v0
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        y = y.transpose(1, 2).contiguous().view(B, T, D)
+        x = x + self.proj(y)
+        x = x + self.mlp(self.ln2(x))
+        return x, v0
+
+
+class GPT(nn.Module):
+    def __init__(self, vocab, n_layer, n_embd, n_head, block, cfg):
+        super().__init__()
+        self.cfg = cfg
+        self.tok = nn.Embedding(vocab, n_embd)
+        self.use_rope = cfg.pos == "rope"
+        if not self.use_rope:
+            self.pos = nn.Embedding(block, n_embd)
+        self.blocks = nn.ModuleList([Block(n_embd, n_head, block, cfg, is_first=(i == 0)) for i in range(n_layer)])
+        self.lnf = make_norm(n_embd, cfg)
+        self.head = nn.Linear(n_embd, vocab, bias=False)
+        self.head.weight = self.tok.weight  # tied
+        self.block = block
+
+    def forward(self, idx):
+        B, T = idx.size()
+        x = self.tok(idx)
+        if not self.use_rope:
+            pos = torch.arange(T, device=idx.device)
+            x = x + self.pos(pos)[None]
+        v0 = None
+        for b in self.blocks:
+            x, v0 = b(x, v0)
+        logits = self.head(self.lnf(x))
+        cap = getattr(self.cfg, "logit_cap", 0.0)
+        if cap and cap > 0:
+            logits = cap * torch.tanh(logits / cap)
+        return logits
+
+
+def load_gollem_v6(path, device="cpu"):
+    """Model (eval mode) and tokenizer from a directory with config.json, model.safetensors and tokenizer.json."""
+    from safetensors.torch import load_file
+    from tokenizers import Tokenizer
+
+    with open(os.path.join(path, "config.json"), encoding="utf-8") as f:
+        cfg = types.SimpleNamespace(**json.load(f))
+    model = GPT(cfg.vocab, cfg.n_layer, cfg.n_embd, cfg.n_head, cfg.block, cfg)
+    model.load_state_dict(load_file(os.path.join(path, "model.safetensors")), strict=True)
+    return model.to(device).eval(), Tokenizer.from_file(os.path.join(path, "tokenizer.json"))
+
+
+@torch.no_grad()
+def generate(model, tok, prompt, max_new_tokens=64, temperature=0.0, top_k=None, seed=None):
+    """Continue `prompt`. temperature 0 = greedy; otherwise sampling, optionally top-k."""
+    g = torch.Generator().manual_seed(seed) if seed is not None else None
+    dev = next(model.parameters()).device
+    x = torch.tensor([tok.encode(prompt).ids], device=dev)
+    for _ in range(max_new_tokens):
+        logits = model(x[:, -model.block:])[:, -1].float().cpu()
+        if temperature <= 0:
+            nxt = logits.argmax(-1, keepdim=True)
+        else:
+            logits = logits / temperature
+            if top_k:
+                kth = torch.topk(logits, top_k).values[:, -1:]
+                logits = logits.masked_fill(logits < kth, float("-inf"))
+            nxt = torch.multinomial(torch.softmax(logits, -1), 1, generator=g)
+        x = torch.cat([x, nxt.to(dev)], dim=1)
+    return tok.decode(x[0].tolist())