--- license: mit language: en library_name: transformers pipeline_tag: text-generation tags: - tiny-model - llama - from-scratch - conversational - multi-turn - tool-use - agent-harness - retrieval-augmented - question-answering - attribution - humble-ai - small-language-model - muon - gguf - text-generation-inference widget: - text: "\n\nwho are you\n<|eot|>\n\n" example_title: "Identity" - text: "\n\nwho wrote dracula\n<|eot|>\n\n" example_title: "Reads real prose" - text: "\n\nwhat did i eat yesterday\n<|eot|>\n\n" example_title: "Limits" ---
Loom Weave 3
# Loom Weave 3 **31.5M parameters · 16 layers · 1024 context · Textile Labs** The first Loom that **reads real prose**. Successor to [Loom Weave 2](https://huggingface.co/textilelabs/Loom-Weave-2) — and deliberately **smaller** than it (Weave 2 was 59.65M). Trained from scratch: randomly initialised weights, nothing fine-tuned from anyone's checkpoint. **It decides when to look something up, writes the query, reads what comes back from live Wikipedia, and answers from it — saying it checked rather than knew.** No earlier Loom could read real, messy prose; the family's whole weakness was that curated results read well and live articles didn't. This is the model that closes that gap. ``` you who wrote dracula Loom Weave 3 dracula harness ← Dracula is an 1897 Gothic horror novel by Irish author Bram Stoker. Loom Weave 3 I think Irish author Bram Stoker. Worth saying I checked rather than knew it. ``` ## Why it's smaller than Weave 2 Weave 2 (59.65M) failed for method, not size — a thin instruction veneer over a pretraining register, train and eval written by the same hand. Weave 3 is **half the size** and far better, because tokens-per-parameter and *how* it's trained matter more than parameter count. That has precedent: Loom Spark 3 (12.2M) succeeded and beat Spark 2 (19.9M). Bigger is not the goal; behaviour is. ## What changed The reading skill is a real circuit, and it only formed once two things were right: | lever | what went wrong before | fix | |---|---|---| | **learning rate on the attention matrices** | trained 8× too low, so the copy/reading circuit never formed — the model was fluent but couldn't pull "Paris" out of a sentence containing "Paris" | Muon LR set to the family standard (0.025); reading appeared | | **what the loss grades** | it graded every token equally, spending most of its effort on raw web text | loss is masked to the reply tokens; whole conversations packed into each block | | **how long it trained** | over-trained a tiny corpus into memorised, garbled output | short training, kept the best-validation checkpoint | | **admitting personal limits** | learned "I can't know your *name*" but not "I can't know what you *did*" — so it hallucinated on other personal questions | added varied personal-unknowable examples; it now declines all of them | ## The search harness The model decides a search is needed and writes the query. `harness.py` does the rest: - searches the model's query **and** the subject it can see in your question - prefers the real article over lists, films, albums and disambiguation pages - reads the intro first, further only when the intro has no answer of the right kind - strips brackets and pronunciation guides, so real text looks like training text - **hands back one sentence**, not a paragraph — a model this size misreads paragraphs ## Measured behaviour Same harness and settings a user gets. Run on 2026-09-20. **The acceptance battery**, row by row (total ties the best in the Loom family): | row | Loom Weave 3 | |---|---:| | A · says its own name | 10/12 | | B · its own name under rough typing | 10/12 | | C · 5-turn conversation stays on thread | 5/5 | | D · answers from a search result | **5/5** | | E · follow-up answered from the same result | 2/5 | | F · says it looked, after a lookup | 5/5 | | G · **never** claims a lookup it didn't make | **16/16** | | H · admits what it can't know about you | **8/8** | | I · says when a result doesn't contain the answer | 0/5 | | J · never leaks a search tag with tools off | 28/28 | | K · stops on its own | 12/12 | | L · searches when it should, not for your private things | 19/20 | | **total** | **120/133** | **End to end**, everyday questions it had never seen, live Wikipedia, the model writing its own query, scored on the final answer: | set | decided to search | wrote its own query | answer reached the model | **answered right** | |---|---:|---:|---:|---:| | tuning | 20/20 | 20/20 | 14/20 | **11/20** | | held-out | 20/20 | 20/20 | 12/20 | **6/20** | It is the first Loom to read live prose above a coin-flip on the tuning set — but read the held-out number honestly: on unfamiliar, messy passages it is right about a third of the time. ## Read this before you use it Every point here was measured. - **Held-out live reading is ~30%.** "I looked that up" means it searched — not that it read the result correctly. Run `harness.py --show` and trust the sentence it read. - **It is not a calculator.** Small sums are out of scope (0/6). Use a tool for arithmetic. - **Follow-up questions from the same result are weaker** (~40%). - **It never says a result doesn't contain the answer** — it answers from whatever it read. - **It rarely asks a clarifying question** on an ambiguous request. - **It reads a *supplied* result far better than it reads *live* search** (row D is 100%). - **Harness search is Wikipedia only** — time, weather, news and prices can't be answered. - **It is a small reader/assistant, not a chat stylist.** Voice is plain and brief by design. What it does do, reliably: knows what it is, **stops on its own**, holds a conversation, **admits what it can't know about you (8/8)**, and **never claims a lookup it didn't make (16/16)**. ## Usage — the harness ```bash python3 harness.py "who wrote dracula" python3 harness.py # interactive python3 harness.py --show "how tall is mount everest" # see what it searched and read python3 harness.py --no-tools "who are you" ``` Stdlib + `certifi`. Wikipedia needs no API key. **Never feed a failed lookup back as a result** — the model will answer from the error text; `harness.py` fails loudly instead. ## Usage — Ollama ```bash ollama run hf.co/textilelabs/Loom-Weave-3 "who are you" ``` `template` and `params` are read automatically. ## Usage — transformers ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("textilelabs/Loom-Weave-3") model = AutoModelForCausalLM.from_pretrained("textilelabs/Loom-Weave-3").eval() eot = tok.convert_tokens_to_ids("<|eot|>") def ask(message, tools=False): p = f"\n\n{message}\n<|eot|>\n\n" ids = tok(p, return_tensors="pt", add_special_tokens=False).input_ids with torch.no_grad(): out = model.generate(ids, max_new_tokens=64, do_sample=False, eos_token_id=eot, pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))[0] return tok.decode(out[ids.shape[1]:], skip_special_tokens=False).replace("<|eot|>", "").strip() ``` Prompt format is exact: `\n\n{message}\n<|eot|>\n\n`. ## How it was built | | | |---|---| | architecture | Llama — 16 layers × 384d, GQA (6 heads / 2 KV), SwiGLU, RoPE, tied embeddings | | parameters | 31,469,952 | | context | 1,024 | | vocabulary | 16,384 custom BPE | | optimiser | Muon (LR 0.025) on the 2D hidden matrices, AdamW on embeddings and norms | | loss | masked to the reply tokens; whole conversations packed per block (FFD) | | schedule | warmup → stable → cosine decay (WSD); best-validation checkpoint kept | | corpus | 173,043 conversations; grounded-reading rows carry real encyclopedic prose | | training | ~1,280 steps · ~336M tokens seen · 10 epochs · from random init | | hardware | Kaggle dual T4 (GPU) · ~2 hours | ## Files ``` config.json / model.safetensors the model tokenizer.json / tokenizer_config.json custom BPE tokenizer, 16,384 tokens loom-weave-3-f16.gguf for Ollama / llama.cpp (lookup tags USER_DEFINED) harness.py runnable search harness template / params read automatically by `ollama run hf.co/...` Modelfile for building locally ATTRIBUTION.md required credits for the training corpora ``` ## Training data | slice | source | |---|---| | grounded reading of real prose | **SQuAD 2.0** (CC BY-SA 4.0) · **Wikipedia** (CC BY-SA) | | when to reach for a tool | **MASSIVE** (CC BY 4.0) · **CLINC150** (CC BY 3.0) | | instruction following | **databricks-dolly-15k** (CC BY-SA 3.0) | | multi-turn dialogue structure | **OpenAssistant OASST1** (Apache 2.0) | | tokenizer coverage of real prose | **HuggingFaceFW/fineweb-edu** (ODC-By 1.0) | | identity, limits, warmth, attribution | Textile Labs — written for Loom | No language model wrote any training query, and nothing is fine-tuned from anyone's checkpoint. No real user data was used. See ATTRIBUTION.md. ## License Model: MIT. Training data retains its original licences and attribution.