---
language: en
license: apache-2.0
tags:
  - tiny
  - slm
  - small-language-model
  - from-scratch
  - llama
  - rope
  - swiglu
  - rmsnorm
  - text-generation
  - tinystories
pipeline_tag: text-generation
library_name: transformers
---

# StoryLM-10M

A ~12.6M parameter LLaMA-style causal language model trained from scratch on TinyStories.

## Architecture

| Parameter | Value |
|-----------|-------|
| d_model | 256 |
| n_heads | 8 (MHA) |
| n_kv_heads | 8 |
| n_layers | 8 |
| head_dim | 32 |
| FFN dim | 1024 (SwiGLU, 4x) |
| Vocab size | 8192 |
| Context length | 512 |
| Tied embeddings | Yes |
| Total params | 12,603,648 |

## Training

- **Data**: TinyStories (2,119,719 stories, 533.7M tokens)
- **Steps**: 24,400 (batch 4, seq 512)
- **Optimizer**: AdamW, LR 3e-4, cosine schedule, 500-step warmup
- **Best val loss**: 1.8635
- **Val perplexity**: 6.45

## Generation Samples

```
Prompt: Once upon a time, there was a little
Output: Once upon a time, there was a little

Prompt: The cat sat on the
Output: The cat sat on the

Prompt: In the beginning, the world was
Output: In the beginning, the world was

Prompt: A small robot named
Output: A small ro bot named

Prompt: Every morning, the sun
Output: Every morning, the sun
```

## Notes

- Trained on a single RTX 5090 (32 GB) in ~281 seconds of training time.
- The model is a custom implementation (not HuggingFace transformers-compatible out of the box).
- Weights are stored as a single PyTorch checkpoint (`model.pt`).