# SuperSmallJokeClaude

A 1.64M-parameter tiny transformer trained from scratch on Fable 5 Claude Code assistant traces.

## Specs
- **Params:** 1,644,032 (1.64M)
- **Architecture:** 4-layer transformer, d_model=128, 4 heads, d_ff=256, tied embeddings
- **Vocab:** BPE-8192 (from Claude Code tokenizer)
- **Context:** 512 tokens
- **Training data:** 24,249 tokens (Fable 5 Claude Code assistant traces)
- **Training:** 2000 steps, batch 8, seq 256, AdamW lr=3e-4 cosine schedule
- **Final loss:** 2.90
- **Training time:** 7 seconds on RTX 5090

## What it does

It's a joke — a tiny model trained on tiny data. It will produce degenerate output (repeated tokens, gibberish). It exists to demonstrate that you can train a working transformer in 7 seconds on a single GPU.

## Files
- `final.pt` — full checkpoint (model state dict + config + training metadata)
- `tokenizer.json` — BPE-8192 tokenizer (Claude Code)

## Load it

```python
import torch
from tokenizers import Tokenizer

ckpt = torch.load('final.pt', map_location='cpu')
config = ckpt['config']

# Build model (see train_jokeclaude_v5.py for the class definition)
model = Transformer(**config)
model.load_state_dict(ckpt['model_state_dict'])
model.eval()

tokenizer = Tokenizer.from_file('tokenizer.json')
```

## Disclaimer
This is an experimental/fun artifact. The model is not useful for any real task. It exists because someone asked for a "super small joke claude" and we said why not.