# NanoChat D34 SFT - Project Context

This repository contains fine-tuned checkpoints and training data for NanoChat D34, a 2.2B parameter language model based on Karpathy's nanochat architecture.

## Repository Purpose

This is a **model artifacts repository** for Hugging Face, not a training codebase. It contains:
- Pre-trained model checkpoints (`.pt` files)
- Checkpoint metadata (`.json` files)
- Tokenizer files
- Custom training data used for fine-tuning

## Architecture Overview

```
Base Model (karpathy/nanochat-d34)
         ↓
    Mid-Training (813 steps)
         ↓
    SFT Training (700-719 steps)
         ↓
    This Repository
```

## Key Files

### Checkpoints (use with nanochat repo)
| File | Best For |
|------|----------|
| `model_000719.pt` | General use (best ARC-Easy: 73.9%) |
| `model_000700.pt` | Reasoning tasks (best MMLU: 42.6%) |
| `custom_only_model_000033.pt` | Analytics/SQL domain tasks |
| `full_custom_model_000055.pt` | Mixed general + domain tasks |
| `mid_model_000813.pt` | Starting point for different SFT |

### Training Data
- `training_data/combined_training_nanochat.jsonl` - 1,092 custom examples
- `training_data/full_combined_training_nanochat.jsonl` - 1,812 full dataset

Custom data focuses on **ClickHouse SQL queries for blockchain/trading analytics**.

## Working with This Repository

### To use the model
Clone the [nanochat repo](https://github.com/karpathy/nanochat) and copy checkpoints to `~/.cache/nanochat/`.

### To add training data
Training data format (JSONL, one conversation per line):
```json
[{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
```

### To retrain/fine-tune
1. Use the nanochat repository for training scripts
2. Start from `mid_model_000813.pt` for custom SFT experiments
3. Use training data files as templates for data format

## Model Configuration

All checkpoints share this architecture:
```json
{
  "sequence_len": 2048,
  "vocab_size": 65536,
  "n_layer": 34,
  "n_head": 17,
  "n_kv_head": 17,
  "n_embd": 2176
}
```

## Common Tasks

### Evaluate a checkpoint
```bash
# In nanochat repo
uv run python -m scripts.eval_mmlu --source=sft --model-tag=d34 --step=719
```

### Run inference
```bash
uv run python -m scripts.chat_cli --source=sft --model-tag=d34 --step=719 --temperature=0.6
```

### Add new training examples
1. Create JSONL file with conversation pairs
2. Run SFT training in nanochat repo
3. Upload new checkpoint here

## Related Repositories

- [karpathy/nanochat](https://github.com/karpathy/nanochat) - Training code and scripts
- [victoremnm/nanochat](https://github.com/victoremnm/nanochat) - Fork with setup guide
- [karpathy/nanochat-d34](https://huggingface.co/karpathy/nanochat-d34) - Base model

## Notes

- This is a private repository - requires HF authentication to clone
- Model is MIT licensed (same as base nanochat)
- Best for learning/experimentation, not production use