---
license: apache-2.0
language: en
pipeline_tag: text-generation
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - SLM
  - small-language-model
  - sub-3m
  - from-scratch
  - bpe
  - rope
  - gqa
  - swiglu
  - banana-mind
metrics:
  - accuracy
  - perplexity
---

# BananaMind-IDK-3M

A 2,715,984-parameter small language model trained from scratch on 8B tokens of FineWeb-Edu, requested by @Banaxi-Tech and trained by @Compactbot.

## Architecture

| Parameter | Value |
|-----------|-------|
| Params (learnable) | 2,715,984 |
| Hidden dim (d) | 144 |
| Layers | 6 |
| Query heads | 2 |
| KV heads | 1 (GQA) |
| Head dim | 80 |
| FFN | SwiGLU, 3× expansion |
| Position encoding | RoPE |
| Normalization | RMSNorm |
| Vocab size | 8,192 (BPE) |
| Tied embeddings | Yes |

## Training

- **Data**: 8B tokens of FineWeb-Edu (cycled from a 3.6 GB local sample)
- **Optimizer**: AdamW, lr 5e-3, cosine decay with 200-step warmup
- **Batch**: 4 × grad-accum 8 = effective 32, seq 2048
- **Steps**: 122,070 (reached the 8B-token target)
- **Hardware**: RTX 5090 (shared, batch reduced to fit ~2 GB free VRAM)

## Evaluation (zero-shot, loglikelihood)

| Benchmark | Score |
|-----------|-------|
| PIQA | 53.7% |
| ARC-Easy | 30.77% |
| ARC-Challenge | 16.47% |
| HellaSwag | 26.01% |
| ArithMark-3.0 | — (not scored) |

These are honest numbers for a 2.7M-param model. For reference, chance is 50%/25%/25%/25% respectively.

## Known limitations

This model produces **degenerate greedy generation** — outputs collapse into repetition loops after the first sentence. It learned token-level statistics (grammatical first sentences) but not enough structure to sustain coherent multi-sentence generation. This is expected at 2.7M params even with 8B tokens of training data.

The model is published as a research artifact / data point, not as a usable generation model.

## Files

- `final.pt` — full checkpoint (state_dict + optimizer + step), load with `torch.load('final.pt', map_location='cpu')['model']`
- `config.json` — architecture config
- `tokenizer.json` — BPE tokenizer (8192 vocab, 5922 merges)
- `tokenizer_config.json` — tokenizer settings

## Usage

```python
import torch

ckpt = torch.load('final.pt', map_location='cpu')
state_dict = ckpt['model']  # 128 tensors, 2,715,984 learnable params
```

The model uses a custom architecture (GQA attention + SwiGLU FFN + RoPE + RMSNorm). A loading script is not included; the state_dict keys follow the pattern:
- `model.embed_tokens.weight` [8192, 144]
- `model.layers.{i}.{attn|mlp|norm}.*` for i in 0..5
- `model.norm.weight` [144]
- (lm_head is tied to embed_tokens)