A modified GPT-2 model with only 25 million non-embedding params that outbenches GPT-2(124m), Pythia-70m/160m, and Cerebras-111m, it has ScaledSinusoidal position embeddings, embedding layernorm, no biases, and was trained on only 8 billion tokens of the SlimPajama dataset at home on 2xA6000. (On the graphic it's mis-labeled as cramp-41m)
OLD BENCHMARK
| model | avg | arc | hellaswag | mmlu | truthfulqa |
|---|---|---|---|---|---|
| cramp-25m | 30.57 | 21.76 | 27.35 | 25.53 | 47.66 |
| gpt2 (125m) | 30.06 | 22.1 | 31.6 | 25.86 | 40.67 |
| pythia 70m deduped | 30.25 | 21.08 | 27.17 | 25.26 | 47.51 |
| pythia 70m | 30.46 | 21.59 | 27.29 | 25.9 | 47.06 |
| pythia 160m deduped | 31.16 | 24.06 | 30.34 | 24.95 | 44.34 |
| pythia 160m | 30.58 | 22.78 | 30.34 | 24.95 | 44.26 |
*NEW BENCHMARK
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 25 | acc | 0.1724 | ± | 0.0110 |
| none | 25 | acc_norm | 0.2031 | ± | 0.0118 | ||
| truthfulqa_mc2 | 2 | none | 0 | acc | 0.4767 | ± | 0.0156 |
| hellaswag | 1 | none | 10 | acc | 0.2687 | ± | 0.0044 |
| none | 10 | acc_norm | 0.2773 | ± | 0.0045 | ||
| winogrande | 1 | none | 5 | acc | 0.5028 | ± | 0.0141 |
MMLU
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| world_religions | 0 | none | 5 | acc | 0.1813 | ± | 0.0295 |
| virology | 0 | none | 5 | acc | 0.1928 | ± | 0.0307 |
| us_foreign_policy | 0 | none | 5 | acc | 0.2900 | ± | 0.0456 |
| sociology | 0 | none | 5 | acc | 0.2438 | ± | 0.0304 |
| security_studies | 0 | none | 5 | acc | 0.2367 | ± | 0.0272 |
| public_relations | 0 | none | 5 | acc | 0.2455 | ± | 0.0412 |
| professional_psychology | 0 | none | 5 | acc | 0.2271 | ± | 0.0169 |
| professional_medicine | 0 | none | 5 | acc | 0.4375 | ± | 0.0301 |
| professional_law | 0 | none | 5 | acc | 0.2490 | ± | 0.0110 |
| professional_accounting | 0 | none | 5 | acc | 0.2589 | ± | 0.0261 |
| prehistory | 0 | none | 5 | acc | 0.2963 | ± | 0.0254 |
| philosophy | 0 | none | 5 | acc | 0.2315 | ± | 0.0240 |
| nutrition | 0 | none | 5 | acc | 0.2222 | ± | 0.0238 |
| moral_scenarios | 0 | none | 5 | acc | 0.2313 | ± | 0.0141 |
| moral_disputes | 0 | none | 5 | acc | 0.2168 | ± | 0.0222 |
| miscellaneous | 0 | none | 5 | acc | 0.2708 | ± | 0.0159 |
| medical_genetics | 0 | none | 5 | acc | 0.3000 | ± | 0.0461 |
| marketing | 0 | none | 5 | acc | 0.1923 | ± | 0.0258 |
| management | 0 | none | 5 | acc | 0.1942 | ± | 0.0392 |
| machine_learning | 0 | none | 5 | acc | 0.2054 | ± | 0.0383 |
| logical_fallacies | 0 | none | 5 | acc | 0.2393 | ± | 0.0335 |
| jurisprudence | 0 | none | 5 | acc | 0.2130 | ± | 0.0396 |
| international_law | 0 | none | 5 | acc | 0.2562 | ± | 0.0398 |
| human_sexuality | 0 | none | 5 | acc | 0.2366 | ± | 0.0373 |
| human_aging | 0 | none | 5 | acc | 0.2063 | ± | 0.0272 |
| high_school_world_history | 0 | none | 5 | acc | 0.2700 | ± | 0.0289 |
| high_school_us_history | 0 | none | 5 | acc | 0.2206 | ± | 0.0291 |
| high_school_statistics | 0 | none | 5 | acc | 0.4722 | ± | 0.0340 |
| high_school_psychology | 0 | none | 5 | acc | 0.2257 | ± | 0.0179 |
| high_school_physics | 0 | none | 5 | acc | 0.2384 | ± | 0.0348 |
| high_school_microeconomics | 0 | none | 5 | acc | 0.3403 | ± | 0.0308 |
| high_school_mathematics | 0 | none | 5 | acc | 0.2630 | ± | 0.0268 |
| high_school_macroeconomics | 0 | none | 5 | acc | 0.2051 | ± | 0.0205 |
| high_school_government_and_politics | 0 | none | 5 | acc | 0.2280 | ± | 0.0303 |
| high_school_geography | 0 | none | 5 | acc | 0.3535 | ± | 0.0341 |
| high_school_european_history | 0 | none | 5 | acc | 0.2909 | ± | 0.0355 |
| high_school_computer_science | 0 | none | 5 | acc | 0.2400 | ± | 0.0429 |
| high_school_chemistry | 0 | none | 5 | acc | 0.2759 | ± | 0.0314 |
| high_school_biology | 0 | none | 5 | acc | 0.3161 | ± | 0.0265 |
| global_facts | 0 | none | 5 | acc | 0.2000 | ± | 0.0402 |
| formal_logic | 0 | none | 5 | acc | 0.1825 | ± | 0.0346 |
| elementary_mathematics | 0 | none | 5 | acc | 0.2566 | ± | 0.0225 |
| electrical_engineering | 0 | none | 5 | acc | 0.2414 | ± | 0.0357 |
| econometrics | 0 | none | 5 | acc | 0.2544 | ± | 0.0410 |
| conceptual_physics | 0 | none | 5 | acc | 0.2809 | ± | 0.0294 |
| computer_security | 0 | none | 5 | acc | 0.2000 | ± | 0.0402 |
| college_physics | 0 | none | 5 | acc | 0.3431 | ± | 0.0472 |
| college_medicine | 0 | none | 5 | acc | 0.2197 | ± | 0.0316 |
| college_mathematics | 0 | none | 5 | acc | 0.3100 | ± | 0.0465 |
| college_computer_science | 0 | none | 5 | acc | 0.3100 | ± | 0.0465 |
| college_chemistry | 0 | none | 5 | acc | 0.3400 | ± | 0.0476 |
| college_biology | 0 | none | 5 | acc | 0.2083 | ± | 0.0340 |
| clinical_knowledge | 0 | none | 5 | acc | 0.2189 | ± | 0.0254 |
| business_ethics | 0 | none | 5 | acc | 0.2000 | ± | 0.0402 |
| astronomy | 0 | none | 5 | acc | 0.2237 | ± | 0.0339 |
| anatomy | 0 | none | 5 | acc | 0.3333 | ± | 0.0407 |
| abstract_algebra | 0 | none | 5 | acc | 0.2200 | ± | 0.0416 |
- Downloads last month
- 7