Polish SLM Benchmark

A leaderboard for small Polish language models, with raw task scores and parameter-based groups.

This benchmark was designed around small causal language models and may contain task-specific evaluation errors. See About for caveats and Reproduce for the reference scorer.
<1B paramsBase / Instruct / CPT / RL task primary scores + raw / normalized compositesLoading results…
Leaderboard

Models

# Model ↕ Author ↕ Grp ↕ Params ↕ Type ↕ Composite ↓ Norm.
composite ↕		 Polemo2 in ↕ Polemo2 out ↕ 8tags ↕ Belebele ↕ CBD ↕ DYK ↕ KLEJ NER ↕ PolQA ↕ PPC ↕ PSC ↕
Loading results.json…
About

About the benchmark

Polish SLM Benchmark compares Polish-capable language models below 1B parameters under one reproducible evaluation protocol.

Primary metric policy

Fixed semantic-label tasks use domain-matched PMI. Multiple-choice and reranking tasks keep direct likelihood ranking. Composite is the unweighted mean of the 10 primary task scores. Norm. Composite is an additional summary metric introduced with scoring policy v1.0.2. It standardizes every task against the frozen v1.0.1 reference leaderboard and then averages the 10 standardized task values. This makes tasks with different score ranges comparable without inventing manual task weights.

PolEmo2 IN / OUTdomain PMI → accuracy
8Tagsdomain PMI → accuracy
Belebelechoice likelihood → accuracy
CBDdomain PMI → macro-F1 (6 classes)
DYKdomain PMI → positive-class F1
KLEJ NERdomain PMI → accuracy
PolQAchoice likelihood → accuracy
PPCdomain PMI → accuracy
PSCdomain PMI → positive-class F1
Compositeunweighted mean of 10 primary scores
Norm. Compositemean task z-score vs frozen v1.0.1 reference pool
Norm. Composite — scoring policy v1.0.2. For task i, the normalized value is zᵢ = (scoreᵢ − μᵢ) / σᵢ, where μᵢ and σᵢ are the mean and population standard deviation of that task in the frozen v1.0.1 reference leaderboard. Norm. Composite is the unweighted mean of the 10 task z-scores. A value near 0 means performance close to the historical reference-model average; +1 means roughly one reference standard deviation above it on average, and −1 one below. It is a separate metric beside Composite, not a subtraction from it.
Frozen v1.0.1 normalization reference. The reference statistics are fixed when scoring policy v1.0.2 is introduced, so adding a new model does not change older Norm. Composite values. Reference pool: 14 models. Task statistics: polemo_in: μ=20.134, σ=5.805, polemo_out: μ=16.020, σ=11.586, tags8: μ=38.839, σ=9.140, belebele: μ=23.499, σ=1.198, cbd: μ=8.579, σ=3.994, dyk: μ=23.853, σ=4.510, klej_ner: μ=18.989, σ=4.457, polqa: μ=52.216, σ=1.643, ppc: μ=32.036, σ=8.455, psc: μ=30.476, σ=11.008.
Scorer v1.0.1 corrected domain-PMI blanking for CBD, PPC and PSC after an independent GoLLeM evaluation highlighted discrepancies. Scoring policy v1.0.2 adds Norm. Composite while leaving all primary task scores and the raw Composite unchanged. Scores are versioned with the scorer and should not be treated as infallible.
Scorer repository ↗ Report an evaluation issue ↗ GoLLeM evaluation ↗ KateMajzel / gollem-pl ↗
Reproduce

Reference scorer

Primary task scores are currently reproduced with scorer 1.0.1. Scoring policy 1.0.2 adds Norm. Composite using frozen v1.0.1 reference statistics; the underlying task scores remain unchanged.

Scoring policy v1.0.2

task_z = (primary_score - reference_mean) / reference_std
norm_composite = mean(task_z_1 ... task_z_10)

# reference_mean/reference_std are frozen from the v1.0.1 leaderboard
# 0.00  = historical reference-model mean
# +1.00 = one reference standard deviation above the mean
# -1.00 = one reference standard deviation below the mean
# Norm. Composite is separate from the raw Composite

Run a Hugging Face model

python .\Polish_SLM_Benchmark_v1.0.1.py `
  --source hf `
  --model "OrisTeam/Koliber-v1.0-Base-Preview" `
  --device cuda:0 `
  --dtype bf16 `
  --batch-size 8 `
  --out ".\results_koliber"
GitHub repository ↗ Scorer source ↗ SpeakLeash lm-eval ↗
View current task scorer source — v1.0.1 
Open this section to load the scorer source from GitHub.
Polish SLM Benchmark