winnow-olmoe-math-keep25-onpolicy

On-policy–polished variant of winnow-olmoe-math-keep25. Requires trust_remote_code=True (ragged variable-width experts). 2.09B params, 442/1024 experts deleted.

Results vs the off-policy checkpoint it started from

checkpoint	GSM8K	MATH500	IFEval	HumanEval	MBPP
off-policy step150 (base)	0.391	0.148	0.196	0.000	0.068
on-policy step300 (this)	0.487	0.178	0.299	0.024	0.072

GSM8K +9.6 pts, paired McNemar over all 1319 items (243 wrong to right, 116 right to wrong, net +127), two-sided exact p = 1.8e-11. MATH500 +3.0 pts, p = 1.3e-03. lm-eval-harness, gsm8k_cot_zeroshot flexible-extract, served via vLLM.

Control: the off-policy run had plateaued before the switch (GSM8K 0.365 / 0.408 / 0.391 at steps 50 / 100 / 150), so the gain is attributable to the objective change, not to additional optimizer steps.

Recipe

Warm-started from off-policy step150, then 150 on-policy steps (150 to 300): reverse KL on student rollouts, lr 1e-5, AdamW8bit, wd 0.1, grad clip 1.0, no reference anchor, gold CE rail 0.25 with 0.005/step decay against a combined cached-trajectory set, 256 prompts/step, group size 4, max 2048 new tokens, vLLM rollouts refreshed every step, seed 1223. Rollout finish rate held at 96-99%.