namezz/soft-prompt-interpretability-six-pairs

Six soft-prompt research cases

Prompt-only research checkpoints accompanying the project introduction. No full base model weights, optimizer states or raw training/evaluation answers are included.

63 saved checkpoints: optimizer update zero, every recorded 20-update interval, and final updates. These are not every optimizer step. Each file holds one finite 2D prompt_embeddings tensor, verified against its original training-manifest SHA-256.

Evidence limitations

GPT-OSS uses a corrected-routing evaluation curve. Mellum uses greedy single-sample evaluation outside the identified stochastic DP duplicate defect. Poro, MiMo math, Nemotron code and AReaL8 scores are historical and duplicate-affected, not validated independent-sampling evidence. Best means maximum on the recorded curve, not independent validation or repeated-seed stability. Do not claim all six are stable near/above 100%.

The original math training manifests did not record resolved base-model revisions. Preserve that uncertainty instead of treating current upstream main as historical identity. Code model commits/subfolders are in manifest.json. All prompts have 256 positions; widths differ across models. u0 is an initialized prompt, not the no-prompt student baseline.

Files and use

CHECKPOINTS.md lists every saved point; manifest.json binds each weight path and hash to its score and status. Pair-level CSV/JSON files are also provided. The final checkpoint is not always best. Configs and model IDs are provided, but exact template/position/attention behavior must use the project code.

Download with authenticated Hugging Face access to this private repository. Use a fixed commit revision for reproducibility:

from huggingface_hub import snapshot_download
path = snapshot_download(repo_id="namezz/soft-prompt-interpretability-six-pairs", revision="<commit-sha>")

Read one tensor without downloading the base model:

python inspect_prompt.py --pair math-gptoss --update 180

CPU dependencies: PyTorch and safetensors. For actual inference, consult the project environment and export/evaluation implementation; the inspector is not an inference runner. GPT-OSS needs sink-preserving attention. Code execution must remain isolated. No new upstream-model redistribution license is granted here; follow the upstream model/data terms before redistribution or use.

Included cases

ID	Best update	Best recovery (%)	Score status
math-gptoss	180	92.62	corrected_routing
math-poro	160	135.47	historical_duplicate_affected
math-mimo	140	90.15	historical_duplicate_affected
code-nemotron	184	101.21	historical_duplicate_affected
code-mellum	138	110.34	greedy_outside_duplicate_defect
code-areal8	100	125.00	historical_duplicate_affected