Snowball-67B-A2B-Identity-CD16-2026.10.26

Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.

Status: research checkpoint, not a release. Tracking issue: marin-community/marin#10695.

What it is

Marin Snowball 67B-A2B v0.1 @ adc0078a19a4383bc998fcdb43867b3360baf5f3, plus 16 supervised fine-tuning updates of context distillation. The teacher is v0.1 itself, prompted with the first-person identity system prompt "I am Marin Snowball, an AI model created by Open Athena." The student is trained to give those answers without the system prompt.

About 96% of the supervised tokens in the 16 updates come from non-identity rows. Nearly all of these (95% of all tokens) are v0.1's own answers, replayed to preserve its behaviour. The rest are third-person and persona control rows. The identity rows are under 4% of the tokens.

All 502 tensors are exported in v0.1's dtypes: 476 BF16 weights and 26 F32 router biases.

Training data: open-athena/Snowball-67B-A2B-Identity-CD16-Data-2026.10.26. It has every training row, the training order (only positions 0–127 were used), the mixture settings and the quality-control verdicts, so the data can be improved and the checkpoint retrained.

Recipe

Training code: Marin at commit cc127d54 (tag snowball-identity-cd16-2026.10.26), experiments/post_training/identity_context_distill/ (sft.py, sft_launch.py, runtime_recipe.py, PRECISION_RUNTIME.md). The data-generation and QC scripts are not in the Marin repository; they are in the dataset repo under scripts/, as as-run research code. Part of the capture library used for data generation was not committed; scripts/capture_lib/ is a later working-tree snapshot, and its key sampling values were cross-checked against every recorded request.

1. Base. open-athena/Marin-Snowball-67B-A2B-v0.1 at adc0078a19a4383bc998fcdb43867b3360baf5f3.

2. Identity data (220 rows).

• Teacher and system prompt: the teacher is v0.1 with the system prompt "I am Marin Snowball, an AI model created by Open Athena." One row uses "I am Marin Snowball, an AI model. Open Athena created me." The system prompt goes into the teacher request only. The student row keeps just the user turn and the teacher's answer.
• Sampling:
  • temperature 0.7, top_p 0.95, at most 8,192 completion tokens;
  • one completion per request, with repeated samples sent as separate requests;
  • seed = a per-capture offset + the sample index;
  • student context limit 8,192 tokens.
• Captures (rows in the final set): six in total.

Capture	Thinking	Samples per prompt	Prompts	Sampled	Passed the automatic filter	Rows used
Off-mode, name and creator	off	16 (creator) or 4 (name/negation)	83	740	369	145
On-mode, name and negation	on	408–409	49	20,000	306	55
Main wording-D	on	40	83	3,320	37	13
Creator top-up	on	86–87	29	2,500	26	2
Wording pilot (five system wordings; only wording D kept)	on	4–8 per wording	26	720	8 (wording D)	2
Creator pilot (two wordings)	on	16 per wording	34	1,088	8 + 8	3

• Automatic filter (version 4, revision 2). It checks both the answer and the reasoning. It rejects any of these:
  • an incomplete or empty answer, or reasoning present in thinking-off mode;
  • a wrong identity in the answer or the reasoning;
  • a repetitive trace (an 8-word window repeated 8 or more times);
  • a sequence over 8,192 tokens;
  • any mention of a system prompt or instructions, or the teacher's system text copied;
  • no first-person name in the reasoning (thinking on);
  • a non-canonical spelling of "Marin Snowball" or "Open Athena";
  • invented aliases, negative self-identity predicates, other labs' identities, and unsupported extra identity facts.
• Selection guard. It also rejects unverified organisation or process descriptors, facts outside an allowlist, links to other labs or models, and project-versus-model confusion. The only allowed target facts are the name, "Open Athena" as creator, and at most "the team at Open Athena".
• Caps:
  • at most 4 copies of any exact input + thinking mode, after removing exact duplicate rows;
  • in thinking-off mode, at most 24 rows whose answer is exactly "I am Marin Snowball, an AI model created by Open Athena." (23 are used).
• Human QC:
  • Thinking on: 396 candidate records were read in full (306 new and 90 earlier candidates re-checked under the current filter). 201 were kept; deduplication and the cap left 75.
  • Thinking off: 291 unique candidates were read. 158 were kept; the cap left 145.
  • A second reviewer signed off the final merge of each.

3. Replay and control data (547 rows).

• Generation: plain v0.1 with no system prompt; the same sampling as above (temperature 0.7, top_p 0.95, max 8,192 tokens), one sample per prompt and mode.
  • Replay: 579 prompts, thinking on.
  • Controls: 48 prompts (persona, third-person and negation), thinking on and off.
  • That makes 675 requests.
• Prompt authoring:
  • 640 replay prompts were written blind (192 math, 192 code, 192 chat, 64 fiction). 616 passed a semantic review.
  • 579 remained after an overlap check against 5,377 public benchmark prompt records: GSM8K test, HumanEval, AIME 2024 and 2025, MATH500, MBPP and a LeetCode snapshot. The check used nearest-neighbour and phrase search, not an all-pairs comparison. LiveCodeBench was not checked.
• QC:
  • Every row has a human verdict. Math was checked by hand; there are no gold answers and no code was run.
  • Automatic guard flags (structural problems, unrequested self-identity or creator claims, claims about absent instructions) must be cleared before a row can be kept.
  • The negation controls (6 rows) were excluded by a project ruling: silent compliance with a foreign salutation is a poor target.
  • Result: 547 kept, 122 dropped, 6 excluded.

4. Mixture.

• Component weights: identity 264 and preservation 547, so each identity row is sampled 1.2× as often as a preservation row.
• Shuffle: seed 310, Feistel permutation, mixture block 2,048.
• Format: chat with the base chat template; user turns masked; unpacked.
• The 16 updates used the first 128 positions of the training order. Identity rows are 3,383 of 87,905 supervised tokens (3.85%).

5. Training.

• Framework: Levanter chat SFT at commit cc127d54.
• Optimizer:
  • AdamW, learning rate 2e-6, a constant schedule with a 2-step linear warmup;
  • weight decay 0, gradient clipping 0.5;
  • betas 0.9 / 0.95 and epsilon 1e-8, which are the code defaults.
• Batch and steps: batch size 8, sequence length 8,192, 16 updates (128 sequences).
• Precision:
  • parameters in FP32, compute in BF16;
  • dropless MoE (capacity factor 4.0, 0 dropped tokens recorded);
  • FP32 router combine, FP32 activations, FP32 router dot product, torch_order attention;
  • router biases computed in FP32 and frozen.
• Hardware: 4 nodes × 8 H100 (CoreWeave), 24.7 H100-hours of allocation.

6. Export.

• The last step (step 15, counting from 0) is saved to Hugging Face format in BF16, with the 26 router biases in FP32.
• An audit confirms that the 476 BF16 tensors equal casts of the trained FP32 weights, and that the 26 router biases equal the source.

7. Evals.

• Teacher-forced identity gate, a generation screen (identity, third-person, fiction and persona controls), and paired GSM8K / MATH500.
• The 14-eval suite: MMLU-Pro, GPQA Diamond, IFBench, HumanEval+, MBPP+, MATH500, GSM8K, AIME24, TriviaQA, TruthfulQA, Sotopia-hard, BFCL parity, SWE-bench Verified 100 and OpenThoughts-TBLite 100.
• Details: marin-community/marin#10695.

Identity

Teacher-forced negative log-likelihood of the correct continuation, thinking off, no system prompt, nats per token:

	v0.1	This checkpoint
Name ("Marin Snowball")	8.85	1.45
Creator ("Open Athena")	14.02	0.011

In generation without a system prompt, it names its creator much more often, but its own name only sometimes. On 52 checkable identity prompts it names Open Athena in 38 with thinking off (36 with thinking on), against 0 for v0.1. It calls itself "Snowball" in 2 (6 with thinking on).

Preserved capabilities

On a 14-eval suite (same evaluator code and serve settings as v0.1), these are within noise of v0.1: MMLU-Pro, GPQA Diamond, MATH500, GSM8K, AIME24, HumanEval+, MBPP+, IFBench, SWE-bench Verified (100 tasks), OpenThoughts-TBLite (100 tasks) and Sotopia-hard (−1.4 points, 95% CI [−3.4, +0.5]). TriviaQA improves by 2.4 points. SWE-bench and TBLite use 100 tasks each, so their 95% CIs are about ±8 points; small changes there cannot be detected.

Known regressions

• Function calling: BFCL parity falls by about 20 points (27.6% → 7.3% against v0.1 with the identity system prompt, same evaluator). The model often ends its turn without making a tool call: 109 of 123 trials end without an answer file, against 53 for v0.1 with the system prompt.
• Third-person questions: some questions about other companies' products, such as OpenAI's, get answered as if the product were made by Open Athena. Without a system prompt this happens on 2 of 14 development questions and 4 of 30 unseen ones (v0.1: 0). With the identity system prompt it happens on 23 of the 30 unseen ones (v0.1 with the same system prompt: 1).
• TruthfulQA: −1.4 points (95% CI excludes 0).

We release our tools to the public with the intent that they are used safely, ethically, and responsibly. They are meant to contribute to the public's understanding of science, and aid in its advancement. By using this Marin model, you agree to comply with all applicable law.