Multi-Teacher Distillation Dataset (57,937 traces)

A quality-filtered, deduplicated, multi-teacher SFT corpus combining traces from three frontier models across math, code, reasoning, instruction-following, tool-use, science, long-context, multilingual, and creative dialogue domains.

Teachers

Quick Start

General SFT (recommended entrypoint)

from datasets import load_dataset

# Default config — rebalanced sampling weights
ds = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "sft_balanced")

for batch in ds["train"]:
    messages = batch["messages"]       # list of {role, content, reasoning_content, tool_calls, trainable}
    tools = batch["tools"]             # list of tool definitions
    weight = batch["sampling_weight"]  # float — use in WeightedRandomSampler
    # ... feed to your trainer with responses-only loss masking

Each message in the messages list has this structure:

{
    "role": "assistant",               # system | user | assistant | tool
    "content": "The answer is 30.",    # final text output
    "reasoning_content": "Buzz and...",# chain-of-thought (empty for non-reasoning turns)
    "tool_calls": [],                  # [{id, type, function: {name, arguments}}]
    "tool_call_id": "",                # for tool-role messages
    "name": "",                        # optional tool name
    "trainable": True                  # True only on assistant turns
}

Native GLM-4.7-Flash training (pre-tokenized)

Skip the chat template entirely — input_ids and labels are already computed:

import pyarrow.dataset as ds

table = ds.dataset(".../data/glm47_native", format="parquet")

# Each row:
#   input_ids:          list[int] — exact token IDs from GLM-4.7-Flash tokenizer
#   labels:             list[int] — same length, -100 on non-assistant tokens
#   total_tokens:       int
#   assistant_tokens:   int       — count of labels != -100
#   mask_method:        str       — "prefix-differential" or "structural-redaction"
#   renderer_revision:  str       — pinned tokenizer version

# Train with standard causal LM loss:
#   loss = CrossEntropyLoss(model(input_ids), labels, ignore_index=-100)

Tool-use / agent training

ds = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "sft_tools")
# 5,909 rows with real tool-call trajectories (not JSON-in-content)

# Then RL on deterministic tool tasks:
rl = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "rl_tool_prompts")
# prompt_messages_json + tools_json + reward_contract_json

Dataset Summary

Build pipeline

Base dataset (52,083) ── freeze + inventory
                         ├── K3 supplement: 4,354 accepted → 694 already in base → 3,660 new
                         ├── Qwen V4 production: 2,131 accepted + 70 canary
                         └── raw merge upper bound: 57,944
                         ↓
                    normalize → quality filter → deduplicate
                         ↓
                    57,937 canonical rows (7 near-dups removed)
                         ↓
                    capability taxonomy → mixture analysis → sampling policies
                         ↓
                    exact retokenization (GLM/Qwen/Llama) → 24 training configs

Domain Distribution

Normalized domains

Source breakdown (top 20)

Capability Mixture

Under sft_balanced sampling weights, the effective mixture at training time shifts:

Configs (24 Parquet views)

Training configs

Ablation configs

Infrastructure configs

Column Schemas

sft_balanced / sft / specialist sft_* configs

glm47_native (pre-tokenized)

token_stats (exact multi-tokenizer)

Contains total/assistant token counts for GLM-4.7-Flash, Qwen3-0.6B, and Llama-3.1-8B per row. Key columns: glm47_total_tokens, glm47_assistant_tokens, glm47_loss_ratio, glm47_context_bucket, qwen3_*, llama31_final_*, max_validated_tokens.

canonical (full audit view)

All columns from the schemas above plus: teacher_model, teacher_provider, trace_kind, disposition, ground_truth_json, verifier_passed, family_oracle_passed, reference_agreement, format_score_raw, quality_flags, source_record_hash, prompt_cluster_id, template_cluster_id, dedup_cluster_id, schema_version, source_repository, source_revision, source_license, source_split, source_item_id.

rl_tool_prompts

Token Statistics

All 57,937 rows were retokenized with three pinned tokenizers:

GLM-4.7-Flash context bucket distribution:

Loss mask methods: prefix-differential (most rows — prefix-preserving chat template) and structural-redaction (multi-turn tool rows — diff-based mask). Zero mask failures.

Loss Masking

Two methods, automatically selected per row:

1. Prefix-differential: The chat template is prefix-preserving (adding a new message doesn't change earlier tokens). Labels are assigned by diffing successive template renders — tokens from assistant messages get their real IDs, all others get -100.

2. Structural-redaction: For complex multi-turn tool conversations where the template isn't prefix-preserving. Each assistant payload is redacted in a copy of the conversation, and the diff between full and redacted renders identifies assistant tokens.

Both methods guarantee: no user/system/tool content in the loss, no fabricated labels, at least one proven assistant span per row.

Deduplication

Three layers applied:

1. Exact identity hash — canonical hash of normalized (system, user turns, tools, fixture)
2. Semantic near-dup — 5-gram Jaccard >= 0.9 using blake2b shingles
3. Prompt cluster overlap — rows sharing a prompt cluster with base dataset are deduped

Result: 7 near-duplicates removed from 5,861 eligible supplement rows; zero exact duplicates in final set.

Quality Filtering

K3 double-count guard: 694 K3 rows already in base dataset were identified by ID intersection and excluded from supplements (only 3,660 new K3 rows admitted).

Curriculum Recommendations

The curriculum_stages.json in the hub package defines a multi-stage schedule:

Provenance and License

This dataset is a synthetic research corpus built from teacher model outputs. Each row carries full provenance in the canonical config: source_repository, source_revision, source_license, source_item_id, teacher_model, teacher_provider.

Base dataset rows (52,083) originate from public sources including MetaMathQA (MIT), GSM8K (MIT), SciQ (CC0), ARC (CC-BY-SA), OpenBookQA (CC-BY-SA 4.0), CommonsenseQA (CC-BY-SA), QASC (Apache-2.0), CodeAlpaca, Evol-Code, tulu-3, Dolly, MATH, NuminaMath-CoT, OrcaMath. Supplement rows from K3 (kimi-code/k3) and Qwen V4 (qwen3.8-max-preview) are labeled other; synthetic research corpus.

The repo name retains the historical 51389 from the original v1 release; the actual row count is 57,937.

Known Limitations

• Tools (10.3%) and long-context (2.9%) are below SOTA floors by inventory. The sft_balanced config uses sampling weights to compensate at train time, but genuine coverage gaps remain until additional generation completes.
• Multilingual (0.8%) is critically thin; domain-only classification without a language detector means some multilingual content in base instruction rows may not be tagged.
• Reasoning is over-indexed (84.6%) — appropriate for reasoning-forward students, but generalist trainings should use sft_balanced weights or supplement with non-reasoning data.
• Single-turn dominates (86.7%) — sft_balanced down-weights single-turn to ~56% effective share.
• Full exact tokenization covers GLM-4.7-Flash, Qwen3, and Llama-3.1. Other tokenizers require applying your own chat template to the sft configs.
• This package is a local pre-release (rc3). The qualitative review of the compile is in progress.

Citation

If you use this dataset, cite the teacher models and upstream sources that made it possible:

@misc{qwen38,
  title={Qwen3.8-Max-Preview},
  author={Alibaba Cloud},
  year={2026}
}
@misc{glm52,
  title={GLM-5.2},
  author={Z.AI},
  year={2026}
}
@misc{kimi-k3,
  title={Kimi Code K3},
  author={Moonshot AI},
  year={2026}
}
@misc{metamathqa,
  title={MetaMathQA: Bridging Visual and Textual Reasoning},
  author={Yu, Shuo and others},
  year={2024},
  url={https://github.com/meta-math/MetaMath}
}