vibethinker-3b-mxfp8-mlx

MLX quantization of WeiboAI/VibeThinker-3B for Apple Silicon.

Variant: Block float MX FP8
Disk size: 3046 MB
Quantized by: sahilchachra

Benchmark results

Evaluated on Apple M5 Pro with MLX. Model loaded once; performance and quality measured in a single pass.

Performance

	This model	FP16 baseline
Decode tok/s (steady-state)	78.68	45.34
Prefill tok/s (steady-state)	542.02	341.91
Decode tok/s (avg, long traces)	70.12	44.08
Peak memory (GB)	4.084	6.936
Disk size (MB)	3046	5902

Warmed, short-prompt, chat-templated, thinking disabled. Represents steady-state decode for typical chat use; long thinking traces will be slower due to KV-cache growth.

Quality

Benchmark	This model	FP16 baseline	n
MATH-500 (math reasoning)	96.7% (answered 30/30)	93.3% (answered 30/30)	30
AIME 2024 (competition math)	66.7%	76.7%	30
IFEval (instruction following)	59.1%	61.4%	44
HumanEval (code, pass@1)	66.7%	73.3%	30
MMLU (knowledge, accuracy)	36.0%	40.0%	50

MATH-500 per-level accuracy

Level	This model	FP16 baseline
level 1	100.0%	100.0%
level 2	100.0%	100.0%
level 3	83.3%	83.3%
level 4	100.0%	100.0%
level 5	100.0%	83.3%

Context scaling (decode tok/s)

Context length	Decode tok/s
~128 tokens	74.2
~256 tokens	74.1
~512 tokens	74.1
~1024 tokens	73.3

Prompt format & recommended settings

VibeThinker-3B is a reasoning model (built on Qwen2.5-3B). It thinks inside a <think>…</think> block and then writes its final answer, so the way you call it matters:

• Always apply the chat template (tokenizer.apply_chat_template(..., add_generation_prompt=True)). Passing a raw string skips the <|im_start|> / <|im_end|> markers the model was trained on.
• Give it room to think. It can emit thousands of reasoning tokens before the answer — use max_tokens of at least 8192 (16384+ for hard competition math). Small caps cut it off mid-thought.
• For math, ask for the final answer in \boxed{} — that's how it was trained and how the benchmarks below were scored. Parse the text after </think> for the answer.
• Context length: up to 64K tokens.
• Stop token: these MLX variants set eos_token_id to include <|im_end|> (151645) so generation halts cleanly at the turn boundary.

Recommended sampling (from the original model card): temperature=1.0, top_p=0.95, top_k disabled.

The benchmark numbers above were produced with greedy decoding (temperature 0) for reproducibility — so a variant's score reflects quantization damage, not sampling noise. For everyday use, the recommended sampling settings give better, more diverse reasoning.

Usage

pip install mlx-lm

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("sahilchachra/vibethinker-3b-mxfp8-mlx")

messages = [{"role": "user",
             "content": "Let x be the number of ways to ... . Give the final answer in \\boxed{}."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False,
)

# Reasoning needs a large token budget; sampling per the model card.
sampler = make_sampler(temp=1.0, top_p=0.95)
response = generate(
    model, tokenizer, prompt=prompt,
    max_tokens=16384, sampler=sampler, verbose=True,
)
# `response` contains a <think>...</think> trace followed by the final answer.

All variants in this collection

Model	Variant
sahilchachra/vibethinker-3b-mxfp4-mlx	Block float MX FP4
sahilchachra/vibethinker-3b-mxfp8-mlx	Block float MX FP8 ← this model
sahilchachra/vibethinker-3b-optiq-5bpw-mlx	OptiQ mixed-precision (target 5.0 bpw)

Notes

• Requires Apple Silicon (M1 or later) with MLX
• Benchmarks run on Apple M5 Pro, 24 GB unified memory
• License: see WeiboAI/VibeThinker-3B for the original model's license

Original model

See WeiboAI/VibeThinker-3B for full model details and intended use.