leonsarmiento/Qwen3.8-27B-3bit-mtp-mlx

Qwen3.8-27B VLM with the MTP drafter baked in — the speed variant of this quantization family. Identical 3-bit mixed recipe as Qwen3.8-27B-3bit-mlx, plus the source model's Multi-Token Prediction layer bundled inside the checkpoint, so the engine runs speculative decoding natively — no separate drafter model to download or pair.

Which one should you get?

	Qwen3.8-27B-3bit-mlx	This model (3bit-mtp)
Footprint	12.71 GB	13.04 GB (+0.33 GB)
Peak memory (pp1024/tg128)	13.16 GB	15.69 GB with MTP on / 13.16 GB with MTP off
Speculative decoding	—	✅ bundled, toggleable at runtime
Best for	The RAM-poor — leanest footprint	Machines with ~3 GB headroom that want extra decode speed

⚠️ Lightning MTP speedup is an oMLX-only feature (verified on oMLX v0.6.1, build 2323). LM Studio does not execute bundled MTP layers — this model loads and runs fine there, but the MTP tensors are inert dead weight. LM Studio users: get the clean Qwen3.8-27B-3bit-mlx instead. MTP is also not needed for thinking mode — see below.

Quantization Details

Same recipe as the clean build:

• Embedding layers: 4-bit with group_size=64
• Prediction head (lm_head): 4-bit with group_size=64
• Vision tower: 8-bit with group_size=64
• All other layers: 3-bit with group_size=64

MTP add-on (extracted from the source mtp.* tensors, scottlowry/oQ checkpoint convention):

• 7 norm tensors: bf16, +1.0 MLX RMSNorm convention
• fc.weight: bf16 unquantized (bit-identical to source)
• 7 linear layers: 4-bit affine, group_size=64 (per-tensor config overrides — the 3-bit default doesn't apply to them)
• text_config.mtp_num_hidden_layers: 1, mtp_use_dedicated_embeddings: false

Stats: 3.716 bpw base + 0.33 GB MTP · 13.04 GB · 3 shards · mlx-vlm · vision tower preserved · chat template with reasoning_effort defaulting to low.

Benchmarks (oMLX v0.6.1, Engine Auto, Context: Code/Python, pp1024/tg128)

Config	tg TPS (1×)	tg TPS (4×)	Peak Mem
3bit-mtp, Lightning MTP ON	20.0	90.9	15.69 GB
3bit-mtp, MTP OFF	15.3	67.4	13.16 GB
oQ4e-mtp reference (4-bit + iMatrix), MTP ON	22.3	87.2	18.35 GB
oQ4e-mtp reference, MTP OFF	14.8	59.0	16.41 GB

• +30.7% single-stream decode with MTP on (15.3 → 20.0 tok/s)
• Beats the 4-bit oQ4e reference at batch ≥ 2 (90.9 vs 87.2 tok/s @ 4×) while using 2.66 GB less memory
• Single-stream sits ~10% below a 4-bit verifier: the 3-bit base rejects more speculative drafts (verifier noise) — the accepted trade for the smaller footprint
• Disable TurboQuant KV cache — MTP shows no advantage with it enabled

When to turn MTP off: thinking/reasoning-heavy tasks. Faster token generation gets spent on more reasoning, not faster answers (measured on Qwen3.6-35B MATHQA: −7pp accuracy with MTP on, +20% wall time). With MTP off this model behaves exactly like the clean 3-bit build.

Source

Qwen/Qwen3.8-27B — dense hybrid-attention VLM, 64 layers (48 linear attention + 16 full attention), 262K context, MTP prediction layer.

Instruct Benchmarks

Model	MMLU	MMLU_PRO	HUMANEVAL	MBPP	Avg
Huihui-Gemma-4 26B 3bit-XL	78.0%	71.0%	98.0%	82.0%	82.3%
Gemma-4 26B 3bit-XL	80.0%	65.0%	98.0%	86.0%	82.3%
Qwen3.8-27B 3bit-mtp	80.0%	61.0%	94.0%	70.0%	76.3%
Qwen3.6-35B 3bit-XL	82.0%	63.0%	66.0%	80.0%	72.8%
Huihui-Qwen3.6-35B 3bit-XL	83.0%	63.0%	62.0%	76.0%	71.0%

Instruct mode (thinking off). Sampled: MMLU 100/14042, MMLU_PRO 100/12032, HUMANEVAL 50/164, MBPP 50/500. Wall times not directly comparable across architectures (dense vs MoE active params).

Highlights: HUMANEVAL 94% at dense 3-bit — strong code for the size class. Benchmarks ran in instruct mode; engage Lightning MTP (oMLX only) for decode speed, not for these quality numbers.