primitive-ai/Qwen3.8-Flash-Next-NVFP4

The 180B Flash-Next on one GPU.

Qwen3.8-Flash-Next is 360 GB in BF16 and needs two data-center GPUs in FP8.
This build serves it on one 96 GB Blackwell: 88.8 GiB of VRAM, the 51B n-gram table in host RAM, and no patched runtime.

---

Why this quant

• 🖥️ One 96 GB GPU. 88,828 MiB of VRAM at serve: NVFP4 experts plus the BF16 tail. The n-gram table lives in host RAM (~100 GB) with async prefetch. Any checkpoint that keeps the table in BF16 can take this path, the W4A16 builds measured further down included; the FP8-table checkpoints (official FP8, RadixArk, Inferact) cannot on the shipped image.
• 🎯 92.2 knowledge on a 1,170-item, 9-benchmark suite (n=2 runs: 92.1, 92.2), 99.4% finished, zero request errors. GSM8K 98.0, MMLU-Pro 90.0.
• 🔧 84.6 call / 56.7 abstain on the 200-item tool-calling suite (n=3). It makes correct calls at the same rate as much smaller strong models and never declines to call — the abstention items are what pull a pooled number down, which is why both halves are printed.
• ⚡ 74.4 tok/s single-stream (12.2 ms/token), 483.8 tok/s at concurrency 32, measured prefix-cache-free with distinct seeds; the two seeds agreed within 0.04%.
• 🔀 MTP speculative decoding preserved. All 31 MTP tensors byte-identical to the source; {"method":"mtp","num_speculative_tokens":3} works as Qwen documents.
• 🧩 Stock image, no patches. vllm/vllm-openai:qwen38-flash-next exactly as published. The serve block below carries the two flags that make single-GPU work — without them the server hangs silently or times out. One optional one-file overlay (further down) fixes a flaky upstream startup race that the image predates.

---

Serve it

docker run --gpus all --ipc=host -p 8000:8000 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Two flags are load-bearing on a single GPU. --distributed-executor-backend mp: the default single-GPU executor never starts the n-gram offload worker, and the first forward waits on it forever — the server looks healthy and hangs. VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800: the worker loads a 95 GB table before serving and the 600 s default can expire first. Host needs about 100 GB of free RAM. The reasoning parser and tool-call parser in this block are validated on this exact checkpoint: thinking lands in reasoning with no markup in content, and tool calls arrive as structured tool_calls with valid JSON arguments and finish_reason: tool_calls. If you have seen other NVFP4 cards warn that the automatic MoE backend picks FLASHINFER_TRTLLM and corrupts output: on this image on an RTX PRO 6000 the automatic choice is FLASHINFER_CUTLASS, printed at boot, and every number on this card went through it. Measured on the mixed sibling (same expert kernel), --moe-backend marlin decodes 10% faster single-stream (93.2 vs 84.6 tok/s) and 1–3% slower at concurrency 32, with TTFT up 12%; tool-calling under Marlin 79.5 (n=1). Pick by your concurrency.

---

Not enough host RAM? Put the table on NVMe

The serve command above wants ~100 GB of free host RAM for the n-gram table. With fast local storage you can skip that: this repo ships a one-file overlay (worker_image_disk.py) that maps the table from a file instead. First boot writes 95.4 GB into the store directory; every later boot maps it instantly and skips the table's checkpoint reads.

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 worker_image_disk.py --local-dir .
mkdir -p pledisk_store

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_disk.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/pledisk_store:/pledisk_store \
  -e VLLM_PLE_DISK_OFFLOAD_DIR=/pledisk_store \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Measured on the mixed sibling — both repos ship byte-identical BF16 tables and the same worker path, so the disk behavior transfers; absolute tok/s columns are the sibling's. 8K in / 512 out, prefix-cache-free, two seeds per cell (shown a / b):

config	boot	tok/s @ 1	tok/s @ 32	median TTFT @ 1
table in RAM (command above)	302 s	84.5 / 84.4	516.8 / 523.6	569 / 573 ms
disk, container capped to 48 GB RAM — recommended	263 s	79.4 / 76.8	427.0 / 435.8	571 / 573 ms
disk, uncapped 176 GB host	303–344 s	50.4–62.1	196.7–396.7	1.8–2.9 s
disk, cold page cache	404 s	40.8 / 37.0	134.2 / 290.3	4.6 / 5.2 s
disk, first boot (writes the file)	504 s	—	—	—

Net cost of the disk path, run capped: −8% single-stream, −17% at concurrency 32, TTFT parity with the RAM baseline. The counterintuitive row is the uncapped one, and it reproduces across two boots and four seeds: without a container memory cap, the boot's own 172 GB checkpoint streaming flows through the global page cache and evicts the table it is about to need, so gathers fault back to NVMe mid-decode. A memory cap makes reclaim cgroup-local — the container's checkpoint reads can only evict the container's own cache, and the table stays resident. So on the disk path, always cap the serving container (48 GB is what we validated; --memory 48g --memory-swap 48g).

Accuracy is unaffected — the mapping serves the same bytes. Inside the 48 GB cap the 200-item tool-calling suite scored 78.5 with zero request errors and zero truncations (repeat spread on this suite: 78.0–80.5), and the generation-sanity gate passed on the first-boot and capped configurations. Cold cache is a floor, not a steady state: the two cold @ 32 runs went 134 → 290 tok/s back to back as the cache refilled. Boot times share one caveat: all were measured with the checkpoint at least partially page-cache-resident; a truly cold first read of the 172 GB weights adds its own disk time to any of them.

The overlay targets this exact image. The same change is a draft PR to vLLM — vllm-project/vllm#54070, branch feat/ple-disk-offload — stacked on the PLE CPU-offload PR (vllm-project/vllm#53899).

Where upstream stands (2026-09-03): the architecture merged into vLLM main on 2026-08-31 (#53896), so nightly builds serve this model with the table on GPU, which means TP2 or more. The CPU-offload path (#53899) is still open, so single-GPU serving still means this image or that branch. A second upstream route, #54129, memory-maps the FP8 table straight out of the checkpoint shards; its thread reports 87.3 tok/s single-stream at TP1 on an RTX PRO 6000 with RadixArk's NVFP4, on par with the table resident in RAM on that box.

Known image bug, fix included. The current image (sha256:fc120ece…, unchanged as of 2026-09-03) predates an upstream fix for a startup race in the PLE offload path: vLLM can hang right after CUDA graph capture, looping No available shared memory broadcast block found in 60 seconds (vllm-project/vllm#53960; fixed on 2026-08-29 in the offload PR branch, #53899, which is still unmerged, so the issue stays open). This repo ships the fixed connector as connector_mrv2.py — mount it alongside any of the serve commands here, with or without the other overlays:

  -v $PWD/connector_mrv2.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/connector.py:ro ```

Verified on this build with the fix mounted: normal boot, sanity gate passed, accuracy and
throughput unchanged.

---

## Quantized PLE tables: 49 / 32 / 28.8 GB instead of 95 GB

The table itself also quantizes well. We publish it in **FP8 per-row (49 GB)**, **INT4
group-16 (32 GB)**, and **NVFP4-style e2m1 group-16 (28.8 GB)**, served memory-mapped by a
two-file overlay — host RAM cost becomes page cache only, no container cap needed. Accuracy
holds on both suites for all three (knowledge 92.2 / 92.9 / 92.2 vs 92.2 BF16; tool-calling
n=3 means 77.7 / 78.2 / 78.7 vs 79.2, one ±1.5 band), throughput lands within 5–6% of the
in-RAM BF16 baseline, and MTP keeps most of its speed-up (129.6 tok/s single-stream with the
INT4 table, 128.8 with NVFP4, vs 142.6 in-RAM). Validated end to end inside a 48 GB container:
tool-calling 80.5, 79.4 / 486 tok/s — a 64 GB-RAM host serves this model. Tables, overlay
files, serve command, format spec, and the full measurement table:
[primitive-ai/Qwen3.8-Flash-Next-PLE-quant](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant).

---

## Speculative decoding (MTP)

The MTP tensors are preserved byte-identical, so vLLM's built-in draft path works — add:

```bash
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Real-prompt A/B (measured on the mixed sibling), single stream, thinking on (60-item subset of our eval, decode rate = output tokens over wall time — not comparable to the bench-serve numbers above, and measured on real prompts because random-token benches overstate speculative gains):

speculative config	decode tok/s	strict score
none	91.2	91.7
num_speculative_tokens: 1	does not boot	—
num_speculative_tokens: 2	133.2	93.3
num_speculative_tokens: 3	142.6 (+56%)	93.3
num_speculative_tokens: 3, table on NVMe	77.5–82.3	93.3

Score differences are single-run noise on 60 items; the speed difference reproduces. num_speculative_tokens: 1 hangs this image at startup — the engine core blocks in shm_broadcast before the API server binds, reproduced on three boots (one waited 80 minutes). Use 3.

---

The other 4-bit quants, measured on the same box

When we first ran this (2026-08-28), every vLLM-lane quant of this model was experts-only 4-bit and these were the four public ones; nota-ai's calibrated NVFP4 was added on 2026-09-03 in a separate sitting. Same protocol for all: one RTX PRO 6000, 200-item tool-calling suite, cache-free 8K/512 bench, table in host RAM.

build	expert format	tok/s @ 1	tok/s @ 32	TTFT @ 1	tool-calling
our mixed build	NVFP4 g16 + FP8 attention	84.4	520	570 ms	79.2 (n=3)
Intel AutoRound	INT4 g128 sym, iters=200	82.6	482	683 ms	80.5 (n=1)
lvkaokao RTN	INT4 g128 sym, RTN	82.4	481	683 ms	80.0 (n=1)
cyankiwi AWQ	INT4 g32 asym, AWQ	81.6	422	680 ms	78.0 (n=1)
wtdcode AWQ	INT4 g128 sym, AWQ	82.6	484	682 ms	76.0 (n=1)
nota-ai NVFP4 (added 2026-09-03)	NVFP4 g16, calibrated W4A4	74.5	483	624 ms	78.0 (n=1)

Single tool-calling runs carry a ±1.5 spread, so the middle of this column is one band; the AWQ-vs-RTN-family gap at the edges is larger than that. Two patterns hold regardless: every 4-bit expert format decodes at ~82 tok/s single-stream with ~680 ms TTFT — the speed difference in the top row is the FP8 attention, not the expert format — and the W4A16 builds run on pre-Blackwell GPUs, which NVFP4 does not serve natively. All four keep the n-gram table in BF16, so the quantized-table sidecars apply to them as well.

The nota-ai row is the one calibrated NVFP4 in the field (512 conversations of 4K tokens, W4A4 with a calibrated activation global scale) and the only one we also ran through the knowledge suite: 92.1 strict / 92.6 completed, 99.3% finished, zero errors, against 92.2 for our uncalibrated round-to-nearest NVFP4; tool calling 83.8 call / 55.0 abstain against our 84.6 / 56.7. Speed is identical to our plain build (74.5 vs 74.4 tok/s single-stream, 483 vs 484 at concurrency 32) because the tensors that move at decode are the same format. On this suite, at this precision, calibration did not buy anything measurable. Their card's greedy-decoding table shows NVFP4 at or above BF16 on three of five benchmarks; they call that sampling noise, and our numbers agree with the reading. Not yet measured here: btbtyler09's GPTQ g32 (INT4 on the experts, the shared experts and the QSA projections; BF16 table) and codegyver's NVFP4 + FP8 weight-only QSA/GDN build for DGX Spark (FP8 table, so not single-GPU on this image). One more idea from the field: hampsonw ships an INT4 delta for the 512 experts inside the MTP head (5.2 GB of draft weights down to 1.5 GB per GPU) on top of the Intel build, with acceptance unchanged in their runs. The MTP head here stays BF16.

One kernel finding from the same sitting, on our mixed build: the image auto-selects the FLASHINFER_CUTLASS NVFP4 MoE kernel on this card, and --moe-backend marlin (weight-only dequant into a BF16 GEMM) decodes faster single-stream: 93.2 tok/s against 84.6, two seeds each, at a cost of 1–3% at concurrency 32 (502–512 vs 516) and 12% on TTFT (635 vs 567 ms). Tool-calling under Marlin: 79.5 (n=1; suite band ±1.5). Details in the serve section above.

MTP and the disk-backed BF16 table do not combine well: speculation multiplies table-gather traffic, the working set outgrows the page cache, and the +56% collapses to roughly the no-speculation rate (capped 77.5, uncapped 82.3). With the BF16 table in RAM, use MTP; on a low-RAM host, pair MTP with the INT4 quantized table instead, which keeps 129.6 tok/s.

---

Measured

1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call), under one fixed protocol: temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, no reasoning parser, the last ANSWER: scored. Concurrency 32 on the same single RTX PRO 6000 Blackwell the fit numbers come from. Auto-scored, no LLM judge.

build	size	overall	knowledge	call	abstain	runs k/a	finished	out/answer	tok/s @ 32	tok/s @ 1
this repo	186 GB	90.2	92.2	84.6	56.7	2/3	99.4%	664 tok	483.8	74.4

overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled as 1,370, weighted 85.4% and 14.6% by item count. call is accuracy on the 160 tool-calling items that require a call; abstain is the 40 whose correct action is to call nothing — they are never pooled into one number on our cards, because a model can be strong at one and weak at the other, and this one is exactly that: solid call accuracy, zero abstentions in the system-prompt tool format. Through the native tools= API it does abstain (validated above), so weight the two columns by how your application passes tools.

Same-box numbers for the community W4A16 builds are in the comparison table further up; the section below covers what single-GPU serving requires of a checkpoint.

Knowledge is a mean of 2 runs (92.1, 92.2); tool-calling of 3 (78.5, 80.0, 78.5 pooled). Throughput is prefix-cache-free — --random-prefix-len 0, a distinct seed per run, warm-up seeded apart — and two seeds agreed within 0.04%. On our other models this suite's repeat spread runs to ±0.5 on knowledge and ±1.5 on tool calling; treat gaps inside that as ties.

---

Single-GPU serving: what a checkpoint needs

On the shipped image, VLLM_PLE_CPU_OFFLOAD builds the CPU-side n-gram table as plain BF16, so serving on one 96 GB card there needs a checkpoint that ships the table in BF16. This build does, and so do the community W4A16 builds measured above and nota-ai's NVFP4, which we booted on this card the same way (394 s to healthy, 88,828 MiB, the same footprint as this build; their own card lists two cards as the minimum). With about 100 GB of free host RAM they can all take the single-card path (or far less RAM with the quantized-table sidecars: 49 / 32 / 28.8 GB, page-cache only). Checkpoints that store the table as FP8 with a scale tensor, the official FP8 release and the NVFP4 conversions derived from it, need TP2 or more on this image.

That constraint is loosening upstream. The offload PR branch (#53899) has loaded FP8 and NVFP4 global-scale tables since 2026-08-30, and its thread confirms RadixArk's NVFP4 booting at TP1 on an RTX PRO 6000 with no patches on that branch. Outside vLLM, garnermccloud's SGLang fork streams RadixArk's FP8 table from SSD and reports 164.7 tok/s single-stream with MTP on one RTX PRO 6000 (their prompts, 1,024-token completions, not our protocol). llama.cpp has had the architecture on main since 2026-08-27, so the GGUF lane is real now; GGUF and MLX conversions still do not serve on vLLM.

---

What's quantized to what

tensors	format
all 48 layers' routed experts (gate/up/down_proj, 120.8B params)	NVFP4 (group 16)
n-gram embedding table (51.2B, 128 shards)	BF16, pre-scaled — the only table format the shipped image's offload worker loads (the PR branch has since added FP8 and NVFP4 global-scale tables)
attention, GDN linear-attention, shared experts, routers, MTP, vision, embeddings, norms	BF16, byte-identical to the source

Weights-only round-to-nearest, no calibration. The n-gram table carries the FP8 release's values, materialized in BF16 — the same numbers every runtime materializes at load.

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from one fixed protocol.

The same 1,370 items: a 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). temperature 0.6, top_p 0.95, top_k 20, thinking forced on, a 16,384-token budget, no reasoning parser, scoring the last ANSWER: in the reply. Concurrency 32 on one RTX PRO 6000 Blackwell, each model's rows in one sitting. Auto-scored, no LLM judge. Both halves are means of at least three runs per build.

model	shape	size	overall	knowledge	call	abstain	finished	out	tok/s @ 32
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5	98.9%	1097 tok	1523
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5	97.9%	1429 tok	2204
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0	99.3%	760 tok	1469
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5	99.7%	800 tok	1176
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5	99.7%	651 tok	908
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0	97.3%	995 tok	670
Qwen3.8-Flash-Next (mixed sibling)	180 B MoE (6 B active)	183.7 GB	90.3	92.2	84.8	56.7	99.5%	686 tok	816

The Flash-Next row is the mixed sibling's sitting; this repo's own numbers under the same protocol are in Measured above (90.2 / 92.2 / 84.6 / 56.7), a tie with the mixed build.

Read overall with finished. overall scores an answer that overran the token budget as wrong, but it cannot say whether the model needed the room or failed to stop; finished and out separate those. A gap under 1.0 is a tie. The tok/s column comes from each model's own sitting and drifts a few percent between sittings, so read it as a bracket.

call and abstain are the tool-calling suite's two halves, reported separately. call is accuracy on the 160 items that require a tool call; abstain is the 40 whose correct action is to call nothing. They used to be pooled into one agentic number, and the pooling misled: a model with ordinary call accuracy and unusual abstention discipline outscored models that are better at actually making calls. Weight them by your own workload's mix.

Per-benchmark detail, both halves

benchmark	Laguna-XS-2.1	Nemotron-3.5-Lightning-30B-A3B	Ornith-1.5-35B-A3B	Muse-Glimmer-30B	Qwen3.8-27B	Laguna-S-2.1	Qwen3.8-Flash-Next
knowledge
mmlu_pro	79.0	82.0	89.5	89.0	89.5	81.0	89.0
math_500	76.0	80.0	88.0	74.0	82.0	83.0	86.0
gsm8k	97.0	96.0	99.0	98.0	98.0	99.0	97.0
arc_challenge	94.0	97.3	98.7	98.0	98.0	96.0	98.7
hellaswag	70.7	78.0	86.7	83.3	82.0	76.7	88.7
winogrande	84.7	89.3	89.3	90.7	89.3	86.0	95.3
commonsenseqa	80.8	83.3	86.7	82.5	84.2	85.8	86.7
boolq	90.0	92.0	92.0	90.0	91.0	92.0	94.0
openbookqa	94.0	99.0	98.0	97.0	97.0	94.0	96.0
tool calling
bfcl_v4	83.3	96.7	76.7	86.7	96.7	90.0	96.7
xlam	64.4	73.3	55.6	66.7	77.8	48.9	82.2
toolace	37.8	71.1	71.1	64.4	68.9	48.9	73.3
glaive	87.5	97.5	92.5	100.0	100.0	85.0	100.0
when2call	75.0	77.5	52.5	55.0	55.0	75.0	55.0

---

_{primitive · more models · inference economics for production LLM systems}