JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-3.75bpw-96GB

Swift 1.5 Qwen3.8-Flash-Next — EXL3 3.75 bpw for vLLM on a 96 GB GPU (262k context, 4 sessions, vision, MTP)

Install from this card, not from generic snippets. This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow Requirements & install and Quick start (on Ampere GPUs, TabbyAPI as described below); a plain pip install vllm, Transformers or SGLang cannot load it.

EXL3 quantization (3.75 bits per weight on average) of UkisAI's Swift 1.5 Qwen3.8-Flash-Next (MoE, 125B total / 6B active, plus a 51B n-gram "PLE" table, MTP head and vision), made to serve the whole model on one 96 GB GPU with vLLM (continuous batching): 262,144 tokens per session without YaRN, 4 concurrent sessions in a 1.1M-token KV cache with 15 GB still free, vision and MTP. Validated on a real NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, power-limited to 300 W). It is ≈3.3× closer to the BF16 model than our 2.25 bpw build (KLD 0.013 vs 0.044).

This build needs a 96 GB GPU. For 48 GB GPUs (RTX PRO 5000, L40S, A40) use our 2.25 bpw build.

Our focus. We quantize and evaluate models for one use case: a local Linux system-administration agent (opencode with tool calling) for a university IT department, used in Spanish. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ.

⚠️ Use temperature 1.0 (UkisAI's recommended sampling: temperature 1.0, top_p 0.95, top_k 20 — this repo's generation_config.json). At lower temperatures this model falls into repetition loops on long answers (see Sampling and repetition loops). This comes from the model, not from the quantization: our 2.25 bpw build loops about as often.

What is in this repository (and what is not)

Which one to use

All Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Variant	Weights	Context on the test GPU	KLD vs BF16	Agent tasks	Decode, 1 request	Pick it for
EXL3 2.25 · vLLM	37.68 GiB	262,144 with vision	0.0486	27/30	98.4 tok/s	one 48 GB Ada or Blackwell GPU with vLLM: 262,144 tokens with vision and 4 sessions, fp8 KV cache
EXL3 2.25 · TabbyAPI	37.68 GiB	262,144 with vision	0.0486	26/30	83.3 tok/s	a 48 GB Ampere GPU (A40): TabbyAPI with a Q8 cache (vLLM would need a BF16 KV cache there) and ≈6 GB of host RAM instead of ≈90 GB
EXL3 3.75 · vLLM (this repo)	59.29 GiB	262,144 with vision	0.0151	27/30	112.9 tok/s	one 96 GB GPU: the closest to BF16 (KLD 0.0151)

File(s)	What	Used by
model-*.safetensors, config.json, index	Swift 1.5 Flash-Next weights in EXL3 (3.75 bpw average)	vLLM and exllamav3
ngram_embedding.safetensors (26 GB)	the n-gram (PLE) table in EXL3	exllamav3 only (vLLM cannot read it)
scripts/assemble.py	builds the vLLM directory (downloads the two files below)	vLLM
plugin/qwen38_exl3_vision/	our vLLM plugin 0.3.1: EXL3 MoE experts, vision tower, Qwen3.8-Flash-Next (Qwen4Exp)	vLLM

Download size: 89.9 GB (63.7 GB of EXL3 weights + 26.2 GB EXL3 n-gram table). vLLM users can skip the n-gram table (--exclude ngram_embedding.safetensors, 63.7 GB).

Not included, because they are not ours to redistribute — scripts/assemble.py downloads them from their owners:

• NVIDIA's FP8 PLE table (model-fp8-mtp-ple.safetensors from nvidia/Qwen3.8-Flash-Next-NVFP4, NVIDIA Open Model License; 54 GB download, 51 GB once extracted). vLLM keeps it in pinned host RAM, not on the GPU.
• turboderp's MTP mixer patch (mtp_hyper_connection_mixer_patch.safetensors, 13 MB, from turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5): without it the MTP head does not load.

Results

Family comparison (Swift 1.5 Qwen3.8-Flash-Next (125B-A6B), same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX PRO 6000 Blackwell Server Edition 96 GB (power limit 300 W, 2,270 MHz) with vLLM 0.31.0 + exllamav3 1.5.1 + plugins, and, for the 2.25 bpw build, one A40 48 GB with TabbyAPI (exllamav3 1.6.1, Q8 cache) as described for Ampere GPUs, each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

Variant	Weights	KLD mean / median (internal · public)	Perplexity (BF16 2.1585 · pub. 1.7326)	Needle (8 facts) near max. context	Tools: real opencode request	Agent tasks (10 × 3)	Reasoning xhigh: correct · tokens	Vision
EXL3 2.25 bpw (vLLM) (main)	37.68 GiB	0.0486 / 0.0163 · pub. 0.0443 / 0.0049	2.1907 · pub. 1.7521	8/8 at 241,437	59/60	27/30	48/48 · 13,451	15/15
EXL3 2.25 bpw (TabbyAPI, Ampere) (main)	37.68 GiB	0.0486 / 0.0163 · pub. 0.0443 / 0.0049	2.1907 · pub. 1.7521	8/8 at 241,438	58/60	26/30	48/48 · 10,978	15/15
EXL3 3.75 bpw (vLLM) (main, this repo)	59.29 GiB	0.0151 / 0.0047 · pub. 0.0130 / 0.0012	2.1638 · pub. 1.7418	8/8 at 241,437	59/60	27/30	48/48 · 13,148	15/15

Variant	Context on the test GPU	Sessions	Prefill near max. context	Decode, 1 / 2 / 8 requests (tok/s)	MTP tokens per step	VRAM used at peak
EXL3 2.25 · vLLM	262,144 with vision	4	3,188 tok/s	98.4 / 181.9 / 274.6	2.77	48,417 MiB
EXL3 2.25 · TabbyAPI	262,144 with vision	4	1,298 tok/s	83.3 / 63.2 / 70.9	—	45,219 MiB (272 MiB free on the A40 48 GB)
EXL3 3.75 · vLLM	262,144 with vision	4	3,113 tok/s	112.9 / 177.9 / 274.8	2.82	82,463 MiB

How to read it:

• KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
• Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
• Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
• VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB.
• Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Variant	KV pool (tokens)	Max sessions	32k each	64k each	128k each	Full context each
EXL3 2.25 · vLLM	419,939	4	4	4	3	1 × 262,144
EXL3 3.75 · vLLM	1,122,383	4	4	4	4	4 × 262,144

Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Variant	Configuration	Decode by simultaneous requests (aggregate tok/s)	N sessions at once, needle 8/8	Tools (5 × 3)	VRAM used at peak
EXL3 2.25 · vLLM	Quick start (4 sessions, MTP 3)	1: 98 · 2: 182 · 8: 275	—	59/60	48,417 MiB
EXL3 2.25 · vLLM	many users (16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens)	1: 93 · 4: 235 · 8: 399 · 16: 580	16/16 of 16k	14/15	48,471 MiB
EXL3 3.75 · vLLM	Quick start (4 sessions, MTP 3)	1: 113 · 2: 178 · 8: 275	—	59/60	82,463 MiB
EXL3 3.75 · vLLM	many users (16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens)	1: 89 · 4: 220 · 8: 368 · 16: 532	16/16 of 16k	15/15	82,515 MiB

Earlier measurements

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Check	Result
KL divergence vs Swift 1.5 Flash-Next BF16 (public trace / our sysadmin trace)	0.0130 / 0.0151 (our 2.25 bpw: 0.044 / 0.049)
RTX PRO 6000 (96 GB, 300 W): KV cache 20e9 bytes fp8	1,122,383 tokens: 4 sessions × 262,144 with 15 GB still free at peak (vision + MTP); weights 59.9 GiB on the GPU
Decode speed (1 · 2 · 4 concurrent requests, MTP 3; RTX PRO 6000 at 300 W)	at the recommended T 1.0: 76–97 · 131–132 · 204–228 tok/s; at T 0: 98–119 · 152–154 · 270–281 tok/s (MTP accepts fewer drafts at T 1.0)
Prefill (long prompts)	≈4,200 tok/s; needle-in-a-haystack 8/8 at 62k / 123k / 236k tokens
Vision	15/15
Tool calls with the real opencode request (reasoning xhigh · no reasoning)	29/30 · 30/30
Reasoning problems	18/18 (median 334 tokens of reasoning)
8 concurrent long sessions	8/8
Agent tasks (opencode run, real command execution, 20 tasks × 2)	39/40 with file names in backticks (see notes)

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 2.25 · vLLM build of Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

Temperature	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Loops / 30	24	24	21	21	17	8	0	0
Answers that finished before 16,000 tokens / 30	2	2	5	5	6	8	8	9

From temperature 0.9 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Model (build)	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Qwen3.8-27B · EXL3 3.85	12	12	7	8	1	1	1	0
Qwen3.8-27B-pi · EXL3 3.85	9	4	9	10	4	2	1	0
Qwopus3.8-27B-Flash-V2 · EXL3 3.85	3	1	2	1	2	0	0	0
Swift 1.5 (Qwen3.8-27B) · EXL3 3.85	7	5	3	4	3	1	1	0
Ornith-1.5-9B · EXL3 6.00 (16 GB)	6	2	3	1	1	0	0	0
NeoHorse-1-9B · EXL3 6.00 (16 GB)	6	3	3	1	0	0	0	0
Qwen3.6-35B-A3B · EXL3 4.00	0	0	1	1	0	0	0	0
Ornith-1.5-35B-A3B · EXL3 4.00	1	0	0	0	0	0	0	0
Qwopus3.6-35B-A3B-Coder · EXL3 4.00	9	13	12	6	11	6	5	4
Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM	—	—	13	7	5	2	0	0

Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

• Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers.
• qwen38_exl3_vision 0.4.1 (ours; 0.4.1 is a maintenance release; 0.4.0 added multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/): EXL3 MoE experts (stacked and run with exllamav3's fused MoE kernel), the EXL3 vision tower and the Qwen3.8-Flash-Next (Qwen4Exp) model.

Use exllamav3 1.5.1: with 1.6.x the KLD is the same but tool calls without reasoning got worse in our tests. Python 3.12; a CUDA 13.0 toolkit for flashinfer's JIT kernels (point CUDA_HOME at it if the nvcc from pip does not match the CUDA headers).

uv venv --python 3.12 venv
uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto          # torch 2.13.0
W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl
curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"
git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)
hf download JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-3.75bpw-96GB --local-dir swift-fn-375 --exclude ngram_embedding.safetensors
uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e swift-fn-375/plugin/qwen38_exl3_vision
venv/bin/python swift-fn-375/scripts/assemble.py --src swift-fn-375 --dst swift-fn-375-vllm

Quick start (one 96 GB GPU, e.g. RTX PRO 6000)

The host needs about 90 GB of RAM for this process (the PLE table is pinned in RAM: 51 GB); with a 70 GB limit it was killed for lack of RAM.

export VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
vllm serve swift-fn-375-vllm --served-model-name swift-fn \
  --max-model-len 262144 --kv-cache-dtype fp8 --kv-cache-memory-bytes 20000000000 \
  --max-num-seqs 4 --max-num-batched-tokens 2048 --enable-prefix-caching --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"max_pixels":1003520}' \
  --compilation-config '{"max_cudagraph_capture_size":16}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 16 --speculative-config '{"method":"mtp","num_speculative_tokens":1}' --compilation-config '{"max_cudagraph_capture_size":64}' — 16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens. Aggregate decode 532 tok/s with 16 requests (Quick start: 275 with 8); a single request runs at 89 tok/s (113 with the Quick start); 16 of 16 simultaneous sessions returned all 8 needle facts; 82,515 MiB of VRAM used at peak. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.35). Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "local vLLM",
      "options": { "baseURL": "http://127.0.0.1:8000/v1" },
      "models": {
        "swift-fn": {
          "name": "Swift 1.5 Qwen3.8-Flash-Next EXL3 3.75 (local)",
          "tool_call": true,
          "temperature": true,
          "attachment": true,
          "modalities": { "input": ["text", "image"], "output": ["text"] },
          "limit": { "context": 262144, "output": 32000 },
          "variants": {
            "none":   { "chat_template_kwargs": { "enable_thinking": false } },
            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },
            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },
            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }
          }
        }
      }
    }
  },
  "agent": {
    "build": { "model": "local-vllm/swift-fn", "variant": "xhigh", "temperature": 1.0 },
    "plan":  { "model": "local-vllm/swift-fn", "variant": "xhigh", "temperature": 1.0 }
  }
}

• limit.context = --max-model-len (262144); with output: 32000 opencode compacts the history at ≈context − 32k tokens of input.
• Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests), and set it to 1.0 (see Sampling and repetition loops).
• Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
• Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (</dev/null).

PLE table outside the vLLM process (experimental)

service-ple is an experimental project that keeps this model's PLE table out of the vLLM process: a per-machine service with a shared cache that reads the rows from the local file or forwards them to a central server. See its repository for details and current status.

Important notes

• Plugin 0.3.1 and CUDA graphs (2026-10-14): set max_cudagraph_capture_size ≥ max-num-seqs × (1 + MTP tokens) so that every decode step is replayed from a graph (the Quick start's 16 is enough for its 4 sessions with MTP 3). 0.3.1 runs every step of up to 256 tokens in exllamav3's fused MoE kernel; 0.3.0 only did so up to 25 tokens (10 experts per token × tokens ≤ 256) and fails at start with a larger capture size. RTX PRO 6000 (300 W), Swift 2.25 bpw, 16 sequences, short prompts, total tok/s with 1 / 4 / 8 / 16 requests at once: 0.3.0 with the Quick start's graphs ≤ 16 and MTP 3 → 107 / 223 / 158 / 237; 0.3.1 with graphs ≤ 64 and MTP 1 → 83 / 225 / 351 / 496; 0.3.1, graphs ≤ 64, MTP 3 → 91 / 233 / 334 / 347. So: MTP 3 for one user, MTP 1 with graphs ≥ 2 × sequences for many. KL divergence 0.3.1 vs 0.3.0: 0.0193 (two 0.3.0 runs differ by 0.0200).
• Memory. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization, and read the real headroom with nvidia-smi (the driver reserves ≈450 MiB that vLLM cannot use). 20e9 (1.1M tokens) left 15 GB free at peak on the RTX PRO 6000 with 4 long sessions, vision and MTP: there is room for a larger cache or more sessions.
• VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 and CUDA-graph captures ≤16 are required for the EXL3 MoE path.
• File names in non-English prompts. Like other Qwen3.x models, it sometimes writes an output file under the conventional English name when its Spanish name is a system concept (procesos.csv → processes.csv); names unrelated to the system are kept. Writing file names and paths in backticks fixes it (measured on our 2.25 bpw build: agent 72/80 → 79/80).
• Invented command options. With uncommon CLIs (Slurm sacctmgr, FreeIPA ipa) it may guess option names, like every model we tested. Given the tool's real --help/man output our 2.25 bpw build got them right every time (Slurm QoS 5/5, FreeIPA 5/5); documenting the tools your agent uses is the practical fix.

Quantization recipe

exllamav3 1.5.1 from the Swift 1.5 Flash-Next BF16: per-tensor bit allocation with sc_optimize (target 3.75 bpw, -mink 2: no expert below 2 bits), head 6 bits, vision 6, MTP 4, a shared 4-bit n-gram table; calibration on a trace of Swift 1.5 27B answering Spanish sysadmin/agent prompts at reasoning effort xhigh. We also tried calibrating with the model's own outputs and with exllamav3's default data: both gave a slightly higher KLD.

Integrity and validation

What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).

Artifact integrity

• Every weight file is checked against our local copy with SHA-256 before publication; Hugging Face shows each file's SHA-256 in Files and versions.
• The measurements used the weights downloaded from this repo at commit deebf6d598d5; later commits change only metadata, documentation or the plugin, never the .safetensors files.
• vLLM loads every tensor listed in model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.

Quantization

• Per-tensor bit allocation is recorded by exllamav3 in quantization_config.json; its tensor_storage entry for embed_tokens describes the INT8 qweight + BF16 scales actually stored (updated on 2026-10-14; the plugin only reads the .trellis entries).
• The INT8 embedding conversion (scripts/embed_int8.py) is reproducible: re-running it on the source checkpoint gives the published qweight and scales bit for bit (checked on Ornith-1.5-9B 6.00 bpw); its reported error now uses the stored BF16 scales (0.00806 vs 0.00803 with FP32 scales).
• Fidelity: KL divergence against the BF16 Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) with exllamav3's qbench on two traces (ours and turboderp's public one).

Chat template

• The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).

Runtime

• Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.

Scope: results apply to the revision, software stack (vLLM 0.31.0 + exllamav3 1.5.1 + plugins), hardware and settings stated above; other setups are not tested.

Limitations

• n-gram speculative decoding does not start with this model in vLLM 0.31.0 (--speculative-config '{"method":"ngram",...}' stops at start-up with RuntimeError: PLE inputs were not prepared, tested on 2026-10-14): use its MTP head, as in the Quick start.
• Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with sudo, and give the agent a non-privileged account and a test machine where you can.
• Quality was not measured with standard benchmarks (see UkisAI's card for those); the checks are KL divergence against BF16, long-context retrieval, vision, tool calling with a real agent request, reasoning problems and agent tasks. All numbers come from our machines and one software stack.
• The EXL3 MoE path in vLLM is our own plugin code (not upstream). The RTX PRO 6000 we used is power-limited to 300 W (600 W stock): expect equal or higher speed at full power.
• This repository has not yet been checked by downloading it on a fresh machine.

Feedback welcome

Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).

Credits & license

• Model: Swift 1.5 Qwen3.8-Flash-Next by UkisAI, a derivative of Qwen3.8-Flash-Next by Qwen.
• License: UkisAI's contribution, including these quantized weights, is under the Swift Open License v1.0: free for personal, research, educational, evaluation and commercial use for individuals and organizations with gross annual revenue up to US$1,000,000; above that, commercial use needs a Swift Enterprise License from UkisAI. The base model is under the Qwen Community License 1.0 and its terms still apply (Model-as-a-Service / AI Work Assistant businesses need a separate license from Qwen). See NOTICE for attribution and the list of changes.
• exllamav3 by turboderp (quantization, kernels, qbench and the public trace; the MTP mixer patch); NVIDIA (FP8 PLE table, not redistributed here); OrcaSAQ2-kernel; vLLM.
• Our plugin and scripts/assemble.py are Apache-2.0.