--- language: - es - en license: other license_name: swift-open-license-1.0 license_link: https://huggingface.co/ukisai/Swift-Qwen3.8-Flash-Next/blob/main/LICENSE base_model: ukisai/Swift-Qwen3.8-Flash-Next base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags: - sysadmin - opencode - linux - vllm - exl3 - exllamav3 - qtip - moe - swift - qwen3_8 - mtp - speculative-decoding - long-context - vision - 96gb - rtx-pro-6000 - endpoints_compatible --- # Swift 1.5 Qwen3.8-Flash-Next — EXL3 3.75 bpw for vLLM on a 96 GB GPU (262k context, 4 sessions, vision, MTP) > **Install from this card, not from generic snippets.** This is an EXL3 checkpoint: vLLM loads it only together with exllamav3 and our plugins. Follow [Requirements & install](#requirements--install-vllm-031--exllamav3-151--plugins) and [Quick start](#quick-start-one-96-gb-gpu-eg-rtx-pro-6000) (on Ampere GPUs, TabbyAPI as described below); a plain `pip install vllm`, Transformers or SGLang cannot load it. EXL3 quantization (3.75 bits per weight on average) of [UkisAI's Swift 1.5 Qwen3.8-Flash-Next](https://huggingface.co/ukisai/Swift-Qwen3.8-Flash-Next) (MoE, 125B total / 6B active, plus a 51B n-gram "PLE" table, MTP head and vision), made to serve the **whole model on one 96 GB GPU** with **vLLM** (continuous batching): **262,144 tokens per session without YaRN, 4 concurrent sessions in a 1.1M-token KV cache with 15 GB still free, vision and MTP**. Validated on a real **NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB, power-limited to 300 W)**. It is ≈3.3× closer to the BF16 model than our 2.25 bpw build (KLD 0.013 vs 0.044). > **This build needs a 96 GB GPU.** For 48 GB GPUs (RTX PRO 5000, L40S, A40) use our > [2.25 bpw build](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-2.25bpw-48GB). > **Our focus.** We quantize and evaluate models for one use case: a **local Linux system-administration agent** (opencode with tool calling) > for a university IT department, **used in Spanish**. Our calibration traces include real sysadmin/agent conversations in Spanish, and our quality > checks (tool calls with the real opencode request, agent tasks with real command execution, long-context retrieval, long answers in Spanish) reflect > that use. KL divergence is also reported on turboderp's public trace, so it can be compared and reproduced. Numbers for other domains or languages may differ. > ⚠️ **Use temperature 1.0** (UkisAI's recommended sampling: temperature 1.0, top_p 0.95, top_k 20 — this repo's `generation_config.json`). > At lower temperatures this model **falls into repetition loops** on long answers (see [Sampling and repetition loops](#sampling-and-repetition-loops)). This comes from the > model, not from the quantization: our 2.25 bpw build loops about as often. ## What is in this repository (and what is not) ### Which one to use All Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) variants we publish, measured on 2026-10-14 under the same conditions (see *Family comparison* for every number): | Variant | Weights | Context on the test GPU | KLD vs BF16 | Agent tasks | Decode, 1 request | Pick it for | |---|---|---|---|---|---|---| | [EXL3 2.25 · vLLM](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-2.25bpw-48GB) | 37.68 GiB | 262,144 with vision | 0.0486 | 27/30 | 98.4 tok/s | one 48 GB Ada or Blackwell GPU with vLLM: 262,144 tokens with vision and 4 sessions, fp8 KV cache | | [EXL3 2.25 · TabbyAPI](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-2.25bpw-48GB) | 37.68 GiB | 262,144 with vision | 0.0486 | 26/30 | 83.3 tok/s | a 48 GB Ampere GPU (A40): TabbyAPI with a Q8 cache (vLLM would need a BF16 KV cache there) and ≈6 GB of host RAM instead of ≈90 GB | | **[EXL3 3.75 · vLLM](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-3.75bpw-96GB) (this repo)** | 59.29 GiB | 262,144 with vision | 0.0151 | 27/30 | 112.9 tok/s | one 96 GB GPU: the closest to BF16 (KLD 0.0151) | | File(s) | What | Used by | |---|---|---| | `model-*.safetensors`, `config.json`, index | Swift 1.5 Flash-Next weights in EXL3 (3.75 bpw average) | vLLM and exllamav3 | | `ngram_embedding.safetensors` (26 GB) | the n-gram (PLE) table in EXL3 | exllamav3 only (vLLM cannot read it) | | `scripts/assemble.py` | builds the vLLM directory (downloads the two files below) | vLLM | | `plugin/qwen38_exl3_vision/` | our vLLM plugin 0.3.1: EXL3 MoE experts, vision tower, Qwen3.8-Flash-Next (`Qwen4Exp`) | vLLM | Download size: **89.9 GB** (63.7 GB of EXL3 weights + 26.2 GB EXL3 n-gram table). vLLM users can skip the n-gram table (`--exclude ngram_embedding.safetensors`, 63.7 GB). **Not included, because they are not ours to redistribute** — `scripts/assemble.py` downloads them from their owners: - **NVIDIA's FP8 PLE table** (`model-fp8-mtp-ple.safetensors` from [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4), NVIDIA Open Model License; 54 GB download, 51 GB once extracted). vLLM keeps it in **pinned host RAM**, not on the GPU. - **turboderp's MTP mixer patch** (`mtp_hyper_connection_mixer_patch.safetensors`, 13 MB, from [turboderp/Qwen3.8-Flash-Next-exl3](https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3), branch `3.05bpw_h5_ng5`): without it the MTP head does not load. ## Results ### Family comparison (Swift 1.5 Qwen3.8-Flash-Next (125B-A6B), same conditions) Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX PRO 6000 Blackwell Server Edition 96 GB (power limit 300 W, 2,270 MHz) with vLLM 0.31.0 + exllamav3 1.5.1 + plugins, and, for the 2.25 bpw build, one A40 48 GB with TabbyAPI (exllamav3 1.6.1, Q8 cache) as described for Ampere GPUs, each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's `qbench` (see *How to read it*). | Variant | Weights | KLD mean / median (internal · public) | Perplexity (BF16 2.1585 · pub. 1.7326) | Needle (8 facts) near max. context | Tools: real opencode request | Agent tasks (10 × 3) | Reasoning `xhigh`: correct · tokens | Vision | |---|---|---|---|---|---|---|---|---| | [EXL3 2.25 bpw (vLLM)](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-2.25bpw-48GB) (`main`) | 37.68 GiB | 0.0486 / 0.0163 · pub. 0.0443 / 0.0049 | 2.1907 · pub. 1.7521 | 8/8 at 241,437 | 59/60 | 27/30 | 48/48 · 13,451 | 15/15 | | [EXL3 2.25 bpw (TabbyAPI, Ampere)](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-2.25bpw-48GB) (`main`) | 37.68 GiB | 0.0486 / 0.0163 · pub. 0.0443 / 0.0049 | 2.1907 · pub. 1.7521 | 8/8 at 241,438 | 58/60 | 26/30 | 48/48 · 10,978 | 15/15 | | **[EXL3 3.75 bpw (vLLM)](https://huggingface.co/JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-3.75bpw-96GB) (`main`, this repo)** | **59.29 GiB** | **0.0151 / 0.0047 · pub. 0.0130 / 0.0012** | **2.1638 · pub. 1.7418** | **8/8 at 241,437** | **59/60** | **27/30** | **48/48 · 13,148** | **15/15** | | Variant | Context on the test GPU | Sessions | Prefill near max. context | Decode, 1 / 2 / 8 requests (tok/s) | MTP tokens per step | VRAM used at peak | |---|---|---|---|---|---|---| | EXL3 2.25 · vLLM | 262,144 with vision | 4 | 3,188 tok/s | 98.4 / 181.9 / 274.6 | 2.77 | 48,417 MiB | | EXL3 2.25 · TabbyAPI | 262,144 with vision | 4 | 1,298 tok/s | 83.3 / 63.2 / 70.9 | — | 45,219 MiB (272 MiB free on the A40 48 GB) | | EXL3 3.75 · vLLM | 262,144 with vision | 4 | 3,113 tok/s | 112.9 / 177.9 / 274.8 | 2.82 | 82,463 MiB | How to read it: - **KLD / perplexity**: next-token KL divergence against the BF16 model with exllamav3's `qbench` (lower is better). *Internal*: our sysadmin/agent evaluation set (not published, separate from the calibration data); *public*: turboderp's `qbench_prompts_gen.json`, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from `qbench`, not from this campaign. - **Needle**: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. **Tools**: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (`generation_config.json`); a try counts when it calls a valid tool with valid arguments and the expected command. **Agent**: 10 sysadmin tasks run with `opencode run` and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. **Reasoning**: 6 checkable problems × 8 at `xhigh`; total reasoning tokens over the 48. **Vision**: our 4-image test. - **Prefill / decode**: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than *Sessions*, the extra ones wait). **MTP tokens per step**: vLLM's speculative-decoding metrics over the whole run. - **VRAM really free**: lowest `memory.free` while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. - Older sections below keep their own date and setup; use this table to compare variants. **How many sessions fit** with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than *max sessions* at once): | Variant | KV pool (tokens) | Max sessions | 32k each | 64k each | 128k each | Full context each | |---|---|---|---|---|---|---| | EXL3 2.25 · vLLM | 419,939 | 4 | 4 | 4 | 3 | 1 × 262,144 | | EXL3 3.75 · vLLM | 1,122,383 | 4 | 4 | 4 | 4 | 4 × 262,144 | **Few users or many users.** The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the *many users* option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change. | Variant | Configuration | Decode by simultaneous requests (aggregate tok/s) | N sessions at once, needle 8/8 | Tools (5 × 3) | VRAM used at peak | |---|---|---|---|---|---| | EXL3 2.25 · vLLM | Quick start (4 sessions, MTP 3) | 1: 98 · 2: 182 · 8: 275 | — | 59/60 | 48,417 MiB | | EXL3 2.25 · vLLM | many users (16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens) | 1: 93 · 4: 235 · 8: 399 · 16: 580 | 16/16 of 16k | 14/15 | 48,471 MiB | | EXL3 3.75 · vLLM | Quick start (4 sessions, MTP 3) | 1: 113 · 2: 178 · 8: 275 | — | 59/60 | 82,463 MiB | | EXL3 3.75 · vLLM | many users (16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens) | 1: 89 · 4: 220 · 8: 368 · 16: 532 | 16/16 of 16k | 15/15 | 82,515 MiB | ### Earlier measurements _Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use *Family comparison* above to compare variants._ | Check | Result | |---|---| | KL divergence vs Swift 1.5 Flash-Next **BF16** (public trace / our sysadmin trace) | **0.0130 / 0.0151** (our 2.25 bpw: 0.044 / 0.049) | | RTX PRO 6000 (96 GB, 300 W): KV cache 20e9 bytes fp8 | **1,122,383 tokens**: 4 sessions × 262,144 with **15 GB still free** at peak (vision + MTP); weights 59.9 GiB on the GPU | | Decode speed (1 · 2 · 4 concurrent requests, MTP 3; RTX PRO 6000 at 300 W) | at the recommended **T 1.0: 76–97 · 131–132 · 204–228 tok/s**; at T 0: 98–119 · 152–154 · 270–281 tok/s (MTP accepts fewer drafts at T 1.0) | | Prefill (long prompts) | ≈4,200 tok/s; needle-in-a-haystack 8/8 at 62k / 123k / 236k tokens | | Vision | 15/15 | | Tool calls with the real opencode request (reasoning xhigh · no reasoning) | 29/30 · 30/30 | | Reasoning problems | 18/18 (median 334 tokens of reasoning) | | 8 concurrent long sessions | 8/8 | | Agent tasks (`opencode run`, real command execution, 20 tasks × 2) | **39/40** with file names in backticks (see notes) | ## Sampling and repetition loops Measured on 2026-10-14 with the same software and GPU as *Family comparison*, on our EXL3 2.25 · vLLM build of Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions. | Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | |---|---|---|---|---|---|---|---|---| | Loops / 30 | 24 | 24 | 21 | 21 | 17 | 8 | 0 | 0 | | Answers that finished before 16,000 tokens / 30 | 2 | 2 | 5 | 5 | 6 | 8 | 8 | 9 | From temperature **0.9** up, loops are rare (at most 1 of 30). This repo's default (`generation_config.json`) is **1.0**. Other models, same test and setup (loops / 30): | Model (build) | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | |---|---|---|---|---|---|---|---|---| | Qwen3.8-27B · EXL3 3.85 | 12 | 12 | 7 | 8 | 1 | 1 | 1 | 0 | | Qwen3.8-27B-pi · EXL3 3.85 | 9 | 4 | 9 | 10 | 4 | 2 | 1 | 0 | | Qwopus3.8-27B-Flash-V2 · EXL3 3.85 | 3 | 1 | 2 | 1 | 2 | 0 | 0 | 0 | | Swift 1.5 (Qwen3.8-27B) · EXL3 3.85 | 7 | 5 | 3 | 4 | 3 | 1 | 1 | 0 | | Ornith-1.5-9B · EXL3 6.00 (16 GB) | 6 | 2 | 3 | 1 | 1 | 0 | 0 | 0 | | NeoHorse-1-9B · EXL3 6.00 (16 GB) | 6 | 3 | 3 | 1 | 0 | 0 | 0 | 0 | | Qwen3.6-35B-A3B · EXL3 4.00 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | | Ornith-1.5-35B-A3B · EXL3 4.00 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | Qwopus3.6-35B-A3B-Coder · EXL3 4.00 | 9 | 13 | 12 | 6 | 11 | 6 | 5 | 4 | | Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | — | — | 13 | 7 | 5 | 2 | 0 | 0 | ## Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins) EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's `vllm.general_plugins` entry point: - **[Continuum-AI-Corp/OrcaSAQ2-kernel](https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel)** (commit `7ddffb12c00c24b8bc6b1164723c04ac490a91bc`): EXL3 linear layers. - **`qwen38_exl3_vision` 0.4.1** (ours; 0.4.1 is a maintenance release; 0.4.0 added multi-GPU: tensor parallelism for the dense models, `-tp 2`, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, `-tp 2 --enable-expert-parallel`; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in [`plugin/`](plugin/qwen38_exl3_vision/)): EXL3 MoE experts (stacked and run with exllamav3's fused MoE kernel), the EXL3 vision tower and the Qwen3.8-Flash-Next (`Qwen4Exp`) model. Use **exllamav3 1.5.1**: with 1.6.x the KLD is the same but tool calls without reasoning got worse in our tests. Python 3.12; a CUDA 13.0 toolkit for flashinfer's JIT kernels (point `CUDA_HOME` at it if the `nvcc` from pip does not match the CUDA headers). ```bash uv venv --python 3.12 venv uv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0 W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whl curl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}" git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc) hf download JRamirez-UAB/Swift-1.5-Qwen3.8-Flash-Next-EXL3-3.75bpw-96GB --local-dir swift-fn-375 --exclude ngram_embedding.safetensors uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e swift-fn-375/plugin/qwen38_exl3_vision venv/bin/python swift-fn-375/scripts/assemble.py --src swift-fn-375 --dst swift-fn-375-vllm ``` ## Quick start (one 96 GB GPU, e.g. RTX PRO 6000) The host needs **about 90 GB of RAM** for this process (the PLE table is pinned in RAM: 51 GB); with a 70 GB limit it was killed for lack of RAM. ```bash export VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True vllm serve swift-fn-375-vllm --served-model-name swift-fn \ --max-model-len 262144 --kv-cache-dtype fp8 --kv-cache-memory-bytes 20000000000 \ --max-num-seqs 4 --max-num-batched-tokens 2048 --enable-prefix-caching --mamba-cache-mode align \ --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"max_pixels":1003520}' \ --compilation-config '{"max_cudagraph_capture_size":16}' \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` **For many users** (measured on 2026-10-14, same GPU): the same command with `--max-num-seqs 16 --speculative-config '{"method":"mtp","num_speculative_tokens":1}' --compilation-config '{"max_cudagraph_capture_size":64}'` — 16 sessions of up to 262,144 tokens, MTP 1, CUDA graphs up to 64 tokens. Aggregate decode 532 tok/s with 16 requests (Quick start: 275 with 8); a single request runs at 89 tok/s (113 with the Quick start); 16 of 16 simultaneous sessions returned all 8 needle facts; 82,515 MiB of VRAM used at peak. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step. ## Using it with opencode (how we tested) This checkpoint is meant to be used as a **coding / sysadmin agent from [opencode](https://opencode.ai)** (tested with opencode 1.18.35). Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with `opencode run` agent tasks; context, speed, memory and image tests were run directly against the vLLM API. ```json { "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "swift-fn": { "name": "Swift 1.5 Qwen3.8-Flash-Next EXL3 3.75 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 262144, "output": 32000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/swift-fn", "variant": "xhigh", "temperature": 1.0 }, "plan": { "model": "local-vllm/swift-fn", "variant": "xhigh", "temperature": 1.0 } } } ``` - **`limit.context` = `--max-model-len`** (262144); with `output: 32000` opencode compacts the history at ≈`context − 32k` tokens of input. - **Declare `"temperature": true` in the model**, otherwise opencode sends no temperature to a custom model (checked by capturing the requests), and set it to **1.0** (see [Sampling and repetition loops](#sampling-and-repetition-loops)). - Without `attachment` / `modalities` opencode will not send images; drop them if you serve with `--language-model-only`. - Start opencode inside a project directory (it takes a git snapshot before each step); for `opencode run` from scripts, close stdin (`