Instructions to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NInfer
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with NInfer:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K # Run inference directly in the terminal: llama cli -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K # Run inference directly in the terminal: llama cli -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K # Run inference directly in the terminal: ./llama-cli -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K # Run inference directly in the terminal: ./build/bin/llama-cli -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Use Docker
docker model run hf.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
- LM Studio
- Jan
- vLLM
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
- Ollama
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with Ollama:
ollama run hf.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
- Unsloth Desktop
- Pi
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with Docker Model Runner:
docker model run hf.co/nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
- Lemonade
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Run and chat with the model
lemonade run user.Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer-Q6_K
List all available models
lemonade list
- Hermes Agent
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer:Q6_K" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Card: add Q3 LynnStyle-16G GGUF tier; update GGUF tier chart
Browse files- README.md +46 -2
- assets/header-gguf-tier-table-en.png +2 -2
- assets/header-gguf-tier-table-zh.png +2 -2
|
@@ -40,6 +40,10 @@ A model post-trained with several rounds of SFT and RLOO on top of [Qwen3.8-27B-
|
|
| 40 |
- **GGUF Q2 LynnStyle · reasoning length (P50 / P70 / P90 tokens)**: GPQA 2,989.5 / 8,923.8 / 27,156.1; MMLU 181 / 326.3 / 1,020.5; LCB 5,656.5 / 17,835.8 / 40,067.3.
|
| 41 |
- **GGUF Q2 LynnStyle · speed**: NInfer with built-in Q4 MTP, C8 aggregate 521.0 tok/s (512-token cap, short run); llama.cpp with built-in Q4 MTP, C4 aggregate 174.8 tok/s (4096-token cap, long run). Protocols differ; do not compare directly.
|
| 42 |
- **GGUF Q2 LynnStyle · size**: GGUF 13,276,009,792 bytes (BPW 3.89), NInfer 13,568,617,472 bytes with vision (text + MTP BPW 3.89).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
- **GGUF Q3 LynnStyle · scores**: GPQA 173/198, MMLU 449/500, LCB 92/100 (llama.cpp, C4, 100K full sets; scored before the built-in MTP head was attached (external Q8 MTP draft); same quantized weights).
|
| 44 |
- **GGUF Q3 LynnStyle · reasoning length (P50 / P70 / P90 tokens)**: GPQA 3,125.5 / 7,265.2 / 24,343.1; MMLU 148 / 279.3 / 923.4; LCB 4,421 / 14,232.2 / 34,946.
|
| 45 |
- **GGUF Q3 LynnStyle · speed**: NInfer with built-in Q4 MTP, C8 aggregate 512.3 tok/s (512-token cap, short run); llama.cpp with built-in Q4 MTP, C4 aggregate 191.8 tok/s (4096-token cap, long run). Protocols differ; do not compare directly.
|
|
@@ -118,6 +122,9 @@ The original Qwen3.8 27B (FP8) was measured on two RTX PRO 6000 GPUs at C8 per G
|
|
| 118 |
| GPQA | GGUF Q2 LynnStyle | 177/198 | **2,989.5** / **8,923.8** / **27,156.1** | **3** | **0** |
|
| 119 |
| MMLU | GGUF Q2 LynnStyle | 433/500 | **181** / **326.3** / **1,020.5** | 1 | 0 |
|
| 120 |
| LCB | GGUF Q2 LynnStyle | **90/100** | **5,656.5** / **17,835.8** / **40,067.3** | **1** | **1** |
|
|
|
|
|
|
|
|
|
|
| 121 |
| GPQA | GGUF Q8_0 | 171/198 | **3,102.5** / **7,971** / **28,891.6** | **2** | **0** |
|
| 122 |
| GPQA | GGUF Q6_K | 174/198 | **3,683** / **9,814** / **27,966** | **2** | **2** |
|
| 123 |
| MMLU | Original Qwen3.8 27B (FP8) | 444/500 | 205 / 373 / 1,622 | 1 | 0 |
|
|
@@ -171,6 +178,7 @@ In Lynn: **Settings → Models → Local Qwen3.8-27B → pick Q2 / Q3 → Instal
|
|
| 171 |
| GGUF Q4 LynnStyle · NInfer | `GGUF-NInfer/` | NInfer package from the same Q4 LynnStyle GGUF (`.ninfer`), built-in Q4_K MTP and vision; NInfer-all and ninfer-fusion-kvmem; scored on llama.cpp before the built-in MTP head was attached (external Q8 MTP draft); same quantized weights | 19,913,006,080 (~18.5 GiB) | 5.74 | 175 / 448 / 89 |
|
| 172 |
| GGUF Q4 LynnStyle · Q8MTP | `GGUF/` | The same Q4 LynnStyle GGUF with only the built-in MTP head re-encoded as Q8_0 (`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf`); every other tensor is byte-identical to the original; an alternative with a Q8_0 MTP head (the original's Q4_0 MTP head is supported by llama.cpp); llama.cpp vision via the mmproj in `GGUF/` | 19,832,743,232 (~18.5 GiB) | 5.81 | 175 / 448 / 89 |
|
| 173 |
| GGUF Q4 LynnStyle · Q8MTP · NInfer | `GGUF-NInfer/` | NInfer package from the Q8MTP GGUF above (`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer`), built-in Q8_0 MTP head and vision, model id `qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp`; body weights identical to the original Q4 LynnStyle package, scores carried over | 20,125,342,720 (~18.7 GiB) | 5.81 | 175 / 448 / 89 |
|
|
|
|
| 174 |
| GGUF Q2 LynnStyle | `GGUF/` | Mixed-precision LynnStyle GGUF for llama.cpp (`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf`): lowest type reaches Q2; **built-in Q4 MTP** (not Q8); llama.cpp vision via `--mmproj` with the mmproj in `GGUF/` | 13,276,009,792 (~12.4 GiB) | 3.89 | 177 / 433 / 90 |
|
| 175 |
| GGUF Q2 LynnStyle · NInfer | `GGUF-NInfer/` | NInfer package from the same Q2 LynnStyle GGUF (`.ninfer`), built-in Q4_K MTP and vision; NInfer-all and ninfer-fusion-kvmem; scores from GGUF on llama.cpp | 13,568,617,472 (~12.6 GiB) | 3.89 | 177 / 433 / 90 |
|
| 176 |
| GGUF Q2 LynnStyle · Q8MTP | `GGUF/` | The same Q2 LynnStyle GGUF with only the built-in MTP head re-encoded as Q8_0 (`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf`); every other tensor is byte-identical to the original; an alternative with a Q8_0 MTP head (the original's Q4_0 MTP head is supported by llama.cpp); llama.cpp vision via the mmproj in `GGUF/` | 13,488,346,432 (~12.6 GiB) | 3.95 | 177 / 433 / 90 |
|
|
@@ -190,7 +198,7 @@ In Lynn: **Settings → Models → Local Qwen3.8-27B → pick Q2 / Q3 → Instal
|
|
| 190 |
| Q3 LynnStyle · Q8MTP | 134.9 | 51.7% (2.55) | 138.3 | 0.904 | correct |
|
| 191 |
| Q4 LynnStyle · Q8MTP | 154.8 | 71.0% (3.13) | 135.6 | 0.879 | correct |
|
| 192 |
|
| 193 |
-
BPW for GGUF Q3 LynnStyle is 17303770432 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 5.07; the Q3 NInfer package was 17300551168 bytes before the vision component was added, BPW 5.07. BPW for GGUF Q4 LynnStyle is 19620406592 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 5.75; the Q4 NInfer package was 19617187328 bytes before the vision component was added, BPW 5.74. BPW for GGUF Q2 LynnStyle is 13,276,009,792 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 3.89; the Q2 NInfer package was 13,272,798,720 bytes before the vision component was added, BPW 3.89. BPW for GGUF Q8_0 is 29,069,202,688 × 8 ÷ 27,320,697,856 = 8.51; the Q8_0 NInfer package was 29,065,983,488 bytes before the vision component was added, BPW 8.51. BPW for the two GGUF Q6_K packages is the single file's bytes × 8 ÷ the text + MTP parameter count 27,320,697,856: the GGUF is 23,177,516,384 bytes, giving 6.79, and the NInfer package was 23,084,144,128 bytes before the vision component was added, giving 6.76. The parameter count comes from the GGUF header (866 tensors: text 26,895,998,464 + MTP 424,699,392); vision is not counted in the denominator (neither the llama.cpp mmproj nor the 295,720,448-byte vision part now inside each `.ninfer`). The BF16 / FP8 / NVFP4 tiers in the main repository use 27,781,427,952 (including the vision tower), so BPW should not be compared directly across the two.
|
| 194 |
|
| 195 |
## Best-TPS and concurrency recommendations
|
| 196 |
|
|
@@ -423,6 +431,19 @@ llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf
|
|
| 423 |
|
| 424 |
To use another GGUF tier, only change `-m` to `...-Q3LynnStyle-MTP.gguf`, `...-Q4LynnStyle-MTP.gguf`, `...-Q6_K-MTP.gguf`, or `...-Q8_0-MTP.gguf` (and the `--alias`, e.g. `qwen3.8-27b-coder390-q6k-mtp`, `qwen3.8-27b-coder390-q8-mtp`). MTP is fused inside the GGUF (`blk.64`); do not attach an external draft; the log should show `creating MTP draft context against the target model`. For image input add `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`.
|
| 425 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 426 |
**llama.cpp · vision (image input)**
|
| 427 |
|
| 428 |
```bash
|
|
@@ -455,6 +476,7 @@ llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf
|
|
| 455 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf` | 17,303,770,432 | `9985b3d41fc7dc0bb5d493f7523d4515504912a3a7fc830f66e0fd2f90f9fd95` |
|
| 456 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-MTP.gguf` | 19,620,406,592 | `56c7c605a59764d9f0bb645d4eb335f1574af2c9e74020386139597df0e872f5` |
|
| 457 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf` | 13,276,009,792 | `cad973b3b7cc0d86bd39968209268318a746673c12c9536e5c37408baa7b5e6e` |
|
|
|
|
| 458 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf` | 13,488,346,432 | `62fd6cf6079dfb02b3e3aab68e39b4cdc6a471a51a82b56c9a76068e961d8a4a` |
|
| 459 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf` | 17,516,107,072 | `2c74a65df9d229bfba2eb3a878a25b84d4035d1be4061c9a665af213d98cae58` |
|
| 460 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf` | 19,832,743,232 | `5e2a1784298f696cb38f063962da0ad4acd1fd020c6e7c6e520df54acf8f57eb` |
|
|
@@ -535,6 +557,10 @@ Thanks to the Qwen team for the base model and the official FP8 scheme; to Opus5
|
|
| 535 |
- **GGUF Q2 LynnStyle · 思考长度(P50 / P70 / P90 token)**:GPQA 2,989.5 / 8,923.8 / 27,156.1;MMLU 181 / 326.3 / 1,020.5;LCB 5,656.5 / 17,835.8 / 40,067.3。
|
| 536 |
- **GGUF Q2 LynnStyle · 速度**:NInfer 内置 Q4 MTP,C8 合计 521.0 tok/s(生成上限 512 短测);llama.cpp 内置 Q4 MTP,C4 合计 174.8 tok/s(生成上限 4096 长测)。口径不同,勿直接横比。
|
| 537 |
- **GGUF Q2 LynnStyle · 体积**:GGUF 13,276,009,792 字节(BPW 3.89),NInfer 13,568,617,472 字节,含视觉(文本 + MTP BPW 3.89)。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 538 |
- **GGUF Q3 LynnStyle · 成绩**:GPQA 173/198、MMLU 449/500、LCB 92/100(llama.cpp,C4,100K 全量;成绩在内置 MTP 前测得(外挂 Q8 MTP 草稿),量化权重相同)。
|
| 539 |
- **GGUF Q3 LynnStyle · 思考长度(P50 / P70 / P90 token)**:GPQA 3,125.5 / 7,265.2 / 24,343.1;MMLU 148 / 279.3 / 923.4;LCB 4,421 / 14,232.2 / 34,946。
|
| 540 |
- **GGUF Q3 LynnStyle · 速度**:NInfer 内置 Q4 MTP,C8 合计 512.3 tok/s(生成上限 512 短测);llama.cpp 内置 Q4 MTP,C4 合计 191.8 tok/s(生成上限 4096 长测)。口径不同,勿直接横比。
|
|
@@ -613,6 +639,9 @@ Thanks to the Qwen team for the base model and the official FP8 scheme; to Opus5
|
|
| 613 |
| GPQA | GGUF Q2 LynnStyle | 177/198 | **2,989.5** / **8,923.8** / **27,156.1** | **3** | **0** |
|
| 614 |
| MMLU | GGUF Q2 LynnStyle | 433/500 | **181** / **326.3** / **1,020.5** | 1 | 0 |
|
| 615 |
| LCB | GGUF Q2 LynnStyle | **90/100** | **5,656.5** / **17,835.8** / **40,067.3** | **1** | **1** |
|
|
|
|
|
|
|
|
|
|
| 616 |
| GPQA | GGUF Q8_0 | 171/198 | **3,102.5** / **7,971** / **28,891.6** | **2** | **0** |
|
| 617 |
| GPQA | GGUF Q6_K | 174/198 | **3,683** / **9,814** / **27,966** | **2** | **2** |
|
| 618 |
| MMLU | 原版 Qwen3.8 27B(FP8) | 444/500 | 205 / 373 / 1,622 | 1 | 0 |
|
|
@@ -666,6 +695,7 @@ LCB 以 llama.cpp 的结果为准:LCB 测评脚本请求 NInfer 时收到的回
|
|
| 666 |
| GGUF Q4 LynnStyle · NInfer | `GGUF-NInfer/` | 由同一份 Q4 LynnStyle GGUF 转换的 NInfer 包(`.ninfer`),内置 Q4_K MTP 与视觉;NInfer-all 与 ninfer-fusion-kvmem 均可;成绩在内置 MTP 前测得(外挂 Q8 MTP 草稿),量化权重相同,成绩在 llama.cpp 上测得 | 19,913,006,080(约 18.5 GiB) | 5.74 | 175 / 448 / 89 |
|
| 667 |
| GGUF Q4 LynnStyle · Q8MTP | `GGUF/` | 同一个 Q4 LynnStyle GGUF,只把内置 MTP 头改为 Q8_0(`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf`),其他张量与原包逐字节相同;Q8_0 MTP 头备选版(原包的 Q4_0 MTP 头 llama.cpp 支持);llama.cpp 看图同样用 `GGUF/` 里的 mmproj | 19,832,743,232(约 18.5 GiB) | 5.81 | 175 / 448 / 89 |
|
| 668 |
| GGUF Q4 LynnStyle · Q8MTP · NInfer | `GGUF-NInfer/` | 由上面的 Q8MTP GGUF 转成的 NInfer 包(`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer`),内置 Q8_0 MTP 头和视觉组件,model id `qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp`;主体权重与原 Q4 LynnStyle 包相同,分数沿用原包 | 20,125,342,720(约 18.7 GiB) | 5.81 | 175 / 448 / 89 |
|
|
|
|
| 669 |
| GGUF Q2 LynnStyle | `GGUF/` | llama.cpp 用混合精度 LynnStyle GGUF(`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf`):最低精度到 Q2;内置 **Q4 MTP**(不是 Q8);视觉用 `GGUF/` 内的 mmproj(llama.cpp `--mmproj`) | 13,276,009,792(约 12.4 GiB) | 3.89 | 177 / 433 / 90 |
|
| 670 |
| GGUF Q2 LynnStyle · NInfer | `GGUF-NInfer/` | 由同一份 Q2 LynnStyle GGUF 转换的 NInfer 包(`.ninfer`),内置 Q4_K MTP 与视觉;NInfer-all 与 ninfer-fusion-kvmem 均可;成绩为 GGUF 在 llama.cpp 上测得 | 13,568,617,472(约 12.6 GiB) | 3.89 | 177 / 433 / 90 |
|
| 671 |
| GGUF Q2 LynnStyle · Q8MTP | `GGUF/` | 同一个 Q2 LynnStyle GGUF,只把内置 MTP 头改为 Q8_0(`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf`),其他张量与原包逐字节相同;Q8_0 MTP 头备选版(原包的 Q4_0 MTP 头 llama.cpp 支持);llama.cpp 看图同样用 `GGUF/` 里的 mmproj | 13,488,346,432(约 12.6 GiB) | 3.95 | 177 / 433 / 90 |
|
|
@@ -685,7 +715,7 @@ LCB 以 llama.cpp 的结果为准:LCB 测评脚本请求 NInfer 时收到的回
|
|
| 685 |
| Q3 LynnStyle · Q8MTP | 134.9 | 51.7%(2.55) | 138.3 | 0.904 | 正确 |
|
| 686 |
| Q4 LynnStyle · Q8MTP | 154.8 | 71.0%(3.13) | 135.6 | 0.879 | 正确 |
|
| 687 |
|
| 688 |
-
GGUF Q3 LynnStyle 的 BPW 按 17303770432 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 5.07;Q3 NInfer 加视觉前为 17300551168 字节、BPW 5.07。GGUF Q4 LynnStyle 的 BPW 按 19620406592 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 5.75;Q4 NInfer 加视觉前为 19617187328 字节、BPW 5.74。GGUF Q2 LynnStyle 的 BPW 按 13,276,009,792 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 3.89;Q2 NInfer 加视觉前为 13,272,798,720 字节、BPW 3.89。GGUF Q8_0 的 BPW 按 29,069,202,688 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 8.51;Q8_0 NInfer 加视觉前为 29,065,983,488 字节、BPW 8.51。GGUF Q6_K 两个包的 BPW 按单个文件字节数 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 计算:GGUF 为 23,177,516,384 字节、6.79,NInfer 加视觉前为 23,084,144,128 字节、6.76。参数数按 GGUF 头部统计(866 个张量:文本 26,895,998,464 + MTP 424,699,392);视觉不计入分母:llama.cpp 的外挂 mmproj 和现在打进每个 `.ninfer` 的视觉部分(295,720,448 字节)都不算。主仓库 BF16 / FP8 / NVFP4 各档按 27,781,427,952(含视觉塔)计算,分母不同,不宜直接横向比较 BPW。
|
| 689 |
|
| 690 |
## 最佳 TPS 推荐与并发推荐
|
| 691 |
|
|
@@ -919,6 +949,19 @@ llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf
|
|
| 919 |
|
| 920 |
换其他 GGUF 档只需改 `-m` 为 `...-Q3LynnStyle-MTP.gguf`、`...-Q4LynnStyle-MTP.gguf`、`...-Q6_K-MTP.gguf` 或 `...-Q8_0-MTP.gguf`(并相应改 `--alias`,例如 `qwen3.8-27b-coder390-q6k-mtp`、`qwen3.8-27b-coder390-q8-mtp`)。MTP 已融合进 GGUF(`blk.64`),不要再外挂 draft;日志应出现 `creating MTP draft context against the target model`。需要图像输入时加 `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`。
|
| 921 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 922 |
**llama.cpp · 视觉(图片输入)**
|
| 923 |
|
| 924 |
```bash
|
|
@@ -952,6 +995,7 @@ llama-server -m ./GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf
|
|
| 952 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf` | 17,303,770,432 | `9985b3d41fc7dc0bb5d493f7523d4515504912a3a7fc830f66e0fd2f90f9fd95` |
|
| 953 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-MTP.gguf` | 19,620,406,592 | `56c7c605a59764d9f0bb645d4eb335f1574af2c9e74020386139597df0e872f5` |
|
| 954 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf` | 13,276,009,792 | `cad973b3b7cc0d86bd39968209268318a746673c12c9536e5c37408baa7b5e6e` |
|
|
|
|
| 955 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf` | 13,488,346,432 | `62fd6cf6079dfb02b3e3aab68e39b4cdc6a471a51a82b56c9a76068e961d8a4a` |
|
| 956 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf` | 17,516,107,072 | `2c74a65df9d229bfba2eb3a878a25b84d4035d1be4061c9a665af213d98cae58` |
|
| 957 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf` | 19,832,743,232 | `5e2a1784298f696cb38f063962da0ad4acd1fd020c6e7c6e520df54acf8f57eb` |
|
|
|
|
| 40 |
- **GGUF Q2 LynnStyle · reasoning length (P50 / P70 / P90 tokens)**: GPQA 2,989.5 / 8,923.8 / 27,156.1; MMLU 181 / 326.3 / 1,020.5; LCB 5,656.5 / 17,835.8 / 40,067.3.
|
| 41 |
- **GGUF Q2 LynnStyle · speed**: NInfer with built-in Q4 MTP, C8 aggregate 521.0 tok/s (512-token cap, short run); llama.cpp with built-in Q4 MTP, C4 aggregate 174.8 tok/s (4096-token cap, long run). Protocols differ; do not compare directly.
|
| 42 |
- **GGUF Q2 LynnStyle · size**: GGUF 13,276,009,792 bytes (BPW 3.89), NInfer 13,568,617,472 bytes with vision (text + MTP BPW 3.89).
|
| 43 |
+
- **GGUF Q3 LynnStyle-16G · what it is**: our lowest Q3 tier, recommended for 16 GB VRAM (`GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf`). It changes the earlier Q2 scheme: the lowest type is now IQ3_XXS (no Q6 / Q8 in the body), built with importance-matrix correction and GSQ-RCO (only same-byte GSQ refinements were applied, so the size is unchanged). Built-in Q4_K MTP head (norms F32). Text-only GGUF; vision via the mmproj in `GGUF/`. No NInfer package for this tier yet.
|
| 44 |
+
- **GGUF Q3 LynnStyle-16G · scores**: GPQA 172/198, MMLU 427/500, LCB 90/100 (llama.cpp + DFlash2 external draft, C4, 100K context, max_tokens 94,208, no timeout, reasoning effort xhigh; full sets).
|
| 45 |
+
- **GGUF Q3 LynnStyle-16G · reasoning length (P50 / P70 / P90 tokens)**: GPQA 3,811 / 11,404 / 31,418; MMLU 190 / 361 / 1,665; LCB 6,520 / 21,586 / 43,184. GPQA uses `reasoning_tokens`; the MMLU and LCB records have no separate reasoning field, so `completion_tokens` (including the final answer) is used.
|
| 46 |
+
- **GGUF Q3 LynnStyle-16G · size**: GGUF 13,276,009,920 bytes (~12.4 GiB, BPW 3.89). Speed not measured yet.
|
| 47 |
- **GGUF Q3 LynnStyle · scores**: GPQA 173/198, MMLU 449/500, LCB 92/100 (llama.cpp, C4, 100K full sets; scored before the built-in MTP head was attached (external Q8 MTP draft); same quantized weights).
|
| 48 |
- **GGUF Q3 LynnStyle · reasoning length (P50 / P70 / P90 tokens)**: GPQA 3,125.5 / 7,265.2 / 24,343.1; MMLU 148 / 279.3 / 923.4; LCB 4,421 / 14,232.2 / 34,946.
|
| 49 |
- **GGUF Q3 LynnStyle · speed**: NInfer with built-in Q4 MTP, C8 aggregate 512.3 tok/s (512-token cap, short run); llama.cpp with built-in Q4 MTP, C4 aggregate 191.8 tok/s (4096-token cap, long run). Protocols differ; do not compare directly.
|
|
|
|
| 122 |
| GPQA | GGUF Q2 LynnStyle | 177/198 | **2,989.5** / **8,923.8** / **27,156.1** | **3** | **0** |
|
| 123 |
| MMLU | GGUF Q2 LynnStyle | 433/500 | **181** / **326.3** / **1,020.5** | 1 | 0 |
|
| 124 |
| LCB | GGUF Q2 LynnStyle | **90/100** | **5,656.5** / **17,835.8** / **40,067.3** | **1** | **1** |
|
| 125 |
+
| GPQA | GGUF Q3 LynnStyle-16G | 172/198 | **3,811** / **11,404** / **31,418** | **2** | **2** |
|
| 126 |
+
| MMLU | GGUF Q3 LynnStyle-16G | 427/500 | **190** / **361** / 1,665 | **0** | 0 |
|
| 127 |
+
| LCB | GGUF Q3 LynnStyle-16G | **90/100** | **6,520** / **21,586** / **43,184** | **3** | **3** |
|
| 128 |
| GPQA | GGUF Q8_0 | 171/198 | **3,102.5** / **7,971** / **28,891.6** | **2** | **0** |
|
| 129 |
| GPQA | GGUF Q6_K | 174/198 | **3,683** / **9,814** / **27,966** | **2** | **2** |
|
| 130 |
| MMLU | Original Qwen3.8 27B (FP8) | 444/500 | 205 / 373 / 1,622 | 1 | 0 |
|
|
|
|
| 178 |
| GGUF Q4 LynnStyle · NInfer | `GGUF-NInfer/` | NInfer package from the same Q4 LynnStyle GGUF (`.ninfer`), built-in Q4_K MTP and vision; NInfer-all and ninfer-fusion-kvmem; scored on llama.cpp before the built-in MTP head was attached (external Q8 MTP draft); same quantized weights | 19,913,006,080 (~18.5 GiB) | 5.74 | 175 / 448 / 89 |
|
| 179 |
| GGUF Q4 LynnStyle · Q8MTP | `GGUF/` | The same Q4 LynnStyle GGUF with only the built-in MTP head re-encoded as Q8_0 (`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf`); every other tensor is byte-identical to the original; an alternative with a Q8_0 MTP head (the original's Q4_0 MTP head is supported by llama.cpp); llama.cpp vision via the mmproj in `GGUF/` | 19,832,743,232 (~18.5 GiB) | 5.81 | 175 / 448 / 89 |
|
| 180 |
| GGUF Q4 LynnStyle · Q8MTP · NInfer | `GGUF-NInfer/` | NInfer package from the Q8MTP GGUF above (`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer`), built-in Q8_0 MTP head and vision, model id `qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp`; body weights identical to the original Q4 LynnStyle package, scores carried over | 20,125,342,720 (~18.7 GiB) | 5.81 | 175 / 448 / 89 |
|
| 181 |
+
| GGUF Q3 LynnStyle-16G | `GGUF/` | Our lowest Q3 tier, for 16 GB VRAM (`Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf`): lowest type IQ3_XXS, no Q6 / Q8 in the body; importance-matrix correction + GSQ-RCO (same-byte refinements, size unchanged); **built-in Q4_K MTP**; llama.cpp vision via the mmproj in `GGUF/`; scored with llama.cpp + DFlash2 external draft | 13,276,009,920 (~12.4 GiB) | 3.89 | 172 / 427 / 90 |
|
| 182 |
| GGUF Q2 LynnStyle | `GGUF/` | Mixed-precision LynnStyle GGUF for llama.cpp (`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf`): lowest type reaches Q2; **built-in Q4 MTP** (not Q8); llama.cpp vision via `--mmproj` with the mmproj in `GGUF/` | 13,276,009,792 (~12.4 GiB) | 3.89 | 177 / 433 / 90 |
|
| 183 |
| GGUF Q2 LynnStyle · NInfer | `GGUF-NInfer/` | NInfer package from the same Q2 LynnStyle GGUF (`.ninfer`), built-in Q4_K MTP and vision; NInfer-all and ninfer-fusion-kvmem; scores from GGUF on llama.cpp | 13,568,617,472 (~12.6 GiB) | 3.89 | 177 / 433 / 90 |
|
| 184 |
| GGUF Q2 LynnStyle · Q8MTP | `GGUF/` | The same Q2 LynnStyle GGUF with only the built-in MTP head re-encoded as Q8_0 (`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf`); every other tensor is byte-identical to the original; an alternative with a Q8_0 MTP head (the original's Q4_0 MTP head is supported by llama.cpp); llama.cpp vision via the mmproj in `GGUF/` | 13,488,346,432 (~12.6 GiB) | 3.95 | 177 / 433 / 90 |
|
|
|
|
| 198 |
| Q3 LynnStyle · Q8MTP | 134.9 | 51.7% (2.55) | 138.3 | 0.904 | correct |
|
| 199 |
| Q4 LynnStyle · Q8MTP | 154.8 | 71.0% (3.13) | 135.6 | 0.879 | correct |
|
| 200 |
|
| 201 |
+
BPW for GGUF Q3 LynnStyle is 17303770432 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 5.07; the Q3 NInfer package was 17300551168 bytes before the vision component was added, BPW 5.07. BPW for GGUF Q4 LynnStyle is 19620406592 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 5.75; the Q4 NInfer package was 19617187328 bytes before the vision component was added, BPW 5.74. BPW for GGUF Q2 LynnStyle is 13,276,009,792 × 8 ÷ the text + MTP parameter count 27,320,697,856 = 3.89; the Q2 NInfer package was 13,272,798,720 bytes before the vision component was added, BPW 3.89. BPW for GGUF Q8_0 is 29,069,202,688 × 8 ÷ 27,320,697,856 = 8.51; the Q8_0 NInfer package was 29,065,983,488 bytes before the vision component was added, BPW 8.51. BPW for the two GGUF Q6_K packages is the single file's bytes × 8 ÷ the text + MTP parameter count 27,320,697,856: the GGUF is 23,177,516,384 bytes, giving 6.79, and the NInfer package was 23,084,144,128 bytes before the vision component was added, giving 6.76. The parameter count comes from the GGUF header (866 tensors: text 26,895,998,464 + MTP 424,699,392); vision is not counted in the denominator (neither the llama.cpp mmproj nor the 295,720,448-byte vision part now inside each `.ninfer`). The BF16 / FP8 / NVFP4 tiers in the main repository use 27,781,427,952 (including the vision tower), so BPW should not be compared directly across the two. BPW for GGUF Q3 LynnStyle-16G is 13,276,009,920 × 8 ÷ 27,320,697,856 = 3.89.
|
| 202 |
|
| 203 |
## Best-TPS and concurrency recommendations
|
| 204 |
|
|
|
|
| 431 |
|
| 432 |
To use another GGUF tier, only change `-m` to `...-Q3LynnStyle-MTP.gguf`, `...-Q4LynnStyle-MTP.gguf`, `...-Q6_K-MTP.gguf`, or `...-Q8_0-MTP.gguf` (and the `--alias`, e.g. `qwen3.8-27b-coder390-q6k-mtp`, `qwen3.8-27b-coder390-q8-mtp`). MTP is fused inside the GGUF (`blk.64`); do not attach an external draft; the log should show `creating MTP draft context against the target model`. For image input add `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`.
|
| 433 |
|
| 434 |
+
**llama.cpp · GGUF Q3 LynnStyle-16G · MTP (16 GB VRAM)**
|
| 435 |
+
|
| 436 |
+
```bash
|
| 437 |
+
llama-server -m ./GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf \
|
| 438 |
+
-c 32768 -np 1 -ngl 99 --jinja \
|
| 439 |
+
--temp 1.0 --top-p 0.95 --top-k 20 \
|
| 440 |
+
--host 127.0.0.1 --port 8080 \
|
| 441 |
+
--alias qwen3.8-27b-coder390-q3lynnstyle16g-mtp \
|
| 442 |
+
--spec-type draft-mtp --spec-draft-n-max 4
|
| 443 |
+
```
|
| 444 |
+
|
| 445 |
+
Not yet tested on a 16 GB card; the command above uses one slot at 32K context, raise `-c` / `-np` if memory allows. MTP is fused inside the GGUF (Q4_K, `blk.64`); do not attach an external draft. For image input add `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`. The three-suite scores for this tier were measured with llama.cpp + DFlash2 external draft (C4, 100K context, 94,208 generation cap, no timeout, xhigh); speed not measured yet.
|
| 446 |
+
|
| 447 |
**llama.cpp · vision (image input)**
|
| 448 |
|
| 449 |
```bash
|
|
|
|
| 476 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf` | 17,303,770,432 | `9985b3d41fc7dc0bb5d493f7523d4515504912a3a7fc830f66e0fd2f90f9fd95` |
|
| 477 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-MTP.gguf` | 19,620,406,592 | `56c7c605a59764d9f0bb645d4eb335f1574af2c9e74020386139597df0e872f5` |
|
| 478 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf` | 13,276,009,792 | `cad973b3b7cc0d86bd39968209268318a746673c12c9536e5c37408baa7b5e6e` |
|
| 479 |
+
| `GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf` | 13,276,009,920 | `3872d17684a1b71d38fd36eb5ace047ac61bab11799926455de933fe0b3e60e1` |
|
| 480 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf` | 13,488,346,432 | `62fd6cf6079dfb02b3e3aab68e39b4cdc6a471a51a82b56c9a76068e961d8a4a` |
|
| 481 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf` | 17,516,107,072 | `2c74a65df9d229bfba2eb3a878a25b84d4035d1be4061c9a665af213d98cae58` |
|
| 482 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf` | 19,832,743,232 | `5e2a1784298f696cb38f063962da0ad4acd1fd020c6e7c6e520df54acf8f57eb` |
|
|
|
|
| 557 |
- **GGUF Q2 LynnStyle · 思考长度(P50 / P70 / P90 token)**:GPQA 2,989.5 / 8,923.8 / 27,156.1;MMLU 181 / 326.3 / 1,020.5;LCB 5,656.5 / 17,835.8 / 40,067.3。
|
| 558 |
- **GGUF Q2 LynnStyle · 速度**:NInfer 内置 Q4 MTP,C8 合计 521.0 tok/s(生成上限 512 短测);llama.cpp 内置 Q4 MTP,C4 合计 174.8 tok/s(生成上限 4096 长测)。口径不同,勿直接横比。
|
| 559 |
- **GGUF Q2 LynnStyle · 体积**:GGUF 13,276,009,792 字节(BPW 3.89),NInfer 13,568,617,472 字节,含视觉(文本 + MTP BPW 3.89)。
|
| 560 |
+
- **GGUF Q3 LynnStyle-16G · 简介**:我们最低的 Q3 档,推荐 16G 显存(`GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf`)。相对此前的 Q2 方案做了调整:最低精度改为 IQ3_XXS(主体无 Q6 / Q8),并做了重要性矩阵校正和 GSQ-RCO(只采用同字节的 GSQ 精修,体积不变)。内置 Q4_K MTP 头(norm 为 F32)。GGUF 只含文本,看图用 `GGUF/` 内的 mmproj。本档暂无 NInfer 包。
|
| 561 |
+
- **GGUF Q3 LynnStyle-16G · 成绩**:GPQA 172/198、MMLU 427/500、LCB 90/100(llama.cpp + 外挂 DFlash2 草稿,C4,100K 上下文,生成上限 94,208,不计超时,reasoning effort xhigh,全量)。
|
| 562 |
+
- **GGUF Q3 LynnStyle-16G · 思考长度(P50 / P70 / P90 token)**:GPQA 3,811 / 11,404 / 31,418;MMLU 190 / 361 / 1,665;LCB 6,520 / 21,586 / 43,184。GPQA 用 `reasoning_tokens`;MMLU 和 LCB 的记录没有单独的 reasoning 字段,用的是 `completion_tokens`(含最终答案)。
|
| 563 |
+
- **GGUF Q3 LynnStyle-16G · 体积**:GGUF 13,276,009,920 字节(约 12.4 GiB,BPW 3.89)。速度尚未测试。
|
| 564 |
- **GGUF Q3 LynnStyle · 成绩**:GPQA 173/198、MMLU 449/500、LCB 92/100(llama.cpp,C4,100K 全量;成绩在内置 MTP 前测得(外挂 Q8 MTP 草稿),量化权重相同)。
|
| 565 |
- **GGUF Q3 LynnStyle · 思考长度(P50 / P70 / P90 token)**:GPQA 3,125.5 / 7,265.2 / 24,343.1;MMLU 148 / 279.3 / 923.4;LCB 4,421 / 14,232.2 / 34,946。
|
| 566 |
- **GGUF Q3 LynnStyle · 速度**:NInfer 内置 Q4 MTP,C8 合计 512.3 tok/s(生成上限 512 短测);llama.cpp 内置 Q4 MTP,C4 合计 191.8 tok/s(生成上限 4096 长测)。口径不同,勿直接横比。
|
|
|
|
| 639 |
| GPQA | GGUF Q2 LynnStyle | 177/198 | **2,989.5** / **8,923.8** / **27,156.1** | **3** | **0** |
|
| 640 |
| MMLU | GGUF Q2 LynnStyle | 433/500 | **181** / **326.3** / **1,020.5** | 1 | 0 |
|
| 641 |
| LCB | GGUF Q2 LynnStyle | **90/100** | **5,656.5** / **17,835.8** / **40,067.3** | **1** | **1** |
|
| 642 |
+
| GPQA | GGUF Q3 LynnStyle-16G | 172/198 | **3,811** / **11,404** / **31,418** | **2** | **2** |
|
| 643 |
+
| MMLU | GGUF Q3 LynnStyle-16G | 427/500 | **190** / **361** / 1,665 | **0** | 0 |
|
| 644 |
+
| LCB | GGUF Q3 LynnStyle-16G | **90/100** | **6,520** / **21,586** / **43,184** | **3** | **3** |
|
| 645 |
| GPQA | GGUF Q8_0 | 171/198 | **3,102.5** / **7,971** / **28,891.6** | **2** | **0** |
|
| 646 |
| GPQA | GGUF Q6_K | 174/198 | **3,683** / **9,814** / **27,966** | **2** | **2** |
|
| 647 |
| MMLU | 原版 Qwen3.8 27B(FP8) | 444/500 | 205 / 373 / 1,622 | 1 | 0 |
|
|
|
|
| 695 |
| GGUF Q4 LynnStyle · NInfer | `GGUF-NInfer/` | 由同一份 Q4 LynnStyle GGUF 转换的 NInfer 包(`.ninfer`),内置 Q4_K MTP 与视觉;NInfer-all 与 ninfer-fusion-kvmem 均可;成绩在内置 MTP 前测得(外挂 Q8 MTP 草稿),量化权重相同,成绩在 llama.cpp 上测得 | 19,913,006,080(约 18.5 GiB) | 5.74 | 175 / 448 / 89 |
|
| 696 |
| GGUF Q4 LynnStyle · Q8MTP | `GGUF/` | 同一个 Q4 LynnStyle GGUF,只把内置 MTP 头改为 Q8_0(`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf`),其他张量与原包逐字节相同;Q8_0 MTP 头备选版(原包的 Q4_0 MTP 头 llama.cpp 支持);llama.cpp 看图同样用 `GGUF/` 里的 mmproj | 19,832,743,232(约 18.5 GiB) | 5.81 | 175 / 448 / 89 |
|
| 697 |
| GGUF Q4 LynnStyle · Q8MTP · NInfer | `GGUF-NInfer/` | 由上面的 Q8MTP GGUF 转成的 NInfer 包(`Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.ninfer`),内置 Q8_0 MTP 头和视觉组件,model id `qwen3.8-27b-coder390-Q4LynnStyle-Q8Mtp`;主体权重与原 Q4 LynnStyle 包相同,分数沿用原包 | 20,125,342,720(约 18.7 GiB) | 5.81 | 175 / 448 / 89 |
|
| 698 |
+
| GGUF Q3 LynnStyle-16G | `GGUF/` | 我们最低的 Q3 档,面向 16G 显存(`Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf`):最低精度 IQ3_XXS,主体无 Q6 / Q8;重要性矩阵校正 + GSQ-RCO(同字节精修,体积不变);内置 **Q4_K MTP**;看图用 `GGUF/` 内的 mmproj;成绩用 llama.cpp + 外挂 DFlash2 测得 | 13,276,009,920(约 12.4 GiB) | 3.89 | 172 / 427 / 90 |
|
| 699 |
| GGUF Q2 LynnStyle | `GGUF/` | llama.cpp 用混合精度 LynnStyle GGUF(`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf`):最低精度到 Q2;内置 **Q4 MTP**(不是 Q8);视觉用 `GGUF/` 内的 mmproj(llama.cpp `--mmproj`) | 13,276,009,792(约 12.4 GiB) | 3.89 | 177 / 433 / 90 |
|
| 700 |
| GGUF Q2 LynnStyle · NInfer | `GGUF-NInfer/` | 由同一份 Q2 LynnStyle GGUF 转换的 NInfer 包(`.ninfer`),内置 Q4_K MTP 与视觉;NInfer-all 与 ninfer-fusion-kvmem 均可;成绩为 GGUF 在 llama.cpp 上测得 | 13,568,617,472(约 12.6 GiB) | 3.89 | 177 / 433 / 90 |
|
| 701 |
| GGUF Q2 LynnStyle · Q8MTP | `GGUF/` | 同一个 Q2 LynnStyle GGUF,只把内置 MTP 头改为 Q8_0(`Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf`),其他张量与原包逐字节相同;Q8_0 MTP 头备选版(原包的 Q4_0 MTP 头 llama.cpp 支持);llama.cpp 看图同样用 `GGUF/` 里的 mmproj | 13,488,346,432(约 12.6 GiB) | 3.95 | 177 / 433 / 90 |
|
|
|
|
| 715 |
| Q3 LynnStyle · Q8MTP | 134.9 | 51.7%(2.55) | 138.3 | 0.904 | 正确 |
|
| 716 |
| Q4 LynnStyle · Q8MTP | 154.8 | 71.0%(3.13) | 135.6 | 0.879 | 正确 |
|
| 717 |
|
| 718 |
+
GGUF Q3 LynnStyle 的 BPW 按 17303770432 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 5.07;Q3 NInfer 加视觉前为 17300551168 字节、BPW 5.07。GGUF Q4 LynnStyle 的 BPW 按 19620406592 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 5.75;Q4 NInfer 加视觉前为 19617187328 字节、BPW 5.74。GGUF Q2 LynnStyle 的 BPW 按 13,276,009,792 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 3.89;Q2 NInfer 加视觉前为 13,272,798,720 字节、BPW 3.89。GGUF Q8_0 的 BPW 按 29,069,202,688 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 = 8.51;Q8_0 NInfer 加视觉前为 29,065,983,488 字节、BPW 8.51。GGUF Q6_K 两个包的 BPW 按单个文件字节数 × 8 ÷ 文本 + MTP 参数数 27,320,697,856 计算:GGUF 为 23,177,516,384 字节、6.79,NInfer 加视觉前为 23,084,144,128 字节、6.76。参数数按 GGUF 头部统计(866 个张量:文本 26,895,998,464 + MTP 424,699,392);视觉不计入分母:llama.cpp 的外挂 mmproj 和现在打进每个 `.ninfer` 的视觉部分(295,720,448 字节)都不算。主仓库 BF16 / FP8 / NVFP4 各档按 27,781,427,952(含视觉塔)计算,分母不同,不宜直接横向比较 BPW。GGUF Q3 LynnStyle-16G 的 BPW 按 13,276,009,920 × 8 ÷ 27,320,697,856 = 3.89。
|
| 719 |
|
| 720 |
## 最佳 TPS 推荐与并发推荐
|
| 721 |
|
|
|
|
| 949 |
|
| 950 |
换其他 GGUF 档只需改 `-m` 为 `...-Q3LynnStyle-MTP.gguf`、`...-Q4LynnStyle-MTP.gguf`、`...-Q6_K-MTP.gguf` 或 `...-Q8_0-MTP.gguf`(并相应改 `--alias`,例如 `qwen3.8-27b-coder390-q6k-mtp`、`qwen3.8-27b-coder390-q8-mtp`)。MTP 已融合进 GGUF(`blk.64`),不要再外挂 draft;日志应出现 `creating MTP draft context against the target model`。需要图像输入时加 `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`。
|
| 951 |
|
| 952 |
+
**llama.cpp · GGUF Q3 LynnStyle-16G · MTP(16G 显存)**
|
| 953 |
+
|
| 954 |
+
```bash
|
| 955 |
+
llama-server -m ./GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf \
|
| 956 |
+
-c 32768 -np 1 -ngl 99 --jinja \
|
| 957 |
+
--temp 1.0 --top-p 0.95 --top-k 20 \
|
| 958 |
+
--host 127.0.0.1 --port 8080 \
|
| 959 |
+
--alias qwen3.8-27b-coder390-q3lynnstyle16g-mtp \
|
| 960 |
+
--spec-type draft-mtp --spec-draft-n-max 4
|
| 961 |
+
```
|
| 962 |
+
|
| 963 |
+
本档未在 16G 卡上实测,上面按单路 32K 上下文给出,显存够再调大 `-c` / `-np`。MTP 已融合进 GGUF(Q4_K,`blk.64`),不要再外挂 draft。看图时加 `--mmproj ./GGUF/mmproj-Qwen3.8-27B-Q8_0.gguf`。本档的三科成绩是用 llama.cpp + 外挂 DFlash2 草稿测得(C4,100K 上下文,生成上限 94,208,不计超时,xhigh);速度尚未测试。
|
| 964 |
+
|
| 965 |
**llama.cpp · 视觉(图片输入)**
|
| 966 |
|
| 967 |
```bash
|
|
|
|
| 995 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-MTP.gguf` | 17,303,770,432 | `9985b3d41fc7dc0bb5d493f7523d4515504912a3a7fc830f66e0fd2f90f9fd95` |
|
| 996 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-MTP.gguf` | 19,620,406,592 | `56c7c605a59764d9f0bb645d4eb335f1574af2c9e74020386139597df0e872f5` |
|
| 997 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-MTP.gguf` | 13,276,009,792 | `cad973b3b7cc0d86bd39968209268318a746673c12c9536e5c37408baa7b5e6e` |
|
| 998 |
+
| `GGUF/Qwen3.8-27B-Coder390-Q3LynnStyle16G-MTP.gguf` | 13,276,009,920 | `3872d17684a1b71d38fd36eb5ace047ac61bab11799926455de933fe0b3e60e1` |
|
| 999 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q2LynnStyle-Q8MTP.gguf` | 13,488,346,432 | `62fd6cf6079dfb02b3e3aab68e39b4cdc6a471a51a82b56c9a76068e961d8a4a` |
|
| 1000 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q3LynnStyle-Q8MTP.gguf` | 17,516,107,072 | `2c74a65df9d229bfba2eb3a878a25b84d4035d1be4061c9a665af213d98cae58` |
|
| 1001 |
| `GGUF/Qwen3.8-27B-Coder390-EfficientThink-Q4LynnStyle-Q8MTP.gguf` | 19,832,743,232 | `5e2a1784298f696cb38f063962da0ad4acd1fd020c6e7c6e520df54acf8f57eb` |
|
|
Git LFS Details
|
|
Git LFS Details
|
|
Git LFS Details
|
|
Git LFS Details
|