--- license: openmdw-1.1 base_model: - poolside/Laguna-S-2.1 base_model_relation: quantized library_name: gguf pipeline_tag: text-generation tags: - laguna - laguna-s-2.1 - gguf - llama.cpp - mixture-of-experts - quantized - q4-k-m - q8-0 - dflash - speculative-decoding - turing --- # Laguna S 2.1 — attention-Q8 target + Q4_K_M DFlash (GGUF) Hardware-tuned GGUF pair for running [Poolside Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1) on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload: - a layer-aware target derived from Poolside's official imatrix Q4_K_M GGUF, converting only its 240 F16 attention tensors to Q8_0; - a Q4_K_M conversion of Poolside's official Laguna S 2.1 DFlash draft model. This is a **modified quantized redistribution**, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed. ## Files | File | Size | Tensor layout | Purpose | |---|---:|---|---| | `laguna-s-2.1-attnQ8.gguf` | 67.6 GiB | 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 | Layer-aware target | | `laguna-s-2.1-DFlash-Q4_K_M.gguf` | 622 MiB | 45 Q4_K, 4 Q6_K, 27 F32 | Speculative-decoding draft | The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization. ## Why this layout Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU. The target conversion was intentionally narrow: - **F16 attention → Q8_0** - **Q4_K / Q6_K / F32 → COPY unchanged** - no expert-layer requantization - no router, norm, or bias quantization ## RTX 2080 Ti results AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's `laguna` llama.cpp branch, 262K context, one slot. ### Target-only decode | Prompt language | Official Q4_K_M | attention-Q8 | Change | |---|---:|---:|---:| | English | 19.640 tok/s | **21.048 tok/s** | **+7.2%** | | Traditional Chinese | 19.733 tok/s | **21.259 tok/s** | **+7.7%** | ### With Q4_K_M DFlash | Prompt language | Official target + Q4 draft | attention-Q8 + Q4 draft | Change | |---|---:|---:|---:| | English | 27.812 tok/s | **28.959 tok/s** | **+4.1%** | | Traditional Chinese | 18.906 tok/s | **19.498 tok/s** | **+3.1%** | Strict JSON, native tool-call safety, and executable `min_window` coding gates passed after target quantization. Laguna's known failure mode—spending the full completion budget in reasoning and returning no final content—was unchanged and should be handled at the agent layer. ## Serving with llama.cpp Use [Poolside's llama.cpp fork, `laguna` branch](https://github.com/poolsideai/llama.cpp/tree/laguna). Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair. ```bash ./build/bin/llama-server \ -m laguna-s-2.1-attnQ8.gguf \ -md laguna-s-2.1-DFlash-Q4_K_M.gguf \ --spec-type draft-dflash \ --spec-draft-n-max 4 \ -ngld 99 \ -c 262144 \ -ctk q4_0 -ctv q4_0 \ -ctkd f16 -ctvd f16 \ -t 24 -tb 24 -td 12 -tbd 12 \ -fa on \ -fit on -fitt 7680 -fitc 262144 \ --parallel 1 \ --jinja \ --reasoning on --reasoning-format deepseek \ --host 0.0.0.0 --port 8095 ``` ### Important DFlash caveats **Keep draft KV in F16.** Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5–6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel. **Tune `--spec-draft-n-max` for the hardware and draft precision.** Poolside's official BF16 example uses `n=15`. With this Q4_K_M draft on the tested Turing hybrid-offload system, `n=4` was the best measured point. Larger values reduced acceptance enough to lose throughput. The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (`-ctkd` and `-ctvd`). ## Context and reasoning This package was validated at 262,144 tokens. The native target was trained up to 1M context; see [Poolside's official model card](https://huggingface.co/poolside/Laguna-S-2.1) for the 1M YaRN override and quality caveats. For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short `max_tokens` cap alone can consume the whole budget inside `reasoning_content` before a final answer is emitted. ## Credits - Base model and architecture: [poolside/Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) - Official GGUF and importance matrix: [poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF) - DFlash draft: [poolside/Laguna-S-2.1-DFlash](https://huggingface.co/poolside/Laguna-S-2.1-DFlash) - Layer-aware target conversion, Q4_K_M draft conversion, Turing tuning, and validation: **coolthor** ## License Distributed under the [OpenMDW-1.1 License](https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md), inherited from Laguna S 2.1. A complete copy is included as `LICENSE.md`. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution. --- ☕ If this saved you GPU hours, you can [buy me a coffee](https://buymeacoffee.com/thorlin). --- Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on [ai-muninn.com](https://ai-muninn.com). More quantized models: [huggingface.co/coolthor](https://huggingface.co/coolthor).