How to use from
Ollama
ollama run hf.co/coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Quick Links

Laguna S 2.1 β€” attention-Q8 target + Q4_K_M DFlash (GGUF)

Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:

  • a layer-aware target derived from Poolside's official imatrix Q4_K_M GGUF, converting only its 240 F16 attention tensors to Q8_0;
  • a Q4_K_M conversion of Poolside's official Laguna S 2.1 DFlash draft model.

This is a modified quantized redistribution, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed.

Files

File Size Tensor layout Purpose
laguna-s-2.1-attnQ8.gguf 67.6 GiB 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 Layer-aware target
laguna-s-2.1-DFlash-Q4_K_M.gguf 622 MiB 45 Q4_K, 4 Q6_K, 27 F32 Speculative-decoding draft

The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization.

Why this layout

Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU.

The target conversion was intentionally narrow:

  • F16 attention β†’ Q8_0
  • Q4_K / Q6_K / F32 β†’ COPY unchanged
  • no expert-layer requantization
  • no router, norm, or bias quantization

RTX 2080 Ti results

AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's laguna llama.cpp branch, 262K context, one slot.

Target-only decode

Prompt language Official Q4_K_M attention-Q8 Change
English 19.640 tok/s 21.048 tok/s +7.2%
Traditional Chinese 19.733 tok/s 21.259 tok/s +7.7%

With Q4_K_M DFlash

Prompt language Official target + Q4 draft attention-Q8 + Q4 draft Change
English 27.812 tok/s 28.959 tok/s +4.1%
Traditional Chinese 18.906 tok/s 19.498 tok/s +3.1%

Strict JSON, native tool-call safety, and executable min_window coding gates passed after target quantization. Laguna's known failure modeβ€”spending the full completion budget in reasoning and returning no final contentβ€”was unchanged and should be handled at the agent layer.

Serving with llama.cpp

Use Poolside's llama.cpp fork, laguna branch. Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair.

./build/bin/llama-server \
  -m laguna-s-2.1-attnQ8.gguf \
  -md laguna-s-2.1-DFlash-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 4 \
  -ngld 99 \
  -c 262144 \
  -ctk q4_0 -ctv q4_0 \
  -ctkd f16 -ctvd f16 \
  -t 24 -tb 24 -td 12 -tbd 12 \
  -fa on \
  -fit on -fitt 7680 -fitc 262144 \
  --parallel 1 \
  --jinja \
  --reasoning on --reasoning-format deepseek \
  --host 0.0.0.0 --port 8095

Important DFlash caveats

Keep draft KV in F16. Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5–6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel.

Tune --spec-draft-n-max for the hardware and draft precision. Poolside's official BF16 example uses n=15. With this Q4_K_M draft on the tested Turing hybrid-offload system, n=4 was the best measured point. Larger values reduced acceptance enough to lose throughput.

The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (-ctkd and -ctvd).

Context and reasoning

This package was validated at 262,144 tokens. The native target was trained up to 1M context; see Poolside's official model card for the 1M YaRN override and quality caveats.

For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short max_tokens cap alone can consume the whole budget inside reasoning_content before a final answer is emitted.

Credits

License

Distributed under the OpenMDW-1.1 License, inherited from Laguna S 2.1. A complete copy is included as LICENSE.md. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution.


β˜• If this saved you GPU hours, you can buy me a coffee.


Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.

Provenance

This section documents release ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.

Source lineage

Source repository Revision used for this build Role
poolside/Laguna-S-2.1-GGUF not recorded Immediate official imatrix Q4_K_M target GGUF
poolside/Laguna-S-2.1-DFlash not recorded Immediate DFlash draft input
poolside/Laguna-S-2.1 not recorded Original target lineage

Build details

Tools and versions: Poolside llama.cpp fork, laguna branch; exact build-time commit/build not recorded.

For the target model, only the 240 F16 attention tensors were converted to Q8_0; the 239 Q4_K, 48 Q6_K and 287 F32 tensors were copied without changes. The draft model was converted to Q4_K_M. The published target recipe is copied below; the exact draft-conversion command is not recorded.

The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.

Recorded conversion command (published recipe; not executed for this provenance review):

llama-quantize \
  --token-embedding-type Q4_K \
  --output-tensor-type Q6_K \
  --tensor-type '^blk\.[0-9]+\.attn_(q|k|v|output|gate)\.weight$=Q8_0' \
  --tensor-type '^blk\.(0|1|2|3|4|5|8|11|14|17|20|23|26|29|32|35|38|41|42|43|44|45|46|47)\.ffn_down=Q6_K' \
  --tensor-type 'ffn_down=Q4_K' \
  --tensor-type 'ffn_gate=Q4_K' \
  --tensor-type 'ffn_up=Q4_K' \
  laguna-s-2.1-Q4_K_M.gguf laguna-s-2.1-attnQ8.gguf Q8_0 24

Weight file SHA-256 checksums

The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.

File Bytes SHA-256
laguna-s-2.1-DFlash-Q4_K_M.gguf 652160160 872be33424231b9e70ad95caf5c080825d110a0a4411bd66fa13c379651b56b9
laguna-s-2.1-attnQ8.gguf 72545160800 2d3d6f5a0a4fa534aad7bd1b67b00da59c2961a6c8bbe364ab8dbb376e842b54

Verifying the files

Download the desired files from revision ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.

Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.

Known gaps: Target/draft build-time upstream commits; Exact quantizer commit/build; Complete DFlash conversion invocation.

Sources: existing model card and HF metadata.

Downloads last month
830
GGUF
Model size
118B params
Architecture
laguna
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF

Quantized
(94)
this model