Instructions to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Use Docker
docker model run hf.co/coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
- Ollama
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with Ollama:
ollama run hf.co/coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with Docker Model Runner:
docker model run hf.co/coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
- Lemonade
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Laguna S 2.1 β attention-Q8 target + Q4_K_M DFlash (GGUF)
Hardware-tuned GGUF pair for running Poolside Laguna S 2.1 on a single 22 GB RTX 2080 Ti with CPU/GPU hybrid offload:
- a layer-aware target derived from Poolside's official imatrix Q4_K_M GGUF, converting only its 240 F16 attention tensors to Q8_0;
- a Q4_K_M conversion of Poolside's official Laguna S 2.1 DFlash draft model.
This is a modified quantized redistribution, not a fine-tune and not an official Poolside release. The target's existing Q4_K, Q6_K, router, norm, and bias tensors were byte-preserved; only the F16 attention tensors were changed.
Files
| File | Size | Tensor layout | Purpose |
|---|---|---|---|
laguna-s-2.1-attnQ8.gguf |
67.6 GiB | 239 Q4_K, 48 Q6_K, 240 Q8_0, 287 F32 | Layer-aware target |
laguna-s-2.1-DFlash-Q4_K_M.gguf |
622 MiB | 45 Q4_K, 4 Q6_K, 27 F32 | Speculative-decoding draft |
The original official target GGUF is about 75 GB. Converting its 240 F16 attention tensors to Q8_0 reduced tensor payload by 2,506 MiB while preserving every already-quantized tensor without requantization.
Why this layout
Laguna S 2.1 is a 118B-A8B MoE. Its official Q4_K_M GGUF deliberately leaves attention in F16, while experts use Q4_K/Q6_K and sensitive router/norm/bias tensors remain F32. On a 22 GB Turing card, target weights are mostly CPU-resident, so reducing attention traffic lets llama.cpp fit more target data on GPU.
The target conversion was intentionally narrow:
- F16 attention β Q8_0
- Q4_K / Q6_K / F32 β COPY unchanged
- no expert-layer requantization
- no router, norm, or bias quantization
RTX 2080 Ti results
AI Lab test system: EPYC 7402, 128 GB eight-channel DDR4 ECC, one modified RTX 2080 Ti 22 GB for Laguna, Poolside's laguna llama.cpp branch, 262K context, one slot.
Target-only decode
| Prompt language | Official Q4_K_M | attention-Q8 | Change |
|---|---|---|---|
| English | 19.640 tok/s | 21.048 tok/s | +7.2% |
| Traditional Chinese | 19.733 tok/s | 21.259 tok/s | +7.7% |
With Q4_K_M DFlash
| Prompt language | Official target + Q4 draft | attention-Q8 + Q4 draft | Change |
|---|---|---|---|
| English | 27.812 tok/s | 28.959 tok/s | +4.1% |
| Traditional Chinese | 18.906 tok/s | 19.498 tok/s | +3.1% |
Strict JSON, native tool-call safety, and executable min_window coding gates passed after target quantization. Laguna's known failure modeβspending the full completion budget in reasoning and returning no final contentβwas unchanged and should be handled at the agent layer.
Serving with llama.cpp
Use Poolside's llama.cpp fork, laguna branch. Generic upstream llama.cpp does not yet contain the complete Laguna DFlash decoder contract required by this pair.
./build/bin/llama-server \
-m laguna-s-2.1-attnQ8.gguf \
-md laguna-s-2.1-DFlash-Q4_K_M.gguf \
--spec-type draft-dflash \
--spec-draft-n-max 4 \
-ngld 99 \
-c 262144 \
-ctk q4_0 -ctv q4_0 \
-ctkd f16 -ctvd f16 \
-t 24 -tb 24 -td 12 -tbd 12 \
-fa on \
-fit on -fitt 7680 -fitc 262144 \
--parallel 1 \
--jinja \
--reasoning on --reasoning-format deepseek \
--host 0.0.0.0 --port 8095
Important DFlash caveats
Keep draft KV in F16. Q4_0 draft KV reduced proposal acceptance below 1% and decode to roughly 5β6 tok/s. Q4_K_M draft weights and Q4_0 runtime KV are independent quantizations; their errors do not cancel.
Tune --spec-draft-n-max for the hardware and draft precision. Poolside's official BF16 example uses n=15. With this Q4_K_M draft on the tested Turing hybrid-offload system, n=4 was the best measured point. Larger values reduced acceptance enough to lose throughput.
The target KV can remain Q4_0; the precision warning applies specifically to the DFlash draft K/V cache (-ctkd and -ctvd).
Context and reasoning
This package was validated at 262,144 tokens. The native target was trained up to 1M context; see Poolside's official model card for the 1M YaRN override and quality caveats.
For tool/coding action lanes, consider disabling thinking or adding an empty-final repair. A short max_tokens cap alone can consume the whole budget inside reasoning_content before a final answer is emitted.
Credits
- Base model and architecture: poolside/Laguna-S-2.1
- Official GGUF and importance matrix: poolside/Laguna-S-2.1-GGUF
- DFlash draft: poolside/Laguna-S-2.1-DFlash
- Layer-aware target conversion, Q4_K_M draft conversion, Turing tuning, and validation: coolthor
License
Distributed under the OpenMDW-1.1 License, inherited from Laguna S 2.1. A complete copy is included as LICENSE.md. The files in this repository were modified through quantization; Poolside does not endorse or maintain this redistribution.
β If this saved you GPU hours, you can buy me a coffee.
Made by Thor Lin. Full benchmarks, setup notes and the runs that failed are on ai-muninn.com. More quantized models: huggingface.co/coolthor.
Provenance
This section documents release ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd, inspected on 2026-11-14. Where the exact upstream commit used at build time is unknown, it is listed as not recorded rather than backfilled with today's upstream HEAD.
Source lineage
| Source repository | Revision used for this build | Role |
|---|---|---|
| poolside/Laguna-S-2.1-GGUF | not recorded | Immediate official imatrix Q4_K_M target GGUF |
| poolside/Laguna-S-2.1-DFlash | not recorded | Immediate DFlash draft input |
| poolside/Laguna-S-2.1 | not recorded | Original target lineage |
Build details
Tools and versions: Poolside llama.cpp fork, laguna branch; exact build-time commit/build not recorded.
For the target model, only the 240 F16 attention tensors were converted to Q8_0; the 239 Q4_K, 48 Q6_K and 287 F32 tensors were copied without changes. The draft model was converted to Q4_K_M. The published target recipe is copied below; the exact draft-conversion command is not recorded.
The existing records cover quantization and format conversion only; they don't mention any fine-tuning or merging by coolthor. That reflects the documentation, not an independent weight diff.
Recorded conversion command (published recipe; not executed for this provenance review):
llama-quantize \
--token-embedding-type Q4_K \
--output-tensor-type Q6_K \
--tensor-type '^blk\.[0-9]+\.attn_(q|k|v|output|gate)\.weight$=Q8_0' \
--tensor-type '^blk\.(0|1|2|3|4|5|8|11|14|17|20|23|26|29|32|35|38|41|42|43|44|45|46|47)\.ffn_down=Q6_K' \
--tensor-type 'ffn_down=Q4_K' \
--tensor-type 'ffn_gate=Q4_K' \
--tensor-type 'ffn_up=Q4_K' \
laguna-s-2.1-Q4_K_M.gguf laguna-s-2.1-attnQ8.gguf Q8_0 24
Weight file SHA-256 checksums
The values below are Hugging Face LFS object SHA-256 values for the pinned release above. File sizes and digests agree with pinned tree metadata. They identify full file contents, not LFS pointer text or Git blob IDs. Calibration imatrices are included when present.
| File | Bytes | SHA-256 |
|---|---|---|
laguna-s-2.1-DFlash-Q4_K_M.gguf |
652160160 | 872be33424231b9e70ad95caf5c080825d110a0a4411bd66fa13c379651b56b9 |
laguna-s-2.1-attnQ8.gguf |
72545160800 | 2d3d6f5a0a4fa534aad7bd1b67b00da59c2961a6c8bbe364ab8dbb376e842b54 |
Verifying the files
Download the desired files from revision ea6cd8f8fd070f48a8c9bdbb8ee96838d1639cfd and compare sha256sum <file> (Linux) or shasum -a 256 <file> (macOS) with the table. If cloning with Git LFS, hash the downloaded weight files, not the LFS pointer files. A matching hash confirms your files match this release; it does not prove the weights came only from the listed base model with no other changes.
Checking how the weights were derived requires the exact upstream commit, tool versions, recipe and any calibration or training data, then re-running the build and diffing the weights with a format-aware tool (or comparing file hashes where serialization is deterministic). Because of the gaps listed below, this build can't be reproduced exactly today. Quantized files are expected to differ numerically from BF16 sources, and a raw file hash cannot establish a base-to-quant weight diff. No quantization rerun or weight-diff audit was performed for this provenance review.
Known gaps: Target/draft build-time upstream commits; Exact quantizer commit/build; Complete DFlash conversion invocation.
Sources: existing model card and HF metadata.
- Downloads last month
- 830
4-bit
Model tree for coolthor/Laguna-S-2.1-attnQ8-DFlash-Q4-GGUF
Base model
poolside/Laguna-S-2.1