Add quantized models with per-model cards, MODELFILE, CLI examples, and auto-upload

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Coder-30B-A3B-Instruct-f32_Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

MODELFILE

@@ -0,0 +1,25 @@
+# MODELFILE for Qwen3-Coder-30B-A3B-Instruct-GGUF
+# Used by LM Studio, OpenWebUI, GPT4All, etc.
+
+context_length: 32768
+embedding: false
+f16: cpu
+
+# Chat template using ChatML (used by Qwen)
+prompt_template: >-
+         <|im_start|>system
+       You are a helpful assistant.<|im_end|>
+         <|im_start|>user
+       {prompt}<|im_end|>
+         <|im_start|>assistant
+
+# Stop sequences help end generation cleanly
+stop: "<|im_end|>"
+stop: "<|im_start|>"
+
+# Default sampling
+temperature: 0.6
+top_p: 0.95
+top_k: 20
+min_p: 0.0
+repeat_penalty: 1.1

Qwen3-Coder-30B-A3B-Instruct-f32_Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f9552b2ca5db4a596425d241bad4ba526be5f82bf5f10f40f35ebde8b7ff5b6
+size 11258611392

Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:536f4c5dcd45b35ca2a3b59dd1655c29296867c8791a0b2a09c55e9a34b8b695
+size 14711848640

Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e08750999da8b83dfa780ffd82919c46e93e7e6695fd0824206eaf39bec67414
+size 13292469952

Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2aa0dec45bddb00775d7cfd5d79680435f9431284e76834e49bd7152a5fd499e
+size 18556688064

Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d4f5c50d14f7afefcb4c728d4e2e11d1a1f5318fdb6551745054f51ee9b8fb8
+size 17456010944

Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:637c0f56860e92cf1cacea82d0169b44fd4833dd0278aaf6c3a4c8cdbd830a0b
+size 21725583040

Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:518e5d9c2d18f7a394889e4973a1bce78c05614dc0f9c04a446c03185da9f68c
+size 21080512192

Qwen3-Coder-30B-A3B-Instruct-f32_Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ee845097a4f01f69140748a1e33401a53567d494e448d9c907efd2bfc71ecbe
+size 25092533952

Qwen3-Coder-30B-A3B-Instruct-f32_Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:888b757556626a402b2c2b27d93d56307ddcb92f662c59c91a623dce89cecf1f
+size 32483933888

README.md

@@ -0,0 +1,102 @@
+---
+license: apache-2.0
+tags:
+  - gguf
+  - qwen
+  - llama.cpp
+  - quantized
+  - text-generation
+  - reasoning   - agent   - multilingual
+base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct
+author: geoffmunn
+pipeline_tag: text-generation
+language:
+  - en
+  - zh
+  - es
+  - fr
+  - de
+  - ru
+  - ar
+  - ja
+  - ko
+  - hi
+---
+
+# Qwen3-Coder-30B-A3B-Instruct-GGUF
+
+This is a **GGUF-quantized version** of the **[Qwen/Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct)** language model — a  Converted for use with \llama.cpp\, [LM Studio](https://lmstudio.ai), [OpenWebUI](https://openwebui.com), [GPT4All](https://gpt4all.io), and more.
+
+💡 **Key Features of Qwen3-Coder-30B-A3B-Instruct:**
+
+
+
+
+    @"
+    
+    ## 💡 Why f16 (not f32)?
+    
+    This model uses **FP16 (16-bit floating point)** as its base precision, not full FP32 (32-bit). Here's why:
+    
+    - **FP16 (Half Precision)**:
+      - Uses **~50% less memory** than FP32.
+      - **Sufficient for inference** quality in modern LLMs.
+      - Supported natively by **GPUs (NVIDIA/AMD/Apple)** and optimized in `llama.cpp`.
+      - **No perceptible quality loss** compared to FP32 for most tasks.
+      - Standard for GGUF models in the community.
+    
+    - **FP32 (Full Precision)**:
+      - Rarely used for inference due to **double the RAM/VRAM usage**.
+      - Only needed for **extreme numerical stability** (e.g. scientific simulations).
+      - **Not recommended** for LLM chat or coding tasks.
+    
+    ✅ **Conclusion**: `f16` is the **sweet spot** — high fidelity, efficient, and widely compatible. Quantized versions (Q4_K_M, Q5_K_M, etc.) trade a small amount of this quality for massive speed and memory gains.
+    
+    "@
+    } else {
+    @"
+    
+    ## 💡 Why f32?
+    
+    This model uses **FP32 (32-bit floating point)** as its base precision. This is unusual for GGUF models because:
+    
+    - FP32 doubles memory usage vs FP16.
+    - Modern LLMs (including Qwen3) are trained in mixed precision and **do not benefit** from FP32 at inference time.
+    - Only useful for **debugging**, **research**, or **extreme numerical robustness**.
+    
+    ⚠️ Consider converting from `f32` → `f16` first using `llama-convert` if you control the source.
+    
+"@
+
+
+## Available Quantizations (from f32)
+
+| Level     | Quality       | Speed     | Size      | Recommendation |
+|----------|--------------|----------|-----------|----------------|
+| Q2_K | Minimal | ⚡ Fast | 19.5 GB | Only on severely memory-constrained systems. | | Q3_K_S | Low-Medium | ⚡ Fast | 22.2 GB | Minimal viability; avoid unless space-limited. | | Q3_K_M | Low-Medium | ⚡ Fast | 23.3 GB | Acceptable for basic interaction. | | Q4_K_S | Practical | ⚡ Fast | 27.0 GB | Good balance for mobile/embedded platforms. | | Q4_K_M | Practical | ⚡ Fast | 28.1 GB | Best overall choice for most users. | | Q5_K_S | Max Reasoning | 🐢 Medium | 31.5 GB | Slight quality gain; good for testing. | | Q5_K_M | Max Reasoning | 🐢 Medium | 32.2 GB | Best quality available. Recommended. | | Q6_K | Near-FP16 | 🐌 Slow | 36.5 GB | Diminishing returns. Only if RAM allows. | | Q8_0 | Lossless* | 🐌 Slow | 48.0 GB | Maximum fidelity. Ideal for archival. |
+> 💡 **Recommendations by Use Case**
+>
+  > - - 💻 **Standard Laptop (i5/M1 Mac)**: Q5_K_M (optimal quality)
+- 🧠 **Reasoning, Coding, Math**: Q5_K_M or Q6_K
+- 🔍 **RAG, Retrieval, Precision Tasks**: Q6_K or Q8_0
+- 🤖 **Agent & Tool Integration**: Q5_K_M
+- 🛠️ **Development & Testing**: Test from Q4_K_M up to Q8_0
+
+## Usage
+
+Load this model using:
+- [OpenWebUI](https://openwebui.com) – self-hosted AI interface with RAG & tools
+- [LM Studio](https://lmstudio.ai) – desktop app with GPU support
+- [GPT4All](https://gpt4all.io) – private, offline AI chatbot
+- Or directly via \llama.cpp\
+
+Each quantized model includes its own \README.md\ and shares a common \MODELFILE\.
+
+## Author
+
+👤 Geoff Munn (@geoffmunn)  
+🔗 [Hugging Face Profile](https://huggingface.co/geoffmunn)
+
+## Disclaimer
+
+This is a community conversion for local inference. Not affiliated with Alibaba Cloud or the Qwen team.

SHA256SUMS.txt

@@ -0,0 +1,9 @@
+7f9552b2ca5db4a596425d241bad4ba526be5f82bf5f10f40f35ebde8b7ff5b6  Qwen3-Coder-30B-A3B-Instruct-f32_Q2_K.gguf
+536f4c5dcd45b35ca2a3b59dd1655c29296867c8791a0b2a09c55e9a34b8b695  Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_M.gguf
+e08750999da8b83dfa780ffd82919c46e93e7e6695fd0824206eaf39bec67414  Qwen3-Coder-30B-A3B-Instruct-f32_Q3_K_S.gguf
+2aa0dec45bddb00775d7cfd5d79680435f9431284e76834e49bd7152a5fd499e  Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_M.gguf
+3d4f5c50d14f7afefcb4c728d4e2e11d1a1f5318fdb6551745054f51ee9b8fb8  Qwen3-Coder-30B-A3B-Instruct-f32_Q4_K_S.gguf
+637c0f56860e92cf1cacea82d0169b44fd4833dd0278aaf6c3a4c8cdbd830a0b  Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_M.gguf
+518e5d9c2d18f7a394889e4973a1bce78c05614dc0f9c04a446c03185da9f68c  Qwen3-Coder-30B-A3B-Instruct-f32_Q5_K_S.gguf
+9ee845097a4f01f69140748a1e33401a53567d494e448d9c907efd2bfc71ecbe  Qwen3-Coder-30B-A3B-Instruct-f32_Q6_K.gguf
+888b757556626a402b2c2b27d93d56307ddcb92f662c59c91a623dce89cecf1f  Qwen3-Coder-30B-A3B-Instruct-f32_Q8_0.gguf