QaptaanLM-0.75B-Instruct-GGUF

This repository contains official GGUF quantized formats and turnkey Ollama Modelfiles for QaptaanLM-0.75B-Instruct, an efficient 752M hybrid linear attention foundation programming model.

---

📦 Quantization Matrix & Download Sizes

#	Quantization	File Name	Size	Memory Required	Quality & Description
1	FP16	qaptaan-0.75b-instruct-f16.gguf	1.45 GB	~2.0 GB	Full 16-bit float unquantized baseline master.
2	BF16	qaptaan-0.75b-instruct-bf16.gguf	1.45 GB	~2.0 GB	16-bit BFloat precision GGUF.
3	Q8_0	qaptaan-0.75b-instruct-q8_0.gguf	774 MB	~1.1 GB	8-bit integer quantization (99.9% quality retention).
4	Q6_K	qaptaan-0.75b-instruct-q6_k.gguf	600 MB	~900 MB	6-bit K-quantization for high-fidelity code synthesis.
5	Q5_K_M	qaptaan-0.75b-instruct-q5_k_m.gguf	551 MB	~800 MB	5-bit K-quantization (Medium tensor split - Sweet Spot).
6	Q5_K_S	qaptaan-0.75b-instruct-q5_k_s.gguf	537 MB	~780 MB	5-bit K-quantization (Small tensor split).
7	Q5_0	qaptaan-0.75b-instruct-q5_0.gguf	537 MB	~780 MB	5-bit standard legacy quantization.
8	Q4_K_M ⭐	qaptaan-0.75b-instruct-q4_k_m.gguf	504 MB	~720 MB	Recommended. Best balance of speed, size, and reasoning quality.
9	Q4_K_S	qaptaan-0.75b-instruct-q4_k_s.gguf	481 MB	~700 MB	4-bit K-quantization (Small).
10	Q4_0	qaptaan-0.75b-instruct-q4_0.gguf	478 MB	~700 MB	4-bit legacy quantization.
11	Q3_K_M	qaptaan-0.75b-instruct-q3_k_m.gguf	444 MB	~650 MB	3-bit K-quantization (Medium).
12	Q3_K_S	qaptaan-0.75b-instruct-q3_k_s.gguf	415 MB	~600 MB	3-bit K-quantization (Small).
13	Q2_K	qaptaan-0.75b-instruct-q2_k.gguf	402 MB	~550 MB	2-bit extreme compression for ultra low-memory micro-devices.

---

🚀 Quickstart: Running with Ollama

Every quantization format has a corresponding pre-configured Modelfile (e.g., Modelfile-Q4_K_M, Modelfile-Q8_0):

# 1. Download the desired GGUF weights and Modelfile
huggingface-cli download kaptaan45/QaptaanLM-0.75B-Instruct-GGUF qaptaan-0.75b-instruct-q4_k_m.gguf Modelfile-Q4_K_M --local-dir ./qaptaan

# 2. Create the Ollama model
ollama create qaptaan-instruct -f ./qaptaan/Modelfile-Q4_K_M

# 3. Chat with QaptaanLM
ollama run qaptaan-instruct "Write a Python function to compute the Fibonacci sequence using dynamic programming."

---

🛠️ Quickstart: Running with llama.cpp

Interactive CLI Chat:

./llama-cli -m qaptaan-0.75b-instruct-q4_k_m.gguf \
  -p "<|im_start|>system\nYou are QaptaanLM, an expert AI programming assistant.<|im_end|>\n<|im_start|>user\nWrite a Rust function for binary search.<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 --temp 0.20 --repeat-penalty 1.12 -r "<|im_end|>"

Local OpenAI-Compatible Server:

./llama-server -m qaptaan-0.75b-instruct-q4_k_m.gguf --port 8080 -c 4096

---

License

Released under the Apache 2.0 License.