VishwamAI/vishwamai-distilled-gguf

VishwamAI — Distilled (GGUF, Q4_K_M)

Quantized GGUF build of VishwamAI-distilled, a defensive-first cybersecurity + math/science/coding assistant. It is Gemma 4 E4B (Apache 2.0), fine-tuned in stages and then knowledge-distilled from a larger teacher on verified examples (teacher answers were kept only when their code passed tests). Runs fast and locally with llama.cpp / Ollama / llama-cpp-python on modest hardware (4–6 GB GPU or CPU).

• Merged fp16 weights: kasinadhsarma/vishwamai-distilled
• LoRA adapter: kasinadhsarma/vishwamai-distilled-lora
• Lineage: Gemma 4 E4B → stage-0 SFT → stage-1 (CWE + humor) → distilled (verified teacher data)

⚠️ Authorized / educational use only. Use security capabilities solely on systems you own or are explicitly authorized to test (labs, CTFs, defensive research, education).

How it was distilled

• Teacher: a larger instruct model served via the Hugging Face Inference router.
• Verification-gated: coding answers were executed against tests and kept only if they passed; concept answers (ATT&CK / CWE / math) were kept as written. This prevents the student from copying teacher hallucinations.
• Student: Gemma 4 E4B (~8B incl. embeddings), trained with Unsloth QLoRA, merged to 16-bit, then quantized to GGUF.
• Note on teacher size: per the capacity-gap literature, a mid-size teacher (~12–16B) tends to produce a better ~8B student than a much larger one; teacher choice is a tuning dial.

Files

File	Quant	Size	Use
vishwamai-Q4_K_M.gguf	Q4_K_M	~5.3 GB	Best size/quality balance — recommended default

Run it

Ollama

printf 'FROM ./vishwamai-Q4_K_M.gguf\nSYSTEM "You are VishwamAI, a defensive-first security research assistant and a math/science solver. Be precise, state uncertainty, never fabricate CVE IDs or physical constants."\n' > Modelfile
ollama create vishwamai -f Modelfile
ollama run vishwamai

llama.cpp (OpenAI-compatible local server)

huggingface-cli download kasinadhsarma/vishwamai-distilled-gguf vishwamai-Q4_K_M.gguf --local-dir .
./llama-server -m vishwamai-Q4_K_M.gguf -ngl 99 -c 8192 --host 127.0.0.1 --port 8080
# then open http://127.0.0.1:8080 for the built-in chat UI

llama-cpp-python

from llama_cpp import Llama
llm = Llama.from_pretrained(
    "kasinadhsarma/vishwamai-distilled-gguf",
    filename="vishwamai-Q4_K_M.gguf",
    n_ctx=4096, n_gpu_layers=99,   # n_gpu_layers=0 for CPU
)
print(llm.create_chat_completion(messages=[
    {"role": "system", "content": "You are VishwamAI, a defensive-first security and math/science assistant."},
    {"role": "user", "content": "Give two Sysmon detections for LSASS credential dumping."},
])["choices"][0]["message"]["content"])

Recommended system prompt

You are VishwamAI, a defensive-first security research assistant and a
math/science solver for an authorized practitioner. Be precise, state uncertainty,
never fabricate CVE IDs or physical constants, and use the calculator/CAS tool for
any non-trivial computation.

Evaluation

Benchmark with the project's eval notebook (GSM8K · MMLU · MMMLU) against this GGUF, and compare base vs stage-1 vs distilled under identical settings — the deltas are what matter. For a defensive-security assistant, also report security MCQ (e.g., CyberMetric/SecQA) and a safety set (harmful-refusal / over-refusal / prompt-injection).

Limitations

• ~4B (8B incl. embeddings): a specialist, not a frontier generalist. A small distill set yields only small gains over stage-1 — scale the verified data for real improvement.
• May hallucinate facts (CVE IDs, ATT&CK details, constants). Route factual lookups through a RAG index and keep the "state uncertainty" system prompt.
• Quantized (Q4_K_M): minor quality loss vs fp16; use Q5_K_M / Q6_K for higher fidelity.
• Capacity limit: an 8B cannot equal a 27B+ generally, but a specialized 8B can rival larger general models on its narrow domain (defensive security + coding).

License

Apache 2.0 (inherited from the Gemma 4 base). Provided for authorized security research, education, and lawful use only.