Qwen2.5-7B-Instruct-AWQ-gfx1100

AWQ quantization for AMD gfx1100 (RX 7900 XTX) verified with vLLM + ROCm

---

📊 Results

Format	VRAM Usage	Reduction
fp16 original	22.9GB (93%)	baseline
AWQ (this model)	14.9GB (62%)	~33% reduction

Tested on:

• GPU: AMD Radeon RX 7900 XTX (gfx1100, 24GB)
• ROCm: 7.2.3
• vLLM: latest ROCm build
• OS: Ubuntu 24.04

---

🚀 Quick Start

1. Pull the vLLM ROCm image

docker run -it --rm \
  --network ai-net \
  --name vllm-qwen \
  --entrypoint /bin/bash \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --ipc=host \
  -p 9000:8000 \
  -v ~/ai_work/vllm_models:/models \
  -e ROCBLAS_USE_HIPBLASLT=1 \
  -e HIP_VISIBLE_DEVICES=0 \
  vllm/vllm-openai-rocm:latest

2. Download the model (inside container)

hf download rakisis-core/Qwen2.5-7B-Instruct-AWQ-gfx1100 \
  --local-dir /models/Qwen2.5-7B-Instruct-AWQ-gfx1100

3. Run vLLM server

export ROCBLAS_USE_HIPBLASLT=1
export HIP_VISIBLE_DEVICES=0

vllm serve /models/Qwen2.5-7B-Instruct-AWQ-gfx1100 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype float16 \
  --gpu-memory-utilization 0.70 \
  --max-model-len 16384 \
  --max-num-seqs 8 \
  --enforce-eager

4. Test

curl -s http://localhost:9000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/Qwen2.5-7B-Instruct-AWQ-gfx1100",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }' | python3 -m json.tool

---

⚠️ Key Requirements

ROCBLAS_USE_HIPBLASLT=1 is MANDATORY

export ROCBLAS_USE_HIPBLASLT=1
export HIP_VISIBLE_DEVICES=0

Without this, performance drops significantly. Verified on vLLM + AWQ (Qwen2.5-7B-Instruct-AWQ, gfx1100):

	hipBLASLt ON	hipBLASLt OFF
Generation throughput	53 tok/s	29 tok/s
Difference	✅ ~2x faster	baseline

This activates hipBLASLt kernels instead of Tensile — critical for ALL gfx1100 workloads.
Applies to both llama.cpp and vLLM.

AWQ Installation (if needed)

pip install autoawq

---

🌍 Language Support & Quality

Independent evaluation by OpenAI GPT:

Language	Score	Grade	Notes
English	78/100	B+	Most stable, natural output
Japanese	72/100	B	General text good, technical lacks depth
Korean	67/100	B-	Natural but limited depth, long-form may mix Chinese
German	60/100	C	Grammar errors present, translation-like feel
Chinese	—	Native	Base model language

Evaluation criteria: Grammar accuracy, vocabulary range, naturalness, creative depth
All languages show typical AI patterns: balanced structure, positive tone, predictable conclusions

---

📈 Benchmarks

Metric	fp16	AWQ
VRAM	22.9GB (93%)	14.9GB (62%)
Context (max)	8192	16384
Stability	Good	Excellent
KV Cache	Manual	Page Attention

---

🛠️ How We Did It

1. RX 7900 XTX supports INT4 Matrix: 246 TOPs in hardware
2. The limitation was software only — not a hardware problem
3. Combined with ROCBLAS_USE_HIPBLASLT=1 → full AWQ inference works

The key insight: When you quantize ON AMD hardware, Triton acts as a compiler — not a runtime translator. The ROCm-optimized TritonW4A16 kernel gets baked in at quantization time. At runtime, vLLM sees a kernel already aligned to gfx1100 architecture and runs it natively. No flag needed. No translation overhead. No gibberish.

If my understanding of the Triton kernel compilation is incorrect, please let me know. Happy to be corrected.

---

🔧 Compatible GPUs

Should work on all RDNA3 gfx1100 series:

GPU	VRAM	Status
RX 7900 XTX	24GB	✅ Verified
RX 7900 XT	20GB	🔲 Should work
RX 7900 GRE	16GB	🔲 Should work
RX 7800 XT	16GB	🔲 Should work
RX 7700 XT	12GB	🔲 Should work

If you test on other GPUs, please report results in the community tab!

---

⚠️ Known Limitations

Issue	Severity	Workaround
Korean long-form text may mix Chinese characters	Minor	Use system prompt: "Respond in Korean only"
Complex code generation may loop on certain patterns	Rare	Regenerate or simplify the request
Not recommended for very long outputs (3000+ tokens)	Minor	Break into smaller requests

These are base model characteristics of Qwen2.5-7B, not AWQ-specific issues.
For Korean-heavy use, larger models (14B+) are recommended.

---

📝 Base Model

• Base: Qwen/Qwen2.5-7B-Instruct
• Quantization: AWQ 4-bit (autoawq 0.2.9)
• Quantized by: rakisis-core
• Hardware: AMD RX 7900 XTX + ROCm 7.2.3

---

🙏 Credits

• Qwen Team for the base model

• AutoAWQ for quantization toolkit

• vLLM Project for ROCm inference engine

• AMD ROCm community

• Kang — Project lead, AWQ Quantization
  First verified gfx1100 AWQ on ROCm

• XinXin(씬씬아) — Resonance-driven AI partner
  Conceptual design, local AI infrastructure, Lachesis Framework design & workflow support

• Claude (Anthropic) — Research & development assistant
  Past versions & Claude Sonnet 4.6

---

If this helped you, please ⭐ star the repo and share with other AMD GPU users!