litigerking/Huihui-Qwen3.8-27B-Q4-NInfer

Huihui Qwen3.8-27B Q4 — NInfer artifact

This repository provides the NInfer conversion of the Huihui Qwen3.8-27B abliterated checkpoint. It is packaged for efficient local inference and is not a Transformers-format checkpoint.

Artifact

• File: huihui-qwen3.8-27b-q4.ninfer
• Size: approximately 18.2 GB
• Architecture: Qwen3_5ForConditionalGeneration
• Context length: 262,144 tokens
• Quantization: NInfer groupwise integer conversion with Q4/Q5/Q6 and W8 components, plus BF16/FP32 runtime tensors
• MTP tensors: preserved in the artifact
• Conversion device: NVIDIA A40, CUDA

Running with NInfer

git clone https://github.com/tctiger/ninfer-A40.git
cd ninfer-A40

ninfer-serve /models/huihui-qwen3.8-27b-q4.ninfer \
  --host 0.0.0.0 \
  --port 8080 \
  --max-context 262144 \
  --kv-dtype bf16 \
  --spec mtp \
  --draft-tokens 3 \
  --lm-head-draft

The server exposes an OpenAI-compatible API. Benchmark results vary with GPU, KV-cache type, prompt length, and concurrency.

Safety and intended use

Use this model only for lawful, authorized research, defensive security work, education, and evaluation. The abliterated nature of the source model may reduce refusal behavior; users remain responsible for access control, content filtering, and operational safeguards.

Provenance and license

This artifact is derived from the upstream Huihui Qwen3.8-27B abliterated checkpoint. Please preserve upstream attribution and comply with the upstream model license and usage restrictions. The NInfer conversion does not grant additional rights over the source weights.

Limitations

The file requires an NInfer runtime with Qwen3.8 support. It cannot be loaded directly with transformers or llama.cpp. Validate quality, safety, and tool-use behavior before deployment.