Instructions to use cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4")
model = AutoModelForMultimodalLM.from_pretrained("cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4

SGLang

How to use cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 with Docker Model Runner:
```
docker model run hf.co/cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4
```

Only !!!!!!!!!!!!!!!!!!!!!!!!!!! in response

by pi-null-mezon - opened Apr 23

Discussion

pi-null-mezon

Apr 23

Hi! Thanks for models! Unfortunately this one produces only !!!!!!!!!!!!!!!!!!!... in output. Maybe some ideas why this could happen?

I run on vllm in docker. Previously successgully run same quant of Qwen3.5-27B, but 3.6 does not work properly (

services:
  qwen3.6:
    image: vllm/vllm-openai:nightly
    container_name: qwen3.6-27b
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ['0', '1']
    ports:
      - "9090:8000"
    volumes:
      - /home/alex/.cache:/root/.cache
    ipc: host
    environment:
      - HF_HOME=/root/.cache
      - VLLM_USE_MODELSCOPE=True
      - CUDA_VISIBLE_DEVICES=0,1
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
    restart: always
    command: >
      cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4
      --served-model-name Qwen3.6-27B
      --attention-backend FLASHINFER
      --block-size 32
      --enable-prefix-caching
      --max-num-seqs 4
      --max-model-len 196609
      --max-num-batched-tokens 2048
      --enable-chunked-prefill
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --reasoning-parser qwen3
      --enable-prompt-tokens-details
      --gpu-memory-utilization 0.8
      --tensor-parallel-size 2
      --host 0.0.0.0
      --port 8000

cpatonn

cyankiwi org Apr 23

Could you try FLASH_ATTN attention backend instead?

pi-null-mezon

Apr 24

Thanks! With FLASH_ATTN works properly)

markypizz

Jun 28

•

edited Jun 28

I am seeing this happen as well now, but not always. It seems to be happening pretty deep into a thread, maybe around 60k context and above. I've had it happen 3 or 4 times now out of maybe 50 prompts. I am running on vLLM version 0.23.1rc1.dev245+g9037498c2 with this config:

command: --model cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 -tp 2 --gpu-memory-utilization 0.92 --max-model-len 150000 --max-num-seqs 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":1}' --enable-prefix-caching --trust-remote-code --default-chat-template-kwargs '{"preserve_thinking":true}'

I tried FLASH_ATTN, and it didn't make any difference. I am interfacing with it through QwenCode, and the !!!!!s seem to happen during the thinking phase. Trying to get some more data. If I find anything, I'll share it here.

markypizz

Jul 6

Fixed my above issue. I had to disable mtp. Everything is working great for me now.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment