Instructions to use vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram")
model = AutoModelForMultimodalLM.from_pretrained("vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram

SGLang

How to use vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram with Docker Model Runner:
```
docker model run hf.co/vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram
```

ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram

Weights from bottlecapai/ThinkingCap-Qwen3.8-Flash-Next, quantized using the recipe from Intel/Qwen3.8-Flash-Next-W4A16-AutoRound. Intel supplies the recipe, not replacement weights. ThinkingCap's tokenizer, chat template and vision processor are retained.

Only intended precision deviation from Intel: ngram lookup weights use scaled FP8 E4M3FN instead of BF16. All other precision settings and exclusions are unchanged.

Layer precision

Component	Checkpoint precision / treatment
Main routed MoE experts: `gate_proj`, `up_proj`, `down_proj`	Symmetric INT4, group size 128; W4A16
Ngram lookup tables	FP8 E4M3FN; shared FP32 scale per logical table across splits
Other PLE weights and projections	Original 16-bit precision; BF16 source
Self/linear attention, `in_proj_a`, `in_proj_b`, indexers	Original 16-bit precision; BF16 source
Routers (`mlp.gate`), shared experts and their gates	Original 16-bit precision; BF16 source
Hyper-connection weights	Original 16-bit precision; BF16 source
Vision encoder and merger	Original precision; excluded
Token embeddings and LM head	Original precision; excluded
Entire MTP module, including experts and attention	Original precision; excluded
Norms, hash/index buffers, other non-target tensors	Original dtype; integer buffers remain integer

W4A16 means 4-bit weights and 16-bit activations for INT4 targets. No FP8 activation conversion or FP8 KV-cache requirement is introduced; runtime compute/cache dtypes depend on the inference engine.

Quantization recipe

Uses Intel AutoRound, integrated into Intel Neural Compressor.

Setting	Value
INT4 recipe	W4A16, symmetric, group size 128, 200 tuning iterations
Calibration	Batch 8, sequence length 2048, 128 samples, seed 42; default dataset
Device / export	`auto` / `auto_round` (`auto_round:auto_gptq` packing)
Implementation	AutoRound 0.16.1 for compatibility fixes; Intel used 0.15.0

Same configured INT4 recipe, not a claim of bit-identical weights. Intel's exclusion list is retained:

lm_head,embed_tokens,visual,linear_attn,self_attn,hyper_connection,mlp.gate,shared_expert,in_proj_a,in_proj_b,ple,mtp,indexer

After INT4 export, *.ngram_embedding[.shard_N].weight is converted using FP8(clamp(weight / scale, -448, 448)), where scale = amax / 448 (1 for zero tables). Splits share an FP32 *.ngram_embedding.weight_scale; text_config.ple_embedding_dtype is set to float8_e4m3fn.

Provenance

ThinkingCap source revision: d90c9fedadaf7556df4206e6bea5a6b391b1a58e.
Intel recipe revision: 4c67bf686b7f7fd386bae6b07ab59e8ff1d5b897.
Export/conversion details and package versions: quantization-run.json.
Attribution does not imply endorsement by Intel, bottlecapai or Qwen.

Inference and limitations

Requires AutoRound INT4 and FP8 PLE support, including shared scales and split tables; vLLM 0.32.0 implements these loading paths. Compatibility is not universal. No benchmarks are claimed; Intel's scores do not describe this ThinkingCap variant. Validate quality, multimodal behavior and serving compatibility. Source-model limitations and safety considerations still apply.

License

Qwen Community License 1.0, unchanged from ThinkingCap; the original LICENSE is retained, not replaced by the quantizer's software license. Commercial conditions include separate licensing for certain Model as a Service and AI Work Assistant businesses. Read the full license; this is not legal advice.

Downloads last month: 39

Safetensors

Model size

73B params

Tensor type

I64

I32

BF16

F16

F8_E4M3

Model tree for vcerny/ThinkingCap-Qwen3.8-Flash-Next-W4A16-AutoRound-FP8NGram

Base model

Qwen/Qwen3.8-Flash-Next

Finetuned

bottlecapai/ThinkingCap-Qwen3.8-Flash-Next

Quantized

(5)

this model