Instructions to use incoai/GLM-5.3-Flash-DFlash2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use incoai/GLM-5.3-Flash-DFlash2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="incoai/GLM-5.3-Flash-DFlash2")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("incoai/GLM-5.3-Flash-DFlash2")
model = AutoModel.from_pretrained("incoai/GLM-5.3-Flash-DFlash2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use incoai/GLM-5.3-Flash-DFlash2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "incoai/GLM-5.3-Flash-DFlash2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-Flash-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/incoai/GLM-5.3-Flash-DFlash2

SGLang

How to use incoai/GLM-5.3-Flash-DFlash2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "incoai/GLM-5.3-Flash-DFlash2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-Flash-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "incoai/GLM-5.3-Flash-DFlash2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "incoai/GLM-5.3-Flash-DFlash2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use incoai/GLM-5.3-Flash-DFlash2 with Docker Model Runner:
```
docker model run hf.co/incoai/GLM-5.3-Flash-DFlash2
```

zhijianliu commited on Aug 31

Commit

bf582e4

verified ·

1 Parent(s): dc77ff1

Checkpoint update

Browse files

Files changed (3) hide show

README.md +6 -56
assets/dflash2-figure.png +3 -0
model.safetensors +1 -1

README.md CHANGED Viewed

@@ -30,6 +30,10 @@ Two-tap dynamic convolutions in the backbone keep the draft from decaying
 toward the end of the block. Decoding is lossless: greedy output
 matches the target model exactly, and sampling preserves its distribution.
 ## Quick Start
 Serve with [SGLang](https://github.com/sgl-project/sglang):
@@ -49,62 +53,8 @@ See the [blog post](https://inco.ai/blog/dflash2/) for more details.
 ## Evaluation
-- Runtime: SGLang on four NVIDIA GB300 GPUs (TP4), with TRT-LLM DSA and FlashInfer TRT-LLM MoE for the target, FP8 target KV cache, and FlashAttention 4 for DFlash 2 draft attention
-- Speculation block size: 8 (7 draft tokens per verification step)
-- Sampling: GLM-5.3-Flash's officially recommended parameters (temperature 1.0, top-p 0.95), with the default `Max` reasoning effort
-- Maximum new tokens: 4096
-- Samples: 128 at concurrency 1; 1,024 at concurrency 8 and 32
-We compare autoregressive decoding, GLM-5.3-Flash's native MTP, and DFlash 2.
-All speculative methods propose seven draft tokens per verification step.
-### Acceptance Length
-Acceptance length is the per-request mean of completion tokens divided by verification steps.
-Higher is better.
-| Task | MTP | DFlash 2 |
-| :--- | ---: | ---: |
-| GSM8K | 5.06 | **5.78** |
-| MATH-500 | 4.95 | **5.86** |
-| HumanEval | 4.70 | **5.32** |
-| MBPP | 4.26 | **4.85** |
-| MT-Bench | 3.71 | **4.03** |
-### Throughput
-Throughput is total output tokens divided by end-to-end wall time.
-Each cell shows `output tok/s (speedup vs. autoregressive)`.
-#### Concurrency 1
-| Task | Autoregressive | MTP | DFlash 2 |
-| :--- | ---: | ---: | ---: |
-| GSM8K | 146.8 | 282.6 (1.93×) | **355.4 (2.42×)** |
-| MATH-500 | 157.5 | 323.2 (2.05×) | **438.9 (2.79×)** |
-| HumanEval | 166.6 | 323.5 (1.94×) | **436.8 (2.62×)** |
-| MBPP | 168.2 | 299.5 (1.78×) | **402.2 (2.39×)** |
-| MT-Bench | 169.3 | 231.1 (1.36×) | **293.2 (1.73×)** |
-#### Concurrency 8
-| Task | Autoregressive | MTP | DFlash 2 |
-| :--- | ---: | ---: | ---: |
-| GSM8K | 582.6 | 825.5 (1.42×) | **922.9 (1.58×)** |
-| MATH-500 | 794.9 | 1,267.8 (1.59×) | **1,552.3 (1.95×)** |
-| HumanEval | 897.4 | 1,533.1 (1.71×) | **1,943.5 (2.17×)** |
-| MBPP | 895.8 | 1,428.0 (1.59×) | **1,821.8 (2.03×)** |
-| MT-Bench | 864.0 | 1,084.5 (1.26×) | **1,303.2 (1.51×)** |
-#### Concurrency 32
-| Task | Autoregressive | MTP | DFlash 2 |
-| :--- | ---: | ---: | ---: |
-| GSM8K | 917.3 | 1,203.1 (1.31×) | **1,318.3 (1.44×)** |
-| MATH-500 | 1,635.6 | 2,362.7 (1.44×) | **2,717.0 (1.66×)** |
-| HumanEval | 2,089.8 | 3,511.4 (1.68×) | **4,198.4 (2.01×)** |
-| MBPP | 2,102.0 | 3,168.5 (1.51×) | **3,791.4 (1.80×)** |
-| MT-Bench | 1,831.5 | 2,317.0 (1.27×) | **2,675.0 (1.46×)** |
 ## License

 toward the end of the block. Decoding is lossless: greedy output
 matches the target model exactly, and sampling preserves its distribution.
+<div align="center">
+  <img src="assets/dflash2-figure.png" alt="DFlash 2: parallel block drafting with a candidate path selector" width="100%">
+</div>
 ## Quick Start
 Serve with [SGLang](https://github.com/sgl-project/sglang):
 ## Evaluation
+Benchmark numbers for this checkpoint are being re-measured and will be
+published here once final.
 ## License

assets/dflash2-figure.png ADDED Viewed

Git LFS Details

SHA256: 6d8dcc9a9472bddb644c881fe83050b5343d09bf5ffb0912e5598e55fe671e99
Pointer size: 131 Bytes
Size of remote file: 287 kB

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b33c03475ba7322cf398828f2d8d1be376df30dc05c6b40c28c8ea8da23e410b
 size 2342169800

 version https://git-lfs.github.com/spec/v1
+oid sha256:b038e1d9d1e7833fa3880c2c0135ba9b673013f03da1b29fb831931584759dac
 size 2342169800