Instructions to use byadecha/Qwen3.5-9B-FoRE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use byadecha/Qwen3.5-9B-FoRE with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="byadecha/Qwen3.5-9B-FoRE", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("byadecha/Qwen3.5-9B-FoRE", trust_remote_code=True)
model = AutoModel.from_pretrained("byadecha/Qwen3.5-9B-FoRE", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use byadecha/Qwen3.5-9B-FoRE with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "byadecha/Qwen3.5-9B-FoRE"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-9B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/byadecha/Qwen3.5-9B-FoRE

SGLang

How to use byadecha/Qwen3.5-9B-FoRE with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "byadecha/Qwen3.5-9B-FoRE" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-9B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "byadecha/Qwen3.5-9B-FoRE" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "byadecha/Qwen3.5-9B-FoRE",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use byadecha/Qwen3.5-9B-FoRE with Docker Model Runner:
```
docker model run hf.co/byadecha/Qwen3.5-9B-FoRE
```

Qwen3.5-9B-FoRE

Code (GitHub) | Models (Hugging Face)

The FoRE drafter for Qwen/Qwen3.5-9B, from Fork and Regenerate: MTP-Guided Early Forking and Suffix Regeneration for Block Diffusion Speculative Decoding (Bruh Lemma Yadecha and Li Yang).

It is a block diffusion drafter fine-tuned from z-lab/Qwen3.5-9B-DFlash to draft from visible prefixes of varying lengths. FoRE uses it in both of its stages. In MTP-guided early forking, the target's pretrained MTP module proposes K candidates for the first drafted position and the drafter generates the rest of a block for each. In draft refinement, the drafter regenerates the suffix after each of R replacement tokens. The target verifies all K+R drafts in one batched forward pass, so the output is the target's greedy continuation, up to bf16 near-ties.

The checkpoint runs with the FoRE code and must be paired with Qwen3.5-9B. It has not been tested in other DFlash integrations.

Usage

git clone https://github.com/uncc-efficient-ai/FoRE && cd FoRE
uv sync --locked --extra flash-attn
hf download Qwen/Qwen3.5-9B --local-dir models/Qwen3.5-9B
hf download byadecha/Qwen3.5-9B-FoRE --local-dir models/Qwen3.5-9B-FoRE
uv run fore-generate \
  --target models/Qwen3.5-9B \
  --drafter models/Qwen3.5-9B-FoRE \
  --k 8 --r 8 \
  --prompt "Write a Python function that returns the n-th Fibonacci number."

--k is the number K of MTP-guided drafts and --r the number R of additional drafts from draft refinement. The code's README covers setup and the Python API, and its docs/training.md covers training.

Results

Acceptance length τ (speedup over autoregressive decoding) with greedy decoding, at most 2,048 generated tokens per response, on one NVIDIA H200 NVL GPU with FlashAttention-2, from Table 1 of the paper. Both FoRE columns use this checkpoint.

Benchmark	DFlash	FoRE, K=8, R=0	FoRE, K=8, R=8
GSM8K	6.54 (3.44×)	7.56 (3.91×)	9.77 (4.44×)
MATH-500	7.23 (3.77×)	8.31 (4.16×)	10.64 (4.65×)
AIME25	6.13 (3.13×)	7.30 (3.60×)	9.53 (4.03×)
HumanEval	7.03 (4.11×)	8.10 (4.38×)	10.24 (4.86×)
MBPP	5.31 (2.73×)	6.63 (3.35×)	8.61 (3.88×)
LiveCodeBench	4.79 (2.50×)	5.74 (2.89×)	7.44 (3.20×)
MT-Bench	3.46 (1.80×)	4.45 (2.22×)	5.65 (2.46×)
Mean	5.78 (3.07×)	6.87 (3.50×)	8.84 (3.93×)

Training

Initialization: z-lab/Qwen3.5-9B-DFlash at revision 492f4b532a957a50561e1418e5a3f31690f127f4.
Data: 97,896 responses that Qwen3.5-9B regenerated (temperature 0.7, top-p 0.95, thinking off, at most 3,072 new tokens) for prompts from Nemotron-Post-Training-Dataset-v2 and CodeAlpaca-20k. Prompts whose responses overlapped test items of GSM8K, MATH-500, HumanEval, MBPP or MT-Bench were removed, as were the 1,997 held-out prompts.
Objective: one epoch with the D-PACE loss and visible prefixes of varying lengths in each 16-token block, learning rate 1e-4, on four H200 GPUs.
Checkpoint: the final step, epoch_1_step_24046.

The code repository's docs/training.md gives the full recipe, and its THIRD_PARTY.md lists third-party code, models and data.

License

The weights are released under the Apache License 2.0 (LICENSE). They are fine-tuned from z-lab/Qwen3.5-9B-DFlash, released under the MIT License, whose notice is in dflash-MIT.txt. The target model Qwen3.5-9B is Apache-2.0.