local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE

DeepSeek V4.1 Flash 1017

Run this prepared Flare runtime on four NVIDIA SM120 GPUs with 96 GB VRAM each. It supports 32 concurrent requests, seven DSpark draft tokens per round, text and images, and a 1,048,576-token per-request context limit. Serving uses Flare's native Rust runtime without Python or runtime compilation.

The package contains model weights, compiled CUDA kernels and captured execution graphs. It requires the Flare image below; Transformers, vLLM, SGLang and llama.cpp cannot load this runtime format. MXFP4 expert weights retain losslessly compressed CSF scales, attention uses BF16 arithmetic, and KV storage uses FP8 compression.

Use Linux x86-64, the NVIDIA Container Toolkit and working CUDA peer access between all four GPUs. Four RTX PRO 6000 Blackwell Workstation Edition GPUs were used for serving checks. Prefer GPUs behind the same PCIe switch. Allow about 515 GB of disk space for the runtime files, plus download-cache headroom. The examples keep the 188.8 GiB Engram tables in system RAM; allow at least 256 GiB of host RAM for the tables and serving overhead.

The public main revision contains the model card, license and publisher profile. Runtime files are in gb202. Keep model.safetensors, every content-addressed .safetensors payload and publisher.json together with their original filenames.

Download and run

Install the Hugging Face hf CLI and download the complete C32 runtime:

mkdir -p models flare-state
hf download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
  --revision gb202 \
  --include '*.safetensors' 'publisher.json' \
  --local-dir ./models

Run the primary file in Docker:

docker run --pull=always --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3"' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/models,dst=/models,readonly" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  ghcr.io/local-inference-lab/flare:release \
  run /models/model.safetensors \
  --publisher-keys /models/publisher.json \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 7 \
  --mtp-adaptive-verification --mtp-adaptive-verification-alpha 1 \
  --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory \
  --trust-store /state/trust.toml --host 0.0.0.0 --port 8000

The release image is the release channel; latest is also available. Review and accept the publisher and model license when prompted. The writable state directory preserves that acceptance. Docker selects host GPUs 0–3; Flare addresses them by their container-local indices.

With a compatible Flare executable installed directly on the host:

flare run ./models/model.safetensors \
  --publisher-keys ./models/publisher.json \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 7 \
  --mtp-adaptive-verification --mtp-adaptive-verification-alpha 1 \
  --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory \
  --host 127.0.0.1 --port 8000

Download through Flare

Flare can select gb202, download the complete package and serve it from a persistent cache. The explicit primary filename selects this C32 runtime:

mkdir -p flare-cache flare-state
docker run --pull=always --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3"' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/flare-cache,dst=/cache" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  ghcr.io/local-inference-lab/flare:release \
  run local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
  --revision gb202 --filename model.safetensors \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 7 \
  --mtp-adaptive-verification --mtp-adaptive-verification-alpha 1 \
  --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory --cache-dir /cache \
  --trust-store /state/trust.toml --host 0.0.0.0 --port 8000

For download only, replace run with download and omit the serving flags. This repository is public. If authentication is required, run hf auth login, mount the token file read-only and pass its container path with --token-file.

Send a request

The OpenAI-compatible API accepts model name DeepSeek-V4.1-Flash:

curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"Explain how a hash table works."}],"max_tokens":1024}'

Sampling defaults are temperature 1.0, top-p 1.0, top-k disabled, min-p 0, repetition penalty 1.0, and zero presence/frequency penalties. Request values override them. For nucleus sampling, add "top_p":0.95; top-k and min-p can be set with "top_k":17 and "min_p":0.05.

Images use OpenAI image_url content items with PNG, JPEG or static WebP data URLs. The package supports four simultaneous image-bearing requests within its 32-request total. Each request shares a 4096-visual-token budget across its images; usable image count depends on resolution. Image requests queue when all four feature slots are occupied.

Serving settings

• --max-concurrency defaults to 32. Keep 32 for aggregate throughput; use --max-concurrency 8 to reduce resident execution state. The package cannot exceed 32 active requests.
• --mtp-draft-tokens defaults to 7, with adaptive verification enabled. Keep these settings for the packed graph catalog. Use --mtp-draft-tokens 0 for target-only execution. Positive draft limits 1–6 and fixed-width verification are not supported by this package.
• --scheduler-policy defaults to balanced. The examples recommend throughput for concurrent generation. Use --scheduler-policy balanced to select the default latency/throughput balance.
• --kv-cache-limit defaults to memory, as shown above. Available GPU memory determines the KV pool. The shared logical KV capacity is reported at startup and is not multiplied by the TP rank count. The per-request context limit remains 1,048,576 tokens.
• --memory-limit defaults to the package's zero serving reserve, apart from Flare's driver margin. Dedicated GPUs provide the largest KV pool. To retain 2 GiB per GPU for other work, add --memory-limit=-2G; the KV pool will shrink accordingly.
• Prefill uses captured chunks through 4096 tokens. Changing this prepared limit requires a different runtime package.
• --embedding-table-storage defaults to disk. The examples recommend ram to avoid Engram disk reads during generation. RAM mode keeps one host copy of the tables and transfers the required rows to each rank.

Disk-backed Engram

To avoid holding the full Engram tables in RAM, use fast local NVMe storage with a bounded row cache:

flare run ./models/model.safetensors \
  --publisher-keys ./models/publisher.json \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 7 \
  --mtp-adaptive-verification --mtp-adaptive-verification-alpha 1 \
  --scheduler-policy throughput \
  --embedding-table-storage disk --embedding-table-cache-size 8GiB \
  --kv-cache-limit memory --host 127.0.0.1 --port 8000

--embedding-table-cache-size defaults to zero and applies only to disk storage; it cannot be combined with RAM storage. Disk-backed performance depends on storage latency. Docker containers using disk-backed tables must allow Linux io_uring operations. Engram offload leaves transformer weights in GPU memory.

Preparation

This runtime was prepared with high autotuning effort, a 64-graph budget for packed speculative rounds, and a 128-row speculative-token capacity. It retains seven learned draft tokens per round with GPU prefix selection. The launch examples use adaptive verification scoring alpha 1 and the throughput scheduler.

Comprehensive model-accuracy evaluation has not been run.