---
license: mit
library_name: vibevoice.c
base_model: Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM
tags:
- speculative-decoding
- dflash
- drafter
- speech-recognition
- vibevoice
- awq
- compressed-tensors
pipeline_tag: automatic-speech-recognition
---

# VibeVoice-ASR-DFlash2-Drafter-AWQ-W4A16-ASYM

A [DFlash 2](https://inco.ai/blog/dflash2/) block drafter for **VibeVoice-ASR (7B)**
([Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM](https://huggingface.co/Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM)), for speculative decoding in
[vibevoice.c](https://github.com/Ar4ikov/vibevoice.c). It proposes 8 tokens in
one pass; vibevoice.c checks them in one pass of the model and keeps the ones
the model agrees with, plus one of its own.

**The check is exact:** every checked row is computed with the arithmetic of
the model's own one-token decode step, so the transcript with this drafter is
byte-for-byte the transcript without it — words, timestamps, speakers. The
drafter only changes how many passes it takes.

One thing changes with `--draft`: vibevoice.c then attends with flashinfer
(`--attn auto` picks it), because flashinfer checks several rows in one pass
over the cache where the default fa2 walks it once per row. The transcript is
then byte-for-byte that of `--attn flashinfer` without a drafter — on this
model the default's words, with a timestamp 10 ms apart on a 32-minute file.
`--attn fa2 --draft` keeps the default's transcript bit for bit, and gains
less on long recordings.

This is the drafter with its projections stored as INT4 (AWQ, compressed-tensors
`pack-quantized`, W4A16 asymmetric, groups of 128): 0.55 GB instead
of the BF16 checkpoint's ([Ar4ikov/VibeVoice-ASR-DFlash2-Drafter](https://huggingface.co/Ar4ikov/VibeVoice-ASR-DFlash2-Drafter)).
vibevoice.c loads the codes as they are. Accepted drafts on the held-out
traces: 5.268 tokens per block, against 5.272 in BF16.

## Use

Needs vibevoice.c with DFlash 2 support: branch `dflash2`
([PR #48](https://github.com/Ar4ikov/vibevoice.c/pull/48)), in the next
release. The model and this drafter in one download:
[Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2](https://huggingface.co/Ar4ikov/VibeVoice-ASR-AWQ-W4A16-ASYM-DFlash2) (the INT4 drafter in
`drafter/`, used without `--draft`).

```bash
vv_cli --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --audio talk.wav --draft ./VibeVoice-ASR-DFlash2-Drafter-AWQ-W4A16-ASYM
vv_cli serve --model ./VibeVoice-ASR-AWQ-W4A16-ASYM --draft ./VibeVoice-ASR-DFlash2-Drafter-AWQ-W4A16-ASYM --slots 4
```

| flag | |
|---|---|
| `--draft <dir>` | this repository |
| `--draft-block <n>` | rows of the drafted block the model checks per pass (the drafter drafts 8; without the flag: 4 or 8, whichever keeps more tokens per ms) |
| `--draft-check exact\|fast` | exact (default): the transcript without a drafter on the same `--attn`, bit for bit; fast: flashinfer attention and prefill projections for the check, the greedy transcript within rounding |
| `--draft-quant int4\|f16` | how the runtime holds the drafter (this one is INT4 already) |

It drafts for any VibeVoice-ASR (7B) checkpoint of the same model —
[microsoft/VibeVoice-ASR](https://huggingface.co/microsoft/VibeVoice-ASR), [scerz/VibeVoice-ASR-4bit](https://huggingface.co/scerz/VibeVoice-ASR-4bit) — since it reads
only the model's hidden states, embedding and LM head; it was trained on the
AWQ checkpoint's transcripts, so that is where it drafts best.
On NF4 (bitsandbytes) and INT8 weights only `--draft-check fast` pays: their
decode step has no multi-row twin, so an exact check would cost a step per
row, and vibevoice.c declines the drafter there with a pointer to `fast`.

## Results

vibevoice.c `b72be15` (branch `dflash2`), RTX 3090, VibeVoice-ASR-AWQ-W4A16-ASYM, greedy decoding, decode tokens per second:

|  | plain | drafted | speedup | tokens per block | same transcript |
|---|---|---|---|---|---|
| 20 held-out clips (AMI, VoxConverse, earnings calls, LibriSpeech, FLEURS) | 148 tok/s | 565 tok/s | **3.82x** | 5.32 | 20/20 |
| 2-minute file, 2 speakers | 151 tok/s | 523 tok/s | **3.47x** | 4.64 | yes |
| 32-minute file | 123 tok/s | 296 tok/s | **2.41x** | 3.63 | yes |

Plain = the same binary without `--draft`, both with flashinfer attention, which `--draft` picks by default; `--draft-check exact` with the check width it measures. Against the default without a drafter (fa2): 3.65x on the 2-minute file, 2.78x on the 32-minute file. The same clips and files as the BF16 drafter's card.

## The drafter

* DFlash 2: 5 Qwen3-style layers (hidden 3584, 28/4 heads,
  intermediate 9472), 831M parameters, block 8.
* **KV injection:** for every position the model has processed, the outputs of
  its layers 1, 7, 13, 19 and 25 are concatenated, projected (`fc`) and
  normalized; every drafter layer turns them into keys and values. The drafter
  reads the model's own view of the audio and of the transcript so far.
* **Two-tap dynamic convolution** around attention and MLP (per-row kernels
  predicted from the row), and a **candidate selector** that re-ranks the
  model head's top 16 per row with a pairwise predecessor/successor term
  (rank 256).
* The embedding and the LM head are the model's, frozen. The drafter scores a
  **draft vocabulary** of 32768 ids (`draft_vocab`): the ids its training
  transcripts use, plus every stop id.
* Accepted drafts on 43 held-out clips' traces: **5.268 tokens per
  block** of 8 (the selector's, with every earlier draft right).

## Training

Self-distillation: the drafter learns what *this* model writes, timestamps and
speaker ids included. `tools/dflash` in vibevoice.c holds the pipeline:

1. ~390 hours of audio, 8228 training clips and 43 held out, from LibriSpeech,
   FLEURS (8 languages), SOVA, AMI, earnings calls and VoxConverse — audio
   only; the datasets' own text is never used.
2. The model's own greedy transcripts of every clip (`vv_dflash_data gen`).
3. Traces: every position's token, its role and the five tapped layers'
   outputs, replayed through the runtime (`vv_dflash_data trace`).
4. `train.py` (PyTorch, flex attention): anchors at generated positions, rows
   1..7 scored against the next 7 tokens with weights e^(-k/4); cross-entropy
   of the head plus the selector's; AdamW, cosine schedule, BF16.
5. `awq_drafter.py`: activation-aware scales where they fold into the drafter without changing it (the MLP input, up → down, v → o per KV channel), then a clip search on every projection; INT4 codes stored as compressed-tensors.

## Limits

* vibevoice.c only: the checkpoint keeps the published DFlash 2 names, but the
  KV injection, the selector and the draft vocabulary are as vibevoice.c
  implements them (`docs/DFLASH.md`).
* CUDA only (the CPU path and Metal decode without the drafter). Greedy only.
* Audio the corpus covered little (Mandarin, Russian) keeps fewer drafts; the
  runtime then falls back to plain steps and stays near plain speed.

## License

MIT, like VibeVoice.