Harsha001/Phonon-2-XDNA2

Phonon-2 for the AMD Ryzen AI NPU (XDNA2)

Built from FermionResearch/Phonon-2, which is derived from NVIDIA's Parakeet TDT 0.6B v3.

English speech recognition with the encoder running entirely on the NPU of AMD Ryzen AI 300 and Ryzen AI MAX PCs (XDNA2, e.g. Strix Point, Strix Halo), through Windows ML. No AMD account, no Ryzen AI SDK install, no PyTorch: pip install, and Windows downloads the NPU provider itself.

• The encoder (subsampling, 24 Conformer blocks, projection; 609M parameters) is one static-shape ONNX graph per input length (10 s, 20 s and 36 s). The NPU compiler turns it into BF16 and places all 1,366 operators on the NPU, in a single subgraph, with no fallback to the CPU.
• Feature extraction, the small TDT decoder (LSTM + joint, 18M parameters) and the greedy search run on the CPU with NumPy and ONNX Runtime.
• Longer audio is cut into pieces of up to 36 s at the quietest moment near each cut.

Accuracy

LibriSpeech test sets from the Open ASR Leaderboard (hf-audio/esb-datasets-test-only-sorted), every utterance, scored with the leaderboard's English text normalizer and corpus word error rate (lower is better).

	test-clean	test-other
This repo, encoder on the NPU (BF16)	2.16 %	4.32 %
Same pipeline, full-precision reference on the CPU (Fermion's own transformers loader)	2.14 %	4.34 %
Fermion's published figures for Phonon-2	1.72 %	3.92 %

The NPU is within 0.02 points of the full-precision reference on both sets. Fermion publishes lower numbers from its own runtime; our full-precision reference, measured with the same scripts as the NPU rows, sits 0.42 points above them on both sets (for example it writes "misses" where the reference text has "Mrs."). Compare the NPU against that row.

Speed

Measured on a Ryzen AI MAX+ 395 (Strix Halo, 50 TOPS XDNA2 NPU, 16 Zen 5 cores), steady state, encoder only:

Window	NPU (BF16)	CPU, 16 cores (FP32, ONNX Runtime)
10 s	466 ms (21x realtime)	172 ms (58x realtime)
20 s	738 ms (27x realtime)	318 ms (63x realtime)
36 s	2,001 ms (18x realtime)	607 ms (59x realtime)

Over the full test sets, where every utterance is padded up to its window, the NPU encoder averaged 12 to 13x realtime.

What the NPU is for: on this chip the CPU is faster for a single stream. The NPU's value is that it leaves the CPU and GPU free for other work and is built to use less power (not measured here), which matters for always-on dictation and background transcription on a laptop.

Quick start

Windows 11 on a Ryzen AI 300 / AI MAX PC, Python 3.12, an up-to-date NPU driver.

py -3.12 -m venv .venv; .venv\Scripts\activate
pip install huggingface_hub
hf download Harsha001/Phonon-2-XDNA2 --local-dir Phonon-2-XDNA2
cd Phonon-2-XDNA2
pip install -r requirements.txt
python phonon2_xdna2.py --prepare                 # compiles the three lengths for your NPU, once (about 1.5 hours)
python phonon2_xdna2.py recording.wav             # 16 kHz or any rate; mono or stereo
python phonon2_xdna2.py recording.wav --device cpu

Without --prepare each length compiles the first time audio of that length shows up (about 25 minutes each). The compiled encoders go to npu_cache/ (about 0.8 GB each) and load in two seconds afterwards.

From Python:

from phonon2_xdna2 import Phonon2, load_audio
asr = Phonon2(".", device="npu")
print(asr.transcribe(load_audio("recording.wav")))

Files

File
encoder.weights	the encoder's weights in FP32, 2.44 GB, shared by the three graphs
encoder_T1000.onnx, encoder_T2000.onnx, encoder_T3600.onnx	the encoder for 10 s, 20 s and 36 s of audio (opset 18)
decoder.onnx, joint.onnx	the TDT prediction network (one step) and joint network (opset 17)
mel_filters.npy, vocab.json	128-band mel filter bank and the 8,192-piece vocabulary with the 5 TDT durations
phonon2_xdna2.py, requirements.txt	the runtime: features, NPU setup, chunking, TDT greedy search
build/	the scripts that made these files and measured them

How this was made

1. Phonon-2's container (revision fb09063) was expanded with Fermion's own reference loader (reference_transformers.py). Its quantization-aware-trained encoder weights take one of five learned values per row and expand exactly to dense FP32, so nothing was re-quantized from a lower-precision copy.
2. The encoder was rewritten for a static-shape NPU graph (build/phonon_npu.py): padding masks for the three subsampling stages, the convolution module and attention come in as input tensors, and attention uses a finite additive mask (-10,000) so BF16 never meets infinity. In PyTorch and in ONNX on the CPU it matches the transformers encoder to about 5e-6.
3. Exported with torch.onnx (dynamo, opset 18) for 1,000 / 2,000 / 3,600 feature frames; the three graphs were rewritten to read one shared weight file (build/share_weights.py) and checked bit-identical to the originals.
4. Compiled by the VitisAI execution provider that Windows ML installs (MicrosoftCorporationII.WinML.AMD.NPU.EP 1.8.75), which converts the FP32 graph to BF16 for the NPU.
5. Measured with build/eval_wer.py (accuracy) and build/bench.py (speed). The released runtime itself was checked in a fresh environment with only requirements.txt: compiled from these exact files, its transcripts matched the full-precision reference word for word.

Tried and not shipped: INT8 (A16W8). A16W8 encoders made with AMD Quark (per-channel INT8 weights, INT16 activations, calibrated on LibriSpeech dev) kept transcripts identical to FP32 on the CPU, but the Windows ML NPU provider either stopped while compiling them or ran most of the graph on the CPU (6.3 s per 10 s window). BF16 is the working path through Windows ML today.

A fix you may need elsewhere: the NPU compiler cannot resolve file paths inside C:\Program Files\WindowsApps (Windows gives no short names there), so a first compile fails with Failed to load VFS from C:\PROGRA~1\WindowsApps\. The runtime loads the provider through a directory junction in npu_cache/ep/, which avoids it.

Limits

• Made for Windows 11 with an XDNA2 NPU (Ryzen AI 300 and AI MAX series) and tested on a Ryzen AI MAX+ 395 only. The --device cpu path runs anywhere ONNX Runtime does.
• English only, like Phonon-2. One stream at a time; no batching, no word timestamps in this runtime.
• The first compile per length takes about 25 minutes and needs several GB of free memory.

Licence and credits

The model files are a derivative of Phonon-2 and Parakeet TDT 0.6B v3 and are distributed under CC-BY-4.0 (LICENSE-WEIGHTS-CC-BY-4.0.txt), the same as both.

• Phonon-2 © 2026 Fermion Research (CC-BY-4.0 weights, Apache-2.0 code); its NOTICE is included as NOTICE-Phonon-2 and lists its own training data licences (including CHiME-6, CC-BY-SA-4.0, and SPGISpeech).
• Parakeet TDT 0.6B v3 © NVIDIA Corporation, CC-BY-4.0.
• Changes made here: the weights were expanded to dense FP32 without retraining, the encoder was re-expressed as three static-shape ONNX graphs with explicit masks, the decoder and joint network were exported to ONNX, and the feature extraction and greedy search were re-implemented in NumPy. On the NPU the encoder runs in BF16. Not made, sponsored or endorsed by Fermion Research, NVIDIA or AMD.
• The runtime and build scripts in this repo are Apache-2.0 (LICENSE-CODE-Apache-2.0.txt).