SoFlowRig
Audio-driven MetaHuman Control Rig generation (v1.3-4): 176-D face (104 unidirectional + 72 bidirectional), DiT / Transformer Encoder / Transformer Decoder, frozen wav2vec2-base-960h audio and bge-base-zh-v1.5 global text.
Code: github.com/Uvvvvvvvv/SoFlowRig
Released checkpoints
Paper / recommended weights are the 180,000-step EMA snapshots. Later 400k-step weights overfit and are not released.
Each file contains only ema_model (plus finished_steps, model_type, face_dim=176). Load with inference_mha.py (strict=False).
| File | Backbone | Hidden / depth / heads | Training Vel_MSE / Con_MSE at 180k | Size |
|---|---|---|---|---|
soflowrig_ditb_180000_ema.pt |
DiT-B (default) | 512 / 8 / 8 | 0.04858 / 0.00376 | ~195 MB |
soflowrig_dits_180000_ema.pt |
DiT-S | 384 / 6 / 6 | 0.09283 / 0.00845 | ~88 MB |
soflowrig_transencs_180000_ema.pt |
TransEnc-S | 384 / 6 / 6 | 0.26703 / 0.02591 | ~54 MB |
soflowrig_transdecs_180000_ema.pt |
TransDec-S | 384 / 6 / 6 | 0.23237 / 0.01988 | ~67 MB |
Logged Vel_MSE / Con_MSE are training objectives (100-step averages), not held-out metrics.
# default paper weight
wget https://huggingface.co/go49guy/SoFlowRig/resolve/main/soflowrig_ditb_180000_ema.pt
Inference
python inference_mha.py \
--ckpt soflowrig_ditb_180000_ema.pt \
--config configs/mha_configs/train.yaml \
--audio path/to/clip.wav \
--text "开心、笑容满面、眉毛上扬" \
--nfe 4 --cfg-scale 1.2 --text-cfg 1.5 \
--chunk-size 150 --chunk-overlap 30 \
--output out.json
Set model_type in the YAML to match the checkpoint (DiT-B, DiT-S, TransEnc-S, or TransDec-S).
Training notes
- SoFlow velocity (75%) + consistency (25%), linear path, Euler solution coefficients
- Batch 64, AdamW 1e-4, 1×H20, AMP off
- Face dim 176,
with_rotate=false - Dual classifier-free guidance (independent audio / text scales)
External models (not in this repo)
License
Apache-2.0 for these weights and the companion GitHub code. The MetaHuman dataset and Unreal / MetaHuman Animator assets are not included.