FAMA Γ MEUSLI Γ Villanova: English ASR
Model Details
Model Description
A speechLLM for English automatic speech recognition (ASR), built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.
- Developed by: SpeechTek Unit, Fondazione Bruno Kessler
- Language(s): English
- Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
- Speech encoder: FBK-MT/fama-medium-asr (frozen)
- Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
- Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
- License: CC BY 4.0
- Code: Villanova-Speech-LLM
Uses
Trained for English automatic speech recognition (ASR). The prompt used in training and inference is "Transcribe English speech to text. " (the trailing space is part of the prompt and must be kept).
How to Get Started with the Model
The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/asr/monolingual/finetune_fama_meusli_vnova_en.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:
hf download SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_asr_en_v1
and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_asr_en_v1/model.pt.
Training Details
Training Data
VoxPopuli English (train / validation / test splits).
Training Procedure
Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt used in training and inference is "Transcribe English speech to text. " (the trailing space is part of the prompt and must be kept).
Training Hyperparameters
| Parameter | Value |
|---|---|
| llm_name | villanova-2b |
| llm_dim | 2560 |
| context_length | 4096 |
| encoder_name | fama |
| encoder_dim | 1024 |
| input_type | mel (80-dim filterbank) |
| encoder_projector | linear (Linear 2048β2048, ReLU, Linear 2048β2560) |
| encoder_projector_ds_rate | 2 |
| projector initialisation | MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised |
| peft_method | lora |
| lora r / alpha / dropout | 32 / 32 / 0.05 |
| lora target_modules | q_proj, v_proj |
| freeze_encoder | true |
| freeze_llm | true |
| epochs | 4 |
| warmup_steps | 1000 |
| total_steps | 1000000 |
| lr | 1e-4 |
| optimizer | AdamW (weight_decay 0) |
| batch_size_training | 4 |
| gradient_accumulation_steps | 1 |
| val_batch_size | 4 |
| validation_interval | 5000 |
| num_workers_dataloader | 4 |
| use_fp16 | true |
| enable_ddp | true |
| enable_fsdp | false |
| decoding | beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0 |
Evaluation
WER is computed after removing punctuation and applying Whisper's EnglishTextNormalizer for English and BasicTextNormalizer for Italian. Decoding uses beam search with beam size 4.
Results
| Test set | Metric | Score |
|---|---|---|
| VoxPopuli English (test, 1,842 utterances) | WER % | 6.55 |
- This is the best English ASR result among all configurations tested in the project, including Whisper large-v3-turbo with Villanova-2B (7.09) and with EuroLLM-1.7B (6.67).
- About 1,200 VoxPopuli English utterances are longer than 30 s; FAMA processes them without truncation.
Limitations
- Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding
no_repeat_ngram_size, a mildrepetition_penaltyor a length limit tied to the audio is recommended. - Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.
Acknowledgements
This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 β IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.
Citation
If you use this model, please cite SLAM-LLM and the MEUSLI paper:
@article{ma2026slam,
title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
journal={IEEE Journal of Selected Topics in Signal Processing},
year={2026}
}
@inproceedings{fong25_interspeech,
title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
booktitle={Interspeech 2025},
pages={2003--2007},
year={2025},
doi={10.21437/Interspeech.2025-764}
}
Model tree for SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1
Base model
FBK-MT/fama-medium-asr