FAMA Γ— MEUSLI Γ— Villanova: English ASR

Model Details

Model Description

A speechLLM for English automatic speech recognition (ASR), built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.

  • Developed by: SpeechTek Unit, Fondazione Bruno Kessler
  • Language(s): English
  • Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
  • Speech encoder: FBK-MT/fama-medium-asr (frozen)
  • Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
  • Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
  • License: CC BY 4.0
  • Code: Villanova-Speech-LLM

Uses

Trained for English automatic speech recognition (ASR). The prompt used in training and inference is "Transcribe English speech to text. " (the trailing space is part of the prompt and must be kept).

How to Get Started with the Model

The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/asr/monolingual/finetune_fama_meusli_vnova_en.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:

hf download SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_asr_en_v1

and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_asr_en_v1/model.pt.

Training Details

Training Data

VoxPopuli English (train / validation / test splits).

Training Procedure

Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt used in training and inference is "Transcribe English speech to text. " (the trailing space is part of the prompt and must be kept).

Training Hyperparameters

Parameter Value
llm_name villanova-2b
llm_dim 2560
context_length 4096
encoder_name fama
encoder_dim 1024
input_type mel (80-dim filterbank)
encoder_projector linear (Linear 2048β†’2048, ReLU, Linear 2048β†’2560)
encoder_projector_ds_rate 2
projector initialisation MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised
peft_method lora
lora r / alpha / dropout 32 / 32 / 0.05
lora target_modules q_proj, v_proj
freeze_encoder true
freeze_llm true
epochs 4
warmup_steps 1000
total_steps 1000000
lr 1e-4
optimizer AdamW (weight_decay 0)
batch_size_training 4
gradient_accumulation_steps 1
val_batch_size 4
validation_interval 5000
num_workers_dataloader 4
use_fp16 true
enable_ddp true
enable_fsdp false
decoding beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0

Evaluation

WER is computed after removing punctuation and applying Whisper's EnglishTextNormalizer for English and BasicTextNormalizer for Italian. Decoding uses beam search with beam size 4.

Results

Test set Metric Score
VoxPopuli English (test, 1,842 utterances) WER % 6.55
  • This is the best English ASR result among all configurations tested in the project, including Whisper large-v3-turbo with Villanova-2B (7.09) and with EuroLLM-1.7B (6.67).
  • About 1,200 VoxPopuli English utterances are longer than 30 s; FAMA processes them without truncation.

Limitations

  • Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding no_repeat_ngram_size, a mild repetition_penalty or a length limit tied to the audio is recommended.
  • Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.

Acknowledgements

This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 – IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.

Citation

If you use this model, please cite SLAM-LLM and the MEUSLI paper:

@article{ma2026slam,
  title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
  author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  year={2026}
}

@inproceedings{fong25_interspeech,
  title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
  author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
  booktitle={Interspeech 2025},
  pages={2003--2007},
  year={2025},
  doi={10.21437/Interspeech.2025-764}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1

Finetuned
(7)
this model

Dataset used to train SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1

Collection including SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1