FAMA Γ MEUSLI Γ Villanova: bilingual Italian-English ASR
Model Details
Model Description
A speechLLM for bilingual Italian and English automatic speech recognition (ASR) with a single model, built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.
- Developed by: SpeechTek Unit, Fondazione Bruno Kessler
- Language(s): Italian and English
- Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
- Speech encoder: FBK-MT/fama-medium-asr (frozen)
- Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
- Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
- License: CC BY 4.0
- Code: Villanova-Speech-LLM
Uses
Trained for bilingual Italian and English automatic speech recognition (ASR) with a single model. The prompt used in training and inference is "Transcribe speech to text.".
How to Get Started with the Model
The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/asr/bilingual/finetune_fama_meusli_vnova_bilingual.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:
hf download SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_bilingual_asr_v1
and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_bilingual_asr_v1/model.pt.
Training Details
Training Data
The model was trained on large (en-it), a balanced Italian-English set built from three sources per language, with English cut to the Italian sample counts. Counts are utterances per language:
| Source (Italian / English) | Train | Validation | Test |
|---|---|---|---|
| Common Voice 17.0 (it / en) | 169,771 | 15,149 | 15,155 |
| FLEURS (it_it / en_us) | 2,602 | 391 | 647 |
| MLS Italian / LibriSpeech-960 (test-clean for test) | 59,623 | 1,248 | 1,262 |
| Total, both languages | 463,992 | 33,576 | 34,128 |
Training Procedure
Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt used in training and inference is "Transcribe speech to text.".
Training Hyperparameters
| Parameter | Value |
|---|---|
| llm_name | villanova-2b |
| llm_dim | 2560 |
| context_length | 4096 |
| encoder_name | fama |
| encoder_dim | 1024 |
| input_type | mel (80-dim filterbank) |
| encoder_projector | linear (Linear 2048β2048, ReLU, Linear 2048β2560) |
| encoder_projector_ds_rate | 2 |
| projector initialisation | MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised |
| peft_method | lora |
| lora r / alpha / dropout | 32 / 32 / 0.05 |
| lora target_modules | q_proj, v_proj |
| freeze_encoder | true |
| freeze_llm | true |
| epochs | 4 |
| warmup_steps | 1000 |
| total_steps | 1000000 |
| lr | 1e-4 |
| optimizer | AdamW (weight_decay 0) |
| batch_size_training | 4 |
| gradient_accumulation_steps | 1 |
| val_batch_size | 4 |
| validation_interval | 5000 |
| num_workers_dataloader | 4 |
| use_fp16 | true |
| enable_ddp | true |
| enable_fsdp | false |
| decoding | beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0 |
Evaluation
WER is computed after removing punctuation and applying Whisper's EnglishTextNormalizer for English and BasicTextNormalizer for Italian. Decoding uses beam search with beam size 4.
Results
| Test set | Metric | Score |
|---|---|---|
| large (en-it) test, Italian part | WER % | 8.45 |
| large (en-it) test, English part | WER % | 11.94 |
| large (en-it) test, pooled | WER % | 10.08 |
| Common Voice 17.0 Italian (test) | WER % | 7.09 |
| VoxPopuli English (test; not in the training data) | WER % | 10.12 |
Limitations
- Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding
no_repeat_ngram_size, a mildrepetition_penaltyor a length limit tied to the audio is recommended. - Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.
Acknowledgements
This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 β IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.
Citation
If you use this model, please cite SLAM-LLM and the MEUSLI paper:
@article{ma2026slam,
title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
journal={IEEE Journal of Selected Topics in Signal Processing},
year={2026}
}
@inproceedings{fong25_interspeech,
title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
booktitle={Interspeech 2025},
pages={2003--2007},
year={2025},
doi={10.21437/Interspeech.2025-764}
}
Model tree for SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1
Base model
FBK-MT/fama-medium-asr