FAMA × MEUSLI × Villanova: English-to-Italian speech translation
Model Details
Model Description
A speechLLM for English-to-Italian speech translation (ST), built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.
- Developed by: SpeechTek Unit, Fondazione Bruno Kessler
- Language(s): English and Italian
- Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
- Speech encoder: FBK-MT/fama-medium-asr (frozen)
- Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
- Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
- License: CC BY 4.0
- Code: Villanova-Speech-LLM
Uses
Trained for English-to-Italian speech translation (ST). The prompt used in training and inference is "Translate English speech to Italian text. " (the trailing space is part of the prompt and must be kept).
How to Get Started with the Model
The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/st/monolingual/finetune_fama_meusli_vnova_en2it.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:
hf download SpeechTek/speechllm_fama_meusli_villanova_st_en2it_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_st_en2it_v1
and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_st_en2it_v1/model.pt.
Training Details
Training Data
VoxPopuli English audio, with Italian targets generated by translating the English transcripts with EuroLLM-1.7B-Instruct.
Training Procedure
Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt used in training and inference is "Translate English speech to Italian text. " (the trailing space is part of the prompt and must be kept).
Training Hyperparameters
| Parameter | Value |
|---|---|
| llm_name | villanova-2b |
| llm_dim | 2560 |
| context_length | 4096 |
| encoder_name | fama |
| encoder_dim | 1024 |
| input_type | mel (80-dim filterbank) |
| encoder_projector | linear (Linear 2048→2048, ReLU, Linear 2048→2560) |
| encoder_projector_ds_rate | 2 |
| projector initialisation | MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised |
| peft_method | lora |
| lora r / alpha / dropout | 32 / 32 / 0.05 |
| lora target_modules | q_proj, v_proj |
| freeze_encoder | true |
| freeze_llm | true |
| epochs | 4 |
| warmup_steps | 1000 |
| total_steps | 1000000 |
| lr | 1e-4 |
| optimizer | AdamW (weight_decay 0) |
| batch_size_training | 2 |
| gradient_accumulation_steps | 2 |
| val_batch_size | 4 |
| validation_interval | 5000 |
| num_workers_dataloader | 4 |
| use_fp16 | true |
| enable_ddp | true |
| enable_fsdp | false |
| decoding | beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0 |
Evaluation
BLEU is computed on normalised text. Decoding uses beam search with beam size 4.
Results
| Test set | Metric | Score |
|---|---|---|
| VoxPopuli English→Italian (test) | BLEU | 57.75 |
Limitations
- Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding
no_repeat_ngram_size, a mildrepetition_penaltyor a length limit tied to the audio is recommended. - Translation references (training and test) were generated with EuroLLM-1.7B-Instruct rather than written by human translators, so BLEU measures agreement with those references.
- Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.
Acknowledgements
This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 – IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.
Citation
If you use this model, please cite SLAM-LLM and the MEUSLI paper:
@article{ma2026slam,
title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
journal={IEEE Journal of Selected Topics in Signal Processing},
year={2026}
}
@inproceedings{fong25_interspeech,
title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
booktitle={Interspeech 2025},
pages={2003--2007},
year={2025},
doi={10.21437/Interspeech.2025-764}
}
Model tree for SpeechTek/speechllm_fama_meusli_villanova_st_en2it_v1
Base model
FBK-MT/fama-medium-asr