FAMA × MEUSLI × Villanova: bilingual Italian-English speech translation
Model Details
Model Description
A speechLLM for speech translation in both directions (Italian→English and English→Italian) with a single model, built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.
- Developed by: SpeechTek Unit, Fondazione Bruno Kessler
- Language(s): Italian and English
- Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
- Speech encoder: FBK-MT/fama-medium-asr (frozen)
- Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
- Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
- License: CC BY 4.0
- Code: Villanova-Speech-LLM
Uses
Trained for speech translation in both directions (Italian→English and English→Italian) with a single model. The prompt is chosen from each example's translation direction: "Translate Italian speech to English text. " or "Translate English speech to Italian text. " (trailing spaces included, as in scripts/st/conf/translate_bilingual.yaml).
How to Get Started with the Model
The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/st/bilingual/finetune_fama_meusli_vnova_bilingual.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:
hf download SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_bilingual_st_v1
and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_bilingual_st_v1/model.pt.
Training Details
Training Data
The model was trained on the speech-translation version of large (en-it): the same audio as the bilingual ASR set, with targets generated by translating each transcript with EuroLLM-1.7B-Instruct. Counts are utterances per language:
| Source (Italian / English) | Train | Validation | Test |
|---|---|---|---|
| Common Voice 17.0 (it / en) | 169,771 | 15,149 | 15,155 |
| FLEURS (it_it / en_us) | 2,602 | 391 | 647 |
| MLS Italian / LibriSpeech-960 (test-clean for test) | 59,623 | 1,248 | 1,262 |
| Total, both languages | 463,992 | 33,576 | 34,128 |
Training Procedure
Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt is chosen from each example's translation direction: "Translate Italian speech to English text. " or "Translate English speech to Italian text. " (trailing spaces included, as in scripts/st/conf/translate_bilingual.yaml).
Training Hyperparameters
| Parameter | Value |
|---|---|
| llm_name | villanova-2b |
| llm_dim | 2560 |
| context_length | 4096 |
| encoder_name | fama |
| encoder_dim | 1024 |
| input_type | mel (80-dim filterbank) |
| encoder_projector | linear (Linear 2048→2048, ReLU, Linear 2048→2560) |
| encoder_projector_ds_rate | 2 |
| projector initialisation | MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised |
| peft_method | lora |
| lora r / alpha / dropout | 32 / 32 / 0.05 |
| lora target_modules | q_proj, v_proj |
| freeze_encoder | true |
| freeze_llm | true |
| epochs | 4 |
| warmup_steps | 1000 |
| total_steps | 1000000 |
| lr | 1e-4 |
| optimizer | AdamW (weight_decay 0) |
| batch_size_training | 2 |
| gradient_accumulation_steps | 2 |
| val_batch_size | 4 |
| validation_interval | 5000 |
| num_workers_dataloader | 4 |
| use_fp16 | true |
| enable_ddp | true |
| enable_fsdp | false |
| decoding | beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0 |
Evaluation
BLEU is computed on normalised text. Decoding uses beam search with beam size 4.
Results
| Test set | Metric | Score |
|---|---|---|
| large (en-it) test, Italian→English | BLEU | 49.25 |
| large (en-it) test, English→Italian | BLEU | 41.26 |
Limitations
- Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding
no_repeat_ngram_size, a mildrepetition_penaltyor a length limit tied to the audio is recommended. - Translation references (training and test) were generated with EuroLLM-1.7B-Instruct rather than written by human translators, so BLEU measures agreement with those references.
- Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.
Acknowledgements
This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 – IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.
Citation
If you use this model, please cite SLAM-LLM and the MEUSLI paper:
@article{ma2026slam,
title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
journal={IEEE Journal of Selected Topics in Signal Processing},
year={2026}
}
@inproceedings{fong25_interspeech,
title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
booktitle={Interspeech 2025},
pages={2003--2007},
year={2025},
doi={10.21437/Interspeech.2025-764}
}
Model tree for SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1
Base model
FBK-MT/fama-medium-asr