FAMA × MEUSLI × Villanova: bilingual Italian-English speech translation

Model Details

Model Description

A speechLLM for speech translation in both directions (Italian→English and English→Italian) with a single model, built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.

  • Developed by: SpeechTek Unit, Fondazione Bruno Kessler
  • Language(s): Italian and English
  • Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
  • Speech encoder: FBK-MT/fama-medium-asr (frozen)
  • Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
  • Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
  • License: CC BY 4.0
  • Code: Villanova-Speech-LLM

Uses

Trained for speech translation in both directions (Italian→English and English→Italian) with a single model. The prompt is chosen from each example's translation direction: "Translate Italian speech to English text. " or "Translate English speech to Italian text. " (trailing spaces included, as in scripts/st/conf/translate_bilingual.yaml).

How to Get Started with the Model

The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/st/bilingual/finetune_fama_meusli_vnova_bilingual.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:

hf download SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_bilingual_st_v1

and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_bilingual_st_v1/model.pt.

Training Details

Training Data

The model was trained on the speech-translation version of large (en-it): the same audio as the bilingual ASR set, with targets generated by translating each transcript with EuroLLM-1.7B-Instruct. Counts are utterances per language:

Source (Italian / English) Train Validation Test
Common Voice 17.0 (it / en) 169,771 15,149 15,155
FLEURS (it_it / en_us) 2,602 391 647
MLS Italian / LibriSpeech-960 (test-clean for test) 59,623 1,248 1,262
Total, both languages 463,992 33,576 34,128

Training Procedure

Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt is chosen from each example's translation direction: "Translate Italian speech to English text. " or "Translate English speech to Italian text. " (trailing spaces included, as in scripts/st/conf/translate_bilingual.yaml).

Training Hyperparameters

Parameter Value
llm_name villanova-2b
llm_dim 2560
context_length 4096
encoder_name fama
encoder_dim 1024
input_type mel (80-dim filterbank)
encoder_projector linear (Linear 2048→2048, ReLU, Linear 2048→2560)
encoder_projector_ds_rate 2
projector initialisation MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised
peft_method lora
lora r / alpha / dropout 32 / 32 / 0.05
lora target_modules q_proj, v_proj
freeze_encoder true
freeze_llm true
epochs 4
warmup_steps 1000
total_steps 1000000
lr 1e-4
optimizer AdamW (weight_decay 0)
batch_size_training 2
gradient_accumulation_steps 2
val_batch_size 4
validation_interval 5000
num_workers_dataloader 4
use_fp16 true
enable_ddp true
enable_fsdp false
decoding beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0

Evaluation

BLEU is computed on normalised text. Decoding uses beam search with beam size 4.

Results

Test set Metric Score
large (en-it) test, Italian→English BLEU 49.25
large (en-it) test, English→Italian BLEU 41.26

Limitations

  • Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding no_repeat_ngram_size, a mild repetition_penalty or a length limit tied to the audio is recommended.
  • Translation references (training and test) were generated with EuroLLM-1.7B-Instruct rather than written by human translators, so BLEU measures agreement with those references.
  • Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.

Acknowledgements

This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 – IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.

Citation

If you use this model, please cite SLAM-LLM and the MEUSLI paper:

@article{ma2026slam,
  title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
  author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  year={2026}
}

@inproceedings{fong25_interspeech,
  title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
  author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
  booktitle={Interspeech 2025},
  pages={2003--2007},
  year={2025},
  doi={10.21437/Interspeech.2025-764}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1

Finetuned
(7)
this model

Datasets used to train SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1

Collection including SpeechTek/speechllm_fama_meusli_villanova_bilingual_st_v1