FAMA Γ— MEUSLI Γ— Villanova: bilingual Italian-English ASR

Model Details

Model Description

A speechLLM for bilingual Italian and English automatic speech recognition (ASR) with a single model, built on FBK technology with the SLAM-LLM framework. It combines the frozen FAMA speech encoder, a linear projector initialised from the MEUSLI projector for Villanova-2B, and the frozen Villanova-2B language model adapted with LoRA. This repository contains the 14.16M trained parameters (model.pt): the projector (9.44M) and the LoRA adapters (4.72M). The encoder and the language model are downloaded separately.

  • Developed by: SpeechTek Unit, Fondazione Bruno Kessler
  • Language(s): Italian and English
  • Model type: linear projector + LoRA adapters in a speechLLM framework (encoder and LLM frozen)
  • Speech encoder: FBK-MT/fama-medium-asr (frozen)
  • Language model: VillanovaAI/Villanova-2B-2603 (frozen, LoRA-adapted)
  • Projector initialisation: SpeechTek/MEUSLI_projector_v2_Villanova_2B. MEUSLI was trained with Whisper large-v3-turbo; because its first layer does not match FAMA's output size, that layer is re-initialised and the rest is loaded.
  • License: CC BY 4.0
  • Code: Villanova-Speech-LLM

Uses

Trained for bilingual Italian and English automatic speech recognition (ASR) with a single model. The prompt used in training and inference is "Transcribe speech to text.".

How to Get Started with the Model

The checkpoint can be used for decoding or further finetuning with the shell scripts of the Villanova-Speech-LLM repository (training script: scripts/asr/bilingual/finetune_fama_meusli_vnova_bilingual.sh). First download FAMA, Villanova-2B and the MEUSLI checkpoint into models/ as described in the repository README (the scripts load MEUSLI before this checkpoint), then the checkpoint:

hf download SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_bilingual_asr_v1

and pass it to the scripts with ckpt_path=checkpoints/speechllm_fama_meusli_villanova_bilingual_asr_v1/model.pt.

Training Details

Training Data

The model was trained on large (en-it), a balanced Italian-English set built from three sources per language, with English cut to the Italian sample counts. Counts are utterances per language:

Source (Italian / English) Train Validation Test
Common Voice 17.0 (it / en) 169,771 15,149 15,155
FLEURS (it_it / en_us) 2,602 391 647
MLS Italian / LibriSpeech-960 (test-clean for test) 59,623 1,248 1,262
Total, both languages 463,992 33,576 34,128

Training Procedure

Trained with torchrun on a single NVIDIA A100 64 GB GPU of the CINECA Leonardo Booster partition, with the Villanova-Speech-LLM fork of SLAM-LLM. FAMA receives 80-dimensional filterbank features with CMVN. Every parameter of the projector and of the LoRA adapters is trained; the encoder and the LLM stay frozen. Training targets are lowercase. The learning rate warms up linearly over 1,000 steps and then decays linearly towards zero at total_steps; with 1,000,000 total steps it stays close to 1e-4 for the whole run. The prompt used in training and inference is "Transcribe speech to text.".

Training Hyperparameters

Parameter Value
llm_name villanova-2b
llm_dim 2560
context_length 4096
encoder_name fama
encoder_dim 1024
input_type mel (80-dim filterbank)
encoder_projector linear (Linear 2048β†’2048, ReLU, Linear 2048β†’2560)
encoder_projector_ds_rate 2
projector initialisation MEUSLI v2 for Villanova-2B; encoder_projector.linear1 re-initialised
peft_method lora
lora r / alpha / dropout 32 / 32 / 0.05
lora target_modules q_proj, v_proj
freeze_encoder true
freeze_llm true
epochs 4
warmup_steps 1000
total_steps 1000000
lr 1e-4
optimizer AdamW (weight_decay 0)
batch_size_training 4
gradient_accumulation_steps 1
val_batch_size 4
validation_interval 5000
num_workers_dataloader 4
use_fp16 true
enable_ddp true
enable_fsdp false
decoding beam search, 4 beams, max 200 new tokens, repetition_penalty 1.0

Evaluation

WER is computed after removing punctuation and applying Whisper's EnglishTextNormalizer for English and BasicTextNormalizer for Italian. Decoding uses beam search with beam size 4.

Results

Test set Metric Score
large (en-it) test, Italian part WER % 8.45
large (en-it) test, English part WER % 11.94
large (en-it) test, pooled WER % 10.08
Common Voice 17.0 Italian (test) WER % 7.09
VoxPopuli English (test; not in the training data) WER % 10.12

Limitations

  • Decoding has no repetition guard, so a few utterances per test set can loop until the 200-token limit; on out-of-domain data this affects more utterances. Adding no_repeat_ngram_size, a mild repetition_penalty or a length limit tied to the audio is recommended.
  • Single training run per configuration; differences under about 0.5 WER/BLEU are within run-to-run variation.

Acknowledgements

This work was supported by Ministero delle Imprese e del Made in Italy (IPCEI Cloud DM 27 giugno 2022 – IPCEI-CL-0000007) and European Union (Next Generation EU). We acknowledge the CINECA award under the ISCRA initiative (ISCRA-C project COBRAS), for the availability of high-performance computing resources and support.

Citation

If you use this model, please cite SLAM-LLM and the MEUSLI paper:

@article{ma2026slam,
  title={SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing},
  author={Ma, Ziyang and Yang, Guanrou and Chen, Wenxi and Gao, Zhifu and Du, Yexing and Li, Xiquan and Zheng, Zhisheng and Zhu, Haina and others},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  year={2026}
}

@inproceedings{fong25_interspeech,
  title={Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages},
  author={Fong, Seraphina and Matassoni, Marco and Brutti, Alessio},
  booktitle={Interspeech 2025},
  pages={2003--2007},
  year={2025},
  doi={10.21437/Interspeech.2025-764}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1

Finetuned
(7)
this model

Datasets used to train SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1

Collection including SpeechTek/speechllm_fama_meusli_villanova_bilingual_asr_v1