yztian/VisTrans

PRIM: Towards Practical In-Image Multilingual Machine Translation (EMNLP 2025 Main)

📄Paper arXiv | 💻Code GitHub

Introduction

This repository provides the VisTrans model, trained as part of our work PRIM: Towards Practical In-Image Multilingual Machine Translation.

The VisTrans model is an end-to-end model for In-Image Machine Translation, which handles the visual text and background information in the image separately, with a two-stage training and multi-task learning strategy.

The model is trained on MTedIIMT, and tested on PRIM (see ./PRIM directory). It is also trained and tested on IIMT30k (see ./IIMT30k directory).

Inference

For inference and detailed usage instructions, please refer to our GitHub repository.

Citation

If you find our work helpful, we would greatly appreciate it if you could cite our paper:

@inproceedings{tian-etal-2025-prim,
    title = "{PRIM}: Towards Practical In-Image Multilingual Machine Translation",
    author = "Tian, Yanzhi  and
      Liu, Zeming  and
      Liu, Zhengyang  and
      Feng, Chong  and
      Li, Xin  and
      Huang, Heyan  and
      Guo, Yuhang",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.691/",
    pages = "13693--13708",
    ISBN = "979-8-89176-332-6",
    abstract = "In-Image Machine Translation (IIMT) aims to translate images containing texts from one language to another. Current research of end-to-end IIMT mainly conducts on synthetic data, with simple background, single font, fixed text position, and bilingual translation, which can not fully reflect real world, causing a significant gap between the research and practical conditions. To facilitate research of IIMT in real-world scenarios, we explore Practical In-Image Multilingual Machine Translation (IIMMT). In order to convince the lack of publicly available data, we annotate the PRIM dataset, which contains real-world captured one-line text images with complex background, various fonts, diverse text positions, and supports multilingual translation directions. We propose an end-to-end model VisTrans to handle the challenge of practical conditions in PRIM, which processes visual text and background information in the image separately, ensuring the capability of multilingual translation while improving the visual quality. Experimental results indicate the VisTrans achieves a better translation quality and visual effect compared to other models. The code and dataset are available at: https://github.com/BITHLP/PRIM."
}