Agent I/O Guards
Collection
Small open multilingual models: guards, classifiers, reranker, embeddings, language ID, punctuation. Apache-2.0, ONNX. • 63 items • Updated
Multilingual Reranker (small, 141M)
A cross-encoder reranker for search and RAG in 40+ languages, built on mmBERT-small. Give it a query and candidate passages, for example the top 20-100 hits of a vector or keyword search. It scores each pair, so you can re-sort the candidates and keep the best ones for your LLM. Queries and passages can be in different languages. It is distilled from Qwen3-Reranker-4B into a model about 28x smaller. Apache-2.0, trained on openly licensed web text. ONNX files for CPU and the browser (transformers.js) are included. A larger, more accurate version is available as Horizon-Labs/multilingual-reranker-base. Try it in the browser.
- One output logit per (query, passage) pair: higher = more relevant.
sigmoid(logit)gives a 0-1 relevance score. - Max length: trained with 384 tokens per pair. Longer passages are truncated; split long documents into chunks.
onnx/model_quantized.onnx(int8 embeddings, 268 MB): scores differ from fp32 by 0.001 on average (sigmoid scale) over 1240 benchmark pairs.
Usage
sentence-transformers:
from sentence_transformers import CrossEncoder
model = CrossEncoder("Horizon-Labs/multilingual-reranker-small")
query = "How tall is the Eiffel Tower?"
passages = ["The Eiffel Tower is 330 metres tall.", "La tour Eiffel a été construite pour l'Exposition universelle de 1889.",
"The Statue of Liberty is 93 metres tall."]
print(model.rank(query, passages)) # [{'corpus_id': 0, 'score': ...}, ...]
transformers:
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("Horizon-Labs/multilingual-reranker-small")
model = AutoModelForSequenceClassification.from_pretrained("Horizon-Labs/multilingual-reranker-small").eval()
enc = tok([query] * len(passages), passages, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
scores = model(**enc).logits[:, 0].sigmoid()
transformers.js:
import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";
const tok = await AutoTokenizer.from_pretrained("Horizon-Labs/multilingual-reranker-small");
const model = await AutoModelForSequenceClassification.from_pretrained("Horizon-Labs/multilingual-reranker-small", { dtype: "q8" });
const enc = await tok(new Array(passages.length).fill(query), { text_pair: passages, padding: true, truncation: true });
const { logits } = await model(enc);
Evaluation
nDCG@10 for reranking the candidate lists of public reranking benchmarks (the MTEB versions). Since v1.1 the training data includes MIRACL's training split (real queries; different queries from the MIRACL dev queries tested here), so the MIRACL column is in-domain for this model, as it is for bge-reranker-v2-m3. The other benchmarks were used only for evaluation. Every model sees the same queries and candidates, through the same
script (code/). MIRACL: 60 queries per language with 100 candidates each. Wikipedia (WikipediaRerankingMultilingual):
60 queries per language with 9 candidates each. "Other" = ESCI (es, jp, us), RuBQ, T2Reranking, mMARCO-ja and AskUbuntu.
| model | licence | MIRACL (18 languages) | Wikipedia (16 languages) | other (6 sets) | mean | note |
|---|---|---|---|---|---|---|
| this model (141M) | Apache-2.0 | 0.772 | 0.959 | 0.784 | 0.839 | multilingual |
| Horizon-Labs/multilingual-reranker-base (308M) | Apache-2.0 | 0.800 | 0.967 | 0.798 | 0.855 | multilingual |
| cross-encoder/ms-marco-MiniLM-L6-v2 (22M) | Apache-2.0 | 0.322 | 0.842 | 0.648 | 0.604 | English |
| BAAI/bge-reranker-base (278M) | MIT | 0.683 | 0.858 | 0.735 | 0.758 | Chinese/English |
| Alibaba-NLP/gte-reranker-modernbert-base (149M) | Apache-2.0 | 0.473 | 0.898 | 0.755 | 0.709 | English |
| BAAI/bge-reranker-v2-m3 (568M) | Apache-2.0 | 0.825 | 0.958 | 0.814 | 0.866 | multilingual; trained on MIRACL train |
| Qwen/Qwen3-Reranker-0.6B (596M) | Apache-2.0 | 0.785 | 0.958 | 0.793 | 0.845 | multilingual LLM reranker |
| Qwen/Qwen3-Reranker-4B (4B) | Apache-2.0 | 0.833 | 0.971 | 0.822 | 0.875 | multilingual LLM reranker; our teacher |
- The table shows the released checkpoint. Means over 2 training seeds (mean nDCG@10 .839 / .839).
- For reference, ranking the same candidates by bge-m3 dense-embedding similarity alone scores 0.797 / 0.927 / 0.788 (mean 0.837), so on this test a reranker adds most on top of a weaker first-stage retriever (BM25 or a small embedding model). Full comparison: leaderboard.
- Models ahead of this one: MIRACL: bge-reranker-v2-m3, Qwen3-Reranker-0.6B, Qwen3-Reranker-4B; Wikipedia: Qwen3-Reranker-4B; other: bge-reranker-v2-m3, Qwen3-Reranker-0.6B, Qwen3-Reranker-4B. bge-reranker-v2-m3 and, since v1.1, this model were trained on MIRACL training queries (not the dev queries used here).
Per set (nDCG@10):
| set | this model | bge-reranker-v2-m3 | Qwen3-Reranker-4B (teacher) |
|---|---|---|---|
| AskUbuntu duplicate questions (English) | 0.623 | 0.679 | 0.700 |
| ESCI product search, Spanish | 0.840 | 0.846 | 0.872 |
| ESCI product search, Japanese | 0.855 | 0.861 | 0.869 |
| ESCI product search, English | 0.884 | 0.895 | 0.908 |
| MIRACL ar | 0.823 | 0.845 | 0.876 |
| MIRACL bn | 0.814 | 0.893 | 0.896 |
| MIRACL de | 0.661 | 0.748 | 0.791 |
| MIRACL en | 0.721 | 0.728 | 0.790 |
| MIRACL es | 0.719 | 0.781 | 0.767 |
| MIRACL fa | 0.727 | 0.803 | 0.821 |
| MIRACL fi | 0.793 | 0.838 | 0.856 |
| MIRACL fr | 0.663 | 0.751 | 0.760 |
| MIRACL hi | 0.691 | 0.743 | 0.781 |
| MIRACL id | 0.612 | 0.722 | 0.641 |
| MIRACL ja | 0.810 | 0.840 | 0.873 |
| MIRACL ko | 0.830 | 0.841 | 0.856 |
| MIRACL ru | 0.795 | 0.838 | 0.868 |
| MIRACL sw | 0.872 | 0.904 | 0.859 |
| MIRACL te | 0.838 | 0.961 | 0.912 |
| MIRACL th | 0.803 | 0.878 | 0.887 |
| MIRACL yo | 0.904 | 0.935 | 0.903 |
| MIRACL zh | 0.828 | 0.809 | 0.851 |
| mMARCO (Japanese) | 0.735 | 0.811 | 0.776 |
| RuBQ (Russian) | 0.802 | 0.861 | 0.874 |
| T2Reranking (Chinese) | 0.751 | 0.746 | 0.756 |
| Wikipedia bg | 0.950 | 0.956 | 0.967 |
| Wikipedia bn | 0.943 | 0.964 | 0.971 |
| Wikipedia cs | 0.983 | 0.983 | 0.992 |
| Wikipedia da | 0.979 | 0.955 | 0.988 |
| Wikipedia de | 0.962 | 0.963 | 0.988 |
| Wikipedia en | 0.988 | 0.983 | 0.986 |
| Wikipedia fa | 0.946 | 0.953 | 0.956 |
| Wikipedia fi | 0.979 | 0.973 | 0.988 |
| Wikipedia hi | 0.913 | 0.917 | 0.914 |
| Wikipedia it | 0.960 | 0.969 | 0.966 |
| Wikipedia nl | 0.961 | 0.961 | 0.968 |
| Wikipedia no | 0.909 | 0.931 | 0.945 |
| Wikipedia pt | 0.963 | 0.949 | 0.978 |
| Wikipedia ro | 0.968 | 0.942 | 0.982 |
| Wikipedia sr | 0.982 | 0.973 | 0.983 |
| Wikipedia sv | 0.964 | 0.962 | 0.969 |
Training
- Passages: about 400k passages of 2-6 sentences from FineWeb-2 and FineWeb (ODC-BY) in 47 languages.
- Queries: Qwen3.8-27B (Apache-2.0) wrote a natural question and a keyword query for each passage, in the passage's language, plus English questions for 15% of the non-English passages (cross-lingual search).
- Real queries (since v1.1): 40,203 Wikipedia search queries in 16 languages from the MIRACL training split (Apache-2.0), with their human-judged relevant and non-relevant passages (padded to 16 candidates with random passages of the same language), scored by the same teacher.
- Scale: 334,400 synthetic queries with 16 candidates each (5.35M teacher-scored pairs), sampled evenly across languages (plus the 40,203 MIRACL queries). The synthetic data is published as Horizon-Labs/multilingual-rerank-distill.
- Schedule: 1 epoch, learning rate 5e-5, 16 queries x 16 candidates per step, max 384 tokens per pair.
- Candidates: for each query, the 15 most similar passages in the same language by bge-m3 dense retrieval (hard negatives) plus the source passage.
- Labels: Qwen3-Reranker-4B (Apache-2.0) scored every (query, candidate) pair. The student learns the teacher's ranking with a listwise KL loss over each query's 16 candidates, plus a pointwise loss on the teacher's relevance probability. The teacher's scores are soft labels, so the near-duplicate passages that dense retrieval finds are not wrongly treated as negatives.
- Code:
code/in this repository.
Changelog
- v1.1 (2026-10-16): added 40,203 real search queries from the MIRACL training split (16 languages) to the distillation data. 2-seed means v1.0 -> v1.1: MIRACL .730 -> .772 (in-domain since v1.1), Wikipedia .959 -> .962, other .783 -> .784, mean .824 -> .839. v1.0 stays available under the tag
v1.0.
Limitations
- The model inherits the teacher's judgement, including its mistakes. It is weaker than the teacher, especially on long or technical passages.
- Training queries are mostly LLM-written questions and keyword queries over web text (plus real Wikipedia search queries since v1.1). Very domain-specific search (legal, medical, code) and conversational queries are less covered.
- Pairs are truncated at the max length. Rerank passage-sized chunks, not whole documents.
- Downloads last month
- 729
// npm i @huggingface/transformers import { AutoTokenizer, AutoModelForSequenceClassification } from '@huggingface/transformers'; const tokenizer = await AutoTokenizer.from_pretrained('Horizon-Labs/multilingual-reranker-small'); const model = await AutoModelForSequenceClassification.from_pretrained('Horizon-Labs/multilingual-reranker-small'); const query = 'Which planet is known as the Red Planet?'; const documents = [ 'Mars, known for its reddish appearance, is often referred to as the Red Planet.', 'Venus is often called the twin of Earth because of its similar size and proximity.', ]; const inputs = tokenizer(new Array(documents.length).fill(query), { text_pair: documents, padding: true, truncation: true }); const { logits } = await model(inputs); console.log(logits.sigmoid().tolist()); // one relevance score per document