---
license: apache-2.0
language:
- ar
- en
base_model: oddadmix/Emhotob-5M
pipeline_tag: text-generation
library_name: transformers
tags:
- translation
- msa
- fusha
- english
- arabic
- small-language-model
- slm
- tiny-lm
- chatml
- scaling-study
metrics:
- bleu
- chrf
---

# Emhotob-5M-English-MSA-v1 — Bidirectional English ↔ MSA (~5M params)

A **5.08M-parameter** model that translates **both ways** between **English** and
**Modern Standard Arabic (الفصحى)**. A single set of weights serves both directions;
a direction-specific system prompt selects which way to translate.

Finetuned from [`oddadmix/Emhotob-5M`](https://huggingface.co/oddadmix/Emhotob-5M),
a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

> **Scaling study.** This runs the exact recipe of
> [`oddadmix/50M-English-MSA-v1`](https://huggingface.co/oddadmix/50M-English-MSA-v1) on a base
> ~10× smaller. At 5M **short sentences translate correctly** while longer inputs drift and
> repeat. It is a scaling demonstration, not a production translator; use the 50M sibling for
> fluent output.

## Evaluation

Evaluated on a **deterministic held-out set of 3,000 pairs** (`seed=42`), decoded
**greedily** (`do_sample=False`, no repetition penalty), scored with sacreBLEU:

| Direction | sacreBLEU | chrF |
|---|---:|---:|
| **English → MSA** | **2.79** | 20.23 |
| **MSA → English** | **3.10** | 17.63 |

The saved weights are the best checkpoint by validation loss (`eval_loss = 3.105`, epoch 3 of 3).
For reference, the 50M sibling scores BLEU ~46 (en→msa) / ~50 (msa→en) with the same data and eval.

### Example translations

Real greedy-decoded outputs from the held-out set:

**English → MSA**

| English input | Model output (MSA) | Reference |
|---|---|---|
| Thank you very much, you are so kind. | شكرًا لك، أنت رائع جدًا. | شكرًا جزيلًا لك، أنت لطيف للغاية. |
| I'm just trying to reassure myself. What's the worst… | أنا فقط أتفق معك. ما الذي يحدث؟ … | أنا فقط أحاول أن أطمئن نفسي… |

**MSA → English**

| MSA input | Model output (English) | Reference |
|---|---|---|
| شكرًا جزيلًا لك، أنت لطيف للغاية. | Thank you and you're saying, you're saying. | Thank you very much, you are so kind. |

Short, common sentences come out well; longer inputs lose coherence. 20 samples per direction
with references are in [`eval_bidirectional.json`](./eval_bidirectional.json).

## Usage

ChatML format. **Pick the system prompt for the direction you want:**

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-English-MSA-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_MSA = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."
SYS_TO_EN  = "You are a professional translator. Translate the Modern Standard Arabic text into English."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("Thank you very much, you are so kind.", SYS_TO_MSA))
```

## Training

- **Base model:** `oddadmix/Emhotob-5M` (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000,
  tied embeddings; **5,080,704 params** after resizing for 2 ChatML tokens)
- **Dataset:** `oddadmix/egyptian-msa-2.9-openai-bytedance-translations` (132K rows; this model
  uses the `english` and `msa` columns)
- **Method:** HuggingFace `Trainer`, ChatML, **prompt-masked cross-entropy** (loss only on the
  assistant turn). Each row is exploded into **two** training examples (one per direction).
  Two ChatML special tokens (`<|im_start|>`, `<|im_end|>`) were added and embeddings resized.
- **Hyperparameters:** 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
  bf16 · max length 1024 · `load_best_model_at_end` on `eval_loss`.
- **Split:** 129,009 train / 3,000 deterministic held-out (`seed=42`), scored both directions.

## Limitations

- A 5M model near the **emergence threshold**: reliable only on short, common sentences;
  longer or rarer inputs drift, repeat, or lose meaning.
- For fluent translation use `oddadmix/50M-English-MSA-v1`.

## License

Apache-2.0, inherited from the base model.