---
license: apache-2.0
language:
- ar
pipeline_tag: automatic-speech-recognition
tags:
- automatic-speech-recognition
- arabic
- arabic-dialects
- dialectal-arabic
- nadi-2026
- speech-to-text
- ensemble
- rover
- pipeline
base_model:
- CohereLabs/cohere-transcribe-arabic-07-2026
- openai/whisper-large-v3
---

# NADI 2026 Mixed ASR — dialectal Arabic transcription pipeline

*Part of the [NAMAA at NADI 2026 Shared Task](https://huggingface.co/collections/NAMAA-Space/namaa-at-nadi-2026-shared-task-6ad8822b6702f645a4da58df) collection.*

**This repository contains no trained weights.** NADI 2026 Subtask 1.2 was
addressed with *public* Arabic ASR checkpoints used zero-shot — no fine-tuning
was performed. What is released here is the development pipeline, the measured
ablations, and the bugs that cost the most, which is the part that was expensive
to produce.

## Official result

| Subtask | Selected system | WER | CER |
|---|---|---:|---:|
| **1.2 Mixed ASR** | Cohere Transcribe Arabic, bfloat16, **no post-processing** | **44.65%** | 17.16% |

CodaBench submission 864494, blind set of 427 clips. An earlier record rounded
this to 0.45.

**The selected system used no orthographic post-processing and no hypothesis
combination.** That is the single most important thing on this page, and it is
the opposite of what the development pipeline below was built to do.

### What the available comparisons do and do not show

| Configuration | Dev WER | Blind WER |
|---|---:|---:|
| Whisper-large-v3-turbo, zero-shot (public baseline) | 63.47% | — |
| Whisper-large-v3, LoRA comparison | ~62% | — |
| Cohere int8, five beams | 56% | — |
| Cohere int8, normalisation + medoid selection | 55.48% | 49% |
| **Cohere bfloat16, no post-processing** | — | **44.65%** |

Read that table carefully, because the obvious reading is not supported. The
bfloat16 development WER and the int8-only blind WER were never recorded, so the
two rows that matter most are **not a matched comparison**: they differ in
numerical precision *and* in post-processing *and* in hypothesis selection at
the same time. The 4.35-point blind gap cannot be attributed to any one of them,
and it does not establish a development-to-blind ranking reversal.

What can be said: the configuration that shipped was the simplest one, and
nothing in the record shows the extra machinery earning its place on the blind
set.

The team's other ASR subtask, **1.3 code-switched Arabic–French–English**, reached
**16.98% WER / 6.58% CER** on the blind set using a two-stage rank-32 LoRA
adaptation of FarukSTT. Those adapter weights are not part of this release.

## The single most expensive finding

**The scorer aligns line *k* of your predictions with row *k* of the dataset,
not by `id`.** Dataset rows are shuffled relative to `id`, so writing
predictions in id-sorted order scrambles every pairing. That one line of code
cost **0.9 WER** — more than every modelling decision in this pipeline combined.

It survived an earlier correctness check because that check scored `hyp[perm]`
against `ref[perm]`, which is permutation-invariant: it returns the same number
for every ordering, including the broken one.

> **An order check must permute one side only.** Permuting only the dev
> hypotheses reproduced the failure exactly — 1.3507 / 1.1423 against the
> observed blind 1.3757 / 1.1491, versus 0.5911 / 0.3163 correctly paired.

Two other bugs were found and fixed in the development pipeline:

* **Repetition loops.** Two blind clips decoded into 320- and 62-word loops,
  roughly 380 spurious insertions. Handled with `repetition_penalty` /
  `no_repeat_ngram_size` plus a post-hoc collapse.
* **Long-audio truncation (~7 WER points).** The Cohere processor has
  `max_audio_clip_s = 35.0` and auto-chunks longer audio, returning *more* rows
  than you passed in. A `zip(batch, texts)` silently dropped the extras, so every
  clip over 35 s kept only its first chunk. Blind clips under 35 s decoded at a
  median 2.19 words/second; the 8 clips over 35 s came out at 0.03–1.73 w/s —
  about 500 missing words from a 6,519-word reference.

## The development pipeline

Four public checkpoints, each decoded independently and cached to its own JSONL,
then combined. **This is the development system, not the submitted one** — the
blind submission that scored 44.65% used `cohere_base` alone.

| Key | Model | Family |
|---|---|---|
| `cohere_base` | [`CohereLabs/cohere-transcribe-arabic-07-2026`](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026) | Cohere ASR (~2.07B, Apache 2.0) |
| `whisper_dialect` | [`oddadmix/whisper-large-v3-turbo-arabic-dialectal`](https://huggingface.co/oddadmix/whisper-large-v3-turbo-arabic-dialectal) | Whisper |
| `whisper_lv3` | [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3) | Whisper |
| `cohere_dialect` | [`oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2`](https://huggingface.co/oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2) | Cohere ASR |

An int8 build, [`NAMAA-Space/cohere-transcribe-arabic-07-2026-int8`](https://huggingface.co/NAMAA-Space/cohere-transcribe-arabic-07-2026-int8),
is wired in for low-VRAM runs. NAMAA hosted and quantised the original Cohere
weights; no additional training or fine-tuning was performed on that model, and
its pretraining is not a NAMAA contribution.

Decoding for the selected system: deterministic **five-beam** search,
`max_new_tokens = 440`, `do_sample = False`.

**Combination.** Two combiners, both scored on dev against an oracle ceiling
before being trusted:

* **Medoid selection (MBR)** — pick the candidate with the lowest mean
  normalised edit distance to the others. Conservative; never invents a token no
  system produced. This is what the int8 + normalisation configuration used.
* **ROVER** — align every system to the longest-weighted pivot using `jiwer`
  word alignments, then vote slot by slot. On 600 real dev clips with synthetic
  system diversity, ROVER scored **0.5624** against a best member of **0.6008**.

## Text post-processing, measured

On dev clips matching the blind length distribution (≤40 reference words):

| Change | WER | CER | Verdict |
|---|---:|---:|---|
| baseline | 0.5382 | 0.2451 | |
| fold alef `أإآٱ → ا` | **0.5320** | 0.2425 | keep |
| squeeze character runs to 3 | 0.5382 | **0.2317** | keep — CER only |
| learned word map, 2-fold CV | −0.0005 | | negligible |
| split leading `و` | 0.5872 | 0.2513 | rejected |
| `ى → ي` | 0.5493 | 0.2475 | rejected |
| `ة → ه` | 0.5632 | 0.2514 | rejected |
| `اللي → الي` | 0.5402 | 0.2449 | rejected |

About **0.7 WER points** total on development. The `و`-split looked like a
4-point win from raw token counts — references carry 1,231 standalone `و`
against the system's 113 — and costs 4.9 when actually measured.

Error composition on dev with correct pairing: substitutions 36.9%,
**deletions 16.3%**, insertions 5.5%. The models drop speech rather than invent
it, which is why orthographic rewriting cannot help much — and is consistent
with the selected blind system carrying no post-processing at all.

## A normalizer that silently deleted 83% of the data

An earlier notebook contained an Arabic character class written with literal
glyphs that resolved to `[U+0617-U+064B]` — covering `U+0621-U+064A`, i.e. every
Arabic letter. It deleted **2,980 of 3,152 transcripts** and scored WER 0.9996,
which looks like a bad run rather than a broken pipeline.

Two guards make that unshippable here:

* the text cell asserts `FINAL_NORM("انا") == "انا"` before anything else runs;
* `score()` refuses to return a number when more than a quarter of hypotheses
  are empty, or when normalization drops over half the words.

A bare `wer < 1.0` check passes at 0.9997 and is useless. **Write Arabic
literals in code as `\uXXXX` escapes.**

## Files

| File | What it is |
|---|---|
| `NADI2026_MixedASR_PIPELINE.ipynb` | **Start here.** The development pipeline, self-contained. Open in Colab, set a GPU runtime, put a token in Secrets as `HF_TOKEN`, run top to bottom. |
| `NADI2026_MixedASR_Improved_E2E.ipynb` | v2 — **unverified, see below.** |
| `nadi_asr_one_cell.py` | the same v1 pipeline as a single paste-able cell |
| `DIAGNOSIS.md` | the full write-up of the ordering bug and its proof |
| `tools/` | the ablation scripts that produced the tables above. Nothing in the notebooks imports them. |

`QUICK_MODE = True` (the default) runs `cohere_base` alone — which is also the
configuration that produced the official 44.65%. Set it to `False` for the
4-model development ensemble; every system caches to its own JSONL, so nothing
already computed is recomputed.

## About the v2 notebook

`NADI2026_MixedASR_Improved_E2E.ipynb` extends the pipeline with newer acoustic
checkpoints (`oddadmix/whisper-large-v3-arabic-dialectal-v2`), a librosa silence
trim, a universal degeneracy guard combining words-per-second and type–token
ratio, and an opt-in LoRA fine-tune of the winning Whisper system on the
labelled development set.

**It has never been run end to end on a GPU. None of its numbers are measured.**
It is shipped because the design and the guards are reusable, not because it is
known to beat anything. Treat every claim in it as a hypothesis.

## Data

| | |
|---|---|
| Development | [`UBC-NLP/NADI2026_subtask2_MixedASR`](https://huggingface.co/datasets/UBC-NLP/NADI2026_subtask2_MixedASR) — 3,152 examples |
| Blind test | [`UBC-NLP/NADI2026_subtask1.2_MixedASR_test`](https://huggingface.co/datasets/UBC-NLP/NADI2026_subtask1.2_MixedASR_test) — 427 clips |

Subtask 1.2 scoring normalises whitespace only. No task training split is
provided; the public development repository's name contains `subtask2`, which is
a known naming quirk, not a different task.

## Intended use and limitations

Research on dialectal Arabic ASR and NADI 2026 Subtask 1.2 reproduction.

* A WER of 44.65% means **roughly two words in five are wrong.** These
  transcripts are usable for search, rough gisting and downstream features; they
  are not usable as a record of what a person said.
* Deletions dominate the error profile — output is systematically *shorter* than
  the truth. Do not treat a short transcript as a short utterance.
* These are NADI 2026 Mixed ASR scores. Nothing here is a general Arabic ASR
  benchmark result, and local development scoring was not verified
  byte-identical to the official scorer.
* Experiments were single runs without seed-variance analysis, and the exact
  development subset, Cohere checkpoint revisions and public-model training
  overlap remain unresolved. An audio-overlap audit was not performed.
* The pipeline downloads and runs third-party checkpoints; their own licences
  and limitations apply.

## Provenance and citation

Part of the **NAMAA Community** submission to the NADI 2026 shared task, by
**Mohamed Abdelaziz, Fatima Emad Eldin, Mousa Ashraf Abd El Malak and Omer
Nacar**, released through **[NAMAA-Space](https://huggingface.co/NAMAA-Space)**
(Network for Advancing Modern ArabicNLP & AI).

```bibtex
@inproceedings{namaa_nadi2026,
  title     = {NAMAA Community at NADI 2026 shared task: Mixed-Dialect and
               Code-Switched Arabic ASR and Dialect Identification},
  author    = {Abdelaziz, Mohamed and Emad Eldin, Fatima and
               Abd El Malak, Mousa Ashraf and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing
               Conference (ArabicNLP 2026)},
  year      = {2026}
}
```

Please also cite the NADI 2026 shared-task overview (Sullivan et al., 2026) and
the base ASR checkpoints listed above.