NAMAA-Space/nadi26-mixed-asr-pipeline

NADI 2026 Mixed ASR — dialectal Arabic transcription pipeline

Part of the NAMAA at NADI 2026 Shared Task collection.

This repository contains no trained weights. NADI 2026 Subtask 1.2 was addressed with public Arabic ASR checkpoints used zero-shot — no fine-tuning was performed. What is released here is the development pipeline, the measured ablations, and the bugs that cost the most, which is the part that was expensive to produce.

Official result

Subtask	Selected system	WER	CER
1.2 Mixed ASR	Cohere Transcribe Arabic, bfloat16, no post-processing	44.65%	17.16%

CodaBench submission 864494, blind set of 427 clips. An earlier record rounded this to 0.45.

The selected system used no orthographic post-processing and no hypothesis combination. That is the single most important thing on this page, and it is the opposite of what the development pipeline below was built to do.

What the available comparisons do and do not show

Configuration	Dev WER	Blind WER
Whisper-large-v3-turbo, zero-shot (public baseline)	63.47%	—
Whisper-large-v3, LoRA comparison	~62%	—
Cohere int8, five beams	56%	—
Cohere int8, normalisation + medoid selection	55.48%	49%
Cohere bfloat16, no post-processing	—	44.65%

Read that table carefully, because the obvious reading is not supported. The bfloat16 development WER and the int8-only blind WER were never recorded, so the two rows that matter most are not a matched comparison: they differ in numerical precision and in post-processing and in hypothesis selection at the same time. The 4.35-point blind gap cannot be attributed to any one of them, and it does not establish a development-to-blind ranking reversal.

What can be said: the configuration that shipped was the simplest one, and nothing in the record shows the extra machinery earning its place on the blind set.

The team's other ASR subtask, 1.3 code-switched Arabic–French–English, reached 16.98% WER / 6.58% CER on the blind set using a two-stage rank-32 LoRA adaptation of FarukSTT. Those adapter weights are not part of this release.

The single most expensive finding

The scorer aligns line k of your predictions with row k of the dataset, not by id. Dataset rows are shuffled relative to id, so writing predictions in id-sorted order scrambles every pairing. That one line of code cost 0.9 WER — more than every modelling decision in this pipeline combined.

It survived an earlier correctness check because that check scored hyp[perm] against ref[perm], which is permutation-invariant: it returns the same number for every ordering, including the broken one.

An order check must permute one side only. Permuting only the dev hypotheses reproduced the failure exactly — 1.3507 / 1.1423 against the observed blind 1.3757 / 1.1491, versus 0.5911 / 0.3163 correctly paired.

Two other bugs were found and fixed in the development pipeline:

• Repetition loops. Two blind clips decoded into 320- and 62-word loops, roughly 380 spurious insertions. Handled with repetition_penalty / no_repeat_ngram_size plus a post-hoc collapse.
• Long-audio truncation (~7 WER points). The Cohere processor has max_audio_clip_s = 35.0 and auto-chunks longer audio, returning more rows than you passed in. A zip(batch, texts) silently dropped the extras, so every clip over 35 s kept only its first chunk. Blind clips under 35 s decoded at a median 2.19 words/second; the 8 clips over 35 s came out at 0.03–1.73 w/s — about 500 missing words from a 6,519-word reference.

The development pipeline

Four public checkpoints, each decoded independently and cached to its own JSONL, then combined. This is the development system, not the submitted one — the blind submission that scored 44.65% used cohere_base alone.

Key	Model	Family
cohere_base	CohereLabs/cohere-transcribe-arabic-07-2026	Cohere ASR (~2.07B, Apache 2.0)
whisper_dialect	oddadmix/whisper-large-v3-turbo-arabic-dialectal	Whisper
whisper_lv3	openai/whisper-large-v3	Whisper
cohere_dialect	oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2	Cohere ASR

An int8 build, NAMAA-Space/cohere-transcribe-arabic-07-2026-int8, is wired in for low-VRAM runs. NAMAA hosted and quantised the original Cohere weights; no additional training or fine-tuning was performed on that model, and its pretraining is not a NAMAA contribution.

Decoding for the selected system: deterministic five-beam search, max_new_tokens = 440, do_sample = False.

Combination. Two combiners, both scored on dev against an oracle ceiling before being trusted:

• Medoid selection (MBR) — pick the candidate with the lowest mean normalised edit distance to the others. Conservative; never invents a token no system produced. This is what the int8 + normalisation configuration used.
• ROVER — align every system to the longest-weighted pivot using jiwer word alignments, then vote slot by slot. On 600 real dev clips with synthetic system diversity, ROVER scored 0.5624 against a best member of 0.6008.

Text post-processing, measured

On dev clips matching the blind length distribution (≤40 reference words):

Change	WER	CER	Verdict
baseline	0.5382	0.2451
fold alef أإآٱ → ا	0.5320	0.2425	keep
squeeze character runs to 3	0.5382	0.2317	keep — CER only
learned word map, 2-fold CV	−0.0005		negligible
split leading و	0.5872	0.2513	rejected
ى → ي	0.5493	0.2475	rejected
ة → ه	0.5632	0.2514	rejected
اللي → الي	0.5402	0.2449	rejected

About 0.7 WER points total on development. The و-split looked like a 4-point win from raw token counts — references carry 1,231 standalone و against the system's 113 — and costs 4.9 when actually measured.

Error composition on dev with correct pairing: substitutions 36.9%, deletions 16.3%, insertions 5.5%. The models drop speech rather than invent it, which is why orthographic rewriting cannot help much — and is consistent with the selected blind system carrying no post-processing at all.

A normalizer that silently deleted 83% of the data

An earlier notebook contained an Arabic character class written with literal glyphs that resolved to [U+0617-U+064B] — covering U+0621-U+064A, i.e. every Arabic letter. It deleted 2,980 of 3,152 transcripts and scored WER 0.9996, which looks like a bad run rather than a broken pipeline.

Two guards make that unshippable here:

• the text cell asserts FINAL_NORM("انا") == "انا" before anything else runs;
• score() refuses to return a number when more than a quarter of hypotheses are empty, or when normalization drops over half the words.

A bare wer < 1.0 check passes at 0.9997 and is useless. Write Arabic literals in code as \uXXXX escapes.

Files

File	What it is
NADI2026_MixedASR_PIPELINE.ipynb	Start here. The development pipeline, self-contained. Open in Colab, set a GPU runtime, put a token in Secrets as HF_TOKEN, run top to bottom.
NADI2026_MixedASR_Improved_E2E.ipynb	v2 — unverified, see below.
nadi_asr_one_cell.py	the same v1 pipeline as a single paste-able cell
DIAGNOSIS.md	the full write-up of the ordering bug and its proof
tools/	the ablation scripts that produced the tables above. Nothing in the notebooks imports them.

QUICK_MODE = True (the default) runs cohere_base alone — which is also the configuration that produced the official 44.65%. Set it to False for the 4-model development ensemble; every system caches to its own JSONL, so nothing already computed is recomputed.

About the v2 notebook

NADI2026_MixedASR_Improved_E2E.ipynb extends the pipeline with newer acoustic checkpoints (oddadmix/whisper-large-v3-arabic-dialectal-v2), a librosa silence trim, a universal degeneracy guard combining words-per-second and type–token ratio, and an opt-in LoRA fine-tune of the winning Whisper system on the labelled development set.

It has never been run end to end on a GPU. None of its numbers are measured. It is shipped because the design and the guards are reusable, not because it is known to beat anything. Treat every claim in it as a hypothesis.

Data

Development	UBC-NLP/NADI2026_subtask2_MixedASR — 3,152 examples
Blind test	UBC-NLP/NADI2026_subtask1.2_MixedASR_test — 427 clips

Subtask 1.2 scoring normalises whitespace only. No task training split is provided; the public development repository's name contains subtask2, which is a known naming quirk, not a different task.

Intended use and limitations

Research on dialectal Arabic ASR and NADI 2026 Subtask 1.2 reproduction.

• A WER of 44.65% means roughly two words in five are wrong. These transcripts are usable for search, rough gisting and downstream features; they are not usable as a record of what a person said.
• Deletions dominate the error profile — output is systematically shorter than the truth. Do not treat a short transcript as a short utterance.
• These are NADI 2026 Mixed ASR scores. Nothing here is a general Arabic ASR benchmark result, and local development scoring was not verified byte-identical to the official scorer.
• Experiments were single runs without seed-variance analysis, and the exact development subset, Cohere checkpoint revisions and public-model training overlap remain unresolved. An audio-overlap audit was not performed.
• The pipeline downloads and runs third-party checkpoints; their own licences and limitations apply.

Provenance and citation

Part of the NAMAA Community submission to the NADI 2026 shared task, by Mohamed Abdelaziz, Fatima Emad Eldin, Mousa Ashraf Abd El Malak and Omer Nacar, released through NAMAA-Space (Network for Advancing Modern ArabicNLP & AI).

@inproceedings{namaa_nadi2026,
  title     = {NAMAA Community at NADI 2026 shared task: Mixed-Dialect and
               Code-Switched Arabic ASR and Dialect Identification},
  author    = {Abdelaziz, Mohamed and Emad Eldin, Fatima and
               Abd El Malak, Mousa Ashraf and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing
               Conference (ArabicNLP 2026)},
  year      = {2026}
}

Please also cite the NADI 2026 shared-task overview (Sullivan et al., 2026) and the base ASR checkpoints listed above.