{ "cells": [ { "cell_type": "markdown", "id": "c000", "metadata": {}, "source": [ "# NADI 2026 โ€” Subtask 1.2 Mixed ASR โ€” complete pipeline\n", "\n", "**Self-contained.** No other file, notebook or repo artifact is needed. Run the cells top to bottom.\n", "Only a Hugging Face token is required (Colab โ–ธ ๐Ÿ”‘ Secrets โ–ธ `HF_TOKEN`). Runtime โ–ธ **GPU**.\n", "\n", "Everything here was measured on the 3152-clip dev split. Score history: **1.3757 โ†’ 0.48**.\n", "\n", "---\n", "\n", "## The bugs, and what each was worth\n", "\n", "**1. Line order โ€” cost 0.9 WER, fixed.** The scorer aligns line *k* of `predictions.txt` with **dataset\n", "row *k***. The submission was written in `sorted(ids)` order and the dataset rows are shuffled, so every\n", "hypothesis was scored against a different clip's reference. Permuting only the dev hypotheses into id\n", "order reproduces it exactly: WER 1.3507 / CER 1.1423 against the observed 1.3757 / 1.1491.\n", "\n", "This survived an earlier check because that check scored `hyp[perm]` against `ref[perm]` โ€” which is\n", "permutation-invariant and returns the same number for every ordering. **An order check must permute one\n", "side only.** ยง7 asserts `wer < 1.0`.\n", "\n", "**2. Repetition loops โ€” fixed.** Two blind clips decoded into 320- and 62-word loops, โ‰ˆ380 spurious\n", "insertions on a 6519-word reference.\n", "\n", "**3. Long-audio truncation โ€” worth โ‰ˆ7 WER points, NOT YET CLAIMED.** The Cohere processor config sets\n", "`max_audio_clip_s: 35.0`. Audio past that is auto-split into overlapping chunks, so `processor(...)`\n", "returns **more rows than you passed in**. The old loop did `for i, t in zip(b, texts)` โ€” `zip` silently\n", "drops the extra rows, so long clips kept only their first chunk. On the blind set, clips under 35 s\n", "decode at a median **2.19 words/second**; the 8 clips over 35 s came out at 0.03โ€“1.73 w/s:\n", "\n", "| row | duration | words | rate |\n", "|---|---|---|---|\n", "| 352 | 38.1 s | **1** (`@@@ูุฑุงุบ`) | 0.03 w/s |\n", "| 224 | 49.7 s | **7** | 0.14 w/s |\n", "| 167 | 59.1 s | 50 | 0.85 w/s |\n", "\n", "388 s of audio that should yield โ‰ˆ850 words yielded 339 โ€” roughly **500 missing words**. ยง5 windows long\n", "audio explicitly and stitches on the longest shared word n-gram, for every model family. This needs a\n", "GPU run, which is why no CPU-only rebuild could ever claim it. **It is the largest remaining item.**\n", "\n", "## What was tested and rejected\n", "\n", "Text post-processing is exhausted โ€” about 0.7 WER points total, all of it already in `FINAL_NORM` below.\n", "Measured on the dev clips matching the blind length distribution:\n", "\n", "| change | WER | CER | |\n", "|---|---|---|---|\n", "| baseline | 0.5382 | 0.2451 | |\n", "| fold alef `ุฃุฅุขูฑโ†’ุง` | 0.5320 | 0.2425 | keep |\n", "| squeeze char runs to 3 (`ู‡ู‡ู‡ู‡ู‡โ€ฆ`) | 0.5382 | **0.2317** | keep โ€” CER only |\n", "| learned word map, 2-fold CV | โˆ’0.0005 | | negligible |\n", "| split leading `ูˆ` | 0.5872 | 0.2513 | โœ— |\n", "| `ู‰โ†’ูŠ` | 0.5493 | 0.2475 | โœ— |\n", "| `ุฉโ†’ู‡` | 0.5632 | 0.2514 | โœ— |\n", "| `ุงู„ู„ูŠโ†’ุงู„ูŠ` | 0.5402 | 0.2449 | โœ— |\n", "\n", "The `ูˆ`-split looked like a 4-point win from token counts (references have 1231 standalone `ูˆ` against\n", "our 113) and costs 4.9. Everything left has to come from the acoustic models.\n", "\n", "## Sections\n", "\n", "| ยง | | cost |\n", "|---|---|---|\n", "| 1 | config + install | โ€” |\n", "| 2 | text pipeline | CPU |\n", "| 3 | dev references + scoring harness | CPU, ~2 min |\n", "| 4 | model registry, long-audio-safe transcription | โ€” |\n", "| 5 | run models on dev, score each | GPU |\n", "| 6 | combine โ€” MBR / ROVER / weighted / oracle ceiling | CPU |\n", "| 7 | blind transcription + submission, guards asserted | GPU |\n", "| 8 | push results back to HF | โ€” |\n", "\n", "`QUICK_MODE = True` in ยง1 runs one model over a small dev subset end to end first, so you get a valid\n", "submission in ~30 minutes before committing to the full ensemble." ] }, { "cell_type": "markdown", "id": "c001", "metadata": {}, "source": [ "## 1. Config, install, login" ] }, { "cell_type": "code", "id": "c002", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ------------------------------- CONFIG -------------------------------\n", "QUICK_MODE = True # True: 1 model, 200 dev clips - a valid submission fast.\n", " # False: full ensemble (4 models, 500 dev clips, ~2-3 h on A100)\n", "DEV_EVAL_N = 200 if QUICK_MODE else 500\n", "RUN_DEV = True # score on dev before submitting. Turn off only if you already know the winner.\n", "HF_REPO = \"FatimahEmadEldin/Nadi\" # where ยง8 pushes; set to None to skip pushing\n", "PUSH_PREFIX = \"claude/standalone\"\n", "# ----------------------------------------------------------------------\n", "\n", "import sys, subprocess, os, json, gc, time, re, random\n", "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-U\",\n", " \"transformers\", \"accelerate\", \"datasets\", \"soundfile\", \"librosa\",\n", " \"jiwer\", \"huggingface_hub\", \"pyarrow\", \"fsspec\"], check=True)\n", "import torch\n", "print(\"torch\", torch.__version__)\n", "HAS_GPU = torch.cuda.is_available()\n", "print(\"GPU:\", torch.cuda.get_device_name(0) if HAS_GPU else \"NONE - sections 1-3 and 6 still work\")\n", "if HAS_GPU:\n", " torch.backends.cuda.matmul.allow_tf32 = True; torch.backends.cudnn.allow_tf32 = True\n", "os.makedirs(\"out\", exist_ok=True)" ] }, { "cell_type": "code", "id": "c003", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import getpass\n", "from huggingface_hub import login, HfApi\n", "tok = None\n", "try:\n", " from google.colab import userdata; tok = userdata.get(\"HF_TOKEN\")\n", "except Exception:\n", " tok = os.environ.get(\"HF_TOKEN\")\n", "if not tok:\n", " tok = getpass.getpass(\"HF token: \").strip()\n", "login(token=tok, add_to_git_credential=False)\n", "os.environ[\"HF_TOKEN\"] = tok\n", "\n", "DEV_REPO = \"UBC-NLP/NADI2026_subtask2_MixedASR\"\n", "TEST_REPO = \"UBC-NLP/NADI2026_subtask1.2_MixedASR_test\"\n", "N_TEST, SR = 427, 16000\n", "api = HfApi(token=tok)\n", "print(\"logged in |\", \"quick mode\" if QUICK_MODE else \"full ensemble mode\")\n", "\n", "# If a previous run backed transcripts up (claude/cache/), pull them in - they are\n", "# raw model output and cost real GPU time. Every stage below is keyed off these\n", "# filenames, so restoring them skips the transcription entirely.\n", "RESTORE_CACHE = True\n", "if RESTORE_CACHE:\n", " from huggingface_hub import list_repo_files, hf_hub_download\n", " try:\n", " hits = [f for f in list_repo_files(\"FatimahEmadEldin/Nadi\", token=tok)\n", " if f.startswith(\"claude/cache/\") and f.endswith(\".jsonl\")]\n", " for f in hits:\n", " dst = \"out/\" + os.path.basename(f)\n", " if not os.path.exists(dst):\n", " src = hf_hub_download(\"FatimahEmadEldin/Nadi\", f, token=tok)\n", " open(dst, \"wb\").write(open(src, \"rb\").read())\n", " print(\"restored\", dst)\n", " print(f\"cache: {len(hits)} file(s) found on the hub\")\n", " except Exception as e:\n", " print(\"cache restore skipped:\", type(e).__name__)" ] }, { "cell_type": "markdown", "id": "c004", "metadata": {}, "source": [ "## 2. Text pipeline\n", "\n", "`BEST_NORM` is byte-for-byte the normalization behind the 0.56 dev submission. `FINAL_NORM` adds the\n", "three measured wins. Rules that were tested and hurt are deliberately absent โ€” see the table above." ] }, { "cell_type": "code", "id": "c005", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# --------------------------------------------------------------------------------\n", "# Arabic is written as \\uXXXX escapes ON PURPOSE, never as literal characters.\n", "# Combining marks reorder visually vs logically, and a literal class like\n", "# \"[-...]\" silently became [U+0617-U+064B] in an earlier draft of this\n", "# notebook -- a range that swallows U+0621-U+064A, i.e. EVERY ARABIC LETTER. The\n", "# normalizer then deleted the entire transcript and produced 427 blank lines.\n", "# Escapes cannot be reordered by an editor, a terminal, or a copy-paste.\n", "# --------------------------------------------------------------------------------\n", "# U+0617-U+061A small high marks | U+064B-U+0652 tanween..sukun | U+0670 superscript alef | U+0640 tatweel\n", "_DIAC = re.compile(\"[\\u0617-\\u061a\\u064b-\\u0652\\u0670\\u0640]\")\n", "# U+0617-U+061A small high marks | U+064B-U+0652 tanween..sukun | U+0670 superscript alef | U+0640 tatweel\n", "\n", "_PUNCT_CHARS = \"\\u061b\\u060c\\u061f\" + \"`\\u00f7\\u00d7_()*^%][/:\\\"{}~\\u00a6+|!\\u2026\\u2013\\u2019\\u201d\\u201c;?.,\" # arabic semicolon/comma/question + ascii + typographic\n", "_PUNCT = re.compile(\"[\" + re.escape(_PUNCT_CHARS) + \"]\")\n", "_BRACKETED = re.compile(r\"[\\[\\(\\{<][^\\]\\)\\}>]{0,40}[\\]\\)\\}>]\")\n", "_TAGWORDS = re.compile(r\"\\b[Ss][Pp][Ee][Aa][Kk][Ee][Rr]\\s*[A-Za-z0-9]?\\b\")\n", "_JUNK = re.compile(\"[#@\\u266a\\u2669\\u266b\\u266c\\\\*\\\\$\\\\&<>\\\\\\\\]+\")\n", "_LATIN = re.compile(r\"[A-Za-z]+\")\n", "_ELONG = re.compile(r\"(.)\\1{3,}\")\n", "_ALEF = re.compile(\"[\\u0623\\u0625\\u0622\\u0671]\") # alef: hamza above/below, madda, wasla\n", "_PLAIN_ALEF = \"\\u0627\"\n", "\n", "# mined from dev by aligning hyp/ref: count>=10, P(ref|hyp)>=0.5, edit distance<=1,\n", "# validated 2-fold out of sample. Only one rule cleared the bar.\n", "LEARNED_MAP = {\"\\u0634\\u064a\\u0621\": \"\\u0634\\u064a\"} # shay' -> shay\n", "\n", "def norm_ws(t): return re.sub(r\"\\s+\", \" \", t).strip()\n", "def BEST_NORM(t): return norm_ws(_PUNCT.sub(\" \", _DIAC.sub(\"\", t)))\n", "\n", "def strip_junk(t):\n", " t = _BRACKETED.sub(\" \", t); t = _TAGWORDS.sub(\" \", t)\n", " t = _JUNK.sub(\" \", t); t = _LATIN.sub(\" \", t)\n", " return norm_ws(t)\n", "\n", "# max_run: identical adjacent words kept. max_phrase_run: identical n-gram loops kept.\n", "# grid-searched on dev -> strip=1, max_run=3, max_phrase_run=3\n", "def collapse_repeats(text, max_run=3, max_ngram=5, max_phrase_run=3):\n", " w = text.split()\n", " if not w: return text\n", " out, run = [], 0\n", " for tok in w:\n", " if out and tok == out[-1]:\n", " run += 1\n", " if run >= max_run: continue\n", " else:\n", " run = 0\n", " out.append(tok)\n", " w = out\n", " for n in range(max_ngram, 1, -1):\n", " i, out = 0, []\n", " while i < len(w):\n", " gram = w[i:i+n]\n", " if len(gram) < n:\n", " out.extend(w[i:]); break\n", " reps, j = 1, i + n\n", " while w[j:j+n] == gram:\n", " reps += 1; j += n\n", " if reps > max_phrase_run:\n", " out.extend(gram * max_phrase_run); i = j\n", " else:\n", " out.extend(gram); i += n\n", " w = out\n", " return \" \".join(w)\n", "\n", "# ุฃุฅุขูฑ -> ุง. dev short subset: WER 0.5382 -> 0.5320. Free option: a no-op if the scorer\n", "# already folds alef, a gain if it does not. ู‰->ูŠ and ุฉ->ู‡ were tested and HURT.\n", "def fold_alef(t): return _ALEF.sub(_PLAIN_ALEF, t)\n", "\n", "# ู‡ู‡ู‡ู‡ู‡ู‡ู‡ู‡...(216 chars in the blind set) -> ู‡ู‡ู‡. Word-level collapse cannot catch this:\n", "# it is one token, not a repeated token. CER 0.2425 -> 0.2317; WER unchanged.\n", "# keep=3 because the references themselves contain 165 runs of 3+.\n", "def squeeze_elong(t, keep=3): return _ELONG.sub(lambda m: m.group(1) * keep, t)\n", "\n", "def apply_learned(t): return \" \".join(LEARNED_MAP.get(w, w) for w in t.split())\n", "def CLEAN_NORM(t): return norm_ws(collapse_repeats(strip_junk(BEST_NORM(t))))\n", "def FINAL_NORM(t): return norm_ws(apply_learned(squeeze_elong(fold_alef(CLEAN_NORM(t)))))\n", "\n", "def max_word_run(text):\n", " w = text.split(); best = cur = 1\n", " for a, b in zip(w, w[1:]):\n", " cur = cur + 1 if a == b else 1; best = max(best, cur)\n", " return best if w else 0\n", "\n", "# self-test: if any of these fail, STOP - the normalizer is broken and would\n", "# silently blank out the transcripts (this exact failure happened once, see the note above)\n", "_t1 = \"###\" + \"\\u0623\\u0646\\u0627\" + \" \" + \"\\u0647\" * 19 + \" [Speaker B ] \" + \"\\u0634\\u064a\\u0621\"\n", "assert fold_alef(\"\\u0623\") == \"\\u0627\", \"alef folding broken\"\n", "assert BEST_NORM(\"\\u0627\\u0646\\u0627\") == \"\\u0627\\u0646\\u0627\", \"BEST_NORM is deleting Arabic letters - check _DIAC\"\n", "assert FINAL_NORM(_t1) == \"\\u0627\\u0646\\u0627 \\u0647\\u0647\\u0647 \\u0634\\u064a\", FINAL_NORM(_t1)\n", "assert FINAL_NORM((\"\\u0648 \" * 9).strip()) == \"\\u0648 \\u0648 \\u0648\"\n", "assert FINAL_NORM(\"\") == \"\"\n", "print(\"text pipeline ready, self-test passed\")\n", "print(\" sample:\", FINAL_NORM(_t1))" ] }, { "cell_type": "markdown", "id": "c006", "metadata": {}, "source": [ "## 3. Dev references and the scoring harness\n", "\n", "Reads only the `id` and `transcription` columns of the dev parquet shards โ€” column pruning, so no audio\n", "is downloaded here (~2 min).\n", "\n", "`score()` is the tool that was missing all along: it measures a change *before* you spend a submission on\n", "it. It reports the full dev set and the **short subset** (โ‰ค40 reference words), which matches the blind\n", "length distribution (median 5.3 s, 15.3 ref words/clip) and is the honest predictor." ] }, { "cell_type": "code", "id": "c007", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import fsspec, pyarrow.parquet as pq\n", "from jiwer import wer, cer\n", "\n", "DEV_SHARDS = [\"dev-00000-of-00009-051641e73732d26d\", \"dev-00001-of-00009-f2505c9f0c41e739\",\n", " \"dev-00002-of-00009-574478cbf86a090c\", \"dev-00003-of-00009-f2d81e6bc3114920\",\n", " \"dev-00004-of-00009-797f0063ead966d6\", \"dev-00005-of-00009-c68f72797dac98b6\",\n", " \"dev-00006-of-00009-e6e6831129f93d1e\", \"dev-00007-of-00009-7fea04d4729de732\",\n", " \"dev-00008-of-00009-f28e4000cf5e2fa3\"]\n", "\n", "if os.path.exists(\"out/dev_ids_refs.json\"):\n", " _d = json.load(open(\"out/dev_ids_refs.json\", encoding=\"utf-8\"))\n", " dev_ids, dev_refs = _d[\"ids\"], _d[\"refs\"]\n", "else:\n", " fs = fsspec.filesystem(\"http\", client_kwargs={\"headers\": {\"Authorization\": \"Bearer \" + os.environ[\"HF_TOKEN\"]}})\n", " dev_ids, dev_refs = [], []\n", " for nm in DEV_SHARDS:\n", " url = f\"https://huggingface.co/datasets/{DEV_REPO}/resolve/main/data/{nm}.parquet\"\n", " t0 = time.time()\n", " with fs.open(url, \"rb\", block_size=4 * 1024 * 1024) as fh:\n", " t = pq.ParquetFile(fh).read(columns=[\"id\", \"transcription\"])\n", " dev_ids += t.column(\"id\").to_pylist(); dev_refs += t.column(\"transcription\").to_pylist()\n", " print(f\" {nm[:12]} {t.num_rows} rows {time.time()-t0:.0f}s total {len(dev_ids)}\", flush=True)\n", " json.dump({\"ids\": dev_ids, \"refs\": dev_refs}, open(\"out/dev_ids_refs.json\", \"w\", encoding=\"utf-8\"),\n", " ensure_ascii=False)\n", "\n", "DEV_N = len(dev_refs)\n", "DEV_REFS = [norm_ws(r) for r in dev_refs]\n", "\n", "_short = [i for i in range(DEV_N) if len(DEV_REFS[i].split()) <= 40]\n", "_long = [i for i in range(DEV_N) if len(DEV_REFS[i].split()) > 60]\n", "random.Random(0).shuffle(_short)\n", "DEV_EVAL = sorted(set(_short[:DEV_EVAL_N]) | set(_long))\n", "print(f\"\\ndev {DEV_N} clips | eval subset {len(DEV_EVAL)} \"\n", " f\"({sum(len(DEV_REFS[i].split()) for i in DEV_EVAL)} ref words; blind has 6519), \"\n", " f\"including all {len(_long)} long clips so the windowing gets exercised\")\n", "\n", "def score(hyp_by_index, idxs, label, norm=FINAL_NORM, quiet=False):\n", " # references stay in DATASET order; only the hypothesis is ever permuted.\n", " # wer(ref[perm], hyp[perm]) is the same number for every perm - that invariance\n", " # is exactly what hid the ordering bug, so never write a check that way.\n", " h = [norm(hyp_by_index.get(i, \"\")) for i in idxs]\n", " r = [DEV_REFS[i] for i in idxs]\n", " w, c = wer(r, h), cer(r, h)\n", " empty = sum(1 for x in h if not x.strip())\n", " if not quiet:\n", " print(f\" {label:46s} WER {w:.4f} CER {c:.4f}\"\n", " + (f\" [{empty} EMPTY]\" if empty else \"\"))\n", " # A blanking normalizer scores ~0.9997, which slips past a bare `wer < 1.0`\n", " # check. That happened once and cost a full GPU run, so check the cause, not\n", " # just the symptom.\n", " raw_words = sum(len(str(hyp_by_index.get(i, \"\")).split()) for i in idxs)\n", " out_words = sum(len(x.split()) for x in h)\n", " assert empty < 0.25 * len(h), (\n", " f\"{label}: {empty}/{len(h)} hypotheses are EMPTY after normalization. \"\n", " f\"The normalizer is deleting text - this is not a model problem. \"\n", " f\"Check that BEST_NORM('\\u0627\\u0646\\u0627') == '\\u0627\\u0646\\u0627'.\")\n", " assert raw_words == 0 or out_words > 0.5 * raw_words, (\n", " f\"{label}: normalization dropped {raw_words - out_words} of {raw_words} words. \"\n", " f\"Check the _DIAC character class.\")\n", " assert w < 1.0, f\"{label}: WER >= 1 means the pairing is wrong, not the model\"\n", " return w, c\n", "print(\"scoring harness ready\")" ] }, { "cell_type": "markdown", "id": "c008", "metadata": {}, "source": [ "## 4. Models and long-audio-safe transcription\n", "\n", "Ensembling only pays when systems make **different** mistakes. The int4/int8/full/dialectal Cohere builds\n", "are all the same 2B model, so their errors are correlated and combining them gains little. This list\n", "mixes architectures on purpose.\n", "\n", "| key | model | arch | reported |\n", "|---|---|---|---|\n", "| `cohere_base` | `CohereLabs/cohere-transcribe-arabic-07-2026` | Conformer 2B | your system, full precision rather than int8 |\n", "| `whisper_dialect` | `oddadmix/whisper-large-v3-turbo-arabic-dialectal` | Whisper turbo | **WER 0.344 / CER 0.115** vs 0.590 / 0.278 base, ~40k dialectal clips |\n", "| `whisper_lv3` | `openai/whisper-large-v3` | Whisper | maximally different errors from Cohere |\n", "| `cohere_dialect` | `oddadmix/โ€ฆ-dialectal-v2` | Conformer 2B | Gulf โˆ’0.153, Iraqi โˆ’0.239, Moroccan โˆ’0.165, but **+0.044 overall** |\n", "\n", "`cohere_dialect`'s own card says it is worse overall than base โ€” it is here as a *voter*, and ยง5 decides\n", "from dev whether it earns its place.\n", "\n", "Long audio is handled in every path: Cohere is windowed manually at `max_audio_clip_s` with overlap and\n", "stitched on the longest shared word n-gram, so nothing depends on the processor's row count matching the\n", "batch. Whisper and CTC go through the pipeline's own 30 s/5 s chunking." ] }, { "cell_type": "code", "id": "c009", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from tqdm.auto import tqdm\n", "\n", "MODELS = {\n", " \"cohere_base\": dict(id=\"CohereLabs/cohere-transcribe-arabic-07-2026\", family=\"cohere\"),\n", " \"whisper_dialect\": dict(id=\"oddadmix/whisper-large-v3-turbo-arabic-dialectal\", family=\"whisper\"),\n", " \"whisper_lv3\": dict(id=\"openai/whisper-large-v3\", family=\"whisper\"),\n", " \"cohere_dialect\": dict(id=\"oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2\", family=\"cohere\"),\n", " \"cohere_int8\": dict(id=\"NAMAA-Space/cohere-transcribe-arabic-07-2026-int8\", family=\"cohere\"),\n", " \"w2v2_xlsr\": dict(id=\"jonatasgrosman/wav2vec2-large-xlsr-53-arabic\", family=\"ctc\"),\n", "}\n", "ACTIVE = [\"cohere_base\"] if QUICK_MODE else [\"cohere_base\", \"whisper_dialect\", \"whisper_lv3\", \"cohere_dialect\"]\n", "print(\"active systems:\", ACTIVE)\n", "\n", "GEN = dict(max_new_tokens=440, num_beams=5, repetition_penalty=1.1, no_repeat_ngram_size=8)\n", "GEN_STRICT = dict(max_new_tokens=440, num_beams=1, repetition_penalty=1.3, no_repeat_ngram_size=5)\n", "BATCH = 8\n", "_S = {}\n", "\n", "def unload():\n", " global _S\n", " _S = {}; gc.collect()\n", " if HAS_GPU: torch.cuda.empty_cache()\n", "\n", "def load_model(key):\n", " global _S\n", " unload()\n", " mid, fam = MODELS[key][\"id\"], MODELS[key][\"family\"]\n", " if fam == \"cohere\":\n", " from transformers import AutoProcessor, CohereAsrForConditionalGeneration\n", " proc = AutoProcessor.from_pretrained(mid)\n", " mdl = CohereAsrForConditionalGeneration.from_pretrained(mid, device_map=\"auto\").eval()\n", " dt = next((p.dtype for p in mdl.parameters() if p.dtype in (torch.float16, torch.bfloat16)), torch.float32)\n", " fe = proc.feature_extractor\n", " _S = dict(key=key, fam=fam, proc=proc, mdl=mdl, dt=dt,\n", " max_s=float(getattr(fe, \"max_audio_clip_s\", 35.0)),\n", " ov_s=float(getattr(fe, \"overlap_chunk_second\", 5.0)))\n", " print(f\"{key}: {mid} | {dt} | max_clip {_S['max_s']}s overlap {_S['ov_s']}s\")\n", " else:\n", " from transformers import pipeline\n", " _S = dict(key=key, fam=fam,\n", " pipe=pipeline(\"automatic-speech-recognition\", model=mid, device=0,\n", " torch_dtype=torch.float16, chunk_length_s=30, stride_length_s=5))\n", " print(f\"{key}: {mid} | pipeline, 30s/5s chunking\")\n", " return _S\n", "\n", "def _cast(d, dt):\n", " return {k: (v.to(dt) if (torch.is_tensor(v) and torch.is_floating_point(v)) else v) for k, v in d.items()}\n", "\n", "@torch.no_grad()\n", "def _cohere_decode(auds, gen):\n", " a = [np.asarray(x, dtype=\"float32\") for x in auds]\n", " try:\n", " inp = _S[\"proc\"](a, sampling_rate=SR, language=\"ar\", return_tensors=\"pt\", padding=True).to(_S[\"mdl\"].device)\n", " inp.pop(\"audio_chunk_index\", None)\n", " out = _S[\"mdl\"].generate(**_cast(inp, _S[\"dt\"]), **gen)\n", " txt = [t.strip() for t in _S[\"proc\"].batch_decode(out, skip_special_tokens=True)]\n", " if len(txt) != len(a):\n", " raise RuntimeError(\"processor chunked - per-clip fallback\") # never let zip() paper over this\n", " return txt\n", " except Exception:\n", " res = []\n", " for x in a:\n", " inp = _S[\"proc\"](x, sampling_rate=SR, language=\"ar\", return_tensors=\"pt\").to(_S[\"mdl\"].device)\n", " inp.pop(\"audio_chunk_index\", None)\n", " o = _S[\"mdl\"].generate(**_cast(inp, _S[\"dt\"]), **gen)\n", " res.append(\" \".join(t.strip() for t in _S[\"proc\"].batch_decode(o, skip_special_tokens=True)).strip())\n", " return res\n", "\n", "def _stitch(a, b, max_overlap_words=25):\n", " # drop the longest word n-gram that ends `a` and starts `b` (the overlap region)\n", " aw, bw = a.split(), b.split()\n", " if not aw or not bw: return \" \".join(aw + bw)\n", " for k in range(min(max_overlap_words, len(aw), len(bw)), 1, -1):\n", " if aw[-k:] == bw[:k]:\n", " return \" \".join(aw + bw[k:])\n", " return \" \".join(aw + bw)\n", "\n", "def _cohere_long(arr, gen):\n", " win = int((_S[\"max_s\"] - 1.0) * SR)\n", " hop = int((_S[\"max_s\"] - 1.0 - _S[\"ov_s\"]) * SR)\n", " pieces = [arr[s:s + win] for s in range(0, max(len(arr) - win + hop, 1), hop)]\n", " pieces = [p for p in pieces if len(p) > int(0.4 * SR)]\n", " outs = []\n", " for k in range(0, len(pieces), 2):\n", " outs += _cohere_decode(pieces[k:k + 2], gen)\n", " text = outs[0] if outs else \"\"\n", " for nxt in outs[1:]:\n", " text = _stitch(text, nxt)\n", " return text\n", "\n", "def transcribe(ds, indices, jsonl, tag=\"\"):\n", " # returns {dataset_index: raw_text}. Resume-safe: rerun after a disconnect and it continues.\n", " done = {}\n", " if os.path.exists(jsonl):\n", " for line in open(jsonl, encoding=\"utf-8\"):\n", " r = json.loads(line); done[r[\"i\"]] = r[\"raw\"]\n", " todo = [i for i in indices if i not in done]\n", " if not todo:\n", " print(f\"{tag}: all {len(indices)} cached\"); return done\n", " dur = {i: ds[i][\"audio\"][\"array\"].shape[0] / SR for i in todo}\n", " f = open(jsonl, \"a\", encoding=\"utf-8\")\n", " def _save(i, t):\n", " done[i] = t\n", " f.write(json.dumps({\"i\": i, \"raw\": t}, ensure_ascii=False) + \"\\n\")\n", " f.flush(); os.fsync(f.fileno())\n", "\n", " if _S[\"fam\"] == \"cohere\":\n", " mx = _S[\"max_s\"]\n", " long_i = [i for i in todo if dur[i] > mx]\n", " short_i = sorted([i for i in todo if dur[i] <= mx], key=lambda i: dur[i])\n", " print(f\"{tag}: {len(short_i)} short + {len(long_i)} long (windowed), {len(done)} cached\")\n", " for k in tqdm(range(0, len(short_i), BATCH), desc=f\"{tag} short\"):\n", " b = short_i[k:k + BATCH]\n", " for i, t in zip(b, _cohere_decode([ds[i][\"audio\"][\"array\"] for i in b], GEN)):\n", " _save(i, t)\n", " for i in tqdm(long_i, desc=f\"{tag} long\"):\n", " _save(i, _cohere_long(ds[i][\"audio\"][\"array\"], GEN))\n", " bad = [i for i in indices\n", " if max_word_run(FINAL_NORM(done.get(i, \"\"))) >= 4 or not FINAL_NORM(done.get(i, \"\")).strip()]\n", " if bad:\n", " print(f\"{tag}: re-decoding {len(bad)} degenerate/empty\")\n", " for i in bad:\n", " arr = ds[i][\"audio\"][\"array\"]\n", " alt = (_cohere_long(arr, GEN_STRICT) if arr.shape[0] / SR > mx\n", " else _cohere_decode([arr], GEN_STRICT)[0])\n", " a, b_ = FINAL_NORM(done.get(i, \"\")), FINAL_NORM(alt)\n", " if b_.strip() and (not a.strip() or max_word_run(b_) < max_word_run(a)):\n", " _save(i, alt)\n", " else:\n", " kw = dict(generate_kwargs={\"language\": \"ar\", \"task\": \"transcribe\"}) if _S[\"fam\"] == \"whisper\" else {}\n", " for i in tqdm(todo, desc=tag):\n", " out = _S[\"pipe\"]({\"raw\": np.asarray(ds[i][\"audio\"][\"array\"], dtype=\"float32\"),\n", " \"sampling_rate\": SR}, **kw)\n", " _save(i, (out[\"text\"] or \"\").strip())\n", " f.close()\n", " return done\n", "print(\"transcription ready\")" ] }, { "cell_type": "markdown", "id": "c010", "metadata": {}, "source": [ "## 5. Run each model on dev and score it" ] }, { "cell_type": "code", "id": "c011", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datasets import load_dataset, Audio\n", "\n", "DEV_SYS, DEV_SCORE = {}, {}\n", "if RUN_DEV:\n", " dev_ds = load_dataset(DEV_REPO, split=\"dev\").cast_column(\"audio\", Audio(sampling_rate=SR))\n", " assert len(dev_ds) == DEV_N and list(dev_ds[\"id\"]) == dev_ids, \"dev row order changed\"\n", " for key in ACTIVE:\n", " load_model(key)\n", " DEV_SYS[key] = transcribe(dev_ds, DEV_EVAL, f\"out/dev_{key}.jsonl\", tag=f\"dev/{key}\")\n", " unload()\n", " print(\"\\n=== single-system dev WER (eval subset) ===\")\n", " for key in ACTIVE:\n", " DEV_SCORE[key] = score(DEV_SYS[key], DEV_EVAL, key)\n", " BEST_SINGLE = min(DEV_SCORE, key=lambda k: DEV_SCORE[k][0])\n", " print(f\"\\nbest single: {BEST_SINGLE} WER {DEV_SCORE[BEST_SINGLE][0]:.4f}\")\n", "\n", " long_in_eval = [i for i in DEV_EVAL if len(DEV_REFS[i].split()) > 60]\n", " if long_in_eval:\n", " hw = sum(len(FINAL_NORM(DEV_SYS[BEST_SINGLE][i]).split()) for i in long_in_eval)\n", " rw = sum(len(DEV_REFS[i].split()) for i in long_in_eval)\n", " print(f\"long-clip length ratio {hw/rw:.2f} over {len(long_in_eval)} clips \"\n", " f\"(was 0.71 before the windowing fix; 1.00 is the target)\")\n", "else:\n", " BEST_SINGLE = ACTIVE[0]\n", " print(\"dev scoring skipped\")" ] }, { "cell_type": "markdown", "id": "c012", "metadata": {}, "source": [ "## 6. Combination\n", "\n", "**MBR** โ€” per clip keep the candidate with the smallest average character edit distance to the others.\n", "Conservative: the output is always a real hypothesis from a real system.\n", "\n", "**ROVER** โ€” align every system to a pivot and majority-vote each slot, deletions voting as ฮต so a word\n", "only one system produced cannot win by default. Fixes errors *inside* a clip, but can produce word salad\n", "when systems disagree structurally.\n", "\n", "**Weighted ROVER** โ€” the same, each vote weighted by `1/WER` on dev.\n", "\n", "**Oracle** โ€” the candidate that actually scores best per clip. Not submittable; it is the ceiling. If the\n", "oracle sits close to the best single system, the systems are too correlated for any combiner to help.\n", "\n", "The combiner code below is unit-tested inline. On 600 real dev clips with three synthetically diversified\n", "systems it scored ROVER 0.5624 against a best member of 0.6008, recovering 57% of the gap." ] }, { "cell_type": "code", "id": "c013", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import jiwer\n", "\n", "def _ed(a, b):\n", " if a == b: return 0\n", " la, lb = len(a), len(b)\n", " if not la or not lb: return max(la, lb)\n", " prev = list(range(lb + 1))\n", " for i, ca in enumerate(a, 1):\n", " cur = [i]\n", " for j, cb in enumerate(b, 1):\n", " cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))\n", " prev = cur\n", " return prev[-1]\n", "\n", "def mbr_pick(cands):\n", " cands = [c for c in cands if c.strip()] or [\"\"]\n", " if len(cands) == 1: return cands[0]\n", " costs = [sum(_ed(c, o) / max(len(c), len(o), 1) for o in cands if o is not c) for c in cands]\n", " return cands[costs.index(min(costs))]\n", "\n", "def rover(cands, weights=None):\n", " toks = [c.split() for c in cands]\n", " if weights is None: weights = [1.0] * len(toks)\n", " live = [(t, w) for t, w in zip(toks, weights) if t]\n", " if not live: return \"\"\n", " if len(live) == 1: return \" \".join(live[0][0])\n", " piv = max(range(len(live)), key=lambda k: len(live[k][0]) * live[k][1])\n", " pt = live[piv][0]\n", " slots = [[(w, live[piv][1])] for w in pt]\n", " for k, (tk, wt) in enumerate(live):\n", " if k == piv: continue\n", " al = jiwer.process_words([\" \".join(pt)], [\" \".join(tk)]).alignments[0]\n", " covered = set()\n", " for ch in al:\n", " if ch.type in (\"equal\", \"substitute\"):\n", " for off, r in enumerate(range(ch.ref_start_idx, ch.ref_end_idx)):\n", " h = ch.hyp_start_idx + off\n", " if r < len(slots) and h < len(tk):\n", " slots[r].append((tk[h], wt)); covered.add(r)\n", " elif ch.type == \"delete\":\n", " for r in range(ch.ref_start_idx, ch.ref_end_idx):\n", " if r < len(slots): slots[r].append((None, wt)); covered.add(r)\n", " for r in range(len(slots)):\n", " if r not in covered: slots[r].append((None, wt))\n", " out = []\n", " for s in slots:\n", " tally = {}\n", " for w, wt in s: tally[w] = tally.get(w, 0.0) + wt\n", " best = max(tally.items(), key=lambda kv: (kv[1], kv[0] is not None))\n", " if best[0] is not None: out.append(best[0])\n", " return \" \".join(out)\n", "\n", "# inline unit tests - a broken combiner must not reach the GPU run\n", "assert rover([\"a b c\", \"a b c\", \"a x c\"]) == \"a b c\", \"majority should fix a substitution\"\n", "assert rover([\"a b c\", \"a c\", \"a b c\"]) == \"a b c\", \"majority should restore a deletion\"\n", "assert rover([\"a b c\", \"a b z c\", \"a b c\"]) == \"a b c\", \"minority insertion should be rejected\"\n", "assert rover([\"a b c\", \"\", \"a b c\"]) == \"a b c\", \"empty system should be ignored\"\n", "assert mbr_pick([\"a b c\", \"a b c\", \"z z z\"]) == \"a b c\", \"mbr should pick the consensus\"\n", "print(\"combiner unit tests passed\")\n", "\n", "def combine(sysdict, idxs, method, keys, dev_wer=None):\n", " wts = [1.0 / max(dev_wer[k][0], 1e-3) for k in keys] if (method == \"wrover\" and dev_wer) else None\n", " out = {}\n", " for i in idxs:\n", " cands = [FINAL_NORM(sysdict[k].get(i, \"\")) for k in keys]\n", " out[i] = mbr_pick(cands) if method == \"mbr\" else rover(cands, wts)\n", " return out" ] }, { "cell_type": "code", "id": "c014", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "WINNER, RESULTS = BEST_SINGLE, {}\n", "if RUN_DEV and len(ACTIVE) > 1:\n", " RESULTS = {k: DEV_SCORE[k][0] for k in ACTIVE}\n", " print(\"=== combinations on the dev subset ===\")\n", " for m in (\"mbr\", \"rover\", \"wrover\"):\n", " RESULTS[m] = score(combine(DEV_SYS, DEV_EVAL, m, ACTIVE, DEV_SCORE), DEV_EVAL,\n", " f\"[{m}] all {len(ACTIVE)} systems\", norm=lambda t: t)[0]\n", " if len(ACTIVE) > 2:\n", " print()\n", " for drop in ACTIVE: # a system dragging the ensemble down shows up here\n", " keys = [k for k in ACTIVE if k != drop]\n", " RESULTS[\"mbr::\" + \",\".join(keys)] = score(\n", " combine(DEV_SYS, DEV_EVAL, \"mbr\", keys, DEV_SCORE), DEV_EVAL,\n", " f\"[mbr] without {drop}\", norm=lambda t: t)[0]\n", " orc = {i: min([FINAL_NORM(DEV_SYS[k].get(i, \"\")) for k in ACTIVE],\n", " key=lambda c: jiwer.process_words([DEV_REFS[i]], [c or \" \"]).wer) for i in DEV_EVAL}\n", " print()\n", " score(orc, DEV_EVAL, \"[oracle] best candidate per clip (CEILING, not submittable)\", norm=lambda t: t)\n", "\n", " WINNER = min(RESULTS, key=lambda k: RESULTS[k])\n", " print(f\"\\n>>> winner: {WINNER} WER {RESULTS[WINNER]:.4f}\"\n", " f\" (best single {BEST_SINGLE} {DEV_SCORE[BEST_SINGLE][0]:.4f})\")\n", " if RESULTS[WINNER] >= DEV_SCORE[BEST_SINGLE][0] - 0.002:\n", " WINNER = BEST_SINGLE\n", " print(\" gain is inside the noise of this subset - falling back to the single model\")\n", "print(\"submitting with:\", WINNER)" ] }, { "cell_type": "markdown", "id": "c015", "metadata": {}, "source": [ "## 7. Blind transcription and submission" ] }, { "cell_type": "code", "id": "c016", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import zipfile\n", "\n", "test_ds = load_dataset(TEST_REPO, split=\"test\").cast_column(\"audio\", Audio(sampling_rate=SR))\n", "test_ids = list(test_ds[\"id\"])\n", "assert len(test_ds) == N_TEST, f\"expected {N_TEST} clips, got {len(test_ds)}\"\n", "\n", "if WINNER in (\"mbr\", \"rover\", \"wrover\"): NEEDED, METHOD = ACTIVE, WINNER\n", "elif WINNER.startswith(\"mbr::\"): NEEDED, METHOD = WINNER[5:].split(\",\"), \"mbr\"\n", "else: NEEDED, METHOD = [WINNER], None\n", "print(\"systems needed:\", NEEDED, \"| method:\", METHOD or \"single\")\n", "\n", "TEST_SYS = {}\n", "for key in NEEDED:\n", " load_model(key)\n", " TEST_SYS[key] = transcribe(test_ds, list(range(N_TEST)), f\"out/test_{key}.jsonl\", tag=f\"blind/{key}\")\n", "unload()\n", "\n", "if METHOD:\n", " merged = combine(TEST_SYS, range(N_TEST), METHOD, NEEDED, DEV_SCORE)\n", " lines = [merged[i] for i in range(N_TEST)]\n", "else:\n", " lines = [FINAL_NORM(TEST_SYS[WINNER][i]) for i in range(N_TEST)]\n", "lines = [l if l.strip() else \"\\u0646\\u0639\\u0645\" for l in lines] # a blank line scores worse than any guess" ] }, { "cell_type": "code", "id": "c017", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def build(name, lines, note):\n", " txt, zp = f\"out/{name}.txt\", f\"out/{name}.zip\"\n", " with open(txt, \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n", " f.write(\"\\n\".join(lines) + \"\\n\")\n", " if os.path.exists(zp): os.remove(zp)\n", " with zipfile.ZipFile(zp, \"w\", zipfile.ZIP_DEFLATED) as z:\n", " z.write(txt, \"predictions.txt\") # exactly one file, at the zip root\n", " with zipfile.ZipFile(zp) as z: inside = [(i.filename, i.file_size) for i in z.infolist()]\n", " assert len(inside) == 1 and inside[0][0] == \"predictions.txt\" and inside[0][1] > 2000, inside\n", " assert len(lines) == N_TEST, f\"{len(lines)} != {N_TEST}\"\n", " assert all(l.strip() for l in lines), \"blank line\"\n", " assert all(l == l.strip() for l in lines), \"untrimmed line\"\n", " print(f\" {name:38s} lines={len(lines)} words={sum(len(l.split()) for l in lines):5d} \"\n", " f\"zip={os.path.getsize(zp)}B [{note}]\")\n", " return zp\n", "\n", "# line k is the transcript of dataset row k - NEVER sorted(ids). This is the 0.9-WER bug.\n", "FINAL = build(f\"predictions_{WINNER}\", lines, \"UPLOAD THIS\")\n", "for key in TEST_SYS:\n", " build(f\"single_{key}\", [FINAL_NORM(TEST_SYS[key][i]) for i in range(N_TEST)], \"fallback\")\n", "\n", "durs = [test_ds[i][\"audio\"][\"array\"].shape[0] / SR for i in range(N_TEST)]\n", "rate = sorted(len(lines[i].split()) / durs[i] for i in range(N_TEST) if durs[i] <= 35)\n", "med = rate[len(rate) // 2]\n", "print(f\"\\nlong-clip check (median {med:.2f} w/s on clips under 35 s):\")\n", "for i in [i for i in range(N_TEST) if durs[i] > 35]:\n", " w = len(lines[i].split())\n", " print(f\" row {i:3d} {durs[i]:5.1f}s -> {w:3d} words ({w/durs[i]:.2f} w/s)\"\n", " + (\" <-- STILL TRUNCATED\" if w / durs[i] < 0.5 * med else \" ok\"))\n", "\n", "_t = open(f\"out/predictions_{WINNER}.txt\", encoding=\"utf-8\").read()\n", "print(f\"\\njunk check: char runs 4+ {len(re.findall(r'(.)\\1{3,}', _t))} | \"\n", " f\"'#' {_t.count('#')} | latin {len(re.findall(r'[A-Za-z]', _t))} | \"\n", " f\"word-repeat runs>=4 {sum(1 for l in _t.splitlines() if max_word_run(l) >= 4)}\")\n", "\n", "try:\n", " from google.colab import files; files.download(FINAL)\n", "except Exception:\n", " print(\"download manually:\", FINAL)" ] }, { "cell_type": "markdown", "id": "c018", "metadata": {}, "source": [ "## 8. Push results back" ] }, { "cell_type": "code", "id": "c019", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if HF_REPO:\n", " for fn in sorted(os.listdir(\"out\")):\n", " p = f\"out/{fn}\"\n", " if os.path.isfile(p) and os.path.getsize(p) < 200 * 1024 * 1024:\n", " api.upload_file(path_or_fileobj=p, path_in_repo=f\"{PUSH_PREFIX}/{fn}\",\n", " repo_id=HF_REPO, commit_message=f\"standalone pipeline, winner={WINNER}\")\n", " print(\"pushed\", f\"{PUSH_PREFIX}/{fn}\")\n", "else:\n", " print(\"HF_REPO is None - not pushing\")" ] }, { "cell_type": "markdown", "id": "c020", "metadata": {}, "source": [ "---\n", "## After the quick run\n", "\n", "`QUICK_MODE = True` gives you a valid submission from one model with the long-audio fix applied โ€” that\n", "alone is the โ‰ˆ7-point item. Then set `QUICK_MODE = False` and rerun: every system caches to its own\n", "jsonl, so the work already done is not repeated.\n", "\n", "## Reading ยง6\n", "\n", "Compare best single, best combination, and the oracle.\n", "\n", "* **Oracle far below the best single** โ€” the systems disagree productively; add `w2v2_xlsr` (CTC makes\n", " completely different errors) or route per dialect with your Task 2 classifier, since `cohere_dialect`\n", " is reported at โˆ’0.15 to โˆ’0.24 WER on Gulf/Iraqi/Moroccan.\n", "* **Oracle close to the best single** โ€” the systems share their mistakes. Stop ensembling; fine-tune on\n", " dev's 3152 labelled in-domain clips instead. That is the remaining lever, and the model is currently\n", " zero-shot.\n", "* **A combination beating the best single by under ~1 point** โ€” noise at this subset size. The notebook\n", " falls back to the single model automatically.\n", "\n", "## Where the error actually is\n", "\n", "Dev, correct pairing: substitutions 36.9%, **deletions 16.3%**, insertions 5.5%. The model drops speech\n", "rather than inventing it, and 3.7% of dev clips produce under half the reference length. ยง4 recovers the\n", "truncation share of that; the rest is the model.\n", "\n", "## Two invariants\n", "\n", "1. Line *k* of `predictions.txt` is dataset row *k*. Never `sorted(ids)`.\n", "2. An order check must permute **one side only**. `wer(ref[perm], hyp[perm])` is identical for every\n", " `perm` โ€” that is how the 1.3757 submission passed its check." ] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" }, "accelerator": "GPU" }, "nbformat": 4, "nbformat_minor": 5 }