{ "cells": [ { "cell_type": "markdown", "id": "c000", "metadata": {}, "source": [ "# NADI 2026 - Subtask 1.2 Mixed ASR - v2 pipeline (larger models, harder cleanup, dev fine-tune)\n", "\n", "**Self-contained.** No other file or repo artifact is required beyond an `HF_TOKEN`\n", "(Colab > key icon Secrets > `HF_TOKEN`). Runtime > **GPU** (A100 recommended once you turn on\n", "fine-tuning; T4/L4 is enough for `QUICK_MODE`).\n", "\n", "Score history: `1.3757` (id-sorted bug) `-> 0.48` (order fix) `-> ~0.45` (4-model ensemble,\n", "the version this notebook extends). Current public best on the leaderboard: `0.44`.\n", "\n", "## What changed since the 0.45 run\n", "\n", "| # | change | why | expected |\n", "|---|---|---|---|\n", "| 1 | swap in `whisper-large-v3-arabic-dialectal-v2` (1.54B, new) and the `-v2` turbo checkpoint | oddadmix's own balanced-test WER for these is 0.3199 / 0.3324, both better than the `whisper-large-v3-turbo-arabic-dialectal` (no `-v2`) this repo was using | a stronger single system feeding the ensemble |\n", "| 2 | silence trim (librosa) + words/sec + type-token-ratio degeneracy check on **every** clip, **every** model family | the old check only caught long Cohere clips with an exact-repeat word run; \"keyboard spam\" / hallucinated loops happen on short clips and in Whisper too, usually triggered by silence padding | fewer catastrophic per-clip WER spikes, which move the aggregate metric more than uniform small gains |\n", "| 3 | LoRA fine-tune of the winning Whisper system on dev, scored on the held-out `DEV_SEL` slice it never trains on | the prior README named this the largest remaining lever and never did it | the one change that can plausibly clear 0.44; everything else here is polish |\n", "\n", "Section 4 lists every model with its self-reported WER so you can judge the swap yourself.\n", "Nemotron's ASR fine-tune scored worse anyway (0.432) and needs the separate NeMo toolkit, so\n", "it is documented but not wired in - not worth the fragility for a lower-scoring system.\n", "\n", "## Sections\n", "\n", "| section | | GPU? |\n", "|---|---|---|\n", "| 1 | config, install, login, cache restore | - |\n", "| 2 | text pipeline (`FINAL_NORM`) | CPU |\n", "| 3 | dev references + scoring harness | CPU, ~2 min |\n", "| 4 | model registry | - |\n", "| 5 | audio preprocessing: silence trim + universal degeneracy guard | - |\n", "| 6 | transcription engine (long-audio windowing kept from the 0.48 fix) | - |\n", "| 7 | run models on dev, score each | GPU |\n", "| 8 | **LoRA fine-tune** the best Whisper system on dev (excludes `DEV_SEL`) | GPU, slow |\n", "| 9 | combine: MBR / ROVER / weighted ROVER / oracle ceiling | CPU |\n", "| 10 | blind transcription + submission, guards asserted | GPU |\n", "| 11 | push results back to HF | - |\n", "\n", "`QUICK_MODE = True` (default) gives a valid submission from one model, no fine-tuning, in well\n", "under an hour. Set it `False` for the full ensemble, and separately set `DO_FINETUNE = True` when\n", "you are ready to spend the GPU time on section 8 - it is the step most likely to move the score,\n", "and the one you should not skip if 0.44 is the target.\n", "\n", "**Deadline close?** Set `DEADLINE_MODE = True` in section 1. It forces one model\n", "(`DEADLINE_MODEL_KEY`), a `DEV_SANITY_N`-clip sanity check instead of a real dev sweep (just\n", "enough to catch a broken model/normalizer before it burns your remaining time), skips the\n", "long-clip diagnostic, skips fine-tuning, and goes straight to the blind test. Whisper-family\n", "transcription is batched (`batch_size=BATCH`, matching the Cohere path) rather than one clip per\n", "`pipe()` call - a python-level loop over single-item calls never engages a pipeline's batching no\n", "matter how it was constructed, which was the actual bottleneck in one live run (5.63s/clip)." ] }, { "cell_type": "markdown", "id": "c001", "metadata": {}, "source": [ "## 1. Config, install, login, cache restore" ] }, { "cell_type": "code", "id": "c002", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ------------------------------- CONFIG --------------------------------------\n", "QUICK_MODE = True # True : 1 model, 200 dev clips -> a valid submission fast\n", " # False: full ensemble, 500 dev clips (~2-3h on an A100)\n", "DO_FINETUNE = False # True : run section 8 (LoRA fine-tune). Real GPU time (~1-2h/epoch\n", " # for ~2700 clips on an A100). Leave False for your first pass.\n", "FT_MAX_CLIPS = 2000 # cap on fine-tune training clips, for session-length control\n", "FT_EPOCHS = 2\n", "TRIM_SILENCE = True # librosa.effects.trim before every model sees the audio\n", "ADD_QWEN3_ASR = False # opt-in extra ensemble voter; skipped automatically if the\n", " # installed transformers build lacks AutoModelForMultimodalLM\n", "RUN_DEV = True # score on dev before submitting. Leave True.\n", "HF_REPO = \"FatimahEmadEldin/Nadi\" # None to skip pushing\n", "PUSH_PREFIX = \"claude/improved_e2e\"\n", "RESTORE_CACHE = True # pull previously computed transcripts from claude/cache/\n", "DEV_EVAL_N = 200 if QUICK_MODE else 500\n", "\n", "# Deadline mode overrides everything above: ONE model, a tiny sanity check (not a real dev\n", "# sweep), no ensemble, no fine-tune, straight to the blind test. Turn this on when a\n", "# submission deadline is close and QUICK_MODE's ~1h isn't fast enough or safe enough to gamble\n", "# on the full ensemble. DEADLINE_MODEL_KEY must be a key in the MODELS dict defined in section 4.\n", "DEADLINE_MODE = False\n", "DEADLINE_MODEL_KEY = \"cohere_base\" # pick whichever single model you already trust most\n", "DEV_SANITY_N = 30 # just enough to catch a broken model/normalizer, not a selection metric\n", "if DEADLINE_MODE:\n", " RUN_DEV, DO_FINETUNE, DEV_EVAL_N = True, False, DEV_SANITY_N\n", " print(f\"DEADLINE MODE: {DEADLINE_MODEL_KEY} only, {DEV_SANITY_N}-clip sanity check, \"\n", " f\"no ensemble, no fine-tune, straight to the blind test\")\n", "# -------------------------------------------------------------------------------\n", "\n", "import sys, subprocess, os, json, gc, time, re, random, glob, zipfile, getpass\n", "\n", "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-U\",\n", " \"transformers\", \"accelerate\", \"datasets\", \"soundfile\", \"librosa\",\n", " \"jiwer\", \"huggingface_hub\", \"pyarrow\", \"fsspec\", \"peft\"], check=True)\n", "\n", "import torch, numpy as np\n", "import fsspec, pyarrow.parquet as pq, jiwer\n", "from jiwer import wer, cer\n", "from tqdm.auto import tqdm\n", "from huggingface_hub import login, HfApi, hf_hub_download, list_repo_files\n", "from datasets import load_dataset, Audio\n", "\n", "DEV_REPO = \"UBC-NLP/NADI2026_subtask2_MixedASR\"\n", "TEST_REPO = \"UBC-NLP/NADI2026_subtask1.2_MixedASR_test\"\n", "N_TEST, SR = 427, 16000\n", "\n", "HAS_GPU = torch.cuda.is_available()\n", "print(\"torch\", torch.__version__, \"| GPU:\",\n", " torch.cuda.get_device_name(0) if HAS_GPU else \"NONE (sections 1-3, 5, 9 still work)\")\n", "if HAS_GPU:\n", " torch.backends.cuda.matmul.allow_tf32 = True\n", " torch.backends.cudnn.allow_tf32 = True\n", "os.makedirs(\"out\", exist_ok=True)" ] }, { "cell_type": "code", "id": "c003", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "_tok = None\n", "try:\n", " from google.colab import userdata\n", " _tok = userdata.get(\"HF_TOKEN\")\n", "except Exception:\n", " _tok = os.environ.get(\"HF_TOKEN\")\n", "if not _tok:\n", " _tok = getpass.getpass(\"HF token: \").strip()\n", "login(token=_tok, add_to_git_credential=False)\n", "os.environ[\"HF_TOKEN\"] = _tok\n", "api = HfApi(token=_tok)\n", "print(\"logged in |\", \"QUICK mode\" if QUICK_MODE else \"FULL ensemble mode\",\n", " \"| fine-tune\", \"ON\" if DO_FINETUNE else \"off\")\n", "\n", "# Raw model output (out/*.jsonl) is the expensive artifact - it costs real GPU time\n", "# and nothing downstream depends on how it was produced, so restoring it lets a\n", "# disconnected/rerun session skip transcription entirely.\n", "if RESTORE_CACHE and HF_REPO:\n", " try:\n", " hits = [f for f in list_repo_files(HF_REPO, token=_tok)\n", " if f.startswith(\"claude/cache/\") and f.endswith(\".jsonl\")]\n", " for f in hits:\n", " dst = \"out/\" + os.path.basename(f)\n", " if not os.path.exists(dst):\n", " open(dst, \"wb\").write(open(hf_hub_download(HF_REPO, f, token=_tok), \"rb\").read())\n", " print(\" restored\", dst)\n", " print(f\"cache: {len(hits)} file(s) on the hub\")\n", " except Exception as e:\n", " print(\"cache restore skipped:\", type(e).__name__)" ] }, { "cell_type": "markdown", "id": "c004", "metadata": {}, "source": [ "## 2. Text pipeline\n", "\n", "Byte-identical to the pipeline that scored 0.48 / ~0.45, with the same guard rails. Text\n", "post-processing was measured exhausted in the prior round (~0.7 WER points, all applied here) -\n", "this notebook's new gains are meant to come from acoustics (section 5-6) and fine-tuning\n", "(section 8), not from more regex.\n", "\n", "Arabic below is written as `\\uXXXX` escapes **on purpose**. A literal character class silently\n", "became `[U+0617-U+064B]` in an earlier draft - a range that covers `U+0621-U+064A`, i.e. every\n", "Arabic letter - and the normalizer deleted 2980 of 3152 transcripts, scoring WER 0.9996. Escapes\n", "cannot be silently reordered by an editor or terminal the way combining marks can." ] }, { "cell_type": "code", "id": "c005", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# U+0617-U+061A small high marks | U+064B-U+0652 tanween..sukun | U+0670 superscript alef | U+0640 tatweel\n", "_DIAC = re.compile(\"[\\u0617-\\u061a\\u064b-\\u0652\\u0670\\u0640]\")\n", "_PUNCT_CHARS = (\"\\u061b\\u060c\\u061f\"\n", " + \"`\\u00f7\\u00d7_()*^%][/:\\\"{}~\\u00a6+|!\\u2026\\u2013\\u2019\\u201d\\u201c;?.,\")\n", "_PUNCT = re.compile(\"[\" + re.escape(_PUNCT_CHARS) + \"]\")\n", "_BRACKETED = re.compile(r\"[\\[\\(\\{<][^\\]\\)\\}>]{0,40}[\\]\\)\\}>]\")\n", "_TAGWORDS = re.compile(r\"\\b[Ss][Pp][Ee][Aa][Kk][Ee][Rr]\\s*[A-Za-z0-9]?\\b\")\n", "_JUNK = re.compile(\"[#@\\u266a\\u2669\\u266b\\u266c\\u2018\\\\*\\\\$&<>\\\\\\\\]+\") # incl. U+2018, missing from _PUNCT\n", "_LATIN = re.compile(r\"[A-Za-z]+\")\n", "_ELONG = re.compile(r\"(.)\\1{3,}\")\n", "_ALEF = re.compile(\"[\\u0623\\u0625\\u0622\\u0671]\") # alef: hamza above/below, madda, wasla\n", "_PLAIN_ALEF = \"\\u0627\"\n", "_FILLER = \"\\u0646\\u0639\\u0645\" # never ship a blank line\n", "LEARNED_MAP = {\"\\u0634\\u064a\\u0621\": \"\\u0634\\u064a\"} # mined on dev, 2-fold CV\n", "\n", "def norm_ws(t): return re.sub(r\"\\s+\", \" \", t).strip()\n", "def BEST_NORM(t): return norm_ws(_PUNCT.sub(\" \", _DIAC.sub(\"\", t)))\n", "\n", "def strip_junk(t):\n", " t = _BRACKETED.sub(\" \", t); t = _TAGWORDS.sub(\" \", t)\n", " t = _JUNK.sub(\" \", t); t = _LATIN.sub(\" \", t)\n", " return norm_ws(t)\n", "\n", "def collapse_repeats(text, max_run=3, max_ngram=5, max_phrase_run=3):\n", " # kill decoder loops: exact adjacent-word repeats and repeated n-gram phrases.\n", " # grid-searched on dev: strip=1, max_run=3, max_phrase_run=3 -> WER 0.5866, CER 0.3123.\n", " w = text.split()\n", " if not w: return text\n", " out, run = [], 0\n", " for tok in w:\n", " if out and tok == out[-1]:\n", " run += 1\n", " if run >= max_run: continue\n", " else:\n", " run = 0\n", " out.append(tok)\n", " w = out\n", " for n in range(max_ngram, 1, -1):\n", " i, out = 0, []\n", " while i < len(w):\n", " gram = w[i:i + n]\n", " if len(gram) < n:\n", " out.extend(w[i:]); break\n", " reps, j = 1, i + n\n", " while w[j:j + n] == gram:\n", " reps += 1; j += n\n", " if reps > max_phrase_run:\n", " out.extend(gram * max_phrase_run); i = j\n", " else:\n", " out.extend(gram); i += n\n", " w = out\n", " return \" \".join(w)\n", "\n", "def fold_alef(t): return _ALEF.sub(_PLAIN_ALEF, t)\n", "def squeeze_elong(t, keep=3): return _ELONG.sub(lambda m: m.group(1) * keep, t)\n", "def apply_learned(t): return \" \".join(LEARNED_MAP.get(w, w) for w in t.split())\n", "def CLEAN_NORM(t): return norm_ws(collapse_repeats(strip_junk(BEST_NORM(t))))\n", "def FINAL_NORM(t): return norm_ws(apply_learned(squeeze_elong(fold_alef(CLEAN_NORM(t)))))\n", "\n", "def max_word_run(text):\n", " w = text.split(); best = cur = 1\n", " for a, b in zip(w, w[1:]):\n", " cur = cur + 1 if a == b else 1; best = max(best, cur)\n", " return best if w else 0\n", "\n", "def type_token_ratio(text):\n", " w = text.split()\n", " return (len(set(w)) / len(w)) if w else 1.0\n", "\n", "# self-test: STOP here if this fails. This exact failure mode shipped once and\n", "# looked like a model problem for hours - it is a normalizer bug, not acoustics.\n", "_ANA = \"\\u0627\\u0646\\u0627\"\n", "assert BEST_NORM(_ANA) == _ANA, \"normalizer is deleting Arabic letters - check _DIAC\"\n", "assert fold_alef(\"\\u0623\") == _PLAIN_ALEF\n", "_t1 = \"###\" + _ANA + \" \" + \"\\u0647\" * 19 + \" [Speaker B ] \" + \"\\u0634\\u064a\\u0621\"\n", "assert FINAL_NORM(_t1) == _ANA + \" \\u0647\\u0647\\u0647 \\u0634\\u064a\", FINAL_NORM(_t1)\n", "assert FINAL_NORM(\"\") == \"\"\n", "print(\"text pipeline verified |\", FINAL_NORM(_t1))" ] }, { "cell_type": "markdown", "id": "c006", "metadata": {}, "source": [ "## 3. Dev references and the scoring harness\n", "\n", "Unchanged from the 0.48/0.45 pipeline. Reads only the `id` and `transcription` columns of the\n", "dev parquet shards (column pruning - no audio downloaded here). `DEV_SEL` is a fixed, seeded\n", "random subset matching the blind length distribution (<=40 ref words) - it is **the** selection\n", "metric for every decision the rest of this notebook makes, including which clips are excluded\n", "from fine-tuning in section 8." ] }, { "cell_type": "code", "id": "c007", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DEV_SHARDS = [\"dev-00000-of-00009-051641e73732d26d\", \"dev-00001-of-00009-f2505c9f0c41e739\",\n", " \"dev-00002-of-00009-574478cbf86a090c\", \"dev-00003-of-00009-f2d81e6bc3114920\",\n", " \"dev-00004-of-00009-797f0063ead966d6\", \"dev-00005-of-00009-c68f72797dac98b6\",\n", " \"dev-00006-of-00009-e6e6831129f93d1e\", \"dev-00007-of-00009-7fea04d4729de732\",\n", " \"dev-00008-of-00009-f28e4000cf5e2fa3\"]\n", "\n", "if os.path.exists(\"out/dev_ids_refs.json\"):\n", " _d = json.load(open(\"out/dev_ids_refs.json\", encoding=\"utf-8\"))\n", " dev_ids, dev_refs = _d[\"ids\"], _d[\"refs\"]\n", "else:\n", " _fs = fsspec.filesystem(\"http\", client_kwargs={\"headers\": {\"Authorization\": \"Bearer \" + _tok}})\n", " dev_ids, dev_refs = [], []\n", " for nm in DEV_SHARDS:\n", " url = f\"https://huggingface.co/datasets/{DEV_REPO}/resolve/main/data/{nm}.parquet\"\n", " t0 = time.time()\n", " with _fs.open(url, \"rb\", block_size=4 * 1024 * 1024) as fh:\n", " t = pq.ParquetFile(fh).read(columns=[\"id\", \"transcription\"])\n", " dev_ids += t.column(\"id\").to_pylist(); dev_refs += t.column(\"transcription\").to_pylist()\n", " print(f\" {nm[:12]} {t.num_rows} rows {time.time()-t0:.0f}s total {len(dev_ids)}\", flush=True)\n", " json.dump({\"ids\": dev_ids, \"refs\": dev_refs},\n", " open(\"out/dev_ids_refs.json\", \"w\", encoding=\"utf-8\"), ensure_ascii=False)\n", "\n", "DEV_N = len(dev_refs)\n", "DEV_REFS = [norm_ws(r) for r in dev_refs]\n", "_sh = [i for i in range(DEV_N) if len(DEV_REFS[i].split()) <= 40]\n", "_lo = [i for i in range(DEV_N) if len(DEV_REFS[i].split()) > 60]\n", "random.Random(0).shuffle(_sh)\n", "DEV_SEL = sorted(_sh[:DEV_EVAL_N]) # blind-like clips: THE selection metric\n", "DEV_LONG = [] if DEADLINE_MODE else sorted(_lo) # long clips: windowing diagnostic only, skipped under a deadline\n", "DEV_EVAL = sorted(set(DEV_SEL) | set(DEV_LONG))\n", "FT_POOL = sorted(set(range(DEV_N)) - set(DEV_SEL)) # everything section 8 is ALLOWED to train on\n", "print(f\"dev {DEV_N} clips | selection set {len(DEV_SEL)} | long diagnostic {len(DEV_LONG)} \"\n", " f\"| fine-tune pool (DEV_SEL excluded) {len(FT_POOL)}\")\n", "\n", "def score(hyp_by_index, idxs, label, norm=FINAL_NORM):\n", " # references stay in DATASET order; only the hypothesis is ever permuted. This is the\n", " # correct-by-construction shape of a scoring check: wer(ref[perm], hyp[perm]) is identical\n", " # for every perm, which is exactly what hid the id-sort ordering bug the first time.\n", " h = [norm(hyp_by_index.get(i, \"\")) for i in idxs]\n", " r = [DEV_REFS[i] for i in idxs]\n", " w, c = wer(r, h), cer(r, h)\n", " empty = sum(1 for x in h if not x.strip())\n", " print(f\" {label:50s} WER {w:.4f} CER {c:.4f}\" + (f\" [{empty} EMPTY]\" if empty else \"\"))\n", " raw_w = sum(len(str(hyp_by_index.get(i, \"\")).split()) for i in idxs)\n", " out_w = sum(len(x.split()) for x in h)\n", " assert empty < 0.25 * len(h), (\n", " f\"{label}: {empty}/{len(h)} hypotheses EMPTY after normalization - the normalizer is \"\n", " f\"deleting text, not a model problem.\")\n", " assert raw_w == 0 or out_w > 0.5 * raw_w, (\n", " f\"{label}: normalization dropped {raw_w - out_w} of {raw_w} words. Check _DIAC.\")\n", " assert w < 1.0, f\"{label}: WER >= 1 means the pairing is wrong, not the model\"\n", " return w, c\n", "print(\"scoring harness ready\")" ] }, { "cell_type": "markdown", "id": "c008", "metadata": {}, "source": [ "## 4. Model registry\n", "\n", "Self-reported WER is oddadmix's own balanced-test number (~2600 clips, undiacritized) - not this\n", "task's dev set, but the best apples-to-apples signal available before spending a GPU pass here.\n", "\n", "| key | model | family | params | self-reported WER |\n", "|---|---|---|---|---|\n", "| `cohere_base` | `CohereLabs/cohere-transcribe-arabic-07-2026` | Conformer | 2B | (your prior best single) |\n", "| `whisper_large_v2` | `oddadmix/whisper-large-v3-arabic-dialectal-v2` | Whisper | **1.54B** | **0.3199** - new, the largest model here |\n", "| `whisper_turbo_v2` | `oddadmix/whisper-large-v3-turbo-arabic-dialectal-v2` | Whisper | 809M | 0.3324 - upgrade of the `v1` (no `-v2`) checkpoint the 0.45 run used |\n", "| `cohere_dialect` | `oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2` | Conformer | 2B | reported worse overall than `cohere_base`, but different mistakes on Gulf/Iraqi/Moroccan |\n", "| `w2v2_xlsr` | `jonatasgrosman/wav2vec2-large-xlsr-53-arabic` | CTC | 300M | weakest single system, kept only for architectural diversity |\n", "| `qwen3_asr` *(opt-in)* | `oddadmix/qwen3-asr-1.7b-arabic-dialectal-v2` | LLM-decoder | 1.7B | 0.478 - weak alone, but a genuinely different failure mode |\n", "\n", "Not included: `nemotron-3.5-asr-arabic-dialectal-v2` (0.432, needs the separate NeMo toolkit\n", "rather than `transformers` - not worth the added fragility for a system that scores worse than\n", "both Whisper checkpoints anyway). `whisper_lv3` (`openai/whisper-large-v3`, zero-shot) is kept in\n", "the registry for reference but dropped from the default `ACTIVE` list - it was in the 0.45\n", "ensemble as a diversity vote, but both new Whisper dialectal checkpoints should already dominate\n", "it on dialectal Arabic." ] }, { "cell_type": "code", "id": "c009", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "MODELS = {\n", " \"cohere_base\": dict(id=\"CohereLabs/cohere-transcribe-arabic-07-2026\", family=\"cohere\"),\n", " \"whisper_large_v2\": dict(id=\"oddadmix/whisper-large-v3-arabic-dialectal-v2\", family=\"whisper\"),\n", " \"whisper_turbo_v2\": dict(id=\"oddadmix/whisper-large-v3-turbo-arabic-dialectal-v2\", family=\"whisper\"),\n", " \"cohere_dialect\": dict(id=\"oddadmix/cohere-transcribe-arabic-07-2026-dialectal-v2\", family=\"cohere\"),\n", " \"w2v2_xlsr\": dict(id=\"jonatasgrosman/wav2vec2-large-xlsr-53-arabic\", family=\"ctc\"),\n", " \"whisper_lv3\": dict(id=\"openai/whisper-large-v3\", family=\"whisper\"),\n", " \"qwen3_asr\": dict(id=\"oddadmix/qwen3-asr-1.7b-arabic-dialectal-v2\", family=\"qwen3\"),\n", "}\n", "\n", "if DEADLINE_MODE:\n", " assert DEADLINE_MODEL_KEY in MODELS, f\"{DEADLINE_MODEL_KEY!r} is not a MODELS key\"\n", " ACTIVE = [DEADLINE_MODEL_KEY]\n", "else:\n", " ACTIVE = [\"whisper_large_v2\"] if QUICK_MODE else \\\n", " [\"cohere_base\", \"whisper_large_v2\", \"whisper_turbo_v2\", \"cohere_dialect\", \"w2v2_xlsr\"]\n", " if not QUICK_MODE and ADD_QWEN3_ASR:\n", " ACTIVE.append(\"qwen3_asr\")\n", "print(\"active systems:\", ACTIVE)" ] }, { "cell_type": "markdown", "id": "c010", "metadata": {}, "source": [ "## 5. Audio preprocessing: silence trim + universal degeneracy guard\n", "\n", "Two changes from the 0.45 pipeline, both aimed at \"keyboard spam\" / repetition-loop clips rather\n", "than at the bulk WER:\n", "\n", "* **Silence trim.** `librosa.effects.trim(top_db=30)` removes leading/trailing near-silence\n", " before any model sees the audio. Silence padding is a documented trigger for decoder\n", " hallucination loops in both CTC and seq2seq ASR models - the model has nothing to condition on\n", " and falls back to repeating whatever it just said.\n", "* **Universal degeneracy check.** The 0.45 pipeline only re-decoded clips over 35s whose Cohere\n", " output had an exact 4+ word repeat. `is_degenerate()` below also flags clips whose output\n", " implies an implausible speaking rate (words/sec far outside the corpus distribution) or whose\n", " **type-token ratio** is too low for its length - this catches near-duplicate loops with minor\n", " variation (a word repeated with a slightly different spelling each time) that an exact-match\n", " repeat counter misses - and it runs for\n", " every model family, not just Cohere." ] }, { "cell_type": "code", "id": "c011", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import librosa\n", "\n", "def trim_silence(arr, sr=SR, top_db=30):\n", " if not TRIM_SILENCE or len(arr) < sr // 4:\n", " return arr\n", " trimmed, _ = librosa.effects.trim(np.asarray(arr, dtype=\"float32\"), top_db=top_db)\n", " return trimmed if len(trimmed) > sr // 4 else arr # never trim a clip to (near) nothing\n", "\n", "def _pre_collapse(t):\n", " return norm_ws(strip_junk(BEST_NORM(t)))\n", "\n", "def is_degenerate(text, dur_s, lo_wps=0.15, hi_wps=6.0, min_ttr_words=8, min_ttr=0.35):\n", " # Two independent signals. Word rate is measured on the PRE-collapse text: collapse_repeats\n", " # caps any exact repeat loop at 3 words, so measuring rate on the already-collapsed text would\n", " # hide exactly the pathology this is meant to catch. max_word_run / type-token-ratio are\n", " # measured on the final (post-collapse) text, to catch near-duplicate loops that vary slightly\n", " # each repeat (typo'd re-runs of the same phrase) and so never match collapse_repeats' exact\n", " # n-gram equality check.\n", " raw = _pre_collapse(text)\n", " if not raw.strip():\n", " return True\n", " if dur_s > 0.3:\n", " wps = len(raw.split()) / dur_s\n", " if wps < lo_wps or wps > hi_wps:\n", " return True\n", " t = FINAL_NORM(text)\n", " if max_word_run(t) >= 4:\n", " return True\n", " w = t.split()\n", " if len(w) >= min_ttr_words and type_token_ratio(t) < min_ttr:\n", " return True\n", " return False\n", "\n", "assert is_degenerate(\"\", 5.0) is True # empty\n", "assert is_degenerate(\"\\u0643\\u0644\\u0645\\u0629 \" * 30, 2.0) is True # 30 words in 2s: implausible rate\n", "assert is_degenerate(\"\\u0627\\u0646\\u0627 \\u0630\\u0647\\u0628\\u062a\", 3.0) is False # normal short utterance\n", "print(\"degeneracy guard ready\")" ] }, { "cell_type": "markdown", "id": "c012", "metadata": {}, "source": [ "## 6. Transcription engine\n", "\n", "Same long-audio windowing as the 0.48 fix (Cohere's processor silently returns more rows than it\n", "was given past `max_audio_clip_s`; this windows explicitly and stitches on the longest shared\n", "word n-gram, so nothing depends on the processor's row count matching the batch). The degeneracy\n", "re-decode now applies to every family via `is_degenerate()`, using a stricter greedy decode as\n", "the fallback." ] }, { "cell_type": "code", "id": "c013", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "GEN = dict(max_new_tokens=440, num_beams=5, repetition_penalty=1.1, no_repeat_ngram_size=8)\n", "GEN_STRICT = dict(max_new_tokens=440, num_beams=1, repetition_penalty=1.3, no_repeat_ngram_size=5)\n", "BATCH = 8\n", "_S = {}\n", "\n", "def unload():\n", " global _S\n", " _S = {}; gc.collect()\n", " if HAS_GPU: torch.cuda.empty_cache()\n", "\n", "def load_model(key, model_obj=None, processor_obj=None):\n", " # model_obj/processor_obj let section 8 hand in a fine-tuned model without a repo id.\n", " global _S\n", " unload()\n", " fam = MODELS[key][\"family\"]\n", " mid = MODELS[key][\"id\"]\n", " if model_obj is not None:\n", " if fam == \"whisper\":\n", " from transformers import pipeline\n", " _S = dict(key=key, fam=fam,\n", " pipe=pipeline(\"automatic-speech-recognition\", model=model_obj,\n", " tokenizer=processor_obj.tokenizer,\n", " feature_extractor=processor_obj.feature_extractor,\n", " device=0 if HAS_GPU else -1,\n", " torch_dtype=torch.float16, chunk_length_s=30, stride_length_s=5,\n", " batch_size=BATCH))\n", " print(f\"{key}: fine-tuned in-memory model | pipeline, 30s/5s chunking, batch_size={BATCH}\")\n", " else:\n", " raise NotImplementedError(\"in-memory loading only wired for the whisper family\")\n", " return _S\n", " if fam == \"cohere\":\n", " from transformers import AutoProcessor, CohereAsrForConditionalGeneration\n", " proc = AutoProcessor.from_pretrained(mid)\n", " mdl = CohereAsrForConditionalGeneration.from_pretrained(mid, device_map=\"auto\").eval()\n", " dt = next((p.dtype for p in mdl.parameters()\n", " if p.dtype in (torch.float16, torch.bfloat16)), torch.float32)\n", " fe = proc.feature_extractor\n", " _S = dict(key=key, fam=fam, proc=proc, mdl=mdl, dt=dt,\n", " max_s=float(getattr(fe, \"max_audio_clip_s\", 35.0)),\n", " ov_s=float(getattr(fe, \"overlap_chunk_second\", 5.0)))\n", " print(f\"{key}: {mid} | {dt} | max_clip {_S['max_s']}s overlap {_S['ov_s']}s\")\n", " elif fam == \"qwen3\":\n", " try:\n", " from transformers import AutoProcessor, AutoModelForMultimodalLM\n", " import soundfile as sf, tempfile\n", " proc = AutoProcessor.from_pretrained(mid, trust_remote_code=True)\n", " mdl = AutoModelForMultimodalLM.from_pretrained(mid, device_map=\"auto\",\n", " trust_remote_code=True).eval()\n", " _S = dict(key=key, fam=fam, proc=proc, mdl=mdl, sf=sf, tempfile=tempfile)\n", " print(f\"{key}: {mid} | loaded (experimental)\")\n", " except Exception as e:\n", " print(f\"{key}: SKIPPED - {type(e).__name__}: {e}. \"\n", " f\"This model needs a newer transformers than is installed; \"\n", " f\"ACTIVE will drop it automatically.\")\n", " _S = dict(key=key, fam=\"unavailable\")\n", " else:\n", " from transformers import pipeline\n", " _S = dict(key=key, fam=fam,\n", " pipe=pipeline(\"automatic-speech-recognition\", model=mid, device=0 if HAS_GPU else -1,\n", " torch_dtype=torch.float16 if HAS_GPU else torch.float32,\n", " chunk_length_s=30, stride_length_s=5, batch_size=BATCH))\n", " print(f\"{key}: {mid} | pipeline, 30s/5s chunking, batch_size={BATCH}\")\n", " return _S\n", "\n", "def _cast(d, dt):\n", " return {k: (v.to(dt) if (torch.is_tensor(v) and torch.is_floating_point(v)) else v)\n", " for k, v in d.items()}\n", "\n", "@torch.no_grad()\n", "def _cohere_decode(auds, gen):\n", " a = [np.asarray(x, dtype=\"float32\") for x in auds]\n", " try:\n", " inp = _S[\"proc\"](a, sampling_rate=SR, language=\"ar\",\n", " return_tensors=\"pt\", padding=True).to(_S[\"mdl\"].device)\n", " inp.pop(\"audio_chunk_index\", None)\n", " out = _S[\"mdl\"].generate(**_cast(inp, _S[\"dt\"]), **gen)\n", " txt = [t.strip() for t in _S[\"proc\"].batch_decode(out, skip_special_tokens=True)]\n", " if len(txt) != len(a):\n", " raise RuntimeError(\"row count mismatch - per-clip fallback\") # THE 0.48-fix bug\n", " return txt\n", " except Exception:\n", " res = []\n", " for x in a:\n", " inp = _S[\"proc\"](x, sampling_rate=SR, language=\"ar\",\n", " return_tensors=\"pt\").to(_S[\"mdl\"].device)\n", " inp.pop(\"audio_chunk_index\", None)\n", " o = _S[\"mdl\"].generate(**_cast(inp, _S[\"dt\"]), **gen)\n", " res.append(\" \".join(t.strip() for t in\n", " _S[\"proc\"].batch_decode(o, skip_special_tokens=True)).strip())\n", " return res\n", "\n", "def _stitch(a, b, max_overlap_words=25):\n", " aw, bw = a.split(), b.split()\n", " if not aw or not bw: return \" \".join(aw + bw)\n", " for k in range(min(max_overlap_words, len(aw), len(bw)), 1, -1):\n", " if aw[-k:] == bw[:k]:\n", " return \" \".join(aw + bw[k:])\n", " return \" \".join(aw + bw)\n", "\n", "def _cohere_long(arr, gen):\n", " win = int((_S[\"max_s\"] - 1.0) * SR)\n", " hop = int((_S[\"max_s\"] - 1.0 - _S[\"ov_s\"]) * SR)\n", " pieces = [arr[s:s + win] for s in range(0, max(len(arr) - win + hop, 1), hop)]\n", " pieces = [p for p in pieces if len(p) > int(0.4 * SR)]\n", " outs = []\n", " for k in range(0, len(pieces), 2):\n", " outs += _cohere_decode(pieces[k:k + 2], gen)\n", " text = outs[0] if outs else \"\"\n", " for nxt in outs[1:]:\n", " text = _stitch(text, nxt)\n", " return text\n", "\n", "def _qwen3_decode(arr, sr=SR):\n", " with _S[\"tempfile\"].NamedTemporaryFile(suffix=\".wav\") as f:\n", " _S[\"sf\"].write(f.name, np.asarray(arr, dtype=\"float32\"), sr)\n", " inputs = _S[\"proc\"].apply_transcription_request(\n", " audio=f.name, language=\"Arabic\").to(_S[\"mdl\"].device, _S[\"mdl\"].dtype)\n", " out_ids = _S[\"mdl\"].generate(**inputs, max_new_tokens=256)\n", " gen_ids = out_ids[:, inputs[\"input_ids\"].shape[1]:]\n", " return _S[\"proc\"].decode(gen_ids, return_format=\"transcription_only\")[0].strip()\n", "\n", "def transcribe(ds, indices, jsonl, tag=\"\"):\n", " # {dataset_index: raw_text}. Resume-safe: rerun after a disconnect and it continues.\n", " # Every clip is silence-trimmed, decoded, checked with is_degenerate(), and re-decoded with a\n", " # stricter greedy config if it looks degenerate - this now runs for every family, not just\n", " # long Cohere clips.\n", " if _S.get(\"fam\") == \"unavailable\":\n", " print(f\"{tag}: model unavailable, returning empty (ensemble/combine will skip it)\")\n", " return {}\n", " done = {}\n", " if os.path.exists(jsonl):\n", " for line in open(jsonl, encoding=\"utf-8\"):\n", " r = json.loads(line); done[r[\"i\"]] = r[\"raw\"]\n", " todo = [i for i in indices if i not in done]\n", " if not todo:\n", " print(f\"{tag}: all {len(indices)} cached\"); return done\n", " arrs = {i: trim_silence(ds[i][\"audio\"][\"array\"]) for i in todo}\n", " dur = {i: arrs[i].shape[0] / SR for i in todo}\n", " f = open(jsonl, \"a\", encoding=\"utf-8\")\n", " def _save(i, t):\n", " done[i] = t\n", " f.write(json.dumps({\"i\": i, \"raw\": t}, ensure_ascii=False) + \"\\n\")\n", " f.flush(); os.fsync(f.fileno())\n", "\n", " if _S[\"fam\"] == \"cohere\":\n", " mx = _S[\"max_s\"]\n", " long_i = [i for i in todo if dur[i] > mx]\n", " short_i = sorted([i for i in todo if dur[i] <= mx], key=lambda i: dur[i])\n", " print(f\"{tag}: {len(short_i)} short + {len(long_i)} long (windowed), {len(done)} cached\")\n", " for k in tqdm(range(0, len(short_i), BATCH), desc=f\"{tag} short\"):\n", " b = short_i[k:k + BATCH]\n", " for i, t in zip(b, _cohere_decode([arrs[i] for i in b], GEN)):\n", " _save(i, t)\n", " for i in tqdm(long_i, desc=f\"{tag} long\"):\n", " _save(i, _cohere_long(arrs[i], GEN))\n", " elif _S[\"fam\"] == \"qwen3\":\n", " for i in tqdm(todo, desc=tag):\n", " try:\n", " _save(i, _qwen3_decode(arrs[i]))\n", " except Exception as e:\n", " print(f\" {tag} row {i}: qwen3 decode failed ({type(e).__name__}), leaving empty\")\n", " _save(i, \"\")\n", " else:\n", " # batched, not one clip per pipe() call - a python-level for loop over single-item\n", " # calls never engages the pipeline's own batch_size, no matter how it was constructed.\n", " # This was the actual bottleneck on a live run: 5.63s/it single-clip on an A100/L4,\n", " # which is GPU-idle-between-calls time, not model compute time.\n", " kw = dict(generate_kwargs={\"language\": \"ar\", \"task\": \"transcribe\",\n", " \"repetition_penalty\": GEN[\"repetition_penalty\"],\n", " \"no_repeat_ngram_size\": GEN[\"no_repeat_ngram_size\"]}) \\\n", " if _S[\"fam\"] == \"whisper\" else {}\n", " for k in tqdm(range(0, len(todo), BATCH), desc=tag):\n", " b = todo[k:k + BATCH]\n", " batch_in = [{\"raw\": np.asarray(arrs[i], dtype=\"float32\"), \"sampling_rate\": SR} for i in b]\n", " outs = _S[\"pipe\"](batch_in, **kw)\n", " for i, out in zip(b, outs):\n", " _save(i, (out[\"text\"] or \"\").strip())\n", "\n", " bad = [i for i in indices if is_degenerate(done.get(i, \"\"), dur.get(i, 0.0))]\n", " if bad and _S[\"fam\"] != \"qwen3\":\n", " print(f\"{tag}: re-decoding {len(bad)} degenerate/implausible-rate clips with a stricter config\")\n", " for i in bad:\n", " arr = arrs.get(i, trim_silence(ds[i][\"audio\"][\"array\"]))\n", " d = dur.get(i, arr.shape[0] / SR)\n", " if _S[\"fam\"] == \"cohere\":\n", " alt = _cohere_long(arr, GEN_STRICT) if d > _S[\"max_s\"] else _cohere_decode([arr], GEN_STRICT)[0]\n", " else:\n", " kw2 = dict(generate_kwargs={\"language\": \"ar\", \"task\": \"transcribe\", \"num_beams\": 1,\n", " \"repetition_penalty\": GEN_STRICT[\"repetition_penalty\"],\n", " \"no_repeat_ngram_size\": GEN_STRICT[\"no_repeat_ngram_size\"]}) \\\n", " if _S[\"fam\"] == \"whisper\" else {}\n", " alt = (_S[\"pipe\"]({\"raw\": np.asarray(arr, dtype=\"float32\"), \"sampling_rate\": SR}, **kw2)[\"text\"] or \"\").strip()\n", " old, new = FINAL_NORM(done.get(i, \"\")), FINAL_NORM(alt)\n", " still_bad_new = is_degenerate(new, d)\n", " if new.strip() and (not old.strip() or not still_bad_new or max_word_run(new) < max_word_run(old)):\n", " _save(i, alt)\n", " f.close()\n", " return done\n", "print(\"transcription engine ready\")" ] }, { "cell_type": "markdown", "id": "c014", "metadata": {}, "source": [ "## 7. Run each active model on dev and score it" ] }, { "cell_type": "code", "id": "c015", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DEV_SYS, DEV_SCORE = {}, {}\n", "BEST_SINGLE = ACTIVE[0]\n", "if RUN_DEV:\n", " dev_ds = load_dataset(DEV_REPO, split=\"dev\").cast_column(\"audio\", Audio(sampling_rate=SR))\n", " assert len(dev_ds) == DEV_N and list(dev_ds[\"id\"]) == dev_ids, \"dev row order changed\"\n", " for key in list(ACTIVE):\n", " load_model(key)\n", " DEV_SYS[key] = transcribe(dev_ds, DEV_EVAL, f\"out/dev_{key}.jsonl\", tag=f\"dev/{key}\")\n", " if not DEV_SYS[key]:\n", " print(f\" dropping {key} from ACTIVE - it produced no output\")\n", " ACTIVE.remove(key)\n", " unload()\n", " print(\"\\n=== single systems on the dev selection set ===\")\n", " for key in ACTIVE:\n", " DEV_SCORE[key] = score(DEV_SYS[key], DEV_SEL, key + \" [blind-like]\")\n", " BEST_SINGLE = min(DEV_SCORE, key=lambda k: DEV_SCORE[k][0])\n", " print(f\"\\nbest single: {BEST_SINGLE} WER {DEV_SCORE[BEST_SINGLE][0]:.4f}\")\n", "\n", " if DEV_LONG:\n", " hw = sum(len(FINAL_NORM(DEV_SYS[BEST_SINGLE][i]).split()) for i in DEV_LONG)\n", " rw = sum(len(DEV_REFS[i].split()) for i in DEV_LONG)\n", " print(f\"long-clip length ratio {hw/rw:.2f} over {len(DEV_LONG)} clips (1.00 is the target)\")\n", "else:\n", " print(\"dev scoring skipped\")" ] }, { "cell_type": "markdown", "id": "c016", "metadata": {}, "source": [ "## 8. LoRA fine-tune the best Whisper system on dev\n", "\n", "This is the step the prior README called out and never did. It trains on `FT_POOL`\n", "(`DEV_N - len(DEV_SEL)` clips, capped to `FT_MAX_CLIPS`) and is scored on `DEV_SEL`, which is\n", "never in the training set - so the WER printed here is a genuine held-out number, directly\n", "comparable to the zero-shot score for the same system in section 7.\n", "\n", "Only a Whisper-family system is fine-tuned. `CohereAsrForConditionalGeneration`'s training\n", "inputs are not documented clearly enough here to write a training loop with confidence, and\n", "shipping a guessed training API for a 2B model is a good way to waste an hour of A100 time on\n", "silent garbage. Whisper's LoRA recipe (`peft`, `target_modules=[\"q_proj\", \"v_proj\"]`) is the\n", "one HF has published and it is used verbatim.\n", "\n", "If no Whisper-family system is in `ACTIVE`, or `DO_FINETUNE` is off, this section is a no-op." ] }, { "cell_type": "code", "id": "c017", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "FT_BASE_KEY = None\n", "if DO_FINETUNE:\n", " whisper_keys = [k for k in ACTIVE if MODELS[k][\"family\"] == \"whisper\"]\n", " if not whisper_keys:\n", " print(\"no whisper-family system in ACTIVE - nothing to fine-tune, skipping section 8\")\n", " else:\n", " FT_BASE_KEY = min(whisper_keys, key=lambda k: DEV_SCORE.get(k, (1.0, 1.0))[0])\n", " print(f\"fine-tuning base: {FT_BASE_KEY} ({MODELS[FT_BASE_KEY]['id']})\")" ] }, { "cell_type": "code", "id": "c018", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if DO_FINETUNE and FT_BASE_KEY:\n", " from transformers import (WhisperForConditionalGeneration, WhisperProcessor,\n", " Seq2SeqTrainer, Seq2SeqTrainingArguments)\n", " from peft import LoraConfig, get_peft_model\n", " import torch.utils.data as tud\n", "\n", " ft_mid = MODELS[FT_BASE_KEY][\"id\"]\n", " ft_train_idx = FT_POOL[:FT_MAX_CLIPS]\n", " print(f\"fine-tune set: {len(ft_train_idx)} clips (of {len(FT_POOL)} available, DEV_SEL excluded)\")\n", "\n", " processor = WhisperProcessor.from_pretrained(ft_mid, language=\"Arabic\", task=\"transcribe\")\n", " base_model = WhisperForConditionalGeneration.from_pretrained(ft_mid, device_map=\"auto\")\n", " base_model.generation_config.language = \"ar\"\n", " base_model.generation_config.task = \"transcribe\"\n", " base_model.generation_config.forced_decoder_ids = None\n", "\n", " lora_cfg = LoraConfig(r=32, lora_alpha=64, target_modules=[\"q_proj\", \"v_proj\"],\n", " lora_dropout=0.05)\n", " ft_model = get_peft_model(base_model, lora_cfg)\n", " ft_model.print_trainable_parameters()\n", "\n", " # Seq2SeqTrainer already batches training (per_device_train_batch_size below is real\n", " # batching - this is not the section-6 inference bug). The actual waste here was\n", " # recomputing silence-trim + feature extraction INSIDE __getitem__, i.e. every epoch\n", " # re-does CPU work that only needs to happen once per clip. Precompute once, in parallel\n", " # (numpy/librosa release the GIL enough for a thread pool to help, no multiprocessing\n", " # pickling headaches in a notebook), and the Dataset below just indexes into memory.\n", " from concurrent.futures import ThreadPoolExecutor\n", "\n", " def _extract(i):\n", " arr = trim_silence(dev_ds[i][\"audio\"][\"array\"])\n", " feats = processor.feature_extractor(arr, sampling_rate=SR).input_features[0]\n", " labels = processor.tokenizer(FINAL_NORM(DEV_REFS[i])).input_ids\n", " return {\"input_features\": feats, \"labels\": labels}\n", "\n", " with ThreadPoolExecutor(max_workers=8) as ex:\n", " ft_items = list(tqdm(ex.map(_extract, ft_train_idx), total=len(ft_train_idx),\n", " desc=\"pre-extracting fine-tune features (once, not per-epoch)\"))\n", "\n", " class DevClips(tud.Dataset):\n", " def __init__(self, items):\n", " self.items = items\n", " def __len__(self):\n", " return len(self.items)\n", " def __getitem__(self, i):\n", " return self.items[i]\n", "\n", " def collate(batch):\n", " feats = torch.tensor(np.array([b[\"input_features\"] for b in batch]), dtype=torch.float32)\n", " lab_ids = [torch.tensor(b[\"labels\"]) for b in batch]\n", " labels = torch.nn.utils.rnn.pad_sequence(lab_ids, batch_first=True,\n", " padding_value=processor.tokenizer.pad_token_id)\n", " labels = labels.masked_fill(labels == processor.tokenizer.pad_token_id, -100)\n", " return {\"input_features\": feats, \"labels\": labels}\n", "\n", " train_ds = DevClips(ft_items)\n", " # Whisper's log-mel input is always a fixed 30s window regardless of clip length, so batch\n", " # size scaling is predictable - bump per_device_train_batch_size here (16, 24...) if your\n", " # GPU has headroom; effective batch size is per_device_train_batch_size *\n", " # gradient_accumulation_steps. dataloader_num_workers/pin_memory overlap batch collation\n", " # with GPU compute, though most of the win here is the precompute above.\n", " args = Seq2SeqTrainingArguments(\n", " output_dir=\"out/whisper_lora\", per_device_train_batch_size=16,\n", " gradient_accumulation_steps=1, learning_rate=1e-3, warmup_steps=50,\n", " num_train_epochs=FT_EPOCHS, fp16=HAS_GPU, logging_steps=25, save_strategy=\"no\",\n", " dataloader_num_workers=2, dataloader_pin_memory=True,\n", " report_to=[], remove_unused_columns=False, predict_with_generate=False)\n", " trainer = Seq2SeqTrainer(model=ft_model, args=args, train_dataset=train_ds,\n", " data_collator=collate)\n", " trainer.train()\n", "\n", " ft_model.eval()\n", " FT_KEY = \"whisper_ft\"\n", " MODELS[FT_KEY] = dict(id=f\"{ft_mid}+lora(dev,{len(ft_train_idx)}clips)\", family=\"whisper\")\n", " load_model(FT_KEY, model_obj=ft_model, processor_obj=processor)\n", " DEV_SYS[FT_KEY] = transcribe(dev_ds, DEV_SEL, f\"out/dev_{FT_KEY}.jsonl\", tag=f\"dev/{FT_KEY}\")\n", " DEV_SCORE[FT_KEY] = score(DEV_SYS[FT_KEY], DEV_SEL, FT_KEY + \" [held-out, never trained on]\")\n", " base_wer = DEV_SCORE[FT_BASE_KEY][0]\n", " print(f\"\\n{FT_BASE_KEY} zero-shot WER {base_wer:.4f}\")\n", " print(f\"{FT_KEY} fine-tuned WER {DEV_SCORE[FT_KEY][0]:.4f}\")\n", " if DEV_SCORE[FT_KEY][0] < base_wer - 0.002:\n", " print(\"fine-tune improved the held-out score - adding it to ACTIVE\")\n", " ACTIVE.append(FT_KEY)\n", " unload()\n", " else:\n", " print(\"fine-tune did NOT clear noise-level improvement over the zero-shot model - \"\n", " \"not added to ACTIVE. Try more epochs/clips, or a lower learning rate.\")\n", " del MODELS[FT_KEY]\n", " unload()\n", "else:\n", " print(\"section 8 skipped (DO_FINETUNE is off, or no whisper-family system was active)\")" ] }, { "cell_type": "markdown", "id": "c019", "metadata": {}, "source": [ "## 9. Combination: MBR / ROVER / weighted ROVER / oracle ceiling\n", "\n", "Unchanged combiners from the 0.45 pipeline, inline-tested before they ever touch real output.\n", "\n", "**MBR** keeps the candidate closest to the others - conservative, output is always a real\n", "hypothesis. **ROVER** aligns every system to a pivot and majority-votes each slot, with\n", "deletions voting as an explicit option so a word only one system produced cannot win by default.\n", "**Weighted ROVER** weights each vote by `1/WER` measured in section 7/8. **Oracle** is the\n", "per-clip best candidate - not submittable, but if it sits close to the best single system, the\n", "systems are too correlated for any combiner to help and the lever to pull is fine-tuning or a\n", "genuinely different architecture, not more combiner tuning." ] }, { "cell_type": "code", "id": "c020", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def _ed(a, b):\n", " if a == b: return 0\n", " la, lb = len(a), len(b)\n", " if not la or not lb: return max(la, lb)\n", " prev = list(range(lb + 1))\n", " for i, ca in enumerate(a, 1):\n", " cur = [i]\n", " for j, cb in enumerate(b, 1):\n", " cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))\n", " prev = cur\n", " return prev[-1]\n", "\n", "def mbr_pick(cands):\n", " cands = [c for c in cands if c.strip()] or [\"\"]\n", " if len(cands) == 1: return cands[0]\n", " costs = [sum(_ed(c, o) / max(len(c), len(o), 1) for o in cands if o is not c) for c in cands]\n", " return cands[costs.index(min(costs))]\n", "\n", "def rover(cands, weights=None):\n", " toks = [c.split() for c in cands]\n", " if weights is None: weights = [1.0] * len(toks)\n", " live = [(t, w) for t, w in zip(toks, weights) if t]\n", " if not live: return \"\"\n", " if len(live) == 1: return \" \".join(live[0][0])\n", " piv = max(range(len(live)), key=lambda k: len(live[k][0]) * live[k][1])\n", " pt = live[piv][0]\n", " slots = [[(w, live[piv][1])] for w in pt]\n", " for k, (tk, wt) in enumerate(live):\n", " if k == piv: continue\n", " al = jiwer.process_words([\" \".join(pt)], [\" \".join(tk)]).alignments[0]\n", " covered = set()\n", " for ch in al:\n", " if ch.type in (\"equal\", \"substitute\"):\n", " for off, r in enumerate(range(ch.ref_start_idx, ch.ref_end_idx)):\n", " h = ch.hyp_start_idx + off\n", " if r < len(slots) and h < len(tk):\n", " slots[r].append((tk[h], wt)); covered.add(r)\n", " elif ch.type == \"delete\":\n", " for r in range(ch.ref_start_idx, ch.ref_end_idx):\n", " if r < len(slots): slots[r].append((None, wt)); covered.add(r)\n", " for r in range(len(slots)):\n", " if r not in covered: slots[r].append((None, wt))\n", " out = []\n", " for s in slots:\n", " tally = {}\n", " for w, wt in s: tally[w] = tally.get(w, 0.0) + wt\n", " best = max(tally.items(), key=lambda kv: (kv[1], kv[0] is not None))\n", " if best[0] is not None: out.append(best[0])\n", " return \" \".join(out)\n", "\n", "assert rover([\"a b c\", \"a b c\", \"a x c\"]) == \"a b c\"\n", "assert rover([\"a b c\", \"a c\", \"a b c\"]) == \"a b c\"\n", "assert rover([\"a b c\", \"a b z c\", \"a b c\"]) == \"a b c\"\n", "assert rover([\"a b c\", \"\", \"a b c\"]) == \"a b c\"\n", "assert mbr_pick([\"a b c\", \"a b c\", \"z z z\"]) == \"a b c\"\n", "print(\"combiners verified\")\n", "\n", "def combine(sysdict, idxs, method, keys, dev_wer=None):\n", " wts = [1.0 / max(dev_wer[k][0], 1e-3) for k in keys] if (method == \"wrover\" and dev_wer) else None\n", " out = {}\n", " for i in idxs:\n", " c = [FINAL_NORM(sysdict[k].get(i, \"\")) for k in keys]\n", " out[i] = mbr_pick(c) if method == \"mbr\" else rover(c, wts)\n", " return out" ] }, { "cell_type": "code", "id": "c021", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "WINNER, RESULTS = BEST_SINGLE, {}\n", "if RUN_DEV and len(ACTIVE) > 1:\n", " RESULTS = {k: DEV_SCORE[k][0] for k in ACTIVE}\n", " print(\"=== combinations on the dev selection set ===\")\n", " for m in (\"mbr\", \"rover\", \"wrover\"):\n", " RESULTS[m] = score(combine(DEV_SYS, DEV_SEL, m, ACTIVE, DEV_SCORE), DEV_SEL,\n", " f\"[{m}] all {len(ACTIVE)} systems\", norm=lambda t: t)[0]\n", " if len(ACTIVE) > 2:\n", " print()\n", " for drop in ACTIVE:\n", " keys = [k for k in ACTIVE if k != drop]\n", " RESULTS[\"mbr::\" + \",\".join(keys)] = score(\n", " combine(DEV_SYS, DEV_SEL, \"mbr\", keys, DEV_SCORE), DEV_SEL,\n", " f\"[mbr] without {drop}\", norm=lambda t: t)[0]\n", " orc = {i: min([FINAL_NORM(DEV_SYS[k].get(i, \"\")) for k in ACTIVE],\n", " key=lambda c: jiwer.process_words([DEV_REFS[i]], [c or \" \"]).wer)\n", " for i in DEV_SEL}\n", " print()\n", " score(orc, DEV_SEL, \"[oracle] best candidate per clip (CEILING, not submittable)\", norm=lambda t: t)\n", " WINNER = min(RESULTS, key=lambda k: RESULTS[k])\n", " print(f\"\\n>>> winner: {WINNER} WER {RESULTS[WINNER]:.4f}\"\n", " f\" (best single {BEST_SINGLE} {DEV_SCORE[BEST_SINGLE][0]:.4f})\")\n", " if RESULTS[WINNER] >= DEV_SCORE[BEST_SINGLE][0] - 0.002:\n", " WINNER = BEST_SINGLE\n", " print(\" gain is inside this subset's noise - falling back to the single model\")\n", "print(\"\\nsubmitting with:\", WINNER)" ] }, { "cell_type": "markdown", "id": "c022", "metadata": {}, "source": [ "## 10. Blind transcription and submission" ] }, { "cell_type": "code", "id": "c023", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_ds = load_dataset(TEST_REPO, split=\"test\").cast_column(\"audio\", Audio(sampling_rate=SR))\n", "test_ids = list(test_ds[\"id\"])\n", "assert len(test_ds) == N_TEST, f\"expected {N_TEST} clips, got {len(test_ds)}\"\n", "\n", "if WINNER in (\"mbr\", \"rover\", \"wrover\"): NEEDED, METHOD = ACTIVE, WINNER\n", "elif WINNER.startswith(\"mbr::\"): NEEDED, METHOD = WINNER[5:].split(\",\"), \"mbr\"\n", "else: NEEDED, METHOD = [WINNER], None\n", "print(\"systems needed:\", NEEDED, \"| method:\", METHOD or \"single\")\n", "\n", "TEST_SYS = {}\n", "for key in NEEDED:\n", " if key == \"whisper_ft\":\n", " load_model(key, model_obj=ft_model, processor_obj=processor)\n", " else:\n", " load_model(key)\n", " TEST_SYS[key] = transcribe(test_ds, list(range(N_TEST)), f\"out/test_{key}.jsonl\", tag=f\"blind/{key}\")\n", "unload()\n", "\n", "if METHOD:\n", " merged = combine(TEST_SYS, range(N_TEST), METHOD, NEEDED, DEV_SCORE)\n", " lines = [merged[i] for i in range(N_TEST)]\n", "else:\n", " lines = [FINAL_NORM(TEST_SYS[WINNER][i]) for i in range(N_TEST)]\n", "lines = [l if l.strip() else _FILLER for l in lines]" ] }, { "cell_type": "code", "id": "c024", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "_BUILT = set()\n", "def build(name, lines, note):\n", " txt, zp = f\"out/{name}.txt\", f\"out/{name}.zip\"\n", " _BUILT.add(f\"{name}.txt\"); _BUILT.add(f\"{name}.zip\")\n", " with open(txt, \"w\", encoding=\"utf-8\", newline=\"\\n\") as f:\n", " f.write(\"\\n\".join(lines) + \"\\n\")\n", " if os.path.exists(zp): os.remove(zp)\n", " with zipfile.ZipFile(zp, \"w\", zipfile.ZIP_DEFLATED) as z:\n", " z.write(txt, \"predictions.txt\") # exactly one file, at the zip root\n", " with zipfile.ZipFile(zp) as z:\n", " inside = [(i.filename, i.file_size) for i in z.infolist()]\n", " assert len(inside) == 1 and inside[0][0] == \"predictions.txt\" and inside[0][1] > 2000, inside\n", " assert len(lines) == N_TEST, f\"{len(lines)} != {N_TEST}\"\n", " assert all(l.strip() for l in lines), \"blank line\"\n", " assert all(l == l.strip() for l in lines), \"untrimmed line\"\n", " print(f\" {name:38s} lines={len(lines)} words={sum(len(l.split()) for l in lines):5d} \"\n", " f\"zip={os.path.getsize(zp)}B [{note}]\")\n", " return zp\n", "\n", "# line k is the transcript of DATASET row k - never sorted(ids). That was the 0.9-WER bug.\n", "FINAL = build(f\"predictions_{WINNER.replace(':', '_').replace(',', '-')}\", lines, \"UPLOAD THIS\")\n", "for key in TEST_SYS:\n", " build(f\"single_{key}\", [FINAL_NORM(TEST_SYS[key][i]) for i in range(N_TEST)], \"fallback\")\n", "\n", "durs = [test_ds[i][\"audio\"][\"array\"].shape[0] / SR for i in range(N_TEST)]\n", "_r = sorted(len(lines[i].split()) / durs[i] for i in range(N_TEST) if durs[i] <= 35)\n", "_med = _r[len(_r) // 2] if _r else 0.0\n", "print(f\"\\nlong-clip check (median {_med:.2f} w/s under 35 s):\")\n", "for i in [i for i in range(N_TEST) if durs[i] > 35]:\n", " w = len(lines[i].split())\n", " print(f\" row {i:3d} {durs[i]:5.1f}s -> {w:3d} words ({w/durs[i]:.2f} w/s)\"\n", " + (\" <-- STILL TRUNCATED\" if durs[i] > 0 and w / durs[i] < 0.5 * _med else \" ok\"))\n", "\n", "_txt = open(FINAL.replace(\".zip\", \".txt\"), encoding=\"utf-8\").read()\n", "print(f\"junk check: char runs 4+ {len(re.findall(r'(.)' + chr(92) + '1{3,}', _txt))} | \"\n", " f\"'#' {_txt.count('#')} | latin {len(re.findall('[A-Za-z]', _txt))} | \"\n", " f\"word-repeat runs>=4 {sum(1 for l in _txt.splitlines() if max_word_run(l) >= 4)}\")\n", "\n", "try:\n", " from google.colab import files; files.download(FINAL)\n", "except Exception:\n", " print(\"download manually:\", FINAL)\n", "\n", "print(\"\\nDONE. Upload\", FINAL, \"to Codabench.\")" ] }, { "cell_type": "markdown", "id": "c025", "metadata": {}, "source": [ "## 11. Push results back to HF" ] }, { "cell_type": "code", "id": "c026", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "if HF_REPO:\n", " for fn in sorted(os.listdir(\"out\")):\n", " p = f\"out/{fn}\"\n", " if fn.endswith((\".txt\", \".zip\")) and fn not in _BUILT:\n", " print(\"skipped stale\", fn); continue\n", " if os.path.isfile(p) and os.path.getsize(p) < 200 * 1024 * 1024:\n", " dest = f\"claude/cache/{fn}\" if fn.endswith(\".jsonl\") else f\"{PUSH_PREFIX}/{fn}\"\n", " api.upload_file(path_or_fileobj=p, path_in_repo=dest, repo_id=HF_REPO,\n", " commit_message=f\"improved e2e pipeline, winner={WINNER}\")\n", " print(\"pushed\", dest)\n", "else:\n", " print(\"HF_REPO is None - not pushing\")" ] }, { "cell_type": "markdown", "id": "c027", "metadata": {}, "source": [ "---\n", "## Reading the results\n", "\n", "* **Section 7 (`BEST_SINGLE`)** tells you whether the two new Whisper-v2 checkpoints actually\n", " beat `cohere_base` on this task's dev set - their self-reported numbers are from a different\n", " benchmark, so this is the first real test.\n", "* **Section 8** is the one number worth reading closely: fine-tuned WER on `DEV_SEL` vs the same\n", " system zero-shot in section 7, both computed on data the fine-tune never trained on. If it\n", " clears noise, it is added to the ensemble automatically; if not, the section says so and moves\n", " on rather than shipping an unproven checkpoint.\n", "* **Section 9's oracle** - if it sits far below the best single system, the ensemble still has\n", " room (try turning on `qwen3_asr` or `w2v2_xlsr` for more architectural diversity); if it sits\n", " close, stop tuning the ensemble and put more time into section 8 instead.\n", "\n", "## Two invariants, unchanged\n", "\n", "1. Line *k* of `predictions.txt` is dataset row *k*. Never `sorted(ids)`.\n", "2. An order check must permute **one side only** - `wer(ref[perm], hyp[perm])` is identical for\n", " every `perm`, which is exactly how the very first, badly-ordered submission passed its own\n", " check and scored 1.3757." ] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" }, "accelerator": "GPU" }, "nbformat": 4, "nbformat_minor": 5 }