# Subtask 1.2 Mixed ASR — why the blind test scored 1.3757 while dev scored 0.55

## Verdict

`predictions.txt` was written in **`id`-sorted order**. The Codabench scorer aligns line *k* of
`predictions.txt` with **dataset row *k*** — the order `load_dataset(...)` yields. Both the dev and the
blind test rows are shuffled with respect to `id`, so id-sorting scrambles the pairing: every hypothesis
was scored against some other clip's reference.

The model is fine. Nothing about the acoustics, the normalization, or the ensemble is at fault.

## Evidence

`scoring_result/detailed_scores.json` from the failed run:

```
word_errors 8968   word_total 6519   wer 1.3757
char_errors 36700  char_total 31938  cer 1.1491
```

8968 errors against a 6519-word reference, while the submitted hypotheses contain 6367 words — roughly
the same total length as the reference. Errors exceeding the reference length with matching total length
is the signature of misalignment, not of a bad model: it means hypotheses of one length were aligned
against references of an unrelated length.

Reproducing it on dev, where references are available. Taking the 3152 dev hypotheses that are already in
this repo (`submissions/asr_cohere_int8/transcripts_dev_FULL.jsonl`) and permuting **only the
hypotheses** into `id` order, leaving references in dataset order:

| pairing | WER | CER |
|---|---|---|
| hyp dataset order vs ref dataset order | **0.5911** | 0.3163 |
| hyp **id-sorted** vs ref dataset order | **1.3507** | 1.1423 |
| the actual blind-test score            | **1.3757** | 1.1491 |

1.3507 / 1.1423 versus 1.3757 / 1.1491 — the bug reproduces the observed score to within the difference
between the two sets. Nothing else produces a WER above 1.0 of that shape.

Confirming from the other direction: the dev-phase submission that scored **0.56** on the leaderboard was
written in dataset order (`submissions/asr_cohere_int8/final_submission/run_info.txt` →
`order: dataset (primary), id-sorted (backup)`), and the blind-test notebook
(`Subtask 2 Mixed ASR/Blind test/NADI2026_MixedASR_BlindTest_Fix.ipynb`, cell 17) switched to:

```python
order_id = sorted(range(n), key=lambda i: ids[i])   # <-- id order the scorer requires
```

That comment is the bug. `id`-sorting moves **0 of 3152** dev rows and 426 of 427 blind rows to a
different position, so the two orders share essentially nothing.

## Why the earlier check said "id order confirmed correct"

`NADI2026_MixedASR_Improve_Ensemble.ipynb` printed `FINAL id-sorted dev WER 58.44%` and concluded id
order was verified. It scored `hyp` in id order against `ref` in id order:

```python
order_id = sorted(range(n), key=lambda i: ids[i])
hyp_id = [BEST_NORM(done[i]) for i in order_id]
ref_id = [norm_ws(refs[i])   for i in order_id]
wer(ref_id, hyp_id)
```

WER is invariant under applying the same permutation to both sides. That test returns the identical
number for every possible ordering, so it can never fail. Confirmed in §2 of the fix notebook: it prints
0.5911 for both the correct pairing and the id-sorted-both-sides pairing.

**Rule:** an order check must permute one side only.

## Secondary issue: decoder repetition loops

Independent of the ordering, two blind-test clips collapsed into repetition loops — 320 and 62 identical
consecutive words. On a 6519-word reference those alone are roughly 380 spurious insertions, about 5.8
WER points. Nine lines also carried `###` artifacts (`#` is absent from the punctuation-stripping set),
and dev output contains `[Speaker B ]` tags whose brackets get stripped while the Latin text survives as
insertions.

Post-hoc cleanup, grid-searched on the full dev split with correct pairing:

| normalization | WER | CER |
|---|---|---|
| `BEST_NORM` (what was submitted) | 0.5911 | 0.3163 |
| + strip junk / speaker tags / Latin | 0.5900 | 0.3144 |
| + collapse repeat loops | 0.5877 | 0.3147 |
| both (`CLEAN_NORM`, max_run=3, max_phrase_run=3) | **0.5866** | **0.3123** |

Small on dev because only 14 of 3152 dev clips degenerate; larger on the blind test, where the two
catastrophic clips are 2 of 427.

## The fix

`claude/blindtest_orderfix/predictions_FIXED_dataset_order.zip` — 427 lines, dataset order,
`CLEAN_NORM`, rebuilt from the existing `submissions/asr_blindtest/transcripts_test_b5.jsonl` with no
re-inference, so the acoustics are byte-identical to the run that produced the 1.3757 file. Only the line
order and the loop cleanup differ.

Also included, in case the leaderboard allows more attempts:

| file | what it is |
|---|---|
| `predictions_FIXED_dataset_order.zip` | **upload this** — dataset order + cleanup |
| `variant_dataset_order_no_cleanup.zip` | dataset order, exact 0.56-dev normalization — isolates the cleanup's effect |
| `variant_id_sorted_BROKEN.zip` | byte-identical to the 1.3757 submission, kept as the control |

Expected result: **≈0.55–0.59 WER**, matching dev.

`claude/NADI2026_MixedASR_BlindTest_ORDERFIX_claude.ipynb` reproduces all of the above in Colab: §2 the
proof on dev (CPU), §3 the rebuild (CPU, ~30 s), §4 an optional GPU re-run with
`repetition_penalty=1.1`, `no_repeat_ngram_size=8` and a >5 words/second re-decode guard, §5 pushes
artifacts back here.

## Guard for next time

```python
assert wer(ref_in_dataset_order, hyp) < 1.0, "WER >= 1 means the pairing is wrong, not the model"
```

Any submission file written with `sorted(range(n), key=lambda i: ids[i])` is suspect. The dialect-ID and
intent submissions carry an id per row so the scorer aligns them by key, not by position — those are
safe. Only the plain-one-line-per-clip formats are exposed.