Praveenrajus/tamil-spacing-restorer

Tamil Spacing & Diacritic Restorer (v3)

Restores word spacing and vowel/diacritic marks in unspaced Tamil text — including classical / Agasthiyar medical Tamil and palm-leaf manuscript (PLM) text — with a deterministic constrained decoder (no free generation, no hallucinated content).

What's new in v3 — source-routed Stage-2

v2.1 applied the diacritic corrector + period inserter to every input. Locked-test analysis showed this trades sources: it helps PLM a lot (32/58 examples improved, 0 harmed) but never helps Agasthiyar (0/52 helped, 5 harmed) and can touch modern text (2 harms) — which made the model feel worse than v1 on classical Agasthiyar input.

v3 adds a tiny source router (char 2–4-gram TF-IDF + logistic regression, 99.86% validation accuracy) that classifies each raw input as agasthiyar / plm / modern_csv and fires Stage-2 only on PLM-like text, where it is validated to help. Agasthiyar and modern inputs get the pure Stage-1 output — which matches or beats v1.

Result: v3 Pareto-dominates both v1 and v2.1 on the locked test — every source is as good or better than both previous versions.

Locked-test results (uCER %, lower is better / exact-match %, higher is better)

Source	v1	v2.1	v3
Agasthiyar uCER	0.715	0.730	0.708
Agasthiyar EM	84.13	80.95	84.13
PLM uCER	6.531	5.930	5.930
Modern uCER	0.046	0.050	0.049
Modern EM	97.16	96.86	97.01

• vs v1: Agasthiyar better, PLM much better (−0.60 uCER), modern within 0.003 uCER / 0.15 EM pts (well inside the ≤0.60% uCER, ≤1 EM pt acceptance bar).
• vs v2.1: better or equal on every source.

Also investigated for v3 and ruled out (frontier analysis)

• From-scratch transformer tagger (15M & 34M params, two seeds): plateaued far below v1 on Agasthiyar/modern (modern EM ~50% vs 98%). The dilated-conv inductive bias is decisively better for this char-level task at this data scale.
• From-scratch conv tagger with domain upsampling (two mixes): PLM beat v1 by 45% (4.0 vs 7.26 gCER on val) but the 519 unique domain examples overfit at 20–30× repetition and distorted the general segmenter — Agasthiyar 9× worse than v1, modern EM ~58%. v1's strength is its 65k-example modern training; small domain data cannot reproduce it from scratch.
• Period-position model (torch MLP word classifier): +0.006 recall but precision capped at 0.967 < 0.98 gate (16 no-period Agasthiyar verse references are a hard ceiling), and worse domain gCER than v2.1. Not shipped.
• Guarded wrong-word repair and generalized diacritic corrector: both negative (zero recovered spans / regressed every metric). Not shipped.

Usage

pip install torch python-Levenshtein scikit-learn huggingface_hub

from tamil_spacing import TamilSpacingRestorer

# Accept the (free) license terms on the model page first, then:
restorer = TamilSpacingRestorer.from_pretrained("Praveenrajus/tamil-spacing-restorer")

restorer.restore("கூகூபொடுவதுநலலவெலலநசக்கரை")  # default mode = "correct"
# 'கூகூ போடுவது நல்ல வெல்ல நசக்கரை'

Inference modes

• correct — default / recommended. Stage-1 tagger + source-routed Stage-2: the diacritic corrector + period inserter fire only on PLM-like input (router = char n-gram classifier). This is the single best mode and the only one the live demo uses: it beats v1 on classical Tamil and preserves modern Tamil, with no hallucinated content.
• fast — Stage-1 constrained tagger only (fastest; no corrector, no period inserter).
• rerank — Stage-1 n-best + lexicon/LM reranker (experimental). Kept for API users; not the default — on the locked test it degrades modern Tamil (exact-match 96.86% → 59%) and raises unsupported insertions (5 → 58).

restorer.restore(text)                  # "correct" (default, routed)
restorer.restore(text, mode="fast")     # Stage-1 only
restorer.restore(text, mode="rerank")   # experimental; can degrade modern Tamil

Loading previous versions

v1 and v2/v2.1 are preserved and always loadable via revisions:

TamilSpacingRestorer.from_pretrained("Praveenrajus/tamil-spacing-restorer", revision="v1")
TamilSpacingRestorer.from_pretrained("Praveenrajus/tamil-spacing-restorer", revision="v2")
TamilSpacingRestorer.from_pretrained("Praveenrajus/tamil-spacing-restorer", revision="v2.1")

Method

Stage-1 (tagger): HF-v1 weights, gently retrained with an L2-to-init anchor + EMA + PLM-heavy source mix, then weight-space interpolated back toward HF v1 at α=0.22 (seed-44 balanced run). Dilated char-level CNN (12 layers, d=384) with a constrained grapheme decoder — deterministic, cannot emit invalid Tamil.

Stage-2 (diacritic corrector): constrained short→long compound-vowel repair, lexicon + grapheme-LM (order 6) grounded, calibrated (delta=0.1, min_freq=2) on held-out validation. Fires only when the router classifies input as PLM.

Stage-2c (period inserter): grounded insertion of sentence-ending . from train-safe, domain-only per-word period-follow statistics (thresh=0.9, min_count=3). Same PLM-only routing.

Source router (v3): char 2–4-gram TF-IDF (60k features) + logistic regression, trained on train-split raws only (never val/test), 99.86% validation accuracy. source_router.pkl. If the router file is absent, correct mode falls back to v2.1 behavior (Stage-2 on all inputs).

Files

• model.pt — Stage-1 interpolated weights (dilated CNN)
• config.json — model config
• source_router.pkl — v3 source router (TF-IDF + logistic regression)
• domain_lexicon.json — train-safe domain lexicon (274,394 words)
• domain_grapheme_lm.json — grapheme n-gram LM (order 6)
• diacritic_corrector.json — Stage-2 config (delta, min_freq)
• period_stats.json — Stage-2c domain period-follow statistics (train-safe)
• period_inserter.json — Stage-2c config (thresh, min_count)
• tamil_spacing/ — self-contained inference package (codec, model, reranker, restorer, diacritic_corrector, period_inserter, router)
• examples/ — CPU/GPU inference examples
• test_inference.py — smoke tests

Live demo

Space: https://huggingface.co/spaces/Praveenrajus/tamil-spacing-restorer-demo

License

Proprietary — Copyright © Praveen Raj U S. Do not redistribute. Gated access (auto-approved) on the model page.