A LoRA fine-tune of IndicF5 for improved Bhojpuri pronunciation and prosody, trained on the SYSPIN Bhojpuri corpus.
## What this is A rank-32 LoRA adapter (10.1M / 347M trainable params, 2.9%) fine-tuned on top of AI4Bharat's **IndicF5** (F5-TTS architecture) using **9.4 hours** of the SYSPIN Bhojpuri corpus (2 studio speakers, male + female). Zero-shot voice cloning: give it a short reference clip + its transcript, and target text, and it synthesizes that text in the reference voice/style. This is a personal, self-funded learning project — not a production system. Read the caveats below before using it for anything beyond experimentation. ## Does it actually help? Measured against stock IndicF5 on a 32-sentence held-out diagnostic set built specifically to probe Bhojpuri-vs-Hindi pronunciation contrasts (व→ब, श/ष→स, ण→न, avagraha, verb endings, retained final vowels), using pitch-contour correlation against real recordings of the same sentences (DTW-aligned): | Measure | Stock IndicF5 | This LoRA | Change | |---|---|---|---| | Pitch-contour correlation (↑ better) | 0.502 | **0.601** | +0.099 | | Pitch RMSE, semitones (↓ better) | 3.759 | **3.207** | −0.552 | | Pitch range ratio vs. real speech (→1.0 ideal) | 1.457 | **1.157** | closer to 1.0 | | Spectral (MFCC) distance (↓ better) | 49.355 | **47.079** | −2.276 | Improved on **all 8** pronunciation categories tested, biggest gains on `avagraha` (0.628→0.785) and `retroflex_n` (0.549→0.696).  *Left: training/validation loss over the 8,000-update run — note it plateaus hard after ~update 2,500-3,000; most of the gain happens early. Right: per-category pitch-correlation vs. stock on the 32-sentence diagnostic set.* **Read the numbers honestly, not as a guarantee:** - These are pitch/spectral *proxies* for melody and prosody, not a measure of correct pronunciation or intelligibility — no native-speaker listening evaluation has been done yet. - 32 diagnostic sentences, one seed, one real reference recording per sentence. - The training slice (9.4h) is on the smaller side for this kind of fine-tune — treat this as a promising checkpoint, not a finished model. - "Duration drift" in this project's own diagnostics is identical across all models (F5 sets output length from text) and isn't a model quality signal. ## Usage This is a LoRA adapter on IndicF5's custom F5-TTS architecture (not a `transformers` model), so it's loaded through this project's own code rather than `AutoModel`. Verified working end-to-end: ```bash git clone https://github.com/gentleman101/bhojpuri-f5-tts.git cd bhojpuri-f5-tts ./scripts/bootstrap.sh # sets up the venv, downloads IndicF5 base weights python scripts/pull_run.py lora_10h_r32 # downloads this checkpoint from this repo python scripts/infer.py \ --ref-audio your_reference.wav --ref-text "transcript of your reference clip" \ --text "अरे भाई, ई त बहुते बढ़िया बा!" \ --adapter runs/lora_10h_r32/checkpoints/step_0008000 \ --out out.wav ``` `--ref-audio` is a 3-10s clip of any voice you want to clone (any language/speaker — zero-shot), with `--ref-text` as its exact transcript. Full training/inference code, configs, and the diagnostic set: **[github.com/gentleman101/bhojpuri-f5-tts](https://github.com/gentleman101/bhojpuri-f5-tts)** ## Data & attribution Fine-tuned on the **SYSPIN_S1.0 Bhojpuri corpus**, © Indian Institute of Science (IISc) Bengaluru, SPIRE Lab, released under **CC-BY-4.0**. Please cite: > Abhayjeet et al., "SYSPIN_S1.0 Corpus — A TTS Corpus of 900+ hours in nine Indian Languages", 2025. Base model: [ai4bharat/IndicF5](https://huggingface.co/ai4bharat/IndicF5) (MIT license). ## License & ethical use - **Code** (training/inference scripts, this repo's tooling): MIT, see the GitHub repo. - **Model weights**: derived from CC-BY-4.0 data; released under CC-BY-4.0 with the attribution above. - **This model can reproduce the voice characteristics of the two SYSPIN speakers.** It is released for research/educational use consistent with SYSPIN's stated purpose (building open TTS technology for underserved Indian languages). Do not use it to impersonate these or any other individuals, to create deceptive or fraudulent audio, or to imply endorsement by the original speakers. Audio generated with this model is synthetic — disclose it as such wherever it's used. ## Training details - Base: IndicF5 (F5-TTS arch, ~330M params), LoRA rank 32, alpha 32, targeting attention + feedforward projections. - Data: `syspin_10h` split, 5,462 training clips / 100 validation clips. - 8,000 updates, batch ≈38,400 frames/update, AdamW, peak LR 1e-4 (800-update linear warmup, linear decay to 0), bf16, EMA decay 0.999. - Hardware: single A100 40GB, `torch.compile` enabled (~1.8x speedup over eager). - Final validation loss: 0.6173 (EMA weights).