IOTEverythin/voxi-duo-en-in

Voxi-Duo

A LoRA adapter for Moshi that makes it behave as a recruiter's voice agent speaking Indian English: it introduces itself, screens candidates, and schedules or reschedules interviews, while listening and speaking at the same time.

This is a research prototype, version 0.2. It is not ready for real callers. See Limitations.

What it is

Base model	kyutai/moshiko-pytorch-bf16, CC-BY-4.0
Method	LoRA, rank 64, on the temporal and depth transformers; audio codec frozen
Training data	197 synthetic two-channel calls, 3.30 hours
Training	250 steps, batch 8 x 70 s, learning rate 1e-5, one A100-80GB
Agent persona	"Ananya", one fixed voice
Language	Indian English

Results

Both models were played the same 12 held-out callers (voices and scripts not seen in training), first 60 seconds of each call.

Measure	Untouched Moshi	This adapter
Calls where it uses task vocabulary	33.3%	100.0%
Caller's details said back	0.0% of 26	19.2% of 26
Share of the call it speaks	18.3%	27.1%
Talks over the caller	14.9%	16.0%
Reply gap, median	2.00 s	0.70 s
Stops after being interrupted, median	1.25 s	0.55 s
Words in repetition loops	0.0%	0.0%

Held-out loss went from 2.25 to 1.64 over training.

Estimated speech quality (SQUIM PESQ, 1 to 4.5): 3.01 for this adapter, against 3.06 for the training voice after Moshi's codec, which is the ceiling, and 3.11 for untouched Moshi. Pitch movement: 3.02 semitones, against 3.11 in the training audio.

The callers in this test are recordings. They do not react to the model, so these numbers describe behaviour, not whether a call succeeds. A model that talks more overlaps a fixed recording more, so the talk-over figure is not a clean measure of turn-taking.

Limitations

• It invents facts. Moshi takes no written context. The adapter has no way to know a candidate's name, the role, or which interview slots are free, and it makes them up. Do not connect it to real callers or a real calendar.
• Synthetic training data. The calls were written from templates and spoken by a text-to-speech system. The model has not heard real callers.
• One domain. Recruitment calls only: screening, scheduling, rescheduling.
• Small evaluation. 12 calls, one run, one seed. No human listening study and no live conversations were part of the evaluation.
• Inherited from Moshi: audio passes through a 1.1 kbps codec, which limits how natural any voice can sound.

How the data was made

1. Call scripts from templates, with ground-truth details and timing events: backchannels, interruptions, self-corrections, thinking pauses, and waits while the agent checks a system. Each agent line carries a tone.
2. Speech from Chatterbox (MIT), which copies voice and accent from a short reference clip. Chatterbox marks its output with an inaudible watermark, so the training audio carries it.
3. Reference clips from GLOBE (CC0, from Mozilla Common Voice), filtered to speakers who reported an Indian accent. The agent's voice is copied from one such volunteer recording.
4. Transcripts from Whisper large-v3; calls whose audio strayed from the script were dropped.

Use

pip install moshi huggingface_hub
huggingface-cli download Joshuant/voxi-duo-en-in lora.safetensors config.json --local-dir ./adapter
python -m moshi.server --hf-repo kyutai/moshiko-pytorch-bf16 \
    --lora-weight ./adapter/lora.safetensors --config-path ./adapter/config.json

Then open http://localhost:8998 and talk to it.

Files

File	What
lora.safetensors	The adapter
config.json	Model config with the LoRA settings; the loader needs it
scores/	Evaluation numbers and training curves
samples/	Replies from this adapter: left channel model, right channel caller

License and credits

Released under CC-BY-4.0, the license of the base model.

• Moshi by Kyutai (CC-BY-4.0)
• moshi-finetune by Kyutai (Apache-2.0)
• Chatterbox by Resemble AI (MIT)
• GLOBE and Mozilla Common Voice (CC0)
• The synthetic-data approach follows PersonaPlex