BHĀVA · voice and face, together
A talking AI persona for Indian languages. Its speech and its facial motion are generated by one joint transformer, in the same denoising loop — not a voice made first and a face animated to it afterwards.
Hindi · English · Hinglish · Kannada · Tamil · Marathi · Telugu 466 M-parameter joint model runs on one 12 GB laptop GPU AI-generated personasDemo recorded live turns — every reply generated on the laptop
One photo, any language
Officer Priya answering in Kannada.
A second persona, male voice, answering in Hindi.
16 live turns, 2 personas — every reply in the user's language, including switching mid-conversation.
How it works
A Thinker LLM (Gemma 4, llama.cpp) writes the reply; the joint MM-DiT generates the mel spectrogram and 69 face-motion numbers per frame together; Vocos makes the waveform, LivePortrait the 3D face, LatentSync redraws the mouth from the voice.
In each of 22 layers the face stream's queries attend over its own keys and the voice's — one clock (RoPE) for sound and face.
Voice and face emerge from noise — together
The model's prediction at each of 20 denoising steps: spectrogram (voice), motion codes and the rendered face.
Real attention: layer 0 locks onto whole syllables, middle layers onto the exact instant; the face spends up to 68 % of its attention on the voice (layer 11).
The face in 3D
One photo → the 3D face driven by the joint model → lips redrawn from the voice.
21 learned 3D keypoints move the face.
Which keypoint owns which region; dense 3D flow; occlusion.
67,522 frames of real people talking; the orange path is one generated sentence.
Measured
| Metric | What it checks | Result |
|---|---|---|
| Speech CER (Whisper large-v3-turbo round trip) | intelligibility | 0.032 mean on 16 live turns, 7 languages |
| SyncNet confidence / offset | lips match the voice | 7.59 mean, offset 0 on all 16 (a real speaker: 5.83) |
| Lip stage ablation (same clips) | why a pixel mouth | 3D render only 3.4–4.3 → MuseTalk 5.3–5.8 → LatentSync 9.1–10.0 |
| Audio artefacts | end-of-clip bursts | 4/48 at 12 steps → 0/48 at 20 steps |
| Latency | first video on screen | median ≈ 20 s (not yet real time) |
Responsible use
Both faces are AI-generated images; no real person's likeness is used. The two voice prompts are 7-second clips of volunteer speakers from Google's FLEURS corpus (CC-BY 4.0) — for deployment a consented voice actor would be recorded. Every output is labelled AI-generated. The system runs fully offline, so a citizen's conversation never leaves the device. Known limits: the mouth is redrawn by a separate stage after the joint model, and replies are not yet real time.