NatVC
Real-Time Neural Voice Conversion
Changing one person's voice into another's, live, with no noticeable delay
- +0.3 MOS over StreamVC
- Multi-GPU training
Built with
PyTorch
Accelerate (multi-GPU)StreamVCLLVC
TensorBoard
ONNX
Code is private (company / client work)
Overview
Voice conversion makes one speaker sound like another while keeping what they say. Doing it live is hard: every millisecond of processing adds delay the listener can hear. My MSc thesis at Centrale Lyon, carried out during my internship at SteelSeries, asked whether a StreamVC-based model could convert any voice to any other voice in real time, within the latency budget of a live application, without losing naturalness.
I built a PyTorch and ONNX voice-conversion system with end-to-end latency below 80 ms, trained it on multiple GPUs with Accelerate and a multi-scale mel loss, and added real-time multi-voice blending based on the LLVC architecture.
Through targeted fine-tuning and waveform post-processing, the system improved MOS (mean opinion score, a listener rating of naturalness) by 0.3 points over the StreamVC baseline. This is industry work, so the description stays high level and the code is proprietary.
The question
Can a StreamVC-based model perform any-to-any voice conversion in real time, under the latency budget of a live application, without sacrificing naturalness?
Key features
- Converts any input voice to a target voice in real time.
- Runs with end-to-end latency below 80 ms.
- Blends multiple target voices in real time, building on the LLVC architecture.
- Exports the model to ONNX for real-time inference.
- Trains across multiple GPUs with Accelerate, tracked in TensorBoard.
How it works
Start from StreamVC
A streaming voice-conversion architecture served as the baseline, chosen because it processes audio as it arrives.
Train at scale
Multi-GPU distributed training with Accelerate, using a multi-scale mel loss to compare generated and target audio at several resolutions.
Blend voices
Real-time multi-voice blending based on the LLVC architecture.
Refine quality
Targeted fine-tuning and waveform post-processing raised MOS by 0.3 points over the StreamVC baseline.
Deploy for real time
A PyTorch and ONNX system that keeps end-to-end latency below 80 ms.
Tech stack
Model development and training
Distributed training across GPUs
StreamVC
Streaming voice-conversion baseline architecture
LLVC
Basis for real-time multi-voice blending
Tracking training runs
Real-time inference export
What I learned
- Latency is a first-class design constraint that reshapes the whole architecture
- Loss design is what drives perceived naturalness, not model size alone
Limitations
- Industry work — implementation kept high-level, code is proprietary
Related projects
JointVC — Disentangled Voice Conversion, Trained in One Run
Private for now: I’m still exploring this research, so the code isn’t published yet.
Ongoing research: a tiny real-time voice converter whose content encoder, speaker encoder and decoder learn together, and the open problem at its heart, disentanglement: keeping who speaks out of what is said.
Built with:
PyTorchONNXHugging Face Transformerstorchaudio
PairVC — Real-Time Voice Conversion from Paired Speech
GitHub: PairVCLive demo: PairVC
A tiny causal voice converter that turns your voice into a target voice live, 20 ms at a time on one CPU core. It learns from a big offline teacher model, on paired speech.
Built with:
PyTorchONNX
Streamlit
JavaScript
Kaggle
