JointVC
Disentangled Voice Conversion, Trained in One Run
Teaching one network what is said and another who says it, at the same time, without letting them mix.
- Ongoing research
- Trained in one run
- No target audio
Built with
PyTorch
ONNX
Hugging Face Transformerstorchaudio
Private for now: I’m still exploring this research, so the code isn’t published yet.
Overview
JointVC is the research I am working on now, and I am still discovering it. It is a tiny streaming any-to-any voice converter in the StreamVC family: a content encoder hears what is said, a speaker encoder hears who says it, and a decoder rebuilds the speech in any voice, 20 ms at a time.
The concept at its heart is disentanglement: the content must carry the words but no trace of the speaker, and the speaker embedding must carry the voice but not the words. StreamVC gets there by training in stages. My idea is to train all three networks together, in one run, with no target audio, keep them apart with explicit leakage defences, and measure the leak instead of claiming it is gone.
This project is private for now: I am still exploring it and see potential in it, so the code stays closed until the open questions are answered.
The question
Can a voice converter learn what is said and who says it in one joint training run, with no target audio, and still keep the two apart?
Key features
- Content, speaker and decoder trained together from step 0, with no target audio
- Leakage defences: gradient reversal, a LEACE eraser, leakage-gradient projection and a unit bottleneck
- A harmonic (DDSP) decoder branch, with pitch and loudness taken from the reference
- Independent probes that try to recover the speaker from the content, at every validation
- MOS, WER, PESQ and STOI evaluation
Tech stack
The three networks, the leakage defences and the training loop
Streaming export
Pretrained speech models used as judges and optional content targets
Audio loading and features
What I learned
- Disentanglement has to be measured with independent attacks, never assumed: statistics that read “independent” can still be broken by a simple probe.
Limitations
- Research in progress: the defences that work on toy data do not yet transfer to real speech, and the timbre of the converted voice is still off.
Related projects
PairVC — Real-Time Voice Conversion from Paired Speech
GitHub: PairVCLive demo: PairVC
A tiny causal voice converter that turns your voice into a target voice live, 20 ms at a time on one CPU core. It learns from a big offline teacher model, on paired speech.
Built with:
PyTorchONNX
Streamlit
JavaScript
Kaggle
NatVC — Real-Time Neural Voice Conversion
Code is private (company / client work)
Any-to-any voice conversion that runs in real time under a live-application latency budget — my Centrale Lyon MSc thesis, carried out during my internship at SteelSeries.
Built with:
PyTorchAccelerate (multi-GPU)StreamVCLLVC
TensorBoard
