PairVC

Real-Time Voice Conversion from Paired Speech

Audio · Oct 2026 · In progress

Your voice into a target voice, live: a tiny causal network trained on a big teacher’s paired speech.

MCD 6.53 → 2.70 dB
20 ms chunks, one CPU core
Runs in the browser

Built withPyTorchONNXStreamlitJavaScriptKaggle

Overview

PairVC turns your voice into one of several target voices while you speak. It is a fully causal waveform-to-waveform network that converts audio 20 ms at a time on one CPU core.

It is trained on paired data: every source clip comes with the same speech already converted to each target voice by a big, slow, offline teacher. PairVC learns to reproduce the teacher’s output, so it gets the teacher’s voices at a fraction of the cost.

It continues the real-time voice conversion work I started in my MSc thesis (NatVC).

The question

Can a tiny streaming network learn a big, slow voice converter’s voices well enough to run live on a CPU?

Key features

Fully causal: 20 ms chunks, 60 ms latency, one CPU core
Teacher → student distillation on synthetic parallel speech
GAN training: multi-scale mel loss and HiFi-GAN-style discriminators
ONNX export, a live in-browser demo and a Streamlit app with a live microphone
Metrics: MCD, F0 error, voicing, DNSMOS

How it works

Paired data from a teacher
A big offline voice converter turns each source clip into every target voice; the pairs become the training set.
A tiny causal network
A causal convolutional encoder squeezes the audio 320× into 64-dim frames (one per 20 ms), and a mirrored decoder rebuilds the waveform in the target voice, conditioned through FiLM.
Live conversion
Exported to ONNX, the model streams 20 ms chunks from the microphone, in Python or directly in the browser.

Examples

The validation clip at every evaluation: source, PairVC at that step, the teacher, and the validation loss.

Log-mel spectrograms on a speaker never seen in training: before training, after training, and the teacher, for both target voices.

The Streamlit app converting a live microphone, 20 ms at a time, with its latency measured.

Tech stack

PyTorch
The model, GAN losses and distributed training
ONNX
Export and real-time inference, in Python and in the browser
Streamlit
Playground app with a live microphone
JavaScript
In-browser demo with ONNX Runtime Web and an AudioWorklet
Kaggle
GPU training notebook

What I learned

A mel-only warm-up before the GAN losses got the small model past the point where it learned no harmonics at all.
Objective metrics (MCD, F0 error, voicing, DNSMOS) say more than the loss: the timbre was learned before the pitch.

Limitations

The small CPU model has not learned the target pitch yet: the output is still breathy, and the full-size model on a GPU is the next step.
No pretrained voice converter is published, only the small demo model.

Related projects

JointVC — Disentangled Voice Conversion, Trained in One Run
Audio · Oct 2026 · In progress
Private for now: I’m still exploring this research, so the code isn’t published yet.
Ongoing research: a tiny real-time voice converter whose content encoder, speaker encoder and decoder learn together, and the open problem at its heart, disentanglement: keeping who speaks out of what is said.
Built with: PyTorchONNXHugging Face Transformerstorchaudio
Read more → JointVC
NatVC — Real-Time Neural Voice Conversion
Audio · Sep 2025 · Completed
Code is private (company / client work)
Any-to-any voice conversion that runs in real time under a live-application latency budget — my Centrale Lyon MSc thesis, carried out during my internship at SteelSeries.
Built with: PyTorchAccelerate (multi-GPU)StreamVCLLVCTensorBoard
Read more → NatVC