MusicGen
Transformer vs LSTM vs RNN for Music
Three neural networks compete to write folk-style tunes from text notation.
- 3 architectures
- 3 tokenizers
Built with
PyTorch
Keras
Streamlitmusic21FluidSynth + pydub
Poetry
Overview
I wanted a controlled comparison of the three classic sequence models (Transformer, LSTM, and RNN) on the same music data, rather than one heavily tuned model and a guess about why it worked.
MusicGen represents songs as ABC notation, a text format for sheet music, and trains each architecture to continue that text one token at a time. I implemented the Transformer, LSTM, and RNN from scratch in PyTorch; the LSTM and RNN also have versions built on PyTorch's own layers. I added three tokenizers (a character-level one, a byte-pair-encoding tokenizer I trained myself, and a Keras word tokenizer) so I could see how tokenization changes the results.
A Streamlit app ties it together. You pick a model, give it a few opening notes, and it generates tunes that are rendered to audio and can be downloaded as MP3.
The question
On the same music corpus, how do Transformer, LSTM, and RNN compare for symbolic music generation?
Key features
- Transformer, LSTM, and RNN generators trained on the same ABC-notation corpus, each with recorded training and validation loss and accuracy curves.
- From-scratch LSTM and RNN cells alongside versions built on PyTorch's own layers.
- Three interchangeable tokenizers (character-level, custom byte-pair encoding, and Keras), each with unit tests.
- Streamlit app with a model selector, a box for starting ABC notes, and sliders for length, temperature, and top-k.
- Generated ABC is converted to MIDI and then to audio, so each tune can be played in the browser, downloaded as an MP3, or viewed as its ABC source.
How it works
Songs as text
Each tune's ABC notation is wrapped in start and end tokens, and the tunes are joined into one long training text.
Tokenize
The text is split into tokens by one of three tokenizers: individual characters, merges learned with byte-pair encoding, or Keras word tokens.
Train next-token predictors
A decoder-only Transformer, an LSTM, and an RNN each learn to predict the next token, while training and validation loss and accuracy are tracked.
Sample new music
From the user's opening notes, the model samples tokens one at a time, controlled by temperature and top-k.
Render to audio
The generated ABC is parsed with music21, written to MIDI, synthesized to WAV with FluidSynth, and converted to MP3 for playback and download.
Examples
Tech stack
What I learned
- A controlled architecture comparison teaches more than one heavily-tuned model
- Tokenization strategy visibly shifts generation quality
Limitations
- Symbolic music only; no raw-audio synthesis