Google's open-source neural synth is creating totally new sounds

NSynth Super, part of the Google Brain AI unit, uses machine learning to interpret music and come up with something completely different

13 Mar 2018

What do you get when you cross a sitar, a marimba, a dog and a car? Allow Google's neural synthesiser to play it to you, with hundreds of variations in timbre, pitch and tone. The company's new plaything, known as the NSynth Super, enables musicians to use machine learning to produce never-before-heard sounds and paw at the limits of computer-enhanced creativity.

The work is part of Magenta, a research project that sits under Google Brain – Google’s deep learning artificial intelligence unit – which explores the role of machine learning in creating art and music. The NSynth research was first detailed in May 2017 and now Google is open-sourcing the hardware specs and interface to let anyone hack together their own high-tech instrument. Google describes the Nysnth Super as an “open source experimental instrument”.

Previous forays into artificially intelligent music have generally involved generating melodies to supply missing parts: people have played duets with computers, or an orchestra’s missing instruments have had their parts filled in. The NSynth algorithm takes things further by letting musicians create totally new sounds. Using deep neural networks, it learns the characteristics of different instruments and is able to blend these parts to form new wholes.

The algorithm can also discern what makes each sound unique. “In principle we have two sounds – the sound of a snare and the sound of a bass. The algorithm creates all the sound that exists between, but it’s not just mixing them together – it actually understands the quality of the sounds, so in the case of the snare and the bass it will produce a sound in between, that will somehow have the attack of the snare, with the hit, but it also has the harmonics of the bass,” says João Wilbert, creative technology lead at Google’s Creative Lab in London. This ability to preserve defining qualities produces genuinely unusual sounds.

The original algorithm was trained on over 300,000 instrument sounds, making it by far the largest dataset of musical notes publicly available. Magenta built NSynth using TensorFlow – Google’s machine learning technology, which it opensourced back in 2015 – and all their models and tools are also open source and available on GitHub.

“How do we make it accessible without needing loads of code to understand it?” says team lead Peter Semple. In the past, the same team built Project Bloks, a hands-on coding activity which uses building blocks to harness kids’ tactility and get them building and sequencing technological structures.

Here, Wilbert and creative technologist Zebedee Pedersen wanted to build a tool that anyone could use. Both Wilbert and Pedersen make and play their own music, and designed the NSynth Super to be cheap to build – it can be constructed from a few sheets of perspex, some 3D printed knobs at each corner, and a Raspberry Pi.

Users can upload their own recorded ‘sound pack’ of 16 pre-processed source sounds and let the algorithm do its work as they drag their fingers across the screen to create new acoustics, somewhere within the matrix of the four source sounds at each corner, using the dials to define the sound space represented in the touchpad. This touchpad is in turn represented on an analogue screen with a map of dots that light up in response to your finger’s movements. It’s been designed to fit in with existing music-making methods, so you can plug in a keyboard, say, using a MIDI controller.

Pedersen says the algorithm lends sounds their own particular grainy qualities and that in their experience musicians enjoyed being able to mess around with the limitations as well as the new possibilities. Adding a strange new texture to a sound as classic as the piano, for instance, was something the NSynth opened up. In the future, say Wilbert and Pedersen, they hope to be able to build the technology to create in-between sounds in real time – but it’s not there yet, and would need to interpolate sounds at a very high speed.

Ultimately, the purpose of the project is to get humans and machines collaborating, not competing. "We don't want to create something that generates the music, the notes themselves, because that's actually what a musician does. We wanted to give them the complementary part,” says Wilbert.