Exploring the Eerie World of AI-Generated Deepfake Music

January 20, 2021 - 8 minutes read

“New” music generated by OpenAI, an artificial intelligence (AI) development and research company, uses data from millions of songs to create tunes with existing voices and sounds. For example, a song by Frank Sinatra contains the lyrics “It’s Christmas time! It’s hot tub time!” It sounds like him, and the accompanying music sounds like his style, but it lurches between keys and sometimes sounds garbled.

OpenAI’s algorithm generates music from multiple genres and musical styles. Besides Sinatra, they’ve created deepfake music of Elvis, Katy Perry, Tupac, Celine Dion, Simon and Garfunkel, and more. The algorithm learned from 1.2 million songs scraped from the web, along with the lyrics and metadata like length, artist info, release date, and more.

AI’s Ear for Music

Based on the algorithm’s training, it can output music in the average length of a song. If you want to mimic a specific artist, just feed the algorithm songs from the artist and wait for a “new” song to come out of the other end.

Dr. Matthew Yee-King is an electronic musician and researcher at Goldsmiths, University of London. He says that the OpenAI algorithm is “really impressive” because it dissects songs into abstract units of music, categorized like a custom dictionary. It does so at “three different layers of time”, which results in the AI “[finding] and [piecing] together the relevant bits of the ‘dictionary’ to create something in musical space.”

But music lovers say that some of the music samples that the algorithm outputs are creepy. Many of the artists are dead and the AI makes them croon out non-sensical phrases or sentences. Some listeners even go as far as to describe the AI-generated songs as “the screams of the damned” and “SOUNDS F***ING DEMONIC”.

It turns out that the music industry is actually implementing more AI algorithms these days. Some are being used to fill in pitch or auto-tune a singer’s voice, while others are being used for more interesting applications, like bringing back a dead artist like Tupac. As these algorithms continue to diversify and expand, deepfake music is predicted to create legal, ethical, and political ramifications for the music industry.

Google’s Magenta Project is a machine learning application used as “a tool in the creative process”. It has released open-source APIs that allow artists to create human-AI co-authored songs and compose machine-generated sounds. Amper Music, a startup, generates custom AI-produced music for content requiring audio.

Legal Ramifications

We don’t need a crystal ball to predict that deepfakes are going to lead to ethical, legal, and intellectual property problems. Anyone could get their hands on deepfake software and create a jingle or tune “by” Snoop Dogg for their commercial or film. Do you have to pay Snoop Dogg for using his likeness and voice? Do you have to get his permission before and after generating the song? On the other hand, streaming services like Spotify and radio stations could add similar-sounding AI-generated music to playlists, without the need to pay anyone for royalties and copyright.

Legal departments from music agencies are already working to mitigate AI-generated deepfake music. In 2020, New York City-based Roc Nation filed DCMA (Digital Millennium Copyright Act) requests against a YouTuber to take down AI-generated music that mimics Jay-Z’s voice. The song used Jay-Z’s voice to rap Billy Joel and Shakespeare. According to the filing, “This content unlawfully uses an AI to impersonate our client’s voice.” This case is the first of its kind and only the beginning of what’s to come.

Regulations and Responsibility

The International Federation of the Phonographic Industry declined to speak about how artists and the integrity of their creations should be protected. Similarly, in the UK, the British Phonographic Industry company doesn’t want to speak out about how the music industry will deal with this new threat to original music. In the UK, there’s not really a basis for the legal protection of musicians.

According to Rupert Skellett, the head of Legal for Beggars Group, there are two copyrights for music in the UK: one for lyrics and music notation and one for the sound recording. “If someone hasn’t used the actual recording, you’d have no legal action against them in terms of copyright with regards to the sound recording,” he says. There might be an issue with sharing the recording with others, but Skellett says the U.S. would take more issue with impersonating famous people for commercial gain.

Limitless Potential

Many people, musicians and listeners alike, are enraptured by the creative potential and possibilities that AI brings into the music world. Yee-King even postulates, “If you’ve got a statistical model of millions of songs, you can ask the algorithm: what haven’t you seen? You can find that blank space, and then create something new.”

Mat Dryhurst is a podcaster and music artist who has spent the last several years researching and developing AI and similar technologies. He says AI is taking the art of sampling and running with it, creating “an extraordinary power and responsibility.”

Using deepfakes creates several different questions. What makes us respond to specific types of music? How can we teach AI to generate more logical, smoother, and catchier music? What makes a musical artist special?

One day, AI will reach a level of musical ability that rivals the world’s top artists. Yee-King envisions a world where AI can generate the perfect music for you based on your mood, environment, and current needs. Companies like Endel are already working on this type of on-demand AI-generated music.

Riffing Off Human Ingenuity

While AI sounds cool and fun to experiment with, we must not create a shift in the music industry that sends the message that we don’t want human artists to try new things or collaborate with others. We still need humans to participate in the creation and the creative side of music. After all, music is often created based on an artist’s emotions which then causes emotions in the listener.

Would you listen to AI-generated music? Have you tried the Endel app? Let us know in the comments below!