music/audio analysis in python...building an auto-dj. goal. • create mix-ins and mix-outs that...
TRANSCRIPT
![Page 1: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/1.jpg)
Vivek Jayaram
MUSIC/AUDIO ANALYSIS IN PYTHON
![Page 2: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/2.jpg)
WHY AUDIO SIGNAL PROCESSING? • My background as a DJ and CS student
• Music is everywhere! So many possibilities
• Many parallels to computer vision
![Page 3: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/3.jpg)
SOME APPLICATIONS• Shazam - How does it recognize songs?
![Page 4: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/4.jpg)
SOME APPLICATIONS• Speech to Text – Siri, Android
![Page 5: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/5.jpg)
OTHER APPLICATIONS• Classify a song into genre • Find interesting segments of songs • Recommendations • Generate Audio Automatically
![Page 6: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/6.jpg)
OVERVIEW• Basics of Audio • Sampling and Representation • Fourier Transformations • Building an Auto-DJ Software • Finding Interesting Segments of Songs
![Page 7: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/7.jpg)
Waves and Frequencies
BASICS OF AUDIO
![Page 8: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/8.jpg)
BASICS OF SOUND• Most basic type of sound is a sine wave
![Page 9: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/9.jpg)
BASICS OF SOUND• Frequency determines pitch, amplitude determines volume
• Doubling the frequency creates octave (same note)
• “Nice” ratios generally make nice intervals
![Page 10: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/10.jpg)
BASICS OF SOUND• Can combine sine waves to make intervals (Perfect Fifth Below)
![Page 11: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/11.jpg)
WHAT MAKES A SOUND DISTINCT?• If a piano and guitar playing A are both 440, why do they sound different?
• Each has different amount of overtones
• Frequencies at 440hz, 880hz, 1320hz, 1760hz …
• Ratio of each determines timbre
![Page 12: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/12.jpg)
WHAT MAKES A SOUND DISTINCT?
![Page 13: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/13.jpg)
WHAT MAKES A SOUND DISTINCT• Add all those sine waves together
![Page 14: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/14.jpg)
Sampling and Representation
HOW IS AUDIO STORED IN COMPUTERS?
![Page 15: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/15.jpg)
SAMPLING• Sound in the real world is a continuous wave • Computers are discrete. Need to sample
![Page 16: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/16.jpg)
SAMPLING • Music is just array of heights sampled at regular intervals • Music normally sampled at 44khz • Space vs quality tradeoff • Issues with high frequencies
![Page 17: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/17.jpg)
SAMPLING
![Page 18: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/18.jpg)
FOURIER TRANSFORMS
![Page 19: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/19.jpg)
MOTIVATIONS• Array of numbers doesn’t tell us much about audio • Want a more representative feature • Frequency is everything • Can we get frequencies?
![Page 20: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/20.jpg)
FOURIER TRANSFORM
![Page 21: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/21.jpg)
FOURIER TRANSFORM
![Page 22: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/22.jpg)
FOURIER TRANSFORMS• Decompose any wave into sine frequencies
• Theory is outside scope
• Height is amplitude of that frequency
![Page 23: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/23.jpg)
FOURIER TRANSFORMS IN PYTHON• FT woks on continuous, infinitely long waves
• Alternative calculates discrete, short time TF
• Take small section of audio (.1 sec), calculate frequencies
![Page 24: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/24.jpg)
LIBROSA• Don’t reinvent the wheel! • Can get frequencies in two lines of code • y, sr = librosa.load(“song.mp3”)
• D = librosa.stft(y)
![Page 25: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/25.jpg)
![Page 26: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/26.jpg)
GET MUSICAL PITCHES• Frequencies are nice but can we do more? • ~440 is an A, so is 880 etc. • “Bin” frequencies at different octaves to get amount of each note • Get 12 x num_samples array
• y, sr = librosa.load(“song.mp3”)• S = np.abs(librosa.stft(y)**2) # Get magnitude of stft• chroma = librosa.feature.chroma_stft(S=s, sr=sr)
![Page 27: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/27.jpg)
![Page 28: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/28.jpg)
BUILDING AN AUTO-DJ
![Page 29: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/29.jpg)
GOAL• Create mix-ins and mix-outs that sound good
• Manually select mix-in and mix-out point for many songs of equal lengths
• Try to figure out which songs mash well with each other
• Create the mix and output result
![Page 30: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/30.jpg)
MASHABILITY BASED ON FREQUENCIES• Want to see how well two songs sound while played over each other
• Compute chromagram for each
• See how similar they are on a frame by frame basis
![Page 31: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/31.jpg)
![Page 32: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/32.jpg)
RESULT• Find best mixes • Synchronize beats using librosa • Output result using EchoNest for python • Code at github.com/vivjay30/AutoDJ/
![Page 33: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/33.jpg)
Project with Google
FINDING INTERESTING PARTS OF SONGS
![Page 34: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/34.jpg)
GOAL• Wanted 10 second clips for a “Guess the Song Game”
• Random selection won’t suffice
• Need those clips to be interesting/recognizable parts of the song
• Idea: Look for 10s clip that repeats itself the most number of times
![Page 35: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/35.jpg)
CHROMAGRAMS• Chromagrams used and worked excellently
• Robust against changes in instrumentation
• Chorus in different octave still looks exactly the same
• Distills piece down to notes
![Page 36: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/36.jpg)
TIME-TIME SIMILARITY MATRIX• Chromagram is long array, compare each sample to every other sample
• Point (x,y) represents how similar time x and y are
• Example shown for “Scream and Shout”
![Page 37: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/37.jpg)
![Page 38: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/38.jpg)
![Page 39: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/39.jpg)
FINDING SEGMENTS• Repeated segments show up as diagonal lines
• Look for these diagonal lines and group together to find most repeated segment
• Unfortunately can’t play samples because code belongs to Google
![Page 40: MUSIC/AUDIO ANALYSIS IN PYTHON...BUILDING AN AUTO-DJ. GOAL. • Create mix-ins and mix-outs that sound good • Manually select mix-in and mix-out point for many songs of equal lengths](https://reader036.vdocuments.site/reader036/viewer/2022081621/612ec96e1ecc515869430891/html5/thumbnails/40.jpg)
KEY TAKEAWAYS• Don’t reinvent the wheel. Libraries exist for everything
• Frequencies are important and are an accurate representation of music
• Not always important to understand theory to use application