![Page 1: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/1.jpg)
Julius Smith Music 421 Applications Lecture – 1 / 50
Historical Overview of Audio Spectral Modeling
Julius Smith
CCRMA, Stanford University
Music 421 Applications Lecture
January 7, 2014
![Page 2: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/2.jpg)
Milestones in Audio Spectral Modeling
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 2 / 50
• Fourier’s theorem (1822)
• Telharmonium (1898)
• Voder (1920s)
• Vocoder (1920s)
• Hammond Organ (1930s)
• Phase Vocoder (1966)
• Digital Organ (1968)
• Additive Synthesis (1969)
• FM Brass Synthesis (1970)
• Synclavier 8-bit FM/Additive synthesizer (1975)
• FM singing voice (1978)
• Sinusoidal Modeling (1985)
• Sines + Noise (1988)
• Sines + Noise + Transients (1988,1996,1998,2000)
• Inverse FFT synthesis (1992)
• Future Directions
![Page 3: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/3.jpg)
Telharmonium (1898)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 3 / 50
![Page 4: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/4.jpg)
Telharmonium (Cahill 1898)
Julius Smith Music 421 Applications Lecture – 4 / 50
U.S. patent 580,035:
“Art of and Apparatus for Generating and Distributing Music Electrically”
![Page 5: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/5.jpg)
Telharmonium Rheotomes
Julius Smith Music 421 Applications Lecture – 5 / 50
Forerunner of the Hammond Organ Tone Wheels
![Page 6: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/6.jpg)
Telharmonium Rotor: Forerunner of Hammond Organ Tone
Wheels
Outline
Telharmonium
• Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 6 / 50
![Page 7: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/7.jpg)
The Voder (1939)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 7 / 50
![Page 8: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/8.jpg)
The Voder (Homer Dudley — 1939 Worlds Fair)
Julius Smith Music 421 Applications Lecture – 8 / 50
http://davidszondy.com/future/robot/voder.htm[<
![Page 9: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/9.jpg)
Voder Keyboard
Julius Smith Music 421 Applications Lecture – 9 / 50
http://www.acoustics.hut.fi/publications/files/theses/
lemmetty mst/chap2.html — (from Klatt 1987)
![Page 10: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/10.jpg)
Voder Schematic
Julius Smith Music 421 Applications Lecture – 10 / 50
http://ptolemy.eecs.berkeley.edu/~eal/audio/voder.html
![Page 11: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/11.jpg)
Voder Demos
Outline
Telharmonium
Voder
• Voder Keyboard
• Voder Schematic
• Voder Demos
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 11 / 50
• Video
• Audio
![Page 12: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/12.jpg)
The Channel Vocoder (1928)
(“Voice Coder”)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 12 / 50
![Page 13: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/13.jpg)
Vocoder Analysis & Resynthesis (Dudley 1928)
Outline
Telharmonium
Voder
Channel Vocoder
• Vocoder Examples
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 13 / 50
Analysis:
• Ten analog bandpass filters between 250 and 3000 Hz:
Bandpass → rectifier → lowpass filter → amplitude envelope
• Voiced/Unvoiced decision made
• Fundamental frequency F0 measured for voiced case
Synthesis:
• Ten matching bandpass filters driven by a
“buzz source” (voiced), or
“hiss source” (unvoiced)
• Bands were scaled by amplitude envelopes and summed
• Said to have an “unpleasant electrical accent”
Related Speech Models:
• The Vocoder is an early source-filter model for speech
• Linear Predictive Coding (LPC) of speech is another
![Page 14: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/14.jpg)
Vocoder Filter Bank Analysis/Resynthesis
Outline
Telharmonium
Voder
Channel Vocoder
• Vocoder Examples
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 14 / 50
f
A
f
A
f
A
Analysis Synthesis
Data Compression,Transmission,Storage,Manipulation,Noise reduction, ...
Processing
magnitude, ormagnitude andphase extraction
x(t)
x0(t)
x1(t)
xN−1
x(t)
x0(t)
x1(t)
xN−1
![Page 15: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/15.jpg)
Channel Vocoder Sound Examples
Outline
Telharmonium
Voder
Channel Vocoder
• Vocoder Examples
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 15 / 50
• Original
• 10 channels, sine carriers
• 10 channels, narrowband-noise carriers
• 26 channels, sine carriers
• 26 channels, narrowband-noise carriers
• 26 channels, narrowband-noise carriers, channels reversed
• Phase Vocoder: Identity system in absence of modifications
• The FFT Phase Vocoder next transitioned to the Short-Time
Fourier Transform (STFT) (Allen and Rabiner 1977)
![Page 16: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/16.jpg)
The Phase Vocoder (1966)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 16 / 50
![Page 17: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/17.jpg)
Phase Vocoder Analysis for Additive Synthesis (1976)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 17 / 50
ωk ω0
Channel FilterResponse
ak
∆ωk
Analysis Model Synthesis Model
ak(t) ωk+∆ωk(t)
A F
Out
Sine Osc
• Early “channel vocoder” implementations (hardware) only
measured amplitude ak(t) (Dudley 1939)
• The “phase vocoder” (Flanagan and Golden 1966) added phase
tracking in each channel
• Portnoff (1976) developed the FFT phase vocoder,
which replaced the heterodyne comb in computer-music
additive-synthesis analysis (James A. Moorer)
• Inverse FFT synthesis (Rodet and Depalle 1992) gave faster
sinusoidal oscillator banks
![Page 18: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/18.jpg)
Amplitude and Frequency Envelopes
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 18 / 50
ak(t)
∆ωk(t) = φk(t)
0
t
t
![Page 19: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/19.jpg)
Additive Synthesis (1969)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 19 / 50
![Page 20: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/20.jpg)
Classic Additive-Synthesis Analysis (Heterodyne Comb)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
• Additive Analysis
• Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 20 / 50
John Grey 1975 — CCRMA Tech. Reports 1 & 2
(CCRMA “STANM” reports — available online)
![Page 21: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/21.jpg)
Classic Additive-Synthesis (Sinusoidal Oscillator Envelopes)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
• Additive Analysis
• Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 21 / 50
John Grey 1975 — CCRMA Tech. Reports 1 & 2
(CCRMA “STANM” reports — available online)
![Page 22: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/22.jpg)
Classic Additive Synthesis Diagram (Computer Music, 1960s)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
• Additive Analysis
• Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 22 / 50
noise
FIR
A1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)
Σ
y(t) =4
∑
i=1
Ai(t) sin
[∫
t
0
ωi(t)dt+ φi(0)
]
![Page 23: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/23.jpg)
Classic Additive-Synthesis Examples
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
• Additive Analysis
• Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 23 / 50
• Bb Clarinet
• Eb Clarinet
• Oboe
• Bassoon
• Tenor Saxophone
• Trumpet
• English Horn
• French Horn
• Flute
• All of the above
• Independently synthesized set
(Synthesized from original John Grey data)
![Page 24: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/24.jpg)
Frequency Modulation Synthesis
(1973)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 24 / 50
![Page 25: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/25.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 26: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/26.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 27: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/27.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 28: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/28.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 29: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/29.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 30: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/30.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 31: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/31.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 32: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/32.jpg)
Frequency Modulation (FM) Synthesis
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 25 / 50
FM synthesis is normally used as a spectral modeling technique
• Discovered and developed (1970s) by John M. Chowning
(CCRMA Founding Director)
• Key paper: JAES 1973 (vol. 21, no. 7)
• Commercialized by Yamaha Corporation:
DX-7 synthesizer (1983)
OPL chipset (SoundBlaster PC sound card)
Cell phone ring tones
• On the physical modeling front, synthesis of vibrating-string
waveforms using finite differences started around this time:
Hiller & Ruiz, JAES 1971 (vol. 19, no. 6)
![Page 33: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/33.jpg)
FM Formula
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 26 / 50
x(t) = Ac sin[ωct+ φc +Am sin(ωmt+ φm)]
where
(Ac, ωc, φc) specify the carrier sinusoid
(Am, ωm, φm) specify the modulator sinusoid
Can also be called phase modulation
![Page 34: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/34.jpg)
Simple FM “Brass” Patch (Chowning 1970–)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 27 / 50
Jean-Claude Risset observation (1964–1969):
Brass bandwidth ∝ amplitude
gA F
FA
Out
fc = f0
fm = f0
![Page 35: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/35.jpg)
FM Harmonic Amplitudes (Bessel Function of First Kind)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 28 / 50
Harmonic number k, FM index β:
0
2
4
6
8
10
05
1015
2025
30
−0.5
0
0.5
1
Order k
Argument β
J k(β)
![Page 36: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/36.jpg)
Frequency Modulation (FM) Examples
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 29 / 50
All examples by John Chowning unless otherwise noted:
• FM brass synthesis
• Low Brass example
• Dexter Morril’s FM Trumpet
• FM singing voice (1978)
Each formant synthesized using an FM operator pair
(two sinusoidal oscillators)
• Chorus
• Voices
• Basso Profundo
• Other early FM synthesis
• Clicks and Drums
• Big Bell
• String Canon
![Page 37: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/37.jpg)
FM Voice
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
• FM Synthesis
• FM Formula
• FM Patch
• FM Spectra
• FM Examples
• FM Voice
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 30 / 50
FM voice synthesis can be viewed as compressed modeling of
spectral formants
Modulation Frequency (all three)
Frequency f0
Magnitude
Carrier 1 Carrier 2
Carrier 3
![Page 38: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/38.jpg)
Sinusoidal Modeling Synthesis
(1988)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 31 / 50
![Page 39: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/39.jpg)
Tracking Spectral Peaks in the Short-Time Fourier Transform
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 32 / 50
dB magPeaktracking
QuadraticPeakInterpolation
Frequencies
Amplitudes
Phases
window w(n)
s(t)
atan
FFT
• STFT peak tracking at CCRMA: mid-1980s (PARSHL program)
• Motivated by vocoder analysis of piano tones
• Influences: STFT (Allen and Rabiner 1977),
ADEC (1977), MAPLE (1979)
• Independently developed for speech coding by McAulay and
Quatieri at Lincoln Labs (1985)
![Page 40: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/40.jpg)
Example Spectral Trajectories
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 33 / 50
f
t
![Page 41: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/41.jpg)
Parametric Spectral Modeling
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 34 / 50
A1(t) A2(t) A3(t) A4(t)f1(t) f2(t) f3(t) f4(t)
∑white noiseu(t) filter(t)
ht(τ)
y(t) =4∑
i=1
Ai(t) cos[
∫
t
0ωi(t)dt+ φi(0)
]
+ (ht ∗ u)(t)
![Page 42: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/42.jpg)
Sines + Noise Sound Examples
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 35 / 50
Xavier Serra thesis demos (Sines + Noise signal modeling)
• Piano
Original
Sinusoids alone
Residual after sinusoids removed
Sines + noise model
• Voice
Original
Sinusoids
Residual
Synthesis
![Page 43: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/43.jpg)
Musical Effects with Sines+Noise Models
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 36 / 50
• Piano Effects
Pitch downshift one octave
Pitch flattened
Varying partial stretching
• Voice Effects
Frequency-scale by 0.6
Frequency-scale by 0.4 and stretch partials
Variable time-scaling, deterministic to stochastic
![Page 44: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/44.jpg)
Cross-Synthesis with Sines+Noise Models
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 37 / 50
• Voice “modulator”
• Creaking ship’s mast “carrier”
• Voice-modulated creaking mast
• Same with modified spectral envelopes
![Page 45: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/45.jpg)
Sines + Transients Sound Examples
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 38 / 50
In this technique, the sinusoidal sum is phase-matched at the
cross-over point only (with no cross-fade).
• Marimba
Original
Sinusoidal model
Original attack, followed by sinusoidal model
• Piano
Original
Sinusoidal model
Original attack, followed by sinusoidal model
![Page 46: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/46.jpg)
Multiresolution Sines + Noise + Transients
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 39 / 50
Why Model Transients Separately?
• Sinusoids efficiently model spectral peaks over time
• Filtered noise efficiently models spectral residual vs. t
• Neither is good for abrupt transients in the waveform
• Phase-matched oscillators are expensive
• More efficient to switch to a transient model during transients
• Need sinusoidal phase matching at the switching times
Transient models:
• Original waveform slice (1988)
• Wavelet expansion (Ali 1996)
• MPEG-2 AAC (with short window) (Levine 1998)
• Frequency-domain LPC
(time-domain amplitude envelope) (Verma 2000)
![Page 47: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/47.jpg)
Time Scale Modification of Sines + Noise + Transients Models
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 40 / 50
transien
ts
transien
ts
transien
ts
transien
ts
sines +noise
sines +noisesines +noise
sines +noise
sines +noise
sines +noise
sines +noise
sines +noise
sines +noise
original signal
time-scaledsignal
time
Time-Scale Modification (TSM) becomes well defined :
• Transients are translated in time
• Sinusoidal envelopes are scaled in time
• Noise-filter envelopes also scaled in time
• Dual of TSM is frequency scaling
![Page 48: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/48.jpg)
Sines + Noise + Transients Time-Frequency Map
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 41 / 50
0 50 100 150 200 250−1
−0.5
0
0.5
1
time [milliseconds]
ampl
itude
0 50 100 150 200 2500
2
4
6
8
10
12
14
16
freq
uenc
y [k
Hz]
(Levine 1998)
![Page 49: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/49.jpg)
Corresponding Analysis Windows
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 42 / 50
0 50 100 150 200 250time [milliseconds]
ampl
itude
tran
sien
thi
gh o
ctav
em
iddl
e oc
tave
low
oct
ave
![Page 50: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/50.jpg)
Quasi-Constant-Q (Wavelet) Time-Frequency Map
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 43 / 50
0 50 100 150 200 250
−2
−1
0
1
2
x 104
time [milliseconds]
ampl
itude
0 50 100 150 200 2500
2
4
6
8
10
12
14
16
freq
uenc
y [k
Hz]
![Page 51: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/51.jpg)
Bark-Band Noise Modeling at High Frequencies (Levine 1998)
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 44 / 50
0 500 1000 1500 2000 2500 3000 3500 4000 450030
35
40
45
50
55
60
65
70
75
80
85
frequency [Hz]
mag
nitu
de [d
B]
![Page 52: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/52.jpg)
Amplitude Envelope for One Noise Band
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 45 / 50
0 50 100 150 200 250 300 35040
50
60
70
80
Orig
inal
Mag
. [dB
]
0 50 100 150 200 250 300 35045
50
55
60
65
70
75
80
time [milliseconds]
LSA
Mag
. [dB
]
![Page 53: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/53.jpg)
Sines + Noise + Transients Sound Examples
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 46 / 50
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
(http://ccrma.stanford.edu/~scottl/thesis.html)
“It Takes Two” by Rob Base & DJ E-Z Rock
• Original
• MPEG-AAC at 32 kbps
• Sines+transients+noise at 32 kbps
• Multiresolution sinusoids
• Residual Bark-band noise
• Transform-coded transients (AAC)
• Bark-band noise above 5 kHz
![Page 54: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/54.jpg)
Time Scale Modification using Sines + Noise + Transients
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
• Sinusoidal Modeling
• Spectral Trajectories
• Sines + Noise
• S+N Examples
• S+N FX
• S+N XSynth
• Sines + Transients
• S + N + Transients
• S+N+T TSM
• S+N+T Freq Map
• S+N+T Windows
• HF Noise Modeling
• HF Noise Band
• S+N+T Examples
Future
Julius Smith Music 421 Applications Lecture – 47 / 50
Scott Levine Thesis Demos (Sines + Noise + Transients at 32 kbps)
(http://ccrma.stanford.edu/~scottl/thesis.html)
Time-Scale Modification (pitch unchanged)
• S+N+T time-scale factors [2.0, 1.6, 1.2, 1.0, 0.8, 0.6, 0.5]
S+N+T Pitch Shifting (timing unchanged)
• Pitch-scale factors [0.89, 0.94, 1.00, 1.06, 1.12]
![Page 55: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/55.jpg)
Summary and Future Prospects
Outline
Telharmonium
Voder
Channel Vocoder
Phase Vocoder
Additive Synthesis
FM Synthesis
Sinusoidal Modeling
Future
Julius Smith Music 421 Applications Lecture – 48 / 50
![Page 56: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/56.jpg)
Spectral Modeling History Highlights
Julius Smith Music 421 Applications Lecture – 49 / 50
• Bernoulli’s modal sums (1733)
• Fourier’s initial theorem (1822)
• Telharmonium (1906)
• Hammond organ (1930s)
• Channel Vocoder (1939)
• Phase Vocoder (1966)
• “Additive Synthesis” (1969)
• FFT Phase Vocoder (1976)
• Sinusoidal Modeling
(1977,1979,1985)
• Sines+Noise (1989)
• Sines+Transients (1989)
• TF Reassignment (1995)
• Sines+Noise+Transients
(1998)
Perceptual audio coding:
• Princen-Bradley filterbank
(1986)
• K. Brandenburg thesis (1989)
• Auditory masking usage
• Dolby AC2
• Musicam
• ASPEC
• MPEG-I,II,IV
(S+N+T “parametric sounds”)
![Page 57: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/57.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 58: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/58.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 59: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/59.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 60: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/60.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 61: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/61.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 62: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/62.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?
![Page 63: Historical Overview of Audio Spectral Modeling - CCRMAjos/pdf/SMS.pdf · Historical Overview of Audio Spectral Modeling Julius Smith ... • Digital Organ ... Forerunner of Hammond](https://reader031.vdocuments.site/reader031/viewer/2022020412/5b1ec0fb7f8b9a22028bf4b0/html5/thumbnails/63.jpg)
Future Prospects
Julius Smith Music 421 Applications Lecture – 50 / 50
Observations:
• Sinusoidal modeling of sound is “Unreasonably Effective”
• Basic “auditory masking” discards ≈ 90% information
• Interesting neuroscience observation:
“... most neurons in the primary auditory cortex A1 are silent
most of the time ...”
(from “Sparse Time-Frequency Representations”, Gardner and Magnesco,
PNAS:103(16), April 2006)
• What is the right “psychospectral model” for sound?
The cochlea of the ear is a real-time spectrum analyzer
How is the “ear’s spectrogram” represented at higher levels?