singing voice synthesis in the context of music technology...
TRANSCRIPT
![Page 1: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/1.jpg)
Singing voice synthesis in the context
of music technology research
Xavier Serra Music Technology Group
Universitat Pompeu Fabra, Barcelona http://mtg.upf.edu
![Page 2: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/2.jpg)
Musical Sound Synthesis
![Page 3: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/3.jpg)
Theremin Lev Termen, 1917
RCA theremin, 1929 Clara Rockmore “Vocalise” by Rachmaninoff
![Page 4: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/4.jpg)
Moog Synthesizer
Walter Carlos “Switched on Bach”, 1968
Moog synthesizer from the 60s
Robert Moog, 1964
![Page 5: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/5.jpg)
FM synthesis John Chowning, 1972
DX7 by Yamaha, 1983
![Page 6: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/6.jpg)
Waveguide synthesis Julius Smith, 1985
VL1 by Yamaha, 1994
![Page 7: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/7.jpg)
Sampling
Fairlight, 1980
![Page 8: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/8.jpg)
New musical interfaces
Reactable, 2005
![Page 9: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/9.jpg)
Singing Voice Synthesis
![Page 10: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/10.jpg)
Physical model from 50s
Max Mathews, 1965
Kelly, Lochbaum and Mathews, 1961 “Daisy”
![Page 11: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/11.jpg)
Linear Predictive Coding
Charles Dodge “The days ahead”, 1975
![Page 12: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/12.jpg)
FOF synthesis Xavier Rodet, 1977
Aria “Queen of the Night” from the Magic Flute by Mozart, 1979
![Page 13: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/13.jpg)
SinSy Keiichi Tokuda et al., 2009
![Page 14: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/14.jpg)
Vocaloid and Hatsune Miku
![Page 15: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/15.jpg)
Sample editing
Harmonic + Stochastic analysis
sines
residual
Timbre (Resonances, Excitation)
Stationary Note attack Note release Note to note Articulation Vibrato
Excitation + Resonance
model estimation
Database Creation
![Page 16: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/16.jpg)
residual
1/F
Flat residual excitation
Flat harmonic excitation
EpR spectral Amplitude
Filtering
Filtering
EpR spectral Phase
sines
residual
HpS synthesis
Synthesis
Demo from 2001
![Page 17: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/17.jpg)
Yamaha, 2004
![Page 18: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/18.jpg)
![Page 19: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/19.jpg)
Yamaha, 2006
![Page 20: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/20.jpg)
![Page 21: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/21.jpg)
![Page 22: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/22.jpg)
![Page 23: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/23.jpg)
![Page 24: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/24.jpg)
![Page 25: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/25.jpg)
![Page 26: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/26.jpg)
![Page 27: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/27.jpg)
Some challenges in singing voice research
![Page 28: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/28.jpg)
![Page 29: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/29.jpg)
Carnatic music
T. M. Krishna
![Page 30: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/30.jpg)
Beijing Opera
Zhao Xiu Jun
![Page 31: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/31.jpg)
Intonation
![Page 32: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/32.jpg)
Melodic Motives
Discovery Induction Extraction
Matching Retrieval
+ Characterization
Transform
N dimensions
![Page 33: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/33.jpg)
Lyrics and music
Chinese is a tonal language
ma ma1 ma2 ma3 ma4 妈 麻 ⻢马 骂 吗
“mum” “hemp” “horse” “to scold” “?”
Hypothesis: performers try to reflect tone contours in the melody in order to make lyrics understandable
![Page 34: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/34.jpg)
Conclusion
• Musical instruments <--> Voice • Singing voice synthesis <--> Speech synthesis • Timbre models --> Musical models
• Lessons learned from Hatsune Miku
![Page 35: Singing voice synthesis in the context of music technology ...ssw8.talp.cat/papers/ssw8_KN3_Serra.pdf · in the context of music technology research Xavier Serra Music Technology](https://reader030.vdocuments.site/reader030/viewer/2022040402/5e8067ce86abdb01760956aa/html5/thumbnails/35.jpg)
Thanks