automatic and data driven pitch contour manipulation with functional data analysis
DESCRIPTION
Automatic and Data Driven Pitch Contour Manipulation with Functional Data Analysis. Michele Gubian, Lou Boves Radboud University Nijmegen Nijmegen, The Netherlands Francesco Cangemi Laboratoire Parole et Langage University of Provence, Aix-en-Provence, France. Outline. - PowerPoint PPT PresentationTRANSCRIPT
Automatic and Data DrivenPitch Contour Manipulationwith Functional Data Analysis
Michele Gubian, Lou BovesRadboud University NijmegenNijmegen, The Netherlands
Francesco CangemiLaboratoire Parole et LangageUniversity of Provence, Aix-en-Provence, France
2
Outline Pitch Contour Manipulation
Context and problem
Sketch of proposed approach
Use of Functional Data Analysis (FDA)
Case study
Data preparation
Functional PCA
Functional synthesis and listening
Conclusions
3
Context Languages can express oppositions using intonation
Question/Statement opposition in Neapolitan Italian
QUESTION STATEMENT
“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)
What are the intonation cues that listeners use?
Perceptual experiments where listeners judge stimuli whose pitch (F0) contour has been manipulated
STEP 1: extract pitch contours from speech data
STEP 2: modify pitch contours
STEP 3: re-synthesize speech
4
Pitch Contour Manipulation
Use of an intonation model
Stylization
Manual changestime
F0
POSSIBLE IMPROVEMENTS Handle dynamic detail
Locally (e.g. concavity/convexity)
Long range correlation
Derive useful variation modes directly and automatically from data
5
A data driven approach
Functional
Data
Analysisx
6
Question/Statement opposition in Neapolitan Italian
DATA 2 male speakers
3 carrier sentences (read speech)
“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)
“Valeria viene alle nove (?)” = Valeria arrives at 9 (?)
“Amelia dorme da nonna (?)” = Amelia sleeps at grandma’s (?)
2 modalities = Q / S
5 repetitions
2 x 3 x 2 x 5 - 3 discarded = 57 utterances
7
Data Preparation
Sampled F0 curves have to be turned into functions
A basis of functions (B-splines) expresses each original curve
Decide how much detail to retain (smoothing)
8
Data Preparation (2) Landmark registration
Align points in time that are deemed as having the same
meaning across the dataset
9
ClassicPrincipal Component Analysis (PCA)
age25 65
salary
xx
xxx
x
xx
xxx
xx xx xxxx
xxxx
xx x
xxx
xx
x
xx
x
x
PC1
PC2
10
Functional PCA
11
PC-based signal reconstruction
+ 1.65 x - 0.46 x
mean(t) PC1(t) PC2(t)
12
Manipulated stimuli
13
Conclusions A data driven approach is possible in the exploration of
intonation phenomena
FDA provides automatic tools to describe variation in a set
of pitch contours extracted from real utterances
provided that the relevant landmarks are annotated
The same tools allow to construct artificial contours with
desired perceptual characteristics
Smooth and global variation are applied
Variations come from a statistical analysis of data
The process is automatic
14