amul&’modaltrainingapproach · pdf fileintroduc&on" •...
TRANSCRIPT
A mul&-‐modal training approach to improve cochlear implant users' ability to handle simultaneous talk
Amy Beeston and Emina Kur&ć Mul&modality in Language Research
Leeds ·∙ 26 June 2014
University of Sheffield • Prof Bill Wells, Human Communica&on Sciences • Prof Guy Brown, Computer Science • Dr Emina Kur&ć and Amy Beeston, Research Associates, Computer Science • Ella Page, project student, Human Communica&on Sciences Sheffield Teaching Hospitals NHS Trust • Dr Harriet Crook, Clinical Scien&st, Neurotology • Erica Bradley, Speech & Language Therapist, Neurotology Cochlear implant (CI) user panel • 5 cochlear-‐implanted par&cipants recruited from Sheffield Cochlear Implant Service
Project team
!"#$!%&'(&)&(&"%&*
%+,-$.&('"+./&(*
0111234524
/&&106255
-78'9$!"-:;7-& <1:16:<1=0.!,,'7.>'/&&,-$"=36'%&.&-&(>'($7;,?&))!&@;,=='3)A
!"#$!%&
;&-7!@,'$)'%?7(B& %?7(B& #7- -$-7@@!"&'"$C
!
97>.&"-';+&';7-& =5:14:<1=0
)$('-?&'(&%$#&(>'$)')&&'(&*;/,'%?&%D
'03C02 '1C11 '03C0211= !"'&#&"-'$)'7"'!"#$!%&'A+&(>%$"-7%-'($!,!"'.%)&&@>'$"1==0'216'<611
!
-$-7@' '03C02
!EFGHIJ'(JKJLJEIJ'0111234524'M/&&106255N'KGL'O'03C02'HP'EGQ'RSJ'TER'UTVTWXJ'WV'GEJ'GK'YZJ'[JYZGRP'WJXGQ*\'/V'/TE]'-LTEPKJL':'/7%,':'?G[J'/TE]HE^_',GLY'%GRJ'132211`'/TE]'7IIGSEY'=<<a0663C'!/7"'B/0<%!-!132211=<<a0663`',b!)-'%!-!B/<@\'/V';JWHY:'%LJRHY'ITLR'WV'YJXJUZGEHE^'GE'1==0'<a=<141'GL'1==0'<a=05<aC\'!E'UJLPGE'YG'YZJ'%TPZHJLP'$KKHIJ`'JHYZJL'TY'YZJ'(GVTX'?TXXT[PZHLJ'?GPUHYTX`'BXGPPGU'(GTR`',ZJKKHJXR',=1'<c)'GL'TY'YZJ'"GLYZJLE'BJEJLTX'?GPUHYTX`'
?JLLHJP'(GTR`',ZJKKHJXR',6'a7+\'/V'UGPY'd'%ZJeSJP`'UGPYTX'GLRJLP'JYIC'PZGSXR'WJ'ILGPPJR'f7:%'9TVJJ'$EXVf'TER'[TRJ'UTVTWXJ'YG',ZJKKHJXR'-JTIZHE^'?GPUHYTX'"?,')GSERTYHGE'-LSPY
9XJTPJ'JEIXGPJ'T'PYT[UJR'TRRLJPPJR'JEFJXGUJ'HK'T'LJIJHUY'HP'LJeSHLJR
9XJTPJ'RJYTIZ'YZHP'PJIYHGE'TER'KGLQTLR'QHYZ'VGSL'LJ[HYYTEIJ'YG*%TPZHJLP'$KKHIJ`'(GVTX'?TXXT[PZHLJ'?GPUHYTX`'BXGPPGU'(GTR`',ZJKKHJXR',=1'<c)'GL%TPZHJLP'$KKHIJ`'"GLYZJLE'BJEJLTX'?GPUHYTX`'?JLLHJP'(GTR`',ZJKKHJXR',6'a7+C
)HETEIHTX',JLFHIJP`',TXJP'@JR^JL`'%XGI]'-GQJL`'"GLYZJLE'BJEJLTX'?GPUHYTX`'?JLLHJP'(GTR`',ZJKKHJXR`',6'a7+-JXC'1==0'<<441a6'')Tg'1==0'<<441a0
#7-'(&B'"GC'460'011='46
'03C02 '1C11
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
Introduc&on
• Losing your hearing is not simply the absence of sound – for individual: loss of capacity to take part in social life – addi&onal societal s&gma: being treated differently*
• Cochlear implanta&on is not a cure for deafness – it is a prosthe&c subs&tute.
• Approx 324,200 people worldwide implanted by end of 2012† – 10,000 cases in UK† – c. 113 per month in UK‡
• Insufficient evidence/training materials to guide clinicians to help cochlear implant (CI) users deal with simultaneous talk.
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
* Lamb and Archibold (2013), Adult Cochlear Implanta&on: Evidence and experience. Tech rep, The Ear Founda&on. † NIH Publica&on No. 11-‐4798 (2013-‐11-‐01). ‡ Bri&sh Cochlear Implant Group, Annual Update on UK Cochlear Implant Provision (2011-‐2012).
Electric hearing
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
www.mrc-‐cbu.cam.ac.uk
• Acous&cal vibra&ons are delivered to brain as electrical signals
Challenges for CI users
• Cochlear implant (CI) users need op&mum conversa&onal sejngs (quiet, one partner, awareness to avoid overlap).
• Q: “When no background noise is present, can you effortlessly par&cipate in conversa&ons with friends or family members (e.g., aner dinner)?”
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• Mean response for 26 adults = 5.7
MED-‐EL Hearing Implant Sound Quality Index (HISQUI) v23015_2.0, English.
always never
“almost always”
“frequently”
Challenges for CI users
• Q: “When mul&ple people are talking simultaneously, can you effortlessly follow discussions of friends and family members?”
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
MED-‐EL Hearing Implant Sound Quality Index (HISQUI) v23015_2.0, English.
• Mean response for 26 adults = 2.8 • Unfamiliar voices are s&ll more difficult to follow • Limits of the device? Conversa&onal/social experience?
always never
“occasionally” “rarely”
Social signals and behaviours
• Mul&ple behavioural cues combine to produce a social signal
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
Vinciarelli et al. (2009). Social signal processing: Survey of an emerging domain. Image and Vision Compu&ng 27, 1743–1759.
– Physical appearance – Gesture and posture – Face/eyes behaviour – Space and environment – Linguis'c content – Vocal behaviour *
* prosodic features, e.g., pitch, loudness, dura&on, silence, …
• Stronger reliance for CI users on non-‐auditory cues? • Is vocal non-‐verbal behaviour informa&ve or distrac&ng?
CI sound encoding
• 22 electrodes vs. 30,000 hair cells. • Low spectral resolu&on: hard to dis&nguish one talker from
another (especially concurrent talkers). • Poor representa&on of voice F0: hard to iden&fy social ac&ons
realised by prosodic cues (e.g., turn compe&&on).
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
www.cochlear.com
Scene analysis problem
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• Parse single-‐channel stream without cues from pitch or loca&on
Kur&ć et al. (2012). A corpus of spontaneous mul&-‐party conversa&on in Bosnian Serbo-‐Croa&an and Bri&sh English. Proc. LREC, Istanbul.
Conversa&onal dataset
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• 3 hours of informal talk, na&ve English (UK)
• 16% of total talk &me is overlapping talk
• 41% of speaker turns are overlapped by another speaker
Kur&ć et al. (2012). A corpus of spontaneous mul&-‐party conversa&on in Bosnian Serbo-‐Croa&an and Bri&sh English. Proc. LREC, Istanbul.
• Lip-‐reading impossible; Gaze, gesture, posture, … • Segmenta&on by turn construc&on unit (TCU), transcrip&on,
overlap compe&&veness, xml data in ELAN annota&on format
CI-‐user feedback
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• Web-‐based ques&onnaire – hearing conversa&ons – taking part in conversa&ons – age, gender, and hearing status (implant type) per ear
• CI-‐user panel
– 5 expert CI-‐users recruited for dura&on of project – conversa&onal experiences, awareness of overlap – grading speech material – grading user tasks – evaluate training sonware – listening skills – (later, speaking skills)
Listening skill development
• Objec&ve – Understand mul&-‐party conversa&on • Visualisa&on of acous&c informa&on • CI-‐user difficulty ra&ngs for speech material and tasks
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
Reported difficul&es – iden&fy talker – recognise speech – understand ac&on
i. Speech material
• Graded exposure to mul&-‐party conversa&on – Incremental learning through repe&&on
• Classify by number/gender of talkers, turn transi&ons
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
ii. User tasks
• Widen conversa&onal experience – Rate difficulty for CI user
• Focus on non-‐linguis&c aspects – iden&fy number of talkers – iden&fy same/different gender – iden&fy clear/overlap
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
N G T speech collec'on 0 0 0 1 talker 1 0 0 2 talkers, same gender, clear 1 0 1 2 talkers, same gender, overlap 1 1 0 2 talkers, diff. gender, clear 1 1 1 2 talkers, diff. gender, overlap
Number of talkers, N (single = 0, mul&ple = 1)
Gender, G (same = 0, different = 1)
Turn exchange, T (clear = 0, in overlap = 1)
(0, 0, 0) T G
E
Speaking skill development
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• Objec&ve: par&cipate in mul&-‐party conversa&on • 1. gather voice norms (pitch, intensity, tempo) • 2. use overlaps appropriately
– Iden&fy entry &mes (especially in overlap) – Control pitch/loudness (show compe&&veness)
Conclusions and further work
Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć
• Some CI-‐users keen to develop mul&-‐party conversa&onal skills – varied popula&on – different experiences/strategies
• Informal talk is extremely challenging – missing some turn-‐taking cues – mul&modal presenta&on of acous&c cues
• Learning interac&onal strategies for handling overlapping talk – complement technological advances – increase social par&cipa&on
• Sonware may be more widely relevant – hearing or communica&ve difficul&es