computational modeling of spoken language processing: a …€¦ · computational modeling of...
TRANSCRIPT
![Page 1: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/1.jpg)
The 30th Annual Conference of the Cognitive Science Society
Computational Modeling of Spoken Language Processing:
A hands-on tutorial
![Page 2: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/2.jpg)
Computational Modeling of Spoken LanguageProcessing: A hands-on tutorial
Ted StraussNew School University
Dan MirmanJim Magnuson
University of Connecticut Department of Psychologyand
Haskins Laboratories
![Page 3: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/3.jpg)
Goals
• General– Illustrate principles of modeling using spoken word
recognition as an example domain
• Specific– Intensive instruction in using jTRACE to prepare
participants to do their own modeling
![Page 4: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/4.jpg)
Plan• Module 1: Introduction to SWR and TRACE• Module 2: Tour of jTRACE• Module 3: Classic simulations• Module 4: Scripting• Module 5: Linking hypotheses• Module 6: Lab time, Q&A, one-on-one
![Page 5: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/5.jpg)
Module 1
•Motivations for modeling
•Review of speech perception andspoken word recognition (SWR)models
•Introduction to TRACE
![Page 6: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/6.jpg)
Principles of spoken word recognition• Current theories share three core principles
(cf. Marslen-Wilson, 1993)
– As a word is heard:1. Multiple words are activated2. Activation depends on
a. Similarity to the inputb. Word frequency (prior probability)
3. Activated words compete for recognition
![Page 7: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/7.jpg)
Fundamental problems for SWR
• Precisely which items are activated (similaritymetric)?
• Segmentation / alignment problems• How is competition resolved?• Fluent speech vs. isolated words• Learning• Connections to production, semantics
(word/sentence/beyond)
![Page 8: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/8.jpg)
Why model?• Minor differences in similarity metric, competition
mechanism, etc., lead to intuitive differences• What are the precise differences?• With just a few assumptions operating
simultaneously, analytic prediction becomesdifficult if not intractable
• Prediction via simulation– Forces precise specification of assumptions– When faced with demands of real processing, simpler
solutions may emerge– OR seemingly logical predictions may be falsified
![Page 9: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/9.jpg)
Psychological models vs. ASR
• Keep in mind: our goal is to developpsychological models
• These will not perform as well as ASR systems• No current psychological models of word
recognition work directly with speech• But ASR systems seem to operate very differently
than human speech recognition and are notpsychologically tractable
![Page 10: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/10.jpg)
Different kinds of modelsCohort (Marslen-Wilson & Welsh, 1978;Marslen-Wilson, 1987)
Verbal /box and arrow
Neighborhood Activation Model(Luce, 1986; Luce & Pisoni, 1998)
Mathematical
TRACE (McClelland & Elman, 1986)
Shortlist/Merge (Norris, 1994; Norris et al., 2000)
PARSYN (Auer, Luce et al., 2000)
SRNs (Elman, 1990; Gaskell & Marslen-Wilson, 1997;Magnuson et al., 2003)
Plaut & Kello (1999)
ART (e.g., Grossberg et al., 1997, 2000)
Simulating
![Page 11: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/11.jpg)
Comparing types of models
• Nature of the competitor set• Cohort and NAM make conflicting predictions• Can simulated time course help resolve the
conflict?
![Page 12: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/12.jpg)
Verbal/box & arrow• Cohort I, II: precise verbal
models• Make optimal use of speech:
activate based on matches,inhibit based on mismatch
• Exploits temporal nature ofspeech for segmentation
• Predictions: ordinal/relative toinformation density
• Evidence: cat primes sugar(via candy) but not chair (viasat) k ae t
toadghostcoatcoastkeencavecatchcastcandycastlecatcattlecatapult
catchcastcandycastlecatcattlecatapult
catcattlecatapult
![Page 13: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/13.jpg)
Mathematical• Neighborhood Activation Model (NAM)
Luce (1986), Luce & Pisoni (1998)
• Mathematical model– Described as a processing model, but most significant
contribution: simple, concise encapsulation of theoreticalassumptions
– Does not address segmentation/alignment
1. Operationalize neighbor (1-phoneme shortcut,segment-by-segment similarity)
2. Recognition facility (frequency weightedneighborhood probability) ≈
!
ft
fn"• Evidence: FWNPR accounts for more variance than
any other factor!
![Page 14: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/14.jpg)
Competitor setsExample: cat
CohortsNeighbors
cab can castcad cam cattlecap calf calve
caskcampcafécabincandycandlecaverncaptaincanteencampaign …
cameracatatoniccatapultcatalogcasserolecabernet …
cutkit
caughtcoat
cot
atpat
gnatbatmat
fatvattathatratsat …
![Page 15: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/15.jpg)
Simulating• To simulate, you must grapple with the
practical implications of theoreticalassumptions
• Also, many other details– How to make input analogous to speech– How to map model time to real time– How to link model performance to human task– How to gauge model success and failure
![Page 16: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/16.jpg)
Simulating: TRACE (McClelland & Elman, 1986)
Features over time
beaker
i s Rb
speaker Allopenna, Magnuson,& Tanenhaus (1998):Human eye trackingdata highly similar toTRACE predictions
![Page 17: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/17.jpg)
Why use TRACE?• Excellent predictions for broad range of
phenomena• Representative of current models
– Dynamics/time course– Embodies 3 key principles (multiple activation;
activation proportional to similarity & prior probability;competition)
• Relatively transparent parameters• Shortcomings
– Brute force approach to solving alignment– Does not learn
![Page 18: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/18.jpg)
Introduction to TRACE
• Architecture• Connectivity• Flow of activation• Input representation• Processing time vs. slice time
![Page 19: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/19.jpg)
TRACE architecture: Connectivity
• Bottom-up (feed-forward) excitatoryconnections
– Input→feature– Feature→phoneme– Phoneme→word
• Top-down (feed-back) excitatoryconnections
– Word→phoneme– Phoneme→feature*
• Lateral inhibitionwithin layers
![Page 20: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/20.jpg)
TRACE input representation
• Designed to approximate several importantfacts about the speech signal and speechperception– Perceptual similarity rooted in acoustic
similarity rooted in event (production)similarity
– Speech signal is extended over time– Speech sounds (phonemes) overlap in time
from one to the next
![Page 21: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/21.jpg)
TRACE: Input representation• The input to TRACE
is a matrix of 7 featurevectors with 9 levelseach
• Features are based onacoustic-phoneticfeatures– consonantal, vocalic,
diffuseness, acuteness,voicing, burst, power
- ^ b r ^ p t -
![Page 22: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/22.jpg)
Coarticulation
- ^ b r ^ p t -
![Page 23: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/23.jpg)
Similarity/t/ ↔ /d/ vs. /t/ ↔ /a/
Cycle
Activation
tdpkb
![Page 24: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/24.jpg)
Activation of phonemes & words• Bottom-up (feed-
forward) excitationcauses initialactivation
• Excitationincreases withtemporal overlapand similarity
![Page 25: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/25.jpg)
Lexical-to-phoneme feedback• Word units feed
back to constituentphonemes
• Constituentphonemes thatwere not “heard”(earlier or later) vialexical feedback
![Page 26: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/26.jpg)
Within-layer competition (lateral inhibition)
![Page 27: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/27.jpg)
Two kinds of time1. “Real” time (cycles): passing of time,
during which speech input is beingpresented to the model continuously
![Page 28: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/28.jpg)
Two kinds of time
2. Temporalalignmentof units;“slice”number
![Page 29: Computational Modeling of Spoken Language Processing: A …€¦ · Computational Modeling of Spoken Language Processing: A hands-on tutorial Ted Strauss New School University Dan](https://reader034.vdocuments.site/reader034/viewer/2022052016/602e30295e63c267a95673b1/html5/thumbnails/29.jpg)
Next: Module 2, tour of jTRACE