learning narrative schemas nate chambers, dan jurafsky stanford university ibm watson research...
TRANSCRIPT
Learning Narrative SchemasLearning Narrative Schemas
Nate Chambers, Dan Jurafsky
Stanford University
IBM Watson Research Center Visit
Two Joint TasksTwo Joint Tasks
Events in a Narrative Semantic Roles
suspect, criminal, client, immigrant, journalist, government, …
police, agent, officer, authorities, troops, official, investigator, …
ScriptsScripts
• Background knowledge for language understanding
Restaurant Script
Schank and Abelson. 1977. Scripts Plans Goals and Understanding. Lawrence Erlbaum.
Mooney and DeJong. 1985. Learning Schemata for NLP. IJCAI-85.
• Hand-coded• Domain dependent
ApplicationsApplications
• Coreference• Resolve pronouns (he, she, it, etc.)
• Summarization• Inform sentence selection with event confidence scores
• Aberration Detection• Detect surprise/unexpected events in text
• Story Generation• McIntyre and Lapata, (ACL-2009)
• Textual Inference• Does a document infer other events
• Selectional Preferences• Use chains to inform argument types
The ProtagonistThe Protagonist
protagonist:
(noun)
1. the principal character in a drama or other literary work
2. a leading actor, character, or participant in a literary work or real event
Inducing Narrative RelationsInducing Narrative Relations
1. Dependency parse a document.
2. Run coreference to cluster entity mentions.
3. Count pairs of verbs with coreferring arguments.
4. Use pointwise mutual information to measure relatedness.
Chambers and Jurafsky. Unsupervised Learning of Narrative Event Chains. ACL-08
Narrative Coherence AssumptionVerbs sharing coreferring arguments are semantically connected
by virtue of narrative discourse structure.
Chain Example (ACL-08)Chain Example (ACL-08)
Schema Example (new)Schema Example (new)
Police, Agent, Authorities
Judge, OfficialProsecutor, Attorney
Plea, Guilty, InnocentSuspect, Criminal,Terrorist, …
Narrative SchemasNarrative Schemas
Integrating Argument TypesIntegrating Argument Types
• Use verb relations to learn argument types. • Record head nouns of coreferring arguments.
• Use argument types to learn verb relations.• Include argument counts in relation scores.
The typhoon was downgraded
Sunday as it moved inland from the
coast, where it killed two people.
downgrade-o, move-s, typhoon
move-s, kill-s, typhoon
downgrade-o, kill-s, typhoon
€
sim(ei,e j )
€
sim(ei,e j ,a)
Learning SchemasLearning Schemas
€
narsim(N,v j ) = maxC i
chainsim(Ci,< v j ,d >)d∈Dv j
∑
Argument InductionArgument Induction
SuspectGovernmentJournalistMondayMemberCitizenClient……
€
score( )
• Induce semantic roles by scoring argument head words.
Training DataTraining Data
• 1.2 million New York Times articles• NYT portion of the Gigaword Corpus• David Graff. 2002. English Gigaword. Linguistic Data Consortium.
• Stanford Parser• http://nlp.stanford.edu/software/lex-parser.shtml
• OpenNLP coreference• http://opennlp.sourceforge.net
• Lemmatize verbs and noun arguments.
Learned ExamplesLearned Examples
court, judge, justice, panel, Osteen, circuit, nicolau, sporkin, majority
law, ban, rule, constitutionality, conviction, ruling, lawmaker,
Learned ExamplesLearned Examples
company, inc, corp, microsoft, iraq, co, unit, maker, …
drug, product, system, test, software, funds, movie, …
Database of SchemasDatabase of Schemas
• ~500 unique schemas, 10 events each• Temporal ordering data• Available online soon.
EvaluationsEvaluations
• Compared to FrameNet• High precision when overlapping• New type of knowledge not included
• Cloze Evaluation• Predict missing events• Far better performance than vanilla distributional approaches
Future WorkFuture Work
• Improved information extraction• Extract information across multiple predicates.
• Knowledge Organization• Link news articles describing subsequent events.
• Core AI Reasoning• Automatic approach to learning causation?
• NLP specific tasks• Coreference, summarization, etc.
Thanks!Thanks!
• Unsupervised Learning of Narrative Schemas and their ParticipantsNathanael Chambers and Dan JurafskyACL-09, Singapore. 2009.
• Unsupervised Learning of Narrative Event ChainsNathanael Chambers and Dan JurafskyACL-08, Ohio, USA. 2008.
• Jointly Combining Implicit Constraints Improves Temporal OrderingNathanael Chambers and Dan JurafskyEMNLP-08, Waikiki, Hawaii, USA. 2008.
• Classifying Temporal Relations Between EventsNathanael Chambers, Shan Wang, Dan JurafskyACL-07, Prague. 2007.
Cloze EvaluationCloze Evaluation
1. Choose a news article at random.
2. Identify the protagonist.
3. Extract the narrative event chain.
4. Randomly remove one event from the chain.
• Predict which event was removed.
Cloze ResultsCloze Results
• Outperform the baseline distributional learning approach by 36%
• Including participants improves further by 10%
Comparison to FrameNetComparison to FrameNet
• Narrative Schemas• Focuses on events that occur together in a narrative.
• FrameNet (Baker et al., 1998)
• Focuses on events that share core roles.
Comparison to FrameNetComparison to FrameNet
• Narrative Schemas• Focuses on events that occur together in a narrative.• Schemas represent larger situations.
• FrameNet (Baker et al., 1998)
• Focuses on events that share core roles.• Frames typically represent single events.
Comparison to FrameNetComparison to FrameNet
1. How similar are schemas to frames?• Find “best” FrameNet frame by event overlap
2. How similar are schema roles to frame elements?• Evaluate argument types as FrameNet frame elements.
FrameNet Schema SimilarityFrameNet Schema Similarity
1. How many schemas map to frames?• 13 of 20 schemas mapped to a frame• 26 of 78 (33%) verbs are not in FrameNet
2. Verbs present in FrameNet• 35 of 52 (67%) matched frame• 17 of 52 (33%) did not match
FrameNet Schema SimilarityFrameNet Schema Similarity
traderisefall
Exchange
Change Position on a Scale
Two FrameNet FramesOne Schema
• Why were 33% unaligned?• FrameNet represents subevents as separate frames• Schemas model sequences of events.
FrameNet Argument SimilarityFrameNet Argument Similarity
2. Argument role mapping to frame elements.• 72% of arguments appropriate as frame elements
law, ban, rule, constitutionality,conviction, ruling, lawmaker, tax
INCORRECT
FrameNet frame: EnforcingFrame element: Rule
XX Event ScoringXX Event Scoring
€
chainsim(Ci,< acquit,subj >) =maxa
sim(< acquit,subj >,ei,a)i=1
n
∑
€
+score(Ci,a)
XX Argument InductionXX Argument Induction
• Induce semantic roles by scoring argument head words.
€
score( ) = (1− λ )pmi(ei,e j )j= i+1
n
∑i=1
n−1
∑ +λ log( freq(ei,e j , ))
How often do events shareany coreferring arguments?
How often do they shareargument ?
= criminal?
€
score( ) = sim(ei,e j , )j= i+1
n
∑i=1
n−1
∑
ResultsResults
Chains
Schemas
Typed Chains
Typed Schemas
10.1%
ResultsResults
1. We learned rich narrative structure.
• 10.1% improvement over previous work
2. Induced semantic roles characterizing the participants in a narrative.
3. Verb relations and their semantic roles can be jointly learned and improve each other’s results.
• Selectional preferences improve verb relation learning.
XX Semantic Role InductionXX Semantic Role Induction
• Supervised Learning• PropBank (Palmer et al., 2005),• FrameNet (Baker et al., 1998), • VerbNet (Kipper et al., 2000)
• Bootstrapping from a seed corpus• (Swier and Stevenson, 2004), (He and
Gildea, 2006)
• Unsupervised, pre-defined roles• (Grenegar and Manning 2006)
• WordNet inspired• (Green and Dorr, 2005), (Alishahi and
Stevenson, 2007)
SuspectGovernmentJournalistMondayMemberCitizenClient…