creating subjective and objective sentence classifiers from unannotated texts ellen riloff...
TRANSCRIPT
Creating Subjective and Objective Sentence Classifiers from Unannotated Texts
Ellen RiloffUniversity of Utah
(Joint work with Janyce Wiebe at theUniversity of Pittsburgh)
What is Subjectivity?
• Subjective language includes opinions, rants, allegations, accusations, suspicions, and speculation.
• Distinguishing factual information from subjective information could benefit many applications, including:
– information extraction– question answering– summarization– spam filtering
Previous Work on Subjectivity Classification
• Document-level subjectivity classification (e.g., [Turney 2002; Pang et al. 2002; Spertus 1997])
But most documents contain subjective and objective sentences. [Wiebe et al. 01] reported that 44% of sentences in their news corpus were subjective!
• Sentence-level subjectivity classification [Dave et al. 2003; Yu et al. 2003; Riloff, Wiebe, & Wilson 2003]
Goals of our research
• Create classifiers that label sentences as subjective or objective.
• Learn subjectivity and objectivity clues from unannotated corpora.
• Use information extraction techniques to learn subjective nouns.
• Use information extraction techniques to learn subjective and objective patterns.
Outline of Talk
• Learning subjective nouns with extraction patterns
• Automatically generating training data with high-precision classifiers
• Learning subjective and objective extraction patterns
• Naïve Bayes classification and self-training
Information Extraction
• Information extraction (IE) systems identify facts related to a domain of interest.
• Extraction patterns are lexico-syntactic expressions that identify the role of an object. For example:
<subject> was killed
assassinated <dobj>
murder of <np>
Learning Subjective Nouns
Goal: to learn subjective nouns from unannotated texts.
Method: applying IE-based bootstrapping algorithms that were designed to learn semantic categories.
Hypothesis: extraction patterns can identify subjective contexts that co-occur with subjective nouns.
Example: “expressed <dobj>” concern, hope, support
Extraction Examples
expressed <dobj> condolences, hope, grief, views, worries
indicative of <np> compromise, desire, thinking
inject <dobj> vitality, hatred
reaffirmed <dobj> resolve, position, commitment
voiced <dobj> outrage, support, skepticism, opposition, gratitude, indignation
show of <np> support, strength, goodwill, solidarity
<subj> was shared anxiety, view, niceties, feeling
Meta-Bootstrapping [Riloff & Jones 99]
Unannotated Texts
Best Extraction Pattern
Extractions (Nouns)
Ex: hope, grief, joy, concern, worries
Ex:expressed <DOBJ>
Ex: happiness, relief, condolences
Basilisk [Thelen & Riloff 02]
extraction patterns andtheir extractions
corpus
seedwords
semanticlexicon
5 best candidate words
Pattern Pool
best patterns
CandidateWord Pool
extractions
Subjective Seed Words
cowardice embarrassment hatred outragecrap fool hell slanderdelight gloom hypocrisy sighdisdain grievance love twitdismay happiness nonsense virtue
Subjective Noun Results
• Bootstrapping corpus: 950 unannotated FBIS documents (English-language foreign news)
• We ran each bootstrapping algorithm for 400 cycles, generating ~2000 words.
• We manually reviewed the words and labeled them as strongly subjective or weakly subjective.
• Together, they learned 1052 subjective nouns (454 strong, 598 weak).
Examples of Strong Subjective Nouns
anguish exploitation pariahantagonism evil repudiationapologist fallacies revengeatrocities genius roguebarbarian goodwill sanctimoniousbelligerence humiliation scumbully ill-treatment smokescreencondemnation injustice sympathydenunciation innuendo tyrannydevil insinuation venomdiatribe liarexaggeration mockery
Examples of Weak Subjective Nouns
aberration eyebrows resistantallusion failures riskapprehensions inclination sincerityassault intrigue slumpbeneficiary liability spiritbenefit likelihood successblood peaceful tolerancecontroversy persistent trickcredence plague trustdistortion pressure unitydrama promiseeternity rejection
Outline of Talk
• Learning subjective nouns with extraction patterns
• Automatically generating training data with high-precision classifiers
• Learning subjective and objective extraction patterns
• Naïve Bayes classification and self-training
Initial Training Data Creation
rule-based subjectivesentenceclassifier
rule-basedobjectivesentenceclassifier
subjective & objective sentences
unlabeled texts
subjective clues
Subjective Clues
• entries from manually developed resources [Levin 93; Ballmer & Brennenstuhl 81]
• Framenet lemmas with frame element experiencer [Baker et al. 98]
• adjectives manually annotated for polarity [Hatzivassiloglou & McKeown 97]
• n-grams learned from corpora [Dave et al. 03; Wiebe et al. 01]
• words distributionally similar to subjective seed words [Wiebe 00]
• subjective nouns learned from extraction pattern bootstrapping [Riloff et al. 03]
Creating High-Precision Rule-Based Classifiers
• a sentence is subjectivesubjective if it contains 2 strong subjective clues
• a sentence is objective if:
– it contains no strong subjective clues
– the previous and next sentence contain 1 strong subjective clue
– the current, previous, and next sentence together contain 2 weak subjective clues
GOAL: use subjectivity clues from previous research to build a high-precision (low-recall) rule-based classifier
Data Set
• The MPQA Corpus contains 535 FBIS texts that have been manually annotated for subjectivity.
• Our test set consisted of 9,289 sentences from the MPQA corpus.
• We consider a sentence to be subjective if it has at least one private state of strength medium or higher.
• 54.9% of the sentences in our test set are subjective.
Accuracy of Rule-Based Classifiers
SubjRec SubjPrec SubjFSubj RBC 34.2 90.4 46.6
ObjRec ObjPrec ObjFObj RBC 30.7 82.4 44.7
Generated Data
We applied the rule-based classifiers to 298,809 sentences from (unannotated) FBIS documents.
52,918 were labeled subjective
47,528 were labeled objective
training set of over 100,000 labeled sentences!
Outline of Talk
• Learning subjective nouns with extraction patterns
• Automatically generating training data with high-precision classifiers
• Learning subjective and objective extraction patterns
• Naïve Bayes classification and self-training
Representing Subjective Expressions with Extraction Patterns
• Extraction patterns can represent linguistic expressions that are not fixed word sequences.
drove [NP] up the wall- drove him up the wall- drove George Bush up the wall- drove George Herbert Walker Bush up the wall
step on [modifiers] toes- step on her toes- step on the mayor’s toes- step on the newly elected mayor’s toes
gave [NP] a [modifiers] look- gave his annoying sister a really really mean look
The Extraction Pattern Learner
• Used AutoSlog-TS [Riloff 96] to learn extraction patterns.
• AutoSlog-TS needs relevant and irrelevant texts as input.
• Statistics are generated measuring each pattern’s association with the relevant texts.
• The subjective sentences were called relevant, and the objective sentences were called irrelevant.
<subject> passive-vp <subj> was satisfied<subject> active-vp <subj> complained<subject> active-vp dobj <subj> dealt blow<subject> active-vp infinitive <subj> appears to be<subject> passive-vp infinitive <subj> was thought to be<subject> auxiliary dobj <subj> has position
active-vp <dobj> endorsed <dobj>infinitive <dobj> to condemn <dobj>active-vp infinitive <dobj> get to know <dobj>passive-vp infinitive <dobj> was meant to show <dobj>subject auxiliary <dobj> fact is <dobj>
passive-vp prep <np> opinion on <np>active-vp prep <np> agrees with <np>infinitive prep <np> was worried about <np>noun prep <np> to resort to <np>
Relevant Irrelevant
[The World Trade Center], [an icon] of [New York City], was intentionally attacked very early on [September 11, 2001].
Parser Extraction Patterns:<subj> was attackedicon of <np>was attacked on <np> Syntactic Templates
AutoSlog-TS(Step 1)
AutoSlog-TS (Step 2)
Relevant Irrelevant
Extraction Patterns Freq Prob<subj> was attacked 100 .90icon of <np> 5 .20was attacked on <np> 80 .79
Extraction Patterns:<subj> was attackedicon of <np>was attacked on <np>
Identifying Subjective and Objective Patterns
AutoSlog-TS generates 2 statistics for each pattern:F = pattern frequencyP = relevant frequency / pattern frequency
We call a pattern subjective if F 5 and P .95(6364 subjective patterns were learned)
We call a pattern objective if F 5 and P .15(832 objective patterns were learned)
Examples of Learned Extraction Patterns
Subjective Patterns<subj> believes<subj> was convincedaggression against <np>to express <dobj>support for <np>
Objective Patterns<subj> increased production<subj> took effectdelegation from <np>occurred on <np>plans to produce <dobj>
Patterns with Interesting Behavior
PATTERN FREQ P(Subj | Pattern)<subj> asked 128 .63<subj> was asked 11 1.0
<subj> was expected 45 .42was expected from <np> 5 1.0
<subj> talk 28 .71talk of <np> 10 .90<subj> is talk 5 1.0
<subj> put 187 .67<subj> put end 10 .90
<subj> is fact 38 1.0fact is <dobj> 12 1.0
Augmenting the Rule-Based Classifiers with Extraction Patterns
SubjRec SubjPrec SubjFSubj RBC 34.2 90.4 46.6Subj RBC 58.6 80.9 68.0 w/Patterns
ObjRec ObjPrec ObjFObj RBC 30.7 82.4 44.7Obj RBC 33.5 82.1 47.6 w/Patterns
Outline of Talk
• Learning subjective nouns with extraction patterns
• Automatically generating training data with high-precision classifiers
• Learning subjective and objective extraction patterns
• Naïve Bayes classification and self-training
Naïve Bayes Classifier
• We created an NB classifier using the initial training set and several set-valued features:
– strong & weak subjective clues from RBCs
– subjective & objective extraction patterns
– POS tags (pronouns, modals, adjectives, cardinal numbers, adverbs)
– separate features for each of the current, previous, and next sentences
Naïve Bayes Training
extractionpattern learner
training set
objectivepatterns
subjectivepatterns Naïve Bayes
training
POSfeatures
subjectiveclues
Naïve Bayes Results
SubjRec SubjPrec SubjFNaïve Bayes 70.6 79.4 74.7
ObjRec ObjPrec ObjFNaïve Bayes 77.6 68.4 72.7RWW03 74 70 72(supervised)
RWW03 77 81 79(supervised)
Self-Training Process
best N sentences
Naïve Bayesclassifier
unlabeledsentences
extractionpattern learner
training set
objectivepatterns
subjectivepatterns Naïve Bayes
training
POSfeatures
subjectiveclues
Self-Training Results
SubjRec SubjPrec SubjFSubj RBC w/Patts 1 58.6 80.9 68.0Subj RBC w/Patts 2 62.4 80.4 70.3
ObjRec ObjPrec ObjFObj RBC w/Patts 1 33.5 82.1 47.6Obj RBC w/Patts 2 34.8 82.6 49.0
Naïve Bayes 1 70.6 79.4 74.7Naïve Bayes 2 86.3 71.3 78.1
Naïve Bayes 1 77.6 68.4 72.7Naïve Bayes 2 57.6 77.5 66.1
RWW03 (supervised) 77 81 79
RWW03 (supervised) 74 70 72
Conclusions
• We can build effective subjective sentence classifiers using only unannotated texts.
• Extraction pattern bootstrapping can learn subjective nouns.
• Extraction patterns can represent richer subjective expressions.
• Learning methods can discover subtle distinctions between very similar expressions.
THE END
Thank you!
Related Work
• Genre classification (e.g., [Karlgren and Cutting 1994; Kessler et al. 1997; Wiebe et al. 2001])
• Learning adjectives, adj. phrases, verbs, and N-grams [Turney 2002; Hatzivassiloglou & McKeown 1997;
Wiebe et al. 2001]
• Semantic lexicon learning [Hearst 1992; Riloff & Shepherd 1997; Roark & Charniak 1998; Caraballo 1999]
– Meta-Bootstrapping [Riloff & Jones 99]
– Basilisk [Thelen & Riloff 02]
What is Information Extraction?
Extracting facts relevant to a specific topic from narrative text.
Example Domains
Terrorism: perpetrator, victim, target, date, location
Management succession: person fired, successor, position, organization, date
Infectious disease outbreaks: disease, organism, victim, symptoms, location, date
Information Extraction from Narrative Text
• Role relationships define the information of interest …keywords and named entities are not sufficient.
Researchers have discovered how anthrax toxin destroys cells and rapidly causes death ...
Troops were vaccinated against anthrax, cholera, …
Ranking and Manual Review
• The patterns are ranked using the metric:
• A domain expert reviews the top-ranked patterns and assigns thematic roles to the good ones.
RlogF (patterni) = Fi
Ni
* log2 (Fi)
Fi is the # of instances of patterni in relevant textsNi is the # of instances of patterni in all texts
Semantic Lexicons• A semantic lexicon assigns categories to words.
•Semantic dictionaries are hard to come by, especially for specialized domains.
•WordNet [Miller 90] is popular but is not always sufficient. [Roark & Charniak 98] found that 3 of every 5 words learned by their system were not present in WordNet.
politician humantruck vehiclegrenade weapon
The Bootstrapping Era
Unannotated Texts
+
= KNOWLEDGE !
Meta-Bootstrapping
Unannotated Texts
Best Extraction Pattern
Extractions (Nouns)
Ex: anthrax, ebola, cholera, flu, plague
Ex:outbreak of <NP>
Ex: smallpox,tularemia, botulism
Semantic Lexicon (NP) Results
Iter Company Location Title Location Weapon(Web) (Web) (Web) (Terror) (Terror)
1 5/5 (1.0) 5/5 (1.0) 0/1 (0) 5/5(1.0) 4/4(1.0)
10 25/32 (.78) 46/50 (.92) 22/31 (.71) 32/50 (.92) 31/44 (.70)
20 52/65 (.80) 88/100 (.88) 63/81 (.78) 66/100 (.66) 68/94 (.72)
30 72/113 (.64) 129/150 (.86) 86/131 (.66) 100/150 (.67) 85/144 (.59)
Basilisk
extraction patterns andtheir extractions
corpus
seedwords
semanticlexicon
5 best candidate words
Pattern Pool
best patterns
CandidateWord Pool
extractions
The Pattern Pool
RlogF (patterni) = Fi
Ni
* log2 (Fi)
Fi is the number of category members extracted by patterni
Ni is the total number of nouns extracted by patterni
where:
Every extraction pattern is scored and the best patternsare put into a Pattern Pool.
The scoring function is:
Scoring Candidate Words
1. collecting all patterns that extracted it
2. computing the average number of category members extracted by those patterns.
Each candidate word is scored by:
j=1
Ni
AvgLog (wordi) =log2 (Fj + 1)
Ni
Building
0
20
40
60
80
100
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct
Le
xic
on
E
ntr
ies
BA-1
MB-1
Event
0
50100
150
200250
300
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct L
ex
ico
n
En
trie
s
BA-1
MB-1
Human
0
200
400
600
800
1000
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct L
ex
ico
n
En
trie
s
BA-1
MB-1
Location
0
100
200
300
400
500
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct L
ex
ico
n
En
trie
s BA-1
MB-1
Time
0
10
20
30
40
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct
Le
xic
on
En
trie
s
BA-1
MB-1
Weapon
0
20
40
60
80
0 200 400 600 800 1000 1200
Total Lexicon Entries
Co
rre
ct
Le
xic
on
En
trie
s
BA-1
MB-1
Bootstrapping a Single Category
Bootstrapping Multiple Categories
A Smarter Scoring Function
diff (wi,ca) = AvgLog (wi,ca) - max (AvgLog(wi,cb)) b a
We incorporated knowledge about competing semantic categories directly into the scoring function.
The modified scoring function computes the difference between the score for the target category and the best score among competing categories.