ambiguity, statistical word sense discovery and...
TRANSCRIPT
![Page 1: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/1.jpg)
Ambiguity,Statistical Word Sense Discovery
and Semantic Role Labeling
Andrew McCallum
Computer Science DepartmentUniversity of Massachusetts Amherst
Including slides from Chris Manning and Dan Klein.
![Page 2: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/2.jpg)
Inherent Ambiguity in SyntaxFed raises interest rates 0.5%in effort to control inflation
NY Times headline 17 May 2000S
NP VP
NNP
FedV NP NP PP
raisesinterest rates
NN NN0.5 in NN VP
V VP
V NP
NN
CD NN PP NP%
effortto
controlinflation
![Page 3: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/3.jpg)
Where are the ambiguities?
Fed raises interest rates 0.5 % in effort tocontrol inflation
Part-of-speech ambiguities Syntactic attachmentambiguities
Word sense ambiguities: Fed →”federal agent”interest →a feeling of wanting to know or learn more
Semantic interpretation ambiguities above the word level.
NNP NNSVBZ
NNSVBZ
NNSVBZ
VB
CD NN
![Page 4: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/4.jpg)
Effects of V/N Ambiguity (1)
S
NP VP
NNP
Fed
V NP
raises
interest rates
NN NN
![Page 5: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/5.jpg)
Effects of V/N Ambiguity (2)
S
NP VP
N
Fed
N NP
raises interest
rates
V
N
![Page 6: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/6.jpg)
Effects of V/N Ambiguity (3)
S
NP VP
N
Fed
N NP
raises interest rates
N V
NCD
%0.5
![Page 7: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/7.jpg)
Ambiguous Headlines
• Iraqi Head Seeks Arms• Juvenile Court to Try Shooting Defendant• Teacher Strikes Idle Kids• Stolen Painting Found by Tree• Kids Make Nutritious Snacks• Local HS Dropouts Cut in Half• British Left Waffles on Falkland Islands• Red Tape Holds Up New Bridges• Clinton Wins on Budget, but More Lies Ahead• Ban on Nude Dancing on Governor’s Desk
![Page 8: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/8.jpg)
Natural Language Computingis hard because
• Natural language is:– highly ambiguous at all levels– complex and subtle– fuzzy, probabilistic– interpretation involves combining evidence– involves reasoning about the world– embedded a social system of people interacting
• persuading, insulting and amusing them• changing over time
![Page 9: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/9.jpg)
Probabilistic Models of Language
The tools of probability:• Bayesian Classifiers (not rules)• Hidden Markov Models (not DFAs)• Probabilistic Context Free Grammars
• …other tools of Machine Learning, AI, Statistics
To handle this ambiguity and to integrateevidence from multiple levels we turn to:
![Page 10: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/10.jpg)
Another Area whereProbabilistic Combination of Evidence
Won
![Page 11: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/11.jpg)
Attachment Ambiguity
• Where to attach a phrase in the parse tree?• “I saw the man with the telescope.”
– What does “with a telescope” modify?– Is the problem AI complete? Yes, but…
– Proposed simple structural factors• Right association [Kimball 1973]
‘low’ or ‘near’ attachment = ‘early closure’ of NP• Minimal attachment [Frazier 1978]
(depends on grammar) = ‘high’ or ‘distant’ attachment= ‘late closure’ (of NP)
![Page 12: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/12.jpg)
Attachment Ambiguity
• Such simple structural factors dominated inearly psycholinguistics, and are still widelyinvoked.
• In the V NP PP context, right attachment getsright 55-76% of the cases…
• But this means that it gets wrong 33-45% ofthe cases!
![Page 13: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/13.jpg)
Attachment Ambiguity
• “The children ate the cake with a spoon.”• “The children ate the cake with frosting.”
• “Joe included the package for Susan.”• “Joe carried the package for Susan.”
• Ford, Bresnan and Kaplan (1982):“It is quite evident, then, that the closure effects inthese sentences are induced in some way by thechoice of the lexical items.”
![Page 14: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/14.jpg)
Simple model
• (Log) likelihood ratio– A common and good way of comparing between
two exclusive alternatives– Same idea as a naïve Bayes classifier
– if >0, attach to verb, if <0 attach to noun– For example,
P(with a spoon | ate) > P(with a spoon | cake)
![Page 15: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/15.jpg)
Attachment, Problematic Example
• “Chrysler confirmed that it would end its troubledventure with Maserati.”
• w C(w) C(w, with)end 5156 607venture 1442 155
• Get wrong answer:P(with|end) = (607/5156) = 0.118P(with|venture) = (155/1442) = 0.107
• Should also express preference for attaching ‘low’.
![Page 16: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/16.jpg)
Other attachment issues
• There are attachment questions other thanprepositional phrases– adverbial, participial, noun compounds– Examples
door bell manufacturer[door bell] manufacturerUnix system administratorUnix [system administrator]
– Data sparseness is a bigger problem with many of these
• In general, indeterminacy is quite common– “We have not signed a settlement agreement with them.”– Either reading seems equally plausible.
![Page 17: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/17.jpg)
Lexical acquisition, semantic similarity
• Previous models give same estimate to allunseen events.
• Unrealistic - could hope to refine that basedon semantic classes of words
• Examples– “Susan ate the cake with a durian.”– “Susan had never eaten a fresh durian before.”– Although never seen “eating pineapple” should be
more likely than “eating holograms” becausepineapple is similar to apples, and we have seen“eating apples”.
![Page 18: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/18.jpg)
An application: selectional preferences
• Most verbs prefer arguments of a particulartype. Such regularities are called selectionalpreferences or selectional restrictions.
• “Bill drove a…” Mustang, car, truck, jeep
• Selectional preference strength: how stronglydoes a verb constrain direct objects
• “see” versus “unknotted”
![Page 19: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/19.jpg)
Measuring selectional preference strength
• Assume we are given a clustering of (direct object) nouns.Resnick (1993) uses WordNet.
• Selectional association between a verb and a class
Proportion that its summand contributes to preference strength.
• For nouns in multiple classes, disambiguate as most likelysense:
![Page 20: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/20.jpg)
Selection preference strength(made up data)
Noun class c P(c) P(c|eat) P(c|see) P(c|find)people 0.25 0.01 0.25 0.33furniture 0.25 0.01 0.25 0.33food 0.25 0.97 0.25 0.33action 0.25 0.01 0.25 0.01SPS S(v) 1.76 0.00 0.35
A(eat, food) = 1.08A(find, action) = -0.13
![Page 21: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/21.jpg)
Selectional Preference Strength example(Resnick, Brown corpus)
![Page 22: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/22.jpg)
But how might we measureword similarity for word classes?
• Vector spaces
![Page 23: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/23.jpg)
But how might we measureword similarity for word classes?
• Vector spacesword-by-word matrix B
![Page 24: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/24.jpg)
Similarity measures for binary vectors
![Page 25: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/25.jpg)
Cosine measure
![Page 26: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/26.jpg)
Example of cosine measure onword-by-word matrix on NYT
![Page 27: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/27.jpg)
Probabilistic measures
![Page 28: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/28.jpg)
Neighbors of word “company”[Lee]
![Page 29: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/29.jpg)
Clustering words into topics withLatent Dirichlet Allocation
[Blei, Ng, Jordan 2003]
Sample a distributionover topics, θ
For each document:
Sample a topic, z
For each word in doc
Sample a wordfrom the topic, w
Example:
70% Iraq war30% US election
Iraq war
“bombing”
GenerativeProcess:
![Page 30: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/30.jpg)
STORYSTORIES
TELLCHARACTER
CHARACTERSAUTHOR
READTOLD
SETTINGTALESPLOT
TELLINGSHORT
FICTIONACTION
TRUEEVENTSTELLSTALE
NOVEL
MINDWORLDDREAM
DREAMSTHOUGHT
IMAGINATIONMOMENT
THOUGHTSOWNREALLIFE
IMAGINESENSE
CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE
WATERFISHSEA
SWIMSWIMMING
POOLLIKE
SHELLSHARKTANK
SHELLSSHARKSDIVING
DOLPHINSSWAMLONGSEALDIVE
DOLPHINUNDERWATER
DISEASEBACTERIADISEASES
GERMSFEVERCAUSE
CAUSEDSPREADVIRUSES
INFECTIONVIRUS
MICROORGANISMSPERSON
INFECTIOUSCOMMONCAUSING
SMALLPOXBODY
INFECTIONSCERTAIN
Example topicsinduced from a large collection of text
FIELDMAGNETICMAGNET
WIRENEEDLE
CURRENTCOIL
POLESIRON
COMPASSLINESCORE
ELECTRICDIRECTION
FORCEMAGNETS
BEMAGNETISM
POLEINDUCED
SCIENCESTUDY
SCIENTISTSSCIENTIFIC
KNOWLEDGEWORK
RESEARCHCHEMISTRY
TECHNOLOGYMANY
MATHEMATICSBIOLOGY
FIELDPHYSICS
LABORATORYSTUDIESWORLD
SCIENTISTSTUDYINGSCIENCES
BALLGAMETEAM
FOOTBALLBASEBALLPLAYERS
PLAYFIELD
PLAYERBASKETBALL
COACHPLAYEDPLAYING
HITTENNISTEAMSGAMESSPORTS
BATTERRY
JOBWORKJOBS
CAREEREXPERIENCE
EMPLOYMENTOPPORTUNITIES
WORKINGTRAINING
SKILLSCAREERS
POSITIONSFIND
POSITIONFIELD
OCCUPATIONSREQUIRE
OPPORTUNITYEARNABLE
[Tennenbaum et al]
![Page 31: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/31.jpg)
STORYSTORIES
TELLCHARACTER
CHARACTERSAUTHOR
READTOLD
SETTINGTALESPLOT
TELLINGSHORT
FICTIONACTION
TRUEEVENTSTELLSTALE
NOVEL
MINDWORLDDREAM
DREAMSTHOUGHT
IMAGINATIONMOMENT
THOUGHTSOWNREALLIFE
IMAGINESENSE
CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE
WATERFISHSEA
SWIMSWIMMING
POOLLIKE
SHELLSHARKTANK
SHELLSSHARKSDIVING
DOLPHINSSWAMLONGSEALDIVE
DOLPHINUNDERWATER
DISEASEBACTERIADISEASES
GERMSFEVERCAUSE
CAUSEDSPREADVIRUSES
INFECTIONVIRUS
MICROORGANISMSPERSON
INFECTIOUSCOMMONCAUSING
SMALLPOXBODY
INFECTIONSCERTAIN
FIELDMAGNETICMAGNET
WIRENEEDLE
CURRENTCOIL
POLESIRON
COMPASSLINESCORE
ELECTRICDIRECTION
FORCEMAGNETS
BEMAGNETISM
POLEINDUCED
SCIENCESTUDY
SCIENTISTSSCIENTIFIC
KNOWLEDGEWORK
RESEARCHCHEMISTRY
TECHNOLOGYMANY
MATHEMATICSBIOLOGYFIELD
PHYSICSLABORATORY
STUDIESWORLD
SCIENTISTSTUDYINGSCIENCES
BALLGAMETEAM
FOOTBALLBASEBALLPLAYERS
PLAYFIELD
PLAYERBASKETBALL
COACHPLAYEDPLAYING
HITTENNISTEAMSGAMESSPORTS
BATTERRY
JOBWORKJOBS
CAREEREXPERIENCE
EMPLOYMENTOPPORTUNITIES
WORKINGTRAINING
SKILLSCAREERS
POSITIONSFIND
POSITIONFIELD
OCCUPATIONSREQUIRE
OPPORTUNITYEARNABLE
Example topicsinduced from a large collection of text
[Tennenbaum et al]
![Page 32: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/32.jpg)
Collocations
• An expression consisting of two or morewords that correspond to some conventionalway of saying things.
• Characterized by limited compositionality.– compositional: meaning of expression can be
predicted by meaning of its parts.– “strong tea”, “rich in calcium”– “weapons of mass destruction”– “kick the bucket”, “hear it through the grapevine”
![Page 33: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/33.jpg)
Collocations important for…
• Terminology extraction– Finding special phrases in technical domains
• Natural language generation– To make natural output
• Computational lexicography– To automatically identify phrases to be listed in a dictionary
• Parsing– To give preference to parses with natural collocations
• Study of social phenomena– Like the reinforcement of cultural stereotypes through language (Stubbs
1996)
![Page 34: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/34.jpg)
Contextual Theory of Meaning
• In contrast with “structural linguistics”, which emphasizes abstractions,properties of sentences
• Contextual Theory of Meaning emphasizes the importance of context– context of the social setting (not idealized speaker)– context of discourse (not sentence in isolation)– context of surrounding words
Firth: “a word is characterized by the company it keeps”• Example [Halliday]
– “strong tea”, coffee, cigarettes– “powerful drugs”, heroin, cocaine– Important for idiomatically correct English, but also social implications of
language use
![Page 35: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/35.jpg)
Topics Modeling Phrases
• Topics based only on unigrams oftendifficult to interpret
• Topic discovery itself is confused becauseimportant meaning / distinctions carried byphrases.
• Significant opportunity to provide improvedlanguage models to ASR, MT, IR, etc.
![Page 36: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/36.jpg)
Topical N-gram Model
z1 z2 z3 z4
w1 w2 w3 w4
y1 y2 y3 y4
θ
φ1
T
D
. . .
. . .
. . .
α
WTW
ψ γ1 γ2β φ2
[Wang, McCallum 2005]
![Page 37: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/37.jpg)
LDA Topic
LDA
algorithmsalgorithmgenetic
problemsefficient
Topical N-grams
genetic algorithmsgenetic algorithm
evolutionary computationevolutionary algorithms
fitness function
![Page 38: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/38.jpg)
Topic Comparison
learningoptimalreinforcementstateproblemspolicydynamicactionprogrammingactionsfunctionmarkovmethodsdecisionrlcontinuousspacessteppoliciesplanning
LDAreinforcement learningoptimal policydynamic programmingoptimal controlfunction approximatorprioritized sweepingfinite-state controllerlearning systemreinforcement learning rlfunction approximatorsmarkov decision problemsmarkov decision processeslocal searchstate-action pairmarkov decision processbelief statesstochastic policyaction selectionupright positionreinforcement learning methods
policyactionstatesactionsfunctionrewardcontrolagentq-learningoptimalgoallearningspacestepenvironmentsystemproblemstepssuttonpolicies
Topical N-grams (2) Topical N-grams (1)
![Page 39: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/39.jpg)
Topic Comparison
motionvisualfieldpositionfiguredirectionfieldseyelocationretinareceptivevelocityvisionmovingsystemflowedgecenterlightlocal
LDAreceptive fieldspatial frequencytemporal frequencyvisual motionmotion energytuning curveshorizontal cellsmotion detectionpreferred directionvisual processingarea mtvisual cortexlight intensitydirectional selectivityhigh contrastmotion detectorsspatial phasemoving stimulidecision strategyvisual stimuli
motionresponsedirectioncellsstimulusfigurecontrastvelocitymodelresponsesstimulimovingcellintensitypopulationimagecentertuningcomplexdirections
Topical N-grams (2) Topical N-grams (1)
![Page 40: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/40.jpg)
Topic Comparison
wordsystemrecognitionhmmspeechtrainingperformancephonemewordscontextsystemsframetrainedspeakersequencespeakersmlpframessegmentationmodels
LDAspeech recognitiontraining dataneural networkerror ratesneural nethidden markov modelfeature vectorscontinuous speechtraining procedurecontinuous speech recognitiongamma filterhidden controlspeech productionneural netsinput representationoutput layerstraining algorithmtest setspeech framesspeaker dependent
speechwordtrainingsystemrecognitionhmmspeakerperformancephonemeacousticwordscontextsystemsframetrainedsequencephoneticspeakersmlphybrid
Topical N-grams (2) Topical N-grams (1)
![Page 41: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/41.jpg)
Unsupervised learning oftopic hierarchies
(Blei, Griffiths, Jordan & Tenenbaum, NIPS 2003)
![Page 42: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/42.jpg)
Joint models of syntax and semantics (Griffiths,Steyvers, Blei & Tenenbaum, NIPS 2004)
• Embed topics model inside an nth orderHidden Markov Model:
Document-specific distribution over topics
![Page 43: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/43.jpg)
FOODFOODSBODY
NUTRIENTSDIETFAT
SUGARENERGY
MILKEATINGFRUITS
VEGETABLESWEIGHT
FATSNEEDS
CARBOHYDRATESVITAMINSCALORIESPROTEIN
MINERALS
MAPNORTHEARTHSOUTHPOLEMAPS
EQUATORWESTLINESEAST
AUSTRALIAGLOBEPOLES
HEMISPHERELATITUDE
PLACESLAND
WORLDCOMPASS
CONTINENTS
DOCTORPATIENTHEALTH
HOSPITALMEDICAL
CAREPATIENTS
NURSEDOCTORSMEDICINENURSING
TREATMENTNURSES
PHYSICIANHOSPITALS
DRSICK
ASSISTANTEMERGENCY
PRACTICE
BOOKBOOKS
READINGINFORMATION
LIBRARYREPORT
PAGETITLE
SUBJECTPAGESGUIDE
WORDSMATERIALARTICLE
ARTICLESWORDFACTS
AUTHORREFERENCE
NOTE
GOLDIRON
SILVERCOPPERMETAL
METALSSTEELCLAYLEADADAM
OREALUMINUM
MINERALMINE
STONEMINERALS
POTMININGMINERS
TIN
BEHAVIORSELF
INDIVIDUALPERSONALITY
RESPONSESOCIAL
EMOTIONALLEARNINGFEELINGS
PSYCHOLOGISTSINDIVIDUALS
PSYCHOLOGICALEXPERIENCES
ENVIRONMENTHUMAN
RESPONSESBEHAVIORSATTITUDES
PSYCHOLOGYPERSON
CELLSCELL
ORGANISMSALGAE
BACTERIAMICROSCOPEMEMBRANEORGANISM
FOODLIVINGFUNGIMOLD
MATERIALSNUCLEUSCELLED
STRUCTURESMATERIAL
STRUCTUREGREENMOLDS
Semantic classes
PLANTSPLANT
LEAVESSEEDSSOIL
ROOTSFLOWERS
WATERFOOD
GREENSEED
STEMSFLOWER
STEMLEAF
ANIMALSROOT
POLLENGROWING
GROW
![Page 44: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/44.jpg)
GOODSMALL
NEWIMPORTANT
GREATLITTLELARGE
*BIG
LONGHIGH
DIFFERENTSPECIAL
OLDSTRONGYOUNG
COMMONWHITESINGLE
CERTAIN
THEHIS
THEIRYOURHERITSMYOURTHIS
THESEA
ANTHATNEW
THOSEEACH
MRANYMRSALL
MORESUCHLESS
MUCHKNOWN
JUSTBETTERRATHER
GREATERHIGHERLARGERLONGERFASTER
EXACTLYSMALLER
SOMETHINGBIGGERFEWERLOWER
ALMOST
ONAT
INTOFROMWITH
THROUGHOVER
AROUNDAGAINSTACROSS
UPONTOWARDUNDERALONGNEAR
BEHINDOFF
ABOVEDOWN
BEFORE
SAIDASKED
THOUGHTTOLDSAYS
MEANSCALLEDCRIEDSHOWS
ANSWEREDTELLS
REPLIEDSHOUTED
EXPLAINEDLAUGHED
MEANTWROTE
SHOWEDBELIEVED
WHISPERED
ONESOMEMANYTWOEACHALL
MOSTANY
THREETHIS
EVERYSEVERAL
FOURFIVE
BOTHTENSIX
MUCHTWENTY
EIGHT
HEYOUTHEY
ISHEWEIT
PEOPLEEVERYONE
OTHERSSCIENTISTSSOMEONE
WHONOBODY
ONESOMETHING
ANYONEEVERYBODY
SOMETHEN
Syntactic classes
BEMAKEGET
HAVEGO
TAKEDO
FINDUSESEE
HELPKEEPGIVELOOKCOMEWORKMOVELIVEEAT
BECOME
![Page 45: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/45.jpg)
Corpus-specific factorization(NIPS)
Sem
antic
sSy
ntax
![Page 46: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/46.jpg)
REMAINED
5 8 14 25 26 30 33IN ARE THE SUGGEST LEVELS RESULTS BEEN
FOR WERE THIS INDICATE NUMBER ANALYSIS MAYON WAS ITS SUGGESTING LEVEL DATA CAN
BETWEEN IS THEIR SUGGESTS RATE STUDIES COULDDURING WHEN AN SHOWED TIME STUDY WELLAMONG REMAIN EACH REVEALED CONCENTRATIONS FINDINGS DIDFROM REMAINS ONE SHOW VARIETY EXPERIMENTS DOES
UNDER REMAINED ANY DEMONSTRATE RANGE OBSERVATIONS DOWITHIN PREVIOUSLY INCREASED INDICATING CONCENTRATION HYPOTHESIS MIGHT
THROUGHOUT BECOME EXOGENOUS PROVIDE DOSE ANALYSES SHOULDTHROUGH BECAME OUR SUPPORT FAMILY ASSAYS WILLTOWARD BEING RECOMBINANT INDICATES SET POSSIBILITY WOULD
INTO BUT ENDOGENOUS PROVIDES FREQUENCY MICROSCOPY MUSTAT GIVE TOTAL INDICATED SERIES PAPER CANNOT
INVOLVING MERE PURIFIED DEMONSTRATED AMOUNTS WORK
THEYAFTER APPEARED TILE SHOWS RATES EVIDENCE ALSO
ACROSS APPEAR FULL SO CLASS FINDINGAGAINST ALLOWED CHRONIC REVEAL VALUES MUTAGENESIS BECOME
WHEN NORMALLY ANOTHER DEMONSTRATES AMOUNT OBSERVATION MAGALONG EACH EXCESS SUGGESTED SITES MEASUREMENTS LIKELY
Syntactic classes in PNAS
![Page 47: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/47.jpg)
Semantic highlighting Darker words are more likely to have been generated from the topic-based “semantics” module:
![Page 48: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/48.jpg)
Semantic Role Labeling (SRL)
• Characterize clauses as relations with roles:
• Want to more than which NP is the subject (but not much more):• Relations like subject are syntactic, relations like agent or message are
semantic• Typical pipeline:
– Parse, then label roles– Almost all errors locked in by parser– Really, SRL is quite a lot easier than parsing
![Page 49: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/49.jpg)
SRL Example
![Page 50: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/50.jpg)
PropBank / FrameNet
• FrameNet: roles shared between verbs• PropBank: each verb has it’s own roles• PropBank more used, because it’s layered over the treebank (and so has
greater coverage, plus parses)• Note: some linguistic theories postulate even fewer roles than FrameNet
(e.g. 5-20 total: agent, patient, instrument, etc.)
![Page 51: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/51.jpg)
PropBank Example
![Page 52: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/52.jpg)
PropBank Example
![Page 53: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/53.jpg)
PropBank Example
![Page 54: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/54.jpg)
Shared Arguments
![Page 55: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/55.jpg)
Path Features
![Page 56: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/56.jpg)
Results
• Features:– Path from target to filler– Filler’s syntactic type, headword, case– Target’s identity– Sentence voice, etc.– Lots of other second-order features
• Gold vs parsed source trees
– SRL is fairly easy on gold trees
– Harder on automatic parses
![Page 57: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/57.jpg)
Outline
• Role Discovery (Author-Recipient-Topic Model, ART)
• Group Discovery (Group-Topic Model, GT)
• Enhanced Topic Models– Correlations among Topics (Pachinko Allocation, PAM)
– Time Localized Topics (Topics-over-Time Model, TOT)
– Markov Dependencies in Topics (Topical N-Grams Model, TNG)
• Bibliometric Impact Measures enabled by Topics
Social Network Analysis with Topic Models
Multi-Conditional Mixtures
![Page 58: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/58.jpg)
Groups and Topics
• Input:– Observed relations between people– Attributes on those relations (text, or categorical)
• Output:– Attributes clustered into “topics”– Groups of people---varying depending on topic
![Page 59: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/59.jpg)
Discovering Groups fromObserved Set of Relations
Admiration relations among six high school students.
Student Roster
AdamsBennettCarterDavisEdwardsFrederking
Academic Admiration
Acad(A, B) Acad(C, B)Acad(A, D) Acad(C, D)Acad(B, E) Acad(D, E)Acad(B, F) Acad(D, F)Acad(E, A) Acad(F, A)Acad(E, C) Acad(F, C)
![Page 60: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/60.jpg)
Adjacency Matrix Representing Relations
FEDCBA
FEDCBAFEDCBA
G3G3G2G1G2G1
G3G3G2G1G2G1
FEDCBA
FEDBCA
G3G3G2G2G1G1
G3G3G2G2G1G1
FEDBCA
Student Roster
AdamsBennettCarterDavisEdwardsFrederking
Academic Admiration
Acad(A, B) Acad(C, B)Acad(A, D) Acad(C, D)Acad(B, E) Acad(D, E)Acad(B, F) Acad(D, F)Acad(E, A) Acad(F, A)Acad(E, C) Acad(F, C)
![Page 61: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/61.jpg)
Group Model:Partitioning Entities into Groups
2S
v
!
2G
! ! !
Stochastic Blockstructures for Relations[Nowicki, Snijders 2001]
S: number of entities
G: number of groups
Enhanced with arbitrary number of groups in [Kemp, Griffiths, Tenenbaum 2004]
BetaDirichlet
Binomial
S
gMultinomial
![Page 62: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/62.jpg)
Two Relations with Different Attributes
FEDBCA
G3G3G2G2G1G1
G3G3G2G2G1G1FDBECA
G2G2G2G1G1G1
G2G2G2G1G1G1
FDBECA
Student Roster
AdamsBennettCarterDavisEdwardsFrederking
Academic Admiration
Acad(A, B) Acad(C, B)Acad(A, D) Acad(C, D)Acad(B, E) Acad(D, E)Acad(B, F) Acad(D, F)Acad(E, A) Acad(F, A)Acad(E, C) Acad(F, C)
Social Admiration
Soci(A, B) Soci(A, D) Soci(A, F)Soci(B, A) Soci(B, C) Soci(B, E)Soci(C, B) Soci(C, D) Soci(C, F)Soci(D, A) Soci(D, C) Soci(D, E)Soci(E, B) Soci(E, D) Soci(E, F)Soci(F, A) Soci(F, C) Soci(F, E)
FEDBCA
![Page 63: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/63.jpg)
The Group-Topic Model:Discovering Groups and Topics Simultaneously
bN
w
t
B
T
!
!
DirichletMultinomial
Uniform
2S
v
!
2G
! ! !
BetaDirichlet
Binomial
S
gMultinomial
T
[Wang, Mohanty, McCallum 2006]
![Page 64: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/64.jpg)
Inference and EstimationGibbs Sampling:- Many r.v.s can beintegrated out- Easy to implement- Reasonably fast
We assume the relationship is symmetric.
![Page 65: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/65.jpg)
Dataset #1:U.S. Senate
• 16 years of voting records in the US Senate (1989 – 2005)
• a Senator may respond Yea or Nay to a resolution
• 3423 resolutions with text attributes (index terms)
• 191 Senators in total across 16 yearsS.543Title: An Act to reform Federal deposit insurance, protect the deposit insurancefunds, recapitalize the Bank Insurance Fund, improve supervision and regulationof insured depository institutions, and for other purposes.Sponsor: Sen Riegle, Donald W., Jr. [MI] (introduced 3/5/1991) Cosponsors (2)Latest Major Action: 12/19/1991 Became Public Law No: 102-242.Index terms: Banks and banking Accounting Administrative fees Cost controlCredit Deposit insurance Depressed areas and other 110 terms
Adams (D-WA), Nay Akaka (D-HI), Yea Bentsen (D-TX), Yea Biden (D-DE), YeaBond (R-MO), Yea Bradley (D-NJ), Nay Conrad (D-ND), Nay ……
![Page 66: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/66.jpg)
Topics Discovered (U.S. Senate)
carepolicypollutionpreventionemployeelawresearchelementarybusinessaidpetrolstudents
taxcongressgasdrugaidtaxnuclearchildren
insuranceforeignwateraidlabormilitarypowerschool
federalgovernmentenergyeducation
EconomicMilitaryMisc.EnergyEducation
Mixture of Unigrams
Group-Topic Model
assistancebusinessdiseasesresearchdisabilitywagecommunicableenergymedicareminimumdrugstax
careincomecongressgovernmentmedicalcongresstariffaid
insurancetaxchemicalsfederalsecurityinsurancetradeschoolsociallaborforeigneducation
Social Security+ Medicare
EconomicForeignEducation+ Domestic
![Page 67: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/67.jpg)
Groups Discovered (US Senate)
Groups from topic Education + Domestic
![Page 68: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/68.jpg)
Senators Who Change Coalition the mostDependent on Topic
e.g. Senator Shelby (D-AL) votes with the Republicans on Economicwith the Democrats on Education + Domesticwith a small group of maverick Republicans on Social Security + Medicaid
![Page 69: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/69.jpg)
Dataset #2:The UN General Assembly
• Voting records of the UN General Assembly (1990 - 2003)
• A country may choose to vote Yes, No or Abstain
• 931 resolutions with text attributes (titles)
• 192 countries in total
• Also experiments later with resolutions from 1960-2003
Vote on Permanent Sovereignty of Palestinian People, 87th plenary meeting
The draft resolution on permanent sovereignty of the Palestinian people in theoccupied Palestinian territory, including Jerusalem, and of the Arab population inthe occupied Syrian Golan over their natural resources (document A/54/591)was adopted by a recorded vote of 145 in favour to 3 against with 6 abstentions:
In favour: Afghanistan, Argentina, Belgium, Brazil, Canada, China, France,Germany, India, Japan, Mexico, Netherlands, New Zealand, Pakistan, Panama,Russian Federation, South Africa, Spain, Turkey, and other 126 countries.Against: Israel, Marshall Islands, United States.Abstain: Australia, Cameroon, Georgia, Kazakhstan, Uzbekistan, Zambia.
![Page 70: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/70.jpg)
Topics Discovered (UN)
callsisraelcountriessecuritysituationimplementation
syriapalestineuseisraelhumanweapons
occupiedrightsnuclear
Securityin Middle East
Human RightsEverythingNuclear
Mixture ofUnigrams
Group-TopicModel
israelspacenationsoccupiedraceweaponspalestinepreventionunitedhumanarmsstatesrightsnuclearnuclear
Human RightsNuclear ArmsRace
NuclearNon-proliferation
![Page 71: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/71.jpg)
GroupsDiscovered(UN)The countries list for eachgroup are ordered by their2005 GDP (PPP) and only 5countries are shown ingroups that have more than5 members.
![Page 72: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/72.jpg)
Groups and Topics, Trends over Time (UN)
![Page 73: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/73.jpg)
Outline
• Role Discovery (Author-Recipient-Topic Model, ART)
• Group Discovery (Group-Topic Model, GT)
• Enhanced Topic Models– Correlations among Topics (Pachinko Allocation, PAM)
– Time Localized Topics (Topics-over-Time Model, TOT)
– Markov Dependencies in Topics (Topical N-Grams Model, TNG)
• Bibliometric Impact Measures enabled by Topics
Social Network Analysis with Topic Models
Multi-Conditional Mixtures
![Page 74: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/74.jpg)
Social Networks in Research Literature
• Better understand structure of our ownresearch area.
• Structure helps us learn a new field.• Aid collaboration• Map how ideas travel through social networks
of researchers.
• Aids for hiring and finding reviewers!
![Page 75: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/75.jpg)
Traditional Bibliometrics
• Analyses a small amount of data(e.g. 19 articles from a single issue of a journal)
• Uses “journal” as a proxy for “research topic”(but there is no journal for information extraction)
• Uses impact measures almost exclusivelybased on simple citation counts.
How can we use topic models to create new, interesting impact measures?
![Page 76: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/76.jpg)
Our Data
• Over 1 million research papers,gathered as part of Rexa.info portal.
• Cross linked references / citations.
![Page 77: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/77.jpg)
Finding Topics with TNGTraditional unigram LDA
run on 1 milliontitles / abstracts
(200 topics)
...select ~300k papers onML, NLP, robotics, vision...
Find 200 TNG topics among those papers.
![Page 78: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/78.jpg)
Topical Bibliometric Impact Measures
• Topical Citation Counts
• Topical Impact Factors
• Topical Longevity
• Topical Diversity
• Topical Precedence
• Topical Transfer
[Mann, Mimno, McCallum, 2006]
![Page 79: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/79.jpg)
Topical DiversityEntropy of the topic distribution among
papers that cite this paper (this topic).
LowDiversity
HighDiversity
![Page 80: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/80.jpg)
Topical Diversity
Can also be measured on particular papers...
![Page 81: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/81.jpg)
Topical PrecedenceWithin a topic, what are the earliest papers that received more than n citations?
“Early-ness”
Information Retrieval:
On Relevance, Probabilistic Indexing and Information Retrieval,Kuhns and Maron (1960)
Expected Search Length: A Single Measure of Retrieval Effectiveness Basedon the Weak Ordering Action of Retrieval Systems,
Cooper (1968)Relevance feedback in information retrieval,
Rocchio (1971)Relevance feedback and the optimization of retrieval effectiveness,
Salton (1971)New experiments in relevance feedback,
Ide (1971)Automatic Indexing of a Sound Database Using Self-organizing Neural Nets,
Feiten and Gunzel (1982)
![Page 82: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/82.jpg)
Topical PrecedenceWithin a topic, what are the earliest papers that received more than n citations?
“Early-ness”
Speech Recognition:
Some experiments on the recognition of speech, with one and two ears,E. Colin Cherry (1953)
Spectrographic study of vowel reduction,B. Lindblom (1963)
Automatic Lipreading to enhance speech recognition, Eric D. Petajan (1965)
Effectiveness of linear prediction characteristics of the speech wave for...,B. Atal (1974)
Automatic Recognition of Speakers from Their Voices,B. Atal (1976)
![Page 83: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/83.jpg)
Topical Transfer
Transfer from Digital Libraries to other topics
WebBase: a repository of Web pages11Web Pages
Trawling the Web for Emerging Cyber-Communities
12Graphs
Lessons learned from the creation anddeployment of a terabyte digital video
12Video
On being ‘Undigital’ with digital cameras:extending the dynamic...
14Computer Vision
Trawling the Web for Emerging Cyber-Communities, Kumar, Raghavan,... 1999.
31Web Pages
Paper TitleCit’sOther topic
![Page 84: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/84.jpg)
Topical TransferCitation counts from one topic to another.
Map “producers and consumers”
![Page 85: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/85.jpg)
Outline
• Role Discovery (Author-Recipient-Topic Model, ART)
• Group Discovery (Group-Topic Model, GT)
• Enhanced Topic Models– Correlations among Topics (Pachinko Allocation, PAM)
– Time Localized Topics (Topics-over-Time Model, TOT)
– Markov Dependencies in Topics (Topical N-Grams Model, TNG)
• Bibliometric Impact Measures enabled by Topics
Social Network Analysis with Topic Models
Multi-Conditional Mixtures
![Page 86: Ambiguity, Statistical Word Sense Discovery and …people.cs.umass.edu/~mccallum/talks/potts-semantics2006.pdf · Statistical Word Sense Discovery and Semantic Role Labeling ... –“Susan](https://reader031.vdocuments.site/reader031/viewer/2022030423/5aaad0e87f8b9a72188ec501/html5/thumbnails/86.jpg)
Topic Model Musings
• 3 years ago Latent Dirichlet Allocationappeared as a complex innovation...but now these methods & mechanics arewell-understood.
• Innovation now is to understanddata and modeling needs,how to structure a new model to capture these.