natural logic and natural language inference bill maccartney stanford university / google, inc. 8...
TRANSCRIPT
Natural Logic and Natural Language Natural Logic and Natural Language InferenceInference
Bill MacCartneyStanford University / Google, Inc.
8 April 2011
2
Two disclaimersTwo disclaimers
• The work I present today isn’t exactly fresh• Essentially, it’s my dissertation work from 2009• I hope it can usefully provide context for more recent
work
• I’m a computer scientist, not a semanticist or a logician• Consequently, I emphasize pragmatism over rigor
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
3
Natural language inference (NLI)Natural language inference (NLI)
• Aka recognizing textual ‘entailment’ (RTE)
• Does premise P justify an inference to hypothesis H?• An informal, intuitive notion of inference: not strict logic• Emphasis on variability of linguistic expression
• Necessary to goal of natural language understanding (NLU)
• Can also enable semantic search, question answering, …
P Every firm polled saw costs grow more than expected,even after adjusting for inflation.
H Every big company in the poll reported cost increases.yes
Some
Some no
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
4
NLI: a spectrum of approachesNLI: a spectrum of approaches
lexical/semanticoverlap
Jijkoun & de Rijke 2005
patternedrelation
extraction
Romano et al. 2006
semanticgraph
matching
MacCartney et al. 2006Hickl et al. 2006
FOL &theoremproving
Bos & Markert 2006
robust,but shallow
deep,but brittle
naturallogic
(this work)
Problem:imprecise easily confounded by negation, quantifiers, conditionals, factive & implicative verbs, etc.
Problem:hard to translate NL to FOLidioms, anaphora, ellipsis, intensionality, tense, aspect, vagueness, modals, indexicals, reciprocals, propositional attitudes, scope ambiguities, anaphoric adjectives, non-intersective adjectives, temporal & causal relations, unselective quantifiers, adverbs of quantification, donkey sentences, generic determiners, comparatives, phrasal verbs, …
Solution?
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
5
What is natural logic?What is natural logic? ( ( natural deduction) natural deduction)
• Characterizes valid patterns of inference via surface forms• precise, yet sidesteps difficulties of translating to FOL
• A long history• traditional logic: Aristotle’s syllogisms, scholastics, Leibniz, …• the term “natural logic” was introduced by Lakoff (1970)• van Benthem & Sánchez Valencia (1986-91): monotonicity
calculus• Nairn et al. (2006): an account of implicatives & factives
• We introduce a new theory of natural logic• extends monotonicity calculus to account for negation &
exclusion• incorporates elements of Nairn et al.’s model of implicatives
• …and implement & evaluate a computational model of it
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
6
‘‘Entailment’ relations in past Entailment’ relations in past workwork
X is a man
X is a woman
X is a hippo
X is hungry
X is a fish
X is a carp
X is a crow
X is a bird
X is a couch
X is a sofa
Yesentailment
Nonon-entailment
2-wayRTE1,2,3
Yesentailment
Nocontradiction
Unknowncompatibility
3-wayFraCaS,
PARC, RTE4
P = Qequivalence
P < Qforward
entailment
P > Qreverse
entailment
P # Qnon-entailment
containmentSánchez-Valencia
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
7
16 elementary set relations16 elementary set relations
? ?
? ?
y
x
x
y
Assign sets x, y to one of 16 relations, depending on emptiness or non-emptiness of each of four partitions
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
empty
non-empty
8
16 elementary set relations16 elementary set relations
x ^ y x ‿ y
x y x ⊐ y
x ⊏ y x | y x # y
But 9 of 16 are degenerate: either x or y is either empty or universal.
I.e., they correspond to semantically vacuous expressions, which are rare outside logic textbooks.
We therefore focus on the remaining seven relations.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
9
The set of 7 basic entailment The set of 7 basic entailment relationsrelations
Venn symbol
name example
x y equivalence couch sofa
x ⊏ y forward entailment(strict)
crow ⊏ bird
x ⊐ y reverse entailment(strict)
European ⊐ French
x ^ y negation(exhaustive exclusion)
human ^ nonhuman
x | y alternation(non-exhaustive exclusion)
cat | dog
x ‿ y cover(exhaustive non-exclusion)
animal ‿ nonhuman
x # y independence hungry # hippo
Relations are defined for all semantic types: tiny ⊏ small, hover ⊏ fly, kick ⊏ strike,this morning ⊏ today, in Beijing ⊏ in China, everyone ⊏ someone, all ⊏ most ⊏ some
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
10
|
x R y
Joining entailment relationsJoining entailment relations
fish human nonhuman^
y zS
?
?
⋈
⊏ ⋈ ⊏ ⊏
⊐ ⋈ ⊐ ⊐
^ ⋈ ^
R ⋈ R
⋈ R R
⊏
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
11
Some joins yield unions of Some joins yield unions of relations!relations!
x | y y | z x ? z
couch | table table | sofa couch sofa
pistol | knife knife | gun pistol ⊏ gun
dog | cat cat | terrier dog ⊐ terrier
rose | orchid orchid | daisy rose | daisy
woman | frog frog | Eskimo woman # Eskimo
What is | | ?⋈
| | {, ⊏, ⊐, |, #}⋈
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
12
Of 49 join pairs, 32 yield relations in ; 17 yield unions
Larger unions convey less information — limits power of inference
In practice, any union which contains # can be approximated by #
The complete join tableThe complete join table
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
13
will depend on:• the lexical entailment relation generated by e:
(e)• other properties of the context x in which e is
applied
( , )
Lexical entailment relationsLexical entailment relations
x e(x)
compound expression
atomic edit: DEL, INS, SUB
entailment relation
Example: suppose x is red car
If e is SUB(car, convertible), then (e) is ⊐If e is DEL(red), then (e) is ⊏
Crucially, (e) depends solely on lexical items in e, independent of context x
But how are lexical entailment relations determined?
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
14
Lexical entailment relations: Lexical entailment relations: SUBsSUBs
(SUB(x, y)) = (x, y)
For open-class terms, use lexical resource (e.g. WordNet)for synonyms: sofa couch, forbid prohibit
⊏for hypo-/hypernyms: crow ⊏ bird, frigid ⊏ cold, soar ⊏ rise
| for antonyms and coordinate terms: hot | cold, cat | dog
or | for proper nouns: USA United States, JFK | FDR
# for most other pairs: hungry # hippo
Closed-class terms may require special handlingQuantifiers: all ⊏ some, some ^ no, no | all, at least 4 ‿ at most
6
See paper for discussion of pronouns, prepositions, …
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
15
Lexical entailment relations: DEL Lexical entailment relations: DEL & INS& INS
Generic (default) case: (DEL(•)) = ⊏, (INS(•)) = ⊐• Examples: red car ⊏ car, sing ⊐ sing off-key• Even quite long phrases: car parked outside since last week ⊏ car• Applies to intersective modifiers, conjuncts, independent
clauses, …• This heuristic underlies most approaches to RTE!• Does P subsume H? Deletions OK; insertions penalized.
Special cases• Negation: didn’t sleep ^ did sleep• Implicatives & factives (e.g. refuse to, admit that): discussed
later• Non-intersective adjectives: former spy | spy, alleged spy # spy• Auxiliaries etc.: is sleeping sleeps, did sleep slept
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
16
The impact of semantic The impact of semantic compositioncomposition
How are entailment relations affected by semantic composition?
f
@
f
@
x y
?
The monotonicity calculus provides a partial answer UP
⊏ ⊏⊐ ⊐# #
DOWN ⊏ ⊐⊐ ⊏# #
NON ⊏ #⊐ ## #
If f has monotonicity…
How is (x, y) projected by f?
But how are other relations (|, ^, ‿) projected?
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
@ means fn application[ ]
negation ⊏ ⊐⊐ ⊏^ ^| ‿‿ |# #
not happy not glad
isn’t swimming # isn’t hungry
17
A typology of projectivityA typology of projectivityProjectivity signatures: a generalization of monotonicity classes
not French ‿ not Germannot more than 4 | not less than 6
not human ^ not nonhuman
didn’t kiss ⊐ didn’t touchnot ill ⊏ not seasick
In principle, 77 possible signatures, but few actually realized
↦Each projectivity signature is a map
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
intersectivemodification
⊏ ⊏⊐ ⊐^ || |‿ ## #
18
A typology of projectivityA typology of projectivityProjectivity signatures: a generalization of monotonicity
classesEach projectivity signature is a mapIn principle, 77 possible signatures, but few actually realized
↦
negation
⊏ ⊐⊐ ⊏^ ^| ‿‿ |# #
metallic pipe # nonferrous pipe
live human | live nonhumanFrench wine | Spanish wine
See my disseration for projectivity of various quantifiers, verbs
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
19
Projecting through multiple Projecting through multiple levelslevels
⊏
⊏
⊐
⊐
⊐
a shirtnobody can without enter
@
@
@
@
clothesnobody can without enter
@
@
@
@
Propagate entailment relation between atoms upward, according to projectivity class of each node on path to root
nobody can enter without a shirt ⊏ nobody can enter without clothes
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
20
Implicatives & factives Implicatives & factives [Nairn et al. 06][Nairn et al. 06]
signature
example
implicatives
+ / – he managed to escape
+ / o he was forced to sell
o / – he was permitted to live
implicatives
– / + he forgot to pay
– / o he refused to fight
o / + he hesitated to ask
factives + / + he admitted that he knew
– / – he pretended he was sick
o / o he wanted to fly
9 signatures, per implications (+, –, or o) in positive and negative contexts
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
21
Implicatives & factivesImplicatives & factives
signature
example(DEL
)(INS)
implicatives
+ / – he managed to escape he escaped
+ / o he was forced to sell ⊏ he sold ⊏ ⊐
o / – he was permitted to live ⊐ he lived ⊐ ⊏
implicatives
– / + he forgot to pay ^ he paid ^ ^
– / o he refused to fight | he fought | |
o / + he hesitated to ask ‿ he asked ‿ ‿
factives + / + he admitted that he knew ⊏ he knew ⊏ ⊐
– / – he pretended he was sick | he was sick | |
o / o he wanted to fly # he flew # #
We can specify relation generated by DEL or INS of each signature
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Room for variation w.r.t. infinitives, complementizers, passivation, etc.Some more intuitive when negated: he didn’t hesitate to ask | he didn’t askFactives not fully explained: he didn’t admit that he knew | he didn’t know
22
Putting it all togetherPutting it all together
1. Find a sequence of edits e1, …, en which transforms p into h. Define x0 = p, xn = h, and xi = ei(xi–1) for i [1, n].
2. For each atomic edit ei:
1. Determine the lexical entailment relation (ei).
2. Project (ei) upward through the semantic composition tree of expression xi–1 to find the atomic entailment relation (xi–1, xi)
3. Join atomic entailment relations across the sequence of edits:(p, h) = (x0, xn) = (x0, x1) ⋈ … ⋈ (xi–1, xi) ⋈ … ⋈ (xn–1, xn)
Limitations: need to find appropriate edit sequence connecting p and h;tendency of ⋈ operation toward less-informative entailment relations; lack of general mechanism for combining multiple premises
Less deductive power than FOL. Can’t handle e.g. de Morgan’s Laws.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
23
An exampleAn example
P The doctor didn’t hesitate to recommend Prozac.
H The doctor recommended medication.yes
i ei xi lex atom join
The doctor didn’t hesitate to recommend Prozac.
1 DEL(hesitate to)The doctor didn’t recommend Prozac.
2 DEL(didn’t)The doctor recommended Prozac.
3 SUB(Prozac, medication)The doctor recommended medication.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
‿ ||
^^ ⊏
⊏ ⊏ ⊏ yes
24
Different edit orders?Different edit orders?i ei lex atom join
1 DEL(hesitate to) ‿ | |
2 DEL(didn’t) ^ ^ ⊏
3 SUB(Prozac, medication) ⊏ ⊏ ⊏
i ei lex atom join
1 DEL(didn’t) ^ ^ ^
2 DEL(hesitate to) ‿ ‿ ⊏
3 SUB(Prozac, medication) ⊏ ⊏ ⊏
i ei lex atom join
1 SUB(Prozac, medication) ⊏ ⊏ ⊏
2 DEL(hesitate to) ‿ | |
3 DEL(didn’t) ^ ^ ⊏
i ei lex atom join
1 DEL(hesitate to) ‿ | |
2 SUB(Prozac, medication) ⊏ ⊐ |
3 DEL(didn’t) ^ ^ ⊏
i ei lex atom join
1 DEL(didn’t) ^ ^ ^
2 SUB(Prozac, medication) ⊏ ⊐ |
3 DEL(hesitate to) ‿ ‿ ⊏
i ei lex atom join
1 SUB(Prozac, medication) ⊏ ⊏ ⊏
2 DEL(didn’t) ^ ^ |
3 DEL(hesitate to) ‿ ‿ ⊏
Intermediate steps may vary; final result is typically (though not necessarily) the same
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
25
Implementation & evaluationImplementation & evaluation
The NatLog system: an implementation of this model in codeFor implementation details, see [MacCartney & Manning 2008]
Evaluation on FraCaS test suite183 NLI problems, nine sections, three-way classificationAccuracy 70% overall; 87% on “relevant” sections (60% coverage)Precision 89% overall: rarely predicts entailment wrongly
Evaluation on RTE3 test suiteLonger, more natural premises; greater diversity of inference typesNatLog alone has mediocre accuracy (59%) but good precisionHybridization with broad-coverage RTE system yields gains of 4%
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
Natural logic is not a universal solution for NLIMany types of inference not amenable to natural logic approachOur inference method faces many limitations on deductive
power
More work to be done in fleshing out our accountEstablishing projectivity signatures for more quantifiers, verbs,
etc.Better incorporating presuppositions
But, our model of natural logic fills an important nichePrecise reasoning on negation, antonymy, quantifiers,
implicatives, …Sidesteps the myriad difficulties of full semantic interpretationPractical value demonstrated on FraCaS and RTE3 test suites
26
ConclusionsConclusions
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
:-) Thanks! Questions?
27
Backup slides followBackup slides follow
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
28
An example involving exclusionAn example involving exclusion
P Stimpy is a cat.
H Stimpy is not a poodle. yes
i ei xi lex atom join
Stimpy is a cat.
1 SUB(cat, dog)Stimpy is a dog.
2 INS(not)Stimpy is not a dog.
3 SUB(dog, poodle)Stimpy is not a poodle.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
| ||
^^ ⊏
⊐ ⊏ ⊏ yes
29
An example involving an An example involving an implicativeimplicative
P We were not permitted to smoke.
H We smoked Cuban cigars. no
i ei xi lex atom join
We were not permitted to smoke.
1 DEL(permitted to)We did not smoke.
2 DEL(not)We smoked.
3 INS(Cuban cigars)We smoked Cuban cigars.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
⊐ ⊏⊏
^^ |
⊐ ⊐ | no
30
de Morgan’s Laws for quantifiersde Morgan’s Laws for quantifiers
P Not all birds fly.
H Some birds do not fly. yes
i ei xi lex atom join
Not all birds fly.
1 DEL(not)All birds fly.
2 SUB(all, some)Some birds fly.
3 INS(not)Some birds do not fly.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
^ ^^
⊏⊏ ‿
^ ‿ ⊏⊐‿#wtf??
31
de Morgan’s Laws for quantifiers de Morgan’s Laws for quantifiers (2)(2)
P Not all birds fly.
H Some birds do not fly. yes
i ei xi lex atom join
Not all birds fly.
1 DEL(not)All birds fly.
2 INS(not)All birds do not fly.
3 SUB(all, some)Some birds do not fly.
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
^ ^^
|^ ⊐
⊏ ⊏ ⊏⊐‿#wtf??
32
A more complex exampleA more complex example
P Jimmy Dean refused to move without blue jeans.
H James Dean didn’t dance without pants. yes
i ei lex atom join
1 SUB(Jimmy Dean, James Dean)
2 DEL(refuse to) | | |
3 INS(did) |
4 INS(n’t) ^ ^ ⊏
5 SUB(move, dance) ⊐ ⊏ ⊏
6 DEL(blue) ⊏ ⊏ ⊏
7 SUB(jeans, pants) ⊏ ⊏ ⊏
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
33
A more complex example (2)A more complex example (2)
P Jimmy Dean refused to move without blue jeans.
H James Dean didn’t dance without pants. yes
i ei lex atom join
1 INS(did)
2 INS(n’t) ^ ^ ^
3 DEL(blue) ⊏ ⊐ |
4 SUB(jeans, pants) ⊏ ⊐ |
5 SUB(move, dance) ⊐ ⊐ |
6 DEL(refuse to) | ‿ ⊏
7 SUB(Jimmy Dean, James Dean) ⊏
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion
34
A more complex example (3)A more complex example (3)
P Jimmy Dean refused to move without blue jeans.
H James Dean didn’t dance without pants. yes
i ei lex atom join
1 INS(did)
2 INS(n’t) ^ | |
6 DEL(refuse to) | | ⊏⊐|#
3 DEL(blue) ⊏ ⊏ •
4 SUB(jeans, pants) ⊏ ⊏ •
5 SUB(move, dance) ⊐ ⊐ •
7 SUB(Jimmy Dean, James Dean) •
Introduction • Entailment Relations • Joins • Lexical Relations • Projectivity • Implicatives • Inference • Evaluation • Conclusion