Empirical Evaluation of Pronoun Resolution and Clausal Structure
Joel Tetreault and James AllenUniversity of Rochester
Department of Computer Science
RST and pronoun resolution Previous work suggests that breaking apart
utterances into clauses (Kameyama 1998), or assigning a hierarchical structure (Grosz and Sidner, 1986; Webber 1988) can aid in the resolution of pronouns:
1. Make search more efficient (less entities to consider)2. Make search more successful (block competing
antecedents) Empirical work has focused on using segmentation
to limit accessibility space of antecedents Test claim by performing an automated study on a
corpus (1241 sentence subsection of PennTreebank; 454 3rd person pronouns)
Rhetorical Structure Theory A way of organizing and describing
natural text (Mann and Thompson, 1988)
It identifies a hierarchical structure Describes binary relations between
text parts
Experiment Create coref corpus that includes PT
syntactic trees and RST information Run pronoun algorithms over this
merged data set to determine baseline score LRC (Tetreault, 1999) S-list (Strube, 1998) BFP (Brennan et al., 1987)
Develop algorithms that use clausal information to compare with baseline
Corpus 52 Wall Street Journal Articles from
1995 Penn Treebank 1273 sentences, 7594 words, 454
third person pronouns Pronoun Corpus annotated in same
manner as Ge and Charniak (1998) RST corpus from RST Discourse
Treebank (Marcu et al., 2002)
Pronoun Corpus( (S (S (NP\-SBJ\-\1#-290~1 (DT The) (NN package) ) (VP (VBD was) (VP (VBN termed) (S (NP\-SBJ (\-NONE\- \*\-\1) ) (ADJP\-PRD (JJ
excessive) )) (PP (IN by)
(NP\-LGS (DT the) (NNP Bush) (NN administration) ))))) (\, \,) (CC but) (S (NP\-SBJ (PRP#OBJREF-290~2 it) ) (ADVP (RB also) ) (VP (VBD provoked) (NP…..
RST Corpus(SATELLITE (SPAN |4| |19|) (REL2PAR ELABORATION-
ADDITIONAL) (SATELLITE (SPAN |4| |7|) (REL2PAR CIRCUMSTANCE)
(NUCLEUS (LEAF |4|) (REL2PAR CONTRAST) (TEXT _!THE PACKAGE WAS TERMED EXCESSIVE BY THE BUSH |ADMINISTRATION,_!|))
(NUCLEUS (SPAN |5| |7|) (REL2PAR CONTRAST) (NUCLEUS (LEAF |5|) (REL2PAR SPAN)
(TEXT _!BUT IT ALSO PROVOKED A STRUGGLE WITH INFLUENTIAL CALIFORNIA LAWMAKERS_!))
Baseline Results
Algorithm
% Right (S) % Right (C)
LRC 80.8% 76.4%
S-list 73.4% 70.0%
BFP 59.5% 48.7%
Naïve 50.7% 56.0%
LRC Algorithm While processing utterance’s
entities (left to right) do: Push entity onto Cf-list-new, if
pronoun, attempt to resolve first: Search through Cf-list-new (l-to-r) taking
the first candidate that meets gender, agreement constraints, etc.
If none found, search past utterance’s Cf-lists starting from previous utterance to beginning of discourse
LRC Error Analysis (89 errors) (24) Minimal S
“the committee said the company reneged on its obligations”
(21) Localized Errors “…to get a customer’s 1100 parcel-a-week
load to its doorstep” (15) Preposed Phrase
“Although he was really tired, John managed to drive 10 hours without sleep”
LRC Errors (2) (12) Parallelism
“It more than doubled the Federal’s long term debt to 1.9 billion dollars, thrust the company into unknown territory – heavy cargo – and suddenly exanded its landing rights to 21 countries from 4.
(11) Competing Antecedents “The weight of Lebanon’s history was also against
him, and it is a history…” (4) Plurals referring to companies
“The Ministory of Construction spreads concrete…. But they seldom think of the poor commuters.”
LRC Errors (3) (2) Genitive Errors
“Mr. Richardson wouldn;t offer specifics regarding Atco’s proposed British project, but he said it would compete for customers…”
Advanced Approaches Grosz and Sidner (1986)– discourse
structure is dependent on intentional structure. Attentional state is modeled as a stack that pushes and pops current state with changes in intentional structure
Veins Theory (Ide and Cristea, 2000) – position of nuclei and satellites in a RST tree determine DRA (domain of referential accessibility) for each clause
G&S Accessibility
e3
e4
e5
e6, p1
e1, e2
Search Order:
e6, e5, e4, e1, e2
Veins Theory Each RST discourse unit (leaf) has an
associated vein (Cristea et al., 1998; Ide and Cristea, 2000)
Vein provides a “summary of the discourse fragment that contains that unit”
Contains salient parts of the RST tree – the preceding nuclei and surrounding satellites
Veins determined by whether node is a nucleus or satellite and what its left and right children are
Veins Algorithm Use same data set augmented with head and
veins information (automatically computed) Exception: RST data set has some multi-child
nodes, assume all extra children are right children Bonus: areas to the left of the root are potentially
accessible – makes global topics introduced in the beginning accessible
Implementation – search each unit in the entity’s DRA starting with most-recent and left-to-right within clause. If no antecedent is found, use LRC to search.
Transforms
Goal of transforms – flatten corpus a bit to create larger segments, so more entities can be considered
SAT – merge satellite leaf into its sibling if sibling is a subtree with all leaves
SENT – merge clauses together in RST tree back into sentence
ATT – merge clauses that are in attribution relation
Transform Examples
NucleusLeaf C1
SatelliteLeaf C2
Subtree Root
Nucleus * Sat-leaf C3
(1) ORIG Subtree Root
C1 C3
(2) SAT
C2
Subtree Root(3) SENT
C1 + C2 + C3
(4) ATT Subtree Root
C1 + C2 C3
* C1 and C2 are in an Attribution relation
SAT example
S.A. Brewing would make a takeoveroffer for all of Bell Resources
if it exercises the option
according to the commission.
Nucleus Sat-Leaf (attribution)
Nuc-Leaf Sat-Leaf (condition)
Nucleus
S.A. Brewing would make a takeoveroffer for all of Bell Resources
if it exercises the option according to the commission
Sat-LeafNuc-Leaf Sat-Leaf
Nucleus
ORIGINAL
TRANSFORM
SENT example
Under the plan, Costa Rica will buyback roughly 60% of its bank debt at a deeply discounted price
according to officials
Nuc-Leaf Satellite (attribution)
Nuc-Leaf Sat-Leaf (elaboration)
Nucleus
ORIGINAL
TRANSFORM
involved in the agreement.
Under the plan, Costa Rica will buyback roughly 60% of its bank debt at a deeply discounted price, according to officials involved in the agreement
Nuc-leaf
ATT example
said Douglas Myers, Chief ExecutiveOf Lion Nathan.
Nuc-Leaf Sat-Leaf (attribution)
Satellite (summary)
ORIGINAL
TRANSFORM
Lion Nathan has a concluded contract withBond and Bell Resources,
Sat-leaf (summary)
Lion Nathan has a concluded contract withBond and Bell Resources, said Douglas Myers,Chief Executive of Lion Nathan
Results
Transform
Veins (S) Veins (C) GS (S*) GS (S) GS (C)
Original 78.9 76.7 72.3 78.9 71.4
ATT 79.3 78.2 73.7 79.3 76.3
SAT 78.9 76.4 73.6 79.1 73.9
SENT N/A N/A 78.5 80.8 N/A
SENT-SAT N/A N/A 79.7 80.8 N/A
Long Distance Resolution 10 cases in corpus of pronouns with
antecedents more than 2 utterances away, most in ATT relations
LRC gets them all correct, since no competing antecedents (“him”, “their”)
Veins (w/o ATT) gets 6 out of 10 With the transforms, all algorithms
get 100%
Conclusions Two ways to determine success of decomposition
strategy: intrasentential and intersentential resolution
Intra: no improvement, better to use grammatical function
Inter: LDR’s…. Hard to draw concrete conclusions Need more data to determine if transforms give a good
approximation of segmentation Using G&S accessibility of clauses doesn’t seem to
work either At the minimum, even if a method performs the
same, it has the advantage of a smaller search space
Future Work Error analysis shows determining
coherence relations could account for several intrasentential cases
Use rhetorical relations themselves to constrain accessibility of entities
Annotating human-human dialogues in TRIPS 911 domain for reference, already been annotated for argumentation acts (Stent, 2001)