Download - Curs 5 Rezoluţia anaforei - continuare
![Page 1: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/1.jpg)
Curs 5Rezoluţia anaforei - continuare
![Page 2: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/2.jpg)
Designing Test-Beds for General Anaphora Resolution
Work done in collaboration with:
Oana [email protected]
University of Saarland, Germany
DAARC’04
![Page 3: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/3.jpg)
Evaluation of a minimal AR system
markablesRE-extractorinput outputAR-engine
![Page 4: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/4.jpg)
Evaluation of a minimal AR system
re-test
re-gold
RE-extractor re-test-coref-test
re-gold-coref-gold
re-gold-coref-test
AR-engine
P, R, F
Test the RE-extractor
Test the whole system globally
P, R. F
Test only the AR-engine
P, R. F
input
![Page 5: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/5.jpg)
The Orwell corpus
• Chapters 1, 2, 3 and 5 from George Orwell’s “Ninety eighty four”
• Automatic detection of markables– POS-tagging– FDG parser– markable = any construction dominated by a
noun/pronoun– detection of head and lemma (given)– deletion of relative clauses
![Page 6: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/6.jpg)
The Penn Treebank corpus
• 7 files from WSJ• Extraction of markables from the PTB-
style constituency trees– Collin’s rules to extract head– WordNet script for lemma– dependency links between words
![Page 7: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/7.jpg)
Dimensions of corpora
Orwell PTB
No. of sentences 983 281
No. of words 19520 6244
No. of REs 5474 1853
Average no. of REs per sentence 5.5 6.6
Pronouns 1902 136
No. of DEs 2768 1546
![Page 8: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/8.jpg)
Markables• Generally, conformant with MUC-7 and ACE
criteria• Differences:
– do not include relative clauses – each term of an apposition is taken separately ([Big
Brother], [the primal traitor])– conjoined expressions are annotated individually
([John] and [Mary], [hills] and [mountains])– modifying nouns appearing in noun-noun modification
are not marked separately ([glass doors], [prison food], [the junk bond market])
![Page 9: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/9.jpg)
Markables
What do we mark? • noun phrases
– definite: (the principle, the flying object) – indefinite: (a book, a future star) – undetermined (sole guardian of truth) – names (Winston Smith, The Ministry of Love)– dates (April) – currency expressions ($40) – percentages (48%)
• pronouns– personal (I, you, he, him, she, her, it, they, them) – possessive (his, her, hers, its, their, theirs) – reflexive (himself, herself, itself, themselves) – demonstrative (this, that, these, those) – wh-pronouns – when they replace an entity (which, who, whom, whose, that)
• numerals– when they refer to entities (four of them, the first, the second)
![Page 10: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/10.jpg)
Our model: primary attributes• Lexical & morphological:
– lemma– number– POS– headForm
• Syntactic:– synt-role– dependency-link– npText– includedNPs– isDefinite, isIndefinite,– predNameOf
• Semantic:– isMaleName, isFemaleName, isFamilyName, isPerson– HeSheItThey
• Positional:– offset– sentenceID
![Page 11: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/11.jpg)
Our model: knowledge sources
• For each attribute there is a knowledge source that fetches the value using:– The POS tagger output– The FDG structure– Large name databases– The WordNet hierarchy– Punctuation
![Page 12: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/12.jpg)
Knowledge sources - HeSheItThey
• HeSheItThey = [Phe, Pshe, Pit, Pthey]– for pronouns – straightforward;– for NPs:
• n = # synsets of the head• f = # synsets which are hyponyms of <female>• m = # synsets which are hyponyms of <male>• p = # synsets which are hyponyms of <person>If NP is plural: Phe=0, Pshe=0, Pit=0, Pthey=1
Else: Phe= , Pshe= , Pit= , Pthey=0n
mpm )(*21
n
fpf )(*21
npn
![Page 13: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/13.jpg)
Our model: rules• Demolishing rules:
– IncludingRule: prohibits coreference between nested REs • Certifying rules:
– PredNameRule – ProperNameRule – whRule
• Promoting/demoting rules: – HeSheItTheyRule– RoleRule– NumberRule– LemmaRule– PersonRule– SynonymyRule– HypernymyRule– WordnetChainRule
![Page 14: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/14.jpg)
Our model: the whRules (example)
• Rules for detecting the antecedent of a wh-pronoun:– Case1:
I saw [a blond boy] who was playing in the garden.
– Case2:[The colour of the chair] which was underneath the
table…[The atmosphere of happiness] which she carried
with her.
![Page 15: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/15.jpg)
Our model: domain of referential accessibility
• Linear
![Page 16: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/16.jpg)
Evaluation of the RE-extractor
re-test
re-gold
RE-extractor re-test-coref-test
re-gold-coref-gold
re-gold-coref-test
AR-engine
P, R, F
Test the RE-extractor
input
When a gold-test pair of markables match?
![Page 17: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/17.jpg)
Evaluation of the RE-extractor
re-test
re-gold
re-test-coref-test
re-gold-coref-gold
re-gold-coref-test
AR-engine
P, R, F
Test the RE-extractor
When a gold-test pair of markables match?
• head matching (HM): if they have the same head
PTB-HMP=0,90R=0,95F=0,92
Orwell-HMP=0,85R=0,94F=0,89
RE-extractor
gold
test
markable
markable
input
![Page 18: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/18.jpg)
Evaluation of the RE-extractor
re-test
re-gold
re-test-coref-test
re-gold-coref-gold
re-gold-coref-test
AR-engine
Test the RE-extractor
When a gold-test pair of markables match?
• partial matching (PM): if they have the same head and the mutual overlap is higher than 50% (compared to the longest span)
PTB-PMP=0,78R=0,82F=0,79
P, R, F
Orwell-PMP=0,74R=0,80F=0,76
RE-extractor
gold
test
l1
l
l / l1 > 0.5
input
l2
![Page 19: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/19.jpg)
Evaluation of the AR-engine•Same set of markables (on the “identity of head” criterion)
–For each anaphor in the gold: •If it belongs to a chain that doesn’t contain any other anaphor, then we
look in the test set to see if it belongs to a similar trivial chain– if yes will get the value 1;
gold
test Ci = 1
i
![Page 20: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/20.jpg)
Evaluation of the AR-engine•Same set of markables (on the “identity of head” criterion)
–For each anaphor in the gold: •If it belongs to a chain that doesn’t contain any other anaphor, then we
look in the test set to see if it belongs to a similar trivial chain– if yes will get the value 1; – otherwise it will get the value 0;
gold
test Ci = 0
i
![Page 21: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/21.jpg)
Evaluation of the AR-engine•Same set of markables (on the “identity of head” criterion)
–For each anaphor in the gold: •If the anaphor belongs to a chain containing other n anaphors, then we
look in the test set and count how many of these n anaphors belong to the chain corresponding to the current test-set anaphor (we note this number with m). The ratio m/n will be the value assigned to the current anaphor.
gold
test ci = 2/30 11
1 11
i
![Page 22: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/22.jpg)
Evaluation of the AR-engine•Same set of markables (on the “identity of head” criterion)
–For each anaphor in the gold: •If the anaphor belongs to a chain containing other n anaphors, then we look in
the test set and count how many of these n anaphors belong to the chain corresponding to the current test-set anaphor (we note this number with m). The ratio m/n will be the value assigned to the current anaphor.
–Then we add this number for all anaphors and divide by no. of anaphors: ci / N
gold
test ci = 2/30 11
1 11
i
![Page 23: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/23.jpg)
Evaluation of the AR-engine working on coreferences
RE-extractor AR-enginere-test re-test-coref-test
re-gold
re-gold-coref-gold
re-gold-coref-test
input
PTBSR = 0.69
MUC F = 0.75
OrwellSR = 0.66
MUC F = 0.72
![Page 24: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/24.jpg)
Evaluation of the whole system• Possibly different sets of markables, identified on the “identity of head”
criterion, as found in gold and test, with possibly different spans– same global formula but the contribution of each markable is factored by
the mutual overlapping score, showing the test versus gold overlapping of markables
mosi = b/a
gold
test
a
b
![Page 25: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/25.jpg)
Evaluation of the whole system• Possibly different sets of markables, identified on the “identity of head”
criterion, as found in gold and test, with possibly different spans– same global formula but the contribution of each markable is factored by
the mutual overlapping score, showing the test versus gold overlapping of markables
ci = (0.7+0.5)/3 = 1.2/3
gold
test
R = ci / Ng
0 0.50.7
1 11
i
![Page 26: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/26.jpg)
Evaluation of the whole system• Possibly different sets of markables, identified on the “identity of head”
criterion, as found in gold and test, with possibly different spans– “misses” (failings to find certain markables) influence R– “false-alarms” (markables erroneously considered in the test) influence P
gold
test
0 0.7
1 11
i
false-alarm miss
![Page 27: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/27.jpg)
Evaluation of the whole system
RE-extractor AR-enginere-test re-test-coref-test
re-gold
re-gold-coref-gold
OrwellSR = 0,55
re-gold-coref-test
input
PTBSR = 0,61
![Page 28: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/28.jpg)
Commentaries• RE-extractor module gives better results on PTB
than on Orwell– human syntactic annotation versus automatic FDG
structure detection• AR module gives slightly better results on PTB than
on Orwell– news (finance) versus belles-lettres– heads: in PTB – extracted by rules relying on the human
syntactic annotation, in Orwell – extracted by rules relying on the FDG results
• Difficult to compare with other approaches/authors– apparently we are in the upper class– BUT: not the same corpus, not the same evaluation metric
![Page 29: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/29.jpg)
Transferring Coreference Chains through Word Alignment
In collaboration with Oana Postolache and Constantin Orăsan
[email protected] [email protected]
University of Saarland, GermanyUniversity of Wolverhampton, United Kingdom
LREC’06
![Page 30: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/30.jpg)
The goal
• Automatic annotation of coreference chains for languages with sparser resources (Romanian).
• Difficulties:En: John broke his arm recently. It hurts badly.
Ro: Ion şi-a rupt braţul recent. Îl doare rău.
![Page 31: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/31.jpg)
English-Romanian parallel corpus
• George Orwell’s novel “1984”– 6,411 sentences– the English version is in the process of being
manually annotated with coreference chains (now we have half of the corpus)
• Experimental data: three parts from the first chapter– ~13K words.– 638 sentences– the Romanian version manually annotated for
evaluation purposes
![Page 32: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/32.jpg)
Coreference information annotated
• Conformant with MUC-7 and ACE 2003.• Referential expressions are:
– Noun-phrases: definite, indefinite & undetermined– Proper names– Pronouns and wh-pronouns– Numerals
• The REs include only restrictive clauses• The term of an apposition is taken separately• Conjoined expressions are taken individually• Noun premodifiers are not marked
![Page 33: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/33.jpg)
Experiment: Automatic word alignment
• We used the Romanian-English aligner COWAL (Tufiş et al., 2006)
• Performance: 83.30% F-measure
• The first ranked system out of 37 at ACL’05 shared task on word alignment
![Page 34: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/34.jpg)
Experiment:Extraction of the Romanian REs• For an Eng RE with words e1, e2, … en, we
extract the Rom set of words r1, r2, …rm, surface ordered
• Heads are transferred through the alignment from Eng to Rom (1:n)
• We consider the Rom RE as the span of words between r1 and rm
![Page 35: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/35.jpg)
![Page 36: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/36.jpg)
Experiment 1:Extraction of the Romanian REsFour situations:
1. An Eng RE has a corresponding Rom RE with ONE head.2. An Eng RE has a corresponding Rom RE with ONE OR MORE
heads.3. An Eng RE has a corresponding Rom RE with NO head.4. An Eng RE has NO corresponding Rom RE.
Only REs conforming to 1. and 2. are considered. The head of the Rom RE is taken as the leftmost
head whose POS is Noun, Pronoun or Numeral.
![Page 37: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/37.jpg)
Experiment 2:Coreference chains transfer
• As the Eng REs are clustered in chains referring to the same entity, and we have the corresponding Rom REs, we simply “import” the clustering.
• As not all Eng REs have a corresponding Rom RE, the no. of clusters between Eng and Rom may differ.
• Also there are differences between the lengths of corresponding clusters.
![Page 38: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/38.jpg)
Evaluation
• Transferred data (system) is compared against gold standard data (manual) for Rom.
![Page 39: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/39.jpg)
Evaluation of the RE heads
• We only consider the heads of the system REs and the heads of the gold standard REs.
Precision 95.7
Recall 69.5
F-measure 80.5
![Page 40: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/40.jpg)
Evaluation of the RE spans (1/2)• All REs
The overlaps between the system REs and the gold standard REs: 2 * #(wordsSystemRE wordsGoldRE)
Overlap= ------------------------------------------------------- #wordsSystemRE + #wordsGoldRE
Precision 86.8Recall 63.0F-measure 73.0
![Page 41: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/41.jpg)
Evaluation of the RE spans (2/2)
The previous numbers reflect also the penalties for not having a certain REs in the system, or having wrong REs (errors also contained in the heads evaluation).
Only correct system REs:– System REs with a correct head against the
corresponding gold REs.
Span overlap 90.7
![Page 42: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/42.jpg)
Evaluation of coreference chains• All system REs against the gold REs
• The correct systems REs against the gold REs
MUC F-score 63.88B-cubed F-score 82.60
MUC F-score 87.72B-cubed F-score 93.14
![Page 43: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/43.jpg)
Error analysis (1/3)Incorrect detection of Rom REs
• Wrong alignment• Eng. adjs/advs/verbs translated in Rom by nouns
En: naturally sanguine faceRo: faţă sangvină de la natură (face sanguine from the nature)
• Choices of the Rom translatorEn: The actual writing would be easyRo: Scrisul în sine era o treabă uşoară (The writing itself was an easy job)
• Eng noun premodifiers translated in Rom as prepositional phrase postmodifiers or possesives
En: a forced labour campRo: un lagăr de muncă silnică (a camp of forced labour)
![Page 44: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/44.jpg)
Error analysis (2/3)Errors in the spans overlap
• Wrong alignment• Triggered by the choice of translation:
En: [Someone with a comb and a piece of toilet paper] was trying to keep tune with the music.
Ro: [Cineva se străduia, cu un pieptene şi o bucată de hârtie igienică], să ţină isonul muzicii.([Someone was trying, with a comb and a piece of toilet paper], to keep tune with the music.)
![Page 45: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/45.jpg)
Error analysis (3/3)Incorrect detection of coreference
chains
• Errors due to translation choice:En: The sky was a harsh blue.
A predicative noun – subject relationship.
Ro: Cerul era de un albastru strident.(The sky was as a harsh blue.)
![Page 46: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/46.jpg)
Conclusions• What and why?
– An automatic method for projecting coreference chains in parallel corpora
– To augment the scarce resources of coreference information
– A preprocessing step prior to manual correction in the annotation effort
• How good?– References: high precision (> 95%) but smaller recall (~
70%)– Coreference chains: relatively high F-measure (> 90%)
for correct REs
![Page 47: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/47.jpg)
Our anaphoric… references
• Cristea,D., Dima,G.E. (2001): An integrating framework for anaphora resolution. In Information Science and Technology, Romanian Academy Publishing House, Bucharest, vol. 4, no. 3-4, p 273-291.
• Cristea,D., Postolache,O.-D., Dima,G.E., Barbu,C. (2002): AR-Engine – a framework for unrestricted co-reference resolution. In Proceedings of The Third International Conference on Language Resources and Evaluation, LREC-2002, Las Palmas, Spain.
• Cristea, D., Dima, G.E., Postolache, O.D., Mitkov, R. (2002): Handling complex anaphora resolution cases. In Proceedings of the Discourse Anaphora and Anaphor Resolution Colloquium, Lisbon, September 18-20, 2002.
• Postolache, O. and Cristea, D. (2004): Designing Test-beds for General Anaphora Resolution, in Proceedings of the Discourse Anaphora and Anaphor Resolution Colloquium – DAARC, St. Miguel, Portugal.
• Cristea,D.; Postolache,O.D. (2005): How to deal with wicked anaphora, in António Branco, Tony McEnery and Ruslan Mitkov (eds.): Anaphora Processing: Linguistic, Cognitive and Computational Modelling, Current Issues in Linguistic Theory, Benjamin Publishing Books, ISBN 90-272-4777-3 (Eur)/1-58811-621-2 (USA).
• Postolache, O., Cristea, D., Orasan, C. (2006). Transferring Coreference Chains through Word Alignment. In Proceedings of LREC-2006, Geneva, May 2006.
![Page 48: Curs 5 Rezoluţia anaforei - continuare](https://reader030.vdocuments.site/reader030/viewer/2022020200/56813d42550346895da7009f/html5/thumbnails/48.jpg)
Contest on AR @ 6th DAARCMarch 2007, Lagos, Portugal
Call to be issued soon…
• On English only, 4 tracks: – with markables given, resolve only pronouns– with markables given, resolve all anaphors– no markables given, resolve only pronouns– no markables given, resolve all anaphors