hoo 2012: a report on the preposition and determiner error correction shared task
DESCRIPTION
HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task. Robert Dale, Ilya Anisimoff and George Narroway Centre for Language Technology Macquarie University. Overview. Background: The HOO 2011 Pilot Round The HOO 2012 Prepositions and Determiners Challenge - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/1.jpg)
HOO 2012: A Report on the Preposition
and Determiner Error Correction Shared Task
Robert Dale, Ilya Anisimoff and George Narroway
Centre for Language Technology
Macquarie UniversityBEA 2012-06-07 1
![Page 2: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/2.jpg)
Overview
• Background: The HOO 2011 Pilot Round• The HOO 2012 Prepositions and
Determiners Challenge• Results• Conclusions
BEA 2012-06-07 2
![Page 3: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/3.jpg)
BEA 2012-06-07 3
What is HOO?
• ‘Helping Our Own’–We want to use NLP technology to help people write better NLP papers
• Initially targeting –non-native speakers– ‘superficial’ errors – grammar, style, punctuation
![Page 4: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/4.jpg)
BEA 2012-06-07 4
The HOO 2011 Pilot Round Data
• ~1k word extracts from ACL papers authored by non-native speakers
• Texts marked up with an XML variant of the Cambridge Learner Corpus tagset
• Development data: 19 extracts–Annotated by two professional copy editors
• Test data: 19 extracts from same documents–Annotated by one editor and checked by one organiser
![Page 5: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/5.jpg)
HOO 2011: Errors by Type
BEA 2012-06-07 5
![Page 6: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/6.jpg)
HOO 2011: Participation
• Google Groups discussion list with 40+ members
• 6 teams participated in the pilot round • Results presented at European NLG
Workshop 2011• See Dale and Kilgarriff [2010, 2011] for
details
BEA 2012-06-07 6
![Page 7: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/7.jpg)
Overview
• Background: The HOO 2011 Pilot Round• The HOO 2012 Prepositions and
Determiners Challenge• Results• Conclusions
BEA 2012-06-07 7
![Page 8: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/8.jpg)
Preposition Errors
BEA 2012-06-07 8
Error Type Original CorrectionIncorrect Preposition
I could only travel on July
I could only travel in July
Missing Preposition
I am looking forward your reply
I am looking forward to your reply
Unwanted Preposition
I have booked a flight to home
I have booked a flight home
![Page 9: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/9.jpg)
Article Errors
BEA 2012-06-07 9
Error Type Original CorrectionIncorrect Article
wich was situeded on a seaside
wich was situeded on the seaside
Missing Article
I will give you all information I will give you all the information
Unwanted Article
One of my hobbies is the photography
One of my hobbies is photography
![Page 10: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/10.jpg)
The Data
• Focussed only on preposition and article errors
• All other errors left uncorrected but not annotated
• Training:–1000 exam scripts from the Cambridge Learner Corpus
–375 words per script, one error every 44 words
• Testing:–100 previously unseen exam scripts–180 words per script, one error every 40 words
BEA 2012-06-07 10
![Page 11: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/11.jpg)
Error Type Breakdown
BEA 2012-06-07 11
![Page 12: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/12.jpg)
Overview
• Background: The HOO 2011 Pilot Round• The HOO 2012 Prepositions and
Determiners Challenge• Results• Conclusions
BEA 2012-06-07 12
![Page 13: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/13.jpg)
Participation
• 26 teams registered interest, 14 teams made final submissions
BEA 2012-06-07 13
![Page 14: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/14.jpg)
Evaluation Metrics
• Scope:–Preps, Dets, Both
• Scores:–Detection, Recognition, Correction
• Measures: –Precision, Recall, F-Score
BEA 2012-06-07 14
![Page 15: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/15.jpg)
Managing Annotation Errors
• Teams submitted results (standoff annotations) on test data within one week of test data release
• Teams invited to nominate corrections to the test data
• Change requests assessed and implemented in the gold standard
• Initially submitted results rescored against revised gold standard
• Results shown here are before revisions: see paper for numbers after revisionsBEA 2012-06-07 15
![Page 16: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/16.jpg)
Prepositions
BEA 2012-06-07 16
![Page 17: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/17.jpg)
Determiners
BEA 2012-06-07 17
![Page 18: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/18.jpg)
Prepositions and Determiners
BEA 2012-06-07 18
![Page 19: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/19.jpg)
Approaches Taken• Maximum Entropy classifier [ET, KU, LE,
NA, TH]• Naïve Bayes classifier [CU, TC, UI]• N-gram LM [ET, JU, VT]• Rule-based component [ET, JU, LE]• Decision-tree classifier [UT, VA]• Average Perceptron classifier [UI]• Confidence-weighted linear classifiers
[NU]• CRF tagger [TH]• SVM classifier [UD]BEA 2012-06-07 19
![Page 20: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/20.jpg)
Data Used• FCE data as provided [CU, KU, LE, NA,
TC, UI, UT, VT]• Google 1T [ET, UD, UI, UT, VA]• BNC [LE, UT, TH]• FCE data corrected [LE, NU]• Google Books n-gram corpus [TC]• Reuters [LE]• Complete FCE dataset* [CU]• San Jose Mercury News and
MetaMetrics Lexile corpora* [ET]BEA 2012-06-07 20
![Page 21: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/21.jpg)
Features Used• Lexical items [ET,
KU, LE, NU, UI, UT, TC, TH, VA]
• POS tags [ET, KU, LE, NU, UD, UI, UT, TC, TH]
• Phrase-chunk based features [ET, KU, LE, NU, UD, UI, UT, TH]
• Parse-based features [CU, KU, NA]
• Web counts [UD, NU]
• Original article [UI]• Word clusters [UT]• Noun countability
[UT]• Dependency n-gram
counts [NU]• Learner’s L1 [KU]• Length+ending
encoding [VT]
BEA 2012-06-07 21
![Page 22: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/22.jpg)
Overview
• Background: The HOO 2011 Pilot Round• The HOO 2012 Prepositions and
Determiners Challenge• Results• Conclusions
BEA 2012-06-07 22
![Page 23: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/23.jpg)
Outcomes
• A high-participation shared task with an emphasis on public data sets and replicability
• A set of freely-available flexible tools for evaluating a wide range of text-level error correction tasks
• A benchmarking dataset• See www.correcttext.org to get the
tools and the existing data sets
BEA 2012-06-07 23
![Page 24: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/24.jpg)
Limitations and Issues
• Hard to get data, expensive to construct
• Annotation errors and disagreements• Is it valid to look at one or two error
types in isolation?
BEA 2012-06-07 24
![Page 25: HOO 2012: A Report on the Preposition and Determiner Error Correction Shared Task](https://reader034.vdocuments.site/reader034/viewer/2022042703/568168f1550346895ddff7b7/html5/thumbnails/25.jpg)
Thank you to:• Cambridge University Press• the BEA organisers, and • all the participants for taking part
BEA 2012-06-07 25