18-12-07 | 1/21 › off-line answer extraction / jori mur question answering
Post on 31-Mar-2015
219 Views
Preview:
TRANSCRIPT
18-12-07 | 1/21
› Off-line answer extraction / Jori Mur
Question Answering
18-12-07 | 2/21
Overview› Using patterns to extract answers off-line› High precision, low recall
• Simple patterns not robust enough• Answer not in one sentence• No binary relation in question• Hard to come up with new patterns
18-12-07 | 3/21
Joost
18-12-07 | 4/21
Off-line answer extraction using patterns
› /[Noun] [Name]/... premier Balkenende ...
› /[Name], [Noun]/... Balkenende, premier ...
› /[Name] [Verb] as [Noun]/... Balkende heeft als premier ...
› /[Name] [be] [Det] [Noun]/... Balkenende is de premier ...
› ...
18-12-07 | 5/21
High precision, low recall
› Tjong Kim Sang, Bouma en de Rijke. Developing Offline Strategies for Answering Medical Questions, 2005
› Bernardi, Jijkoun, Mishne en de Rijke.Selectively Using Linguistic Resources Throughout the Question Answering Pipeline,2003
18-12-07 | 6/21
Investigating experiment› Joost› 99 function questions from CLEF2003 and
CLEF2004• Who was the author of Die Gotterdämmerung?• Of which company is Christian Blanc president?
› 15 patterns• /[ADJ][NOUN][NAME]/
Franse president Sarkozy• /[NAME], [NOUN] van [COUNTRY]/
Sarkozy, president van Frankrijk
18-12-07 | 7/21
Results› Extraction
› QA
Judgement # extracted factsCorrect 58Partially correct 15Incorrect 27Total 100
Judgement # questionsCorrect 35No answer found 46Incorrect 18Total 99
18-12-07 | 8/21
Error analysis: 46 No answer found
› In which company is Christian Blanc president?
› Christian Blanc, president of the rather destitute French aviation company Air France
Reason # QuestionsAnswers could not be found 14Patterns don't match 20Reference resolution needed 6Other 6Total 46
› Who is the daughter of Deng Xiaopeng?› Will Deng Xiaoping make it until the
Chinese New Year? (...) when Deng's youngest daughter, Deng Rong, recently said (...)
› Who is the head of state of Australia?› The prime-minister of Australia explained his
plan to transfer the role of the head of state from the British queen Elizabeth to (...)
18-12-07 | 9/21
Error analysis: 18 Incorrect
› Who is the German minister of Economy?› Who was the president of the US during the
second World War?
Reason # QuestionsModifiers ignored 12Other 6Total 18
18-12-07 | 10/21
4 Challenges
› Simple patterns are not robust enough
› Not easy to think of new patterns
› Answer not in one sentence
› Not always binary relations
18-12-07 | 11/21
1. Simple patterns are not robust enough
... George Bush, yesterday he became 61, said today as 43rd president of the US ...
... George Bush said as president of the US ...
PredmSubj Obj ObjMod
Subj Predm Obj ObjMod
› <said, subj, George Bush>
› <said, predm, as>
› <as, obj, president>
› <president, mod, of>
› <of, obj, US>
18-12-07 | 12/21
2. Answer not in one sentence
Balkenende was happy with the result, allthough he did not win the elections.
18-12-07 | 13/21
2. Evaluation and Results CoreferenceMUC score : |S|-|p(S)|
|S| -1
Recall: 45.6%
Precision: 67.9%
ExtractionWithout coreference With coreference127.711 facts 146.210 facts
QAWithout coreference With coreference153/233 (65.7%) 159/233 (68.2%)Question 20050107:
Wie was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde? ‘Who was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope?’.
The answer in the AD of September 19th, 1995:Bowersox was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde. ‘Bowersox was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope’.
18-12-07 | 14/21
3. Not always binary relations› When was Jimi Hendrix born?
Jimi Hendrix – 1942
› Who is the president of Amerika?Bush – president – Amerika
› Who was the president of Amerika in 1944?Roosevelt – president – Amerika – 1944
› Who succeeded president Roosevelt as president of Amerika in the second world war?Truman – president – Amerika – WO II Truman – successor – Roosevelt
18-12-07 | 15/21
3. Dependency relation overlap Q: Who succeeded president Roosevelt as president of
Amerika in the second world war?A1: George Bush, president of Amerika said that ...A2: At the end of the second world war Harry S. Truman succeeded president Roosevelt as president of Amerika.
A1: <president, mod, of> <of,obj,Amerika>
A2: <succeed, subj, X> <of, obj, Amerika>
<succeed, obj, president Roosevelt> <world war, det, the>
<succeed, predm, as> <world war, mod, second>
<as, obj, president>
<president, mod, of>
18-12-07 | 16/21
4. Hard to transfer to new question type› Learn patterns in bootstrapping way
1. Start with 10 seed-pairs2. Find dependency path between two terms3. Replace terms by variables to create pattern4. Use patterns to find new facts5. Use new facts to find new patterns
Find balance precision/recall3a. Filter patterns
18-12-07 | 17/21
4. Calculate precision patterns› Replace only the answer term with a variable.
› P = Ca/Co› Co : the total number of times the pattern
matched a phrase in the corpus› Ca : the total number of times the pattern
matched a phrase in the corpus containing the correct answer term
18-12-07 | 18/21
4. Experiment and Results learning› Stop at precision of table < 0.5 or
more than 100 patterns › 24 capital question from Clef, 3 turned out
NILFreq > 1 # patterns # facts P of facts QA
24 179127 <0 5 of 21
Freq > 1, P>=0.511 2403 0.76 16 of 21
143 3572 0.55 19 of 21
Freq > 1, P>=0.759 2375 0.75 16 of 21
93 2792 0.72 17 of 21125 3031 0.61 17 of 21
After 1st round
After 1st roundAfter 2nd round
After 1st roundAfter 2nd roundAfter 3rd round
18-12-07 | 19/21
4. Nice examples of found patterns› ... is na H de grootste stad van L...
Petersburg is na Moskou de grootste stad van Rusland
› ... de L-adj tenniskampioenschappen in H...de open Franse tenniskampioenschappen in Parijs
› ... de L-adj president opent in H...De Tjechische president Havel opent op 14 juni in Praag
› ... de L-adj ministers komen bijeen in H...Volgende maand komen de Europese minister van industrie opnieuw bijeen in Brussel
18-12-07 | 20/21
Summary› Qatar – off-line answer extraction module in
Joost
High precision, low recall
• Dependency patterns• Coreference resolution• Score for dependency overlap• Learning patterns
18-12-07 | 21/21
Questions?
top related