18-12-07 | 1/21

› Off-line answer extraction / Jori Mur

Question Answering

18-12-07 | 2/21

Overview› Using patterns to extract answers off-line› High precision, low recall

• Simple patterns not robust enough• Answer not in one sentence• No binary relation in question• Hard to come up with new patterns

18-12-07 | 3/21

Joost

18-12-07 | 4/21

Off-line answer extraction using patterns

› /[Noun] [Name]/... premier Balkenende ...

› /[Name], [Noun]/... Balkenende, premier ...

› /[Name] [Verb] as [Noun]/... Balkende heeft als premier ...

› /[Name] [be] [Det] [Noun]/... Balkenende is de premier ...

› ...

18-12-07 | 5/21

High precision, low recall

› Tjong Kim Sang, Bouma en de Rijke. Developing Offline Strategies for Answering Medical Questions, 2005

› Bernardi, Jijkoun, Mishne en de Rijke.Selectively Using Linguistic Resources Throughout the Question Answering Pipeline,2003

18-12-07 | 6/21

Investigating experiment› Joost› 99 function questions from CLEF2003 and

CLEF2004• Who was the author of Die Gotterdämmerung?• Of which company is Christian Blanc president?

› 15 patterns• /[ADJ][NOUN][NAME]/

Franse president Sarkozy• /[NAME], [NOUN] van [COUNTRY]/

Sarkozy, president van Frankrijk

18-12-07 | 7/21

Results› Extraction

› QA

Judgement # extracted factsCorrect 58Partially correct 15Incorrect 27Total 100

Judgement # questionsCorrect 35No answer found 46Incorrect 18Total 99

18-12-07 | 8/21

Error analysis: 46 No answer found

› In which company is Christian Blanc president?

› Christian Blanc, president of the rather destitute French aviation company Air France

Reason # QuestionsAnswers could not be found 14Patterns don't match 20Reference resolution needed 6Other 6Total 46

› Who is the daughter of Deng Xiaopeng?› Will Deng Xiaoping make it until the

Chinese New Year? (...) when Deng's youngest daughter, Deng Rong, recently said (...)

› Who is the head of state of Australia?› The prime-minister of Australia explained his

plan to transfer the role of the head of state from the British queen Elizabeth to (...)

18-12-07 | 9/21

Error analysis: 18 Incorrect

› Who is the German minister of Economy?› Who was the president of the US during the

second World War?

Reason # QuestionsModifiers ignored 12Other 6Total 18

18-12-07 | 10/21

4 Challenges

› Simple patterns are not robust enough

› Not easy to think of new patterns

› Answer not in one sentence

› Not always binary relations

18-12-07 | 11/21

1. Simple patterns are not robust enough

... George Bush, yesterday he became 61, said today as 43rd president of the US ...

... George Bush said as president of the US ...

PredmSubj Obj ObjMod

Subj Predm Obj ObjMod

› <said, subj, George Bush>

› <said, predm, as>

› <as, obj, president>

› <president, mod, of>

› <of, obj, US>

18-12-07 | 12/21

2. Answer not in one sentence

Balkenende was happy with the result, allthough he did not win the elections.

18-12-07 | 13/21

2. Evaluation and Results CoreferenceMUC score : |S|-|p(S)|

|S| -1

Recall: 45.6%

Precision: 67.9%

ExtractionWithout coreference With coreference127.711 facts 146.210 facts

QAWithout coreference With coreference153/233 (65.7%) 159/233 (68.2%)Question 20050107:

Wie was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde? ‘Who was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope?’.

The answer in the AD of September 19th, 1995:Bowersox was piloot van de missie die de astronomische satelliet, de Hubble Space Telescope, repareerde. ‘Bowersox was pilot of the mission that repaired the astronomic satellite, the Hubble Space Telescope’.

18-12-07 | 14/21

3. Not always binary relations› When was Jimi Hendrix born?

Jimi Hendrix – 1942

› Who is the president of Amerika?Bush – president – Amerika

› Who was the president of Amerika in 1944?Roosevelt – president – Amerika – 1944

› Who succeeded president Roosevelt as president of Amerika in the second world war?Truman – president – Amerika – WO II Truman – successor – Roosevelt

18-12-07 | 15/21

3. Dependency relation overlap Q: Who succeeded president Roosevelt as president of

Amerika in the second world war?A1: George Bush, president of Amerika said that ...A2: At the end of the second world war Harry S. Truman succeeded president Roosevelt as president of Amerika.

A1: <president, mod, of> <of,obj,Amerika>

A2: <succeed, subj, X> <of, obj, Amerika>

<succeed, obj, president Roosevelt> <world war, det, the>

<succeed, predm, as> <world war, mod, second>

<as, obj, president>

<president, mod, of>

18-12-07 | 16/21

4. Hard to transfer to new question type› Learn patterns in bootstrapping way

1. Start with 10 seed-pairs2. Find dependency path between two terms3. Replace terms by variables to create pattern4. Use patterns to find new facts5. Use new facts to find new patterns

Find balance precision/recall3a. Filter patterns

18-12-07 | 17/21

4. Calculate precision patterns› Replace only the answer term with a variable.

› P = Ca/Co› Co : the total number of times the pattern

matched a phrase in the corpus› Ca : the total number of times the pattern

matched a phrase in the corpus containing the correct answer term

18-12-07 | 18/21

4. Experiment and Results learning› Stop at precision of table < 0.5 or

more than 100 patterns › 24 capital question from Clef, 3 turned out

NILFreq > 1 # patterns # facts P of facts QA

24 179127 <0 5 of 21

Freq > 1, P>=0.511 2403 0.76 16 of 21

143 3572 0.55 19 of 21

Freq > 1, P>=0.759 2375 0.75 16 of 21

93 2792 0.72 17 of 21125 3031 0.61 17 of 21

After 1st round

After 1st roundAfter 2nd round

After 1st roundAfter 2nd roundAfter 3rd round

18-12-07 | 19/21

4. Nice examples of found patterns› ... is na H de grootste stad van L...

Petersburg is na Moskou de grootste stad van Rusland

› ... de L-adj tenniskampioenschappen in H...de open Franse tenniskampioenschappen in Parijs

› ... de L-adj president opent in H...De Tjechische president Havel opent op 14 juni in Praag

› ... de L-adj ministers komen bijeen in H...Volgende maand komen de Europese minister van industrie opnieuw bijeen in Brussel

18-12-07 | 20/21

Summary› Qatar – off-line answer extraction module in

Joost

High precision, low recall

• Dependency patterns• Coreference resolution• Score for dependency overlap• Learning patterns

18-12-07 | 21/21

Questions?