Upload File

question answering - stanford university · 2016-12-21 · dan%jurafsky% 2 ques%on(answering(what...

  • Home
  • Documents
  • Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass
73
Question Answering What is Ques+on Answering?

Upload: ngobao

Post on 20-Aug-2018

216 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

What  is  Ques+on  Answering?  

Page 2: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

2  

Ques%on  Answering  

What do worms eat?

worms

eat

what

worms

eat

grass

Worms eat grass

worms

eat

grass

Grass is eaten by wormsbirds

eat

worms

Birds eat worms

horses

eat

grass

Horses with worms eat grass

with

worms

Ques%on: Poten%al-Answers:

One  of  the  oldest  NLP  tasks  (punched  card  systems  in  1961)  Simmons,  Klein,  McConlogue.  1964.  Indexing  and  Dependency  Logic  for  Answering  English  Ques+ons.  American  Documenta+on  15:30,  196-­‐204  

Page 3: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Ques%on  Answering:  IBM’s  Watson  •  Won  Jeopardy  on  February  16,  2011!  

3  

WILLIAM WILKINSON’S “AN ACCOUNT OF THE PRINCIPALITIES OF

WALLACHIA AND MOLDOVIA” INSPIRED THIS AUTHOR’S

MOST FAMOUS NOVEL

Bram  Stoker  

Page 4: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Apple’s  Siri  

4  

Page 5: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Wolfram  Alpha  

5  

Page 6: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

6  

Types  of  Ques%ons  in  Modern  Systems  

•  Factoid  ques+ons  •  Who  wrote  “The  Universal  Declara4on  of  Human  Rights”?  •  How  many  calories  are  there  in  two  slices  of  apple  pie?  •  What  is  the  average  age  of  the  onset  of  au4sm?  •  Where  is  Apple  Computer  based?  

•  Complex  (narra+ve)  ques+ons:  •  In  children  with  an  acute  febrile  illness,  what  is  the                              efficacy  of  acetaminophen  in  reducing  fever?  

•  What  do  scholars  think  about  Jefferson’s  posi4on  on                      dealing  with  pirates?  

Page 7: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Commercial  systems:    mainly  factoid  ques%ons  

Where  is  the  Louvre  Museum  located?   In  Paris,  France  

What’s  the  abbrevia+on  for  limited  partnership?  

L.P.  

What  are  the  names  of  Odin’s  ravens?   Huginn  and  Muninn  

What  currency  is  used  in  China?   The  yuan  

What  kind  of  nuts  are  used  in  marzipan?   almonds  

What  instrument  does  Max  Roach  play?   drums  

What  is  the  telephone  number  for  Stanford  University?  

650-­‐723-­‐2300  

Page 8: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Paradigms  for  QA  

•  IR-­‐based  approaches  •  TREC;    IBM  Watson;  Google  

•  Knowledge-­‐based  and  Hybrid  approaches  •  IBM  Watson;  Apple  Siri;  Wolfram  Alpha;  True  Knowledge  Evi    

8  

Page 9: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Many  ques%ons  can  already  be  answered  by  web  search  

•  a  

9  

Page 10: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

IR-­‐based  Ques%on  Answering  

•  a  

10  

Page 11: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

11  

IR-­‐based  Factoid  QA  

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

DocumentDocumentDocument

Page 12: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

IR-­‐based  Factoid  QA  •  QUESTION  PROCESSING  

•  Detect  ques+on  type,  answer  type,  focus,  rela+ons  •  Formulate  queries  to  send  to  a  search  engine  

•  PASSAGE  RETRIEVAL  •  Retrieve  ranked  documents  •  Break  into  suitable  passages  and  rerank  

•  ANSWER  PROCESSING  •  Extract  candidate  answers  •  Rank  candidates    

•  using  evidence  from  the  text  and  external  sources  

Page 13: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Knowledge-­‐based  approaches  (Siri)  

•  Build  a  seman+c  representa+on  of  the  query  •  Times,  dates,  loca+ons,  en++es,  numeric  quan++es  

•  Map  from  this  seman+cs  to  query  structured  data    or  resources  •  Geospa+al  databases  •  Ontologies  (Wikipedia  infoboxes,  dbPedia,  WordNet,  Yago)  •  Restaurant  review  sources  and  reserva+on  services  •  Scien+fic  databases  

13  

Page 14: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Hybrid  approaches  (IBM  Watson)  

•  Build  a  shallow  seman+c  representa+on  of  the  query  •  Generate  answer  candidates  using  IR  methods  

•  Augmented  with  ontologies  and  semi-­‐structured  data  

•  Score  each  candidate  using  richer  knowledge  sources  •  Geospa+al  databases  •  Temporal  reasoning  •  Taxonomical  classifica+on  

14  

Page 15: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

What  is  Ques+on  Answering?  

Page 16: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Answer  Types  and  Query  Formula+on  

Page 17: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Factoid  Q/A  

17  

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

DocumentDocumentDocument

Page 18: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Ques%on  Processing  Things  to  extract  from  the  ques%on  

•  Answer  Type  Detec+on  •  Decide  the  named  en%ty  type  (person,  place)  of  the  answer  

•  Query  Formula+on  •  Choose  query  keywords  for  the  IR  system  

•  Ques+on  Type  classifica+on  •  Is  this  a  defini+on  ques+on,  a  math  ques+on,  a  list  ques+on?  

•  Focus  Detec+on  •  Find  the  ques+on  words  that  are  replaced  by  the  answer  

•  Rela+on  Extrac+on  •  Find  rela+ons  between  en++es  in  the  ques+on  18  

Page 19: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Question Processing They’re the two states you could be reentering if you’re crossing Florida’s northern border

•  Answer  Type:    US  state  •  Query:    two  states,  border,  Florida,  north  •  Focus:  the  two  states  •  Rela+ons:    borders(Florida,  ?x,  north)  

19  

Page 20: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Type  Detec%on:  Named  En%%es  

•  Who  founded  Virgin  Airlines?  •   PERSON    

•  What  Canadian  city  has  the  largest  popula4on?  •   CITY.  

Page 21: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Type  Taxonomy  

•  6  coarse  classes  •  ABBEVIATION,  ENTITY,  DESCRIPTION,  HUMAN,  LOCATION,  NUMERIC  

•  50  finer  classes  •  LOCATION:  city,  country,  mountain…  •  HUMAN:  group,  individual,  +tle,  descrip+on  •  ENTITY:  animal,  body,  color,  currency…  

21  

Xin  Li,  Dan  Roth.  2002.  Learning  Ques+on  Classifiers.  COLING'02  

Page 22: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

22  

Part  of  Li  &  Roth’s  Answer  Type  Taxonomy  

LOCATION

NUMERIC

ENTITY HUMAN

ABBREVIATIONDESCRIPTION

country city state

datepercent

money

sizedistance

individual

title

group

food

currency

animal

definition

reason expression

abbreviation

Page 23: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

23  

Answer  Types  

Page 24: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

24  

More  Answer  Types  

Page 25: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  types  in  Jeopardy  

•  2500  answer  types  in  20,000  Jeopardy  ques+on  sample  •  The  most  frequent  200  answer  types  cover  <  50%  of  data  •  The  40  most  frequent  Jeopardy  answer  types  he,  country,  city,  man,  film,  state,  she,  author,  group,  here,  company,  president,  capital,  star,  novel,  character,  woman,  river,  island,  king,  song,  part,  series,  sport,  singer,  actor,  play,  team,    show,                              actress,  animal,  presiden+al,  composer,  musical,  na+on,                                      book,  +tle,  leader,  game  

25  

Ferrucci  et  al.  2010.  Building  Watson:  An  Overview  of  the  DeepQA  Project.  AI  Magazine.  Fall  2010.  59-­‐79.  

Page 26: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Type  Detec%on  

•  Hand-­‐wrioen  rules  •  Machine  Learning  •  Hybrids  

Page 27: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Type  Detec%on  

•  Regular  expression-­‐based  rules    can  get  some  cases:  •  Who  {is|was|are|were}  PERSON  •  PERSON  (YEAR  –  YEAR)  

•  Other  rules  use  the  ques%on  headword:    (the  headword  of  the  first  noun  phrase  ater  the  wh-­‐word)    

• Which  city  in  China  has  the  largest  number  of  foreign  financial  companies?  

• What  is  the  state  flower  of  California?  

Page 28: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Type  Detec%on  

•  Most  oten,  we  treat  the  problem  as  machine  learning  classifica+on    • Define  a  taxonomy  of  ques+on  types  • Annotate  training  data  for  each  ques+on  type  •  Train  classifiers  for  each  ques+on  class                              using  a  rich  set  of  features.  •  features  include  those  hand-­‐wrioen  rules!  

28  

Page 29: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Features  for  Answer  Type  Detec%on  

•  Ques+on  words  and  phrases  •  Part-­‐of-­‐speech  tags  •  Parse  features  (headwords)  •  Named  En++es  •  Seman+cally  related  words    

29  

Page 30: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Factoid  Q/A  

30  

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

DocumentDocumentDocument

Page 31: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Keyword  Selec%on  Algorithm  

1.  Select  all  non-­‐stop  words  in  quota+ons  2.  Select  all  NNP  words  in  recognized  named  en++es  3.  Select  all  complex  nominals  with  their  adjec+val  modifiers  4.  Select  all  other  complex  nominals  5.  Select  all  nouns  with  their  adjec+val  modifiers  6.  Select  all  other  nouns  7.  Select  all  verbs    8.  Select  all  adverbs    9.  Select  the  QFW  word  (skipped  in  all  previous  steps)    10.  Select  all  other  words    

Dan  Moldovan,  Sanda  Harabagiu,  Marius  Paca,  Rada  Mihalcea,  Richard  Goodrum,  Roxana  Girju  and  Vasile  Rus.  1999.  Proceedings  of  TREC-­‐8.  

Page 32: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Choosing keywords from the query

32

Who coined the term “cyberspace” in his novel “Neuromancer”?

1 1

4 4

7

cyberspace/1 Neuromancer/1 term/4 novel/4 coined/7

Slide  from  Mihai  Surdeanu  

Page 33: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Answer  Types  and  Query  Formula+on  

Page 34: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Passage  Retrieval  and  Answer  Extrac+on  

Page 35: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Factoid  Q/A  

35  

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

DocumentDocumentDocument

Page 36: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

36  

Passage  Retrieval  

•  Step  1:  IR  engine  retrieves  documents  using  query  terms  •  Step  2:  Segment  the  documents  into  shorter  units  

•  something  like  paragraphs  

•  Step  3:  Passage  ranking  •  Use  answer  type  to  help  rerank  passages  

Page 37: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Features  for  Passage  Ranking  

•  Number  of  Named  En++es  of  the  right  type  in  passage  •  Number  of  query  words  in  passage  •  Number  of  ques+on  N-­‐grams  also  in  passage  •  Proximity  of  query  keywords  to  each  other  in  passage  •  Longest  sequence  of  ques+on  words  •  Rank  of  the  document  containing  passage  

Either  in  rule-­‐based  classifiers  or  with  supervised  machine  learning  

Page 38: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Factoid  Q/A  

38  

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

DocumentDocumentDocument

Page 39: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Answer  Extrac%on  

•  Run  an  answer-­‐type  named-­‐en+ty    tagger  on  the  passages  •  Each  answer  type  requires  a  named-­‐en+ty  tagger  that  detects  it  •  If  answer  type  is  CITY,  tagger  has  to  tag  CITY  •  Can  be  full  NER,  simple  regular  expressions,  or  hybrid  

•  Return  the  string  with  the  right  type:  •  Who is the prime minister of India (PERSON)  Manmohan Singh, Prime Minister of India, had told left leaders that the deal would not be renegotiated.!

•  How tall is Mt. Everest? (LENGTH)  The official height of Mount Everest is 29035 feet!

Page 40: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Ranking  Candidate  Answers  

•  But  what  if  there  are  mul+ple  candidate  answers!      

 Q: Who was Queen Victoria’s second son?!•  Answer  Type:    Person  

•  Passage:  The  Marie  biscuit  is  named  ater  Marie  Alexandrovna,  the  daughter  of  Czar  Alexander  II  of  Russia  and  wife  of  Alfred,  the  second  son  of  Queen  Victoria  and  Prince  Albert  

Page 41: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Ranking  Candidate  Answers  

•  But  what  if  there  are  mul+ple  candidate  answers!      

 Q: Who was Queen Victoria’s second son?!•  Answer  Type:    Person  

•  Passage:  The  Marie  biscuit  is  named  ater  Marie  Alexandrovna,  the  daughter  of  Czar  Alexander  II  of  Russia  and  wife  of  Alfred,  the  second  son  of  Queen  Victoria  and  Prince  Albert  

Page 42: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Use  machine  learning:  Features  for  ranking  candidate  answers  

Answer  type  match:    Candidate  contains  a  phrase  with  the  correct  answer  type.  PaZern  match:  Regular  expression  paoern  matches  the  candidate.  Ques%on  keywords:  #  of  ques+on  keywords  in  the  candidate.  Keyword  distance:  Distance  in  words  between  the  candidate  and  query  keywords    Novelty  factor:  A  word  in  the  candidate  is  not  in  the  query.  Apposi%on  features:  The  candidate  is  an  apposi+ve  to  ques+on  terms  Punctua%on  loca%on:  The  candidate  is  immediately  followed  by  a                                    comma,  period,  quota+on  marks,  semicolon,  or  exclama+on  mark.  Sequences  of  ques%on  terms:  The  length  of  the  longest  sequence                                                                    of  ques+on  terms  that  occurs  in  the  candidate  answer.    

Page 43: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Candidate  Answer  scoring  in  IBM  Watson  

•  Each  candidate  answer  gets  scores  from  >50  components  •  (from  unstructured  text,  semi-­‐structured  text,  triple  stores)  

•  logical  form  (parse)  match  between  ques+on  and  candidate  •  passage  source  reliability    •  geospa+al  loca+on  •  California    is    ”southwest  of  Montana”  

•  temporal  rela+onships  •  taxonomic  classifica+on  43  

Page 44: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

44  

Common  Evalua%on  Metrics  

1. Accuracy  (does  answer  match  gold-­‐labeled  answer?)  2. Mean  Reciprocal  Rank  

•  For  each  query  return  a  ranked  list  of  M  candidate  answers.  •  Query  score  is  1/Rank  of  the  first  correct  answer    •  If  first  answer  is  correct:  1    •  else  if  second  answer  is  correct:  ½  •  else  if  third  answer  is  correct:    ⅓,    etc.  •  Score  is  0  if  none  of  the  M  answers  are  correct  

•  Take  the  mean  over  all  N  queries  

MRR =

1rankii=1

N

∑

N

Page 45: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Passage  Retrieval  and  Answer  Extrac+on  

Page 46: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Using  Knowledge  in  QA  

Page 47: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Rela%on  Extrac%on  

•  Answers:  Databases  of  Rela+ons  •  born-­‐in(“Emma  Goldman”,  “June  27  1869”)  •  author-­‐of(“Cao  Xue  Qin”,  “Dream  of  the  Red  Chamber”)  •  Draw  from  Wikipedia  infoboxes,  DBpedia,  FreeBase,  etc.  

•  Ques+ons:  Extrac+ng  Rela+ons  in  Ques+ons  Whose  granddaughter  starred  in  E.T.?  

(acted-in ?x “E.T.”)! (granddaughter-of ?x ?y)!47  

Page 48: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Temporal  Reasoning  

•  Rela+on  databases  •  (and  obituaries,  biographical  dic+onaries,  etc.)  

•  IBM  Watson  ”In  1594  he  took  a  job  as  a  tax  collector  in  Andalusia”  Candidates:  •  Thoreau  is  a  bad  answer  (born  in  1817)  •  Cervantes  is  possible  (was  alive  in  1594)  

48  

Page 49: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Geospa%al  knowledge  (containment,  direc%onality,  borders)    

•  Beijing    is  a  good  answer  for    ”Asian  city”  •  California    is    ”southwest  of  Montana”  •  geonames.org:  

49  

Page 50: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Context  and  Conversa%on    in  Virtual  Assistants  like  Siri  

•  Coreference  helps  resolve  ambigui+es  U:  “Book  a  table  at  Il  Fornaio  at  7:00  with  my  mom”  U:  “Also  send  her  an  email  reminder”  

•  Clarifica+on  ques+ons:  U:  “Chicago  pizza”  S:  “Did  you  mean  pizza  restaurants  in  Chicago                                                                                                                              or  Chicago-­‐style  pizza?”  

50  

Page 51: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Using  Knowledge  in  QA  

Page 52: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Ques+on  Answering  in  Watson  (Deep  QA)  

Page 53: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Ques%on  Answering:  IBM’s  Watson  •  Won  Jeopardy  on  February  16,  2011!  

53  

WILLIAM WILKINSON’S “AN ACCOUNT OF THE PRINCIPALITIES OF

WALLACHIA AND MOLDOVIA” INSPIRED THIS AUTHOR’S

MOST FAMOUS NOVEL

Bram  Stoker  

Page 54: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

The  Architecture  of  Watson  

DocumentDocumentDocument

(1) Question Processing

From Text Resources

Focus Detection

Lexical Answer Type

Detection

Question

Document and

Passsage Retrieval passages

DocumentDocumentDocument

QuestionClassification

Parsing

Named Entity Tagging

Relation Extraction

Coreference

From Structured Data

Relation Retrieval

DBPedia Freebase

(2) Candidate Answer Generation

CandidateAnswer

CandidateAnswer

CandidateAnswerCandidateAnswerCandidate

AnswerCandidate

AnswerCandidateAnswerCandidate

AnswerCandidateAnswerCandidate

AnswerCandidate

AnswerCandidate

Answer

(3) Candidate Answer Scoring

Evidence Retrieval

and scoring

AnswerExtractionDocument titlesAnchor text

TextEvidenceSources

(4) Confidence

Merging and

Ranking

TextEvidenceSources

Time from DBPedia

Space from Facebook

Answer Type

Answerand

Confidence

CandidateAnswer

+ Confidence

CandidateAnswer

+ Confidence

CandidateAnswer

+ ConfidenceCandidate

Answer+

ConfidenceCandidateAnswer

+ Confidence

LogisticRegression

AnswerRanker

MergeEquivalentAnswers

Page 55: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Stage  1:  Ques%on  Processing  

•  Parsing  •  Named  En+ty  Tagging  •  Rela+on  Extrac+on  •  Focus  •  Answer  Type  •  Ques+on  Classifica+on  

55  

Page 56: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Poets  and  Poetry:  He  was  a  bank  clerk  in  the  Yukon  before  he  published  “Songs  of  a  Sourdough”  in  1907.      

THEATRE:  A  new  play  based  on  this  Sir  Arthur  Conan  Doyle  canine  classic  opened  on  the  London  stage  in  2007.      

authorof(focus,“Songs  of  a  sourdough”)    publish  (e1,  he,  “Songs  of  a  sourdough”)    in  (e2,  e1,  1907)    temporallink(publish(...),  1907)    

GEO  

COMPOSITION  PERSON  

YEAR  

YEAR  

Named  En)ty  and  Parse   Focus   Answer  Type   Rela)on  Extrac)on  

GEO  

Page 57: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Focus  extrac%on  

•  Focus:  the  part  of  the  ques+on  that  co-­‐refers  with  the  answer  

•  Replace  it  with  answer  to  find  a  suppor+ng  passage.    

•  Extracted  by  hand-­‐wrioen  rules    •  "Extract    any  noun  phrase  with  determiner  this”      •  “Extrac+ng  pronouns  she,  he,  hers,  him,  “!57  

Page 58: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Lexical  Answer  Type  

•  The  seman+c  class  of  the  answer  •  But  for  Jeopardy  the  TREC  answer  

   type  taxonomy  is  insufficient  •  DeepQA  team  inves+gated  20,000  ques+ons  •  100  named  en++es  only  covered  <50%  of  the  ques+ons!  •  Instead:  Extract  lots  of  words:  5,000  for  those  20,000  ques+ons  

58  

LOCATION

NUMERIC

ENTITY HUMAN

ABBREVIATIONDESCRIPTION

country city state

datepercent

money

sizedistance

individual

title

group

food

currency

animal

definition

reason expression

abbreviation

Page 59: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Lexical  Answer  Type  

•  Answer  types  extracted  by  hand-­‐wrioen  rules    •  Syntac+c  headword  of  the  focus.  • Words  that  are  coreferent  with  the  focus  •  Jeopardy!  category,  if  refers  to  compa+ble  en+ty.    

Poets  and  Poetry:  He  was  a  bank  clerk  in  the  Yukon  before  he  published  “Songs  of  a  Sourdough”  in  1907.    59  

Page 60: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Rela%on  Extrac%on  in  DeepQA  

•  For  the  most  frequent  30  rela+ons:  •  Hand-­‐wrioen  regular  expressions  •  AuthorOf:  •  Many  paoerns  such  as  one  to  deal  with:  •  a  Mary  Shelley  tale,  the  Saroyan  novel,  Twain’s  travel  books,  a1984  Tom  Clancy  thriller  

•  [Author]  [Prose]  •  For  the  rest:  distant  supervision  

60  

Page 61: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Stage  2:  Candidate  Answer  Genera%on  

61  

Page 62: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Extrac%ng  candidate  answers  from  triple  stores  

•  If  we  extracted  a  rela+on  from  the  ques+on  …  he  published  “Songs  of  a  sourdough”  

(author-of ?x “Songs of a sourdough”)!

•  We  just  query  a  triple  store  • Wikipedia  infoboxes,  DBpedia,  FreeBase,  etc.  •  born-­‐in(“Emma  Goldman”,  “June  27  1869”)  •  author-­‐of(“Cao  Xue  Qin”,  “Dream  of  the  Red  Chamber”)  •  author-­‐of(“Songs  of  a  sourdough”,  “Robert  Service”)     !

62  

Page 63: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Extrac%ng  candidate  answers  from  text:  get  documents/passages  

1.  Do  standard  IR-­‐based  QA  to  get  documents  Robert  Redford  and  Paul  Newman  starred  in  this  depression-­‐era  griter  flick.    (2.0  Robert  Redford)  (2.0  Paul  Newman)  star  depression  era  griter  (1.5  flick)    

 

!63  

Page 64: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Extrac%ng  answers  from  documents/passages  

•  Useful  fact:  Jeopardy!  answers  are  mostly  the  +tle  of  a  Wikipedia  document    •  If  the  document  is  a  Wikipedia  ar+cle,  just  take  the  +tle  •  If  not,  extract  all  noun  phrases  in  the  passage  that  are  Wikipedia  document  +tles  • Or  extract  all  anchor  text    <a>The Sting</a>!  

!

64  

Page 65: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Stage  3:  Candidate  Answer  Scoring  

•  Use  lots  of  sources  of  evidence  to  score  an  answer  •  more  than  50  scorers  

•  Lexical  answer  type  is  a  big  one  •  Different  in  DeepQA  than  in  pure  IR  factoid  QA  •  In  pure  IR  factoid  QA,  answer  type  is  used  to  strictly  filter  answers  

•  In  DeepQA,  answer  type  is  just  one  of  many  pieces  of  evidence  

65  

Page 66: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Lexical  Answer  Type  (LAT)  for  Scoring  Candidates  

•  Given:  •  candidate  answer      &        lexical  answer  type  

•  Return  a  score:  can  answer  can  be  a  subclass  of  this  answer  type?  •  Candidate:  “difficulty  swallowing”  &  LAT  “condi4on”  1.  Check  DBPedia,  WordNet,  etc  

•  difficulty  swallowing  -­‐>  Dbpedia  Dysphagia  -­‐>  WordNet  Dysphagia  •  condi4on-­‐>  WordNet  Condi4on  

2.  Check  if  “Dysphagia”  IS-­‐A  “Condi+on”  in  WordNet  •  Wordnet  for  dysphagia   66    

Page 67: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Rela%ons  for  scoring  

•  Q:  This  hockey  defenseman  ended  his  career  on  June  5,  2008  •  Passage:  On  June  5,  2008,  Wesley  announced  his  re+rement  

ater  his  20th  NHL  season  

•  Ques+on  and  passage  have  very  few  keywords  in  common  •  But  both  have  the  Dbpedia  rela+on  ActiveYearsEndDate() !

67  

Page 68: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Temporal  Reasoning  for  Scoring  Candidates  

•  Rela+on  databases  •  (and  obituaries,  biographical  dic+onaries,  etc.)  

•  IBM  Watson  ”In  1594  he  took  a  job  as  a  tax  collector  in  Andalusia”  Candidates:  •  Thoreau  is  a  bad  answer  (born  in  1817)  •  Cervantes  is  possible  (was  alive  in  1594)  

68  

Page 69: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Geospa%al  knowledge  (containment,  direc%onality,  borders)    

•  Beijing    is  a  good  answer  for    ”Asian  city”  •  California    is    ”southwest  of  Montana”  •  geonames.org:  

69  

Page 70: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Text-­‐retrieval-­‐based  answer  scorer  •  Generate  a  query  from  the  ques+on  and  retrieve  passages  •  Replace  the  focus  in  the  ques+on  with  the  candidate  answer  •  See  how  well  it  fits  the  passages.    •  Robert  Redford  and  Paul  Newman  starred  in  this  depression-­‐era  

gri`er  flick  •  Robert  Redford  and  Paul  Newman  starred  in  The  S+ng  

70  

Page 71: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Stage  4:  Answer  Merging  and  Scoring  

•  Now  we  have  a  list  candidate  answers  each  with  a  score  vector  •  J.F.K [.5 .4 1.2 33 .35 … ]!•  John F. Kennedy [.2 .56 5.3 2 …]!

•  Merge  equivalent  answers:  J.F.K.  and  John  F.  Kennedy  •  Use  Wikipedia  dic+onaries  that  list  synonyms:    •  JFK,  John  F.  Kennedy,  John  Fitzgerald  Kennedy,  Senator  John  F.  Kennedy,  President  Kennedy,  Jack  Kennedy  

•  Use  stemming  and  other  morphology  

71  

Page 72: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Dan  Jurafsky  

Stage  4:  Answer  Scoring  

•  Build  a  classifier  to  take  answers  and  a  score  vector  and  assign  a  probability  

•  Train  on  datasets  of  hand-­‐labeled  correct  and  incorrect  answers.  

72  

Page 73: Question Answering - Stanford University · 2016-12-21 · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms eat grass Worms eat grass worms eat grass Grass

Question Answering

Ques+on  Answering  in  Watson  (Deep  QA)  


Worms Eat Our Garbage Selected Worksheets from Trina Ball

WHAT IS PhotosynthesisAntelopes eat the grass, lions eat antelopes, but when the lions die, their bodies become food for the grass, and so the ‘circle of life’ goes on…. “The

Food Chains How energy is passed! Some animals eat only plants. They eat grass, leaves, tree bark, fruit, nuts, seeds, and parts of flowers. Say the

How To Eat Fried Worms Vocabulary Flash Cards. limp, walk awkwardly HOBBLE

How To Attract And Keep Birds In Your Garden...Meal worms are a popular food, according to the latest bird poll. Many birds will eat meal worms, dead or alive. The worms can be purchased

Worms, Worms, WORMS! Mrs. Howard’s Life Science. Worms, Worms, WORMS! Roundworms Flatworms Segmented worms Worm Lab Activity Mrs. Howard’s Life Science

Worms for Composting, Worms for Bait and Worms for Waste Processing

BY JORDAN TABER * A zebras food is really dry. * Zebras eat a variety of grass. * They eat shrubs, herbs, leaves, twigs, and bark. Zebras eatmany typesof

WORM FARMING - Compost Collective · 2019-07-17 · GETTING STARTED WHAT TO FEED YOUR WORMS Worm farming uses composting worms such as tiger worms to eat through a mixture of food

Launtau wild Buffalo By Calvin Lai. What does the buffalo eat They Eat grass and young shrubs for each three meals. They are vegetarian

Let Worms Eat Your Garbage, and Improve Your Soil!! Rhianna Simes Master Recycler Coordinator

Recycling With Worms - Study Guidebullfrogfilms.com/guides/wormsguide.pdfI. Worms Eat Our Garbage: Classroom Activities for a Better Environment Ages 9-14 Pages can be photocopied

Web viewJean-Claude et Philippe vont chez le médecin ... I eat gummy worms for dinner every day!

How to Eat Fried Worms

Worms and Parasites. Parasites Worms How does a person get worms?

27 September – 3 October. Frogs are carnivores, which mean they eat meat. Small frogs usually eat insects like spiders, bugs and worms. Bigger frogs eat

How to Eat Fried Worms By: Thomas Rockwell

pavelspoetry.com€¦  · Web viewTHE DREADFUL . In sweet spring grass recline and eat, Cool the blades that touch your feet, Nothing threatens, eat your fill . Nothing near you

Food chains and Webs - hspcrswart.co.za · Various food chains and how they are linked: • Worms, grasshoppers, mice, guinea fowls and antelope eat grass and plants. • Herbivores

How to Eat Fried Worms - WordPress.com€¦ · “How about worms?” Alan asked Billy. Tom’s sister’s cat squirmed out from under the porch and rubbed against Billy’s knee

& SustainAbility Servicesenvirohouse.org.au.tmp.anchor.net.au/sites/default/files... ·  · 2016-04-11tray up, and the worms will ‘eat ... • Black 100% recycled plastic • fly

Lecture #11 Worms, Worms & Worms Phylum Platyhelminthes Phylum Nematoda Phylum Annelida

British Reptiles€¦ · • amphibians; • spiders; • worms; • insects. Grass Snakes Grass snakes are very difficult to spot because they are fast moving. They are found in

OWNER’S MANUAL - Hungry Bin · worms breathe through their skin, anaerobic conditions prevent the worms from breathing properly, and may cause them to die. Worms can eat roughly

I will tell about horses. What horses eat? Horses eat grass,hay,apples,sugar,oat. Horses eat grass,hay,apples,sugar,oat

Worms Eat Your Garbage

Easy worm farming :: web - NSW Environment & Heritage...Easy worm farming. Pick a well-shaded spot so that your worms don’t get too hot. 1 Choose the site Worms like to eat vegetable

Warm Up List (use notes if needed) 3 animals that eat only plants and grass 3 animals that eat only meat 3 animals that eat both

Prairie Dog By Celeste Kowalik. Prairie Dogs Eat Plants, Grass, and Insects

Rhinos: They are very big. They have got four legs. They can run. They eat grass

Cows eat grass, don’t they? Results of a social science

Question Answering - Cleveland State Universitycis.csuohio.edu/~sschung/CIS660/QALectureNotesStanford.pdf · Dan%Jurafsky% 2 Ques%on(Answering(What do worms eat? worms eat what worms

Lab Worms, worms & worms. Class Turbellaria planaria dugesia

Why Do Chinese People Eat Snakes, Ants, And Worms for Medicine_The World of Chinese

Many animals need to eat plants to stay alive. These animals are called Herbivores. They eat plants such as leaves, grass, flowers, seeds, fruit, and