interacve machines: from quesons to experience
TRANSCRIPT
Interac(veMachines:FromQues(onstoExperience
GiuseppeRiccardiUniversityofTrento,Italy
TheVision
• WhyNaturalLanguageQues(on/Answer?• Interac(vemachines
• Interac(veeco‐systems
Forthesakeofaskingques(ons• Knowledgewilldoit?
– Whywaterincreasesitsvolumewhenitfreezes?
Forthesakeofaskingques(ons• Knowledgewilldoit?
Why water increases its volume when it freezes?
– GOOGLE.comCHACHA.com
Q:WhenwaterfreezesitsvolumeincreasesbyhowmuchpercentAA:Whenwaterfreezesat0°Citsvolumeincreasesbyabout9%underSTP(StandardTemperatureandPressure).
Isthatcorrect?
Knowledge‐BasedQ&A
• Precisionoftheanswers• Accountabilityoftheresources• Limita(onofDocumentspace(vsWEB)
• Bri^lenessofthenaturallanguageinterface
“Computersareuseless.Theycanonlygiveanswers.”—PabloPicasso,(1881‐1973).
SearchEngineQueries
Searchvastamountofdatausing(almost)naturallanguage
Thesearchenginesimple“drill”1. ShortQuery(“universita’Trento“,“rifugio
PassoSanPellegrino“)
2. HumanParseofRetrievedDocuments3. HumanDecisionwhetherto
A. FollowuponlinksORB. GOTO1
Knowledge‐BasedQ&A
DEEPQAProject• IBMResearchFundedproject
– BuildQAengine(WATSON)capableofchallengingHumansintheJeopardy!Game.
– Domainisvirtuallyopen.NODBratherunstructureddata(lightorfullsupervision).
• OpenCollabora(veResearchAgreementbtwUTrentoandIBMYorkTown– MIT,UTexas,CMU,USC
• Utrento– Interac(veMachinestoresolveQues(onsintoAc(ons/Decisions/Tasks
– MachineLearningmodelsforParsingSentencesforQAClassifica(on,RerankingHypotheses(targetAnswers)
NaturalLanguageParsingandapplica(onstoDEEPQA
wAlessandroMoschi=
Outline• Mo(va(ons
• TwoimportantproblemsinJeopardy– Ques(onClassifica(on– AnswerSelec(on
• Results• Conclusions
LetusConsideroneJeopardyCue
• Whenhitbyelectrons,aphosphorgivesoffelectromagneGcenergyinthisform
• Solu(ons:photons/light• Electrons,phosphorandelectromagneGcenergyareinarela(onshipwhichgivesthesolu(on
• Howrepresen(ngandusingsuchrela(oninamachine?
Issuesandsolu(ons• Howexploi(ngshallowseman(cinforma(on?• Howdealingwithnoiseanderrorsofseman(cparsing?
• Designinghand‐crakedrulestodealwithanykindoferrorsornoiseisnotrealis(c
• Weneedsta(s(calmethodsasthey:– “canac(vatetherules”thatmaximizetheprobabilityofsuccess
– canautoma(callylearnsuchprobabili(esfromdata
ImportantTasksinJeopardy
Ques?on
Ques(on/TopicAnalysis
Hypothesis&EvidenceScoring
SokFiltering Synthesis
FinalMerging&Ranking
QueryDecomposi(on
HypothesisGenera(on
Answer&Confidence
AnswerReranking
Ques(onClassifica(on
Classifica(oninDefini(onvsnotDefini(oninJeopardy
• Usually,todothisistoloseagamewithoutplayingit
(solu(on:forfeit)
• Whenhitbyelectrons,aphosphorgivesoffelectromagneGcenergyinthisform
OurApproach
• SupervisedApproach– Posi(veandnega(veexamplesofdefini(onques(ons
– Syntac(cinforma(onisintui(velyimportant• Applyoff‐the‐shelfparsers
– Asthisisanewtask,toextractfeatures,weexploittreekernels,e.g.thePar(alTreeKernel(Moschim,ECML2006)
ParseTree
ROOT
SBARQ
WHADVP
WRB
When
S
VP
VBN
hit
PP
IN
by
NP
NNS
electrons
,
,
NP
DT
a
NN
phosphor
VP
VBZ
gives
PRP
RP
off
NP
NP
JJ
electromagnetic
NN
energy
PP
IN
in
NP
DT
this
NN
form
ROOT
SBARQ
WHADVP
WRB
When
S
VP
VBN
hit
PP
IN
by
NP
NNS
electrons
,
,
NP
DT
a
NN
phosphor
VP
VBZ
gives
PRP
RP
off
NP
NP
JJ
electromagnetic
NN
energy
PP
IN
in
NP
DT
this
NN
form
PAS
A0
electrons
predicate
hit
AM
When
PAS
A0
a phosphor
predicate
give off
A1
energy
AM
in this form
PAS
A0
electrons
predicate
hit
A1
it
AM
if
PAS
A0
a phosphor
predicate
give off
A1
photons
1
Similaritybasedonthenumberofcommonsubstructures
NP
D N
VP
V
hit
a phosphor
Apor(onofthesubstructureset
ExperimentsonJeopardyQues(onClassifica(on
Model Precision Recall F1 RBC 28.27 70.59 40.38 BOW 46.55 50.65 48.51 CSK 39.07 77.12 51.86 PTK 47.84 57.84 52.37 PTK+WSK+CSK+RBC
67.66 66.99 67.32
RuleBasedClassifier(RBC)
OnlyWordOverlap(BOW) CategorySubsequences(CSK)
Par(alTreeKernel(PTK)
66.7%ofrela(veimprovementontherule‐basedclassifier
OurApproachtoAnswerRe‐ranking
• Learnaclassifierof<ques(on,answer>pairs– Posi(ve:theansweriscorrect– Nega(ve:otherwise
• Kernelapproach– Severalkernelsappliedtobothques(onsandanswers
AnexampleofJeopardyQues(on
Baseline Model
Ques(on
Answer
Methodology:1‐ApplyingPTKwithoutanyextraannota(onandevaluatethemodelasbaseline.
!"#$%&'(
)'$*#+(
Best Model Methodology:1‐Applyinglemma(za(onandstemminginleaveslevel.2‐Addananchortopre‐terminalandhigherlevelsifthesub‐treesaresharedinQandA.3‐Ignorestopwordsinmatchingprocedure.Ques(on
Answer
!"#$%&'(
Precision/Recall Evaluation
Precision Recall F1
BaselinewithPTK(Par(alTreeKernel)
17.1% 59.37% 26.56
BestModel(STK)Syntac(cTreeKernel
17.71% 5.59% 8.50
BestModelwithPTK 24.10% 61.12% 34.57
Ranking Precision at 1 and 5 best
Evalua?onon1‐best
Evalua?onon5‐best
Accuracy Accuracy
Baseline 0.22 0.22
PTK(Baseline) 0.23 0.19
PTK(BestModel) 0.33 0.23
Conclusions• WeuseofpowerfulMLalgorithms
– e.g.SupportVectorMachines– robusttonoise
• Abstractrepresenta(onsofexamples– Similarityfunc(ons(KernelMethods)
• ModelingQues(onseman(cswithadvancedsyntac(candshallowseman(cstructures
• SokwarealreadyintegratedintheJeopardySystem
Ques(onstoAc(ons
SocialNetworks
GiuseppeRiccardi 33
SocialInterac(onsCustomer CompanyCi?zen PublicAdmin.Pa?ent DoctorCi?zen Poli?calPartyUser UserWorker SupervisorPeer PeerRela?ve Rela?ve…………… ……………
Cancomputersbe
partofthesocialnetwork?
NetworkofAgents
HAgent
MAgent2
Yahoo!
Kayak
HAgent2
Blog
HAgent1
MAgent1
GiuseppeRiccardi 36
HAgent
HAgent2
Chacha
Kayak
Blog1
Blog2
HAgent1
MAgent1
Informa(onBandwidth(Entropy)Cost($,(me2taskcomple(on)Reliability(Trust,Performance)
Human–ComputerInterac(on
H
ARTICULATION
OBSERVATION
TYPING
GESTURES
SPEECH
VISION
VISION
HEARING
Spring2010 37……….
Human–ComputerInterac(on
H C
ARTICULATION OBSERVATION
PRESENTATIONOBSERVATION
TYPING
GESTURES
SPEECH
VISION
VISION
GUI
TTSHEARING
Spring2010 38……….
……….
INTERFACE
Spring2010 39
Automatic Speech Understanding
FeatureExtrac(on A
Pa^ernMatching Ŵ
Given the acoustic observation sequence A=a1,a2,…,am,
what is the most likely “word” sequence W=w1,w2,…,wn?
LEHTAHSPREYLetuspray
Le^ucespray
Natural Language Query to DB
Find the best flight from New York to Paris tomorrow business class!
TASK:Transac?onal
USERCONSTRAINTS
Object
Spring 2009 41
Machine Understanding • User:
“Find the best flight from New York to Paris tomorrow business class”
• Speech Recognition: “ Find the bass flight from Newark to Paris
tomorrow business class”
• Modeling Uncertainty: – @action=Request-Reservation (0.9) – @origin=Newark (0.5) – @time-departure=Tuesday (0.7) – @destination=Paris (0.8)
Humanerrstoo!
Cooperative Task SW-HW Help Desk
UHiGoodMorningOHi,HowMayIHelpYou?UIamRobertaSicconicalling
fromCulturalAffairsatCityHall.
UIhadmadearequestforapasswordchangeyesterday
OOkdoyouhavetherequesttrackid?
UUhmNoIcannotfindOOkdoyouhavethedateof
therequest?UWellthatwasyesterdayO...okIthinkIcanfindit..IgotitOIt’sforapasswordreset.
URight.TheproblemisthatIchangedthepasswordwhenIfirstloggedin..
OYouweresupposedtochangefirst(meyouloggedin.Nowlet’strytogethertologin
OcanyoutellmeyouRVSofyourcomputer
UWellletmesee.ThisisanewPCtome.WherecanIfindit?
OUsuallythetagisrightnexttothebaseofthechassynexttothepowerswitch.Itreads“inventariose^oreinforma(co”.
UInventario..Se^ore...
PersonalIden?fica?on
ProblemStatementTicketRecordRetrieval
ProblemResolu?on(USER)
ProblemResolu?on(PARTI)
10/13/2008 GiuseppeRiccardi
DialogModels(DM)
MarkovDecisionProcesses
Conversa(onalAgents
Conversa(onalAgents
Observa(ons+Interpreta(on+Uncertainty+Decisions+Interac(on=Experience
Evalua(on• Howgoodaremachines?
– Accomplishingtasks
– Acceptancebytheusers,bythenetwork– LifeSpan(Evolveover(me)
• “Research Systems are difficult to evaluate while Commercial systems are ea$y to evaluate “(Paek,2007)
DEMOS
Conclusion
• ThereareQ&Amachinetaskswhichwillgive(par(al)benefits
• Machinesmayfulfill(social)roles
• Theul?mategoalistounderstandthecollabora?veinterac?onofmachine/humanagents– Language– Interac?onModel– SocialEqua?on