![Page 1: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/1.jpg)
1 / 28
http://www.flickr.com/photos/joshmichtom/4311110421/
Guardian: A Crowd-Powered Spoken Dialog System for Web APIs
Ting-Hao (Kenneth) HuangCarnegie Mellon University
Walter S. LaseckiUniversity of Michigan
Jeffrey P. BighamCarnegie Mellon University HCOMP
2015
![Page 2: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/2.jpg)
2 / 28
What time is it?
It’s 9:30.
Kenneth’s apartment.
![Page 3: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/3.jpg)
3 / 28
How was the Pirates game last night?
!
Kenneth’s apartment.
![Page 4: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/4.jpg)
4 / 28
How was the Steelers game yesterday?
!
Kenneth’s apartment.
![Page 5: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/5.jpg)
5 / 28
Is the movie Martian still playing in theaters?
!
Kenneth’s apartment.
![Page 6: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/6.jpg)
6 / 28
Use Web APIs to EmpowerDialog Systems
![Page 7: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/7.jpg)
7 / 28
Gap between User & Machine
?
![Page 8: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/8.jpg)
8 / 28
A Crowdsourcing Solution
![Page 9: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/9.jpg)
9 / 28
Two Challenges
term
location
Hi, I’m in San Diego.Any Chinese restaurants here?
Define Parameters
Extract Parameters
![Page 10: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/10.jpg)
10 / 28
How Do Dialog Systems Usually Do?
term
location
Hi, I’m in San Diego.Any Chinese restaurants here?
Define Parameters
Extract Parameters
![Page 11: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/11.jpg)
11 / 28
Bridging this Gap is Expensive
• Define Parameters requires Experts– Experts are expensive.– Most services are not designed for dialog systems.– Unsupervised Slot Induction
• Extract Parameters requires Data– (Which we don’t have.)– Supervised Slot Filling
• Slot Filling / Entity Recognition– No labeled data
• State Tracking– No dialogue data
– Unsupervised Slot Filling
![Page 12: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/12.jpg)
12 / 28
Can the Crowd Do It?
term
location
Hi, I’m in San Diego.Any Chinese restaurants here?
Define Parameters
Extract Parameters
![Page 13: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/13.jpg)
13 / 28
Define Parameters
term
location
Hi, I’m in San Diego.Any Chinese restaurants here?
Define Parameters
Extract Parameters
![Page 14: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/14.jpg)
14 / 28
How machines understand a Web API?
1. Use which parameters ?
2. Ask user what questions to elicit these parameters?
Yelp Search API 2.0 has22 parameters.
![Page 15: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/15.jpg)
15 / 28
Parameter Rating Problem
offset
term
location
sw_latitude
sw_longitude
category_filter
accuracy
deals_filter
radius_filter
...
offset
term
location
sw_latitude
sw_longitude
category_filter
accuracy
deals_filter
radius_filter
...
Pick good parameters for the dialog system.
![Page 16: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/16.jpg)
16 / 28
How about just do a survey?
TaskParameter Name / Desc
![Page 17: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/17.jpg)
17 / 28
Baselines
MAP MRR0
0.10.20.30.40.50.60.70.80.9
1
Not UnnaturalAsk SiriAsk a Friend
Average results of 8 Web APIs’ parameters
Results are not so good...
![Page 18: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/18.jpg)
18 / 28
Match Questions with Parameters
offset
I like Chinese food.
What do you want to eat?? !
I’m in Pittsburgh.
Which city are you in?? !
Dinner.
Is it dinner or lunch?? ! ...
location?
!
term?
!!
?!
?!
?!
?!category_filter
?!
?!
?!
?!
?!
?!
?!
?! ?
!?
! ?!
?!
?!
?!
?!
?!
?!?
!
?!
?!
?!
?!
?!?
!
?!
?!
?!
?!?
!?
!?
!
?!
?!
?!
?!
term
location
sw_latitude
sw_longitude
category_filter
Better Parameter
Yelp API
Question Collection
Parameter Filtering
Qestion-Parameter Matching
![Page 19: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/19.jpg)
19 / 28
Evaluation on Parameter Ranking
MAP MRR0
0.10.20.30.40.50.60.70.80.9
1Question MatchingNot UnnaturalAsk SiriAsk a Friend
Question Matching outperforms all baselines.
Average results of 8 Web APIs’ parameters
![Page 20: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/20.jpg)
20 / 28
Questions Collected Already!
1. Use which parameters ?
2. Ask user what questions to elicit these parameters?
Yelp Search API 2.0 has22 parameters.
![Page 21: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/21.jpg)
21 / 28
Extract Parameters
term
location
Define Parameters
Extract Parameters
Hi, I’m in San Diego.Any Chinese restaurants here?
![Page 22: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/22.jpg)
22 / 28
Dialog ESP Game
Hi, I’m in San Diego.
AnswerAggregate
Location =San Diego
Recr
uite
d Pl
ayer
s
Time Constraint
![Page 23: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/23.jpg)
23 / 28
Guardian: A Crowd-Powered Spoken Dialog System for Web APIs
3
2 Call Web APIHi, I’m in San Diego.Any Chinese restaurants here?
1 Talk and Extract Parameter
Interpret Result to User
Mandarin Wok Restaurant is good ! It’s on 4227 Balboa Ave.
term = Chinese
location = San Diego
YelpSearchAPI 2.0
{ ... "name": "Mandarin Wok Restaurant”,..."address":["4227Balboa Ave”,...], …}
JSON
![Page 24: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/24.jpg)
24 / 28
Engineering Challenges
• Real-time Response ……..…..……… Retainer Model• Converse with User ……………………………….. Chorus • Speech Recognition ………………... Web Speech API• Parameter Extraction ………..…… Dialog ESP-Game• JSON Visualization ………………..….. JSON Visualizer• Response Generation Assistant ………………. jQuery• Workflow Control ………………. Game-like Interface• Dialog Management …………. Finite-state Machine• Crowdsourcing Platform …………. Mechanical Turk
![Page 25: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/25.jpg)
25 / 28
System Evaluation
Web API
TaskFind Chineserestaurants in
Pittsburgh.
Check current weather
by using a zip code.
Find information of “Titanic”.
Valid JSON 9 / 10 9 / 10 6 / 10
Task Completion 10 / 10 9 / 10 10 / 10
Domain Referenced TCR 0.96 0.94 0.88
![Page 26: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/26.jpg)
26 / 28
Guardian: A Hybrid Framework
Annotate Data on the Fly !
![Page 27: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/27.jpg)
27 / 28
What’s next?
• More Automations– Slot Filling / Entity Recognition– Dialog Management– Response Generation
• 1,000+ APIs?• Future of Dialog Systems– What if you can really talk to a machine…– On wearable device?
![Page 28: Guardian: A Crowd-Powered Spoken Dialog System for Web APIs](https://reader036.vdocuments.site/reader036/viewer/2022062400/5879701d1a28ab37368b5e99/html5/thumbnails/28.jpg)
28 / 28
Thank you!
( SORT OF )
http://www.flickr.com/photos/joshmichtom/4311110421/