trailblazer: enabling blind web users to blaze trails through the web
DESCRIPTION
IUI talk on TrailBlazer, work by Jeffrey P. Bigham, Tessa Lau, and Jeffrey Nichols.TRANSCRIPT
![Page 1: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/1.jpg)
Jeffrey P. BighamUniversity of Washington [email protected]
Enabling Blind Users to Blaze Trails Through the Web
Tessa Lau and Jeffrey NicholsIBM Almaden Research
{tessalau, jwnichols}@us.ibm.com
start
![Page 2: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/2.jpg)
Scale of Access
37 Million Blind People in the World(125 million with low vision)
Increasing Aging Population Situational Impairments
Access technology benefits 25% of web users1
[1] “The Market for Accessible Technology. “ Forrester Research, 2003.
![Page 3: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/3.jpg)
Web Potential for Blind People
Web Advantages Convertible to Accessible Form Shop From Home
Blind People Read 7x as Many Books
The Law U.S. Government Requires Accessibility1,2
NFB vs. Target Corporation2
Making Access Possible is Not Enough Should be Intuitive and Fun
[1] Section 508 of the U.S. Rehabilitation Act.[2] Americans with Disabilities Act.
![Page 4: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/4.jpg)
![Page 5: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/5.jpg)
![Page 6: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/6.jpg)
X
X
X
X
h
h
h
e b b
START
Keyboard Shortcuts
![Page 7: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/7.jpg)
X
X
X
X
<span class=“heading”>
e b b
START
<span class=“heading”>
<span class=“heading”>
.heading { font-weight: bold; }
Keyboard Shortcuts
![Page 8: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/8.jpg)
Task: Placing Book on Hold
1. Enter “Book You Want”
2. Click “Find” Button
3. Find Correct Result
4. Click “Place Hold” Button
5. Click “Request” Button
6. Verify Queue Position
![Page 9: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/9.jpg)
Task: Placing Book on Hold
1. Enter “Book You Want”
2. Click “Find” Button
3. Find Correct Result
4. Click “Place Hold” Button
5. Click “Request” Button
6. Verify Queue Position
![Page 10: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/10.jpg)
Task: Placing Book on Hold
1. Enter “Book You Want”
2. Click “Find” Button
3. Find Correct Result
4. Click “Place Hold” Button
5. Click “Request” Button
6. Verify Queue Position
![Page 11: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/11.jpg)
Task: Placing Book on Hold
1. Enter “Book You Want”
2. Click “Find” Button
3. Find Correct Result
4. Click “Place Hold” Button
5. Click “Request” Button
6. Verify Queue Position
![Page 12: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/12.jpg)
How can blind people more effectively complete web tasks?
Avoid Linear Searches Direct Users to Important Content
Users Maintain Control Interface should be understandable Guide users not direct them
Flexible and Adaptable Work in full web domain
Compatibility with Tools Users KnowSupplement to existing tools (screen readers)
![Page 13: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/13.jpg)
CoScripter1:Recording and Replaying Tasks
Pseudo-Natural Language Commands
[1] Little et al. “Koala: capture, share, automate, personalize business processes on the web.” CHI 2007.
1. go to “http://www.united.com”2. enter your “flight number” (e.g., 0168) into the “Flight number” textbox3. enter your “airport” (e.g., SJC) into the “From:” textbox4. click the “Check” button
CoScript Repository
![Page 14: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/14.jpg)
Example
TrailBlazerPlacing a book on hold at the Seattle Public Library.
![Page 15: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/15.jpg)
Set Focus & Provide Contextual Help
Step 2 of 15: select “Books” from the “Search” listbox
“Search” Listbox
“Previous Step” Button
“Play from Here” Button
“Next Step” Button
Starts Here
...
...
Step description.
Inline PlaybackControls
![Page 16: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/16.jpg)
Formative Evaluation
![Page 17: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/17.jpg)
Generalizing with Suggestions
Choose from one of thefollowing suggestions:
Click the “Check” button.Enter into the “From:” textbox.Enter into the “To:” textbox.Select from the “Date:” listbox.Select from the “Time:” listbox.
![Page 18: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/18.jpg)
Providing Suggestions
User enters a short task description (query) “flight status on united” “place hold on book at Seattle library”
(etc.)
Machine learning model ranks all actions.
FeaturesTask DescriptionRelated Scripts from the Script RepositoryUser’s History of Actions
Apply existing scripts to new tasks or new sitesNo need to explicitly find script first
Click the “profile” link.Click the “Home” link.Click the “Books” link.…Enter into the “Name” textbox.Enter into the “Zip” textbox.…Click the “Submit” button.Click the “Check Now” button.…Turn on the “Yes” Radio Button.….
Example Action List
x1 x2 x6
…
y
![Page 19: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/19.jpg)
0.080.270.100.760.20
Generalizing Scripts to New Sites
“flight status on united”Task Description
Repository
1. go to “www.aa.com”2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox3. click the fifth “GO” button
Flight Status on American Airlines
1. go to “http://www.southwest.com/”2. select “SJC” from the “Select the Departure City:” listbox3. enter your “flight number” into the first “Enter the Flight Number” textbox4. click the first “View Flight Status Information” button
Flight Status on Southwest Airlines
Click “my profile” linkClick “Flight Information” linkEnter into “Mileage Plus” textboxEnter into “flight number” textboxClick “Check” button
Possible Actions
similarity( , )
![Page 20: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/20.jpg)
Click “my profile” linkEnter into ‘password’ password boxClick “Login” buttonEnter into “flight number” textboxClick “Check” button
0.080.880.100.360.20
Using History to Match Sub-TasksLast Action
Repository
Update Facebook Status
Reserve UW Meeting Room
Possible Actions
1. go to “www.facebook.com”2. enter your “facebook username” into the “username” textbox3. enter your “facebook password” into the password box4. Click “Login” button5. …
7. …8. enter your “CSE login” into the “username” textbox9. enter your “CSE password” into the password box10. …
0.000.870.460.140.04
0.000.750.420.13
“enter into ‘username’ textbox”
similarity( , )
![Page 21: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/21.jpg)
Evaluation
Tested on 102 Actions 15 Most Popular CoScripts User-provided title used as task description 6.8 steps per script 42 possible actions
Leave-One-Out Cross-Validation Removed from repository too
Top 5 in 76% of cases
CoScriptRepository(5.1 words on avg.)
Suggestions Provided
![Page 22: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/22.jpg)
Ongoing and Future Work
User Studies What happens when TrailBlazer is wrong? Incentives vs. Cost
Better Model Continuous Learning Popularity, Past Behavior, etc.
Improved Web Access Voice Over Mobile Phones Small Screens
![Page 23: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/23.jpg)
Conclusion
TrailBlazer makes web access more efficient Record, replay and share trails Targets non-visual use Uses existing repository of trails
Suggestion Component Finds appropriate trails Generalizes existing scripts to new tasks and sites
![Page 24: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/24.jpg)
The End
Thanks to:Allen Cypher, Clemens Drews, Jimmy Lin, Yevgen Borodin, Charles Chen and T.V. Raman.
Want to hire me?
![Page 25: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/25.jpg)
Directing Users to Content
Direct to Content Avoid inefficient linear searches
5 Participants/20 Regions Mechanical Turk
3 Commonalities Semantic Desciption (78%) Heading of Region (53%) Words Used in Region (37%)
Clip the “search results” region
![Page 26: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/26.jpg)
Directing Users to Content
Direct to Content Avoid linear searches
5 Descriptions of 20 Regions Mechanical Turk 27 Users
Clip the “search results” region
Clip the region containing “batting average”
![Page 27: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/27.jpg)
0.080.270.100.760.20
Generalizing Scripts to New Sites
“flight status on united”Task Description
Repository
1. go to “www.aa.com”2. enter your “Flight Number” (e.g. 144) into the “Flight Number:” textbox3. click the fifth “GO” button
Flight Status on American Airlines
1. go to “http://www.southwest.com/”2. select “SJC” from the “Select the Departure City:” listbox3. enter your “flight number” into the first “Enter the Flight Number” textbox4. click the first “View Flight Status Information” button
Flight Status on Southwest Airlines
Click “my profile” linkClick “Flight Information” linkEnter into “Mileage Plus” textboxEnter into “flight number” textboxClick “Check” button
Possible Actions Actions a Scripts s1: enter1: flight1: number1: textbox
2: button1: city2: click1: departure3: enter4: flight3: go1: information1: listbox4: number1: select1: SJC1: status2: textbox1: view2: your
Similarity = cos(a, s)
![Page 28: Trailblazer: Enabling Blind Web Users to Blaze Trails Through the Web](https://reader033.vdocuments.site/reader033/viewer/2022060115/5579a976d8b42ac1148b4cfa/html5/thumbnails/28.jpg)
Click “my profile” linkEnter into ‘password’ password boxClick “Login” buttonEnter into “flight number” textboxClick “Check” button
0.080.880.100.760.20
Matching Partial ScriptsLast Action
Repository
Update Facebook Status
Reserve UW Meeting Room
Possible Actions Action a
Scripts s
1: enter2: password1: box
box: 0.9button: 0.1click: 0.0CSE: 1.2enter: 2.5facebook: 1.2go: 0.0login: 0.1password: 1.8textbox: 1.6
W(si) = d * W(si-1) + cos(t, si)
1. go to “www.facebook.com”2. enter your “facebook username” into the “username” textbox3. enter your “facebook password” into the password box4. Click “Login” button5. …
7. …8. enter your “CSE login” into the “username” textbox9. enter your “CSE password” into the password box10. …
0.30 * 0.000.30 * 0.00 + 0.870.30 * 0.87 + 0.200.30 * 0.46 + 0.000.30 * 0.14 + 0.00
0.30 * 0.00 + 0.000.30 * 0.00 + 0.750.30 * 0.75 + 0.200.30 * 0.43 + …
0.000.870.460.140.04
0.000.750.420.13
d = 0.30
“enter into ‘username’ textbox”