3a8 picture driven computing in assistive
TRANSCRIPT
www.aegis-project.eu
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY
Silvio Bonfiglio1 - George Kourousias2
1. FIMI, a Barco Company, Saronno (Italy)2. Synchrotron Radiation Facility, ELETTRA, Trieste (Italy)
1st International AEGIS Conference 7-8 October 2010, University of Seville
Domain: Improved Accessibility in ICT
Why?
It may positively affect and improve our society
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
Two Paths to Accessibility
i.e. a product that – without add-ons – can be experienced with similar ease of use by any user independently by the disability.
DIRECT ACCESS Universal Design (UD) – Design for All
INDIRECT ACCESS Multi-modal, Personalized Accessible Design
i.e. a product that is made accessible for a specific category of users by incorporating as an add on a specific AT componentor an extra interaction modality.
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
Picture Driven Computing
•We aim to introduce it as an Assistive Technology
•Adding a Modality to the GUI of the computer
•Accessibility through Indirect Access
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
We define as Picture-driven Computing the Computing paradigm that
has as core Data Abstraction the Visual Information in a 2-
dimensional spatial domain. This Visual Information is what intends
to be visible on a graphical Output device.
Example: a Picture-Driven Search
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
•Using Picture objects instead of keywords
Sikuli, a Picture-Driven language
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
•Developed recently at MIT by T. Yeh, T.H. Chang and R.C.Miller for the search and
automation of GUI elements using screenshots
•Based on Sikuli-script , a scripting module that allows programmers to use
screenshots of GUI elements to control them programmatically without any API.
•Python
•Open Source & Multiplatform
•We use it as our main paradigm for Picture-Driven Development of AT
Sikuli: How it Works
A. a computer vision library
B. a mechanism to programmatically “dictate” Input actions
C. a mature underlying language (Python) for traditional programatic control
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
Sikuli: An Example
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
Goal: clicks all the “minimize window” buttons on the screenPicture-Object Input: target image of the button
i) then the systems computer vision (A) will locate the x,y position of that on the screen
ii) (B) the “mouse control” will “send” the mouse on x,y, and
iii) through a for or while loop (C) will click (B) for all the targets that (A) has found.
Sikuli as an Accessibility Design enabler
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
Automation Scripts: where Sikuli script - following a predefined set of mouse
and keyboard actions- preforms a task that may be difficult otherwise to the user with a
disability.
Event Raisers: where the system can keep monitoring the user's screen for
specific graphical objects and raise an event/alarm and consequently trigger a set of
programmatic actions.
Bridge Systems: where applications can communicate with other software even if
an API is not provided.
We Identify the following 3 classes:
+ Hybrid approaches
Future work
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville
•Further research on Picture-driven Computing.
•Comparison of “indirect access” Accessibility Design versus the
“direct” one (Universal Design) for the computer as an AT-aid.
•Additional development of Sikuli AT Scripts.
•Evaluation of the technical issues and performance of Sikuli.
Acknowledgments
further info:
PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville