can you see me now? disability assistance in mobile devices mark stump graduate student department...
TRANSCRIPT
Can you see me now? Disability assistance in mobile devicesMark StumpGraduate StudentDepartment of Computer EngineeringRochester Institute of Technology
Agenda
•Introduction•Computer Vision •Android•Android/CV Fusion and Challenges•Problem•Preliminary Implementation and Results•Questions/Answers
Computer Vision
•How computers see and perceive the world
•Very diverse field with much up and coming development
Image Source:http://startupblog.files.wordpress.com/2009/10/webcam.jpg
Image Source: http://www.elexp.com/test/ping.jpg
Computer Vision•Subject Areas
▫Feature Detection Edge Detection Blob Detection Corner Detection
▫Recognition Optical Character
Recognition (OCR) Pose Estimation Gesture Recognition
•And many, many moreImage Source: http://stevehanov.ca/blog/index.php?id=62
Face Detection• Recognizes a face in an image
or video feed• Has many applications
▫Digital Cameras Autofocus
▫Security Face as a security Code
▫Advertising▫Energy Saving
Televisions detecting when someone is not looking at the screen and decreasing the brightness Image Source:
http://blog.cordiner.net/wp-content/uploads/2010/02/lenna_output.jpg
Face Detection• Various algorithms developed• Find faces through various
methods▫Comparing against a large
training set▫Looking for components of a
face in a scene (eyes, mouth, etc)
• With further development, can extract other information▫Gender▫Mood
Image Source: http://www.dcviews.com/press/images/Face-detection.jpg
Face Tracking•After a face is detected
▫Follow it as it moves through a scene
•Completed many ways▫Fast face detection▫Color Analysis▫Body analysis▫Template Matching
Image Source: http://www.globalspec.com/NpaPics/38/121978_020520089775_ExhibitPic.JPG
Android Introduction• Created by Google• “Software stack for mobile
devices that includes an operating system, middleware and key applications”
• Generally used in Cell Phones▫HTC Hero▫HTC Magic▫Motorola Droid
• Also beginning to branch other devices▫Dell Mini 5 Tablet PC▫Acer Aspire One D250 Netbook▫Barnes and Noble Nook eReader
Image Source: http://en.wikipedia.org/wiki/File:Android_and_cupcake.jpg
Android SDK•Freely available •Currently in release 2.2
(FroYo)•Provides for interfacing with
various portions of an Android device▫Screen▫Wireless (Bluetooth, Wifi)▫Motion Sensing▫GPS▫Camera▫Telephony
Image Source: http://developer.android.com/assets/images/home/froyo-android.png
Platform
•Applications primarily written in Java•Plugin available for Eclipse IDE•Runs of native “Dalvik Virtual Machine”
▫Register-based Architecture▫Used for platforms with limited resources
(like a phone)•Applications can also be written in C/C++
▫Android Native Development Kit (NDK)
Device Development (HTC Magic)• Android Phone, marketed by T-Mobile as
‘myTouch 3G’• Specifications
▫Operating System: Android 1.6▫ Processor: Qualcomm® MSM7200A™, 528 MHz▫ Memory
512 MB ROM 288 MB RAM
▫ Display: 3.2-inch TFT-LCD flat touch-sensitive screen with 320x480 HVGA resolution
▫ Connectivity Bluetooth 2.0 Wi-Fi 802.11 b/g Proprietary USB Interface (Mini-USB 2.0/Audio Jack in
one)▫ 3.2 Megapixel Camera with auto focus
Device Development (Motorola Droid)• Android Phone, marketed by Motorola on Verizon
Network• Specifications
▫Operating System: Android 2.1▫ Processor: Arm® Cortex™, 550 MHz▫ Memory
512 MB ROM 256 MB RAM
▫ Display: 3.7-inch 16:9 Widescreen touch-sensitive screen with 480x854 WVGA resolution
▫ Connectivity Bluetooth 2.1 + EDR Wi-Fi 802.11 b/g Micro USB Interface
▫ 5 Megapixel Camera with auto focus and dual-LED Flash
Image Source: http://www.softsailor.com/wp-content/uploads/2009/10/Motorola-Droid-Multimedia-Station.jpg
Image Source:http://smartphonesfans.com/wp-content/uploads/2010/03/motorola-droid.jpg
Device Development (HTC Evo)• Android Phone, marketed by HTC on Sprint
Network• Specifications
▫Operating System: Android 2.1▫ Processor: Qualcomm® Snapdragon™ 8650, 1 GHz▫ Memory
1 GB ROM 512 MB RAM
▫ Display: 4.3-inch touch-sensitive screen with 800x480 WVGA resolution
▫ Connectivity Bluetooth 2.1 + EDR Wi-Fi 802.11 b/g Micro USB Interface HDMI Output
▫ 8 Megapixel Camera with auto focus and dual-LED Flash With 1.3 MP front-facing camera
Image Source: http://nexus404.com/Blog/wp-content/uploads2/2010/03/Sprint-HTC-EVO-4G-1.jpg
Image Source:http://www.hypebeast.com/image/2010/03/htc-evo-phone-4.jpg
Android/Computer Vision
•Potential for almost any application to be developed
•Numerous Augmented Reality applications already available for platform
Image Source:http://www.uwplatt.edu/web/presentations/PennState/ar/pix/augmented-reality-hud.jpg
Google Goggles• Application created by Google Labs• “Visual Search Application”• Can take pictures of various objects and
complete searches based on them▫Landmarks▫Books▫Artwork▫Places▫Wine▫Logos▫Business Cards
Can pull phone numbers from a card to import into phone to call, store, etc.
Google Goggles - Landmark
Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_landmark.jpg
Google Goggles – Business Card
Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_contact.png
Google Goggles – Wine
Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_wine.jpg
Scotoma (Blind Spots)•“A spot in the visual
field in which vision is absent or deficient”
•Causes▫Macular Degeneration
Degeneration of the eye▫Optic neuritis
Inflammation, Degeneration of the Optic Nerve
▫And others..Image Source: http://en.wikipedia.org/wiki/File:CentralScotoma.jpg
Scotoma Treatments•Treatments center
around stopping the spot from growing larger
•For most cases, no cure•Effects (Very Situational)
▫Impairs functional acuity▫Can affect color
perception
Image Source: http://www.incm.cnrs-mrs.fr/pperso/portraits/scotoma.png
Scotoma Effects (Continued)• ‘Eccentric Viewing’
▫Also called Preferred Retinal Loci (PRL)
▫“This is similar to looking slightly away from an object at night in order to see it better” Dan Roberts, via
Macular Generation Support, www.mdsupport.org
Image Source: http://www.acbvi.org/albums/Vision/images/112p_donut.jpg
My Goal!
•To develop a system that will aid a person in compensating with various types of scotoma▫Through the use of
mobile devices
Finally, my development•Program to help those with various scotoma
interact•Limit (and possibly eliminate) ‘eccentric
viewing’ •Goals
▫Run efficiently and quickly▫Provide easy-to-use and configurable interface▫Be implementable on a mobile device▫Implement cutting-edge face detection methods
developed in-house ‘Random Projection’-Based Methods
Current Status•Prototype Implemented on a lab machine•Relatively Inefficient Implementation•Relies heavily on OpenCV graphical libraries for
interface▫OpenCV is a freely available computer vision
library that provides many standard algorithms and aids for implementation
▫Originally launched by Intel in 1999•Uses ‘off-the-shelf’ algorithms for face
detection▫Viola-Jones Detection
Initial Android Development•Program written to perform edge detection
on various images•Uses standard edge detection method
▫Sobel Kernel •Further extended to implement the on-
board camera on the phone
Initial Android Development
HTC Magic (Java-Based Implementation)
HTC Magic (OpenCV-Based Implementation)
Initial Android Development
Motorola Droid(Java-Based Implementation)
Motorola Droid(OpenCV-Based Implementation)
Future•Research into optimizing hand-offs between
native libraries and interface•Optimize/Port previous lab-machine code to
Android•Experiment with different interface types
for configuration and use
References/Further Information• Merriam-Webster’s Online Dictionary
▫ http://www.merriam-webster.com/dictionary• Causes of Scotoma
▫ http://www.wrongdiagnosis.com/s/scotoma/causes.htm• Scotoma – Texas School for the Blind and Visually
Impaired▫ http://www.tsbvi.edu/Education/anomalies/scotoma.htm
• Eccentric Viewing▫ http://www.mdsupport.org/library/eccentric.html
• Energy saving of TV by face detection▫ http://portal.acm.org.offcampus.lib.washington.edu/citati
on.cfm?doid=1389586.1389693• Google Goggles Information
▫ http://www.google.com/mobile/goggles/