team iralar breanna heidenburg -- michael lenisa -- daniel wentzel advisor: dr. malinowski

Team IRALARBreanna Heidenburg -- Michael Lenisa -- Daniel Wentzel

Advisor: Dr. Malinowski

The Project◦ Why is it important

The Goals◦ System breakdown

Image recognition Point transformation User Interface

The Results

What is our project?

Track a user’s eye and use the information to control a computer cursor

Enhances Human Computer Interaction

◦Speed of use◦Hands-free use

3 Part System◦ Image Processing Application◦ Calibration and Mapping system◦ GUI designed for gaze-based interaction

Systems developed concurrently and independently

Separate Applications at run-time

Hardware and Image Processing Application

Hardware◦ Camera

QuickCam Pro for Notebooks Visible Spectrum Camera

◦ Polarizer Tiffen 25mm polarizing filter Removes glare from eye reflections

◦ Lighting Diffuse LEDs Slightly distracting to the user, but necessary to provide

light for the camera

LitEye LE-500◦ High resolution (SVGA)◦ Color Display◦ Translucent or opaque operation◦ Stationary relative to user’s eye

Real time pupil tracking system◦ Developed in C using OpenCV image processing libraries◦ Traditional image processing and blob tracking

Capabilities◦ Locate and determine center of pupil in image◦ Low light and high reflection environments◦ All eye colors◦ Data logging and static test modes◦ Packaged into self contained Windows installer for easy

deployment onto any computer

The Process◦ Capture Image◦ Extract Red Channel◦ Smooth image◦ Apply binary threshold◦ Locate blobs◦ Reject false positives◦ Determine center of pupil blob◦ Adapt threshold◦ Repeat

Summary◦The Good

Dynamically adapts to changing lighting conditions and eye types

Maintains performance in low-light and specularly noisy conditions

◦The Bad Still relies on Logitech camera drivers Extreme reflections still cause problems

Raw Capture

Pre-Processed Image

Completed Recognition

Examples of performance in poor conditions

Low Light Difficult False Positive

Calibration and point mapping

System for mapping the location of the center of the pupil to a pixel on a computer screen

Reasons◦ Geometry

The eye is not flat but a screen is not

◦ User Customization All eyes are different Everyone wears the HMD differently

◦ User Training Calibration system also acts as a quick tutorial

3 dimensional best fit plane◦ Currently using a 4th degree best fit

Xpix = A1 + Xeye*B1 + Yeye*C1

Ypix = A2 + Xeye*B2 + Yeye*C2

Calibration sub-system determines these coefficients

How do we solve the problem?

◦ Multiple Variable Linear Regression – Least Squares

Y = B0 + B1x1 + … + Bkxk

◦ Uses matrix algebra to obtain a coefficient matrixB[] = (X’X)-1X’Y

Results◦ pixelX = 224.9 + 4.8*eyeX – 2.9*eyeY◦ pixelY = 1612.5 + 2.0*eyeX – 5.6*eyeY

B[] B[] == ((XX’’XX))-1-1XX’’YY

How do we click?◦ Monitor eye movements◦ Identify pauses

◦ When eye position is within a small percentage for a certain amount of time

◦ Currently set at 5 frames (~200 mS)◦ Generally, it takes 230 mS for a hand to click a

mouse.

Communications and Custom GUI interface

Why is it important?◦ Allows multiple processes to communicate◦ Allows relay of time sensitive information

UDP vs. TCP◦ UDP -> ‘Connectionless’◦ TCP -> ‘Connection oriented’

Multi-threading◦ Necessary for running multiple pieces of code in a

single process

Multi-threading issues present

Multi-threading◦ Issues with public variable usage◦ Solutions we are using:

Critical Section Raises thread priority (does not allow for interrupts)

Receive request for

data (over UDP)

Raise Thread priority

Lower Thread Priority

Reply to

request

Read Variable

Custom GUI for Gaze Tracking Applications

Why?◦ Gaze tracking accuracy limited by inherent

properties of human vision system◦ Traditional GUI too small and intrusive for use

with transparent HMD◦ Demonstrate applications of gaze tracking

Achieved via a combination of Hardware and Software

Software:WPF & C#◦ XAML

(eXtensible Application Markup Language)◦ Very similar to XML◦ Uses ‘code-behind’ in a similar style to javascript

‘code-behind’ is in C#

Multiple applications have been created within the interface

Multiple screens for functionality testing◦ even games

Additional ability to minimize interface and interact with external applications is present

Calibration Screen Main Screen

What did our test results show?

Who helped us out?

Northrop Grumman LitEye

Dr. Malinowski and the EE faculty Mr. Mattus & Mr. Schmidt