Research Background:Depth Exam Presentation

Susan Kolakowski

March 20, 2006

Committee:

Juan Cockburn, Chair

Jeff Pelz, Adviser

Andrew Herbert

Mitchell Rosen

Carl Salvaggio

Research Background

• Introduction

• Human Visual System

• Eye Movements

• Eye Trackers– RIT Wearable Eye Tracker

• My Research

Introduction

• Why are eye trackers used?– Objective measure of where people look– Interest in Human Visual System

• Examples:– Understanding Behaviors: How do humans read?– Improving Skill: Train people to move their eyes as

an expert would.– Improving Quality: What parts of an image are

important to the image’s overall quality?

The Human Eye

IrisPupil Cornea

Ciliary Muscle

Retina

Optic Nerve

Fovea

Eyelens

Optic Axis

Human Visual System

• What we see is determined by– How our rods and cones are connected and

distributed– How our brain processes this information– What we already accept as truth (previous

knowledge)– How we move our eyes throughout a scene

The Retina• Contains two types of photoreceptors

– Rods that offer wide field of view (and night vision)– Cones that provide high acuity (and color vision)

The Craik-O’Brien Illusion

Lateral Inhibition

Affect of Previous Knowledge

Rotating Mask

Affect of Previous Knowledge

The Fovea

• At its center: contains only cones (no rods)

• Perceive greatest detail and color vision– To get the most detailed representation of a

scene, must move your eyes rapidly so that different areas of the scene fall on your fovea

• Along visual axis - lowest potential for aberrations

(fovea covers <0.1% of the field)

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Serial Execution

Eye Movements…• Saccades

• Smooth Pursuit

• Optokinesis (OKN)

• Vestibular-Ocular Reflex (VOR)

• Fixations

… and lack thereof

Fixations• Stabilizations of the eye for higher acuity

at a given point

• Drifts and tremors of the eye occur during fixations such that the view is always changing slightly

Saccades• Rapid ballistic movement of eye from one

position to another

• Shift point of gaze such that a new region falls on the fovea

Eye Movements

X X X

Smooth Pursuit• Smooth eye movement to track a moving

target

• Involuntary - can’t be produced without a moving object

Eye Movements

X X

Optokinesis• Invoked to stabilize an image on the retina

• Eye rotates with large object or with its field-of-view

Eye Movements

Vestibular-Ocular Reflex• Invoked to stabilize an image on the retina

• Stabilizes an image as the head or body moves relative to the image

X

Eye Trackers

• Invasive– Painful devices which discomfort subject’s eye

• Restrictive– Devices that require strict stabilization of

subject’s head, not allowing for natural movement

• Modern Video-Based Trackers– Remote - constrained to 2D stimuli– Head-mounted - allows natural movement

Intrusive Eye Trackers

• Delabarre 1898

Mechanical stalk

• Yarbus 1965

Intrusive Eye Trackers

• Robinson 1963, Search Coils

Video-based Eye Trackers

• Early 1970’s, Limbus

RESTRICTIVE

Video-based Eye Trackers

• Cornsweet and Crane 1973, Dual Purkinje

RESTRICTIVE

Video-based Eye Trackers

• Dark-Pupil • Bright Pupil

Early 1970’s

Video-based Eye Trackers

• Head-Mounted • Remote

R.I.T. Wearable Eye Tracker

Video-based Eye Trackers

SCENE CAMERA

IR LED

EYE CAMERA

How it works

• Off-axis illumination

R.I.T. Wearable Eye Tracker

• Off-line processing

Example Video

My Research

• Objective: Improve the performance of video-based eye trackers in the processing stage.– Compensate for camera movement with

respect to the subject’s head– Reduce noise

LOWER PRECISION

R.I.T. Wearable Eye Tracker• Advantage:

– Subject is less constrained, can perform more natural tasks

• Disadvantage:– Camera (eye tracker)

not stabilized - need to account for any movement of camera relative to head

Lower Precision• Need to account for movement of camera

with respect to the head requires additional data: corneal reflection

• Corneal Reflection data is not as precise as Pupil data.

Analysis of Disadvantages

Too bad we can’t just use the Pupil data

Oversimplifying Assumption• Assumption: When the camera moves

with respect to the head, the pupil and corneal reflection move the same amount.

• To account for camera movement:

Analysis of Disadvantages

€

P −CR

Why this assumption is wrong• Corneal Reflection data comes from the

center of the reflection off the curved outer surface of the eye

• Pupil data comes from the center of the flat virtual image of the pupil inside the eye.

DON’T MOVE THE SAME AMOUNT

WHEN THE CAMERA MOVES

Result of Oversimplification

• P-CR vector difference changes with camera movement– Artifacts in final data

X X X

The Solution

• Determine the actual relationship between the pupil and corneal reflection during BOTH:– Camera movements– Eye movements

• Use these relationships to develop a new equation in terms of pupil and corneal reflection position

Eye Movements

Camera Movements

Camera and Eye Gains• Eye Gain: amount corneal reflection moves when

pupil moves 1 degree during an eye movement

• Camera Gain: amount corneal reflection moves when pupil moves 1 degree during a camera movement€

eye_ gain =ΔCR

ΔP

⎛

⎝ ⎜

⎞

⎠ ⎟eye

≅ 0.5146

€

cam_ gain =ΔC

ΔP

⎛

⎝ ⎜

⎞

⎠ ⎟camera

≅ 0.8524

€

Pcam = Ptrack − Peye

The Equations

€

Pcam =Ptrack ⋅E −CRtrack

E −C

€

Pcam ⋅E = Ptrack ⋅E −CReye

€

Pcam ⋅E −CRcam = Ptrack ⋅E −CReye −CRcam

€

Pcam (E −C) = Ptrack ⋅E −CRtrack

4 Initial Equations

€

Peye, Pcam,CReye,CRcam4 Unknowns:€

Ptrack = Pcam + Peye(1)

€

CRtrack =CRcam +CReye(2)

€

E =CReyePeye

(3)

€

C =CRcamPcam

(4)

Added Benefit

• Can smooth Camera array without loss of information from Pupil array:

• Assuming camera moves more slowly than eye moves.

• Result is on same level as Pupil only data€

Peye = Ptrack − Pcam

Determining the Gains• Eye Gain: (Instruct subject to…)

– Look at center of field-of-view.– Keep camera and head perfectly still.– Look through calibration points.

• Cam Gain: (Instruct subject to…)– Look at center of field-of-view.– Keep eye fixated while moving the camera on

nose.

Eye Gain Results

Eye Gain Results

y = 0.5161x + 0.3322

R2 = 0.9878

Camera Gain Results

Camera Gain Results

y = 0.8143x + 4.5981

R2 = 0.9768

Camera Gain Results

y = 0.8143x + 4.5981

R2 = 0.9768

Camera Gain Results

y = 0.8143x + 4.5981

R2 = 0.9768

slope = average gain = 0.8524 of 5 subjects

Testing the Algorithm• Collect data:

– 5 subjects look through 9 calibration points while moving the eye tracker’s headgear

• Extract eye movements:– Use average gains to calculate Camera array– Smoothed Camera array– Subtracted smoothed Camera array from

Pupil array Eye array

Horizontal Results

Results

X X X

X X X

X X X

Horizontal Results

Results Continued

X X X

X X X

X X X

Horizontal Results

Results Continued

X X X

X X X

X X X

Vertical Results

Results Continued

X X X

X X X

X X X

Vertical Results

Results Continued

X X X

X X X

X X X

Vertical Results

Results Continued

X X X

X X X

X X X

Vertical Results

Results Continued

X X X

X X X

X X X

Vertical Results

Results Continued

X X X

X X X

X X X

Noise ReductionResults Continued

Noise ReductionResults Continued

Conclusions

• Successful application to head-mounted video-based eye trackers– Use same gain values for all subjects

• Final Eye array precision is on the order of the Pupil array precision– Noise due to Corneal Reflection data is

reduced

Next Steps• Calibration - Eye array represents eye movement in head - need to map this to the world (via scene camera)

Next Steps• Calibration - Eye array represents

eye movement in head - need to map

this to the world (via scene camera)• Investigate realistic camera movements and

alternative smoothing options for Camera array

Next Steps• Calibration - Eye array represents

eye movement in head - need to map

this to the world (via scene camera)• Investigate realistic camera movements and

alternative smoothing options for Camera array• Obtain gain values for larger group of subjects

Next Steps• Calibration - Eye array represents

eye movement in head - need to map

this to the world (via scene camera)• Investigate realistic camera movements and

alternative smoothing options for Camera array• Obtain gain values for larger group of subjects• Test on larger eye movements

Next Steps• Calibration - Eye array represents

eye movement in head - need to map

this to the world (via scene camera)• Investigate realistic camera movements and

alternative smoothing options for Camera array• Obtain gain values for larger group of subjects• Test on larger eye movements• Revision for remote trackers

Questions, Suggestions…