toward accurate and robust cross-ratio based gaze trackers through learning from simulation (etra...
DESCRIPTION
Jia-Bin Huang, Qin Cai, Zicheng Liu, Narendra Ahuja, and Zhengyou Zhang Towards Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation Proceedings of ACM Symposium on Eye Tracking Research & Applications (ETRA), 2014 ETRA 2014 Best Paper AwardTRANSCRIPT
![Page 1: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/1.jpg)
Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through
Learning from SimulationJia-Bin Huang1, Qin Cai2, Zicheng Liu2,
Narendra Ahuja1, and Zhengyou Zhang2
21
![Page 2: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/2.jpg)
Why?
• Multimodal natural interaction• Gaze + touch, gesture, speech
If I were an iron man…
![Page 3: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/3.jpg)
Why?• Understanding user attention and intention
![Page 4: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/4.jpg)
Why?• Understanding interaction among people
Before sunrise1995
![Page 5: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/5.jpg)
ScleraLimbus
PupilIris
Glint
Cornea (like a spherical mirror)
Mike @ Monster University
![Page 6: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/6.jpg)
Geometric Model of an Eye
![Page 7: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/7.jpg)
Gaze Estimation using Pupil Center and Corneal Reflections
Interpolation-based
Cross-Ratio based
Model-based
![Page 8: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/8.jpg)
Model-based Gaze Estimation
• Detailed geometric modeling between light sources, corneal, and camera [Guestrin and Eizenman, 2006]
• Pros• Accurate (reported performance < 1o)• 3D gaze direction• Head pose invariant
• Cons• Need careful hardware calibration
Figure from [Guestrin and Eizenman, 2006]
![Page 9: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/9.jpg)
Interpolation-based Gaze Estimation
• Learn polynomial regression from subject-dependent calibration• Directly map from normalized to Point of Regard (2D PoR)
[Cerrolaza et al., 2008]
• Pros• Simple to implement• No need for hardware calibration
• Cons• Head pose sensitive
![Page 10: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/10.jpg)
Cross-Ratio based Gaze Estimation
• Gaze estimation by exploiting invariance of a plane projectivity [Yoo et al. 2002]
• Pros• Simple to implement• No need for hardware calibration• Head pose invariant
• Cons• Large subject dependent bias occur
because simplifying assumptions Figure from [Coutinho and Morimoto 2012]
![Page 11: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/11.jpg)
The Basic Form of Cross-Ratio Method
Image
Corneal
Display
![Page 12: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/12.jpg)
Two Sources of Errors [Kang et al. 2008]
• Angular deviation of visual axis and optical axis
• Virtual image of pupil center is not coplanar with corneal reflections
![Page 13: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/13.jpg)
Improve Accuracy for Stationary Head
CR [Yoo-2002]
CR-Multi [Yoo-2005]
CR-HOM [Kang-2007]
CR-HOMN [Hansen-2010]
CR-DV [Coutinho-2006]
No correction
Scale correction
Scale and translation correction
Homography correction
Homography correction + Residual interpolation
![Page 14: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/14.jpg)
Improve Robustness for Head MovementsNo adaptation Adapt to eye
depth variationsAdapt to eye movementsAssumptions 1) weak perspective2) fixed eye parameters.
CR [Yoo-2002] CR-DD [Coutinho and Morimoto 2010]
PL-CR [Coutinho and Morimoto 2012]
![Page 15: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/15.jpg)
Accuracy of Gaze Prediction for Stationary Head
Robustness to Head Movement
No adaptation
CR [Yoo-2002]
CR-Multi [Yoo-2005]
CR-DV [Coutinho-2006]
CR-HOM [Kang-2007]
CR-HOMN [Hansen-2010]
No correction
Scale correction
Scale and translation
correction
Homography correction
Homography correction + Residual interpolation
CR-DD [Coutinho-2010]
Adapt to eye depth variations only
PL-CR [Coutinho-2012]
Adapt to eye movementsAssumptions 1) weak perspective2) fixed eye parameters.
Adapt to eye movementsNo assumptions on 1) weak perspective 2) fixed eye parameters
This paper
![Page 16: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/16.jpg)
How? The Main Idea
• Build upon the homography normalization method [Hansen et al 2010]
• Improving accuracy and robustness simultaneously by introducing theAdaptive Homography Mapping
![Page 17: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/17.jpg)
Adaptive Homograph Mapping
• Two types of predictor variables
• : capture the head movements relative to the calibration position• Affine transformation between the glints quadrilateral
• : capture gaze direction for spatially-varying mapping• Pupil center position in the normalized space
• : polynomial regression of degree two with parameter
![Page 18: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/18.jpg)
Training Adaptive Homography Mapping• Exploit large amount of simulated data• the set of sampled head position in 3D• the set of calibration target index in the screen space
• Objective function
![Page 19: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/19.jpg)
Minimizing the Objective Function
• Minimize an algebraic error at each sampled head position
• Use the solution from algebraic error minimization as initialization Minimize the re-projection errors using the Levenberg-Marquardt algorithm
![Page 20: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/20.jpg)
Visualize the Training Process
• Eye gaze prediction results using the bias-correcting homography computed at the calibration position
![Page 21: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/21.jpg)
RMSE Error Comparisons Using Different Training Models• Differences are small in linear
regression• Linear model is not
sufficiently complex
• Compensation using both predictor variables achieve the lowest errors
![Page 22: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/22.jpg)
Linear Regression
![Page 23: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/23.jpg)
Linear Regression
Adding the normalized pupil centercorrected spatially-varying errors
![Page 24: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/24.jpg)
Quadratic Regression
![Page 25: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/25.jpg)
Quadratic Regression
![Page 26: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/26.jpg)
Experimental Results – Synthetic data • Setup• Screen size 400mm x 300mm• Four IR lights• Camera 13mm focal length, placed slighted below the screen border
(FoV~31 degree)
• Calibration position and eye parameters• Eye parameters from [Guestrin and Eizenman, 2006]
![Page 27: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/27.jpg)
Stationary Head Varying corneal radius
![Page 28: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/28.jpg)
Stationary HeadVarying pupil-corneal distance
![Page 29: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/29.jpg)
Stationary HeadVarying (horizontal) angle between optical/visual axis
![Page 30: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/30.jpg)
Stationary HeadVarying (vertical) angle between optical/visual axis
![Page 31: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/31.jpg)
Head Movements Parallel to the Screen
![Page 32: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/32.jpg)
Head Movement along Depth Variation
![Page 33: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/33.jpg)
Tested at Another Head Position
![Page 34: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/34.jpg)
Noise Sensitivity Analysis
![Page 35: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/35.jpg)
Effect of Sensor Resolution (at calibration)
Focal Length = 13 mm Focal Length = 35 mm
![Page 36: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/36.jpg)
Effect of Sensor Resolution (at new position)
Focal Length = 13 mm Focal Length = 35 mm
![Page 37: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/37.jpg)
Real Data Evaluation – Programmable Hardware Setup
Off-axis IR light sources
Stereo camera (We use one only in this work)
On-axis ring light
![Page 38: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/38.jpg)
Real Data Evaluation – Feature Detection
• Detecting glints and pupil center
![Page 39: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/39.jpg)
Averaged Gaze Estimation Error
at calibration position
![Page 40: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/40.jpg)
Averaged Gaze Estimation Error
Calibrated at 600mm from screenCalibrated at 500mm from screen
![Page 41: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/41.jpg)
Conclusions
• A learning-based approach for simultaneously compensating (1) spatially varying errors and (2) errors induced from head movements
• Generalize previous work on compensating head movements using glint geometric transformation [Cerroaza et al. 2012] [Coutinho and Morimoto 2012]
• Leveraging simulated data avoid the tedious data collection
![Page 42: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/42.jpg)
Future Work
• Consider subject-dependent parameters in the learning and inference the adaptive homography adaptation
• Integrate binocular information, please see poster
Zhengyou Zhang, Qin Cai, Improving Cross-Ratio-Based Eye Tracking Techniques by Leveraging the Binocular Fixation Constraint
• Extensive user study using a physical setup
![Page 43: Toward Accurate and Robust Cross-Ratio based Gaze Trackers Through Learning From Simulation (ETRA 2014)](https://reader038.vdocuments.site/reader038/viewer/2022102922/5462bac7b4af9f531c8b490b/html5/thumbnails/43.jpg)
Comments or questions?
Jia-Bin Huang [email protected]
Narendra [email protected]
Zhengyou Zhang [email protected]
Zicheng [email protected]