Eye Detection And Iris Center Tracking With Eyelashes Occlusion Correction 1MD SOHEL RANA, 2MD ABDUL AWAL and 3MD ZAHIDUL ISLAM

1,3Dept. of Information and Communication Engineering, Islamic University, Kushtia, Bangladesh 2Dept. of Computer Science, College of Engineering and Technology, IUBAT, Dhaka, Bangladesh

1ice_sohelrana@yahoo.com, 2bongabdo@gmail.com, 3zahidimage@gmail.com

Abstract: Eye detection and iris center tracking is one of the most challenging problems in computer vision area. In this paper, we propose a new method for eye detection and iris center tracking with removing eyelashes occlusion. To do so firstly, we detect the face region using Haar classifier. Eyes region is detected by vertical and horizontal edge information. For accurate measurement of iris center, the intensity of the detected eye sub image is scaled. Otsu thresholding method is used for extracting binarization of eye sub image. Then, exact iris contour has selected from two largest iris contour candidates. Finally, ellipse fitting method is applied in the selected iris contour. The center of the ellipse will be the center of iris. A center correction method also has been explained for correction of the inaccurate center detection due to thick patched lower intensity eyelashes with iris.

Keywords: Iris center, Edge maxima searching, Intensity scaling, Ellipse fitting, Eyelashes occlusion

1. INTRODUCTION It is a challenging issue for developers of new human-computer interfaces is to provide a more interactive and natural way of interaction with computer systems, avoiding excessive use of hand and finger movement. Eye is one of the important organs which can be used for many types of human computer interface such as gaze tracking, drowsiness measuring, fixation analysis and many other facial biometrics [1, 2, 3]. In the field of human computer interface, eye tracking takes a prime role. In this paper, we proposed a remote web cam based eye tracking method. The method is divided in two modules: rough eye region detection and iris center detection from each eye region. Kun Peng and Liming Chen proposed horizontal and vertical edge projection based method for rough eye region detection [4]. But this method is good for straight forward looking face. If face is slightly tilted then edge maxima cannot allocate exact eye region. Haar classifier based approach can be used for eye region detection [5]. But, for eye detection, classifier based approach is too sensitive to false eye detection and it needs lots of training data. In this paper, window based edge maxima searching method is used for localizing rough eye region which solves both face tilting and false eye region detection problem. Gouqing Xu, Yangheng Wang, Jituo Li and Xiaoxu Zhou proposed an eye model based system [6]. They shows iris center and corner detection under different condition including naked-eye indoor, glasses indoor, naked-eye outdoor. Template matching based approach is also used for eye and iris center tracking [7, 8]. In case of template matching based approach, various rotated, scaled and oriented template is used for matching. But, it is also computationally expensive and good contrast of image is required. Alberto De Santis, Daniela Iacoviello proposed four level segmentation based eye and pupil tracking method [9]. By this method iris can be tracked at different head pose. But, there is large error for detection of exact center of the iris. M. Hassaballah proposed iris detection technique which is based on window based entropy calculation in gray scale eye sub image [10]. G.C. Feng and P.C. Yuen proposed a variance projection function based eye detection method

[11]. This method is computationally expensive. K. Nguyen proposed five stage iris recognition technique based on low resolution and high resolution image features [12]. Tsuyoshi Kawaguchi and Mohmed Rizon proposed intensity and edge information based iris selection method [13]. Recently, infrared light source is used for iris tracking [14, 19]. L. Ma developed a segmentation method which approximates the pupil center coordinates and applies canny edge detection and Hough transformation only in the iris region determined by the center of the pupil [15]. But, Circular Hough based iris center detection method is affected much for eyelids and eyelashes [16]. In particular, iris segmentation becomes difficult in gray scale image when eyelashes patched with exact iris region. Jian-Gang Wang, Eric Sung and Ronda Venkateswarlu proposed iris segmentation and contour ellipse fitting based iris center detection method [17]. They used morphological operation to segment iris from eyelids. But, there is no guarantee that morphological operation will always segment iris from eyelids. In our paper, iris center correction method is proposed. In this paper, exact iris contour selection and ellipse fitting based approach is used for iris center detection. The paper is arranged as: section 2 describes the eye tracking method from detected face, section 3 describes the iris center tracking method and finally experimental results and conclusion are drawn in section 4 and 5. 2. ROUGH EYE REGION DETECTION We used openCV (Open Source Computer Vision) Haar classifier for detection of face ROI (Region Of Interest). 2.1. PREPROCESSING To reduce the noise, the face region has been smoothed. Then, Skin-color segmentation is done in YCrCb color space.

Figure 1: (A) Vertical edge maxima searching using window (B) Vertical

line (L1 and L2) drawing (C) Horizontal edge maxima searching using window (D) Two horizontal line (L3 and L4) drawing

Canny edge detection method is applied for edge detection from the skin color segmented face ROI. 2.2. EDGE MAXIMA SEARCHING Now, maximum number of edge pixel is searched using a window over the edge sub image of face. If the width and height of face ROI sub image are 푤 and ℎ respectively, then for vertical line drawing, the searching rectangle’s width 푆 and height 푆 are taken as shown below:

푆 =푤푝 , 푤ℎ푒푟푒푝 = 4~8

푆 = ℎ

But, from experiment, we have seen that 푝 = 6 is best suited. The window is updated at every 푤/12 position in horizontal direction (figure 1(A)). Edge maxima searching algorithm is divided into two parts which are described in the next section.

Vertical line drawing: Figure 1(A) shows edge maxima searching starts from left sides of face ROI. Count number of edge pixels over each window. Stop searching when left edge of window reaches 푤/2 distance from left edge of face ROI. Find the maximum edge pixel counted window. Draw a vertical line 퐿1 at 푤/12 distance from maximum edge pixel counted window’s left side. Another vertical line 퐿2 is drawn using same process (figure 1(B)). This time the window starts searching from 푤/2 and stops when it reaches푤.

Horizontal line drawing: Here, same algorithm is used as previous. But, searching window is updated in the vertical direction. In this case, the search window’s width and height are taken as shown below:

푆 = 푑푖푠푡푎푛푐푒푏푒푡푤푒푒푛퐿1푎푛푑퐿2

푆 =ℎ푝 ,푤ℎ푒푟푒푝 = 8~14

From experiment we have seen that 푝 = 12 is best suited. The window is updated every ℎ/24 position in vertical direction. Searching starts from ℎ/12 and ends when upper edge of searching window reaches ℎ/2 position (figure 1(C)). Draw a horizontal line 퐿3 at distance ℎ/24 from maximum edge pixel counted window’s upper edge. Now, next line 퐿4 is drawn parallel to 퐿3 at distance ℎ/4 from퐿3.

A vertical line is drawn at the middle point of 퐿1 and 퐿2 from line 퐿3 to퐿4 (figure 1(D)). Marks the intersection points of all those line and makes two rectangles. Then rectangle 퐸1 will be left eye region and rectangle 퐸2 will be right eye region (Figure 2). Figure 2 has shown the detected eye region using edge maxima searching. If

푊푖푑푡ℎ표푓퐸1 +푊푖푑푡ℎ표푓퐸2 < (퐹푎푐푒푅푂퐼),

then rough eye region will not be correctly positioned. In this case, the located eye region of previous frame is considered in present frame. Actually, lighting variation of different tilted face is the cause of wrong vertical edge maxima founding. Hence, eye region is not positioned correctly.

3. IRIS CENTER TRACKING After extracting left and right eye region, each gray scale eye sub image is used for iris center detection. 3.1. INTENSITY SCALING 퐼(푥, 푦) is an intensity of (푥, 푦) coordinate in eye sub image. Then, after intensity scaling,

퐼 (푥, 푦) = 푘 ∗ 퐼(푥, 푦) (1) In equation (1) 푘 is scaling factor and 퐼 (푥, 푦) is the intensity of (푥, 푦) coordinate in intensity scaled eye sub image. The value of 푘 is taken as4~8. Experimentally we show that 푘 = 6 is best suited. After intensity scaling, binarized it using Otsu thresholding method. 3.2. IRIS CONTOUR SELECTION Canny edge detection method is used for edge detection. After edge detection, find contours and bound each contour by a rectangle. If any bounded rectangle of contour follows the rules: 푐표푛푡표푢푟 . ℎ푒푖푔ℎ푡 ≥ ∗ 푐표푛푡표푢푟 .푤푖푑푡ℎ, then that contour is rejected from iris contour. Here, 푐표푛푡표푢푟 . ℎ푒푖푔ℎ푡 is the height of bounded contour rectangle and 푐표푛푡표푢푟 .푤푖푑푡ℎ is the width of bounded contour rectangle. This condition is heuristically chosen. Largest two rectangles are selected as iris contour candidate. Because of, contour of eyebrows or eyelashes can be iris contour candidate. After this, the exact iris contour is chosen from the two selected contours. Average intensity of the iris region will be lower than the region eyebrows or outer corner side eyelashes region. For calculation average, the inner region of contours at intensity scaled sub image is used. Sometimes corneal reflection may increase the average of the actual iris region. For average calculation following equation is used,

퐴푣푒푟푎푔푒 = ∑ ( , ) , 푤ℎ푒푟푒퐼(푥, 푦) < 250 (2)

Figure 2: Detected eye region

In equation (2) 퐼(푥, 푦) is the intensity of (푥, 푦) coordinate and 푁 is the total number of considered pixel in iris contour. In figure 3(E), there are three contours bounded by three rectangles. Bounded rectangle contour B3 has been rejected. It shows that the height of B3 is larger than width. Bounded contour B1 and B2 are selected iris contour candidate. The average intensity B1 contour is lower than B2 contour of intensity scaled sub image. So, B1 is selected as the iris contour. After selection of exact iris contour, ellipse fitting algorithm is applied to find the iris center. 3.3. ELLIPSE FITTING Finally, ellipse is fitted to this selected contour. Least square ellipse fitting based method is applied for fitting [18]. An ellipse is a one type of conic which can be specified by second order polynomial,

푓(푥, 푦) = 푎푥 + 푐푥 + 푏푥푦 + 푑푥 + 푒푦 + 푓 = 0 (3)

Where 푎, 푏, 푐, 푑, 푒, 푓 are coefficients of the ellipse and (푥, 푦) are the points lying on it. From the output of ellipse fitting, the center of the ellipse will be the center of iris.

3.4. CORRECTION OF DETECTED CENTER After center detection, inaccurate center could be detected due to eyelashes patching with iris. Outer corner side’s eyelashes are thick. So, when iris resides close to the outer corner, the thick lower intensity eyelashes region patched with iris region. This case, detected center by ellipse fitting is inaccurate (figure 4(B)).

Figure 3: Iris contour selection A) Eye sub image B) Result of intensity scaling C) Binarization D) Contours of binarized image E) Bounding

rectangle of the contours

Figure 4: A) Correct detected center after ellipse fitting B) Correction required due to ellipse fitting in patched eyelashes contour C) Contour points

which are used for ellipse refitting

After center detection by ellipse fitting, every time our system will check that is it needed for center correction? Figure 5(a) shows a bounded rectangle of binarized iris image. Here, 퐶 is the detected center by ellipse fitting. 퐴퐵퐶퐷 is the bounded rectangle of the selected contour which drawn in binarized eye sub image. It shows two square regions 푅1 (Green square in figure 5) and 푅2 (Yellow square in figure 5) where lengths of both regions퐼푟 . Here, 퐼푟 is height of bounded rectangle. If upper left coordinate of square 푅2 is (푋1, 푌1):

푋1 = 푥 푣푎푙푢푒표푓푐표표푟푑푖푛푎푡푒퐶 − 퐼푟 /4 (4)

푌1 = 푦 푣푎푙푢푒표푓푐표표푟푖푑푛푎푡푒퐴 (5)

The upper right corner coordinate of square 푅1 is will be(푋2, 푌2):

푋2 = 푥 푣푎푙푢푒표푓푐표표푟푑푖푛푎푡푒퐶 (6)

푌2 = 푦 푣푎푙푢푒표푓푐표표푟푑푖푛푎푡푒퐴 (7)

If total number of iris pixel in 푅1 > total number of iris pixel in 푅2. Then center correction is required, else center correction is not required. Figure 5(b) shows that the total number iris pixel in 푅1< total number of iris pixel in 푅2. So, here correction is not required. Here, we considered the left eye sub image for center correction. For right eye sub-image, the rectangle 푅1and푅2 is taken in opposite direction than left eye sub image. Now, if is center correction is required. Then, again ellipse is fitted into the iris contour. If 퐶푅(푥, 푦) is a contour point then the point 퐸퐹(푥, 푦) which is taken for ellipse refitting follows the following condition:

퐸퐹(푥, 푦) = 퐶푅(푥, 푦), 푖푓푥 푣푎푙푢푒표푓푐표표푟푑푖푛푎푡푒퐴 +퐼푟 > 퐶푅(푥, 푦) (8)

In case of left eye, Equation (8) is considered (figure 4(C)). For right eye opposite direction contour points are taken. Finally, the center of this ellipse is considered as corrected center.

Figure 5: a) Correction needed binarized region b) Correction not needed

binarized region

Figure 6: Results of eye detection and iris center tracking

4. EXPERIMENTAL RESULTS The algorithm is implemented using MS (Microsoft) Visual Studio C++ and OpenCV (Open source Computer Vision). We used very low cost 2 megapixel, 640*480 formatted webcam. The experiment is done on different focal distance with total 980 consequent video frames. Frame rate of our proposed method is about 16 fps. Manually, we intended ground truth and corresponding iris center point frame by frame. If ground truth is 푔(푥, 푦) and iris center is (푥, 푦) , then root mean square error of n frame is calculated by equation (9)

푅푀푆퐸 =∑ { ( , ) ( , )} (9)

In figure 6 detected eye region and iris center is shown with various gaze direction of three people. Here, some erroneous result is shown. In case of first person, both eye region detection and center detection results are good (figure 6(a, b, c, d)). Here, this person’s eyelashes are not thicker. For second person (figure 6(e, f, k, l)), iris center detection result with different gaze direction is shown. This person has thicker eyelashes and our proposed system has removed this occlusion. For third person (figure 6(g, h, i, j)), lighting variation is occurred. Hence, eye region detection and center detection results are not robust. Specially, in figure 6(j), here ellipse fitting is occurred into wrong contour and detected center is incorrect.

For eye region detection, if the face is properly allocated, then our detection result of eye rough region is 98% accurate. The results of the correct center estimation at less than a given Euclidian distance from actual center are presented in table 1. This table shows 64% centers are detected in accuracy below the distance of 0.5 pixels. 95% iris centers are detected with less than a distance of 1.5 pixels. More than 99% iris centers are detected with distance greater than 1.5 pixels. So, here good accuracy is acquired. About 200 frames come for center correction from 780 frames, because of outer corners patched eyelashes occlusion.

Table 1. Percentage of correct detection of iris center

Table 2. Mean square error comparison

Figure 7: Error with presence of eyelashes occlusion correction vs without

occlusion correction

More than 83% frames are corrected below 1.5 pixels. In figure 7 shows the comparison of distances from ground truth to the corrected center and without corrected center. The performance comparison between our proposed method and Jian-Gang Wang [17] method has been shown in table 2 by means of Root mean square error (RMSE). So, our correction method has increased the efficiency of iris center detection at presence of outer corners patched eyelashes with iris. Although, we experimented this method with small amount of video frames, but hope that the efficiency will not degrade with large amount of video frames. 5. CONCLUSION In this paper, a new algorithm for eye detection and iris center tracking is proposed with eyelashes occlusion correction. As experimental result shows a good precision is acquired by this method. One of the major drawbacks is that the corneal reflection at edge part may help to increase the error rate. The proposed system requires good lighting condition. This method can be used as an application of various human-computer interfaces. This method also can be used for other facial biometrics based application.

References [1] Zhu J., Yang J., “Subpixel eye gaze tracking,” Fifth

IEEE International Conference on Automatic Face and Gesture Recognition, 2002.

0

2

4

6

8

10

12

14

0 50 100 150 200 250

Mea

n Sq

uare

Err

or

Outer corner eyelashes occluded frames

Ellipse fitted center with outer corners eyelashes occlusion

Corrected center

Pixel 0-0.5 0.5-1.5 1.5-3 3-5 Iris center 64% 95 % 99 % 99 %

Algorithm J Wang Proposed method

RMSE 2.45 1.33

a b c d f

g h ia

j k l

e

[2] Li D., Babcock J., Parkhurst D. J., “A low-cost head-mounted eye-tracking solution,” Association for Computing Machinery, March 2006.

[3] Santis A.D., Iacoviello D., “Robust real time eye tracking for computer interface for disabled people,” Computer Methods and Programs in Biomedicine, 2009.

[4] Peng K., Chen L., Ruan S., Kukharev G., “A Robust Algorithm for Eye Detection on Gray Intensity Face without Spectacles,” JCS&T, 2005.

[5] Viola P. and Jones M. J., “Rapid object detection using a boosted cascade of simple features,” IEEE CVPR, 2001.

[6] Xu G., Wang Y., Li J., Zhou X., “Real time detection of eye corners and iris center from images acquired by usual camera,” International Journal of Intelligent Engineering and Systems, Vol.3, No.1, 2010.

[7] Du Y., Arslanturk E., “Video-based noncooperative iris segmentation,” IEEE Transactions on Systems, Man, and Cybernetics, 2010.

[8] Liu H., Liu Q., “Robust Real-time Eye Detection and Tracking for Rotated Facial Images under Complex Conditions,” Sixth International Conference on Natural Computation, 2010.

[9] Santis A.D., Iacoviello D., “Robust real time eye tracking for computer interface for disabled people,” Computer Methods and Programs in Biomedicine, 2009.

[10] Hassaballah M., Murakami K., Ido S., “An Automatic Eye Detection Method for Gray Intensity Facial Images,” International Journal of Computer Science Issues, Vol. 8, Issue 4, No. 2, July 2011.

[11] Feng G.C., Yuen P.C., “Variance projection function and its application to eye detection for human face recognition,” Pattern Recognition Letters, 1998.

[12] Nguyen K., Fookes C., Sridharan S., Denman S., “Feature-domain super-resolution for iris recognition,” 18th IEEE International Conference on Image Processing, 2011.

[13] Kawaguchi T., Rizon M., “Iris detection using intensity and edge information,” The Journal of Pattern Recognition Society, 2003.

[14] Mock K., Hoanca B., Weaver J., Milton M., “Poster: Real-time continuous iris recognition for authentication using an eye tracker,” The ACM Computing Classification System, October 16-18, 2012.

[15] Ma L., Tan T., Wang Y., Zhang D., “Efficient iris recognition by characterizing key local variations,” IEEE Transactions on Image Processing, vol. 13, No. 6, June 2004.

[16] Wojcikiewicz W., “Hough Transform Line detection in Robot soccer,” Coursework for Image processing, Heriot Watt University, 14th March 2008.

[17] Wang J., Sung E., Venkateswarlu R., “8-29 on eye gaze determination via iris contour,” IAPR Workshop on Machine Vision Applications, Nov. 28-30, 2000.

[18] Fitzgibbon, Pilu M., Fisher R. B., “Direct Least Square Fitting of Ellipses,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 476-480, 1999.

[19] Maskeliunas R., Raudonis V., “ROBOSOFA-Low Cost Multimodal I/O Fusion for Smart Furniture,”

International Arab Journal of Information Technology, Vol. 10, No. 4, July 2013.

Md. Sohel Rana has received his B.Sc degree from the Department of Information and Communication Engineering, Islamic University, kushtia, Bangladesh. Currently, He is an M.Sc student of the same department. His research fields of interest are machine vision, eye

tracking, objects feature extraction, statistical learning etc. Presently, he is working to develop eye tracking system.

Md. Abdul Awal has successfully possessed a degree in MS in Telecommunication from University of Information Technology & Science (UITS) and Bachelor in Computer Engineering from American International University of Bangladesh (AIUB). In 2007 he has

joined as a Faculty in Department of Computer Science under College of Engineering and Technology at IUBAT - International University of Business Agriculture and Technology, Dhaka, Bangladesh. Besides he has been working with IBM Bangladesh Private Limited.

Dr. Md. Zahidul Islam has received his B.Sc. and M.Sc. degrees from the Department of Applied Physics & Electronic Engineering, University of Rajshahi (RU), Bangladesh in 2000 and 2002 respectively. In 2003, he has joined as a Lecturer in the

Department of Information and Communication Engineering, Islamic University (IU), Kushtia, Bangladesh. He has done his Ph.D research on Visual Object Tracking System from the Department of Computer Engineering at Intelligent Image Media & Interface Lab, Chonnam National University (CNU), South Korea. In August 2011, Dr. Islam has been successfully awarded his PhD from the same department. Besides, he has done his research internship in 3D Vision Lab in Samsung Advanced Institute of Technology (SAIT), Suwon, South Korea. Dr. Islam has also other research interests like computer vision, 3D object, human and motion tracking and tracking articulated body, genetic algorithm etc. Currently he is an Associate Professor and head of the dept. of Information and Communication Engineering, Islamic University (IU), Kushtia, Bangladesh.