AN EYEAN EYE MOUSE SYSTEM USING MOUSE SYSTEM USING FPGAFPGA

Professor: Yih-Ran Sheu Student: Dinh Viet Thang(M992B206)

CONTENTCONTENT

I. AbstractI. AbstractI. AbstractI. Abstract

II. II. IntroductionIntroductionII. II. IntroductionIntroduction

III. System DesignIII. System DesignIII. System DesignIII. System Design

IV. IV. Experimental ResultExperimental ResultIV. IV. Experimental ResultExperimental Result

V. ConclusionsV. ConclusionsV. ConclusionsV. Conclusions

I. AbstractI. Abstract

- The objective of this paper present a set of techniques integrated into an eye-mouse system using a field programmable gate array (FPGA).

- Our system requires a camera (TRDB-D5M), FPGA-DE2 board and a personal computer. A field programmable gate array (FPGA) is used to control the CCD camera and execute the image processing operations and control eye-mouse of computer via RS232 protocol.

- The coordinate of eye-mouse in our system is determined by the system based on FPGA imaging analysis. No computer is necessary in image processing. Good efficiency can be seen using FPGA hardware approach to determine the eye-mouse coordinate per image frame.

- The color information of eye is used to estimate the direction of pupil movement and then uses the direction information to control the mouse of computer.

II. IntroductionII. Introduction

•Nowadays, eye tracking and eye mouse system more and more are widely. There are many eye tracking system using image processing by computer supported by many image processing tools such as Matlab, Visual C, Visual Basic…etc, but these systems reduce the speed of the computer and sometimes cause unexpected situations.

• Thus, processing image to detect eye movements using FPGA hardware bring an alternative solution. In this paper, we practically design and develop the improved eye-mouse device with TRD-D5M camera, FPGA hardware, and computer. An eye tracking systems is a system that can track the movements of the user's eyes.

•Eye mouse system is a very good approach for people who do not have the ability to perform the movements of common control on computer such as mouse drag and mouse click…etc.

•In this paper, we perform image processing to detect pupil movement in FPGA hardware then sends control information to the computer to control a cursor on the screen. There are several different ways to track the directions of eye movements. Each approach has its own advantages, disadvantages, and limitations. We use information about eye color to detect the movement of the eye.

•There are many color spaces such as RGB, HSI, YUV, YIQ ... We choose the YUV color space because it has many advantages and simple. 

•The image processing in FPGA is not as good as image processing in personal computer but this process is totally independent, thus increasing the processing speed of the system. The eye mouse system is used to access computers via eye movements.

III. System Design

•The eye-mouse system consists of two modules. The first module including second -stage algorithm is proposed to implement image processing and pupil movement detection on FPGA hardware.

•An efficient pupil detection based on the pupil color information is employed to find the pupil and corner of user’s eye. After this, some algorithms is used to detect the direction information of pupil/corner of eye then transmit to computer via RS232 communication protocol.

•The second module is software on computer including cursor control block, user interface block, and application. The computed direction information of pupil movements is used to control the mouse of computer.

•Through the graphic user interface, users can implement some application.

•The component of eye mouse system is introduced as follows.

a. Software architecture system

b. Figure Hardware architecture system

2.1 HARWARE SPECCIFICATIONS

•The system consists of a personal computer, FPGA hardware with TRD-D5M camera and RS232 cable, FPGA hardware with a video capture, zoom buttons and adjust contrast.

•A real-time eye-tracking system requires an image capturing device operating at an extremely high frame rate to be able to effectively track the pupil in the presence of saccadic motion.

•Therefore, a high-frame rate image capture device is necessary in order to build a general eye tracker system for various human-computer interaction applications [1].

•The TRDB_D5M Kit satisfies most of the above requirements to develop a 5 Mega Pixel Digital Camera on the Altera DE2-70 / DE2 boards.

•2.2 THE ALGORITHM SYSTEM2.2 THE ALGORITHM SYSTEM

• The image input through the camera is transformed by FPGA hardware to obtain the image data of the plane coordinate.

• Every point (i,j) on the coordinate plane has its own image data. With this image information, the system is able to identify the pupil of eye from the image[5].

• Our methods based on color information about the pupil of eye  to determine the pixel according to the definition given.

• The first problem is to represent images in the appropriate color space that allows to distinguish two components: composition luminance, luminous intensity and color components.

• RGB color space generally inappropriate because it mixes the two components.

2.2.1 Image processing of an eye-mouse 2.2.1 Image processing of an eye-mouse systemsystem

•- Although there are other color models such as CMY, YIQ and HSI, in this paper YUV color model is chose because of its advantages. In YUV color model, the luminance information is represented with Y component while the color information is represented by the U and V components.

•- The main idea of using color-space information is to increase the information available about an image. The pupil feature detection system based on eigenfeature approach is used for gray-level Y and U, V color information of image. YUV color components can be obtained by using the RGB color coefficients as follows [6].

B

G

R

0.100 0.515 - 0.615

0.437 0.289 - 0.148 -

0.114 0.587 0.299

V

U

Y

•a. The definition of thresholdinga. The definition of thresholding

g(i,j) =

Tjif

Tjif

),(,0

),(,255

b. Calculation for the coordinate of the pupil center:

YX CC,

KK

YK

XK 11

1,

1

Where K is the total number of the dark spot at which the gray level is lower than the threshold after dynamic binarizing of the eye image. X and Y are coordinates of the black pixel. [5]

2.2.2 Eye movement

detection After we have located the pupil center point from the eye we can proceed to extract the direction information from the pupil movements.

The directions of the pupil movements are quantized into three directions: right, left, and center.

The third different position of pupil center point corresponding to the third region on the picture with size 320x240 pixel is  illustrated as follows.

Figure 6. The 3 different regions of pupil movement corresponding to the 3 directions of mouse.

•2.2.3 Communication between FPGA hardware and computer

• The direction information of pupil is transmitted to computer by RS232 or USB protocol communication.

• This paper we use RS232 protocol communication. Visual Basic software is used to receive the direction information of the eye movements from FPGA hardware.

• Although the baud rate of RS232 port is lower than USB, this speed is enough to satisfy our requirement.

•3. EXPERIMENTAL RESULTS3. EXPERIMENTAL RESULTS•3.1 Eye Tracking Experiment3.1 Eye Tracking Experiment- In this paper, we only perform a simple mouse eye system.

The obtained results are quite accurate because only three regions of the window is active. - For other complicated applications which require the pupil of the eye moving to more precise location of the window, only information detected from the color of the eye of people is not enough. - In addition, changes in light conditions also reduce the accuracy of the system. - Therefore, in these cases we need to combine information about color and other methods as detect edges, Hough Transform to upgrade the system. Some exact results are shown in the picture.

•3.2 Mouse Control and communication Aid3.2 Mouse Control and communication Aid

- We can use the eye tracking system combined with the communication aid, as shown in Figure.7. - The communication aid can be designed to contain 3 cells for each window to express our needs. A mouse click is required when a cell is active. - The mouse click can be started after the cursor has occupied the cell for more than a specified period (2 second). The center of the window is used to rest the cursor.

Figure 7. Communication Aid a) The Home application cell b) The Emergency cell

IV. ConclusionsIV. Conclusions- In this paper we propose an automatic and real time detecting pupil movement direction method and then implement a simple eye-mouse system. - The particular feature of this method is using the color information to detect the pupil movement. In this paper, we only use the color information so the results have some problem when the lighting conditions change. - Furthermore, the eye’s pupil is detected with a predefined camera position only, so this method has some limitations. However, the algorithm is efficient because it is only based on low level signals and thresholds for simple image processing. - So,  in a limited application, this algorithm can work well. Currently, we have simulated our process by Visual Basic and then convert to Verilog program for FPGA hardware. - In our future work, we plan to use more information of eye, edge information, and the Hough transform to get better results.

References[1]. Ashlt talukder, john-michael morookian, s. monacos, r. lam, c. lebaw, a. bond Intelligent Instruments and Systems Technology Group, In-Situ Instruments Section, Real-time Non-Invasive Eye tracking and Gaze-point Determination for Human-Computer Interaction and Biomedicine.[2]. Mu-chun su, kuo-chung wang, gwo-dong chen Department of Computer Science and Information Engineering,National Central University, Chung Li, Taiwan, An eye tracking system and its application in aids for people with severe disabilities.[3]. Hong LIU, Yuwen WU, Hongbin ZHA National Lab. On Machine Perception, Peking University, 100871, Beijing, China, Eye states detection from color facial image sequence[4]. Hasan Demirel, Department of Electrical & Electronic Engineering Eastern Mediterrnean University, Color spase analysis for facial feature detection. [5].Chern-Sheng Lin, Hsien- Tse Chen, Tien-Gern Lin, Mau- Shiun Yeh, Chuen- Lin Tien, Development and Application of An infrared eye-mouse control system