Driver status monitoring based on Neuromorphic visual processing

Dongwook Kim, Karam Hwang, Seungyoung Ahn, and Ilsong Han Cho Chun Shik Graduated School for Green Transportation

Korea Advanced Institute of Science and Technology 335 Gwhak-ro, Yuseong-gu, Daejeon 305-701

REPUBLIC OF KOREA dwkim88@kaist.ac.kr, i.s.han@kaist.ac.kr

Abstract: - In this paper, the driver status monitoring based on neurmorphic visual processing, which is robust at luminance and noise, is proposed and verified with experimentation. The infrared camera 640x480(VGA) resolution with 30 FPS (frames per second) are adopted for input image. The algorithm performed with openCV libraries both PC and Raspberry Pi. The processing sequence can be categorized as two part; one is feature extraction and the other is convolution for compute correlation with embedded each status features. After the calculation, the results overlapped in original input image to check the status of driver. The performance is 92% with 20 FPS calculation time for PC, and 88% with 7 FPS calculation time for Raspberry Pi. Key-Words: - Visual processing, Neuromorphic processing, Driver status monitoring, Vehicle safety

1 Introduction With increased popularity of personal vehicles, the

number of car accidents have increased as well. To prevent such accidents, there have been many researches that have been conducted toward safe driving which utilize sensors such as image sensor, radar, LIDAR (Line detection and ranging), ultrasonic sensor, and others [1-4]. The current emerging technology which is known as the advanced driver assistance system (ADAS), can secure the safety for the driver as well as the pedestrian. Research of obstacle or pedestrian detected in front of the vehicle are being steadily conducted both in academic institutions and industries [5, 6]. The potential dangers on the road can be the main cause to traffic accidents, but the driver’s condition can also be another cause to traffic accidents. Delphi, Denso and Toyota have conducted research in developing systems that senses the driver’s emotions and warns the driver if significant change in emotion is detected [7-11]. In this paper, a neuromorphic algorithm that is impervious to noise and light intensity is proposed, which is used to detect the driver’s status [12]. The proposed algorithm was verified by monitoring the driver’s status through two experiments; the first experiment with a pre-recorded video, and the second experiment with a real-time processing with infrared camera. Moreover, the algorithm was also implemented in Raspberry Pi with infrared camera connection.

2 Neuromorphic algorithm based visual processing

To detect the driver’s status, various methods can be used [7-11]. In this paper, the neuromorphic algorithm has been used as it is a robust algorithm that is nearly unaffected by the changes in the environment. With this algorithm, a high detection rate could be acquired despite the constant variation in outside environment. 2.1 Neuromorphic algorithm The neuromorphic algorithm is based on the neural network theory experiment conducted by Hubel and Wisel [13]. In this experiment, an electrode was inserted into the cerebral cortex of the cat, and its reaction strength was observed whenever a visual stimulus was given. Based on the results as shown in Fig. 1, a significant change in reaction was shown in the cerebral visual cortex when a directional visual stimulus was given. Based on the neuron theory of McCulloch-Pitts Model of neuron, it has been stated that there is a difference in stimulation when the stimulus of equal intensity is given with variations in weight [14]. As illustrated in the neural network as shown in Fig. 2, each simulated input are weighted based on a given arbitrary value. Whether the total stimuli is over or under than threshold T, the output value is determined.

Recent Advances on Computer Engineering

ISBN: 978-1-61804-336-8 150

Fig. 1. Visual stimuli of Hubel and Wiesel’s experiments; Response of the cat’s cortex when a rectangular slit of light of differnt orientations.

Fig. 2 McCulloch-Pitts neuron model. 2.2 Neuromorphic based feature extraction

Based on the experiments conducted by Hubel- Wisel and McCulloch-Pitts model of neuron, a filter that can detect the specific characteristics of the image based on its given weight and direction have been applied which allowed the algorithm to be robust and become impervious to noise and brightness characteristics [13]. Fig. 3 shows the input image and the processed image output based on the orientation filter. As shown in Fig. 3(b), only the edges of the image were acquired.

Fig. 3 Applying orientation filter in original image. (a) original image, (b) results of orientation feature extraction from oritinal image.

2.3 Driver status recognition After acquiring the characteristics of the image

through the orientation filter, the driver’s status was detected using the template matching method, where the system matches the processed output image with pre-defined images stored in the database. Each pre-defined image represents a specific status of the driver. As illustrated in Fig. 4, the similarity of the two images are detected using the image convolution method, where the specific status of the driver is determined if the similarity value between the two images is higher than a specific value.

Fig. 4. Neuromorphic algorithm based driver status monitoring. Feature extraction with orientation filter(left), correlation determination to recognize the status based on the pre-defined template(right).

3 Implementation and experimentation of neuromorphic algorithm

An experiment that processes the acquired images through PC and Raspberry Pi was conducted to verify the image processing procedure based on neuromorphic algorithm. The camera used for the experiment is an infrared camera which displays uniform results regardless of day or night time.

Fig. 5 (a) shows the experiment conducted in PC based on the given image output by the infrared camera during night time environment, and Fig. 5 (b) shows the experiment conducted in Raspberry Pi based on the given image output by the infrared camera.

Fig. 5. Experimental setup, (a) PC base experiment with infrared camera, (b) Raspberry Pi based experiment with infrared camera

Recent Advances on Computer Engineering

ISBN: 978-1-61804-336-8 151

3.1 Experimental setup In the experiment, the PC and Raspberry Pi was set via openCV library where both controllers were set to receive images from the infrared camera at 30 fps(frames per second), where each image was sized at 640x480. With the processed output image, the algorithm will determine the driver’s status based on the image’s characteristic similarity determination via template matching. To observe the accuracy of template matching, the test subject’s face template has been set in left, right, and front configurations as shown in Fig. 6(a), and observed how the algorithm will determine the driver’s state based on the configured template. Additional templates has been implemented into the algorithm as shown in Fig. 6(b), where each template shows the driver’s eye fully open, half open, and fully closed. With this detection, the system will be able to detect whether the driver is drowsing.

Fig. 6. Template data base for calculating correlation, (a) head direction template, (b) eye open template. 3.2 Results 3.2.1 Experimental results based on PC The left image in Fig. 7 shows the image characteristics through image processing, and the right image shows the processed image superimposed into the original image to determine the driver status. As shown in Fig. 7(a), the head movement direction could be determined as left, right, and front direction, and Fig. 7(b) shows how much the eye is closed. When processing the input image at the rate of 30 fps, it takes 50 mili-seconds to process a single image. The processed image was recorded, and the accuracy of the head movement and eye closing detection has been recorded in Table. 1. Because of the difference in hardware performance between the PC and Raspberry Pi, the calculating time and latancy is slightly different as well. However, both systems achieved similar level of accuracy.

Table . 1 Accuracy rate for driver status monitoring based on PC experiment

Fig. 7. Experimental results image, (a) Extract feature image from original image, (b) overlapped recognized status result to original image.

Drowsiness Inattentiveness

Number of frames 1172 1559 Negative recognize

frames 72 172

Positive recognize frames

1100 1387

Accuracy 93.9 % 88.9 %

Recent Advances on Computer Engineering

ISBN: 978-1-61804-336-8 152

3.2.2 Experimental results based on Raspberry Pi A similar experiment described in Section 3.2.1 was conducted using Raspberry Pi and it’s results were observed. Since the Raspberry Pi only requires 10-Watts, the driver’s status monitoring could be applied in various environments. The left-side image in Fig. 8 shows the output results of the infrared camera, and the right-side image shows the determined characteristics based on the applied filter of the algorithm. In addition, a function was implemented where it measures the time of how long the driver’s status was kept at its same position, and if necessary, an alarm was triggered if the same position was utilized for a significant time. The two images were processed in real time during night time, and the processing speed was at 6.5~7 fps, which takes 130~150ms to process one image. Similar to the procedure done for the PC, the processed image was recorded and the accuracy was recorded in Table 2. Therefore, in case of situations where the eye is closed for more than 3 seconds or when the face direction is not maintained in the front direction for more than 5 seconds, the warning alarm will occur.

Fig. 8 Experimental results using Raspberry Pi with infrared camera. (a) Head direction recognition image (left), feature extraction from original image (right), (b) drowsiness status recognition image (left), feature extraction image (right).

Table. 2 Accuracy rate for driver status monitoring based on Raspberry Pi

Drowsiness Inattentiveness

Number of frames 775 1529 Negative recognize

frames 66 182

Positive recognize frames

709 1347

Accuracy 91.5% 88.1 %

4 Conclusion The neuromorphic algorithm based visual processing for driver status monitoring is proposed and verified through experimentation. The infrared camera is used for more robustness in luminance factor with 30 FPS, 640x480 resolution. Based on the results, the monitoring system was able to detect drowsiness and inattentive with over 90% accuracy and 88 % accuracy, respectively. The performance of the PC and the Raspberry Pi was set at 20 fps and 7 fps, respectively. Through the neuromorphic algorithm based visual processing, a driver status detection and warning system was designed. The drowsiness and inattentiveness of the driver was recognized, and a warning alarm was triggered in case the driver maintained the same state for a specific duration of the time. Through this system, it can be expected that car accidents due to driver error can be significantly reduced. Acknowledgement: The research was promoted as part of ‘Technology Development for Behavior Improvement and Violation Control of High Risk Drivers’ that was carried out by Ministry of Land, Infrastructure and Transport in Korea. References: [1] K. Takagi, K. Morikawa, and M. Saburi “Road

environment recognition using on-vehicle LIDAR,” Intelligent Vehicles Symposium, IEEE, pp. 120-125, 2006.

[2] G. Tabuel, R. Sharma, J. S. Yang, “An Experimental Study of a Lane Departure Warning System Based on Optical Flow and Hough Transform Methods,” WSEAS Transaction on Systems, 13.1, pp. 105-115, 2014.

[3] J. M. R. Behana, C.A. Cruz, A.Z. Lopez, and A. Ferreyra-Ramirez, “Speed booms detection for a ground vehicle with computer vision,” Proc. of the 12th WSEAS, pp. 258-264, 2010.

[4] M. Bertozzi, L. Bombini, P. Cerri, P. Medici, P.C. Antonello, and M. Miglietta, “Obstacle

Recent Advances on Computer Engineering

ISBN: 978-1-61804-336-8 153

detection and classification fusing radar and vision,” Inteligent Vehicles Symposium, IEEE. pp. 608-613, 2008.

[5] T. Ogawa, H. Sakai, Y. Suzuki, K. Takagi, K. Morikawa, “Pedestrian detection and tracking using in-vehicle lidar for automotive application,” Intelligent Vehicles Symposium, IEEE. pp. 734-739, 2011.

[6] Q. Ji and X. YangX1. “Real-Time Eye, Gaze, and Face Pose Tracking for Monitoring Driver Vigilance”, Real-Time Imaging 8(5), pp.357-377, 2002.

[7] J. Fukuda, K. Adachi, M. Nishida and E. Akutsu, “Development of driver’s drowsiness detection technology,” Toyta Technical Review, vol 45, pp. 34-40, 1995.

[8] T. Nakagawa, T. Kawachi, S. Arimitsu, M. Kanno, K. Sasaki, and H. Hosaka, “Drowsiness detection using spectrum analysis of eye movement and effective stimuli to keep driver awake,” DENSO Technical Review, vol. 12, pp. 113-118, 2006.

[9] E. Rogado, J. Garcia, R. Barea, L. Bergasa and E. Lopez, “Driver Fatigue Detection System,” IEEE Int. Conf. Robotics and Biomimetics, 2009.

[10] Q. Ji, Z. Zhu, P. Lan, “Real-Time Nonintrusive Monitoring and Prediction of Driver Fatiue,” IEEE Transactions on Vehicular Technology, Vol. 53, 2004.

[11] L. Bergasa, J. Neuvo, M. Sotelo, R. Barea, M. Lopez, “Real-time system for monitoring driver vigilance,” IEEE Trans. Intel. Transport Syst., 2006.

[12] W. J. Han, and I. S. Han, “Bio-inspired visual signal processing based on neuromorphic circuit,” Proc. WSEAS, pp. 131-136, 2010.

[13] D. H. Hubel, and T. N. WIESEL, “Receptive fields of single neurons in the cat’s striate cortex,” The journal of physiology 148.3, pp. 574-591, 1949.

[14] S. Haykin, N. Network, “A comprehensive foundation,” Neural Networks 2, 2004.

Recent Advances on Computer Engineering

ISBN: 978-1-61804-336-8 154