neural network based detection of drowsiness with eyes
TRANSCRIPT
Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=titr20
Download by: [UCL Library Services] Date: 05 October 2016, At: 07:08
IETE Technical Review
ISSN: 0256-4602 (Print) 0974-5971 (Online) Journal homepage: http://www.tandfonline.com/loi/titr20
Neural Network Based Detection of Drowsinesswith Eyes Open using AR Modelling
Hyungseob Han & Uipil Chong
To cite this article: Hyungseob Han & Uipil Chong (2016): Neural Network BasedDetection of Drowsiness with Eyes Open using AR Modelling, IETE Technical Review, DOI:10.1080/02564602.2015.1118362
To link to this article: http://dx.doi.org/10.1080/02564602.2015.1118362
Published online: 29 Feb 2016.
Submit your article to this journal
Article views: 10
View related articles
View Crossmark data
Neural Network Based Detection of Drowsiness with Eyes Open using ARModelling
Hyungseob Han and Uipil Chong
Department of Electrical and Computer Engineering, University of Ulsan, Ulsan, South Korea
ABSTRACTThis paper proposes a method of neural network based drowsiness detection with eyes open usingpower spectrum analysis and auto-regressive modelling. After the electroencephalogrammeasurements are complete, alertness, transient, and drowsy periods are classified according toalpha spectrum changes and alpha-blocking phenomena. Although the subject’s eyes are open,alpha spectrum changes such as drowsiness patterns are detected. Consequently, drowsinessdetection with eyes open is applied into the proposed system. The neural network based proposedmethod shows that LPC (linear predictive coding) coefficients are the proper feature vectors andaverage classification rate is about 92%.
KEYWORDSDrowsiness detection; EEG;Feature extraction; LPCcoefficients; Neural network;Multi-layer perceptron
1. INTRODUCTION
Drowsiness state is the transition period between alert-ness and the onset of drowsiness is associated with Hori’ssleep stage 1 [1]. A number of methods for driver’sdrowsiness detection system have been studied, and theyare mainly classified into three approaches. The firstapproach is to observe the driver’s behaviours related todrowsiness, such as the inclination of the driver’s head,sagging posture, decline in gripping force on the steeringwheel, and lane departures using a camera to track roadmarkings [2]. The second approach is to measure thephysiological signals of drivers and to analyse them, suchas electroencephalogram (EEG), electrocardiogram(ECG), electromyogram (EMG), electrooculogram(EOG), and skin electric potential [3�5]. The thirdapproach is to analyse facial image changes using imageprocessing techniques with regard to eyelid movement,such as eye-blinking frequency, average eye-closurespeed, and percentage of eye closure [6,7]. Among theseapproaches, there is a general agreement that analysis ofEEG waveforms and eye blink patterns can reliablydetect driver fatigue or drowsiness [8].
Since image processing techniques are non-invasive andnon-contact, additional wearing devices are not neededand system installation is not difficult. Measuring theeye-blinking frequency and eye closure duration ismostly used to determine the degree of drowsy drivingby such criteria: eye blinks of 0.3�0.4 seconds durations[8] and inter-eye blink intervals of 6�8 seconds [9].Although drowsiness detection models using image
processing techniques have the advantage of applying tovehicles practically, accuracy and precision are lowerthan analysing a physiological signal. Especially, sincethe image processing techniques are restricted to detecteyes’ closed state, they cannot be applied for detectingdrowsiness with the eyes open. Also, the analysis of theeye-blinking frequency and eye closure duration basedon image processing techniques cannot detect thedrowsy state with eyes open. These limitations of vision-based systems can make dangerous situations in the realdriving environment. In this paper, EEG-based analysisby changes of alpha power spectrum and alpha-blockingphenomena is applied to detect drowsiness when subjectsget drowsy in an eyes-open state [6].
A combination of various techniques in frequencydomain and statistical methods has been applied toextract feature vectors. Especially, Fourier-based fre-quency domain techniques have been applied for physio-logical signal analysis with artificial neural networks[6,10,11]. However, the frequency domain techniquesusing Fourier-based approaches have performance limi-tations as follows. The first is that the Fourier-basedapproaches assume data out of window to be either zeroor repetitive. This assumption can generate spectralsmearing. The second is that spectrum energy is dis-torted by windowing. The third is not to show a fine res-olution for short data [12�14]. Since physiological signalhas characteristics of non-stationary and non-linear,abnormal states can occur instantaneously undertransient condition. Fourier-based frequency analysis is
© 2016 IETE
IETE TECHNICAL REVIEW, 2016http://dx.doi.org/10.1080/02564602.2015.1118362
not a proper approach so alternative analysis methodsshould be needed for physiological analysis. Featureextraction technique in this paper is based on AR (auto-regressive) time series modelling, which has a fine resolu-tion with small sample sizes and sampling rates com-pared to Fourier-based approaches. Especially, LPC(linear predictive coding), a kind of AR time seriesmodelling, is proposed and its coefficients are used asfeature vectors of EEG data for each state.
2. BACKGROUND
2.1 EEG
The electrical activity of the brain is commonly analysedto monitor brain activity. Neural functions rely onelectrical events within the plasma membrane of neu-rons. The brain with billions of neurons and their activitygenerate an electrical field that can be captured byplacing electrodes on the outer surface of the skull. Theelectrical activity changes constantly, as nuclei and corti-cal areas are stimulated or not. Data for reporting theelectrical activity of the brain is called an EEG and theseelectrical patterns observed are called brain waves.Table 1 shows a comparison table of EEG rhythmicactivity frequency bands [6].
3. PROPOSED METHOD
In this paper, the overall drowsy monitoring procedure isdescribed in Figure 1. For the monitoring, input EEGdata are obtained from the human EEG measuring sys-tem. Feature vectors are extracted from the obtaineddata by LPC analysis. Extracted feature vectors aredivided into training and test data-set. After the trainingdata are trained by the back propagation algorithm andfeed-forward network of the multi-layer perceptron(MLP) neural network, the detection and classificationmodel is completed. This proposed model performs theclassification for each state using the test data.
3.1 Extraction of feature vectors
The electrical activity of the brain signals are representedby harmonic sinusoids. In the spectrum of each case,
sharp peaks are found at harmonics. Since the AR modelis based on a structure of all-pole form, it is possible forspectrum estimation for peaks [12]. In this paper, LPC,which is one of AR time series modelling, is proposed asfeature extraction method for each state. The LPC differ-ence equation is represented as Equation (1). Gx(n)ands(n) are the value of the weighted present input and thepresent output, respectively. The value of p is LPC order.If aj is the estimate of aj, the estimation error is given byEquation (2). To minimise the mean squared error ofe(n), the partial derivatives with respect to aj are set tozero for j D 1, …, p. The p number of simultaneousequations is generated. In this paper, aj (j D 1, …, p)can be solved using Durbin’s recursive algorithm.aj computed is the LPC coefficient [13].
s nð ÞDGx nð ÞCXp
jD 1
ajs n� jð Þ (1)
e nð ÞD s nð ÞCXp
jD 1
ajs n� jð Þ (2)
This paper used the LPC coefficients as the feature vec-tors for three states. Since the optimal number of LPCorder about EEG data is determined to 4, the value of j isset to 4 [6]. Table 2 shows four LPC coefficients for EEGaccording to alertness, transient, and drowsy state.
3.2 Multi-layer perceptron
The MLP neural network, which is one of the supervisedlearning methods, has been applied into various fieldssuch as machine learning, recognition system, and signaloptimisation. Compared to other learning methods, ithas an advantage of simple implementation and a
Table 1: Comparison of EEG frequency band
Band Range (Hz) State Location
Delta <4 Adult slow-wave sleep in babies VariableTheta 4�7 Drowsiness in adults and teens Occipital, temporalAlpha 8�13 Relaxed/reflecting closing the eyes Occipital, parietalBeta 14�31 Active thinking, focus, hi-alert,
anxiousFrontal
Figure 1: Proposed driver drowsy monitoring system
Table 2: LPC coefficients of EEG signals
State Coeff.(aj) Alertness Transient Drowsiness
1 0.3324 ¡0.7790 ¡0.19992 0.0263 ¡0.3104 ¡0.55093 0.1121 0.2480 0.56404 ¡0.0842 0.0831 0.2362
2 H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS
relatively low memory requirement [15�17]. The MLPconsists of input layer, hidden layer, and output layerand each neuron is fully connected to each other withweights w. It is shown as Figure 2. The MLP trainingprocedure iteratively updates connection weightsthrough feed-forward process and back propagationalgorithm and minimises error between computed out-put vector z and target vector t. From these procedures,trained network model can be completed [18]. In thissystem, input vector x and output vector z correspond toLPC coefficients and driver’s states respectively.
One of the important roles of hidden neurons and hid-den layer is to separate significant features from inputfeatures. Therefore, setting the number of hidden neu-rons and the size of hidden layer becomes an importantpart to determine performance of the neural network.Also, because optimisation for the number of hiddenneurons varies according to types of problems to besolved, the proper number of hidden neurons that satis-fies requirements for each system should be determined.In this paper, the number of hidden layers is set to onebased on universal approximation theory [14]. In orderto optimise the proposed MLP structure, we experi-mented by decreasing the number of hidden neuronsfrom ten to one. From the experiment, the classificationperformance gradually decreases from under three.Therefore, the number of hidden neurons is set to three.Based on Kearns’s recommendation, the ratio of trainingdata-sets and the test data-sets is set 7:3 and 20% of thetraining data-set is allocated into validation subset [18].Parameters of MLP neural network are set in the sameway as [19] and are described in Table 3.
4. ENVIRONMENT AND SEGMENTATION
4.1 Experiment environment
The experiments were conducted in the anechoic cham-ber at the University of Ulsan as in Figure 3. A total of
10 subjects (ages from 24 to 60) were selected throughthe Epworth Sleepiness Scale (ESS)() survey. To reduceenvironmental noise in EEG, all electronic devices wereturned off except necessary equipment. At that time,indoor temperature was 22�C and indoor humiditywas 54%.
LAXTHA PolyG-I equipment of the 16-channel wasused, and using TeleScan software of LAXTHA, EEGwas recorded by using eight electrodes (Fp1, Fp2, F3, F4,P3, P4, O1, O2) placed according to the international10-20 system with the reference electrode on the rightearlobe as shown in Figure 4. The EEG signals were
Figure 2: MLP architecture
Table 3: Parameters of MLP neural network [19]
Parameters Configuration
Learning algorithm Scaled conjugate gradientsLearning rate 0.05Transfer function Tangent sigmoidTraining method Batch trainingPerformance function MSE (mean square errors)
Figure 3: Measurement of EEG
Figure 4: Electrodes placement in 10/20 system [6]
H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS 3
amplified and a sampling frequency is 256 Hz. To reducethe contact impedance between EEG electrodes and cor-tex, the surface of the electrodes were cleaned with alco-hol swab before recording and the electrodes wereplaced.
The experiments were performed at midnight in a noise-less environment. Subjects had resting time in order toadapt to the new situation by sitting in a prepared chairbefore experimentation. After adaptation, subjects gotinto the car and were required to perform the drivingsimulation concentrating on the screen. Each subjectsimulated a 40-minute work session two times. TeleScan(LAXTHA Inc.) was applied to the EEG data acquisitionand analysis. Figure 5 shows samples of raw EEG datarecord using TeleScan. A video camera, which wassynchronised with the data, was located in front of thesubject and was used to capture the video image of thesubject’s face. EEG of each subject was measured at 40-minute intervals two times.
4.2 Data segmentation
Figure 6 shows the process that classifies acquired datathat depend on states.
In this research, definition for alertness state and drowsi-ness state are set to the Hori’s standards. According toHori’s standard, when state changes from alertness todrowsiness (sleep stage 1), there is drop in eye-blinkingrate. Also there are changes for brain wave. In the occipi-tal area, alpha (8�13Hz) wave decreases, with thesealpha-block phenomena occur along with theta wave(4�7Hz) arises over the area [11]. Depending on candi-dates’ health status, classification was driven into alert-ness (A), transient (T), and drowsiness (D) states.Moreover, subjects’ brain wave and time drowsing weresynchronised with visual inspection.
Changing window size, the segmented data for A, T, andD states were generated into 100 data group AD fA 0½ �;. . . ;A 99½ �g; TD fT 0½ �; . . . ;T 99½ �g; DD fD 0½ �; . . .D 99½ �g for each state.
With this data group, when order of the model is called p,AR coefficient of states, a k½ �D a1 k½ �; . . . ap k½ �
� �;
t k½ �D t1 k½ �; . . . tp k½ �� �
; d k½ �D ðd1 k½ �; . . . dp k½ �Þ areextracted where k D 0, 1, …, 99. Total number of trainingset and test data-set are 300.
The vectors extracted formally are used as input vector ofneural network, learned in MLP neural network andclassified into one of the states above.
5. EXPERIMENT RESULTS AND DISCUSSION
5.1 EEG analysis with drowsiness state
According to Hori’s standard, when state changes fromalertness to drowsiness (sleep stage 1), there is drop ineye-blinking rate. Also there are changes for brain wave.In the occipital area, alpha (8�13Hz) wave decreases,with these alpha-block phenomena occur along withtheta wave (4�7Hz) arises over the area [11]. Figures 7and 8 show the change of the alpha and theta wave
Figure 5: Sample of raw EEG data record
Figure 6: Data classification and segmentationFigure 7: The changes of alpha waves power spectrum in drowsi-ness period
4 H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS
power spectrum, respectively. In Figure 7, two sectionsfrom 75 to 90 seconds and from 110 to 125 secondsshow that alpha waves in occipital area increase anddecrease. Simultaneously, a gradual increase in thetawaves is shown in Figure 8. These two sections are calledalpha-block phenomena. Therefore, in this paper, thesesections were classified into drowsiness state.
5.2 EEG analysis with transient state
In drowsiness state, every section does not always corre-spond to such alpha-blocking patterns. Alpha wave pat-terns like Figure 7 are shown in Figure 9 although thesubject’s eyes are still being open. The subject kept his eyesopen from 235 to 289 seconds and his video recordingsare shown in Figure 10. In Figure 9, the alpha power spec-trum shows a growing trend so that a state of the subjectcan be considered to be drowsy, but his state kept his eyesopen. For accurate analysis, we have taken medical advicesfrom a neurologist in Ulsan University Hospital. In themedical opinion, these sections could be distinguished asdrowsiness with eyes open [6].Especially, in this paper,these sections were defined as transient state.
5.3 Classification results
In order to evaluate the effectiveness of extracted fea-tures, each state classification was performed using EEG
data obtained from different subjects in Chapters 7 and8. Based on the experimental results, an average classifi-cation rate of the proposed method is about 92%. Table 4represents the classification results based on the MLPneural network with respect to EEG data. This tabledescribes how many signals are properly classifiedamong 30 input signal set. For example, in Table 4, as 30Drowsiness data were applied into the system, all caseswere perfectly classified into the Drowsiness category.
In [11], the drowsiness detection system using frequencyanalysis and support vector machine was proposed andshowed approximately 95% accuracy but data length of10 seconds was used so it cannot guarantee performancein short time data. In [20], an independent componentanalysis method was proposed as the main technique ofthe drowsiness detection method and the results of clas-sification represented about 83% accuracy. According to
Figure 8: The changes of theta waves power spectrum in drowsi-ness period
Figure 9: The changes of alpha waves power spectrum in drowsi-ness period (eyes open) [6]
Figure 10: Changes of subject’s eyes in experiment [6]
Table 4: Classification result for EEG test data
Categories
Input data Alertness Transient Drowsiness
Alertness 26 3 1Transient 1 27 2Drowsiness 0 0 30
Note: Bold in numbers (26, 27, and 30) indicates only correct classificationnumber. For example, in the case of alertness, the figure “26” means a correctclassification number for the 30 input data-set.
H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS 5
these classification results, not only does the proposedsystem shows excellent results but its classification proce-dure is computationally efficient and fast because MLP isused as the classifier and the LPC coefficients used as fea-tures can be computed by linear equations.
Considering that proposed system is applied in fieldapplications, there is a very strong possibility of exposureto noise and vibration generated in vehicles. Additionalclassification experiments adding white Gaussian noiseare needed. Experiments are performed changing thenoise level in test data and the classification rate declinesabout 78%. From these results, LPC coefficients are frag-ile to noise. Therefore, by applying the proposed systemsusing LPC coefficients in field applications, the accuracyof data acquisition and noise reduction process have totake precedence.
6. CONCLUSIONS
A vision-based drowsy detection system has been widelystudied and practically applied for vehicles. However,image processing techniques cannot be fundamental solu-tions for drowsiness detection because its accuracy andprecision is not only lower than compared to a physiologi-cal signal but it also cannot detect drowsiness with the eyesopen. However, the proposed method properly classifiesthe transient state, which is drowsiness with the eyes open.
This paper proposes EEG-based drowsiness detectionsystem using LPC coefficients and MLP neural network.In order to evaluate the effectiveness of extracted fea-tures, the classification was performed as using EEGmeasured from different subjects. From the experimentalresults, the proposed system classifies EEG raw data intoeach state. Based on previous studies, the proposed sys-tem has an excellent classification performance.
Since LPC coefficients are computed by a solution of lin-ear equations, the feature extraction procedure of the pro-posed system is computationally fast. Also, MLP used asthe classifier does not need reference data in every teststep compared to other classification methods. Thus, theproposed system fulfills both high classification perfor-mance and computational efficiency so that it can beimplemented with real-time systems and sufficient to beapplied into real driving test if the accuracy of data acqui-sition and noise cancellation pre-processing are required.
ACKNOWLEDGMENTS
This research was supported by University of Ulsan.
DISCLOSURE STATEMENT
No potential conflict of interest was reported by the authors.
REFERENCES
[1] M. V. M. Yeo, X. Li, and E. P. V. Wilder-Smith, “Charac-teristic EEG differences between voluntary recumbentsleep onset in bed and involuntary sleep onset in adriving simulator,” Clin. Neurophysiol., Vol. 118,pp. 1315�23, Jun. 2007.
[2] Q. Ji, Z. Zhu, and P. Lan,, “Real-time nonintrusive moni-toring and prediction of driver fatigue,” IEEE Trans. onVeh Technol, Vol. 53, no. 4, pp. 1052�68. Jul. 2004
[3] H. Kataoka, H. Yoshida, A. Saijo, M. Yasuda, and M.Osumi, “Development of a skin temperature measuringsystem for non-contact stress evaluation,” in Proceedingsof the 20th Annual International Conference of the IEEEEngineering in Medicine and Biology Society, HongKong, 1998, pp. 940�3.
[4] A. Bunde, S. Havlin, J. W. Kantelhardt, T. Penzel, J.-H.Peter and K. Voigt, “Correlated and uncorrelated regionsin heart-rate fluctuations during sleep,” Phys. Rev. Lett.,Vol. 85, pp. 3736�9, Oct. 2000.
[5] A. G. Correa, L. Orosco, and E. Laciar, “Automaticdetection of drowsiness in EEG records based on multi-modal analysis,” Med. Eng. Phys., Vol. 36, pp. 244�9,Feb. 2014
[6] H. Han, U. Chong, “Detection of drowsiness with eyes-open using EEG,” J. Strategic and Int. Stud., Vol. 9, no. 1,pp. 79�86, Jan. 2014.
[7] D. Anirban, A. George, S. L. Happy, and A. Routray. “Avision-based system for monitoring the loss of attentionin automotive drivers,” IEEE Trans. Intell. Transp. Syst.,Vol. 14, no. 4, pp. 1�14, Dec. 2013.
[8] Hart, W. M., Adler’s Physiology of the Eye: ClinicalApplication 9th ed. Philadelphia: Mosby, 1992.
[9] Doughty, M. J., “Further assessment of gender- and blinkpattern-related differences in the spontaneous eyeblinkactivity in primary gaze in young adult humans,” Optom-etry Vision Sci. Vol. 79, pp. 439�47, Jul. 2002.
[10] J. D. Wu, and T. R. Chen, “Development of a drowsinesswarning system based on the fuzzy logic images analysis,”Expert Syst. Appl., Vol. 34, pp. 1556�61, Feb. 2008.
[11] M. V. M. Yeo, X. Li, K. Shen, and E. P. V. Wilder-Smith,“Can SVM be used for automatic EEG detection ofdrowsiness during car driving?,” Saf. Sci., Vol. 47pp. 115�6, Jan 2009.
[12] V. Lawhern, W. D. Hairston, and K. Robbins, “Optimal fea-ture selection for artifact classification in EEG time series,”The Abbreviated journal title should be “Found. Aug-mented Cogn., LNCS”, Vol. 8027, pp 326�34, Jul. 2013.
[13] V. Lawhern, S. Kerick, and K. A. Robbins,“Detectingalpha spindle events in EEG time series using adaptiveautoregressive models,” BMC Neurosci., Vol. 14, no. 101,pp. 1�16, Sep. 2013.
[14] H. Han, S. Cho, and U. Chong, Fault diagnosis system usingLPC coefficients and neural network, Ulsan: Strategic Tech-nology (IFOST), International Forum, 2010, pp. 87�90.
[15] R. P. Lippmann, “Pattern classification using neural net-work,” IEEE Commun. Mag., Vol. 27, no. 11, pp. 47�50,Nov. 1989.
6 H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS
[16] T. Ahsan, T. Jabid, and U.-P Chong, “Facial expressionrecognition using local transitional pattern on gaborfiltered facial images”, IETE Tech Rev, Vol. 30, no. 1,pp.47�52, Jan�Feb 2013.
[17] D. Guan, and W. Yuan, “A survey of mislabeled trainingdata detection techniques for pattern classification,” IETETech Rev, Vol. 30, no. 6, pp.524�30, Nov�Dec 2013.
[18] S. Haykin, Neural Networks: A Comprehensive Founda-tion. New Jersey: Prentice-Hall, 1999, pp.156�248.
[19] H. Han, S. Cho, and U. Chong, “Feature vector decisionmethod of various fault signals for neural-network-basedfault diagnosis system,” Trans. Korean Soc. Noise Vib.Eng., Vol. 20, no.11, pp. 1009�17, 2010.
[20] C.-T. Lin, R.-C. Wu, S.-F. Liang, W.-H. Chao, Y.-J. Chen,and T.-P. Jung,“EEG-based drowsiness estimation forsafety driving using independent component analysis,”IEEE Trans.On Circuit and Syst I, Vol. 52, no. 12, pp.2726�38, Dec. 2005.
AuthorsHyungseob Han received his BS and MSdegrees in computer engineering fromthe University of Ulsan, Ulsan, Korea in2009 and 2011, respectively. He has beenstudying computer engineering at theUniversity of Ulsan to earn his PhDdegree since 2011. He worked for theSchool of Electrical Engineering at theUniversity of Ulsan as a guest professor
in 2013 and 2014. His current research interests include bio-medical signal processing, fault detection and diagnosis in theplants, nonlinear signal analysis, and feature extractionalgorithms.
E-mail: [email protected]
Uipil Chong received the BS degree inelectrical engineering from the Universityof Ulsan, Ulsan, Korea, in 1978, and hisMS degree in electrical engineering fromKorea University, Seoul, Korea in 1980.He studied in the field of computer engi-neering of Oregon State University andreceived MS degree in 1985 and receivedPhD degree at New York University
(POLY), in 1997. In January of 1997, Dr Chong joined theSchool of Electrical Engineering, University of Ulsan in UlsanCity, Korea where he has been promoted to full professor since2006. He has more than 300 papers and holds 30 Korean pat-ents in the area of digital signal processing, fault detection anddiagnosis, biomedical engineering, and multimedia applica-tions. He is a member of IEEE since 1993 and Eta Kappa Nusince 1995. Currently, he is the head of Whale Research Insti-tute in University of Ulsan.
E-mail: [email protected]
H. HAN AND U. CHONG: NEURAL NETWORK BASED DETECTION OF DROWSINESS 7
本文献由“学霸图书馆-文献云下载”收集自网络，仅供学习交流使用。
学霸图书馆（www.xuebalib.com）是一个“整合众多图书馆数据库资源，
提供一站式文献检索和下载服务”的24 小时在线不限IP
图书馆。
图书馆致力于便利、促进学习与科研，提供最强文献下载服务。
图书馆导航：
图书馆首页 文献云下载 图书馆入口 外文数据库大全 疑难文献辅助工具