ARTIFICIAL PASSENGERARTIFICIAL PASSENGER

What is an artificial What is an artificial passenger?passenger?

Natural language e-companion.Natural language e-companion.

Sleep preventive device in cars to Sleep preventive device in cars to overcome drowsiness.overcome drowsiness.

Life safety system.Life safety system.

What does it do?What does it do?

Detects alarm conditions through Detects alarm conditions through sensors.sensors.

Broadcasts pre-stored voice Broadcasts pre-stored voice messages over the speakers.messages over the speakers.

Captures images of the driver.Captures images of the driver.

Devices that are used in Devices that are used in APAP

The main devices that are used in this The main devices that are used in this artificial passenger are:-artificial passenger are:-

Eye tracker.Eye tracker.

Voice recognizer or speech Voice recognizer or speech recognizer.recognizer.

About APAbout AP

The AP is an artificial intelligence–based The AP is an artificial intelligence–based companion that will be resident in companion that will be resident in software and chips embedded in the software and chips embedded in the automobile dashboard. automobile dashboard.

The system has a conversation planner The system has a conversation planner that holds a profile of you, including that holds a profile of you, including details of your interests and profession. details of your interests and profession.

A microphone picks up your answer A microphone picks up your answer and breaks it down into separate and breaks it down into separate words with speech-recognition words with speech-recognition software. software.

A camera built into the dashboard A camera built into the dashboard also tracks your lip movements to also tracks your lip movements to improve the accuracy of the speech improve the accuracy of the speech recognition.recognition.

A voice analyzer then looks for signs A voice analyzer then looks for signs of tiredness by checking to see if the of tiredness by checking to see if the answer matches your profile. Slow answer matches your profile. Slow responses and a lack of attention are responses and a lack of attention are signs of fatigue.signs of fatigue.

If you reply quickly and clearly, the If you reply quickly and clearly, the system judges you to be alert and system judges you to be alert and tells the conversation planner to tells the conversation planner to continue the line of questioning. continue the line of questioning.

If your response is slow or doesn’t If your response is slow or doesn’t make sense, the voice analyzer make sense, the voice analyzer assumes you are dropping off and assumes you are dropping off and acts to get your attention. acts to get your attention.

DetectingDetecting driver vigilancedriver vigilance

Aiming a single camera at a head of a Aiming a single camera at a head of a driver. driver.

Detecting frequency of up and down Detecting frequency of up and down nodding and left to right rotations of nodding and left to right rotations of the head within a selected time period. the head within a selected time period.

Determining frequency of eye Determining frequency of eye blinkings and eye closings. blinkings and eye closings.

HOW DOES TRACKING DEVICE HOW DOES TRACKING DEVICE WORK?WORK?

Data collection and analysis is Data collection and analysis is handled by eye-tracking software. handled by eye-tracking software.

Data are stored as a series of x/y Data are stored as a series of x/y coordinates related to specific grid coordinates related to specific grid points on the computer screen. points on the computer screen.

Our head tracker consists of tracking the Our head tracker consists of tracking the lip corners, eye centers, and side of the lip corners, eye centers, and side of the face.face.

““Occlusion” of the eyes and mouth often Occlusion” of the eyes and mouth often occurs when the head rotates or the eyes occurs when the head rotates or the eyes close, so our system tracks through such close, so our system tracks through such occlusion and can automatically reinitialize occlusion and can automatically reinitialize when it mis-tracks.when it mis-tracks.

Representative ImageRepresentative Image

Eye TrackerEye Tracker

Monitoring SystemMonitoring System

Tracking includes the following Tracking includes the following stepssteps

Automatically initialize lips and eyes Automatically initialize lips and eyes using color predicates and connected using color predicates and connected components. components.

Track lip corners using dark line Track lip corners using dark line between lips and color predicate between lips and color predicate even through large mouth movement even through large mouth movement like yawning. like yawning.

Construct a bounding box of the head.Construct a bounding box of the head.

Determine rotation using distances Determine rotation using distances between eye and lip feature points and between eye and lip feature points and sides of the face. sides of the face.

Determine eye blinking and eye Determine eye blinking and eye closing using the number and intensity closing using the number and intensity of pixels in the eye region.of pixels in the eye region.

The lip and eye colors ((RED, BLUE, The lip and eye colors ((RED, BLUE, GREEN)RGB) are marked in the GREEN)RGB) are marked in the image offline.image offline.

Mark the lip pixels in the image is Mark the lip pixels in the image is important.important.

Each pixel has an Red(R), Green)G), and Each pixel has an Red(R), Green)G), and Blue(B) component. For a pixel that is Blue(B) component. For a pixel that is marked as important, go to this location in marked as important, go to this location in the RGB array indexing on the R, G, B the RGB array indexing on the R, G, B components.components.

This array location can be incremented by This array location can be incremented by equation (1): equation (1): exp(−1.0*( exp(−1.0*( j*j+k*k+i*i j*j+k*k+i*i )/(2*sigma*sigma)); )/(2*sigma*sigma)); (1) (1) where: sigma is approximately 2; where: sigma is approximately 2;

If a color, or pixel value, is marked as If a color, or pixel value, is marked as important multiple times, its new important multiple times, its new value can be added to the current value can be added to the current value. Pixel values that are marked value. Pixel values that are marked as unimportant can decrease the as unimportant can decrease the value of the RGB indexed location via value of the RGB indexed location via equation (2) as follows: equation (2) as follows: exp(−1.0*( exp(−1.0*( j*j+k*k+i*i j*j+k*k+i*i )/(2*(sigma−1)*(sigma−1))). (2) )/(2*(sigma−1)*(sigma−1))). (2) where: sigma is approximately 2; where: sigma is approximately 2;

The values in the array which are The values in the array which are above a threshold are marked as above a threshold are marked as being one of the specified colors. being one of the specified colors. Another RGB array is generated of Another RGB array is generated of the skin colors, and the largest non-the skin colors, and the largest non-skin components above the lips are skin components above the lips are marked as the eyesmarked as the eyes

First a dark pixel is located First a dark pixel is located

The system goes to the eye center of the The system goes to the eye center of the previous frame and finds the center of previous frame and finds the center of mass of the eye region pixelsmass of the eye region pixels

Look for the darkest pixel, which Look for the darkest pixel, which corresponds to the pupilcorresponds to the pupil

This estimate is tested to see if it is close This estimate is tested to see if it is close enough to the previous eye locationenough to the previous eye location

Feasibility occurs when the newly Feasibility occurs when the newly computed eye centers are close in computed eye centers are close in pixel distance units to the previous pixel distance units to the previous frame's computed eye centers. This frame's computed eye centers. This kind of idea makes sense because kind of idea makes sense because the video data is 30 frames per the video data is 30 frames per second, so the eye motion between second, so the eye motion between individual frames should be relatively individual frames should be relatively small. small.

If new points are too far away, the system If new points are too far away, the system searches a window around the eyes and searches a window around the eyes and finds all non-skin connected components finds all non-skin connected components in approximately a 7×20 pixel window, in approximately a 7×20 pixel window, and finds the slant of the line between the and finds the slant of the line between the lip corners using equation (5). This lip corners using equation (5). This equation finds the slope between two equation finds the slope between two points in general. points in general. (( (( y y 2 2 −y −y 1 )/( 1 )/( x x 2 2 −x −x 1 )) (5) 1 )) (5) where: x 1 ,y 1 is the coordinate of a where: x 1 ,y 1 is the coordinate of a feature; and x 2 ,y 2 is the coordinate of feature; and x 2 ,y 2 is the coordinate of the other corresponding feature. the other corresponding feature.

The system selects the eye centroids The system selects the eye centroids that have the closest slant to that of that have the closest slant to that of the slant between the lip corners the slant between the lip corners using equation (5). These two stages using equation (5). These two stages are called the eye black hole tracker. are called the eye black hole tracker.

The detection of eye occlusion is done by The detection of eye occlusion is done by analysing the bright regions.analysing the bright regions.

As long as there are eye-white pixels in As long as there are eye-white pixels in the eye region the eyes are open. If not the eye region the eyes are open. If not blinking is happening. To determine what blinking is happening. To determine what eye-white color is, in the first frame of eye-white color is, in the first frame of each sequence we find the brightest pixel each sequence we find the brightest pixel in the eye region and use this as the eye in the eye region and use this as the eye white color.white color.

If the eyes have been closed for If the eyes have been closed for more than approximately 40 of the more than approximately 40 of the last approximately 60 frames, the last approximately 60 frames, the system declares that driver has his system declares that driver has his eyes closed for too long.eyes closed for too long.

OutputOutput

Onsite alarms within the vehicleOnsite alarms within the vehicle

Remote alarmsRemote alarms

Other applicationsOther applications

1) Cabins in airplanes.1) Cabins in airplanes.

2) Water craft such as boats.2) Water craft such as boats.

3) Trains and subways.3) Trains and subways.

SUMMARY SUMMARY

Method for monitoring driver Method for monitoring driver alertnessalertness

Sufficient time to avert an accident.Sufficient time to avert an accident.

Monitoring of full facial occlusion of Monitoring of full facial occlusion of the driver. the driver.

THANK YOUTHANK YOU