ARTIFICIAL ARTIFICIAL PASSENGERPASSENGER (A Sleep Prevention Dialogue

Based Car System)

Prepared ByJesalpura Riddhi

09-IT-14Guided By

Mansi Parmar

IndexIndex IntroductionIntroduction Why such system Why such system Artificial passenger Artificial passenger What does it doWhat does it do Devices Devices Working component Working component Advantages Advantages DisadvantagesDisadvantages ApplicationApplication Future enhancementFuture enhancement ConclusionConclusion BibliographyBibliography

IntroductionIntroduction IBM (International business machines IBM (International business machines

corporation, NY) has developed a software corporation, NY) has developed a software that holds a conversation with the driver to that holds a conversation with the driver to determine whether the driver can respond determine whether the driver can respond alertly enough, called “alertly enough, called “Artificial PassengerArtificial Passenger”.”.

This was designed to makeThis was designed to make long solo journeys safer andlong solo journeys safer and more bearable.more bearable.

Why Such System?Why Such System? According to a national survey in UK and USA, it According to a national survey in UK and USA, it

is observed that the driver fatigue annually is observed that the driver fatigue annually causescauses

• 10000 crashes10000 crashes• 1500 deaths 1500 deaths • 7100 injuries 7100 injuries

Majority of off-road accidents observed, were Majority of off-road accidents observed, were caused by eye closure of half and even 2-3 caused by eye closure of half and even 2-3 seconds, where the normal human eye blinks at seconds, where the normal human eye blinks at 0.2-0.3 seconds.0.2-0.3 seconds.

What is an Artificial What is an Artificial Passenger?Passenger?

Natural language e-companion.Natural language e-companion.

Sleep preventive device in cars to Sleep preventive device in cars to overcome drowsiness.overcome drowsiness.

Life safety system.Life safety system.

What Does It Do?What Does It Do?Detects alarm conditions through sensors.Detects alarm conditions through sensors.

Broadcasts pre-stored voice messages Broadcasts pre-stored voice messages over the speakers.over the speakers.

Captures images of the driver.Captures images of the driver.

Condition SensorCondition Sensor

Mobile Indicator Mobile Indicator DeviceDevice

About Artificial PassengerAbout Artificial PassengerThe AP is an The AP is an Artificial IntelligenceArtificial Intelligence

based companion that will be resident in based companion that will be resident in software and chips embedded in the software and chips embedded in the automobile dashboard. automobile dashboard.

The system has a conversation planner The system has a conversation planner that holds a profile of you, including details that holds a profile of you, including details of your interests and profession. of your interests and profession.

A microphone picks up your answer and A microphone picks up your answer and breaks it down into separate words with breaks it down into separate words with speech-recognition software. speech-recognition software.

A camera built into the dashboard also A camera built into the dashboard also tracks your lip movements to improve the tracks your lip movements to improve the accuracy of speech recognition. accuracy of speech recognition.

A voice analyzer then looks for signs of A voice analyzer then looks for signs of tiredness by checking to see if the answer tiredness by checking to see if the answer matches your profile. Slow responses and matches your profile. Slow responses and lack of attention are signs of fatigue.lack of attention are signs of fatigue.

If you reply quickly and clearly, the system If you reply quickly and clearly, the system judges you to be alert and tells the judges you to be alert and tells the conversation planner to continue the line conversation planner to continue the line of questioning. of questioning.

If your response is slow or doesn’t make If your response is slow or doesn’t make sense, the voice analyzer assumes you sense, the voice analyzer assumes you are dropping off and acts to get your are dropping off and acts to get your attention.attention.

If driver displays signs of fatigue, the If driver displays signs of fatigue, the artificial passenger might be programmed artificial passenger might be programmed to open all the windows, sound a buzzer, to open all the windows, sound a buzzer, increase background music volume, or increase background music volume, or even spray the driver with ice water. even spray the driver with ice water.

Devices Used in Artificial Devices Used in Artificial PassengerPassenger

Eye trackerEye tracker Voice recognizer or speech recognizerVoice recognizer or speech recognizer Natural language processorNatural language processor Driver analyzerDriver analyzer Conversational plannerConversational planner AlarmAlarm MicrophoneMicrophone CameraCamera

Working ComponentsWorking ComponentsEye-TrackerEye-Tracker Collecting eye movement data requires both hardware and software.Collecting eye movement data requires both hardware and software.

Hardware:: Head-Mounted systems or remote Hardware:: Head-Mounted systems or remote systemssystems

Both systems measure the corneal reflection of the infrared LED, Both systems measure the corneal reflection of the infrared LED, which illuminates and generates a reflection off the surface of the eye.which illuminates and generates a reflection off the surface of the eye.

This action causes the pupil to appear as a bright disk in contrast to This action causes the pupil to appear as a bright disk in contrast to surrounding iris and creates a small glint underneath the pupil which surrounding iris and creates a small glint underneath the pupil which is used for calibration and tracking.is used for calibration and tracking.

Head mounted systemsHead mounted systems use multiple data points to record use multiple data points to record eye movement and measures pupil glint from multiple angles.eye movement and measures pupil glint from multiple angles.

Remote systemsRemote systems ,in contrast, measure the orientation of the eye ,in contrast, measure the orientation of the eye relative to a fixed unit such as camera.relative to a fixed unit such as camera.

Monitoring SystemMonitoring System

Software:Software: Data collection and analysis is handled Data collection and analysis is handled

by eye-tracking software.by eye-tracking software. Software catalogs eye-tracking in one of Software catalogs eye-tracking in one of

the two waysthe two ways1.1. ERICA’s Eye Gaze software uses a ERICA’s Eye Gaze software uses a

small red x to represent eye movement.small red x to represent eye movement.2.2. In other, data is stored as a series of x/y In other, data is stored as a series of x/y

coordinates related to specific grid points coordinates related to specific grid points on the computer screenon the computer screen

Eye TrackerEye Tracker

Working Working Components( cont…)Components( cont…)

Automatic Speech RecognitionAutomatic Speech Recognition

There are two ASRs used in the system:There are two ASRs used in the system:

First one is “speaker independent” and used for First one is “speaker independent” and used for decoding voice signals of the driver.decoding voice signals of the driver.

Second one operates with voice car media and Second one operates with voice car media and decodes tapes, audio , telephones, mails etc.decodes tapes, audio , telephones, mails etc.

Working Working Components( contd…)Components( contd…)

Natural Language ProcessorNatural Language Processor Processes the decoded signals of voice or text data provided Processes the decoded signals of voice or text data provided

from the ASR. from the ASR. Identifies related meanings from the contents of the decoded Identifies related meanings from the contents of the decoded

messages.messages.

Produces variant of responses.Produces variant of responses.

This output goes to the driver analyzer as an input.This output goes to the driver analyzer as an input.

Working Working Components( contd…)Components( contd…)

Driver AnalyzerDriver Analyzer

Receives the textual data and voice data. Receives the textual data and voice data.

Measures the time response using a clock.Measures the time response using a clock.

Time responses, conclusions about driver’s Time responses, conclusions about driver’s alertness will be passed on to the alertness will be passed on to the conversational planner.conversational planner.

Working Working Components( contd…)Components( contd…)

Conversational PlannerConversational Planner It is the heart of the system. It is the heart of the system.

Instructs the language generator to produce the Instructs the language generator to produce the response.response.

If the driver is in perfect condition CP instructs the If the driver is in perfect condition CP instructs the language generator to continue the conversation language generator to continue the conversation otherwise it will be instructed to change the otherwise it will be instructed to change the conversation.conversation.

Working Components Working Components (contd…)(contd…)

AlarmAlarm If the CP (conversational planner) receives If the CP (conversational planner) receives

information that the driver is about to fall asleep information that the driver is about to fall asleep an alarm system is activated.an alarm system is activated.

MicrophoneMicrophone For picking up words and separate For picking up words and separate them by some internally used softwarethem by some internally used software for conversation. for conversation.

Working Components Working Components (contd…)(contd…)

CameraCameraThis will track the lip movements of the This will track the lip movements of the

driver.driver.Used to improve the accuracy of the Used to improve the accuracy of the

speech recognition.speech recognition.

..

Showing the dashboard of the car where the whole artificial system is generally Showing the dashboard of the car where the whole artificial system is generally attached.attached.

Steps Involved in Steps Involved in TrackingTracking

Automatically initialize lips and eyes using Automatically initialize lips and eyes using color predicates and connected color predicates and connected components. components.

Track lip corners using dark line between Track lip corners using dark line between lips and color predicate even through large lips and color predicate even through large mouth movement like yawning. mouth movement like yawning.

Construct a bounding box of the head.Construct a bounding box of the head.

Determine rotation using distances Determine rotation using distances between eye and lip feature points and between eye and lip feature points and sides of the face. sides of the face.

Determine eye blinking and eye closing Determine eye blinking and eye closing using the number and intensity of pixels in using the number and intensity of pixels in the eye region.the eye region.

Determine driver vigilance level using all Determine driver vigilance level using all acquired information.acquired information.

DetectingDetecting Driver Driver VigilanceVigilance Aiming a single camera at a head of the driver. Aiming a single camera at a head of the driver.

Detecting frequency of up and down nodding and left to Detecting frequency of up and down nodding and left to right rotations of the head within a selected time period right rotations of the head within a selected time period with the camera. with the camera.

Determining frequency of eye blinking and eye closing.Determining frequency of eye blinking and eye closing.

Determining frequency of yawning of the driver within the Determining frequency of yawning of the driver within the selected time period with the cameraselected time period with the camera

Generating an alarm signal in real time if the frequency Generating an alarm signal in real time if the frequency value of the up and down nodding, the left to right value of the up and down nodding, the left to right rotations, the eye blinking, the eye closings, the yawning rotations, the eye blinking, the eye closings, the yawning exceeds a selected threshold value. exceeds a selected threshold value.

AdvantagesAdvantages It is broadly used to prevent accident.It is broadly used to prevent accident.Artificial component establishes interface Artificial component establishes interface

with other drivers very easily.with other drivers very easily. If the driver gets a heart attack or drunk , it If the driver gets a heart attack or drunk , it

will send signals to vehicles nearby about will send signals to vehicles nearby about this so driver there become alert.this so driver there become alert.

DisadvantagesDisadvantages It is costly. It can only be installed to high profiled

ApplicationsApplications Interface with Neighboring CarsInterface with Neighboring Cars - -Determines if a driver Determines if a driver

presents a high safety risk for ex. falling asleep, tired, inexperienced presents a high safety risk for ex. falling asleep, tired, inexperienced or under the influence of alcohol and signals the cars nearby to be or under the influence of alcohol and signals the cars nearby to be careful of the driver.careful of the driver.

Medical ApplicationMedical Application -The system can monitor a driver and -The system can monitor a driver and detect if they are sick ,for ex. having a stroke or heart attack.detect if they are sick ,for ex. having a stroke or heart attack.

In any problem it alerts the vehicles near by , so the driver there can In any problem it alerts the vehicles near by , so the driver there can become alert.become alert.

Opens and closes the doors and windows of the car automatically.Opens and closes the doors and windows of the car automatically.

It is also used for the entertainment.It is also used for the entertainment.

Also used in cabins in airplanes, trains, boats etcAlso used in cabins in airplanes, trains, boats etc

Future ImplementationFuture ImplementationWill provide us with shortest time routing Will provide us with shortest time routing

based on road conditions changing based on road conditions changing because of weather and traffic, information because of weather and traffic, information about the cars on the route, destination about the cars on the route, destination requirement (as-flight has been delayed requirement (as-flight has been delayed etc)etc)

ConclusionConclusionMethod for monitoring driver alertness Method for monitoring driver alertness Sufficient time to avert an accident. Sufficient time to avert an accident. Successful implementation of Artificial Successful implementation of Artificial

passenger would allow use of various passenger would allow use of various services in car like reading emails, services in car like reading emails, navigation, downloading music files, navigation, downloading music files, voice games etc without compromising voice games etc without compromising on driver safetyon driver safety

BibliographyBibliography L R Bhal et al. “Performance of the IBM speech L R Bhal et al. “Performance of the IBM speech

recognition systemrecognition system www.freepatentsonline.com/4682348.htmlwww.freepatentsonline.com/4682348.html www.slideshare.com www.slideshare.com www.about.comwww.about.com www.answers.comwww.answers.com www.google.comwww.google.com www.wikipedia.com www.wikipedia.com   

Thank Thank You…You…