Emotional Design

• D. Norman (Emotional Design, 2004)

• Model with three levels

– Visceral (lowest level)

– Behavioral (middle level)

– Reflective (top level)

Emotional Intelligence (EI)

• IQ is not the only indicator of intelligence (Emotional Intelligence book by Daniel Goleman, 1995)

• EI: Awareness and ability to manage one’s emotions in a healthy manner.

• EI: Ability to sense, perceive, understand, and assess own and other people’s emotions

Is Mr. Spock intelligent?

• Spock is only rational

• Descarte’s Error (Damasio, 1994)

• Artificial intelligence searches unlimited

search space to make a rational decision

• Missing ‘somatic markers’ that associate

feelings with decisions

Damasio’s Somatic Marker Hypothesis

• Originated from the observation of individuals who had sustained damage to the ventromedial prefrontal cortex.

• Normal intellectual function

• Normal Neuropsychological function

• Normal on tests sensitive to frontal lobe function

• However, severe impairment in personal and social decision making and conduct. – Difficulty with planning in the immediate, and future.

– No longer able to make personally advantageous decisions

– Often sustain social, personal, economic losses

• The only deficit that could be detected was one in which these individuals failed to display emotion in situations in which emotion would be normatively expected.

• This led Damasio to posit that these individuals manifest a deficit in reasoning that is secondary to deficits in emotional processing.

What is Affect?

• The type and degree of emotion a person

displays

• The experienced, subjective, and

conscious aspect of feeling or emotion

– Positive

– Negative

– Neutral

Affect Theory

• Developed by Silvan S. Tomkins in 1962

• Tomkins book Affect Imagery (3 vols.)

• Believed that the affect system is the

motivating force in human life.

• Organized affect into 3 main categories:

– Positive, negative, and neutral

– Each has a low/high intensity label

Tomkins nine affects

• Positive:

• Enjoyment/Joy - smiling, lips wide and out

• Interest/Excitement - eyebrows down, eyes tracking, eyes looking, closer listening

• Neutral:

• Surprise/Startle - eyebrows up, eyes blinking

• Negative:

• Anger/Rage - frowning, a clenched jaw, a red face

• Disgust - the lower lip raised and protruded, head forward and down

• Dissmell (reaction to bad smell) - upper lip raised, head pulled back

• Distress/Anguish - crying, rhythmic sobbing, arched eyebrows, mouth lowered

• Fear/Terror - a frozen stare, a pale face, coldness, sweat, erect hair

• Shame/Humiliation - eyes lowered, the head down and averted, blushing

Affective Computing

• Computing that relates to, arises from, or

deliberately influences emotions

• Coined by Rosalind Picard

– Founder and director of the Affective Computing

Research Group at the MIT Media Lab.

– Her book, Affective Computing (1997) lays the

groundwork for giving machines the skills of

emotional intelligence.

Affective computing is related to other computing disciplines:

Artificial Intelligence (AI),

Virtual Reality (VR) and

Human Computer interaction (HCI).

Questions that need to be answered:

What is an affective state (typically feelings, moods, sentiments etc.)?

Which human communicative signals convey information about affective state?

How various kinds of affective information can be combined to optimize inferences

about affective states?

How to apply affective information to designing systems?

Affective Computing

• Recognize emotions

• Express emotions

• ‘Have’ emotions

Recognize Emotions

• Bio-signals (wearable sensors)

• Brain Signals, skin temperature, blood

pressure, heart rate, respiration rate

• Facial Expressions

• Speech/Vocal expressions

• Gestures

• Limbic movements

• Text

Recognition

• we need an emotion model that allows us to

differentiate between emotional states

• we need a classification scheme that uses

specific features from an input signal to recognize

the user’s emotions

Text

Sentiment Analyzing Discussion Board

http://socialxyz.com/SAD/

The Natural Language Toolkit, or more commonly NLTK, is a

suite of libraries and programs for symbolic and statistical

natural language processing (NLP) for the Python

programming language.

Speech

Paralinguistic Features of Speech – how is it said?

Prosodic features (e.g., pitch-related feature, energy-related features,

and speech rate)

Spectral features (e.g., MFCC - Mel-frequency cepstral coefficient

and cepstral features)

Spectral tilt, LFPC (Log Frequency Power Coefficients)

F0 (fundamental frequency of speech), Long-term spectrum

Studies show that pitch and energy contribute the most to affect

recognition

Speech disfluencies (e.g., filler and silence pauses)

Context information (e.g., subject, gender, and turn-level features

representing local and global aspects of the dialogue)

Nonlinguistic vocalizations (e.g., laughs and cries, decode other

affective signals such as stress, depression, boredom, and excitement)

Feature Extraction Pre-processing

Speech Signal

Classification

Classified Result Audio recordings collected in call centers

and, meetings, Wizard of Oz scenarios

interviews and other dialogue systems

• Accuracy rates from speech are somewhat lower

(35%) than facial expressions for the basic emotions .

• Sadness, anger, and fear are the emotions

that are best recognized through voice, while

disgust is the worst.

]M. Pantic, N. Sebe, J. F. Cohn, and T. Huang. Affective multimodal human-computer interaction. In ACM International Conference on Multimedia (MM), 2005. Rafael A. Calvo, Sidney D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications

Lucey, P.; Cohn, J.F.; Kanade, T.; Saragih, J.; Ambadar, Z.; Matthews, I.; , "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression," Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on , vol., no., pp.94-101, 13-18 June 2010

Facial Expressions

Example: Active Appearance Model (AAM)

(AAM) based system which uses AAMs to track the face

and extract visual features. Support vector machines are used

(SVMs) to classify the facial expressions and emotions.

Bio-signals 22

• Physiological signals derived from Autonomic

Nervous System (ANS) of human body.

– Fear for example increases heartbeat and

respiration rates, causes palm sweating, etc.

• Psychological Metrics used are:

– GSR - Galvanic Skin Resistance

– RESSP - Respiration

– BVP - Blood Pressure

– Skin Temperature

• Electroencephalogram (EEG), Electrocardiogram

(ECG), Electrodermal activity (EDA),

Electromyogram (EMG)

• Skin conductivity sensors, blood volume sensors, and

respiration sensors may be integrated with shoes,

earrings or watches, and T-shirts

Huaming Li and Jindong Tan. 2007. Heartbeat driven medium access control for body sensor networks. In Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments (HealthNet '07). ACM, New York, NY, USA, 25-30.

gesture and body motion information is an important modality for human affect recognition; combination of face and gesture is 35% more accurate than facial expression alone.

Two categories of Body-Motion-based affect recognition

Stylized

The entirety of the movement encodes a particular emotion.

Non-stylized

More natural - knocking door, lifting hand, walking etc.

Gestures

Fusion

• Neural Networks (NN) • Hidden Markov Models (HMM)

• K-Nearest Neighbors (KNN)

• Linear Discriminant Analysis (LDA)

• Support Vector Machines (SVM)

• Gaussian Mixture Models (GMM)

• Discriminant Function Analysis (DFA)

• Sequential Forward Floating Search (SFFS)

Frequently used Detection and

Estimation Techniques

Express emotions

• Kismet (Breazeal and Scassellati, 2002)

• Emotional expression for

communication and social co-ordination

• Emotion for organisation of behaviour

(action selection, attention and learning)

• Arbib and Fellous (2004)

• More effective expression than humans:

• Human expression identified 50% of the

time. Computer expression identified

70% of the time (Elliott, 1997).

https://www.youtube.com/watch?v=PtCIbGjJV4c

1

3

2

4

1

3

2

4

1. Listening

2. Understand

3. Confused

4. Waving goodbye

EXPERIMENTS AND RESULTS

35 35

Navigation Agent

Safety Agent

Driving Aid Agent

Affective Multimedia Agent

Audio Linguistic / Non-linguistic

Facial Expression

Seat Pressure

Actions •Steering Movement

•Interaction with Gas / Break Paddle

Bio-signals

Stress Level Basic Emotions

Feature Detector

Feature Detector

Feature Detector

…………...

Feature Estimator

Complex Emotions

……

Route Selection

Inter agent

communication

to aid decision

making

Notify

in case of

Emergency

Speed, ABS,

Traction Control

Music, Climate

Control

Alert the Driver

Have emotions

• Can machines feel?

• How would we know?

Criteria for having emotions

• System has behavior that appears to arise from emotions

• System has fast ‘primary’ emotional responses to certain inputs

• System can cognitively generate emotions

• System can have emotional experience

• System’s emotions interact with other processes (e.g. memory)