models cognitive modeling - cslcsl.anthropomatik.kit.edu/.../km2012-v8-affectivemodels.pdf ·...
TRANSCRIPT
1/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
Affective Models
Cognitive Modeling
Dominic Heger,
Felix Putze,
Tanja Schultz
21.6.2012
2/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Outline
• Affective Computing
• Emotion Models • Descriptive models
• Appraisal theories
• Big Five Personality Theory
• Affect Recognition • Facial expressions, Voice, Physiology
• Examples for Affective Models in Systems • ALMA
• Kismeth
• Affect Expression in Emotional Robots
3/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
1. Affective Computing
4/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Affective Computing
• Affective Computing is a very active research topics
• Affective Computing (Rosalind Picard, MIT Press 1997) • Computing that relates to, arises from, or deliberately influences
emotions (p. 3)
• How and why computers might be designed to recognize, express, and “have” emotions
• “Affect refers to the experience of feeling or emotion”1
• → Emotion and Affect are often used equivalently
• Affective States in Human Computer Interaction • Emotion
• Mood
• Activation, Workload, … 1APA Dictionary of Psychology (2006)
5/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Media Equation
• Reeves & Nass: Media equation • Media = real life
• Humans treat computers, televisions, and other media in the same way they treat people
• Apply social rules from human interaction to machines
Users react to social signals sent by the system
Users expect systems to react to social signals they send
Machine, which do not follow those rules are perceived as unintelligent and impolite
• Numerous experiments:
People are polite to computers
Treat computers with female voice differently than computes with male voice (gender differences)
Etc.
6/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Fields for affective computers
• Human-Computer Interaction is unnatural • Adaptation to the user
• Humanoids and Embodied Conversational Agents
• Text based communication and virtual reality do not carry emotional information • Emotion carrying communication and telepresence
• Emotion recognition → communication channel → emotion expression
• Monitoring of humans • Call centers, elderly care, surveillance, etc.
• Educational systems • Emotions are known to influence learning
• E.g. Yerkes-Dodson law, flow, etc.
7/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Emotion Theory
• Numerous theories have been proposed
• Theories have different focus and different schools
• Conceptualization by six aspects (components1) of emotions • Subjective experiences (feelings)
• Psychophysiological changes (bodily sensations)
• Appraisals of events (triggered by a stimulus)
• Emotion regulation
• Motor expressions (face, voice, gestures)
• Action tendencies
• Changes in one component can lead to corresponding changes in others (interrelated)
1Fontaine, et al. (2007): “The world of emotions is not two-dimensional”
8/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Why do we have Emotions?
• Functions of emotions • Matter of communication
Essential part of social interaction
• Regulate intensity and duration of actions
E.g. influences motivation
• Influence on learning
• Decision making and intuitions
“Selling is a transfer of emotions”
• Control behavior according to needs and situation
E.g. Signals danger situations (faster responses than rational decisioning)
• Etc.
• Emotional Intelligence is important in human life
10/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Frontal Lobe – Emotions – Decision Making
• Descartes’ Error (Damasio, 1994) • No dualist separation of mind and body / rationality and emotions
• In complex and uncertain situations with limited time emotions play critical role in decision making (e.g. gut feelings)
• Somatic marker hypothesis: emotional memory sends decision signals
• Emotions guide behavior and decision making
• Rationality requires emotional input
• Case of Phineas Gage (19th century) • One of most famous cases in neuro psychology
• Accident damaged left frontal lobe by iron rod
• No fundamental inabilities but strong change in mental attitude, emotions, and decision making abilities
11/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
2. Emotion Models
12/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Discrete/Categorical Models
• Anger, disgust, fear, joy, sadness, or surprise ?
13/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Discrete/Categorical Models
• Six or more basic emotions • Darvin, Ekman, Plutchik, …
• E.g. Anger, fear, disgust, surprise, joy, sadness, etc.
• Distinctive universal signals (e.g. facial expressions)
• Cross-culturally displayed and recognized
14/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Different Basic Emotion Models
From Ortony and Turner, 1990: “What’s basic about basic emotions”
15/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Dimensional Models
• Describe a small number of dimensions
• Emotion is point in this multi-dimensional space • Russell (1980): Bipolar circumplex model (dims: Valence and Arousal)
• Mehrabian (1995): PAD model (Pleasure, Arousal, Dominance)
Valence (quality): unpleasant to pleasant
Positive vs. negative affective states
Arousal (quantity): calm to excited
Mental and/or physical activity level
Dominance (control): weak to strong
Control or lack of control over others or situations
• Mostly generated from data corpora (data-driven approach) • Emotional adjectives, Facial expressions, Emotional experiences, …
• Statistical methods to find latent dimensions in data corpora
Factor analysis, Principal Component Analysis, …
16/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Dimensional Models
• How many dimensions do emotion models need? • Depends on application and goals
• E.g. Difficult to discriminate anger, fear, and stress in Valence-Arousal-Model
17/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Relationship Discrete and Dimensional Models
Adapted from Russel 1980 (by Picard)
18/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Plutchik’s theory of basic emotions
• 1960-1980s Emotion – A Psychoevolutionary Synthesis
• 3D circumplex model of emotions • Analogous to color wheel
• 8 Primary bipolar emotions
• Different intensities of emotions
• Combines discrete and dimensional models
• Can be mixed (secondary emotions)
19/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Appraisal Models
• Appraisals • Mental representations of a situation by an individual
(including interpretations and explanations)
• Emotions in Appraisal Models • Emotions are cognitive appraisals of antecedent situations
• I.e. responses based on evaluations of situations and events
• Appraisal Theory can explain shortcomings of other theories • Variability and degree of response in emotional reactions
• Similar situations and different reactions
• Similar reaction to different situations
20/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Emotion Model
• Ortony, Clore und Collins (1988)
• One of the most famous emotion models for virtual emotional characters
• Several more or less simplified implementations for exist
• Distinguishes 22 emotion categories by evaluating
• Consequences of events (Happy or unhappy)
• Aspects of objects (Like or dislike)
• Actions of agents (approve or disapprove)
21/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Model Structure
22/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Emotion Model
• Knowledge (goals, standards, attitudes) needed for evaluation • Specified by designer of the character
• World model: E.g. Large Table with events, actions, objects, affected emotion category,…
• Intensity variables for each emotion category • After classification of the emotion category
• Depend on agent’s goals and history of events
• Example: User gives agent bunch of bananas • Traversal of the flow diagram from the perspective of the agent
• Appraisal evaluation results in emotion
23/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Model Structure
• Consequences of giving bananas for the user → Pity
24/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Model Structure
• Consequences of giving bananas for the agent itself (agent is hungry) → Satisfaction
25/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Model Structure
• Action of the user (agent respects selfless action) → Admiration
26/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s OCC Model Structure
• Aspect of the object (agent has passion for bananas) → Love
27/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Component Process Model of Emotion
• Scherer (2001): “Appraisal Considered as a Process of Multilevel Sequential Checking”
• Cognitive appraisal modeled by Stimulus Evaluation Checks (SEC) • “Minimal set […] to account for the differentiation of the major families of
emotional states”
• Fixed sequence of checks
• Four Appraisal Objectives (types or classes of information required for an organism to prepare an appropriate reaction) 1. Relevance Detection
2. Implication Assessment
3. Coping Potential Determination
4. Normative Significance Evaluation
28/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s CPM Stimulus Evaluation Checks
1. Relevance Detection
Novelty Check (e.g. predictability, familiarity)
Intrinsic Pleasantness Check (likely to result in pleasure or pain?)
Goal Relevance Check (e.g. relevance for own goals and needs)
2. Implication Assessment
Causal Attribution Check (reason for the event)
Outcome Probability Check (how certain is the expected result?)
Discrepancy from Expectation Check (is the event as expected?)
Goal/Need Conduciveness Check (helpful for own goals?)
Urgency Check (direct action required?)
29/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s CPM Stimulus Evaluation Checks
3. Coping Potential Determination
Control Check (How far is the event controllable?)
Power Check (How far am I able to control the event?)
Adjustment Check (How well deal with consequences of an event?)
4. Normative Significance Evaluation
Internal Standards Check (Matches my ideals and moral?)
External Standards Check (e.g. social consequences of an action, compatibility with standards of a salient reference group?)
30/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Component Process Model
• Predict specific changes caused by patterns of SEC results
• Response patterns for organismic subsystems • Example response patterns for Novelty Check:
31/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Component Process Model
• Major “modal“ emotions (happiness, disgust, anxiety, …) can be predicted by SECs • Relationship between organismic responses patterns and SECs
• Relationship between SECs and modal emotion
• Specific response profiles by vocal and facial expressions for modal emotions exist
33/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Example: Attitude vs Mood vs Emotion
• Attitude (evaluation of a concept in general) • “I find it uncomfortable to go to the dentist.”
• Mood (longer lasting, general feeling) • “I am nervous because I have to go to the dentist tomorrow.”
• Emotion (short-lived, stimulus-triggered feeling) • “I just heard someone scream in the doctor’s room. I am frightened.”
34/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Personality Modeling
Numerous Personality theories exist
One of many definitions of personality:
Major differences to emotions Personality is stable across long periods of time
Not caused by an event or appraisal of an event
Personality fundamentally coins human behavior
„Characteristics of the person that account for
consistent patterns of feeling, thinking, and
behaving“ [Pervin et al., 2001]
35/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Big Five Theory of Personality Traits
Five personality traits describing general dimensions of human personality
Personality can be seen as point in a 5-dim space
Very widespread and widely researched psychological personality theories
Based on the Lexical Hypothesis: All aspects relevant to describe personality are encoded as words in language
Developed using factor analysis
Elaborated personality assessment by questionnaires → NEO-FFI
36/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Big Five Dimensions
Neuroticism: Stability/lability of (negative) emotional experiencing
Extraversion: Sociability, positive emotions, excitement-seeking,...
Openness to Experience: Interest in new experiences and impressions
Agreeableness: Altuism, desire for harmony,...
Conscientiousness: Achievment striving, self-discipline,...
Self-assured, talkative, optimistic, cheerful
Preference of being on their own
Worried, concerned, unsure, anxious
Calm, balanced, carefree, collected
Fancyfull, imaginative, prefer diversification
Conventional, practical, straightforward
Dutiful, reliable, precise, squeamish
Careless, incurious, less systematic
Benevolent, caring, helpful, cooperative
Antagonistic, egocentric, misrustfull
37/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
3. Emotion Recognition
38/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Empirical Affect Modeling
• Empirical Modeling of Affect, i.e. Emotion Recognition • Speech, visual signals, biophysiological signals
• Emotional Speech Classification • Prosody features
Pitch variables (F0 level, range, contour, jitter, etc.)
Voice quality (articulation manner, voice timbre, etc.)
Speaking rate, …
• No set of voice cues to reliably discriminate among emotions (Juslin & Scherer, 2005)
• Syntax, Semantic, dialog strategy, …
39/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Visual Signals
• Facial Action Coding System (Ekman & Friesen) • Comprehensive description of facial expressions
• Action Units (AUs) for all visually distinguishable facial movements
• Intensity of AUs (5 point scale)
• Multiple AUs can be combined
• Mapping between basic emotions and combinations of AUs
• MPEG-4 Facial Animation Parameters • Facial Expressions
• Visemes
• Other visual signals: Gestures, Pose, ...
40/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Physiological Biosignals
• Several physiological correlates to emotions • Blood pressure, Heart rate, Respiration rate
Arousal: Increase
Negative stimuli: Ambiguous results, most studies report decrease (at least for picture viewing)
• Skin conductance
Arousal
Emotional stimuli (positive and negative): increase
• Brain activity
No specific emotion areal
Reward system (dopamine pathways): ventral tegmentum, medial forebrain, nucleus accumbens
Several correlation to activity in amygdala, hippocampus (limbic system), asymmetry at prefrontal cortex, etc.
42/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Affective Data Collection
• Picard et al., 2001: “Toward machine emotional intelligence” • Spontaneous vs. posed
Asked to elicit a certain emotion?
• Lab setting vs. real-world
Usual environment of the subject?
• Expression vs. feeling
External or internal, sender or receiver?
• Open recording vs. hidden recording
Is subject aware of data collection?
• Emotion-purpose vs. other-purpose
Does subject know that she is part of an emotion experiment?
• Challenges • Segmentation and annotation (ground truth) is difficult
• No common test data to compare recognition rates
43/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Emotion and Expression
• Difference between true emotional state and communicated state
• Smiling associated with happiness, but: • happiness is neither necessary not sufficient for smiling
• ∃ evidence that smiles appear limited to social circumstances
• ∃ evidence that smiles occur in humorous rather than pleasant circumstances
• ∃ evidence that some expressions are uncorrelated with emotion • r = 0.78 between cognitive appraisal of unexpectedness and self-
reported surprise
• r = 0.46 between self-reported surprise and facial expression
44/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
4. Affective Models in Systems
45/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s ALMA – A Layered Model of Affect
• Example for virtual characters: VirtualHuman (DFKI, 2005)
• Goal: Human-like conversational characters, to generate a Learning group experience for the user
• Layered model of affect • Short-term affect: Emotions (OCC, PAD)
• Medium-term affect: Mood (PAD)
• Long-term affect: Personality (Big Five)
• Map OCC emotions to PAD space
• Big Five personality traits as default PAD mood
• Example for a mood change • Slightly anxious mood
• Several fear emotion events
• Mood changes to fully anxious
Student Sven and teacher Valerie
46/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s ALMA – A Layered Model of Affect
• XML definitions for each character for • Five personality traits,
• Appraisal rules
• Modeled affect used to • Select wording and phrasing
• Select dialog strategies
• Trigger idle gestures
• Change the characteristics of conversational gestures
• Control of facial expressions
47/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Affect Expressive Robots
• Kismet (Breazeal, MIT, 1990s) • Anthropomorphic robot head for face-to-face interaction with humans
• Sensors: Cameras, microphones (prosody)
• Affect expressive abilities to communicate likes/dislikes: Tone of voice, Facial expressions, Postures
• Cognitive architecture to model drives and emotions
48/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s
• Video!!!
49/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Kismet’s Cognitive Architecture
Breazeal (2003), “Emotion and sociable humanoid robots”
50/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Kismet’s Cognitive Architecture
• Visual and auditory stimuli sensed by the robot
• Filtered by feature extractors (e.g. color, motion, pitch, etc.)
• High-level perceptual system (binds features by releaser processes representing current state of the robot)
• Affective appraisal phase: Active releasers are tagged with affective information • 4 types of appraisal
Intensity
Relevance
Intrinsic Pleasantness
Goal Directedness
→ Arousal (energy), valence (favorable), and stance (approachable)
51/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Kismet’s Cognitive Architecture
• Appraisals are filtered through the emotion elicitors for each emotion process
• Emotion arbitration phase: emotion processes compete for activation in a winner-take-all scheme
• Winner evokes a corresponding facial expression, body posture, and vocal quality (expressive motor system)
• May also evoke a corresponding behavioral response by the corresponding behavior in the behavior system
52/52
Co
gnit
ive
Mo
de
ling
– A
ffec
tive
Mo
del
s Affect Expressive Robots
• Albert (UCSD, 2009) • Video
• Hiroshi Ishiguro (Osaka, ATR) • Telepresence humanoid robots
http://www.youtube.com/watch?v=pkpWCu1k0ZI