socially intelligent sensing - hayley hung - tu delft - behavior design ams
DESCRIPTION
Tenure Track Assistant Professor gave this presentation at the first Behavior Design AMS meetup on Thursday, September 19th in Amsterdam.TRANSCRIPT
![Page 1: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/1.jpg)
1Hayley Hung, TUDelft
Socially Intelligent Sensing
Hayley Hung
Pattern Recognition and Bioinformatics GroupIntelligent Systems Department
![Page 2: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/2.jpg)
2Hayley Hung, TUDelft
What is social behaviour?
![Page 3: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/3.jpg)
3Hayley Hung, TUDelft
What is Human Face-to-face interaction for ?
Makes our lives easier:Relationships TrustCo-operationPersuade/influence othersInformation sharing How could technology help
us to understand/interpret/socially relevant behaviour?How could this help to influence/enhance our experience?
![Page 4: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/4.jpg)
4Hayley Hung, TUDelft
Research Mission Statement
To develop algorithms that can model and understand non-verbal human social behaviour in real life situations. And through this, to understand how to build systems that can enhance people's quality of life by behaving with socially aware intelligence.
Develop algorithms that are perceptive to human social behaviour: Social Signal Processing, Machine LearningEnhancing people's quality of life: Human Machine Interaction, Ambient Intelligent Environments, Design, Architecture.Socially aware intelligence: Social and Behavioural Psychology, Ethnography.
![Page 5: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/5.jpg)
Hayley Hung, TUDelft
What can you say about this picture?
![Page 6: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/6.jpg)
Hayley Hung, TUDelft
What can you say about this picture?
Relaxed postureGestures
Vocal Behaviour
Mutual Gaze
Interpersonal Distance
![Page 7: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/7.jpg)
7Hayley Hung, TUDelft
Current Research Frontier
Person detection
Person tracking
Gaze detection
Body pose estimation
Group detection
Social and Behavioural Pscychology, Ethnography
Activity modelling
Action recognition
Attraction Estimation
Rapport Estimation
Role Recognition
Personality estimation
Dominance Estimation
![Page 8: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/8.jpg)
8Hayley Hung, TUDelft
Current Research Frontier
Relationship intimacy estimation
Conversation quality estimation
Person detection
Person tracking
Gaze detection
Body pose estimation
Group detection
Conversational event estimation
Personality estimation
Relationship quality estimation
Social and Behavioural Pscychology, Ethnography
Activity modelling
Action recognition
New Problem
Definitions
![Page 9: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/9.jpg)
9Hayley Hung, TUDelft
How to model social behaviour
Sensor Data Feature and Cue Extraction
Data AnnotationSocial BehaviourModelling and Classification
Model PerformanceEvaluation
![Page 10: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/10.jpg)
10Hayley Hung, TUDelft
Task 1: Estimating Attraction
Source: http://catinbag.blogspot.nl/2010/07/fatal-attraction.htmlVeenstra and Hung, “Do They Like Me? Using Video Cues to Predict Desires during Speed-dates” in ICCV Workshops 2011
![Page 11: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/11.jpg)
11Hayley Hung, TUDelft
Speed Dating, Non-verbal cues and Attraction
Can proximity-related video cues be used to automatically predict attraction in speed-dates?
![Page 12: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/12.jpg)
12Hayley Hung, TUDelft
Automated Position Extraction
- =
![Page 13: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/13.jpg)
13Hayley Hung, TUDelft
Automated Position Extraction
![Page 14: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/14.jpg)
14Hayley Hung, TUDelft
A Sped up Speed Date:
![Page 15: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/15.jpg)
15Hayley Hung, TUDelft
Speed Dating Results
Predicting attractionVariance in position is best feature predictor for women (70%).Variance in position of the women and synchrony both perform well (70%) for men.
Fusion of all synchrony features
Fusion of all movement features
![Page 16: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/16.jpg)
16Hayley Hung, TUDelft
Speed Date Experiments : Conclusion
The video channel can indeed be a source of valuable information in speed-dates Results differ per gender:
Movement synchrony information is more important for males than females.For females, information on the movement of their male counterpart gives good results
![Page 17: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/17.jpg)
17Hayley Hung, TUDelft
Task 2: Classifiying Social Actions using a Single Wearable Accelerometer
Hung, Englebienne, Kools, “Estimating Social Actions”, Ubicomp 2013
![Page 18: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/18.jpg)
18Hayley Hung, TUDelft
Modelling Human Social Behaviour in Dense Crowds
How can we model instantaneous social behaviour in extremely large crowds?
![Page 19: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/19.jpg)
19Hayley Hung, TUDelft
Our Goal
To develop methods to automatically measure socially relevant behaviour and moods in dense crowds using just a single tri-axial accelerometer
First step: detect socially relevant actions
Speaking; Laughing; Gesturing; Stepping; Drinking
![Page 20: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/20.jpg)
20Hayley Hung, TUDelft
Our Goal
Use insights from Social Psychology:
Speakers move more than listeners (McNeill 2000)
Laughter and joking correlated with sudden bursts of motion (Kendon 1990)
Synchronised motion during conversation (Kendon 1990,Chartrand and Bargh 1999)
![Page 21: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/21.jpg)
21Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording area
![Page 22: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/22.jpg)
22Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording area
![Page 23: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/23.jpg)
23Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)Experiment Stages: Briefing; Meeting and Mingling; Team formation (groups of 4); Quiz; Award GivingPrizes for top 3 teams5mx6m recording areaEach participant wore a sensing badge
![Page 24: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/24.jpg)
24Hayley Hung, TUDelft
Classifying social actions
Social Actions: speaking, laughing, gesturing, drinking, or stepping
(%) Gesture Step Drink Laugh SpeechPrecision 59 100 100 100 64Recall 24 21 21 38 82F-measure
34 35 35 56 72
![Page 25: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/25.jpg)
25Hayley Hung, TUDelft
Social Action Conclusion
It is possible to detect socially relevant behaviour.Can we detect when people are in the same conversational group?Could we even detect
personality traits quality of people's interaction?Quality of people's relationships?...etc
![Page 26: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/26.jpg)
26Hayley Hung, TUDelft
Looking to the Future...
The way we behave socially can exhibit strong detectable patterns, which are robust to noise.
How simple can the extracted features be?How could socially aware systems benefit better design?
![Page 27: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/27.jpg)
27Hayley Hung, TUDelft
Potential Applications: Human Robot Interaction
![Page 28: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/28.jpg)
28Hayley Hung, TUDelft
Applications: Urban Planning
Behaviour in Public SpacesHow can we measure statistically generaliseable changes as a result of interventions?
![Page 29: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/29.jpg)
29Hayley Hung, TUDelft
Potential Applications: Organisational behaviour
![Page 30: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/30.jpg)
30Hayley Hung, TUDelft
Acknowledgements/Collaborators
Arno Veenstra
Gwenn Englebienne Jeroen Kools
Ben KroseMaarten van Steen
Matt Dobson Claudio Martella
Domenic Vossen
![Page 31: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/31.jpg)
31Hayley Hung, TUDelft
Workshop at ACM Multmedia (acmmm13.org/)Barcelona , October 22
Human Behaviour Understanding for the Interactions in Arts, Creativity, Entertainment and Edutainment
Albert Ali Salah (Bogazici University, Turkey)Oya Aran (Idiap Research Institute, Switzerland)Hatice Gunes (Queen Mary University of London, UK)
![Page 32: Socially intelligent sensing - Hayley Hung - TU Delft - Behavior Design AMS](https://reader034.vdocuments.site/reader034/viewer/2022042815/55626dfcd8b42ae87d8b56f3/html5/thumbnails/32.jpg)
32Hayley Hung, TUDelft
Data:Sensors
Each participant wore:indoor positioning deviceProximity and Acceleration sensor
12 participants wore wireless microphone (for annotation)3 fish eye cameras (for annotation)3 accelerometer readings failed (software bug)