emotive media - visualization and analysis of human bio-feedback data
TRANSCRIPT
Emotive MediaVisualisation and Analysis of Human Bio-Feedback Data
26/08/2016 @ Curtin HIVE, Perth, Australia
A/Prof. Artur Lugmayr
www.artur-lugmayr.com
TriggerPeak of
ExpectationsDisillusion Enlightenment
Plateau of Productivity
Speech Recognition
Artificial Intelligence
Deep Learning
Open AI
Gesture Recognition
Automated Narratives
Traditional Media
IoT
Affective Computation
Smart Home
Smart Accessories
Brain Interfaces
Sensor Interfaces
Biosensors
Big Data NOW2-5 YEARS5-10 YEARS> 10 YEARS
Technologies –Between Hype and Maturity
QoE (Quality of Experience)
Intelligent Machines
Practical Applications
• Health care and wellness
• eLearning
• Robotics and digital pets
• Identification and safety
• Sentiment and text analysis
• Small gadgets and wearables
• Sports analysis
• User-experience research
• Quality of Experience (QoE)
• Analysis of archives and media
• Advertisement analytics
• Game behaviour and analytics
• Organizational management
• Intelligent machines
• …
Affective Computation
Neurosciences
Media Studies and Communication
Theories
User Experience, and Human-Computer-Interaction
Psychology of ‘The Human Mind’
Computational Models
Involved Research Fields
http://muvis.cs.tut.fi/index.html
Previously…. the conveyer belt for data processing
In the Future… we need a ‘cognitive’ conveyer belt for knowledge and wisdom
· Situation
· Object
· Collaboration
· Behaviour
· Perception
· Attention
· Arousal
· Emotion
· Experience
· Feelings
· Cognition
MindCommunication
Cause
· Language
· Personality
· Aesthetics
· Emotional Binding
· Artificial Intelligence
· Motives and BelievesConcepts
The Key in Emotional Media are Higher Level Concepts, as e.g. Language, Aesthetics, and Personality
Lugmayr, Artur, Tillmann Dorsch and Pabo Roman Humanes. "Emotional Ambient Media." Handbook of Research on Synthetic Emotions and Sociable Robotics: New Applications in Affective Computing and Artificial Intelligence. IGI Global, 2009. 443-459. Web. 27 Aug. 2016. doi:10.4018/978-1-60566-354-8.ch022
EmotionalState
Signal Level(Signals)
Concept Level(Concepts)
Symbol Level(Symbols)
Pattern Level(Patterns)
ActionMind
Reaction
Level
CAPTURE
SIMULATE
CONCEPT
RECOGNIZE
LEARN
COMMUNICATE
Emotional Media are Dual – Media that Understand Emotions, and Media that Simulate Emotions
“getting computers to ‘think’ like humans, [and] AI’s next natural step seems to be getting computers to ‘feel’ like humans”
J. Krikke and B. Alfonsi, "In the News," Intelligent Systems, IEEE, vol. 21, pp. 102-104, c3, 2006
• Hayley Hung, Hatice Gunes, Introduction to Social Signal Processing, ACM Multimedia 2015: Emotional and Social Signals Tutorial• New Social Signals in a New Interaction World: The Next Frontier for Social Signal Processing, A Vinciarelli, AS Pentland, Systems, Man, and
Cybernetics Magazine, IEEE 1(2), 2015
“A Social Signal is a communicative or informative signal that, either directly or indirectly provides information concerning social interactions, social emotions, social attitudes or social relations”
• http://www.theaustralian.com.au/news/deputy-opposition-leader-julie-bishop-says-peter-slipper-cannot-remain-as-speaker/story-e6frg6n6-1226491792937
• http://www.abc.net.au/pm/content/2015/s4331627.htm
NOTE! Due to eventual copyright reasons, I removed 2 images of two Australian Politicians. You want to know what was here? Click on the links below…
Semantics & Meaning (Stock Exchange Sentiment)
Macroeconomic IndicatorsInterest rateInflationDomestic creditIndustrial productionUnemployment rateConsumer sentimentConsumer pricesGrowth durable goodsGDP….
Implicit Sentiment IndicatorsCall/Put ratioOpen interestIPOsIPO first day returnsMutual fund flows…
Explicit Sentiment IndicatorsBoerse Frankfurt sentiment indexEconomic sentiment index (ESI)ZEW IndicatorIfo Business Climate Index…
Textual Sentiment Sources SourcesNewslettersRSS FeedsNews…
Sentiment IndicatorSentiment indexTrend and trend strength(Resistance/Support)(Hot news)
Textual Sentiment
Analysis
Text Mining
Sentiment AggregationSentiment
Model
Technical IndicatorsVolatilityVolumeTurnoverRSI, CCI, ARMS, …….
Lugmayr and G. Gossen, “Evaluation of methods and techniques for language based sentiment analysis for DAX30 stock exchange - a first concept of a"LUGO" sentiment indicator,” in Proceedings of the 5th International Workshop on Semantic Ambient Media Experience (SAME) - in Conjunction withPervasive 2012, A. Lugmayr, T. Risse, B. Stockleben, J. Kaario, B. Pogorelc, and E. S. Asensio, Eds. Newcastle, UK: Tampere University of Technology (TUT),2012.
Techniques in Social Signal Recognition
• Verbal communication
• Non-verbal communication• Gestures, facial expressions, postures, gaze, pupils, vocal behaviour, speech
activity, personal space, seating arrangements, …
• Media archive analytics• Multimedia content analysis, low-level features, semantic relations, …
• Social media analytics
• Media achieve analytics
• Hayley Hung, Hatice Gunes, Introduction to Social Signal Processing, ACM Multimedia 2015: Emotional and Social Signals Tutorial
Bio-Feedback Visualization
https://processing.org/Examples: https://processing.org/examples/ https://www.youtube.com/watch?v=SWCkEKrgaXk
NOTE! Due to eventual copyright reasons, I removed 2 images of the example. You want to know what was here? Click on the links below…
Emotional Interaction and Interfaces
Lugmayr, A., Dorsch, T. & Humanes, P.R., 2009. Emotional Ambient Media. In
Handbook of Research on Synthetic Emotions and Sociable Robotics: New
Applications in Affective Computing and Artificial Intelligence. IGI Global.
Empathy Glasses (CHI 2016)
• Masai, Katsutoshi et al. “Empathy Glasses.” CHI (2016)., https://www.semanticscholar.org/paper/Empathy-Glasses-Masai-Kunze/8cc9f1a26ef16490d4688bb9928fe91e3f71d02f
NOTE! Due to eventual copyright reasons, I removed the images of the example. You want to know what was here? Click on the links below…
Personality
Personality
Personality
Personality
Personality
Personality
Personality
Personality
PersonalityCollaborative
Space
Personality ComputationBits of personality everywhere (Portable Personality – P2), similar to Microsoft’s Cortana and Apple’s Siri
NOTE! Due to eventual copyright reasons, I removed 2 images of the example. You want to know what was here? Click on the links below…
P2 is similar to the film idea in ‘The Golden Compass’. Check it out: https://en.wikipedia.org/wiki/The_Golden_Compass_(film)
S. Reymann, V. Bruns, and A. Lugmayr, "P2 - Portable Personality a Middleware Solution for SmartUser Profile Management and Distribution," in Interactive TV: A Shared Experience, TISCP AdjunctProceedings of EuroITV 2007, Amsterdam, 2007
S. Uhlmann and A. Lugmayr. Media in the Ubiquitous Era: Ambient, Social and Gaming Media, chapterPortable Personality and its Personalization Algorithms: An Overview and Directions, pages 66–93. IGIGlobal, Hershey, USA, 2011. ISBN 9781609607746.
P2 – Portable Personality www.portablepersonality.org
Provider
P2 Service P2 ServiceP2 Mobile Service
Consumer
Service
S. Reymann, V. Bruns, and A. Lugmayr, "P2 - Portable Personality a Middleware Solution for SmartUser Profile Management and Distribution," in Interactive TV: A Shared Experience, TISCP AdjunctProceedings of EuroITV 2007, Amsterdam, 2007
S. Uhlmann and A. Lugmayr. Media in the Ubiquitous Era: Ambient, Social and Gaming Media, chapterPortable Personality and its Personalization Algorithms: An Overview and Directions, pages 66–93. IGIGlobal, Hershey, USA, 2011. ISBN 9781609607746.
Free UX Testing Tool: The LudoVico UX Machine for Physiological Sensor Data Recording, Analysis, and Visualization for User Experience Design Experiments, SEACHI Workshop, SIGCHI, 2016
The Ludovico UX MachineWWW.UX-MACHINE.COM (freely available sometimes in 2016)
The LudoVico UX Machine
Medical Sensor Data: Collection and interpretation of medical sensor data, such as EMC, EMG, breath, pulse, body temperature, body position, and galvinistic skin response.
Smart Environment Sensor Data: data coming from smart city environments, mashup data, location based data, mobile phone data.
Body Motion Sensor Data: Collection and interpretation of body motion sensor data through multiple sensor attached to a human body.
Extendible Platform: A plug-in concept and a light-weight sensor data protocol allows high extendibility of the platform, add new sensor types, additional features, new analysis techniques, and new functionality.
Freely Available: The LudoViko UX Tool is freely available for download: http://www.artur-lugmayr.com starting from mid-2016. Subscribe to our newsletter to get more information.
The Key is Visualisation
Data AnalysisRe-
presentationPresentation
Interaction
Cognition &
Perception
Communication
Context
VA
LE
NC
E
AROUSALLOW
(passive)HIGH
(active)
NEGATIVE
POSITIVE
… and developing user-interfaces that people understand in particular domains….
…a film-maker might not understand complex time-series data in form of diagrams, but a user-interface that he is comfortable with as e.g. Adobe Premiere integrating an ‘emotional monitor’…
Marshall McLuhan• Electronic culture as follow up culture for 400 years of print culture
• Media extend human senses: sight, hearing, touch, and smell
• Media technologies change society and culture: extensions of our bodies change mind & society affecting our mental functions • generating ideas, perception of the world, emotions, experiences
• The effect of McLuhan’s thesis:• Technology DETERMINES social outcome• Technology enables man extended and a live model outside the mind• Cybernetics
• Examples: light bulb, hammer, 3D cinema, ambient media ...
https://en.wikipedia.org/wiki/Marshall_McLuhan
1964 Understanding Media: The Extensions of Man; 1st ed. McGraw Hill, NY; reissued by MIT Press, 1994, with introduction by Lewis H. Lapham; reissued by Gingko Press, 2003. ISBN 1-58423-073-8.
