1 Copyright © 2014 Tata Consultancy Services Limited

Fusing Personal Context with Physical and Physiological Context for creating value-added crowd-sensing applications

22nd May 2015

Arpan PalPrincipal Scientist, Innovation LabsTata Consultancy Services Ltd.

2

Pioneer & Leader in Indian IT

TCS was established in 1968

One of the top ranked global software service provider

Largest Software service provider in Asia

300,000+ associates

USD 15Billion+ annual revenue

Global presence – 55+ countries, 119 nationalities

First Software R&D Center in India

Tata Consultancy Services (TCS) at a Glance

Bangalore, India1

Chennai, India2

Cincinnati, USA3

Delhi, India4

Hyderabad, India5

Kolkata, India6

Mumbai, India7

Peterborough, UK8

Pune, India9

2000+ Associates in Research, Development and Asset Creation

Singapore10

Innovation @ TCS

TCS Connected Universe Platform (TCUP)• M2M Communication• Distributed Computing• Sensor Integration and Management• Analytics ServicesContext-aware Applications• Healthcare• Insurance• Retail• Manufacturing• Smart Building / Campus• Smart Villages / Cities

Overview

10 Corporate Innovation Labs

Co-innovation Network (COIN) with Academia and Industry

Internet-of-Things Research

Three stage Innovation Process – Explore, Enable. Exploit

3

Agenda

Context Discovery using IoT

Application Use Cases – Physical Context

Evacuation, Insurance, Retail

Physiological Sensing – Mobile and Wearable

HRV, BP, EEG, GSR

Behavioral Model from Physiological Sensing

4

The Internet of Everything

Humans

Physical Objects and Infrastructu

re

Computing Infrastructu

re

Peo

ple

Con

text

Dis

cove

ry

PhysicalContext Discovery

INTERNET OF EVERYTHING

Physical Context

DiscoveryWhat is happening,

where and when

People Context Discovery

Who is doing what, where and when, who is

thinking what

Internet of

Digital

Internet of

Things

Internet of

Humans

ABI Research. May 7, 2014

New Business / Pricing Models

5

Understanding the People Context

Non-intrusive, un-obtrusive sensing

Identity, Location, Activity, Physiology

Understand Behavior – Individuals / Groups

Quantified Self

Customer becomes the focus, not the product or service – key is understanding the Customer, Extend B2B to B2B2C

Using Wearable's and Nearables

(mobile phone, camera, mic, ….)

6

Context Discovery - Multi-dimensional Fusion

• Panic• Stress• Like / Dislike

• Weather• Environment

• Network• Likes and

Dislikes

• Location• Activity• Proximity

Physical Social Media

PhysiologySurroundings

Contextual Information

7

Click to edit Master title styleApplication Use Cases

8

Application Use Cases

• Floor plan based capacity planning

• Location based recommendation

• Behavioral Sensing – panic / proximity

Emergency Evacuation

• Hard Cornering / Braking / Harsh Acceleration from Accelerometer

• Driver Scoring• Road / Traffic /

Weather Condition

• Behavioral Sensing - Stress

Driving Behavior

• User profiling from usage / social media

• Location based Recommendation

• Environmental Effect

• Behavioral Sensing – Buying urge / group behavior

Consumer Behavior

Wearable sensing, nearable sensing and crowd sensing

9

Sensing Physical Context of People – Location and Activity

Indoor Localization – Bldg, Mall• Entry-Exit using RFID and

Magnetometer• Zoning using Wi-Fi• Fine-grained positioning using Inertial

Navigation

Activity Detection - Wellness• Walking / Brisk Walking / Jogging /

Running using Accelerometer Signature

• Orientation and Placement agnostic• Calorie Burnt using Activity based

models

Magnetometer – Entry/Exit

RFID Fusion

WiFi -Zoning Bluetooth -Proximity

98% 99.7% 97% 96%

(Accuracy ~2m)

(Accuracy ~ 98%)

Publicationso Nasimuddim Ahmed et. al., ""SmartEvacTrak: A People Counting and Coarse-Level Localization Solution for Efficient

Evacuation of Large Buildings“, CASPER'15 workshop of IEEE Percom 2013o Vivek Chandel et.al., "AcTrak - Unobtrusive Activity Detection and Step Counting using Smartphones“, Mobiquitous 2013

10

Sensing Physical Context of People – Traffic and Driving

Traffic Sensing – City Authority• Congestion Modeling from historical

location data crowd sensed from vehicles

• Honk Detection from crowd sensed audio data

• Road Condition Monitoring from crowd sensed Accelerometer data

Driving Behavior - Insurance• Hard Cornering / Breaking / Harsh

Acceleration from Accelerometer Analytics

Publicationso Maiti, Santa, et al. "Historical data based real time prediction of vehicle arrival time." ITSC 2014o Ghose, Avik et. al., "Road condition monitoring and alert application: Using in-vehicle smartphone as internet-connected

sensor.“, Percom Workshops 2012.o Tapas Chakravarthy et. al., “MobiDriveScore — A system for mobile sensor based driving analysis: A risk assessment

model for improving one's driving”, ICST 2013

11

Physiological Sensing – Mobile Phone and Wearable

12

Behavioral Sensing using Physiology

Referenceso Levenson, Robert W. "Blood, sweat, and fears." Annals of the New York Academy of Sciences 1000, no. 1 (2003):o Näätänen, R et.al., "A model for the role of motivational factors in drivers' decision-making." Accident Analysis &

Prevention 6, no. 3 (1974)o GW Evans, “Environmental stress”, 1984o Bechara, Antoine et. al., "Emotion, decision making and the orbitofrontal cortex." Cerebral cortex 10, no. 3 (2000):o Mauss, Iris B et, al., "The tie that binds? Coherence among emotion experience, behavior, and physiology." Emotion 5, no.

2 (2005): 175.

• Heart Rate Variability

• Blood Pressure

• EEG• GSR

13

Physiological Sensing – Heart Rate, BP and HRV

PPG Signal

Field Trials at TCS Office and Indian VillagesTie-up with HospitalsWearable variant pilot for Crane Operator Monitoring in Factories

14

rMSSD, DSD, SDNN, nn50, PNN50, nn20, pNN20

Physiological Sensing – Results

~2 bpm error in

Heart

Rate~92% Accuracy in blood

pressure

Publicationso Arpan Pal et. al., "A Robust Heart Rate Detection using Smart-phone Video", in MobileHealth workshop of Mobihoc

2013o Aishwarya Visvanathan et. al., "Smart Phone Based Blood Pressure Indicator", in MobileHealth workshop of Mobihoc

2014.o Anirban Duttachoudhury et.al., "Demo – Estimating Blood Pressure and ECG from Photoplethysmograph

using Smart Phones", SenSys 2014 – BEST DEMOo Banerjee, Rohan et al. "Noise Cleaning and Gaussian Modeling of Smart Phone Photoplethysmogram to

improve Blood Pressure Estimation“, ICASSP 2015o Nasim Ahmed et al. “Feasibility Analysis for Estimation of Blood Pressure and Heart Rate using A Smart

Eye Wear”, WearSys workshop in Mobisys 2015

~89% Accuracy in HRV -

SDNN

15

Psycho-Physiological Sensing – EEG and GSR

GSR

Mental tasksCognitive LoadVisual Attention(VA)Memory(M)Logic(L)Arithmetic(A)Other(O)

Emotion(E) Stress(S)

  

EE

G

arte

fact

re

mov

al

AP

I (V

A),

AP

I(M

),…

AP

I(S

)

Application

Fus

ion

 Fea

ture

E

xtra

ctio

n (

ind

ivid

ual t

ask

)

16

Psycho-Physiological Sensing – Results

~80% Accuracy, showing

hierarchical keyboard is easy to

use than QWERTY

~75% Accuracy

o “Evaluation of Different onscreen keyboard layouts using EEG signals”, SMC 2013o “EEG-Based Fuzzy Cognitive Load Classification”, FUZZ IEEE 2013o “Unsupervised Approach for Measurement of Cognitive Load using EEG Signals”, BIBE

2013

17

Physiological Sensing for Behavior Modeling

18

Behavioral Modeling using Physiology

Pietro Cipresso et. al., “Psychometric modeling of the pervasive use of Facebook through psychophysiological measures: Stress or optimal experience?”, Computers in Human Behavior , 49 (2015) 576–587, Elsevier

19

Behavioral Modeling using Physiology – Early Results

Tetris-like game designed for Bored and Flow State Stimulio Submitted: “Dynamic Assessment of Learners' Mental State for an

Improved Learning Experience “, Frontiers of Education 2015

Should be extendable to other use cases

20

Looking Ahead - Challenges

Need to take care of Battery Power Issue

Need to address Privacy Issue

Each sensor may be very accurate on its own – fusion is the key

Right feature selection for the given use case would be critical

Lack of multi-sensor Dataset needs to be addressd

Option• Do controlled experiments on diverse set of sample subjects using physiological

sensing and create simplified aggregate models • Use the Model in the field (e.g. - % of people who do not follow the evacuation

recommendation can help in creating a probabilistic model)• Would need Individual training or constant wearing of sensors for individual

models – Driving / Shopping Behavior

TCUP – the TCS IoT

platform can be used to

collect multi-sensor data in an

efficient way

21

More References

o Karel A. Brookhuis, Dick de Waard, Monitoring drivers’ mental workload in driving simulators using physiological measures, Accident Analysis & Prevention, Volume 42, Issue 3, May 2010, Pages 898-903, ISSN 0001-4575, http://dx.doi.org/10.1016/j.aap.2009.06.001.

o J.A. Healey and R.W. Picard, "Detecting Stress during Real-world Driving Task using Physiological Sensors", Intelligent Transportation System, IEEE Trans, , Vol. 6, No. 2, June (2005) 156-166.

o Jordan Smith, Neil Mansfield, Diane Gyi, Mark Pagett, Bob Bateman, Driving performance and driver discomfort in an elevated and standard driving position during a driving simulation, Applied Ergonomics, Volume 49, July 2015, Pages 25-33, ISSN 0003-6870, http://dx.doi.org/10.1016/j.apergo.2015.01.003.

o Gianluca Borghini, Laura Astolfi, Giovanni Vecchiato, Donatella Mattia, Fabio Babiloni, Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness, Neuroscience & Biobehavioral Reviews, Volume 44, July 2014, Pages 58-75, ISSN 0149-7634, http://dx.doi.org/10.1016/j.neubiorev.2012.10.003.

o David P. Wyon, Inger Wyon, Fredrik Norin, Effects of moderate heat stress on driver vigilance in a moving vehicle, Ergonomics, Vol. 39, Iss. 1, 1996.

o Markku Kilpeläinen, Heikki Summala, Effects of weather and weather forecasts on driver behaviour, Transportation Research Part F: Traffic

o Psychology and Behaviour, Volume 10, Issue 4, July 2007, Pages 288-299, ISSN 1369-8478, http://dx.doi.org/10.1016/j.trf.2006.11.002.

o Mauss, Iris B., Robert W. Levenson, Loren McCarter, Frank H. Wilhelm, and James J. Gross. "The tie that binds? Coherence among emotion experience, behavior, and physiology." Emotion 5, no. 2 (2005): 175.

o Levenson, Robert W. "Blood, sweat, and fears." Annals of the New York Academy of Sciences 1000, no. 1 (2003): 348-366.o Bechara, Antoine, Hanna Damasio, and Antonio R. Damasio. "Emotion, decision making and the orbitofrontal cortex."

Cerebral cortex 10, no. 3 (2000): 295-307.

Thank You

IT ServicesBusiness SolutionsConsulting

arpan.pal@tcs.com

With Avik Ghose, Aniruddha Sinha, Tanushyam Chattopadhyay, Arindam Pal, Debatri Chatterjee