mining minds presenter12-july-2014khu information curation layer low level context-awareness

Mining Minds

Presenter

12-July-2014

KHU

Information Curation LayerLow Level Context-awareness

/Information Curation Layer 2

/Low Level Context-awareness

• Introduction• Motivation• Related Works• Architecture• Tools and Technologies• Development Timeline• Current Status

3

/Introduction 4

Physical Activities

Emotional States

Social Interaction

• Heterogeneous information source around people• Daily physical activities• Social interactions• Psychological states

• Automatically collect and provide basic information to the system

/Introduction 5

Physical Activities

Emotional States

Social Interaction

Knowledge

Information

Data

• Extract information directly from raw data

• Play an important role to get useful knowledge from people• Daily habit• Behavior

/Motivation 6

• Collect, process different kind of data• Sensory data• Social data

• Analyze and extract useful information at low-level• Physical activities• Social activities• Emotional states

• Provide the interface to interact with higher level and database manager

/Related Works 7

• Activity Recognizer• [WiiRemote] The Wii Remote™ Plus controller is the heart of the

motion gaming experience on your Wii console.• [Han2012] tried to overcome the limitation of accelerometer based

activity recognition. Accelerometer is used to recognize walking, running, and stay, and audio, GPS and wifi are used to recognize bus and subway.

• There lots of segmentation works such as graph-cut based segmentation by [Pourjam2013], and mean-shift algorithm by [Atefian2013] have been proposed for human body segmentation.

/Low Level Context-awareness 8

• Emotion Recognizer• [MITMindReader] MIT’s Mind Reader software can scan faces in a

crowd to determine audience mood, a tool that may replace opinion polls and help public speakers tailor their words for maximum impact.

• Various types of classifiers have been used for the task of speech emotion recognition such as HMM, GMM, SVM, etc. [Ayadi2011].

• Several emotion research works tried to separate the original complex multiple emotion classification problem by applying hierarchical approach with combination of different classifiers [Lee2011].

/Architecture 9

High Level Context-awareness

HDFS Data Access Interface

Low Level Context-awareness

Raw Data

Personal Information

SNSInteractio

nAnalyzer

ActivityRecognize

r

EmotionRecognize

r

/Architecture 10




Social Data(Twitter)

Attribute The-

saurus

Thesaurus Manager

System Data

Morpheme Manager

Analysis

Syntax Analyzer

Morpheme Analyzer

Attribute Extractor

Emotion Extractor

Attribute-Emotion Mapping Module

Positive, Negative Analyzer

Compilation

DMBS Connector

ArchiveListener

Remote Control Request Module

Twitter Analyzer

Sentiment

Activity Recognizer

Wearable Sensor based AR

Data Acquisition

FeatureExtraction

Training Models

Classifying

Smartphone based AR

PreprocessingFeature

Extraction

GPS Validation

Decision Maker

Video based AR

Data Acquisition

SegmentationFeature

Extraction

Classifying

Emotion Recognizer

Audio based ER

Preprocessing Classification Tree Construction

FeatureExtraction

Classifying

Video based ER

Face Detection HMM Training

FeatureExtraction

HMM Testing

Physiological sensor based ER

Statistical Feature Extraction

Non-Param Cumulative Sum Auto Associate

Neural Network

Deci

sion

Fusi

on

Synch

roniz

ati

on

Pro

babili

ty C

om

pu

tati

on

Sensory Data(Acc, GPS, Video)

Sensory Data(Heart rate, Video,

Audio)

Personal Information(behavior, interest)

Activity Label(standing, sitting, running,

…)

Emotion Label(happy, angry, boredom,

…)

/Architecture 11





Attribute The-

saurus

Thesaurus Manager

System Data

Morpheme Manager

Analysis

Syntax Analyzer

Morpheme Analyzer

Attribute Extractor

Emotion Extractor



Compilation

DMBS Connector

ArchiveListener


Twitter Analyzer

Sentiment

Activity Recognizer


Data Acquisition

FeatureExtraction

Training Models

Classifying

Smartphone based AR


Extraction

GPS Validation

Decision Maker

Video based AR

Data Acquisition

SegmentationFeature

Extraction

Classifying

Emotion Recognizer

Audio based ER


FeatureExtraction

Classifying

Video based ER


FeatureExtraction

HMM Testing




Neural Network

Deci

sion

Fusi

on

Synch

roniz

ati

on

Pro

babili

ty C

om

pu

tati

on



Audio)



…)


…)

• SNS Analyzer - Twitter• Take input from Twitter

API in schema format• Analyze Twitter data in

different contexts• Activity

• Emotion

• Behavior

• Provide the output based on keyword

/Architecture 12





Attribute The-

saurus

Thesaurus Manager

System Data

Morpheme Manager

Analysis

Syntax Analyzer

Morpheme Analyzer

Attribute Extractor

Emotion Extractor



Compilation

DMBS Connector

ArchiveListener


Twitter Analyzer

Sentiment

Activity Recognizer


Data Acquisition

FeatureExtraction

Training Models

Classifying

Smartphone based AR


Extraction

GPS Validation

Decision Maker

Video based AR

Data Acquisition

SegmentationFeature

Extraction

Classifying

Emotion Recognizer

Audio based ER


FeatureExtraction

Classifying

Video based ER


FeatureExtraction

HMM Testing




Neural Network

Deci

sion

Fusi

on

Synch

roniz

ati

on

Pro

babili

ty C

om

pu

tati

on



Audio)



…)


…)

• Activity Recognizer• Take input from

different sensors• Wearable sensors

• Smartphone’s sensors

• Video sensors

• Recognize activities based on specific machine learning algorithms for each kind of data

• Provide output as activity label in text format

/Architecture 13





Attribute The-

saurus

Thesaurus Manager

System Data

Morpheme Manager

Analysis

Syntax Analyzer

Morpheme Analyzer

Attribute Extractor

Emotion Extractor



Compilation

DMBS Connector

ArchiveListener


Twitter Analyzer

Sentiment

Activity Recognizer


Data Acquisition

FeatureExtraction

Training Models

Classifying

Smartphone based AR


Extraction

GPS Validation

Decision Maker

Video based AR

Data Acquisition

SegmentationFeature

Extraction

Classifying

Emotion Recognizer

Audio based ER


FeatureExtraction

Classifying

Video based ER


FeatureExtraction

HMM Testing




Neural Network

Deci

sion

Fusi

on

Synch

roniz

ati

on

Pro

babili

ty C

om

pu

tati

on



Audio)



…)


…)

• Emotion Recognizer• Take input from different

sensors• Audio sensor

• Video sensors

• Physiological sensors

• Recognize emotions based on specific machine learning algorithms for each kind of data• Apply Fusion technique to

increase confident of predict output from different decisions.

• Provide output as emotion label in text format

/Tools and Technologies 14

• Tools for development• MATLAB• Eclipse• Android SDK

• Technologies• Machine Learning

• Platforms• Microsoft Windows• Android OS

/Development Timeline 15

1st year

2nd year

Interface Definition

Adapter Development

Component Modification

Component Validation

Evaluate Components based on collected data

Component Adjustment

Output

Modified Compo-nents

Validated Compo-nents

Evaluation Report

1st Integration Phase

2nd Evaluation Phase

Final module

Interface Report

/Current Status 16

• Social Interaction Analyzer• Need to define the input and output to interact with Tapacross’s

engine

• Activity and Emotion Recognizer• Each individual module is available and ready for integration

/References 17

[WiiRemote] https://www.nintendo.com/wii/what-is-wii/#/controls[Han2012] Manhyung Han, La The Vinh, Young-Koo Lee and Sungyoung Lee, "Comprehensive Context Recognizer Based on Multimodal Sensors in a Smartphone", Journal of Sensors, vol. 12, no. 9, pp. 12588-12605, 2012[Pourjam2013] Pourjam, E., Ide, I., Deguchi, D., & Murase, H. Segmentation of Human Instances Using Grab-cut and Active Shape Model Feedback. In proceddings of MVA2013 IAPR International Conference on Machine Vision Applications, pp. 77–80, May 20–23, 2013.[Atefian2013] Atefian, M., & Mahdavi-Nasab, H. (2013). A Robust Mean-Shift Tracking Using GMM Background Subtraction, J. Basic. Appl. Sci. Res., vol. 3, no. 4, 596–607, 2013.[MITMindReader] http://trac.media.mit.edu/mindreader/[Ayadi2011] Ayadi, M.E., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44 (3), 572 - 587 (2011).[Lee2011] C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan. Emotion recognition using a hierarchical binary decision tree approach. Speech Commun., 53(9-10):1162-1171, Nov. 2011.

QuestionsThank You!

[email protected]

•18

mining minds presenter12-july-2014khu information curation layer low level context-awareness

Documents

system slide

useful information

database manager slide

basic information

higher level

emotion research works

based segmentation

negative analyzer positive