dr. partha pratim roy - bdasp.iiita.ac.in · indian institute of technology roorkee applications of...

INDIAN INSTITUTE OF TECHNOLOGY ROORKEE

Applications of Machine Learning using Sensors

Dept. of Computer Science and Engineering

Indian Institute of Technology, Roorkee, India

Dr. Partha Pratim Roy

Email: proy.fcs@iitr.ac.in

2

Some Applications using Sensors

Sensor based Activity AnalysisGesture Recognition and Rendering Brain Computer Interface

3

Leap Motion and its working area

IR Cameras

IR Blasters

4

Sensor’s output

5

Applications

5

Touchless navigation

Sign language interface

3D Air printing

Augmented Reality

Physical Rehabilitation

Consumer electronic interface

Some Applications of Gesture Recognition

6

Recognition and Rendering of shapes on 2D display device

● Analysis of the motion of fingers in 3D space

● Recognition of natural gestures of 2D/3D geometric and non

geometric shapes using Leap Motion device.

● Rendering the shapes on 2D display device. MuPAD note

book was used for this purpose.

7

Sample of Single-finger Gestures

8

Sample of Multiple-finger Gestures

9

Rendering of Shapes

Natural gestures and their corresponding shape rendering

➢ Single-finger gesture showing a pyramid

➢ Multiple-finger gesture showing a sphere

10

Dataset Details

● Collected data for 36 distinct geometric and non-geometric shapes.● Collected data from 10 volunteer. Each volunteer plays each

gesture for 15 times.

11

Feature Extraction (Extended Npen++ features)

Gesture direction: At every time instant t gesture direction is defined as:

12

Feature extraction contd.

Aspect: The aspect of the trajectory characterizes the height-to-width ratio of the bounding box constituting the neighboring points of [x(t), y(t), z(t)] and is defined as:

13

Feature extraction contd.

● Curliness: Curliness feature as denoted by C(t) measures the deviation from a straight line in the vicinity of point P(t).

● Lineness: Lineness L(t) is defined as the average square of distance between every point in the cubical box of P and the straight-line joining the first and last point

14

Feature extraction contd.

Slope: In 3D, slope is defined as the direction ratios representedby l, m and n with respect to x, y and z dimensions of the straightline joining the start and endpoints within the bounding box.

Classification by Hidden Markov Model (HMM)

15

Achieved 92.87% accuracy when evaluated using 5-fold cross-validation method.

Variations in 3D shape rendering using parameters extracted from

gesture, e.g. “Cylinder”

16

17

Sign Language Recognition

• A sign language is a visual language that uses a system of

manual and non-manual signs as the means of

communication.

• SLR aims to develop methods and algorithms that correctly

identify a given sequence and able to convert it into

text/speech.

• Manual Signs- it involves finger-spelling (air-writing), hand

shapes, orientation and movements of the hands and fingers

during signing.

• Non-manual Signs- it involves various facial expressions,

head tilting, shoulder raising and mouthing that are added

with manual signs.

18

Continuous SLR

• It specify the recognition of continuous signs form the

sentences in the SLR.

• Recognition will take place as per the signs available in

vocabulary.

• A continuous SLR system has been proposed by modifying

the LSTM architecture.

A Mittal, P Kumar, PP Roy, B Raman, B. B. Chowdhuri, Continuous Sign Language Recognition using Leap motion, Pattern Recognition Letters,2018.

19

Continuous SLR

20

Manual Sign Implementation (air-writing)

1. Air-writing: It means writing in the 3D space without pen-paper.

Challenges:

a. No stroke information (No pen up-down information).

b. Continuous writing (words are attached with each other).

c. Large variation among same word written by the same writer.

d. No predefined plane or axis for writing.

e. Noisy signal or Jitter.

P Kumar, R Saini, PP Roy, DP Dogra. Study of text segmentation and recognition using leap motion sensor. IEEE Sensors Journal. 2016;17(5):1293-301.

21

Manual Sign Implementation (air-writing)

22

Air-Signature Authentication

"Analysis of 3D Signatures Recorded Using Leap Motion Sensor", Multimedia Tools and Applications, 2017.

26

Non-manual Sign Recognition

• An example where the manual-signs for the two different

words are same but have different facial expressions.

“Independent Bayesian Classier Combination based Sign Language Recognition using Facial Expression”, Information Sciences,2017.

27

Non-manual Sign Recognition

• Multimodal System set up

28

SLR Learning Interface

• A SLR learning interface has been developed using Blender

tool and NLP techniques.

• The interface works by entering the isolated or complete

sentence to get corresponding signs.

“3D Avatar based Sign Language Learning System”, ACM TALLIP, 2018. (Submitted)

29

Video Surveillance

30

Video Surveillance – Moving Object Detection

“Moving object detection using modified temporal differencing and local fuzzy thresholding”. The Journal of Supercomputing 73(3): 1120-1139 (2017)

31

Continuous Tracking

Ahmed et al., "Surveillance Scene Representation and Trajectory Anomaly Detection Using Aggregation of Multiple Concepts", Expert Systems with Applications, 2018.

32

Human Activity Analysis

A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)

33

Human Activity Analysis using Depth Sensors

34

Human Activity Analysis

"A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)

35

Human Interaction Monitoring System

"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).

36

Human Interaction Monitoring System

"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).

37

Electroencephalogram (EEG)

• The electroencephalogram (EEG) measures theactivity of large numbers (populations) of neurons.

• First recorded by Hans Berger in 1929.

• Electrodes measure voltage-differences at the scalpin the microvolt (μV) range.

• Voltage-traces are recorded with millisecondresolution – great advantage over brain imaging(fMRI, MEG).

38

Beyond Clinical Significance of EEG

EEG is used by researchers for the development of various BCI applications:

a. Neuro-Marketing

b. Emotion detection

c. Gaming

d. Security and authentication

e. ERP (Event Related Potential)

f. Wheel Chair control, mouse movement etc.

39

Methods for recording EEG

a. Invasive: recordings that are made with theelectrodes implanted within the depth of brainsurgically.

b. Non-Invasive: recordings that are obtainedfrom electrodes attached over the scalpsurface.

40

EEG Devices

41

EEG Acquisition

Standard placements of electrodes on the human scalp: A, auricle; C, central; F, frontal; Fp, frontal pole; O, occipital; P, parietal; T, temporal.

42

BCI Applications using EEG

43

Sample EEG recordings

44

EEG Characteristics

▪ EEG potentials are good indicators of brain state.

▪ They often display rhythmic patterns at characteristic frequencies

45

EEG Characteristics

• EEG rhythms correlate with patterns of behavior (level of attentiveness, sleeping,

waking, seizures, coma).

• Rhythms occur in distinct frequency ranges:

– Gamma: 20-60 Hz (“cognitive” frequency band)

– Beta: 14-20 Hz (activated cortex)

– Alpha: 8-13 Hz (quiet waking)

– Theta: 4-7 Hz (sleep stages)

– Delta: less than 4 Hz (sleep stages, especially “deep sleep”)

• Higher frequencies: active processing, relatively de-synchronized activity (alert

wakefulness, dream sleep).

• Lower frequencies: strongly synchronized activity (nondreaming sleep, coma).

46

Extraction of Gamma Features (32-100 Hz)

Barjinder Kaur, Dinesh Singh, and Partha Pratim Roy. "A Novel framework of EEG-based user identification

by analyzing music-listening behavior." Multimedia Tools and Applications: 2017

47

Features

• Root Mean Square (RMS)

• Standard Deviation (SD)

• Energy

• Mean (M)

48

Neurophone: Assistive Framework

Pradeep Kumar, Rajkumar Saini, Pawan Sahu, Partha Pratim Roy, Debi Prosad Dogra, Balasubramaninan

Raman, ‘Neuro-phone: An assistive framework to operate Smartphone using EEG signals’, IEEE Tensymp

2017.

49

Basic Applications

50

Results for Neurophone

51

Age and Gender Recognition by EEG

Barjinder Kaur, Partha Pratim Roy, Dinesh Singh. “Age and Gender Classification using BCI.“ Neural Computing & Applications, 2018.

52

Deep Learning Architecture

Results using Deep LSTM-BLSTM

54

Bio-signal framework to secure Mobile devices

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017

55

BCI Applications using EEG

56

Framework overview

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017

57

Attacking Scenario

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to

secure mobile devices, Journal of Network and Computer Applications, 2017

58

Customer choice prediction in Online Shopping

Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of

EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.

59

Architecture

Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of

EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.

60

Dataset Samples

Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of

EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.

61

Choice Prediction Results

• Best results have been recorded in Theta band waves.

62

Fusion of EEG Response and Sentiment Analysis of Products Review

63

The main objectives of the paper are as follows:

➢ A multimodal scheme is framed to improve the overall rating

performance of a product by fusing EEG signals with

sentiment score.

➢ The multimodal optimization is performed using ABC

optimization algorithm to maximize the overall system rating.

64

VADER Sentiment Analysis

➢ It is a lexicon of sentiment-related words and rule based

sentiment analysis

➢ It maps words to sentiment by building a lexicon or a

dictionary of sentiment.

➢ It produces three sentiment components.

Positive, Neutral, Negative.

➢ Other lexicon methods are TextBlob and NaiveBayes

Analyzer.

65

Pre-processing steps on the customer reviews

66

Comparative Analysis among VADER, TextBlob and NaiveBayes of customer’s reviews

67

Acknowledgment

INDIAN INSTITUTE OF TECHNOLOGY ROORKEE

Thank you

Dr. Partha Pratim Roy

Email: proy.fcs@iitr.ac.in