dr. partha pratim roy - bdasp.iiita.ac.in · indian institute of technology roorkee applications of...
TRANSCRIPT
INDIAN INSTITUTE OF TECHNOLOGY ROORKEE
Applications of Machine Learning using Sensors
Dept. of Computer Science and Engineering
Indian Institute of Technology, Roorkee, India
Dr. Partha Pratim Roy
Email: [email protected]
2
Some Applications using Sensors
Sensor based Activity AnalysisGesture Recognition and Rendering Brain Computer Interface
3
Leap Motion and its working area
IR Cameras
IR Blasters
4
Sensor’s output
5
Applications
5
Touchless navigation
Sign language interface
3D Air printing
Augmented Reality
Physical Rehabilitation
Consumer electronic interface
Some Applications of Gesture Recognition
6
Recognition and Rendering of shapes on 2D display device
● Analysis of the motion of fingers in 3D space
● Recognition of natural gestures of 2D/3D geometric and non
geometric shapes using Leap Motion device.
● Rendering the shapes on 2D display device. MuPAD note
book was used for this purpose.
7
Sample of Single-finger Gestures
8
Sample of Multiple-finger Gestures
9
Rendering of Shapes
Natural gestures and their corresponding shape rendering
➢ Single-finger gesture showing a pyramid
➢ Multiple-finger gesture showing a sphere
10
Dataset Details
● Collected data for 36 distinct geometric and non-geometric shapes.● Collected data from 10 volunteer. Each volunteer plays each
gesture for 15 times.
11
Feature Extraction (Extended Npen++ features)
Gesture direction: At every time instant t gesture direction is defined as:
12
Feature extraction contd.
Aspect: The aspect of the trajectory characterizes the height-to-width ratio of the bounding box constituting the neighboring points of [x(t), y(t), z(t)] and is defined as:
13
Feature extraction contd.
● Curliness: Curliness feature as denoted by C(t) measures the deviation from a straight line in the vicinity of point P(t).
● Lineness: Lineness L(t) is defined as the average square of distance between every point in the cubical box of P and the straight-line joining the first and last point
14
Feature extraction contd.
Slope: In 3D, slope is defined as the direction ratios representedby l, m and n with respect to x, y and z dimensions of the straightline joining the start and endpoints within the bounding box.
Classification by Hidden Markov Model (HMM)
15
Achieved 92.87% accuracy when evaluated using 5-fold cross-validation method.
Variations in 3D shape rendering using parameters extracted from
gesture, e.g. “Cylinder”
16
17
Sign Language Recognition
• A sign language is a visual language that uses a system of
manual and non-manual signs as the means of
communication.
• SLR aims to develop methods and algorithms that correctly
identify a given sequence and able to convert it into
text/speech.
• Manual Signs- it involves finger-spelling (air-writing), hand
shapes, orientation and movements of the hands and fingers
during signing.
• Non-manual Signs- it involves various facial expressions,
head tilting, shoulder raising and mouthing that are added
with manual signs.
18
Continuous SLR
• It specify the recognition of continuous signs form the
sentences in the SLR.
• Recognition will take place as per the signs available in
vocabulary.
• A continuous SLR system has been proposed by modifying
the LSTM architecture.
A Mittal, P Kumar, PP Roy, B Raman, B. B. Chowdhuri, Continuous Sign Language Recognition using Leap motion, Pattern Recognition Letters,2018.
19
Continuous SLR
20
Manual Sign Implementation (air-writing)
1. Air-writing: It means writing in the 3D space without pen-paper.
Challenges:
a. No stroke information (No pen up-down information).
b. Continuous writing (words are attached with each other).
c. Large variation among same word written by the same writer.
d. No predefined plane or axis for writing.
e. Noisy signal or Jitter.
P Kumar, R Saini, PP Roy, DP Dogra. Study of text segmentation and recognition using leap motion sensor. IEEE Sensors Journal. 2016;17(5):1293-301.
21
Manual Sign Implementation (air-writing)
22
Air-Signature Authentication
"Analysis of 3D Signatures Recorded Using Leap Motion Sensor", Multimedia Tools and Applications, 2017.
26
Non-manual Sign Recognition
• An example where the manual-signs for the two different
words are same but have different facial expressions.
“Independent Bayesian Classier Combination based Sign Language Recognition using Facial Expression”, Information Sciences,2017.
27
Non-manual Sign Recognition
• Multimodal System set up
28
SLR Learning Interface
• A SLR learning interface has been developed using Blender
tool and NLP techniques.
• The interface works by entering the isolated or complete
sentence to get corresponding signs.
“3D Avatar based Sign Language Learning System”, ACM TALLIP, 2018. (Submitted)
29
Video Surveillance
30
Video Surveillance – Moving Object Detection
“Moving object detection using modified temporal differencing and local fuzzy thresholding”. The Journal of Supercomputing 73(3): 1120-1139 (2017)
31
Continuous Tracking
Ahmed et al., "Surveillance Scene Representation and Trajectory Anomaly Detection Using Aggregation of Multiple Concepts", Expert Systems with Applications, 2018.
32
Human Activity Analysis
A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)
33
Human Activity Analysis using Depth Sensors
34
Human Activity Analysis
"A novel framework of continuous human-activity recognition using Kinect. Neurocomputing311: 99-111 (2018)
35
Human Interaction Monitoring System
"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).
36
Human Interaction Monitoring System
"Kinect sensor-based interaction monitoring system using the BLSTM neural network in healthcare”. International Journal of Machine Learning and Cybernetics (2018).
37
Electroencephalogram (EEG)
• The electroencephalogram (EEG) measures theactivity of large numbers (populations) of neurons.
• First recorded by Hans Berger in 1929.
• Electrodes measure voltage-differences at the scalpin the microvolt (μV) range.
• Voltage-traces are recorded with millisecondresolution – great advantage over brain imaging(fMRI, MEG).
38
Beyond Clinical Significance of EEG
EEG is used by researchers for the development of various BCI applications:
a. Neuro-Marketing
b. Emotion detection
c. Gaming
d. Security and authentication
e. ERP (Event Related Potential)
f. Wheel Chair control, mouse movement etc.
39
Methods for recording EEG
a. Invasive: recordings that are made with theelectrodes implanted within the depth of brainsurgically.
b. Non-Invasive: recordings that are obtainedfrom electrodes attached over the scalpsurface.
40
EEG Devices
41
EEG Acquisition
Standard placements of electrodes on the human scalp: A, auricle; C, central; F, frontal; Fp, frontal pole; O, occipital; P, parietal; T, temporal.
42
BCI Applications using EEG
43
Sample EEG recordings
44
EEG Characteristics
▪ EEG potentials are good indicators of brain state.
▪ They often display rhythmic patterns at characteristic frequencies
45
EEG Characteristics
• EEG rhythms correlate with patterns of behavior (level of attentiveness, sleeping,
waking, seizures, coma).
• Rhythms occur in distinct frequency ranges:
– Gamma: 20-60 Hz (“cognitive” frequency band)
– Beta: 14-20 Hz (activated cortex)
– Alpha: 8-13 Hz (quiet waking)
– Theta: 4-7 Hz (sleep stages)
– Delta: less than 4 Hz (sleep stages, especially “deep sleep”)
• Higher frequencies: active processing, relatively de-synchronized activity (alert
wakefulness, dream sleep).
• Lower frequencies: strongly synchronized activity (nondreaming sleep, coma).
46
Extraction of Gamma Features (32-100 Hz)
Barjinder Kaur, Dinesh Singh, and Partha Pratim Roy. "A Novel framework of EEG-based user identification
by analyzing music-listening behavior." Multimedia Tools and Applications: 2017
47
Features
• Root Mean Square (RMS)
• Standard Deviation (SD)
• Energy
• Mean (M)
48
Neurophone: Assistive Framework
Pradeep Kumar, Rajkumar Saini, Pawan Sahu, Partha Pratim Roy, Debi Prosad Dogra, Balasubramaninan
Raman, ‘Neuro-phone: An assistive framework to operate Smartphone using EEG signals’, IEEE Tensymp
2017.
49
Basic Applications
50
Results for Neurophone
51
Age and Gender Recognition by EEG
Barjinder Kaur, Partha Pratim Roy, Dinesh Singh. “Age and Gender Classification using BCI.“ Neural Computing & Applications, 2018.
52
Deep Learning Architecture
Results using Deep LSTM-BLSTM
54
Bio-signal framework to secure Mobile devices
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017
55
BCI Applications using EEG
56
Framework overview
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to secure mobiledevices, Journal of Network and Computer Applications, 2017
57
Attacking Scenario
Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, A bio-signal based framework to
secure mobile devices, Journal of Network and Computer Applications, 2017
58
Customer choice prediction in Online Shopping
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
59
Architecture
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
60
Dataset Samples
Mahendra Yadava, Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, Debi Prosad Dogra, Analysis of
EEG signals and its application to neuromarketing, Multimedia Tools and Applications, 2017.
61
Choice Prediction Results
• Best results have been recorded in Theta band waves.
62
Fusion of EEG Response and Sentiment Analysis of Products Review
63
The main objectives of the paper are as follows:
➢ A multimodal scheme is framed to improve the overall rating
performance of a product by fusing EEG signals with
sentiment score.
➢ The multimodal optimization is performed using ABC
optimization algorithm to maximize the overall system rating.
64
VADER Sentiment Analysis
➢ It is a lexicon of sentiment-related words and rule based
sentiment analysis
➢ It maps words to sentiment by building a lexicon or a
dictionary of sentiment.
➢ It produces three sentiment components.
Positive, Neutral, Negative.
➢ Other lexicon methods are TextBlob and NaiveBayes
Analyzer.
65
Pre-processing steps on the customer reviews
66
Comparative Analysis among VADER, TextBlob and NaiveBayes of customer’s reviews
67
Acknowledgment
