ece 494 capstone design final design presentation smartphone based human behavioral analysis andrew...

ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron Stewart April 18, 2014 Advisor: Dr. Fei Hu

Outline 4/18/2014Smartphone Based Human Behavioral Analysis2 Project Recap & Goal System Diagram Subsystem Breakdown & Team Roles Sensor Data Extraction & Analysis App Development & Activity Recognition DTW Algorithm Environment Sensors NN Algorithm Raw Memory Extraction & Analysis HMM Algorithm Administration

Project Recap & Goal 4/18/2014Smartphone Based Human Behavioral Analysis3 Create a system for tracking and detecting a users behavioral patterns though the phones sensors and internal memory logs. Can be used in multiple areas: Healthcare Activity Monitoring Homeland Security

System Diagram 4/18/20144Smartphone Based Human Behavioral Analysis

Behavior Recognition Phone Data Sensor Data Extraction & Analysis Accelerometer Light Sound Proximity GPS Raw Memory Extraction & Analysis dc3dd Call Logs Text Messages Machine Learning Algorithms Hidden Markov Model Neural Networks Dynamic Time Warping Subsystem Breakdown & Team Roles 4/18/20145Smartphone Based Human Behavioral Analysis Andrew Jackson Android Sensor App DTW Algorithm Aaron Stewart Raw Memory Analysis Team Leader Robbie Rosati Machine Learning Algorithms Raw Memory Extraction Michael Armstrong Raw Memory Extraction HMM Algorithm Andy McWilliams HMM Algorithm Raw Memory Analysis

Sensor Data Extraction & Analysis Aaron Stewart

Android, iOS, or Windows Phone We chose to go with Android. No development license fees Majority of market Many sensors available Open file system Previous Android programming experience Cheaper phone prices 4/18/20147Smartphone Based Human Behavioral Analysis

Samsung Galaxy S4 Mini 4/18/2014Smartphone Based Human Behavioral Analysis8 New phone with many sensors Fairly inexpensive for an unlocked phone Smaller size great for testing in pockets

Accelerometer LightSoundProximityGPS 4/18/2014 Smartphone Based Human Behavioral Analysis 9 Sensors

Sensor Types 4/18/2014Smartphone Based Human Behavioral Analysis10 Accelerometer Measures acceleration with gravity value Reports data in 3 values, one for each axis Example: Could tell if user is walking or stationary GPS Reports GPS coordinates See changes in location Example: Could tell if user is driving

Sensor Types 4/18/2014Smartphone Based Human Behavioral Analysis11 Light Intensity of light Reports data in a single value Example: Indoors/Outdoors, Weather, Time of day Sound Detects sound waves Reports loudness Example: Voice recognition Proximity Changes depending on surrounding objects Boolean value Example: Device is in tight space such as pocket or purse

Behavior Analysis App Version 1.0 4/18/2014Smartphone Based Human Behavioral Analysis12 Shows values from each sensor in real-time as the sensor updates Can save the data to a file on phone Move file to computer to analyze with MATLAB Graph the data Validate that it makes sense

Testing with App 3/10/2014Smartphone Based Human Behavioral Analysis13 Most sensor data comes in a set of 3 points Few have single values Tests related to typical smartphone user behaviors Used MATLAB for data graphing and filtering

Light Sensor Test (Different Exposure) 4/18/2014Smartphone Based Human Behavioral Analysis14

Accelerometer Test (Answer Call) 4/18/2014Smartphone Based Human Behavioral Analysis15 XYZXYZ

App Development and Activity Recognition Andrew Jackson

4/18/2014 Smartphone Based Human Behavioral Analysis 17

Behavior Analysis App Version 2.0 4/18/2014 Smartphone Based Human Behavioral Analysis 18 Integrates DTW machine learning algorithm Allows training of the algorithm from the phone no computer required Once trained, will recognize where the phone is with almost 100% accuracy Visual Upgrades All features from previous app rolled over

Further App Improvements 4/18/2014Smartphone Based Human Behavioral Analysis19 Increased accuracy of activity recognition by Increasing sensor sensitivity Polling each sensor more times a second Tweaking DTW algorithm code Using multiple sensors at the same time Continued adding new activities Added speech recognition By demo day: Add in GPS coordinates Further explore light, sound, and proximity sensor possibilities

Accelerometer, Light, and Proximity 4/18/2014 Smartphone Based Human Behavioral Analysis 20 Chose to work with accelerometer first because it can give some of the most useful data about the phone Applying DTW on accelerometer data Next, applied light and proximity sensors to get better results

Recognized Activities Last Presentation Now 4/18/2014 Smartphone Based Human Behavioral Analysis 21 Walking (with phone in hand) Talking on phone Sitting on table Holding in hand Walking (with phone in pocket) Running Stairs Sitting Driving

4/18/2014 Smartphone Based Human Behavioral Analysis 22 App Demonstration

Dynamic Time Warping Andrew Jackson

Dynamic Time Warping Measures similarity between two sequences which may vary in time or speed Calculates an optimal match between the two given sequences or time series A distance-like quantity is measured between the two series Smartphone Based Human Behavioral Analysis4/18/201424

DTW Time Series 4/18/2014Smartphone Based Human Behavioral Analysis25 Time series can be accelerated/decelerated as much as necessary to give an optimal match. Cost: 3.3084e+05Cost: 2.7239e+06

Applying DTW to the Project 4/18/2014Smartphone Based Human Behavioral Analysis26 Allows training for different activities with disregard to time Supports three-dimensional data Provides extremely accurate results Runs in O(n 2 ) time

FastDTW 4/18/2014Smartphone Based Human Behavioral Analysis27 Open-source algorithm based on original DTW Runs in O(n) time Slightly less accurate Easy to port to Android

Environmental Sensors Robbie Rosati

Proximity Sensor 4/18/2014Smartphone Based Human Behavioral Analysis29 Gives proximity in cm Most phones only return binary values near and far Ability to check whether phone is pressed against ear, in pocket, etc. neural networks

Light Sensor 4/18/2014Smartphone Based Human Behavioral Analysis30 Detects ambient luminosity in lux Useful for indoor/outdoor detection Could be included in gestures, used with DTW

GPS 4/18/2014Smartphone Based Human Behavioral Analysis31 Can get users latitude and longitude Could improve detection for if user is driving Worst sensor with battery life Need to only use it occasionally

Microphone 4/18/2014Smartphone Based Human Behavioral Analysis32 Speaker recognition via NN algorithm Passive or active detection Could use to detect loudness of rooms Also see things in sound waveform like snoring

Neural Networks Robbie Rosati

Transition from Support Vector Machine to Neural Networks 4/18/2014Smartphone Based Human Behavioral Analysis34 Implemented SVM into our Behavior Analysis app Ran too inefficiently for phone hardware Difficult to train with the sound sensor Therefore, decided to use an alternative algorithm that would fit our needs. Decided on Neural Networks, a popular machine learning algorithm for speech recognition

Neural Networks 4/18/2014Smartphone Based Human Behavioral Analysis35 Algorithm used for machine learning and pattern recognition Inspired by the way the brain recognizes objects and sound Presented as systems of interconnected neurons that can compute values Difficult to train in a short time so used an API from Google to offload processing from the phone Integrates speech recognition into the app

Google Speech API 4/18/2014Smartphone Based Human Behavioral Analysis36 Open source API for speech recognition Could be coded into our Behavior Analysis app Uses NN to interpret speech Can be presented in text with further coding However, the app requires an internet connection for this function since it streams audio to remote servers

4/18/2014 Smartphone Based Human Behavioral Analysis 37 App Demonstration

Raw Memory Extraction & Analysis Michael Armstrong

4/18/2014 Smartphone Based Human Behavioral Analysis 39

Overview 4/18/2014Smartphone Based Human Behavioral Analysis40 Retrieving the physical image of a device is our goal. Immense variety of phones with an array of OS and applications. Current solutions are time consuming and/or very expensive. Access to deleted data A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.

Samsung Galaxy S3 Mini 4/18/2014Smartphone Based Human Behavioral Analysis41 Relatively inexpensive Compatible with teams SIM cards SD card slot Has some sensors in case we need it as back up for sensor testing

Testing Options 4/18/2014Smartphone Based Human Behavioral Analysis42 Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary. Backup Software Flashing box Linux Forensics Software dc3dd

dc3dd 4/18/2014Smartphone Based Human Behavioral Analysis43 Terminal based utility for Linux Parses a partition bit-by-bit and creates binary image Advantages: Exactly what we need, easy to use Disadvantage: Parses through empty space and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issue

Steps for Extraction 4/18/2014Smartphone Based Human Behavioral Analysis44 Gain root access to Android using simple utility Extract a copy of the database file Shrink the partition of a USB flash drive to the smallest possible size and copy the database file to the partition Use dc3dd to parse the partition and create the binary image for the database file

Viewing the Image File 4/18/2014Smartphone Based Human Behavioral Analysis45 Any hex viewer application should able to view the data inside the image Chose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for

Finding Desired Data 4/18/2014Smartphone Based Human Behavioral Analysis46 Messages and numbers are encoded as ASCII Locate corresponding hex data using the search function in Ghex Verify by comparing the located hex data to ASCII values Process should work for any type of file because the ASCII data will there regardless of file type

Example: Text Message 4/18/2014Smartphone Based Human Behavioral Analysis47

Strings and Grep Commands 4/18/2014Smartphone Based Human Behavioral Analysis48 Strings - Captures groups of letters separated by spaces in a file Grep - Searches for an expression matching the users input expression

Hidden Markov Model Andy McWilliams

Hidden Markov Model Smartphone Based Human Behavioral Analysis4/18/201450 Way of predicting the next possible output in a sequence of events Equivalent to recognizing or interpreting that particular sequence Slightly complicated because there can be multiple ways to produce the same observed output

Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral Analysis51 Accomplished through MATLAB There will be different matrices such as the transition matrix and emission matrix Transition matrix probability of transitioning from one state to the next Emission matrix probability that one state emits the next state

Hidden Markov Model 4/18/2014Smartphone Based Human Behavioral Analysis52 1. Model Evaluation Problem What is the probability of the observation? Given an observed sequence and an HMM, how probable is that sequence? Forward algorithm 2. Path Decoding Problem What is the best hidden state sequence for the observation? Given an observed sequence and an HMM, what is the most likely state sequence that generated it? Viterbi algorithm 3. Model Training Problem How to estimate the model parameters? Given an observation, can we learn an HMM for it?

Example with Binary 4/18/2014 Smartphone Based Human Behavioral Analysis 53

HMM MATLAB 4/18/2014Smartphone Based Human Behavioral Analysis54 Answers received contain new transition and emission matrices with probabilities There are three files in which we work with to execute the code Multiple Assumptions must be made in order to create the three files

Three Files of MATLAB 4/18/2014Smartphone Based Human Behavioral Analysis55 1. Markovsource is a data file that contains multiple fields of phone numbers, dates, etc. 2. The transfer file contains matlab code to convert the data into the necessary values 3. The markovTEmatrix file contains the transition and emission matrices These three files are used to receive the state probabilities for each matrix

Assumptions 4/18/2014Smartphone Based Human Behavioral Analysis56 There are three kinds of fields within the data file such as phone numbers with 10 digits, dates, and other non- numerical alphabetic fields Within the matrices, the first ten rows represent the date fields probabilities The next ten rows represent the phone number fields There is an extra 1 or 2 rows that are used for the non- numerical alphabetic fields

Transition and Emission Matrices 4/18/2014Smartphone Based Human Behavioral Analysis57

Multiple MATLAB Functions 4/18/2014Smartphone Based Human Behavioral Analysis58 Hmmdecode - used to calculate the state probabilities of a sequence of emissions Hmmviterbi used to calculate the most probable state path for a hidden Markov model Hmmtrain used to calculate likelihood estimates for the transition and emission matrices from a certain sequence Hmmestimate can be used if the sequence is already known, but it has something missing

State Probabilities 4/18/2014Smartphone Based Human Behavioral Analysis59

Administration Andy McWilliams

Validation Plan 4/18/2014Smartphone Based Human Behavioral Analysis61 Sensor Data Extraction & Analysis Retrieve real-time data from all chosen sensors Graph and analyze each set of data by performing tests Raw Memory Extraction & Analysis Test each raw memory extraction option Retrieve binary image of databases Machine Learning Algorithms Use algorithms to recognize patterns in users behavior Use HMM algorithm to retrieve deleted data

Budget 4/18/201462Smartphone Based Human Behavioral Analysis Also received $400 from Dr. Hus budget to purchase Samsung Galaxy S4 Mini

Schedule Smartphone Based Human Behavioral Analysis4/18/201463

Impacts Healthcare, Activity Monitoring, Forensics National Science Foundation Dr. Hu will use project findings to help teach CPS at rural schools in a virtual environment 4/18/201464Smartphone Based Human Behavioral Analysis

Thank you! Questions?

ece 494 capstone design final design presentation smartphone based human behavioral analysis andrew...

Documents