ece 494 capstone design final design presentation smartphone based human behavioral analysis

ECE 494 Capstone DesignFinal Design Presentation

Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron Stewart

April 18, 2014 Advisor: Dr. Fei Hu

Cyber Physical

Team

2

Outline

4/18/2014Smartphone Based Human Behavioral Analysis

Project Recap & Goal System Diagram Subsystem Breakdown & Team Roles Sensor Data Extraction & Analysis App Development & Activity Recognition DTW Algorithm Environment Sensors NN Algorithm Raw Memory Extraction & Analysis HMM Algorithm Administration

3

Project Recap & Goal


Create a system for tracking and detecting a user’s behavioral patterns though the phone’s sensors and internal memory logs.

Can be used in multiple areas: Healthcare Activity Monitoring Homeland Security

4

System Diagram


Sensors

Hidden Markov Model

Graphs

Binary Data

Acoustic

Accelerometer

Light

Reverse Engineering Hidden Markov Model

Neural Networks

Dynamic Time Warping

Call log, Deleted phone numbers, deleted texts

Speech recognition, shouting, snoring, coughing

Walking, Running, Going up stairs, Sitting, Standing

Daytime/nighttime, inside/outside, in or out of pocket

Internal Memory

5

Behavior Recognition

Phone Data

Sensor Data Extraction &

Analysis

Accelerometer

Light

Sound

Proximity

GPS

Raw Memory Extraction &

Analysis

dc3dd

Call Logs

Text Messages

Machine Learning

Algorithms

Hidden Markov Model

Neural Networks


Subsystem Breakdown & Team Roles


Andrew Jackson• Android Sensor App• DTW AlgorithmAaron Stewart• Raw Memory Analysis• Team LeaderRobbie Rosati • Machine Learning

Algorithms• Raw Memory ExtractionMichael Armstrong• Raw Memory Extraction• HMM AlgorithmAndy McWilliams• HMM Algorithm • Raw Memory Analysis

Sensor Data Extraction & Analysis

Aaron Stewart

7

Android, iOS, or Windows PhoneWe chose to go with

Android. No development license

fees Majority of market Many sensors available Open file system Previous Android

programming experience Cheaper phone prices


8

Samsung Galaxy S4 Mini


New phone with many sensors

Fairly inexpensive for an unlocked phone

Smaller size great for testing in pockets

Smartphone Based Human Behavioral Analysis9

Accelerometer

Light

SoundProximity

GPS

4/18/2014

Sensors

10

Sensor Types


Accelerometer• Measures acceleration with gravity

value• Reports data in 3 values, one for

each axis• Example: Could tell if user is walking

or stationaryGPS• Reports GPS coordinates• See changes in location• Example: Could tell if user is driving

11

Sensor Types


Light• Intensity of light• Reports data in a single value• Example: Indoors/Outdoors, Weather, Time of

daySound• Detects sound waves• Reports loudness• Example: Voice recognition

Proximity• Changes depending on surrounding objects• Boolean value• Example: Device is in tight space such as pocket

or purse

12

Behavior Analysis App Version 1.0


Shows values from each sensor in real-time as the sensor updates

Can save the data to a file on phone

Move file to computer to analyze with MATLAB

Graph the data Validate that it makes

sense

13

Testing with App


Most sensor data comes in a set of 3 points

Few have single values

Tests related to typical smartphone user behaviors

Used MATLAB for data graphing and filtering

14

Light Sensor Test (Different Exposure)


15

Accelerometer Test (Answer Call)


X

Y

Z

App Development and Activity Recognition

Andrew Jackson


17

Sensors

Hidden Markov Model

Graphs

Binary Data

Acoustic

Accelerometer

Light


Neural Networks






Internal Memory

18

Behavior Analysis App Version 2.0


Integrates DTW machine learning algorithm

Allows training of the algorithm from the phone – no computer required

Once trained, will recognize where the phone is with almost 100% accuracy

Visual Upgrades All features from previous

app rolled over

19

Further App Improvements


Increased accuracy of activity recognition by Increasing sensor sensitivity Polling each sensor more times a

second Tweaking DTW algorithm code Using multiple sensors at the same

time Continued adding new activities Added speech recognitionBy demo day:

Add in GPS coordinates Further explore light, sound, and

proximity sensor possibilities

20

Accelerometer, Light, and Proximity


Chose to work with accelerometer first because it can give some of the most useful data about the phone

Applying DTW on accelerometer data

Next, applied light and proximity sensors to get better results

21

Recognized ActivitiesLast Presentation Now


Walking (with phone in hand)

Talking on phone Sitting on table Holding in hand

Walking (with phone in pocket)

Running Stairs Sitting Driving

Smartphone Based Human Behavioral Analysis22 4/18/2014

App Demonstration


Andrew Jackson

Smartphone Based Human Behavioral Analysis

24

Dynamic Time Warping Measures similarity between two sequences

which may vary in time or speed Calculates an optimal match between the two

given sequences or time series A distance-like quantity is measured between

the two series

4/18/2014

Time

25

DTW Time Series


Time series can be accelerated/decelerated as much as necessary to give an optimal match.

Cost: 3.3084e+05

Cost: 2.7239e+06


26

Applying DTW to the Project

4/18/2014

Allows training for different activities with disregard to time

Supports three-dimensional data Provides extremely accurate results Runs in O(n2) time

27

FastDTW


Open-source algorithm based on original DTW

Runs in O(n) time

Slightly less accurate

Easy to port to Android

Execution Time of FastDTW on Large Time Series

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000Length of Time Series

Tim

e (s

econ

ds)

DTWFastDTW (radius=100)FastDTW (radius=20)FastDTW (radius=0)

Environmental Sensors

Robbie Rosati

29

Proximity Sensor


Gives proximity in cm Most phones only return binary values “near”

and “far” Ability to check whether phone is pressed

against ear, in pocket, etc.

30

Light Sensor


Detects ambient luminosity in lux

Useful for indoor/outdoor detection

Could be included in gestures, used with DTW

31

GPS


Can get user’s latitude and longitude

Could improve detection for if user is driving

Worst sensor with battery life

Need to only use it occasionally

32

Microphone


Speaker recognition via NN algorithm Passive or active detection Could use to detect loudness of rooms Also see things in sound waveform – like

snoring

Neural Networks

Robbie Rosati

34

Transition from Support Vector Machine

to Neural Networks


Implemented SVM into our Behavior Analysis app

Ran too inefficiently for phone hardware

Difficult to train with the sound sensor

Therefore, decided to use

an alternative algorithm that would fit our needs. Decided on Neural

Networks, a popular machine learning algorithm for speech recognition

35

Neural Networks


Algorithm used for machine learning and pattern recognition

Inspired by the way the brain recognizes objects and sound

Presented as systems of interconnected neurons that can compute values

Difficult to train in a short time so used an API from Google to offload processing from the phone

Integrates speech recognition into the app

36

Google Speech API


Open source API for speech recognition

Could be coded into our Behavior Analysis app

Uses NN to interpret speech

Can be presented in text with further coding

However, the app requires an internet connection for this function since it streams audio to remote servers

Smartphone Based Human Behavioral Analysis37 4/18/2014

App Demonstration

Raw Memory Extraction & Analysis

Michael Armstrong


39

Sensors

Hidden Markov Model

Graphs

Binary Data

Acoustic

Accelerometer

Light


Neural Networks






Internal Memory

40

Overview


Retrieving the physical image of a device is our goal. Immense variety of phones with an array of OS

and applications. Current solutions are time consuming and/or very

expensive. Access to deleted data

A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.

41

Samsung Galaxy S3 Mini


Relatively inexpensive

Compatible with team’s SIM cards

SD card slot

Has some sensors in case we need it as back up for sensor testing

42

Testing Options


Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary. Backup Software Flashing box Linux Forensics Software dc3dd

43

dc3dd


Terminal based utility for Linux Parses a partition bit-by-bit and creates binary

image Advantages: Exactly what we need, easy to

use Disadvantage: Parses through empty space

and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issue

44

Steps for Extraction


Gain root access to Android using simple utility

Extract a copy of the database file Shrink the partition of a USB flash drive to the

smallest possible size and copy the database file to the partition

Use dc3dd to parse the partition and create the binary image for the database file

45

Viewing the Image File


Any hex viewer application should able to view the data inside the image

Chose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for

46

Finding Desired Data


Messages and numbers are encoded as ASCII

Locate corresponding hex data using the search function in Ghex

Verify by comparing the located hex data to ASCII values

Process should work for any type of file because the ASCII data will there regardless of file type

47

Example: Text Message


48

‘Strings’ and ‘Grep’ Commands


Strings - Captures groups of letters separated by spaces in a file

Grep - Searches for an expression matching the user’s input expression

Hidden Markov Model

Andy McWilliams


50

Hidden Markov Model

4/18/2014

Way of predicting the next possible output in a sequence of events

Equivalent to recognizing or interpreting that particular sequence

Slightly complicated because there can be multiple ways to produce the same observed output

51

Hidden Markov Model


Accomplished through MATLAB There will be different matrices such as the

transition matrix and emission matrix Transition matrix – probability of transitioning

from one state to the next Emission matrix – probability that one state

emits the next state

52

Hidden Markov Model


1. Model Evaluation Problem What is the probability of the observation? Given an observed sequence and an HMM, how

probable is that sequence? Forward algorithm

2. Path Decoding Problem What is the best hidden state sequence for the

observation? Given an observed sequence and an HMM, what is the

most likely state sequence that generated it? Viterbi algorithm

3. Model Training Problem How to estimate the model parameters? Given an observation, can we learn an HMM for it?

Smartphone Based Human Behavioral Analysis53

Example with Binary

4/18/2014

54

HMM MATLAB


Answers received contain new transition and emission matrices with probabilities

There are three files in which we work with to execute the code

Multiple Assumptions must be made in order to create the three files

55

Three Files of MATLAB


1. Markovsource is a data file that contains multiple fields of phone numbers, dates, etc.

2. The transfer file contains matlab code to convert the data into the necessary values

3. The markovTEmatrix file contains the transition and emission matrices

These three files are used to receive the state probabilities for each matrix

56

Assumptions


There are three kinds of fields within the data file such as phone numbers with 10 digits, dates, and other non-numerical alphabetic fields

Within the matrices, the first ten rows represent the date field’s probabilities

The next ten rows represent the phone number fields

There is an extra 1 or 2 rows that are used for the non-numerical alphabetic fields

57

Transition and Emission Matrices


58

Multiple MATLAB Functions


Hmmdecode - used to calculate the state probabilities of a sequence of emissions

Hmmviterbi – used to calculate the most probable state path for a hidden Markov model

Hmmtrain – used to calculate likelihood estimates for the transition and emission matrices from a certain sequence

Hmmestimate – can be used if the sequence is already known, but it has something missing

59

State Probabilities


Administration

Andy McWilliams

61

Validation Plan


Sensor Data Extraction & Analysis Retrieve real-time data from all chosen sensors Graph and analyze each set of data by performing

tests Raw Memory Extraction & Analysis

Test each raw memory extraction option Retrieve binary image of databases

Machine Learning Algorithms Use algorithms to recognize patterns in user’s

behavior Use HMM algorithm to retrieve deleted data

62

Budget


Samsung Galaxy S3

Mini

$225

$75

Capstone Budget

Also received $400 from Dr. Hu’s budget to

purchase Samsung Galaxy S4 Mini


63

Schedule

4/18/2014

Research and Purchase

Build Sensor Testing App

Test Sensors and Analyze Data

Add DTW Support for App

Train DTW With Gestures

Build App Version 2.0

Add Speech Recognition Support

Research Image Extraction Tools

Create and View a Generic Binary Image

Root Android and Extract Text Database

Create Binary Image of Database File

Succesfully Create HMM Matrices

Use HMM on Binary Data

8/24/2

013

10/13

/2013

12/2/

2013

1/21/2

014

3/12/2

014

CompletedIn - ProgressNot Started

64

Impacts Healthcare, Activity Monitoring, Forensics National Science Foundation Dr. Hu will use project findings to help

teach CPS at rural schools in a virtual environment


Thank you! Questions?

ece 494 capstone design final design presentation smartphone based human behavioral analysis

Documents

human behavioral analysis20

proximity4182014 smartphone

users behavioral patterns

light sensor

sensor sensitivitypolling

sensor updates

previous app

data graphing