ece 494 capstone design final design presentation smartphone based human behavioral analysis

Download ECE 494 Capstone Design Final Design  Presentation Smartphone Based  Human Behavioral Analysis

Post on 24-Feb-2016




0 download

Embed Size (px)


ECE 494 Capstone Design Final Design Presentation Smartphone Based Human Behavioral Analysis. Andrew Jackson Michael Armstrong - PowerPoint PPT Presentation


Slide 1

ECE 494 Capstone DesignFinal Design Presentation

Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron StewartApril 18, 2014 Advisor: Dr. Fei Hu

Outline4/18/2014Smartphone Based Human Behavioral Analysis2Project Recap & GoalSystem DiagramSubsystem Breakdown & Team RolesSensor Data Extraction & AnalysisApp Development & Activity RecognitionDTW AlgorithmEnvironment SensorsNN AlgorithmRaw Memory Extraction & Analysis HMM AlgorithmAdministration

Project Recap & Goal4/18/2014

Smartphone Based Human Behavioral Analysis3Create a system for tracking and detecting a users behavioral patterns though the phones sensors and internal memory logs.Can be used in multiple areas:HealthcareActivity MonitoringHomeland Security

System Diagram4/18/2014

4Smartphone Based Human Behavioral Analysis

Subsystem Breakdown & Team Roles 4/18/2014

5Smartphone Based Human Behavioral Analysis5Sensor Data Extraction & AnalysisAaron StewartAndroid, iOS, or Windows PhoneWe chose to go with Android.No development license feesMajority of marketMany sensors availableOpen file systemPrevious Android programming experienceCheaper phone prices


7Smartphone Based Human Behavioral Analysis

Samsung Galaxy S4 Mini4/18/2014

Smartphone Based Human Behavioral Analysis8New phone with many sensors

Fairly inexpensive for an unlocked phone

Smaller size great for testing in pockets


Smartphone Based Human Behavioral Analysis9SensorsSensor Types 4/18/2014

Smartphone Based Human Behavioral Analysis10Sensor Types 4/18/2014

Smartphone Based Human Behavioral Analysis11 Behavior Analysis App Version 1.04/18/2014

Smartphone Based Human Behavioral Analysis12Shows values from each sensor in real-time as the sensor updates Can save the data to a file on phone Move file to computer to analyze with MATLABGraph the dataValidate that it makes sense

Testing with App3/10/2014Smartphone Based Human Behavioral Analysis13Most sensor data comes in a set of 3 points

Few have single values

Tests related to typical smartphone user behaviors

Used MATLAB for data graphing and filtering

Light Sensor Test (Different Exposure) 4/18/2014

Smartphone Based Human Behavioral Analysis14

Accelerometer Test (Answer Call)4/18/2014

Smartphone Based Human Behavioral Analysis15




App Development and Activity RecognitionAndrew Jackson4/18/2014

Smartphone Based Human Behavioral Analysis17

Behavior Analysis App Version 2.04/18/2014

Smartphone Based Human Behavioral Analysis18Integrates DTW machine learning algorithm Allows training of the algorithm from the phone no computer requiredOnce trained, will recognize where the phone is with almost 100% accuracyVisual UpgradesAll features from previous app rolled over

Further App Improvements4/18/2014

Smartphone Based Human Behavioral Analysis19Increased accuracy of activity recognition by Increasing sensor sensitivityPolling each sensor more times a secondTweaking DTW algorithm codeUsing multiple sensors at the same time Continued adding new activitiesAdded speech recognitionBy demo day:Add in GPS coordinatesFurther explore light, sound, and proximity sensor possibilities

Accelerometer, Light, and Proximity4/18/2014

Smartphone Based Human Behavioral Analysis20Chose to work with accelerometer first because it can give some of the most useful data about the phoneApplying DTW on accelerometer dataNext, applied light and proximity sensors to get better results

Recognized ActivitiesLast PresentationNow4/18/2014

Smartphone Based Human Behavioral Analysis21Walking (with phone in hand)Talking on phoneSitting on tableHolding in hand

Walking (with phone in pocket)RunningStairs SittingDriving4/18/2014

Smartphone Based Human Behavioral Analysis22App DemonstrationDynamic Time WarpingAndrew JacksonDynamic Time WarpingMeasures similarity between two sequences which may vary in time or speedCalculates an optimal match between the two given sequences or time seriesA distance-like quantity is measured between the two series

Smartphone Based Human Behavioral Analysis4/18/2014


DTW Time Series4/18/2014

Smartphone Based Human Behavioral Analysis25Time series can be accelerated/decelerated as much as necessary to give an optimal match.

Cost: 3.3084e+05Cost: 2.7239e+06Applying DTW to the Project4/18/2014

Smartphone Based Human Behavioral Analysis26Allows training for different activities with disregard to time Supports three-dimensional dataProvides extremely accurate resultsRuns in O(n2) time


Smartphone Based Human Behavioral Analysis27Open-source algorithm based on original DTWRuns in O(n) timeSlightly less accurateEasy to port to Android

Environmental SensorsRobbie RosatiProximity Sensor4/18/2014

Smartphone Based Human Behavioral Analysis29Gives proximity in cmMost phones only return binary values near and farAbility to check whether phone is pressed against ear, in pocket, etc.neural networks

Light Sensor4/18/2014

Smartphone Based Human Behavioral Analysis30Detects ambient luminosity in luxUseful for indoor/outdoor detectionCould be included in gestures, used with DTW


Smartphone Based Human Behavioral Analysis31Can get users latitude and longitude

Could improve detection for if user is driving

Worst sensor with battery life

Need to only use it occasionally


Smartphone Based Human Behavioral Analysis32Speaker recognition via NN algorithmPassive or active detectionCould use to detect loudness of roomsAlso see things in sound waveform like snoring

Neural NetworksRobbie RosatiTransition from Support Vector Machine to Neural Networks4/18/2014

Smartphone Based Human Behavioral Analysis34Implemented SVM into our Behavior Analysis appRan too inefficiently for phone hardwareDifficult to train with the sound sensor

Therefore, decided to use an alternative algorithm that would fit our needs.Decided on Neural Networks, a popular machine learning algorithm for speech recognition

Neural Networks4/18/2014

Smartphone Based Human Behavioral Analysis35Algorithm used for machine learning and pattern recognitionInspired by the way the brain recognizes objects and soundPresented as systems of interconnected neurons that can compute valuesDifficult to train in a short time so used an API from Google to offload processing from the phone Integrates speech recognition into the app

Google Speech API4/18/2014

Smartphone Based Human Behavioral Analysis36Open source API for speech recognition Could be coded into our Behavior Analysis appUses NN to interpret speech Can be presented in text with further codingHowever, the app requires an internet connection for this function since it streams audio to remote servers


Smartphone Based Human Behavioral Analysis37App DemonstrationRaw Memory Extraction & AnalysisMichael Armstrong4/18/2014

Smartphone Based Human Behavioral Analysis39


Smartphone Based Human Behavioral Analysis40Retrieving the physical image of a device is our goal.Immense variety of phones with an array of OS and applications.Current solutions are time consuming and/or very expensive.Access to deleted data

A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.Samsung Galaxy S3 Mini4/18/2014

Smartphone Based Human Behavioral Analysis41Relatively inexpensive

Compatible with teams SIM cards

SD card slot

Has some sensors in case we need it as back up for sensor testing

Testing Options4/18/2014

Smartphone Based Human Behavioral Analysis42Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary.Backup SoftwareFlashing boxLinux Forensics Softwaredc3dd dc3dd 4/18/2014

Smartphone Based Human Behavioral Analysis43Terminal based utility for Linux Parses a partition bit-by-bit and creates binary imageAdvantages: Exactly what we need, easy to useDisadvantage: Parses through empty space and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issueSteps for Extraction4/18/2014

Smartphone Based Human Behavioral Analysis44Gain root access to Android using simple utilityExtract a copy of the database fileShrink the partition of a USB flash drive to the smallest possible size and copy the database file to the partitionUse dc3dd to parse the partition and create the binary image for the database fileViewing the Image File4/18/2014

Smartphone Based Human Behavioral Analysis45Any hex viewer application should able to view the data inside the imageChose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for

Finding Desired Data4/18/2014

Smartphone Based Human Behavioral Analysis46Messages and numbers are encoded as ASCII

Locate corresponding hex data using the search function in Ghex

Verify by comparing the located hex data to ASCII values

Process should work for any type of file because the ASCII data will there regardless of file typeExample: Tex