Enhancing the Measurement of Clinical Outcomes Using Microsoft Kinect

Philip Breedon and Francesco Luke Siena Design for Health and Wellbeing Research Group Nottingham Trent University

Bill Byrom and Willie Muehlhausen Product Innovation ICON Clinical Research

2

Presentation Overview

Overview of Clinical Trials

Motion Capture Platforms in Healthcare

Review of Kinect Applications for Outcomes Measurement

Example Measurement System

Overview of Clinical Trials

Bill Byrom, ICON

29% Do not advance

55% Do not advance

40% Do not advance

• Clinical trials rely upon robust and validated methodologies to measure health status and to

detect treatment-related changes in health status over time

• In some cases outcomes measures used rely on subjective ratings by the investigators at

each study research site.

– performance, balance, movement or mobility based on observation of the patient conducting a

specified movement or activity.

• Subjective ratings are not very sensitive to detecting small improvements

– Inter-rater reliability

• Objective measures preferred

– More sensitive

– Less prone to rater variability

– Able to measure detailed or subtle aspects of movement and mobility.

5

Objective measurement

Motion Capture Platforms in Healthcare

Luke Siena, Nottingham Trent University

1. 3D Camera Systems and Sensors

2. Benefits of Motion Capture Platforms

3. Comparison Of Key Hardware & Utilities

4. Understanding The Progress Within The Motion Capture Platform Market

7

Motion Capture Platforms in Healthcare

8

3D Camera Systems & Sensors

9

Motion Capture Platforms in Healthcare

• 3D camera systems and sensors have great potential to continue having a positive impact on the market in a variety of industries, especially within health care and clinical platforms.

• Hardware specification improvements may still be required when considering accurate tracking of fine or rapid movements, and therefore the sampling rates associated with the capture of this data may need to improve.

• The application of motion capture camera systems and technology in clinical and home health care applications, especially within the rehabilitation sector is constantly evolving.

• Platforms such as Neuroforma, JINTRONIX, Stroke Recovery with Kinect and Face To Face have recently been developed, amongst others.

• There is a growing body of applications utilising motion capture technology that study or encourage movement in wellness, healthcare and clinical research.

• The area of rehabilitation is constantly exploring ways of providing engaging environments through regular exercise regimes to enable patient feedback and correction.

• Ensuring exercises are being performed correctly for optimal benefit.

• Enabling remote assessment and adjustment of exercise regimes between clinic visits ensures regular patient contact and reviews which can be monitored.

10

Benefits Of Motion Capture Platforms In Healthcare

11

Comparison of Microsoft Kinect 1.0 & 2.0 For HealthCare Utility Applications

Function Kinect 1.0 Kinect 2.0

RGB Camera (Pixel) 1280 × 1024 or 640 × 480 1920 × 1080

Depth Camera (Pixel) 640 × 480 512 × 424

Sampling Rate (FPS / Hz) 30 FPS 30 FPS

SDK 1.8 Compatibility Yes No

SDK 2.0 Compatibility No Yes

Face Tracking Yes Yes

Expression Recognition No (Possible With Additional Algorithms) Yes

Bone Orientations No Yes

Body Joint Forces No Yes

Hand Tracking No (Possible With Additional Tools) Yes

Muscle Simulation No Yes

Heart Rate Measurement No Yes

* Price & Specifications as of May 2016

Capability / Function Intel RealSense SR300 Kinect 2.0

RGB Camera (Pixel) 1080p at 30 FPS, 720p at 60 FPS 1920 × 1080 at 30 FPS

Depth Camera (Pixel) Up to 640 x 480 at 60 FPS (Fast

VGA, VGA), HVGA at 110 FPS

512 × 424 at 30 FPS

Skeletal Joint Definition Points 22 26

Face Tracking & Recognition Yes Yes

Expression Recognition Yes Yes

Gesture Recognition Yes Yes

Hand Tracking Yes Yes

Audio Stream Dual Array Microphones 4-Mic-Array

Connectivity (USB) 3.0 3.0

Approx. price (USD)* 130 190

12

Comparison of Intel® RealSense TM SR300 and Microsoft Kinect 2.0 For HealthCare Systems

* Price & Specifications as of May 2016

13

Why Champion The Intel® RealSense™ ?

• The Intel camera offers greater resolution and sampling rate in comparison to Kinect 2.0, which may offer advantages when tracking fine or fast movements.

• One of the novelties of the Intel RealSense 3D camera range is its versatility for integration into a variety of platforms, yet at the same time it remains affordable.

• Intel have developed a number of Intel RealSense camera systems which can be integrated into a variety of platforms whether this be Desktop PC’s, All-In-One PC’s, 2 In 1 PC’s, external camera systems, smartphones and tablet kits and even a robotics.

Review of Kinect Applications for Outcomes Measurement

Bill Byrom, ICON

1. Gait and balance

2. Upper extremity movement

3. Chest wall motion analysis

4. Facial analysis

15

Four main areas of measurement

• Various performance tests

proposed – Short walking tests

– Treadmill walking tests

– Balance tests

• Spline interpolation to estimate

100 Hz sampling frequency

• Custom error correction

technique to improve data

artefact identification

16

Gait and balance

Pfister A. et al. (2014)

17

Gait and balance

Ref Performance Measure Indication Comparator n Validation evidence

[1] Treadmill walking tests

Hip/knee flexion/extension Stride timing

Healthy volunteers (HV)

VICON motion capture

28 Kinect underestimated flexion, overestimated extension. Stride timing often well correlated.

[2] Short walk Velocity, stride length, hip/knee ROM

MS + HV PRO (MSWS) ClinRO (EDSS)

20 Able to distinguish MS form controls Reliability good except step width and hip ROM

[3] 6 m walk Step length, foot swing velocity, mean and peak gait velocity, asymmetry

Stroke 10mWT, TUG, Step test

30 Kinect parameters reliable: ICCs > 0.8 Feasible to instrument gait analysis

[4] Standing, stepping, walk on spot, UPDSS

Various PD + HV VICON motion capture

19 Good for gross movements Poor for fine movement Good correlation with VICON (r > 0.8)

[5] Short max speed walk

Speed; L/R, Up/Down and 3D deviation; speed deviation

MS + HV 25 foot walk test

44 Able to differentiate MS and controls Good concordance with 25-foot walk test

18

Upper extremity movement

Lin J-L. et al. (2014)

• Range of motion and

reaching volume

estimated from various

performance tests – Standard range of motion

movements

– Movement task

Upper extremity movement

Ref Performance Measure Indication Comparator n Validation evidence

[6] Shoulder movement

Shoulder flexion, abduction, rotation

Adhesive capsulitis + HV

Goniometer 27 ICCs: 0.864-0.942

[7] FMA / ARAT Shoulder/elbow/wrist flexion, abduction, rotation

Stroke Impulse motion cap. + clinician ass.

9 MC: R2 = 0.64, p < 0.001 Clin. Ass: R2 = 0.86, p < 0.001

[8] Arm movement

Shoulder flexion, abduction, rotation, extension

Healthy volunteers (HV)

Goniometer 10 r = 0.86 to 0.99

[9] Arm movement

3D workable reaching space

HV Impulse motion capture

10 R2 = 0.79

[10] Pediatric Functional Assessment

Index finger and thumb, wrist, elbow, shoulder ROM

HV Clinician assessment 12 “Technically sound approach”

[11] Movement task

Involuntary movements / dyskinesia

HV Clinician assessment

4 Cohen’s kappa 0.85, p < 0.05

[12] Fugl-Meyer, WMFT, ARAT

Shoulder, elbow and wrist position

HV Optitrack motion capture

10 “Kinect is sufficiently accurate and responsive”

[13] Arm/hand movements

Machine learning identification

MS Differentiate MS from HV

1041 “Automated MS assessment possible”

• Four Kinect cameras used to generate a 3D image

of the chest

• Performance test: – Quiet breathing for 20 s, followed by a relaxed vital

capacity (VC) manoeuver (maximum inspiration and

expiration) and followed by 20 s of quiet breathing.

• Tidal volume, Respiratory Rate, and minute

ventilation compared to spirometry – Good concordance for

• Cystic Fibrosis patients: r>0.8656

• Healthy volunteers r> 0.922

20

Chest wall analysis

21

Facial analysis

Face to Face solution

• Rehabilitation system for facial paralysis in

stroke patients.

• Recognizes facial expressions

• Facial exercise performance is assessed by

the system and scored according to how

well the user can undertake each of the

defined set of expressions.

• Potential to apply to providing longitudinal

objective measures of change to assess

treatment effects.

22

Summary findings

• May be less able to measure fine or rapid movements – Sampling rate of camera

– Resolution and depth of vision

• Joint detection accuracy with conventional SDK may limit some applications

• May provide a low cost alternative to specialist labs or subjective endpoints in large scale trials

Example Measurement System

Bill Byrom, ICON

• Objectives

• Understand how to develop applications using the Kinect Windows SDK

• Demonstrate the concept of health outcomes measurement using Kinect

• Input into definition of future requirements

24

Proof of concept: shoulder ROM

25

Proof of concept: shoulder ROM

Thank you

Any questions?

@billbyrom

https://uk.linkedin.com/in/bill-byrom-7136975

References [1] Pfister A. et al. (2014). Comparative abilities of Microsoft Kinect and Vicon

3D motion capture for gait analysis . J Med Eng Technol; 38: 274-280. [8] Lin J-L. et al. (2014). Assessment of range of shoulder motion using Kinect.

Gerontechnology ; 13:249

[2] Gholami F. et al. (2015). https://arxiv.org/pdf/1508.02405v1.pdf [9] Kurillo G. et al. (2013). Evaluation of upper extremity reachable workspace using Kinect camera. Technology and Health Care ; 21:641–656

[3] Clarke R.A. et al. (2015). Instrumenting gait assessment using the Kinect in people living with stroke: reliability and association with balance tests. J NeuroEngineering and Rehab; 12:15-23.

[10] Rammer J.R. et al. (2014). Evaluation of Upper Extremity Movement Characteristics during Standardized Pediatric Functional Assessment with a Kinect-Based Markerless Motion Analysis System. Conf Proc IEEE Eng Med Biol Soc. 2014: 2525–2528.

[4] Galna B. et al. (2014). Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson’s disease. Gait & Posture; 39: 1062–1068

[11] Li S.et al. (2015). Quantitative Assessment of ADL: A Pilot Study of Upper Extremity Reaching Tasks. J Sensors; Article ID 236474.

[5] Behrens J. et al. (2014). Using perceptive computing in multiple sclerosis - the Short Maximum Speed Walk test. J NeuroEngineering and Rehab; 11:89-98.

[12] Webster D. et al. (2014). Experimental Evaluation of Microsoft Kinect’s Accuracy and Capture Rate for Stroke Rehabilitation Applications. IEEE Haptics Symposium 2014.

[6] Lee S.H. et al. (2015). Measurement of Shoulder Range of Motion in Patients with Adhesive Capsulitis Using a Kinect. PLOS ONE10(6): e0129398.

[13] Kontschieder P. et al. (). Quantifying Progression of Multiple Sclerosis via Classification of Depth Videos. , in (Golland, P. et al. eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014, Volume 8674 of the series Lecture Notes in Computer Science); pp 429-437.

[7] Olesh E.V. (2014). Automated Assessment of Upper Extremity Movement Impairment due to Stroke. PLoS ONE 9(8): e104487

[14] Harte J.M. et al. (2015). Chest wall motion analysis in healthy volunteers and adults with cystic fibrosis using a novel Kinect-based motion tracking system. Med. Biol. Eng. Comput.; DOI 10.1007/s11517-015-1433-1.

[15] Breedon P. et al. (2014). First for Stroke: using the Microsoft' Kinect' as a facial paralysis stroke rehabilitation tool. Int J Integrated Care (IJIC), 14.