gymskill - a personal trainer for physical exercises
DESCRIPTION
We present GymSkill, a personal trainer for ubiquitous monitoring and assessment of physical activity using standard fitness equipment. The system records and analyzes exercises using the sensors of a personal smartphone attached to the gym equipment. Novel fine-grained activity recognition techniques based on pyramidal Principal Component Breakdown Analysis (PCBA) provide a quantitative analysis of the quality of human movements. In addition to overall quality judgments, GymSkill identifies interesting portions of the recorded sensor data and provides suggestions for improving the individual performance, thereby extending existing work. The system was evaluated in a case study where 6 participants performed a variety of exercises on balance boards. GymSkill successfully assessed the quality of the exercises, in agreement with the professional judgment provided by a physician. User feedback suggests that GymSkill has the potential to serve as an effective tool for motivating and supporting lay people to overcome sedentary, unhealthy lifestyles. GymSkill is available in the Android Market as "VMI Fit"TRANSCRIPT
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
GymSkill: A Personal Trainer for Physical Exercises
Andreas Möller, Luis Roalter, Stefan Diewald, Johannes Scherr,
Matthias Kranz
Technische Universität München, Germany
Nils Hammerla, Patrick Olivier, Thomas Plötz
Newcastle University,
United Kingdom
March 22 PerCom 2012, Lugano, Switzerland
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outline
• Motivation
• Automatic Assessment of Physical Exercises
• Case Study
• Conclusion
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 2
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Motivation
• Physical activity is required for healthy lifestyle • Problem: people do not exercise enough
– Lack of knowledge of correct exercise execution for fast improvement – High level of long-term motivation needed
• Advantages of a personal trainer: – Continuous supervision and professional feedback – Individualized advice and motivation – Minimization of injury risk
• Disadvantages of a personal trainer: – Expensive – Privacy
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 3
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
• Scientific focus on – Activity recognition – Wearable sensors
• Health and Fitness Systems – Heart rate monitor, foot pod, GPS watch, …
• No skill assessment
– Fixed activity tracking (Wii balance board, Kinect) • Gaming aspect • Dedicated hardware
Related Work
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 4
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
GymSkill
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 5
• Smartphone-based personal trainer • Monitoring and assessment of physical exercises
– Based on phone sensor data – No need for additional sensors
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outline
• Motivation
• Personal Health and Fitness Systems
• Automatic Assessment of Physical Exercises
• Case Study
• Conclusion
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 6
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
GymSkill: Automatic Assessment of Physical Exercises
• We look at rocker board exercises • Example use case for equipment-based training • Defined quality criteria:
tilt angles, speed, smoothness of movement, touching the ground… • Phone is attached to the board • During exercise: Basic situated feedback • After exercise: Fine-grained analysis of exercise quality and reasons for
quality differences
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 7
move back and forth move left and right balance on the center
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Skill Assessment Principle
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 8
Sensor Data Recording
Sensor Data Processing PCBA Analysis
Skill Assessment
Detailed Skill Report
Real-Time Feedback
Skill Level (Score)
Output Rendering
Log File
AJAX
Smartphone Server“Cloud“
Simple Analysis
HTTP
User Feedback
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Direct Feedback While Training
• Repetition count • Visual feedback of board movement • Warning when tilted too far • Optional acoustic feedback
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 9
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Post-Exercise Analysis
• Performed for each exercise run (e.g.: 10 repetitions of tilting back and forth) • Global Analysis
– Smoothness and continuity of movement – Global motion quality – Usage of board‘s degrees of freedom
• Local Analysis – Identify „interesting portions“ of sensor data – What is „interesting“?
In recurrent data, this means unusual data compared to the rest – E.g. participant hesitates or gets stuck
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 10
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Global Analysis
• Estimation of motion axis (providing the dominant signal) • Comparison of the empirical distribution
to ideal distribution function („gold standard“) • Usage of normalized and un-normalized functions to determine
smoothness and utilization of board‘s degrees of freedom • Transformation into a performance quality metric between 0 and 1
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 11
�� �� � � ��
����
����
����
����
���
����
������ �������
������������ �����
��������
���! � "��!�
���#
���
���#
���
������������ �$�
�� ������
%&���'�������
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Local Analysis
• Assumption: sensor data of a movement shares (unknown) statistical properties
• Unusual portions of a sequence violate this assumption and can be identified • Extension to PCA: Principal Component Breakdown Analysis • PCA model is learned from local neighborhood
(using sliding window technique) • Frames are projected to lower-dimensional subspace using PCA • Reconstruction errors used as a measure for motion quality • Problem: ideal window size not known • Solution: multi-scale comparison (iteratively growing window)
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 12
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
User Feedback
• Visual feedback – PCA-based assessment diagram
(red and yellow parts contain irregularities)
• Textual feedback – Based on global and local metrics – Rule-based combination of aspects trigger
textual cues
• “Thumb” feedback – Overall assessment at a glance
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 13
Try to be more continuous in your motion!
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outline
• Motivation
• Automatic Assessment of Physical Exercises
• Case Study
• Conclusion
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 14
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Case Study
• Set of 20 exercises developed by sports medicine specialist
• 6 participants, 5 days of training (20 different exercises twice a day)
• 1200 exercise records • Goal 1: Collection of training data
– Identification of criteria and assessment by physician
• Goal 2: Qualitative evaluation of prototype – Questionnaire study
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 15
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Case Study: Trial Assessment Example 1
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 16
PCBA: Continuity
Time [s]5 10 15 20
!2 !1 0 1 20
0.05
0.1
0.15
0.2
displacement [std]
frequency
General motion
observedideal
!max 0 +max0
0.05
0.1
0.15
0.2
0.25Angle usage
displacement [°]
frequency
Try to be more continuous in your motion!
You touched the ground 3 times.
Your movement is not ideal.
! Move back and forth in a continuous motion.
Try to move similarly to both sides of the board.
! You do not utilise the full range of angles!
! You lean towards the front!
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Case Study: Trial Assessment Example 2
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 17
PCBA: Continuity
Time [s]5 10 15
!2 !1 0 1 20
0.05
0.1
0.15
0.2
displacement [std]
frequency
General motion
observedideal
!max 0 +max0
0.05
0.1
0.15
0.2Angle usage
displacement [°]
frequency
Your movement is continuous, nice!
You did not touch the ground!
Overall you perform the movement nicely!
Try to move similarly to both sides of the board.
! You do not utilise the full range of angles!
! You lean towards the front!
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Qualitative Feedback
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 18
Fully agree
Not agree at all
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Qualitative Feedback
• Feature wishlist
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 19
Fully agree
Not agree at all
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Outline
• Motivation
• Automatic Assessment of Physical Exercises
• Case Study
• Conclusion
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 20
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Summary
• Mobile skill assessment of overall exercise quality • Identification of typical exercising errors • “Personal trainer” • Participant feedback indicates potential for long-term exercising motivation
Further work: • Long-term study on training progress • Assessment and feedback entirely on mobile phone • Generalization of assessment model
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 21
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
GymSkill in Google Play
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 22
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Thank you for your attention! Questions?
? ? 04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 23
[email protected] www.vmi.ei.tum.de/team/andreas-moeller.html
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Paper Reference
• Please find the associated paper at: https://vmi.lmt.ei.tum.de/publications/2012/percom2012-preprint.pdf
• Please cite this work as follows: • Andreas Möller, Luis Roalter, Stefan Diewald, Johannes Scherr, Matthias
Kranz, Nils Hammerla, Patrick Olivier, Thomas Plötz GymSkill: A Personal Trainer for Physical Exercises In: 2012 IEEE International Conference on Pervasive Computing and Communications (PerCom2012), Lugano, Switzerland, March 2012, pp. 213-220
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 24
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
If you use BibTex, please use the following entry to cite this work:
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 25
@INPROCEEDINGS{6199869, author={M\"{o}ller, Andreas and Roalter, Luis and Diewald, Stefan and Scherr, Johannes and Kranz, Matthias and Hammerla, Nils and Olivier, Patrick and Pl\"{o}tz, Thomas}, booktitle={Pervasive Computing and Communications (PerCom), 2012 IEEE International Conference on}, title={GymSkill: A personal trainer for physical exercises}, year={2012}, month={march}, volume={}, number={}, pages={213 -220}, doi={10.1109/PerCom.2012.6199869}, ISSN={},}
Technische Universität München Fachgebiet Verteilte Multimodale Informationsverarbeitung Prof. Dr. Matthias Kranz
Image Sources
• Slide 5 – gpsreview.net – amazon.com – spieleradar.de – golem.de
04.01.13 Andreas Möller et al. - GymSkill: A Personal Trainer for Physical Exercises 26