metrics and benchmarking for assisted wheelchair navigation

Download Metrics and Benchmarking for Assisted Wheelchair Navigation

If you can't read please download the document

Upload: crisurdiales

Post on 01-Jun-2015

263 views

Category:

Technology


1 download

DESCRIPTION

A brief survey on the different metrics that can be used in the field of assistive wheelchair navigation, including usability questionnaires, clinical scales, task and psych metrics, etc. It also includes obstacle courses and a benchmarking proposal. An invited presentation to ICORR'13 workshop CanWheel (http://www.canwheel.ca/events/upcoming/icorr-2013-workshops/)

TRANSCRIPT

  • 1. ICORR 2013 - Seattle TEC2011-29106 Cristina Urdiales Metrics and benchmarking for assisted wheelchair navigation Grupo ISIS (UMA)

2. ICORR 2013 - Seattle TEC2011-29106 Wheelchair metrics There are no universally accepted metrics/procedures to evaluate navigation There are benchmarking procedures in established obstacle courses BUT they do not provide information on everyday navigation at preferred environments for example ... The Valutazione Funzionale Mielolesi (VFM) Tufts Assessment of Motor Performance (TAMP) Wheelchair Skills Test (WST) 3. ICORR 2013 - Seattle TEC2011-29106 Wheelchair metrics There are well known questionnaires that can be applied to any situation; e.g.: TLX (General): overall workloadl task difficulty, time pressure, performance, mental/sensory effort, physical effort, frustration level, stress level, fatigue, activity type. PIADS (Assistive devices): self-esteem, well-being, quality of life, embarrassment, eagerness to try new things MPT (Assistive technology): peoples condition, ADL, opinion on technology FEW (wheelchair users): ergonomics QUEST (wheelchair users): usability, weight or dimensions of the device, service delivery, repair and servicing ... but they are usually subjetive and too general i.e. users give opinions and they do not need to provide information on their specific condition QUEST PIADS 4. ICORR 2013 - Seattle TEC2011-29106 Condition metrics Questionnaires can be (and usually are) complemented with clinical scales to characterize the user ; e.g.: (MMSE) or Folstein (Cognitive): quantitative measure of cognitive status in adults GDS (Cognitive): constructed to assess depression and general well-being in the elderly Barthel Index (functional): measures how a persons copes with the activities of daily living CIRS (physical): cumulative effect of diverse conditions but they only provide information about the state of the user at the moment the scale was obtained and each covers specifics, not the global picture. These questionnaires can be complemented with biometrics sensors to obtain information in real time, but wearables are often not comfortable enough to carry for prolongued times and only a few provide stable readings during activity (e.g. pulsioximeters) 5. ICORR 2013 - Seattle TEC2011-29106 Course-related metrics Wheelchair propulsion, assessed in terms of period of time, a fixed distance or longest distance possible. Transfer from and to the wheelchair, usually examining the performance in different transfers. Negotiation of kerbs, with kerbs height ranging from 0.025 to 0.15m, and some requiring ascending and descending the kerb. Ascending slopes, defined in terms of inclination (ranging from 1 to 11 degrees) and length (ranging from 3 to 21m). Traversing tracks, i.e. slalom, figure of eight and obstacle course. Sprinting over a fixed distance (ranging from 6.5 to 30m). Performing a wheelie. Managing brakes, negotiating doors or loading the wheelchair into a car. Typically measured while the user completes a certain preestablished course. Usually very specific and objetive. 6. ICORR 2013 - Seattle TEC2011-29106 Path-related metrics Deviation with respect to the canonical trajectory (often quantified into a few bins as ok, acceptable and wrong) Success/failure Typically measured when there is a pre-established visual path to track. Very objetive. Can be combined with task metrics (see next slide) 7. ICORR 2013 - Seattle TEC2011-29106 Task metrics Task success (or) degree of success Task completion time Number of collisions Distance travelled All quantitative measures that can be objectively related to the users performance with the wheelchair while performing a given task Global (the task needs to be finished to obtain them) Smoothness, related to sharp direction changes Safety, related to obstacle proximity Directness, related to the skill of keeping directions Local (can be obtained at any given location) Time and Distance to Stop (Ts, Ds) Time and Distance to Collision (Tc, Dc) Obstacle clearance (distance when an obstacle is avoided) Dynamic (the task needs to be finished to obtain them) 8. ICORR 2013 - Seattle TEC2011-29106 Psych metrics Psych metrics aim at (indirectly) evaluating the users (mental) state. These metrics are very dependent on interfaces and ergonomics and, consequently, have been widely used in those fields. Interaction number (with the interface)/ amount of information exchanged between user and machine (usability) Number of error situations (usability) Joystick Variation (more than 10%) (mental workload) Intervention level, defined as the portion of time that the user moves a joystick (mental workload) Disagreement, defined as the angle between the user command and the wheelchair motor command (frustration, stress) Inconsistency, defined as the variation of the user output when facing similar situations (cognitive skills) 9. ICORR 2013 - Seattle TEC2011-29106 Benchmarking It consists of trying to find how a generic user would respond to every faced situation Generic users do not exist, but they can be modelled from a large enough number of subjects via data mining If situations are defined at high level, there are infite variations. If they are defined at local level, there are only a few1 1 J. Minguez, J. Osuna, and L. Montanor, A divide and conquer strategy based on situations to achieve reactive collision avoidance in troublesome scenarios, IEEE Trans. on Robotics, 2009. 10. ICORR 2013 - Seattle TEC2011-29106 Data mining Given a large enough sample space, we split it into classes where users respond similarly to similar situation. Their prototypes are equal to the average of all samples within the bin weighted by their respective efficiency, i.e.the best solution taken by the largest number of users 11. ICORR 2013 - Seattle TEC2011-29106 User profiling and benchmarking Using the proposed prototype user we can ... Subject 14 (Guillain-Barre, MMSE=23, Barthel=28) see what skills any given user lacks with respect to the average user see what efficiency a given user has in a task in comparison with the average user Subject 813 (rep. Brain stroke, MMSE=20.3, Barthel=29) 12. ICORR 2013 - Seattle TEC2011-29106 Conclusions and future work (Standard) Benchmarking procedures and metrics are of key importance in wheelchair navigation Typically we need to combine usability questionnaires, clinical indexes and several metrics to see the bigger picture Profiling helps greatly the analysis of results through personalization Can we use real time metrics to (indirectly predict potential health issues (alarms)? Can we use profiles for better personalization and adaptation of assistance? 13. ICORR 2013 - Seattle TEC2011-29106 Cristina Urdiales Grupo ISIS (UMA) Metrics and benchmarking for assisted wheelchair navigation