Adaptive Rehabilitation using Mixed-reality at Home: The ARM at

Home study

RIC Margaret Duff, Meghan Buell, and W. Zev RymerEmory Steven Wolf and Aimee ReissASU Pavan Turaga, Nicole Lehrer, Michael Baran, Vinay Venkataraman, Loren Olson and Todd IngallsCMU Thanassis Rikakis

Adaptive mixed reality rehabilitation

- Computational assessment of movement - Abstract audio and visual feedback, evaluation and adaptation

- Tangible sensing objects provide functional goals

- Increase engagement and enhance motor learning through self-assessment of movement

- Recover pre-morbid movement patterns and reduce compensation while increasing function

Completed at Banner Baywood Medical Center

Evaluating outcomes of mixed reality compared to traditional

therapy

AMRR improves function and kinematics

- Both groups improved in the Wolf Motor Function Test

- Every AMRR participant saw at least a 30% improvement in composite kinematic impairment measure (KIM), with a much more consistent distribution of improvement

% c

hang

e

% c

hang

e

*

AMRR group

Control group

Issues to address

- Neither group reported a significant change in impaired arm use / quality in ADLs

- Long-term plan to both encourage functional and movement quality improvements

- Continue therapy at home and with a greater variety of tasks

Scaling AMRR theories for home therapy

Home AMRR system

- An engaging therapy environment at home

- Task repetition, variability and intensity

- Easy to use and understand in a largely unsupervised environment

- Useful information (feedback) about task completion and movement quality

Feedback examples

Pilot study of unsupervised training

- Test feasibility and effectiveness

- Examine how people with stroke use and accept the system

- Determine what further work is needed to accommodate the needs of the greatest percentage of people

Study protocol

- 1 week (3 sessions) of supervised training- 4 weeks (12 sessions) of unsupervised training- Pre, post, and 4 week follow-up evaluationsParticipant demographics

Median Range

Age (years) 59.5 49 - 69

Months post stroke

27 14 - 44

Fugl-Meyer (/66) 48.5 37 - 55

N = 6 (6M, 0 F)

Wolf Motor Function Test

Both FAS and time improve after therapy and are mostly retained at follow-up

Fugl-Meyer and Motor Activity Log

FM scores improve after therapy and are retained at follow-up

MAL scores are inconsistent after therapy and at follow-up

Kinematic results of trained task

Velocity peak trained to about .6 m/sInconsistent changes in horizontal

trajectory

Participant acceptance of system

Preliminary outcomes

- System was stable throughout

- Successful unsupervised training

- FM and WMFT improved after training

- Kinematics and MAL were

inconsistent

Current and future work

- Improved hand function sensing

- Track ADLs objectively and transfer therapy gains to everyday

- Increased adaptability of therapy protocols

- Better classification of movement impairments

Hand function sensing

More adaptive therapy protocols

Current Protocol- Two set therapy tracks

Future Considerations- Progression based on ability- Objects that vary more in complexity

and weight- Variability within a set of reaches- Dissociate objects from table

Assessing the classifiersProblem - building high level metrics of efficiency for complex movements with reduced sensing

Assessing new metrics for classifying movement-Correlation to kinematic assessment of simple tasks-Therapist ratings of simple to complex tasks, each rated in terms of overall performance and component performance -Components that are being trained on do not have one-to-one mapping with therapist ratings, which implies no supervised training data to build classifiers-But there is weak supervision !

Kinematic classifiers that drive feedback

Curvedness – Measure of spatial error

Too Fast / Too Slow – Measure of deviation in velocity profile

Smoothness – Measure of variability in velocity profile

Cone grasp(simple, trained)

Elevated touch(simple, semi-trained)

Transport cylinder(complex, trained)

Therapist rated tasks

Video recordings of 3 tasks (5 trials each), performed once a week independent of therapy

Treating therapist rated each reach, presented randomly

Rating system was developed to assign a score for each of the following:

1. Initial impression of overall trial (Modified FAS)

2. Trajectory 3. Compensation4. Hand manipulation (grasp, touch)5. Transport phase (if transport task)6. Release phase (if transport task)7. Final impression of overall trial (Modified

FAS)

If needed, explanation recorded if final impression is different than initial impression

Therapist rated tasks

How does therapist rating help in tuning these classifiers?

Assume a linear model of kinematic classifiers

W1

W2

W3

W4

Cumulative Classifier Score

W1F1(T1) + W2F2(T2) + W3F3(T3) + W4F4(T4) + noise =

Cost function =

Use Nelder-Mead’s Simplex (fminsearch algorithm in MATLAB) to perform optimization

Movement quality assessment

Therapist Rating (R)

{Initial impression of overall score}

Initial Results

– 3 participants (mild impairment) recorded at Emory

– 4 video recorded sessions each

– Total of 55 reaches to grasp the cone

Initial ResultsBefore optimization: Observe the overlap in score distributions, which implies classifiers are not tuned properly

4 5

Means are close

Initial ResultsOptimizing only the combination weights: The score distribution overlap does not get affected, suggesting that the problem really lies with the classifiers

4 5

Initial Results

4 5

Reduced overlap

After optimization of weights and thresholds: score distribution overlap reduces

Conclusions

- Changes in therapy protocols and tasks needed to benefit a larger subset of the population

- Movement classifiers need to be generalized and improved, while staying accurate

- Monitor and encourage transfer of therapy strategies to everyday life

Thank You!!!