GeST – Computer Delivered Gesture Therapy for People with

Aphasia

Abi.Roper.1@City.ac.uk Division of Language and Communication Science

City University London

Today•Gesture Therapy•Existing Computer Therapies

•The GReAT Project

•Designing and Refining a Computer Gesture Therapy - Gest

•Gest Demonstration

•Delivering a Computer Therapy

•Gest Pilot Study

•Outcomes and Implications

Previous Studies of Gesture Therapy

To compensate for speech (e.g. Daumuller & Goldenberg, 2010)

To facilitate speech (e.g. Boo & Rose, 2011; Marangolo et al, 2010; Rose & Douglas, 2008)

Findings

• Even people with severe aphasia can improve gesture production

• Treatments with a gesture component can enhance naming

But …

Gains are often very modest

Gains may relate to the intensity of therapy (Caute, 2012)

Computer Therapy for Aphasia

• Sentence Shaper – Speech production Exercises sentence level

• “The program is most likely to be effective for people who are able to produce some speech, and whose executive function and/or memory allows them to utilize a software program that requires executing certain actions in order (for example, turning a sound recorder on, speaking then turning the recorder off) and self-monitoring (playing back their utterances in order to correct and expand them).”

SWORD Spoken Word Production Exercises. Developed for Apraxia of Speech.

Software Design and Useability Strengths• Mouse only control: All participants reported that they were

able to use the mouse / touchpad (following instruction) to navigate the programme.

• AP1’s wife: “It’s about making it simple. Easy to read.” • All participants responded that the programme never once

crashed.• AOS 29’s Husband: “I could do me back garden!” Reports

being happy that his wife was doing something independently.

• AP2’s wife: [whilst her husband was using the therapy] “I got a lot of gardening done!”

Computer Therapy for Aphasia

Software Design and Usability LimitationsMouse Access: AOS 29 reported that it took some time

to learn to use the mouse but she got used to using it. Reported that a touch screen would have made the laptop a lot easier to use.

SLT Comments: “Some patients needed prompting to remind them that they need to click. Some patients timed out before providing a response and were awarded a fail. Is it possible to give even longer than 25 seconds?”

The GReAT Project

What’s different and novel about the GReAT project?

• Use of Gesture Recognition Technology (Gesture Recognition in Aphasia Therapy)

• Software Design and Useability Focus

• Participatory Design Process

Project Aims

• To develop an affordable, computer-based technology that can be used in therapy at home to help people with severe aphasia to gesture.

• To evaluate the efficacy of the technology within a pilot therapy study

Project Structure

• Phase 1: Designing a prototype gesture therapy using participatory design methods.

• Phase 2: Testing and piloting the prototype

Project TeamHuman Computer Interaction Design & Language and Communication Science

Stephanie Wilson Sam Muscroft Julia Galliers Jane Marshall

Naomi Cocks Tim Pring Abi Roper

Phase 1

• Designing a prototype gesture therapy using participatory design methods.

Consultants

Justine Everson Gerald Hartup Carol Watson

Philip Pepper Emma Buswell

Consultants

• Role: to test and feedback about relevant technology.

• Person Specifications: – Expressive aphasia language difficulties. – Able to attend university once or twice a month for

participatory design sessions.

• Recruited through in house clinic and through links with the Stroke Association Communication Support

Co-ordinators.

• Employed by City University London as Casual Staff members.

Methods: Participatory Design Sessions

• Participatory design – engaging end users in design process

• Sessions explored offline gesture therapy, computer gesture recognition, interaction within 3D worlds and computer interfaces.

• Consultants took part in 9 sessions each

• Project team involved in each session- 1 HCID Researcher- 1 HCID Developer- 1 Speech and Language Therapist Researcher- 2 or 3 Consultants

Session Structure

1. Introduction to scheduled activities

2. Round table gesture activity

3. Demonstration of Technology

4. Trial use of technology by one consultant - followed by interview at computer

5. Tea break

6. Trial use of technology by remaining consultant(s)

Participatory Design

What did we learn from the Sessions?

1. Consistency2. Simplicity3. Pace4. Reliability5. Rewards6. Individual Differences7. Potential of ‘gaming’.

Key Features of GeSTSeparate keyboard Gesture recognition

Gestures presented in isolation & in context 3D worlds

OK ← →

The Prototype

OK

← →

Tool Design

3 Levels

Tool Design• User sees gesture demonstrated twice

• Is invited to copy the gesture

• User monitors their production attempt on screen

• Receives reward and feedback for each correctly recognised gesture

Demonstration Video

Using the Therapy at home

•Pilot study (Coming up next)•How does this work at home?Key differences between lab and home – User practising independently, User intending to practise daily. User practising in non-lab conditions.

Things to consider when setting up

•Lighting conditions

•Safety and permanence (negotiate!)

•User comfort and access

Things to consider when training

• Develop the user’s confidence in the system. (Be confident yourself)

Demonstrate:

1. Allow user to observe entirely2. Allow user to observe and operate

interaction buttons3. Allow user to operate alone but with

support as needed (confidence)

Things to consider when training

•Reinforce how to switch the computer on and off several times.

•Make an appointment to come back in one week to review.

•At review appointment, observe and re-train difficult procedures.

Phase 2

• The Pilot Study

Questions• Will practice with Gest improve participants’ production of

gestures &/or spoken words?• Will improvements be specific to items that feature in the

programme?• Will gains occur when Gest is used without ongoing

therapist support?• Will gains be maintained after Gest is withdrawn?• What are participants’ views about Gest?• What are carers’ views about Gest? (where relevant)• Is Gest easy and enjoyable to use?

Participants

• 9 people with severe aphasia– Consent to take part– Fluent pre-stroke users of English– Naming score <20% – Able to recognise pictures– No known dementia or other cognitive impairment

Consent

Screening

Tests (1)

3 Weeks Practice

Tests (2)

Phase 1 with weekly visits from therapist

3 Weeks Practice

Phase 2 with no weekly visits from therapist

Tests (3)

3 weeks no tool

Tests (4)Total time commitment: about 14 weeks

Practice Phases

• Each last 3 weeks• Each practise 15 gestures with the tool• Phase 1: Weekly visits from therapist• Phase 2: Initial but no weekly visits

Tests

• 60 items– Gesture from picture– Name from picture What is the

name of this?

How would you gesture this?Items:

30 practised with Gest

15 familiarised only

15 controls

Scoring Gestures

• Gesture tests are filmed• 4 Scoring videos created• Each video contains 60 gestures in random

order:– 15 from test 1– 15 from test 2– 15 from test 3– 15 from test 4

• Scores– Recognition Score– Rating Score

• Scorers are ‘blind’ to the time of assessment

Scoring Gestures

Results

Gesture Recognition

Statistical Analysis

• Time x Group interaction• Planned comparisons:

– Items that were treated with Supported Gest changed over time, other groups did not

– Supported Gest items improved significantly following practice (between time 1 and time 2), but not in the other phases

– Recognition of Supported Gest items was significantly improved against baseline even at time 4

Interim Conclusions

• Repeated testing without intervention (Control items) did not improve gesture production

• Independent practice with Gest and familiarisation did not improve gesture production.

• Using Gest with therapist support improved gesture production. However:– Gains were modest – Differences between the groups were very small.

Usage Logs

• Record– Number of sessions– Length of sessions– Levels of programme accessed– Number of gestures recognised

More Interim Conclusions

• Benefits from Supported Gest may reflect– Therapist input (‘The Abi Factor’)– Usage

More Results

Naming

More Conclusions

• Using Gest did not facilitate naming of the gesture targets

• This was despite the repeated inclusion of spoken names in the programme

Qualitative Observations:Some ‘Carer’ Comments

Independence of Use

• ‘She uses it all on her own, I don’t know how to operate it’

• The first session I stayed with L, after that I’ve helped only if she’s found something particularly frustrating’

• All comment that the participant initiated use of Gest

Enjoyment

• All say that the participant enjoyed Gest

• ‘he likes it when they clapped’

• ‘some of the gestures are particularly fitting and she enjoyed rainbow’

Views about Technology

• ‘I was a technophobe and when they said ‘computer’ I thought it was going to cause problems. I thought I wouldn’t understand and he wouldn’t understand it. But it’s so ‘easy’

Reservations

• Carry over to real life (1 carer):

• ‘while she works on it here (points to computer) it doesn’t necessarily translate’

• She wanted a hankie last night and didn’t make a gesture’

Conclusions• Will practice with Gest improve participants’ production of

gestures?Yes

• Will improvements be specific to items that feature in the programme?

Yes• Will gains occur when Gest is used without ongoing therapist

support?No

• Will gains be maintained after Gest is withdrawn?Yes

• Will practice with Gest improve naming of target itemsNo

Conclusions• What are participants’ views about Gest?

Very Positive

• What are carers’ views about Gest? Very Positive

• Is Gest easy and enjoyable to use?

Yes

Acknowledgements

The Research Councils UK Digital Economy Programme The Stroke Association

Consultants and their familiesParticipants and their families

All our wonderful students who helped to score data

Thank YouGReAT@city.ac.uk

www.soi.city.ac.uk/great

Find Out More

Website: www.soi.city.ac.uk/great

Facebook: www.facebook.com/aphasiatech

Vimeo: www.vimeo.com/aphasiatech

Email: GReAT@city.ac.uk