goal attainment scaling - aacpdm · 2018. 10. 12. · 1.goal attainment scaling = gas 2.gas is a...

Occupational Therapy and Physical Therapy

Goal Attainment ScalingClinical Utility and Training Methods

Jen Angeli, Karen Harpster, Amber Sheehan


FINANCIAL DISCLOSURE

AACPDM 72nd Annual Meeting

October 9-13, 2018

Speaker Names: Amber Sheehan, Karen Harpster, Jen Angeli

1. Disclosure of Relevant Financial Relationships

We have no financial relationships to disclose.

2. Disclosure of Off-Label and/or investigative uses:

We will not discuss off label use and/or investigational use in my presentation.


Grab your phone.

In the internet browser, type:

kahoot.it


https://kahoot.com/welcomeback/

https://create.kahoot.it/login

https://kahoot.com/welcomeback/

https://create.kahoot.it/login


Learning objectives

• Synthesize the literature on GAS training and clinical utility

• Describe pilot effort to train therapists on GAS utilization, per best available evidence

• Propose best practice for GAS standardization

• Discuss feasibility of implementation


First, we get on the same page.

1.Goal Attainment Scaling = GAS

2.GAS is a criterion-referenced measure

that helps us to quantify a degree to

which personal goals are achieved.

3.It was developed in 1968 and was first

used in mental health.

4.It is becoming more popular in

rehabilitation health, as an index of

intervention efficacy.


The GAS parts

• Document a set of patient-identified goals

• Identify the construct of change that matters most to the patient/family

• Observe/discuss/document current performance on each goal

• Agree upon a desired short term outcome

• Discuss what performance looks like if it is a little better than the desired short term outcome, or much better than the desired short term outcome.


For example:

I want to ride my bike to the end of

Victory Avenue (0.5 miles).Performance Interpretation GAS

interval

Completed MS150 Much better than expected +2

Rode to park for new zipline (1

mile)

Better than expected +1

See above Short term goal 0

Rode to Joe’s house for dinner

(3 houses down)

Progress, but did not achieve

goal

-1

Cannot ride bike Baseline level of performance -2


Extra parts• Weight each goal for importance (patient/caregiver)

and difficulty (therapist)

• Put it all in a big formula that will calculate a T score.

• Use the T score or change in T scores to interpret

or Minimal Detectable Change = 10 points


What’s nice about GAS

• It’s patient-centric

• It provides a single, objective summary of

performance

• Progress reflects meaningful change in a

prioritized area


What’s not nice about GAS

• It’s hard to learn.

• It feels like it’s not clinically feasible.

• It’s only valid if you follow the rules,

particularly those associated with scaling.


What we learned about GAS

in a systematic review

1. Is it being used in pediatric rehab?

2. What’s the rigor of published pediatric

studies employing the GAS?

3. Is GAS a responsive outcome measure?


• 51 studies in pediatric rehab were included

in our review

• They weren’t very strong with respect to

rigor and quality

• Despite this, GAS appeared to detect

meaningful change in ~ 60% of studies


• Higher Sackett levels = more rigorous study

design


• 17 quality criteria that group broadly into categories describing:– content validity of the scale

– reliability of scale construction

– reliability of scale rating

– additional items related to training and examiner bias

• Higher quality scores = greater validity in GAS application

• See handout for criteria!


Quality summary

• Of the 51 studies included, less than half (19 studies) were categorized as Sackett levels I, II, or III.

• Average quality score = 4.57 points (/17)

Individual studies lost the most points for failure of Goal Attainment Scaling scales to adhere to the criteria of:

-unidimensionality (1/51)

-time-specificity (3/51)

-reliability of scale construction-equidistance of levels (3/51)

-inter-rater reliability (4/51)


Why does it matter?

• GAS T scores may inflate (and indicate

progress) simply because of bias

introduced from failure to create valid and

reliable scales.


What did we do?


Our Goal:

• To utilize the GAS as an objective

outcome measure with a high level of

reliability in clinical practice

– Start with episodic care summer programs

• Clinic-based GAS session

http://www.acmsoft.com/Services/think-big-start-small.html


https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-

oZ74Bg&q=let%27s+do+it&oq=let%27s+do+it&gs_l=img.3..0l10.18304.28268..28880...0.0..0.105.1091.7j4......2....1..gws-wiz-

img.....0..35i39j0i67.3puL26l830Q#imgrc=FjAo6UDp_7yAjM:&spf=1539226018390

https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-

oZ74Bg&q=Reading+research+articles&oq=Reading+research+articles&gs_l=img.3..35i39j0i24.3052.5474..5884...0.0..0.99.10

78.12......1....1..gws-wiz-img.......0j0i7i30j0i7i5i30j0i8i7i30j0i5i30.4Kmw2Xj2-aY


https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=tby-

W8HkDMqf_QbHgLToAg&q=variation&oq=variation&gs_l=img.3..35i39j0l9.2142.2834..3640...0.0..0.102.191.1j1......1..

..1..gws-wiz-img.0D_0uKckWAo#imgrc=nTk8YeFP9SO0NM:&spf=1539226998417

• Level of parent and child involvement

• Video tape GAS performance vs no video

• Documentation

• Setting the construct of change


Develop a process

• Video tape all GAS goal performances

• Watch videos during team meetings

– Develop documentation (importance, difficulty,

level of change, etc)

• Review goals using GAS checklist

• Review GAS levels with caregivers

http://www.10minutebiztools.com/Startup-Trial-and-Error.html


GAS in Research: Refine the

process even further• Independent teams

– COPM/treatment team

– GAS team

• Training to ensure specificity of COPM goal E.g. “John runs back and forth and puts holes in his socks” “John will sit for 5 minutes to complete a fine motor task”

• Develop a script to elicit meaningfulness of goals


What happened next:

Therapist Training

• Developed a training manual, based on current evidence– includes procedures and criteria for scale

construction, scoring, and interpretation of outcome data

• Provide formal training – Introduced the training manual and demonstrate

application via case study

– Trainees independently review materials and construct a scales based on the same video

– Review scale with each trainee using quality criteria to promote adequate interrater reliability


What happened next:

GAS Visit via telehealth

• GAS scales are developed by 2

independent raters based on video taped

performance of goals

– GAS scales compared for sameness

• Child scored on both

– GAS checklist implemented

– Feedback given regarding scale construction

within 1 week of GAS completion


GAS through telehealth....

• Volume control

• Test video call with caregiver

– Ensure they understand how to use

technology, camera angle, etc.

• Make sure the caregiver knows the child

needs to be present for session


GAS through telehealth....

• Make sure the supplies for the goals are readily available

• Making sure the video stays on the child during goal completion

• Considering an informal scale to capture how much assistance the caregiver needs to participate in co-creation of scale

– seems to be impacting validity of the scales we create


of doing GAS through

telehealth

• Materials usually readily available

• Natural environment

• Generalization occurs naturally

• Empowering the caregiver


Criteria for GAS appraisal

• Eliminated items

– Goals specific to aim of intervention

– Pretest score same across all goals/ patients

– Specification of follow up time

– Adequate therapist training

– Report of inter-rater reliability

– Report of examples of goals and scaling

• Added items suggested by other sources


Results thus far

• 2/5 trained therapists had no prior GAS

background

• Telehealth goal scaling completed with 6

patients, each by one of 3 therapists

– Average 9.4/15 scaling quality criteria met

– After feedback provided, no additional sessions

yet to assess potential change in quality

– Two trained therapists have not yet had a goal

scaling session scheduled


Results thus far, cont.

• Most easily achieved criteria

– Mutually exclusive goals

– All five levels defined

• similar to the literature


Results thus far, cont.

• Other easily achieved criteria

• Scale uni-dimensionality

• Verification of pre-intervention performance

• Equidistance of scale levels

– frequent absence in the literature, achieved here due to

initial training and study design


Future plans

• Study design

– Confirm post-test scale score with caregiver


Future plans

• Study design

– Determine interrater reliability by correlating

the ratings of the original goal scaler with a

second rater on 20-30% of goals (Krasny-

Pacini et al., 2017; King et al., 1999)

http://www.clearwatercounty.org/departments/building_plan/BuildingSub/buildingcontacts/index.php

http://www.clearwatercounty.org/departments/building_plan/BuildingSub/buildingcontacts/index.php


Future plans

• Reporting results

– Adequate example of goals and scaling reported

– Results in literature were most commonly stated

as:

• a raw post-intervention T-score (27 of 51 studies)

• change in T-score (3 of 51)

• number or % of goals that were achieved (i.e., at or

above the “0” level on the scale; 23 of 51 studies)

• effect size (4 of 51)


Zip Coat

• Goal: Mom would like Peter to be able

to zip his coat independently

• Construct of Change: Level

of independence


Let's Practice


Zip Coat



2:

1:

0:

-1:

-2: Needs help with engaging zipper and stabilizing coat to pull up zipper


Zip Coat



2:

1:

0:

-1: Stabilize coat and pull up zipper with coat on the table

-2: Needs help with engaging zipper andstabilizing coat to pull up

zipper


Zip Coat



2:

1:

0: Stabilize coat and pull zipper up with coat on


-2: Needs help with engaging zipper andstabilizing coat to pull up zipper


Zip Coat



2:

1: Engage and pull up zipper with coat on the table





Zip Coat



2: Engaging and pull up zipper while the coat is on

1: Engage and pull up zipper with coat on the table





Jennifer Angeli, DPT, PhD

Physical Therapist, Cincinnati Children’s Hospital Medical Center

Assistant Professor, Rehabilitation Sciences, University of Cincinnati

513.252.5722

[email protected]

Amber Sheehan, OTR/L

Occupational Therapist, Cincinnati Children’s Hospital Medical Center

513.636.8526

[email protected]

Karen Harpster, OTD, PhD

Occupational Therapist, Cincinnati Children’s Hospital Medical Center

Assistant Professor, Rehabilitation Sciences, University of Cincinnati

513.803.3604

[email protected]


References

• Grant, M., & Ponsford, J. (2014). Goal attainment scaling in brain injury rehabilitation: strengths, limitations and recommendations for future applications. [KH, SS]. Neuropsychol Rehabil, 24(5), 661-677. doi: 10.1080/09602011.2014.901228

• Harpster, K., Sheehan, A., Foster, E. A., Leffler, E., Schwab, S. M., & Angeli, J. M. (2018). The methodological application of goal attainment scaling in pediatric rehabilitation research: a systematic review. Disability and rehabilitation, 1-10.

• Kiresuk T.J. & Sherman R.E. (1968). Goal Attainment Scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4(6), 443–453.

• Kiresuk, T.J., Smith, A., & Cardillo, J.E. 1994. Goal Attainment Scaling: Applications, Theory, and Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates.

• Mailloux, Z., May-Benson, T. A., Summers, C. A., Miller, L. J., Brett-Green, B., Burke, J. P., et al. (2007). The Issue Is—Goal attainment scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61, 254–259.

• McDougall, J. & Wright, V. (2009). The ICF-CY and Goal Attainment Scaling: Benefits of their combined use for pediatric practice. Disability and Rehabilitation, 31 (16): 1362–1372.

• Ruble, L., McGrew, J.H., & Toland, M.D. (2012). Goal Attainment Scaling as an outcome measure in randomized controlled trials of psychosocial interventions in autism. J Autism Dev Disord, 42(9): 1974–1983

• Schaaf, R.C. et al. (2013). An intervention for sensory difficulties in children with autism: A randomized trial. J Autism Dev Disord, online.

goal attainment scaling - aacpdm · 2018. 10. 12. · 1.goal attainment scaling = gas 2.gas is a...

Documents