goal attainment scaling: its use in evaluating pediatric ... attainment scaling.pdf · goal...

Goal Attainment Scaling:Its Use in Evaluating

Pediatric Therapy Programs

Gillian A. KingJanette McDougallRobert J. Palisano

Janet GritzanMary Ann Tucker

ABSTRACT. Goal attainment scaling is becoming an increasinglypopular technique for evaluating the functional goal attainment of chil-dren receiving pediatric therapy services. This article reports on theexperiences of the authors in conducting formal program evaluationsusing this individualized measurement approach. Goal attainment scal-ing is described, its utility is assessed, and issues in its use are identi-fied. The article considers the pros and cons of the technique, highlightsthe key decisions required to use goal attainment scaling effectively,and provides standard criteria and procedures for its use in pediatricsettings. [Article copies available for a fee from The Haworth Document Deliv-

Gillian A. King, PhD, is Investigator, CanChild Centre for Childhood DisabilityResearch, McMaster University and Research Program Manager, Thames ValleyChildren’s Centre. Janette McDougall, MA, is Research Associate, Thames ValleyChildren’s Centre. Robert J. Palisano, ScD, PT, is Professor and Director, Program inMovement Science, Medical College of Pennsylvania, Hahnemann University, andCo-Investigator with CanChild. Janet Gritzan, MClSc, is Speech-Language Patholo-gist and Mary Ann Tucker, BSc, is Manager of School-Age and Adolescent Services,both at Thames Valley Children’s Centre.

The authors sincerely thank the service providers, parents, teachers, and childrenwho assisted with this research. The authors also thank Paul Stolee for his usefulfeedback on this article.

This research has been made possible through funding provided by CanChild andThames Valley Children’s Centre.

Physical & Occupational Therapy in Pediatrics, Vol. 19(2) 1999� 1999 by The Haworth Press, Inc. All rights reserved. 31

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

PHYSICAL & OCCUPATIONAL THERAPY IN PEDIATRICS32

ery Service: 1-800-342-9678. E-mail address: [email protected]<Website: http://www.haworthpressinc.com>]

KEYWORDS. Goal attainment scaling, therapy, program evaluation,functional outcomes, children with special needs, rehabilitation

Many questions need to be considered when designing outcomeevaluation studies for children receiving occupational, physical, orspeech-language therapy, either in the community or in a health carecentre. One of the fundamental questions is whether to use a standard-ized or individualized measurement approach–or both.1,2 Individual-ized methods indicate whether single individuals have achieved thegoals of intervention. These methods also provide clear goals andpriorities for intervention, ensure the ongoing relevance of the child’sgoals, and reflect a client-centered perspective to service delivery.1,3

One of the most widely-used individualized approaches is goal at-tainment scaling (GAS),1 which provides an individualized, criterion-referenced measure of change. The GAS procedure involves: (a) defin-ing a unique set of goals for each child, (b) specifying a range ofpossible outcomes for each goal (on a scale recommended to containfive levels, from �2 to +2),4 and (c) using the scale to evaluate thechild’s functional change after a specified intervention period. As wehave used it, a score of �2 represents the child’s baseline level beforeintervention, �1 represents improvement that is less than the ex-pected level of attainment after intervention, 0 represents the expectedlevel of attainment after intervention, and +1 and +2 represent levelsof attainment that exceed expectations but represent outcomes that thechild is thought to be capable of achieving under favorable conditions.(Examples of these five-point GAS scales written for children receiv-ing occupational, physical, and speech-language therapy are presentedin Tables 1 to 3. The examples reflect subcategories of goals for threeareas (productivity, mobility, and communication), which are oftentargeted for intervention in the school setting.)

GAS was initially used to measure the impact of intervention in themental health field.5 Since then, it has been used widely to evaluatehealth services, educational programs, and social services.6,7 In 1979,GAS was considered the most popular outcome evaluation techniquein the human sciences.8

There are two main reasons for measuring outcomes in the field of

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 33

TABLE 1. Examples of Goal Attainment Scales for Children Receiving Occupa-tional Therapy Services to Address Classroom Productivity

��

��

�� !"#��$� � ��$��%��%�&�&�&�&'�� $� �� $��$��'�� $�� (�� ( ��&

��)$�� $�� $��(� ��&� �� $��&��$��

��

�� *(��$ �! � �� +� � �� !",��$� � ��$��%�� ( �� $� �� $��$��'

�� $�� (�� $��(� ��&��)$�� $�� $��&��$��

-*(��$�� +� � �� !"!.��$� � ��$��%��( �� $� �� $��$��'�� (�� $��(� ��

��)$�� $�� $��

� �� /! � �� +� � �� !"!.��$� � ��$��%��*(��$�� ( �� (��+ ��$� ��$ ��$��$��'

��0�� $�(��$�� (�� $��(� �� )$�� $

��

�� /� � �� +� � �� (��) � ��$��%��*(��$�� ( �� (��+ ��$ ��$ ��$��$��'

0��#�� $�(��$�� (�� $�(��$��%�� $�(��$�� )$�� $ �� + �� '

��

pediatric therapy: (1) to evaluate outcomes for a specific child (toimprove services to that child), and (2) to determine the effectivenessof a service or program as a whole.9 GAS can be used for bothpurposes–to document therapeutic change in individual children or toexamine change in groups of children. The focus of this article is onthe latter–the use of GAS in program evaluation studies.

GAS has been used in a number of studies evaluating pediatrictherapy services and programs, most fairly recent.10-17 Four studiestook place in early intervention settings and focused on the attainmentof motor development goals by infants or preschool children.14-17

Three other studies focused on the attainment of rehabilitation therapy

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


TABLE 2. Examples of Goal Attainment Scales for Children Receiving PhysicalTherapy Services to Address Mobility

��

�� 1 ��

�� 2��$��.�� & ��%�� ' 1 ��+� ��+�+ �� $�� $ ��& + �� $�� ()�)��( �� ,34�� $�( ��(�� &�� $ �� (� ��$�� %� � �(� �� $ �� $�� $�(��

+��"�++�+�� ((� �$'

�� *(��$ �! 2��$��.�� & ��%�� ' 1 ��+� ��+�+ �� $�� $ ��& + �� $��

��()�)��( �� .��,�� $�( ��(�� &�� $�� (� ��$�� % ��$�� $�(��+��"�++

+�� '

-*(��$�� 5 2��$��.�� & ��%�� ' 1 ��+� ��+�+ �� $�� $ ��& + �� $�� ( �� 4�� $�( ��(�� &�� $ �� (� ��$ ��%(��+�� $ �� "�++�+�� '

� �� /! 2��$��.�� & ��%�� ' 1 ��+� ��+�+ �� *(��$�� $�� $ ��& + �� $��

�� ( �� 4�� $��( ��(�� &�� $�� (� ��$��

�� /� 2��$��.�� & ��%�� ' 1 ��+� ��+�+ �� *(��$�� $�� $ ��& + �� $�� $�(��$��

�� ( �� 4�� (�� &��$�(��$�� $�(��$��%��

��(� ��"�� '

goals by school-aged children.10,11,13 A recent study by Brown et al.12

examined the effects of physical therapy intervention on the attain-ment of gross motor goals in individuals ranging from 3 to 30 yearswith severely limited physical and cognitive abilities. Thus, there isgrowing evidence that GAS is a useful way to measure therapeuticchange. GAS has shown that children receiving pediatric therapy in-tervention achieve goals in a variety of spheres–classroom productiv-ity, mobility, and communication. Still, questions remain about how toproperly conduct GAS and concerns are often raised about its reliabil-

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 35

TABLE 3. Examples of Goal Attainment Scales for Children ReceivingSpeech-Language Therapy Services to Address Communication

��

�� (�� $�6 �$�� (�� $�6 �$�� (�� $�6 �$��+��$� �� $�6�� 7�� $�6��

�� 6 �$��88�99��$�� 6 �$��88�99��$�� 6 �$��88+99��$��,4:�� +�� $�(�� $�(�� $�( ��%�3�3& ,5:�� $�( �� !;��+��5� �� ' �(��( �� ,4:�� %�3�3&�!;��+

%�3�3&�,��+�!5� �� 5� �� '� �'

�� *(��$ �1 6 �$��88�99��$�� 6 �$��88�99��$�� 6 �$��88+99��$�� ,4:�� $ +�� $�(�� $�(��

( �� ,5:�� $�� $�� ,4:��

-*(��$�� 5 6 �$��88�99��$�� 6 �$��88�99��$�� 6 �$��88+99��$��,4:�� +�� $�(�� $�(�� $�� ,5:�� $��

� �� $�� ,4:��

� �� /! 6 �$��88�99��$�� 6 �$��88�99��$�� 6 �$��88+99��$��*(��$�� ,4:�� +�� $�(�� $�(��

�(�� ,5:�� (��(�� ,4:��

�� /� 6 �$��88�99��$�� 6 �$��88�99��$�� 6 �$��88+99��$��*(��$�� ,4:�� +�� $�(�� $�(��

�(�� ,5:�� $�� (�� ,4:��

ity and validity.8,18 The appropriate use of GAS depends on a clearunderstanding of its strengths and weaknesses19 and thoughtful deci-sion-making concerning the issues that arise when using GAS in pe-diatric settings.

The aim of this article is to assist potential users to decide whetheror not to use GAS and to provide information about how to implementGAS with a minimum of bias. Practical tips and guidelines are pre-sented based on our combined experience with five studies that usedGAS in three types of pediatric settings: (a) a multidisciplinary,school-based therapy program encompassing occupational, physical,and speech-language therapy services,10,11 (b) physical therapy ser-vices for infants addressing mobility and other developmentalgoals,15,16 and (c) a residential setting for children and adults with

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


severely limited physical and cognitive abilities.12 These studies wereconducted in the United States or in Canada. Based on feedback fromtherapists involved in these studies, our own observations, and guide-lines in the general GAS literature, we have devised standard criteriaand procedures for the application of GAS in the formal evaluation ofpediatric therapy programs.

Other articles have provided information on the technical aspectsof using GAS, specifically the steps involved in setting the scales andrating outcomes.13-23 Most of these articles have dealt with the ap-plication of GAS in non-therapeutic settings, such as special educa-tion services,20,22 human services programs,21 and mental healthservices.23 On a practical level, the most useful publications arethose by Kiresuk and Lund,24 who discuss typical errors in creatingGAS scales and answer commonly-asked questions about biases ingoal setting and rating, and Smith,23 who provides guidelines aroundwho should set goals and who should rate goal attainment.

To date, publications have not addressed the specific issues thatarise in using GAS in pediatric therapy: Issues such as how to ap-proach goal selection and goal definition for children with specialneeds, and where to set the baseline on the GAS scale when a childhas a progressive or non-progressive condition. General guidelinesfor the use of GAS (such as those laid out by Kiresuk et al.6) must beadapted to particular intervention settings and populations because ofthe different issues that arise in each setting.25 The present articleidentifies key decision points and provides criteria and guidelines forthe systematic use of GAS in pediatric therapy settings, includingschool-based therapy programs and early intervention programs.

DECIDING WHETHER OR NOT TO USE GASTO EVALUATE PEDIATRIC THERAPY PROGRAMS

A number of organizational conditions are necessary for the suc-cessful implementation of a program evaluation study using GAS.They include: (a) a motivated team whose members are committed tothe evaluation and who share a common drive toward improvement oftherapy services, (b) adequate orientation and training of therapists,(c) the availability of people to coach therapists in the proper applica-tion of GAS, so that both technical and practical issues are addressedin an integrated fashion, and (d) sufficient resources allocated to do the

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 37

job.24 We often have employed a study coordinator to oversee the datacollection and training and to ensure the technical quality of the GASscaling and rating procedures.

The pros and cons of using GAS have been outlined in many ar-ticles.1,6,8,13,15,16,18,19,23,26 A primary strength of GAS is its ability tomeasure change in performance, whereas most standardized measuresare discriminative tools designed to measure post-intervention status(based on norms for children without special needs) and have not beenvalidated as responsive to clinically significant change.6,16,19 Clinicalsignificance refers to the magnitude of an effect in real-world terms.27

GAS is criterion-referenced, rather than norm-referenced, making itpotentially responsive to small changes that are perceived by children,families, and teachers as important for daily function. GAS may beparticularly useful for children with low cognitive functioning, sincestandardized measures may not be sensitive to the small but meaning-ful changes targeted for these individuals.

Relatively few standardized measures address functional outcomesthat are appropriate for children with special needs within a contextsuch as school (e.g., children’s ability to walk from the bus to theschool classroom). Standardized assessments of function often aredesigned to measure a broad range of abilities. Some of these areasmay not reflect therapy goals and not be relevant to particular chil-dren. Pediatric studies provide a fair amount of evidence that GAS andparallel standardized measures provide scores that are only moderate-ly correlated with one another.10,11,15-17 For formal program evalua-tion purposes, we recommend the use of both GAS and standardizedmeasures.10,11

In addition to the ability to measure change in the performance ofindividual children, GAS has other advantages: clinical utility, rele-vance, client involvement, and acceptability.1,23 GAS is ideally suitedto collaborative goal setting between a therapist, child, parent, andother professionals (such as teachers). Its collaborative use reflects aclient- or family-centered approach to service delivery.1,3,28 Otherpotential advantages of GAS include: (a) improved clarity of therapyobjectives for both therapists and clients, (b) improved conceptualiza-tion and delivery of the intervention, (c) more realistic client andtherapist expectations of therapy, (d) increased client satisfaction, and(e) increased motivation of the client toward improvement, providedby the very existence of goals.1,6,13,26

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


There are a number of potential limitations in using GAS in aprogram evaluation study: (a) biases in goal scaling and rating canoccur, (b) training and standardized implementation procedures arerequired, which are time-consuming (when therapists are unfamiliarwith GAS), and (c) GAS can interfere with day-to-day practice be-cause, when conducting a program evaluation, therapists should notmodify a goal in the course of the intervention. There are two reasonsfor this. First, the study intervention period may not be sufficientlylong for change to be expected on a new or modified goal. Secondly,therapists may elect to change goals they discover that they cannotmeet, which undermines the utility of GAS.

The major drawback to GAS is the possibility of bias in the use ofthe tool, which can affect its validity.8,13,16,18 Unintentional bias canoccur in goal scaling (so goals are overly easy to attain) or in goalrating (showing children make improvements that are not in factreal). Reliability and validity can be improved, however, by compre-hensive training of raters, adequate definitions of the levels of goalattainment, and the use of multiple raters.6,18 As well, there areaspects of how services are delivered in pediatric settings that natu-rally reduce the possibility and extent of bias in goal scaling andrating. A collaborative goal setting model (a common feature of amultidisciplinary, family-centered approach to service delivery)helps to ensure that goal levels are meaningful and ratings are validbecause both are based on a consensus involving several individualswho are knowledgeable about the child and invested in ensuring thatthe child makes real gains.26,29 Thus, collaborative goal setting helpsto ensure that therapy goals are meaningful to the child and familyand not simply easy goals that therapists set on their own and can besure of attaining, which is a criticism raised by many.13,25

We have observed differences between the rehabilitation disciplineswith respect to the ease of writing appropriate functional goals. TheGAS scaling format appears easiest to apply for speech therapy goalsand harder to apply for physical therapy and occupational therapygoals. As well, in King et al.,11 speech-language pathologists found iteasier to set goals that could be integrated into the child’s function inthe school setting. We speculate that the established hierarchy of thedevelopment of speech sounds may assist speech-language patholo-gists in setting goals in the area of articulation. Physical and occupa-tional therapists needed to give more thought to the process of writing

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 39

the various goal attainment levels. For instance, physical therapistsfound it hard to set equal intervals between goal levels for goalstargeting unique mobility difficulties and found it hard to establishrelevant goal levels for high functioning children. Interestingly, therehas been relatively little application of GAS in the speech-languagearea. The majority of applications have dealt with physical and occu-pational therapy goals.12,14-17,29

HOW TO USE GAS APPROPRIATELY AND EFFECTIVELYIN A PEDIATRIC CONTEXT

Overview of GAS Procedure

We recommend that the child’s treating therapist participate in thegoal setting process and that a therapist not providing services to thechild do the goal rating. For each goal, the treating therapist provides awritten description of the child’s baseline level of performance. In afamily-centered service delivery model, this baseline level of perfor-mance would be vetted with clients (i.e., teachers, parents, and oftenthe children themselves). The treating therapist, in conjunction withclients, also provides a written description of the expected level ofperformance for the child at the end of intervention (corresponding tothe 0 rating). We have found it most effective for the other levels of thescale ( �1, +1, and +2) then to be written by the treating therapist inconjunction with a person trained to oversee the quality of the GASscales (i.e., a person who understands the steps of the scaling proce-dure, the necessary criteria, and pitfalls to avoid). The final step is apeer review of the GAS scales by the therapists for each disciplinewho do the ratings of goal attainment at the end of the intervention(with the assistance of a person well-versed in the GAS scaling meth-odology). If a formal program evaluation is being done, a researchassistant may be involved. However, any well-trained person couldhelp identify the goal attainment levels and assist in the peer review ofthe GAS scales.

Ensuring the Technical Quality of the GAS Procedure

One needs to ensure that the goal scales are reliable (i.e., that arating made by one therapist observing the child’s performance is

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


comparable to the rating made by a different therapist) and that biasdoes not occur in goal scaling (overly easy goals) or goal rating (im-provements that are not real). The reliability and validity of GASscales can be improved by various procedures: ensuring that treatingtherapists have a minimum level of experience so they can set realisticgoals in conjunction with children and parents; providing comprehen-sive training to therapists; using collaborative goal setting and peerreview in the goal selection phase; ensuring well-written goalsthrough training, peer review, and use of a standard procedure andchecklist; and using independent raters (i.e., raters who do not have apersonal investment in the outcome).

Table 4 outlines the questions that need to be considered in usingGAS effectively, criteria that should be met to ensure reliability andvalidity of the procedure, and procedures and tools that can be used tomeet the criteria. These questions and criteria are based on the au-thors’ experiences in using GAS and on recommendations in the liter-ature.

How Much Clinical Experience Is Necessary to Set AppropriateGoals? One year of full-time clinical experience in the setting ofinterest (a school-based therapy program or an early interventionprogram) is ideal. This amount of time provides therapists withenough exposure to different types of goals and different children sothat they can estimate the performance levels that children will mostlikely attain. We did not meet this criterion in our own study11 (only83% of therapists had over a year experience in providing school-based intervention) and realize that the experience level and numberof staff in the program implementing the evaluation are limitingfactors.

The ability to decide on the key variables that must change for goalattainment is a skill that improves with experience. Experienced thera-pists develop competency in assessing child and environmental factorsaffecting performance, are able to generate various ideas about pos-sible variables to change, and are able to focus in on the variable theybelieve is the most amenable to change (based on their past knowledgeand experience). For example, speech-language pathologists workingon communication goals in the area of articulation select particulartarget sounds based on knowledge of the developmental hierarchy inattainment of speech sounds, the child’s stimulation potential for thatsound in isolation, the importance of that sound in increasing the

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 41

TABLE 4. How to Use Goal Attainment Scaling Effectively

��

��

<�� 1 � �(�� $� ��+�! 7��*(� ��$�*(� �� +�(�$�� *(� ��3 � �(��$�(( �( ��= ��3

<�� $ 1 � �(�� $� ��(( �*��; ��1��4�+� �� +� �� +��(��+�� + ��$�$�� (��= ��3 ��$� ��(�3

<�� $�>�� 1 ��( ��+�� $��= �� ?

!3 �� +��$� �� -�(��

� � �3 ��% � �(��

�� $��

��@��

��$��

�� &

(� ��&��$"� � ��$'3

�3 ��+ ��(��

(��+��%�3�3&��+� ��+� � A��(�� +��

( �� ( ��$��)�)$�� 3

+��&� �� @� ��+��

�� $��+�� &� � �

��(�� )��$'3

<�� $�>�� -� ��+� �� $? -� � �� %�3�3&��$�>�� $� �� +� �� !3�� (��&�� )��$+��)(��'= �� ( ��$� ��$��?

�3� �(��+�� +� �

��$ 1 � �(��

03� �� ( ��

#3� �� (�� 6�� $�>��+� ��

�� 1 �� $?� ��

!3 �� +��

��+�� $��+ A��+��$� $��( �� ( ��$� ��$� ��

�3� ��(( �*��>�� $�>��+�� %�3�3&� �� + ��/!��/�� $ ��%�� !'

�� !&��3'

03 �(��+��(� ��$�+� ��

#3 �+��$��+� ��

%�� +��'&

��(��

43�� +�� $�(��$�� (��9��( ��%�� +�� (� ��+� ��'

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


TABLE 4 (continued)

��

��

<�� $�>�� !3�B�� $��$�� (�� A��$�(��$�� 3�� = ��CD��$��( ��$�� 3

�3�B�$�� 88(� +� ��$��$�E��+�DCDD� ��$�(��$�� 3

<�� 9��$��$�� $�� -�� $�� $= �� $��&�( �+� �� +� ��$��

��1)�� 3 ��$� ��+�� (��3

child’s intelligibility, and the consensus of the caregiver. Determininga child’s potential for change during the intervention period is basedon clinical judgment (i.e., experience, the underlying cause of thechild’s articulation disorder, the type of error with the targeted sounds,the number of other sounds that are in error).

How Much Instruction and Training Is Necessary for Therapists?Orientation and training of therapists is necessary for the successfuluse of GAS.13 The steps and time requirements included in Table 5 arebased on Kiresuk et al.’s6 recommendations and our own experiences.Many therapists play the dual role of treating therapist and ratingtherapist (only for children to whom they do not provide intervention),which requires approximately 12 hours of training.

The training procedures in Table 5 incorporate the recommenda-tions of therapists involved in our evaluation studies. For instance,therapists recommended that a list of common errors in creating GASscales (and solutions) would be useful (Table 6), as would an invento-ry of potential goals.

To develop skill in writing the scales, we recommend the use ofsmall group sessions in which therapists practice putting goals intoGAS format (i.e., identifying five levels of possible goal attainment).A series of case scenarios can be used for practice and discussion.Therapists also can practise putting goals in GAS format for two orthree children from their caseload. At this point, it is useful for them tohave access to a resource binder with examples of the five-point rating

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 43

TABLE 5. Orientation and Training of Therapists to the Goal Attainment ScalingProcedure

��1 � �(�� %�� '

� � ��

� <��$�� *��(��+�� (��+��$�� +��

1 ��1 � �(��F�� >��%0� �� '� �� (�( �� (��+�� +��

� ��)�)��$�� %�� '� ��$�� +�� G��$��

B��1 � �(��F��B�� >��%#� �� '� B�� (�� +�� +��

� B��+�� ( ��$� �� <��$��

��%!� �� '� G��$��

scale applied to goals in particular areas (e.g., mobility goals, commu-nication goals).

How Can One Ensure Adequate Goal Selection? It is important thatthe selected goals are meaningful to the child and family and reflectthe primary focus of therapy for the child. We therefore recommendthat treating therapists be involved in goal selection rather than inde-pendent goal setters.30

How Can One Ensure Adequate Goal Scaling? The literature refersto six criteria for good goal writing: Goals should be relevant, under-standable, measurable, behavioral, attainable, and time-limited.13,18

We have gone beyond these general criteria to specify criteria relevantto each of the goal levels and those dealing with the properties of thescale as a whole (see Table 4).

Three criteria concerning the scale as a whole require more ex-planation. First, although the GAS scale examples in Tables 1 through3 did not specify a time period, it should be noted that a standard time

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


TABLE 6. Common Errors in Creating GAS Scales

��

�� $ 7+� ��*(��$��%�3�3&�5��'��+�� 1 ��*(��$��+�� %�3�3&�88�� $��

�� $��(� ��$�� (��%�3�3&�88�� E'&��$�++�� (��+� �� !55�� *��(�� $�� (��&� � �+� �� 3 �� E'3

�� 1� �� (��+�� &�� " � �( �+�� (��+� �� +

�� +�� 3 ��( �+�� $�� 3

��(��H� �� $�� 2��$�� +�� +� ��3�1 ��$��( ��+� � �� (� +� ��

�� *(��$�� $� ��$�� 3�7+�� 3 $��&� ��$��

��+��&�� $ �� &( ��$�$�� $�++� �� 3

A��>�� $�� + ��+� ��>�� 7�� &��&� � ��+� ��3

/!��$�/�� +� �� $�!��3

��7 �� $�� 1 ��+� �� A� �� +� �� ( ��+� �� $�� $�� $�� $��

��$�%�3�3&� ��+� �� +� � �� $3�� '�� +� �� +� � �� $�%�3�3&� ��+ �� '3

A��2�++� �� $��( ��$1��%�3�3&�6��& �� $�� ( ��&�� $� �+� 6 ��&�� ' �� &�� $��+�� $�� 3 $�++� ��(��%�3�3&�88��3�3�399'3��

B�$��$�� $��$ �� +�� 7��(�� $�� $��$�� (��3�� %�3�3&� ��/!�� $�� +�� $��%�3�3&�/!��$��

�(��+��$��88#5��$�45�� 99��$� � 88�� #5�� $��(��45��/��(��+��$��8845��$ �� E��$�/��$��88�� E'3�7+�� $��*��45�� & 45�� $��(��.5�� E'�� /!��$� ��/��$�� (��+�� 3�� $&��(��$�� %�3�3&��+�� $��( �� $��$�� $��(��&�� $��%�3�3&� ��/! �� $�� '��(��+��$��88#5��$ ��+��+�345�� E��$� ��/��(��+��$��88.5��$�;5�� EI��+�� $��44�� &�� /!�� /�� '3

�� 7��$�++�� * �� +�� +��&��(�� (��(�� 3�7+�� $� �((�� 3� ��((� �� * ��&��$��(�� $9��(� +� ��3

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 45

period (the length of the intervention period as prescribed by the studydesign) was set for all goals.

An example may clarify criterion 4. The following goal includesthree possible variables of change, namely distance, time, and level ofassistance: ‘‘Walks 100m with platform walker in 8 minutes with twohands on walker to assist with steering.’’ In order to write an appropri-ate scale, the therapist must decide which is the most important vari-able of change. That variable would then be altered in each of thewritten goal levels, with the other variables held constant.

With respect to criterion 5, if physical assistance is required, thisshould be explicitly stated in the written goal, either as a constantfactor or as the variable that changes over time. The key idea is thatgoal attainment levels should reflect change in the child’s behavior,not unacknowledged variations in the therapist’s physical assistance. Itis permissible to write goals where the physical assistance of someoneelse is explicitly stated as the variable that changes over time, so thatthe goal scale shows meaningful changes in the level of assistance thechild requires to perform a task. An example of the gradations thatcould be included in a scale focusing on changes in level of assistanceis: physical and verbal assistance required (�2), verbal assistancewith checklist required (�1), checklist and verbal cueing or prompt-ing required (0), verbal cueing/prompting required (+1), and com-pletely independent (+2).

According to strict research methodology,8 the therapist who setsthe levels of goal attainment should not be the same therapist whoprovides the treatment. This is a difficult criterion to meet since, inactual clinical practice, the treating therapist is involved in goal set-ting, often in conjunction with the client.10,11,23 This is defensiblewhen one involves an independent rating therapist and a trainer/studycoordinator in the review of the goal attainment levels (who follows astandard procedure with set criteria). In our experience, even highlyexperienced therapists have some difficulty identifying the baselineand expected levels of the goals. Assistance and review by others isuseful, appreciated, and necessary. Figure 1 provides a checklist thatcan be used in the review of written goals by a trainer and independentrating therapist.

How Can One Ensure Adequate Goal Rating? Cardillo30 addressesthe selection of raters and the decision about the timing of the goalrating session, but provides little information on how to conduct the

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


FIGURE 1. Goal Attainment Scaling Checklist–Goal Review Procedure

J��+�6� ��(��?

� 1 � �(��?�-*(��$��%�3�3&�� +�5'

��

�� !��

1 ��+� �� (� ��3

1 � �� (( �*��>�� 3

1 � ��(� ��$�+� �� +� ��3

1 �� +��$��+ ��%� &��+��+��&�� +��$��+� ��'3

� ��"��

�� !��

��

�(��+�� +� �� $

�� ( ��

�� (��

actual goal rating session. Our experiences have led to some recom-mendations. First, it is important to consider the child’s view of therating situation. When children are aware that a new person is comingto watch a session, they may be very motivated to perform well forthis visiting person. The treating therapist can reduce the ‘‘hype’’regarding the rater’s visit by informing the child in advance of the visitand assuring the child that his/her regular performance is what iscalled for. The rating therapist should minimize the effect of his/herpresence by sitting quietly in the back of the room and making notesdiscreetly.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 47

A second issue concerns the behavior of the treating therapist inthe session. For goals whose attainment cannot be observed undernaturally occurring circumstances at school, we have found it mostappropriate for the treating therapist to interact with the child andrequest performance of the behavior outlined in the goal. (Somegoals require that the child be set to the task because of infrequentnaturally occurring opportunities to display the behavior.) The treat-ing therapist begins by orienting the child to the task. If prompting,cueing or some other support is required, the therapist starts with the0 level of the scale and prompts for performance up or down the scaledepending on the child’s success at the 0 level. Brown et al.12 alsoused a prompting procedure and allowed up to three trials per goal(for individuals with severely limited physical and cognitive abili-ties). Similarly, we have found that children may need more than oneattempt to demonstrate their true ability. Some children react to thepresence of the rating therapist by showing silly behavior and two orthree attempts are required before they calm down and demonstratetrue performance.

How Is the Summary Score Determined? For program evaluationpurposes, users need to calculate an appropriate summary score toreflect the overall goal attainment of children in the therapy or earlyintervention program. The recommended procedure is to convert chil-dren’s outcome scores into aggregate T-scores (see Cardillo andSmith,4 for a complete discussion of T-scores and other summaryscores). T-scores can be calculated using a statistical computer soft-ware package such as the Statistical Package for the Social Sciences(SPSS). Mean aggregate T-scores facilitate reliability analyses andcomparisons across children, and provide an overall evaluation ofchildren’s performance. Another advantage of using T-scores is thatthey can be compared to other standardized scores.

T-scores can be computed using the formula developed by Kiresukand Sherman:5

T = 50 +(10 � WiXi)

� (1 � r) �Wi 2 + r (�Wi 2)

In this formula, 50 represents the mean and 10 is the standarddeviation. Wi represents the weighing for a particular goal and Xi

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


represents the score for each goal. The r represents the expected over-all intercorrelation among outcome scores. The formula for computingthe T-score assumes a relatively low correlation among goals of .30.Kiresuk and Sherman5 found this correlation useful because it yields astandard deviation of 10 units. Cardillo and Smith4 strongly recom-mend against the use of differential weighing of GAS goals. The useof this formula may appear time-consuming and difficult, but the needfor manual computation is rare. If goals are not weighted and thesuggested intercorrelation of .30 is used, tables are available that allowthe quick and easy conversion of outcome scores into T-scores forclients with up to eight scored scales (see Kiresuk et al.6).

Conceptual Issues Arising in the Use of GAS

Validity Issues: How to Ensure that Goal Attainment Is Real. Twobasic strategies can be used to address the issue of validity:(1) supplement the use of GAS with measures that provide moreconventional estimates of post-treatment status (i.e., standardizedmeasures) to provide a more comprehensive assessment of outcome,6or (2) employ randomly selected control goals (after Brown et al.12).In Brown et al., therapists created GAS scales both for treatmentgoals that were practiced in the physical therapy setting and forcontrol goals that were set but not addressed in practice. The sub-jects’ progress on treatment goals was significantly greater than theirprogress on the control goals, suggesting that the therapy interven-tion was the factor contributing to improved goal attainment and thatthe goal attainment was real.

How to Set the Baseline in the GAS Scale. This is an importantconsideration. When no deterioration is expected in children’s perfor-mance, such as when the child has a non-progressive, chronic condi-tion, �2 can be defensibly used as the child’s baseline.10,11 Whenevaluating goal attainment of individuals with severely limited physi-cal and cognitive abilities, Brown et al.12 used �2 as a baseline butcaptured lower performance on a test day by using a score of �3.When evaluating the performance of children with progressive condi-tions who may deteriorate in function over time, it would make senseto set the baseline at �1. If the expectation is that decreased perfor-mance will occur and the goal is to minimize performance loss, itwould make sense to write goal levels in terms of gradations of dimin-ished performance.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 49

How to Best Demonstrate Reliability. It is important to report thereliability of the goal ratings. There are various types of reliabilityincluding inter-rater stability (independent raters’ agreement overtime) and inter-rater reliability (independent raters’ agreement atthe same point in time).8 In King et al.,10 we computed a measure ofinter-rater stability, which involved correlating T-scores based onthe ratings of the treating and rating therapists obtained on separateoccasions. We found a correlation of .6, which indicates a moderatelevel of agreement between the rating therapists and the treating thera-pists on the amount of improvement the children made on their goals,and is typical of estimates of stability obtained when ratings by differ-ent individuals are obtained on different occasions.6 In our secondstudy, we felt it was more appropriate to obtain a measure of inter-rater reliability from two independent rating therapists who ratedthe goals on the same occasion.8 Using this procedure, the inter-ra-ter reliability (Intraclass Correlation Coefficient) was .98. To demon-strate that the GAS scales are reliable, we recommend that othersexamine and report inter-rater reliability estimates from two inde-pendent raters.

Brown et al.12 computed measures of inter-rater reliability betweena treating therapist who directly observed levels of goal attainment in aphysical therapy setting and an independent rater who scored goalattainment based on videotapes of the same sessions. This is a goodway to reduce performance demands or reactive behavior created bythe presence of an unfamiliar, independent rater.

What Does It Mean When the Change Exceeds Therapists’ Expecta-tions? Studies have found that GAS ratings are often higher than theexpected level of 0 on the 5-point scale.10,11,15,16 Since the ratingswere reliable, it appears that finding change exceeding therapists’expectations is not due to biases in the rating procedure, but rather inthe goal setting. There is therefore fairly strong evidence that expectedoutcomes may be underestimated systematically by therapists. Themultiple reasons for this are hard to disentangle. Therapists may setlow levels of expected goal attainment to be cautious and ensuresuccess for the child. It is also possible that therapists underestimatethe therapy gains that children can make, which may be due to notfully taking children’s motivation into account. The underestimationof expected outcomes should be discussed with therapists during train-ing in future studies.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


How to Approach Goal Selection and Definition. When selecting agoal, therapists need to be aware of the underlying factor that is beingtargeted for change–whether the major limiting factor is the child’sendurance or motivation, for example, or aspects of the physical envi-ronment. Goal definition therefore is based on the therapist’s workinghypothesis of the best way to assist the child to perform the task.

The therapist’s hypothesis about the major factor limiting the child’ssuccessful performance is sometimes evident from looking at the goalscale that is set. We have found that when the variable being changeddeals with a quality (such as accuracy) or a quantity (such as number ofletters formed correctly), then the working hypothesis seems to deal withsome aspect internal to the child. On the other hand, when the variablebeing changed deals with the level of physical or verbal assistance pro-vided, or levels of equipment support, then the variable of change corre-sponds to the factor of critical importance in the therapist’s view. Underthese conditions, the written goal levels provide a window on the thera-pist’s view of the factors limiting the child’s goal achievement.

Implications for Therapists and Managers

This article should help therapists and managers make informeddecisions about whether or not to use GAS, based on an understandingof the questions they need to ask and the requirements for using GASappropriately and effectively in pediatric program evaluation studies.Moreover, the criteria and procedural guidelines presented here shouldprovide useful assistance to those using GAS in pediatric settings.Information from GAS can to be used to improve programs, to assistin meeting accreditation requirements, and to demonstrate account-ability to governing bodies and funders.

GAS has been criticized due to its potential for bias when imple-mented without thought and care.18 We have provided a more hopefulor balanced perspective here. GAS can be implemented appropriatelywhen guidelines and standard procedures are used. Under these condi-tions, it is well worth the effort to use GAS to evaluate pediatrictherapy programs. Service providers, managers, parents, and childrenthemselves benefit from the knowledge provided by goal attainmentscaling.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

King et al. 51

REFERENCES

1. Russell D, King G, Palisano R, Law M. Measuring individualized outcomes(Research Report No. 95-1). Hamilton, ON: McMaster University and Chedoke-McMaster Hospitals, CanChild Centre for Childhood Disability Research; 1995.

2. Russell D, King G, Palisano R, Law M. Measuring individualized outcomes.Proceedings of the American Academy for Cerebral Palsy and Developmental Medi-cine. 1996; 25.

3. Zaza C, Stolee P, Prkachin K. The application of goal attainment scaling inchronic pain settings. Journal of Pain and Symptom Management. 1999; 55.

4. Cardillo JE, Smith A. Psychometric issues. In: Kiresuk TJ, Smith A, CardilloJE, eds. Goal attainment scaling: Applications, theory, and measurement. Hillsdale,NJ: Lawrence Erlbaum Associates; 1994; 173-212.

5. Kiresuk TJ, Sherman RE. Goal attainment scaling: A general method for eval-uating comprehensive community mental health programs. Community Ment HealthJ. 1968; 4:443-453.

6. Kiresuk TJ, Smith A, Cardillo JE. Goal attainment scaling: Applications,theory, and measurement. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994.

7. MacKay G, Somerville W, Lundie J. Reflections on goal attainment scaling(GAS): Cautionary notes and proposals for development. Educational Research.1996; 38:161-172.

8. Cytrynbaum S, Ginath Y, Birdwell J, Brandt L. Goal attainment scaling: Acritical review. Eval Q. 1979; 3:5-40.

9. Law M, King GA, MacKinnon E, Russell DJ. Quality performance: Design-ing clinical services around person-centered outcomes. In: Gardner JF, Nudler S, eds.Quality performance in human services: Leadership, values, and vision. Baltimore,MD: Paul H. Brookes; 1999; 81-106.

10. King G, Tucker M, Alambets P, Gritzan J, McDougall J, Ogilvie A, Husted K,O’Grady S, Brine M, Malloy-Miller T. The evaluation of functional, school-basedtherapy services for children with special needs. A feasibility study. Phys Occup TherPediatr. 1998; 18: 1-27.

11. King G, McDougall J, Tucker MA, Gritzan J, Malloy-Miller T, Alambets P,Cunning D, Thomas K, Gregory K. An evaluation of functional, school-based thera-py services for children with special needs. Manuscript submitted for publication;1999.

12. Brown DA, Effgen SK, Palisano RJ. Performance following ability-focusedphysical therapy intervention in individuals with severely limited physical and cogni-tive abilities. Phys Ther. 1998; 78:934-947.

13. Clark MS, Caudrey DJ. Evaluation of rehabilitation services: The use of goalattainment scaling. Int Rehabil Med. 1983; 5: 41-45.

14. Maloney FP, Mirrett P, Brooks C, Johannes K. Use of goal attainment scalingin the treatment and ongoing evaluation of neurologically handicapped children. AmJ Occup Ther. 1978; 32: 505-510.

15. Palisano RJ, Haley SM, Brown DA. Goal attainment scaling as a measure ofchange in infants with motor delays. Phys Ther. 1992; 72: 432-437.

16. Palisano RJ. Validity of goal attainment scaling with infants with motor de-lays. Phys Ther. 1993; 73: 651-658.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.


17. Stephens TE, Haley SM. Comparison of two methods for determining changein motorically handicapped children. Phys Occup Ther in Pediatr. 1991; 11(1): 1-17.

18. Ottenbacher KJ, Cusick, A. Goal attainment scaling as a method of clinicalservice evaluation. Am J Occup Ther. 1990; 44: 519-525.

19. Ottenbacher KJ, Cusick A. Discriminative versus evaluative assessment:Some observations on goal attainment scaling. Am J Occup Ther. 1993; 47: 349-354.

20. Carr RA. Goal attainment scaling as a useful tool for evaluating progress inspecial education. Except Child. 1979; October: 88-95.

21. Kiresuk TJ, Lund SH. Goal attainment scaling. In: Atkisson C, ed. Evaluationof human service programs. New York, NY: Academic Press; 1978; 341-370.

22. Shuster SK, Fitzgerald N, Shelton G, Barber P, Desch S. Goal attainment scal-ing with moderately and severely handicapped preschool children. Journal of the Di-vision for Early Childhood. 1984; Winter: 26-37.

23. Smith A. Introduction and overview. In: Kiresuk TJ, Smith A, Cardillo JE,eds. Goal attainment scaling: Applications, theory, and measurement. Hillsdale, NJ:Lawrence Erlbaum Associates; 1994; 1-14.

24. Kiresuk TJ, Lund SH. Implementing goal attainment scaling. In: Kiresuk TJ,Smith A, Cardillo JE, eds. Goal attainment scaling: Applications, theory, and mea-surement. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994; 119-134.

25. Lewis A, Spencer JH, Haas GL, DiVittis A. Goal attainment scaling: Rele-vance and replicability in follow-up of inpatients. J Nerv Ment. 1987; 175: 408-417.

26. Mitchell T, Cusick A. Evaluation of a client-centered paediatric rehabilitationprogram using goal attainment scaling. Australian Occupational Therapy Journal.1998; 45: 7-17.

27. Bain BA, Dollaghan CA. Clinical forum: Treatment efficacy. The notion ofclinically significant change. Language, Speech, and Hearing Services in Schools.1991; 22: 264-270.

28. Rosenbaum P, King S, Law M, King G, Evans J. Family-centered service: Aconceptual framework and research review. Phys Occup Ther Pediatr. 1998; 18:1-20.

29. Stolee P, Zaza C, Pedlar A, Myers AM. Clinical experience with goal attain-ment scaling in geriatric care. Journal of Aging and Health. 1999; 11: 96-124.

30. Cardillo JE. Goal setting, follow-up, and goal monitoring. In: Kiresuk TJ,Smith A, Cardillo JE, eds. Goal attainment scaling: Applications, theory, and mea-surement. Hillsdale, NJ: Lawrence Erlbaum Associates; 1994; 39-59.

Phys

Occ

up T

her

Pedi

atr

Dow

nloa

ded

from

info

rmah

ealth

care

.com

by

Uni

vers

ity o

f B

ritis

h C

olum

bia

on 0

9/15

/10

For

pers

onal

use

onl

y.

goal attainment scaling: its use in evaluating pediatric ... attainment scaling.pdf · goal...

Documents