Evidence for whom?: ASHA’s National OutcomesMeasurement System

Robert Mullen*

American Speech-Language-Hearing Association, 10801 Rockville Pike,

Rockville, MD 20852-3279, USA

Received 20 November 2003; accepted 6 April 2004

Abstract

The growth of managed care and increasing demands by school systems resulted in of a number of

new audiences looking for the evidence underlying the practice of speech-language pathology (SLP).

Third-party payers, operating in an environment emphasizing ‘‘return-on-investment,’’ sought data

linking expenditures on SLP services with tangible outcomes affecting resource utilization, such as

required level of care and ability to discharge. ASHA’s National Outcomes Measurement System

(NOMS) was developed to address these types of needs. This paper provides an overview of NOMS

and its intended uses in the field.

Learning outcomes: (1) Readers will understand the rationale behind the development of NOMS.

(2) Readers will be aware of the strengths and limitations of NOMS.

# 2004 Elsevier Inc. All rights reserved.

Keywords: Evidence-based practice; National Outcomes Measurement System; Functional Communication

Measures

Spurred on by the growth of managed care, the 1990s saw the emergence of a number of

new audiences looking for the evidence underlying the practice of speech-language

pathology. Third-party payers, operating in an environment emphasizing ‘‘return-on-

investment,’’ sought data linking expenditures on SLP services with tangible outcomes

affecting resource utilization, such as required level of care and ability to discharge. In

school settings, an increased emphasis on accountability led administrators, principals, and

teachers to look for evidence that SLP intervention translated into observable improvement

Journal of Communication Disorders 37 (2004) 413–417

* Tel.: þ1 301 897 5700x4265.

E-mail address: rmullen@asha.org (R. Mullen).

0021-9924/$ – see front matter # 2004 Elsevier Inc. All rights reserved.

doi:10.1016/j.jcomdis.2004.04.004

in childrens’ classroom performance. In both settings, SLPs and administrators sought

data that they could use to advocate for staffing levels and to educate patients and their

families.

What united these quite diverse data needs was the fact that they almost always

involved the presentation of data to non-SLP audiences. This called into question the

extent to which data from clinical SLP research could be effectively used to address these

evidence needs. While clinical research may help us understand how to get a 4-year-old

girl from a Mean Length of Utterance (MLU; Brown, 1973) of 1.5 to an MLU of 4.0,

what the parents need to know is that this means the difference between their daughter

saying ‘‘spoon’’ versus ‘‘I dropped my spoon’’ or ‘‘I need a spoon.’’ Clinical research may

provide evidence that, using a certain protocol under certain conditions with certain

patients, SLPs can typically get those patients from a 0 to an 8 on the Complex Ideational

Material sub-score of the Boston Diagnostic Aphasia Examination (Goodglass, Kaplan, &

Barresi, 2000). While an SLP would know that this suggests that the person is much

more likely to be able to successfully live at home, it may not be terribly meaningful to a

payer.

In 1994, the American Speech-Language-Hearing Association (ASHA) established a

Task Force to develop within the Association a clearinghouse for functional outcomes

data. The clearinghouse would serve as a repository for such data, which could then

be analyzed and disseminated to clinicians as a way to meet these evidence needs.

Unfortunately, psychometrically sound, functional, and clinically meaningful measure-

ment systems could not be found, and ASHA undertook development of its own

outcomes system, which came to be known as the National Outcomes Measurement

System (NOMS).

NOMS is the umbrella term for three separate but related nationwide data collection

systems for speech-language pathology treatment outcomes. As it was the arena of adult

health care in which SLPs most acutely recognized an unmet need for outcomes data, the

first data collection system to be developed and launched was the Adult component of

NOMS. Data collection for the Adult component began in late 1998, followed by the Pre-

Kindergarten (ages 3–5) component and Schools component in 1999.

All three NOMS components have a similar structure. When a patient is admitted to SLP

services, the SLP submits an Admission form, via the Internet or as a scannable document,

which captures basic demographic and diagnostic data about the individual receiving

services. Additionally, the SLP scores the patient on one or more of a series of seven-point

scales to measure the patient’s level of functional communication. These scales, referred to

as the Functional Communication Measures (FCMs), are disorder-specific (see Table 1).

The SLP scores the patient on whichever FCM(s) correspond to the patient’s treatment

plan.

When the patient is discharged from SLP services, the SLP completes a Discharge

form. This form is used to record data about the amount and type of services provided

and to record discharge scores on the same FCMs previously gathered with the

Admission form. Movement along the FCM scales from admission to discharge is

used as the outcome measure, and it is compared to the patient characteristics and

service delivery variables in order to learn more about which factors are associated with

positive and negative outcomes. Participating sites have access to an on-line database

414 R. Mullen / Journal of Communication Disorders 37 (2004) 413–417

from which they can generate reports of their site’s data as well as comparable national

data.

1. Methodological limitations and strengths

1.1. Limitations

1.1.1. Patient controls

In contrast to experimental study designs, the case series approach utilized in NOMS

does not contain any control groups. Because there are no absence-of-treatment or placebo

groups against which to compare the outcomes of the patients who did receive treatment, a

causal link between treatment and outcome cannot be established.

1.1.2. Treatment protocols

Recording of detailed descriptions of treatment protocols are beyond the scope of

NOMS, so little can be learned about the relative effectiveness of different approaches.

Further study is needed to ascertain the usefulness of the FCMs as appropriate outcomes

measures in effectiveness studies.

1.1.3. Reliability

Movement away from strictly clinical measures to more functional measures necessarily

introduces more subjectivity into the scoring of the FCMs. Reliability is much more easily

established on a test which measures how often a patient successfully completes eight

well-defined tasks, for example, than for a measure of the extent to which a student’s

participation in the classroom is limited by his speech production difficulties. In addition,

the clinician who is responsible for the patient’s treatment determines the ratings. In most

cases, that means the SLP is also being evaluated in part on the extent to which his/her

patients make progress, providing an unfortunate incentive for inflated scoring.

Table 1

Adult Functional Communication Measure: attention

Level Ability

1 Attention is non-functional. The individual is generally unresponsive to most stimuli.

2 The individual can briefly attend with consistent maximal stimulation, but not long enough to

complete even simple living tasks.

3 The individual maintains attention over time to complete simple living tasks of short duration

with consistent maximal cueing in the absence of distracting stimuli.

4 The individual maintains attention during simple living tasks of multiple steps and long duration

within a minimally distracting environment with consistent minimal cueing.

5 The individual maintains attention within simple living activities with occasional minimal cues

within distracting environments. The individual requires increased cueing to start, continue,

and change attention during complex activities.

6 The individual maintains attention within complex activities, and can attend simultaneously to

multiple demands with rare minimal cues. The individual usually uses compensatory strategies when

encountering difficulty. The individual has mild difficulty or takes more than a reasonable amount of

time to attend to multiple tasks/stimuli.

R. Mullen / Journal of Communication Disorders 37 (2004) 413–417 415

1.2. Strengths

1.2.1. Local data

This approach affords sites, whether health-care facilities or school systems, the

opportunity to utilize findings based on their own patients, eliminating concerns about

the extent to which studies in the scholarly literature are applicable to their specific

circumstances. National data from similar treatment settings are also provided to provide

context in which to interpret a site’s data. Additionally, the data are based on functionally-

written FCMs, which non-clinical audiences should find easier to understand than many

clinical measures in widespread use. Finally, NOMS is an approved outcomes measure-

ment system under the Joint Commission on the Accreditation of Healthcare Organiza-

tions’ (JCAHO) ORYX Initiative, allowing participating facilities to use NOMS in meeting

their accreditation requirements.

1.2.2. Statistical power

As of the Fall 2003, the Adult NOMS database contained records of over 100,000 patient

stays, with the Schools database containing approximately 15,000 records, and 8000 in

Pre-Kindergarten. These large data sets provide ample statistical power for most descrip-

tive analyses.

1.2.3. Hypothesis generation

While NOMS data cannot conclusively demonstrate a direct causal link between SLP or

audiologic interventions and patient outcomes, the data are a valuable starting point for the

development of hypotheses for testing using experimental study designs.

2. Conclusion

A critical analysis of the evidence supporting clinical practice in speech-language

pathology, and alignment of practice with this evidence remains at the core of evidence-

based practice. However, the past decade has seen the emergence of interested parties from

outside of the profession who are also demanding evidence. For these other groups, it will

not be enough to demonstrate that SLP interventions are research-based. Rather, they will

need to be convinced that these interventions result in tangible improvements in patients’

lives. Neither NOMS’ case-series approach nor most other clinical research are ideally

suited to addressing these evidence needs. Treatment effectiveness research combining

functional patient outcome measures with scientifically rigorous controls will be needed to

meet these challenges.

Appendix A. Continuing education

1. The National Outcome Measurement System was developed as an answer to:

a. Third-party payers’ desire to see tangible outcomes of SLP services

b. The emergence of evidence-based practice

416 R. Mullen / Journal of Communication Disorders 37 (2004) 413–417

c. An increased emphasis on accountability within the school systems

d. A need for additional measures that could be used to educate patients and their

families

e. All of the above

2. The third-party payer is:

a. Typically not a SLP

b. May not appreciate the results of clinical research

c. Typically unaware of the correlation between a test score and functional ability

level

d. Looking for a ‘‘return on investment’’

e. All of the above

3. The NOMS is:

a. A clearing house for functional outcomes data

b. Used to meet the need for evidenced-based practice

c. Both a and b

d. Developed by the American Psychological Association

e. Based on available psychometric and clinical data

4. NOMS collects data in the areas of:

a. Adult treatment

b. Adolescent treatment

c. School-age treatment

d. Kindergarten treatment

e. Both a and c

5. Strengths of NOMS include:

a. The ability to establish a causal link between treatment and outcomes

b. May be used to meet accreditation requirements

c. Provides ample statistical power for descriptive analysis

d. Both b and c

e. Functional Communication Measures are scored subjectively

References

Brown, R. (1973). A first language: The early stages. Cambridge, MA: Harvard University Press.

Goodglass, H., Kaplan, E., & Barresi, B. (2000). Boston Diagnostic Aphasia Examination (3rd ed.). New York:

Lippincott, Williams, Wilkins.

R. Mullen / Journal of Communication Disorders 37 (2004) 413–417 417