evaluation methods in disease management studies

Dis Manage Health Outcomes 2008; 16 (5): 365-373EXPERT OPINION 1173-8790/08/0005-0365/$48.00/0

© 2008 Adis Data Information BV. All rights reserved.

Evaluation Methods in DiseaseManagement Studies2004–07

Thomas Wilson,1,2 Martin MacDowell,1,3 Patricia Salber,1,4 Gary Montrose1,5 and Carolyn Hamm1,6

1 Population Health Impact Institute, Loveland, Ohio, USA2 Trajectory Healthcare, LLC, Loveland, Ohio, USA3 UIC College of Medicine at Rockford, Rockford, Illinois, USA4 Universal American Corp., Houston, Texas, USA5 Montrose Healthcare Strategies, LLC, Denver, Colorado, USA6 Walter Reed Army Medical Center, Washington, DC, USA

A variety of program evaluation designs are available to assess the impact of disease or care managementAbstractprograms, which can make it difficult to compare outcomes of different interventions. The need to compareprograms has resulted in consideration of standardizing evaluations of disease management programs; however,recommendations on the conduct of such evaluations have not been widely adopted. The purpose of this article isto examine the consistency of study characteristics of disease management peer reviewed evaluations over a3-year period (1 January 2004–28 February 2007) and to suggest questions that must be answered to ensure basictransparency of methods and metrics.

Study designs vary considerably among the current literature on evaluations of disease managementinterventions involving US health plans (25 studies). The mechanism for defining the intervention populationswere not consistent, even among interventions focused on a single disease, and evaluations employed bothadministrative and clinical data. The current literature included both randomized (n = 10) and non-randomizedstudies (n = 15). The referent population varied among the non-randomized studies, and included data from thepre-intervention period and both concurrent and historical control groups. The outcome metrics used in theevaluations included mortality and readmission rates, as well as time to readmission and various cost parameters.The majority of reviewed studies corrected for the confounding variables of age and sex, and a high proportioncorrected for a range of other confounding factors.

In conclusion, the evaluations of disease management programs in the literature cannot be consideredstandardized. To increase the transparency and validity of disease management intervention evaluations, werecommend consideration of five basic questions regarding intervention descriptions, intervention population,referent population, outcomes metrics, and confounding variables. Standardization on such basic parameters is anecessary step towards being able to assess the quality and validity of evaluations. Such standardization isessential for comparing the effectiveness of alternative programs, and to enable data-driven value-basedpurchasing decisions.

A 2003 article commissioned by the Disease Management management: follow-up, case-control, cross-sectional, ecological,Association of America (DMAA) listed seven different study quasi-experimental, benchmark, and post-only.[1] In 2005, thedesign types that could be used to examine the impact of disease

366 Wilson et al.

DMAA Outcomes Guide[2] listed 12 different ‘study design char- In 2006, a study was commissioned by CorSolutions, a diseaseacteristics’: management company, and performed by the RAND Corporation.

It recommended “in the absence of a control group … the use of1. randomized controlled trial (RCT)benchmark data as rough estimate of secular trend and statistical2. pre-post with the patient as their own controlestimates to the degree possible.”[9] Also in 2006, Sexner et al.[10]

3. pre-post historical control design with concurrent controlrecommended a “retrospective cohort pre-post baseline populationgroupbased savings analysis” where results are “compared with the4. pre-post historical control with actuarial adjustmenttrend of the non-diseased population … the most appropriate5. pre-post historical control with no actuarial adjustmentavailable comparison group.” In 2006, Linden[11] recommended6. pre-post multiple baseline (with randomized controls)that the condition-specific admission rate over an extended base-7. benchmark studyline be used as a reference in the evaluation of disease manage-

8. pre-post time seriesment interventions measuring condition-specific admissions. In

9. post-only control groupDecember 2007, the American Academy of Actuaries recommen-

10. post-only/no control groupded that “causality” was not the goal, but “correlation” or associa-

11. cross-sectional (no comparison) tion was “acceptable to most buyers,” and recommended an actua-12. simple time-series (e.g. run charts). rial method, with a strong stress on equivalence.[12] These reviews

In a 2007 book published by the DMAA,[3] multiple methods did not examine the level of standardization of evaluation today.for evaluating disease management programs were described, Other reviews of the disease management literature have beenbased upon the disciplines of health services research, epidemiolo- conducted, but these have primarily focused on the value ofgy, econometrics, and actuarial sciences. disease management, not the methods used to determine that

These articles highlighted the multiplicity of design options value.[13-20]

available to assess the impact of disease management. ManyThere is a need to review the current level of standardization

industry experts are concerned with this lack of standardizationamong disease management impact studies. The purpose of this

and are working to develop a standard method to enable diseasepaper is to assess the processes and methods used to assess

management programs to be compared side by side.[4] The purposeoutcomes in peer-reviewed, US-based disease management evalu-

of this article is to examine the consistency of disease managementation studies performed with the involvement of a health plan and

peer reviewed evaluations over a 3-year period. This is an essentialpublished between 2004 and 2007. The specific outcomes/effects

initial step in considering the variations present in the publishedof the disease management programs will not be examined.

literature. The goal is to develop a framework for assessing diseaseTo this end, we performed a search in MEDLINE using the keymanagement evaluations that is robust to the variety of disease

search criteria ‘insurance claims’ or ‘administrative claims’ ormanagement interventions and evaluation methods. Valid evalua-‘chronic illness care’ or ‘chronic care model’ or ‘population-basedtion findings should be used when comparing program effective-management’ or ‘disease-management’ or ‘disease management’ness and outcomes.or ‘care management’ and ‘return on investment’ or ‘ROI’ orThis push toward methods standardization was started in ear-‘economic’ or ‘financial’ or ‘savings’ amongst articles publishednest in 2003 when the disease management company Americanbetween 1 January 2004 and 31 December 2006. Abstracts wereHealthways (now known as Healthways) and Johns Hopkinsreviewed by three panelists (P. Salber, M. MacDowell, and T. Wil-University suggested that a “pre-post study was sufficient at thisson) and articles were excluded if they did not involve a healthtime” (the inclusion of an actuarial referent was recommended inplan, focused only on prescription-based disease management,the appendix).[5] This model was never widely adopted[6] and otherfeatured interventions that were managed by employers, or de-approaches have been recommended. The most recent attempt wasscribed studies that were not performed in the US. This resulted inby the DMAA itself, which, in December 2006, recommendedthe selection of 23 articles.[21-43] Two additional articles that met“use of pre-post study design that incorporates, whenever possible,the criteria above and were published between 1 January 2007 andan equivalent, concurrent comparison group” and suggested sever-28 February 2007, were added to the review at a later date.[44,45]al more detailed recommendations including “risk adjustment”Each article was reviewed by M. MacDowell, C. Hamm, andand the use of “non-chronic trend” as a reference.[7] This book was

updated with more detail and clarification in 2007[8] and 2008.[4] T. Wilson.

© 2008 Adis Data Information BV. All rights reserved. Dis Manage Health Outcomes 2008; 16 (5)

Evaluation Methods in Disease Management Studies 367

The goal was to describe the level of standardization and rather benchmarks (as recommended by RAND), articles, or non-variability of the study characteristics listed below. The consensus standard comparisons (e.g. the pre- intervention period in a pre-opinion regarding the evaluation design characteristics was post study) or articles, or non-standard comparisons (e.g. the pre-reached by three reviewers (M. MacDowell, C. Hamm, and period in a pre-post evaluation design).T. Wilson).

1.5 Confounding Factors1. Evaluation Design Characteristics

These are factors, which may or may not be measured, thatDescriptions of the following characteristics must be present in could be independently related to the outcome metric but are not

order to achieve transparency of methods and metrics. part of the intervention. Common confounding variables are ageand sex. These factors can confound a causal relationship. Here is

1.1 Intervention a typical example: if the age distribution in the interventionpopulation was 40% elderly, while in the concurrent referent theThis is what is done to the population. This is not an element ofage distribution was 60% elderly, the better outcomes in thethe evaluation design; it is rather the thing the study is evaluating.intervention population could be totally or partially a result of theyounger population in the intervention group, rather than the1.2 Intervention Populationintervention itself. Other potential confounding factors include

This is the group of people in which the disease management differences between the intervention population and the referent inprogram is implemented. benefit design, disease severity, the presence of co-morbid condi-

tions, the propensity of participants to take care of their health, and1.3 Outcomes Metrics

provider mix. Importantly, it is unlikely that all confoundingfactors can be taken into account in any study. Thus, positive (orThese are the targeted measures that the intervention is de-negative) results from disease management studies should neversigned to impact. Although the term includes the word ‘outcomes,’be thought of as ‘proof.’ Confounders are variables that are onlythese measures may or may not be caused by the intervention.useful when there is a referent other than the pre-interventionIndeed, a basic requirement of an evaluation is the use of these inperiod in a simple pre-post study in a single cohort of patients.the baseline period, to compare with the post period in the inter-

Each evaluation design characteristic was examined by threevention population only, to the pre-period in the external referentreviewers (M. MacDowell, C. Hamm, and T. Wilson).(if available).

2. Characteristics of the Reviewed Disease1.4 ReferentManagement Evaluations

This is what the intervention population is compared with and,ideally, is designed to estimate values for the outcome metrics in

2.1 Interventionsthe intervention population in the absence of the intervention.Broadly defined, referents include control groups, expert opinion,

The majority of the 25 articles that met our selection criteria forthe defined time period for the ‘pre-intervention’ period in a pre/

disease management interventions were ‘telephonic nurse man-post-intervention study (involving either the same individuals or a

agement,’ ‘monthly group meetings,’ ‘senior life management,’different group of people), historical control groups from a time

‘care advocacy,’ ‘immunization reminders,’ ‘home-based teach-period prior to that of the intervention population, and the concur-

ing,’ ‘interventions to enhance fitness,’ ‘interventions to enhancerent period to the intervention population.

care management,’ and ‘collaborative care.’ See table I for aThe referent can be chosen by an experimental method (e.g.

complete description of interventions.randomization) or a non-experimental method (e.g. people whoopt out, staggered roll-out, an employer group that did not partici- 2.2 Intervention Population Definition:pate in disease management, a matched control group, a previousperiod in time for a patient population). We have used the term There was a large number of different disease/condition states‘referent,’ rather than the more common ‘reference population’ as represented in the 25 articles. These included congestive heartthere are some referents in place that are not populations, but failure (CHF; n = 7), diabetes mellitus (n = 2), asthma (n = 3),


368 Wilson et al.


Tab

le I

. D

escr

iptio

n of

cite

d ar

ticle

s re

view

ed (

as d

escr

ibed

by

the

auth

ors

of t

he a

rtic

les)

Stu

dyD

esig

n/na

ture

Mai

n at

trib

utes

Inte

rven

tion

Out

com

e m

etric

sC

onfo

undi

ng m

etric

sof

RC

Tof

stu

dypo

pula

tion

Mar

tinR

CT

Med

icar

e,S

enio

r lif

e/ca

reS

F-6

, re

sour

ce u

se m

easu

red

by a

dmis

sion

rat

esA

ge,

sex

(all

othe

rs w

ere

outc

omes

, e.

g. S

F)

et a

l.[21]

(indi

vidu

al)

CH

F,

etc.

man

agem

ent

and

bed-

days

per

tho

usan

d pe

r ye

ar,

mem

ber

satis

fact

ion,

and

cos

ts m

easu

red

by p

aid

clai

ms

Tin

kelm

anN

on-R

CT

Ast

hma

Mul

tiple

Tot

al c

osts

Age

, se

xan

d W

ilson

[22]

inte

rven

tion

DM

Gal

brea

thR

CT

CH

FM

ultip

leA

ll-ca

use

mor

talit

y, E

F,

% o

n gu

idel

ine-

base

d R

x,A

ge,

sex,

rac

e, m

arita

l sta

tus,

fun

ctio

nal s

tatu

s,et

al.[2

3](in

divi

dual

)in

terv

entio

n D

Mco

sts,

sta

ysB

P,

med

ical

his

tory

, pu

lse,

pha

rmac

othe

rapy

DeB

usk

RC

TLo

w-r

isk

Nur

se c

ase

Tim

e to

reh

ospi

taliz

atio

n, r

easo

ns f

or r

e-A

ge,

sex,

rac

e, m

arita

l sta

tus,

edu

catio

n le

vel,

et a

l.[24]

(offi

ces)

CH

Fm

anag

emen

t,ho

spita

lizat

ion

HF

his

tory

(pr

evio

us d

iagn

osis

of

HF

, ba

selin

ete

leph

onic

NY

HA

cla

ss I

–II,

III–I

V),

cau

se o

f H

F (

coro

nary

dise

ase,

hyp

erte

nsio

n, v

alvu

lar

dise

ase,

unkn

own)

, sy

mpt

oms

befo

re h

ospi

taliz

atio

n(a

ngin

a, a

typi

cal c

hest

pai

n, e

xert

iona

l sho

rtne

ssof

bre

ath,

cou

gh,

paro

xysm

al n

octu

rnal

dys

pnea

,pe

riphe

ral e

dem

a, o

rtho

pnea

)

Sac

kett

Non

-RC

TP

rena

tal

Pre

nata

lLB

W,

cost

sN

one

et a

l.[25]

prog

ram

with

case

man

agem

ent

Vill

agra

Non

-RC

TD

iabe

tes

Mul

tiple

Cos

t (in

patie

nt,

outp

atie

nt,

prof

essi

onal

ser

vice

s,A

ge,

sex

and

Ahm

ed[2

6]in

terv

entio

n D

MR

x dr

ugs,

oth

er),

use

(da

ys,

ER

, vi

sits

,ad

mis

sion

s),

qual

lity

indi

cato

rs (

HbA

1c,

AC

E,

dila

ted

retin

al e

xam

, m

icro

albu

min

, lip

id,

toba

cco

use)

Ber

gR

CT

All

mem

bers

Dire

ct m

ail

Influ

enza

/pne

umon

ia a

dmis

sion

/ER

, in

fluen

zaA

ge,

sex,

hou

seho

ld s

ize,

mem

ber

mon

ths,

dua

let

al.[2

7](in

divi

dual

)m

arke

ting

ofva

ccin

atio

ns p

neum

onia

vac

cina

tions

, es

timat

edco

vera

ge,

cond

ition

pre

vale

nce

(eig

ht c

omm

onim

mun

izat

ion

cost

/sav

ings

(m

ultip

lyin

g th

e es

timat

ed a

bsol

ute

chro

nics

)di

ffere

nce

in u

tiliz

atio

n an

d ut

iliza

tion

orva

ccin

atio

n)

Ber

gN

on-R

CT

CH

FT

elep

honi

c D

MA

nnua

l cos

t, $U

S (

med

ical

, R

x).

Med

ical

ser

vice

Dem

ogra

phic

(m

onth

s pr

e, m

ean)

and

co-

et a

l.[28]

utili

zatio

n (a

nnua

lized

rat

e pe

r 10

00),

pre

scrip

tion

mor

bidi

ty (

% C

AD

, C

OP

D,

hype

rten

sion

, di

abet

esdr

ug u

se (

annu

al d

ays

supp

ly p

er p

erso

n),

mel

litus

, ar

thrit

is,

dem

entia

, de

pres

sion

)pr

escr

iptio

n dr

ug u

se b

asel

ine

(% w

ho h

adpr

escr

iptio

n)

Sco

ttR

CT

≥11

outp

atie

ntM

onth

ly g

roup

Per

pat

ient

clin

ic v

isits

, ph

arm

acy

fills

, ho

spita

lA

ge,

sex,

mar

ital s

tatu

s, %

with

a c

hron

icet

al.[2

9](in

divi

dual

)vi

sits

mee

tings

adm

issi

ons,

hos

pita

l obs

erva

tion

adm

issi

ons,

cond

ition

(14

list

ed),

no.

of

Rx,

mob

ility

lim

itatio

n,pa

tient

s an

dho

spita

l out

patie

nt v

isits

, pr

ofes

sion

al s

ervi

ces,

depr

essi

on,

% o

f m

emor

y pr

oble

ms,

live

alo

ne,

doct

ors

emer

genc

y vi

sits

, sk

illed

nur

sing

fac

ility

no b

asic

AD

L de

ficits

, no

adv

ance

d A

DL

defic

its,

adm

issi

ons,

hom

e he

alth

vis

its;

cost

(by

ser

vice

no h

ouse

hold

AD

L de

ficits

, fa

ir or

poo

r he

alth

line)

, Q

OL

(mul

tiple

)st

atus

Con

tinue

d ne

xt p

age



Tab

le I

. C

ontd

Stu

dyD

esig

n/na

ture

Mai

n at

trib

utes

Inte

rven

tion

Out

com

e m

etric

sC

onfo

undi

ng m

etric

sof

RC

Tof

stu

dypo

pula

tion

Ros

tR

CT

Dep

ress

ion

Enh

ance

d ca

reD

ays

free

of

depr

essi

on,

prog

ram

and

out

patie

ntN

one

liste

det

al.[3

0](g

roup

s)m

anag

emen

tco

sts,

pat

ient

tim

e an

d tr

ansp

orta

tion

cost

s,Q

ALY

s

Cat

ovN

on-R

CT

Ast

hma

Hom

e-ba

sed

Hos

pita

l adm

its,

ER

adm

itsS

ex,

race

, ag

e, r

egio

n of

cou

ntry

, ur

ban/

rura

let

al.[3

1]te

achi

ngs

Hud

son

Non

-RC

TC

HF

Rem

ote

ER

vis

its,

inpa

tient

adm

issi

ons,

re-

adm

issi

ons

Non

e (s

trat

ified

by

sex

and

age)

et a

l.[32]

mon

itorin

g

van

Von

noN

on-R

CT

CH

FM

ultip

leC

HF

-rel

ated

hos

pita

l sta

y or

out

patie

nt v

isit,

tot

alA

ge,

sex,

con

ditio

ns (

rheu

mat

ic,

hype

rten

sive

,et

al.[3

3]in

terv

entio

n D

Mco

st d

iffer

ence

s pr

e-po

st (

adju

sted

for

leng

th o

fis

chem

ic,

pulm

onar

y he

art

dise

ases

), d

rugs

prog

ram

par

ticip

atio

ns)

(dio

xin,

diu

retic

s, e

tc),

use

d nu

rsin

g ho

me,

use

dho

me

care

, no

. of

dia

gnos

is c

odes

Del

aron

deR

CT

Ast

hma

Mul

tiple

Cha

nges

in a

sthm

a m

edic

atio

n us

e, p

hysi

cian

Age

, se

xet

al.[3

4]in

terv

entio

n D

Mof

fice

visi

ts,

ER

vis

its,

hosp

italiz

atio

ns,

and

QO

L

Dun

agan

RC

TC

HF

Nur

se,

Mor

talit

y, r

e-ad

mis

sion

or

ED

vis

it, a

ny r

e-A

ge,

sex,

rac

e, m

arita

l sta

tus,

hel

p liv

ing

atet

al.[3

5](in

divi

dual

s)te

leph

onic

adm

issi

on,

prim

ary

HF

rea

dmis

sion

, an

yho

me,

hea

lth w

ith d

aily

livi

ng,

isch

emic

etio

logy

read

mis

ion

or d

eath

of H

F,

LVE

F,

NY

HA

cla

ssifi

catio

n, A

CE

reg

imen

,di

scha

rge

crea

tinin

e, C

harls

on s

core

, S

F-1

2,B

ack

Inve

ntor

y S

cale

, M

LHF

Q (

phys

ical

and

men

tal)

Haw

kins

Non

-RC

TS

peci

al n

eeds

Ris

kM

embe

r sa

tisfa

ctio

n, a

dmits

, LO

S,

ED

tot

al c

osts

Non

eet

al.[3

6]ch

ildre

nas

sess

men

tan

d ca

sem

anag

emen

t

Sch

wer

ner

Non

-RC

T14

chr

onic

,M

ultip

leP

MP

M,

tren

d14

chr

onic

, co

mpl

ex c

ondi

tions

et a

l.[37]

com

plex

inte

rven

tion

DM

cond

ition

s

Wis

eN

on-R

CT

Pop

ulat

ion

Med

ical

PM

PM

(al

l, IP

, O

P,

Rx,

Pro

f),

qual

ity (

HE

DIS

:A

ge,

sex,

hea

lth p

lan

bene

fits

(est

imat

ed),

et a

l.[38]

and

DM

HbA

1c t

estin

g an

d co

ntro

l, ey

e ex

am,

lipid

exa

m,

num

ber

of h

ealth

con

ditio

ns,

prev

alen

ce o

f he

alth

LDL

<13

0, n

ephr

opat

hy m

onito

ring)

cond

ition

s

Dal

yR

CT

Chr

onic

ally

Cas

eH

ospi

taliz

atio

n re

-adm

issi

on,

re-a

dmitt

ed w

ithin

Age

, A

PA

CH

E s

core

, G

lasc

ow,

pre-

hosp

ital

et a

l.[39]

(indi

vidu

al)

criti

cal i

ll >

3m

anag

emen

t48

hou

rs,

deat

h at

re-

adm

issi

on,

days

to

first

re-

adm

issi

on c

o-m

orbi

d sc

ore,

med

icat

ion

pre-

days

adm

issi

on,

tota

l re-

hosp

italiz

atio

n da

ys;

disc

harg

eho

spita

l sco

re,

leng

th o

f ho

spita

l sta

y, s

ex,

race

,m

echa

nica

ldi

spos

ition

(lo

ng-t

erm

acu

te c

are,

reh

abili

tatio

n,pr

imar

y IC

D-9

dia

gnos

isve

ntila

tion,

nurs

ing

hom

e, d

eath

), d

isch

arge

sta

tus

(with

post

-dis

char

geox

ygen

, tr

ache

otom

y, v

entil

ator

, co

gniti

veim

pairm

ent)

; S

F-8

sco

re p

hysi

cal f

unct

ioni

ng a

ndm

enta

l fun

ctio

ning

, pa

tient

AD

L/A

DL

scor

e

Con

tinue

d ne

xt p

age

370 Wilson et al.


Tab

le I

. C

ontd

Stu

dyD

esig

n/na

ture

Mai

n at

trib

utes

Inte

rven

tion

Out

com

e m

etric

sC

onfo

undi

ng m

etric

sof

RC

Tof

stu

dypo

pula

tion

Sid

orov

[40]

Non

-RC

TC

HF

Mul

tiple

Ass

essm

ent

of L

VH

, pr

escr

ibed

an

AC

E o

rN

one

inte

rven

tion

DM

cont

rain

dica

tion

(LV

EF

140

%),

pre

scrib

ed a

β-

adre

noce

ptor

ant

agon

ist;

inpa

tient

day

s, E

Rvi

sits

, to

tal c

laim

s, E

R c

laim

s; f

inan

cial

(out

patie

nt,

inpa

tient

, C

HF

-rel

ated

cla

ims)

Gry

pma

Non

-RC

TD

epre

ssio

nC

olla

bora

tive

Util

izat

ion

of s

ervi

ces,

ann

ualiz

ed c

osts

,A

ge,

sex

et a

l.[41]

Car

ede

pres

sion

sco

res

(bas

elin

e an

d 6-

mon

th a

ndM

anag

emen

t:pe

rcen

tage

exp

erie

ncin

g 50

% im

prov

emen

t in

IMP

AC

Tde

pres

sion

) an

tidep

ress

ant

use,

tre

atm

ent

cont

acts

Sha

nnon

RC

TH

igh

use,

>65

CA

pro

gram

:U

tiliz

atio

n in

ser

vice

s (p

rimar

y ca

re p

hysi

cian

,A

ge,

sex,

mea

sure

s of

hea

lth s

tatu

s (n

ot s

how

n)et

al.[4

2](in

divi

dual

)ye

ars

refe

rral

s to

spec

ialis

t, ho

spita

l adm

issi

ons,

hos

pita

l day

s,ho

me-

and

ER

)co

mm

unity

-ba

sed

serv

ices

Sw

eene

yN

on-R

CT

Life

-lim

iting

Inte

nsiv

eH

ome

care

day

s, h

ospi

ce d

ays,

reh

abili

tatio

nA

ge,

sex,

bro

ad I

CD

-9 c

ateg

orie

s (t

op 6

of

16:

et a

l.[44]

diag

nosi

spa

tient

-cen

tere

dda

ys,

ER

vis

its,

hosp

ital d

ays,

hos

pita

lne

opla

sms,

circ

ulat

ory,

dig

estiv

e sy

stem

, ne

rvou

sm

anag

emen

tad

mis

sion

s; in

patie

nt a

dmis

sion

s (n

ause

a an

dan

d se

nse,

ill-d

efin

ed,

inju

ry,

and

pois

onin

g)vo

miti

ng,

anem

ia,

fluid

dis

orde

r an

d de

hydr

atio

n,fe

ver,

nut

ritio

nal d

efic

ienc

y, m

etab

olis

m,

anxi

ety)

,m

orta

lity/

surv

ival

Ngu

yen

Non

-RC

TD

iabe

tics

Enh

ance

dH

ospi

tal a

dmis

sion

s (a

ll, d

iabe

tes

rela

ted)

,A

ge,

sex,

HbA

1c le

vels

, ch

roni

c di

seas

e bu

rden

,an

dfit

ness

pro

gram

annu

al p

rimar

y ca

re v

isits

(al

l, di

abet

es r

elat

ed),

prev

entio

n sc

ore

(pre

vent

ion

scor

e is

the

sum

of

Ack

erm

an[4

5]he

alth

care

cos

ts (

tota

l, pr

imar

y ca

re,

inpa

tient

the

num

ber

of t

imes

a s

ubje

ct r

ecei

ved

colo

nco

sts)

(no

te:

stra

tifyi

ng v

aria

bles

: le

vel o

fca

ncer

scr

eeni

ng,

[feca

l occ

ult

bloo

d te

st o

rpa

rtic

ipat

ion

in e

xerc

ise

prog

ram

). N

ote:

ave

rage

flexi

ble

sigm

oido

scop

y],

a sc

reen

ing

inpa

tient

cos

ts f

or m

embe

rs w

ho h

ad in

patie

ntm

amm

ogra

m,

pros

tate

can

cer

scre

enin

g, a

nco

sts

influ

enza

vac

cine

, or

a p

neum

ococ

cal v

acci

nedu

ring

the

2 ye

ars

imm

edia

tely

pre

cedi

ng t

hein

dex

date

)

Fire

ston

eN

on-R

CT

CA

D,

CH

F,

Mul

tiple

Qua

lity

mea

sure

s (L

DL

test

, st

atin

Rx,

Dis

ease

pre

vale

nce

et a

l.[43]

asth

ma,

inte

rven

tion

DM

antih

yper

tens

ive

Rx)

clin

ic v

isits

(ph

ysic

ian,

ER

,di

abet

esad

mis

sion

s, d

ays,

Rx

cost

s)

AD

L =

act

iviti

es o

f da

ily li

ving

; A

PA

CH

E =

Acu

te P

hysi

olog

y an

d C

hron

ic H

ealth

Eva

luat

ion;

BP

= b

lood

pre

ssur

e; C

A =

car

e ad

voca

te;

CA

D =

cor

onar

y he

art

dise

ase;

CH

F =

cong

estiv

e he

art f

ailu

re; C

OP

D =

chr

onic

obs

truc

tive

pulm

onar

y di

seas

e; D

M =

dis

ease

man

agem

ent;

ED

= e

mer

genc

y de

part

men

t; E

F =

eje

ctio

n fr

actio

n; E

R =

em

erge

ncy

room

;H

bA

1c =

one

type

of g

lyco

syla

ted

hem

oglo

bin;

HE

DIS

= H

ealth

Pla

n E

mpl

oyer

Dat

a an

d In

form

atio

n S

et; H

F =

hea

rt fa

ilure

; IC

D-9

= In

tern

atio

nal C

lass

ifica

tion

of D

isea

ses,

ver

sion

9; IM

PA

CT

= I

mpr

ovin

g M

ood-

Pro

mot

ing

Acc

ess

to C

olla

bora

tive

Tre

atm

ent;

IP =

inpa

tient

; L

BW

= lo

w b

irth

wei

ght;

LD

L =

low

-den

sity

lipo

prot

ein;

LO

S =

leng

th o

f st

ay;

LV

EF

=le

ft ve

ntric

ular

eje

ctio

n fr

actio

n; L

VH

= l

eft

vent

ricul

ar h

yper

trop

hy;

ML

HF

Q =

Min

neso

ta L

ivin

g w

ith H

eart

Fai

lure

que

stio

nnai

re;

NY

HA

= N

ew Y

ork

Hea

rt A

ssoc

iatio

n; O

P =

outp

atie

nt; P

MP

M =

per

mem

ber

per

mon

th; P

rof =

pro

fess

iona

l; Q

AL

Y =

qua

lity-

adju

sted

life

-yea

r; Q

OL

= q

ualit

y of

life

; RC

T =

ran

dom

ized

con

trol

led

tria

l; R

x =

pre

scrip

tion;

SF

-6=

Out

com

es S

tudy

36-

Item

Sho

rt-F

orm

Hea

lth S

urve

y.


chronic diseases (coronary artery disease, CHF, diabetes, and for a complete description of outcome metrics employed in theasthma; n = 1), total health plan population (without regard to studies.diagnosis; n = 3), depression (n = 2), high utilizers (n = 2), severechronic disease (n = 2), life-limiting diagnoses (n = 1), discharged 2.5 Confounding Factorswith ventilator (n = 1), prenatal (n = 1), and special needs chil-dren (n = 1). Five of the articles listed no confounding variables. Four stud-

The criteria used to define the intervention population were not ies adjusted results for differences in the age and sex of compara-consistent among evaluations of programs for a single disease. For tor groups (all were non-randomized); the other 16 adjusted resultsexample, among studies of CHF interventions, some defined the for differences in other variables, as well as age and sex.study population using the diagnostic codes (i.e. International Among the studies with non-randomized designs, the con-Classification of Diseases, version 9 [ICD-9]) from administrative founding variables that were adjusted for included the number ordata, with or without health plan membership requirements or type of co-morbidities, HEDIS metrics (that did not require chartutilization history (regarding inpatient stays, emergency room review – as is done in the HEDIS hybrid method, which utilizesvisits, etc.). In addition, some used other factors (e.g. the ability to both administrative claims data and chart review to derive thecommunicate with the patient’s primary care provider) as a secon- numerator and denominator of metrics), and prevalence of specificdary criteria for selection. The randomized trials generally used national drug codes (NDC), number of prescriptions, coverageclinical parameters (e.g. left ventricular ejection fraction) to define type, benefit design, and the length of time in the plan/program.the intervention population. There was no commonly applied Other confounding variables that were not assessed using adminis-standard for selecting populations. trative data included household size, marital status, educational

level, the presence/absence of left ventricular hypertrophy, labora-2.3 Referent Types tory parameter values, left ventricular ejection fraction, and New

York Heart Association class. See table I for a complete descrip-Ten of the 25 studies used randomization to select the referent

tion of confounding factors in the studies reviewed.population; randomization was mostly at the individual level(n = 7),[21,23,27,29,34,35,39,42] while two studies randomized physician

3. Conclusions and Recommendationsoffices, rather than individuals.[24,30] One study[34] employedrandomization for one sub-study and analyzed a combined popula-

Clearly, we do not see any trend towards the standardization oftion that included both the intervention group and referents for the

intervention populations, outcome metrics, or even a commonother sub-studies.

definition of ‘disease management.’ within current evaluations ofThe other 15 studies employed non-randomized designs.

disease management programs. Thus, comparison of outcomesTwo[32,36] used the pre-intervention period as the primary referent

between different interventions would appear to be very difficult,for the intervention period, three[25,41,43] used an historical referent

if not impossible.(two of these used more than one design), ten used a concurrent

While we await a standard evaluation design, the best choicereferent (three of these used more than one de-

for value-based purchasing will be to use a basic list of criteriasign).[22,26,28,31,33,37,38,40,44,45] None used expert opinion; however,

when assessing the methods used in a disease management evalua-one study did cite an expert to adjust for benefit design differences

tion. Such a list – if standardized – would at least enable the studybetween referent and intervention populations.

methods to be compared as is done in ‘level of evidence’ scoring inthe evidenced-based medicine area.[46-50] A standardized evalua-2.4 Outcome Metricstion is essential for two reasons: (i) to assess level of confidence inthe evaluation design (and to compare this across programs); andOutcome metrics included mortality, admission, re-admission,(ii) to assess level of confidence in the results.time to re-admission, and cost (total, admission, out-patient, pre-

scription). These were often related to specific diagnoses. Other Disease management evaluation studies – published or not –metrics included days free of depression, the percentage of low may include any number of technical and non-technical elements,birth-weight babies among all births, quality-of-life survey out- but the foundational criteria needed to assess the validity of thesecomes, patient satisfaction survey outcomes, and Healthcare Ef- studies is transparency of methods and metrics. Based upon thefectiveness Data and Information Set (HEDIS) metrics. See table I simple criteria we used to assess the methods used in a representa-


372 Wilson et al.

Disease Management Outcomes Summit; 2002 Nov 7–10; Palm Springs (CA).tive list of 25 articles, five simple questions that would provideNashville (TN): American Healthways Inc., 2003

increased transparency are listed below: 6. Glabman M. ‘Take my word for it’: the enduring dispute over measuring DM’seconomic value. Managed Care April 2006 [online]. Available from URL:1. Is the intervention used in the disease management programhttp://managedcaremag.com/archives/0604/0604.dmvalue.html [Accessedclearly stated?2008 Nov 27]

2. Is the population in which the intervention is implemented 7. Disease Management Association of America. Outcomes guidelines report. Vol I.Washington, DC: DMAA, 2006clearly defined?

8. Disease Management Association of America. Outcomes guidelines report. Vol II.3. Is the referent used in the disease management evaluationWashington, DC: DMAA, 2007

clearly defined? 9. Mattke S, Bergamo G, Balakrishnan A, et al. Measuring and reporting the perform-ance of disease management programs. Santa Monica (CA): RAND, 2006 Aug4. Are the outcome metrics used to compare the intervention and

10. Sexner S, Baker K, Gold D. Guidelines for analysis of economic returns fromreferent group clearly defined and consistently assessed?health management programs: the art of health promotion. Art Health Prom

5. Are key confounding factors identified, measured, and proper- 2006 Jul/Aug: 1-17

11. Linden A. What will it take for DM to demonstrate a return on investment? Newly taken into account when making a comparison between theperspectives on an old theme. Am J Manag Care 2006; 12: 217-22

outcome metrics obtained for the intervention and referent groups?12. American Academy of Actuaries’ Disease Management Work Group. Disease

Other groups have suggested checklists for disease manage- management: a public policy practice note. Washington, DC: American Acade-my of Actuaries, 2007ment, and we encourage the reader to review these more compre-

13. Weingarten SR, Henning JM, Badamgarav E, et al. Interventions used in diseasehensive lists. They range from the general to the specific: general management programmes for patients with chronic illness: which ones work?

Meta-analysis of published reports. BMJ 2002; 325: 1-8checklists have been offered by Mathematica,[51] the National14. Ofman JJ, Badamgarav E, Henning JM, et al. Does disease management improveManaged Health Care Congress (now The Health/Care Evaluation

clinical and economic outcomes in patients with chronic diseases? A systematicworkgroup),[52] the DMAA,[53] and the American Heart Associa- review. Am J Med 2004; 117: 182-92

tion.[54,55] Specific checklists from health services research[56] and 15. Holtz-Eakin D. An analysis of the literature on disease management programs.Congressional Budget Office, 2004 Oct [online]. Available from URL: http://actuarial sciences[57] are also available. However, using these morewww.cbo.gov/ftpdocs/59xx/doc5909/10-13-DiseaseMngmnt.pdf [Accessed

comprehensive lists are not useful unless some detail is presented 2008 Oct 28]

16. Krause D. Economic effectiveness of disease management: a metaanalysis. Dis– made transparent – regarding the five issues we discuss.Manage 2005; 8: 114-34An article is in preparation that will examine the quality and

17. Goetzel RZ, Ozminkowski RJ, Villagra VG, et al. Return on investment in diseasestrength of these 25 studies using more sophisticated quantitative management: a review. Health Care Financ Rev 2005; 26: 1-19

scoring criteria. This article will take the reader beyond the basics 18. Wilson TW, MacDowell M, Montrose G. The prerequisite importance of judgingthe potential usefulness of workplace health promotion studies: beyond “Pub-listed here to enable a more detailed look at the very importantlished in a Peer-Reviewed Journal” and “Strength of Study Design”. J Health

issues related to equivalence and statistical significance. Productivity 2006; 1: 23-8

19. Pelletier KR. A review and analysis of the clinical and cost-effectiveness studies ofcomprehensive health promotion and disease management programs at the

Acknowledgments worksite: update VI 2000–2004. J Occup Environ Med 2005 Oct; 47 (10):1051-8

20. Mattke S, Seid M, Sai M. Evidence for the effect of disease management: is $1This article was funded in part by an unrestricted educational grant frombillion a year a good investment? Am J Manag Care 2007; 13: 670-6the Health Industry Forum at Brandeis University provided to the Population

21. Martin DC, Berger ML, Anstatt DT, et al. A randomized controlled open trial ofHealth Impact Institute.population-based disease and case management in a Medicare Plus Choicehealth maintenance organization. Prev Chronic Dis 2004; 1 (4): A05 [online].Available from URL: http://www.pubmedcentral.nih.gov/articlerender.fcgi?arReferences tid=1277945 [Accessed 2008 Nov 24]

1. MacDowell M, Wilson TW. Framework for assessing causality in disease manage-22. Tinkelman D, Wilson S. Asthma disease management: regression to the mean or

ment programs [commissioned paper by Disease Management Association ofbetter? Am J Manag Care 2004 Dec; 10 (12): 948-54

America]. Washington, DC: DMAA, 200323. Galbreath AD, Krasuski RA, Smith B, et al. Long-term healthcare and cost2. Outcomes Consolidation Steering Committee. Disease management program

outcomes of disease management in a large, randomized, community-basedguide: including principles for assessing disease management programs. Wash-population with heart failure. Circulation 2004; 110 (23): 3518-26ington, DC: DMAA, 2005: 33

24. DeBusk RF, Miller NH, Parker KM, et al. Care management for low-risk patients3. Lindner A, Wilson T, Berg G, et al. Perspectives in disease management evalua-with heart failure: a randomized, controlled trial. Ann Intern Med 2004; 141 (8):tion: health services research, epidemiology, econometrics, and actuarial. In:606-13Hay J, Rindress D, editors. Measuring disease management. Yardley (PA):

25. Sackett K, Pope RK, Erdley WS. Demonstrating a positive return on investment forDMAA, 2007a prenatal program at a managed care organization: an economic analysis.4. Disease Management Association of America. Outcomes guidelines report. Vol. 3.J Perinat Neonatal Nurs 2004 Apr-Jun; 18 (2): 117-27Washington, DC: DMAA, 2008

5. American Healthways and Johns Hopkins University. Standard outcomes metrics 26. Villagra VG, Ahmed T. Effectiveness of a disease management program forand evaluation methodology of disease management programs. 2nd Annual patients with diabetes. Health Aff 2004; 23 (4): 255-66



27. Berg GD, Thomas E, Silverstein S, et al. Reducing medical service utilization by 43. Firestone B, Bartlett J, Selby J. Can disease management reduce health care costsencouraging vaccines: randomized controlled trial. Am J Prev Med 2004 Nov; by improving quality? Health Aff 2004; 23 (6): 63-7527 (4): 284-8

44. Sweeney L, Halpert A, Waranoff J. Patient-centered management of complex28. Berg GD, Wadhwa S, Johnson AE. A matched-cohort study of health services patients can reduce costs without shortening life. Am J Manag Care 2007; 13:

utilization and financial outcomes for a heart failure disease-management 84-92program in elderly patients. J Am Geriatr Soc 2004 Oct; 52 (10): 1655-61

45. Nguyen HQ, Ackerman R. Impact of a managed-Medicare physical activity benefit29. Scott JC, Conner DA, Venohr I, et al. Effectiveness of a group outpatient visit on health care utilization and costs in older adults with diabetes. Diabetes Care

model for chronically ill older health maintenance organization members: aJan 2007; 30 (1): 43-8

2-year randomized trial of the cooperative health care clinic. J Am Geriatr Soc46. Centre for Evidence-Based Medicine. Levels of evidence [online]. Available from2004 Sep; 52 (9): 1463-70

URL: http://www.cebm.net/index.aspx?o=1025 [Accessed 2008 Nov 27]30. Rost K, Pyne JM, Dickinson LM, et al. Cost-effectiveness of enhancing primary

47. Agency for Healthcare Research and Quality. US Preventive Services Task Forcecare depression management on an ongoing basis. Ann Fam Med 2005 Jan-Feb;3 (1): 7-14 [online]. Available from URL: http://www.ahrq.gov/clinic/uspstf/usp

stopics.htm [Accessed 2008 Nov 27]31. Catov JM, Marsh GM, Youk AO, et al. Asthma home teaching: two evaluationapproaches. Dis Manag 2005; 8 (3): 178-87 48. Consolidated Standards of Reporting Trials (CONSORT) [online]. Available from

URL: http://www.consort-statement.org/?o=1011 [Accessed 2008 Nov 27]32. Hudson LR, Hamar GB, Orr P, et al. Remote physiological monitoring: clinical,financial, and behavioral outcomes in a heart failure population. Dis Manag 49. The Cochrane Collaboration [online]. Available from URL: http://www.cochrane.2005; 8 (6): 372-81 org/ [Accessed 2008 Nov 27]

33. vanVonno CJ, Ozminkowski RJ, Smith MW, et al. What can a pilot congestive 50. Des Jarlais DC, Lyles C, Crepaz N, the TREND Group. Improving the reportingheart failure disease management program tell us about likely return on invest- quality of nonrandomized evaluations of behavioral and public health interven-ment? A case study from a program offered to federal employees. Dis Manag

tions: the TREND statement. Am J Public Health 2004; 94 (3): 361-62005 Dec; 8 (6): 346-60

51. Chen A, Brown R, Archibald N, et al. Best practices in coordinated care.34. Delaronde S, Peruccio DL, Bauer BJ. Improving asthma treatment in a managed

Mathematica 2000 [online]. Available from URL: http://www.mathematica-care population. Am J Manag Care 2005 Jun; 11 (6): 361-8

mpr.com/publications/PDFs/bestpractices.pdf [Accessed 2008 Nov 27]35. Dunagan WC, Littenberg B, Ewald GA, et al. Randomized trial of a nurse-

52. Wilson TW, Gruen J, Thar W, et al. Assessing return on investment of defined-administered, telephone-based disease management program for patients with

population disease management interventions. Jt Comm J Qual Saf 2004; 30heart failure. J Card Fail 2005 Jun; 11 (5): 358-65(11): 614-21

36. Hawkins MR, Diehl-Svrjcek B, Dunbar LJ. Caring for children with special53. Fitzner K, Siderov J, Fetterolf D, et al. Principles for assessing disease managementhealthcare needs in the managed care environment. Lippincotts Case Manag

outcomes. Dis Manage 2004; 7: 191-2012006 Jul-Aug; 11 (4): 216-23

54. Faxon DP, Schwamm LH, Pasternak RC, et al. American Heart Association’s37. Schwerner H, Mellody T, Goldstein AB, et al. Evaluating the impact of a diseaseExpert Panel on Disease Management. Improving quality of care throughmanagement program for chronic complex conditions at two large northeast

health plans using a control group methodology. Dis Manag 2006 Feb; 9 (1): disease management: principles and recommendations from the American34-44 Heart Association’s Expert Panel on Disease Management. Circulation 2004

Jun 1; 109 (21): 2651-438. Wise CG, Bahl V, Mitchell R, et al. Population-based medical and diseasemanagement: an evaluation of cost and quality. Dis Manag 2006 Feb; 9 (1): 55. Krumholz HM, Brindis RG, Brush JE, et al. Standards for statistical models used45-55

for public reporting of health outcomes. Circulation 2006; 113: 456-6239. Daly BJ, Douglas SL, Kelley CG, et al. Trial of a disease management program to 56. Linden A, Roberts N. A user’s guide to the disease management literature:

reduce hospital readmissions of the chronically critically ill. Chest 2005 Aug;recommendations for reporting and assessing program outcomes. Am J Manage

128 (2): 507-17Care 2005; 11 (2): 81-90

40. Sidorov J. Reduced health care costs associated with disease management for57. Duncan I, Owen R, Dove H. Testing actuarial methods for evaluating disease

chronic heart failure: a study using three methods to examine the financialmanagement savings outcomes. Society of Actuaries, 2006 [online]. Availableimpact of a heart failure disease management program among medicare advan-from URL: http://www.soa.org/research/files/pdf/Paper_8_TestingActuaritage enrollees. J Card Fail 2006 Oct; 12 (8): 594-600al%20 Methods.pdf [Accessed 2008 Nov 27]

41. Grypma L, Haverkamp R, Little S, et al. Taking an evidence-based model ofdepression care from research to practice: making lemonade out of depression.Gen Hosp Psychiatry 2006 Mar-Apr; 28 (2): 101-7

Correspondence: Dr Thomas Wilson, Population Health Impact Institute,42. Shannon GR, Wilber KH, Allen D. Reductions in costly healthcare service utiliza-

10663 Loveland-Madeira Rd. #210, Loveland, OH 45140, USA.tion: findings from the Care Advocate Program. J Am Geriatr Soc 2006 Jul; 54(7): 1102-7 E-mail: [email protected]


evaluation methods in disease management studies

Documents