An EvaluationFramework for

CommunityHealth Programs

June 2000

The Center for theAdvancement of Community Based Public Health

5102 Chapel Hill Boulevard, Durham, North Carolina 27707

Produced by:

The Center for the Advancement of Community Based Public Health

Quinton E. BakerDenise A. DavisRon Gallerani

Victoria SánchezClaire Viadro

Based on:

Framework for Program Evaluation in Public Health(Centers for Disease Control and Prevention)

Community Tool Box: A Framework for Program Evaluation(University of Kansas)

Supported in part by CDC Cooperative Agreement No. U48/CCU409664 - 07awarded to the University of South Carolina

5102 Chapel Hill Boulevard, Durham, NC 27707Phone: 919.403.2124 Fax: 919.401.9268 E-mail: center@cbph.org www.cbph.org

An Evaluation Framework forCommunity Health Programs

1Introduction to the Framework 1-5

2Overview of Program Evaluation 6-12

3Step 1: Engage Stakeholders 13-18

4Step 2: Describe the Program 19-28

5 Step 3: Focus the EvaluationDesign 29-38

6 Step 4: Gather and AnalyzeEvidence 39-48

7Step 5: Justify Conclusions 49-55

8 Step 6: Ensure Use and ShareLessons Learned 56-62

9Glossary 63-65

10Program Evaluation Resources 66-67

..................................................................................................................

PrefaceAt the Center for the Advancement of Community Based

Public Health (CA-CBPH) we believe that involvingcommunities as full partners and stakeholders in programdevelopment, implementation, and evaluation is vital forbuilding community capacity. Our belief, and commitment tobuilding community capacity, is grounded in principles whichhold that: (1) improvements in health are best achievedthrough the full participation of communities in identifyinghealth issues and creating programs to address them; and(2) in partnerships among entities such as community-based organizations, health agencies, and universities, eachpartner has contributions of equal value for achieving goals(e.g., creating healthy communities).

An Evaluation Framework for Community HealthPrograms can be used by community-based organizationsand community health workers to enhance their under-standing of evaluation and to build their capacity to moreactively and aggressively participate in evaluation efforts oftheir programs.

We believe however, that the framework is only onecomponent in building community capacity and thatadditional efforts are needed to encourage and supportmore inclusive evaluation processes. Additional efforts canbuild upon the Framework’s emphases on the development ofmeasures of program “success” that are relevant to allpartners; on considering what is important for communitiesand other stakeholders to know about their programs; andon producing data that are useful in community settings.

We hope that you will include the Framework in yourprogram “toolbox” and that you find application for it in yourwork. We welcome comments about the Framework.

1

..................................................................................................................

Introduction tothe Framework

This document presents a framework that emphasizesprogram evaluation as a practical and ongoing processthat involves program staff, community members, as wellas evaluation experts. The overall goal of the frameworkis to help guide and inform the evaluation process.

The document is not a comprehensive manual on howto conduct program evaluation. There are already manyexcellent resources that meet the technical aspects ofprogram evaluation. Instead, the framework promotes acommon understanding of program evaluation. It providesa conceptual roadmap that can be adapted to a varietyof settings and within many different groups and communities.The framework may also help individuals or groups with littleformal training or experience in program evaluation enterinto the evaluation decision-making processes, includingconsultation with professional evaluators.

The framework is intended to help those involved inprogram evaluation address the following six questions:

Guiding Questions for Program Evaluation

1. Who is the evaluation for?2. What program are we evaluating?3. What methods will we use in conducting our evaluation?4. How will we gather and analyze information that is credible and in what forms?5. How will we justify our conclusions?6. How can we be assured that what we learn will be used?

In work we do— especiallyat the community basedorganization (CBO) level—evaluation is invaluable,especially if thinking aboutcommunity based entitiesor communities being atthe center of public healthand the work that happensin the public health arena.The ability to have evaluationskills is really key.

Funding agencyrepresentative

2

..................................................................................................................

The framework also addresses the quality of evaluationby asking the question: Will the evaluation be a goodevaluation? In evaluation terms, standards convey qualityand include four key areas to consider:

Four Standards of Program Evaluation

1. Utility (Is the evaluation useful?)2. Feasibility (Is the evaluation viable and practical?)3. Propriety (Is the evaluation ethical?)4. Accuracy (Is the evaluation correct?)

Program evaluators view these four standards as theinitial yardstick by which to judge the quality of programevaluation efforts.

How the document is organizedThe framework is presented in seven main sections.

We present an overview of program evaluation in the firstsection. We then discuss six steps in program evaluation(illustrated on page 3) in the following sections. These stepsaddress the guiding questions listed on page 1. Althoughin the real world the steps may not always strictly follow thissequence, as presented each step provides the foundationfor the next. A glossary and program evaluation resourcesare included at the end of the document.

The discussion of each step includes:◆ Definition of the step◆ Why the step is important◆ What’s involved in completing the step◆ Applying standards◆ Questions (application to your program)

Symbolsto Watch For...

Questions (applicationto your program)

Key ideas to noteand remember

City of Hopecase example

3

..................................................................................................................

Steps inProgram Evaluation

STANDARDSAccuracy

Propriety FeasibilityUtility

1

26

STEPS

4

5 3

EngageStakeholders

Describethe

Program

Focus theEvaluation

DesignGather

and AnalyzeEvidence

JustifyConclusions

Ensure Useand Share

Lessons Learned

4

..................................................................................................................

Throughout each step, a case example illustrates theapplication of the framework in a community program.As just one example, the case is not a rigid or universalillustration of the program evaluation framework. Its mainpurpose is to illustrate the framework’s concepts with aspecific health problem, in a particular setting, and witha particular group of people. We invite you to think abouthow to apply the concepts to your own programs inconsidering the questions at the end of each section.

City of HopeCase Example

Overview: The City of Hope received funding fromthe Centers for Disease Control and Prevention todesign and implement a community-based inter-vention to reduce alcohol injury and death in youngmen. This funding evolved as an outgrowth ofcommunity concern. The problem was documentedthrough a community assessment conducted bycommunity residents, community-based agencyrepresentatives, university faculty, and local advocacygroups. This original group believed that achievingthis goal would require a long-term commitment ofmany people and organizations and formalized theircommitment by establishing the Partnership to ReduceAlcohol-related Injury and Death (PRAID) coalition.

Problem: Alcohol-related injuries and death in theCity of Hope are five times the state rate and two timesthe rate of cities of similar size and demographics inthe region. The assessment revealed a large numberof alcohol-related car injuries in two primary locationswithin a three mile radius of the city. An internal recordreview by the local police department showed thatover a three-year period, 70% of these injuriesinvolved males under the age of 21. A related statisticwas the increased number of youth treated for alcohol-related motor vehicle injuries on weekends in thehospital’s emergency department. In sum, the

5

..................................................................................................................

assessment found that a disproportionate number ofmales between the ages of 15 and 21 were injured orkilled during weekend periods and that a highpercentage of these injuries and deaths wereassociated with alcohol-related motor vehicle crashes(MVCs).

Action Plan: The PRAID coalition chose a multi-pronged strategy to reduce alcohol related injury anddeath experienced by male youth in their community.They proposed to:

1. increase patrols on routes and highways, setup alcohol check points on weekends—specifically on problem routes, and advertisethe strict enforcement of legal and monetarypenalties associated with drinking and driving(driving while under the influence).

2. promote alcohol education programs in allcity schools.

3. require that all students enroll in driverseducation with a mandatory three hours on therisks associated with alcohol and driving.

4. reduce the hours for selling liquor in the county.

Evaluation Plan: The PRAID coalition used thesix steps presented in this framework to think aboutthe evaluation. Some of the members had someevaluation experience, others had none. They knewhowever that the evaluation component was impor-tant to document their efforts as well as to try todetermine if their intervention made a difference.

City of HopeCase Example

6

..................................................................................................................

How many times have you heard yourself say “I’ll neverdo that again!” or, “I’ll try this and see how it works out,” or,“I really liked that restaurant more than the other”? Theseexamples show how we use evaluation in our everydaylives to make decisions, to test a new idea, or to makecomparisons.

When we reflect on our work with communities, we maythink of evaluation as a complex, formal, and difficultprocess. We might equate the evaluation process with theneed to collect program information that documents our workor to compare a group that participates in a program witha group that doesn’t. Many times the value of evaluation isnot immediately apparent to people who work to deliver theactual services (e.g., nurses in a local health department)or to the people who benefit from programs. Sometimeswe lose sight of evaluation’s benefits when funding agenciesemphasize documenting the numbers of people servedor when results are not shared with communities becauseof time pressures to get things done.

However, program evaluation, in fact, does have benefitsfor people in communities who carry out programs toimprove community health. As we developed this document,we spoke to individuals around the country who work dailywith community programs. On the following pages, theyshare their experiences and thoughts on the benefits andapplications of program evaluation.

Overview ofProgram Evaluation

7

..................................................................................................................

Reflect on progress; see where we’re going and where we’re coming from,and improve programs

Evaluation has helped us to focus and study and seewhere we’re at and where we need to go so that wecan find our own guns to put forth a better package . . . .

Community nurse

We forget sometimes to look at “Is it really working theway it’s already set up?” because we haven’t done anevaluation of the program . . . . Do we know it’s set up ina way that actually is having change?

Policy analyst

Influence policy makers and funders

The community needs to speak to funders. We need toimpact funders . . . . The benefit to evaluation is being ableto funnel upward and to trumpet upward, not only berecipients of something.

Community health worker

Build community capacity and engage communities

. . . There’s a capacity-building that goes throughout theprocess, and not that you’re going to make researchersout of everybody, but [you’re] elevating everybody’sunderstanding and ability to really become engaged.

CBO director

Benefits of Program Evaluation

8

..................................................................................................................

Share what works and what doesn’t work with other communities

We want to answer: Does the success of our program havereplication in other places, and if so what are the universalsthere that then can be used as information for programdevelopment, planning, evaluation, at other sites?

CBO director

Ensure funding and sustainability

Evaluation is really about people. But it comes throughin numbers. And then those numbers, when presentedto the person that gives out the funds, it has credibility.

Community nurse

I’ve worked with a lot of small . . . community based groupsthat as a result of the evaluation they were able to highlightthose things in their annual reports or in their fundingrequests to different funders and so forth, and as a resultof that, be able to get funds. And actually, one of them hasbeen able to double their funding base and also the amountthat they get as a result of some very systematic programevaluations we’ve done together.

Program evaluation consultant

Strengthen accountability

You have so many boards where consumers are members,they have fiduciary responsibilities . . . . You do have peoplewho might not have a lot of training in program design,implementation, evaluation, but ultimately have and arequite aware of having a responsibility for the impact andoutcomes of those programs. And so, again, I think that it’simperative that those people have access to evaluate andunderstand things that they are legally responsible for.

Community health worker

9

..................................................................................................................

In addition to benefits identified by people who work dailyin communities, program planners and evaluators believeprogram evaluation to be important when there is a need to:

◆ collect evidence on a program’s effectivenessor impact

◆ be accountable to funders, volunteers, staff,and boards

◆ identify ways to improve a program:● assess the needs of individuals, groups,

communities● improve the usefulness of program materials● determine what works, what doesn’t and why

◆ clarify program plans◆ improve communication among those involved in the

programIn sum, program evaluation is beneficial and important.

It is a means to gain feedback and ensure that everyone isworking toward successful program implementation.Sometimes informal evaluation is enough. However, whenthe stakes are high— when a good deal of time or money isinvolved, when a funder requires it, or when many peoplemay be affected— then it may make sense for yourorganization to use evaluation procedures that are moreformal, visible, and justifiable.

What do we mean by program evaluation?Thus far we’ve identified and discussed several aspects

of program evaluation, without presenting definitions of theterms. As we define it, a program is a series of activitiessupported by a group of resources intended to achievespecific outcomes among particular individuals, groups,and communities. The term program also refers to theeffort that is being evaluated. It may apply to any actionthat seeks to improve outcomes for whole communities,for more specific sectors (e.g., schools, work places), or forsub-groups (e.g., youth, people experiencing violence orHIV/AIDS). Examples of different types of programs include:

10

..................................................................................................................

◆ Direct service interventions (e.g., a programthat offers free breakfast to improve nutrition forgrade school children)

◆ Community mobilization efforts (e.g., an effort toorganize a boycott of California grapes to improvethe economic well-being of farm workers)

◆ Research initiatives (e.g., an effort to find outwhether disparities in health outcomes based onrace can be reduced)

◆ Advocacy work (e.g., a campaign to influencethe state legislature to pass legislation regardingtobacco control)

◆ Training programs (e.g., a job training programto reduce unemployment in urban neighborhoods)

Program evaluation is the systematic collection,analysis, and reporting of information about a program toassist in decision-making. Many of us already assess ourefforts without necessarily calling it evaluation. We assessthe value and impact of our work all the time when weask questions, consult partners, make assessments basedon feedback, and then use those judgments to improveour work.

When we conduct program evaluation, we may answerone or more of the following questions:

◆ What have we done?◆ How well have we done it?◆ Whom have we done it to?◆ How much have we done?◆ How effective has our program been?◆ What could we do better or differently?

Program planning and implementation are directly related to program evaluation.As you review the framework, it will be helpful to remember that each is partof a larger, integrated program cycle. Thus, while we focus on the programevaluation process here, evaluation is not really separate from theprogram planning and implementation processes.

11

..................................................................................................................

These questions fit one of three common types ofprogram evaluation: process evaluation, intermediate orshort-term (impact evaluation), and longer-term (outcome)evaluation. In general:

Process evaluation answers questions abouthow the program is implemented and how the programoutcomes are achieved. It focuses on questions such as:Is the program being implemented as planned? How isthe program achieving its objectives? What activities wereconducted? What materials or services did participantsreceive? What did people experience? How is our coalitionworking? Do we have the “right” stakeholders? In addition,process evaluation tracks the strengths and weaknesses ofthe program and seeks to identify what parts of the programare working and which are not.

Intermediate or short-term evaluation (impactevaluation) answers questions about the short-termeffects or benefits of a program— as opposed tolong-term outcomes such as injury or death. It focuses onquestions such as: What effects did the program have?Can the effects be attributed to the program? Did programparticipants’ knowledge, attitudes, beliefs, or behaviorschange as a result of the program? Did the training programachieve its objectives? What happened as a result of thecoalition’s efforts? In short, intermediate evaluationaddresses the factors that are believed to precede, and thatare linked to, longer-term outcomes. In Step 2— Describethe Program— we describe a process (i.e., logic model) thatcan be used to illustrate the relationship between programactions and outcomes.

Long-term evaluations (outcome) often focus onhealth status, injury (morbidity), death (mortality), or systemschanges. In many health programs the long-term goals areso distant that evaluating them is beyond the range of thespecific program evaluation. Outcome evaluation questionsare generally related to the overall program goal: Whatchange in injury or death occurred because of the program?What is the current prevalence (how many cases of “x”

12

..................................................................................................................

exist?) or what is the current incidence of “x” (how manynew cases of “x” occurred this year?)?

Different professions and academic disciplines use avariety of names to describe the different types of evaluation.Some evaluators consider formative evaluation— used toprovide ongoing feedback for program improvement— as asub-set of process evaluation. We agree with them for thepurposes of this document. In some evaluation circles,summative evaluation is used to describe both short- andlong-term evaluation processes. Some consider programmonitoring as another type of evaluation, although ingeneral, its use is limited to measuring program progress.

The importance of evaluation lies in its purpose and not in specific terminology.We encourage you to first think about what you want to find out about yourprogram, how the information will be used, by whom, and for what purposes.Answers to these initial questions will serve to guide your evaluationprocess before you “name” the type of evaluation you are using.

Additional program evaluation terms used in thisdocument are included in the Glossary. We encourage youto add your own notes or examples to the glossary to buildyour understanding.

13

..................................................................................................................

Stakeholders are:

◆ people who care about what will be learned from the evaluation and aboutwhat will be done with the knowledge gained

Community people, health agency people, andrepresentatives from community-based organizations havedescribed stakeholders as:

◆ people with interest in the program◆ all the necessary partners◆ collaborators◆ those affected by the program◆ shareholders◆ all who have the same shared interest

Another way of thinking about evaluation stakeholdersis to think about the type of stakeholder. Individuals andgroups from many different arenas can be categorized inthree main groups of stakeholders, depending on whattheir role is in the planning and evaluation process. Thefollowing diagram illustrates these three groups.

Definition

EngageStakeholders

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 12

6

35

4

14

..................................................................................................................

The primary users of the evaluation are made up ofpeople from the two other groups. The primary users arethe specific individuals or groups who are in a position todecide about and/or do something with the results. Asuccessful evaluation will designate its primary intendedusers— such as community based organizations, groupsof citizens, program staff and funders— early in theevaluation’s development, and will maintain frequentinteraction with users to be sure that the evaluationspecifically addresses their values and needs.

Why is it important to include differentstakeholders?

Evaluation cannot be done in isolation. Almosteverything done in community health and developmentwork involves partnerships— alliances among differentorganizations, board members, those affected by theproblem, and others. Therefore, any serious effort to

Types of Stakeholders

Primaryusersof the

evaluation

Those involvedin implementing

theprogram

clientsfamily members

neighborhood organizationsacademic institutions

elected and appointed officialsadvocacy groups

community residents

community memberssponsors

collaboratorscoalition partnersfunding officialsadministrators

managersstaff

ñ

ñ

Those servedor affected by

theprogram

ñ

15

..................................................................................................................

evaluate a program must consider the different values heldby the partners or stakeholders.

Stakeholders must be part of the evaluation so that theirunique perspectives are understood. When stakeholdersare not appropriately involved, evaluation findings may beignored, criticized, or resisted.

What’s involved in identifying stakeholders?When identifying stakeholders we ask:◆ Who cares about the program?◆ What do they care about?◆ Which individuals support the program?◆ Which individuals are openly skeptical of,

or antagonistic toward, the program?

Opening an evaluation to opposing perspectives andenlisting the help of potential program opponents canstrengthen the evaluation’s credibility. Likewise, individualsor groups who could be adversely or inadvertently affectedby changes arising from the evaluation have a right to beinvolved. This means including those who would be affectedif program services were expanded, altered, limited, orended because of the evaluation.

The amount and type of stakeholder involvement will bedifferent for each program evaluation. In many instancesstakeholders will be directly involved in designing andconducting the evaluation. They can be kept informedabout progress of the evaluation through periodic meetings,reports, and other means of communication.

Some community initiatives form an evaluation team—made up of various stakeholders— as part of their overallplanning and evaluation process. While not all membershave program evaluation experience or training, at leastsome members do. This type of arrangement can workwell when there is a commitment to build the capacity ofcommunity-based organizations, local advocacy groups,or interested citizens to learn about and participate inprogram evaluation.

16

..................................................................................................................

Early on, an informal project planning committee cametogether to plan the program and the evaluation. Becausethey knew how important it was to include a variety ofstakeholders, they identified and recruited communitystakeholders (parents, students, and local businessowners) in the initial phases of the project through publicforums, by word of mouth, and as they conducted theircommunity assessment. As these stakeholders— whobecame members of the PRAID coalition— met and sharedtheir ideas, the following interests emerged:

After the project staff were on board, staff spent manyhours with stakeholders— in meetings or one-on-onetalks— to understand what the stakeholders caredabout and to find ways to share concerns with otherstakeholders. They relied on formal and informalcommunication to make sure that as many stakeholdervoices were heard as possible.

As the project matured, new stakeholders becameinvolved, including the restaurant association president anda representative from the regional beverage (alcohol)distributor. The latter were primarily interested in balancingthe negative image of drinking and driving among teenswith responsible drinking in legal drinkers (e.g., adults).Their views were important as the PRAID coalition refinedits program plan and focused the evaluation.

Mothers Against DrunkDriving (MADD) Reducing death and injury among young people

Students Against Drunk Preventing alcohol use in teens, especiallyDriving (SADD) related to motor vehicle crashes (MVCs)

Health department Protecting residents from alcohol-related MVCs

Hospital Reducing the resources spent on MVCs

High schools Changing teens’ positive image of drinking

Police department Reducing everyone’s risk on the highways

City officials Changing the town’s image

School of Public Health Improving the health of the community

IdentifyingStakeholders

Early Stakeholders What They Cared About Most

17

..................................................................................................................

◆ Have you included individuals and organizations thatwill be affected by the evaluation in your evaluationplanning group?

◆ Have you considered adding new stakeholders asyour program evaluation is implemented?

◆ Are participants in the evaluation planning grouptrustworthy and competent?

StandardsQuestions

Utility: Ensures thatthe evaluation is usefuland answers questionsthat are directly relevantto users.

Propriety: Ensuresthat the evaluation is anethical one, conductedwith regard for the rightsand interests of thoseinvolved.

◆ Are individuals clear about what is to be done, how,by whom, and when?

◆ Is there a written understanding?◆ Have steps been taken to assure that all stakeholders

and the population served will be respected and theirvalues honored?

◆ Have conflicts of interest been discussed to ensurethat the results or findings will not be compromised?

Applying standardsIncluding stakeholders in evaluation planning and

implementation is one way to ensure a quality evaluation tomeet evaluation standards. The two standards that applymost directly to Step 1— Engage Stakeholders— are utilityand propriety. As you carry out this step, the questionspresented in the following table will help you to clarify andachieve these standards.

In summary, Step 1— Engage Stakeholders—represents a process through which many voices are heard.As the first step, it makes the benefits of the evaluationclear to all stakeholders. Completing this step helps ensurethat the focus of the evaluation— and ultimately the resultsof the evaluation— supports the needs of the stakeholders.

Standards for Step 1: Engage Stakeholders

18

..................................................................................................................

Who are the people who fall into the three types of stakeholder categories,i.e., those involved in implementing the program, those served or affectedby the program, and the primary users of the evaluation?

How could you go about finding out what each stakeholder cares about?

What communication strategies could you use to ensure that different interestsare represented?

What challenges or barriers might you face in identifying and recruitingstakeholders?

How could you deal with these challenges or barriers?

Think about a program evaluation in which you are a stakeholder or a programfor which you would be responsible for identifying and bringing people together.The following questions may help you think about how to approach Step1—Engage Stakeholders.

19

..................................................................................................................

Definition

A program description:

◆ summarizes the program being evaluated

◆ explains what the program is trying to accomplish and how it tries tobring about those changes

◆ illustrates the program’s core components

◆ establishes a program’s ability to make changes

◆ specifies its stage of development

◆ describes how the program fits into the larger organizational andcommunity environment

Why is it important to describe the program?How a program is described sets the frame of reference

for all future decisions about its evaluation. For example,one program may be described as “attempting to strengthenenforcement of existing laws that discourage underagedrinking.” Another program may be described as “aprogram to reduce drunk driving by teens.” The firstprogram specifies enforcement, where the second programis more broadly defined as reducing drunk driving. The

Describethe Program

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 22

6

35

4

20

..................................................................................................................

ProgramDescription

different emphases of these two programs will shape thedirection of evaluation efforts.

In addition, the program description allows members ofthe evaluation group to compare the program to similarefforts, and makes it easier to figure out what parts of theprogram brought about what effects.

Different stakeholders may have different ideas aboutwhat the program is supposed to achieve and why. Forexample, in a program to reduce teen pregnancy somestakeholders may believe this means increasing access tocontraceptives, while others may believe it means focusingon abstinence. Evaluations done without agreement on theprogram definition aren’t likely to be very useful. In manycases, the process of working with stakeholders to developa clear and logical program description will bring benefitslong before data are available to measure programeffectiveness. (This is a good example of why it’s importantto identify stakeholders [Step 1] before proceeding with thenext steps.)

As we will see in the next step, PRAID coalitionmembers decided to focus on reducing injuries anddeath in young men— a broad goal. Coalition membershad different ideas, however, about how to reduce theinjury and death rates. Police departmentrepresentatives were sure that increased penalties fordrinking and driving were the key to reducing injuryand death; school administrators were convinced thateducation was key. Through a participatory planningprocess, coalition members and other stakeholdersdeveloped a long-term plan that incorporated differentideas and priorities within the broad goal of reducingalcohol-related injuries and deaths among young men.Their program description spelled out their plan toachieve this goal.

21

..................................................................................................................

What’s involved in describing, or what are theelements of, a program?

There are seven elements of a program description.A well defined program description lays the foundation forfocusing the evaluation (Step 3).

1. Statement of need2. Expectations3. Activities4. Resources5. Logic model6. Context7. Stage of development

The first five elements are related in a very concreteway to the development of the program. The last twoelements— the context and stage of program development—are important, yet are larger than the program itself andare outside the technical description of the program.The first four elements are shown in the following illustration.

Statement of Need

What we need toknow to describethe problem/issue

Include:Who is affected?How big is the

problem?Is it changing?How is it changing?

Expectations

What our resultswill be

Define:What are our

expectations?What are immediate,

intermediate,and long-termconsequences?

What are theobjectives, goals,mission and visionof our program?

Activities

What we needto do to changethe problem

Identify:What are the specific

strategies andactions we needto take?

Resources

What resourceswe need

Determine:◆ time◆ talent◆ equipment◆ information◆ money◆ other assets

Four elements in describing a program

22

..................................................................................................................

◆In

crea

se h

ighw

ay p

atro

ls/e

nfor

cem

ent

in th

e co

unty

(e.g

., al

coho

l che

ckpo

ints

)◆

Incr

ease

DU

I san

ctio

ns fo

r mal

es u

nder

21

◆In

crea

se th

e pa

rtici

patio

n of

you

ng m

enin

alc

ohol

edu

catio

n pr

ogra

ms

and

driv

er’s

edu

catio

n

◆R

educ

e th

e ho

urs

for s

ellin

g liq

uor

in th

e co

unty

◆In

crea

se le

gal a

nd fi

nanc

ial p

enal

ties

for

alco

hol v

endo

rs s

ellin

g to

mal

es u

nder

21

◆B

uild

and

sus

tain

coa

litio

n of

inte

rest

ed p

artie

s

◆S

uppl

emen

tal f

undi

ng fo

r che

ckpo

ints

◆Tr

aini

ng fo

r law

enf

orce

men

t offi

cial

s

◆A

lcoh

ol e

duca

tion

curr

icul

a◆

Fund

ing

for a

lcoh

ol a

nd d

river

s’ e

duca

tion

prog

ram

s

◆S

taff

to p

erfo

rm “

stin

g” o

pera

tions

tom

onito

r ven

dors

◆R

esou

rces

to d

isse

min

ate

info

rmat

ion

abou

t the

pro

gram

to a

lcoh

ol o

utle

ts

◆C

omm

unity

gro

ups,

pro

fess

iona

lor

gani

zatio

ns a

nd m

unic

ipal

age

ncie

s◆

Pro

gram

sta

ff

Exp

ecta

tions

Act

iviti

esR

esou

rces

Goal

: Red

uce

alco

hol-r

elat

ed in

jury

and

dea

th

Obje

ctiv

e: B

y 20

05 re

duce

the

inci

denc

e of

alc

ohol

-rel

ated

MV

Cs

in m

ales

und

er 2

1 by

50%

.

Stat

emen

t of N

eed

(the

pro

blem

): T

hree

-yea

r inc

reas

e in

the

num

ber o

f alc

ohol

-rel

ated

mot

or v

ehic

le c

rash

es (M

VC

s) a

mon

gm

ales

und

er 2

1

Case

Exa

mpl

e: E

lem

ents

of t

he C

ity o

f Hop

e’s

prog

ram

to re

duce

inju

ry a

nd d

eath

in m

ale

yout

h

Sub

-Obj

ectiv

e A

:

Incr

ease

lega

l san

ctio

ns a

nden

forc

emen

t for

driv

ing

unde

rth

e in

fluen

ce (D

UI)

Sub

-Obj

ectiv

e B

:

Incr

ease

kno

wle

dge

amon

gun

dera

ge y

outh

of t

he ri

sks

asso

ciat

ed w

ith d

rinki

ng a

nd d

rivin

g

Sub

-Obj

ectiv

e C

:

Elim

inat

e th

e av

aila

bilit

y of

alc

ohol

to m

ales

und

er 2

1

Sub

-Obj

ectiv

e D

:

Incr

ease

col

labo

rativ

e pr

oble

m-

solv

ing

amon

g co

mm

unity

, age

ncy,

and

univ

ersi

ty s

take

hold

ers

23

..................................................................................................................

A logic model is a map that graphically illustrates howyour program activities will lead to the expected short-term—and long-term— outcomes. A logic model will make it clearwhether your program makes sense and whether theexpected outcomes are realistic given the planned activities.In short, the model shows what your program is expectedto achieve and how it is expected to work, based on anexpected chain of events that:

Link the clear specification of theproblem/issue you are addressing to

What you believe is needed to change the problem(the investments) to

The procedures, activities, and products it produces to

The shorter term outcomes to

The final consequences (long-term outcomes)

(Adapted from University of Wisconsin Evaluating Collaboratives: Reachingthe Potential, July 1999, G3658-8)

The logic model synthesizes the main program elementsinto a picture of how the program is supposed to work.Often displayed in a flow chart, map, or table, the logicmodel portrays the sequence of steps leading to programresults. A simple logic model might look like this:

InvestmentsProblemor issue

Procedures,activities,and products

Short-termoutcomes(related tosub-objectives)

Longer-termoutcomes(related toprogram goal)

Pregnantteens arenot usingprenatalservices

If:There is aninvestmentof time andmoney

to developa resourcedirectory

then:teens willbe informedabout availableservices

then:teens will beable to gainaccess tothe servicesto meettheir needs

Needs, expectations,and activities are reflectedin the developmentof the Logic Model.

24

..................................................................................................................

Logic models have several benefits. They:◆ allow stakeholders to improve and focus

program direction◆ reveal assumptions about conditions for

program effectiveness◆ provide a frame of reference for one or more

evaluations of the program◆ can be a basis for estimating the program’s effect

on endpoints that are not directly measured

InvestmentsProblemor issue

Procedures,activities,and products

Short-termoutcomes(related tosub-objectives)

Longer-termoutcomes(related toprogram goal)

Alcohol-related injuryand deathin malesunder 21

If:◆ law

enforcementresources arecommitted

◆ educationalresourcesand skills arecommitted

◆ people cometogether

◆ to strengthenenforcementof DUI laws

◆ to providealcohol anddriver’seducation

◆ to fosterchange inalcohol sales

then we’ll see:◆ increased

citations

◆ increasedknowledgeabout risks

◆ decreasedsales

◆ changesin drinkingand drivingbehaviors

then:there will bea reductionin injuriesand deathsdue to alcohol

A simple model for the program is illustrated inthe following:

In “story form” the logic model describes a programthat says: “When law enforcement resources arecommitted, where educational resources and skillsare committed, and when people come together, therewill be stronger law enforcement, increased provision

Logic Model

25

..................................................................................................................

When PRAID coalition members came together todiscuss how they could reduce alcohol-related injuries,they first talked about past efforts. The local MADDchapter had successfully advocated for new alcoholpolicies at the state level but had had little successwith the county commission in the past. Two newcounty commissioners, however, had recently beenelected; one of them had lost a nephew in an

ProgramContext

of education services, and efforts to change alcoholsales policies. When these activities are in place, wewill see an increase in penalties for driving under theinfluence, increased evidence of knowledge aboutalcohol-related risks, and a decrease in alcohol salesto underage youth. These changes will ultimately leadto a reduction in injuries and deaths related to alcoholin the under-21 male population in our area.”

A description of a program’s context considers theimportant features of the environment in which the programoperates. This includes understanding the context interms of:

◆ history◆ politics◆ geography◆ social and economic conditions◆ what other organizations have done

A realistic and responsive evaluation will be sensitiveto a broad range of potential influences on the program.An understanding of the context lets users interpret findingsaccurately and assess the findings’ generalizability. Forexample, a program to improve housing in an inner-cityneighborhood might be a tremendous success, but wouldlikely not work in a small town on the other side of thecountry without specific adaptation.

26

..................................................................................................................

Evaluation Goal

alcohol-related motor vehicle crash— a potential allyfor their initiative. Stakeholders also discussed thelocal DARE program and how it had been part ofalcohol prevention programming for younger childrenfor several years. The PRAID coalition knew that thepolice department was heavily invested in the DAREprogram, raising concerns that resources to fund theproposed PRAID strategies might be perceived ascompeting with existing programs such as DARE.Finally, the coalition members thought about how theirprogram’s educational components could be maderelevant to a wide variety of students— acknowledgingcultural differences, and also recognizing thatindividuals learn in many different ways. As the PRAIDstakeholders refined their program and the evaluation,they were sensitive to these issues.

A program’s stage of development also affects theevaluation process. For example, an evaluation of a newprogram may differ significantly from an evaluation of aprogram that has existed for a number of years. One wayof viewing the intent of evaluation at different stages isthrough the different goals of evaluation at three commonlyrecognized stages:

Stage of Program

Planning: program activities are untested

Implementation: program activitiesare being field tested and modified

Effects or outcomes: enough timehas passed for the program’s effectsto emerge

To refine plans as much as possible

To see what happens in the “real world”and to improve operations

To identify and understand the program’sresults, including those that wereunintentional

27

..................................................................................................................

Applying standardsAs in the previous step, you can assure that the

evaluation is a quality one by incorporating standards intothe program description. The two standards that applymost directly to Step 2— Describe the Program— areaccuracy and propriety. As you carry out this step, thequestions presented in the following table can help you toclarify and achieve these standards.

◆ Is the program description complete?◆ Have you documented the context of the program so

that likely influences on the program can be identified?

StandardsQuestions

Accuracy: Ensuresthat the findings canbe considered correct.

Propriety: Ensuresthat the evaluation is anethical one, conductedwith regard for the rightsand interests of thoseinvolved.

◆ Is the evaluation complete and fair in assessing allaspects of the program, including its strengths andweaknesses?

Standards for Step 2: Describe the Program

In summary, Step 2— Describe the Program— outlinesthe foundation to ensure a well thought out program evalu-ation. It is important to keep in mind that the breadth anddepth of a program description will vary for each programevaluation, meaning that many different activities may be partof developing the description. Multiple sources of informationmay be pulled together to construct a well-rounded descrip-tion. Discussion with stakeholders can confirm the accuracyof an existing program description. Descriptions of what’sgoing on may be checked against direct observation ofactivities in the field. Addressing contextual factors (such asstaff turnover, inadequate resources, political pressures, orstrong community participation) that may affect the programcan broaden an otherwise narrow program description.

28

..................................................................................................................

How are your program’s goals, objectives, and strategies defined?

How are your program’s activities, processes, and products linkedto the program’s outcomes?

What resources might be available to implement the program?

What else might be happening in your community that could have an impacton your program? What other programs have been tried, and by whom?

Is your program new or has it existed for a year or more? (As you read thenext step, think about how this could influence how you focus your evaluation.)

Think about a program evaluation in which you are involved. The followingquestions may help you think about how to approach Step 2— Describethe Program.

29

..................................................................................................................

Definition

Focusing the evaluation design means:

Why is it important to focus the evaluationdesign?

This question involves determining the users andthe uses of the evaluation. It gets at the direct purposeof the evaluation. Some common purposes could be to:

◆ test program effectiveness◆ make a case to change program practices◆ assess the effects of a program on a specific

population◆ justify continued financial or political support

Additional considerations include making sure thatthe chosen evaluation approach answers stakeholderquestions and that the process provides continuousfeedback at all levels of program operation.

◆ carrying out advance planning about where the evaluation is headed andwhat steps it will take to get there

◆ developing a well-focused plan or strategy to improve the usefulness of theevaluation to intended audiences

It’s a stipulation within thegrant to have an evaluationplan in place from the start.[It’s] necessary that periodicreports are submitted to thefunder on where the evalua-tion stands in the process.

Community Member

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 32

6

35

4

Focus theEvaluationDesign

30

..................................................................................................................

What’s involved in focusing an evaluationdesign?

Once you have determined who or what the evaluationis for, there are several steps to focus the evaluationdesign. They include:

◆ determining the information needs of your variousstakeholders

◆ assessing the best techniques to describe andmeasure your program activities

◆ assessing what qualitative and quantitative datamay be available to you

◆ determining the design method that best answersthe key questions set by stakeholders

◆ preparing a written agreement that summarizes theevaluation procedures and specifies the roles andresponsibilities of all involved

There are seven basic issues to consider when focusingan evaluation:

ñ1. Users

2. Uses

3. Purpose

4. Questions

ñ5. Design

6. Methods

7. Agreements

ñ

Users are the specific persons who will use evaluationfindings. Because they directly experience the conse-quences of the trade-offs that are part of any evaluation,they have a right to participate in choosing the focus for theevaluation. When users are encouraged to clarify intendeduses and identify priority questions and preferred methods,the evaluation is more likely to focus on things that willinform and influence future actions.

31

..................................................................................................................

Uses describe what will be done with what is learnedfrom the evaluation. Information collected may have varyinguses, which should be described in detail when designingthe evaluation. This directly feeds into defining the purposeof the evaluation and the questions to be asked.

Usersand Uses

[Evaluation] is demandedwhen we work with externalorganizations. There’sa demand to have anevaluation conducted andbe able to have, you know,meaningful quantitativedata to show results.

Community Member

Users Uses

Police Department

PRAID Coalition

City Officials

Could use the information fromthe evaluation to determine howto allocate their internal resourcesto better serve the public.

Could employ evaluationinformation to bolster datarequired for additional fundingfor program sustainability.

Could use evaluation informationto formulate new legislationrelated to each objective.

Illustrated here is a list of evaluation users and theirintended uses for the information:

Purpose refers to the general intent of the evaluation.A clear purpose serves as the basis for the evaluationquestions, design, and methods. Community organizationsmight become involved in evaluation to:

◆ gain new knowledge about program activities◆ improve or fine-tune existing program operations

(e.g., program processes or strategies)◆ determine the effects of a program by providing

evidence concerning the program’s contributionsto a long-term goal

◆ affect program participants by acting as acatalyst for self-directed change (e.g., teaching

32

..................................................................................................................

The primary purpose of the evaluation as proposedby the PRAID coalition was to determine whether thestated program goals could be achieved by the year2005. The coalition wished to gain insight as towhether the new approaches they had outlined inthe form of specific objectives would have anymeasureable effect on alcohol-related injury and deathin young men. In addition, coalition members wereinterested in the processes by which the primary goalwould be achieved. They viewed the building ofrelationships among PRAID coalition members,community members, service providers, and localgovernment representatives as key to the project’sability to effectively implement its action plan.Recognizing that the implementation of specificprogram objectives had the potential to buildcommunity resources and skills as well as communitycapacity, coalition members decided that theevaluation should also focus on measuring anddescribing changes in these community processes.

Purpose

Questions establish the aspects of the programthat will be addressed and encourage stakeholders toreveal the questions they believe the evaluation shouldanswer. Clearly worded questions that address changesin individuals, within organizations, or in whole counties(units of analysis) will help guide method selection andevidence gathering.

evaluation skills to staff and stakeholders)◆ build capacity or the ability to develop trusting

working relationships among coalition members,health care providers and local governmentrepresentatives

33

..................................................................................................................

The questions that the PRAID coalition membersidentified were strategically linked to the programobjectives to reveal what they believed the evaluationshould address. Questions

(Table continued on page 34)

ProgramObjectives

EvaluationQuestions

InterventionStrategies

Short-termOutcomes

Long-termOutcomes

Increase legalsanctions fordriving underthe influence(DUI)

Will legalsanctions andincreased high-way patrolsreduce theincidence ofDUI behaviorin malesunder 21?

◆ Increasepatrols andenforcement(e.g., alcoholcheckpoints)

◆ IncreaseDUI sanctionsfor malesunder 21

Increasedcitations

Reduction inalcohol-relatedmotor vehiclecrashes

Increaseknowledgeamong malesunder 21about the risksassociatedwith drinkingand driving

Will increasedparticipation ofmales under 21in alcoholand driver’seducationresult in loweralcohol-relatedmorbidity andmortality?

Increasedparticipation ofmales in alcoholeducation anddriver’s educa-tion programs

◆ Increasedknowledgeabout risksassociatedwith alcohol

◆ Changesin drinkingand drivingbehaviors

Reductionin injury anddeath relatedto alcoholin theunder-21 malepopulation

Limit theavailabilityof alcohol tomales under 21

Will increasedlegal penaltiesand time ofday restrictionson the saleof alcohol tomales under 21result in lessunderagedrinking anddriving?

◆ Reducehours ofliquor sellingestablish-ments inthe country

◆ Increase legaland financialpenaltiesfor alcoholvendors sell-ing to malesunder 21

Decreasedsales to malesunder 21

Reductionin injury anddeath relatedto alcoholin theunder-21 malepopulation

34

..................................................................................................................

(Table continued from page 33)

ProgramObjectives

EvaluationQuestions

InterventionStrategies

Short-termOutcomes

Long-termOutcomes

Increase citizenparticipation;develop leader-ship, resources,and social andinterorganiza-tional skills

Will the processof workingcollectivelytoward theachievementof the programgoal developcommunitycapacity?

◆ Regularmeetings ofthe PRAIDcoalitionsubgroupsto assist inprogramplanning,implementa-tion andevaluation

Increasedsense ofcommunity;improved socialintegration

Improvedcommunityproblem-solving

Design refers to how the evaluation’s questions,methods, and overall processes are constructed. Theevaluation should:

◆ be organized from the start to achieve specificagreed-upon uses

◆ have a clear purpose that is focused on the utilityof what is learned

◆ help those who carry out the evaluation to knowwho will do what with the findings

The process of developing a clear design can highlightthe ways that stakeholders, through their many contributions,can improve the evaluation and facilitate the use of the results.

When choosing evaluation methods, evaluation planninggroups usually consider whether the methods will allowstakeholders to effectively answer the questions that arebeing raised. Information on how the evaluation data will beused is key to decisions about methods. Different evaluationquestions require different kinds of data that, in turn, requiredifferent types of methods. Counting how many timesan event occurs calls for quantitative methods, whereasqualitative methods might be used to develop an under-standing of the social context in which an event occurs.

35

..................................................................................................................

The PRAID coalition was interested in reducingalcohol-related injury and death among thecommunity’s male youth. In selecting a design toevaluate the interventions’ longer-term effects,the coalition determined that there were no nearbycommunities of like size and demographics suitableto serve as a comparison group. Taking into accountresources and the evaluation questions of interest tostakeholders, the coalition decided that theirstrategy would include assessing trends over time,identifying key issues, and determining therelationships among key events and outcomes. ThePRAID coalition also considered testing the effec-tiveness of the alcohol and driver’s education pro-grams by comparing males under 21 in the City ofHope with males under 21 in a like-sized city who didnot get the training. Due to financial constraints andpolitical considerations, however, this approach wasnot considered feasible.

Designand Methods

Different evaluation designs can require a varietyof methods, including the following:

◆ systematic participant observation◆ structured or semi-structured interviews◆ focus groups◆ descriptive or analytical surveys

Agreements summarize the evaluation procedures,clarify everyone’s role and responsibilities, and describehow the evaluation procedures will be implemented.Elements of an agreement include: statements concerningthe intended users, uses, purpose, questions, design, andmethods as well as a summary of the deliverables, timelineand budget. An agreement might be a legal contract, amemorandum of understanding, or a detailed protocol.Creating an agreement establishes a mutual understandingof the activities associated with the evaluation. It alsoprovides a basis for modification if necessary.

36

..................................................................................................................

Applying standardsAs we have noted in previous steps, you can help

ensure the quality of your evaluation by consideringrelevant evaluation standards throughout the evaluationprocess. The standards that most directly apply toStep 3— Focus the Evaluation Design— are utility,feasibility, propriety, and accuracy. As youparticipate in focusing the evaluation design, the questionspresented in the following table can help you to clarifyand achieve these standards.

Standards for Step 3: Focus the Evaluation Design

Questions Standards

◆ How can the evaluation be planned, conducted,and reported so that it, in turn, encourages useby stakeholders?

Utility: Ensures thatthe evaluation is usefuland answers questionsthat are directly relevantto users.

◆ Are the evaluation procedures practical? Will they keepdisruption of daily activities to a minimum?

◆ Have you considered the political interests and needs ofvarious groups in planning the evaluation?

◆ Have you assessed the costs of technical resourcesand time?

Feasibility: Ensuresthat the evaluation makessense, takes into con-sideration the interestsof various groups, andcan be cost effective.

Propriety: Ensuresthat the evaluation is anethical one, conductedwith regard for the rightsand interests of thoseinvolved.

◆ Have you addressed issues of conflict openlyand honestly?

◆ Have you planned and implemented sound, ethicalconsistant procedures to ensure findings are correct?

Accuracy: Ensuresthat the findings areconsidered correct.

◆ Have you described the purposes and proceduresof the evaluation in detail?

◆ Can the purposes and procedures be indentifiedand assessed?

37

..................................................................................................................

In summary, Step 3— Focus the Evaluation Design—represents a process through which a design is structuredto capture the information that all stakeholders withinthe effort agree are critical. It ensures that the evaluationdesign meets the needs of all users, that the processanswers the questions that have been raised, and that theevaluation takes into account constraints imposed by timeand the availability of technical resources.

38

..................................................................................................................

What kinds of information needs are your users likely to have?

How could the information produced by the evaluation be used?

What are possible evaluation questions for your program?

What types of evidence could help you show that the program had theintended effect?

How could you go about getting the evidence you need?

Have you thought about who could help you with the technical and design aspectsof your evaluation?

Think about a program evaluation in which you are involved. The followingquestions may help you approach how you think about Step 3— Focus theEvaluation Design.

39

..................................................................................................................

Definition

Gathering and analyzing evidence means:

◆ assembling:● the raw material of a good evaluation● information that gives a well-rounded picture of the program● those data that have been analyzed and synthesized

◆ presenting meaningful results

Why is it important to gather credible evidence?Stakeholders should view the information gathered as

believable, trustworthy, and relevant to their questions.Credibility is based on the questions asked at the beginningof the evaluation process and stakeholders’ motives forasking them. In other words, standards of credibility dependon the questions asked.

Having credible evidence strengthens evaluation resultsas well as the recommendations that follow from them.Recognizing that all types of data have limitations, you canstrengthen the credibility of an evaluation design by usingmultiple procedures for gathering, analyzing and interpretingdata. Increased up-front participation by stakeholders alsoenhances credibility because they will be more likely to

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 42

6

35

4

Gatherand Analyzethe Evidence

40

..................................................................................................................

accept the evaluation’s conclusions and act on its recom-mendations.

In some situations, you may need to ask evaluationspecialists for advice on the most appropriate method(s)to use, given the data restrictions, resource constraintsand standards of evidence set by stakeholders. This isespecially important in situations where concern for dataquality is high or where errors of inference would haveserious consequences.

An essential part of a good evaluation is a review ofwhat is known, what has been done before and what hasbeen done elsewhere. Reviewing published literature andunpublished documents can help strengthen the processof designing an evaluation. Program participants and otherstakeholders also can be important sources of backgroundinformation. Familiarity with other evaluations or researchon similar interventions (or different interventions designedto address the same problem) can help you developcriteria for judging your own program.

What’s involved in gathering and analyzingevidence?

The following features of evidence gathering typicallyaffect how credible an evaluation is seen as being:

1. Indicators2. Sources of evidence3. Quality4. Quantity5. Logistics6. Analysis and synthesis

Indicators are used to judge programs. An indicator isa category of change. Indicators translate general conceptsabout a program and its expected effects into specific,measureable parts. They should reflect the aspects of theprogram that are most meaningful to monitor. Severalindicators are usually needed to track the implementationand effects of a complex program or intervention.

41

..................................................................................................................

Category of Change

Service delivery improvement(a measure of program capacity)

Changes in individual knowledge, skills, behavior

Changes in population health status

Changes in the environment

Logic models (see Step 2) can help develop indicators.A logic model can be used as a template to define a fullspectrum of indicators along the pathway that leads fromprogram activities to the program’s expected effects. Bothqualitative and quantitative indicators can be developed.

Examples of Measures

◆ Client satisfaction level◆ Number of patients immunized

◆ Participant behavior

◆ Injury rates◆ Death rates

◆ Programs, policies and practices

Sources of evidence in an evaluation may bepeople, documents, or observations. More than one sourcemay be used to gather evidence for a given indicator.In fact, selecting multiple sources of evidence providesan opportunity to include different perspectives about theprogram, thereby enhancing the evaluation’s credibility.For instance, program documentation reflecting an internal(staff) perspective could be combined with key informantinterviews with program users. Mixing perspectives providesa more comprehensive view of the program. In addition,the integration of qualitative and quantitative informationcan yield evidence that is more complete and useful, meetingthe needs and expectations of a wider range of stakeholders.

Selected Sources of Evidence◆ Written surveys ◆ Group assessments◆ Personal interviews ◆ Diaries or journals◆ Observation ◆ Geographical mapping◆ Document analysis ◆ Expert or peer review◆ Case studies ◆ Concept mapping◆ Logs, activity forms, registries

Examples of indicators include:

42

..................................................................................................................

Indicatorsand Sourcesof Evidence

The PRAID coalition identified a variety of possibleindicators that related to the program’s objectives.Possible sources of evidence included police records,emergency room data, questionnaire data from acommunity-level survey to assess and monitor theextent of the problem, new community assessmentdata, key informant interviews with youth attendingthe local high schools, and so forth. Coalitionmembers were also interested in using standardizedmeasures that would make it possible to measurechanges over time and across sites. Finally, to assessprogram progress coalition members identifiedprocess indicators and sources of evidence todocument short-term outcomes.

Objectives

Increase legal sanctionsfor DUI (driving under theinfluence) for males under 21

Use driver’s education andalcohol education programsto increase knowledgeamong young men aboutthe risks associated withdrinking and driving

Limit the availability ofalcohol to males under 21through increased legaland financial penalties foralcohol sales to this group

Indicators

◆ Number of DUI chargesfor underage youth

◆ Level of participationin driver’s educationand alcohol educationprograms

◆ Number of on-siteinspections of communityliquor vendors

◆ Liquor licenses on file

Sources of Evidence

◆ Police records of violations◆ Juvenile court filings

◆ School records forattendance at driver’seducation and alcoholeducation programs

◆ Violations records ofliquor vendors

◆ Inventory of liquor-servingestablishments

43

..................................................................................................................

Quality refers to the appropriateness and integrityof information gathered in an evaluation. Well-definedindicators make it easier to collect high quality data thatare reliable and informative. Other factors that affectquality include:

◆ instrument design◆ data collection procedures◆ training of those involved in data collection◆ source selection◆ coding◆ data management◆ routine error checking

Obtaining quality data involves trade-offs (i.e., breadthvs. depth). Thus, stakeholders must decide at the beginningof the evaluation process what is most important. A practicalevaluation should strive for a level of quality that will matchstakeholders’ standards for credibility.

Quality

In determining what was most important to them, thePRAID coalition decided that uniform instruments(unchanged or standardized) should be used toimprove the quality of the evaluation’s survey data.Thus, the same data collection instrument wasused at each data collection site. In addition, thecoalition developed an interview guide for keyinformant interviews. Other steps taken to assurequality included training interviewers. Trainingwas also provided for individuals involved in datacollection, coding, and data management. Thetraining emphasized standardized recordkeepingto ensure the completeness of data. Unexpectedchanges in one agency’s recordkeeping practicesrequired the coalition to reconsider whether andhow the agency’s records could be used; the coalitionalso considered substitute sources of evidence thatmight be available. Coalition members were careful

44

..................................................................................................................

Quantity refers to the amount of evidence gatheredin an evaluation. Evaluation stakeholders should estimatein advance the amount of information that will be required,and establish criteria to decide when to stop collecting data.Quantity affects the level of confidence or precision userscan have (i.e., how sure are we that what we’ve learned istrue?). It also partly determines whether the evaluation willbe able to detect program effects. All evidence collectedshould have a clear, anticipated use.

to pick sources of data that were fairly complete aswell as accessible (e.g., police blotter records).All these actions served to reduce the amount oferror within the data and improved the data’soverall reliability.

Quantity

PRAID coalition members decided to use bothqualitative and quantitative data to develop as wide apicture as possible of the program. First, they decidedto use key informant interviews to elicit informationfrom the under-21 population. After a series ofinterviews had been conducted, the interviewersbegan reaching a “saturation” point in data collection,where different key informants began to expresssimilar patterns or themes. Second, the coalitioncollected police, juvenile court, and hospital data forthe three-year period before the intervention, andannually throughout the intervention. Third, thecoalition’s university partners helped other stake-holders determine the quantitative evidence requiredto assess the program’s effectiveness. Specifically,the coalition determined the sample size necessaryto detect a difference in alcohol-related youth injuriesand deaths as a result of the intervention.

45

..................................................................................................................

Logistics are the methods, timing, and physical infra-structure for gathering and handling evidence. Peopleand organizations have cultural preferences that dictateacceptable ways of asking questions and collecting infor-mation, and influence who is perceived as an appropriateperson to ask the questions (i.e., someone known withinthe community versus a stranger from a local health agency).The techniques used to gather evidence in an evaluationmust be in keeping with a given community’s cultural norms.Data collection procedures should also protect confidentiality.

Logistics

Analysis and synthesis are methods to discoverand summarize evaluation findings. In evaluations that usemultiple methods, patterns in evidence are detected by:

◆ isolating important findings (analysis)◆ combining different sources of information to reach

a larger understanding (synthesis)

Analysis and synthesis involve deciding how to organize,classify, compare, and display information. These decisionsare guided by the questions being asked, the types of dataavailable and, especially, by input from stakeholders andprimary intended users.

PRAID coalition members determined that the studypopulation of young men would be more likely toparticipate in focus groups if peers and youth leaderswere used to market and facilitate the groups. Thefocus group meetings were scheduled in locationswhere young men in the town normally congregated.

Analysisand Synthesis

Members of the evaluation team examined pre- andpost-intervention questionnaire data as well asqualitative interview data for analysis and synthesis.The following questions were raised by coalitionmembers during this phase: What are the common

46

..................................................................................................................

Applying standardsAs we have noted in previous steps, you can help ensure

that your evaluation is a quality one by considering relevantevaluation standards throughout the evaluation process.The two standards that most directly apply to Step 4—Gather and Analyze Evidence— are utility and accuracy.As you participate in gathering and analyzing evidence, thequestions presented in the following table can help you toclarify and achieve these standards.

ideas, themes, and patterns within the qualitativeand quantitative data? What do the frequency datashow? Are there any surprising findings? Did allavailable males within the three designated highschools participate in the education programs? Theanswers to these questions were then displayedgraphically in charts that tracked changes in beliefs,behavior, and participation in alcohol education.Graphs were also created to show the number ofcitations issued to young men within each geographicarea over a given period of time.

StandardsQuestions

Utility: Ensures thatthe evaluation is usefuland answers questionsthat are directly relevantto users.

Standards for Step 4: Gather and Analyze Evidence

◆ Have you reached an agreement on techniquesto analyze and synthesize findings before datacollection begins?

◆ Will the information collected address pertinent issuesabout the program and is this responsive to needs ofyour stakeholders?

(Table continued on page 47)

47

..................................................................................................................

In summary, Step 4— Gather and Analyze Evidence—represents a process through which information about theprogram in which you are engaged can be gathered andsynthesized for subsequent presentation. It ensures thatthe benefits of evaluation (the uses of this information) areclear to all stakeholders and that the processes followedmeet everyone’s agreement.

StandardsQuestions

Standards for Step 4: Gather and Analyze Evidence(Table continued from page 46)

Accuracy: Ensures thatthe evaluation findingsare considered correct.

◆ Are the sources of information used in the programdescribed in adequate detail?

◆ Do the information-gathering procedures addressinternal validity and reliability issues?

◆ Is there a system in place for identifying andcorrecting errors?

◆ Has the process of analyzing quantitative andqualitative data been effective in answering yourkey evaluation questions?

48

..................................................................................................................

What expertise and resources could you draw on for help indefining your methods?

What sources of information could you use in the evaluation (people,documents, observations)?

What systematic processes could you use to gather information?

How could you check for errors and make corrections as part of thedata gathering process?

What data analysis and synthesis techniques could you consider?

How could your analysis process answer key stakeholder questions effectively?

Think about a program evaluation in which you are involved. The followingquestions may help you think about how to approach Step 4— Gather andAnalyze Evidence.

49

..................................................................................................................

Definition

Why is it important to justify conclusions?The evidence gathered for an evaluation does not

necessarily speak for itself. To substantiate and justifyconclusions, it is important to carefully consider the evidencefrom a variety of stakeholder perspectives. Conclusionsbecome justified when they are linked to the gatheredevidence and judged against values that stakeholders agreeupon. When communities, agencies and other stakeholdersagree that the conclusions are justified, they will use theevaluation results with more confidence.

Conclusions can be strengthened by searching foralternative explanations and showing why the evidencedoes not support them. Where there are different butequally well-supported conclusions, it can be helpful topresent each set of conclusions with a summary of itsstrengths and weaknesses.

Justifying conclusions involves:

◆ making claims about the program based on the evidence gathered

◆ justifying the claims by comparing the evidence against stakeholder values

JustifyConclusions

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 52

6

35

4

50

..................................................................................................................

What’s involved in justifying conclusions?Four principal elements are involved in justifying

conclusions:

The central objective of the community-basedintervention program was to reduce by 50% theincidence of alcohol-related motor vehicle crashes(MVCs) in males under 21 by the year 2005. Becausean important purpose of the evaluation was todetermine the program’s effects, it made sense to thePRAID coalition to use this program objective as oneof the standards for making judgments about theprogram. The objective of building coalitions andcommunity partnerships represented another keymeans of judging the program.

ProgramStandards

ñProgramStandards Interpretation ñJudgmentsñ Recommendations

Program standards (not to be confused with theevaluation standards discussed throughout this document)are fundamental to sound evaluation. They reflect stake-holders’ values about the program. Program standards:

◆ allow evaluators and stakeholders to makejudgments about the program

◆ provide alternative ways to compare program results

When stakeholders articulate and negotiate their values,these become the standards for considering a program“successful,” “adequate,” or “unsuccessful.” Examples ofprogram standards that can be used to assess programresults include community values or norms, professionalstandards, program objectives, fixed performance criteria,performance by previous or similar programs, considerationsof feasibility and sustainability, social equity, and otherstakeholder values.

51

..................................................................................................................

Interpretation

Interpretation is the effort to figure out what theevaluation findings mean. Interpretation:

◆ draws on information and perspectives thatstakeholders bring to the evaluation

◆ is strengthened through active stakeholderparticipation and interaction

Uncovering facts about a program’s performance is notenough to draw conclusions. The facts must be interpretedto understand their practical significance.

Imagine that we have stepped three years into thefuture. At that point in time, the evaluation findingsshow a modest reduction in alcohol-related motorvehicle crashes among male youth in the project’sgeographic area. As the PRAID coalition gathers tointerpret these findings, they may try to figure outwhether each of the three broad program strategies(increased enforcement/sanctions, comprehensiveeducation, reduced alcohol sales) has had an impacton the documented decline in alcohol-related crashes.

Citations for Alcohol Sales to Minorsand

Driving Citations to Underage Drinkers

Num

bers

of C

itatio

ns

700

600

500

400

300

200

100

Year 1 Year 2 Year 3

Alternatively, the coalitionmembers may focus oninterpreting the combinedcontributions of the set ofintervention strategies,rather than zeroing in on anysingle strategy.

The graph illustrates datacollected over a three-yearperiod. It summarizescitations issued to alcoholvendors (e.g., liquor stores,bars) for sales of alcohol tominors and alcohol-relateddriving citations issued tounderage drinkers in the

687

63 85 52

543

310

Alcohol sales citations

Driving citations

52

..................................................................................................................

City of Hope (population 22,781). Data can beanalyzed and interpreted to ascertain whethercollective intervention strategies have had any effecton alcohol-related MVCs.

Judgments are statements about a program’s merit,worth, or significance. They are formed when findings andinterpretations are compared against one or more selectedprogram standards. In forming judgments about a program:

◆ multiple program standards can be applied◆ stakeholders may reach different or even conflicting

judgments

Conflicting claims about a program’s quality, value,or importance often indicate that stakeholders are usingdifferent program standards or values in making theirjudgments. This type of disagreement can prompt stake-holders to clarify their values and reach consensus onhow the program should be judged.

Judgments

The evaluation documented an increase in highwaypatrols for underage drinking and driving in someareas of the county on selected weekends. Policedepartment stakeholders, who viewed any im-provement in performance as a sign of success,viewed this increase as a reason to judge the pro-gram positively. However, coalition members fromMADD, SADD, and community-based organizationshad an expectation that countywide patrols wouldbe increased every weekend; as a result, theirjudgment of program performance was less positivethan that of the police.

53

..................................................................................................................

Recommendations are actions to consider as aresult of an evaluation. Recommendations requireinformation beyond what is necessary to form programjudgments. Recommendations:

◆ can strengthen an evaluation when they anticipateand react to what users want to know

◆ may undermine an evaluation’s credibility if they arenot supported by enough evidence, or are not inkeeping with stakeholders’ values

The chances that recommendations will be relevant andwell received can be increased by sharing draft recommen-dations, soliciting reactions from multiple stakeholders, andpresenting options instead of directive advice.

Recommendations

After holding a community forum to further interpretthe evaluation results and make judgments aboutthe program, the PRAID coalition members used theresults to come up with recommendations that theypresented to the county commissioners and schoolboards. For example, because the evaluationsuggested that police patrols on weekends had hadan impact on young men’s drinking and drivingbehavior, the coalition recommended that the level ofpolice patrolling be permanently increased. However,the evaluation results showed low levels ofparticipation by high school students in the program’seducational initiatives, leading the coalition membersto recommend that community-based (rather thanschool-based) strategies be developed to raise youngmen’s awareness about the dangers of alcohol.

Applying standardsAs we have noted in the previous steps, you can help

ensure that your evaluation is a quality one by consideringrelevant evaluation standards throughout the evaluationprocess. The two standards that most directly apply to

54

..................................................................................................................

◆ Have you carefully described the perspectives,procedures and rationale used to interpret the findings?

◆ Have stakeholders considered different approachesfor interpreting the findings?

StandardsQuestions

Utility: Ensures thatthe evaluation is usefuland answers questionsthat are directly relevantto users.

Accuracy: Ensures thatthe evaluation findingsare considered correct.

◆ Can you explicitly justify your conclusions?◆ Are the conclusions fully understandable to

stakeholders?

Standards for Step 5: Justify Conclusions

In summary, Step 5— Justify Conclusions— involvesinterpreting evaluation results so that they make sense to allstakeholders and reflect stakeholders’ values about what isimportant. Completing this step can help stakeholdersconsider what actions to take as a result of the evaluation.

Step 5— Justify Conclusions— are utility and accuracy.As you participate in justifying conclusions, the questionspresented in the following table can help you to clarify andachieve these standards.

55

..................................................................................................................

How could you involve stakeholders in interpreting evaluationresults for your program?

If stakeholders had conflicting judgments about your program, how could youreach consensus?

What information could you use to develop recommendations for action?

How might you share recommendations with other stakeholders?

Think about a program evaluation in which you are involved. The followingquestions may help you approach how you think about Step 5— JustifyConclusions.

56

..................................................................................................................

Why is it important to ensure use and sharelessons learned?

It is naive to assume that lessons learned in anevaluation will necessarily be used in decision-making andsubsequent action. Evaluation participants must make adeliberate effort to promote use of the evaluation findings.They also have a responsibility to prevent misuse offindings. Factors that influence whether results are usedappropriately include: evaluator credibility; disclosure offindings; report clarity, timeliness, and impartiality; andchanges in the program or organization context. Ensuringuse requires thinking strategically from the earliest stagesof the evaluation, as well as diligently looking foropportunities to communicate results and influenceprogram decisions and/or policy makers.

Using evaluation results appropriately means:

◆ making stakeholders aware of the evaluation procedures and findings

◆ considering findings in program-related decisions or actions

◆ ensuring that those who participate in the evaluation perceive it as beneficial

[Evaluation] is a good thingto do for a lot of reasons.But let us be clear that manydecisions get made politicallyand have little or nothingto do with good data andthe value of what’s going on.

CBO Director

Definition

Ensure Useand ShareLessons Learned

STANDARDSUtility AccuracyPropriety Feasibility

1

STEP 62

6

35

4

57

..................................................................................................................

The very process of doing evaluation is important. Whenindividuals are exposed to the logic, reasoning, and valuesthat guide evaluation, their thinking and behavior maychange profoundly. Participation in an evaluation may:

◆ encourage stakeholders to base decisions onsystematic judgments instead of on unfoundedassumptions

◆ prompt stakeholders to clarify their understandingof program goals, thereby improving their abilityto function as a team

◆ help stakeholders clarify what is really importantthrough the process of defining indicators

◆ make outcomes matter by changing thereinforcements connected with achieving positiveresults, as when funders offer “bonus grants” or“outcome dividends” to programs that showsignificant community change and improvement

What’s involved in ensuring use and sharinglessons learned?

Four elements are important in making sure that thefindings from an evaluation are used:

Preparation refers to the steps taken to get readyto eventually use the evaluation findings. Throughpreparation, stakeholders can:

◆ strengthen their ability to translate new knowledgeinto appropriate action

◆ discuss how potential findings might affectdecision-making

◆ explore positive and negative implications ofpotential results and identify different optionsfor program improvement

ñ Feedback ñFollow-upñ DisseminationPreparation

58

..................................................................................................................

The preliminary evidence of increased highwaypatrols for underage drinking and driving drew mixedreactions from various members of the PRAIDcoalition. Police department representatives perceivedthe staffing levels required for increased patrolling tobe unsustainable due to competing priorities and publicsafety requirements, whereas citizen and MADDrepresentatives thought the benefits of the patrolswere probably worth the tradeoffs. By preparing forthe evaluation results early on, the primary users andother stakeholders were able to begin discussingpossible decisions and actions.

Preparation

Feedback is the communication that occurs amongeveryone involved in the evaluation. Feedback, necessaryat all stages of the evaluation process, creates anatmosphere of trust among stakeholders. Early in anevaluation, the process of giving and receiving feedbackkeeps an evaluation on track by keeping everyone informedabout how the program is being implemented and how theevaluation is proceeding. As the evaluation progressesand preliminary results become available, feedback helpsensure that primary intended users and other stakeholdershave opportunities to comment on evaluation decisions.Valuable feedback can be obtained from stakeholdersby holding discussions during each step of the evaluationand routinely sharing interim findings, provisionalinterpretations, and draft reports.

To provide feedback to and obtain feedback from eachtype of stakeholder involved in the PRAID coalition,an evaluation task force was formed, coordinatedby a funded staff person. From the earliest stagesof the evaluation, the staff person planned andcoordinated quarterly meetings for task forcemembers, who then went back to their own groups(e.g., high schools, police, the local health department,

Feedback

59

..................................................................................................................

hospitals, beverage distributors, MADD, SADD, othercommunity-based organizations) to share programprogress. As the evaluation progressed and prelimi-nary evaluation results became available, the staffperson also used a variety of other approaches toensure that all the coalition members remained wellinformed and able to provide feedback. Theseincluded more frequent updates through e-mail, Web-based communications, and interim print reports, aswell as meetings and discussions with the differentgroups represented by the task force.

Although follow-up refers to the support that manyusers need throughout the evaluation process, this step,in particular, refers to the support that is needed after usersreceive evaluation results and begin to reach and justifytheir conclusions. Active follow-up:

◆ reminds users of the intended uses of what hasbeen learned

◆ can help to prevent misuse of results by ensuringthat evidence is applied to the questions that werethe evaluation’s central focus

◆ prevents lessons learned from becoming lostor ignored in the process of making complex orpolitical decisions

As evaluation results for the intervention werecompiled and analyzed, the task force staff personcontinually reminded task force members of theevaluation’s intended uses. Although health careproviders and citizens were interested in consideringthe implications of results showing a decline in alcoholsales to underage males, other coalition memberssuch as liquor distributors and city officials (concernedwith alcohol-related tax revenues) sought to drawattention away from such findings. The follow-up ofthe staff person helped keep discussion focused onthe most important evaluation results.

Follow-up

60

..................................................................................................................

I left a meeting on theevaluation of Project X andit wasn’t for the communityat all. It was really for policymakers. . . . If you reallywant people to use it, [youneed to pay attention to]the use of language andusing other multiple mediaformats for this real complexinformation . . . . CBO Director

The project produced and disseminated evaluationreports that had limited technical jargon and used lotsof graphics, stories, and examples. In addition to usingreports and other written products for dissemination,PRAID coalition members sustained momentum andbroadened the evaluation’s reach by makingpresentations at school board meetings, gatheringsof public officials, and other community events.Community members used the project’s lessonsto begin planning and developing new community-based initiatives.

Dissemination

Applying standardsAs we have already noted, you can help ensure the

quality of your evaluation by considering relevant evaluationstandards throughout the evaluation process. The threestandards that most directly apply to Step 6— Ensure Useand Share Lessons Learned— are utility, propriety, andaccuracy. As you use your own evaluation results, thequestions presented in the following table can help you toclarify and achieve these standards.

Dissemination is the process of communicatingevaluation procedures or lessons learned to relevantaudiences in a timely, unbiased, and consistent manner.Regardless of how communications are structured, the goalfor dissemination is to achieve full disclosure and impartialreporting. Planning effective communications requires:

◆ advance discussion of the reporting strategy withintended users and other stakeholders

◆ consideration of the timing, style, tone, messagesource, vehicle, and format of information products

61

..................................................................................................................

◆ Are participants in the evaluation group trustworthyand competent?

◆ Do reports clearly describe the program, including itscontext, and the evaluation’s purposes, procedures,and findings?

◆ Have you shared significant mid-course findings andreports with users so that the findings can be usedin a timely fashion?

◆ Have you planned, conducted, and reported theevaluation in ways that encourage follow-throughby stakeholders?

StandardsQuestions

Utility: Ensures thatthe evaluation is usefuland answers questionsthat are directly relevantto users.

◆ Have you ensured that the evaluation findings(including the limitations) are made accessible toeveryone affected by the evaluation and otherswho have the right to receive the results?

Propriety: Ensuresthat the evaluation isethical, conducted withregard for the rightsand interests of thoseinvolved.

Accuracy: Ensures thatthe evaluation findingsare considered correct.

◆ Do the evaluation reports impartially and fairly reflectevaluation findings?

◆ Have you tried to avoid the distortions that can becaused by personal feelings and other biases?

Standards for Step 6: Ensure Use and Share Lessons Learned

In summary, Step 6— Ensure Use and Share LessonsLearned— emphasizes the importance of translating resultsinto action. Through the process of preparation, feedback,follow-up, and dissemination, the benefits of evaluationbecome clear to stakeholders. This step ensures that thelessons learned from an evaluation are shared in such away as to influence program decisions, policy makers, andcommunity-based initiatives.

62

..................................................................................................................

How could you ensure that stakeholders receive and providefeedback throughout the evaluation process?

How could you make sure that lessons learned are used?

What support is available to follow up on evaluation results?

What types of communication strategies might be appropriatefor your program and stakeholders?

Think about a program evaluation in which you are involved. The followingquestions may help you think about how to approach Step 6— Ensure Useand Share Lessons Learned.

63

Glossary

The commitment, resources, and skills that a communitycan mobilize and use to address community issuesand problems and strengthen community assets; thecharacteristics of communities that affect their ability toidentify and address social and economic health issues;the cultivation and use of transferable knowledge, skills,systems, and other resources to affect community— andindividual— level change.

Information produced by and used in evaluation— includesnumbers, words, pictures, or any type of information used.

A blueprint, strategy, or outline to answer questions abouta program. Includes a clear statement about the purposeand plans for gathering, processing, and interpreting theinformation needed.

Data collection options and strategies selected to match orfit the overall design and answer the evaluation questions.Methods depend on knowing who the information is for,how it will be used, what types of information are neededand when, and the resources available.

A subset of process evaluation. Information collected fora specific period of time, often during the start-up or pilotphase of a project, to refine and improve implementationand solve unanticipated problems. This can includeprogram monitoring efforts to provide ongoing feedbackand assess intermediate outcomes.

The following glossary of terms is a compilation of definitions from numerousevaluation sources.

Community Capacity

Data

Evaluation Design

Evaluation Methods

Formative Evaluation

64

Assesses whether a program has achieved desiredintermediate changes in individuals, population groups,or organizations.

Benchmarks used to measure or test changes. Indicatorscan be at the level of individuals (e.g., behavioral changes),organizations (e.g., service delivery hours), or communities(e.g., unemployment rates, quality of life variables).

Examines the effects of a program on health status,usually defined in terms of morbidity (illness, injury) andmortality (death) rates. Determines the long-term effectsof a program or intervention.

Addresses questions related to how a program isimplemented. Compares what was supposed to happenwith what actually happened. Answers questions aboutwhy the program succeeded, failed, or requires revising.

A series of activities supported by a group of resourcesintended to achieve specific outcomes among particulargroups.

The systematic collection, analysis, and reporting of infor-mation about the activities, characteristics, and outcomesof programs to make judgements about the program,improve program effectiveness, and/or inform decisionsabout future programming. Utilization-focused programevaluation is done for and with specific, intended primaryusers for specific, intended uses.

Information gathered from interviews, observations, ordocuments. May include detailed descriptions of situations,events, people, interactions, observed behaviors, andpeople’s own thoughts about their experiences, attitudes,and beliefs. Other data sources are excerpts or entirepassages from documents, correspondence, records andcase histories.

Impact Evaluation

Indicators

Outcome Evaluation

Process Evaluation

Program

Program Evaluation

Qualitative Data

65

Information from questionnaires, tests, standardized(fixed, unchanging) observation instruments, andprogram records. Focuses on things that can be counted,categorized, and subjected to statistical analysis.

A subset of cases (e.g., individuals, records, communities)selected from a population. In quantitative research, largesamples generally enhance confidence in survey results.A minimum total number in a sample is important for somestatistical tests.

People who care about what will be learned from theevaluation and about what will be done with the knowledgegained.

Conducted after program completion, often for the benefitof external, decision-makers (e.g., funding agencies,oversight offices, other users). Provides data to supportjudgements about a program’s worth so that decisionscan be made about continuation or expansion.

Patterns or recurrent ideas that emerge as qualitativedata are collected, analyzed, and interpreted.

The primary focus of data collection and analysis.Can be individuals, groups of people (e.g., classrooms),neighborhoods, cities, states, or countries. Units mayalso be events or incidents (e.g., service delivery failures).Each unit may require different kinds of data. The focusof analysis also varies and affects what can be said (e.g.,if the unit of analysis is a city, one cannot make a state-ment about the impact of the program on an individual).

Quantitative Data

Sample Size

Stakeholders

Summative Evaluation

Themes

Units of Analysis

66

..................................................................................................................

Program EvaluationResourcesWeb-based resources:

Centers for Disease Control and Prevention: http://www.cdc.gov/eval/

W.K. Kellogg Foundation: http://www.wkkf.org/Publications/evalhdbk/

University of Kansas: http://ctb.lsi.ukans.edu/ctb

University of Toronto: http://www.utoronto.ca/shp/hcu

University of Wisconsin Extension: http://www.uwex.edu/ces/pdante/evaluat.htm/

Publications

Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New Approaches to Evaluating CommunityInitiatives, New York, NY: Aspen Institute, 1995.

Fawcett SB, Paine-Andrews A, Francisco VT, Schulz J, Ritchter KP, et al. Evaluatingcommunity initiatives for health and development. In Evaluating Health PromotionApproaches, edited by I Rootman and D McQueen. Copenhagen, Denmark: World HealthOrganization — in press.

Fawcett SB, Sterling TD, Paine Andrews A, Harris KJ, Francisco VT, et al. EvaluatingCommunity Efforts to Prevent Cardiovascular Diseases. Atlanta, GA, Centers for DiseaseControl and Prevention, National Center for Chronic Disease Prevention and HealthPromotion, 1995.

Fetterman DM, Kaftarian SJ, Wandersman A. Empowerment Evaluation: Knowledge andTools for Self-assessment and Accountability. Thousand Oaks, CA: Sage Publications, 1996.

Patton MQ, Utilization-focused Evaluation. Thousand Oaks, CA: Sage Publications, 1997.

67

..................................................................................................................

Publications, continued

Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach. Newbury Park,CA: Sage Publications, 1999.

Shadish WR, Cook TD, Leviton LC. Foundations of Program Evaluation. Newbury Park,CA: Sage Publications, 1991.

Taylor-Powell E, Steele S, Douglas M. Planning a program evaluation. Madison, WI:University of Wisconsin Cooperative Extension, 1996 (see web based entry on page 66).

University of Toronto, Health Communication Unit at the Center for Health Promotion,Evaluating Health Promotion Programs (see web based entry on page 66).