facilitator guide/case study

FACILITATOR GUIDE/CASE STUDY

Evaluating Public Health Programs

Created: 2013

Evaluating Public Health Programs. Atlanta, GA: Centers for Disease Control and Prevention (CDC), 2013.

EVALUATING PUBLIC HEALTH PROGRAMS

FACILITATOR GUIDE/CASE STUDY |2

Table of Contents

38TEVALUATING PUBLIC HEALTH PROGRAMS CASE STUDY: ALCOHOL INJURY38...............................................................................................................................3 T

38TOVERVIEW38T ........................................................................................................................ 3 38TINSTRUCTIONS 38T .................................................................................................................. 3 38TBACKGROUND: PART I 38T ...................................................................................................... 3

38T38T 38TBACKGROUND: PART II 38T ..................................................................................................... 7 38TBACKGROUND: PART III 38T .................................................................................................. 10 38T 38TBACKGROUND: PART IV 38T .................................................................................................. 12



Evaluating Public Health Programs Case Study: Alcohol Injury OVERVIEW In this skill assessment, participants will engage in the six steps of the CDC Framework for Program Evaluation: (1) engage stakeholders; (2) describe the program or project; (3) focus the evaluation design; (4) gather credible evidence; (5) justify and state conclusions; and (6) ensure use of evaluation and share lessons learned. The total amount of time for this case study is estimated to be 3 hours. The time allotment per section varies, and is listed at the beginning of each step. Please allow additional time for discussion of each step.

Please keep in mind that certain components of this assessment may need to be modified for specific country needs.

INSTRUCTIONS Instruct participants to complete each of the six steps separately in small groups. Ask participants to read the background information below in their small groups. Then ask participants to answer the questions for one section at a time, taking time to review the answers at the end of each section.

This exercise was designed as a teaching tool, based on common and scientifically proven methods from the United States, to teach effective program evaluation and its application across health topics.

BACKGROUND: PART I In 2005, the City of Changsha, located in the province of Hunan, China, received funding from the China Centers for Disease Control and Prevention (CDC) to design and implement a community-based intervention to reduce alcohol injury and death in Chinese men. As part of this funding award, the Changsha City Health Department was expected to work with a variety of stakeholders, from the private and public sector, in reducing alcohol related road traffic injuries (RTIs).

RTIs were identified as a public health problem through a community assessment conducted by community residents, community-based agency representatives, university faculty, and local advocacy groups. This original group believed that in order to reduce RTIs in Changsha, a long-term commitment of many people would be needed. Therefore, organizations formalized their commitment by establishing the Changsha Partnership to Reduce Alcohol related Injury and Death (PRAID) coalition.



In the initial phases of program planning and evaluation, the Changsha PRAID coalition knew how important it was to include a variety of stakeholders. They identified and recruited stakeholders through public forums, by word of mouth, and as they conducted their community assessment. As these stakeholders met and shared their ideas, the following interests emerged:

Table 1: Stakeholder Interests Early Stakeholders What Stakeholders Cared About Most

Families Against Drunk Driving (FADD) Reducing death, injury, and financial burden on families as the result of alcohol injuries

Changsha City Health Department Protecting residents from alcohol related road traffic injuries (RTI)

Chinese Ministry of Health Promoting national level policies that protect citizens from RTIs

Hunan CDC Protecting residents from alcohol related RTIs

Hospitals Reducing the resources spent on treating RTIs

Private Driving Schools Delivering effective driving curriculum for new drivers

Traffic Police Reducing everyone’s risk on the roads and highways

Mental Health Institute Improving the health of the community Hunan Provincial Transportation Department

Reducing death and injury among drivers across roads in Hunan

Changsha PRAID spent many hours in meetings and one-on-one talks to understand what the stakeholders cared about and to find ways to share concerns with other stakeholders. They relied on formal and informal communication to make sure that as many stakeholder voices were heard.

As the project matured, new stakeholders became involved, including the Changsha restaurant association president and a representative from the provincial beverage (alcohol) distributor. The latter were primarily interested in balancing the negative public image of drinking and driving among men, many of whom are relatively inexperienced drivers. Their views were important as Changsha PRAID refined its program plan and focused the evaluation.

Answer the following questions for Step 1 only. (40 minutes, including reading of previous pages)



Step 1: Engage Stakeholders 1. Identify the stakeholders for whom you will involve in a program evaluation. Consider

the Utility standard when filling out the three columns in the table below. In particular,think about:• Who will use the results of the evaluation?

• Who can influence the use of the findings?

(answers in italics for facilitator only)

Persons involved in program operations

Persons served or affected by the program

Primary users of the evaluation results

• Staff from the City ofChangsha (grantee)

• Changsha PRAIDCoalition

• China CDC (technicalassistance)

• Communications/mediastaff

• New male driversat risk for alcoholrelated injuriesand death

• FADD• Changsha City

HealthDepartment

• Chinese MoH• Hospitals• Hunan CDC• Traffic police• Private driving

schools• Mental Health

Institute• Hunan Provincial

TransportationDepartment

• Changsharestaurantassociation

• Provincialbeveragedistributor

• City of Changsha• Changsha PRAID

Coalition• China CDC• Health officials• Hunan Provincial

TransportationDepartment

• FADD

2. Refer to your completed table (question 1) and answer the following questions. Whoamong the stakeholders willa. Enhance credibility of the program:

_________________________________________________________



b. Implement the program changes:_________________________________________________________

c. Advocate for changes:_________________________________________________________

d. Fund, authorize, or expand the program:_________________________________________________________

3. Complete the table below as follows:a. List one stakeholder from question 1 from each of the categories (person involvedin program operations, person served or affected by the program, primary user of evaluation results).

b. Describe UhowU to engage each stakeholder in the evaluation process.

c. Describe a reasonable Utime commitmentU for each stakeholder’s involvement (e.g., quarterly meetings, weekly phone calls).

(Sample responses in italics for facilitator only)

Stakeholder How to Engage Stakeholder Reasonable Time Commitment

Persons involved in program operations

May have them help with evaluation design and/or conducting the evaluation

Hold quarterly meetings for the duration of the program

Persons served or affected by the program

Invite them to initial meetings to determine their needs and what results they want out of the program

Hold an end of project meeting to share evaluation results

Primary users of the evaluation results

Engage them in designing the evaluation, keep them engaged by sending them interim progress reports

Hold quarterly or monthly meetings to keep them engaged; Hold an end of project meeting to discuss recommendations and to ensure findings are used

Answer the following questions for Step 2 only. (1 hour) Step 2: Describe the Program (Participant Version, page 8)

Read the information below about the program goals and objectives. Then answer the questions that follow.



BACKGROUND: PART II Changsha PRAID coalition members decided to focus on reducing injuries and death in young Chinese men – a broad goal. Different ideas were posed on how to reduce the injury and death rates. Through a participatory planning process, coalition members and other stakeholders developed a long-term plan that incorporated different ideas and priorities within the broad goal of reducing alcohol related injuries and deaths among Chinese men.

Statement of Need (the problem): Three-year increase in the number of alcohol-related road traffic injuries (RTIs) among Chinese males with less than one year of holding a valid driver’s license.

Goal: Reduce alcohol-related injury and death.

Objective: By 2014, reduce the incidence of alcohol related RTIs in Chinese males with less than one year of holding a valid driver’s license by 50%.

1. Based on these objectives and the background information, identify and list thefollowing elements of the project on the table below OR as a logic model on thefollowing page (or a flip chart):• A minimum of UthreeU inputs

• A minimum of UthreeU activities

• A minimum of UthreeU outputs

• A minimum of UoneU short-term outcome, UoneU intermediate outcome and UoneU long-term outcome

Inputs Activities Outputs Outcomes Examples of inputs: • China CDC

funding• Community

partners/stakeholders

• ChangshaPRAID coalitionmembers

• Changsha CityHealth Dept.

• Volunteers• City of

Changsha staff

Examples of Activities: • Increase traffic

police units andenforcement inHunan Province(e.g., sobrietycheckpoints)

• Increasesanctions fornew male driversdriving under theinfluence ofalcohol

Examples of Outputs: • Strengthen

enforcement ofdrunk drivinglaws

• New maledrivers areexposed toalcohol injuryeducation

• Othercommunitymembers/drivers

Short-term: • Increased

number ofcitations andsobrietycheckpoints

• Increasedknowledge andawarenessabout risks

Intermediate: • Changes in

drinking and



• Staff time andeffort

• Free and earnedmedia

• Resources todisseminateinformationabout theprogram toalcohol outlets

• Staff to performmonitoringoperations

• Training for lawenforcementofficials

• Alcoholeducationcurricula

• Increase thenumber ofdriver’seducationprograms thatcover alcoholuse/RTI

• Changes inpolicy andregulatory action

• Build andsustain coalitionof interestedparties

• Publicity/mediacampaign (TVcommercials,radio) forgeneralcommunity anddrivers whoalready have avalid driver’slicense

will be exposed to alcohol injury education (via media campaign)

• EngagedChangshaPRAID coalition

• Communitymobilization toaddress alcoholproblem

driving behaviors • Reducing the

number ofdrivers found tobe intoxicated

Long-term: • Reduction in

injuries anddeaths due toalcohol amongdrivers with validdriver’s license(reducemorbidity andmortality)

• Saferroads/highwaysfor drivers,pedestrians

• Decreasedfinancial cost onhospitals andfamilies fromtreatment andrecovery due toalcohol relatedRTIs



2. Based on the elements that you have defined above, create a logic model for this project (below). Be sure to sequencethe activities. After sketching out the logic model, use flip chart paper to create the model with your group. You willpresent your logic model to the class.

Figure 1: Logic Model Example

Inputs Activities Outputs Outcomes

Short-term Intermediate Long-term

China CDC funding

Community partners and stakeholders

PRAID coalition members

Volunteers

Changsha City Health Department

Staff time and effort

Free and earned media

Resources to disseminate information about the program to alcohol outlets

Training for law enforcement officials

Alcohol education curricula

Staff to perform monitoring

ti

Increase traffic police enforcement (e.g. sobriety checkpoints)

Increase sanctions for driving under the influence of alcohol

Policy and regulatory

Increase the number of driver’s education programs that cover alcohol use/RT I

Build and sustain coalition of interested parties

Publicity/media campaign (TV and radio)

• Engaged ChangshaPRAID coalition

•• Community

mobilization toaddress alcohol problems

New male drivers, along with community, are exposed to alcohol injury education

Strengthen enforcement of drunk driving laws

Increased knowledge and

awareness about risks

Increased citations and

sobriety checkpoints

Changes in drinking and

driving behaviors

Reduction in injuries and deaths due to alcohol

Safer roads/highways for drivers and pedestrians

Reduction in drivers found

to be intoxicated

Reduction in treatment and recovery costs in hospitals & by families



Notes to Facilitator. During the review of responses, ask the following questions if participants chose to develop a logic model.

Answer the following questions for Step 3 only.

Step 3: Focus the Evaluation (Participant Version, page 11)

Read the background information below about the request for an evaluation. Then answer the questions that follow.

BACKGROUND: PART III After the program had been implemented for 3 years, the Hunan CDC and China CDC wanted to determine if a reduction in road traffic injuries among male drivers with less than 1 year of holding a driver’s license can be attributed to 2 features of the program: 1) the increase in sobriety checkpoints and 2) the increase in driver’s education programs that cover alcohol use and RTIs. The stakeholders wanted to share the results with a neighboring province that was considering a similar program.

1. What is the UpurposeU of this evaluation? Mark all that apply. a. Show accountability (Correct Answer)b. Examine program implementationc. Determine program improvementd. Facilitate judgment about a program’s fate

2. Consider the purposes of the evaluation. What type of evaluation is this? Please circleyour answer.a. Process evaluationb. Outcome evaluationc. Both process and outcome evaluation (Correct Answer)

Utility a. Is the level of detail appropriate for the intended user(s)?b. Is the logic model clear to those who need to use the informationto make decisions related to the evaluation?

Feasibility c. How does the logic model show the relationship betweenprogram activities and intended outcomes?

Propriety d. Does the description include sufficient detail for users to criticallyassess the content?

Accuracy e. Would diverse stakeholders consider the logic model areasonable representation of the program?



3. Who are the UusersU for this evaluation? • Hunan CDC

• China CDC

• Neighboring province

4. What is the UuseU for this evaluation? • To learn if a reduction in RTIs among male drivers with less than 1 year of holding

a driver’s license can be attributed to the increase in sobriety check points and/orthe increase in driver’s education programs that cover alcohol use and RTIs.

5. Based on the utility and feasibility standards, is what you are being asked to evaluatea reasonable request? Yes or no? Please explain. If you answered no, what can youevaluate that is related to the request?• Yes, you should be able to collect data on the number of sobriety checkpoints and

citations and participation in the driver’s education programs.

6. Based on your responses to the above questions, what are at least two examples ofquestions your evaluation would ask?• What was the number of sobriety checkpoints conducted by traffic police units in

the last 3 years?

• What was the number of citations for male drunk drivers in the last 3 years?

• How did the number of citations change during the 3 years?

• How many new male drivers were exposed to alcohol injury education during the 3years?

7. Has the program been operating long enough to have a chance of reaching itsintended outcomes?• Yes, it has been operating for 3 years.

Answer the following questions for Step 4 only. (20 minutes) Step 4: Gather Credible Evidence (Participant Version, page 13) 1. Record two evaluation questions you identified in Step 3 in the first column.

2. Identify and list indicators for each question in the second column.

3. Identify and list the data sources or methods you will use to collect data about theindicators in the third column. Consider data sources that will enhance the credibilityof the data with stakeholders.



Evaluation Question Indicators Data Sources / Methods

What was the number of sobriety checkpoints conducted by traffic police units in the last 3 years?

Number of sobriety checkpoints conducted per week/ month/year by location.

Traffic police documents, sobriety checkpoint log sheets

What was the number of citations for male drunk drivers in the last 3 years?

Number of citations for male drunk drivers per week/month/year by location.

Police records

How did the number of citations change during the 3 years?

Number of citations for male drunk drivers each year

Police records

How many new male drivers were exposed to alcohol injury education during the 3 years?

Level of participation in alcohol injury education

Attendance records

Answer the following questions for Step 5 only. (20 minutes) Step 5: Justify conclusions (Participant Version, Page 13)

Read the background information below about the findings from the evaluation. Then answer the questions that follow.

BACKGROUND: PART IV The central objective of the community-based intervention program was to reduce the incidence of alcohol related RTIs in Chinese males with less than 1 year of holding a valid driver’s license by 50 percent. After 3 years, there had been a modest reduction in RTIs among male drivers with less than 1 year of holding a valid driver’s license in Changsha, Hunan. The program attributed this reduction to the increase in sobriety checkpoints conducted by traffic police units between 2012 and 2014. 30 sobriety checkpoints were conducted annually in 2012, whereas in 2014, the number of checkpoints doubled to 62.

The number of citations for male drunk drivers initially increased with the implementation of sobriety checkpoints. As communities witnessed an increase in traffic police presence and sanctions, citations decreased.



The graph below illustrates data collected over a three-year period. It summarizes alcohol-related citations issued to male drunk drivers who held a valid driver’s license for less than 1 year in Changsha. Traffic police stakeholders, who viewed any improvement in performance as a sign of success, viewed this increase as a reason to judge the program positively.

Additionally, the attendance records showed low participation levels in the program’s educational initiatives. When surveyed, the male participants in the rural areas reported lack of awareness and accessibility to driver’s education programs that cover alcohol use/RTI as the main reasons for not attending the program.

Figure 2: Graph of Citations for Male Drunk Drivers

1. Write a minimum of two recommendations for each evaluation result in the spacebelow.• Because the program attributed the reduction of the incidence of alcohol related

RTIs in males to the increase in sobriety checkpoints between 2012 and 2014, it isrecommended that the level of traffic police patrolling be permanently increased.

• The evaluation results showed low levels of participation by males in the program’seducational initiatives mainly due to lack of awareness and accessibility to theprograms; it is recommended that materials are distributed to new drivers whenthey receive their license about the education programs and that more programsare made available in rural areas.

2. Refer to your recommendations from the previous question, and consider the fourstandards for evaluation. Then complete the following table and justify your answers.

63

85

52

020406080

100

2012 2013 2014Num

ber o

f Cita

tions

* less than 1 year of holding a valid driver's license

Citations for male drunk drivers*, Changsha, Hunan, 2012-2014

Driving Citations



Question Answer Yes

Answer No

Justification

Utility: Have different interpretations of the findings been considered?

� Yes � No

Feasibility: Are the recommendations realistic for the program to implement?

� Yes � No

Propriety: Are the conclusions and recommendations reflective and respectful of key stakeholders, including those served by the program?

� Yes � No

Accuracy: Can the conclusions be explicitly justified?

� Yes � No

Facilitator’s note: Responses will vary.

Answer the following questions for Step 6 only. (20 minutes) Step 6: Ensure Use of Evaluation (Participant Version, page 15)

1. Refer to your work in Step 3 (Focus the Evaluation) where you identified the users forthis evaluation. Select two of these users (i.e., stakeholders), and explain how you willensure that the evaluation findings, including limitations, are made accessible.• Answers will depend on the type of stakeholders engaged and the purpose and

use of the evaluation. (If the stakeholder is a community based organization, suchas Changsha PRAID, then it would be best to present the findings in a useful andmeaningful way, such as a community town hall meeting)

• Preliminary process evaluation results should be shared early on so thatimplementation can be modified for future training sessions.

• Consider what is important to each stakeholder (from Step 1 of the evaluation),and make the results relevant to their needs.

2. What will you do to make sure that the evaluation is reported in a manner thatencourages follow-through by the stakeholders?• If the stakeholder is a community based organization, holding multiple community

town hall meetings would encourage follow-through and the use of simple



handouts that highlight success stories can be used, rather than graphs, tables and the use of technical language.

• Engage stakeholders involved in program operations from the beginning withevaluation planning to increase buy-in and ensure utility of the results. Meet withthese stakeholders regularly to provide interim findings, and encourage them toparticipate in decision-making.

• A formal presentation, written report, and/or executive summary can be part of thereporting for officials and other key decision makers. Considering the fact that high-level officials may not have time to read the entire report, fact sheets or anexecutive summary are most useful.

3. In the table below, list the two stakeholders you identified in question 1 and thencomplete the remaining columns with regards to communicating the evaluation resultsto each stakeholder.

Stakeholder What to Communicate

Method of Communication

Frequency

Changsha City Health Department

Implication of results showing a modest decline in RTIs

Community, town hall meetings and discussions

Quarterly

Changsha PRAID Coalition

Implication of results showing a modest decline in RTIs; lessons learned; outcome impacts

Presentations, written report (e.g., executive summary)

Quarterly, end of project

China CDC Implication of results; challenges faced in implementation; lessons learned; collaboration opportunities

Interim progress report, annual reports, meetings, phone calls, e-mail messages

Every 6 months

facilitator guide/case study

Documents