What Do Programs Want?What Do Programs Want?

Jeanette Treiber, PhDProject DirectorCenter for Program Evaluation and ResearchUC Davis School of Medicine

Objective

Determine local agencies’ evaluation capacity building needs

Background California Department of Public Health

(CDPH)’s Tobacco Control Program is carried out by

61 Public Health Departments Approximately 40 Competetive Grantees

More Background CDPH Strategy

Norm Change Community base Community

Assessments 3yr work plans Evaluation

Evaluation ComponentsLocal Program Evaluation requirements

Comprehensive Evaluation Plan Process and Outcome Evaluation Internal or external evaluators Progress Reports Final Evaluation Reports

Tobacco Control Evaluation Center (TCEC)

TCEC provides Evaluation Capacity Building services to California TC programs through:

TATrainingToolsFER scoring and feedback

Rationale for TCEC Services“…pressures have been accompanied by

increasing awareness (and often frustration) on the part of funders as they confront the limited ability of the organizations they fund to meet the challenge of evaluating programs in ways that are likely to yield either genuine outcome-oriented accountability or useful information for formative improvement programs” Stevenson et.al. 2002

Evaluation Capacity Building Observed need and response in past

decade Various approaches to CB Various definitions Various approaches to measuring effect of CB Still little research

Method

1. A 21 item electronic survey (using the online service surveymonkey.com) to a list of 170 project directors and evaluators.

2. Analysis of access database for technical assistance from 2004 to 2009 (approx. 4.5 yrs)

Results (survey)Need for training or materials High or

Moderate Need

Using culturally competent evaluation methods ……………….79.4 %

Writing evaluation plans ……………………………………….59.9 %

Adapting/developing data collection instruments ……………...57.1 %

Interpreting evaluation results…………………………………..54.1 %

As a reference during evaluation activities…………...................51.5 %

Writing evaluation reports……………………………………….50 %

Collecting evaluation data……………………………………….42.9 %

Projected Use of TCEC (survey)

Projected use Response Percent

Finding, developing, or adapting data collection Instruments………………………………………..68.6

Writing an evaluation plan ……………………….57.1

As a reference during evaluation activities ………………………………48.6

To build overall evaluation capacity of the project …………………………….42.9

Preferred Format (survey)

Webinar 68 %

“How-to” evaluation guides on website

57.1%

Individual consultation by phone or e-mail

45.7%

Website strategy exchange 42.9%

Requests for Assistance (TA Log)

Total number of TA requests between 11/2/04 and 7/16/09

512Average: 11 requests/month

Up to 27 requests/agency total

Main Request types (TA Log)

Type of requests

0

50

100

150

200

250

300

Plans tools Analysis Other

Type of Data Collection Instrument

020406080

100120140160180200

Key informantInterviews

Surveys Observation tools

39

189

44

TA DemandNumber of projects that have requested TA

0

10

20

30

40

50

60

70

LLA Compet

Total number ofprojects

Total number ofprojectsrequesting TA

(79%)(71%)

FER Score and TA requestsFinal Evaluation report scores for Local Lead

Agencies were correlated with frequency of TA request.

No statistically significant correlation was found.

Summary/Conclusion Majority of local health departments AND

community organizations in CA TC require evaluation assistance

Greatest need is with DCIs, developing Eval plans, and help conducting evaluation with diverse populations.

Most DCI dev. need is with surveys Webinars are the most desired mode of

training delivery Individual assistance requests vary greatly

in frequency, scope, and content

Limitations Response rate to survey was only 20% TA assistance was not consistently entered

into log in the same manner Requests for TA are somewhat dependent

on marketing of TA TCEC team has changed personnel

throughout the 4.5 years

What do Programs Want?

•Flexible, Varied Evaluation Assistance;

•Convenient Delivery

ReferencesCalifornia Department of Public Health Services,

Tobacco Control Program. http://www.cdph.ca.gov/programs/tobacco/Pages/default.aspx

Innovation Network – Transforming Evaluation for Social Change. http://www.innonet.org/

Preskill, Hallie. (2005). Building Evaluation Capacity. 72 Activities for Teaching and Training. Thousand Oaks (Sage).

More ResourcesStevenson, John, Paul Florin, Dana Scott Mills, and

Marco Andrade. (2002). Building Evaluation Capacity in Human Service Organizations: A Case Study. Evaluation and Program Planning 25, 233-243.

Tobacco Control Evaluation Center (TCEC). Center for Program Evaluation and Research, UC Davis. http://programeval.ucdavis.edu

University of Wisconsin Extension. Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/index.html

ContactJeanette Treiber, PhDProject DirectorCenter for Program Evaluation and ResearchTobacco Control Evaluation CenterUC Davis School of Medicine

jtreiber@ucdavis.eduhttp://programeval.ucdavis.edu1616 Da Vinci CourtDavis, CA 95618