ins and outs of program evaluation
DESCRIPTION
TRANSCRIPT
Program EvaluationA Practical/Painless Approach
NLA/NEMA Conference – Oct. 14, 2010
About the presenter
Kathryn BrockmeierResearch Analyst/ Special Projects Associate, Nebraska Library Commission
Grant writerEvaluator
About you
Library staffTrusteeFriends/Foundation board member
About this presentation
Introduction to program evaluation: designs, methods, and techniques of analysisReal-world examples
NLA/NLC & IMLS Internship programNLC/IMLS Scholarship programNLC One Book for Nebraska Kids & Teens
Q&A
What is a program?
Stems from mission and goalsGoal becomes program
A collection of organizational resources geared to accomplish a certain major goal or set of goals
What is program evaluation?
Collecting informationMaking decisions“What do we need to know?”
For…Needs assessment, return on investment study, grant application/report
Benefits of evaluation
Investigate what works/doesn’t workShare results with stakeholdersDemonstrate effectivenessImprove staff’s involvementModel the program
Types of program evaluationto meet program objectives
Implementation evaluation (process evaluation)
Program implementation objectives
Outcome evaluationParticipant (stakeholder and individual) outcome objectives
Steps in the evaluation process
#1: Assemble an evaluation team
#2: Prepare for the evaluation#3: Develop an evaluation plan#4: Collect evaluation
information#5: Analyze your evaluation
information#6: Prepare the evaluation
report
1. Who will conduct the evaluation?
Outside evaluator supported by in-house staffIn-house evaluator supported by program staff and outside consultantIn-house evaluator supported by program staff
2. How to prepare for the evaluation
a. Decide what to evaluateb. Build a logic model of your
programc. State your program
implementation and participant outcome objectives in measurable terms
d. Identify the context for your evaluation
2a. Decide what to evaluate
Entire program, program component, service, activityDepends on financial and staff resourcesGrant-driven or funding-drivenAt start-up of programTo check effectiveness
2b. Logic models
Assumptions about your target audienceProgram interventions
Implementation objectives
Immediate outcomesImmediate participant outcome objectives
Intermediate outcomesLong-term outcomes
2c. SMART objectivesSpecificMeasurableAchievableRelevantTime-framed
http://www.marchofdimes.com/HI_SMART_objectives.pdf
By the end of ___________________ (enter timeframe), increase _______________ (what will change?) by _______ (enter a percentage) % or more for ___________________ (who will experience the change?) by _________________ (how will the change occur?).
2d. Evaluation context
Assumptions about your target audienceProgram interventions
Implementation objectives
Immediate outcomesImmediate participant outcome objectives
Intermediate outcomesLong-term outcomes
3. Develop the planDiscuss the evaluation frameworkEvaluating implementation objectives
Types of info needed, Sources of info, Selection of info sources, Methods for collecting info, Confidentiality, Timeframe, Analysis methods
Evaluating participant outcome objectives
Do program participants demonstrate changes in knowledge, attitudes, behaviors, or awareness?Were the changes the result of the program’s interventions?
Evaluation practices and procedures
4. “Data” collection
When and where info will be collectedWho are the respondents?Methods
Survey/questionnaireInterviewFocus groupExisting recordsObservation
5. Analyze the information
Descriptive statisticsFrequency distributionsSummary statistics—measures of central tendency, rangeCross-tabulationFilters
Content analysisCase studies
6. Write the report
AudienceFormatWriting tipsOral presentationsVisual aids
Considerations
Resources
The Program Manager’s Guide to Evaluation, from the Administration for Children and Families
http://bit.ly/cNVlQ8
Basic Guide to Program Evaluation, from the Free Management Library
http://www.managementhelp.org/evaluatn/fnl_eval.htm