ins and outs of program evaluation

Program EvaluationA Practical/Painless Approach

NLA/NEMA Conference – Oct. 14, 2010

About the presenter

Kathryn BrockmeierResearch Analyst/ Special Projects Associate, Nebraska Library Commission

Grant writerEvaluator

kathryn.brockmeier@nebraska.gov402-471-4002

About you

Library staffTrusteeFriends/Foundation board member

About this presentation

Introduction to program evaluation: designs, methods, and techniques of analysisReal-world examples

NLA/NLC & IMLS Internship programNLC/IMLS Scholarship programNLC One Book for Nebraska Kids & Teens

Q&A

What is a program?

Stems from mission and goalsGoal becomes program

A collection of organizational resources geared to accomplish a certain major goal or set of goals

What is program evaluation?

Collecting informationMaking decisions“What do we need to know?”

For…Needs assessment, return on investment study, grant application/report

Benefits of evaluation

Investigate what works/doesn’t workShare results with stakeholdersDemonstrate effectivenessImprove staff’s involvementModel the program

Types of program evaluationto meet program objectives

Implementation evaluation (process evaluation)

Program implementation objectives

Outcome evaluationParticipant (stakeholder and individual) outcome objectives

Steps in the evaluation process

#1: Assemble an evaluation team

#2: Prepare for the evaluation#3: Develop an evaluation plan#4: Collect evaluation

information#5: Analyze your evaluation

information#6: Prepare the evaluation

report

1. Who will conduct the evaluation?

Outside evaluator supported by in-house staffIn-house evaluator supported by program staff and outside consultantIn-house evaluator supported by program staff

2. How to prepare for the evaluation

a. Decide what to evaluateb. Build a logic model of your

programc. State your program

implementation and participant outcome objectives in measurable terms

d. Identify the context for your evaluation

2a. Decide what to evaluate

Entire program, program component, service, activityDepends on financial and staff resourcesGrant-driven or funding-drivenAt start-up of programTo check effectiveness

2b. Logic models

Assumptions about your target audienceProgram interventions

Implementation objectives

Immediate outcomesImmediate participant outcome objectives

Intermediate outcomesLong-term outcomes

2c. SMART objectivesSpecificMeasurableAchievableRelevantTime-framed

http://www.marchofdimes.com/HI_SMART_objectives.pdf

By the end of ___________________ (enter timeframe), increase _______________ (what will change?) by _______ (enter a percentage) % or more for ___________________ (who will experience the change?) by _________________ (how will the change occur?).

2d. Evaluation context

Assumptions about your target audienceProgram interventions

Implementation objectives

Immediate outcomesImmediate participant outcome objectives

Intermediate outcomesLong-term outcomes

3. Develop the planDiscuss the evaluation frameworkEvaluating implementation objectives

Types of info needed, Sources of info, Selection of info sources, Methods for collecting info, Confidentiality, Timeframe, Analysis methods

Evaluating participant outcome objectives

Do program participants demonstrate changes in knowledge, attitudes, behaviors, or awareness?Were the changes the result of the program’s interventions?

Evaluation practices and procedures

4. “Data” collection

When and where info will be collectedWho are the respondents?Methods

Survey/questionnaireInterviewFocus groupExisting recordsObservation

5. Analyze the information

Descriptive statisticsFrequency distributionsSummary statistics—measures of central tendency, rangeCross-tabulationFilters

Content analysisCase studies

6. Write the report

AudienceFormatWriting tipsOral presentationsVisual aids

Considerations

Resources

The Program Manager’s Guide to Evaluation, from the Administration for Children and Families

http://bit.ly/cNVlQ8

Basic Guide to Program Evaluation, from the Free Management Library

http://www.managementhelp.org/evaluatn/fnl_eval.htm