regional conferences 2008-2009 planning, doing, and using evaluation
TRANSCRIPT
![Page 1: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/1.jpg)
REGIONAL CONFERENCES 2008-2009REGIONAL CONFERENCES 2008-2009Planning, Doing, and Using EvaluationPlanning, Doing, and Using Evaluation
![Page 2: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/2.jpg)
What are the experiences you have had in evaluation?
![Page 3: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/3.jpg)
![Page 4: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/4.jpg)
Introduction3 Components to Program Evaluation
Planning• Design of the evaluation
Doing• Collection of data, analysis, interpretation
Using• Applying and disseminating the results
![Page 5: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/5.jpg)
Planning Evaluations
• Determining Information Needs – purpose of doing an evaluation
• Developing an Evaluation Strategy – logic model, indicators, measures, questions, logistics
![Page 6: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/6.jpg)
Doing Evaluation
• Collecting and compiling data
• Analyzing information
• Developing conclusions and recommendations
• Writing the report
![Page 7: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/7.jpg)
Using Evaluations
• Developing a communications plan• Developing strategies and action plans to
ensure use of evaluation findings
![Page 8: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/8.jpg)
Planning EvaluationsSteps to ensuring readiness – a critical
success factor for evaluation:• Identify any barriers and facilitators• Identify and involve key stakeholders with
expressed interest in evaluation• Identify an evaluation “champion” • Research existing best practices, previous
designs and measures and academic literature pertaining to program
![Page 9: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/9.jpg)
Planning EvaluationSteps to ensuring readiness (cont’d)
• Collaborate with other agencies to build a network of support
• Establish understanding of why evaluation is being carried out
• Develop evaluation framework, ensuring involvement of key stakeholders and staff
• Construct program logic model
![Page 10: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/10.jpg)
Planning an Evaluation: Developing an Evaluation Framework
Steps to ensuring readiness (cont’d)
• Develop evaluation questions• Identify indicators for all outcomes being
measured• Select appropriate measures and sources of
information • Identify responsibilities for data collection
![Page 11: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/11.jpg)
In your experience, offer examples of:
1) Two barriers you have encountered in the implementation of evaluation in your organization, and…
2) Two facilitators you have encountered in the implementation of evaluation in your organization.
![Page 12: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/12.jpg)
Program Logic Model
A logic model is a visual diagram that shows causal relationships between the various components of a program, i.e., linking program activities with intended results. It is also the foundation for the design of evaluation questions.
![Page 13: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/13.jpg)
Program Logic Models - Example
Inputs Activities Outputs Outcomes
What must the program have in order to function well?
What must you do to achieve your intended results?
How much do you do for how many people over what time?
What difference does the program make for the target population?
![Page 14: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/14.jpg)
Program Logic ModelComponents:• Inputs – funding and resources • Activities – what is being delivered in the
program; the services you provide• Outputs – what is produced – the tangible
products that result from the activities • Outcomes – the impacts of the program
related to the target population that occur as a result of program delivery
![Page 15: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/15.jpg)
Program Logic ModelOutcomes: Short-term – client changes that can be
directly attributed to program delivery; Intermediate – client changes that can be
directly attributed to program delivery; can be related to short-term outcomes long-term goal(s)
Long-term goal – Overall program goal; the ideal state where program objectives have been met; the community need/problem has been addressed
![Page 16: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/16.jpg)
Program Logic Models - Example
Marathon Runner
Goal: to improve race performance over last year’s time
![Page 17: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/17.jpg)
Can you identify Inputs, Activities, Outputs, and Outcomes that might be involved in this training
program?
![Page 18: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/18.jpg)
Program Logic Models - Example• Inputs: new running shoes; dedicated time in
daily schedule; training resources; gym membership
• Activities: interval training; distance runs; strength training
• Outputs: 2 x 1-hour strength training sessions/week; 2 x 1-hour interval sessions/week; 2 x 2-hour distance runs/week
• Outcomes: increased speed; increased endurance
![Page 19: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/19.jpg)
Indicators and Measures
![Page 20: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/20.jpg)
Indicators and MeasuresIndicators: evidence that you’ve attained your goal
and that the outcomes have been achieved• Select the particular outcomes you want to measure.
Ask “How will you know when this has been achieved?” “What is the evidence and how will it be measured?”
• Ensure quality – appropriateness & integrity of information, and quantity – power to detect effects with minimal burden on respondents for providing information
![Page 21: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/21.jpg)
INPUTS ACTIVITIES OUTPUTS Short-term Outcomes
Intermediate -term Outcomes
Final Outcome
PROCESS EVALUATIONPROCESS EVALUATION OUTCOME EVALUATIONOUTCOME EVALUATION
![Page 22: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/22.jpg)
Process Evaluations
Key Issues:• Service Delivery – the extent to which
program components are being delivered as planned
• Coverage – extent of target population participation in program
• Bias – extent to which subgroups of the designated target population participate (or don’t participate)
![Page 23: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/23.jpg)
Outcome Evaluations
Key Issues:
• Program Effectiveness/Impact – is the program achieving the intended outcomes within the targeted population, e.g., changes in knowledge/understanding, attitudes, and behaviour/functioning
![Page 24: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/24.jpg)
ACTIVITY:
Part A Developing evaluation questions
Part B Developing a framework
![Page 25: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/25.jpg)
Susan Kasprzak, Research Associate
Tanya Witteveen, Research Associate
![Page 26: REGIONAL CONFERENCES 2008-2009 Planning, Doing, and Using Evaluation](https://reader035.vdocuments.site/reader035/viewer/2022062511/55150d5d550346a80c8b59a3/html5/thumbnails/26.jpg)
Visit our website for more information:
www.onthepoint.ca