how to conduct an evaluation
DESCRIPTION
HOW TO CONDUCT AN EVALUATION. Jerome De Lisle. 2012 Specialization Courses. Introduction to the Evaluation of Educational & Social Systems (4 Credits) Definitions & History (1) Profession & Competencies (1) Issues & Standards (2) Targets-Systems, Programmes, Curricula (3) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/1.jpg)
HOW TO CONDUCT AN EVALUATION
Jerome De Lisle
![Page 2: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/2.jpg)
2012 Specialization Courses Introduction to the Evaluation
of Educational & Social Systems (4 Credits)
Definitions & History (1) Profession & Competencies (1) Issues & Standards (2) Targets-Systems, Programmes,
Curricula (3) Benefits, Challenges, & Practice (2) Evaluation in Schools (2) Evaluation in Communities (2)
Evaluating National Systems National Assessment (5) Cross-Country & International
Assessments (5) Project (3)
Evaluation Designs (4 Credits)
Evaluation Models (4) Evaluation Designs (6)
Protocol Qual, Quan, MM
The Practice of Evaluation Designs (3)
Evaluation Project (9 Credits)
Develop an evaluation workplan (8) Implement workplan (12) Write & present report (6)
![Page 3: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/3.jpg)
Reading Free material
http://teacherpathfinder.org/School/Assess/assess.html http://www.tbs-sct.gc.ca/eval/pubs/func-fonc/func-fonc_e.
asp#s5 http://www.managementhelp.org/evaluatn/fnl_eval.htm http://www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/STAR
T.HTM#TOC http://www.utas.edu.au/pet/sections/developing.html http://www.chrc-caribbean.org/Downloads/files/
Evaluation%20workplan.pdf#search=%22developing%20an%20evaluation%20design%22
![Page 4: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/4.jpg)
Reading Base Texts
![Page 5: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/5.jpg)
Reading - references
![Page 6: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/6.jpg)
Evaluate what? Education system/quality Schools/Institutions Programs/Projects/Processes/Products People
![Page 7: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/7.jpg)
![Page 8: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/8.jpg)
Performance Management of Programmes
Performance Measurement is an ongoing monitoring and reporting of program accomplishments, against progress towards pre-established goals. Both Program Evaluation and Performance Measurement can help identify areas of programs that need improvement, and whether the program or project is achieving its goals and objectives and the reason why.
![Page 9: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/9.jpg)
Evaluation in the light of Performance Management
A focused Program Evaluation will examine specifically identified factors of a program in a more comprehensive way than from experience that occurs day-to-day.
![Page 10: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/10.jpg)
A Focus on Program Evaluation
Program Evaluation is “the identification, clarification, and application of defensible criteria to determine an object’s worth”--Fitzpatrick, Sanders, & Worthen, 2003
In educational evaluation, that “object” might be a- Programme: “Reading Recovery program” Project: “violence reduction in schools” Process: “the transition from primary to secondary
school” “teacher practices in one special classes Product: “a new textbook series for reading”
![Page 11: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/11.jpg)
Basic Steps in Evaluation
Clarifying the Evaluation Requests & Responsibilities
Setting Boundaries & Analyzing the Evaluation Context
Focus the Evaluation: Identifying & Selection Evaluation Questions & Criteria
Develop a plan to conduct evaluation –- Evaluation Design & Data Collection Strategy
Analyze Data
Write Report
Conduct Study
Chapters in Worthen et al. 2003
![Page 12: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/12.jpg)
Define the Purpose and Scope
The first step is to define the purpose and scope of the evaluation. This is required in order to set limits to the
evaluation, confining it to a manageable size.
This step also involves deciding on the goals and objectives for the evaluation.
As well as the audience for the evaluation results
![Page 13: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/13.jpg)
Define the Purpose and Scope
The audience for evaluation may be very restricted (primarily internal) or may include a wide range of stakeholders and the general public.
The scope of the evaluation will depend on the evaluation's purpose and the information needs of the intended audience. For example, it is appropriate to design a limited
evaluation if the programme has already been evaluated and to target only certain parts of the program which have been changed, revised, or modified or only on certain objectives previously only partially achieved.
![Page 14: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/14.jpg)
Focus the Evaluation Most frequently this step is done through
the development of a set of evaluation questions.
In turn the evaluation questions will influence choices of model and in the “evaluation design”
![Page 15: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/15.jpg)
Focus the Evaluation Developing evaluation questions
requires that the object of the evaluation is fully described and discussions held with stakeholders.
Evaluation questions should be prioritized and examined in relation to the time and resources available.
![Page 16: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/16.jpg)
Sample Evaluation Questions
Evaluation Questions Criteria 1. How many students are exposed to the instructional unit? Number of students in class.
2. What are the demographic characteristics of the students? Grade, Age, Sex, Racial/Ethnic Group.
3. What is the level of the students' basic skills (language and mathematics ability)?
Scores on nationally standardized achievement tests.
4. What is the level of students' knowledge and skills prior to being exposed to the instructional unit being evaluated?
Scores on pre-tests of knowledge and skills related to objectives of curriculum being evaluated.
![Page 17: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/17.jpg)
![Page 18: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/18.jpg)
Types of Questions (CIPP Framework)
Context questions are written to identify needs of target populations and
opportunities to address those needs/ to determine how well project goals address stated needs
Input questions are written to define capabilities, project strategies and
designs, the goals (e.g., equipment, facilities, staff) Process questions
are written to define deficiencies in the process or implementation, how were resources allocated, and what barriers threaten success
Product questions are written to define outcomes, judge their worth, and
describe lessons learned from the project.
![Page 19: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/19.jpg)
Action Questions? 'Action' or improve questions deal with
matters the project team can readily respond to, to rectify or improve an aspect of the innovation. Other questions might focus on more general or 'big
picture' outcomes not as directly linked to action. Some of these may be ‘prove questions’.
It is important to ask both specific, action-oriented as well as more general, 'big picture’ type questions.
![Page 20: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/20.jpg)
High-value questions? Some questions are particularly useful to
ask because of their high 'pay-off‘. For these questions,
little other information in the area answers are of great interest to the major
stakeholders answers will inform or highlight areas that can
readily be improved Answers are feasibly obtained given the time and
resources available.
![Page 21: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/21.jpg)
‘Prove’ & ‘improve’ questionshttp://www.publichealth.arizona.edu/chwtoolkit/PDFs/Logicmod/chapter4.pdf#search=%22evaluation%20%2B%20logic%20model%22
![Page 22: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/22.jpg)
Strategy for obtaining questions
Theory-Driven Models and logic modelling have a built in mechanism for developing EQs
However, logic models may be used in other approaches.
![Page 23: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/23.jpg)
Strategy for obtaining questions
Logic Models illustrates the purpose and content of your program and makes it easier to develop meaningful evaluation questions from a variety of program vantage points: context, implementation and results (which includes outputs, outcomes, and impact).
![Page 24: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/24.jpg)
Framing questions using the logic model
![Page 25: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/25.jpg)
![Page 26: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/26.jpg)
On logic models The term "logic model" comes from
evaluation, but as the term suggests, they are a basic element of programming that communicates the logic behind a program.
![Page 27: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/27.jpg)
On logic models A logic model’s purpose is to
communicate the underlying "theory" or set of assumptions or hypotheses that program proponents have about why the program will work, or about why it is a good solution to an identified problem.
![Page 28: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/28.jpg)
What do logic models look like Logic models are typically diagrams, flow
sheets, or some other type of visual schematic conveying relationships between contextual factors and programmatic inputs, processes, and outcomes. The scheme shows links in a chain of reasoning about "what causes what," in relationship to the desired outcome or goal. The desired outcome or goal is usually shown as the last link.
![Page 29: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/29.jpg)
How to develop a Logic Model Review and clarify the links between
activities and outcomes (impacts). Add inputs and outputs for each
activity. Construct a draft model. Review and revise.
![Page 30: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/30.jpg)
Use a logic model framework
![Page 31: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/31.jpg)
![Page 32: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/32.jpg)
Evaluating the evaluation questions
Who will use the information? Would an answer reduce the present
uncertainty? Would an answer yield important information? Is this question merely of passing interest Would the omission of the question limit the
scope of the evaluation? Will the answer impact the course of events? Is it feasible to answer this question given the
real life constraints?
![Page 33: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/33.jpg)
Other ways to focus an evaluation
Criteria, indicators, and standards are often used in Quantitative Evaluations Along with each question, multiple criteria may
be specified and used to judge the program. Indicators can then be developed for each
criteria The level of performance expected on each
indicator may also be specified. This is considered a standard or benchmark
![Page 34: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/34.jpg)
Definitions A criterion is an attribute or activity necessary to
fulfill evaluation objectives and overall goals – e.g. performance on the national mathematics assessment
![Page 35: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/35.jpg)
Definitions An indicator is a continuous factor used to
describe a construct of interest. It is a quantitative or qualitative measure of programme performance which demonstrates change and which details the extent to which programme results are being or have been achieved. - e.g. the number of students in each category of performance on the national mathematics assessment
![Page 36: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/36.jpg)
Definitions Standards are descriptors used to describe
the performance level associated with a particular rating or grade on a given criterion or dimension of achievements.
Standards are based on indicators and will answer the question “How good is good enough?” e.g. 75% of the school’s students will be in the advanced and proficient categories in the state mathematics assessment
![Page 37: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/37.jpg)
SamplesGoal Indicator Standard [Benchmark]
50 wireless computers will be purchased.
The number of wireless computers on mobile carts will increase from 10 in Year One to 30 in Year Two, to 50 in Year Three.
Each School will add one wireless access point.
By Oct. 1, each school will have installed wireless access points. By Nov. 1, all wireless computers will be configured to reach access points.
The district will create 10 classroom mini-labs consisting of five workstations, scanner, printer, and digital camera.
By Oct. 15, mini-lab equipment will be ordered. By Nov. 15, mini-lab equipment will be installed in 10 classrooms. By Dec. 15, teacher orientation training will be conducted.
The student computer ratio in classrooms will 6:1 in all classrooms.
The current student computer ration of 20:1 will decrease to 10:1 in Year One, and 6:1 in Year Two.
By May, all students will have equitable access to technology, including network and Internet-based resources within labs, media centers, classrooms, and homes.
50 new software titles will be purchased according to needs identified in teacher surveys.
By Nov., available software will increase from 20 titles to 50 titles.
http://www.seirtec.org/_evaluation/eval2.html
![Page 38: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/38.jpg)
Differentiating Questions, Criteria, Indicators
![Page 39: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/39.jpg)
Questions, Criteria, Indicatorshttp://ec.europa.eu/agriculture/rur/eval/evalquest/b_en.pdf#search=%22evaluation%20questions%2C%20indicators%2C%20standards%2C%20criteria%22
![Page 40: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/40.jpg)
![Page 41: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/41.jpg)
From Questions to Design (& beyond)
Evaluation Level
What Questions Are Addressed?
How Will Information Be
Gathered?
What Is Measured or
Assessed?
How Will Information Be
Used?
1. Participants' Reactions
Did they like it?
Was their time well spent?
Did the material make sense?
Will it be useful?
Was the leader knowledgeable and helpful?
Were the refreshments fresh and tasty?
Was the room the right temperature?
Were the chairs comfortable?
Questionnaires administered at the end of the session
Initial satisfaction with the experience
To improve program design and delivery
2. Participants' Learning
Did participants acquire the intended knowledge and skills?
Paper-and-pencil instruments
Simulations
Demonstrations
Participant reflections (oral and/or written)
Participant portfolios
New knowledge and skills of participants
To improve program content, format, and organization
![Page 42: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/42.jpg)
From questions to design3. Organization Support and Change
What was the impact on the organization?
Did it affect organizational climate and procedures?
Was implementation advocated, facilitated, and supported?
Was the support public and overt?
Were problems addressed quickly and efficiently?
Were sufficient resources made available?
Were successes recognized and shared?
District and school records
Minutes from follow-up meetings
Questionnaires
Structured interviews with participants and district or school administrators
Participant portfolios
The organization's advocacy, support, accommodation, facilitation, and recognition
To document and improve organizational support
To inform future change efforts
4. Participants' Use of New Knowledge and Skills
Did participants effectively apply the new knowledge and skills?
Questionnaires
Structured interviews with participants and their supervisors
Participant reflections (oral and/or written)
Participant portfolios
Direct observations
Video or audio tapes
Degree and quality of implementation
To document and improve the implementation of program content
![Page 43: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/43.jpg)
From Questions to design
Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press. See http://www.gse.harvard.edu/hfrp/eval/issue32/qanda.html
5. Student Learning Outcomes
What was the impact on students?
Did it affect student performance or achievement?
Did it influence students' physical or emotional well-being?
Are students more confident as learners?
Is student attendance improving?
Are dropouts decreasing?
Student records
School records
Questionnaires
Structured interviews with students, parents, teachers, and/ or administrators
Participant portfolios
Student learning outcomes:
• Cognitive (Performance & Achievement)
• Affective (Attitudes & Dispositions)
• Psychomotor (Skills & Behaviors)
To focus and improve all aspects of program design, implementation, and follow-up
To demonstrate the overall impact of professional development
![Page 44: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/44.jpg)
Defining “Evaluation Design” An evaluation design is a detailed
specification of the strategy used to collect data, including the groups to study, the units in the group, how the units will be selected, and the time intervals at which they are studied
See http://www.nsf.gov/ehr/rec/evaldesign.jsp
![Page 45: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/45.jpg)
Models and Approaches Different evaluation designs are usually
associated with specific models. Full Evaluation Models are discussed in the Stufflebeam (2002).
![Page 46: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/46.jpg)
Overall Evaluation Approach
Evaluation Design
Relationship between evaluation design & approaches
![Page 47: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/47.jpg)
Selected Models & Approaches Behavioural Objectives Approach. This
approach focuses on the degree to which the objectives of a program, product, or process have been achieved. The major question guiding this kind of evaluation is, “Is the program, product, or process achieving its objectives?”
![Page 48: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/48.jpg)
Selected Models & Approaches Responsive Evaluation. This approach
calls for evaluators to be responsive to the information needs of various audiences or stakeholders. The major question guiding this kind of evaluation is, “What does the program look like to different people?”
![Page 49: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/49.jpg)
Selected Models & Approaches Consumer-Oriented Approaches. The
emphasis of this approach is to help consumers choose among competing programs or products. The major question addressed by this evaluation is, “Would an educated consumer choose this program or product?”
![Page 50: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/50.jpg)
Selected Models & Approaches Utilization-Focused Evaluation. According
to Patton (1997), “utilization focused program evaluation is evaluation done for and with specific, intended primary users for specific, intended uses” (p. 23). Stakeholders have a high degree of involvement. The major question is:“What are the information needs of stakeholders, and how will they use the findings?”
![Page 51: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/51.jpg)
Selected Models & Approaches Empowerment Evaluation. This approach,
as defined by Fetterman (2001), is the “use of evaluation concepts, techniques, and findings to foster improvement and self-determination” (p. 3). The major question characterizing this approach is, “What are the information needs to foster improvement and self-determination?”
![Page 52: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/52.jpg)
Selected Models & Approaches Theory-Driven Evaluation. This approach to
evaluation focuses on theoretical rather than methodological issues. The basic idea is to use the program’s rationale or theory to understand the program’s development and impact. The major focusing questions are, “How is the program supposed to work? What are the assumptions underlying the program’s development and implementation?”
![Page 53: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/53.jpg)
Selected Models & Approaches Expertise/Accreditation Approaches. The
accreditation model relies on expert opinion to determine the quality of programs. The purpose is to provide professional judgments of quality. The question addressed in this kind of evaluation is, “How would professionals rate this program?”
![Page 54: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/54.jpg)
Selected Models & Approaches Goal-Free Evaluation. This approach
focuses on the actual outcomes rather than the intended outcomes of a program. Thus, the evaluator has minimal contact with the program managers and staff and is unaware of the program’s stated goals and objectives. The major question in this kind of evaluation is, “What are all the effects of the program, including any side effects?”
![Page 55: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/55.jpg)
Evaluation Designs Quantitative
Experimental Quasi-Experimental Non-Experimental
Qualitative Case Study Grounded Theory
Mixed Methods
![Page 56: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/56.jpg)
Choosing Evaluation Designs In studies of what works, critical
question of the quality of evidence. Need to consider causal links in
evaluating intervention effectiveness Manage validity threats
![Page 57: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/57.jpg)
Quantitative Evaluation Designs There is no perfect design
Each design has strengths and weaknesses There are always trade-offs – time, costs,
practicality Acknowledge trade-offs and potential
weaknesses Provide some assessment of their likely impact
on your results and conclusion
![Page 58: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/58.jpg)
Quasi & Experimental Designs Quasi-experimental designs
Strategy #1: Add a control group Strategy #2: Take more measurements (time series
designs) Strategy #3: Stagger the introduction of the intervention Strategy #4: Reverse the intervention Strategy #5: Measure multiple outcomes
Experimental designs Experimental designs with “before” and “after”
measurements Experimental designs with “after”-only measurements
![Page 59: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/59.jpg)
Experimental Evaluation Designs
Experimental Designs all share one distinctive element- random assignment to treatment and control groups.
![Page 60: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/60.jpg)
Experimental Evaluation Designs
Experimental design is the strongest design choice when interested in establishing a cause-effect relationship. Experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.
![Page 61: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/61.jpg)
Quasi-Experimental Designs Most quasi-experimental designs are
similar to experimental designs except that the subjects are not randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment.
Like the experimental designs, quasi-experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.
Types of quasi-experimental designs include: comparison group pre-test/post-test design, time series and multiple time series designs, multiple time series designs, non-equivalent control group, and counterbalanced designs.
![Page 62: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/62.jpg)
Quasi-Experimental Designs Quasi-experimental designs also
prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.
![Page 63: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/63.jpg)
Quasi-Experimental Designs Types of quasi-experimental designs
include: comparison group pre-test/post-test design, time series and multiple time series designs, multiple time series designs, non-equivalent control group, and counterbalanced designs.
![Page 64: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/64.jpg)
Non Experimental Quan Designs
Includes “causal-comparative”, correlational and case study (and multi-site case studies) designs.
Also include mixed methods research designs.
Common in theory driven evaluation
![Page 65: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/65.jpg)
Mixed Method Designs Several typologies currently available –
Most popular are Creswell and Plano Clarke (2007, 2010) and Teddlie and Tashakkori (2009)
Weight, emphasis, timing, and strands are important in classifications
![Page 66: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/66.jpg)
![Page 67: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/67.jpg)
![Page 68: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/68.jpg)
![Page 69: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/69.jpg)
![Page 70: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/70.jpg)
![Page 71: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/71.jpg)
![Page 72: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/72.jpg)
![Page 73: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/73.jpg)
![Page 74: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/74.jpg)
![Page 75: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/75.jpg)
![Page 76: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/76.jpg)
![Page 77: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/77.jpg)
![Page 78: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/78.jpg)
Specific Designs: Concurrent/ Parallel (Teddlie & Tashakkori)
![Page 79: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/79.jpg)
Specific Designs: Sequential
![Page 80: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/80.jpg)
![Page 81: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/81.jpg)
![Page 82: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/82.jpg)
Practical steps in developing an evaluation design
Your evaluation design plan may be presented to the sponsor.
An evaluation design matrix should be developed
![Page 83: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/83.jpg)
Evaluation Design Plan Must include
collecting data analyzing data reporting results getting the results used
![Page 84: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/84.jpg)
Using a Design Matrix
![Page 85: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/85.jpg)
Developing an Evaluation Design
Identify the question(s) to be addressed Select measurement instruments & data
sources Select a model and/or design, Select a sample Develop an analysis plan Develop a timeline for study implementation.
![Page 86: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/86.jpg)
Identify the question(s) to be addressed
Your evaluation questions are the centerpiece of the evaluation.
They are used to develop criteria and indicators as well as measurement instruments/data collection strategies
Different evaluation models focus on different question types
![Page 87: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/87.jpg)
Select measurement instruments & data sources
There are multiple ways of answering most questions.
Generally evaluators seek information from multiple sources
![Page 88: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/88.jpg)
Select a model and/or design
The design you choose depends to a considerable extent on the question you are trying to address and the level of rigour that is required.
Sampling is a critical factor that may be neglected impacting on generalizability.
Choose a model that captures your intention best.
Models are considered in the key texts.
![Page 89: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/89.jpg)
Develop an analysis plan Always specify upfront how you will
analyze the data that you will be collecting - especially in quantitative studies
![Page 90: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/90.jpg)
Develop a timeline for study implementation
Develop a timeline for designing or selecting your instruments, collecting your data, and reporting.
Don’t be over-optimistic with the timelines
![Page 91: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/91.jpg)
Reporting & Communicating There is a need to organize and
consolidate the final report Sections
Background (the project’s objectives and activities);
Evaluation questions (meeting stakeholders’ information needs);
Methodology (data collection and analysis); Findings Conclusions (and recommendations).
![Page 92: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/92.jpg)
Evaluating the evaluation report
A well-written report should provide a concise context for understanding the conditions in which results were obtained as well as identify specific factors that affected the results.
It is necessary to balance description with interpretation and analysis.
recommendations should express views based on the total project experience
![Page 93: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/93.jpg)
![Page 94: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/94.jpg)
The Players-Jennifer Greene
![Page 95: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/95.jpg)
The Players-Michael Q. Patton
My name is Michael Quinn Patton and I am an independent evaluation consultant. That means I make my living meeting my clients’ information needs. Over the last few years, I have found increasing demand for innovative evaluation approaches to evaluate innovations. In other words, social innovators and funders of innovative initiatives want and need an evaluation approach that they perceive to be a good match with the nature and scope of innovations they are attempting. Out of working with these social innovators emerged an approach I’ve called developmental evaluation that applies complexity concepts to enhance innovation and support evaluation use.
![Page 96: HOW TO CONDUCT AN EVALUATION](https://reader035.vdocuments.site/reader035/viewer/2022081507/5681649e550346895dd687de/html5/thumbnails/96.jpg)
The Players- Daniel StufflebeamFounder and director, Ohio State University Evaluation Center, 1963-73Dr. Daniel L. Stufflebeam has wide experience in evaluation, research, and testing. He holds a Ph.D. from Purdue University and has held professorships at The Ohio State University and Western Michigan University. He directed the development of more than 100 standardized achievement tests, including eight forms of the GED Tests; led the development of the evaluation field's Program and Personnel Evaluation Standards; established and directed the internationally respected Evaluation Center, directed the federally funded national research and development center on teacher evaluation and educational accountability, and developed the widely used CIPP Evaluation Model. He has conducted evaluations throughout the U.S., and in Asia, Europe, and South America. His clients have included foundations, universities, colleges, school districts, government agencies, the U.S. Marine Corps, a Catholic Diocese, and others. He has served as advisor to many federal and state government departments, the United Nations, World Bank, Open Learning Australia, several foundations, and many other organizations.