overview of chapters 11 – 13, & 17
DESCRIPTION
Overview of Chapters 11 – 13, & 17. Chapter 11. Clarifying the evaluation request and responsibilities An organization has requested that you to complete an evaluation. What questions do you feel are important to ask before beginning the evaluation? - PowerPoint PPT PresentationTRANSCRIPT
Overview of Chapters 11 – 13, & 17
Clarifying the evaluation request and responsibilities
1. An organization has requested that you to complete an evaluation. What questions do you feel are important to ask before beginning the evaluation?
2. Are there times when you would decline a request for evaluation? Why?
3. What are some of the advantages and disadvantages in having an evaluation conduct by an external evaluator? By an internal evaluator?
(take 5 minutes with a partner to discuss)
Chapter 11
1. Identify the stakeholdersKey questions:Who are the sponsors and clients?Who are the managers and staff?Are there other interest groups, such as agencies, elected
officials, the public at large?
2. Identify the purpose for the evaluationKey questions: Why is the evaluation being requested?What question will it answer?How will the results be used? By whom?
Clarifying the Evaluation Request and Responsibilities
3. Determine if the Evaluation is AppropriateKey questions:How will the results be used?Will the results produce trivial information?Is the program too new?Is the evaluation purpose ethical?
Clarifying the Evaluation Request and Responsibilities
Steps to follow: 1. Clarify the intended program model or theory
2. Examine the program in implementation to determine whether it matches the model and could achieve the goals or objectives
3. Explore different approaches to determine if they meet the stakeholders information needs
4. Agree on evaluation priorities and intended uses of the study
Clarifying the Evaluation Request and Responsibilities
Internal vs. External evaluator
Internal: know the organization, history, and decision-making style and would be around to encourage use of evaluation results
External: bring a greater objectivity and sometimes specialized skill for a particular project or evaluation
Who will evaluate the program?
Setting Boundaries and Analyzing the Evaluation Context
1. After determining who the stakeholders are, how do you determine who should be involved in the evaluation process?
2. What should the evaluator consider in analyzing the political context in which an evaluation will occur?
(take 5 minutes to discuss with a partner)
Chapter 12
The evaluator should identify and communicate with each stakeholder group or its representative:
1. To learn about the groups concerns2. To understand how they will use the
evaluation results
This will help the evaluator weigh their input during the evaluation
Setting Boundaries and Analyzing the Evaluation Context
Do evaluators need to be aware of Squeaking wheels &Powerful Stakeholders?
You need to bring about a “democratic dialogue” if possible, with a balance in the number of people involved and diversity within the group.
Setting Boundaries and Analyzing the Evaluation Context
Greene (2005) identified four groups:1. People who have decision authority over
the program2. People who have direct responsibility for
the program3. People who are the intended beneficiaries
of the program
Setting Boundaries and Analyzing the Evaluation Context
Scriven (2007) adds to the list:
Political supporters or opponents
Community Leaders
Public in General
Setting Boundaries and Analyzing the Evaluation Context
Political Context:
1. Who will gain to gain/lose?2. Which individuals and groups have power in the
setting?3. How should the evaluator relate/communicate to the
groups?4. Will all groups cooperate?5. Which groups have a vested interest in the
outcomes?6. What safeguards should be incorporated into the
evaluation agreement?
Setting Boundaries and Analyzing the Evaluation Context
Identifying and selecting the evaluation questions and criteria
1. What is the focus of evaluation questions?2. What are good sources for evaluation
questions?3. Who should be involved in the divergent
and convergent phases? Why?
Chapter 13
Evaluation Questions:
Provide focus to the evaluationSpecify the information the evaluation will
provideGuide choices for data collection, analysis,
and interpretation
Identifying and selecting the evaluation questions and criteria
Good Sources:
Existing Standards in the FieldResearch LiteratureContent ExpertsEvaluators Own Experience
Identifying and selecting the evaluation questions and criteria
Divergent: Cronbach (1982), “Opening one’s mind to be entertained at least briefly as prospects for investigation” (p. 210).
Fitzpatrick, et al. (2011), “throw a broad net and learn from many possible sources” (p. 316).
Identifying and selecting the evaluation questions and criteria
Convergent: Three reasons for reducing the range of variables.
1. Budget2. Manageability/Complexity3. Attention Span of Audience
The evaluator aims for the maximum bandwith.Cronbach (1982)
*Typically completed by an advisory group with representatives from different stakeholder groups.
Identifying and selecting the evaluation questions and criteria
1. What considerations are important in tailoring the reporting of evaluation results to audiences?
2. What are some of the ways results can be communicated to stakeholders?
3. What should be included in an evaluation report?
(take 5 minutes to discuss with a partner)
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
Different audiences have different information needs and knowledge levels.
Reading abilityFamiliarity with the programAttitude toward the programRole in decision makingFamiliarity with evaluation practices/methodsAttitude and interest in evaluationExperience in using evaluation results
Torres, Preskill, and Pointek (2005)
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
If possible (Patton, 2008) points out that evaluation data are sued more if the evaluator discusses and negotiates the format, style, and organization of reports with primary users.
Other Key Points: Avoid Jargon, Use simple direct language, Use examples, anecdotes,
be interesting…
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding
Key Elements of an Evaluation Report (p. 470)I. Executive SummaryII. Introduction to the ReportIII. Focus of the EvaluationIV. Brief Overview of Evaluation Plan &
ProceduresV. Presentation of ResultsVI. Conclusion and RecommendationsVII. Minority Reports or RejoindersVIII. Appendices
Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding