Program Evaluation Strategies to Improve Teaching for Learning

Rossi Ray-Taylor and Nora MartinRay.Taylor and Associates

MDE/NCA Spring School Improvement Conference2008

Introductions

Overview for the session

Evaluation comfort levels

0

5

10

15

20

25

30

conductingevaluation

beingevaluated

notcomfortable

somewhatcomfortable

verycomfortable

Research driven school design

Rigor, relevance, relationships

Evaluation can provide information about

• Definition and evidence of the problem (needs assessment)

• Contextual information about the factors related to the problem and solutions

• The input factors and resources• The interventions, processes and

strategies employed• Outcomes, results and effectiveness

Program evaluation can be a form of action research

Elements of a sound evaluation

• Clear statement of what the project is intended to do and why– Needs assessment– Theory of action– Clear, measurable statement of goal attainment

• Appropriate evaluation methods and tools • Transparency

Evaluation can lay the ground work for accountability

Key concepts

• Identify the total system impact of program and financial decisions

• Institutions led by data and evidence of results

Types of Data

• Achievement

• Demographic

• Program

• Perception

• Process

• Costs

Qualities of Data & Information• Time series, repeated data collection–

how does the effect change over time?

• Cohort analysis – overall effect on one group over time

• Benchmarking & standards – makes the data relative, establishes context and comparables

• Triangulation – looking for information in multiple indicators

• Pattern & trend analysis

• Leading & lagging indicators

Practical matters

• Accuracy

• Reliability

• Validity

• Accessibility

Evaluation

Employ evaluation strategies from the very beginning of a project and assemble and review effectiveness data. State measurable outcomes, document processes and review progress throughout the life of the project.

“Where outcomes are evaluated without knowledge of implementation, the results seldom provide a direction for

action because the decision maker lacks information about what produced the outcomes (or lack of

outcomes).” Michael Quinn Patten

Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”,

Victoria L. Bernhardt, 1998

Central Evaluation Questions

• Did the program / project do what was intended?– Did the project stick to the plan?

There may be valid reasons for varying from the plan – if so what are they?

• What is the theory of change for the project?

• Why is this project being carried out this way and why is it judged to be the most appropriate way?– These questions are important because

they help when judging the impact of changes in the plan.

• What is the context for the project?– Issues include past trends, local politics,

resource distribution, threats, opportunities, strengths and weaknesses

• What actually happened during the course of the project?

• Who was served by the project? Why?

• Who was not served by the project? Why?

• Was the project valued by the intended audience?

• What were the inputs, resources – both real e.g., financial and material – and intellectual?

• What was the “cost” of the project?

• What were the results /outcomes?• Were goals and objectives met?• What were the intended and unintended

consequences?• What was the impact on the overall system?• Was there a process impact – did the project

result in a change in the way that business is done?

A successful project begins and ends with a good evaluation design

How do these questions improve teaching for learning?

The best and most “sticky”, lasting interventions have the following

components

• They are based in research and evidence

• They are locally constructed and customized

• They are targeted to a clear view of the problem to be solved

• They are built for sustainability

• They are “owned” – not just a matter of compliance

• They are designed to build local capacity

Understand change and apply research about change to

improve teaching for learningChange takes trust

Change takes building relationshipsChange takes endurance (time)

Change takes knowledge of research

Examine policies and practices that serve as barriers and those

that serve as catalysts to achievement

Evaluate

Audit the systemMeasure results

Change to goal not just awareness or implementation

Meta Evaluation

Use meta evaluation strategies to look for results across projects and interventions

Evaluation readiness

Identify and clearly state program goals and outcomes

Transform these goals into measurable objectives

Define program theory and supporting research

Develop the formative and summative evaluation plan

Develop the plan to gather data, deploy evaluation resources and gather information from and report to stakeholders

Design for continuous feedback and transparency

Consider system evaluation policies and expectations

At the time of proposal initiatives, programs and projects should be designed to include program evaluation

“It’s easy to make judgments – that’s evaluation. It’s easy to ask questions about impact – that’s evaluation. It’s easy to disseminate reports –

that’s evaluation. What’s hard is to put all those pieces together in a meaningful whole which tells

people something they want to know and can use about a matter of importance. That’s

evaluation.” HalcolmQuoted in “Data Analysis for Comprehensive Schoolwide Improvement”,

Victoria L. Bernhardt, 1998

Program evaluation planning document Goal: What does this project intend to achieve? State goals in observable terms. Limit the project to 3-5 overall goals.

Objective

State project objectives in measurable terms

Measure/Criteria

How will the objective be measured? What tools or methods will be used?What are the criteria for success?

Evaluation activities & questions

What activities and tasks need to be accomplished to carry out evaluation of this objective?

State the evaluation questions

Time line

When will data be collected and reported?

Person responsible

Who is responsible for this task?

Resources and budget

What is the budget and what other resources will be needed?

In the beginning you think. In the end you act. In between you negotiate the possibilities. Some people move from complexity to simplicity and on into catastrophe. Others move from simplicity to

complexity and onward into full scale confusion. Simplification makes action possible in the face of overwhelming complexity. It

also increases the odds of being wrong. The trick is to let a sense of simplicity inform our thinking, a sense of complexity inform our

actions, and a sense of humility inform our judgments…” Michael Quinn Patten (p. 143 Bernhardt)

Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”,

Victoria L. Bernhardt, 1998

Resources

• Center for Evaluation & Education Policy www.ceep.indiana.edu

• The Evaluation Center; Western Michigan University

More Resources

• What Works in Schools Translating Research into Action by Robert J. Marzano

• Data Analysis for Comprehensive Schoolwide Improvement; Victoria L. Bernhardt

More Resources

• The “Data Wise” Improvement Process: Eight steps for using test data to improve teaching and learning, by Kathryn Parker Boudett et al in Harvard Education Letter, January/February 2006 Volume 22, Number 1.

Rossi Ray-Taylor, PhD rossi@raytaylorandassoc.org

Nora Martin, PhDnora@raytaylorandassoc.org

Ray.Taylor and Associates2160 S. Huron Parkway, Suite 3

Ann Arbor, Michigan 48104www.raytaylorandassoc.org