assessing impact evaluating staff development second edition

ASSESSINGIMPACT

Evaluating Staff DevelopmentSecond EditionJoellen Killion

Corwin Press, 2008A joint publication with the National Staff Development Council

Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council 2

As you get settled . . .

Write any questions you have about evaluating staff development on index cards.

One question per card, please.

Intended outcome: think like evaluators and build staff development evaluation into the planning process for the TSSRRP grant.



Describe the eight-step evaluation process and the components of each step.

Define evaluation in relationship to staff development. Understand how a staff development program’s theory

of change and logic model influence the success of a staff development program and its evaluation.

Construct an evaluation framework for a staff development program.

Acquire strategies, tools, and resources to assist in evaluating staff development.


Norms

Share experiences to enrich others.

Ask questions. Learn by doing. Set aside any preconceived

notions about evaluating staff development.

Apply to your own work.

THINK S – A – L – S – A !

A systematic, purposeful process of

studying, reviewing, and analyzing data

gathered from multiple sources in order to

make informed decisions about a

program.


Evaluation is:

Page 8IN THE BOOK


Key ElementsSystematic How rigorous is the process?

Is it conducted in accordance with standards and guidelines?

Standards Does it have merit and/or worth?Does it meet predetermined standards of success?

Audience Who will use the evaluation?For whom is the evaluation being done?

Intended Uses How will the evaluation be used?What decisions will be made as a result of the evaluation?

Pages 8-9IN THE BOOK


Scientifically Based Research vs. Evaluation

Scientifically based research identifies interventions that have been “proven” to be successful under conditions that allow generalizations that these interventions will produce similar “positive” results in other situations.

Evaluation determines whether the selected interventions (based on scientific research or not) have merit (achieved its intended results), worth (perceived as valuable), and/or impact under these conditions.


Purposes of Evaluation

Determine merit and/or

worth

Assess impact

Identify improvements

Provide accountability


Means goals vs. End goals

Provide training Increase student achievement


Evaluation to Assess Impact

Requires the program’s goal to be about increasing student achievement.

Requires a theory of change to explain how the program will produce intended results.


Program vs. Event A program is a set of

purposeful, planned actions and the support system necessary to achieve the identified goals. Effective staff development programs are ongoing, coherent, and linked to student achievement. Page 11-13

IN THE BOOK


Staff Development vs.

Staff Development Program

Talk it over.


Planning1. Assess

Evaluability

2. Formulate Evaluation Questions

3. Construct Evaluation Framework

Conducting4. Collect Data

5. Organize, Analyze, & Display Data

6. Interpret Data

Reporting7. Disseminate

and Use Findings

8. Evaluate the Evaluation

Evaluating Staff Development


Self-Assessment

+ understand

— not sure

? don’t understand


Types of Evaluation

Planning

Formative

Summative



Black Box Evaluations

Input Output

Actions Results

Glass Box Evaluations

?



ProfessionalDevelopment

Action

Student Achievement

Results

Black Box

?


Black Box

Student Achievement


CurriculumDevelopment

Nonacademicfactors

?


Student Achievement

Results


Actions

Glass Box


Student Achievement


Coaching/Follow-upInstructional ResourcesImplementation MonitoringStudent Assessment

Glass Box


Black Box vs. Glass Box

What is the difference?

Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council23

Assumptions1. The staff development program is data-driven,

research-based, and well-defined (p. 21).

2. The school, district, or regional agency has the capacity, including fiscal and human capital, to implement both the program and evaluation with fidelity to their designs (p. 22).

3. Key stakeholders in the school, district, or agency intend to use the evaluation results to make decisions about the program (p. 23).

24

Step 1:Assess

Evaluability


The degree to which a proposed plan for staff development clearly articulates what the staff development program is and how it is expected to produce change.

Step 1: Assess EvaluabilityPlanning Phase


Read and Talk1. Turn to p. 33.2. Read last paragraph3. Turn to p. 344. Read first 2 paragraphs

Can you think of an example?

What was the issue?


Four Components of

Assessing Evaluability

Examine program goals, objectives, standards for success

Examine program’s theory of change

Examine program’s logic model

Adjust program based on assessment


Clarifying Your ProgramWhat are the program’s goals?

Goal is desired state. Improve physical well-being.

Objectives describe the changes expected. Lose 15 pounds in six months.

Activities explain how to achieve the goals and objectives. Walk daily. Eat healthful foods.



Goal, Objective, or Activity? Develop responsible citizens. Increase student proficiency in applying the

scientific process in a laboratory setting as measured by a performance task.

Provide opportunities for professional learning. Establish professional learning communities. Increase student performance 15% on state

assessment in mathematical problem solving.

G

GA

A

O


SMART Goals

Specific change Measurable change Attainable, realistic outcome Results driven (student achievement) Timebound

—Garmston, JSD, Summer 1997, pp. 64-65

Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council

Sample SMART Goals

By 2010, student achievement in mathematics at all grade levels 3–8 will increase by 15% as measured on the state assessment.

By 2010, 90% of the 10th-grade students will demonstrate proficiency in writing on the state writing assessment.


SHORT-TERMKnowledge

Skills

Teachers gain knowledge and skills.

Increase teachers’ content knowledge.

MEDIUMAttitudes

AspirationsBehaviors

Teachers gain aspirations, attitudes, and behaviors.

Change teachers’ instructional practices.

LONG-TERMIntended Results

Students gain aspirations, attitudes, behaviors, knowledge, skills.

Increase student achievement.

ResultsWhat results occur from professional learning for

individuals, schools, communities …


Setting Standards for Acceptable Performance

Specifies how good is good enough

Specifies “success” in advance

Provides a benchmark/baseline for comparison before and after staff development

Page 39IN THE BOOK


Sample Standards for Success Increased student achievement

8% overall increase on state assessment

7% more students in the advanced category and 20% fewer students in the not-proficient category

25% more underrepresented students in the proficient category


Types of Change

Knowledge Attitude Skill Aspiration Behavior

Page 38IN THE BOOK


KASABKnowledge Conceptual understanding of information,

theories, principles, and research

Attitude Beliefs about the value of particular information or strategies

Skill Strategies and processes for applying knowledge

Aspiration Desires, or internal motivation, to engage in a particular practice

Behavior Consistent application of knowledge and skills

Page 38IN THE BOOK

Assessing Impact: Evaluating Staff Development, Second Edition Corwin Press, 2008 A joint publication with the National Staff Development Council37

Student Teacher Principal

Knowledge Students know problem-solving strategies.

Teachers know multiple problem-solving strategies and when to use them.

Principals know what to look for in a problem-solving lesson.

Attitude Students enjoy problem solving.

Teachers believe students’ competence in math is important for success both in and out of school.

Principals believe teachers make a difference in students’ academic success.

Skill Students know how to use their problem solving strategies.

Teachers know how to assess students’ problem-solving ability.

Principals know how to observe and give feedback.

Aspiration Students want to participate in higher- level math classes.

Teachers want all students to learn math.

Principals want all teachers and students to succeed.

Behavior Students apply mathematical reasoning to solve problems in and outside of math.

Teachers provide students with daily practice in problem solving.

Principals monitor student problem-solving scores on benchmark tests and identify students who need extra help.


Theory of Change Identifies the components/actions of a

program (what the program does)

Specifies the relationship among the components to explain how the change occurs (sequence of actions)

Delineates the underlying assumptions upon which the program is based



Source: University of Wisconsin Extension Service

Everyday Theory of Change

Headache Get pills Take pills Headache gone; return to work; contribute to economic well-being of community


Theory of Change

Components/Actions of the program

Sequence of components to show how results will occur

Assumptions underlying the program


Teachers participate in collaborative learning experiences.

Teachers implement new learning in their instruction.

Student performance increases.

IN THE BOOK


Activity

Form groups of two to three people.

In each group, make five cards.

S Asp BAttK


Possible Theories of Change

K + S B + Att + Asp

Asp + Att K B


Activity Using the index cards you made,

create two to three possible theories of change. You might begin with a familiar one.

For each theory of change you create, identify its underlying assumptions.

For each theory of change, also identify the potential implications for key decisions about professional development.


Sample Theory of Change

Review the theory of change on page 42. Start at 12 o’clock.

Read the assumptions on which this program is based on the first paragraph on page 43.

State in your own words what a theory of change is. Page 42

IN THE BOOK


Write the goals, objectives, standards of success, and indicators of success for your case.

Create a theory of change for the staff development program you want to evaluate. When you are satisfied, map it on the large chart paper. Write your assumptions below the diagram.

Post your chart.


Students

Teachers

Coaches

Layering Theories of Change for Systemic Reform

Principals

Central Office

Organization


What is the value of having a theory

of change for a staff development

program?

Talk it over.


Logic Model Specific kind of action plan of the program

specifying the flow of changes (results) that will occur Inputs/resources Activities/processes Initial outcomes Intermediate outcomes Results

Results (Goals)

Inputs Activities IntermediateOutcomes

InitialOutcomes


Homework Assignment Read pages 46-51 Create a logic model for at least 2 actions

or components of your staff development program’s theory of change. (Worksheet #4)

Assess your progress using the Evaluability Assessment Rubric on page 51.


Whew! My brain is tired!51


Review of Session Outcomes

Describe the eight-step evaluation process and the components of each step. √

Define evaluation in relationship to staff development. √ Understand how a staff development program’s theory

of change and logic model influence the success of a staff development program and its evaluation. √─




Final Activity

What have you learned

about evaluating staff

development that will

help you as you develop

the process for

evaluating your own staff

development program?

ASSESSINGIMPACT




Logic Model Specific kind of action plan of the program

specifying the flow of changes (results) that will occur Inputs/resources Activities/processes Initial outcomes Intermediate outcomes Results

Results (Goals)

Inputs Activities IntermediateOutcomes

InitialOutcomes



Inputs/Resources

Outputs/Activities

Initial Outcomes

K-Knowledge S-Skills

IntermediateOutcomesA-Attitudes

A-AspirationsB-Behaviors

Intended Results

Results

Formative Questions Summative Question

Program Goals/Objectives

Implementation Impact

Logic Model: Planning and Evaluating

Theory of Change


What’s the value of having both a theory of change

and a logic model

for a staff development

program?

Talk it over.

58

Step 2: Formulate Evaluation

Questions


Step 2: FormulateEvaluation Questions

Planning PhaseCarefully and thoughtfully crafted evaluation questions give structure and focus to the evaluation framework.

Page 53IN THE BOOK


A Good Question . . .

Focuses your evaluation. Guides the design of the evaluation

framework. Clarifies what the intended users

want to know. Facilitates both design of evaluation

and use of findings.


Types of Evaluation Questions Need

What is the problem we want to solve? Design/Planning

How do we prepare the staff? Process

Is the program being implemented as planned?

ImpactWere the intended results achieved?



Different Questions Require Different Kinds of Evaluation Processes and Evidence

Did student achievement increase by 3% as measured by the state test?

What components of the program do teachers perceive to be most beneficial in changing their instruction?

What changes in teachers’ instructional practices occurred?

To what degree was the program implemented?


Characteristics of a Good Question

Reasonable Appropriate Answerable Specific in terms of

program success Specific in terms of

measure of performance Page 60

IN THE BOOK


ActivityRead the

examples of evaluation

questions and the

commentary on pages 60–63.


Sample Evaluation Questions Improve student writing performance by 8% as

measured on the district grade-level writing rubric: Did students’ classroom writing performance (as measured by the district’s grade-level rubric) increase from last year to this year by 8% as a result of teachers’ use of process writing?

Increase student performance in math problem solving by 5% on the state math assessment: Did students’ scores on the state math assessment in the area of problem solving increase by 8% in classrooms where teachers were using the new instructional strategies to teach students how to solve problems?


Evaluation Question

What is/are your evaluation question(s)? Be sure that the questions align with the

program’s goals. Review your theory of change to see if it is possible to answer your questions given the program’s design.


Share your evaluation questions with a partner and ask for his/her commentary

about whether your question is:

Reasonable. Appropriate. Answerable. Specific in terms of program’s

success. Specific in terms of measurement.

68

Step 3:

Construct the Evaluation Framework


Step 3: Construct the Evaluation Framework

Planning Phase

Decisions about the evaluation Who will conduct the evaluation? What do you need to know (questions)? How will you know (evidence needed)? Who or what will tell you (data sources)? How, when, and where will you collect evidence (data-

collection methods)? How will you plan to analyze the data? When will the evaluation be completed?


Beginning Steps of an Evaluation Framework

I need to know . . . By collecting the following kinds of

information . . . Information will be most useful if it

comes from . . . The data can best be collected

through . . .


Mini-Lessons Impact Means . . . Measuring Change Experimental vs.

Quasi-Experimental Design

Data Sources Data-Collection

Methods


Mini-Lesson 1: Assessing Impact

Assessing impact means measuring change.

Change in what?


Mini-Lesson 2:Data Sources

Sources are the people, places, documents, etc., that provide the information you need.

What and who are possible data sources for each of your evaluation questions?


How effectively are teachers implementing inquiry-based instruction in their science lesson?

Possible data sources: Teachers Students Principals/administrators Student test scores Science kit checkout Lesson plans

Test question analysis Observational report Standards Pre- & post-teacher

tests Student work Teacher reflection


Mini-Lesson 2: Triangulation of Data Sources

Multiple sources

of information for the same kind

of information


Types of Data Sources

Authentic Approximate


Determining Data Sources

Goal Standard of Success

Indicator of Success

Data Source

Increase student achievement in mathematics

By 10% in three years

Fair and reliable measures of mathematics achievement

• State assessment

• Level tests• Schoolwide

common assessments


Mini-Lesson 3:Data Collection Methods

Specific tools or processes used to collect the desired data Surveys Interviews Logs Observations Tests Focus Groups Documents/Artifacts Expert Judgments Etc.


Sample Data Collection Instruments

Review the sample data collection tools on pages 159–194.

Consider how these tools might be adapted for your evaluation.


Determine Appropriateness of Data-Collection Methodologies

Financially feasible Reasonable in terms of human

capacity Appropriate for type of evidence

desired Authentic vs. approximate


TaskComplete the evaluation framework for an evaluation you want to do. Program objectives Evaluation questions Type of design Comparison group Data sources Data-collection methods Timeline Page 70

IN THE BOOK

ASSESSINGIMPACT



83

Step 4:Collect

Data


Step 4: Collect DataConducting Phase

Systematic collection of data

Page 97IN THE BOOK


Possible Ways to Score Lesson Plans

Is complete: yes or no Includes basic features, such as a standard,

instructional strategy, and a method of assessment (present or not present)

Meets standards of quality (rubric), e.g., appropriateness of the instructional strategy and assessment process for the standard


Collect Data

The key in data collection is to be methodical and systematic.


Collect Data In your data

collection process, what do you want to be particularly thoughtful about?

88

Step 5:Organize, Analyze, and Display Data


Step 5: Organize, Analyze, and Display Data

Conducting Phase

Page 101IN THE BOOK


Analyzing Data

Describing Counting Factoring Clustering Comparing Seeking

trends/patterns

Examining outliers Finding covariation Eliminating rival

explanations Modeling Univariate analysis



Ways to Analyze Data Review pages 103–105. Which one(s) will be most appropriate for

your data and evaluation question?


Ways to Analyze


Sample AnalysisLevel of Teacher Use Student Achievement

Routine Use

Mechanical Use

Nonuse Use


What are the best ways to display your data for the target audiences of your evaluation?


Step 6:Interpret Data


Step 6: Interpret DataConducting Phase

Making meaning from data Forming and supporting claims of merit, worth,

and/or impact Clarifying attribution and contribution Making recommendations

Page 109IN THE BOOK


Evidence and Claim

Evidence is the data.

Claim is the statement of value or worth, the conclusion drawn based on the data.


Evidence or Claim

Read the statements on the handout.

Determine which are claims and

which are evidence.


Types of Claims

Merit Worth Impact Attribution vs. Contribution


Rigor of Claims

Are the claims strong or weak?

Are the claims major or minor?


Attribution vs. Contribution

Attribution: Program is solely responsible for producing results.

Contribution: Program was involved in producing results.


Forming Recommendations

Revisions to make Actions to take Decisions to make Possibilities to consider


Interpreting Data

Involve stakeholders Form and support claims Clarify attribution and contribution Make recommendations

104

Step 7:Disseminate and

Use Findings


Step 7: Disseminateand Use Findings

Reporting Phase

Consider audiences Determine types of reports

Interim versus final Oral versus written

Choose dissemination strategies

Page 119IN THE BOOK


“Simplicity as a virtue means that we seek clarity, not complexity”

(Patton, 1997 p. 310)

“To write so that you could not be misunderstood”

(Personal communication, 2001)


Disseminate Findings

Who wants to know what you learned? In other words, who is/are the audience(s) for your evaluation report? To whom are you obligated to report? To whom do you want to report?

What is the best format in which to present your findings so that they will be clear and easily understood?


Disseminate Findings

Look at the possible components of an evaluation report on page 123.

Consider one audience for your results.

Which components will be appropriate for that audience?

109

Step 8:Evaluate

the Evaluation


Step 8: Evaluate the Evaluation Reporting Phase

Resources Design Analysis Interpretation Findings Reporting Evaluator

Page 129IN THE BOOK


Was the evaluation . . .

EffectiveWorth doingWell done Efficient


Planning1. Assess

Evaluability

2. Formulate Evaluation Questions

3. Construct Evaluation Framework

Conducting4. Collect Data

5. Organize, Analyze, & Display Data

6. Interpret Data

Reporting7. Disseminate

and Use Findings

8. Evaluate the Evaluation

Evaluating Staff Development


Review of Session Outcomes

Describe the eight-step evaluation process and the components of each step.

Define evaluation in relationship to staff development. Understand how a staff development program’s theory

of change and logic model influence the success of a staff development program and its evaluation.



Intended outcome: think like evaluators and build staff development evaluation into the planning process for the TSSRRP grant.



Final Activity

What have you learned

about evaluating staff

development that will

help you as you develop

the process for

evaluating your own staff

development program?