evaluating the impact of professional development: five ... · evaluating the impact of...
TRANSCRIPT
1
Evaluating the Impact of
Professional Development:
Five Levels of Evidence
Colorado Transition Leadership Institute
June 22, 2011June Gothberg , Western Michigan University
Agenda
Information on evaluation
Information on five levels of evaluation
Tools for evaluation
Hands-on practice
2
Intended Outcomes
Increase knowledge of five levels of
evaluation
Increase knowledge of the continuum of the
five levels of evaluation starting from
participant satisfaction and ending with
student outcomes
Intended Outcomes
Include appropriate potential indicators and
data sources to evaluate each TAP expected
outcome
Increase knowledge of evaluation tools
Demonstrate knowledge by creating an
evaluation tool
3
Purpose
Improve
student
outcomes!
Purpose of Our Work
Process
Improve
student
outcomes by
improving
what we do!
Purpose of Our Work
4
Proof
Evaluate
improvement
efforts to
determine
effect on
student
outcomes
Purpose of Our Work
Evaluation Connection
Program
planning
Program
evaluation
The best evaluation plans are created during
the strategic planning phase.
5
Evaluation Connection
Evaluation compares the measured impacts
and outcomes against the strategic plan
Evaluation looks at:
the original objectives
what was accomplished
how it was accomplished.
Evaluation Connection
Program
planning
Identify
outcomes
Program
evaluation
Measure
outcomes
6
Lessons Learned
What we’ve learned from local plan analysis:
If teams don’t anticipate their outcomes, they can’t
tell if they’ve achieved them
If teams don’t plan evaluation during the institute
planning phase, evaluation is not likely to happen
When evaluation doesn’t happen, teams don’t know
if they have accomplished what they planned
Our Focus: To Link
Planning and Evaluation
GoalsActivities
Outputs
Outcomes
7
Pull the Pieces Together!
GoalsActivities
Outputs
Outcomes
What is Evaluation?
A study designed and conducted to assist an
audience to assess an object’s merit and
worth.1
The identification of defensible criteria to
determine an evaluation object’s value (merit
or worth), quality, utility, effectiveness, or
significance.2
1. Stufflebeam, D. (1999)
2. Worthen, Sanders, Fitzpatrick (1997)
8
What is Evaluation?
Evaluation is judging the:
Merit
Worth
Or significance of things
near synonyms are quality/value/importance
1. Stufflebeam, D. (1999)
2. Worthen, Sanders, Fitzpatrick (1997)
What is Evaluation?
1. Scriven, M. (1990)
Value
Merit
Value to an individual consumer
Worth
Value to an institution or
collective
9
Evaluation Questions
In general most teams want to answer
these three questions:
Did we do what we said we were going to do?
How well did we do it?
What difference did it make?
In other words what were our outcomes?
Assessing your Outcomes
Outcomes
10
Assessing your Outcomes
What do
you want to
achieve?
Assessing your Outcomes
What do
you want to
achieve?
How will you
know you
achieved it?
11
Assessing your Outcomes
Short-term Long-term
Assessing your Outcomes
Short-termChanges in skills,
attitudes,
knowledge,
behavior, and
decision making
Long-termChanges in status
or life condition
12
Five Levels of Evaluation
Kirkpatrick’s four-level training evaluation
model
Reaction
Learning
Behavior
Results
Five Levels of Evaluation
Tom Guskey adapted this for his work on
professional development evaluation to
include a 5th level and what may be the most
important level for our work today:
Student learning outcomes
13
Five Levels of Evaluation
NSTTAC adapted Gusky’s model for capacity
building and technical assistance evaluation
Guskey based on evaluating a one-time linear
professional development event
NSTTAC’s model adapted for complex multi-site,
multi-level evaluation efforts
Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press
Five Levels of Evaluation
Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press
Level 5: Student learning outcomes
Level 4: Participant use of new knowledge and skills
Level 3: Organization support and change
Level 2: Participant learning
Level 1: Participant reactions
14
NSTTAC Examples - Level 1
Level 1 – Participant reactions
Likert-like scale evaluations of institutes,
cadre meetings, workshops
Achievement of intended outcomes
Usefulness of information
Relevance of materials
Qualitative open ended questions
What worked and what didn’t
Colorado Example - Level 1
Gothberg & Goldsby, 2010, Colorado Transition Leaders Institute Evaluation 2010
15
Level 1 – Report Table
NSTTAC Examples - Level 2
Level 2 – Participant learning
Pre-post tests
New knowledge and skills of participants:
student, teacher, and parent instruments
Analysis of products
Development of IEPs
16
Colorado Example - Level 2
Wirth & Lowenthal, 2009, I-13 Training Pre-then-Post Evaluation
Level 2 – Report Table
Note. Frequency (f) represents the number of participants with a correct answer on the pretest and
posttest. Dependent t test (across all items) revealed a significant difference between pretest scores and
posttest scores, t(396)=-22.06, p < .0001.
17
Level 2 – Report Graph
NSTTAC Examples - Level 3
Level 3 – Organization support and change
Analysis of teacher reports regarding curriculum
implementation Identification of facilitators and barriers to curriculum
implementation, including administrative support
Analysis of annual performance reports (APRs) to
determine
Change in data collection procedures
Alignment of strategic plans (from institutes) with
improvement activities in “determination ” areas
Change in target indicators
18
Colorado Example - Level 3
Gothberg, 2010, Windsor Implementation Evaluation 2010
Level 3 – Report Table
19
NSTTAC Examples - Level 4
Level 4 – Participant use of new knowledge and skills
Analysis of state and local strategic plans (from
institutes)
To document and improve the implementation of
program content
To assess growth from year to year
Evaluation of local curriculum implementation
To assess if and how participants applied their new
knowledge at the classroom level
Colorado Example - Level 4
Kohler & Coyle, 2006, Self-determination Curriculum Evaluation, Tool 1
20
Level 4 – Report Table
NSTTAC Examples - Level 5
Level 5 – Student learning outcomes
Questions
What was the impact on students?
What was the impact performance or achievement?
What was the impact physical or emotional well-being?
Is student attendance improving?
Are dropouts decreasing?
What’s measured
Student learning outcomes: Cognitive, affective,
psychomotor
21
Level 5 - Student Learning
Ways to measure cognitive learning (knowledge):
Standardized achievement assessments
Standardized performance assessments (often scored
with rubrics)
Teacher-developed classroom assessments
Group tasks or activities
Portfolios and other collections of students’ work
Grades or marks
Questionnaires and interviews
School records
Level 5 - Student Learning
Ways to measure affective learning (attitudes &
dispositions) :
Questionnaires
Interviews
22
Level 5 - Student Learning
Ways to measure psychomotor learning (skills &
behaviors) :
Observations
Questionnaires
Interviews
School records
Level 5 - Student Learning
Ways to measure psychomotor learning (skills &
behaviors) :
Incidences in school vandalism
Student behavioral problems
Disciplinary recommendations
Suspensions
Tardiness
Absenteeism
Student participation in service organizations
Use of library, media center, or technology center
23
Level 5 - Student Learning
Ways to measure psychomotor learning (skills &
behaviors) :
Number of special service referrals
Dropout rates
Graduation rates
Post-school outcomes
NSTTAC Examples - Level 5
Level 5 – Student learning
Analysis of APRs and SPP/APR Indicators
To determine school and student improvement on
federal performance and compliance indicators
To demonstrate the overall impact of capacity
building
To assess impact of capacity building model at the
state and local levels
Student portfolios and oral reports
To measure student learning outcomes
24
NSTTAC Examples - Level 5
Level 5 – Student learning
Importance of student voice Student assessment gives us a quantitative view of what the
students learning level and yearly process may be, but when
you incorporate student voice into your evaluation work, you
may see beyond the test and into the impact happening at
the classroom level.
Please listen to the following clip from a student focus group:
Colorado Example - Level 5
Kohler, Coyle, & Gothberg, 2009, Self-determination Curriculum Evaluation, Tool 4
25
Level 5 – Report Table
LEVEL 5 EVALUATION: STUDENT LEARNING OUTCOMES
EVALUATION Durant High School Tool 4: Student Self-Assessment of Student Involvement
Durant, OK, March 11, 2009
PURPOSE To measure extent self-determination courses have impacted student learning outcomes as seen in student involvement on the IEP
EVALUATION
QUESTIONS
Q1. Did the student attend their IEP?
Q2. How much did the student contribute in the IEP meeting?
DATA
SOURCES
Tool 4: Student Assessment of Student Involvement
OUTCOME Q1. The percentage of students that attended their IEP:
100% Students attended their IEP
Q2. The percentage of students that felt they contributed somewhat to yes in the IEP meeting:
100% Identified their post-secondary goals
100% Provided information about their strengths
100% Provided information about their limitations or problem areas
100% Provided information about their interests
100% Provided information about the courses they want to take
100% Reviewed their past goals and performance
100% Asked for feedback or information from the other participants at their IEP meeting
100% Identified the support they need
100% Summarized the decisions made at the meeting
IMPLICATION A significant increase in the number and extent of students involved in their own IEP as self-assessed
Evaluation Help
NSTTAC Evaluation Toolkit
A tool for “data-based” decision-making
Provides “real-life” examples for various states’
evaluation instruments
Samples for your use
Technical Assistance
26
Evaluation Tools
NSTTAC Evaluation Toolkit examples
The entire *updated* Toolkit may be
downloaded from:
www.nsttac.org
Evaluation Tools
NSTTAC Evaluation Toolkit
A tool for “data-based” decision-making
Provides “real-life” examples for various
states’ evaluation instruments
Samples for your use
27
Evaluation Planning Tool
Evaluation Planning Tool
28
Evaluation Planning Tool
Evaluation Planning Tool
29
Hands-on Activity
Please get into small groups
In an envelope are several section three
goal sheets
As a group, please select one for your
practice activity
Hands-on Activity
Use the handouts (Evaluation Planning, Five
Levels, and Evaluation Planning Tool) to
plan an evaluation of your selected goal
Create one evaluation product (survey,
observation guide, interview script)
Nominate someone to report out
30
Thank you!!
June Gothberg
Western Michigan University