models for evaluating msp projects workshop for state...
TRANSCRIPT
![Page 1: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/1.jpg)
Models for Evaluating MSP ProjectsWorkshop for State Leaders and Evaluators of
Teacher Professional DevelopmentWashington, DC
April 24, 2008
Norman L. WebbSenior Research Scientist
Wisconsin Center for Education Research
University of WisconsinMadison, Wisconsin
![Page 2: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/2.jpg)
![Page 3: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/3.jpg)
Program StagesPlanningImplementationImpact
![Page 4: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/4.jpg)
First Steps in Initiating an Evaluation
Finding an evaluatorExternal/internalIndependent/within program
Cost for evaluationScope for evaluation
![Page 5: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/5.jpg)
Evaluation for Program Planning
![Page 6: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/6.jpg)
Compatibility between Program Design and Evaluation Design
![Page 7: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/7.jpg)
No causation without manipulation
![Page 8: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/8.jpg)
Experimental and Quasi-Experimental Designs
Pre-Post TestPre-Post Test with a Control GroupPost Test OnlyPost Test with a Control GroupRandom AssignmentMultiple Levels
Teacher – StudentSchool – Student
Trend Analysis—Baseline, Periodic Assessments
![Page 9: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/9.jpg)
Professional Development Program DesignSelection of Teachers
AllVolunteersPurposeful
Selection of SchoolsAllVolunteersPurposefulStratified
![Page 10: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/10.jpg)
Evaluation of Program Theory
Research BasedLogic
![Page 11: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/11.jpg)
![Page 12: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/12.jpg)
Simon, 1997 Mathematics Teaching Cycle
![Page 13: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/13.jpg)
Logic Model Framework for Listing Program Activities
![Page 14: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/14.jpg)
Professional Development Activities
F6)F5)F4)MTLs monthly two-day workshops on leadership
E6)Teachers -bi-weekly grade level meetings
E5)E4)
D6)Teachers -one-week summer institute
D5)Teachers -after school weekly classes
D4)MTLs support other teachers
ACTIVITIES
Curriculum and Instruction
![Page 15: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/15.jpg)
Professional Development Interim Outcomes
L6)Teachers learn to manage reflective teaching classroom
L5)Teachers evaluate students’understanding based on viable model of how mathematical understanding develops
L4)
K6)Teachers make instructional and curriculum decisions based on students’understanding
K5)Teachers gain skills at recognizing student mathematical thinking and experiment with techniques
K4)
J6)J5)Teachers build deeper understanding of content and student thinking
J4)MTLs gain leadership skills and recognize teachers’ growth in reflective teaching
I6)Teachers independently use reflective teaching strategies
I5)Teachers build confidence in reflective teaching of mathematics
I4)MTLs’ beliefs, attitudes, and practices become aligned with reflective teaching
INTERIM
Curriculum and Instruction
![Page 16: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/16.jpg)
Program Implementation and Evaluation
![Page 17: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/17.jpg)
Guskey’s Five Critical Levels of Evaluating Professional Development
1. Participants’ Reactions, 2. Participants’ Learning, 3. Organization Support for Change, 4. Participants’ Use of New Knowledge and
Skills5. Student Learning Outcomes.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. As reported by Monsaas and McGee-Brown, Evaluating Professional Learning Communities Using Mixed, Quantitative and Qualitative Methodology, MSP Evaluation Summit, September 2005. CD available from Consortium for Building Evaluation Capacity, Utah State University.
![Page 18: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/18.jpg)
Some Issues Related to Evaluating the Implementation of Professional Development
Time FrameProof-of-Concept StudiesBreadth vs. DepthFormative Studies Aligned Measurement InstrumentsLiving with ImperfectionAnalytic Horizon Mismatch with Funding
![Page 19: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/19.jpg)
Frequently Used Methods in Evaluation of Professional Development Programs
Case StudiesExperimental DesignsMulti-Variant DesignsTrend Analyses
![Page 20: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/20.jpg)
Frequently Used Tools in Evaluating Professional Development Programs
SurveysObservationsInterviewsFocus GroupsTeacher LogsContent Knowledge Tests
![Page 21: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/21.jpg)
Web Sites with Evaluation Instruments
www.consortium-chicago.orghttp://ed.uwn.edu/CAREI/cetphttp://[email protected]://www.addingvalue.org/
![Page 22: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/22.jpg)
Less Frequently Used Tools
Content Coverage MapsConcept MapsStudent Work Samples
![Page 23: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/23.jpg)
Program Impact
![Page 24: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/24.jpg)
Math Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other schools
Q15%median95%Q3
Value-Added Effect on Mathematics Scores from Grade 4 to Grade 5, 2003-2005
![Page 25: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/25.jpg)
Spanish Grade 5
-10
-5
0
5
10
15
AlACiMa Cohort I Schools Other Schools
Q15%median95%Q3
Value-Added Effect on Spanish Scores from Grade 4 to Grade 5, 2004-2005
![Page 26: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/26.jpg)
0.90-0.120.58-0.07Spanish
0.061.860.681.26Mathematics
Pr > │t │t ValueStandard ErrorAlACiMaParameter Estimate
Grade 5Subject Matter
Value Added Effect of AlAciMaSchools for Grade 5 by Content Area for Cohort 1 2004-2005
![Page 27: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/27.jpg)
600
620
640
660
680
700
720
740
760
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore
4th8th10th
MMSD Mean Math Scale Scores for Grades 4, 8, and 10 2000-2005
![Page 28: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/28.jpg)
0
100
200
300
400
500
600
700
800
900
1000
2000 2001 2002 2003 2004 2005
Year
Scal
e Sc
ore
MMSD Math Mean, First and Third quartile, and 5th and 95th Percentile Scores for Grade 8 2000-2005
![Page 29: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/29.jpg)
n = 1761
n = 1726
n = 1729n = 1697
n = 1666n = 1706
-0.1
0
0.1
0.2
0.3
0.4
2000 2001 2002 2003 2004 2005
Year
Effe
ct S
ize*
Normalized Scale Score Effect Size Between MMSD and WI for Math Grade 8 2000-2005
![Page 30: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/30.jpg)
Evaluators Need to be Realistic
Program design will change(27 of 40 evaluators, 68%)
Collecting student data is challengingAccess to school, teachers, and students can be difficultThere is a trade off between collecting data and writing about resultsQualitative and quantitative data collections requires a balance
![Page 31: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/31.jpg)
What Has Worked for MSP Evaluators
Continual Communication with StakeholdersConducting Case StudiesAnalysis of Teacher-Learning Logs a Value to Stakeholders
![Page 32: Models for Evaluating MSP Projects Workshop for State ...programs.ccsso.org/content/pdfs/AnalyzingState... · zPre-Post Test zPre-Post Test with a Control Group zPost Test Only zPost](https://reader034.vdocuments.site/reader034/viewer/2022042621/5f6dcb3f6d844867046e5f21/html5/thumbnails/32.jpg)
Final Thoughts
It is not the direction of the wind, but the set of the sails that guides the boat.