evaluation in michigan ’ s model steve goodman sgoodman@oaisd
DESCRIPTION
Evaluation in Michigan ’ s Model Steve Goodman [email protected] National PBIS Leadership Forum October, 2011. http://miblsi.cenmi.org. Mission Statement. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/1.jpg)
Evaluation in Michigan’s Model
Steve [email protected]
National PBIS Leadership Forum
October, 2011
http://miblsi.cenmi.org
![Page 2: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/2.jpg)
Mission StatementTo develop support systems and sustained implementation of a data-driven, problem solving model in schools to help students become better readers with social skills necessary for success.
![Page 3: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/3.jpg)
Nicole Matthews Data Entry
Nicole Matthews Data Entry
Anna HarmsEvaluation Coordinator
Anna HarmsEvaluation Coordinator
Ed HuthData Analyst
Ed HuthData Analyst
Jennifer RollenhagenMeasurement and
Evaluation Specialist
Jennifer RollenhagenMeasurement and
Evaluation Specialist
Evaluation contributes to Project by:
•Developing and providing resources to enhance local capacity related to measurement and evaluation, consistent with the implementation research. Evaluation supports the competencies and capacity necessary for implementation specialists and local districts to engage in effective data-based decision making as part of an integrated behavior and reading RtI model.
•Reporting on program activities and project outcomes to evaluate and improve effectiveness and efficiencies of the project to ensure value added to consumers and stakeholders. This is accomplished by implementing Plan-Do-Study-Act cycles.
MiBLSi Project Evaluation Team
![Page 4: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/4.jpg)
Several Purposes of MiBLSi Assessments
• Audit
– for “taking stock” of current strengths/weaknesses and action planning
• Formative evaluation
– for improving program while it is in the process of being implemented
• Summative evaluation
– for improvement of future reiterations
![Page 5: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/5.jpg)
Internal EvaluationInternal Evaluation(within the project)(within the project)
![Page 6: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/6.jpg)
Feedback Feedback MiBLSi Value-Added Work System
Capital
Resources
People Materials Information
Work Systems: Providing the RtI practices and the supports for the these practices to take place successfully within schools and districts
Stakeholders/Funders
Stakeholders/FundersInvestments:
•Funding•Visibility•Political support
Returns:•Addressing critical issues (Discipline/Ethnicity)•Addressing program directives (State Performance Plan)
Consumers (schools, districts,
ISDs)
Consumers (schools, districts,
ISDs)
Valued RtI Products/ Services
FinancialEvaluationProfessional Learning
Technical Assistance
![Page 7: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/7.jpg)
Worker
Organizational Level
Worker
Worker
Process Level
Evaluation
Tech. Assistance
Prof. Learning
Finance
Consumers
Stake Holders
Worker
Evaluation At Organizational, Process and Performer (worker) level: At each , measurement takes place at determined interval. This information is compared to established standards and provided as feedback for the system.
![Page 8: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/8.jpg)
Levels of Internal Evaluation
Level Description Tools
Organizational Evaluating impact of the RtI project •Stakeholders
• Addressing critical issues (e.g., Discipline/Ethnicity)
• Addressing program directives (e.g., State Performance Plan)
•Consumers• Valued results (e.g., Improved
quality program, fidelity of implementation, successful student outcomes)
• Annual reports
• Measures of program quality
• Measures of fidelity of implementation
• Measures of student outcomes
• Survey of consumer satisfaction
Process Evaluating operational productivity and efficiency of each unit within organization
Project management tools•Fasttrack, MS Project
•Basecamp
Performer (worker)
Evaluating worker productivity and efficiency
• Job models
![Page 9: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/9.jpg)
Job Model: Measurement and Evaluation Specialist
![Page 10: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/10.jpg)
File Maker Pro Data Base
![Page 11: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/11.jpg)
External EvaluationExternal Evaluation(outside the project)(outside the project)
![Page 12: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/12.jpg)
Building StaffBuilding StaffBuilding StaffBuilding Staff
Building Building Leadership Leadership
TeamTeam
Building Building Leadership Leadership
TeamTeam
LEA District LEA District Leadership Leadership
TeamTeam
LEA District LEA District Leadership Leadership
TeamTeam
Collecting information to evaluate implementation effects and using this information for continuous improvement
•Fidelity of implementation (across schools)•Systems integrity (district-LEA)•Student success (district-wide)
•Fidelity of implementation (state)•Systems integrity (project)•Student success (project-wide)
•Fidelity of implementation (across grades)•Systems integrity (school)•Student success (school-wide)
•Student success/Intervention effectiveness
ISD ISD Leadership Leadership
TeamTeam
ISD ISD Leadership Leadership
TeamTeam
MiBLSi ProjectMiBLSi ProjectMiBLSi ProjectMiBLSi Project
•Fidelity of implementation (across districts)•Systems integrity (district-ISD)•Student success
![Page 13: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/13.jpg)
Assessments
Elementary Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• Benchmarks for Advanced Tiers (BAT)• Dynamic Indicators of Basic Early Literacy
Skills (DIBELS)• Planning and Evaluation Tool (PET) for
Effective Schoolwide Reading Programs• Effective Reading Support Team
Implementation Checklist• Special Education Data Collection Form• Schoolwide Reading Analysis Support Page
Middle/Junior High Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• ORF/MAZE through AIMSWeb• School-Wide Evaluation and Planning
Tool for Middle School Literacy (SWEPT)
• Middle School Reading Team Implementation Checklist
• Special Education Data Collection Form
![Page 14: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/14.jpg)
![Page 15: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/15.jpg)
![Page 16: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/16.jpg)
![Page 17: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/17.jpg)
Building LevelBuilding Level
![Page 18: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/18.jpg)
Assist Teams in Using Data for Decision-making
• First Year– Winter systems review
– Spring Data Review
• Second Year– Fall data review
– Winter data review
– Spring data review
• Third Year– Fall data review
– Winter data review
– Spring data review
![Page 19: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/19.jpg)
Assessment Booklet
• Description of assessments
• Data collection schedule
• Data summary• Data forms and
assessment forms
![Page 20: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/20.jpg)
Team Evaluation of Outcome, Process and Systems Data
![Page 21: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/21.jpg)
Assessment Schedule (for Cohort 7 from MiBLSi website)
![Page 22: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/22.jpg)
Video examples for completing and submitting PBIS assessments
![Page 23: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/23.jpg)
Improving the accuracy and Consistency of Recording Office Discipline Referrals
![Page 24: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/24.jpg)
Developing Fluency with Discipline Referral Categories
Example Exercise 2: Match the example situation below to the correct problem behavior on the discipline categories answer sheet. Write the letter in the column for Exercise 2.
![Page 25: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/25.jpg)
District LevelDistrict Level
![Page 26: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/26.jpg)
Increase 8%
Decrease 14.6%
Focus on Implementing with Fidelityusing Benchmarks of Quality (BoQ)/ODR ’06-’07 and ’07-’08
![Page 27: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/27.jpg)
District Implementation Tracking Form
![Page 28: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/28.jpg)
Leadership-Implementation Support Team Self-Assessment
![Page 29: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/29.jpg)
Lesson Learned• Teams need to be taught how to
analyze and use data
• Emphasis on directing resources to need and removing competing activities
• As we grow, it is even more important to systematic gather data that is accurate and then act on the data for continuous improvement
• More work is needed in developing feedback cycles
![Page 30: Evaluation in Michigan ’ s Model Steve Goodman sgoodman@oaisd](https://reader035.vdocuments.site/reader035/viewer/2022070405/56813fbb550346895daa9b06/html5/thumbnails/30.jpg)
“Even if you’re on the right track, you’ll get run over if you just sit there”
- Will Rogers