tom jenkins, ed.d. educational consultation services, llc wilmington, nc
DESCRIPTION
The Formative Evaluation Component of Your MTIM Data Based Instructional Decision Making from the Building Level to the Individual Student Level. Tom Jenkins, Ed.D. Educational Consultation Services, LLC Wilmington, NC. Formative Evaluation. - PowerPoint PPT PresentationTRANSCRIPT
The Formative Evaluation Component of Your MTIM
Data Based Instructional Decision Making from the Building Level to the Individual Student
Level
Tom Jenkins, Ed.D.
Educational Consultation Services, LLCWilmington, NC
Formative Evaluation
• Collecting data to make determinations about the effectiveness of your MTIM implementation
Collecting data to make determinations about the effectiveness of instruction with students – and everything in between
Managing Complex Change= Change
Confusion=
=
=
=
=
+ + + +Vision Skills Incentives ResourcesAction Plan
+ + + +Skills Incentives Resources Action Plan
+ + + +Vision Incentives ResourcesAction Plan
+ + + +Vision Skills ResourcesAction Plan
+ + + +Vision Skills IncentivesAction Plan
+ + + +Vision Skills Incentives Resources
Adapted from Knoster, T.
Anxiety
Resistance
False Starts
Frustration
5Big Ideas
ciCore Instruction
siSupplemental
Instruction
iiiIntensive
Instruction
Efforts lack focus and priority. There is not a
focus on important priority skills for
improvement.+ +3
Assessments(Screening, diagnostic,
progress)+ + =
BIBig Ideas
iiInitial Instruction
Overall low achievement. Student
learning problems across all subgroup
areas.+ +3
Assessments(Screening, diagnostic,
progress)+ + =
Lack of direction to know what needs to be
improved, which students need
intervention, and whether or not
interventions have been effective.
+ +3
Assessments(Screening, diagnostic,
progress)+ + =
siStrategic
Instruction
Lack of resources due to attempts to provide intensive interventions
for those students whose needs could be met through strategic
interventions.+ +
3Assessments
(Screening, diagnostic, progress)
+ + =
iiiIntensive
Instruction/Intervention
Gap increases between average and "at risk students" Continued low performance for some subgroups.+ +
3Assessments
(Screening, diagnostic, progress)
+ + =
IDMSuccess+ +
3Assessments
(Screening, diagnostic, progress)
+ + =
ciCore Instruction
ciCore Instruction
ciCore Instruction
ciCore Instruction
BIBig Ideas
BIBig Ideas
BIBig Ideas
BIBig Ideas
siSupplemental
Instruction
siSupplemental
Instruction
siSupplemental
Instruction
siSupplemental
Instruction
iiiIntensive
Instruction
iiiIntensive
Instruction
iiiIntensive
Instruction
iiiIntensive
Instruction
Analyzing Your MTIM Implementation
• Building consensus is necessary in three stages– pre-implementation– implementation– maintenance
Analyzing Your MTIM Implementation
• Pre-implementation tasks– Identify the purpose of
implementation and the critical data
• whose performance will improve as a result of implementation?
• what aspect of performance will improve?
• how will you know?
• take some time to think about this
Analyzing Your MTIM Implementation
• Pre-implementation tasks continued– Identify the mechanisms that will cause
student performance to change• enhanced delivery of Core• screening• intervening• progress monitoring
Analyzing Your MTIM Implementation
• Pre-implementation tasks continued– Present plan to stakeholders
• allow for stakeholder input• create forms of communication for stakeholder
groups• don’t forget your school board• disagreement is normal and should not be
overlooked
Analyzing Your MTIM Implementation
• Implementation tasks– Consensus is most
easily gained through success!
– implement where leadership is strong
– assess treatment integrity
– start small– provide support– communicate success
to stakeholders
Analyzing Your MTIM Implementation
• Maintenance tasks– Build RTI/TIM into the school culture
• set up team meetings that are on the calendar• present success stories and current status data at
staff meetings
Analyzing Your MTIM Implementation
• Maintenance tasks continued– Training is required
each and every year• assessment tool
training• intervention training• new staff• survey your staff
activity
Analyzing Your MTIM Implementation
• Maintenance tasks continued– maintain communication with and present
results to stakeholders– show connection between implementation
and state accountability measures
Analyzing Your MTIM Implementation
• Things that reduce consensus and support for RTI implementation– lack of identifiable team or individual who is in
charge– lack of focus on what RTI is about– asking classroom teachers to do everything– not communicating to critical stakeholders– not identifying how or when students move
between tiers
Data For Each Tier - Where Do They Come From?
• Tier 1: Universal Screening, accountability assessments, grades, classroom assessments, referral patterns, discipline referrals
• Tier 2: Universal Screening - Group Level Diagnostics (maybe), systematic progress monitoring, formative assessment large-scale assessment data and classroom assessment
• Tier 3: Universal Screenings, Individual Diagnostics, intensive and systematic progress monitoring, formative assessment, other informal assessments
Building MTIM Infrastructure
• Formative evaluation process
• Informed by data• Highly involved school-based leadership
team (SBLT)• School-based MTIM coach
– Provide Technical Assistance– Interpretation and Use of Data– Facilitates regular data meetings for building
and grade levels
Formative Evaluation Component of Infrastructure
• What we need:– Screening system for
identifying students at risk– Diagnostic assessment
tools for identifying specific needs of students identified by screening
– Systematic, explicit, research based instructional strategies – differentiated instruction
– Progress monitoring plan– Evaluation of whether
instruction is effective
Data For Each Tier – Where Do They Come From?
Addl.Diagnostic
Assessment
InstructionResults
Monitoring
IndividualDiagnostic
IndividualizedIntensive
weekly
Screen All Students
Monthly
Screening
Bench-Mark
Assessment
AnnualTesting
Behavior Academics
None ContinueWithCore
Instruction with differentiation
GradesClassroom
AssessmentsYearly Assessments
StandardProtocol
SmallGroupInterventionBy skill
2 times/month
Step 1Step 2 Step 3 Step 4
Supplemental
1-5%
5-10%
80-90%
Core
Intensive
Tier I - Core/Benchmark
• Universal Screening – Academics: Screen all students, begin in
kindergarten; 3 times per year with appropriate early literacy and math measures
– More intense instruction and monitoring within classroom for students below cut scores
– See worksheet
Cut Score Worksheet• Step One: Put all student scores on
the university screening measure on a histogram type chart.
• Step Two: Calculate typical Growth Rate of specific skills. Three formulas can be used here.
• EOYBM – BOYBM / 36 weeks = GR• Or• EOYBM – MOYBM / 18 weeks = GR• Or• MOYBM – BOYBM / 18 weeks = GR• Step Three: Determine the Targeted
Growth Rate for students. Two formulas can be used here depending on the desired amount of ambitiousness.
GR * 1.5 = TGR
• OrGR * 2.0 = TGR
Cut Score Worksheet• Step Four: Calculate the Growth Goal for
the instructional period. • TGR * NWI (18 or 36) = GG• Step Five: Calculate the Cut Score for
determination of level of instruction. Two formulas can be used here depending on the length of the instructional period used in step four.
MOYBM – GG = CS
• OrEOYBM – GG = CS
• Step Six: Using the Cut Score place a line of demarcation on the histogram created in step one. Any students above the Cut Score should obtain the GG via Core instruction. Any students below the Cut Score may need Strategic instruction to obtain the TGR and GG. Students in need of Intensive instruction should be identified using progress monitoring data during Strategic instruction implementation. Progress monitoring within all three tiers allows for students movement between the tiers during the instructional period.
Cut Score Worksheet Activity
• Knowing that your MOYBM is 40 and your BOYBM is 20 what would be the cut score using a accelerator of 1.5?
• BOYBM = 20• MOYBM = 40• 18 weeks of
instruction/intervention• Accelerator of 1.5
Cut Score Worksheet Activity
• Step Two: MOYBM – BOYBM / 18 = GR40 – 20 / 18 = 1.11
• Step Three: GR * 1.5 = TGR1.11 * 1.5 = 1.67
• Step Four: TGR * NWI = GG1.67 * 18 = 29.97
• Step Five: MOYBM – GG = CS40 – 29.97 = 10.03
• Step Six: All students above the score of 10.03 should be able to meet the expected Growth Goal via Core instruction. All students below the score of 10.03 would probably be initially placed in Strategic instruction to obtain the Targeted Growth Rate and necessary Growth Goal.
05
101520253035
1 3 5 7 9 11
Series1
Universal Screening ResultsFidelity Check: Are you doing the right
thing?
• Assess success of instructional program– Percent of students at or above benchmarks– If necessary, examine curriculum, instruction,
or both
• Identify students below benchmarks– Interventions within general education
classroom– Assess progress and consider need for more
intensive interventions
Decision Rules: What is a “Good” Response to Intervention?
• Positive Response
– Gap is closing
– Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range
– Level of “risk” lowers over time
• Questionable Response
– Rate at which gap is widening slows considerably, but gap is still widening
– Gap stops widening but closure does not occur
• Poor Response
– Gap continues to widen with no change in rate.
Performance
Time
Positive Response to Intervention
Expected Trajectory
Observed Trajectory
Decision Rules: Linking RtI to Intervention Decisions
• Positive
• Continue intervention with current goal
• Continue intervention with goal increased
• Fade intervention to determine if student(s) have acquired functional independence.
Decision Rules: What is a “Questionable” Response to Intervention?
• Positive Response
– Gap is closing
– Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range
• Questionable Response
– Rate at which gap is widening slows considerably, but gap is still widening
– Gap stops widening but closure does not occur
– Level of “risk” remains the same over time
• Poor Response
– Gap continues to widen with no change in rate.
Performance
Time
Questionable Response to Intervention
Expected Trajectory
Observed Trajectory
Decision Rules: Linking RtI to Intervention Decisions
• Questionable
– Was intervention implemented as intended?
• If no - employ strategies to increase implementation integrity
• If yes -
– Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.
Decision Rules: What is a “Poor” Response to Intervention?
• Positive Response
– Gap is closing
– Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range
• Questionable Response
– Rate at which gap is widening slows considerably, but gap is still widening
– Gap stops widening but closure does not occur
• Poor Response
– Gap continues to widen with no change in rate.
– Level of “risk” worsens over time
Performance
Time
Poor Response to Intervention
Expected Trajectory
Observed Trajectory
Decision Rules: Linking RtI to Intervention Decisions
• Poor
– Was intervention implemented as intended?
• If no - employ strategies in increase implementation integrity
• If yes -
– Is intervention aligned with the verified hypothesis? (Intervention Design)
– Are there other hypotheses to consider? (Problem Analysis)
– Was the problem identified correctly? (Problem Identification)
Building Your MTIM
Thoughts, fleshing out your pyramid, and action plan activity
• Is your district/school ready for a universal screening process? Consider……– Do you have a person with the ability to be a
data coach?– Who would be your school leadership
team/data analysis team?– Current screening assessment tools– Staff ability to analyze data and make
instructional decisions– Do you have consensus?
Data For Each Tier – Where Do They Come From?
Addl.Diagnostic
Assessment
InstructionResults
Monitoring
IndividualDiagnostic
IndividualizedIntensive
weekly
Screen All Students
ODRsMonthly
Bx Screening
Bench-Mark
Assessment
AnnualTesting
Behavior Academics
None ContinueWithCore
Instruction with differentiation
GradesClassroom
AssessmentsYearly Assessments
StandardProtocol
SmallGroupInterventionBy skill
2 times/month
Step 1Step 2 Step 3 Step 4
Supplemental
1-5%
5-10%
80-90%
Core
Intensive
Need for CBM Type Assessments
• All reading probes scored corrects per minute
Need for CBM Type Assessments
• Math computations are scored by correct digits per minute
Need for CBM Type Assessments
• Correct Sequences for written expression
• Two words form a sequence, word and punctuation form a sequence.
• Most words and punctuation are used twice
• Three minutes to brainstorm, write, and edit
Tier II Strategic and Tier III Intensive Progress Monitoring
• Progress monitoring is essential for four reasons– There is no guarantee that interventions will be successful, thus
the intervention must be “tested” to evaluate effectiveness– Increased emphasis of specific outcomes for students, data
base must be generated to guide intervention decision making– Pre/post testing has be shown to be unreliable (small amount of
data) and provides too little data to allow for instructional decision making – progress monitoring allows for evaluation of level of performance and rate of learning
– Research has shown that progress monitoring is associated with improved educational outcomes
Research has shown that it works!
• If we use research validated reading practices, monitor students’ progress and make changes to instruction based on what we find, between 95 and 100 percent of children can become proficient readers.
• Torgesen, 2000, Learning Disabilities Research and Practice.
• Individual Differences in Response to Early Intervention in Reading: The Lingering Problem of Treatment Resisters
Tier II Strategic and Tier III Intensive Progress Monitoring
• Essential components that must be in place for successful progress monitoring within each tier– A well-defined behavior– Identification of student’s current level of performance
(baseline)– Instruction/Intervention– Goal– A measurement strategy– Graph– Decision-making plan
Tier II Strategic and Tier III Intensive Progress Monitoring
• Target behaviors that are observable and measurable
• Skill specific assessments
• Focus on enabling skills– Skills that are prerequisite skills for more
complex skills– Deficiencies in enabling skills often adversely
affects performance on global assessments
Tier II Strategic and Tier III Intensive Progress Monitoring
• Enabling skills for reading– Phonemic awareness– Alphabetic understanding– Fluency– Sight words– Comprehension
Tier II Strategic and Tier III Intensive Progress Monitoring
• Enabling skills for math– Number sense– Facts– Computation– Applications– Problem solving
• Enabling skills for written expression– Mechanics– Expression
Tier II Strategic and Tier III Intensive Progress Monitoring
• Enabling skills for behavior– Social skills– Work completion– Compliance– Problem solving skills
Tier II Strategic and Tier III Intensive Progress Monitoring
• Goal setting…………….• Standard against which progress can be compared• Allows for aimline to be established• Possible goals
– Level of behavior that is expected – several ways to establish this
– Most frequently used is accelerated growth rates for academics and percent of time expectations for behavior
Tier II Strategic and Tier III Intensive Progress Monitoring
• To identify an accelerated growth rate– Take the growth rate that is calculated and
multiply it by 1.5 to obtain a slightly ambitious growth rate or
– Take the growth rate that is calculated and multiply it by 2 to obtain a more ambitious growth rate
– Then multiply by the number of weeks of intervention
Tier II Strategic and Tier III Intensive Progress Monitoring
• For behavior research indicates that a 75% level of performance can be used for non threatening behaviors
• For behaviors that are threatening or dangerous a 100% level of performance should be used
Tier II Strategic and Tier III Intensive Progress Monitoring
• Goal calculation activity– Using the data below what would Nicole’s goal be, in
each area, for an intensive intervention plan that was implemented for nine weeks using a 1.5 accelerator?
Skill Nicole’s Baseline
District 3rd Grade BOYBM
District 3rd Grade MOYBM
Sight Word Phrase Fluency
53 81 91
Oral Reading Fluency
83 131 157
Tier II Strategic and Tier III Intensive Progress Monitoring
• Graph provides a visual representation of a large amount of data
• A visual representation of student’s acquisition of skills and allows for easier analysis of progress
• Paper and pencil or electronic based graphing
Charting Activity
• Activity– Plot the baseline data
• Monday – 14• Tuesday – 10• Wednesday – 6• Thursday – 5• Friday – 10
– Plot the median and signify with a heavy dark line
Charting Activity
• Plot the goal at the end of eight weeks– 40
• Indicate the aimline
Charting Activity
• Plot the first two weeks progress monitoring data– Week One
• Tuesday – 16• Thursday – 14
– Week Two• Monday – 10• Wednesday – 18• Friday – 16
• Make an informed decision regarding the effectiveness of the intervention
Charting Activity
• Plot progress monitoring data for next two weeks– Week Three
• Tuesday – 14• Thursday - 10
– Week Four• Monday – 15• Wednesday – 16• Friday – 16
• Make an informed decision regarding the effectiveness of the intervention
Charting Activity
• Phase two of the intervention– Plot the progress monitoring of the next two weeks– Week Five
• Tuesday – 24• Thursday - 26
– Week Six• Monday – 26• Wednesday – 24• Friday – 26
• Make an informed decision regarding the effectiveness of the intervention
Charting Activity
• Phase two of the intervention– Plot the progress monitoring of the next two weeks– Week Seven
• Tuesday – 28• Thursday - 30
– Week Eight• Monday – 32• Wednesday – 36• Friday – 38
• Make an informed decision regarding the effectiveness of the intervention
• Final decision?
Charting Activity Continued
• Trend lines• Line that you draw through a series of data points
that represents the student’s actual rate of progress
• If trend line slope is flatter than aimline slope, then adjust intervention
• If trend line slope is steeper than aimline slope, then adjust goal
• If slopes are the same, make no change
Charting Activity Continued
• Trend line procedures– Draw vertical line in middle of graph, half data
points on one side, half on the other – if odd number of data points, put line through middle point
– Draw a vertical line in first half of data to separate data half and half
– Draw a vertical line in second half of data to separate data half and half
Charting Activity Continued
• Trend line procedures– Draw a horizontal line through median of first
half of data – form an intersection with vertical line
– Draw a horizontal line through median of second half of data – form an intersection with vertical line
– Connect the two intersections– Make your decision, is it the same decision?
Formative Evaluation Case StudySkill Prior To
Tier 3 Intervention
Nicole’s Baselines
District 3rd Grade Fall
Mean
Proficiency WBP
Phoneme ID 32,32,30 41 28 22
Blend ID 3,4,6 24 11 5
Sight Word ID 50,53,56 81 68 58
Words In Passages
80,83,84 131 105 80
Addition 0,0,5 13 9 7
Subtraction 2,1,2 5 1 1
Multiplication 7,5,4 11 4 2
Math Concepts
3,2,1 7 4 2
Formative Evaluation Case StudySkill District 3rd
Grade Winter Mean
Proficiency WBP
Phoneme ID 54 43 36
Blend ID 32 16 9
Sight Word ID 91 77 68
Words In Passages
157 114 94
Addition 14 10 8
Subtraction 6 3 2
Multiplication 21 15 11
Math Concepts
10 6 5
Formative Evaluation Case Study
• Nicole– Progress monitoring data for addition
• Week one 8,3,13• Week two 13,9,8• Week three 11,3,11• Week four 13,12,12
Formative Evaluation Case Study
• Nicole – Progress monitoring data for blends
• Week one 7,8,6• Week two 6,5,5• Week three 4,6,4• Week four 4,4,2• Week five 6,6,7• Week six 5,8,10
Formative Evaluation Case Study
• Nicole – Progress monitoring data for sight words
• Week one 61,63• Week two 64,49• Week three 51,53• Week four 50,52• Week five 56,48• Week six 44,50
Formative Evaluation Case Study
• Nicole– Based on your progress monitoring data,
complete the Analysis of Intervention Plan• Identify median score of last three progress
monitoring data points for each skill• Compare current level of performance to
previously established goals• Make a decision about the results of the
intervention plan and the next step• Is this a student that needs to be considered for
specially designed instruction?
0
20
40
60
80
100BenchmarkGoal
Egbert
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Student in January of First Grade
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Benchmark=24
Baseline=11
0
20
40
60
80
100 Line 1
Line 2
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchm=24
Baseline=11
Benchmark Line
Student goal line
0
20
40
60
80
100Bench
Goal
Student
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Monitor Students’s Progress Relative to Goal
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchm=24
Baseline=11
Benchmark Line
Student goal line
0
20
40
60
80
100Bench
Goal
student
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Formative Evaluation: Change Intervention
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchm24
Baseline=11
Benchmark
Student goal line
ChangeIntervention
0
20
40
60
80
100Bench
Goal
student
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Continue Intervention and Monitor Progress
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bnch=24
Baseline=11
Benchmark
Student goal line
ChangeIntervention
0
20
40
60
80
100Class
Bench
studentLine 4
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Continue Intervention and Monitor Progress
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchmrk=24
Baseline=11
Benchmark
Student goal line
ChangeIntervention
Fade Tier ?
0
20
40
60
80
100Class
Goal
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bnchmrk=24
Baseline=11
Benchmark Line
Student goal line
0
20
40
60
80
100ClassGoalstudent
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Monitor Student Progress Relative to Goal
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchmrk=24
Baseline=11
Benchmark line
Student goal line
0
20
40
60
80
100ClassGoalstudent
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Change Student Intervention
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Bchmk=24
Baseline=11
Benchmark line
Student goal line
ChangeIntervention
0
20
40
60
80
100ClassGoalstudent
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Implement Revised Intervention and Continue to Monitor Progress
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Benchmark Line
Student goal line
ChangeIntervention
0
20
40
60
80
100ClassGoalstudent
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Implement Second Intervention Revision
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Benchmark
student goal line
ChangeIntervention
0
20
40
60
80
100ClassGoalstudent
Weeks
Wo
rds
Co
rre
ct
Per
Min
ute
Gap Not Closing: Consider Eligibility and More Intensive Interventions
0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
Benchmark
ChangeIntervention
ClassWCM=54
StudentWCM=32
Tier II Strategic and Tier III Intensive Progress Monitoring
• If the decision is to adjust an intervention, small changes or refinements are recommended before major changes
• However, changes should be substantial enough that it has a possibility to result in improved student performance
• If making an adjustment, do not make two at the same time. It may result in the team being unable to determine what caused increased student performance
• Consider intensity vs methodology
Building Your MTIM
Thoughts, fleshing out your pyramid, and action plan activity
• Is your district/school ready for implementation of a multi-tiered assessment model? Consider……– Current diagnostic and formative assessment
tools– Staff ability to analyze data and make
instructional decisions– Do you have consensus?– Where are the gaps in your pyramid?
Questions?
• Dr. Tom Jenkins, Director• Educational Consultation Services, LLC• Wilmington, NC• (910) 367-7209• [email protected]• www.educationalconsultationservices.com
Thanks
• Dan Reschly
• Dave Tilly
• George Batsche
• Ed Shapiro
• Tracy Hall