september 23 , 2013 12:30 pm – 1:45 pm
DESCRIPTION
S.B. 10-191 Implementation: Implementing the Colorado Model System for Measures of Student Learning. September 23 , 2013 12:30 pm – 1:45 pm. Before we begin…. - PowerPoint PPT PresentationTRANSCRIPT
S.B. 10-191 Implementation:
Implementing the Colorado Model System
for Measures of Student
Learning September 23, 2013 12:30 pm – 1:45 pm
Before we begin…
Find your work group – it ’s important where you sit. Try to sit with members of your school or district.
High Knowledge
Low Knowledge
NextHigh ComfortLow Comfort
On the wall you will see a chart that will be used to capture everyone’s level of comfort and knowledge with Measures of Student Learning. Place a dot on the chart the best represents your level of comfort and knowledge currently .
VisionAll students in Colorado will become educated and
productive citizens capable of succeeding in a globally competitive workforce.
MissionThe mission of CDE is to shape, support, and safeguard a statewide education system that prepares all students for
success in a globally competitive world.
Together We Can
Successful studentsPrepare students to thrive in their education and in a globally competitive workforce. Ensure every student is on track to graduate postsecondary and workforce ready. Increase achievement for all students and close achievement gaps. Ensure students graduate ready for success in postsecondary education and the workforce. Increase national and international competitiveness for all students.
Great teachers and leadersEnsure effective educators for every student and effective leaders for every school and district. Increase and support the effectiveness of all educators. Optimize the preparation, retention, and effectiveness of new educators. Eliminate the educator equity gap.
Outstanding schools and districtsBuild the capacity of schools and districts to meet the needs of Colorado students and their families. Increase school and district performance. Turnaround the state’s lowest performing districts and schools. Foster innovation and expand access to a rich array of high quality school choices for students.
Best education system in the nationBuild the best education system in the nation. Lead the nation in policy, innovation, and positive outcomes for students. Operate with excellence, efficiency, and effectiveness to become the best SEA in the nation. Attract and retain outstanding talent to CDE.
Goals
Students
Educators
Schools/ Districts
State
Driving Questions
What do we want students, educators, schools, and districts to know and be able to do?
How will we know if expectations are met?
How will we respond when help is needed and to support continued growth?
Colorado Academic Standards Assessments
• RTI• PBIS• Targeted interventions• IEPs
Educator Quality Standards
Educator evaluations
• Induction• Mentoring• Professional development plans• Remediation plans
Performance Indicators
School and district performance frameworks
• Unified planning• Priority• Turnaround
Students
Educators
Schools/Districts
When we intenti onally integrate… Coherent and rigorous academic standards
Innovative and engaging learning options
Supported and effective educators
Aligned and meaningful assessments
Statewide and district accountability
Expanding Student Learning
We can personalize learning and ignite the potential of every student.
Introductions
How many of you have attended a CDE training or a training in your
school/district on Measures of
Student Learning?
Welcome and Purpose IntroductionsMeasures of Student Learning
Steps 1-5 Create a template/pie chart
Feedback and critical questioning Decision Framework
Work Session and Next Steps Closing – Feedback and Reflection
Agenda Measures of Student Learning
Training Objectives: Desired Outcomes
Why am I here?To understand the shift from student
growth to measures of student learning To gain a clear understanding of the 5
steps for including measures of student learning in your evaluation system
To determine next steps for your school, district or BOCES
Meeting Norms: How We Will Play Together
How will we work together… Honor everyone’s ti me Listen to learn and apply to your context Balance parti cipati on and share airspace Paraphrase, probe, pause Suspend judgment Commit to practi ce and follow-through Avoid technological distracti ons Have a sense of humor
Measures of Student Learning
STATE COUNCIL FOR EDUCATOR EFFECTIVENESS
Framework for System to Evaluate Teachers
Definition of Teacher Effectiveness
I. Know Content
50% Professional Practice Standards 50% Student Growth MeasuresWeighting: How Much Does
Each Standard Count Towards Overall Performance?
Observations of Other Measures Teaching Aligned with
CDE Guidelines
State Other Assessments Other MeasuresSummative for Non-tested Aligned with Assessments Areas CDE Guidelines
Match of test to teaching assignments
Weighting:Scoring Framework: How Do Measures of Quality Standards
Result in a Determination of Individual Performance?
Performance RatingsIneffective Partially Effective Effective Highly Effective
Quality StandardsII. Establish
EnvironmentIII. Facilitate
LearningIV. Reflect on
PracticeV. Demonstrate
LeadershipVI. Student
Growth
Appeals Process
From Student Academic Growthto
Measures of Student Learning
Measures of Student Learning Using multiple measures to
determine student learning over time.
Colorado Growth Model (CGM)
establishes technical
measure of “growth.”
BrainstormHow do you measure student learning in your
district/school?
Measures of Student Learning
Approaches for Selecting and Using Multiple Measures in Educator
Evaluation
Step 1: Review of the assessment requirements
Step 3: Select and weight multiple measures of student learning
Step 2: Determine student learning measures in your district
Step 4: Set student learning outcome targets and scales
Step 5: Combine the results to get a single student outcomes rating
Decision Framework
Teacher Evaluations
50% Student Academic Growth
VI. Responsibility for student academic growth
Evaluated using the following:(1) a measure of individually-attributed growth, (2) a measure of collectively-attributed growth; (3) when available, statewide summative assessment results; and (4) for subjects with statewide summative assessment results available in two consecutive grades, results from the Colorado Growth Model.
Refers to outcomes on a measure that are
attributed to an individual licensed person,
e.g. DRA2 outcomes for a 1st Grade Teacher’s
students
Refers to outcomes on a measure attributed to two or more licensed
personnel, e.g. 10th gr. Math TCAP –
All Secondary math teachers in school
Table Talk:At your table, discuss if you want more
collective or individual attribution in your evaluation system. Consider the advantages and unintended
consequences of both collective and individual. Be prepared to share your thinking.
Step 1: Collective vs. Individual Attribution
Approaches for Selecting and Using Multiple Measures in Educator
Evaluation
Step 1: Review of the assessment requirements
Step 3: Select and weight multiple measures of student learning
Step 2: Determine student learning measures in your district
Step 4: Set student learning outcome targets and scales
Step 5: Combine the results to get a single student outcomes rating
Decision Framework
Determine how student learning is currently measured in your districtConduct an assessment inventory to identify
what is currently being used to measure student learning
Identify where gaps exist
Step 2: Measures of Student Learning
A Step-By-Step Guide for Selecting and Using Student Learning
Outcomes
Step 1: Review of the assessment requirements
Step 3: Select and weight multiple measures of student learning
Step 2: Determine student learning measures in your district
Step 4: Set student learning outcome targets and scales
Step 5: Combine the results to get a single student outcomes rating
Decision Framework
What do we value? How might we categorize our teachers?
How will our decisions reflect our values?
Categorizing Teachers
No Categorization
Categorization
Group Decisions
Individual Decisions
Example:
Categorizing Teachers
No Summative/CGMAll Other Teachers
Colorado Growth Model (CGM)
Reading, Writing, Math (Gr. 4-10)
2.
State SummativeReading, Writing, Math
(Gr. 3)
Science (Gr. 5, 8, 10)
Social Studies (Gr. 4, 7, HS)
ACT (HS)WIDA ACCESS (ELL)
What are the pros and cons of creating groups of like teachers based on the types of measures that must be included in their body of evidence?
Based on what is expected in law, what categories of teachers are possible?
Select and weight multiple measures of student learning to be included in educator evaluations.What can we do as a district to build comparable
bodies of evidence in like groups of teachers?Let’s try it!
Measures of Student Learning Tool
Step 3: Measures of Student Learning
Create a template using the Measures of Student Learning tool. Questi ons to consider:
What assessments must we include?How much influence will they have on the evaluation?
Create a pie chart for a category of teacher or individual teacher that shows your thinking.
Include a rationale.
Creating Specific Weighting Templates
Pie Chart Feedback
Round RobinEach group should select a spokesperson to
share your thinking and rationale for your choices.
Audience should consider feedback on:The collective vs. the individual attributionTypes of assessmentsWeights
Debrief/Reflection
Did the feedback help shape any of your thinking?
Any aha’s?
What might you reconsider/change if anything?
Districts, BOCES and schools may decide aft er completi ng the Assessment Inventory that a locally created assessment would likely be included as a measure in evaluati on.
Districts and BOCES can use the Assessment Review Tool to show they are taking steps to ensure that the locally created measurements they select are fair, valid, and reliable.
Using Local Assessments in Evaluation
Assessment Review Tool
Criteria used in this tool:Alignment Scoring Fair and UnbiasedOpportunities to Learn
Assessment Review Tool
Assessment Support
Content Collaborati ves P-12 educators from around the state gathered to identi fy and
create a high-quality assessment resource bank, which is aligned to the new Colorado Academic Standards and may be used in the context of Educator Eff ecti veness evaluati ons.
The Content Collaborati ves, CDE, along with state and nati onal experts, will establish examples of student learning measures within each K – 12 content area including:
Cohort I
Dance Drama & Theatre Arts Music Reading, Writing and Communicating
Social Studies Visual Arts
Cohort IIPhysical Education Science World Languages Comprehensive Health
Mathematics CTE
Decision Points How will our district create templates for different
teacher roles and the types of assessments that exist?Who will participate in the process? What do we need to know in order to make good
decisions?What resources (internal and external) do we
have/need?How will we know if we have made good decisions?
Categorizing, Weighting, and Selecting Measures Reflection
Step 1: Review of the assessment requirements
Step 3: Select and weight multiple measures of student learning
Step 2: Determine student learning measures in your district
Step 4: Set student learning outcome targets and scales
Step 5: Combine the results to get a single student outcomes rating
Decision Framework
Approaches for Selecting and Using Multiple Measures in Educator
Evaluation
Use the Student Learning Objective Process to set outcome targets and scales based on data from the selected measures.
Step 4: Measures of Student Learning
What is the Student Learning Objective Process? Enables educators to utilize academic standards
to establish learning outcomes for students, monitor progress and evaluate the degree to which students achieve those outcomes
In educator evaluation systems, this process serves as a: Method of measuring student growth, progress or
mastery of the standards
Student Learning Objective Process
The essential steps of the Student Learning Objective Process include: Understanding the Colorado Academic Standards in order to
select learning outcomes Collecting baseline information to inform target and scale
setti ng Assessing quality, attainment level and rigor of student learning
targets and scales Monitoring student learning (formative practice) Determining attainment of student learning targets and scales Reflecting and refining the Student Learning Objective Process
Student Learning Objective Process
Student Learning Objective Process
Turn and Talk:• Why is it important to start the Student Learning
Objective process with the Colorado Academic Standards?
• How does this first step support the work done in professional practice?
• What tool is in place to support you with part of this process?
THE PROCESS OF: Understanding the Colorado Academic Standards
THE PROCESS OF: Selecting Learning Outcomes for Target
Setting
THE PROCESS OF: Procuring or Developing Assessments Using
Quality Criteria
Student Learning Objective Process
THE PROCESS OF: Collecting Baseline Information
THE PROCESS OF: Setting Student Learning Targets
THE PROCESS OF: Setting Appropriate Scales for Measurement
THE PROCESS OF: Understanding the Colorado Academic Standards
THE PROCESS OF: Selecting Learning Outcomes for Target Setting
THE PROCESS OF: Procuring or Developing Assessments Using Quality Criteria
Possible Student Learning Targets and Scales
Examples of Student Learning Targets and Scales:16 of my 21 students will increase their reading
proficiency by 1-2 grade levels. 9 more students will receive + at the end of the
year. The median score on my end of course
assessment last year was 73%. At the end of the course for this year, the median score will be at least 78%.
Student Learning Objective Process:
Strategies for Target Setting Possible strategies to consider when determining how
students will be captured when target setti ng: Individual – Teacher sets individual goals for students and then
determines how many students met their individual goals Whole Class – Teacher sets a goal that would be applied to all
students in a class Subgroup of Students – Teacher sets a goal for a group(s) of
students with similar baseline data levels Case Load – Teacher sets a goal for the specific students he or she
supports
Some of these strategies may work bett er with some teaching assignments. Can you think of which teaching assignments might
work best with which strategy based on your local context?
Student Learning Objective Process:
Strategies for Target Setting Possible strategies to consider when setti ng learning targets for
students: All targets are indicators of growth using a variety of data sources: Proficiency data Targets – students meeting grade level
expectations Examples: a % of students will pass the end-of-course exam; a % of
students will score a 3 or bett er on the AP exam Growth data Targets – students growing over the course of
instruction Examples: a % of students will progress one fi tness level; a % of
students will make projected growth or bett er on the MAP assessment Averaging data Targets – students’ average score on an
assessment Example: Students will answer, on average, 80% of the questions
correctly on the end-of-course exam
Possible Student Learning Targets
What are the targets in these examples?16 of my 21 students will increase their reading
proficiency by 1-2 grade levels. 9 more students will receive + at the end of the
year. The median score on my end of course
assessment last year was 73%. At the end of the course for this year, the median score will be at least 78%.
Student Learning Objective Process:
Strategies for Scale SettingTeachers set ranges (how many students are
expected to meet the set target) across 4 rating levels: above expected, expected, less than expected, and much less than expected
Teachers collaborati vely set ranges with their evaluator Ranges will vary from year to year and possibly class to
class as baseline data will vary due to student beginning points
Collaboratively setti ng ranges will allow teachers and evaluators to calibrate about expected student learning outcomes
You can use the Measures of Student Learning Tool to record the targets and scales you have established.
When Colorado Growth Model data is available the scale indicating expected student learning outcomes has been established for you.
Student Learning Objective Process:
Strategies for Scale Setting
Possible Student Learning Scales
What are the scales in these examples?16 of my 21 students will increase their reading
proficiency by 1-2 grade levels. 9 more students will receive + at the end of the year. The median score on my end of course assessment
last year was 73%. At the end of the course for this year, the median score will be at least 78%.
Student Learning Objective Process: Target Setting
PracticeWith a partner, use the Setti ng Student Learning
Targets and Scales worksheet to analyze one set of data.Portfolio/Performance OutcomesBaseline Reading LevelsEnd of Course Summative Exam ResultsRubric Outcomes
Practice setti ng a student learning outcome target for expected outcomes.
Student Learning Objective Process:
Scale Setting Practice Think about what your expected outcomes are
(targets). Establish the ranges for each of the categories
below.
How did you decide on the ranges? Are your expectations rigorous, yet att ainable?
ALL 4 sets of dataPerformance Assessment/Portfolio Outcomes
Reading Levels (beginning of the year)
Cumulative exam
Rubric Results
Student Learning Objective Process:
Target and Scale Setting
66 89 68 75 74 80 94 99 60 55 71 73 95 96 84 82 79 81 98 74 86
6 4 1 5 5 2 6 2 5 2 1 6 3 3 5 3 1 2 6 4 4
18 24 24 30 38 40 40 50 50 50 50 50 60 60 60 60 70 70 70 80 80
+ - + - + - + - - + + -
Student Learning Objective Process
THE PROCESS OF: Assessing Quality, Attainment Level and Rigor of Student Learning Targets and Scales
THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning
Targets THE PROCESS OF: Setting Appropriate Scales for
Measurement
THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes
for Target Setting THE PROCESS OF: Procuring or Developing
Assessments Using Quality Criteria
Student Learning Objective Process
Determine Target and Scale Quality Will your district establish quality criteria for
the use of Student Learning Targets and Scales?Criteria for establishing quality targets and
scales might include:Approval processesCollaboratively establishedLevel of attainmentRigor Based on previous dataMeet or exceed past student performance
Student Learning Objective Process
THE PROCESS OF: Monitoring Student Learning (Formative Practice)
THE PROCESS OF: Assessing Quality, Attainment Level and Rigor of Student Learning Targets and Scales
THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning Targets THE PROCESS OF: Setting Appropriate Scales for Measurement
THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes for Target Setting THE PROCESS OF: Procuring or Developing Assessments Using Quality
Criteria
Student Learning Objective Process
THE PROCESS OF: Determining Attainment of Student Learning Targets and Scales
THE PROCESS OF: Monitoring Student Learning (Formative Practice)
THE PROCESS OF: Assessing Quality, Attainment Level and Rigor of Student Learning Targets and Scales
THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning Targets THE PROCESS OF: Setting Appropriate Scales for Measurement
THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes for Target Setting THE PROCESS OF: Procuring or Developing Assessments Using Quality
Criteria
Performance Assessment/Portfolio Outcomes
Reading Levels (beginning of the year)
End of year cumulative exam
Rubric results
Student Learning Targets:Reflection on Pre and Post
Data
75 90 76 89 80 99 90 91 80 70 82 79 94 95 98 95 66 87 98 85 71
5 5 3 6 6 4 3 5 6 6 4 5 4 4 6 5 3 6 5 6 5
30 40 38 38 50 50 50 60 50 60 70 70 60 70 70 80 80 70 80 80 80
+ + - + + + + ++
- + +
Student Learning Objective Process
THE PROCESS OF: Reflecting and Refining Student Learning Objective Process
THE PROCESS OF: Determining Attainment of Student Learning Targets and Scales
THE PROCESS OF: Monitoring Student Learning (Formative Practice)
THE PROCESS OF: Assessing Quality, Attainment Level and Rigor of Student Learning Targets and Scales
THE PROCESS OF: Collecting Baseline Information THE PROCESS OF: Setting Student Learning Targets THE PROCESS OF: Setting Appropriate Scales for Measurement
THE PROCESS OF: Understanding the Colorado Academic Standards THE PROCESS OF: Selecting Learning Outcomes for Target Setting THE PROCESS OF: Procuring or Developing Assessments Using Quality
Criteria
Considerations when including the Student Learning Objective Process in evaluation:Assessment literacy of educators creating targetsData literacy of educators Time and effort required of teacher and evaluator Focus of school or district (all students and/or subgroups of
students) Available information or data in school or district Ease or readiness level of teachers and evaluators Possible approval process of strategies selected Number of targets wanted as a district/building
Considerations when Setting Student Learning Targets
ReflectionWhat are the pros and cons of using the Student
Learning Objective Process?What are teachers already doing to support process?
Are there groups of teachers that already do something very similar to creating Student Learning Targets and Scales?
What additional ideas do you have for making the Student Learning Objective Process more manageable?
Student Learning Objective Process
Step 1: Review of the assessment requirements
Step 3: Select and weight multiple measures of student learning
Step 2: Determine student learning measures in your district
Step 4: Set student learning outcome targets and scales
Step 5: Combine the results to get a single student learning outcomes rating
Decision Framework
Approaches for Selecting and Using Multiple Measures in Educator
Evaluation
Combine the results of multiple measures to get a single measures of student learning rating.
Step 5: Measures of Student Learning
Draft Example
Combining Outcomes
Putti ng It All Together The Decision Framework helps districts
aggregate the 50% Professional Practices and the 50% Measures of Student Learning to reach a final rating of one of the following: Highly EffectiveEffectivePartially Effective Ineffective
Decision Framework
Not Evident
Partially Proficient
ProficientAccom-plished
Exem-plary
1
1
2
2
4
0
20.0% 108 * 10 / 24 = 45.03
4
2
1
1
1
20.0% 108 * 12 / 24 = 54.04
3
2
1
2
2
0
1
30.0% 162 * 15 / 32 = 75.9
1
1
2
15.0% 81 * 4 / 12 = 27.0
2
1
1
4
15.0% 81 * 8 / 16 = 40.5
100% 540 242.4Pts earned =
Weights by std
Points Possible
* earned / possible =
Points Earned
f. Creates a well managed learning environment
Overall Rating for Quality Standard II
Rating
Proficient
Proficient
a. Provides aligned instruction
b. Demonstrates knowledge of literacy development
c. Demonstrates knowledge of mathematics (All)
d. Demonstrates knowledge of content
e. Develops interconnected lessons
f. Makes instruction and content relevant to students
Overall Rating for Quality Standard I
II. Learning Environment
a. Fosters a predictable, caring learning environment
b. Demonstrates a commitment and respect for diversity
c. Engages students
III. Instruction
a. Demonstrates knowledge of current developmental science
b. Plans and delivers data driven instruction
c. Demonstrates knowledge of effective instructional practices
d. Integrates and utilizes technology
e. Establishes high expectations for students
f. Provides opportunities to develop leadership qualities
g. Communicates to students effectively (objectives)
h. Uses appropriate methods to assess
Overall Rating for Quality Standard III
d. Adapts teaching to meet individual needs
e. Works and communicates with families
Proficient
V. Leadership
Partially Proficient
ProficientOverall Rating
a. Demonstrates leadershipb. Contributes to the teaching professionc. Advocates for schools and studentsd. Demonstrates high ethical standards
Overall Rating for Quality Standard V
IV. Reflection
a. Analyzes student data and applies to instruction
b. Links professional growth to professional goals
c. Is able to respond to a complex, dynamic environment
Overall Rating for Quality Standard IV
Proficient
Quality Standard
Element
I. Content and
Pedagogical Knowledge
Prof
essi
onal
Pra
ctice
s Re
port
How to combine professional practice and measures of student learning ratings
Measures of Student Learning
Prof
essi
onal
Pra
ctice
s
540
Exemplary(433 to 540 pts)
Accomplished(325 to 432 pts)
Proficient(217 to 324 pts)
Partially Proficient(109to 216pts)
Basic(0 to 108pts)
Much Lower Than Expected Growth
(0 to 134 pts)
Lower Than Expected Growth(135 to 269 pts)
Expected Growth(270 to 404 pts)
Higher Than Expected Growth(405 to 540 pts)
Highly Effective
730-1080Effective
487-729Partially Effective
244-486Ineffective0-243
0
0
540
2. Annual
Orientation
3.Self-
Assessment
4.Review of
Annual Goals and
Performance Plan
5.Mid-Year Review
6.Evaluator
Assessment
7.End-of-Year
Review
8.Final Ratings
9.Goal-Setting
and Performance
Planning
1.Training
1.Training
Evaluation CyclePrincipal/AssistantPrincipals and Teachers
May 15Within the first two weeks of
school.
End of September.
Prior to the beginning of
Spring Semester
Train: Prior to the beginning of
School.Orient: Within
the first week of School.
End of MayMid-JuneEnd of June
Review and finalize compiled results
from measures of student learning
Include any new district or school
decisions for measuring student
learning
Confirm measures used to determine student learning– Finalize baseline
data
Review available data from measures
to determine if students are on
track
Consider decision points, priorities and what you value.
Identify next steps for your Performance Evaluation Committ ee.
What are your next steps?
Let’s Get to Work!
CDE Educator Eff ecti veness website: htt p://www.cde.state.co.us/EducatorEff ecti veness/ FAQ’s, Fact Sheets Policy and State Model Evaluation System Info Six Steps guidance document
http://www.cde.state.co.us/EducatorEffectiveness/StudentGrowthGuide.asp
CDE Educator Eff ecti veness e-newslett erhttp://www.cde.state.co.us/scripts/communications/EE
Newsletter.aspStay informed by signing up for this monthly newsletter
Where are you now?
High Knowledge
Low Knowledge
Next
High ComfortLow Comfort
On the wall you will see a chart that will be used to capture everyone’s level of comfort and knowledge with Measures of
Student Learning. Place a dot on the chart the best represents your level of comfort and knowledge now that we are near the
end of the training.
Feedback
Please take a few minutes to complete your feedback form.
Contact Us
EE Leadership Katy Anthes: Executive Director
[email protected] Toby King: Director
[email protected] Jean Williams: Rubric Evaluation
Specialist [email protected]
Colorado Legacy Foundation Mike Gradoz: Director
Communications Amy Skinner
[email protected] Katie Lams:
Britt Wilkenfeld: Data Fellow [email protected]
Tricia Majors: Project Mgr. [email protected]
Implementati on Support and Development
Courtney Cabrera [email protected]
Sed Keller [email protected]
Dawn Pare [email protected]
Bob Snead [email protected]
Chris Vance [email protected]
Contact Us
Colorado Department of Higher Education Jennifer Arzberger