equip rubric & effective ccss feedback training session: math

EQuIP Rubric & Effective CCSS Feedback Training Session: Math

2

Getting to Know You

Meet and greet your table mates:

1.Name?

2.Position/school/teaching assignment?

3.What system or structure does your school have for evaluating and sharing lessons/feedback on lessons?

EQuIP = Educators Evaluating the Quality of Instructional Products

EQuIP is an initiative of Achieve’s American Diploma Project (ADP) Network designed to identify high-quality materials aligned to the Common Core State Standards (CCSS). The outcome of that effort was the development of a quality review process designed to determine the quality and alignment of instructional lessons and units to the CCSS.

The objectives are two-fold:

•Increase the supply of high quality lessons and units aligned to the CSSS that are available to elementary, middle, and high school teachers as soon as possible; and,

•Build the capacity of educators to evaluate and improve the quality of instructional materials for use in their classrooms and schools.

Overview and Objectives of EQuIP

EQuIP Quality Review ProcessThe EQuIP quality review process is a collegial process that centers on the use of criteria-based rubrics for English language arts (ELA)/literacy and mathematics. The criteria are organized into four dimensions:

The Four Dimensions1.Alignment to the depth of the CCSS2.Key shifts in the CCSS3.Instructional support4.AssessmentAs educators examine instructional materials against the criteria in each dimension, they are able to use common standards for quality and generate evidence-based commentary and ratings on the quality and alignment of materials.

EQuIP Quality Review: Process & Dimensions

4

5

The Rubric Organization

Criteria that define the rubric are organized to describe quality in four dimensions.

6

The Rubric Design

The EQuIP rubrics are designed to evaluate:• Lessons that include instructional activities and assessments

aligned to the CCSS that may extend over a few class periods or days

• Units that include integrated and focused lessons aligned to the CCSS that extend over a longer period of time

• The rubric is NOT designed to evaluate a single task or activity• The rubrics do not require a specific template for lesson or unit

design

Use the EQuIP quality review process to determine the quality and alignment of lessons and units to the Common Core State Standards (CCSS) in Mathematics

During this session, we will:•Develop our abilities to use EQuIP criteria to provide observations about CCSS-aligned instructional materials and make suggestions for improvement•Develop a common understanding of the EQuIP quality review process•Develop a common understanding of the rating scale and descriptors for the four rubric dimensions and the rating categories and descriptors for overall ratings•Develop our abilities to use EQuIP criteria, rating scales and rating descriptors to accurately rate instructional materials

Session Goals

7

1. CCSS: Before beginning a review, all members of a review team are familiar with the CCSS.

2. Inquiry: Review processes emphasize inquiry rather than advocacy and are organized in steps around a set of guiding questions.

3. Respect & Commitment: Each member of a review team is respected as a valued colleague and contributor who makes a commitment to the EQuIP process.

4. Criteria & Evidence: All observations, judgments, discussions and recommendations are criterion and evidence based.

5. Constructive: Lessons/units to be reviewed are seen as “works in progress.” Reviewers are respectful of contributors’ work and make constructive observations and suggestions based on evidence from the work.

6. Individual to Collective: Each member of a review team independently records his/her observations prior to discussion. Discussions focus on understanding all reviewers’ interpretations of the criteria and the evidence they have found.

7. Understanding & Agreement: The goal of the process is to compare and eventually calibrate judgments to move toward agreement about quality with respect to the CCSS.

EQuIP Quality Review: Principles & Agreements

8

9

• Go to www.achieve.org/equip• Download EQuIP Rubric Feedback form for

your content area• If you are using a Mac you will need the

most recent version of Adobe Reader• If you are using a tablet you will need the

free Adobe Reader App

Getting Started

http://www.achieve.org/equip

Using the Electronic Quality Review Rubric PDF Form

10

Start by identifying yourself and the lesson/unit.

Using the Quality Review Rubric PDF Form

For each dimension:

•Select the checkbox for each criterion for which clear and substantial evidence is found.

•Make observations and suggestions related to criteria and evidence.

•Determine a rating for each dimension based on checked criteria and observations.

11

Giving Feedback

Writing effective feedback is vital to the EQuIP Quality Review Process. Below are the four qualities of effective feedback. •Criterion-based: Written comments are based on the criteria used for review in each dimension. No extraneous or personal comments are included. •Evidence Cited: Written comments suggest that the reviewer looked for evidence in the lesson or unit that address each criterion of a given dimension. Examples are provided that cite where and how the criteria are met or not met. •Improvement Suggested: When improvements are identified to meet criteria or strengthen the lesson or unit, specific information is provided about how and where such improvement should be added to the material.•Clear Communication: Written comment are constructed in a manner keeping with basic grammar, spelling, sentence structure and conventions.

Feedback Example 1: Mathematics

In the introduction the lesson/unit lists three standards (5.NF.3, 5.NF.4 and 5.MD.1) as targets and directly hits two of the three. The measurement standard is partially addressed but not completely. While there are some fractional units used in the lesson, conversion is not clearly enough addressed. More emphasis on unit conversion is suggested if 5.MD.1 is to be a target.

All eight of the Math Practices are listed in the introduction as targets for this lesson/unit. Only those that are CENTRAL to the lesson should be cited with any others either removed or mentioned only as supporting standards.This lesson/unit is correctly balanced with both procedural and conceptual understanding, including both practicing the procedures and interpreting of fractions as division of whole numbers. Solving word problems, including some involving area and fractional units, in this lesson/unit provides evidence of conceptual understanding.

Is this feedback criterion-based?

Was evidence cited?

Was there an improvement suggested?

Is there clear communication?


Criterion-based: Yes

Evidence Cited: Partially

Improvement suggested: Partially

Clarity Provided: Yes•This is effective feedback. The reviewer mentions that two standards are well-treated but one needs more emphasis. •The reviewer cites evidence of word problems that support conceptual understanding but could have been more specific about the how to better align to the measurement standard.•The improvement, “ more emphasis on conversion is suggested” could be more effective if it suggested at least one of those possibilities. In addition the suggestion related to the Math Practices could be more effective if specific examples were provided as to which MP’s should be deleted or identified as supporting those that are central.•The written comments are constructed in a manner in keeping with basic grammar, spelling, sentence structure and conventions. They communicate clearly.


The lesson/unit targets three standards, which are highlighted in the introduction. The first activity targets the first standard listed. The second, although an engaging activity, is not aligned to any of the standards listed. Most of the suggested activities are not well-suited to address the standards and I would not use them in my classroom. My students would not be able to do the third activity at all. Some of the activities could be fun for students and might be helpful with some adjustments.

Is this feedback criteria-based?Was evidence cited?Was there an improvement suggested?Is there clear communication?


Criterion-based: NoEvidence Cited: No Improvement suggested: PartialClarity Provided: Yes•The language of the criteria is not effectively used. Comments about the reviewer’s students is inappropriate and not helpful.•Other than the “first activity targets the first standard,” there is little specific evidence cited in this feedback.•The reviewer suggests that “some of the activities could be… helpful with some adjustments.” This hints at a suggestion for improvement but there are no specifics as to what type of “adjustments” are needed.•The feedback is clear and the written comments are constructed in a manner in keeping with basic grammar, spelling, sentence structure and conventions.

Quality Review Steps

Step 1: Review Materials

Step 2: Apply Criteria in Dimension I: Alignment to the Depth of the CCSS

Step 3: Apply Criteria in Dimensions II–IV

Step 4: Apply an Overall Rating and Provide Summary Comments

Step 5: Compare Overall Ratings and Determine Next Steps

17

18

Criteria for Dimension I: Alignment to the Depth of the CCSS

The lesson/unit aligns with the letter and spirit of the CCSS:

Targets a set of grade-level CCSS mathematics standard(s) to the full depth of the standards for teaching and learning

Standards for Mathematical Practice that are central to the lesson are identified, handled in a grade-appropriate way and well connected to the content being addressed

Presents a balance of mathematical procedures and deeper conceptual understanding inherent in the CCSS

19

The Dimension Rating Scale and Descriptors

Rating Scale for Dimensions I, II, III, IV: •3: Meets most to all of the criteria in the dimension •2: Meets many of the criteria in the dimension •1: Meets some of the criteria in the dimension. •0: Does not meet the criteria in the dimension Descriptors for Dimensions I, II, III, IV: •3: Exemplifies CCSS Quality - meets the standard described by criteria in the dimension, as explained in criterion-based observations. •2: Approaching CCSS Quality - meets many criteria but will benefit from revision in others, as suggested in criterion-based observations. •1: Developing toward CCSS Quality - needs significant revision, as suggested in criterion-based observations. •0: Not representing CCSS Quality - does not address the criteria in the dimension.

Melanie Alkire

I'm not clear about why we need both of these lists of descriptors. Can't they be combined?

20

Apply Criteria in Dimension I: Alignment

Compare Criterion-Based Checks, Observations and Feedback

• What is the pattern within our team in terms of the criteria we have checked?

• Do our observations and feedback reference the criteria and evidence (or lack of evidence) in the instructional materials?

• Does our feedback include suggestions for improvement(s)?

21


22


23

Criteria for Dimension II: Key Shifts in the CCSS

The lesson/unit addresses key shifts in the CCSS:

Focus: Lessons and units targeting the major work of the grade provide an especially in-depth treatment, with especially high expectations. Lessons and units targeting supporting clusters have visible connection to the major work of the grade and are sufficiently brief. Lessons and units do not hold students responsible for material from later grades.

Coherence: The content develops through reasoning about the new concepts on the basis of previous understandings and provides opportunities for students to transfer knowledge and skills within and across domains and learning progressions.

24

Criteria for Dimension II: Key Shifts in the CCSS Rigor: Requires students to engage with and demonstrate challenging

mathematics with appropriate balance among the following:

o Application: Provides opportunities for students to independently apply mathematical concepts in real-world situations and problem solve with persistence, choosing and applying an appropriate model or strategy to new situations

o Conceptual Understanding: Provides opportunities for students to demonstrate conceptual understanding through challenging problems, questions, and writing and speaking about their understanding

o Procedural Skill and Fluency: Expects, supports and provides guidelines for procedural skill and fluency with core calculations and mathematical procedures (when called for in the standards for the grade) to be performed quickly and accurately

25



26

Apply Criteria in Dimension II: Key Shifts





27


28


29


30

Criteria for Dimension III: Instructional Supports

The lesson/unit is responsive to varied student learning needs:Includes clear and sufficient guidance to support teaching and learning of the targeted standards, including, when appropriate, the use of technology and media.Uses and encourages precise and accurate mathematics, academic language, terminology, and concrete or abstract representations (e.g., pictures, symbols, expressions, equations, graphics, models) in the discipline.Engages students in productive struggle through relevant, thought-provoking questions, problems and tasks that stimulate interest and elicit mathematical thinking.Addresses instructional expectations and is easy to understand and use.

31

Criteria for Dimension III: Instructional Supports Provides appropriate level and type of scaffolding, differentiation,

intervention and support for a broad range of learners

Supports diverse cultural and linguistic backgrounds, interests and styles

Provides extra supports for students working below grade level

Provides extensions for students with high interest or working above grade level

32

Criteria for Dimension III: Instructional SupportsA unit or lesson should: Recommends and facilitates a mix of instructional approaches for a variety of learners such as using multiple representations (including models), using a range of questions, checking for understanding, flexible grouping, pair-share, etc.

Gradually removes supports, requiring students to demonstrate their mathematical understanding independently

Demonstrates an effective sequence and a progression of learning where the concepts or skills advance and deepen over time

Expects, supports and provides guidelines for procedural skill and fluency with core calculations and mathematical procedures (when called for in the standards for the grade) to be performed quickly and accurately

33



34

Apply Criteria in Dimension III: Instructional Supports





35


36


37

Criteria for Dimension IV: Assessment

The lesson/unit regularly assesses whether students are mastering standards-based content and skills: Is designed to elicit direct, observable evidence of the degree to which a student can independently demonstrate the targeted CCSS Assesses student proficiency using methods that are accessible and unbiased, including the use of grade-level language in student promptsIncludes aligned rubrics, answer keys and scoring guidelines that provide sufficient guidance for interpreting student performanceA unit or longer lesson should: Uses varied modes of curriculum-embedded assessments that may include pre-, formative, summative and self-assessment measures

38



39

Apply Criteria in Dimension IV:Assessment





40


41


42

Overall Rating and Summary Comments

Overall Rating for the Lesson/Unit:

E: Exemplar – Aligned and meets most to all of the criteria in dimensions II, III, IV (total 11 – 12)

E/I: Exemplar if Improved – Aligned and needs some improvement in one or more dimensions (total 8 – 10)

R: Revision Needed – Aligned partially and needs significant revision in one or more dimensions (total 3 – 7)

N: Not Ready to Review – Not aligned and does not meet criteria (total 0 – 2)

43

Compare Summary Comments and Determine Next Steps

Guiding questions to synthesize criterion-based observations and suggestions: •How do the observations and suggestions for improvement compare?•Are the observations and suggestions criterion-based?•Does this example serve as a model of CCSS instruction? What are its strengths? Areas for improvement?

44


45


46

Reflection on Session Goals

• Did we use the EQuIP criteria to frame and explain evaluation of evidence found in instructional materials?

• Did we develop a common understanding of EQuIP criteria among reviewers?

• Are there any criteria or evidence about which reviewers disagree?• Did we develop reviewers’ abilities to use EQuIP criteria, rating

scales/categorizations and rating descriptors to accurately rate instructional materials?

• To what degree were there differences among reviewers when checking criteria, assigning dimension ratings and assigning overall ratings? What do you think caused these differences?

47

Reflection on Session Goals

• How might you use this evaluation process in your schools?

equip rubric & effective ccss feedback training session: math

Documents

quality review processdevelop

review team

supply of high quality

review processes

rubric dimensions

use of criteria

process dimensions

alignment of lessons