Pilot Review Process

Oregon Rubric & Quality Review Training Session: Mathematics

What is a Pilot Review?

• Oregon has specific laws that outlines how state reviews are carried out and when (ORS 337 & OAR 581-11).– 30+ year old laws (Basal focused, seven year review cycle)– Review is sustained by publisher fees– State math review has been moved from 2014 to 2016– Will work in the 2015 session to update the state review process

• Districts are able to independently review and adopt at any time, provided they use the board approved criteria (OAR 581-022-1622)– Pilot process to support the local review of materials– Summer 2014: SOESD and Hillsboro “Pilot” review process for math– Fall 2014: share results and supporting documents– Summer 2015: aim to support more regional reviews of math

Session Goals

Use the Oregon quality review process (based on EQuIP and IMET) to determine the quality and alignment of lessons and units to the Common Core State Standards (CCSS) in mathematics

During this session, reviewers will:• Develop their abilities to use Oregon math materials criteria to provide

observations about CCSS-aligned instructional materials and make suggestions for improvement

• Develop a common understanding of the Oregon quality review process• Develop a common understanding of the rating scale and descriptors for the four

rubric dimensions and the rating categories and descriptors for overall ratings• Develop their abilities to use the criteria, rating scales and rating descriptors to

accurately rate instructional materials

Oregon Quality Review: Principles & Agreements

1. CCSS: Before beginning a review, all members of a review team are familiar with the CCSS.

2. Inquiry: Review processes emphasize inquiry rather than advocacy and are organized in steps around a set of guiding questions.

3. Respect & Commitment: Each member of a review team is respected as a valued colleague and contributor who makes a commitment to the review process.

4. Criteria & Evidence: All observations, judgments, discussions and recommendations are criterion and evidence based.

5. Constructive: Lessons/units to be reviewed are seen as “works in progress.” Reviewers are respectful of contributors’ work and make constructive observations and suggestions based on evidence from the work.

6. Individual to Collective: Each member of a review team independently records his/her observations prior to discussion. Discussions focus on understanding all reviewers’ interpretations of the criteria and the evidence they have found.

7. Understanding & Agreement: The goal of the process is to compare and eventually calibrate judgments to move toward agreement about quality with respect to the CCSS.

Using the Quality Review Rubric

The Three Shifts in Mathematics

•Focus: Strongly where the Standards focus•Coherence: Think across grades and link to major topics within grades•Rigor: Require conceptual understanding, fluency, and application

Activity 1:Finding Focus & Coherence

Looking for Focus

• Taking a closer look at the major work of each grade

• Two levels of focus:• What’s in/What’s out• The shape of the content that is in

Activity 1

Finding Focus in the CCSS

10

HS content

Activity 2a:Finding Rigor

Rigor: In Major Topics, Pursue Conceptual Understanding, Procedural Skill and Fluency, and Application

• The CCSSM require a balance of: Conceptual understanding Procedural skill and fluency Application of skills in problem solving situations

• Pursuit of all three requires equal intensity in time, activities, and resources.

The three legged stool

How do the Standards signal Rigor?

• Conceptual Understanding:3.NF.1 Understand a fraction 1/b as the quantity formed by 1 part when a whole is partitioned into b equal parts; understand a fraction a/b as the quantity formed by a parts of size 1/b.

• Procedural Skill and Fluency:5.NBT.5 Fluently multiply multi-digit whole numbers using the standard algorithm.

• Application:7.NS.3 Solve real-world and mathematical problems involving the four operations with rational numbers.

Application

• Real-world problems (single- and multi-step)

• Non-routine problems

• Varied problem types (see Tables in CC.OA progressions)

• Enhance major work of the grade

• Constructing models (6-12)

Conceptual Understanding

• Problems can (and should sometimes) be brief

• Explaining reasoning is one way to address conceptual understanding

• Problems and exercises should be grade-level appropriate

• Connections between representations are emphasized

Procedural Skill and Fluency

• Purely procedural problems

• Opportunistic strategies; writers are thoughtful about numbers used

• Repeated practice

• Procedures are built from conceptual understanding

Activity 2b: Focus & Rigor

19

Activity 2b:Looking for rigor in texts

20

Implement tasks that promote reasoning and problem solving

• Dan Meyer: “Math Class Needs a Makeover”

Need to develop patient problem solvers

• What are characteristics of a task that places:– A low-level cognitive demand on students?– A high-level cognitive demand on students?

• What does it mean for students to be “patient” or “impatient” problem solvers?– How can task selection and implementation

condition students to be one of these types of problem solvers?

Layers of a math problem

24

Activity 3Practice using OR-IMET

25

Using the Quality Review Rubric

OR-IMET

27

28

Using the Quality Review Rubrichttp://tinyurl.com/odemath-omet

For each dimension:

• Make observations and suggestions related to criteria and evidence.

• Determine a rating for each dimension based on checked criteria and observations.

• Additional comments to improve the rating of the material in this section

Grouping of Math Practices

Reasoning and Explaining2. Reason abstractly and quantitatively3. Construct viable arguments and critique the reasoning of others

Modeling and Using Tools4. Model with mathematics5. Use appropriate tools strategically Seeing Structure and Generalizing7. Look for and make use of structure8. Look for and express regularity in repeated reasoning

Overarching Habits of Mind of a Productive Mathematical Thinker1. Make sense of problems and persevere in solving them6. Attend to precision

Adapted from (McCallum, 2011)

29

Look for evidence

30

31

Giving Feedback

Writing effective feedback is vital to the Quality Review Process. Below are the four qualities of effective feedback. • Criteria-based: Written comments are based on the criteria used for review

in each dimension. No extraneous or personal comments are included. • Evidence Cited: Written comments suggest that the reviewer looked for

evidence in the lesson or unit that address each criterion of a given dimension. Examples are provided that cite where and how the criteria are met or not met.

• Improvement Suggested: When improvements are identified to meet criteria or strengthen the lesson or unit, specific information is provided about how and where such improvement should be added to the material.

• Clarity Provided: Written comment are constructed in a manner keeping with basic grammar, spelling, sentence structure and conventions.

Feedback Example #1: Mathematics

This unit clearly targets three CCSS, which are noted in the overview. The overview also indicates which Standards for Mathematical Practice are central to the lesson. The activities throughout the unit present a balance of mathematical procedures and deeper conceptual understanding of the standards. The activities reinforce the standards and are well-connected to the content. I think the activities might be challenging with a large class with classroom management issues.

Is this feedback criteria-based?Was evidence cited?Was there an improvement suggested?Is clarity provided?

Initial findingsPilot review SOESD & Hillsboro

Programs Reviewed – Summer 2014

Southern Oregon ESD

• Elementary School– Bridges (K-5)– Engage NY (K-5)– Math Expressions– My Math– Investigations (incomplete

materials submitted)

• Middle School– Core Focus– Connected Math 3– Agile Mind– Go Math– Engage NY (6-8)

35

Programs Reviewed – Summer 2014

Southern Oregon ESD• High School

– HMH HS math (unpublished)– Big Ideas– College Prep Math– Core Plus– Pearson Math– Engage NY (attempted -

incomplete)

Hillsboro Regional Review• High School

– HMH HS math (unpublished)– College Prep Math– Pearson Math– McGraw Hill Math– CK-12– Engage NY (attempted -

incomplete)

36

Preliminary Results:Elementary (SUM 14)

37

Program Name Publisher Review Site “3 or 4” Count

Percentage “4”

Bridges MLC SOESD 10 50%

Engage NY OER/Eureka SOESD 9 0%

Math Expressions HMH SOESD 7 0%

My Math McGraw-Hill SOESD 2 0%

Preliminary Results:Elementary (SOESD: SUM 14)

38

Program Name 1 2 3 4 5 6 7 8 9 10

Bridges 3 3 4 3 3 4 4 3 4 4

Engage NY 3 3 3 3 3 3 3 3 2 3

Math Expressions

3 3 2 3 3 3 2 3 2 3

My Math 3 2 3 2 2 2 2 2 2 2

Preliminary Results:Middle School (SUM 14)

39

Program Name Publisher Review Site “3 or 4” Count

Percentage “4”

Core Focus SMC SOESD 10 90%

Connected Math 3 Pearson SOESD 10 80%

Agile Mind Agile Mind SOESD 9 20%

Go Math HMH SOESD 6 0%

Engage NY OER/Eureka SOESD 2 0%

Preliminary Results:Middle School (SOESD: SUM 14)

40

Program Name 1 2 3 4 5 6 7 8 9 10

Core Focus 4 4 4 4 4 4 4 3 4 4

Connected Math 3 3 4 4 4 3 4 4 4 4 4

Agile Mind 3 3 4 3 3 3 3 2 4 3

Go Math 3 3 2 2 3 2 2 2 3 3

Engage NY 2 2 2 2 2 2 3 2 3 2

Preliminary Results:High School (SOESD & HSD: SUM 14)

41

Program Name Publisher Review Site “3 or 4” Count

Percentage “4”

Core Plus Math HMH SOESD 10 80%College Prep Math CPM SOESD 10 80%Big Ideas HMH SOESD 10 80%HMH Math Unpublished SOESD 10 70%HMH Math Unpublished HSD 10 70%Pearson Math Pearson SOESD 7 10%College Prep Math CPM HSD 6 0%Pearson Math Pearson HSD 5 0%McGraw Hill McGraw-Hill HSD 3 0%CK-12 OER HSD 2 0%Engage NY SOESD/HSD IncompleteAgile Mind SOESD Incomplete

Preliminary Results:High School (SOESD & HSD: SUM 14)

42

Program Name Review Site

1 2 3 4 5 6 7 8 9 10

Core Plus Math SOESD 4 4 4 4 4 4 3 3 4 4College Prep Math SOESD 4 4 4 4 3 3 4 4 4 4Big Ideas SOESD 4 4 3 4 4 4 3 4 4 4HMH Math (unpub.) SOESD 4 4 4 4 4 3 3 4 4 3HMH Math (unpub.) HSD 4 4 4 4 4 3 4 4 3 3Pearson Math SOESD 3 3 2 2 3 3 2 3 4 3College Prep Math HSD 3 3 3 3 2 3 2 3 2 2Pearson Math HSD 3 3 3 3 3 2 2 2 2 2McGraw Hill HSD 3 3 2 2 3 2 2 2 2 2CK-12 HSD 3 2 1 1 3 2 2 1 1 2Engage NY SOESD

(partial) 2 2 2 2 3

Lessons Learned & Moving Forward

43

Lessons Learned from Sum 14

• Training & Calibration is difficult and non-trivial– Significant Refinement from SOSED to HSD– Importance of providing practice with real programs

• Understanding quality criteria valuable regardless if doing a formal review– Spill over effect of understanding concepts like focus and rigor in a

new context– Need to understand quality as materials are organized or created

• Strong interest in this work– Need for both purchase and creation of materials– Need to continue in 2015

44

Looking ahead to 2014-15

• Establishment of review cohort– Provide training to 3-4 leaders from 6-7 regions in the state (~24

statewide) – Can review materials/facilitate reviews Sum 2015

• PLT conferences– First day general sessions/Second day breakouts (math)– Fall training on Finding, evaluating, & modifying resources

• Multi-State OER collaborative– ~10 states have agreed to support the development of CCSS OER

courses in ELA and Math (including Oregon)– Coordinated by CCSSO and Creative Commons– RFP Fall 2014, courses as early as Sum/Fall 2015

45

Questions?

• Mark FreedMathematics Education SpecialistOregon Department of Educationmark.freed@state.or.us503-947-5610

46