Standard Setting: Standard Setting: MCAS High School Science & MCAS High School Science &

Technology/Engineering Technology/Engineering (STE) Tests(STE) Tests

Sheraton Four Points HotelNorwood, MA

August 14-16, 2007

Tuesday, August 14Tuesday, August 14Overview of Plenary Overview of Plenary

SessionSession Welcome/Introductions Overview of MCAS Program and

the High School STE Tests Purpose of 2007 Standard Setting Body of Work Method and

Procedures Ground Rules for Standard Setting Agenda (Tuesday-Thursday)

Department of Department of Education Education Sal Beatini, MCAS Science Test Developer Bob Bickerton, Associate Commissioner Joyce Bowen, Science Specialist Katie Bowler, MCAS Science Test Development Lead Per Christiansen, MCAS Science Test Developer Kevin Dwyer, MCAS History Test Development Lead Jake Foster, State Science Coordinator Mark Johnson, Director of MCAS Test Development Bob Lee, MCAS Chief Analyst Matt O’Connor, Administrator for Administration, Analysis

and Reporting Kit Viator, Director of Student Assessment

Measured Measured Progress Progress Ann Adjutant, Physics Developer

Liz Burton, Psychometrician Lee Butler, Administrative Assistant Lisa Ehrlich, Assistant Vice President Kevin Haley, Manager of Data Analysis Renee Jordan, Service Center Representative Mark Peters, Program Assistant Miechelle Poulin, Program Assistant Michael J. Richards, Program Manager Kevin Sweeney, Assistant Vice President, Research & Analysis David Tong, Assistant Director, MCAS Program Management Karen Whisler, Lead Science Developer Eric Wigode, Director of MCAS Test Development

Biology PanelistsBiology PanelistsErin Bengiovanni Biology Teacher Joseph P. Keefe TechnicalCarol Bernon Science Department Head Barnstable High SchoolSheila Blair Science Dept. Head/Biology Teacher Hull High SchoolRonit Carter Educational Consultant In Harmony with EducationAndrew Creamer Biology Teacher Classical High SchoolAmy Deacon Biology Teacher Pentucket Regional HighJennifer Dube Science Instructor Greater Lawrence Technical HighEileen Dufour Biology II Instructor Greater Lowell Technical HighMary Dulko Biology Teacher Sharon High SchoolRichard Fardy Biology Teacher Wilmington High SchoolMartha Hogan Academic Chair, Science Salem High SchoolElizabeth Hufnagel Science Teacher Brookline High SchoolJulie Jonelis Biology Teacher Sabis International CharterSarah Juhlin Science Instructor Blackstone Valley RegionalKelly Kallin Biology Teacher Sharon High SchoolAaron Mathieu High School Biology Teacher Acton-Boxborough RegionalJim Perry Science Chair Mansfield High SchoolBeth Raccula Science Teacher Walpole High SchoolDianne Rees Director of Science Braintree High SchoolJulie Robertson Upper School Biology Teacher Foxborough Regional CharterCarla Romney Faculty Member Boston UniversityDeborah Sakelakos MCAS Program Administrator Bedford High SchoolErnestine Struzziero Science Teacher Lynnfield High School

Chemistry PanelistsChemistry PanelistsSteven Abdow Chemistry Instructor Durfee High School

Thomas Bartosek Biology/Chemistry Teacher Canton High School

Catherine Botsford Milne Chemistry Teacher Attleboro High School

Patricia Brandl Science Teacher Medford High School

Barbara Brigante Teacher Northeast Metro. Regional

Stacia Brown Science Curriculum Coord. Oakmont Regional HS

Lisa Buron Science Teacher Mansfield High School

Tony DiLuna Science Teacher Woburn Memorial High School

Mary Duane Science Teacher Tantasqua Regional High School

Anthony Fimognari Chemistry Teacher East Longmeadow High School

Catherine Francis Chemistry Teacher North Reading High School

Boriana Georgieva Chemistry Teacher Swampscott High School

Joyce Gleason Educational Consultant

Esther Hines Chemistry Teacher Billerica Memorial High School

Glenn Horner Executive/Scientist DCE Aprilis

Shawn Kenner Chemistry Teacher Sharon High School

Ronee Krashes Teacher Framingham High School

Peter Nassiff Head of Science Department Burlington High School

Diane Perito Chemistry Teacher Malden High School

Nitzan Resnick Director South Area Solomon Schecter Day School

Michael Thompson Chemistry Teacher Amherst Regional High School

Linda Weber Chemistry Teacher Natick High School

Tara Yohan Biology/Chemistry Teacher Amherst Regional High School

Introductory Physics PanelistsIntroductory Physics PanelistsAnne Marie Brooks Science Teacher Wareham High School

Darren Burris Math Instructor Boston Collegiate Charter

Norma Chico High School Physics & Chemistry Doherty Memorial

Steven Cogger BMW Technician BMW Peabody

Marilyn Decker Senior Program Director, Science Boston Public Schools

Michael Doherty Teacher Andover High School

Brian Dukeman Science Teacher Whitman Hanson Regional

Erin Marie Flanagan Teacher Wachusett Regional High School

Cynthia Givens Teacher Andover High School

Naila Jirmanus Science Program Coordinator Sabis International Charter

Christine John Physics Teacher Belmont High School

Mon Luke Teacher Bedford High School

Kristin Newton Physics Teacher Cambridge Rindge & Latin

Julie Shea High School Science Teacher Revere High School

Mary Ellen Stephen Physical Science & Chemistry Teacher Plymouth South High School

John Sullivan Retired Physics Teacher Quincy Public Schools

Sue Vincent Teacher Turners Fall High School

Amy Winston Science Department Chair Newton North High School

Technology/Engineering PanelistsTechnology/Engineering Panelists

Joseph Clement Department Head Beverly High School

Kevin Connolly Technology Teacher Bedford High School

Leigh Estabrooks Inventeams Associate MIT

Thomas Fischer Technology Eng. Teacher Hopedale Jr. Sr. High School

Pamela Glass Math/Science Curriculum Coordinator Westport Community School

Ramiro Gonzalez Engineering & Technology Teacher Boston Arts Academy

William Hankey Engineering Teacher Doherty Memorial High School

Rebecca Lothrop Teacher Clearway School

Charles Pouliot Teacher Lawrence Public Schools

Lee Pulis Curriculum Developer Museum of Science

Bruce Rawley Technology/Engineering Teacher Millbury High School

Anthony Ruscito Technology Teacher Bedford Public Schools

Lynn Russell Biology Teacher New Leadership Charter

Mike Stevens Technology/Engineering Teacher Maynard Public Schools

Victor Valenti Retired Engineer Compaq

Catherine Wolf High School Science Teacher Brookline High School

Standard Setting Standard Setting FacilitatorsFacilitators

Biology – Lisa Ehrlich

Chemistry – Margaret Hill

Introductory Physics – Dona Carling

Technology/Engineering – Tim

Crockett

Purpose of MCAS Purpose of MCAS ProgramProgram

Inform/improve curriculum and instruction

Evaluate student, school, and districtperformance according to Curriculum Framework content standards and MCAS performance standards

Determine eligibility for high school diploma (Competency Determination)

Massachusetts Massachusetts High High School School Competency Competency Determination Determination (CD)(CD)RequirementsRequirements

Meet local graduation requirements Attain passing standard (scaled

score of 220) or higher on high school MCAS tests or equivalent on MCAS Alternate Assessment

To earn diploma, a student must:

Selected Features of Selected Features of MCASMCAS

Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards

100% of questions used to determine student scores released annually

Measures performance of ALL students educated with public funds

Results reported according to scaled scores and performance levels

Historical Background of the Historical Background of the MCAS TestsMCAS Tests

First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10)

1998

Class of 2003 first graduating class required to earn a CD (ELA and Math)

STE tests in grades 5 and 8 introduced

2003

Class of 2010 first graduating class required to earn a CD in STE

2010

Massachusetts Education Reform Law passed

1993

2001

Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced

Newly required NCLB tests (grades 3-8) introduced

2006

1996 First STE Curriculum Framework introduced

1998-2000 Integrated HS STE test administered

2001-2002 Revised STE Curriculum Framework (major revisions); no STE testing

2003 Question tryout for tests in Biology, Chemistry, Introductory Physics, and Technology/Engineering; no results reported

2004-2005 Pilot testing of STE tests; item-level results onlyBOE decides class of 2010 to pass one of STE tests

to earn CD

2006 High School STE CF revised (minor revisions)

2007 Operational test9th graders may earn CD by passing one of four

STE tests (ELA and Math requirements also apply)

Development of Massachusetts STE Development of Massachusetts STE Curriculum Framework (CF) Curriculum Framework (CF) Standards and AssessmentsStandards and Assessments

MCAS STE Testing ProgramMCAS STE Testing Program

End-of-course tests offered in grades 9 and 10:– Biology – Chemistry– Introductory Physics– Technology/Engineering

Students required to pass one of four tests in order to earn CD

High School STE High School STE MCAS Test DesignMCAS Test Design

Each of the four tests includes:

40 multiple-choice items

1 point each

5 open-response items

4 points each

60 total raw

score points

==

MCAS Scaled Scores/Performance Levels

FailingNeeds

Improvement

Proficient Advanced

200

220

240

260

280

High School Science and High School Science and Technology/ Engineering Test Technology/ Engineering Test

Reporting CategoriesReporting CategoriesBiology

Rep. Category % of items (+/- 5%)

Biochemistry and Cell Biology

25

Genetics 20

Anatomy & Physiology

15

Ecology 20

Evolution & Biodiversity

20

ChemistryRep. Category % of items

(+/- 5%)

Properties of Matter and Thermochemistry

25

Atomic Structure & Periodicity

25

Bonding & Reactions

30

Solutions, Equilibrium, & Acid-Base Theory

20

Science and Science and Technology/Engineering Technology/Engineering

Test Reporting Test Reporting CategoriesCategories

Introductory Physics

Rep. Category % of items (+/- 5%)

Motion & Forces

40

Heat & Heat Transfer

15

Waves & Radiation

25

Electromagnetism

20

Technology/Engineering

Rep. Category % of items (+/- 5%)

Engineering Design

20

Construction & Manufacturing

20

Fluid & Thermal Systems

30

Electrical & Communications Systems

30

Rationale:Rationale: End-of-Course STE Test End-of-Course STE Test

MenuMenu Promote deeper assessment of

content than integrated test Support diverse course offerings

and scheduling flexibility for high schools

Provide students choice of assessment for CD requirement

Efforts to Establish Efforts to Establish “Equivalence” “Equivalence” of STE Testsof STE Tests

The STE tests: are based on parallel test design

and development processes are based on performance level

descriptors that are comparable in scope and rigor

have been designed to have similar psychometric properties

External Validation of Efforts External Validation of Efforts to Establish “Equivalence” of to Establish “Equivalence” of

STE TestsSTE TestsPerformance level descriptors used in

standard setting have been externally validated by Massachusetts teachers

Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst

Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

Content Standards Content Standards vsvs. . Performance StandardsPerformance Standards

Content standards = “What”Describe the knowledge and skills students should acquire in a particular content and grade

Performance Standards = “How well”Describe student work on MCAS tests at the Needs Improvement, Proficient, and Advanced levels

General MCAS General MCAS Performance Level Performance Level

DescriptorsDescriptorsNeeds Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems.

Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems.

AdvancedStudents at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

Needs Improvement

On MCAS, a student at this level

Proficient

On MCAS, a student at this level

Advanced

On MCAS, a student at this level

Conceptual Understanding and Factual Knowledge

Demonstrates a partial understanding of some facts, concepts, principles, and theories

Uses basic scientific terms

Demonstrates a solid understanding of many facts, concepts, principles, and theories

Uses appropriate scientific terms

Demonstrates a comprehensive, in-depth understanding of many facts, concepts, principles, and theories

Applies scientific terms in an appropriate context

Scientific Process and Skills

Interprets simple data and creates generalized questions without necessarily using the scientific method

Identifies a problem to be solved

Uses most of the steps of the scientific method to design and interpret experiments

Finds solutions to a variety of problems

Designs and evaluates scientific experiments and generates full interpretations of data

Finds solutions to complex problems

Application/Synthesis

Makes simple predictions about a specific topic

Makes predictions based on information given

Justifies predictions in a general sense

Makes sophisticated predictions

Synthesizes a wide array of information from multiple sources

Applies knowledge to abstract or novel situations

GeneralGeneral STE Performance Level STE Performance Level DescriptorsDescriptors

PurposePurpose

High School Science & Technology/ Engineering: To establish cut scores for MCAS performance levels

Failing Needs Improvement

Proficient

Advanced

Cut score

needed

Cut score

needed

Cut score

needed

Linking Performance Linking Performance Standards with Student Standards with Student

WorkWork What is standard setting?

Establishment of cut scores to distinguish between performance levels

What is your job?Use the PLDs to evaluate student work and make recommendations for where cut scores should be set

Purpose of Standard Purpose of Standard SettingSetting

Determine cut scores for reporting assessment results

Answer the question:– How much is enough?

General Phases of Standard General Phases of Standard SettingSetting

Data-collection phase

Policy-making/decision-making phase

Selected Standard-Setting Selected Standard-Setting MethodsMethods

Angoff Bookmark Body of Work

Choosing a Standard-Choosing a Standard-Setting MethodSetting Method Prior usage/history Recommendation/

requirement by policy-making authority

Type of assessment

Body of Work method chosen for MCAS tests inHigh School Science

& Technology/Engineering

What is the Body of Work What is the Body of Work Procedure?Procedure?

Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds.

HS STE Standard Setting:HS STE Standard Setting:Panelists examine student work

that has not been previously classified and determine how that work

should be classified.

How to Classify Student How to Classify Student WorkWork

Performance Level Descriptors• General• Content specific

Materials you will need:

Bodies of Student Work• Responses to constructed-response

questions• Multiple-choice summary sheet

Rating Forms

How to Classify Student How to Classify Student WorkWork

Examine the student’s responses to multiple-choice questions

Examine the student’s responses to open-response questions

Judge the student’s knowledge and skills demonstrated relative to the PLDs

Panelists do not need to reach consensus on the classifications

How to Classify Student How to Classify Student WorkWork

To help prepare you to do these ratings, you will

spend time becoming familiar with the following:

Test Performance level descriptors

• It is important that all panelists have a clear, common understanding of the PLDs.

Bodies of student work• Multiple-choice items• Constructed-response items

How to Classify Student How to Classify Student WorkWork

You will have the opportunity to discuss your classifications and change them if desired.

Don’t worry! We have procedures, materials, and staff to assist you in this process.

What Next?What Next?Break into content-area groups:

Take the assessment Discuss the Performance Level Descriptors Complete the Item Map Complete training round Complete individual ratings Receive feedback from first round of ratings Discuss ratings and revise Receive feedback from revised ratings Discuss feedback and provide final ratings Complete an evaluation form

Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting

5.We should use this time to rework HS Science and Tech/Eng. performance level descriptors.

8.Standard setting is a great opportunity to rewrite Curriculum Framework standards.7.The process is rigged.

6.This is a good time to vent about all the things you hate about MCAS.

Top 8 Most Top 8 Most Misunderstood Misunderstood Things about Things about Standard SettingStandard Setting

1.Disagreement is bad.

4.Standard setting is scoring.

3.Only HS Science and Tech/Eng. scholars should be doing this work.

2.Only teachers should be doing this work.

Ground RulesGround Rules Role of facilitator is to “facilitate” and keep

process on track Process solely focused on recommending

performance standards (cut scores) for MCAS MCAS performance level descriptors are

integral to process but are not up for debate Panelists’ recommendations are vital;

however, final cut scores determined by the MDOE

Each panelist must be in attendance for the duration of the process for his/her judgments to be considered

Each panelist must complete evaluation form at the end of the event

Cell phones off, please!

Agenda Agenda Tuesday, August 14Plenary 9:00 am – 10:30 am

Break 10:30 am – 10:45 amWork session 10:45 am –12:00 pmLunch 12:00 pm – 1:00 pmWork session 1:00 pm – 4:00 pm

Wednesday, August 15Breakfast 7:30 am – 8:30 amWork session 8:30 am – 12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – 4:00 pm

Agenda Agenda Thursday, August 16Breakfast 7:30 am – 8:30 amWork session 8:30 am – 12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – Until completion

Room Room AssignmeAssignmentsntsPhysics: 101Chemistry: 102Biology: 103Tech/Eng: 104

Questions?Questions?