april 2008getting involved in star1 star item writing, assessment review panels, standard setting,...

April 2008 Getting Involved in STAR 1

STARSTAR Item Writing, Item Writing,

Assessment Review Panels, Assessment Review Panels, Standard Setting,Standard Setting,and Field Testingand Field Testing

WebcastWebcastApril 23, 2008

START TIME 9:00 a.m.Presenters

Jonathan Halling - ETSStephanie Sartain Herrera - ETS

Mary Anne Arcilla - ETSMark McLean - CDE


AgendaAgenda

1. Life of a Test Item2. Item Writer Training3. Assessment Review Panels (ARPs)4. Standard Setting5. Field Testing6. Open Forum


Life of a Test ItemLife of a Test Item


Study state guiding documents

ARP & CDEdevelop blueprints

SBE action

Develop item writer

materials

CDE review of item writer materials

Train item writers

Receive itemsPrepare, review, and revise items

ARP & CDE review items

Revise itemsSPARreview

Create field test sets with CDE review

Conduct field tests

Score and process data from field test

ARP & CDE review data

from field test

Create operational forms

ARP & CDE review forms

Administer operational test



Develop item writer

materials


Train item writers





Conduct field tests






from field test


SBE action




SBE action

Develop item writer

materials


Conduct field tests






from field test


Train item writers







SBE action

Develop item writer

materials


Train item writers








Conduct field tests



from field test




SBE action

Develop item writer

materials


Train item writers





Conduct field tests




from field test




Item Writer TrainingItem Writer Training




SBE action

Develop item writer

materials


Train item writers





Conduct field tests


CDE & ARP review data

from field test





Item WriterItem WriterTrainingTraining

• What will be covered by ETS in the training session?– What makes a good item– The item-writing process– How to write items specifically to meet California

standards

• Who should attend? Persons who:– Possess knowledge of California content

standards – Are current or recent teachers– Have at least three (3) years teaching experience



• How many are trained? – 20–30 attendees per content area in CAPA, CST,

CMA, and STS

• Where does training take place?– 2008: Long Beach, CA

• When does the 2008 training occur?– July 15–16: CAPA items – July 17–18: CST, CMA, and STS items



• Benefits– Professional development– Increased familiarity with California content

standards– Potential for becoming professional item writer

• Expenses– $150 per day or substitute teacher reimbursement– Travel and lodging expenses paid by ETS


Assessment Review PanelsAssessment Review Panels(ARPs)(ARPs)


Assessment Review PanelsAssessment Review Panels

Panel members meet in an advisory capacityto ETS and CDE to review:

• Item specifications and blueprints• Proposed new test items• Data collected on field-tested items • Proposed new operational test forms• Proposed released test questions (RTQs)

– CST only

• Differential Item Functioning (DIF)




SBE action

Develop item writer

materials


Train item writers





Conduct field tests



from field test





Assessment Review PanelsAssessment Review PanelsItem Specifications and BlueprintsItem Specifications and Blueprints

• Help define recommended blueprints for new tests– Develop foundation for future item and test

form development – Examples:

• Revised General Mathematics– Conducted April 2, 2008

• California Modified Assessment (Grades 9–11)




SBE action

Develop item writer

materials


Train item writers





Conduct field tests



from field test





Assessment Review PanelsAssessment Review PanelsItem ReviewItem Review

• Advise if the items developed are:– Measuring the standard – Grade-level appropriate– Free from bias

• Ethnic • Socioeconomic• Gender• Geographical

– Interesting and appropriate to students

• Suggest ways in which an item may be adjusted to meet the above criteria, if necessary

• Do NOT rewrite items




SBE action

Develop item writer

materials


Train item writers





Conduct field tests



from field test





Assessment Review PanelsAssessment Review PanelsData ReviewData Review

• Recommend whether field-tested items should be included in the “bank” and hence be available to be selected for inclusion on future operational tests– Review statistics on the item’s performance

as a field-test item– No editing of items is performed at this

stage


Assessment Review PanelsAssessment Review Panels Differential Item Functioning (DIF)Differential Item Functioning (DIF)

ReviewReview• Occurs when item performance is significantly

different for members of reference groups and comparison groups

• Participants representing various demographic groups review test questions to determine whether or not items are appropriate

• One-day meeting, conducted after field testing

Note: DIF review panel consists of different participants from other ARPs




SBE action

Develop item writer

materials


Train item writers





Conduct field tests



from field test





Assessment Review PanelsAssessment Review PanelsForms ReviewForms Review

• Last stage in the process of building the test

• Assembled operational forms:– Contain a range of difficulty of items– Match the blueprint and statistical

specifications– Avoid clueing

• No editing of items


Assessment Review PanelsAssessment Review PanelsReleased Test Question (RTQ) Released Test Question (RTQ)

ReviewReview• After use on operational test forms,

some questions are released for review and use by the public– Contain a range of item difficulty– Are proportional to the blueprint – Released annually

• No editing of items


ARP Meeting SchedulesARP Meeting Schedules


2008 ARP Meeting Schedule2008 ARP Meeting ScheduleCSTCST

• Item Review – April 2–5

• Data Review– July 29–30

• Forms Review– September 23

• Released Test Questions– October 29

Note: ARP members are expected to attend meetings at ALL stages of development


2008 ARP Meeting Schedule2008 ARP Meeting Schedule CMA CMA

• Item Review– Grades 3–8

• September 9–12

• Data and Forms Review– Grades 6–8

• November 19–20

• Data Review – Grades 3–5

• To be scheduled early 2009



2008 ARP Meeting Schedule2008 ARP Meeting ScheduleCAPACAPA

• Item Review – May 13–14

• Data Review– September 25

• Forms Review– To be scheduled early 2009



2008 ARP Meeting Schedule2008 ARP Meeting Schedule STS STS

• Item Review– RLA grades 8–11, Algebra I, and Geometry

• April 8–11– RLA and Math grades 2–7

• July 8–11

• Data Review– RLA and Math for grades 2–7

• October 8–9

• Forms Review– Grades 2–4

• October 10



Becoming an ARP MemberBecoming an ARP Member


The ProcessThe Process

1. Review the ARP member requirements2. Obtain application form:

• E-mail [email protected]• Call Kristin Elizalde 925-808-2009

3. Submit a completed application form4. Application reviewed by:

• ETS content experts• CDE Standards and Assessment staff• State Board of Education staff

5. Application approved or denied

Note: Process can take several weeks


ARP Members Needed!ARP Members Needed!

• California Standards Tests– Science (Grades 5 and 8)

– History–Social Science (Elementary and Middle School)

– ELA (All grades)

• California Modified Assessment– English–Language Arts (ELA)

– Mathematics

– Science

Note: CMA – Middle and High School especially needed!


ARP Members Needed!ARP Members Needed!

• California Alternate Performance Assessment– English–Language Arts (ELA)– Mathematics– Science

• Standards-based Tests in Spanish– Reading/Language Arts (RLA)– Mathematics, Algebra I, and Geometry

Note: STS – Middle and High School especially needed!


Benefits of Being an ARP Benefits of Being an ARP MemberMember

• Become more familiar with the testing program

• Professional development– Learn more about what attributes a high

quality test item should display– Learn more about assessing how an item

performs on a test

• Contribute to the development of high-quality tests

• All meeting attendance expenses covered


ARP Member RequirementsARP Member Requirements


ARP Member Requirements ARP Member Requirements All Panels All Panels

• Educators from– Urban areas– Rural areas– Suburban areas

• Experienced in teaching K–12 students from diverse populations


ARP Member RequirementsARP Member RequirementsCST, CMA, CAPA, and STSCST, CMA, CAPA, and STS

• Bachelor’s or higher degree in content area or related subject

• Knowledge of California content standards• Familiarity with student capabilities• K–12 teachers must also have:

– Three or more years teaching courses in grades K–12• ELA, math, science, or history–social science

• County, district, or university content specialists must also have:– Three or more years as a district or county content

specialist or university instructor• ELA, math, science, or history–social science


Additional ARP Member Additional ARP Member Requirements Requirements

for CMAfor CMA• English–language arts, mathematics, and science only• K–12 teachers

– Three or more years of experience teaching ELA, math, or science in grades K–12 and experience working with special education students

– Experience with more than one type of disability

• School administrators, district/county content/program specialists, or university educators– Three or more years of experience as a school administrator,

district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science



for CAPAfor CAPA• English–language arts, mathematics, and science only• K–12 teachers

– Experience with students with severe cognitive disabilities – Experience with more than one type of disability– Three or more years of K–12 experience teaching special

education or teaching ELA, math, or science in a general education classroom

• School administrators, district/county content/program specialists, or university educators– Three or more years of experience as a school administrator,

district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science



for STSfor STS

• Reading/language arts (RLA) and mathematics only

• Bilingual and biliterate in Spanish • Three or more years experience

teaching students in Spanish• Bachelor’s or higher degree in Spanish

or a subject area related to bilingual education, reading/language arts, or mathematics


BreakBreak10 Minutes10 Minutes


Standard SettingStandard Setting


What is Standard Setting?What is Standard Setting?• Process used to recommend the

minimum test score needed for a student to be considered performing at a certain performance level– For example, the minimum test score needed to be

at the proficient level

• Bookmark standard setting methodology – Participants review test items that have been

sequenced from the easiest to most difficult item– Participants identify the items that likely exceed the

ability of a student at each of the California performance levels

– The difficulty level of these identified items form the basis for the minimum test scores needed to reach each performance level


Standard SettingStandard SettingCAPA Levels I–V CAPA Levels I–V

(ELA, Math, and Science) (ELA, Math, and Science)

• Three-day meeting – September 16–18, 2008

• Participant requirements same as ARP members


Standard SettingStandard SettingCMA Grades 3–5CMA Grades 3–5

(ELA, Math, and Science) (ELA, Math, and Science)

• Two three-day meetings – ELA

• September 24–26, 2008

– Math and Science• October 1–3, 2008



Standard SettingStandard SettingSTS Grades 2–4STS Grades 2–4RLA and Math RLA and Math

• Three-day meeting – RLA

• January/February 2009

– Math• January/February 2009



Standard SettingStandard SettingMember Requirements Member Requirements

• Educators from:– Urban, rural, and suburban areas– Large and small districts– Geographically dispersed throughout California

• Experienced in teaching students in appropriate grades/content areas and from diverse populations

• Community members– Parent Teacher Association (PTA)– School board– Business


Standard SettingStandard SettingFor CAPA, CMA and STSFor CAPA, CMA and STS

• Application Form– E-mail [email protected]

or– Call Kristin Elizalde 925-808-2009

• Reimbursement for:– Travel and accommodation expenses– Substitute teacher for school/district

Apply now!


Field TestingField Testing


Field TestField TestCalifornia Modified Assessment California Modified Assessment

(CMA)(CMA)

• September 15–29, 2008• Eligible students

– IEP– Not taking CAPA– Administered to grades 5, 7, 8, and 9

• ELA (grades 7, 8, and 9)• ELA writing (grades 5 and 8)• Mathematics (grades 7 and 8)• Science (grade 9)

– Student may test in one or more content areas– 2008 CST results

• Far below basic• Below basic


QuestionsE-mail: [email protected]

Subject: 2008 CMA Field TestTelephone: STAR TAC 800-955-2954

CMA Field TestCMA Field Test


Questions

E-mail: [email protected]

Subject: April Webcast

Open ForumOpen Forum


Next WebcastNext WebcastSTAR 2008: Making Sense of the STAR 2008: Making Sense of the

STAR Data CDSTAR Data CD

May 21, 2008

Webcast starts at 9:00 a.m.

april 2008getting involved in star1 star item writing, assessment review panels, standard setting,...

Documents

field test arp cde review

items spar review slide

field test slide

test item slide

field test sets

operational test slide

assessment review panels

field testing webcast