transitioning to new assessments

CALIFORNIA DEPARTMENT OF EDUCATIONTom Torlakson, State Superintendent of Public Instruction

Reauthorization of the Statewide Pupil Assessment System and

SMARTER Balanced Assessment Consortium (SBAC) Update

CA Co-op Directors Conference Patrick Traynor, Ph.D.

Director, Assessment Development and Administration Division

Transitioning to New Assessments

TOM TORLAKSONState Superintendent of Public Instruction

California Current Assessments

• Standardized Testing and Reporting Program (STAR)• California Standards Tests (CSTs)• Early Assessment Program (EAP)• California Alternate Performance Assessment (CAPA)• California Modified Assessment (CMA)• Standards-based Tests in Spanish (STS)

• California High School Exit Exam (CAHSEE)• California English Language Development Test (CELDT)• General Educational Development Test (GED)• California High School Proficiency Exam (CHSPE)• National Assessment of Educational Progress (NAEP)• Physical Fitness Test (PFT)



California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to:•Develop recommendations addressing 16 areas toward reauthorization of the assessment system•Develop a plan for transitioning to new “high quality assessments”•Consult with specific stakeholders:

• The State Board of Education• Public Schools Accountability Act (PSAA) Committee• Measurement experts from California’s private and public universities• Experts in assessing students with disabilities and English learners• Teachers, administrators, and governing boards from California’s local

educational agencies• Parents

•Recommendations and Plan due to the Legislature fall 2012

3


“High-quality assessment”

• Means an assessment designed to measure a pupil’s knowledge of, understanding of, and ability to apply critical concepts through the use of a variety of item types and formats, including, but not limited to, items that allow for open-ended responses and items that require the completion of performance-based tasks

4


16 Areas of Consideration

The recommendations shall consider including all of the following elements in the reauthorized assessment system:

(1) Aligning assessments to standards

(2) Implementing common assessments developed by state collaborative

(3) Conform to ESEA reauthorization

5


16 Areas of Consideration (Cont.)

(4) Measurement of achievement at a point in time and over time for groups and subgroups of pupils and for individual pupils.

(5) Allow for comparison from one year to the next as a reflection of growth over time

(6) Valid, reliable, and fair for all students including English learners (ELs) and students with disabilities

6



(7) Assessments of ELs using primary language assessments

(8) Ensure no bias with respect to race, ethnicity, culture, religion, gender, or sexual orientation

(9) Incorporate a variety of item types including open-ended and performance-based tasks

7



(10) Generating multiple measures of pupil achievement, which, when combined with other measures, can be used to determine the effectiveness of instruction and the extent of learning

(11) Assess science and history–social science in all grade levels at or above grade 4

(12) Assess understanding and ability to use technology necessary for success in the 21st century classroom and workplace

8


9


(13) Formative and interim assessments that provide timely feedback for purposes of continually adjusting instruction to improve learning

(14) Use test administration and scoring technologies that will allow the return of test results to parents and teachers as soon as possible

(15) Minimizing testing time(16) Including options for diagnostic

assessments for pupils in grade 2


Diagnostic Assessment• Ed Code 60603 defines diagnostic assessment

as:

“ . . . assessment of the current level of achievement of a pupil that serves both of the following purposes:

1) The identification of particular academic standards or skills a pupil has or has not yet achieved.

2) The identification of possible reasons that a pupil has not yet achieved particular academic standards or skills.”

10


Formative Assessment

Ed Code 60603 defines formative assessment as:

“Assessment tools and processes that are embedded in instruction and are used by teachers and pupils to provide timely feedback for the purposes of adjusting instruction and to improve learning.”

11


Interim Assessments

Ed Code 60603 defines interim assessments as:“ . . . an assessment that is given at regular and specified intervals throughout the school year, is designed to evaluate a pupil’s knowledge and skill relative to a specific set of academic standards, and produces results that can be aggregated by course, grade level, school, or local education agency in order to inform teachers and administrator at the pupil, classroom, school, and local education agency levels.”

12


Summative Assessment

Ed Code 60603 defines “achievement test” as:

“ . . . any standardized test that measures the level of performance that a pupil has achieved in the core curriculum areas.”

13


Relationship of Assessments

14

Perie, Marion, Gong, Wurtzel, 2007


Common Assessment Formats

• Paper and pencil testing– Most common type of format– Used for all assessments

• Computer based testing (CBT)– Uses fixed form but is administered using a

computer

• Computer adaptive testing (CAT)– Presents harder or easier questions, depending on

how the student performs as the test proceeds– Can provide greater accuracy with fewer questions

15


1616

Computer Adaptive Testing

• The system will adjust the difficulty of items throughout the assessment

Correct or Incorrect

Difficulty of Next Item

Correct More Difficult

Incorrect Easier


17

Computer Adaptive Testing20 Items


18

Six Item Types

• Selected Response• Short Constructed Response• Extended Constructed Response• Performance Tasks• Technology-Enabled• Technology-Enhanced


The Smarter Balanced Assessment Consortium

(SBAC)

• Area of Consideration 2 (of 16): Implementing common assessments developed by state collaborative

• Assessments for grades 3 - 8 and 11 aligned to the Common Core State Standards (ELA and Math)

• Operational across Consortium states in the 2014–15 school year

• Meets ESEA assessment requirements 19


2020

25 states educating approximately 20 million of the nation’s public K to 12 students

21 governing, 4 advisory states

An Ever Changing Landscape


SBAC Procurements Managed by the State of Washington*

RFP ID RFP Name Contractor RFP Work

Released Start Date

RFP-00 Project Management Partner WestEd 06/04/2010 10/01/2010

RFP-01 Communications Services Provider GMMB 02/25/2011 06/03/2011

RFP-02 Technology Readiness Tool Pearson 05/20/2011 12/07/2011

RFP-03 IT Systems Architecture Measured Progress 05/27/2011 09/15/2011

RFP-04 Item Specifications Measured Progress/ETS 07/27/2011 12/01/2011

RFP-05 Psychometric Services ETS 10/07/2011 02/06/2012

RFP-06 Accessibility and Measured Progress 09/22/2011 01/04/2012

Accommodations Policy Guidelines

RFP-07 Item Authoring/Item Pool Pacific Metrics 12/09/2011 04/03/2012

RFP-08 Item/Task Materials Development Measured Progress 10/14/2011 02/06/2012

RFP-09 Test and CAT Specifications ETS 10/25/2011 02/06/2012

RFP-11/ Test Engine Development for Pilot 03/07/2012 08/20/2012

and Field Test

RFP-18/ and CAT Simulations

RFP-20

RFP-12 Initial Achievement Level CTB/McGraw-Hill 03/14/2012 07/13/012

Descriptors *Status current as of 8/22/12


SBAC Procurements Managed by the State of Washington (cont.)

RFP ID RFP Name Contractor RFP Work

Released Start Date

RFP-13 Translations 06/15/2012 07/15/2012

RFP-14 Item/Task Writing/Review — Pilot CTB/McGraw-Hill 12/13/2011 03/28/2012

RFP-15 Report Development 04/18/2012 09/2012

RFP-16 Item/Task Development — Field Test 09/2012 01/2013

RFP-17 Scoring 11/2012 02/2013

RFP-19 Test Administration 04/13/2012 09/2012

RFP-21 Standard Setting 01/2014 04/2014

RFP-22 Sustainability Task Force 04/12/2012 09/2012

RFP-23 Formative Assessment Practices 08/2012 11/2012

and Professional Learning


Logical Components

Re-take option

Optional Interim assessment system—

Summative assessment for accountability

Last 12 weeks of year*

DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.

Scope, sequence, number, and timing of interim assessments locally determined

* Time windows may be adjusted based on results from the research agenda and final implementation decisions.

Source: http://www.ets.org

Computer AdaptiveAssessment andPerformance Tasks

PERFORMANCETASKS

• Reading• Writing• Math

END OF YEARADAPTIVE

ASSESSMENT

English Language Arts and Mathematics, Grades 3–8 and High School

Computer AdaptiveAssessment andPerformance Tasks

BEGINNING OF YEAR

END OF YEAR

INTERIM ASSESSMENT INTERIM ASSESSMENT

24


2525

School-level Technology Issues

• Technology-enhanced test items will require compatibility with latest browser technologies

• Test security will require device to run in “kiosk mode”

• Devices must support text-to-speech software

• Test items with multimedia content will require sufficient internal and Internet network bandwidth


26

Technology Readiness Tool

• Assesses current capacity and compares to the technology that is expected to be required for new SBAC assessments

• Areas evaluated: devices, devices to tester ratio, network infrastructure, and staff and personnel

• Performs inventory, gap analysis and recommended areas where LEAs will need to upgrade technology


27

Technology Readiness Tool Data Collection

• First data collection window was from March 20, 2012 to June 30, 2012

• Subsequent data collections will occur each spring and fall through 2014

• Input information for computers expected to be available for online testing in 2014.


2828

Technology Readiness Tool Stats (Cont.)

• 600,000 devices entered into system• Operating systems inventory

– Windows XP: 52%

– Windows 7: 20%

– Mac OS X: 20%

– iOS 5.x: 3%

– Others (Vista, Linux, Windows – Other, iOS 4.x, Google Chrome, Android)


2929

Technology Readiness Tool Stats (cont.)

• Hardware types– Desktops: 67%

– Laptops: 19%

– Netbooks: 6%

– Tablets: 4%

– Thin Clients: 4%


30

Guidelines for Purchasing New Hardware

• Inform current and future purchases

• Covering the vast majority of commercially available computers and tablets

• Full guidelines: http://www.smarterbalanced.org/smarter-balanced-assessments/technology


California Work with SBAC: Opportunities for Teacher

Involvement• Feedback on draft content and item specifications• Writing and reviewing of test items and tasks

(2012-13 pilot test; 2013-14 field test)• Range-finding and score validation• Collaborate on designing score reports and Web

tools for digital libraries (2013-14)• Scoring of performance tasks (2014-15 and

beyond)• Nominations for determining Achievement Level

Descriptors (ALD)

31


Current CDE Participation in SBAC Work Groups

32

Group Work Groups InvolvedGroup 1 Item Development + Performance Tasks

Group 2 Technology + Reporting

Group 3 Test Design + Validation and Psychometrics

Group 4 Formative Assessment Practices and Professional Learning + Transition to CCSS

Group 5 Accessibility and Accommodations + Test Administration

Deb Sigman – Executive Committee


Pilot Test

• Pilot Test in early 2013. • More than 10,000 items and

performance tasks • Open to all schools in the

Consortium. • Two million students • Equal percentage of students from

all Governing States33


SBAC Resources• California Department of Education/SBAC

presentations be found on the CDE SBAC Web page at http://www.cde.ca.gov/ta/tg/sa/smarterbalanced.asp

• Join the SBAC CDE electronic mailing list by sending a blank e-mail to: [email protected]

34


35

Assessments to Consider (by content area and grade level)



California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to:•Develop recommendations addressing 16 areas toward reauthorization of the assessment system•Develop a plan for transitioning to new “high quality assessments”•Consult with specific stakeholders:

• The State Board of Education• Public Schools Accountability Act (PSAA) Committee• Measurement experts from California’s private and public universities• Experts in assessing students with disabilities and English learners• Teachers, administrators, and governing boards from California’s local

educational agencies• Parents

•Recommendations and Plan due to the Legislature fall 2012

36


Statewide Assessment Reauthorization Timeline

37

March 21–22: Work Group

Meeting

March 21–22: Work Group

Meeting

April 17–18: Work Group

Meeting

April 17–18: Work Group

Meeting

May 22–23: Work Group

Meeting

May 22–23: Work Group

Meeting

June 12–14: Work Group

Meeting

June 12–14: Work Group

Meeting

July 25–26: Work Group

Meeting

July 25–26: Work Group

Meeting

Regional Public Meetings: 5 meetings between

March and April

Regional Public Meetings: 5 meetings between

March and April

Launch of Online Survey

July 5

Launch of Online Survey

July 5Focus Groups:

July - September

Focus Groups: July -

September

Recommendations to the Legislature in

fall 2012

Recommendations to the Legislature in

fall 2012

September 6: Work Group

Meeting

September 6: Work Group

Meeting

March 2012

April 2012

May 2012

June 2012

July 2012

August 2012

September 2012

October 2012

November 2012

Online Survey closed

September 4

Online Survey closed

September 4

State Board of Education Meeting

State Board of Education Meeting

Report writing and review

Report writing and review

E-mail Account opened

E-mail Account closed


Common Developing Themes

• Diagnostic assessments: ELA and Math in grade 2 and grades 3-12 as needed.

• Formative Assessment Tools: – SBAC: ELA and Math– Explore the state providing formative tools and processes for science

and History/Social-Science

• Interim assessments:– SBAC: ELA and Math Grades 3-8, 11– ELA and Math for all students in grade 2, 9-10– Science/HSS – state provided computerized interim assessments

similar to summative

• Summative: – SBAC assessments in grades 3-8 and 11 (ELA and Math). – Summative assessments for science and HSS in grades 3-12, for ELA

and math in grade 12, and writing in grades 3-12

38


Common Themes (Cont.)

• Matrix sampling in some grades/subjects, specifically science and history–social science (HSS)

• Reduce linguistic complexity of assessments to more accurately measure what students know and can do

• Items that evaluate critical thinking, problem solving, communication, collaboration, creativity, and innovation

• Other measures: student engagement survey, parent survey, graduation rates, attendance

• Multiple measures: within classroom, (e.g., tests, quizzes, projects, essays, journaling, class work, portfolios)

• Using results to satisfy the high school graduation requirement

39


40

SBAC Timeline

Field testing of summative

assessment, training school-

and district-level staff in formative

tools

Field testing of summative

assessment, training school-

and district-level staff in formative

tools

Teams of teachers evaluate formative

assessment practices and

curriculum resources

Teams of teachers evaluate formative

assessment practices and

curriculum resources

Full implementatio

n of assessment

system

Full implementatio

n of assessment

system

Formative Processes, Tools, and Practices Development Begins

Formative Processes, Tools, and Practices Development Begins

Pilot Testing of Summative and Interim Items/Tasks Conducted

Pilot Testing of Summative and Interim Items/Tasks Conducted

Writing and Review

Items/Tasks for Field Testing

(throughout the school year)

Writing and Review

Items/Tasks for Field Testing

(throughout the school year)

Technology readiness

tool available

Technology readiness

tool available

Formative tools available

to teachers

Formative tools available

to teachers

Content and Item

Specifications Development

Content and Item

Specifications Development

Writing and Review of Pilot Items/Tasks (including Cognitive

Labs and Small-Scale Trials)

Writing and Review of Pilot Items/Tasks (including Cognitive

Labs and Small-Scale Trials)


Contact Information• Patrick Traynor, Director

Assessment Development and Administration Division – E-mail: [email protected]

• Jessica Barr, Reauthorization Lead ConsultantAssessment Development and Administration Division– Phone: 916-319-0364– E-mail: [email protected]

• Kristen Brown, SBAC Lead Consultant Assessment Development and Administration Division

– Phone: 916-319-0334– E-mail: [email protected]

• Reauthorization E-Mail Account – [email protected]

41

transitioning to new assessments

Documents