Where are we now? – AB 484 Where are we headed? – Smarter Balanced

SEC. 15. Section 60640 of the Education Code is amended to read:

60640.(a) There is hereby established the Measurement of Academic Performance and Progress, to be known as the MAPP.

(b) Commencing with the 2013–14 school year, the MAPP shall be composed of all of the following:

(1) (A) A consortium summative assessment in English language arts and mathematics for grades 3 to 8, inclusive, and grade 11 that measures content standards adopted by the state board.

(B) In the 2013–14 school year, the consortium summative assessment in English language arts and mathematics shall be a field test only, to enable the consortium to gauge the validity and reliability of these assessments and to conduct all necessary psychometric procedures and studies, including, but not necessarily limited to, achievement standard setting, and to allow the department to conduct studies regarding full implementation of the assessment system. These field tests and results shall not be used for any other purpose, including the calculation of any accountability measure.

The state still needs to receive a waiver from the US Dept. of Ed. to enact it’s provisions.

Development of a Next-Generation Assessment System

How do we know from a student’s performance on a test that they are advancing toward readiness for college and career?

The Smarter Balanced assessments are being developed using the principles of Evidence-Centered Design (ECD)

Three basic elements of ECD are:

1. Stating the claims to be made about test takers, 2. Deciding on the evidence that is required to

support the claims, and 3. Administering the test items that provide the

required evidence.

The result is tighter linkage between test performance and the claims to be made about test takers.

At each grade level within ELA/literacy and mathematics, there is one overall claim encompassing the entire content area and four specific content claims.

Overall Claim – “Students can demonstrate progress toward college and career readiness in English-language arts and literacy.”

Overall Claim – “Students can demonstrate progress toward college and career readiness in mathematics.”

Students will receive a score on each overall claim and scores for the specific content claims.

Overall and specific content claims are delineated into four achievement levels:

thorough understanding to… adequate understanding to… partial understanding to… minimal understanding to…

Items and the collection of items MUST measure deeper understanding and application of concepts.

Therefore, a variety of item types are

necessary. ◦ selected response, short constructed response,

extended constructive response, technology enhanced, and performance tasks.

UCLA’s National Center for Research on Evaluation, Standards, and Student Testing (CRESST) is monitoring the extent to which the two consortia’s (SBAC & PARCC) assessment development efforts are likely to produce tests that measure and support goals for deeper learning.

“Study results indicate that Smarter Balanced and PARCC summative assessments are likely to represent important goals for deeper learning, particularly those related to mastering and being able to apply core academic content and cognitive strategies related to complex thinking, communication, and problem solving.”

The methodology used for defining “Deeper Learning” draws on Norman Webb’s Depth of Knowledge (DOK) classification scheme.

Webb’s system categorized DOK into four

levels

DOK 1: Recall of a fact, term, concept, or procedure; basic comprehension

DOK 2: Application of concepts and/or procedures involving some mental processing

DOK 3: Applications require abstract thinking, reasoning, and/or more complex inferences

DOK 4: Extended analysis or investigation that requires synthesis and analysis across multiple contexts and non-routine applications

Preliminary analysis of SBAC Content Specifications for English Language Arts

CRESST researchers reviewed the assessment targets within each claim and the DOK levels that are specified for each target.

It is of note that there are two or more DOK levels

associated with many of the individual assessment targets.

Our analysis assumes that the number of assessment

targets within each claim provides a gross indicator of the relative emphasis that claim might receive on the end-of-year assessment.

Preliminary analysis of SBAC Content Specifications for English Language Arts

Overall, DOK2 and DOK3 receive the most emphasis, being designated for 46% and 43% respectively of the total assessment targets. DOK1 is designated for about a third of the targets and DOK4 for a quarter of the targets. It is important to keep in mind that multiple DOK levels can be designated for each target and that fidelity of item development, final blueprint rules, and the nature of Smarter Balanced end-of-year performance tasks, will determine the actual distribution of targets and DOK levels that will be administered to students.

Preliminary analysis of SBAC Content Specifications for MATHEMATICS

In contrast to ELA, DOK levels for mathematics appear to differ by claim. For Claim 1, virtually all targets are to be assessed at DOK1 and/or DOK2, and essentially no targets are at DOK3 and DOK4. For Claim 2, DOK1 receives relatively less attention, and there is a corresponding increase in emphasis on DOK3, which is specified for half the targets. For Claims 3 and 4, DOK3 gets the most attention and DOK4 is specified for more than 40% of the targets

According to a recent RAND study (Yuan & Le, 2012) using 2009-10 student enrollment data, it was estimated that:

◦ 0% of students in the US were assessed on deeper learning in mathematics through state tests.

◦ 16% of students were assessed on deeper learning in reading, and

◦ 2-3% were assessed on deeper learning in writing.

Overall, 3-10% of US elementary and secondary students were assessed on deeper learning on at least one state assessment.

New York and Kentucky were the first two states to test their students under stringent Common Core learning standards ◦ 31% of New York students in grades 3-8 met or

exceeded math and English competency standards. A drop of 34% in math and 24% in English.

◦ In Kentucky, the ratio of elementary and middle-

school students score proficient or above declined by about a third.

Purpose: to give students, parents, teachers, administrators the opportunity to become familiar with the online testing environment.

Additionally, assessment items give a glimpse of both the complexity of CCSS and how performance on these standards will be operationalized.

Should NOT be used to guide instructional decisions regarding individual students.

A practice test is available in ELA and math for each of grades 3-8.

Approximately 30 questions per test.

Tests provide a PREVIEW, but do not encompass the full range of content that students may encounter.

No secure login or password required Work cannot be saved; items are NOT scored Information and link to the Practice Tests available

at http://sbac.portal.airast.org/practice-test/ Scoring guides are available through the

“Resources and Documentation” link on the practice test website (see handout).

Additional summative sample items - http://www.ode.state.or.us/search/page/?id=3747