Design frameworkItem developmentCognitive complexitySample items

ELA ITEM DESIGN

4 BUILDING BLOCKS

Learning Progressio

n

Item Design

Outcomes/Scoring

Assessment Quality

BEAR Assessment System Step 2A match between what is taught and what is

assessedConstructive alignment aligning teaching and

assessment to the learning outcomes/standards (Biggs, 2003)

Proposed items are located along the LP map

ITEM DESIGN FRAMEWORK

Learning Outcomes/Standard

sAssessme

nt Task

Teaching &

Instruction

One Framework(Wilson & Sloane, 2000)

Item design framework used by Smarter-Balanced Assessment under the evidence-centered design approach (Mislevy, Steinberg, & Almond, 2003)

Defined as the degree to which expectations and assessments work together to improve and measure students learning

ALIGNMENT FRAMEWORK

1. Categorical concurrence• Commonality between the content categories of the

standards and those of the assessment items2. Range of knowledge correspondence• Number of objectives within each standard covered

by item(s)3. Balance of representation

• Relative coverage of content categories by items in a test

4. Depth of Knowledge consistency• Match between the cognitive demand of items and

the level of cognitive demand communicated by the wording of the objectives

4 CRITERIA TO DETERMINE THE DEGREE OF ALIGNMENT

Universal designDesign item that accurately assess the targeted competency for all students

Ensure item fairness – make sure that items are equally difficult for groups of equal ability (e.g. males and females; urban and rural)

Vocabulary & languageuse content-specific language appropriate to the assessed grade

For non-content-specific material, use vocabulary/language from previous grade levels

ITEM DEVELOPMENT

Grade appropriateness Design items that assess a primary content

domain/standard of the appropriate grade “For non-reading items, the reading level is

approximately one grade level below the grade level of the test, except for specifically assessed content terms or concepts” (SBAC, 2012)

Using items to link tests For pre-post test designs, include some items that

appeared on previous test(s) to measure student progress If the time between tests is relatively long (i.e. 2-3

months), the same test can be used If the time is short (i.e. 2-3 weeks), pick a few

items to reuse and include new ones

ITEM DEVELOPMENT

Use of Modified Bloom’s TaxonomyDefinition

An example of Cognitive Rigor Matrix (Hess, et al., 2009)

Demonstration on how to align standards and proposed item(s) on the LP map

COGNITIVE COMPLEXITY

Modified by Anderson & Krathwohl (2001)

MODIFIED BLOOM’S TAXONOMY

Old Bloom’s Taxonomy (Bloom, 1956)

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

COGNITIVE RIGOR MATRIX

SAMPLE ITEMS FOCUS ON APPLICATION PROCESS

Process Level 1: Recall & Reproduction

Level 2: Skills & Concepts

Level 3: Strategic Thinking/Reasoning

Level 4: Extended Thinking

Apply:Carry out or use a procedure in a given situations; carry out (apply to a familiar task) or use (apply) to an unfamiliar task.

Follow simple/routine procedures.Apply an algorithm or formula.

Solve routine problems applying multiple concepts or decision points.

Use or apply concepts to solve non-routine problems.

Select or devise an approach among many alternatives to solve a novel problem.

SAMPLE ITEMS

Intended levels: Grade 6,7,8 Intended Claim: Claim #1

Students can read closely and analytically to comprehend a range of increasingly complex literary and informational texts.

Strand: Reading Literary TextAnchor Domain: Craft & StructureAnchor Standards:

Interpret words and phrases as they are used in a text, including determining technical, connotative, and figurative meanings, and analyze how specific word choices shape meaning or tone.

Standards: 8.RL.4, 7.RL.4, 6.RL.4Assessment Target: Target #7 on Language use

SAMPLE ITEMS’ SPECIFICATION

  Grade 6 Grade 7 Grade 8DOK 3 2 3

Grade-specific Standard:

RL-4

Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of a specific word choice on meaning and tone.

Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of rhymes and other repetitions of sounds (e.g., alliteration) on a specific verse or stanza of a poem or section of a story or drama.

Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of specific word choices on meaning and tone, including analogies or allusions to other texts.

Assessment Standards for

Target #7 – Language Use

Interpret figurative language use (e.g., personification, metaphor), literary devices, or connotative meanings of words and phrases used in context and their impact on reader interpretation

Interpret impact or intent of figurative language use (e.g., alliteration, onomatopoeia, imagery), literary devices (e.g., flashback, foreshadowing), or connotative meanings of words and phrases used in context and their impact on reader interpretation

Determine or interpret impact or intent of figurative language/literary devices or connotative meanings of words and phrases used in context and the impact of those word choices on meaning and tone.

SAMPLE ITEM GRADE 8

Stimulus Text: Below is an excerpt from American Indian Stories. As you read the excerpt, consider how the author’s word choice develops certain aspects of the narrator’s experience and then answer the questions that follow.  

“The Legends”From American Indian Stories

By Zitkala-SaAt the arrival of our guests I sat close to my mother, and did not leave her side without first asking her consent. I ate my supper in quiet, listening patiently to the talk of the old people, wishing all the time that they would begin the stories I loved best. At last, when I could not wait any longer, I whispered in my mother's ear, "Ask them to tell an Iktomi story, mother." Soothing my impatience, my mother said aloud, "My little daughter is anxious to hear your legends." By this time all were through eating, and the evening was fast deepening into twilight. As each in turn began to tell a legend, I pillowed my head in my mother's lap; and lying flat upon my back, I watched the stars as they peeped down upon me, one by one. The increasing interest of the tale aroused me, and I sat up eagerly listening to every word. The old women made funny remarks, and laughed so heartily that I could not help joining them. The distant howling of a pack of wolves or the hooting of an owl in the river bottom frightened me, and I nestled into my mother's lap. She added some dry sticks to the open fire, and the bright flames leaped up into the faces of the old folks as they sat around in a great circle.  

SAMPLE ITEM GRADE 8, CONT’D

LP Standard SBAC Standard Cognitive Rigor

8.RL.4. Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of specific word choices on meaning and tone, including analogies or allusions to other texts.

Determine or interpret impact or intent of figurative language/literary devices or connotative meanings of words and phrases used in context and the impact of those word choices on meaning and tone.

Analyze, Level 3 (Analyze author’s craft/concept)

SAMPLE ITEM GRADE 7

The Forsaken MermanBy Matthew Arnold Come, dear children, let us away;Down and away below!Now my brothers call from the bay,Now the great winds shoreward blow,Now the salt tides seaward flow;Now the wild white horses play,Champ and chafe and toss in the spray.Children dear, let us away!This way, this way! Call her once before you go—Call once yet!In a voice that she will know:"Margaret! Margaret!"Children's voices should be dear(Call once more) to a mother's ear;

Children's voices, wild with pain—Surely she will come again!Call her once and come away;This way, this way!"Mother dear, we cannot stay!The wild white horses foam and fret."Margaret! Margaret! Come, dear children, come away down;Call no more!One last look at the white-wall'd townAnd the little grey church on the windy shore,Then come down!She will not come though you call all day;Come away, come away!

Children dear, was it yesterdayWe heard the sweet bells over the bay?In the caverns where we lay,Through the surf and through the swell,The far-off sound of a silver bell?Sand-strewn caverns, cool and deep,Where the winds are all asleep;Where the spent lights quiver and gleam,Where the salt weed sways in the stream,Where the sea-beasts, ranged all round,Feed in the ooze of their pasture-ground;Where the sea-snakes coil and twine,Dry their mail and bask in the brine;Where great whales come sailing by,Sail and sail, with unshut eye,Round the world for ever and aye?When did music come this way?Children dear, was it yesterday?

Stimulus text: This poem is spoken in the voice of a male mermaid, whose wife has decided to leave the ocean and return to life on land.

SAMPLE ITEM GRADE 7, CONT’D

LP AND COGNITIVE RIGOR

LP Standard SBAC Standard Cognitive Rigor

7.RL.4. Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of rhymes and other repetitions of sounds (e.g., alliteration) on a specific verse or stanza of a poem or section of a story or drama.

Interpret impact or intent of figurative language use (e.g., alliteration, onomatopoeia, imagery), literary devices (e.g., flashback, foreshadowing), or connotative meanings of words and phrases used in context and their impact on reader interpretation

Analyze, Level 2 (Identify use of literary devices)

SAMPLE ITEM GRADE 6

The highlighted sentence from Summer on Wheels includes a literary device• What does the literary device mean?• Why did the author most likely select the literary device for this

description?Write a 2-3 sentence answer responding to these questions

LP AND COGNITIVE RIGOR

LP Standard SBAC Standard Cognitive Rigor

6.RL.4. Determine the meaning of words and phrases as they are used in a text, including figurative and connotative meanings; analyze the impact of a specific word choice on meaning and tone.

Interpret figurative language use (e.g., personification, metaphor), literary devices, or connotative meanings of words and phrases used in context and their impact on reader interpretation.

Analyze, Level 3 (Analyze author’s craft, viewpoint, or potential bias)

Bloom, B . S . (1956) . Taxonomy o f educat iona l ob ject ives . Handbook I : The Cogn i t ive Domain . New York , NY: Dav id McKay Co.

Anderson , L . W. , & Krathwohl , D . (2001) . A Taxonomy for Learn ing , Teach ing , and Assess ing : A Rev is ion o f B loom's Taxonomy o f Educat iona l Ob ject ives . New York , NY: Longman.

Common Core Standards In i t ia t ive . (2014) . Common core s tandards fo r Eng l i sh language ar ts & l i te racy in h i s tory /soc ia l s tud ies , sc ience, and techn ica l sub jects . Retr ieved f rom http : / /www.corestandards .org /EL A-L i teracy /

Hess , K . , Car loc , D . , Jones , B . , & Walkup, J . , (2009 ) . What exact ly do “ fewer, c learer, and h igher s tandards” rea l ly look l i ke in the c lassroom? Us ing a cogn i t ive r igor matr ix to ana lyze curr i cu lum, p lan lessons , and implement assessments . Paper presented at CCSSO, Detro i t , M ich igan .

Ni tko , A. J . , & Brookhart , S . (2007) . Educat iona l assessment o f s tudents . Upper Sadd le R iver , N J : Pearson Educat ion , Inc .

McMi l lan , J . H. (2007) . Classroom assessment . P r inc ip les and pract ice fo r eff ect ive s tandard-based inst ruct ion (4 th ed . ) . Boston : Pearson - A l lyn & Bacon .

Oregon Department o f Educat ion . (2014, June) . Assessment gu idance . (2014) .CC SS Too lk i t : ELA /L i teracy C la im 1 Sample Summat ive I tems (6 -8) . Retr ieved f rom

http : / /www.ode.s ta te .o r.us /search /page/? id=3714

Webb, N . (2007) . Al ign ing assessments and s tandards . Retr ieved f rom http : / /www.wcer.w isc .edu/news/coverStor ies /a l ign ing_assessments_and_standards .php

Wi lson , M. (2005) . Construct ing measures : An i tem response mode l ing approach . New York , NY: Psycho logy Press , Tay lor & Franc is Group.

Wi lson , M. , & S loane, K . (2000) . From pr inc ip les to pract ice : An embedded assessment system. Appl ied Measurement in Educat ion , 13 (2 ) , pp . 181-208.

Smarter Ba lanced Assessment Consort ium. (2012, Apr i l ) . Genera l i tem spec ifi cat ions . CCSS

BIBLIOGRAPHY

I tem Design (ELA) PPT by the Oregon Department of Educat ion and Berkeley Evaluat ion and Assessment Research Center is l icensed under a CC BY 4.0.

You are free to: Share — copy and red i s t r i bu te the mater ia l i n any med ium o r fo rmat Adapt — remix , t rans fo rm, and bu i l d upon the mater ia l

Under the fo l lowing terms: Attr ibut ion — You mus t g ive approp r ia te c red i t, p rov ide a l i nk to the l i cense , and

ind i ca te i f changes were made. You may do so in any reasonab le manner , bu t no t i n any way tha t sugges ts the l i censo r endo rses you o r you r use .

NonCommerc ia l — You may no t use the mater ia l fo r commerc ia l pu rposes. ShareAl ike — I f you remix , t rans fo rm, o r bu i l d upon the mater ia l , you mus t d i s t r i bu te you r

con t r ibu t i ons under the same l i cense as the o r ig ina l .

Oregon Department of Educat ion welcomes edit ing of these resources and would great ly appreciate being able to learn from the changes made. To share an edited version of this resource, please contact Cr isten McLean, cr isten.mclean@state.or.us.

CREATIVE COMMONS LICENSE