developing quality assessments

DEVELOPING QUALITY ASSESSMENTSPresented by Ellen Vorenkamp for the MI Assessment Consortium at Lewis Cass ISD Best of Formative Assessment Series Session 5

Overview The need for Quality Assessments

Accurate Data Use of Data

Student Growth Educator Evaluation

The Development Process MAC Modules Planning for your Assessment (CASL)

Key Components—Common Core Connections Test Blue Prints Quality Items

Conclusion

Good Assessment applies to all Learners

Regardless of student ability or communication level, our assessments must have a clear purpose and stretch our students to higher levels of doing and thinking

Good assessment design is fundamental to all

Each student is unique As ever – take the best and adapt to our

kids!

Essential Questions to Dialogue Around…

As you develop quality assessments, keep the following questions in mind… What is the purpose of the assessment? Who will use the information? How will it be used? Is the use formative or summative?

A Formative View As you develop quality assessments, keep

the following questions in mind… What is the purpose of the assessment?

To provide teachers immediate information on student learning

Who will use the information? Teachers and students in the classroom

How will it be used? To inform next steps in the learning process

Is the use formative or summative? Formative

A Summative View As you develop quality assessments,

keep the following questions in mind… What is the purpose of the assessment?

Educator Evaluation / Accountability Who will use the information?

Teachers and Administrators How will it be used?

To certify the learning process Is the use formative or summative?

Summative

Think…Pair…Share…What elements are necessary to ensure

quality assessments? List these qualities Discuss why these are important

MAC’s Thinking… 24 Modules that cover the entire

development process… Planning

Modules 1-7 Developing

Modules 8-20 Implementing

Module 21 Reflecting

Modules 21-24

Quality Assessments…

are Reliable and yield Valid data. In order for these two requirements to be met

assessment developers must pay special attention to the following: Standard/Item Alignment Balance of Representation Target-Method Match Quality Items

The best way to ensure your assessment is reliable and valid is to create a test blueprint and follow the blueprint while developing the assessment.

Test Blueprints Requires a bit of planning and

forethought as you begin this process… Several steps should be completed that

will help make developing your test blueprint easier

Are the assessment items tightly aligned with the standards?

Do the assessment items address the targeted standards sufficiently and at all levels of cognitive demand?

Are there an equal number of assessment items per standard? If not, is there “rationale”?

Are there enough assessment items per standard to determine mastery?

Standard Alignment

Items are balanced and should reflect what was taught and how it was taught within the classroom environment

The assessment as a whole should reflect the emphasis of the lesson, unit or other educational experience on which you are assessing.

For example, a social studies assessment might cover 4 related standards or learning targets…

Balance of Representation

Balance of Representation

33%

33%

34%

Lesson Instruction

2-E1.0.32-E1.0.12-E1.0.2

Balance of Representation

33%

33%

34%

Lesson Assessment

2-E1.0.32-E1.0.12-E1.0.2

1) Identify the Depth of Knowledge (DOK) for the standard

2) Match the cognitive demand of the standard with an appropriate item type

3) Determine which item types will be included in the assessment and in what proportion.

Target-Method Match

16

Depth Of Knowledge

SYNTHESIS AND EVALUATION“Putting together elements & parts to form

a whole, then making value judgments about the method.”

   EXTENDED THINKING Requires an investigation; time to think and process multiple conditions of the problem or task.

BLOOM’S TAXONOMY WEBB’S DOK

KNOWLEDGE “The recall of specifics and universals, involving little more than bringing to mind the appropriate material”

COMPREHENSION“Ability to process knowledge on a low level such that the knowledge can be reproduced or communicated without a verbatim repetition.”

APPLICATION“The use of abstractions in concrete situations.”

ANALYSIS“The breakdown of a situation into its component parts.”

RECALLRecall of a fact, information, or procedure (e.g., What are 3 critical skill cues for the overhand throw?)

SKILL/CONCEPT Use of information, conceptual knowledge, procedures, two or more steps, etc.

STRATEGIC THINKINGRequires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer

Wyoming School Health and Physical Education Network (2001).  Standards, Assessment, and Beyond.  Retrieved May 25, 2006, from http://www.uwyo.edu/wyhpenet

Prepared and published by the Educational Policy Improvement Center

Prepared and published by the Educational Policy Improvement Center

Reflection… What are the implications for classroom

instruction?

What are the implications for classroom assessment?

Types/Levels of Standards

Recall and Reproduction– (Knowledge) Facts and concepts we want students to know (Usually DOK Level 1 or 2)

Skills & Concepts – (Skills) Students use their knowledge and reasoning to act skillfully (Usually DOK Levels 2; Can go higher based on context)

Strategic Thinking– (Reasoning) Using what they know to reason and solve problems (Usually DOK Levels 2-4)

Extended Thinking– (Application/Products) use knowledge, reasoning, and skills to create a concrete product (Usually DOK 3-4)

Methods of Assessments

•Multiple Choice

•True-False•Matching

Selected Response

•Diagram•Fill-in-the-blank (words, phrases)

•Essay•Short answer (sentences, paragraphs)

•Web•Concept Map•Flowchart•Graph•Table•Matrix• Illustration

•Presentation

•Movement•Science lab •Athletic skill•Dramatization

•Enactment•Project•Debate•Model•Exhibition•Recital•Performance Task

•Oral questioning

•Observation• Interview •Conference•Process description

•Checklist•Rating scale• Journal sharing

•Thinking aloud a process

•Student self-assessment

•Peer review

ConstructedResponse

Performance Assessment

Observations/Conversations

Adapted from the work of Dr. Robert Marzano23

Target-Method MatchHow well does your method of assessment match your

target?

Target to be

Assessed

Assessment Method

Selected Response/

Short-Response

Extended -Response

Performance Assessment

Observation/Conversation

s

Knowledge/Recall Good match Good match Not a good

match Partial match

Reasoning/Strategic Thinking

Partial match Good match Good match Good match

Performance Skills

Not a good match

Not a good match Good match Partial match

Application/Products

Not a good match Partial match Good match Not a good

match

Putting it all together: Standard Specific Learning Target derived from

Standard Target DOK Level Assessment Method Number of Items/Points

Test Blueprint

Performance Tasks Performance tasks challenge students to apply

their knowledge and skills to respond to complex real-world problems. They can best be described as collections of questions and activities that are coherently connected to a single theme or scenario. These activities are meant to measure capacities such as depth of understanding, writing and research skills, and complex analysis, which cannot be adequately assessed with traditional assessment questions. The performance tasks may take one to two class periods to complete.

Components of a Performance Assessment Performance Assessments:

call upon the examinee to demonstrate skills and competencies (application)

are comprised of 4 basic components: A reason for the assessment

Identify the content and/or skill standard(s) to be assessed A specific performance to be evaluated

Construct the prompt or item and determine the criteria for success Exercises to elicit that performance, and

Describe the way you will collect evidence, when you will collect the evidence and how much evidence you will collect.

Systemic rating procedures Determine the score for proficiency or mastery; determine who will

rate the performance, scoring method (rubric, checklist, anecdotal record, portfolio)

SBAC Performance Tasks (PT) Performance tasks, the most complex of all items, include

the following elements: Integrate knowledge and skills across multiple claims. Measure capacities such as depth of understanding, research skills, and/or complex analysis with relevant evidence. Require student-initiated planning, management of information/data and ideas, and/or interaction with other materials. Reflect a real-world task and/or scenario-based problem Allow for multiple approaches. Represent content that is relevant and meaningful to students. Allow for demonstration of important knowledge and skills. Require scoring that focuses on the essence of the Claim(s) for which the task was written. Seem feasible for the school/classroom environment.

PT Sample: Prior to Test Date

PT Sample: Stimuli and Tasks

PT Sample: Stimuli and Tasks

PT Sample: Stimuli and Tasks

PT Sample: Stimuli and Tasks

PT Sample: Stimuli and Tasks

4th Grade ELA Sample

4th Grade ELA Sample

4th Grade ELA Sample

Additional Sample Items http://www.dese.mo.gov/divimprove/asse

ss/documents/asmt-sbac-math-hs-sample-items.pdf

Scroll down to page 3

Implications for the Classroom

Assessments will evolve to be more rigorous and real world relevant, must match our teaching to this same standard

Assessment data must be used in the moment to inform “next steps” in the learning process

Schools must have a balanced assessment system in place within their classrooms

Not the end of the journey… MAC Modules Much left to do to ensure quality

assessments

A Final Thought… “Remember, formative assessment

works,” says Popham. “When it is used, students learn better. By using this assessment-rooted instructional process, teachers can increase the test-based achievement of their students… ‘Student growth’ will be demonstrated on the tests because, in fact, student growth will have occurred.” “Formative Assessment’s ‘Advocatable

Moment’” by James Popham in Education Week, Jan. 9, 2013 (Vol. 32, #15, p. 29)

Questions and Discussion