robert j. mislevy & min liu university of maryland geneva haertel sri international robert j....
TRANSCRIPT
Robert J. Mislevy & Min Liu University of MarylandGeneva Haertel SRI International
Robert J. Mislevy & Min Liu University of MarylandGeneva Haertel SRI International
The Critical Role of Design Patterns in Large-Scale Assessment
The Critical Role of Design Patterns in Large-Scale Assessment
DR K-12 grant #0733172, “Application of Evidence-Centered Design to State Large-Scale Science Assessment.”
This material is based upon work supported by the National Science Foundation under Grant No. DRL- 0733172. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
DR K-12 grant #0733172, “Application of Evidence-Centered Design to State Large-Scale Science Assessment.”
This material is based upon work supported by the National Science Foundation under Grant No. DRL- 0733172. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Three design challenges Three design challenges Hard-to-assess standards in the
domain of interest e.g., inquiry science skills
Efficient and valid design and development of complex tasks e.g., scenarios, simulations
Accessibility of tasks for diverse learners varying perceptual and expressive
capabilities valid use of assistive technology,
modifications, alternative assessments
Design Patterns in Architecture
Design Patterns in Software Engineering
Design Patterns in Literature
Design PatternsDesign Patterns
Motivation for Assessment Design Patterns
Motivation for Assessment Design Patterns
In-between structure, to connect... Thinking about science learning & inquiry Technical elements of measurement &
delivery Narrative, not technical, contents Some Design Patterns from PADI
Model-Based Reasoning Model Formation; Evaluation; Model Revision; Use
Observational & Experimental Investigations Systems Thinking
Motivation for Assessment Design Patterns
Motivation for Assessment Design Patterns
They lay out a design space for developers Choices, connections, examples Things to be aware of (e.g., research
on Universal Design for Learning) Can improve both Efficiency +
Validity Attributes reflect assessment
argument structure
Assessment ArgumentsAssessment Arguments
What complex of
knowledge, skills, or other
attributes should be
assessed?
What behaviors or
performances should reveal
those constructs?
What tasks or situations
should elicit those
behaviors?
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
StudentModel
EvidenceModel
TaskModel
Mislevy, R.J., & Haertel, G. (2006). Implications for evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25, 6-20.
The Structure of Assessment Design Patterns
The Structure of Assessment Design Patterns
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
The Rationale provides background into the nature of the Focal KSAs, and the kinds of things that people do in what kinds of situations that evidence it. E.g., overview, research links, examples.
StudentModel
EvidenceModel
TaskModel
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
The design pattern is organized around Focal KSAs. They will be
involved in the Student Model, although there may be other KSAs
that are included in the target of inference (e.g., Model Revision—but what models, what context?).
Associated with Characteristic Features of Tasks.
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
Additional KSAs play multiple roles. You need to think about which ones you really DO want to include as targets of inference
(validity) and which ones you really DON’T (invalidity).
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
The Additional KSAs you DO want to include as targets of inference are part of the claim. E.g., knowing Mendel’s laws as well as being able to formulate a model in an investigation.Connected with Variable
Features of Tasks.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
The Additional KSAs you DON’T want to include as targets of inference introduce construct-irrelevant reasons for poor performance. (Especially important for assessing special populations – UDL & accommodations.)
Connected with Variable Features of Tasks & Work Products.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
The Characteristic Features of Tasks help you think about critical features of the tasks
situation you need to get evidence about the Focal KSAs.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
StudentModel
EvidenceModel
TaskModel
Variable Features of Tasks also help you think about data concerning the situation – but now to influence difficulty …
or to bring in or reduce demand for Additional KSAs to avoid alternative explanations.
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
Some Variable Features of Tasks help you match features of tasks and background / knowledge / characteristics of students: Interests, familiarity, previous instruction.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
Potential Work Products help you think about what you want to capture from a performance –product, process, constructed model, written explanation, etc.
Can also call attention to demand for Additional KSAs, & avoid alternative explanations for poor performance
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
Potential Observations are possibilities for the qualities of Work Products – i.e., the data concerning the performance.
StudentModel
EvidenceModel
TaskModel
ATTRIBUTE DESCRIPTION
Rationale How/why this DP addresses evidence about focal KSAs.
Focal Knowledge,
Skills, Abilities
The primary knowledge / skills / abilities (KSAs) targeted by this design pattern.
Additional KSAs
Other knowledge/skills/abilities that may be required by tasks.
Characteristic features
of tasks
Aspects of assessment situations that are needed to evoke evidence about the focal KSAs.
Variable features of
tasks
Aspects of assessment situations that can be varied to shift difficulty or focus.
Potential work
products
What students actually say, do, or make, to produce evidence.
Potential observation
s
Aspects of work products we might identify and evaluate, as evidence about students’ KSAs.
Potential rubrics
Ways of evaluating work products to produce values of observations.
How Design Patterns Support Thinking about the Assessment
Argument
How Design Patterns Support Thinking about the Assessment
Argument
And Potential Rubrics are algorithms/rubrics/rules for evaluating Work Products to interpret evidence from the
student’s performance.
StudentModel
EvidenceModel
TaskModel
Current Catalog of Design PatternsCurrent Catalog of Design Patterns
ECD/PADI related projects have produced over
100 Design Patterns
Domains include science inquiry, science content,
mathematics, economics, model-based reasoning
Span grades 3-16+
Organized around themes, models, and
processes, not surface features or formats of
tasks Support the design of scenario-based, multiple choice,
and performance tasks
Current Catalog of Design PatternsCurrent Catalog of Design Patterns
Subject Areas
Education
Standards
Unifying Themes/ Inquiry
Big Ideas within
Disciplines
Learning Progression
s
Language Proficiency
Total
Science 17 57 4 2 0 80
Mathematics 30 2 3 0 0 35
Economics 0 0 3 0 0 3
Language Arts
30 0 0 0 1 31
Management/ Business
0 7 3 0 0 10
Second Language Acquisition
0 0 0 0 3 3
Grand Total 77 66 13 2 4 162
For more information…For more information…
PADI: Principled Assessment Design for Inquiry
http://padi.sri.com Links to NSF & IES follow-on projects Lots of Tech Reports, interactive online
examples Bob Mislevy home page
http://www.education.umd.edu/EDMS/mislevy/ Links to papers on ECD Cisco applications
Now for the Good Stuff …
Now for the Good Stuff …
Examples of design patterns with content Different projects
Different grain sizes
Different users
How they are being used to tackle pervasive challenges of large-scale assessment.
How they evolved to suit needs of users Same essential structure, but
Representations, language, emphases, and affordances tuned to users and needs