the design phase: using evidence-centered assessment design monty python argument
TRANSCRIPT
The Design Phase:Using Evidence-Centered
Assessment Design
Monty Python argument
Workshop Flow
• The construct of MKT– Gain familiarity with the construct of MKT– Examine available MKT instruments in the field
• Assessment Design– Gain familiarity with the Evidence-Centered Design approach– Begin to design a framework for your own assessment
• Assessment Development– Begin to create your own assessment items in line with your
framework• Assessment Validation
– Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments
The PADI Project – padi.sri.com
What is Evidence-Centered Design?
• Approach developed by ETS• Uses “evidentiary reasoning” to design the underlying
principles of an assessment• Answers these questions:
–What complex of KSAs should be assessed?–What types of evidence would we need to show a test-taker
has these KSAs?–What kinds of assessment items would allow us to gather this
evidence?• Provides a set of tools for structuring a systematic
approach to doing this
Phases of ECD
1) Domain AnalysisBig-picture narrative of the knowledge domain
2) Domain ModelingOrganization of the information in the domain in terms of:
• The aspect of proficiency of the test-taker in the domain• The kinds of things the test-taker might do to provide evidence of
their proficiency• The kinds of situations that might make it possible to provide such
evidence
3) Conceptual Assessment FrameworkBlueprint for the actual assessment that takes all of this into
account
4) The Assessment
Phases of ECD for Assessing MKT
1) Domain Analysis of MKTBig-picture narrative of the
knowledge domain
MKT• Interpreting unconventional forms or
representations• Choosing problems and examples that can
illustrate key curricular ideas• Differentiating between colloquial and
mathematical uses of language• Linking precise aspects of representations• Understanding implications of models and
representations• Evaluating mathematical statements
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Phases of ECD for Assessing MKT
2) Domain Modeling Organization of the information in the domain in terms of:
• The aspect of proficiency of the test-taker in the domain• The kinds of things the test-taker might do to provide evidence of
their proficiency• The kinds of situations that might make it possible to provide such
evidence
Create a “Design Pattern”
Phases of ECD for Assessing MKT
3) Conceptual Assessment FrameworkBlueprint for the actual assessment that takes all of this into
account
Teacher Model
Types of Relationships to Curriculum/PD
AssessmentDevelopment
Curriculum/PDDevelopment
AssessmentDevelopment
Curriculum/PDDevelopment
AssessmentDevelopment
Curriculum/PDDevelopment
Adapting Off-the-Shelf Instruments
• Start with this process for your own assessment needs
• Evaluate the fit of the instrument for your needs
• Important: Do not simply mix and match!
Attributes of a Design Pattern
Framing info • Title, Summary, Rationale
What the test-taker should know
• Focal Knowledge, Skills, and Abilities
Evidence we can collect to show they know it
• Potential observations• Potential work products• Potential rubrics
Kinds of situations that can evoke this evidence
• Characteristic features• Variable features
Other info • Exemplar items, Online resources, References, Misc
Examples of Design Patterns
• MKT for SimCalc
• Co-construct as a group
• Do your own
Foundations of proportionality
Proportionality includes: linearity, rate, function, slope in graphs, interpreting tables with an underlying rate
€
3
5=a
15
€
y = .6x
Find the missing number
Consider the function
NCTM Curriculum Focal Points
Grade 7: Developing an understanding of and applying proportionality, including similarity.
Students extend their work with ratios to develop an understanding of proportionality that they apply to solve single and multistep problems in numerous contexts. They use ratio and proportionality to solve a wide variety of percent problems, including problems involving discounts, interest, taxes, tips, and percent increase or decrease. They also solve problems about similar objects (including figures) by using scale factors that relate corresponding lengths of the objects or by using the fact that relationships of lengths within an object are preserved in similar objects. Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships, including inverse proportionality (xy = k, or y = k/x).
Grade 8: Analyzing and representing linear functions and solving linear equations and systems of linear equations.
Students use linear functions, linear equations, and systems of linear equations to represent, analyze, and solve a variety of problems. They recognize a proportion (y/x = k, or y = kx) as a special case of a linear equation of the form y = mx + b, understanding that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin. Students understand that the slope (m) of a line is a constant rate of change, so if the input, or x-coordinate, changes by a specific amount, a, the output, or y-coordinate, changes by the amount ma. Students translate among verbal, tabular, graphical, and algebraic representations of functions (recognizing that tabular and graphical representations are usually only partial representations), and they describe how such aspects of a function as slope and y-intercept appear in different representations. Students solve systems of two linear equations in two variables and relate the systems to pairs of lines that intersect, are parallel, or are the same line, in the plane. Students use linear equations, systems of linear equations, linear functions, and their understanding of the slope of a line to analyze situations and solve problems.
Students graph proportional relationships and identify the unit rate as the slope of the related line. They distinguish proportional relationships (y/x = k, or y = kx) from other relationships,
including inverse proportionality (xy = k, or y = k/x).
They recognize a proportion as a special case of a linear equation of the form y = mx + b, understanding
that the constant of proportionality (k) is the slope and the resulting graph is a line through the origin.
Students translate among verbal, tabular, graphical, and algebraic representations of functions.
KSAs
• Common Content Knowledge• Proportionality concepts
• Specialized Content Knowledge• Interpreting unconventional forms or representations• Choosing problems and examples that can illustrate key
curricular ideas• Differentiating between colloquial and mathematical uses of
language• Linking precise aspects of representations• Understanding implications of models and representations• Evaluating mathematical statements
Workshop Flow
• The construct of MKT– Gain familiarity with the construct of MKT– Examine available MKT instruments in the field
• Assessment Design– Gain familiarity with the Evidence-Centered Design approach– Begin to design a framework for your own assessment
• Assessment Development– Begin to create your own assessment items in line with your
framework• Assessment Validation
– Learn basic tools for how to refine and validate an assessment • Plan next steps for using assessments
Activity #2Create a Design Pattern
• Find Activity #2 in your binder• Pick a design pattern topic• Work on your own, with a partner, or with a small group• Complete the Design Pattern form• Feedback / Review Process• Discussion to follow
– Show-and-tell of 2 or 3 Design Patterns– Your insights, questions, challenges
Some Useful References(all available on the web)
• Baxter, G. P., & Mislevy, R. J. (2005). The Case for an Integrated Design Framework for Assessing Science Inquiry. PADI Technical Report 5. Menlo Park, CA: SRI International.
• Embretson, S. E. (Ed) (1985). Test Design: Developments in psychology and psychometrics. New York: Academic Press, Inc.
• Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to Evidence-Centered Design. CRESST Technical Paper Series. Los Angeles, CA: CRESST.
• Mislevy, R. J, Hamel, L. et al. (2003). Design Patterns for Assessing Science Inquiry. PADI Technical Report 1. Menlo Park, CA: SRI International.
• PADI Website: http://padi.sri.com
Feedback
• What worked?
• What didn’t work?
• What do you still want to learn?
Q1. Definition of MKT
“By mathematical knowledge for teaching, we mean the mathematical knowledge used to carry out the work of teaching mathematics. Examples of this “work of teaching” include explaining terms and concepts to students, interpreting students’ statements and solutions, judging and correcting textbook treatments of particular topics, using representations accurately in the classroom, and effects of teachers’ mathematical knowledge on student achievement providing students with examples of mathematical concepts, algorithms, or proofs” (Hill, Rowan, & Ball, 2005).“We look across mathematical knowledge needed for or used in teaching, including “pure” content knowledge as taught in secondary, undergraduate, or graduate mathematics courses; pedagogical content knowledge and curricular knowledge (also described by Shulman) both possibly taught in mathematics methods courses; and what is more elusive, knowledge that, while also mathematical, is not typically taught in undergraduate mathematics courses and is not be entirely pedagogical. Mathematical knowledge for teaching, keeping the emphasis on mathematics and acknowledging that teachers may know and use mathematics that is different from what is required for other professions” (Ferrini-Mundy, et. al., 2008)
Question 2
Teacher’s attitudes about being evaluated
Q3. Adapting Off-the-Shelf Instruments
• Start with this process for your own assessment needs
• Evaluate the fit of the instrument for your needs
• Important: Do not simply mix and match!