making sense of data from complex assessments
DESCRIPTION
Making Sense of Data from Complex Assessments. Robert J. Mislevy University of Maryland Linda S. Steinberg & Russell G. Almond Educational Testing Service FERA November 6, 2001. Buzz Hunt, 1986:. How much can testing gain from modern cognitive psychology? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/1.jpg)
FERA 2001 Slide 1November 6, 2001
Making Sense of Data fromComplex Assessments
Robert J. MislevyUniversity of Maryland
Linda S. Steinberg & Russell G. AlmondEducational Testing Service
FERA
November 6, 2001
![Page 2: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/2.jpg)
FERA 2001 Slide 2November 6, 2001
How much can testing gain from modern cognitive psychology?
So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion, then the gains to be
made are minimal.
Buzz Hunt, 1986:
![Page 3: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/3.jpg)
FERA 2001 Slide 3November 6, 2001
Opportunities for Impact Informal / local use Conceptual design frameworks
E.g., Grant Wiggins, CRESST Toolkits & building blocks
E.g., Assessment Wizard, IMMEX Building structures into products
E.g., HYDRIVE, Mavis Beacon Building structures into programs
E.g., AP Studio Art, DISC
![Page 4: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/4.jpg)
FERA 2001 Slide 4November 6, 2001
For further information, see...
www.education.umd.edu/EDMS/mislevy/
![Page 5: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/5.jpg)
FERA 2001 Slide 5November 6, 2001
Don Melnick, NBME:
“It is amazing to me how many complex ‘testing’ simulation systems have been developed in the
last decade, each without a scoring system.
“The NBME has consistently found the challenges in the development of innovative testing methods to lie primarily in the scoring arena.”
![Page 6: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/6.jpg)
FERA 2001 Slide 6November 6, 2001
The DISC Project
The Dental Interactive Simulations Corporation (DISC)
The DISC Simulator The DISC Scoring Engine Evidence-Centered Assessment Design The Cognitive Task Analysis (CTA)
![Page 7: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/7.jpg)
FERA 2001 Slide 7November 6, 2001
Evidence-centered assessment design
The three basic models
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
![Page 8: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/8.jpg)
FERA 2001 Slide 8November 6, 2001
What complex of knowledge, skills, or other attributes should be assessed?
(Messick, 1992)
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
Evidence-centered assessment design
![Page 9: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/9.jpg)
FERA 2001 Slide 9November 6, 2001
What complex of knowledge, skills, or other attributes should be assessed?
(Messick, 1992)
e Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
Student ModelVariables
Evidence-centered assessment design
![Page 10: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/10.jpg)
FERA 2001 Slide 10November 6, 2001
What behaviors or performances should reveal those constructs?
Evidence-centered assessment design
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
![Page 11: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/11.jpg)
FERA 2001 Slide 11November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Work product
Evidence-centered assessment design
![Page 12: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/12.jpg)
FERA 2001 Slide 12November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Work productObservable variables
Evidence-centered assessment design
![Page 13: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/13.jpg)
FERA 2001 Slide 13November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Observable variables
Evidence-centered assessment design
![Page 14: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/14.jpg)
FERA 2001 Slide 14November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What behaviors or performances should reveal those constructs?
Observable variables
Evidence-centered assessment design
Student ModelVariables
![Page 15: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/15.jpg)
FERA 2001 Slide 15November 6, 2001
What tasks or situations should elicit those behaviors?
Evidence-centered assessment design
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
![Page 16: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/16.jpg)
FERA 2001 Slide 16November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What tasks or situations should elicit those behaviors?
Stimulus Specifications
Evidence-centered assessment design
![Page 17: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/17.jpg)
FERA 2001 Slide 17November 6, 2001
Evidence Model(s) Task Model(s)
1. xxxxxxxx 2. xxxxxxxx3. xxxxxxxx 4. xxxxxxxx5. xxxxxxxx 6. xxxxxxxx
Student Model Stat model Evidence
rules
What tasks or situations should elicit those behaviors?
Work Product Specifications
Evidence-centered assessment design
![Page 18: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/18.jpg)
FERA 2001 Slide 18November 6, 2001
Implications for Student Model
SM variables should be consistent with …
The results of the CTA.
The purpose of assessment:
What aspects of skill and knowledge should be used to accumulate evidence across tasks, for pass/fail reporting and finer-grained feedback?
![Page 19: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/19.jpg)
FERA 2001 Slide 19November 6, 2001
Simplified Version of the
DISC Student Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Student Model 2
9/3/99,rjm
Simplified version of DISCstudent model
![Page 20: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/20.jpg)
FERA 2001 Slide 20November 6, 2001
Implications for Evidence Models
The CTA produced ‘performance features’ that
characterize recurring patterns of behavior and
differentiate levels of expertise.
These features ground generally-defined, re-usable
‘observed variables’ in evidence models.
We defined re-usable evidence models for recurring
scenarios for use with many tasks.
![Page 21: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/21.jpg)
FERA 2001 Slide 21November 6, 2001
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
InfoGathAss simplified
9/3/99,rjm
A simplified version of the EMfor InformationGathering
Procedures in the context ofAssessment
An Evidence Model
![Page 22: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/22.jpg)
FERA 2001 Slide 22November 6, 2001
Evidence Models: Statistical Submodel
What’s constant across cases that use the EM» Student-model parents.
» Identification of observable variables.» Structure of conditional probability relationships between
SM parents and observable children.
What’s tailored to particular cases» Values of conditional probabilities» Specific meaning of observables.
![Page 23: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/23.jpg)
FERA 2001 Slide 23November 6, 2001
Evidence Models: Evaluation Submodel
What’s constant across cases» Identification and formal definition of observable variables.» Generally-stated “proto-rules” for evaluating their values.
What’s tailored to particular cases» Case-specific rules for evaluating values of observables--
Instantiations of proto-rules tailored to the specifics of case.
![Page 24: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/24.jpg)
FERA 2001 Slide 24November 6, 2001
“Docking” an Evidence Model
Evidence ModelStudent Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
![Page 25: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/25.jpg)
FERA 2001 Slide 25November 6, 2001
“Docking” an Evidence Model
Evidence ModelStudent Model
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
Information gathering/Usage
Assessment
Adapting to situational constraints
Addressing the chief complaint
Adequacy of examination procedures
Adequacy of history procedures
Collection of essential information
Context
![Page 26: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/26.jpg)
FERA 2001 Slide 26November 6, 2001
Initial Status
Expert .28Competent .43Novice .28
All .33Some .33None .33
![Page 27: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/27.jpg)
FERA 2001 Slide 27November 6, 2001
Expert .39Competent .51Novice .11
All 1.00Some .00None .00
Status after four ‘good’ findings
![Page 28: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/28.jpg)
FERA 2001 Slide 28November 6, 2001
Expert .15Competent .54Novice .30
All .00Some .00None 1.00
Status after one ‘good’ and three ‘bad’ findings
![Page 29: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/29.jpg)
FERA 2001 Slide 29November 6, 2001
“Docking” another Evidence Model
Evidence ModelStudent Model
T rea tm en t P lan n in g
M ed ica l K n o w led g e
C o n tex t
A d eq u acy o f trea tm en t p ro ced u res
In d iv id u a liz a tio n o f p ro ced u res
E ffec t o f trea tm en t o n p a tien t
P e rfo rm an ce o f ex tran eo u s trea tm en t
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
![Page 30: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/30.jpg)
FERA 2001 Slide 30November 6, 2001
“Docking” another Evidence Model
Evidence ModelStudent Model
T rea tm en t P lan n in g
M ed ica l K n o w led g e
C o n tex t
A d eq u acy o f trea tm en t p ro ced u res
In d iv id u a liz a tio n o f p ro ced u res
E ffec t o f trea tm en t o n p a tien t
P e rfo rm an ce o f ex tran eo u s trea tm en t
Communality
Information gathering/Usage
Assessment
Evaluation
Treatment Planning
Medical Knowledge
Ethics/Legal
![Page 31: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/31.jpg)
FERA 2001 Slide 31November 6, 2001
Implications for Task Models
Task models are schemas for phases of cases,
constructed around key features that ... the simulator needs for its virtual-patient data base,
characterize features we need to evoke specified aspects of skill/knowledge,
characterize features of tasks that affect difficulty,
characterize features we need to assemble tasks into tests.
![Page 32: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/32.jpg)
FERA 2001 Slide 32November 6, 2001
Implications for Simulator
Once we’ve determined the kind of evidence we need as evidence about targeted knowledge, how must we construct the simulator to provide the data we need?
Nature of problems» Distinguish phases in the patient interaction cycle.» Use typical forms of information & control availability.
» Dynamic patient condition & cross time cases. Nature of affordances
» Examinees must be able to seek and gather data,» indicate hypotheses,» justify hypotheses with respect to cues, » justify actions with respect to hypotheses.
![Page 33: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/33.jpg)
FERA 2001 Slide 33November 6, 2001
Payoff
Re-usable student-model
» Can project to overall score for licensing
» Supports mid-level feedback as well
Re-usable evidence and task models» Can write indefinitely many unique cases using schemas
» Framework for writing case-specific evaluation rules
Machinery can generalize to other uses & domains
![Page 34: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/34.jpg)
FERA 2001 Slide 34November 6, 2001
Two ways to “score” complex assessments
THE HARD WAY:
Ask ‘how do you score it?’ after you’ve built the assessment and scripted the tasks or scenarios.
A DIFFERENT HARD, BUT MORE LIKELY TO WORK, WAY:
Design the assessment and the tasks/scenarios around what you want to make inferences about, what you need to see to ground them, and the structure of the interrelationships.
Part 2 Conclusion
![Page 35: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/35.jpg)
FERA 2001 Slide 35November 6, 2001
We can attack new assessment challenges by working from generative principles:Principles from measurement and evidentiary
reasoning, coordinated with... inferences framed in terms of current and
continually evolving psychology, using current and continually evolving technologies
to help gather and evaluate data in that light, in a coherent assessment design framework.
Grand Conclusion
![Page 36: Making Sense of Data from Complex Assessments](https://reader036.vdocuments.site/reader036/viewer/2022062314/5681486c550346895db57828/html5/thumbnails/36.jpg)
FERA 2001 Slide 36November 6, 2001