c r e s s t / u c l a model-based assessment: why, what, how, how good, and what next? eva l. baker...
TRANSCRIPT
C R E S S T / U C L A
Model-Based Assessment: Why, What, How, How Good,
and What Next?
Eva L. Baker
National Research Council, Board on Testing and AssessmentBridging the Gap Between Classroom and Large-Scale Assessment Workshop
January 23-24, 2003Washington, DC
UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
C R E S S T / U C L A
Why? Assessment Knowledge: Usable and
UsefulUsable Knowledge
In a form that can be understood
In a form that can be applied
Timed appropriately
May cause rethinking of the problem
Useful Knowledge
Rethinking indicates a new solution path
Adapted to situation
Sufficient to guide solution
Improved outcomes occur as a result
C R E S S T / U C L A
Why Are Some Schools Successful in Using
Assessment Knowledge?
Focus on learning (students and adults)
Constant use of appropriate information (formal and informal)
Focus on feedback and change
Public display and exchange
Community pride in outcomes of students and place
C R E S S T / U C L A
Goals for CRESST Model-Based Assessment (MBA)
Assessment components share a common framework. MBA starts with thinking skills and applies them to content domains to support
Coherent, sustained learning
Spiral teaching-common language
Transfer (application to new situations)
Multipurpose
Learning organization
C R E S S T / U C L A
CRESST Model-Based Assessments (MBA)
Features Research based
Focus on cognition and learning
Abstracted in models based on key learning elements—principles guiding test design and instruction
Operationalized in templates
Reusable and cost-sensitive design/training/scoring
C R E S S T / U C L A
Model-Based AssessmentCognitive Families
ContentUnderstanding
ProblemSolving
Teamwork andCollaboration
MetacognitionCommunication
Learning
C R E S S T / U C L A
Model-Based Assessment Design
Models to templates (to specification) to tests
Template contains domain-independent (transfer) and domain-specific (strategy and knowledge) components
Templates that allow common domain-specific design approaches to be used, e.g., primary sources in history
Scoring requirements
C R E S S T / U C L A
Expert Model—Deep Understanding of Content
(Domain Independent)
Principles or themes (big ideas)
Use of prior knowledge
Explicit relationships
Avoiding misconceptions
Expert performance-based scoring
C R E S S T / U C L A
Specifications forLarge-Scale Use
Standards reference
Place in sequence
Content domain (what’s in and out)
Proportion of effort
Format options
Interpretation rules
Time
C R E S S T / U C L A
Template
Task(s)
Format(s)
Prompt(s) and requirements
Scoring
Directions
Sample
C R E S S T / U C L A
Three Templates for the Model of Deep Content
Understanding
Explanation
Explanation with explicit knowledge
Graphical representation of relationships
C R E S S T / U C L A
Deep Content Understanding
Primary source materials in each domain
Student required to integrate prior knowledge and principles to succeed
Scored by using expert model in subject matter
C R E S S T / U C L A
Content UnderstandingTemplate #1 Explanation
An array of primary source materials
A prompt that asks for an explanation in context
Constructed (written) answer
Evaluated by means of a scoring rubric that operationalizes learning model
C R E S S T / U C L A
Hawaiian History Writing
Assignment: BayonetConstitution
Be sure to show the relationships among your ideas and facts.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history, and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Imagine you are in a class that has been studying Hawaiian history. One ofyour friends, who is a new student in the class, has missed all the classes.Recently, your class began studying the Bayonet Constitution. Your friend isvery interested in this topic and asks you to explain everything that you havelearned about it.
Write an essay explaining the most important ideas you want your friend tounderstand. Include what you have already learned in class about Hawaiianhistory, and what you have learned from the texts you have just read. Whileyou write, think about what Thurston and Liliuokalani said about the BayonetConstitution, and what is shown in the other materials.
C R E S S T / U C L A
EXCERPTS from HAWAIIAN HISTORYPRIMARY SOURCE DOCUMENTS
LILIUOKALANI
For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they becamewealthy, their greed and their love of power increased. Although settledamong us, and drawing their wealth from resources, they were alien to usin their customs and ideas, and desired above all things to secure their own personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdomhighly. He sought honestly to secure it for every class of people, alien ornative. Kalakaua’s highest desire was to be a true sovereign, the chiefservant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having maturedtheir plans in secret, the men of foreign birth rose one day en masse, calleda public meeting, and forced the king to sign a constitution of their ownpreparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race.
C R E S S T / U C L A
ExplanationScoring Rubric
General impression of content quality
Principles or concepts
Prior knowledge
Examples
Misconceptions
Argumentation
C R E S S T / U C L A
Template #2Prior Knowledge and
Explanation
Explicit measurement of knowledge domain before explanation
Uses short answer or selected response
Helps interprets explanation performance
C R E S S T / U C L A
Template #3Knowledge Representation
Same prompts
Key aspects of ideas, supporting facts and views and their relationships
Relationship is explicit
Organizational options
Core and peripheral Hierarchical Cause-and-effect Chronological
Expert scoring
C R E S S T / U C L A
History
C R E S S T / U C L A
Genetics
C R E S S T / U C L A
Bicycle Pump
C R E S S T / U C L A
Brief History of MBA in LAUSD
Content understanding and problem-solving models
Explanation templates
4 subjects, 3 grade levels, 2 languages
Purposes: (1) to clarify expectations; (2) to provided instructionally embedded assessment; (3) to get a measure of school performance
CRESST-managed teacher involvement
C R E S S T / U C L A
LAUSD Process
Teacher design teams
LAUSD standards first
Adapted to success standards
Training cadre of scorers
Training trainers
Supervising scoring
C R E S S T / U C L A
LAUSD Process (cont’d)
Shift in four-topic focus (capacity based) to two and then to one, now back to two
Continual assaults
Curriculum mandates
Accountability pressure (API)
Long-term embedded approach resurfacing
C R E S S T / U C L A
Present LA Situation
Administered in 2, 3, 4, 5, 6, 7, 8, 9
Purpose added regarding promotion
Teacher scored with an audit reported to school
Local sub-districts managing activity
Ongoing validity studies
District review of alternative assessments
C R E S S T / U C L A
C R E S S T / U C L A
CRESST Validation Studies
Score reliability
Task and rater generalizability
Stability of student performance over time
Relationships among measures
Instructional sensitivity
Opportunity to Learn (OTL)
Effect of school composition on performance
Cut-score modeling
C R E S S T / U C L A
LAUSD Grade 7 Students’ Achievement Levels: Comparison of 2002 California Standards Test and Performance
Assignment Scores
Evidence of Predictive Validity
73.7%
49.1%
25.1%
9.3%
21.4%
36.3%41.2%
31.7%
4.9%
14.5%
33.8%
59.0%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
Not Proficient PartiallyProficient
Proficient Advanced
2001 Performance Assignment Scores
% o
f S
tud
ents
in
Dif
fere
nt
Cat
ego
ries
o
f P
erfo
rman
ce i
n C
A S
tan
dar
ds
Tes
t
Below Bas ic
Bas ic
Above Bas ic
C R E S S T / U C L A
LA Scale-Up
Cost and time driven
Maintained by board and union support
Transfer of responsibility
Reduction in technical quality
Reduction in range of measures
Positive evaluation from independent group focusing on changing teaching practices
C R E S S T / U C L A
Continuing R&D Areas
New contexts
Trade-offs (limited number of templates vs. wide range of formats)
Performance over time
Scalability in the long run
Authoring systems to support teacher-developed assessments linked to large- scale assessment
C R E S S T / U C L A
Summary of Assessment Knowledge Requirements
Knowing why
Knowing what to assess: content plus cognitive demands (problem solving, communication, learning to learn, teamwork, content knowledge)
Knowing how: transfer (application to other topics and situations)
Reflecting: applying MBA to teaching
C R E S S T / U C L A
Model-Based AssessmentCognitive Families
ContentUnderstanding
ProblemSolving
Teamwork andCollaboration
MetacognitionCommunication
Learning
C R E S S T / U C L A
Context for Success of Knowledge-Based Reform
Local ownership of knowledge
Infrastructure and stability
Capacity to investigate
Learning
Congruence or peace with external mandates
C R E S S T / U C L A
Usable Knowledge and Support May Get to Useful
Knowledge
For assessment knowledge to be useful, it depends upon the context, capacity, and communication of the teaching system
For assessment knowledge to be useful to students, it must go to the heart of why, what, and how they learn