connecting cognition and assessment: concepts & directions derived from nrc research reports jim...

Connecting Connecting Cognition Cognition

and and Assessment:Assessment:

Concepts & Concepts & Directions Directions

Derived from NRC Derived from NRC Research ReportsResearch Reports

Jim PellegrinoJim PellegrinoUniversity of Illinois, University of Illinois,

ChicagoChicago

The Problem The Problem States face several critical decisions regarding how their assessment systems should be designed and implemented to respond effectively to the multiple provisions of No Child Left Behind. This includes defining the appropriate targets for assessment relative to content standards and determining how those targets might be apportioned across different elements of a comprehensive assessment system that provides information to support the enhancement of learning and instruction as well as accountability. The multiple sets of goals and needs is a major conceptual and operational design challenge.

OverviewOverview NCLB Requirements and Critical IssuesNCLB Requirements and Critical Issues

Advice for Policymakers -- Instructionally Advice for Policymakers -- Instructionally Supportive State Assessment PracticesSupportive State Assessment Practices

Conceptual Basis for Working Through Critical Conceptual Basis for Working Through Critical Issues of Assessment Design and UseIssues of Assessment Design and Use

What’s Needed for Further ProgressWhat’s Needed for Further Progress

Where and How to Learn MoreWhere and How to Learn More

UNTESTED

ESEA/NCLB Key RequirementsESEA/NCLB Key Requirements Annual assessments of all students in Math and Annual assessments of all students in Math and

Reading for Grades 3-8, and once in grades 9-12, Reading for Grades 3-8, and once in grades 9-12, beginning no later than 2005/2006 academic yearbeginning no later than 2005/2006 academic year Math and Reading annual assessments must be aligned with state Math and Reading annual assessments must be aligned with state

academic content and achievement standardsacademic content and achievement standards

Annual assessment of students in science no less Annual assessment of students in science no less than once in each of grades 3-5, 6-9 and 10-12, than once in each of grades 3-5, 6-9 and 10-12, beginning no later than 2007/2008 academic yearbeginning no later than 2007/2008 academic year

Adequate Yearly ProgressAdequate Yearly Progress• 100% of students must meet or exceed a “proficient” level of 100% of students must meet or exceed a “proficient” level of

academic achievement by the 2013-2014 academic yearacademic achievement by the 2013-2014 academic year• Establish intermediate goals for ongoing improvement over the 12 Establish intermediate goals for ongoing improvement over the 12

year periodyear period

Reporting in multiple categories for multiple Reporting in multiple categories for multiple demographic groupsdemographic groups

Some Key IssuesSome Key Issues What gets assessed?What gets assessed?

What standards & at what level of granularity?What standards & at what level of granularity?

How does useable information get derived?How does useable information get derived? Who needs what information, in what form, by Who needs what information, in what form, by

when, and for what purpose?when, and for what purpose?

Are systems of assessments necessary, Are systems of assessments necessary, desirable and/or feasible?desirable and/or feasible? What methods of assessment can be used and What methods of assessment can be used and

for what purposes?for what purposes? How does anyone make sense of data from How does anyone make sense of data from

multiple assessments?multiple assessments?

Commission RequirementsCommission Requirements Requirement 1. Requirement 1. Prioritized Content StandardsPrioritized Content Standards Requirement 2Requirement 2. Unambiguously Described Content . Unambiguously Described Content

StandardsStandards Requirement 3. Requirement 3. Standard by Standard ReportingStandard by Standard Reporting Requirement 4. Requirement 4. Classroom Assessments of State Classroom Assessments of State

Content StandardsContent Standards Requirement 5Requirement 5. Monitoring Curricular Breadth. Monitoring Curricular Breadth Requirement 6Requirement 6. Appropriate Assessment for all Students. Appropriate Assessment for all Students Requirement 7. Requirement 7. Sufficient Test Development TimeSufficient Test Development Time Requirement 8. Requirement 8. Pertinent Professional DevelopmentPertinent Professional Development Requirement 9Requirement 9. Ongoing Evaluation of the System. Ongoing Evaluation of the System

Issues Needing Further GuidanceIssues Needing Further Guidance

What’s the conceptual basis for making What’s the conceptual basis for making decisions among standards?decisions among standards?

How do we specify the meaning of a How do we specify the meaning of a standard?standard? What is assessable and how?What is assessable and how?

What is the process of translating standards What is the process of translating standards into assessment practices?into assessment practices? For large scale tests?For large scale tests? For classroom assessment purposes?For classroom assessment purposes?

How do the system’s pieces come together?How do the system’s pieces come together?

The Committee’s ObjectiveThe Committee’s Objective

To help establish a theoretical foundation for To help establish a theoretical foundation for the design and use of new kinds of the design and use of new kinds of assessments that will help all students learn assessments that will help all students learn and succeed in school. and succeed in school.

Needed are assessments that make as clear Needed are assessments that make as clear as possible to students, their teachers, and as possible to students, their teachers, and other education stakeholders the nature of other education stakeholders the nature of their accomplishments and the progress of their accomplishments and the progress of their learning.their learning.

About the ReportAbout the Report Proposes a vision of educational assessment Proposes a vision of educational assessment

based on contemporary understandings of how based on contemporary understandings of how people learn and how to measure such learning. people learn and how to measure such learning.

Describes an improved approach to assessment Describes an improved approach to assessment design and use, along with promising examples design and use, along with promising examples that include applications of technology. that include applications of technology.

Provides directions for research, development, Provides directions for research, development, policy, and practice for moving the field of policy, and practice for moving the field of assessment forward. assessment forward.

Report Structure & ContentReport Structure & Content

Part IPart I - Introduction & Background- Introduction & Background issues & opportunities in educational assessmentissues & opportunities in educational assessment

Part IIPart II - Scientific Foundations of Assessment- Scientific Foundations of Assessment the sciences of cognition & measurementthe sciences of cognition & measurement

Part IIIPart III - Assessment Design & Use- Assessment Design & Use vision of theory-driven design & its application vision of theory-driven design & its application

Part IVPart IV - Implications & Recommendations- Implications & Recommendations necessary directions for policy, practice & R&Dnecessary directions for policy, practice & R&D

Recommendations Regarding Recommendations Regarding Assessment Policy & PracticeAssessment Policy & Practice

Policy makers are urged to recognize the limitations of current Policy makers are urged to recognize the limitations of current assessments, and to support the development of new systems assessments, and to support the development of new systems of multiple assessments that would improve their ability to of multiple assessments that would improve their ability to make decisions about education programs and the allocation of make decisions about education programs and the allocation of

resourcesresources.. Important decisions should not be based on single test scoreImportant decisions should not be based on single test score Systems should measure growth in achievement over timeSystems should measure growth in achievement over time Emphasis should be shifted from Emphasis should be shifted from assessmentassessment of learningof learning to an to an

increased importance of increased importance of assessment for learningassessment for learning.. Assessments at classroom and large-scale levels should grow out of Assessments at classroom and large-scale levels should grow out of

shared knowledge base about learning and knowingshared knowledge base about learning and knowing

On what conceptual and operational basis can we begin On what conceptual and operational basis can we begin the process of designing and implementing such the process of designing and implementing such systems?systems?

Relating Teaching, Learning & Relating Teaching, Learning & AssessmentAssessment

High-Stakes Summative Tests

ClassroomTeaching &Learning

Level of Impact

Why Focus on Classroom Why Focus on Classroom Formative Assessment?Formative Assessment?

As instruction is occurring, teachers need information to evaluate whether their teaching strategies are working.

They also need information about the current understanding of individual students and groups of students so they can identify the most appropriate next steps for instruction.

Students need feedback to monitor their own learning success and to know how to improve.

Black & Wiliam (1998) reviewed impact of formative assessment practices on learning outcomes -- effect sizes ranging from .5 - 1.0

What’s Needed for Formative What’s Needed for Formative Assessment to WorkAssessment to Work

Sadler’s 3 interconnected elementsSadler’s 3 interconnected elements A clear view of the learning goals A clear view of the learning goals Information about the present state of the Information about the present state of the

learnerlearner Actions to close the gapActions to close the gap

The major challenge is always that ofThe major challenge is always that of Knowing What Students KnowKnowing What Students Know Need conceptually rich systems that link curriculum, Need conceptually rich systems that link curriculum,

instruction and assessmentinstruction and assessment

AAssessment as a Process of ssessment as a Process of Reasoning from EvidenceReasoning from Evidence

cognition cognition model of how students model of how students

represent knowledge & develop represent knowledge & develop competence in the domaincompetence in the domain

observationsobservations tasks or situations that allow tasks or situations that allow

one to observe students’ one to observe students’ performanceperformance

interpretationinterpretation method for making sense of the method for making sense of the

datadata

observation interpretation

cognition

Must be coordinated!

Understanding “Cognition”Understanding “Cognition” Challenge I: Articulating Multiple Explanations Challenge I: Articulating Multiple Explanations

of Thought & Behavior -- of Thought & Behavior -- What we want and What we want and need to understandneed to understand Behavior ranges from micro-processes of rapid perception to Behavior ranges from micro-processes of rapid perception to

macro-processes like problem solving and negotiationmacro-processes like problem solving and negotiation Time periods over which behavior and learning unfolds can Time periods over which behavior and learning unfolds can

vary tremendouslyvary tremendously

Challenge II: Multiple Levels of Explanation -- Challenge II: Multiple Levels of Explanation -- The way we focus the explanationThe way we focus the explanation Cognitive Accounts of Individual Processes and Knowledge Cognitive Accounts of Individual Processes and Knowledge

RepresentationsRepresentations Situated/Sociocultural Accounts of Collective Processes and Situated/Sociocultural Accounts of Collective Processes and

Distributed Knowledge RepresentationsDistributed Knowledge Representations

Cognitive Level AnalysisCognitive Level Analysis

The most critical implications for The most critical implications for assessment are derived from study of the assessment are derived from study of the nature of competence and study of the nature of competence and study of the development of knowledge in specific development of knowledge in specific curriculum domains.curriculum domains.

Characterizing Performance: Task AnalysisCharacterizing Performance: Task Analysis Characterizing Development: Trajectories of LearningCharacterizing Development: Trajectories of Learning Characterizing Knowledge: Forms of RepresentationCharacterizing Knowledge: Forms of Representation

Task Analysis: ArithmeticTask Analysis: Arithmetic

Melissa had 6 pencils. Henry gave her 14 more. Melissa had 6 pencils. Henry gave her 14 more. How many pencils does Melissa have now?How many pencils does Melissa have now?

What are the foundations of competent performance?What are the foundations of competent performance?

Knowledge of concepts like cardinality, setsKnowledge of concepts like cardinality, sets

Knowledge of strategies, like counting, joining setsKnowledge of strategies, like counting, joining sets

Trajectories: ArithmeticTrajectories: Arithmetic Melissa had 6 pencils. Henry gave her 14 more. Melissa had 6 pencils. Henry gave her 14 more.

How many pencils does Melissa have now? How many pencils does Melissa have now?

Direct ModelingDirect Modeling - Represent Sets with Counters, Join, - Represent Sets with Counters, Join, Count All 1…20Count All 1…20

Counting Counting - Represent Sets as Numbers- Represent Sets as Numbers• Count-on from first- 6, 7(1), 8(2), …20(14)Count-on from first- 6, 7(1), 8(2), …20(14)

• Count-on from larger- 14, 15(1), 16(2), …20(6)Count-on from larger- 14, 15(1), 16(2), …20(6)

Derived FactsDerived Facts - 6 + 10 =16, 16 + 4 = 20 - 6 + 10 =16, 16 + 4 = 20

Recall Recall 6 + 14 = 20 6 + 14 = 20

Sociocultural Level AnalysisSociocultural Level Analysis

The most critical implications for assessment The most critical implications for assessment are derived from study of the nature of are derived from study of the nature of practice and forms of participation in practice and forms of participation in communities.communities.

Characterizing Performance: Communal Practices Characterizing Performance: Communal Practices Characterizing Development: Trajectories of Participation Characterizing Development: Trajectories of Participation Characterizing Knowledge: Forms of Mediated ActivityCharacterizing Knowledge: Forms of Mediated Activity

Practice: AccountingPractice: Accounting New Math of the 15th CenturyNew Math of the 15th Century

Development: learning to participate in a guild. Development: learning to participate in a guild. Apprenticeship model of development.Apprenticeship model of development.

Knowledge: mediation of arithmetic operations Knowledge: mediation of arithmetic operations via algorithmsvia algorithms Algorithms of the counting houses are those we Algorithms of the counting houses are those we

teach today.teach today.

Sociocultural Look at ArithmeticSociocultural Look at Arithmetic

Mathematical PracticesMathematical Practices Number TheoryNumber Theory

Conjecture (commutative?)Conjecture (commutative?)

Trajectories of Participation in ArgumentTrajectories of Participation in Argument Cases ---> Generalization--->ProofCases ---> Generalization--->Proof

4 x 7 = 7 x 4 4 x 7 = 7 x 4 ss x ls = ls x ssss x ls = ls x ss

Generalizations About Generalizations About PerformancePerformance

ExpertiseExpertise performance develops in communities that value certain performance develops in communities that value certain

forms of knowledge and activity, like modeling in science.forms of knowledge and activity, like modeling in science. knowledge is tuned to specific patterns of activity, like knowledge is tuned to specific patterns of activity, like

solving certain kinds of problems.solving certain kinds of problems. performance increases in scope and precision with performance increases in scope and precision with

multiple, contextualized experiences. multiple, contextualized experiences. no magic levers: practice, disciplined inquiry.no magic levers: practice, disciplined inquiry.

Implication -- Assessments must be Implication -- Assessments must be designed to capture the complexity of designed to capture the complexity of competent performance, ranging from mental competent performance, ranging from mental processes to participation in forms of practiceprocesses to participation in forms of practice

Generalizations about Generalizations about DevelopmentDevelopment

Not all children learn in the same way or follow the Not all children learn in the same way or follow the same paths to competence. same paths to competence. Conceptual change is often not a simple, uniform Conceptual change is often not a simple, uniform

progression, nor is there movement directly from erroneous progression, nor is there movement directly from erroneous to optimal solution strategies.to optimal solution strategies.

Intermediate forms of knowledge may not resemble expert Intermediate forms of knowledge may not resemble expert forms, so simple building block relations may not hold.forms, so simple building block relations may not hold.

Participation often “starts at the edges” and becomes Participation often “starts at the edges” and becomes progressively more aligned with core disciplinary practicesprogressively more aligned with core disciplinary practices..

Implication - Assessments should identify specific Implication - Assessments should identify specific strategies and forms of activity with respect to the strategies and forms of activity with respect to the role they play in developmental trajectories. (e.g., role they play in developmental trajectories. (e.g., COF is fine at grade 1, not grade 3)COF is fine at grade 1, not grade 3)

Generalizations about Generalizations about Knowledge Knowledge

Disciplinary KnowledgeDisciplinary Knowledge Is organized in ensembles that facilitate its use.Is organized in ensembles that facilitate its use. Is amplified by processes of self regulation, or Is amplified by processes of self regulation, or

“metacognition,” where learners spontaneously evaluate “metacognition,” where learners spontaneously evaluate their knowledge and its limits.their knowledge and its limits.

Is developed in communities that foster identity and Is developed in communities that foster identity and interest. interest.

Implications for Assessment -- multiple “questions”Implications for Assessment -- multiple “questions” Knowledge Issues - Specific Facts, Procedures, SchemasKnowledge Issues - Specific Facts, Procedures, Schemas Reflection Issues - Articulation, EvaluationReflection Issues - Articulation, Evaluation Practice Issues - Why prove? Model? Practice Issues - Why prove? Model?

Why Cognitive Models of Content Why Cognitive Models of Content Knowledge are Critical Knowledge are Critical

Tell us what are the important aspects of Tell us what are the important aspects of knowledge that we should be assessing.knowledge that we should be assessing. Give deeper meaning and specificity to standardsGive deeper meaning and specificity to standards

Give us strong clues as to how such knowledge Give us strong clues as to how such knowledge can be assessedcan be assessed Suggest what can and should be assessed at points Suggest what can and should be assessed at points

proximal or distal to instructionproximal or distal to instruction Can lead to assessments that yield more Can lead to assessments that yield more

instructionally useful information -- within and instructionally useful information -- within and across levels and contextsacross levels and contexts

Can guide the development of systems of Can guide the development of systems of assessmentsassessments Comprehensive, Coherent & ContinuousComprehensive, Coherent & Continuous

Systems of AssessmentsSystems of Assessments

ComprehensiveComprehensive - utilize a range of measurement - utilize a range of measurement approaches that together produce information that approaches that together produce information that can be combined and reported at different levelscan be combined and reported at different levels

CoherentCoherent - the conceptual base for the student - the conceptual base for the student models underlying design of the multiple models underlying design of the multiple assessments should be compatible. There also assessments should be compatible. There also needs to be alignment between curriculum, needs to be alignment between curriculum, instruction and assessment so that everything is instruction and assessment so that everything is working toward a common set of learning goals.working toward a common set of learning goals.

ContinuousContinuous - Sequential observations, over time must - Sequential observations, over time must be conceptually linked so that growth can be be conceptually linked so that growth can be measured. Assessment is viewed as a cumulative measured. Assessment is viewed as a cumulative process rather than a “drop-in-from-the-sky” eventprocess rather than a “drop-in-from-the-sky” event

Assessment Design Assessment Design PrinciplesPrinciples

Assessment design should always be based Assessment design should always be based upon a upon a model of student learningmodel of student learning and a clear and a clear sense of the sense of the inferences about student inferences about student competencecompetence that are desired for the that are desired for the particular context of useparticular context of use. .

Starting with the Student ModelStarting with the Student Model: The student : The student model suggests the most important aspects of model suggests the most important aspects of student achievement that one would want to student achievement that one would want to make inferences about and provides clues make inferences about and provides clues about the types of tasks that will elicit evidence about the types of tasks that will elicit evidence to support those inferences.to support those inferences.

Aspects of Student ModelsAspects of Student Models

Domain specific and empirically basedDomain specific and empirically based Identifies cognitive performances that Identifies cognitive performances that

differentiate expert and novice learners differentiate expert and novice learners Lays out one or more typical progressions Lays out one or more typical progressions

toward competence including milestones or toward competence including milestones or landmark performances along the way.landmark performances along the way.

Can be at various levels of detail; grain size Can be at various levels of detail; grain size depends on assessment purposedepends on assessment purpose

Valuable information sourceValuable information source

connecting cognition and assessment: concepts & directions derived from nrc research reports jim...

Documents