kera 1 © regents of the university of california verifying learning and advanced skills and...

KERA 1 © Regents of the University of California

Verifying Learning and Advanced Skills

and Knowledge

Eva L. BakerNational Center for Research on Evaluation, Standards, & Student TestingGraduate School of Education & Information StudiesUniversity of California, Los Angeles

Korean Educational Research AssociationEducational Innovation for the 21st Century: Sharing Visions and ExperiencesHanyang University, Seoul, Korea

29 October 2010


Presentation Overview

• How have we designed, used, and validated assessments in the past?

• Advanced knowledge and skills call for new methods

• 21st century skills require deeper, more research-based analyses related to learning and learners

• The elastic relationships among practice, research and policy decisions, and research needs


Before the Marriage of Learning and Assessment

Assessment design

• Focused on stable findings rather than change

• Required many items sampled across a broad construct

• Use of item analyses, etc. to improve item performance

• Detect the best

Uses and validity

• Reporting of position in normal distributions

• Technical quality indices were sensitive to numbers of items and wide ranging response

• Correlations and predictions

• Quotas and other limits on access


Is it Time to Flower?

http://www.google.com/imgres?imgurl=http://www.rubythroat.org/images/RoseOfSharon01.jpg&imgrefurl=http://www.lacitycollege.edu/academic/departments/forlang/Kor/Kcourses.html&h=359&w=250&sz=15&tbnid=ioukMy9nXKdceM:&tbnh=121&tbnw=84&prev=/images%3Fq%3Dsouth%2Bkorean%2Bflower&zoom=1&q=south+korean+flower&hl=en&usg=__VJcjOdfCNSfjONMmSgPpY3LNbMs=&sa=X&ei=SV3ATN2yDY7CsAPku8DxCw&sqi=2&ved=0CCMQ9QEwAw


Overview of Assessment of Advanced Skills and Knowledge Needs

Goals and strategies

• Making education more accessible

• High expectations for all

• Emphasis on adaptation and flexibility

• Preparation for global environment

Building in validity

• Different learners, motivations, constraints

• Emphasis on change

• Integration/application interpretive, not just “correct”

• Shared analyses, methods, and technical approaches—convergence and individuality


Beyond “Alignment” Metaphor

• Standards-based systems imply lining up instruction provided and tests used to produce learning (ISD)

• Too limiting as rarely can standards be fully or completely sampled, almost never do they apply directly to future contexts

• Good teaching in 21st century cannot be simply following of approved text and endless repetition to prepare for tests


Towards Coherence and Transfer

• Paradox: to prepare for flexibility takes greater attention to components of learning rather than general constructs

• Biggest change: transfer must be taught and measured

• Harder to prepare for accountability exams

• Both horizontal (subjects) and vertical (grade level) integration


What Are the First Steps?

• Clarifying and developing a reasonable number of standards

• Making sure the standards are valid—by assessing their quality, commonality of interpretation, and relationships to one another

• Logical and empirical processes—building the content ontology


1. Content Clarification by Ontology

• Graphic representation of nodes (what is to be learned) and relationships among them

• Not strictly a teaching sequence

• Verify common interpretations among teachers, educators, content experts, and policymakers

• Determine priorities and fundamentals

• Create a top-level meta-tag system for managing learning and performance database


Common Core Standards Progression


2. Link to Cognitive Demands

• Cognitive demands are learnable/teachable processes that are embedded in subject matter

• Individuals differences may apply, but these are not traits

• They range between very specific to a domain, e.g., trouble shooting, somewhat general, develop a plan for persuasion; to more independent of specific domains, general writing skill


2. Cognitive Demands—21st Century Skills

• Adapt problem solving approaches

• Transfer and apply skills to new situations

• Seek and manage new learning experiences

• Take risks in intellectual or entrepreneurial tasks

• Communicate in a variety of ways

• Expand skills for global work—teamwork, collaboration, language proficiency, cultural situation awareness


Problem Solving Ontology

Mayer, S. (in progress). Problem solving ontology (working document). Los Angeles: University of California, National Center for Researchon Evaluation, Standards, and Student Testing.


3. Integrate Content and Cognition in Model Tasks

• Format may be of multiple types driven by outcome rather than convenience

• Support teacher and student initiative in creating similar tasks

• Assessments used for accountability and formative purposes

• Create reusable objects or templates for assessments (in online environments) with immediate automated scoring

• Monitor in database


English Language Arts


4. Conduct Ongoing Validity Studies

• Systems evolve and validity is a continuing process

• Functional for purposes, accountability, improvement

• Sensitive to instruction not only to student background

• Fair to cultural/linguistic differences, disabled, gender

• Minimal gaming or negative side effects

• Include in database of individuals and items

• Timing


5. More Expansive Goals for 21st Century

• Agility

• Intellectual and emotional resilience

• Persistence

• Societal and family values: dependability, ethics, caring, and joy


Continuing Challenges

• Performance assessments are back, but need research on making them more comparable to one another—validity and fairness

• New psychometrics required if feedback to teachers and learners grows in importance—improvement guidance not rank order

• Costs can only be dealt with by serious use of simulations in design, administration, scoring


Continuing Challenges (Cont’d)

• Teacher evaluation using students’ results very imperfect—but has numbers so may drive out more interpretive models

• Domains of tasks for configurations of standards need to be developed in advance, tried, and sampled. Problem is short time frame, school year, knowledge dependencies

• Need theory of situations (Resnick) to improve process


Research Dreams• Small sample, few—tasks models of validity

• Integrated databases from standards, through learning, through performance, mediated by individual background and recognition of many paths, with room to grow

• New theories of design

• Real investigations of variables, not only evaluation of entire interventions

• Finding ways to re-motivate students for intrinsic learning in this age of distraction, celebrity, and catastrophe


Ready to Fly ???

http://en.wikipedia.org/wiki/File:Black-billed_Magpie.png


http://www.cse.ucla.edu

Eva L. [email protected]

voicefax

email


Backup


Notes from the U.S.• Emphasis on testing expanding with Race to the Top

• Competition funded two consortia to measure Common Core Standards—each with overlapping majorities of states

• Involve both formative and accountability assessment

• Pressure to link teacher evaluation to student work—controversial

• Rethinking role of commercial test producers

• No way to provide standard-by-standard reporting


Speaking Truth to Power

• Not easy unless you are friends

• Policymakers in U.S. have odd mix of research-based ideas and preferences—now almost an overwhelming wave

• Desire conflicting goals without recognizing conflict

kera 1 © regents of the university of california verifying learning and advanced skills and...

Documents