Computer-based Assessment

Paul HorwitzPaul Horwitz

The Concord ConsortiumThe Concord Consortium

Presentation to DR-K12 PI Meeting,Presentation to DR-K12 PI Meeting,

November 10, 2009November 10, 2009

For assessment purposes, is the computer a process innovation or a product innovation?

Inventions that started as process Inventions that started as process innovations and became new productsinnovations and became new products The telephoneThe telephone The refrigeratorThe refrigerator

Computer-assisted assessment as a way to increase productivity

Deliver the identical test onlineDeliver the identical test online Embed questions within visualizationsEmbed questions within visualizations Random assignment of itemsRandom assignment of items Adaptive testingAdaptive testing

Assessment techniques that depend on the computer

Performance assessment based on Performance assessment based on simulations of real systems and processessimulations of real systems and processes

Assessment based on students’ Assessment based on students’ manipulations of causal modelsmanipulations of causal models

Inferences based on tracking of problem-Inferences based on tracking of problem-solving activitiessolving activities

Examples of performance assessment:1. The SPARKS Project

Current ATE ProjectCurrent ATE Project College level: introductory electronicsCollege level: introductory electronics Building self-paced assessments that give Building self-paced assessments that give

students useful feedback and offer an students useful feedback and offer an opportunity to practice skillsopportunity to practice skills

Optionally, assessments generate reports for Optionally, assessments generate reports for instructors as well.instructors as well.

Examples of performance assessment:2. “Evolution Readiness”

Current DR-K12 projectCurrent DR-K12 project Fourth grade science dealing with evolution Fourth grade science dealing with evolution

as emerging from natural selectionas emerging from natural selection Students perform experiments with virtual Students perform experiments with virtual

plant and (soon) animal communitiesplant and (soon) animal communities Actions are logged, analyzed, and used to Actions are logged, analyzed, and used to

report on students’ inquiry skills and report on students’ inquiry skills and content knowledgecontent knowledge

Challenges to innovation in assessment Technological barriersTechnological barriers

Models and simulations take time to buildModels and simulations take time to build Required infrastructure complex and hard to Required infrastructure complex and hard to

maintainmaintain Psychometric barriersPsychometric barriers

How do you score these things?How do you score these things?e.g., what constitutes an “item”?e.g., what constitutes an “item”?

How do you determine validity and reliability?How do you determine validity and reliability?

Importance of innovation in assessment Our understanding of learning has advanced Our understanding of learning has advanced

faster than our ability to assess learningfaster than our ability to assess learning Schools live or die by the results of Schools live or die by the results of

assessments that don’t reflect current assessments that don’t reflect current learning sciences researchlearning sciences research

We “teach to the test” so we don’t teach We “teach to the test” so we don’t teach what we ought to teachwhat we ought to teach