Assessment

What are the questions?What evidence will we

accept?

Diane Ebert-MayDepartment of Plant Biology

Michigan State University

ebertmay@msu.eduhttp://first2.org

Anonymous (Change Magazine, 2001):

Anonymous (Change Magazine, 2001):

“I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

“I believe we would all agree that the absolute best teaching learning-assessment model is the one-on-one Socratic apprenticeship model with unlimited time with the student. But ever since Socrates took on two students rather than only one (to double his income), teachers have had to make compromises in teaching.”

Larry Spence, Penn State

Larry Spence, Penn State

“Today’s graduates cannot meet the demands of workplace or without several more years of learning on the job. They cannot formulate and solve messy real-world problems, work well with others in high-stress team situations, write and speak forcefully and persuasively, or improve their own performance.”

“Today’s graduates cannot meet the demands of workplace or without several more years of learning on the job. They cannot formulate and solve messy real-world problems, work well with others in high-stress team situations, write and speak forcefully and persuasively, or improve their own performance.”

Question 1

How important is it to use multiple kinds of data to assess student learning?

Please respond on a scale if 0-100 in increments of 10:

How important is it to use multiple forms of data to assess student learning?

%

Relative Importance n=127

Question 2

How often do you use data to make instructional decisions?

Please respond on a scale of 0 - 100 in increments of 10:

How often do you use data to make instructional decisions?

n=127Frequency

%

True or False?

Assessing student learning in science parallels what scientists do as researchers.

1.Description:

-What is happening?

2.Cause:

-Does ‘x’ (teaching strategy) affect ‘y’ (understanding)?

3.Process or mechanism:

-Why or how does ‘x’ cause ‘y’?

Parallel: ask questions

We collect data to find out what our students know.

Data helps us understand student thinking about concepts and content.

We use data to guide decisions about course/curriculum/innovative instruction

Parallel: collect data

Quantitative data - statistical analysis

Qualitative data

break into manageable units and define coding categories

search for patterns, quantify

interpret and synthesize

Valid and repeatable measures

Parallel: analyze data

Ideas and results are peer reviewed - formally and/or informally.

Parallel: peer review

What is assessment?

Data collection with the purpose of answering questions about…

students’ understanding

students’ attitudes

students’ skills

instructional design and implementation

curricular reform (at multiple grainsizes)

Why do assessment?

Improve student learning and development.

Provides students and facultysubstantive feedback about student understanding.

Challenge to use disciplinary research strategies to assess learning.

Research Methods

Data collection

approaches

Multiple Choice … … Concept Maps … … Essay … … Interview

high Ease of Assessment low

low Potential for Assessment of Learning high

Theoretical Framework• Ausubel 1968; meaningful learning• Novak 1998; visual representations• King and Kitchner 1994; reflective judgment• National Research Council 1999; theoretical frameworks for assessment

Assessment Gradient

Does active, inquiry-based instructional design influence students’ understanding of evolution and natural selection?

Pre-Posttest Analysis

■ Changes in a population occur through a gradual change in individual members of a population.

■ New traits in species are developed in response to need.

■ All members of a population are genetically equivalent, variation and fitness are not considered.

■ Traits acquired during an individual’s lifetime will be inherited by offspring.

Alternative Conceptions: Natural Selection

Instructional Design

Cooperative groups in class:

Guppy Problem: sexual vs. natural selectionhttp://www.first2.org/resources/inquiry_activities/guppy_activity.htm-PBS film

-Simulation-Analyze data-Written explanation

(AAAS 1999)

Explain the changes that occurred in the tree and animal. Use your current understanding of evolution by natural selection.

Misconception: individuals evolve new traitsMisconception: individuals evolve new traits

% o

f S

tud

ents

n=80; p<.01

Misconception: evolution is driven by needMisconception: evolution is driven by need

% o

f S

tud

ents

n=80; p<.01

In guppy populations, what are the primary changes that occur gradually over time?

In guppy populations, what are the primary changes that occur gradually over time?

a. The traits of each individual guppy within a population gradually change.

b. The proportions of guppies having different traits within a population change.

c. Successful behaviors learned by certain guppies are passed on to offspring.

d. Mutations occur to meet the needs of the guppies as the environment changes.

Anderson et al 2002

Posttest: Student responses to mcPosttest: Student responses to mc

% o

f S

tud

ents

n=171

*

Animal/Tree Posttest: Gain in student understanding of fitnessAnimal/Tree Posttest: Gain in student understanding of fitness

% o

f S

tud

ents

n=80; p<.01

Quantitative Data

Qualitative Data

Design Experiment

Ebert-May et al. 2003 Bioscience

How do assessment questions help us determine students’ prior understanding and progressive thinking about the carbon cycle.

Question

Instructional Design

• Two class meetings on carbon cycle (160 minutes)

• Active, inquiry-based learning– Cooperative groups– Questions, group processing, large lecture

sections, small discussion sections, multi-week laboratory investigation

– Homework problems including web-based modules

• Different faculty for each course– One graduate/8-10 undergraduate TAs per

course

Experimental DesignTwo introductory courses for majors:

Bio 1 - organismal/population biology (faculty A)Bio 2 - cell and molecular biology (faculty B)

Three cohorts:Cohort 1 Bio 1 (n=141)Cohort 2 Bio1/Bio2 (n=63)

Cohort 3 Other/Bio2 (n=40)

Assessment Design

Multiple iterations/versions of the carbon cycle problem Pretest, midterm, final with additional formative assessments during classAdministered during instructionSemester 1 - pretest, midterm, final exam Semester 2 - final exam

Grandma Johnson Problem

Hypothetical scenario: Grandma Johnson had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.

Analysis of Responses

Used same scoring rubric (coding scheme) for all three problems - calibrated by adding additional criteria when necessary, rescoring:

Examined two major concepts: Concept 1: Decomposers respire CO2

Concept 2: Plants uptake of CO2

Explanations categorized into two groups:Organisms (trophic levels)Processes (metabolic)

Code Organisms Code Processes and pathways

1 Decomposers IA Cellular Respiration

IB Release CO 2

2 IIA Pathway of Carbon

Primary producers IIA _1: through Air

IIA _2 : through Root

IIA _3 : no mention about pathway

IIB Make Glucose

II C Photosynthesis

3 Herbivore III Respiration

(glycolysis, Kreb cycle)

4 Carnivore IV Respiration

(glycolysis, Kreb cycle)

Coding Scheme

Corr

ect

Stu

den

t R

esp

on

ses

(%)

Cellular Respiration by Decomposers

Bio1/Bio2 Other/Bio2

Friedmans, p<0.01

Pathway of Carbon in Photosynthesis

Bio1/Bio2

Corr

ect

Stu

dent

Resp

on

ses

(%)

Other/Bio2

Friedmans, p<0.05

IRD Team at MSUJanet Batzli - Plant Biology [U of Wisconsin]Doug Luckie - PhysiologyScott Harrison - Microbiology (grad student)Tammy Long - Plant BiologyJim Smith - ZoologyDeb Linton - Plant Biology (postdoc)Heejun Lim - Chemistry EducationDuncan Sibley - Geology*National Science Foundation

What is the question?

What research and instructional designs?

What data collection methods?

How to analyze and interpret data?

Are findings valid and generalizable?

What are the next questions?

WHO?

What evidence will we accept?