Assessing student learning

Ross NehmAssociate ProfessorCollege of Education and Human Ecology The Ohio State University

Assessment basics

Types of assessment Methods of assessment Assessment data Rubrics and data processing Limitations Questions

Assessment basics

As Susan has discussed, learning goals should be closely tied to assessment strategies.

As you design your I3Us, ask yourself what types of assessment strategies would best evaluate whether or not your learning goals have been met.

Assessment categories

Three commonly used types of assessments include:

Knowledge assessments Attitude and belief (and affective)

assessments Performance assessments

Knowledge assessments Attempt to measure what students know at

varying levels of complexity:

Are students able to recall a particular piece of knowledge?

Are students able to solve a higher-order problem in a particular knowledge domain?

It is useful to measure student abilities at different Bloom levels.

Bloom’s taxonomy Knowledge

list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote, name, who, when, where, etc.

Comprehensionsummarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend

Applicationapply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover

Analysisanalyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer

Synthesiscombine, integrate, modify, rearrange, substitute, plan, create, design, invent, what if?, compose, formulate, prepare, generalize, rewrite

Evaluationassess, decide, rank, grade, test, measure, recommend, convince, select, judge, explain, discriminate, support, conclude, compare, summarize

Topics and Bloom Levels of Questions

Dexter Perkins (University of North Dakota), Karl Wirth (Macalester College), Ed Nuhfer (Idaho State University), presentation at the Student Learning: Observing and Assessing Workshop.

Attitude and belief assessments Attitudes and beliefs play an important role

in science learning, and a goal of your modules may be to alter attitudes.

Do students believe that learning about genomics is important?

Do students like to perform “experiments” using bioinformatics databases?

Performance assessments Performance assessments determine

whether students are able to perform a particular task.

Are students able to download a particular sequence from NCBI in fasta format?

Are students are able to use biology workbench to perform a multiple sequence alignment?

Are students able to work collaboratively to solve a problem?

What assessments should I use?

•Different learning goals will undoubtedly require different (or multiple) types of assessments.

•As you develop each learning goal, think carefully about what type of assessment would best measure what you are trying to achieve.

Assessment methods Each type of assessment (knowledge,

attitude, performance) may require a different assessment method.

Methods may include: Oral interviews Classroom observations Concept mapping Predict-Observe-Explain Paper and pencil tests

Method strengths and weaknesses Different assessment methods have different

strengths and weaknesses.

Oral interviews with students tend to be very informative, but they take large amounts of time to perform and analyze.

Concept maps are excellent tools for exploring the extent of student knowledge integration, but they are difficult to score.

Paper and pencil tests are useful for assessing student learning in large classes, but often they do not provide a rich picture of what students really know.

Examples of methods

Concept mapping

Examples of methods

Oral interviewInterviewer: A number of mosquito populations no longer die when DDT, which is a chemical used to kill insects, is sprayed on them, but many years ago DDT killed most mosquitoes. Could you explain why many mosquitoes don’t die anymore when DDT is sprayed on them?

Participant R: …Well, if at first the DDT killed most mosquitoes and now it’s not killing them [any] more, then a possible explanation would be that when they first started exposing the mosquitoes to the DDT they didn’t have any…their immune system was not that strong to fight the DDT. As time went on they developed some kind of resistance to the DDT…they passed this kind of, um, newly evolved resistance on to the next generation so…passing on this trait from generation to generation…it will start becoming stronger and if the DDT is used on them it wouldn’t kill them...

Interviewer: Can you tell me a little bit more about how that [resistance] would happen, in general terms?

Participant R: …I was watching the discovery channel and there was a man who said he could develop resistance to the venom of a snake…so he started to gradually use little bits of this venom and started injecting venom into his system and from time to time he would increase the amount of venom he took into his system…he got bitten by the snake and to the surprise of the doctors this man actually had some kind of resistance to that venom, in comparison to a normal person who would just die…my guess would be that at first the mosquitoes…from time to time they kept exposing them to this kind of chemical…those will develop some kind of resistance to this kind of chemical for them to survive.

Examples of methods

Likert-scale questionsEach of the statements below expresses a feeling toward biology. Please rate each statement on the extent to which you agree. For each, you may:

A B C D E

strongly agree agree be undecided disagreestrongly disagree

1. Biology is very interesting to me.

2. I don’t like biology, and it scares me to have to take it.

3. I am always under a terrible strain in a biology class.

4. Biology is fascinating and fun.

5. Biology makes me feel secure, and at the same time is stimulating.

6. Biology makes me feel uncomfortable, restless, irritable, and impatient.

http://www.flaguide.org/tools/attitude/biology_attitude_scale.php

Assessment diversity

Assessment type Assessment method

Knowledge Paper and pencil testConcept mappingOral interview

Attitude Likert-scale paper and pencil testClassroom observationOral interview

Performance Computer activityClassroom observationTask analysis

Assessment type + method matrix

I3U

Lear

ing

goal

1

Lear

ning

goa

l 2

Lear

ning

goa

l 3

Lear

ning

goa

l 4

Lear

ning

goa

l 5

Knowledge assessments Concept mapping

Written testWorksheet

Attitude and belief assessments Likert-scale questionairre

Oral interviewPerformance assessments

Computer lab activityGroup project

Methods data scoring Different assessment methods produce different types

of data.

It is often necessary to process the data such that they can be used in analyses of student performance.

For example, a transcribed oral interview is not in itself very helpful. We need to process and interpret the interview.

Likewise, a concept map needs to be scored in some way.

Rubrics Rubrics are used to define the universe of

student performance, knowledge, belief, etc. and map actual student performance within this universe.

Rubrics can be constructed in many different ways: Binary (Right/wrong) Gradational (multistate) categories Likert scales

Simple rubric

-1 = clear evidence of a faulty mental model of natural selection lacking key concepts with numerous misconceptions;

0 = ambiguous evidence: some correct key concepts present, some misconceptions present, but unclear evidence whether an accurate mental model of natural selection is being employed;

1= clear, unambiguous evidence of an accurate working model of natural selection lacking misconceptions and employing at least three key concepts.

Student Inteview scorea 0b 1c 1d -1e 0f 1g 1h -1i 1j 0k 0l 0

m 1n -1o -1p 0

Interview

Example: Learning goal: decrease student use of

misconceptions of natural selection in evolutionary explanations

Type of assessment: Knowledge Oral interview method

Transcribed data Rubric developed to code data Oral interview scoring categories -1, 0, +1 Statistical analysis of distribution frequencies Interpretation of pre-post course gains

Limitations Developing an assessment instrument

(or “test”) that produces meaningful results is much more complicated than most people think.

The bottom line is that most of your assessment results will only be able to be treated as “suggestive” rather than “conclusive”. Why?

Assessment limitations Good assessment relies on:

(1) appropriate experimental design + (2) appropriately chosen and validated

instruments

Most scientists can develop an appropriate experimental design, but are not prepared to develop and validate an instrument.

Without valid and reliable instruments, it is difficult if not impossible to produce a robust conclusion.

Suggestions as you proceed Define your I3U learning goals first Consider the appropriateness of different types of

assessments (e.g., knowledge, attitude) relative to these learning goals

Evaluate the appropriateness of possible assessment methods (e.g., interview, concept maps)

Plan how you will score and evaluate the data you collect using each method

Plan to analyze the scores statistically Use assessments to determine if your learning goals

were met. Keep in mind the limitations of the assessment

approaches you are using.

Questions?