Evaluating Your Students’ Learning & Whether Your

Project Is Succeeding

Innovators’ Toolkit 4Julie Sievers, Center for Teaching Excellence

Why Evaluate Your Project?

To see if it was worth your time and effort

To decide whether your colleagues / department / field should adopt similar strategies in other courses.

To enable you to base your arguments about these issues on evidence, not just anecdote and impression.

Scholarship of Teaching & Learning:

To enable you to present at a conference, or publish in a journal

A Taxonomy of Questions

1. Assessing Effectiveness of Course, Project, Method

• Did I accomplish my goals?

• Does this strategy work?

• Does it work better than other strategies?

“What Works?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000)

Assessment Q: – Similar to program assessment for SACS • “Are you

accomplishing what you set out to do?”

• “How do you know?”

Effectiveness Questions

Success or failure question:

Did students meet the SLOs? Yes or no.• Variation: Did students’ meet this or

that SLO?• Collect evidence similar to that used

for program assessment.

Comparative question:

Did the students meet the SLOs BETTER in this version of the class than in another?• Requires comparison data• May compare to your previous

version, or a standard version you are also offering.

• May compare to a colleague’s course.

Improvement question:

Did the students IMPROVE in their ability to meet an SLO over the course of the semester?• Requires baseline data –

where were students on this SLO in week 1? Where are they in week 15?

• Proves success even if SLO is still not met

Yardstick: learning outcomes

More effectiveness questions

Questions about non-SLO goals

For example: Did this project . . .• Increase student

engagement?• Increase student

motivation?• Facilitate deeper learning?• Improve classroom climate?• Create a classroom that

supports productive failure

Yardstick: Your goals (need to be articulated, but does not have to be an official SLO)

A Taxonomy of Questions2. Inquiry Into Student

Learning

Goal: To better understand • learning processes, • classroom social dynamics, • student errors or misconceptions, • the roles played by various factors

in student learning, etc.

• Descriptive / documentarian• Less oriented around practical

questions of a method, more about fundamental learning issues and processes at work in the course / experience.

“What Is (Happening)?” Q – in Pat Hutchings taxonomy of questions, Opening Lines (2000)

SoTL Q• Common in

Scholarship of Teaching & Learning inquiry

• Less similar to assessment questions.

Inquiry into Student Learning Questions

• What thinking processes do students engage in when learning about X?

• What are students’ beliefs about how they best learn Discipline X?

• How do practices of self-reflection change their learning about X?

• What do students find most difficult about learning in Course X?

• How do students prior understanding of Discipline X affect their ability to acquire new understanding in that field?

• How does their attitude towards Y affect their ability to do work on issue X?

“Here the effort is aimed not so much at proving (or disproving) the effectiveness of a particular approach or intervention but at describing what it [learning] looks like, what its constituent features might be.” – Pat Hutchings

A Taxonomy of Questions3. How Are Things Going?

Goal: To better understand • How students are experiencing

the course• How students are doing – before

they submit a major assignment • Student preconceptions,

misconceptsion• Student study and work habits /

processes

May be:• Anonymous• Ungraded • Diagnostic

Designed to enable you to intervene quickly before semester / unit / project – even class! -- is over

How Are Things Going Questions

Simple / quick tools:• Misconception / Preconception

check• minute paper• Muddiest point (minute) paper• Classroom opinion polls• Course-related self-confidence

surveys• Productive study-time logs• Annotated portfolios• Documented problem

solutions• mid-semester evaluation

(These can also be evidence towards answering your other questions)

See Angelo & Cross, Classroom Assessment Techniques

What’s Your Problem, Anyway?

“One telling measure of how differently teaching is regarded from traditional scholarship or research within the academy is what a difference it makes to have a “problem” in one versus the other.

In scholarship and research, having a “problem” is at the heart of the investigation process; it is the compound of the generative questions around which all creative and productive activity revolves.

But in one’s teaching, a “problem” is something you don’t want to have, and if you have one, you probably want to fix it. Asking a colleague about a problem in his or her research is an invitation; asking about a problem in one’s teaching would probably seem like an accusation.

Changing the status of the problem in teaching from terminal remediation to ongoing investigation is precisely what the movement for a scholarship of teaching is all about.”

Randy Bass, “The Scholarship of Teaching: What’s the Problem?” Inventio: Creative Thinking about Learning and Teaching. (1999)

Activity: Draft Your Question(s)

By Yourself -- 10 minutes

1. Brainstorm a list of questions you wish to answer at end of project

2. Narrow down list to the top 1 or 2

3. Revise the questions to 1. make them more specific2. Make them measurable

With your peer -- 5 min each

1. Explain your top 1 or 2 questions.

2. Peer’s job: Help peer revise questions.

Pairs:• Alex & Yuliya• Gary & Jason• Richard & Mary• Kate & Jimmy• Chris & Rachael

Answering Your Question with Evidence

Evidence . . .

AKA• Data• Artifacts• Student

work

Some things to consider:

• Direct vs. indirect evidence

• Qualitative, quantitative, or both?

• Embedded (“normal educational practice”) vs. add-on

• Product vs. process

Two Examples

From the “How to Start” worksheet in the Vanderbilt SoTL Guide

For literature class, in which a “difficulty log” has been introduced, to be completed as students read a text.

• Analyze the logs, looking for themes or patterns in responses. From the logs, document specific types of difficulty.

From calculus class, in which students have been asked to document their problem solving steps as a way of helping them develop metacognitive skills.

• Looking at scores on test prior to and after the new activity.

Activity: What’s Your Evidence?

By Yourself -- 5 min

Brainstorm the types of evidence you will need to collect to answer your question?

Evidence should fit:• Your questions• Your discipline• Your course• Your timeline

With your peer -- 5 min each

1. Explain your evidence choices.

2. Peer’s job: Help peer refine plan.

Pairs:• Alex & Yuliya• Gary & Jason• Richard & Mary• Kate & Jimmy• Chris & Rachael

Blog Post 5

Post to your blog your question and the evidence you plan to collect to answer that question. Discuss your choices.

Due: Fri, May 29