Active Learning with Clickers and Classroom Voting

Kelly Cline

Carroll College

Helena, Montana

1. What is classroom voting?2. What does the research say?

Conceptual questions and discussions are the key!3. Classroom voting at Carroll College since 20044. How do we use classroom voting?5. Past voting results: A gold mine of information!6. Classifying Questions by past results:

1. Quick Check Questions2. Discussion Questions3. Misconception Magnets4. Checkpoint Questions

7. A call for collaborators: Please try out our questions and send us your voting results!

Outline:

What is classroom voting?

1. The instructor presents a multiple-choice questions to the class.

2. Students are given a few minutes to form an opinion and to discuss the question with their neighbors.

3. Each student votes on the answer thus requiring every student to form an opinion.

4. The instructor leads a class-wide discussion, asking individual students to explain and defend their vote.

Voting has often been run with clickers…

Newer technologies are developing…

• http://www.polleverywhere.com

Students register votes with phones, tablets, web browsers. Votes are displayed on a web page. (Private company with fees)

• http://ripple-core.uoregon.edu/

Students log in with their phones and tablets. (Free and open source.)

After voting is closed, the instructor displays a graph of the results, and leads a class-wide discussion.

Results of Classroom Voting:Students Get Engaged

• Each student must take an active role, trying out a mathematical idea, and discussing it with their peers. Voting breaks them out of passive-lecture-mode, and gets them thinking and participating.

• In general, active learning methods tend to produce deeper conceptual knowledge, which is retained longer.

• Students have fun! They enjoy the class far more, because they are actively involved. Many report that attendance improves and attrition drops.

Peer Instruction: 2 Cycles of VotingOne important style of voting is called “Peer Instruction” originally developed for physics by Eric Mazur at Harvard.

• First the question is posed, and the students must consider and vote on the answer individually: No discussion is allowed. The voting is closed, but the results are not displayed.

• Then the students discuss their answer with their neighbors, and they get to vote again.

• After the second round of voting, the results are displayed, and the instructor guides a discussion.

The Education ResearchTeaching with clickers and classroom voting has been studied extensively at the college/university level, in a wide range of disciplines, across many types of institutions.

Here’s a quick sampler of a few key studies…

In Physics: Test Scores

Improve

In one recent study at the University of British Columbia, two sections of physics enrolling 267 and 271 students, were taught by two experienced instructors until week 12. At this point one section was taught by pair of novice instructors using clickers and voting. An exam was given to both sections. The test class performed dramatically better, with a mean 2.5 standard deviations above the control.

Deslauriers et al. Science. 332 862-864 (2011).

In Physics: Attrition drops substantially

In introductory physics, it was found that sections taught with peer instruction had substantially fewer students drop the course, both at Harvard and at a 2-year college.

Lasry et al. Am. J. Phys. 76 1066-1069 (2008)

In Biology: Students learn from the small group discussions.

In undergraduate genetics, the instructor immediately followed up a 2-cycle vote on one question with a second question on the same topic. No graphs were displayed, there was no class-wide discussion, and the instructor did not teach between the two questions. Yet, student scores improved, particularly on difficult questions.

Smith et al. Science 323 122-124 (2009).

More Results• When students perceive voting as being used primarily

for the teacher’s benefit (for attendance, to quickly grade quizzes), then they are more likely to resent clickers. When students perceive voting as being used primarily for their own benefit, they enjoy and appreciate clickers. (Trees & Jackson 2009)

• Students stay more alert and attentive to the lesson when clickers are used, especially when clicker questions are scattered throughout the period. They pay more attention because they never know when a question may come up! (Hoekstra 2008)

In Math: How do students react?

Post-course surveys were given to 513 students in 26 courses, taught by 14 instructors at 10 different institutions, all of which used voting and discussion as a primary teaching method. (Avg. class size = 20)• 93% said that voting makes the class more fun. • 90% said that voting helps them engage in the material.• 84% said that voting helps them learn. • 77% said they would choose a voting section of a

mathematics class over a non-voting section.

Zullo et al. MAA Notes: Teaching Mathematics with Classroom Voting. 2011.

In Math: The Cornell Study

To try and learn if classroom voting makes a difference, and if so, what types of voting work best, a study was conducted at Cornell:

330 students, in 17 sections of Calculus I, taught by 14 different instructors, who were each free to choose if and how they wanted to use classroom voting.

Common exams were given to all sections.

Miller, Sanatana-Vega, & Terrell PRIMUS, volume 16, #3

How was voting used?

Classes were grouped into 4 categories:

• Deep: Classes asked lots of deep and probing conceptual questions (1-4 times per week) with 2-cycle voting, including regular pre-vote peer discussions.

• Heavy Plus Peer: Voting used 3-4 times per week with 2-cycle voting and regular pre-vote peer discussions.

• Heavy Low Peer: Voting used 3-4 times per week but usually one vote/question with no pre-vote peer discussions.

• Light to Nil: Voting used 1-2 times per week or not at all.

The Results

The final exam was out of 150 points, with 130 being an A, and 116 being a B-.

Classes which asked lots of deep and probing conceptual questions to stimulate peer discussions saw the best results, with statistically significant differences. Voting without discussion had no significant effect.

Who benefited?

• All demographic groups showed statistically significant differences on the final exam between classes with voting and pre-vote peer discussion and those without the discussions.

• The most dramatic difference was on the score of under represented minorities.

• With voting and discussion: Mean score = 117, B-.• Without: Mean score = 106, C.

Classroom Voting at Carroll College

• A liberal arts college in Helena, Montana,1400 students• Math class sizes are usually 15 to 35 students• Graduate 8 to 12 math majors each year• Began voting in Fall 2004, in calculus, using questions

from the Cornell GoodQuestions project, a collection from Mark Schlatter (Centenary College), and others.

• Very positive student reactions• Students actively worked to figure out key ideas during

class through discussion: What more could we want?• NSF Funded: Project MathQuest (2006-2009)• NSF Funded: Project MathVote (2010-2013)

Discussions are the key!

• There are many different things you can do with clickers. The research and our experience indicate that deep questions and small group discussions produce excellent results.

• Learning to “talk math” means learning to “think math.” Students learn to logically defend mathematical ideas!

• We don’t give points or penalties for right or wrong answers. In our small classes, we don’t even have to give participation points. The purpose of the questions is to teach, not to assess and evaluate.

How do we use voting?

On the first day:

• Explain to the class that we’ll be voting on multiple-choice questions several times in most class periods.

• We emphasize that voting is for the students benefit. They will learn math better through voting and discussion than they would just by passively watching the teacher do examples on the board.

• Clearly state a set of rules for voting…

Rules for Voting

1. Everyone must vote.

2. No one is allowed to vote until they have discussed the question with at least one other person.

3. After the vote several students will be called on to explain their votes.

4. It doesn’t matter whether you vote right or wrong, as long as you can explain your thinking. The only unacceptable response is “I just guessed.”

Let the votes trickle in…

It is hard to predict how much time the class will need with any given question.

Tell the students to vote when they’re ready, after some discussion.

The software counts up votes as they are received. When about 60 – 75% of the class has voted, we call for the rest to finish up.

Make a headcount, and require everyone to click in before closing the vote.

Conducting the Post-Vote Class-Wide Discussion

1. Ask a student: “What did you vote for, and why?”

2. Initially, give no feedback as to whether this was right or wrong, and ask the same question of another student, encouraging the students to figure out the correct answer for themselves.

3. If students express contradictory arguments, ask them to respond to each other.

4. If possible, confirm the correct answer after most have already figured it out.

Getting student buy-in• We vote several questions in almost every class

period. Questions are usually interspersed throughout the class.

• Students must see that questions form a vital part of the lesson, rather than being an optional add-on to be squeezed in if time permits.

• Students must see that essential ideas are taught through the questions and resulting discussion.

• The questions must be directly relevant and help prepare students for the assignments and tests which follow.

http://mathquest.carroll.edu• Approximately 2,300 multiple-choice questions

designed for classroom voting in university mathematics

• Questions are freely available for download in pdf format, either in:

• Large font, for cut/pasting into PowerPoint

• Small font, for printing & handing out a numbered list to students

• Teacher’s edition and LaTeX files available with an e-mail request to me: kcline@carroll.edu

Subjects and NumbersAlgebra: 200 questions

Statistics: 107 questions

Precalculus: 226 questions

Differential Calculus: 192 questions

Integral Calculus: 151 questions

Multivariable Calculus: 317 questions

Linear Algebra: 311 questions

Differential Equations: 350 questions

And more!

Topics and CoverageMost of our collections are not closely tied to specific textbooks. Topics are general. For example in linear algebra we begin with:• Systems of Equations• Matrix Representations of Systems of Equations • Gaussian Elimination • Solution Sets of Linear Systems • Linear Combinations • Linear Independence • Rank of a Matrix • Matrix Operations• Matrix Inverses• Determinants • Eigenvalues and Eigenvectors

The Student Edition• Freely available for download by anyone

• Questions numbered sequentially:

The Teacher’s Edition

• Freely available with an e-mail request kcline@carroll.edu

• Question numbers match the student edition

• The correct answer or answers are indicated

• A solution/discussion is usually presented

• Authorship is identified (please share your questions and allow us to include them!)

• Past voting statistics are presented

• Often pre-vote times are recorded as well (min:sec)

Selecting the Right Questions is Essential!

Voting requires a substantial investment of class time, often about 3 to 5 minutes per question, and we vote several times in each class period.

1. Each question must be directly relevant and teach an important idea, thus replacing lecture-time.

2. Students must see how the voting questions help them work through the assignments that follow.

3. Research indicates that the most effective questions provoke small-group discussions about important issues.

That’s a tall order!

How do you find the right questions?• Clickers by themselves are not magic: Using clickers in

your course can dramatically change things for the better, or be a colossal waste of time!

• It’s all about the questions that you use.• Creating really good multiple-choice questions is hard!• It’s rarely obvious whether or not a question is worth

investing the time to make a vote.• What types of questions will be likely to produce good

discussions, both small-group discussions before each vote and class-wide discussions after each vote?

• How can we identify the best questions?

Our Innovation:Record Voting Results

• We began informally recording the percentage of the class voting for each option back in the fall of 2004, just so we could compare how two parallel sections voted on the same questions.

• We realized that this could be useful and interesting information, and it’s not difficult to record.

• We began writing these down on our teacher’s edition every time a question was voted, then adding these to the LaTeX files of our teacher’s edition.

• Friends at other institutions began contributing…

Past voting results are useful for lesson planning

Our collection has 10 or 20 questions available for many topics, and I have time to vote perhaps 3 to 8 questions in a 50 minute class.

Using records of past voting results, I can quickly identify the questions that are more likely to challenge the students and provoke serious discussions.

Every class is different, and there are no guarantees, but past results are a very useful guide!

A Research Project:Study Past Voting Results

• There are many questions for which we now have records of five, ten or more votes.

• Together, these votes can give us insights into student thinking, and common challenges.

• That’s interesting whether you use clickers or not!

• Many questions produce very different responses from the different classes taught by different instructors at different institutions. But some questions produce remarkably similar votes.

Identifying Types of Questions

Quick Check: A strong majority regularly votes correctly.

Discussion Questions: Votes are widely distributed over different answers. We look for the smallest average “winner.”

Misconception Magnets: A strong majority regularly votes for a particular incorrect answer.

Checkpoint Questions: The percent voting correctly varies widely between classes. Look for the largest STD among the percent voting correctly.

Voting Patterns• We have 192 questions on differential calculus.• Each point represents one of the 101 questions for

which we have at least 5 votes.

Quick Check Questions• In the lower right, we see lots of questions where most

students vote correctly most of the time.• Straight-forward practice with a new ideas

A Quick-Check on the Product Rule

• It’s better for students to do the examples themselves, rather than watching us do them on the board.

• But Quick-Check questions rarely produce good discussions…

Identifying Discussion Questions

• Good discussions produce great learning benefits.

• If most of the students agree with each other (rightly or wrongly), small groups will have less to talk about.

• If substantial numbers of students vote for several different options, small groups are more likely to have contrasting opinions, and thus something to discuss!

• We look for questions with the smallest average winner, the option (a,b,c,d) getting the most votes.

Good Discussion Questions are Not Usually Obvious!

• Often I have written a question, expecting it to provoke a good discussion, but the issue is too obvious (or too subtle) and falls flat.

• Other times, a question that appears straight-forward to me brings up a wide variety of important issues, stimulating an excellent discussion.

• Past votes are no guarantee – every class is different! – but they can be a very useful guide.

Second Derivatives

We have spent most of the class period developing an understanding of the meaning contained in a second derivative function. We have answered questions relating the second derivative to concavity, we have matched graphs of functions with graphs of the second derivative, and we have discussed how if the original function gives position as a function of time, then the second derivative gives acceleration.

A Great Question:

In Star Trek: First Contact, Worf almost gets knocked into space by the Borg. Assume he was knocked into space and his space suit was equipped with thrusters. Worf fires his thrusters for 1 second, which produces a constant acceleration in the positive direction. In the next second he turns off his thrusters. In the third second he fires his thruster producing a constant negative acceleration. The acceleration as a function of time is given in Figure 2.31. Which of the following graphs represent his position as a function of time?

(ConcepTests for Calculus by Hughes-Hallett et al.)

The Graphs:

We have about a dozen questions on this topic, but I would never miss asking this one: This is a gem!

Small Avg. Winner: Good Discussions

Here, we calculate that for this question, the average winner gets only 46% of the vote: A very small number!This method usually picks out great discussion questions.

(a) (b) (c) (d) Winner

Class 1 45% 0% 35% 20% 45%

Class 2 13% 0% 48% 39% 48%

Class 3 40% 5% 40% 15% 40%

Class 4 48% 7% 7% 37% 48%

Class 5 35% 0% 41% 18% 41%

Class 6 23% 0% 54% 23% 54%

Avg. 34% 2% 38% 25% 46%

Misconception Magnets

• These are questions where a large majority regularly vote for a specific wrong answer. The pre-vote small group discussions may be brief, but the post-vote discussions can be interesting: Students expect the majority to be correct, and are surprised and intrigued to find out that they were wrong!

• Sometimes small group discussions can be created by simply telling the class that the majority is wrong and calling for another round of voting.

Misconception Magnets• In the lower left, we see a few questions where a strong

majority of students regularly vote incorrectly.

A Misconception Magnet

A Misconception Magnet

Most students apply the product rule and get which is not one of the options, so they vote for (e). But (a), (b), and (c) are all equivalent to this, and so (d) is the correct answer. A few students object that their calculators give them (a) or (b)…

Results from this question.

We identified this question by calculating the average percentage voting for the most population incorrect option: Here this is 57%, the largest of the 101 questions. This is an awesome question for post-vote discussions.

A Misconception MagnetFrom Early Calculus II

We have computed a few left-sums and right-sums, and then tried averaging them, seeing a few examples where this is more accurate.

True or False: Averaging left- and right-sums always improves your estimate. (GoodQuestions)

Our Statistics (T/F): 70/30, 90/10, 90/10, 100/0

The results are quite devastating! A great follow up at this point is to have the students work with their peers to create sketches of counterexamples.

In Differential Equations: Continuity is a rich topic…

We know that a given differential equation is in the form y’(t) = f (y) where f is a continuous function of y. Suppose that f (5) = 2 and f (–1) = –6.

(a) y must have an equilibrium value between y = 5 and y = –1.(b) y must have an equilibrium value between y = 2 and y = –6.(c) This does not necessarily indicate that any

equilibrium value exists.

(a) Because f is continuous, this means that if f goes from positive to negative, then there must be a zero, an equilibrium value between them.

Worse is Better!

Results: Class 1 33/10/57 time 1:45Class 2 57/43/0Class 3 26/15/59 time 2:50Class 4 0/0/100Class 5 50/17/33 time 2:20Class 6 41/0/59 time 2:45

There are two levels to this question, interpreting the notation, then applying the idea of continuity. Both (a) and (b) use the word “must” and students quickly learn to be cautious about such words in mathematics. Good!

Checkpoint Questions• Clickers produce an instant snapshot as to the state of

my class, with results from every student.

• Do they understand, or do we need more discussion?

• Reality check: A few bright and vocal students can easily persuade an instructor that a topic has been mastered, while most of the class is still lost!

• We can identify these from past voting statistics by looking for large standard deviations in the percent voting correctly.

Checkpoint Questions• At the top, we see questions where different classes

voted very differently.

From Calculus I: Optimization

This is false because a local maximum can also occur at a point where the derivative is undefined. Students either recall this, or they do not: a focused question, a key point!

Taking derivatives of logarithms

We have a large standard deviation in the number voting correctly because of one section where no one voted correctly. What happened?

Questions can be used in different ways

In most classes, this was used as a quick-check, a simple question to practice a newly introduced idea.

In the other class, the question was used at the very beginning of the period, before the natural log rule had been presented, with the expectation that students should have read the text.• Students referred to their text, worked together, and

settled for options (c) and (d). • No one voted correctly, but the discussion and follow-

up was great! They worked out most of the problem by themselves!

During Antiderivatives

True or False: If two solutions of a differential equation dy/dx = f (x) have different values at x = 3 then they have different values at every x. (GoodQuestions)

Our Statistics (T/F): 90/10, 10/90, 100/0, 80/20, 85/15

In most classes a large majority of students get this one right, except one class where only 10% got it right. The statistics immediately identify this as a big warning flag: What’s going wrong? What are they missing? What did we do different this time? Some crucial concept has not been mastered! This question is worth asking!

Please join us!

• Please use our questions and record the results of your votes, so that we can put them into the teacher’s edition.

• Past voting statistics are very useful in lesson planning. They also provide fascinating insight into common student errors and issues.

• We have already written several papers analyzing these results. We plan on writing lots more. If we write a paper including your data, we will be happy to include you as a co-author.

Pedagogy is the key,& clickers make it possible

The best results come from a particular teaching technique, not from a particular technology. Clickers can be used in ways that do not engage students in the material and do not involve them in discussions.

The clickers are fun, and the students enjoy registering their votes, but they are really just the “magic feather” the “spoonful of sugar that helps the medicine go down!”

Give Clickers and Voting a Try!• You get immediate feedback.

• Students actively engage in the classroom. It keeps them involved, they have fun, & attendance improves.

• Use voting to replace parts of your lecture, drawing out ideas that you would normally explicitly present.

• Emphasize that the pre-vote discussions are the real engine of this process. When your students are discussing and debating mathematical ideas with their peers, then real learning is taking place!

Our Overall Results• The students love it. They tell us how much fun

they’re having, and complain when we have a non-voting day.

• When asked if they had to choose between a class with voting and one without, a strong majority would choose the one with voting!

• Attendance is outstanding. When students are involved in the class, they are more likely to come.

• Our anecdotal testimony: Our students are learning, they’re having fun, and we’re attracting more math majors and minors.

MAA Notes: Teaching Mathematics with Classroom Voting

• Released September 2011 as an e-book• Edited by Kelly Cline & Holly Zullo • 24 chapters by authors at a wide range of

institutions discussing classroom voting in:

• College Algebra• Calculus• Statistics• Linear Algebra• Differential Equations• Abstract Algebra• And more!

Thank you for your attention!

Any questions?

Looking for Misconception Magnets

We go through the data set, looking for questions where on average the largest percent of students voted for the most popular incorrect answer.

The context:

After studying double integrals, we have introduced triple integrals and how to calculate them. Now, we want to get students to think about what triple integrals might mean…

A simple triple integral

The Results

On average, 57% of students vote (a), the most popular incorrect option. As a result this question ranks highly as a misconception magnet, and our experience confirms that this is a great question. It regularly produces some excellent discussions!

From Linear AlgebraWe discussed systems of two equations and two unknowns, learning to solve them algebraically, graphically, and then by putting an augmented matrix into reduced row echelon form.

Then rather than explicitly teaching what can happen with larger systems, I got the students thinking about these issues with the following question.

A Linear Algebra QuestionA system of 3 linear equations and 3 variables could not have exactly _____ solutions.

a) 1b) 2c) 3d) Infinitee) More than one of these is impossiblef) All of these are possible numbers of solutions

Results indicate a good discussion question:

The Resulting DiscussionOnly a third of the class initially voted for the correct answer, (e) that this system could not have exactly two or three solutions.

Some students drew diagrams and tried to explain things in terms of lines and intersections in two dimensions, which didn’t work. To help, I introduced the idea that a linear equation with three variables can be represented as a plane in three dimensions.

The discussion came to a resolution when a student talked about representing this with an augmented matrix and asked what two or three solutions would look like when the matrix was put in reduced row echelon form.

A Follow-up QuestionA system of 5 linear equations and 7 variables could

not have exactly ______ solutions.a) 0b) 1c) infinited) More than one of these is impossible.e) All of these are possible numbers of solutions.

Students focused more quickly on what the augmented matrix would look like when put in r.r.e.f. Again, the discussion was very fruitful, bringing up lots of key ideas.