using peer instruction for active learning

JMU CFI May Symposium 2016 Dr. Michael S. Kirkpatrick

Teaching, Fast and Slow: Using Peer Instruction for Active Learning

Michael S. Kirkpatrick Department of Computer Science

JMU CFI May Symposium 2016


Agenda

• Introduction • Two systems model of thinking • Heuristics and biases • Why active learning? • Peer Instruction • Formative assessments and ConcepTests • Practice time


Introduction

• Assistant Professor of Computer Science • Primary research focus is CS education

• Misconceptions are central to my courses

• Started at JMU in 2011 • Prior experience teaching mathematics for liberal arts majors at Purdue


Recommended reading


Two systems model

LEFT left

left RIGHT

RIGHT left

RIGHT right

upper lower

LOWER upper UPPER

lower LOWER upper

LEFT left

left RIGHT

RIGHT left

RIGHT right

phlegm NOSE

sneeze FLU FILTHY

muddy clean

HANDS


Two systems model

S O _ P W _ S H IU

S O A P W A S H

W _ S H S O _ P


Two systems model

System 1 • Automatic and quick

• Language processing • Little or no effort • No sense of voluntary

control • Associative machine • Substitutes

easier questions

System 2 • Effortful mental activities

• Calculations and critical thinking

• Depends on system 1 impressions

• Depletes self-control • Lazy controller

• Inhibited by stress, hunger, emotion

What you see is all there is.


Two systems modelRepeated

Experience

Clear Display

Primed Idea

Good Mood

Cognitive Ease

Feels Familiar

Feels True

Feels Good

Feels Effortless


• What is my mood right now?

• Have I caused an accident recently?

• Do I observe some students doing good work?

• How happy are you with your life these days?

• Are you better at driving than the average driver?

• How effective is your style of teaching?

Two systems model


Which group can you make more words from? • XUZONLCJM vs. TAPCERHOB

Which is a more common cause of death? • Strokes or all unintentional accidents/poisonings

combined Which is more common?

• Words that start with “k” or words with “k” as the third letter

Heuristics and biases

Available examples shape

responses



Write this number down on a piece of paper:

25 Are you older or younger than this?



About what percentage of African nations are in the UN? (A) 20% (B) 45% (C) 73% (D)98%



About what percentage of African nations are in the UN? (A) 20% (B) 45% (C) 73% (D)98%

Numbers act as anchors that prime

responses



W A S H

S O A P

phlegm NOSE

sneeze FLU FILTHY

muddy clean

HANDS



Linda is thirty-one years old, single, outspoken, and very bright. She double majored in philosophy and

criminal justice. As a student, she was deeply concerned with issues of discrimination and social

justice, participated in protests against the 2003 invasion of Iraq, and served as a delegate to the

National Black Student Union Conference.



Which of the following is most likely? (A) Linda is a corporate lawyer. (B) Linda is an investment banker for Goldman Sachs. (C) Linda is an elementary school teacher in a wealthy suburban

district. (D) Linda works two jobs as a fast food dishwasher and a

custodian staff manager. (E) Linda is a corporate lawyer who does pro bono work for the

Black Lives Matter movement.



Which of the following is most likely? (A) Linda is a corporate lawyer. (B) Linda is an investment banker for Goldman Sachs. (C) Linda is an elementary school teacher in a wealthy suburban

district. (D) Linda works two jobs as a fast food dishwasher and a

custodian staff manager. (E) Linda is a corporate lawyer who does pro bono work for the

Black Lives Matter movement.

Representative examples suppress base rate statistics knowledge



corporate lawyers

BLM advocates

Linda

Not Linda



Which of these words do you associate more strongly with the passage about Linda?

old vs. liberal

The halo effect creates perceptions that match our emotions


System 1 consequences • Availability heuristic • Priming and anchors • Representative samples & base rate bias • Halo effect: emotionally coherent perceptions • Affect heuristic: preferences influence beliefs • Overconfidence (Dunning-Kruger) and procrastination • Risk assessment and prospect theory



Pedagogical implications

Think-pair-share activity • Think of an instance or a topic that your students

frequently miss on exams. Which biases, heuristics, or system 1/system 2 effects might be contributing to this?

• Pair with someone at your table and describe the instance you thought of.

• Share with everyone at your table, looking for common themes or causes.


Pedagogical implications of system 1 • Computes more than intended (mental shotgun) • Subject to biases and heuristics • Focuses on existing evidence, ignores what’s absent

• “What you see is all there is” • Can be trained by system 2 to detect patterns and execute

skilled intuitions

Pedagogical implications

Active learning and formative assessment train system 1


“Adopting instructional practices that engage students in the learning process is the defining feature of active learning.”

-Michael Prince

ALL


Why active learning?

Pre- vs. post- in physics • Mechanics Diagnostic • Force Concept Inventory • 62 courses (14 trad.) at multiple institutions

• 6542 students (2084) • Worst active learning comparable to best traditional lecture

R.R. Hake, "Interactive-engagement vs traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses," Am. J. Phys. 66, 64- 74 (1998). http://www.physics.indiana.edu/~sdi/ajpv3i.pdf

http://www.physics.indiana.edu/~sdi/ajpv3i.pdf



Metaanalysis of 225 studies • 158 studies: average 0.47 SDs better on CIs/exams

• 67 studies: average failure rate dropped from 33.8% to 21.8%

Heterogeneity analyses indicated no statistically significantvariation among experiments based on the STEM discipline ofthe course in question, with respect to either examination scores(Fig. 2A; Q = 910.537, df = 7, P = 0.160) or failure rates (Fig. 2B;Q = 11.73, df = 6, P = 0.068). In every discipline with more than10 experiments that met the admission criteria for the meta-analysis, average effect sizes were statistically significant foreither examination scores or failure rates or both (Fig. 2, Figs.S2 and S3, and Tables S1A and S2A). Thus, the data indicatethat active learning increases student performance across theSTEM disciplines.For the data on examinations and other assessments, a het-

erogeneity analysis indicated that average effect sizes were lowerwhen the outcome variable was an instructor-written course ex-amination as opposed to performance on a concept inventory(Fig. 3A and Table S1B; Q = 10.731, df = 1, P << 0.001). Al-though student achievement was higher under active learning forboth types of assessments, we hypothesize that the difference ingains for examinations versus concept inventories may be due tothe two types of assessments testing qualitatively different cogni-tive skills. This explanation is consistent with previous research

indicating that active learning has a greater impact on studentmastery of higher- versus lower-level cognitive skills (6–9), andthe recognition that most concept inventories are designed todiagnose known misconceptions, in contrast to course examinationsthat emphasize content mastery or the ability to solve quantitativeproblems (10). Most concept inventories also undergo testing forvalidity, reliability, and readability.Heterogeneity analyses indicated significant variation in terms

of course size, with active learning having the highest impacton courses with 50 or fewer students (Fig. 3B and Table S1C;Q = 6.726, df = 2, P = 0.035; Fig. S4). Effect sizes were sta-tistically significant for all three categories of class size, how-ever, indicating that active learning benefitted students inmedium (51–110 students) or large (>110 students) class sizesas well.When we metaanalyzed the data by course type and course

level, we found no statistically significant difference in activelearning’s effect size when comparing (i) courses for majorsversus nonmajors (Q = 0.045, df = 1, P = 0.883; Table S1D), or(ii) introductory versus upper-division courses (Q = 0.046, df = 1,P = 0.829; Tables S1E and S2D).

Fig. 1. Changes in failure rate. (A) Data plotted as percent change in failure rate in the same course, under active learning versus lecturing. The mean change(12%) is indicated by the dashed vertical line. (B) Kernel density plots of failure rates under active learning and under lecturing. The mean failure rates undereach classroom type (21.8% and 33.8%) are shown by dashed vertical lines.

Fig. 2. Effect sizes by discipline. (A) Data on examination scores, concept inventories, or other assessments. (B) Data on failure rates. Numbers below datapoints indicate the number of independent studies; horizontal lines are 95% confidence intervals.

Freeman et al. PNAS | June 10, 2014 | vol. 111 | no. 23 | 8411

PSYC

HOLO

GICALAND

COGNITIVESC

IENCE

SSE

ECO

MMEN

TARY

S. Freeman et al., “Active learning increases student performance in science, engineering, and mathematics,” Proceedings of the National Academy of Sciences 111(23), 8410- 8415, 2014. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4060654/pdf/pnas.201319030.pdf

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4060654/pdf/pnas.201319030.pdf


Peer Instruction

individual problems from traditional exams on the midtermsin the calculus-based course in 1991 !results reported in Ref.4". Finally, in the second semester of the algebra-basedcourse in Spring 2000 !electricity and magnetism", we in-cluded on the final exam one quantitative problem from theprevious year, when a different instructor had taught thecourse traditionally. We found that the students taught withPI !Spring 2000, N!155" significantly outperformed the stu-dents taught traditionally !Spring 1999, N!178", averaging7.4 out of 10 compared to 5.5 out of 10 !standard deviations2.9 and 3.7, respectively". The improvement of the PI stu-dents over the traditional students corresponds to an effectsize of 0.57. All measures indicate that our students’ quanti-tative problem-solving skills are comparable to or better thanthose achieved with traditional instruction, consistent withthe findings of Thacker et al.19

C. ConcepTest performance

Students’ responses to the ConcepTests themselves pro-vide further insight into student learning. We analyzed stu-dent responses to all of the ConcepTests over an entire se-mester, and find that after discussion, the number of studentswho give the correct answer to a ConcepTest increases sub-stantially, as long as the initial percentage of correct answersto a ConcepTest is between 35% and 70%. !We find that theimprovement is largest when the initial percentage of correctanswers is around 50%.4" In addition, the vast majority ofstudents who revise their answers during discussion changefrom an incorrect answer to the correct answer. Figure 4shows how students change their answers upon discussionfor all of the ConcepTests used during the Fall 1997 semes-ter. The answers are categorized as correct both before andafter discussion !‘‘correct twice’’", incorrect before and cor-rect after discussion !‘‘incorrect to correct’’", correct beforeand incorrect after discussion !‘‘correct to incorrect’’", orincorrect both before and after discussion !‘‘incorrecttwice’’". Nearly half of the correct answers given were ar-rived at after discussion, and students changed from correct

Table I. Force Concept Inventory !FCI" and Mechanics Baseline Test !MBT" results.a

Year MethodFCIpre

FCIpost

Absolute gain(post"pre)

Normalizedgain #g$ MBT

MBT quant.questions N

Calculus-based1990 Traditional !70%" 78% 8% 0.25 66% 62% 1211991 PI 71% 85% 14% 0.49 72% 66% 1771993 PI 70% 86% 16% 0.55 71% 68% 1581994 PI 70% 88% 18% 0.59 76% 73% 2161995 PI 67% 88% 21% 0.64 76% 71% 1811996 PI 67% 89% 22% 0.68 74% 66% 1531997 PI 67% 92% 25% 0.74 79% 73% 117

Algebra-based1998 PI 50% 83% 33% 0.65 68% 59% 2461999 Traditional !48%" 69% 21% 0.40 ¯ ¯ 1292000 PI 47% 80% 33% 0.63 66% 69% 126

aThe FCI pretest was administered on the first day of class; in 1990 no pretest was given, so the average of the1991–1994 pretest is listed. In 1995 the 30-question revised version was introduced !Ref. 15". In 1999 nopretest was given so the average of the 1998 and 2000 pretest is listed. The FCI posttest was administered aftertwo months of instruction, except in 1998 and 1999, when it was administered the first week of the followingsemester to all students enrolled in the second-semester course !electricity and magnetism". The MBT wasadministered during the last week of the semester after all mechanics instruction had been completed. For yearsother than 1990 and 1999, scores are reported for matched samples for FCI pre- and posttest and MBT. No dataare available for 1992 !EM was on sabbatical" and no MBT data are available for 1999.

Fig. 3. Mechanics Baseline Test !Ref. 13" scores for introductory calculus-based physics, Harvard University, Fall 1990–Fall 1997. Average score onentire test !circles" and on quantitative questions !Ref. 17" only !squares" vsyear are shown. Open symbols indicate traditionally taught courses andfilled symbols indicate courses taught with PI. The dotted line indicatesperformance on quantitative questions with traditional pedagogy !1990".

Fig. 4. Answers given to all ConcepTests discussed in Fall 1997, catego-rized as described in the text.

972 972Am. J. Phys., Vol. 69, No. 9, September 2001 C. H. Crouch and E. Mazur

These were NOT “worse” students

These were NOT “worse” students (post - pre)

1 - pre


Peer Instruction

Class structure • Daily or weekly reading quizzes to address basic definitions • Minimum of 15 minutes per concept

• 7-10 minutes of lecture • 5 - 8 minutes for ConcepTest

• Vote first, discuss, vote again, debrief • Exams aligned with concepts

• Not multiple choice, emphasize core ideas • 3-2-1-0 grading scheme

Graded but low-stakes

Can be Canvas/JiTT


Peer Instruction

Reading Quiz Question Without air resistance, an object dropped from a plane flying at constant speed in a straight line will 1. quickly lag behind the plane 2. remain vertically under the plane 3. move ahead of the plane 4. not covered in the reading assignment

Exam Question Two stones are released from rest at a certain height, one after the other. (a) Will the difference in their speeds increase, decrease, or stay the same? (b) Will their separation distance increase, decrease, or stay the same? (c) Will the time interval between the instants at which they hit the ground be smaller than, equal to, or larger than the time interval between the instants of their release?


Peer InstructionConcepTest

A person standing at the edge of a cliff throws one ball straight up and another ball straight down at the same initial speed. Neglecting air resistance, the ball to hit the ground below the cliff with the greater speed is the one initially thrown 1. upward. 2. downward. 3. neither—they both hit at the same speed.

Answer: 3. Upon its descent, the velocity of an object thrown straight up with an initial velocity v is exactly -v when it passes the point at which it was first released.


Peer InstructionConcepTest

Consider the following statement: Bill should quit smoking because people who smoke get lung cancer.

What is the best interpretation of the premise? 1. Most people who smoke get lung cancer. 2. Everyone who smokes gets lung cancer. 3. Some people who some sometimes get lung cancer. 4. People who smoke are more likely to get lung cancer than people who

don’t smoke.

Answer: 4. This example illustrates the principle of charity by introducing the claim of likelihood into the speaker’s assertion.

Peer Instruction in the Humanities Project, Monash University http://artsonline.monash.edu.au/peer-instruction-in-the-humanities

http://artsonline.monash.edu.au/peer-instruction-in-the-humanities


Peer Instruction

Examples of Peer Instruction in the humanities • Have students select the main idea in a text, audio, or

video clip. • Engage students in discussion about a key point in an

image, text, audio, or video clip that may have more than one valid answer.

• Solicit views on contentious points and explain their perspectives to peers.

Peer Instruction Blog https://blog.peerinstruction.net/2013/06/10/3-easy-ways-to-use-clickers-and-peer-instruction-in-the-arts-and-humanities/

https://blog.peerinstruction.net/2013/06/10/3-easy-ways-to-use-clickers-and-peer-instruction-in-the-arts-and-humanities/


ConcepTests

Guidelines for writing ConcepTests • Focus on a single concept • Require conceptual understanding, not

solvable with just equations or facts • Use unambiguous language • Use students’ wrong answers from old exams as options • Avoid converging answers • Un-situate concepts with near and far transfer • Be aware of intrinsic, germaine, and extraneous cognitive load

• Provide worked examples, not just solutions


Far transfer

90 ft

90 ft x

Apply the Pythagorean theorem to the above triangle to find the value of x.

In a baseball diamond, the distance between each base is 90 ft. Which of the following is true about the shortest distance between 1st and 3rd bases (the red line shown above)? 1. It is less than 90 ft. 2. It is between 90 and 180 ft. 3. It is greater than 180 ft.

90 ft 90 ft

90 ft 90 ft


Work time

ConcepTest writing • Recall the learning objective that you identified as

challenging for students in the think-pair-share • Construct a stem for that objective • Write a misconception or two that students have about

the topic • Rephrase the misconception as an answer • Test your ConcepTest on a peer from a different

discipline


Backup slides


ConcepTests

The Force Concept Inventory (FCI) is a validated test to measure learning in physics. Students take the FCI on the first and last day of class. In one study, women scored an average of 60% on the pretest; women in traditional lectures scored an average of 70% on the posttest, whereas women in interactive classrooms scored an average of 85% on the posttest. Men scored an average of 70% on the pretest; men in traditional lectures scored an average of 80% on the posttest, compared with 87% in interactive classrooms.


ConcepTests

Which statement best summarizes the pre- vs. posttest gender performance gap in the physics courses studied?

(A) In both traditional and interactive classrooms, the gender performance gap increased on the post-test.

(B) The gender performance gap increased in traditional lectures but stayed constant in interactive classrooms.

(C) The gender performance gap stayed constant in traditional lectures but decreased in interactive classrooms.

(D)The gender performance gap increased in traditional lectures but decreased in interactive classrooms.


ConcepTests

Cooperative learning closes the gender gap

• Pre-test scores were 10% points higher for men

• Gap persisted with lecture alone

• Post-test results for cooperative classes were almost equal

• Requires more than just interactive lecture

T: traditional lectures IE: interactive lectures IE+: interactive assignments, lectures, tutorials

E. Mazur, “The scientific approach to teaching: Research as a basis for course design,” keynote/plenary talk at the International Computing Education Research Conference (ICER), 2011. http://mazur.harvard.edu/search-talks.php?function=display&rowid=1712


ConcepTests

The evidence supporting active learning suggests that passive engagement with information can contribute to reinforcing misconceptions. One study examined how effective in-class physics demonstrations were in helping to understand concepts. All students began with a reading assignment. Some students then took part in an in-class demonstration; the control group did not observe or take part in a demonstration. The students completed a short test to conclude the experiment. Which group did the worst on the post-test, missing the most points?


ConcepTests

Which group did the worst on the post-test, missing the most points?

(A) Students who did not observe a demo (control group) (B) Students who only observed the demo (C) Students who predicted the outcome before it occurred by

writing down a guess (D) Students who discussed the outcome with peers after

observing what occurred


Two systems model

Christopher Chabris and Daniel Simons The Invisible Gorilla http://www.theinvisiblegorilla.com/videos.html

http://www.theinvisiblegorilla.com/videos.html

using peer instruction for active learning

Documents