Andrew Mason, Ph.D.Kenneth Heller, Leon Hsu, Anne Loyle-Langholz, Qing Xu University of Minnesota, Twin CitiesMAAPT Spring 2011 MeetingSt. Mary’s University, Winona, MN4/30/2011

Supported by NSF DUE #0230830 and DUE #0715615 and by the University of Minnesota

Background Demonstration Preliminary findings

▪ Initial calibration data▪ Currently: Scoring using Rubric for Problem

Solving (Docktor 2009) Methodology for further study

Use coaches in a calculus-based physics course for scientists and engineers Students are asked to volunteer; volunteers

are paid for their participation

Assign students into 2 matched groups Variables for matching: background information, e.g.

HS physics & math level, FCI/CLASS/math pretests

Study #1 (Fall 2010) ~40 students, 1 lecture session of intro

calculus-based class (~20 for each group)

Subset of coaches available – energy (8), momentum (7) – done over 4 weeks (~4 per week)

Retention rate: high in fall (18 of 21) for 15 coaches 2 others completed 12 of 15 All students found them useful

▪ Does this hold for other sections?▪ What will happen with a larger set of coaches?

Student preference for each type Faculty tend to disagree with students

(found type 1 tedious) 1 student initially preferred type 1 but

switched to type 3 after gaining familiarity with physics

Most useful 2nd most useful

Least useful

Type 1 13 3 1

Type 2 0 9 9

Type 3 4 5 8

• Average time range to complete between coaches: between 20 and 40 minutes

Percentage of correct answers seems to correlate with background info

2 sampled “A” students: 80-90% 2 sampled “C” students: 60-70% Students tend to stay on task

Only 2 of 18 had at least one break of more than 5 minutes for a question

Time between questions(one type 1 coach)

Distribution suggests students are taking the tutors seriously Median time = 4.58

s

0

100

200

300

400

500

0 5 10 15 20 25 30F

requ

ency

Time (seconds)

Rubric developed to evaluate student problem solutions Validity, reliability have been tested 2 raters use, discuss with each other until >90%

agreement Five rubric categories (established by

research on expert and novice problem solvers) Useful Description Physics Approach Specific Application of Physics Mathematical Procedures Logical Progression

Study #2 (Spring 2011) 9 students, 1 lecture session of intro calculus-based

class Coaches available for 4 segments: kinematics (3),

dynamics (4), COE (8), COM (7)

Good time to establish baseline of rubric scores for general class Eventual comparison with computer coach

users 2 expert raters – PER researchers 23 students (3 tiers according to pretests);

eventually will expand to ~40 13 problems (8 from quizzes, 5 from final)

- Standard error bars are on the order of +/- 1 to +/- 2 out of 5

Raters: 2 experts/PER researchers

~90 students, 1 lecture session of intro calculus-based class (~45 for each group) Comparison group given equal face time

with problems used in coaches

All coaches available (kinematics, dynamics, energy, momentum, rotational motion) 8 x 5 = 40 total

Short-term questions: Will students use coaches? How will students use them? (keystroke

function) Do they improve students’ problem solving

skills with respect to baseline scores? (rubric scoring of quizzes)

Longer-term questions: Are they adaptable to be used in teaching

other physics courses? Possible software/AI development – refine

beta versions; make it more able to follow student preferences

Can this software be modified by instructors to fit a different problem solving framework?

Summary Initial results seem to have much to say; need to be

expounded upon Currently examining a baseline of exam

performance to compare to future data from computer coach users

Website: http://groups.physics.umn.edu/physed Can look at research, previous talks and publications Try out the coaches! Give us feedback!http://groups.physics.umn.edu/physed/prototypes.html

Treatment and Comparison groups• Treatment group – computer coaching (on Web,

outside of class), 4 problems per week • Comparison group – normal class setting

Data collection• Diagnostic pretests and posttests• Written solutions on quizzes & final exam

• 2×4+5=13 for each student • Problem-solving interviews with students

Can look at individual time on each question for each student

A few questions take some time regardless of student Entering answer into calculator

Can look for patterns in other questions

Chi, Feltovich and Glaser, “Categorization and representation of physics problems by experts and novices,” 1981.

Collins, Brown and Newman, “Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics,” 1989.

Docktor and K. Heller, “Robust assessment instrument for student problem solving,” 2009; also see J. Docktor, dissertation, 2009.

P. Heller and K. Heller, “The Competent Problem Solver: A Strategy for Solving Problems in Physics,” 1995.

P. Heller and Hollabaugh, “Teaching Problem Solving Through Cooperative Grouping. Part 2: Designing Problems and Structuring Groups,” 1992.

Hsu, Heller, Mason, and Xu, Summer AAPT presentation, Portland, OR, 2010.

Larkin, McDermott, D. Simon and H. Simon, “Expert and Novice Performance in Solving Physics Problems,” 1980.

Newell and Simon, “Human Problem Solving,” 1972.

Palincsar and Brown, “Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities,” 1984.

Polya, “How to Solve It,” 1945; 1957. Polya, “Mathematical Discovery,” 1962. Reif and Scott, “Teaching Scientific Thinking

Skills: Students and Computers Coaching Each Other,” 1999; also see L. Scott dissertation, 2001.