loughborough dhd seminar
DESCRIPTION
TRANSCRIPT
Physics Education ResearchThe University of Edinburgh
Diagnosing students’ data handling skills
Simon Bates Ross Galloway
Helen Maynard-Casely
Hilary Singer Kate Slaughter
Physics Education ResearchThe University of Edinburgh
• Diagnostic tests: what, why and how?• Existing diagnostic tests• Why investigate data handling skills?• Development of the diagnostic• Initial results• Development of supporting resources• Summary
Outline
Physics Education ResearchThe University of Edinburgh
Diagnostic tests: what?
• Standardised, statistically validated test.
• Often in multiple choice question (MCQ) format.
• Targeted to address key learning outcomes.
• Often used to identify misconceptions.
Physics Education ResearchThe University of Edinburgh
Diagnostic tests: why?
Ausubel’s Dictum: “Ascertain what the student knows and teach accordingly”
“Ascertain what the student misunderstands and teach accordingly”
Physics Education ResearchThe University of Edinburgh
Diagnostic tests: how?
Often administered using ‘pre / post’ scheme.
Pre-test
Respond
Instruct
Post-test
Physics Education ResearchThe University of Edinburgh
Existing diagnostic tests
• Scientific reasoning (Lawson)• Student attitudes (CLASS) • Graphs (TUG-K) • Forces and statics (FCI) • Electricity and magnetism (BEMA, homegrown) • Quantum mechanics (QMCS, QPCS, homegrown)• Light and spectroscopy Concept Inventory (Bader)• Data handling (UBC/homegrown)• …
Physics Education ResearchThe University of Edinburgh
Force Concept Inventory… a mature, established diagnostic test.
Physics Education ResearchThe University of Edinburgh
Labs: the implicit curriculum• In practical work, we expect students
to acquire data analysis skills in parallel to practical abilities.
• Frequently, these generic skills are not explicitly taught and not effectively assessed.
Physics Education ResearchThe University of Edinburgh
HEA Student Employability Profiles
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0y-
axi
s
x -ax is
Typical question (from UBC pre-prototype)
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0y-
axi
s
x -ax is
A
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0y-
axi
s
x -ax is
B
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0y-
axi
s
x -ax is
C
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0y-
axi
s
x -ax is
D
Physics Education ResearchThe University of Edinburgh
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0
y-a
xis
x -ax is
A
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0
y-a
xis
x -ax is B
0 5 1 0 1 5 2 0-5
0
5
1 0
1 5
2 0
y-a
xis
x -ax is
C0 5 1 0 1 5 2 0
-5
0
5
1 0
1 5
2 0
y-a
xis
x -ax is D
Physics Education ResearchThe University of Edinburgh
No statistically significant difference between 1st, 2nd and 4th year classes.
Response profiles from Edinburgh Physics classes
Physics Education ResearchThe University of Edinburgh
Where we went from here…
• Build on pre-prototype UBC instrument.
• Expand number of questions, increase scope.
• Ensure generic nature so wide applicability: physics, chemistry, engineering, other STEM, etc.
• Ensure test is robust and reliable.
Physics Education ResearchThe University of Edinburgh
Diagnostic test development process
Beichner R. J., (1994) Am. J. Physics 62, 750.
Physics Education ResearchThe University of Edinburgh
23 item MCQ test to address generic data analysis skills. Wide applicability to all STEM disciplines.
Topics covered include:
o precision and accuracy
o approximations
o lines of best fit
o errors and uncertainties
o derivation of quantitative relationships from graphical data
Data Handling Diagnostic – prototype test instrument
Physics Education ResearchThe University of Edinburgh
Trial phase deployment
• Prototype test taken by > 1200 students
• Test deployed in 10 institutions across UK and Ireland (7 Physics & 4 Chemistry depts.)
• Test deployed to all undergraduate years (1st – 5th)
Physics Education ResearchThe University of Edinburgh
Validity and reliability evaluationValidity• test covers appropriate range of skills• test does actually evaluate what you intendedReliability• degree of consistency between scores when
test repeated • test has ability to discriminate student ability
Employ statistical tests and analysis, for individual items and for whole test.
e.g. BEMA: Phys. Rev. Spec. Topics PER 2 010105 (2006)
DHD found to perform satisfactorily
Physics Education ResearchThe University of Edinburgh
Student performance from the trial deployment
0
20
40
60
80
100
120
140
160
180
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Test Score
# o
f S
tud
ents
Histogram of student scoresHistogram of student scores
Physics Education ResearchThe University of Edinburgh
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Question #
Me
an
% C
orr
ec
t
EasyEasy MediumMedium HardHard
Question correct response rates (all students)Question correct response rates (all students)
Physics Education ResearchThe University of Edinburgh
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Question #
% R
espo
nses
A B C D Blank
Question response profilesQuestion response profiles
Physics Education ResearchThe University of Edinburgh
Question 18 Question 18 and response and response
profileprofile
Physics Education ResearchThe University of Edinburgh
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Question #
% C
orr
ec
t
Question correct response rates broken down by classQuestion correct response rates broken down by class
Physics Education ResearchThe University of Edinburgh
Question correct response rates broken down by classQuestion correct response rates broken down by class
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Question #
% C
orr
ec
t
Physics Education ResearchThe University of Edinburgh
45%
50%
55%
60%
65%
70%
1st 2nd 3rd 4th 5th PhD
Year of Study
Me
an
Te
st
Sc
ore
Edinburgh Physics year-by-year mean test scoresEdinburgh Physics year-by-year mean test scores
Physics Education ResearchThe University of Edinburgh
Edinburgh Physics year-by-year question correct response Edinburgh Physics year-by-year question correct response ratesrates
2 consistency test p-values:1st 2nd 3rd 4th 5th PhD
0.038 0.971 0.985 0.934 0.701 0.238
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Question #
% C
orr
ect
1st year 2nd year 3rd year 4th year 5th year PhD
Physics Education ResearchThe University of Edinburgh
• Augment test with stand-alone interactive learning resources
– diagnose and address misconceptions.
Where do we go from here?
Physics Education ResearchThe University of Edinburgh
Learning resources
• Stand-alone and web-based, to support rather than supplant other instruction.
• Accessible direct from questions and by category.
• Feature informal explanations, step-by-step guides,
animations, practice exercises and self-test questions.
Physics Education ResearchThe University of Edinburgh
Physics Education ResearchThe University of Edinburgh
Example: mean, standard deviation and standard error
Physics Education ResearchThe University of Edinburgh
• Random samples drawn from Gaussian distribution.
• mean = 10 std.dev. = 5
• Track evolution of mean, standard deviation and standard error as number of measurements N increases.
Physics Education ResearchThe University of Edinburgh
• Diagnostic tests have been shown to effectively identify learning challenges.
• Data handling skills are important and valued but frequently fall into the ‘implicit curriculum’.
• We have developed a valid, reliable and effective Data Handling Diagnostic.
• We have found that development of these skills and overall student proficiency is not what it might be.
• The diagnostic instrument and its supporting learning resources are ready for use.
Summary
Physics Education ResearchThe University of Edinburgh
http://bit.ly/EdPER
Acknowledgements:We gratefully acknowledge project grant support from the HEA Physical
Sciences Centre and the assistance of our partner institutions in the pilot phase. We also gratefully acknowledge Doug Bonn and James Day of the University of British Columbia for initial ideas and for allowing re-use of three of their items.