16.00 o6.2 d lillis
DESCRIPTION
Physikos 2: David LillisTRANSCRIPT
STATISTICAL MODELLING
FOR IMPROVEMENT OF SECONDARY AND TERTIARY
PHYSICS ASSESSMENTS
PHYSICS: EXTERNAL RESULTS LEVEL 3 BY GENDER (2010)
NUM N A M E
Male 14,630 29.4 47.8 15.9 6.8
Female 8,375 29.5 50.6 14.5 5.4
LEVEL 3 EXTERNAL STANDARDS (2010 RESULTS)
STD TITLE NUM N A M E
90520 Demonstrate understanding of wave systems 6,443 27.2 54.6 13.3 4.9
90521 Demonstrate understanding of mechanical systems
6,480 30.7 46.6 16.6 6.1
90522 Demonstrate understanding of atoms, photons and nuclei
4,862 32.4 46.2 15.1 6.4
90523 Demonstrate understanding of electrical systems
5,220 27.9 47.0 16.9 8.2
PROFILES OF EXPECTED PERFORMANCE (PEPs)
Expected percentage band for each grade, based on
statistical information & changes in examination
format etc
Provide guidance to examiners & markers
Ensure results that are consistent with the standards
and that standards are maintained
2010 PEP BANDS ACTUAL: N = 32.3%; A = 46.2%; M = 15.1%; E = 6.4%
COHORT STRENGTH & STANDARD DIFFICULTY
Sk = Σj ρkjnkj [ Rkj(j) – Rj(j) ] / Σj nkj
Dk = Σj ρkjnkj [ Rkj(j) – Rkj(k) ] / Σj nkj
RESULTS: PHYSICS 90522
Q1ai Q1aii Q1b Q1c Q1d Q2a Q2b Q2c Q2d Q3a Q3b Q3c FINAL
1 -99 2 1 2 3 2 2 1 3 1 2 2
1 1 2 3 1 1 2 1 1 3 2 1 2
2 3 1 3 1 -99 -99 -99 -99 -99 -99 -99 1
1 3 1 3 4 3 3 2 4 3 1 2 3
2 2 1 3 4 3 2 1 4 2 1 -99 2
2 3 2 -99 2 3 3 2 4 3 3 -99 3
PHYSICS 90522 Principal Components Analysis
PHYSICS 90254: Waves (Level 2)
ECONOMICS 90631 (Level 3)Market failure and government interventions
CHEMISTRY 90696 (Level 3) Oxidation-reduction processes
STATISTICS 90643 (Level 3)Probability
PHYSICS 90522: ITEM LOADINGS THE FIRST & SECOND DIMENSIONS
ITEM
DIM 1
DIM 2
Q1ai 0.27 0.29
Q1aii 0.33 0.18
Q1b 0.14 0.21
Q1c 0.31 -0.28
Q1d 0.31 -0.26
Q2a 0.36 -0.06
Q2b 0.28 0.4
Q2c 0.19 -0.58
Q2d 0.35 -0.25
Q3a 0.24 0.04
Q3b 0.27 0.35
Q3c 0.34 -0.01
INTERPRETATION?
FIRST DIMENSION
Understanding of atoms, photons and nuclei
SECOND DIMENSION
Item type (qualitative or quantitative responses)
MEASURES OF INTERNAL CONSISTENCY
Inter-item Correlation, Item Total Correlation &
Cronbach Alpha
Range between – 1 and + 1
Optimal between about 0.4 and 0.85
INTER-ITEM & ITEM TOTAL CORRELATIONS (CRONBACH ALPHA = 0.76)
ITEM Q1 Q2 Q3 TOTAL
Q1 1.00 0.50 0.48 0.58
Q2 0.50 1.00 0.45 0.57
Q3 0.48 0.45 1.00 0.58
EXAMINATIONS IN PHYSICS . . .
Always assess one dominant dimension (cognitive
construct) and at least one other dimension
For practical purposes, form a psychometric scale,
but not a perfect scale
ITEM RESPONSE THEORY
The probability of obtaining in a single item a particular grade or better (Not Achieved, Achieved, Merit or Excellence), for a candidate of ability θ (measured over the entire assessment), is given by the expression:
Pj = { 1 + exp [ ka ( bj – θ ) ] } -1
j indexes the assessment grades Achieved (A) or better, Merit (M) or better, and Excellence (E)
θ is the calculated ability (which you can also think of as a measure of performance)
a is the item discrimination bj is the estimated difficulty of gaining either an A or better, M or better,
or an E grade for the item. k = -1.7 (scales the logistic curve to approximate a cumulative ogive)
ITEM-CHARACTERISTIC CURVE (DICHOTOMOUS)DIFFICULTY: the ability at which P = 0.5
DISCRIMINATION: slope at that point
ITEM-CHARACTERISTIC CURVE (POLYTOMOUS)
IRT PARAMETERS
Item discriminations & grade difficulties estimated
using various models
Ability estimated from performance on all items,
taking account of item difficulties & discriminations
The most objective measure of performance?
90522: EXAMPLE ITEM PARAMETERS (Graded Response Model)
ITEM Discrimination Difficulty (AME)
Difficulty (ME)
Difficulty (E)
Q1aii 1.45 -1.03 0.44 2.08
Q2a 1.70 -1.01 -0.65 4.66
PHYSICS 90522: HISTOGRAM OF CANDIDATE ABILITIES (PERFORMANCES)
PHYSICS 90522: ITEM CHARACTERISTIC CURVES
THE SUFFICIENCY METHOD OF ASSIGNING OVERALL GRADES
For example:
5A => Achieved
3As + 3Ms => Merit
2Ms + 2Es => Excellence
ALIGNMENT OF ACTUAL FINAL GRADES WITH ABILITY-BASED GRADES
Actual Grade
N A M E Candidates
N 78.6 21.4 0.0 0.0 173
Theta A 10.2 79.0 10.2 0.6 353
Grade M 0.8 29.3 58.5 11.4 123
E 0.0 2.0 30.0 68.0 50
THE SCORE-GRADE METHOD
N A M E
0 1 2 3 4 5 6 7 8
SCORE-BASED MARKING
Improved discrimination between grades
Fairer outcomes
Reduction in year-to-year variability
NEW ZEALAND SCHOLARSHIP
SUBJECT Y 13 Cohort
Entries (male)
Entries (female)
Valid results
N S O
Physics 6938 961 323 1066 858 178 24
Chemistry 7323 962 650 1331 1119 188 24
Calculus 7511 1100 435 1263 1026 201 30
Statistics 14466 1128 619 1418 975 395 45
English 13561 683 1042 1339 919 362 53
ALL 9280 9676 14403 10995 2961 384
AWARDING SCHOLARSHIPS IN PHYSICS
RELATIONSHIP BETWEEN PERFORMANCE IN NCEA LEVEL 3 PHYSICS AND SCHOLARSHIP PHYSICS
DIFFERENTIAL ITEM FUNCTIONING (DIF)
Two or more groups of candidates, matched for ability, perform differently on a particular item.
Comparison Groups DIF
Male – Female Yes
European – Māori No
European – Pasifika No
European – Asian No
DIF: ILLUSTRATION(Item 1 from the 2010 Geography 90704 examination)
