16.00 o6.2 d lillis

STATISTICAL MODELLING

FOR IMPROVEMENT OF SECONDARY AND TERTIARY

PHYSICS ASSESSMENTS

PHYSICS: EXTERNAL RESULTS LEVEL 3 BY GENDER (2010)

NUM N A M E

Male 14,630 29.4 47.8 15.9 6.8

Female 8,375 29.5 50.6 14.5 5.4

LEVEL 3 EXTERNAL STANDARDS (2010 RESULTS)

STD TITLE NUM N A M E

90520 Demonstrate understanding of wave systems 6,443 27.2 54.6 13.3 4.9

90521 Demonstrate understanding of mechanical systems

6,480 30.7 46.6 16.6 6.1

90522 Demonstrate understanding of atoms, photons and nuclei

4,862 32.4 46.2 15.1 6.4

90523 Demonstrate understanding of electrical systems

5,220 27.9 47.0 16.9 8.2

PROFILES OF EXPECTED PERFORMANCE (PEPs)

Expected percentage band for each grade, based on

statistical information & changes in examination

format etc

Provide guidance to examiners & markers

Ensure results that are consistent with the standards

and that standards are maintained

2010 PEP BANDS ACTUAL: N = 32.3%; A = 46.2%; M = 15.1%; E = 6.4%

COHORT STRENGTH & STANDARD DIFFICULTY

Sk = Σj ρkjnkj [ Rkj(j) – Rj(j) ] / Σj nkj

Dk = Σj ρkjnkj [ Rkj(j) – Rkj(k) ] / Σj nkj

RESULTS: PHYSICS 90522

Q1ai Q1aii Q1b Q1c Q1d Q2a Q2b Q2c Q2d Q3a Q3b Q3c FINAL

1 -99 2 1 2 3 2 2 1 3 1 2 2

1 1 2 3 1 1 2 1 1 3 2 1 2

2 3 1 3 1 -99 -99 -99 -99 -99 -99 -99 1

1 3 1 3 4 3 3 2 4 3 1 2 3

2 2 1 3 4 3 2 1 4 2 1 -99 2

2 3 2 -99 2 3 3 2 4 3 3 -99 3

PHYSICS 90522 Principal Components Analysis

PHYSICS 90254: Waves (Level 2)

ECONOMICS 90631 (Level 3)Market failure and government interventions

CHEMISTRY 90696 (Level 3) Oxidation-reduction processes

STATISTICS 90643 (Level 3)Probability

PHYSICS 90522: ITEM LOADINGS THE FIRST & SECOND DIMENSIONS

ITEM

DIM 1

DIM 2

Q1ai 0.27 0.29

Q1aii 0.33 0.18

Q1b 0.14 0.21

Q1c 0.31 -0.28

Q1d 0.31 -0.26

Q2a 0.36 -0.06

Q2b 0.28 0.4

Q2c 0.19 -0.58

Q2d 0.35 -0.25

Q3a 0.24 0.04

Q3b 0.27 0.35

Q3c 0.34 -0.01

INTERPRETATION?

FIRST DIMENSION

Understanding of atoms, photons and nuclei

SECOND DIMENSION

Item type (qualitative or quantitative responses)

MEASURES OF INTERNAL CONSISTENCY

Inter-item Correlation, Item Total Correlation &

Cronbach Alpha

Range between – 1 and + 1

Optimal between about 0.4 and 0.85

INTER-ITEM & ITEM TOTAL CORRELATIONS (CRONBACH ALPHA = 0.76)

ITEM Q1 Q2 Q3 TOTAL

Q1 1.00 0.50 0.48 0.58

Q2 0.50 1.00 0.45 0.57

Q3 0.48 0.45 1.00 0.58

EXAMINATIONS IN PHYSICS . . .

Always assess one dominant dimension (cognitive

construct) and at least one other dimension

For practical purposes, form a psychometric scale,

but not a perfect scale

ITEM RESPONSE THEORY

The probability of obtaining in a single item a particular grade or better (Not Achieved, Achieved, Merit or Excellence), for a candidate of ability θ (measured over the entire assessment), is given by the expression:

Pj = { 1 + exp [ ka ( bj – θ ) ] } -1

j indexes the assessment grades Achieved (A) or better, Merit (M) or better, and Excellence (E)

θ is the calculated ability (which you can also think of as a measure of performance)

a is the item discrimination bj is the estimated difficulty of gaining either an A or better, M or better,

or an E grade for the item. k = -1.7 (scales the logistic curve to approximate a cumulative ogive)

ITEM-CHARACTERISTIC CURVE (DICHOTOMOUS)DIFFICULTY: the ability at which P = 0.5

DISCRIMINATION: slope at that point

ITEM-CHARACTERISTIC CURVE (POLYTOMOUS)

IRT PARAMETERS

Item discriminations & grade difficulties estimated

using various models

Ability estimated from performance on all items,

taking account of item difficulties & discriminations

The most objective measure of performance?

90522: EXAMPLE ITEM PARAMETERS (Graded Response Model)

ITEM Discrimination Difficulty (AME)

Difficulty (ME)

Difficulty (E)

Q1aii 1.45 -1.03 0.44 2.08

Q2a 1.70 -1.01 -0.65 4.66

PHYSICS 90522: HISTOGRAM OF CANDIDATE ABILITIES (PERFORMANCES)

PHYSICS 90522: ITEM CHARACTERISTIC CURVES

THE SUFFICIENCY METHOD OF ASSIGNING OVERALL GRADES

For example:

5A => Achieved

3As + 3Ms => Merit

2Ms + 2Es => Excellence

ALIGNMENT OF ACTUAL FINAL GRADES WITH ABILITY-BASED GRADES

Actual Grade

N A M E Candidates

N 78.6 21.4 0.0 0.0 173

Theta A 10.2 79.0 10.2 0.6 353

Grade M 0.8 29.3 58.5 11.4 123

E 0.0 2.0 30.0 68.0 50

THE SCORE-GRADE METHOD

N A M E

0 1 2 3 4 5 6 7 8

SCORE-BASED MARKING

Improved discrimination between grades

Fairer outcomes

Reduction in year-to-year variability

NEW ZEALAND SCHOLARSHIP

SUBJECT Y 13 Cohort

Entries (male)

Entries (female)

Valid results

N S O

Physics 6938 961 323 1066 858 178 24

Chemistry 7323 962 650 1331 1119 188 24

Calculus 7511 1100 435 1263 1026 201 30

Statistics 14466 1128 619 1418 975 395 45

English 13561 683 1042 1339 919 362 53

ALL 9280 9676 14403 10995 2961 384

AWARDING SCHOLARSHIPS IN PHYSICS

RELATIONSHIP BETWEEN PERFORMANCE IN NCEA LEVEL 3 PHYSICS AND SCHOLARSHIP PHYSICS

DIFFERENTIAL ITEM FUNCTIONING (DIF)

Two or more groups of candidates, matched for ability, perform differently on a particular item.

Comparison Groups DIF

Male – Female Yes

European – Māori No

European – Pasifika No

European – Asian No

DIF: ILLUSTRATION(Item 1 from the 2010 Geography 90704 examination)

david.lillis@nzqa.govt.nz

04 – 463 - 4251