assessing the assessments: physics an initial analysis of the june 2003 nys regents physics exam j....

Assessing the Assessments: Physics

An Initial Analysis of the June 2003 NYS Regents Physics Exam

J. Zawicki, SUNY Buffalo State CollegeM. Jabot, SUNY Fredonia

J. Zawicki, M. Jabot 2July 2004

Assessment Purposes

Measure knowledge Measure gain in knowledge Measure preparation (predict success) Sorting (Grading) Degree requirements (benchmarks) …


Curriculum, Assessment and Instruction

•Frameworks

•Syllabi

•Guides

•Blueprints

•Benchmarks

•Objective tests

•Performance assessments

•Portfolios

•Teacher Observations

•Group Activities

•Program Evaluations

Curriculum Standards

Assessment/Evaluation System Instructional Program

alignment

validity

correlation

•Instructional styles

•Print materials

•Equipment

•Facilities

•Technology

•Community


Concepts (Continued)

Difficulty – (Percentage or proportion that are successful on an item)FacilityDifficulty

Discrimination – (How well does the item differentiate between students who understand the subject and those who do not?)


Concepts

Validity – how well the item measures match the target construct. May be qualified as:ConstructContent (Face)Criterion Related

Typically determined by a panel of experts



Reliability – can the results be replicated? Inter-rater (Do two or more raters agree on the score

for an item?) Test/Re-test (Will a student earn similar scores on

different administrations?) Internal Consistency

Criterion referenced tests – have the students met the “standard”



Latency – (How long do students take to complete the test?)

Equitable (Fair) Timed tests (Power tests)


Types of Analysis

Traditional (difficulty, discrimination) Rasch Analysis (item difficulty is equated

to student ability) Cognitive Level (Bloom’s taxonomy

simplified: knowing, using, integrating)


Initial AnalysisPhysics 2004 Format Credit Key R/C 0 R/C 1 R/C 2 R/C 3 R/C 4 R/C (%) Facility Difficulty DE P

Item # 01 MC 1 4 0 1 1 1 181 98 0.98 0.02 -4.10 0.98

Item # 39 MC 1 1 0 178 4 1 0 97 0.97 0.03 -3.39 0.96

Item # 46 MC 1 4 1 72 25 20 64 35 0.35 0.65 0.63 0.30

Item # 20 MC 1 3 0 30 6 56 92 30 0.30 0.70 0.83 0.26

Item # 60 CR 1 2 58 1 0 0 33 0.33 0.67 0.73 0.28

Item # 63 CR 2 3 2 55 1 0 31 0.31 0.69 0.79 0.27

Item # 68 CR 2 37 6 18 0 0 11 0.11 0.89 2.05 0.09

Item # 55 CR 1 50 10 1 0 0 7 0.07 0.93 2.66 0.05


Difficulty Rankings

Easier MC: 1, 39 More Difficult MC: 46, 20 Easier CR: 60, 63 More Difficult CR 68, 55


Easier Multiple Choice


More Difficult Multiple Choice


Easier Constructed Response


More Difficult Constructed Response


Common Threads – Easier Items

Graphing Using charts, tables, graphs Classification (vectors, scalars, …)


Common Threads – More Difficult Items

Mathematical relationships (Modeling) Direct Inverse Linear Exponential

Electric, Magnetic, and Gravitational Interactions; Fields (CSEM, Modeling, Castle, Knight)

Energy vs. Force (FCI, PET, Modeling)


Next Steps

Questions Implications for classrooms (program review) Additional Resources

BSC: SCI685, Evaluation in Science Education SUNY Fredonia: EDU503, Evaluation in the Schools SUNY Buffalo: LAI534, Measurement & Evaluation Of

Science Instruction STANYS 2004 Annual Conference, November 7-9, 2004 Rochester Science Educator’s Conference NYSSELA Perspectives

assessing the assessments: physics an initial analysis of the june 2003 nys regents physics exam j....

Documents