development and validation of an instrument to …...final instrument •had to balance content...
TRANSCRIPT
Development and Validation of an Instrument to Measure Teacher
Knowledge of Inquiry
Gwen Nugent, Jon Pedersen, Greg Welch, Jim Bovaird
University of Nebraska-Lincoln
Background
• Instrument developed as part of national survey of elementary teacher professional development
– Professional development characteristics, teacher knowledge, teacher practice
• 19-item multiple-choice measuring teacher knowledge of:
– Scientific inquiry
– Classroom inquiry
– Pedagogical content knowledge
Development Process
• Review of existing instruments
• Drew questions from:
– Assessing PCK on Inquiry Science Teaching (Schuster,
Cobern, Schwarts, Velom, & Applegate)
– Evolving Inquiry (Doll, Bruning, Horn, & PytlikZillig)
• Developed new items
• Table of specifications developed
Table of Specifications
Reference Source
No. of Questions
Scientific inquiry The diverse ways in which scientists study the natural world and propose explanations based on the evidence derived from their work (NSES, pg. 23:)
6
Inquiry pedagogy [Schuster, et al.] 8
Source Source Source No. questions
Classroom Inquiry
K-4 NSES Standards
Essential features of classroom inquiry
NRC Inquiry Activities
3
Ask questions about objects, organisms, and events
Engage in scientifically oriented questions
Define questions from current knowledge
5
Plan and conduct an investigation
Propose preliminary explanations
1
Gather data Give priority to evidence
Gather evidence 3
Use data to construct explanation
Formulate explanation from evidence
Explain based on evidence
2
Connect explanation to knowledge
Consider other explanations
1
Development Process
• Focus group sessions with pre-service teachers to identify confusing questions and wording
• Resulting pilot assessment had 29 questions
• Pilot test with 164 teachers from two Midwestern states in early 2010 (convenience sample)
Pilot Test Results: Classical Test Theory
• Item difficulty .09 - .99;
• Discrimination .00 - .43
• Item-total correlations -.05 to .34
• Alpha = .54;
Final Instrument
• Had to balance content coverage, length, and psychometric properties
• Final version had 5 scientific inquiry; 7 classroom inquiry; and 7 PCK questions
• Some retained items were revised based on pilot results.
Psychometrics Comparison: Classical Test Theory
Statistic Pilot Items Retained Items
Difficulty .09 - .99 .09 - .91
Discrimination .00 - .43 .01 - .36
Item-total correlation .05 - .34 -.06 - .32
Alpha .54 .54
Psychometric Comparison: IRT
0.00
0.50
1.00
1.50
2.00
2.50
3.00
0.000
0.500
1.000
1.500
2.000
2.500
3.000
-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
Scaled InformationFunction (Original Items)
Scaled InformationFunction (RevisedItems)
Standard Error (OriginalItems)
Standard Error (RevisedItems)
Procedures for Final Survey Administration
• National survey was stratified random sample
• Mail survey conducted spring, 2010 through early spring, 2011
• Participants received mailed survey introduction, scan-form survey, and post card reminder
• Each teacher randomly assigned math, reading, or science version in equal numbers
Final Results from National Survey • Overall return for all subject areas was n = 595
• Teachers taking science measure (n = 142) scored 56% correct overall
– Scientific inquiry 48% correct
– Classroom inquiry 63%
– Pedagogical content knowledge 55%
CTT Comparison
Statistic Pilot Items Retained Items Final Results
Difficulty .09 - .99 .09 - .91 .05 - .91
Discrimination .00 - .43 .01 - .36 .12 - .41
Item-total correlation .05 - .34 -.06 - .32 -.03 - .34
Alpha .54 .54 .54
IRT Comparison
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
-4.0 -3.5 -3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
Sta
nd
ard
Err
or
Info
rmati
on
Ability (Science Inquiry Knowledge)
Revised: Information Pilot: Information Revised: SE Pilot: SE
Future Directions: Data Analytic Model
Contact hr. Perceived Utility
Collaborative
Participation Perceived Knowledge Reported Practice
Practice/feedback Gain
(workshop)
Practice/feedback Knowledge
(classroom)
Future Directions
Knowledge measure will be one of the dependent variables for randomized control trial examining impacts of guided inquiry professional development, including coaching.
Final Results from National Survey
Presentation Wednesday, March 6 at 2:45 in
Curacao 8
Contacts
Gwen Nugent, Ph.D. ([email protected]) Jon Pedersen, Ph.D Greg Welch, Ph.D. Jim Bovaird, Ph.D.
National Center for Research on Rural Education
216 Mabel Lee Hall Lincoln, NE 68588-0235
http://r2ed.unl.edu