questionnaire development measuring validity & reliability james a. pershing, ph.d. indiana...
TRANSCRIPT
Questionnaire Development
Measuring Validity & Reliability
James A. Pershing, Ph.D.Indiana University
Definition of Validity
Instrument measures what it is intended to measure: Appropriate Meaningful Useful
Enables a performance analyst or evaluator to draw correct conclusions
Types of Validity Face Content Criterion
Concurrent Predictive
Construct
Face Validity It looks OK Looks to measure
what it is supposed to measure
Look at items for appropriateness
Client Sample respondents
Least scientific validity measure
Looks Good To Me
Content-Related Validity
Organized review of format and content of instrument
Comprehensiveness Adequate number of
questions per objective
No voids in content By subject matter
experts
Balance
Definition Sample Content Format
Criterion-Related ValiditySubject Instrument A Instrument
BTask Observation
Inventory Checklist John yes noMary no noLee yes noPat no noJim yes yesScott yes yesJill no yes
Usually expressed as a correlation coefficient (0.70 or higher is generally accepted as representing good validity)
How one measure stacks-up against another
Concurrent = at same time
Predictive = now and future
Independent sources that measure same phenomena
Seeking a high correlation
Construct-Related Validity A theory exists
explaining how the concept being measured relates to other concepts
Look for positive or negative correlation
Often over time and in multiple settingsUsually expressed as a correlation coefficient (0.70 or higher is generally accepted as representing good validity)
THEORY
Prediction 1 - Confirmed
Prediction 2 - Confirmed
Prediction 3 - Confirmed
Prediction n - Confirmed
Definition of Reliability The degree to which measures obtained with
an instrument are consistent measures of what the instrument is intended to measure
Sources of error Random error = unpredictable error which is
primarily affected by sampling techniques Select more representative samples Select larger samples
Measurement error = performance of instrument
Types of Reliability Test-Retest Equivalent Forms Internal Consistency
Split-Half Approach Kuder-Richardson Approach Cronbach Alpha Approach
Test-Retest Reliability Administer the same instrument twice to the
same exact group after a time interval has elapsed.
Calculate a reliability coefficient (r) to indicate the relationship between the two sets of scores. r of+.51 to +.75 moderate to good r over +.75 = very good to excellent
T I M E
Equivalent Forms Reliability
Also called alternate or parallel forms Instruments administered to same group at same time Vary:
Calculate a reliability coefficient (r) to indicate the relationship between the two sets of scores.
r of+.51 to +.75 moderate to good r over +.75 = very good to excellent
Response Set: -- Order -- Wording
Stem: -- Order -- Wording
Internal Consistency Reliability
Split-Half Break instrument or
sub-parts in ½ -- like two instruments
Correlate scores on the two halves
Best to consult statistics book and consultant and use computer software to do the calculations for these tests
Kuder-Richardson (KR) Treats instrument as
whole Compares variance
of total scores and sum of item variances
Cronbach Alpha Like KR approach Data scaled or ranked
Reliability and Validity
So unreliable as to be invalid
Fair reliability and fair validity
Fair reliability
but invalid
Good reliability
but invalid
Good reliability and good validity
The bulls-eye in each target represents the information that is desired. Each dot represents a separate score obtained with the instrument. A dot in the bulls-eye indicates that the information obtained (the score) is the information the analyst or evaluator desires.
Comments and Questions