Validity and Item Analysis

Chapter 4

Concerns what instrument measures and how well it does so

Not something instrument “has” or “does not have”

Informs counselor when it is appropriate to use instrument and what can be inferred from results

Reliability is a prerequisite for validity

Validity

Content-Related

Criterion-Related

Construct

Traditional Categories of Validity

Evidence Based on Test Content

Evidence Based on Response Processes

Evidence Based on Internal Structure

Evidence Based on Relations to Other Variables

Evidence Based on Consequences of Testing

Validation Evidence: Standards for Educational and

Psychological Testing

Degree to which the evidence indicates that items, questions, or tasks adequately represent intended behavior domain

Central focus is typically on how the instrument’s content was determined

Content-related validation evidence should not be confused with face validity

Evidence Based on Test Content

Concerns whether individuals respond in a manner consistent with construct measured

May also examine information processing differences by subgroup

Computer simulated performances

Evidence Based on Response Processes

Examining internal structure using factor analysis

Can also examine internal structure of instrument for different subgroups (differential item functioning)

Evidence Based on Internal Structure

Correlational method Convergent and discriminant evidence Multitrait-multimethod matrix

Prediction or instrument-criterion relationship Concurrent and predictive validity Regression Decision Theory

Validity Generalization

Evidence Based on Relations to Other Variables

Correlational Method

Correlational method examines relationship between instrument and criterion or other pertinent variables

Validity coefficient

Multitrait-multimethod matrix examines relationships between instrument and traits that are theoretically related and unrelated to it

Concurrent validity

Predictive validity

Prediction/Instrument-Criterion Relationship

Regression

a = y-intercept b = slopeX = score

xySS ryest

21

Standard Error of Estimate

The margin of expected error in the individual’s predicted criterion score as a result of imperfect validity

Decision Theory: Expectancy Table

Expectancy Table (cont.)

Expectancy Table (cont.)

Method of combining validation studies to determine if validity evidence can be generalized Must have a substantial number of studies Must use meta-analytic procedures

Validity Generalization

Examples of social consequences: Group differences on tests used for employment

selection Group differences in placement in special

education

Counselors should consider both validation evidence and social implications

Evidence-Based Consequences of Testing

Gradual accumulation of evidence

Counselor must evaluate the information to determine if it appropriate for which client under what circumstance

Validation evidence should also be considered in informal assessments

Conclusion on Validation Evidence

Examining and evaluating each item in instrument

Item difficulty:p = # who answered item correctly

total # of individuals

Item discrimination: Extreme group method: d = upper % – lower

% Correlational method

Item Analysis

Focus is on each item and establishing items that measure ability or level of a latent trait

Involves examining item characteristic curve and calibration of each individual item

Parameters: Difficulty Slope (discrimination) Probability of guessing

Item Response Theory

Item Characteristic Curve