psychometric evaluation and calibration plan
DESCRIPTION
Psychometric Evaluation and Calibration Plan. Ron D. Hays, Ph.D. October 14, 2006 ISOQOL (Opala) 2:00-3:30 pm. PROMIS Domains. Physical functioning (Hays/Bjorner) Pain (Revicki/Cook) Fatigue (Lai) Emotional distress (Choi/Reise) Social/role participation (Bode/Hahn). Datasets. - PowerPoint PPT PresentationTRANSCRIPT
1 04/21/23
Psychometric Evaluation Psychometric Evaluation and Calibration Planand Calibration Plan
Ron D. Hays, Ph.D. Ron D. Hays, Ph.D.
October 14, 2006 October 14, 2006
ISOQOL (Opala)ISOQOL (Opala)
2:00-3:30 pm2:00-3:30 pm
2 04/21/23
PROMIS Domains
Physical functioning (Hays/Bjorner)
Pain (Revicki/Cook)
Fatigue (Lai)
Emotional distress (Choi/Reise)
Social/role participation (Bode/Hahn)
3 04/21/23
Datasets
Cancer Item Banks (Northwestern)
Digitalis Investigation Group Study--randomized double-blind placebo-controlled trial evaluating effect of digoxin on mortality in 581 patients with heart failure and sinus rhythm.
IMMPACT--internet-based survey of individuals with chronic pain from the American Chronic Pain Association website
Medical Outcomes Study--observational study of persons with hypertension, diabetes, heart disease, and/or depression in Boston, Chicago, and Los Angeles
WHOQOL-100 data (n = 442 from U.S. field center)
4 04/21/23
Types of AnalysesTypes of Analyses
• Classical Test Theory StatisticsClassical Test Theory Statistics
• IRT Model AssumptionsIRT Model Assumptions
• Model FitModel Fit
• Differential Item FunctioningDifferential Item Functioning
• Item CalibrationItem Calibration
5 04/21/23
Classical Test Theory StatisticsClassical Test Theory Statistics
• Out of rangeOut of range
• Item frequencies and distributionsItem frequencies and distributions
• Inter-item correlationsInter-item correlations
• Item-scale correlationsItem-scale correlations
• Internal consistency reliabilityInternal consistency reliability
6 04/21/23
IRT Model AssumptionsIRT Model Assumptions
• (Uni)dimensionality(Uni)dimensionality
• Local independenceLocal independence
• MonotonicityMonotonicity
7 04/21/23
Sufficient UnidimensionalitySufficient Unidimensionality
• Confirmatory factor modelsConfirmatory factor models
One factorOne factor
Bifactor (general and group factors)Bifactor (general and group factors)
8 04/21/23
Local IndependenceLocal Independence
• After controlling for dominant factor(s), item After controlling for dominant factor(s), item pairs should not be associated.pairs should not be associated.
Look at residual correlations (> 0.20)Look at residual correlations (> 0.20)
9 04/21/23
MonotonicityMonotonicity
• Probability of selecting a response category Probability of selecting a response category indicative of better health should increase as indicative of better health should increase as underlying health increases.underlying health increases.
• Item response function graphs withItem response function graphs with
y-axis: proportion positive for item stepy-axis: proportion positive for item step
x-axis: raw scale score minus item scorex-axis: raw scale score minus item score
10 04/21/23
11 04/21/23
Category Response Curves for Category Response Curves for Samejima’s Graded Response ModelSamejima’s Graded Response Model
0.0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
Theta (q)
P(X
=k|q
)
P (X = 3|q)P (X = 1|q)
P (X = 2|q)
P (X = 4|q)
12 04/21/23
Model FitModel Fit
• Compare observed and expected response Compare observed and expected response frequencies by item and response categoryfrequencies by item and response category
• Items that do not fit and less discriminating Items that do not fit and less discriminating items identified and reviewed by content items identified and reviewed by content expertsexperts
13 04/21/23
Differential Item FunctioningDifferential Item Functioning
• Uniform DIF Uniform DIF
Threshold parameterThreshold parameter
• Non-uniform DIF Non-uniform DIF
Discrimination parameter Discrimination parameter
• Gender, race/ethnicity, age, diseaseGender, race/ethnicity, age, disease
14 04/21/23
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
Trait level
Pro
bab
ilit
y o
f "Y
es"
Res
po
nse
Location DIF Slope DIF
Dichotomous Items Showing DIF(2-Parameter Model)
White
Hispanic
Hispanic
White
15 04/21/23
Item CalibrationItem Calibration
• Item parameters (threshold, discrimination)Item parameters (threshold, discrimination)
• Mean differences for studied disease groupsMean differences for studied disease groups
16 04/21/23
Example of Lessons Learned in Secondary Analyses
Emotional distress
Cannot be adequately modeled as a unidimensional construct.
Limited representation of positive end of construct
Several items having some response options that provide little information.
17 04/21/23
Documentation
Public website: http://www.nihpromis.org/
Peer-reviewed manuscripts, e.g.:
Hays, R. D. et al. (in press). Item response theory analyses of physical functioning items in the Medical Outcomes Study. Medical Care.
Reeve, B. B. (in press). Psychometric evaluation and calibration of health-related quality of life items banks: Plans for the Patient-Reported Outcome Measurement Information System (PROMIS). Medical Care.
18 04/21/23
19 04/21/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
Cancer Fatigue:Cancer Fatigue: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Pain:Cancer Pain: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Social:Cancer Social: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
CSSCD:CSSCD: Cooperative Study of Sickle Cell Disease (pediatric)Cooperative Study of Sickle Cell Disease (pediatric)
CHC:CHC: Chronic Hepatitis C StudyChronic Hepatitis C Study
CHS:CHS: Cardiovascular Health StudyCardiovascular Health Study
DIG:DIG: Digitalis Investigation Group Quality of Life Sub-studyDigitalis Investigation Group Quality of Life Sub-study
IMMPACT:IMMPACT: Multiple Pain ProjectsMultiple Pain Projects
MOS:MOS: Medical Outcomes StudyMedical Outcomes Study
NGHS:NGHS: National Growth and Health Study (peds.)National Growth and Health Study (peds.)
Q-Score:Q-Score: Cancer Quality of Life Project at NWUCancer Quality of Life Project at NWU
WHOQOL:WHOQOL: World Health Organization Quality of Life ProjectWorld Health Organization Quality of Life Project
20 04/21/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
PROMIS Domains
Emotional Distress Fatigue Pain Physical FunctionSocial Role
Participation
CHS(NWU/Cook)
DIG(UCLA/Hays)
Q-Score(NWU/Bode)
CHC(Medtap/Revicki
& Chen)
Cancer Pain(NWU/Lai)
IMMPACT(Medtap/Revicki
& Chen)
CHS(NWU/Cook)
DIG(UCLA/Hays)
CHS(NWU/Cook)
Cancer Social(NWU/Bode)
CHS(NWU/Cook)
Cancer Fatigue(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
MOS(UCLA/Hays& Spritzer)
Q-Score(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
CSSCD(NWU)
NGHS(NWU)
NGHS(NWU)