Download - Psychometric Modeling and Calibration
![Page 1: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/1.jpg)
1 04/20/23
Psychometric Modeling Psychometric Modeling and Calibrationand Calibration
Ron D. Hays, Ph.D. Ron D. Hays, Ph.D.
September 11, 2006 September 11, 2006
PROMIS development process sessionPROMIS development process session
10:45am-12:30 pm10:45am-12:30 pm
![Page 2: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/2.jpg)
2 04/20/23
PROMIS Domains
Physical functioning (Hays/Bjorner)
Pain (Revicki)
Fatigue (Lai)
Emotional distress (Choi/Reise)
Social/role participation (Bode/Hahn)
![Page 3: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/3.jpg)
3 04/20/23
Datasets
Cancer Item Banks (Northwestern)
Digitalis Investigation Group Study--randomized double-blind placebo-controlled trial evaluating effect of digoxin on mortality in 581 patients with heart failure and sinus rhythm.
IMMPACT--internet-based survey of individuals with chronic pain from the American Chronic Pain Association website
Medical Outcomes Study--observational study of persons with hypertension, diabetes, heart disease, and/or depression in Boston, Chicago, and Los Angeles
WHOQOL-100 data (n = 442 from U.S. field center)
![Page 4: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/4.jpg)
4 04/20/23
Types of AnalysesTypes of Analyses
• Classical Test Theory StatisticsClassical Test Theory Statistics
• IRT Model AssumptionsIRT Model Assumptions
• Model FitModel Fit
• Differential Item FunctioningDifferential Item Functioning
• Item CalibrationItem Calibration
![Page 5: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/5.jpg)
5 04/20/23
Classical Test Theory StatisticsClassical Test Theory Statistics
• Out of rangeOut of range
• Item frequencies and distributionsItem frequencies and distributions
• Inter-item correlationsInter-item correlations
• Item-scale correlationsItem-scale correlations
• Internal consistency reliabilityInternal consistency reliability
![Page 6: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/6.jpg)
6 04/20/23
IRT Model AssumptionsIRT Model Assumptions
• (Uni)dimensionality(Uni)dimensionality
• Local independenceLocal independence
• MonotonicityMonotonicity
![Page 7: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/7.jpg)
7 04/20/23
Sufficient UnidimensionalitySufficient Unidimensionality
• Confirmatory factor modelsConfirmatory factor models
One factorOne factor
Bifactor (general and group factors)Bifactor (general and group factors)
![Page 8: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/8.jpg)
8 04/20/23
Local IndependenceLocal Independence
• After controlling for dominant factor(s), item After controlling for dominant factor(s), item pairs should not be associated.pairs should not be associated.
Look at residual correlations (> 0.20)Look at residual correlations (> 0.20)
![Page 9: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/9.jpg)
9 04/20/23
MonotonicityMonotonicity
• Probability of selecting a response category Probability of selecting a response category indicative of better health should increase as indicative of better health should increase as underlying health increases.underlying health increases.
• Item response function graphs withItem response function graphs with
y-axis: proportion positive for item stepy-axis: proportion positive for item step
x-axis: raw scale score minus item scorex-axis: raw scale score minus item score
![Page 10: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/10.jpg)
10 04/20/23
![Page 11: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/11.jpg)
11 04/20/23
Category Response Curves for Category Response Curves for Samejima’s Graded Response ModelSamejima’s Graded Response Model
0.0
0.2
0.4
0.6
0.8
1.0
-3 -2 -1 0 1 2 3
Theta (q)
P(X
=k|q
)
P (X = 3|q)P (X = 1|q)
P (X = 2|q)
P (X = 4|q)
![Page 12: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/12.jpg)
12 04/20/23
Model FitModel Fit
• Compare observed and expected response Compare observed and expected response frequencies by item and response categoryfrequencies by item and response category
• Items that do not fit and less discriminating Items that do not fit and less discriminating items identified and reviewed by content items identified and reviewed by content expertsexperts
![Page 13: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/13.jpg)
13 04/20/23
Differential Item FunctioningDifferential Item Functioning
• Uniform DIF Uniform DIF
Threshold parameterThreshold parameter
• Non-uniform DIF Non-uniform DIF
Discrimination parameter Discrimination parameter
• Gender, race/ethnicity, age, diseaseGender, race/ethnicity, age, disease
![Page 14: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/14.jpg)
14 04/20/23
Item CalibrationItem Calibration
• Item parameters (threshold, discrimination)Item parameters (threshold, discrimination)
• Mean differences for studied disease groupsMean differences for studied disease groups
![Page 15: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/15.jpg)
15 04/20/23
Example of Lessons Learned in Secondary Analyses
Emotional distress
Cannot be adequately modeled as a unidimensional construct.
Limited representation of positive end of construct
Several items having some response options that provide little information.
![Page 16: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/16.jpg)
16 04/20/23
Documentation
eRoom
Public website: http://www.nihpromis.org/
Peer-reviewed manuscripts, e.g.:
Hays, R. D. et al. (submitted). Item response theory analyses of physical functioning items in the Medical Outcomes Study.
Reeve, B. B. (submitted). Psychometric evaluation and calibration of health-related quality of life items banks: Plans for the Patient-Reported Outcome Measurement Information System (PROMIS)
Presentations:
Tomorrow 4:15-6:15
![Page 17: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/17.jpg)
17 04/20/23
Questions?Questions?
![Page 18: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/18.jpg)
18 04/20/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
Cancer Fatigue:Cancer Fatigue: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Pain:Cancer Pain: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
Cancer Social:Cancer Social: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU
CSSCD:CSSCD: Cooperative Study of Sickle Cell Disease (pediatric)Cooperative Study of Sickle Cell Disease (pediatric)
CHC:CHC: Chronic Hepatitis C StudyChronic Hepatitis C Study
CHS:CHS: Cardiovascular Health StudyCardiovascular Health Study
DIG:DIG: Digitalis Investigation Group Quality of Life Sub-studyDigitalis Investigation Group Quality of Life Sub-study
IMMPACT:IMMPACT: Multiple Pain ProjectsMultiple Pain Projects
MOS:MOS: Medical Outcomes StudyMedical Outcomes Study
NGHS:NGHS: National Growth and Health Study (peds.)National Growth and Health Study (peds.)
Q-Score:Q-Score: Cancer Quality of Life Project at NWUCancer Quality of Life Project at NWU
WHOQOL:WHOQOL: World Health Organization Quality of Life ProjectWorld Health Organization Quality of Life Project
![Page 19: Psychometric Modeling and Calibration](https://reader038.vdocuments.site/reader038/viewer/2022103100/56813c60550346895da5e85f/html5/thumbnails/19.jpg)
19 04/20/23
Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis
PROMIS Domains
Emotional Distress Fatigue Pain Physical FunctionSocial Role
Participation
CHS(NWU/Cook)
DIG(UCLA/Hays)
Q-Score(NWU/Bode)
CHC(Medtap/Revicki
& Chen)
Cancer Pain(NWU/Lai)
IMMPACT(Medtap/Revicki
& Chen)
CHS(NWU/Cook)
DIG(UCLA/Hays)
CHS(NWU/Cook)
Cancer Social(NWU/Bode)
CHS(NWU/Cook)
Cancer Fatigue(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
MOS(UCLA/Hays& Spritzer)
Q-Score(NWU/Lai)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
WHOQOL(UCLA/Hays)
CSSCD(NWU)
NGHS(NWU)
NGHS(NWU)