1

Measures in Health Services Research: From Constructs to Care

Jacob Kean, PhD CCC-SLP

Research Health Scientist, Center for Health Information and Communication, Roudebush VA Medical Center

Research Scientist, Regenstrief InstituteAssistant Research Professor, Department of Physical Medicine and

Rehabilitation, Indiana University School of Medicine

2

High-Level Motivation:• To improve the delivery,

quality, cost, access to, and outcomes of care.

Approaches:• Development and

implementation of measures

• Health information exchange

Contexts:• Medical and psychiatric

rehabilitation• Primary care

Constructs

Models

Interpretations

Actions

3

CONSTRUCTS: DELIRIUM/POST-TRAUMATIC AMNESIA

4

Construct: Delirium/PTA

• Post-traumatic amnesia (PTA)– Duration– Measured retrospectively, and later prospectively– Index of injury severity and prognosis

• Prospective measures operationalized the end point

• Prospective measures used to log severity, but poor construct validity when used that way

PTA/PTCS = DeliriumLipowski’s (1987) definition of delirium

Stuss et al. (1999) definition of PTCS

• “Delirium is a transient organic mental syndrome of acute onset, characterized by global impairment of cognitive functions, a reduced level of consciousness, attentional abnormalities, increased or decreased psychomotor activity, and a disordered sleep-wake cycle.”

• “A confusional state can be defined as a transient organic mental syndrome with acute onset characterized by a global impairment of cognitive functions with a concurrent disturbance of consciousness, attentional abnormalities, reduced or increased psychomotor activity, and a disrupted sleep-wake cycle.”

6

Construct: Delirium/PTA

• 3 Factor Model– Attention– Higher-level thinking– Sleep/wake cycle disturbance

• Primary question:– Can we identify the endpoint of this period of

impaired consciousness and track severity with fewer (but construct valid) items?

7

Construct: Delirium/PTA (Kean et al., 2010)

• Sample– 57 with brain injury were recruited, 18 declined, 3 diagnosed

with dementia• N = 36; Age range 19-91• No history of brain injury, substance abuse

• Measures– DSM-IV Diagnostic Criteria for Delirium– Delirium Rating Scale – Revised-98– Delirium Diagnostic Tool – Provisional

• Methods– 3x/week; raters blinded

8

Construct: Delirium/PTA

• Diagnostic accuracy – ROC analysis at cutoff (≤6) resulted in AUC=0.994

(35/36) classified accurately as referenced against the DSM-IV gold standard

• Duration– Correlation between DRS-R98 and DDT-Pro

duration estimates r = 0.975

9

Construct: Delirium/PTA

• Three construct-valid items can accurately – Identify the end of delirium/PTCS following TBI– Measure severity

• Representative papers– Kean & Ryan (2008)– Kean et al. (2010)– Seel et al. (2010)

10

MODELS: ITEM RESPONSE THEORY

11

Models: Item Response Theory

• Resistance to delirium construct in brain injury settings• Our simple, short measure (i.e., DDT-Pro) was limited

– Ordinal-level (nonparametric statistics)• Spurious interactions, underestimation of effect sizes, and impact

on gain scores.

– Person measures are sample-dependent – Focus on group-level metrics (e.g., reliability, error)– Scores obtained from different sets of items (measuring the

same construct) are not directly comparable– Few techniques for validating response patterns and

systematic variation

12

Models: Item Response Theory

• Build on existing PTA measure (Orientation Log)

• Incorporate IRT model to:– Allow better understanding of items and construct– Achieve interval-level measurement– Differentiate ability

Methods (Kean et al., 2011)

• 257 (321) ratings of 90 patients admitted for inpatient rehabilitation following TBI

• 48.25 years (SD 18.87; range from 17 to 93), 75% were male

• Twenty unique items from three scales were administered: O-Log, C-Log, DDT-Pro

• Analyses were conducted with WINSTEPS version 3.6 using a partial credit model

Hybrid Measure Items

Person-Item Maps

•Hybrid measure has improved targeting vs. O-Log

•Separation of both measures suffer due to the poor targeting of more impaired persons

16

Models: Item Response Theory

• IRT model-driven approach– Improves measurement quality and precision– Construct-relevant items contribute to

differentiation of person “ability”• Representative Papers– Kean et al. (2011a, 2011b)– Malec, Kean et al. (2012)– Kean, Malec et al. (2013)– McGuire, Kean et al. (2014)

17

INTERPRETATIONS: RESPONSIVENESS AND SENSITIVITY TO CHANGE

18

Patient-Reported Outcome Measurement Information System (PROMIS)

• The PROMIS assessment system is based on a comprehensive (i.e., physical, mental, social) self-reported health framework composed of many domains

• Domains are represented as unidimensional hierarchies of dozens of items, called “item banks”– Item banks include many items to represent fully the range

of impairment in a given domain• Developed using IRT

– Item banks that can be administered adaptively or assembled as static “short forms”

19

SCOPE Trial (Kroenke et al., 2014)

• Enrolled 250 veterans with moderate to severe and persistent musculoskeletal pain– Mean age of 55.1 years (range, 28 to 65)– 83% were men– Duration of pain was 1 year or longer in 98% of

participants • Tested a telemedicine/collaborative care

intervention• 244 patients who completed both baseline and 3-

month assessments

20

Comparative Responsiveness

• Measures– PROMIS Pain Interference Short Forms– Brief Pain Inventory– SF-36 Bodily Pain– Reference Standard – Patient-reported global

change (Better, Same, Worse)• Standardized Response Means• Standardized Effect Sizes

21

Comparison of SRMBetter Same Worse

BPI Severity 0.71 0.13 -0.47

BPI Interference 0.94 0.38 0.03

BPI Total 0.93 0.31 -0.22

PEG 0.86 0.25 -0.14

SF-36 Bodily Pain 0.71 0.38 0.17

PROMIS-29 0.33 0.29 -0.11

PROMIS-57 0.37 0.30 -0.16

PROMIS Pain 6b 0.51 0.27 -0.02

22

Comparison of SESPain scale Intervention change Control change SES

BPI severity 0.74 (1.83) 0.11 (1.46) 0.38

BPI interference 1.33 (1.94) 0.61 (1.87) 0.37

BPI total 1.04 (1.70) 0.36 (1.45) 0.42

PEG 1.18 (2.07) 0.44 (1.89) 0.37

SF-36 pain 8.24 (16.13) 4.29 (15.76) 0.25

PROMIS®-29 1.81 (5.67) 0.81 (5.88) 0.17

PROMIS®-57 2.05 (5.54) 0.67 (5.81) 0.24

PROMIS® Pain 6b 2.48 (5.27) 0.94 (5.79) 0.28

23

Interpretations: Responsiveness and Sensitivity

• PROMIS Pain Interference short forms were– Less sensitive to change– Less responsive to treatment than BPI– Surprising finding

• Respondent fatigue• Placement of measure in interview• BPI familiarity effect from automated

symptom monitoring

24

Projects Underway

VA (RR&D) Kean – PI (2012-17)

NIDILRR Hammond – PI (2012-17)Role: Co-I

NIH (NIAMS) Monahan – PI (2012-16)Role: Co-I

NIH (NICHD) Malec – PI (2014-16)Role: Co-I

PCORI Kroenke – PI (2014-16)Role: Co-I

ISDH Kean – PI (2014-16)

25

Brain Research in Aggression and Irritability Network (BRAIN): Building Evidence-Based Approaches to Managing Traumatic Brain

Injury (Role: Co-I)

• Aggression and Irritability Impact Measure (AIIM)

• Prior work shows that expression may be less burdensome than impact of irritability/aggression– E.g., mild irritability likely

interferes less with participation of a person with TBI who works in a self-directed and self-paced job.

26

Measurement-Based Telehealth Care of Mild Traumatic Brain Injury (Role: PI)

• VA Telehealth systems for mild TBI are limited

• Assessment of symptoms + self-management aligns with clinical practice guidelines

• Developing measures of self-management

27

Responsiveness and Clinical Validity of PROMIS Pain and Depression Measures (Role: Co-I)

Effectiveness and Patient Selection in Post-Hospital Brain Injury Rehabilitation (Role: Co-I)

• Responsiveness and minimally important differences– PROMIS Pain and

Depression measures– Mayo-Portland

Adaptability Inventory-4

28

Incorporating PROMIS Symptom Measures into Primary Care Practice (Role: Co-I)

• PCORI is interested in integrating PROMIS measures into clinical settings

• Effectiveness trial of providing patient symptom scores to clinicians on symptom improvement, satisfaction with treatment

29

NEXT STEPS: ABERRANT RESPONSE, CSI, PRACTICE-BASED EVIDENCE

30

Aberrant Response

• Persons and items are calibrated together in IRT analysis– Person fit and other metrics used in educational

measurement to detect cheating and identify unique cases for remediation

• Diagnosis and assessment in mild traumatic brain injury depends heavily on self-report

• Symptom validity approaches are coarse

31

Comprehensive Severity Index (CSI®) (Horn et al.)

• Severity defined as “physiologic complexity presented to medical personnel due to the extent and interactions of a patient’s diseases”

• Disease-specific: 5,500 disease-specific groups; over 2,200 distinct criteria. ICD-9 codes trigger disease-specific patient signs, symptoms, and physical findings used to score disease-specific and overall severity levels

32

CSI

Extensive Development

In what ways can CSIchange practice?

How responsive areCSI measures?

Can advanced modelsimprove precision andovercome missing data?

33

SCORE! Trial (Cooper, Bowles et al.)

• Fidelity measurement• RCT Cognitive rehabilitation– 10 hours week/6 weeks

• Implementation (CFIR constructs)– Outer setting– Inner setting– Characteristics of individuals involved– Process

34

Process Factors• Patient Education and Management Strategies• Interventions and surgeries • Medications

Patient Factors• Psychosocial/demographic Factors• Co-occurring Conditions• Severity of Illness and Injury• Genetic information• Measured at Multiple Points in Time

Primary Outcomes• Clinical • Health Status• Functional• Cost/LOS/Encounters• Discharge Disposition• Post-discharge Outcomes

Standardize documentation for :

Control for:

Measure:

Components of Practice-Based Evidence Designs (Horn et al.)

35

Thank You!

• Kurt Kroenke• Patrick

Monahan• Linda Williams• Teresa

Damush

• Flora Hammond

• Jim Malec• Chris Pretz• Al Kozlowski

• Brian Dixon• Laura Myers• Jessica Coffing• Erica Evans