Lecture 6: Reliability and validity of scales (cont)

1. In relation to scales, define the following terms:

- Content validity

- Criterion validity (concurrent and predictive)

- Construct validity (discriminant and convergent)

- Responsiveness

2. Use of proxy respondents

Validity

• Depends on purpose:– screening: discrimination– outcome of treatment: responsiveness,

sensitivity to change– prognosis: predictive validity

Content and face validity

• Judgment of “experts” and/or members of target population

• Does measure adequately sample domain being measured?

• Does it appear to measure what it is intended to measure? (eyeball test)

Criterion validity

• Criterion (“gold” standard)

• Concurrent criterion validity– e.g., screening test vs diagnostic test

• Predictive criterion validity – e.g., cancer staging test vs 5-year survival

Construct validity

• Is the theoretical construct underlying the measure valid?

• Development and testing of hypotheses

• Requires multiple data sources and investigations:– Convergent validity: measure is correlated with

other measures of similar constructs

– discriminant validity: measure is not correlated with measures of different constructs

Responsiveness of measures• Ability to detect clinically important change

over time or differences between treatments

• Requirement of evaluative measures

• Two approaches:– external responsiveness (validation against

change in external criterion)– internal responsiveness: compute effect size of

ratio of change score to measure of variability (different formulae)

Validity of Delirium Index (DI)

• Convergent validity – DI will be correlated with measures of current

health/function:• Current ADL disability (Barthel Index)

• Current severity of illness (clinical judgment and physiological severity score from APACHE)

Validity of Delirium Index (DI)• Discriminant validity:

– DI will be weakly correlated with measures of previous health/function:

• premorbid level of ADL disability

• severity of dementia

• comorbidity (prior)

Validity of DI (cont)

• DI score will predict probability of one-year survival

• DI score will be sensitive to:– changes in medication exposures – changes in environmental factors

Spearman correlation coefficients between Delirium Index and 3 baseline measures of current status

0

0.1

0.2

0.3

0.4

0.5

Barthel Index Clinicalseverity

Physiologicalseverity

Delirium+Dementia (n=165)

Delirium (n=57)

Spearman correlation coefficients between Delirium Index and 3 baseline measures of prior status

0

0.1

0.2

0.3

0.4

0.5

0.6

IADL IQCODE Comorbidity

Delirium+dementia (n=165)

Delirium (n=57)

Delirium severity and survival

• Proportional hazards regression of delirium severity in delirium cohort

• Mean of 1st 2 DI scores

• Results– significant interaction: DI predicted survival in

patients with delirium alone, not in those with dementia

Effects of medications and environmental factors

• Repeated in-hospital measures of DI, medications (medical record), and environmental factors (using checklist)

• Repeated measurements analysis performed to investigate associations of changes in risk factors with change in DI score.

Environmental risk factors for delirium

0 1 2 3 4 5

Cumulative roomchanges (one)

Medical restraint

No clock/watch

Family memberpresent

No reading glasses

Physical restraints

ICU vs medicine

mean DI increase

Source: McCusker et al, JAGS 2001, 49:1327-34

Medications and DI

• Medication changes (increase in # medications, use of anticholinergic medications) were associated with increase in DI score.

Responsiveness of DI

• Internal responsiveness: ability to capture real change over time– effect size (change in mean DI/SD at baseline)– standardized response mean (change in mean

DI/SD of change)

• External responsiveness: are changes in the DI correlated with other measures of change?

Internal responsiveness of the Delirium Index (DI) at 8 weeks

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Effect size Standardized response mean

Delirium+dementia (n=94)

Delirium (n=33)

Validity of DI: Conclusions

• In patients with delirium +/- dementia:– Correlated with measures of current function– Responsive to change over time– Responsive to changes in environmental factors

and medications

• In patients with delirium alone:– Predicts survival

Example: Inflammatory Bowel Disease Questionnaire (IBDQ)

• Disease-specific HRQoL measure

• 30 items covering 4 domains– bowel symptoms– systemic symptoms– emotional function– social function

• How to validate?

Validation of Inflammatory Bowel Disease Questionnaire

• Questionnaires administered to 42 patients with inflammatory bowel disease on 2 occasions, 1 month apart:– IBDQ– Global ratings of function– Global ratings of change by physician and a relative– Disease Activity Index– Emotional function scale from generic HRQoL

instrument

Validation of IBDQ: predictions and results

• Correlation of 0.5 or more between patients’s global rating of change and IBDQ subscale on bowel symptoms (result: 0.42)

• Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on bowel symptoms (result: 0.33)

• Correlation of 0.3 or more between change in Disease Activity Index and IBDQ subscale on systemic symptoms (result: 0.04)

• Correlation of 0.5 or more between change in generic measure of emotional function and IBDQ subscale on emotional function (result: 0.76)