total survey error in disability assessments measuring physical and cognitive capacity in the...

Total Survey Error in Disability Assessments

Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS)

Brad Edwards and Tamara Bruce, Westat

Presented at the International Total Survey Error Workshop Quebec, Canada

June 2011

Overview

NHATS and capacity measures Self report and performance NHATS assessments Strategies for error reduction

- design, training, standardization, parsing out nonresponse

Pretest results National experience Future research2

NHATS

Westat working with a team led by Johns Hopkins Bloomberg School of Public Health

Funded through cooperative agreement from NIA

CAPI panel study with 9,000 Medicare beneficiaries, annual interviews starting 2011

New and improved measures of disability for producing data on trends and trajectories; self-reports, mental and physical assessments, eventually biomarkers, links with admin data

3

Conceptual Framework

Blend of Nagi’s model and WHO perspective

Distinguish explicitly between– Capacity to do something – basic

building blocks

– What people actually do in actual environment – activities

Accommodations may fill gap

4

Capacity Is Critical

Measures of capacity over time are key elements in understanding individual patterns of progression to activity limitations. They allow us to track trends in function that are independent of environmental changes or accommodations, for understanding the disablement process, and as targets for interventions to prevent or slow disability.

5

Reducing Error in Capacity Measures

Questionnaire design Importance of performance measures Challenges for quality management

– New to surveys

– Complex

– Expect high level of interviewer variance

6

Physical Capacity: Upper Extremity

Self report

Able to– Put book on shelf/reach

overhead– Open jar/grasp small object

Performance

Grip strength

7

Physical Capacity: Lower Extremity

Self Report

Able to– Walk 6/3 blocks?– Kneel/bend over?– Lift and carry 20/10

pounds?– Walk up 20/10 stairs?

Performance

Walking speed Balance stands

– Side by side– Semi-tandem– Full-tandem– One leg eyes open– One leg eyes closed

Chair stands

8

Cognitive Capacity: Memory

Self report

At present time? Memory problems

interfere with activities? Memory compared to one

year ago?

Performance

10 word recall– Immediate– Delayed

9

Other Cognitive Performance Measures

Orientation– Day of week, date, naming President and

Vice President Overall cognitive screening/executive

function– Clock drawing

Attention & interference/executive function– Stroop test (computerized)

10

Standardizing Administration

11

Performance measures of capacity can be hard to standardize in large-scale surveys with lay interviewers

NHATS uses multiple methods to improve consistency & accuracy:– Activities Booklet design & use within CAPI framework

– Standardized presentation of cognitive tasks using flash displays

– Survey design & interviewer feedback using CARI coding

– Formal certification process using live respondents

– Follow-up web-based recertification

Training Approaches

Broader scope of demands on NHATS interviewers for performance measures– tests require using a variety of equipment

– navigation of unfamiliar environments

– potentially uncomfortable instructions to respondents

Integration of video components– recruitment

– in-person interviewer training modules such as walking course

– certification & re-certification processes

Walking Course Timing 1

Walking Course Timing 2

Walking Course Timing 3

15

Training Video Demo

16

Item Nonresponse

Performance measures used in depth set of exclusion criteria to minimize burden

High completion rate for both cognitive & performance tasks for those eligible

Comprehensive list of reasons why test was not conducted allow detailed analyses

Majority of unit nonresponse due to inability to complete easier task & safety concerns

Performance related to age, health rating, care setting, memory

17

Respondent Reactions

Avoiding “test” and “performance” Training interviewers how to respond

to requests (e.g., “How did I do?”) Physical activities: “7th inning stretch” Positive feedback at the end Interest in assessing respondent

satisfaction and rapport in relation to nonresponse in later rounds

18

Comparison of Performance and Self Report

High functioning performance group (attempting one-leg balance stands, meeting accuracy thresholds for Stroop) reported better self-reported health and memory and (for the Stroop) had higher performance on two memory tests

19

Comparison with Other Surveys

NLTCS screening questions in NHATS form a bridge from previous trend data to detailed NHATS self reports and assessments

Possibility of future bridge to 6 disability questions developed by NCHS and used on the American Community Survey

20

Conclusions

Reducing error in capacity measures based on performance assessment– sharpens understanding of individual

trajectories and accommodations– is expected to improve predictive ability

for health outcomes NHATS protocol

– standardizes administration– focuses on reasons for nonresponse

21

Conclusions (2)

As a new survey, NHATS has been able to give consideration to the implications of doing these types of assessments from the outset, starting with interviewer recruitment and training. Experience from pretesting has led to refinements of these procedures (e.g., use of videos in recruitment; certification procedures in training) and to the NHATS Activities Booklet. Training materials and data collection instruments will be available later this year at www.nhats.org.

22

Future Research

Interviewer variance study Examination of

– interviewer and respondent conditioning effects – “drift” trajectories– item nonresponse as predictor of unit nonresponse

Error comparison/tradeoffs between self-reports and performance Analysis of cognitive performance and data quality More development

– distance learning– SPC charts, integration with paradata for management – re-certification via WebEX, 2-way video

23

24

For more info, contact:

Brad Edwards

bradedwards@westat.com

or Tamara Bruce tamarabruce@westat.com