speca workshop on disability statistics, dec 13-15, 2006 the definition and measurement of...

SPECA Workshop on Disability Statistics, Dec 13-15, 2006

The Definition and Measurement of Disability: The Work of the Washington Group

Jennifer Madans

National Center for Health Statistics, USA

for the Washington Group

The Washington Group on Disability Statistics

• In June of 2001, the UN International Seminar on the Measurement of Disability recommended the development of principles and standard forms for global indicators of disability to be used in censuses

• There was a broad consensus on the need for population based measures of disability for country use and for international comparisons

The Washington Group was set up to:

• Foster international cooperation in the area of health and disability statistics

• Develop disability measures suitable for censuses and surveys that will provide basic information on disability

• Untangle the web of confusing and conflicting disability estimates

Current Problem• Currently national censuses in developing

countries use one of three types of questions that provide widely differing estimates of national prevalence of disability:

• The three types of questions include• A. Generic question about the presence of a condition

• B. Generic questions about the presence in the household of a person with a disability followed by a list of impairments

• C. Checklist of impairments

Variety in National Estimates

• Developing Countries:• Cyprus – 4.0%• Uganda – 1.2%• Nigeria – 0.5%

• Developed Countries:• United Kingdom – 12.5%

Washington Group: Objectives

• Develop a small set/s of general disability measures

• Recommend extended set/s of items to measure disability as components of population surveys / supplements

• Address methodological issues associated with disability measurement

Preliminary work to meet objectives

• Clarify the purpose of data collection in order to identify appropriate measures

• Understand choices being made when time, expenses and respondent burden limit number of questions

Moving from Concept to Definition to Measurement: The Conceptual Model

ICF selected as the conceptual model:• Common point of reference • Common vocabulary• Does not provide an operational

definition or a way to measure the concepts

Moving from Concept to Definition to Measurement: The Definitional Paradox• There is no single operational definition of disability• Different operational definitions lead to different

estimates• The question you are trying to answer (the purpose)

will determine which definition to use • Need to understand the choices that are being

made when a purpose and a definition are chosen• Need to understand the choices that are being

made when time, expenses and respondent burden limit number of questions

Purpose of Data Collection

3 major classes of purposes at aggregate level

• Service Provision• Monitoring functioning in the population• Assess equalization of opportunities

2 criteria for selection of a purpose• Relevance—particularly for policy makers

and program officials• Feasibility

Purpose: Service provision

• Seeks to identify those with specific needs, usually the most serious problems

• Requires detailed information about the person and the environment

• Influenced by the organization and structure of service organizations within a particular culture

Purpose: Monitoring functioning in the population

• Seeks to identify all those with activity or participation limitation

• Response comparability problematic since participation is culturally and environmentally determined

0

5

1 0

1 5

2 0

2 5

Pro

po

rtio

n (

%)

2 0 0 5 2 0 1 0 2 0 1 5 2 0 2 0 2 0 2 5

Population reporting work limitation

Purpose: Equalization of opportunities

• Seeks to identify all those at greater risk than the general population for limitations in activity or participation

• Disability as a demographic

0

3 0

6 0

9 0

Pro

po

rtio

n (

%)

N o n d is a b le d D is a b le d

% Employed

Applying criteria to select purpose• Service provision:

• Level of detail necessary not feasible in a census format

• Nature of service provision varies across cultures

• Monitoring functioning: • Response comparability problematic since participation

is culturally and environmentally determined

• Assessing equalization of opportunities:• If we conceive of disability toward the most basic

elements of activity, without tying it to participation, we limit the number / types of questions thus enhancing feasibility

Locating Risk in the ICF ModelHealth Condition

Body Functions & Structure Participation

Environmental Factors

Personal Factors

Source: ICF, WHO, 2001

?

Moving from Concept to Definition to Measurement: Measurement of equalization of opportunities

• Locate the definition of disability at the most basic level of activity/participation in core domains

• This level is associated with the ability or inability to carry out basic bodily operations at the level of the whole person (i.e. walking, climbing stairs, lifting packages, seeing a friend across the room)

Benefits of this approach

• Development of a demographic means of understanding disability (can compare persons with and without disability)

• Connection between disability and participation can be made during data analysis

• Effectiveness of programs / policies to promote full participation can be monitored

Possible types of questions

• Questions that measure functioning in various domains such as mobility, cognition, sensory, etc.

• A qualifier would need to ascertain that the action was accomplished without human or mechanical assistance

Possible question options• Mobility

• Walking• Climbing stairs• Bending or stooping• Reaching or lifting• Using hands

• Sensory• Seeing• Hearing

• Communicating• Understanding• Speaking

• Cognitive functions• Learning

• Remembering

• Making decisions

• Concentrating• Emotional functioning

• Interpersonal interactions

• Psychological well-being

Criteria for inclusion of domains

• Cross cultural comparability• Suitability for self-report• Parsimony• Validity across various methodological

modes

WG draft questions for censuses (short disability measure)

Core Questions:1) Do you have difficulty seeing even if wearing glasses?2) Do you have difficulty hearing even if using a hearing

aid?3) Do you have difficulty walking or climbing stairs?4) Do you have difficulty remembering or concentrating?

Additional Questions:5) Do you have difficulty with (self-care such as) washing

all over or dressing?6) Because of a physical, mental, or emotional health

condition, do you have difficulty communicating (for example understanding others or others understanding you)?

Response categories: No - no difficulty; Yes – some difficulty; Yes – a lot of difficulty; Cannot do at all

Objectives

• Identify persons with similar types and degree of limitations in basic activities regardless of nationality or culture

• Represent the majority (but not all) persons with limitations in basic activities in any one nation

• Represent the most commonly occurring limitations in basic activities within any country

• Capture persons with similar problems across countries

Intended use of data

• Development of a demographic means of understanding disability: Can compare persons with and without disability on levels of participation in employment, education, or family life to see if persons with disability have achieved social inclusion

• Monitor effectiveness of programs / policies to promote full participation

• Monitor prevalence trends for persons with limitations in specific basic activity domains

Limitations• One set of measures will not satisfy

multiple needs for disability data• Excluded populations:

• Very young children• Institutionalized population• Homeless• Floating populations

• It is not our purpose to:• identify every person with a disability within every

community • replicate a population evaluated across a wider

range of domains that would be possible with other forms of data collection

Constraints of the census format

• Limited space• Indicated domains reflect types of

questions that would identify largest population for this purpose and those most culturally comparable

• Allows some flexibility for additional culturally relevant questions for specific countries

Pre-test implementation• Implementation protocols developed:

• Objectives and evaluation plan for field / cognitive testing

• Cognitive test plan• Translation• Enumerator training• Sample design issues• Plan for tabulation, analysis and report writing

• Two regional workshops successfully implemented• June 20-22, 2005 / Nairobi, Kenya• September 19-20, 2005 / Rio de Janeiro, Brazil

Standardized testing • 15 countries participating

• 13 funded via World Bank grant, 2 self-funded

• Cognitive tests in 12 countries• Congo, Egypt, Gambia, India, Kenya, Lesotho,

Mauritius, Philippines, Uganda, Mexico, Tanzania, Vietnam

• Field tests in 2 countries• Gambia, Vietnam

• Combined cognitive/field test in 3 countries• Argentina, Brazil, Paraguay

Pre-test objectives

To determine if:

1) the questions are being interpreted as intended by the developers in that they are capturing the important aspects of the functional domains selected and

2) the questions are interpreted consistently across countries.

Pre-test evaluationEvaluation

1) Validity: a) Content validity: How well WG question set

compares with expanded disability measures

b) Criterion related validity: How well individual WG questions compare to relevant similar concept in a comparison measure

c) Face validity: Does the measure ‘look to be valid’?

2) Reliability: test/re-test

Cognitive test

Cognitive test• Objective: to determine if questions are being

interpreted as intended and if interpretation is consistent across countries• WG cognitive test more structured than usual• Ensures a greater level of standardization across

test sites• Understand how the response mechanisms

operate in the different countries

Cognitive test

Components of cognitive test• Purposive samples• Asked WG questions and series of detailed

questions in the same domains• Traditional cognitive probes• Collected information on response process• Analyzed problematic response patterns• Interviewer report on problems respondent had

with questions

Detailed questions on functioning (example, cognition)

WG question: • Do you have difficulty remembering or concentrating?

Detailed questions:• Do you have difficulty remembering the names of people or places?• Do you have difficulty remembering appointments? • Do you have difficulty remembering how to get to familiar places?

• Do you have difficulty remembering important tasks, like taking medications or paying bills?

• Do you have difficulty concentrating on doing something for ten minutes?

• Do you have difficulty learning a new task, for example, learning how to get to a new place?

• Do you have difficulty finding solutions to problems in day to day life?

Information on response process

• Questions to interviewers• Need to repeat any part of the question?• Any difficulty in using response options?• Did respondents ask for clarification or

qualify their answers?• Follow-up probes to understand response

process

Correlates of cognitive difficulty

Unadjusted ORs

95% confidence interval

Concerned 11.18*** 7.34 – 17.05

Learn new task 7.93*** 4.95 – 12.70

Problem solving 7.22*** 4.59 – 11.35

Correlates of cognitive difficulty

Difficulties remembering

Unadjusted ORs

95% confidence interval

People or places

9.12*** 5.85 – 14.23

Appointments 6.55*** 4.36 – 9.82

Finding places 5.20*** 3.43 – 7.86

Important tasks 6.40*** 4.27 – 9.60

Socio- demographic correlates of cognitive difficulty

Unadjusted ORs

95% confidence interval

Gender

Male 1.04 (non-significant) 0.70 – 1.54

Age

35-54

55 +

1.05 (non-significant)

2.05***0.64 – 1.72

1.26 – 3.33

Overall, questions worked well

• Well understood• Responses consistent• Problem areas

• Glasses clause – WG is revisiting but solution has a lot to do with translation protocol

• Communication Question – WG will revisit to see if question can be simplified

• Introduction to the question set

Field test

• Conditions closely approximate how final study will be done

• See how WG core questions function in different countries

• Useful to compare WG set to a larger set of more detailed questions to determine whether the same population is identified by each set

Field test

• Undertaken in Gambia, Vietnam, and South Africa

• Checking for internal consistency• Results comparable to changes seen in other

countries with move towards functional approach to measuring disability

Field test: Analysis• Constructed three thresholds for disability based on

WG census question: • D1=1 if any difficulty• D2=1 if a lot• D3=1 if unable to do

• Constructed three thresholds for disability based on extended questions: • ED1• ED2• ED3

• What was the D measure picking up compared to the ED measures?

Example: Mobility question

• WG question: • Do you have difficulty walking or climbing steps?

• D1=1 if “some difficulty”, “a lot of difficulty”, or “cannot do at all”

• D2=1 if “a lot of difficulty” or “cannot do at all”

• D3=1 if “cannot do at all”

• Extended Questions: • Do you have difficulty going outside of your home?(0)

• Do you have difficulty walking a long distance such as a kilometer (or a mile)?

• ED1=1 if “some”, “a lot” or “cannot do” for either question

• ED2=1 if “a lot” or “cannot do at all” for either question

• ED3=1 if “cannot do at all” for either question

South Africa : N=185D vs ED (2, 3 or 4 Qs)

01020304050607080

Res

pond

ents

(n)

D1 (High) D3 (Low) ED1 (high) ED3 (Low)

Disability data collection recommendations

• Do not use the word disability• Functional approach• Separate functional domains• Scaled responses• Definition of disability can be made ex post,

and hinge on domains and thresholds used

Meeting products and more information

• Executive summaries of meetings, presentations and papers posted on the Washington Group website:• http://www.cdc.gov/nchs/citygroup.htm

• Publication of key papers in a special issue of Research in Social Science and Disability