Brief Historical Overview of the Budapest Initiativeand Testing Activities

20-22 January 2010Palais des Nations, United NationsGeneva, Switzerland

Measuring health statusfor international comparisons…

• What constitutes core health?• What measures should be used?• What standards for producing the data?• Are the data internationally comparable?• What methods should be used for comparisons?

Historical Overview:Establishment of the TF and First Meeting

•May 2004: UNECE sponsored a Joint Meeting on the Measurement of Health Status with WHO and Eurostat

•Broad consensus on a) the need for multiple indicators to provide a full statistical picture of population health, both for individual country use and for international comparisons; and b) the immediate need for an indicator of ‘health state’

•Recommendation to set up a Task Force on the measurement of heath state under the aegis of the UNECE

•The first meeting was held in Budapest in November 2005, and following the City Group convention, now known as the Budapest Initiative (BI)

Historical Overview:Purpose and Objectives

Purpose: To develop a new common instrument to measure health state suitable for inclusion in national interview surveys which will provide basic necessary information on population health.

Objectives:

•To develop a question set to assess overall health state through a number of domains of functioning;

•To describe trends in health over time within a country, across subgroups of a population, and across countries; and

•To do so in the framework of official national statistical systems.

Historical Overview:Framework, Definitions, Domains

• Development of a conceptual framework defining the concepts of health status and health state

• Paper: Health as a Multi-Dimensional Construct and Cross-Population Comparability (2005)

• Adopted a set of criteria for selecting functional domains• Paper: Criteria For and Selection of Domains for the Measurement of

Health Status (2005)

• Selection of domains• Developed a question set to measure selected domains

• Paper: Conceptual and Logistic Issues in Item Construction and Proposal

Questions for Domains (2005)

What is Health State?

Health State measures functional ability as opposed to other aspects of health:

•Determinants and risk factors

•Disease states

•Use of health care•Environment barriers and facilitators

Functioning is measured in terms of ‘capacity’ not ‘performance’

• ‘Within the skin’

•Without the use of aids in a reasonable environment

Health Status and Health States

Health states (capacities)

Determinants of health

Genetic contribution

Lifestyle/behaviour

Death

Physical environment

Physiological risk factors and

risk markers

Diseases, symptoms

and injuries

Domain 1

Domain 2

Domain N2

. .

.

.

HEALTH STATUS

Domain N1

Economic environment

Social environment

Physiological risk factors

and risk markers

Quality of life

Wellbeing

Health State: an individual’s levels of functioning within

a set of health domains.

BI Focus and Guiding Principlesfor the Question Set

• Measure the health states of individuals;• Operationalize health state as functional ability;• Focus on capacity rather than performance;• Measure across a parsimonious number of health

domains, capturing the most variation in health;• Meet high standards of validity and international

comparability; and • Simple and clear, and easily translated into many

languages.

Criteria for the Selection of Domains

•Relevance: face validity, breadth of domains, importance for population health monitoring, draw on selected key ideas of the ICF;

•Feasibility: reasonable for inclusion in health interview surveys, consistency in meaning across multiple social contexts, heterogeneity, parsimony of domains

•Measurement: statistical & structural independence, clear series of levels within domains, within/near the skin, suitability for preference measurement.

In order to identify the domains of health to be included, a set of criteria were established:

Final 7 Domains for BI Questions

• Vision• Hearing• Walking / Mobility• Cognition• Affect• Pain• Fatigue*

The original set of 10 domains included dexterity and social functioning/ relationships, both of which have been dropped. Anxiety was combined with depression in the current domain of affect.

Criteria and Considerationsfor Item Construction

• Number of questions per domain• Ensuring uni-dimensionality• Duration of the recall period• Dealing with assistive devices and medications• Item wording and response categories

• Positive vs. negative wording• 4- vs. 5-category response sets

• Summary/preference health measures

Testing the Question Set

Testing the BI Questions

Round 1:• November – December 2006

• BI-only test (4 countries, 3 languages): Australia, Canada, Italy, U.S.

• Cognitive testing of all 7 domains

• Results reviewed at January 2007 TF meeting

• Separate studies and reports, inconsistent analysis methods, some general findings

• Outcome: BI-M1 question set, submitted to Eurostat

Three rounds of testing have been conducted as part of the development of a BI question set.

Testing the BI Questions

Round 2:• November 2007 – January 2008

• BI-ESS collaborative test (7 countries, 6 languages): Bulgaria, Germany, Portugal, Spain, Switzerland, U.K, U.S.

• Cognitive testing of 6 domains (absent Vision)

• Improvements: Evidence-based methodology & systematic comparative analysis of patterns

• Analysis meeting in U.S. February 2008, all participants

• Results documented in report

• Outcome: Further changes for cognitive testing recommended

WG-BI SimilaritiesSubstantive overlap/Differing perspectives:• Question domains include the same basic and complex activities

• Disability includes interaction with environment and civil rights perspective

Requirements for question sets:• Minimize burden on national data collection

• Parsimony in the number of indicators and measures; domains are succinct, clearly defined

• Reasonable expectation of high quality responses from samples of the general public, demonstrated validity of measures

• International comparability, relevant at national and international level

• Focus on aspects of health that are more likely to produce comparable data

• Need for cross national cognitive and field testing

• Questions must be simple, clear, easily translated into many languages

• Amenable to multi-modes of collection

Testing the BI Questions

Round 3 (Cognitive Test):

• January 2009 – March 2009

• UNESCAP-WG-BI collaborative test (10 countries): Cambodia, Canada, Fiji, Kazakhstan, Maldives, Mongolia, South Africa, Sri Lanka, Philippines, U.S.

• Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities)

• Improvements: On-line data management and analysis tool

• Analysis meeting in U.S. May 2009

• Outcome: Further changes for field test recommended

Testing the BI Questions

Round 3 (Field Test):

• June 2009 – August 2009

• WG-BI-UNESCAP collaborative test (5 countries): Cambodia, Kazakhstan, Maldives, Sri Lanka, Philippines

• Cognitive testing of 11 domains (additions: Upper Body, Communication, Learning, and Life Activities)

• Improvements: Approx 1,000 field interviews at each site

Testing Issues and Methodological Developments

General Issues for Testing

• Cross-cultural comparability: Do the survey questions work consistently across all countries and subgroups?

• Translation comparability: Do terms (both in the question and in the response set) have the same meaning across countries?

• Validity: Do respondents interpret questions consistently regardless of country, language, or demographic? Do respondents use the same thought processes to answer questions?• If not, then, why are there differences? What about the

countries, languages or demographic subgroups generate different response processes?

• How can we “fix” or manage these differences through question design?

Objectives for the Test ProtocolPurpose: To develop systematic comparable method with joint analysis.

• Evidence based

• Joint and coordinated interviewing

Similar protocol

Similar sample

Understanding of differences (at a minimum)

• Joint and coordinated analysis

With interview data

Evidence based (as opposed to opinion)

Lessons Learned

• Semi-structured cognitive interviews offers critical and unique insight into cross-national question performance

• Transparency is critical• Of data from interviews• Of the process for drawing conclusions

• Data collection oversight • Better data management

Because of a physical, mental or emotional problem, do you have difficulty concentrating, remembering or making decisions?

ID Immediate words after question

Response Category

Description of any response Difficulty

Activity, experience or Situation Discussed

took meds ? If so, was this considered?

Way in which answer was formed; pattern of Calculation

S1 "None" No difficulty None Forgetting an umbrella when the forecast is for rain.

no She was thinking about her ability to remember & concentrate in general. Even though she occasionally forgets things it's not a problem. clause ignored

S2 "Sometimes I have trouble rememberingthings"

a little difficulty

Easy to answer, but this difficulty she sees as normal stuff & isn't concerned about it; i.e., this isn't a disability

walking in to the kitchen and forgetting why she's there

no a little difficulty means that sometimes she forgets things - it's frequency of forgetting, not significance of things forgotten; 1st clause ignored

Example BI chart

Q-Notes

• On-line data entry and analysis tool• Allows for continuous oversight• Facilitates quick but thorough analysis

• Designed around analysis principles

Next Steps Toward BI-M2Activities in 2010

Next Steps for the BI Questions

The BI-M1 did not meet all of the objectives for the question set (some domains absent, some room for improvement).

Testing to date has been focused on developing BI-M2.

BI-M2 will be based on lessons learned from:• 2009 UNESCAP cognitive and field tests (reviewed here)

• 2010 European cognitive tests (Round 4)

• 2010 experience with U.S. National Health Interview Survey

2010 Testing - Round 4Comparative Cognitive Interviewing Project

Testing will return to the European region, just as in Round 2.

• Spain (Spanish)

• Portugal (Portuguese)

• Italy (Italian)

• France (French)

• Germany (German)

• Switzerland (French, German, Italian)

• United States (English, Spanish)

Project Group Coalition

Testing will be conducted by:

• Members of the 1st comparative group (from Round 2)

• Methodologists from QEM workshop at NCHS

• US and Canada

Project Goals• Continue methodological work begun in Round 2 testing.

• Narrative interviewing vs. structured probes• Interpretations of vague response categories across cultures and

languages• Use of software for data collection.

• Examine issues raised at QEM.• Use of mixed-method (PROMIS-WG; cognitive and field tests)

• Continue examination of WG/BI extended set, specifically what questions capture.

• Patterns of interpretation• Patterns of calculation• Both in-scope and out-of-scope interpretations

Domains for Inclusion in the Test

• Cognition

• Communication

• Affect

• Learning

• Pain

• Fatigue

• PROMIS subset