Contribution of Survey to Health Contribution of Survey to Health Systems Performance Monitoring: Systems Performance Monitoring:

experience with the experience with the World Health SurveyWorld Health Survey

THE MALAYSIAN EXPERIENCETHE MALAYSIAN EXPERIENCE

28-29 September 200628-29 September 2006Montreux, SwitzerlandMontreux, Switzerland

IntroductionIntroduction

• World Health Survey 2002World Health Survey 2002

– Nationwide community surveyNationwide community survey

– Multistage stratified sampling representative of Multistage stratified sampling representative of

populationpopulation

– Stratified for state & urban rural locationStratified for state & urban rural location

– National & rural/urban location estimatesNational & rural/urban location estimates

– Where possible estimates across various socio-Where possible estimates across various socio-

demographic variables demographic variables

– Institutionalised population excluded (<3%)Institutionalised population excluded (<3%)

IntroductionIntroduction• World Health Survey 2002World Health Survey 2002

– Data collection early March – mid April Data collection early March – mid April

20032003

– 200 personnel of various categories 200 personnel of various categories

including temporary research assistantsincluding temporary research assistants

– MOH facilities & vehiclesMOH facilities & vehicles

– Nationwide publicityNationwide publicity

Field PreparationField Preparation

–Organisational structureOrganisational structure•Advisory/Steering committeeAdvisory/Steering committee•Central Research TeamCentral Research Team•Field Data Collection teamField Data Collection team•Data Entry TeamData Entry Team

Central Research Team

State Teams (12 + 2 states)

State Liaison Officer (1 per state)

2 Teams per state for Peninsular Malaysia + 4 Teams per state for Sabah & Sarawak

( 32 teams)

1 Field Supervisor per

team(32 supervisors)

1 chief scout per state (14 chief scouts)1 scout (PHO) per district (part time)

4 Interviewers (3 temporary staff + 1

nurse) per team(128 data collectors)

1 driver per team

(32 drivers)

National Advisory Committee

ORGANISATIONAL STRUCTUREORGANISATIONAL STRUCTURE

Implementation strategyImplementation strategy

More rural

Less densely

populated areas

Less rural

high density areas (larger sample size)

Very highly density Very highly density

areas (larger areas (larger sample size)sample size)

Mobilization of survey teams across districts & states

– Budget proposal Budget proposal (Oct-02)(Oct-02)

– Translating & Pre-testingTranslating & Pre-testing instrumentsinstruments (Oct-Nov 02)(Oct-Nov 02)

– Road showsRoad shows (16-20 Dec 02)(16-20 Dec 02)

– Recruitment of research assistantsRecruitment of research assistants (Jan-Feb 03)(Jan-Feb 03)

– Field preparation- (sampling & procurement)Field preparation- (sampling & procurement) ((Jan-Feb 03)Jan-Feb 03)

– Identification of EBs and Tagging exerciseIdentification of EBs and Tagging exercise (Jan-Feb 03)(Jan-Feb 03)

– TrainingTraining (17 Feb- 15 Mar 03) (17 Feb- 15 Mar 03)

– ““Launching”Launching” (28 Feb 03)(28 Feb 03)

– Data collectionData collection (Mac-April 03)(Mac-April 03) – Publicity in various mediaPublicity in various media (Feb – April 03)(Feb – April 03)

– Data entryData entry (Mac-April 03)(Mac-April 03)

Survey Survey implementation scheduleimplementation schedule

– Presentation of preliminary findingsPresentation of preliminary findings (July 03)(July 03)

• Programme heads and service providersProgramme heads and service providers

• Share contents of WHS 2002Share contents of WHS 2002

• Identify additional questions relevant for programme needsIdentify additional questions relevant for programme needs

– Further assistance with analysis from WHO Further assistance with analysis from WHO (July 05)(July 05)

– Mini-conference (September 2005)Mini-conference (September 2005)

• Invited resource person from WHO Invited resource person from WHO

• Senior officers from programmes and various operational levelSenior officers from programmes and various operational level– Clinicians, public health specialists, public health engineers, nutritionists, human resource personnelClinicians, public health specialists, public health engineers, nutritionists, human resource personnel

Survey Survey implementation scheduleimplementation schedule

– Report writing (October 2005 – June 2006)Report writing (October 2005 – June 2006)

• 5 volumes5 volumes

• 4 drafts4 drafts

– 3 volumes already with printers (August 2006)3 volumes already with printers (August 2006)

– Proposed presentation to senior management (Nov – Dec 2006)Proposed presentation to senior management (Nov – Dec 2006)

Survey Survey implementation scheduleimplementation schedule

WHS 2002WHS 2002

Snapshot of

Data Quality

WHS 2002

• Sample size = 7528.

• Response Rate = 80.2%

• Analysis (as per WHO) done in 2005.

Sampling

Response Rate

Location

Urban Rural TotalHousehold interviews

Selected 4654 2874 7528

Interviewed 3610 2516 6126

HH RR (%) 77.6 87.5 81.4

Individual interviews

Selected 4654 2874 7528

Interviewed 3554 2484 6038

Individual RR (%) 76.4 86.4 80.2

HH Sample Deviation Index

Individual level Sample Deviation Index

Missing Data

Reliability

Distribution of population by Location

51%

12%

5%

32%

Metropolitan

Urban Large

Urban Small

Rural

HH Level: Sociodemographic Profile(weighted)

• Mean Household size = 4.2

• Male : Female ratio =0.98

• Geographical location: Majority in urban areas

Household size

Q1poorest 4.00

Q2 4.11

Q3 4.16

Q4 4.29

Q5richest 4.89

Missing 3.57

HH Level: Sociodemographic Profile(weighted)

Income by Location (HH Data n=6126)

0% 20% 40% 60% 80% 100%

Metropolitan

Urban Large

Urban Small

Rural

ALL

Q1poorest

Q2

Q3

Q4

Q5richest

QMissing

Presentation & Utilisation of findingsPresentation & Utilisation of findings

• To date, results yet to be formally presented to top management

• General impression of findings– Value added as it provides a better perspective (new

dimensions) of country HS performance – Better performance in some aspects but “eye-opener”

in others!

• Application found downstream at various levels

Utilisation of findingsUtilisation of findings

• National levelNational level– Key input into development of National Health Financing

Mechanism• Volume 5 (Responsiveness section)Volume 5 (Responsiveness section)

– Areas respondents were not satisfied– State of hospital– Utilisation pattern– Cost of care

• Volume 4 (H Expenditure)Volume 4 (H Expenditure)– OOP, perception on risk pooling

• Volume 3 (Coverage)Volume 3 (Coverage)– understanding of current situation of service provision

• Volume 2 (Risk factors)Volume 2 (Risk factors)– What should go into the basic benefit package– ANC, HIV transmission amongst mothers, condom use for prevention

Utilisation of findingsUtilisation of findings

• Programme level

– Responsiveness component used in development of “soft skills”

training modules for health workers & evaluation of front line

customer services

– Input into development of Patient/People - centred services

– Verify effectiveness of current programmes and related activities

– Support (evidence-based) the justification of newly introduced

activity

– Recommend development of new strategies/activities to specific

risk groups

– Identification of new research to look into impact

Potential applicationPotential application

• Evaluation of Mid term review of performance of 9th Malaysia Plan

What we have learnt…..What we have learnt…..

• Objectives have been achievedObjectives have been achieved– Contribution to development of cross-country

measure – Assess country HS performance– Transfer of technology (to some extent!)

What we have learnt…..What we have learnt…..

• Costly affairCostly affair– Human & financial resource intensive

• Need for “buy-in” from all sectorsNeed for “buy-in” from all sectors– To ensure successful survey implementation– To ensure usage of findings

What we have learnt…..What we have learnt…..

• Need for advanced planning (at least Need for advanced planning (at least 1yr!!)1yr!!)– Negotiation with operational managers for

manpower assistance & other logistics– Budget proposal and approval

What we have learnt…..What we have learnt…..• Instrument itselfInstrument itself

– Translation into local language poses a challenge!

– Complicated & lengthy (2.5 hrs mean completion time)

– Some aspects politically sensitive (8000 series were omitted)

– Definitions of certain variables differ from own country• Uneasiness & defensive about country performance by various

programme heads

– E.g. HIV & Human resource• In retrospective

– Should allow for 2 sets of definitions

What we have learnt…..What we have learnt…..

• Analysis & InterpretationAnalysis & Interpretation– Stata software

• Limited expertise

– Complex analysis• CHOPIT (done by WHO team, Geneva)• Some analysis still pending

– E.g. Y star results for adjustments for cross country comparison (responsiveness section still pending)

• Duration of whole activity (data collection to Duration of whole activity (data collection to analysis to report writing)analysis to report writing)– Too long

What we have learnt…..What we have learnt…..

• Lack of expertise to translate research findings into action– e.g. interpreting findings, writing policy briefs

Our conclusion…Our conclusion…

• In general, – WHS 2002 a useful tool for management– Provides good info/added value about our HS

performance not found in routine M & E

• But…. – A costly affair – A painful exercise (blood, sweat & tears!) of

negotiations, personal sacrifices, energy sapping, etc

• And…..– Success requires careful planning

Our recommendations….Our recommendations….

• Have sufficient budget for implementation• Ensure top management commitment• Allow countries to adopt & adapt sections applicable to

them• Need to simplify

– vignettes– health status

• We have the mean score but as there is no benchmark, results not really meaningful

• Need to make instrument brief• Assist countries without capacity to undertake national

community surveys

Our recommendations….Our recommendations….

• Assist countries to market findings to policy makers– Translation into policies– Help to see implications of findings to current

policies

• Need to build greater “in-country” capacity from beginning to end

Thank you