Survey SolutionsCOMPUTER-ASSISTED PERSONAL INTERVIEWING

Michael Lokshin

World Bank

The problem:Average interval between surveys (years)

2

3

4

5

6

7

9

0 2 4 6 8 10

LAC

ECA

World

EAP

SA

SSA

MENA

The problem:Percentage of missing or invalid answers

(questions on learning and literacy)

0% 2% 4% 6% 8% 10% 12% 14% 16% 18%

Malawi

Uganda

Ghana

Nepal

Madagascar

Burundi

Ethiopia

Laos

Only 11 out of 55 countries in Africa had comparable poverty data for2005. The situation has not changed much since.

• 2011: Comprehensive Assessment of CAPI software products is commissioned to University of Maryland by the WB.

… no existing software provides exactly the right mix of features necessary for the sort of surveys conducted by World Bank and its clients.

• 02.2012: LSMS and Computational Tools teams of Research Department of the World Bank with support from Global Strategy of FAO started development.

• 09.2013: First public version of Survey Solutions is released

• 05.2015: Survey Solutions 4.4 is released

Background

Pick 3: Fast, High Quality, Cheap

Ensure data quality and comparability

Improve timeliness of data collection

Cost-effective, sustainable solution for NSOs

From data collection to policy: 1-2 years

From data collection to policy: 4-5 months

Data analysis

Policy decisions

Data collection

Data entry

Data cleaning

Data processing

CAPI

Results:

• Simple yet flexible system for the non-expert users. Typical clients – National Stat Offices

• Functionality for

– data capturing

– survey management

– data management

• Tablet-based with ability to display and navigate through multi-level large questionnaires.

• Support of panel surveys and complex validation algorithms.

• Cost effective system that can be used and supported by NSOs without external TA.

CAPI System Requirements

• CSPro – free, closed source. resides in Bureau of Census USA. Funded by USAID. Windows and Android

• ODK – free, open source software funded by USAID; UWashington, UC Berkeley, UC Davis. A– SurveyCTO, license fee

• Blaise – Statistics Netherlands, license fee. W

• SurveyBe – closed source, license fee. W

• SurveyToGo – closed source, license fee. W

Other CAPI products

𝐷𝑄 = 𝛼𝐷𝐶 ∙ 𝛽𝐷𝑀 ∙ 𝛾𝑆𝑀

DQ – data quality

DC – data capturing

DM – data management

SM – survey management

Survey Solutions : α ≈ β ≈ γ

• Pros:

– A flexible system that can

accommodate complicated

surveys.

• Cons:

– Custom solutions cannot be

easily modified for different

environments

– High costs

– High requirements for the users.

– Questionable sustainability in

the high turnover environments

typical for NSO.

CAPI: Expert-centered approach

• Pros:

– High levels of standardization, solutions could be easily adapted for different surveys in different countries.

– Lower requirements for the users, fast learning curve

– High sustainability

• Cons:

– Less flexible compared to custom solutions of expert-centered software.

CAPI: User-centered approach

• Sustainable, low-cost system for NSO

• Simple, flexible interface for questionnaire development

and testing.

• Tablet interface allows easy navigation through complex

questionnaires.

• Standardized survey management protocol based on the

best practices of data collection

• Intuitive, informative survey status reporting, survey

maps.

• Yet, powerful language for data validation and control of

questionnaire flow.

Survey Solutions: hybrid approach

Survey Solutions

Competition

• Large questionnaires: several thousand questions

• All standard types of questions: – text, numeric, date; multi-choice; dynamic lists;

– Linked questions: Whose cow is it? User can select from the list of household members.

– GPS location; Time; Barcode; Binary files (pictures)

– HQ/Supervisor-filled questions

• Rosters: can be generated from:– Fixed lists; dynamic lists; numeric; multi-choice questions.

– Nested rosters with unlimited degree of nesting

• Interviewer Comments on a question and interview

• Question Instructions

Survey Solutions: data capturing

GPS in action

Plots can be easily and precisely measured by simply walking around the area with the tablets

Ability to measure distances to banks, schools, roads, wells. Maps used for optimal routing

Sensors

Soil Fertility sensor

Water quality testing

Multi-purpose sensor

Vision tests

Portable scales

Wearable devices

Para Data: Adptive Survey Design

• Improve data quality by correcting survey process during the field operation.

• System records all events with timestamps that happen on a tablet: Data entry, Data correction, Responsibility changes, etc.

• Analysis of time per interview, time per question, section. • Changes in productivity over time, for different

interviewers, teams• Quality control, monitoring and evaluation

Prices of tablets start from $60: cheaper than paper questionnaires

Questionnaire delivery, Changes or corrections at no cost

Free software but not free survey

Software might be free but the TA support is very expensive:

Competitor #1: Proposal for a survey in Sri Lanka, 24,000 hhs –$340KCompetitor #2: Actual cost of a survey in Uganda, 10,000 hhs -$350K

Survey Solutions: Cost varies from about $5,000 to 0,000s for large surveys.

• Designer: online tool for questionnaire creation and validation @ solutions.worldbank.org

• Tester: Android app connected to Designer to test questionnaires in real time. @ Google play

• HQ: online tool for centralized survey management, validation, data aggregation and reporting.

• Supervisor: online/offline tool to manage process of data collection on a team supervisor level.

• Tablet CAPI: Android app for data capturing on a tablet.

Survey Solutions: system components

SURVEY PIPELINE: 50+ SURVEYS IN 30+ COUNTRIES

• Africa: Malawi, Uganda (4), Tanzania, Togo, Benin, Madagascar (3), Niger, Cote D’Ivoire (4), Zambia (3), South Africa, Ghana, South Sudan, Mozambique, Nigeria

• MENA: Djibouti (4), Kuwait, Morocco, Tunisia• SAR: Bhutan (3), India, Pakistan• EAP: Thailand (2), Myanmar (7), Pacific Islands (4)• ECA: Armenia, Azerbaijan, Kirgizstan• LAC: St. Lucia (4), Paraguay, Belize

• Survey types: LSMS, HBS, LFS, Enterprise Survey, EDU, Health

• Largest survey: Malawi - 200,000 households• Largest questionnaire: Tanzania – 3,000 questions

Enumerators repeat interviews if errors are detected

Interviewers synchronize theirdevices and upload completedquestionnaires

Supervisors monitor the submissions

Questionnaires with no errors are uploaded to the central server

Supervisors assign households to individual interviewers

WiFi/USB

Researchers design questionnaires using visual tools and upload them to the central server

Interviewers visithouseholds and collect data

HQ distributes the sample lists across teams of enumerators

Internet

WiFi/USB

Internet

Main Website: www.worldbank.org/capi

Online questionnaire designer: solutions.worldbank.org

Youtube channel: Survey Solutions

Real time status of interviews

Map of the survey

Monitor the survey by checking the GPS locationof where and when the interview took place.