gallup world poll ilostat - aapor

Closing Data Gaps in Global Employment MetricsGallup World PollILOSTAT

Copyright © 2015 Gallup, Inc. All rights reserved.

The State of GlobalEmployment Metrics


International Comparability Lacking

• Global employment metrics have been essential tools for informing highlevel political and economic policy for some time now.

• However, quality and consistency of official data varies significantly fromcountry to country and region to region.

• Definitional and methodological differences also significantly impactinternational comparability.

• The ILO invests a lot of work into standardizing, updating, and maintainingthese metrics, but they are ultimately dependent on national statisticaloffices for implementation and reporting.


Closing Gaps in Global Employment Metrics

• Research question: Can we cost-effectively gather global employmentdata that will be useful 1) for policy makers and 2) for economic and socialresearch, that can close the gaps in coverage and comparability thatcurrently exist within the official statistics typically gathered by NationalStatistics Offices (among others)?

• How do Gallup and ILO numbers compare when both reportemployment statistics for a given country in a given year?

• What remedial adjustments might improve comparability,given the methodological, coverage, and definitional variations involved?

• Can Gallup data be used to model missing data where there are noofficial employment statistics reported for a given country in a givenyear?


Sources


ILOSTAT

• The ILOSTAT database “Provides recent labor data for over 100 indicatorsand 165 economies. Annual data prior to 2008 and additional information,such as sources and methods, are still available in LABORSTA and graduallywill be migrated to ILOSTAT.” -http://www.ilo.org/inform/

• The ILO gathers basic country-level employment data from nationalstatistics offices (NSOs) around the world. These data originate variouslyfrom national Labor Force Surveys, Household surveys, population censuses,administrative records, and official estimates.


Gallup World Poll

• Since 2005, Gallup has conducted nationally representative surveys face toface or via telephone in 165 countries (158 in the period 2009-2013) in over140 languages and using various survey modes.

• Face-to-Face/CAPI: In the developing world, including much of LatinAmerica, the former Soviet Union countries, nearly all of Asia, theMiddle East, and Africa, Gallup uses an area frame design for face-to-face interviewing in randomly selected households.

• CATI: In OECD countries, the Persian Gulf, and other countries withreliable RDD sample frames available Gallup uses CATI methodologytargeting landline and mobile phones.

• Questions cover a wide range of topics, including wellbeing, economicconditions, governance issues and community attachment.


ILOSTAT - Definitions


• The “Working Age” definition varies between countries and regions• Less than one third (28%) of countries strictly follow the ILO standard definition

of 15 years of age and older.• Not all countries report true national statistics

• Urban areas only: Argentina, Bolivia, China, Ethiopia, Peru, Uganda• Official estimates/records: Belarus, China, Kazakhstan, Kuwait (2009-10),

Yemen, Bahrain (2011), Guinea

Maximum Ages Minimum Ages

None1012141516

64656670727475

None


Reporting Frequencies, 2009-2013

ILOSTAT Coverage


• Reporting frequency varies considerably between countries and regions• One third (33%) of ILOSTAT countries reported employment statistics

for fewer than three out of the five years 2009-2013• As of March 2015, ILOSTAT had 2014 statistics for four countries

12345

22%

11%

6%5%

56%

Frequency of EmploymentReporting, 2009-2013

Once TwiceThree times Four timesEach year


Gallup World Poll Coverage


• The “Working Age” definition remains consistent across the world, and inline with ILO standards: 15 years of age and older, with no maximum.

• n-sizes are much smaller and vary between countries and years.• Reporting frequency is comparable in economically developed regions and

greatly improved in developing regions.

12345

8%6%

11%

23%52%

Frequency of EmploymentReporting, 2009-2013

Once TwiceThree times Four timesEach year

Reporting Frequencies, 2009-2013


Gallup and ILO Employment Categories• Employed Full-time for an

Employer (Payroll-to-Populationindex)

• Worked total of 30 hours ormore for compensation inthe prior week for one ormore employers.

• Self-Employed Full-Time• Worked 30 hours or more

for themselves in the priorweek. May also work part-time for an employer.

• Part-Time• Worked less than 30 hours

in the prior week.

• Part-Time but Want Full-Time• Worked less than 30 hours

in the prior week but want towork more than 30.

• Unemployed• Did not work in the past

week, but did look for workin the past 4 weeks andwould be able to acceptwork if offered.

• Out of the Workforce• Neither working nor looking

and available for work.Copyright © 2014 Gallup, Inc. All rights reserved.

• Employment-to-Population• All those employed – even

minimally – as a percentageof the Working Agepopulation.

• Unemployment Rate• The unemployed as a

percentage of theEconomically ActivePopulation (Employed andUnemployed).

• Economically Active Population• All those 15 years of age

and older who are workingor looking and available forwork.

• Working Age Population• All those 15 years

of age and older.


Comparisons:Initial,Age-adjusted,Categorically-adjusted


Initial Comparison – Employment to Population


Comparing the ILO’s headline figure ofEmployment to Population (E2P) withWorld Poll E2P data over matchingcountries and years:

• There are 430 country-years with E2Pdata from both sources

• 45% are within the margin of error,another 3% are still within 5%

• However, 21% are more than 10% off.

20%

30%

40%

50%

60%

70%

80%

90%

20% 40% 60% 80% 100%

Gal

lup

Wor

ld P

oll

ILOSTAT

ILO vs Gallup E2P, 2009-2013

45%

3%

31%

21%

ILO vs. Gallup E2P Comparisons, 2009-2013

Within Margin of Error

Within 5%

Within 10%

More than 10% difference


Initial Comparison – Unemployment


Comparing the ILO’s Unemploymentrate figure with World PollUnemployment data over matchingcountries and years:

• There are 453 country-years withUnemployment data from both sources

• 66% are within the margin of error,another 4% are still within 5%

• 15% of data points are more than 10%off.

0%5%

10%15%20%25%30%35%40%45%50%

0% 20% 40%

Gal

lup

Wor

ld P

oll

ILOSTAT

ILO vs Gallup Unemployment, 2009-2013

66%4%

15%

15%

ILO vs. Gallup UnemploymentComparisons, 2009-2013

Within Margin of Error

Within 5%

Within 10%

More than 10% difference


Initial Comparison – E2P Region



Initial Comparison – Unemployment by Region



Comparisons with Age Brackets Aligned


Wherever possible, we filter data on both sides in order to compare equivalentage ranges in each country-year:

• There are 387 country-years where we are able to align the age ranges on E2Pmeasurements.

• However, the average difference in E2P measurements actually increases by0.5 points, from 6.6% to 7.1%

• A similar comparison of age-aligned Unemployment measurements (n=334)shows no change in the average difference at all (4.2%).


Definition and Methodology Differences


• The Military: Some countries exclude military from the target population. (Israelpre-2012)

• Expatriates: Prior to 2013, Gallup only surveyed in Arabic in the Arabian Peninsula,while the non-Arab expatriate populations in countries such as Qatar and Bahrainactually outnumber the Arab population.

• Category Definitions: Nigeria defines unemployment as including all those whocannot find a job working 40 hours per week – a massive divergence from thestandard definition.

• Coverage: Ethiopia only covered urban areas until 2013, whereas rural areas havevery different employment patterns.

• Questionnaire Design: Some Central American countries’ (Honduras, Guatemala)questionnaires go to great lengths to categorize respondents in an employed categoryof some kind, and include unpaid work in the definition of “employed.”

• Estimates: Some countries rely on official estimates or employment registries ratherthan the standard “activity-based” definition promulgated by the ILO.

NOTE:• Volatility in trend on either side occasionally results in significant differences between

Gallup and ILO figures.• In some instances there is at least circumstantial evidence that respondents might tell

a government survey worker one thing and Gallup something else.


59%52%

43%

71%

87% 87% 86% 86% 87%

2009 2010 2011 2012 2013 2014

QATARGALLUP ILO

Data Gap Analysis – Israel and GCC countries E2P

19

68%64%

67%61% 63% 62%

52% 54% 54%59% 60%

2009 2010 2011 2012 2013 2014

ISRAELGALLUP ILO

67% 64%

47%54%

64%

75% 75% 74%

2009 2010 2011 2012 2013 2014

KUWAITGALLUP ILO

48% 45%39% 37%

67%

71% 70%

2009 2010 2011 2012 2013 2014

BAHRAINGALLUP ILO


Data Gap Analysis – Nigeria and Ethiopia

20

3%

12% 12%10%

12%9%

20% 21%24% 24%

29%

2009 2010 2011 2012 2013 2014

NIGERIA UNEMPLOYMENTGALLUP ILO

24%29%

26% 24%28% 29%

20% 21%24% 24%

29%

2009 2010 2011 2012 2013 2014

NIGERIA UNEMPL. (ADJ.)GALLUP ILO

69%77%

58%

47% 48% 49% 52%

76%

53%

2009 2010 2011 2012 2013 2014

ETHIOPIA E2PGALLUP ILO

54%

77%

58%

47% 48% 49% 52%

76%

53%

2009 2010 2011 2012 2013 2014

ETHIOPIA E2P (ADJ.)GALLUP ILO


Data Gap Analysis – Balkans Unemployment

21

2%

7% 6%9% 9%

16%19%

23% 24%22%

2009 2010 2011 2012 2013 2014

SERBIAGALLUP ILO

19% 17%19%

24%

18%23%

32% 32% 31% 31% 29%

2009 2010 2011 2012 2013 2014

MACEDONIAGALLUP ILO

5% 6% 4%9% 9%

24%

27% 28% 28% 27%

2009 2010 2011 2012 2013 2014

BOSNIA & HERZEGOVINAGALLUP ILO

3% 3%6%

8% 8%

19% 20% 20% 20% 20%

2009 2010 2011 2012 2013 2014

MONTENEGROGALLUP ILO


Lessons Learned and Future Directions


Original research question: Can we cost-effectively gather global employment datathat will be useful 1) for policy makers and 2) for economic and social research, that canclose gaps in coverage and comparability?• Yes, ...

• Using moderate n-sizes and a parsimonious survey design, it is possible toreplicate official employment metrics with a relatively high degree of accuracy ina cost effective manner.

• This allows for more frequent reporting of employment metrics in less developedregions, with a higher degree of comparability between countries and regions.

• Where we are able to correct for coverage and methodological differences, Gallupand ILOSTAT measurements do converge, indicating that the underlyingconcepts translate well across countries and regions and that we are “measuringthe same thing.”

• But.• With smaller n-sizes and often only one survey per year, we do see some

additional volatility in these figures that impacts precision.

• More work is necessary.• Need to identify sources of small but persistent biases where they exist.• Experimentation around potential differences in responses to official surveys vs.

third-party surveys.


Other Sources ofError and PotentialRemedialMeasures


Gallup Employment Questions

• Thinking about your WORK SITUATION over the past 7 days, have you beenemployed by an employer — even minimally like for an hour or more — fromwhom you receive money or goods? (This could be for one or moreemployers.)

• In a typical week (7 days), how many hours do you work for an employer?(This could be for one or more employers.)

• Again thinking about the last 7 days, were you self-employed, even minimallylike for an hour or more? This means working for yourself, freelancing, ordoing contract work, OR working for your own or your family's business. Self-employment also includes fishing, doing farm work, or raising livestock foreither YOUR OWN or YOUR FAMILY’S farm or ranch.

• In a typical week (7 days), how many hours do you work as a self-employed individual?

• Do you want to work 30 hours or more per week?

• In the past four weeks, have you actively been looking for employment?"Actively looking" means applying for jobs, searching for jobs, and the like.

• Would you have been able to begin work had you been offered a job within thelast four weeks?


Other Potential Drivers of Differences


• Sampling differences – Gallup uses the best available official estimates for sampleframes and weights results by age, gender and education. However, there is noguarantee that an official survey uses the same frame or method. In addition certaingroups may be more likely to respond/participate in an official survey vs. a third-partysurvey.

• Sample size – Official Labor Force Surveys often have many thousands ofrespondents, but Gallup surveys typically have about 1,000. This sample sizegenerally comes with ±3-5% margins of error.

• Seasonality – Some countries use aggregated surveys that occur more than once peryear, while others depend on one survey in a given year.

• Gallup usually only does one survey, and at varying times.• Timing of surveys can also have strong impacts on measurements during times

of exceptional crisis or expansion.

• Geographic Scope – Sometimes Gallup or a local government excludes a particulararea from a survey for security or political reasons.


Cross-Validationfor EmploymentMetrics


E2P and Unemployment vs. GDP/capita



Gallup’s P2P vs. GDP per capita


• Gallup’s P2P figure does track, however• This relationship also holds within the U.S., state-by-state


P2P Vs. Political, Social Metrics



“Statistics of the economically active population, employment,unemployment and underemployment serve a large variety ofpurposes. … Such information is essential for macro-economicand human resources development planning and policyformulation. When collected at different points in time, thedata provide the basis for monitoring current trends andchanges in the labour market and employment situation,which may be analysed in connection with other economicand social phenomena so as to evaluate macro-economicpolicies.”

- Measurement of Employment, Unemployment andUnderemployment – Current international standards and issues intheir application, Ralf Hussmanns, (ILO Bureau of Statistics), 2007

gallup world poll ilostat - aapor

Documents