a tool for a snapshot assessment of a national statistical system quality conference – 3-5 june...

A tool for a snapshot assessment of a national statistical system

Quality conference – 3-5 June 2014Vienna

Claudia Junker, Head of Unit "Statistical Cooperation", Eurostat

1

Content of the Presentation

1. Background

2. Objectives

3. Presentation of the tool

4. Filling in the tool

5. Using the tool

6. Conclusions

2

Background

Original context is development cooperation, but the tool can be used also in a wider context

Results-based management and performance assessments are central in development

The monitoring of development policies and of aid delivery (and Budget support in particular) require the setting of benchmarks and the construction of systems of indicators

These benchmarks and systems of indicators are fed with public statistics which are provided by the countries themselves

Countries need baseline and benchmarks for assistance projects and their own development

=> What do we know about the quality of these statistics and how can we assess it in a user-friendly way?

3

Objectives:To design a user-friendly tool to help different stakeholders to assess: the quality of the data donors receive from the partner countries to monitor

the results achieved by the national policies and reform programmes, the performance of the support given by the EC to these policies and reform

programmes, The success of assistance projects visible in corresponding official statistical

data The state-of-play in the development of the NSS in a country, The efforts to improve the performance of the NSI/NSS

=> A natural vocation for supporting quality

4

Eurostat's commitment to quality statistics

European Statistics Code of Practice Regular self-assessments by the EU member states Peer reviews and Global assessments with partner countries (East and

South) Support to the African Charter on Statistics and similar regional initiatives

=> A natural vocation for supporting quality

5

Presentation of the tool

The tool consists of two modules:

Module 1: analyse and document the functioning of the national statistical system (NSS) as a whole, with a special focus on institutional, legal and organisational aspects.

Module 2: analyse and document the statistics in specific sectors. A wide range of sectors can be addressed (such as: education, health, agriculture, food security, private sector development, climate change and environment, economic growth, governance, infrastructure, external trade of goods, employment, living conditions, macro-economic stability, population).

6

Presentation of the toolModule 1: Overall National Statistical System (56 questions)

7

Themes Content Number of questions

Mainstreaming

Legal, institutional and strategic foundations (i.e. legal and institutional framework for the whole NSS, , integration and consistency of the statistical development strategy in the national development strategic framework)Organisation and coordination of the NSS

13

Organization

14

Resources

Adequacy of resources for statistics in the NSS (i.e. personnel in adequate quantity, quality of the personnel involved in statistics, equipment and infrastructure, financing)

12

Quality

Determinants of data quality (i.e. quality commitment, professional independence, impartiality, objectivity, methodology and appropriate statistical procedures)

13

Users Relations with users (i.e. relevance, accessibility) 4TOTAL 56

Presentation of the toolModule 1: National Statistical System (56 questions)

Examples of questions: Are statistics used in the development, implementation and monitoring of national development

policies? Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics

development? Is statistical independence a guiding principle for the functioning of the NSS? Does the NSS have mechanisms for assigning of responsibilities in the national statistical process? Is there a human resources policy (recruitment, career development, education and training) for the

NSI? Which proportion of the staff has a (permanent) internet connection? Are there internal handbooks / guidelines / recommendations for the statistical production processes? Are there formal processes in place to consult users about their statistical needs?

8

Presentation of the toolSector matrix

9

Development sectors / Issues

Dimensions Corresponding

statistical sectors

Statistics producers

(NSS) Sources

Main related indicators

Health

Health statistics Ministry of Health DHS, MICS Administrative data from health structures

Child mortality Maternal mortality Main endemic diseases Budget share Other WHO and UNICEF indicators

Education

Education statistics Ministry of Education (Primary, Secondary, Tertiary)

Administrative data from education structures Population data

Literacy rates Enrolment rates Budget share Other UNESCO indicators

Agriculture

Agriculture (economic accounts, price, structures, production)

Ministry of Agriculture, Ministry of Rural Development, Ministry of environment

Agriculture census and surveys Farm register

Production Costs Prices Other FAO indicators

Employment

Offer and demand of work (balance between opportunities and capacities)

Labour market statistics Employment statistics

NSI Employment agency, Social security agency Ministry of Labor Ministry of Budget

Population census Household survey (Labour Force Survey) Administrative. data Public employment

Active population (by gender) Employment rates Unemployment Other ILO indicators

External trade of goods

External trade statistics

NSI Customs Partner countries

Customs records Imports and exports (quantities and values) Terms of trade indexes


Module 2: Sector statistics (between 59 and 61 questions)

10

Themes ContentOne data producer

Several data producers

Part 1 Sector information system assessment

Framework

Legal, institutional and strategic foundations (i.e. legal and institutional framework of the concerned sector information system, integration and consistency with the strategic framework supporting sector statistics development)

7 8

Resources

Adequacy of resources for statistics within the sectorinformation system (i.e. personnel in adequate quantity,quality of the personnel involved in statistics, equipmentand infrastructure, financing)

12 9

QualityDeterminants of data quality (i.e. quality commitment,professional independence, impartiality, objectivity,methodology and appropriate statistical procedures)

22 22

Users Relations with users (i.e. relevance, accessibility) 11 11

Part 2 Assessment of key indicators

IndicatorsSet of key criteria to roughly assess the quality of the indicator

9 9

Number of questions


Module 2: Sector statistics

Examples of questions: Is the development of sector statistics included in the Sector Development Strategies? Which proportion of the staff involved in producing sector statistics is equipped with a modern

computer? Is information on the methods and procedures used for the sector statistics published on the

NSI or Ministry website? Do all users get access to statistical releases at the same time? Are there partnerships or joint studies/works with the scientific community? Are their quality controls and checks on the reporting made by the concerned administrative

units (schools, health centres …)? How much of the statistical outputs in the sector are available via the internet? Is there a contact point to assist users?

11


12

Far from complying with quality requirements

Not totally complying with quality requirements

Complying with quality requirements

Easy colour coding:

Objective criteria – Column in the matrix with management comments for filling in the tables


13

Additional coding:

Recent progress

No changes

Deterioration

More subjective elements


Detailed questions

Potential sources

SituationColor

codingRemarks

Perspective

coding

Identified sources

Initial assessment

Management of the table

Are statistics used in the development, implementation and monitoring of national development policies?

Some evidences found in the main policy papers. However, the reference to official statistics is not systematic and doesn't seem to be articulated around a clear diagnostic and a comprehensive data framework.

Partially ↑ ↑

Was statistical development addressed as part of these development policies?

No evidences found in the main policy papers. No additional information from the field mission

No

Are there statements made by high-level officials (president, ministers, parliament, etc.) on statistics development?

Several statements and articles mentionned during the field mission by the Vice Prime Minister and other high ranking officers from the government

Yes, but occasionaly

Statistics are included in the Government web site but under "Information".

→ →Look in official documents, newspapers and other media for evidence of interest of decision makers in statistics.

Is the Statistical Day/Week celebrated?

Yes, statistical week in October 2010. A leaflet on the SDNSS has been prepared to this occasion.

Not regularly

No plan for further celebrations mentioned during the field mission. Additional clarification to put through the LSB or to find in the WB appraisal report

↓ ↓Regularly' means that the statistical day is celebrated each year or at least every two years.

During the latest celebrations, was the government represented?

The Vice Prime Minister opened and closed the ceremonies. This level is considered high in the lao context.

President or Prime Minister represented

It is not clear if other high level official participated in other events than the openning and closing sessions

→ →Yes' doesn't automatically mean green. Consideration must be given to the authority level of the Government representative. President or Prime Minister means green.

What is the degree of openess vis-a-vis external partners-donors or peers on statistical issues?

Assessment of the capacity to produce core economic indicators - Study made in July 2011 joiintly by the Lao Statistics bureau and the statistical Division of ESCAP (draft version)

Updated GDDS or recent review/assessment

Assessment by ESCAP July 2011. The obstacle to the adhesion to the GDDS must be looked for at the level of the line Ministries and not of the LSB

↑ ↑

Find reviews, studies, assessments of the statistical system that have been made with outside partners or relations established with international systems for assessing statistical data quality (GDDS/SDDS and peer reviews or global/sector assessments mainly).

LSB web-site: www.nsc.gov.laModule 1

Q1 : Statistical development mainstreamed as part of the national development policy

Pri

ori

ty

Statistics should be an integral part of the national development policy, including for the design, monitoring and evaluation of Poverty Reduction Strategies, sector strategies, and other national development plans, as well as assessing progress made towards the MDGs. Most of the policies are linked to the achievements of the MDGS.

Oth

er

Official statements, Government decisions, National Budget, NSDS, National policy documents (SRPC), Reports in the media

Discussion with the LSB Deputy DG on November 13, 2012

Theme 1: Country strategy for statistics

Socio-economic Development Strategy 2020, 7th National Socio-economic Development Plan NSEDP 2011-2015, SDNSS 2010-2020

Look for evidence of whether statistics (i) have been used to design key country policy documents (diagnostics of the situation at macro and sector levels, indicators of achievement) and (ii) are addressed in these documents (attention given to the development and improvement of statistics systems at macro and sector levels). In some cases, the strategic matrix doesn't build on quantitative indicators. It may be necessary to go back to the preparatory studies.

14

Presentation of the toolPart 1 - Assessment at Sector/Ministry level EDUCATION

Detailed questions

Potential sources

SituationColor

codingRemarks

Perspective

coding

Identified sources

Initial assessment

Management of the table

Is there a law or an official agreement that organizes efficiently the production of sector statistics?

No specific law. But there is a practice withinthe Minsitry of Education that has been established during the last 20 years and that involves the schools, the districts and the provinces. There are also regular relations with the LSB (training, consultation on statistical operations)

The law addresses the rights and responsibilities of the line ministries for the production and the dissemination of statistics. A decree from the Ministry of Education established the Information Centre (role and responsibility) in 2006. There will be a new decree soon as ICT and statistics will be split in two different bodies.

→ Statistical law →

In most case, there will not be a specific law at the sector level. It is generally expected that the relations between the NSI and the other data producers in a country are driven by memorandum of understanding that sets the respective responsibilities, including data sets to be exchanged and calendar.

Does the National Statistical Office or other body responsible for statistics in the sector have access to the relevant administrative data?

Yes. No mandate is specifically written, but in practice, data are collected regularly from schools and villages.

Yes partialy

The system is managed by the Education Statistics and Information Technology Center of Education and Sport (SITCES). The system is improving year after year.

↑Interview with Deputy DG SITCES

↑

Oth

er

Is the actual work organization (mandate and attribution) in line with the law or the official agreement?

The LSB is responsible for the coherence of official statistics mainly through its responsibilities in the management and direction of the NSS. The Ministry of Education gather annual data from schools and villages.

YesSee Article 32 of the statistical law. → Statistical law →

To check if mechanisms (communication, dialogue, exchange of data …) that ensure coherence of methods and outputs between the different structures (NSI, sector Ministry, ) involved in the production of sector statistics

Module 2

Theme 1: Legal, institutional and strategic framework at the level of the sector (both the NSI and the sector Ministry)

Q1: Legal and institutional framework supporting the production of the sector statistics

Statistical laws, Government decisions, Memoranda of Understanding, Service Level Agreements, Questionnaire to DG

Pri

ori

ty

15


Part 2. Quality assessment – at the level of Indicators

INDICATOR 1: MinJus - Number of judges

Detailed questions Potential sources SituationColour coding

RemarksPerspective

codingIdentified sources

Initial assessm

entManagement of the table

Are there more than one national indicator on this issue?

No. Data produced by the MinJus

No

Is the definition used in line with the international recommendations?

Questionnaire to the statistical service in charge, Compliance report

National methodologyNo

Is the methods used in line with the international recommendations from?

No

If the methodology has changed in the last years, are the changes documented and old estimates recalculated according to the new methodology?

or No change since 2009

No information available

Methodology changes are not a problem as far as the changes are documented, properly explained to the users, the revised methodology is applied to old series/data.

How long is the time series for this indicator

Since 2009

LSMP 2009-2020 Check if it is a one shot production or a sustainable one well integrated in the national system.

Are the sources used and their coverage in line with the international recommendations?

Vertical informations system

No

Is the data frequency in line with the international recommendations?

Survey data (Demographic and Health Survey – DHS, Multiple Indicator Cluster Survey – MICS, Reproductive and Health survey - RHS)

Reports from Provinces on a quarterly basis

Is the indicator timely enough for the use made by the EC?

Data available for the annual Round Tables

Yes

Is the indicator publicly available and easy to find?

Certainly available for the Round Tables but not directly to the Public

No

16

Filling in the tool

What are the basic sources?

Aspects linked to the legal and policy framework, to the organization and functioning of the system

Country macro-economic and social plan/strategy (Growth and poverty alleviation strategy or any similar policy document)

Generally available on the Government web site

National Strategy for the development of statistics (NSDS)

Information documented and regularly updated on the Paris 21 web site. Also generally available on the NSI web site

The statistical law and related application decrees

Generally available on the NSI web site

Other information on the NSS (coordination, exchanges among the data producers/users)

Sector strategy documents/plans Web sites of the related sector Ministries

Information on the Sector Information system

For Module 2 (mainly theme 1)

For Module 1 (mainly themes 1 and 2)

17

Filling in the tool

What are the basic sources?

Aspects linked to practices: complements to the basic sources

Reports by the NSS (annual reports/plans, self assessments, studies, guides and methods…)

Not always available on the web (paper documents)

Reports by other donors Not always easy to identify and to gather particularly when there is no coordination mechanism for statistics

Contacts/discussions with the actors of the NSS

Direct knowledgeable source but sometimes

For all modules (mainly themes 3, 4 and 5 for Module 1 and themes 2, 3 and 4 for Module 2)

18

Filling in the tool

Who will do it?

Staff from the EC Delegation or other donors alone

Even if only part of the question are answered, the tool can give useful indications, particularly if the objective is to support statistical improvement in areas most in needs

Experts statistician mobilized by the EC Delegation or other donors

This could help for the first exercise, repetition over the years being done then by the Delegation

EC Delegation with other donors

Filling in the tool could be the result of a coordination and information sharing exercise with the donors most interested and involved in the sector

EC Delegation or other donors and National Statistical System (NSS)

This would be the best way to initiate and feed a regular and articulated dialogue on the strengths and weaknesses of the NSS and to potentially open the way to concrete support

EC Delegation, other donors and NSS

Same as the above but involving all the interested donors. In some countries coordination for statistics already exist and could be fed with the results form the tool

NSS alone

This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.

19

In the framework of development cooperation In the framework of the NSS NSS alone

This would be the result of a full appropriation of the tool by the NSS. Certainly the best solution in the long run.

Experts statistician mobilized by the NSS

This could help for the first exercise, repetition over the years being done then by the NSS

Using the tool

Results Module 1: example

Theme 1: Stra

tegy for s

tatistics

Total 1

Theme 2: Organisa

tion of the N

SS

Total 2

Theme 3: Reso

urces

Total 3

Theme 4: Quality

determinants

Total 4

Theme 5: Relation w

ith use

rs

Total 50

1

2

3

4

5

6

7

8

Module 1 - Global results

20

Using the toolResults Module 1: example

Thème 1: Stra

tegy for s

tatistics

Total 1

Thème 2: Organisa

tion of the N

SS

Total 2

Thème 3: Reso

urces

Total 3

Thème 4: Quality

determinants

Total 4

Thème 5: Relation w

ith use

rs

Total 50

1

2

3

4

5

6

7

8

Module 1 - Global results No rigor in quality controls

Poor relevance of and accessibility to data

Strategy exists but not fully mainstreamed in policies

Legal framework in place but poor coordination

Gap in resources allocated to statistics

21

Using the tool


Theme 1: Stra

tegy for s

tatistics

Q1: Mainstr

eamed statisti

cal d

evelopment

Q2: Status o

f statisti

cal s

trategy

Theme 2: Organisa

tion of the N

SS

Q1: Legal fr

amework

Q2: Organisa

tion and coordination

Theme 3: Reso

urces

Q1: Human re

source

s

Q2: Equipment

Q3: Financin

g

Theme 4: Quality

determinants

Q1: Quality

Commitment

Q2: Imparti

ality

Q3: Objecti

vity

Q4: Methodology r

esearch

Theme 5: Relation w

ith use

rs

Q1: Releva

nce

Q2: Acce

ssibilit

y

Module 1 - Detailed results

22

A snapshot of an NSS – using the tool


Theme 1: Stra

tegy for s

tatistics

Q1: Mainstr

eamed statisti

cal d

evelopment

Q2: Status o

f statisti

cal s

trategy

Theme 2: Organisa

tion of the N

SS

Q1: Legal fr

amework

Q2: Organisa

tion and coordination

Theme 3: Reso

urces

Q1: Human re

source

s

Q2: Equipment

Q3: Financin

g

Theme 4: Quality

determinants

Q1: Quality

Commitment

Q2: Imparti

ality

Q3: Objecti

vity

Q4: Methodology r

esearch

Theme 5: Relation w

ith use

rs

Q1: Releva

nce

Q2: Acce

ssibilit

y

Module 1 - Detailed results

GDDS not joined, only a few statements by high-level officials on statistics

Participatory process for the design of the strategy but not translated into wok plans

The law is very comprehensive but the notion of confidentiality is not yet fully internalised

Share of responsibilities and coordination is established but poor execution in practice

Lack of trained and experienced personnel

Number of PC ok but poor internet access

Limited information available on financing

Sectors with no handbooks and no systematic quality controls

No rules for the correction of serious errors

No release calendar

No cooperation with academicsNo systematic consultation with the users

Poor usability of the web site for data dissemination

23

Using the tool

Results Module 2: specific indicators

22%

44%

33%

Number of judges

24

Using the tool

Results Module 2: specific indicators

22%

44%

33%

Number of judgesDefinitions and sources not in line with international recommendations

Only one producer and availability on time

Only short term series and data not easy to find for users

25

Using the tool

An instrument of statistical quality assessment for the EU Delegations

The tables could be filled and updated regularly as a monitoring instrument for the EU Delegations to assess the development of the statistical system of partner countries

26

Using the tool

An instrument of dialogue on statistics between donors (EU or others) and the partner countries

The tool could be embedded in annual reviews or monitoring exercises that could be done in the framework of a political dialogue between the donors and the country authorities,

The tool could ease identifying statistical areas in need to be supported by the donor community and help defining assistance programmes

27

Using the tool

An instrument of assessment of the progress made in the NSS

Filling the tables could be the result of a coordination work within the NSS. The results could be presented and discussed each year (regularly) at a meeting of the National Statistical Council or a similar entity. This would help adjusting work plans.

Examples and good practices could be drawn from exercises carried out in other countries.

28

Conclusions

What the Snapshot is:

In photography: a photograph that is "shot" spontaneously and quickly.The tool is a complex camera (comprehensive and articulated framework) that allows shooting spontaneously (the tool is available and can be used/understood by non-statisticians) and quickly (the tables can be filled in rapidly with available information)

In computing: the capture of an object's state at a specific moment in timeThe tool gives a picture of the statistical system of a country (or of sector statistics in a country) at the moment when the related tables are filled in. This picture can be taken at regular intervals (e.g. yearly) in order to monitor the progress made

29

Conclusions

What the Snapshot is not:

An instrument for rating and/or comparing countries The tool is “process-oriented” and aims at supporting a dialogue among the various actors involved

in the development of statistics in a country

A detailed analysis of the sector statistics There are more sophisticated and targeted frameworks for some statistical sectors (such as

Education/UNESCO) that detail the statistical processes leading to data production in more depth. The tool can be fed from these in-depth analyses and can also be an initial step before going deeper into the analysis.

30

Conclusions

Freely available from Eurostat upon request to send at: [email protected]

Available in three languages: French, English and Spanish Manual and a user-guide E-learning training course on how/why to use it

31

mailto:[email protected]

a tool for a snapshot assessment of a national statistical system quality conference – 3-5 june...

Documents