guidelines for the template for a generic national quality ...the template for a generic national...

Statistical Commission Background document Forty-third session Available in English only 28 February – 2 March 2012 Item 3 (j) of the provisional agenda National quality assurance frameworks

Guidelines for the Template for a Generic National Quality Assurance Framework (NQAF)

Prepared by the Expert Group on National Quality Assurance Frameworks

8 February 2012

GUIDELINES FOR THE TEMPLATE FOR

A GENERIC NATIONAL QUALITY ASSURANCE

FRAMEWORK (NQAF)

(PREPARED BY THE EXPERT GROUP ON NQAF)

CONTENTS I. Introduction ..........................................................................................................................................................1 II. Use of the Generic National Quality Assurance Framework (NQAF) Template............................................1 III. Connection with other quality frameworks .......................................................................................................2 IV. Concepts and terminology ...................................................................................................................................2 V. Content and format of the Guidelines ................................................................................................................2 VI. Acknowledgements...............................................................................................................................................3

TEMPLATE FOR A GENERIC NATIONAL QUALITY ASSURANCE F RAMEWORK (NQAF) .................4

GUIDELINES

1. QUALITY CONTEXT...........................................................................................................................................5 1A. CIRCUMSTANCES AND KEY ISSUES DRIVING THE NEED FOR QUALITY MANAGEMENT ...............................................................5 1B. BENEFITS AND CHALLENGES..........................................................................................................................................................5 1C. RELATIONSHIP TO OTHER STATISTICAL AGENCY POLICIES, STRATEGIES AND FRAMEWORKS AND EVOLUTION OVER TIME...7

2. QUALITY CONCEPTS AND FRAMEWORKS ................................................................................................8 2A. CONCEPTS AND TERMINOLOGY.....................................................................................................................................................8 2B. MAPPING – OF NQAF’S SECTION 3 - TO EXISTING FRAMEWORKS...............................................................................................8

3. QUALITY ASSURANCE GUIDELINES ............................................................................................................8 3A. MANAGING THE STATISTICAL SYSTEM.........................................................................................................................................9

NQAF 1: Coordinating the national statistical system.....................................................................................9 NQAF 2: Managing relationships with data users and data providers..........................................................11 NQAF 3: Managing statistical standards .......................................................................................................14

3B. MANAGING THE INSTITUTIONAL ENVIRONMENT.......................................................................................................................16 NQAF 4: Assuring professional independence...............................................................................................16 NQAF 5: Assuring impartiality and objectivity ..............................................................................................18 NQAF 6: Assuring transparency ....................................................................................................................20 NQAF 7: Assuring statistical confidentiality and security .............................................................................21 NQAF 8: Assuring the quality commitment ....................................................................................................23 NQAF 9: Assuring adequacy of resources......................................................................................................25

3C. MANAGING STATISTICAL PROCESSES.........................................................................................................................................26 NQAF 10: Assuring methodological soundness ...............................................................................................26 NQAF 11: Assuring cost-effectiveness..............................................................................................................29 NQAF 12: Assuring soundness of implementation ...........................................................................................31 NQAF 13: Managing the respondent burden ...................................................................................................33

3D. MANAGING STATISTICAL OUTPUTS............................................................................................................................................36 NQAF 14: Assuring relevance..........................................................................................................................36 NQAF 15: Assuring accuracy and reliability ...................................................................................................38 NQAF 16: Assuring timeliness and punctuality................................................................................................40 NQAF 17: Assuring accessibility and clarity ...................................................................................................42 NQAF 18: Assuring coherence and comparability...........................................................................................45 NQAF 19: Managing metadata ........................................................................................................................47

4. QUALITY ASSESSMENT AND REPORTING ................................................................................................48 4A. MEASURING PRODUCT AND PROCESS QUALITY - USE OF QUALITY INDICATORS, QUALITY TARGETS AND PROCESS

VARIABLES AND DESCRIPTIONS..........................................................................................................................................................48 4B. COMMUNICATING ABOUT QUALITY - QUALITY REPORTS...........................................................................................................50 4C. OBTAINING FEEDBACK FROM USERS...........................................................................................................................................50 4D. CONDUCTING ASSESSMENTS; LABELLING AND CERTIFICATION ................................................................................................51 4E. ASSURING CONTINUOUS QUALITY IMPROVEMENT.....................................................................................................................52

5. QUALITY AND OTHER MANAGEMENT FRAMEWORKS......... ...............................................................54 5A. PERFORMANCE MANAGEMENT....................................................................................................................................................54 5B. RESOURCE MANAGEMENT...........................................................................................................................................................54 5C. ETHICAL STANDARDS...................................................................................................................................................................55 5D. CONTINUOUS IMPROVEMENT.......................................................................................................................................................56 5E. GOVERNANCE...............................................................................................................................................................................56

ANNEXES

ANNEX 1 – MAPPING – OF NQAF’S SECTION 3 - TO EXISTING FRAMEWORKS.................................58 Part 1: Correspondence between the Generic National Quality Assurance Framework Template and the CoP, DQAF, LAC proposal and StatCan..........................................................................................................................................................................58 Part 2: Correspondence between the Generic National Quality Assurance Framework Template and the European Statistics Code of Practice (CoP).......................................................................................................................................................................60 Part 3: Correspondence between the Generic National Quality Assurance Framework Template and the International Monetary Fund’s Data Quality Assessment Framework (DQAF).................................................................................................65 Part 4: Correspondence between the Generic National Quality Assurance Framework Template and the Latin America and the Caribbean Regional Code of Good Statistical Practice (LAC proposal)....................................................................67 Part 5: Correspondence between the Generic National Quality Assurance Framework Template and the Statistics Canada Quality Assurance Framework (StatCan).......................................................................................................................72

ANNEX 2 – SELECTED REFERENCES FOR EACH NQAF LINE .................................................................73

ANNEX 3 – EUROPEAN STATISTICAL SYSTEM'S QUALITY AND PERFORMANCE INDICATORS 93

1

I. Introduction

A. Background

The Template for a Generic National Quality Assurance Framework (NQAF) and this document,

Guidelines for the Template for a Generic National Quality Assurance Framework, have been prepared in

response to decision 41/101 by the United Nations Statistical Commission in 2010, by which it was

recommended that an expert group be established to, inter alia, develop a template for a generic national

quality assurance framework, with accompanying guidelines, to assist countries that may wish to

formulate and operationalize national quality frameworks of their own, or further enhance existing ones.

During the development of the template for a generic NQAF, the expert group was cognizant of the need

to ensure that the template would be – as clearly stipulated by the Statistical Commission - sufficiently

flexible to permit national circumstances to be taken into consideration by countries that chose to apply

the framework template. As noted in the Statistics Canada report to the Statistical Commission

(E/CN.3/2010/2), the development of a generic, one-size-fits-all framework would not be practicable.

Therefore, a generic national quality assurance framework template, accompanied by guidelines to

provide examples and guidance on the possible ways in which an organizing framework could be

formulated and operationalized, was recognized as being more realistic, and therefore was the focus of

the expert group.

II. Use of the Generic National Quality Assurance Framework (NQAF) Template

It should be understood that the application or implementation of the NQAF Template is intended to be

voluntary; the Template is not meant to be prescriptive or viewed as a recommended replacement for

other quality frameworks already adopted or in use by a country’s national statistical office. The NQAF

Template is intended to be a tool to provide the general structure within which individual country-specific

national quality assurance frameworks can be developed by countries that choose to do so. Needless to

say, the components of the Template that may be most applicable to one country might be quite different

for another country, depending upon aspects such as its stage of development, available resources, the

institutional environment within which it operates, and its current most pressing concerns from a quality

perspective.

Therefore, national statistical offices are not expected to strictly apply all of the components of the NQAF

Template, nor is it assumed that the selected elements and mechanisms set out in these Guidelines are

exhaustive or could relate uniformly to all national statistical offices in the same ways. By using the

recommended structure as a starting point, statistical offices are encouraged to determine themselves

which aspects are appropriate to their specific situations, and modify, add to or subtract from the

proposed components to build a framework to fit their needs. A framework created by the national agency

for the national agency will be the most useful type.

2

III. Connection with other quality frameworks

In addition to the prerequisite of flexibility, the expert group took carefully into account the

recommendation by the Statistical Commission that it make use of and align the NQAF Template with the

existing quality frameworks, so as not to “re-create the wheel”. In the design of the NQAF Template, the

European Statistics Code of Practice (CoP), the International Monetary Fund’s Data Quality Assessment

Framework (DQAF), Statistics Canada's quality assurance framework as presented in its 2010 report to

the UN Statistical Commission, and the Proposal for the Structure of a Regional Code of Good Statistical

Practice for Latin America and the Caribbean were used and, to the extent possible, correspondence to

them was adhered to. The mapping of the NQAF Template to each of the above-mentioned frameworks,

which was done by the expert group, will provide assistance to national statistical offices in harmonizing

their quality assurance frameworks with those of regional or international organizations. The detailed

mapping is included in Annex 1 of this document.

IV. Concepts and terminology

While there are several general definitions of quality, one of the most commonly used and succinct

definitions is fitness for use or fitness for purpose. The ISO 9000 Quality Management System’s

definition, cited in the SDMX Metadata Common Vocabulary and in the NQAF expert group’s Glossary, is

the degree to which a set of inherent characteristics fulfils requirements. Over the past twenty years or

so, statistical agencies have arrived at a consensus that the concept of quality of statistical information is

multi-dimensional and that there is no one single measure of data quality. Examples of the common

quality dimensions or components include: relevance; accuracy; reliability; timeliness; punctuality;

accessibility; clarity, interpretability; coherence; comparability; credibility; integrity; methodological

soundness; and serviceability. The dimensions of quality are overlapping and interrelated and, therefore,

the adequate management of each of them is essential if information is to be fit for use. (For information

on other quality-related terms, see the NQAF Glossary at

http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx).

For the purposes of this document, the term statistical agency is used interchangeably with statistical

authority, statistical organization, and statistical office. Throughout these Guidelines, the terms “NQAF

lines” and “elements” are used. Those “NQAF lines” preceded by numbers in brackets in the Template

basically correspond to the “principles” in the CoP and the “elements” in the DQAF. The Guidelines’

“elements” are roughly the same as the “indicators” used to reflect good practice in other frameworks.

V. Content and format of the Guidelines

The one-page NQAF Template to which these Guidelines refer is presented on page 4. The Guidelines

contain brief explanatory texts of the components of sections 1, 2, 4 and 5 of the Template, while more

detailed content is provided for the Template’s section 3, the numbered “NQAF lines”, 1-19. For each

3

numbered NQAF line, a description, followed by elements to be assured, by level (e.g. the national

statistical system or agency level) and by stages of the statistical production process (e.g. programme

design, programme implementation or post-collection evaluation stage) and supporting mechanisms are

included. It should be noted that oftentimes the same elements and mechanisms are repeated across a

number of different NQAF lines. This has been done intentionally to underscore the multi-dimensional

aspect of quality and to allow readers to use parts of the framework independently.

When reviewing or designing a database, output or specific statistics, going through the set of questions

posed in the elements’ lists should be helpful to providers in designing a statistical collection or product

that is fit for purpose, and help other readers in making informed decisions about whether the statistics

produced are fit for use, or are of an acceptable level of quality for their purposes. In order to provide

more information and to help the reader zero in on additional guidance on the individual NQAF lines, a list

of selected references and nationally and internationally developed tools is included for NQAF lines 1-19

in Annex 2.

VI. Acknowledgements

The contributions to the work on the NQAF Template and Guidelines by the expert group members of the

following countries are gratefully acknowledged: Canada, China, Colombia, Egypt, France, Indonesia,

Italy, Japan, Mexico, Niger, Norway, South Africa, Switzerland and Ukraine. Valuable contributions were

also made by the expert group’s observers from the United Nations Economic and Social Commission for

Asia and the Pacific, the United Nations Economic Commission for Europe, Eurostat, the International

Monetary Fund, the World Bank and the United Nations Statistics Division.

4

Template for a Generic National Quality Assurance F ramework (NQAF) (Developed by the Expert Group on NQAF)

1. Quality context

1a. Circumstances and key issues driving the need for quality management 1b. Benefits and challenges 1c. Relationship to other statistical agency policies, strategies and frameworks and evolution over time

2. Quality concepts and frameworks 2a. Concepts and terminology 2b. Mapping to existing frameworks

3. Quality assurance guidelines

3a. Managing the statistical system [NQAF 1] Coordinating the national statistical system [NQAF 2] Managing relationships with data users and data providers [NQAF 3] Managing statistical standards

3b. Managing the institutional environment

[NQAF 4] Assuring professional independence [NQAF 5] Assuring impartiality and objectivity [NQAF 6] Assuring transparency [NQAF 7] Assuring statistical confidentiality and security [NQAF 8] Assuring the quality commitment [NQAF 9] Assuring adequacy of resources

3c. Managing statistical processes

[NQAF 10] Assuring methodological soundness [NQAF 11] Assuring cost-effectiveness [NQAF 12] Assuring soundness of implementation [NQAF 13] Managing the respondent burden

3d. Managing statistical outputs

[NQAF14] Assuring relevance [NQAF15] Assuring accuracy and reliability [NQAF16] Assuring timeliness and punctuality [NQAF17] Assuring accessibility and clarity [NQAF18] Assuring coherence and comparability [NQAF19] Managing metadata

4. Quality assessment and reporting

4a. Measuring product and process quality - use of quality indicators, quality targets and process variables and descriptions

4b. Communicating about quality – quality reports 4c. Obtaining feedback from users 4d. Conducting assessments; labelling and certification 4e. Assuring continuous quality improvement

5. Quality and other management frameworks 5a. Performance management 5b. Resource management 5c. Ethical standards 5d. Continuous improvement 5e. Governance

1. Quality context 5

1. Quality context

1a. Circumstances and key issues driving the need f or quality management

Every organization needs to have a quality management system (or equivalent) in place to ensure quality

in processes and outputs as well as in institutional aspects. Various types of general and internationally-

developed quality management approaches, systems, models and frameworks exist1, and some national

statistical offices may apply one or several of these approaches, in full or in part, for different purposes, or

base their own systems on elements from them while making further adaptations according to their

specific national circumstances. Other national statistical offices, which may in fact already be involved in

a comprehensive range of quality initiatives and activities, may nevertheless lack an overarching

framework to organize, give them context and show how they relate to the various quality tools.

In the context of a national statistical office, systematic quality management typically takes the form of a

quality assurance framework. Several issues have historically underscored the need for national

statistical offices to systematically adopt quality management measures sooner rather than later and

translate them into a formalized quality assurance framework.

In some cases, published errors that caused embarrassment and potential damage to the credibility of the

statistical office and its outputs may have been the catalyst, whereas in other cases, large increases or

decreases in resources was the impetus for the shift by the statistical office towards managing quality in a

more formalized and systematic way. Similarly, government-wide reform initiatives, changes in

management, the restructuring of the national statistical office, or the need to comply with legislation or

regulations are examples of other driving forces leading to a national statistical office’s decision to embark

upon the formulation of a quality assurance framework.

1b. Benefits and challenges

The report by Statistics Canada to the Statistical Commission in 2010 (E/CN.3/2010/2) set the stage for

the expert group to take up the task of developing a generic national quality assurance framework

template that would provide insight into what can be included in a national quality assurance framework.

The report advocated that all national statistical organizations should have a national quality assurance

framework in place – or consider developing one if they have not yet done so. The objective would be to

have in place an overarching framework that would provide context for quality concerns, activities and

initiatives, and explain the relationships between the various quality procedures and tools. Such an

organizing framework has proved to be very useful in providing a single place to record and reference the

1 For example, Total Quality Management; Six Sigma; Balanced Scorecard; the European Foundation for Quality Management Excellence Model; Common

Assessment Framework; International Standardisation Organisation standards; the Object-oriented Quality Management Model, the European Statistics Code of Practice, the African Charter for Statistics; the Data Quality Assessment Framework of the International Monetary Fund; the Quality Framework for Statistical Activities of the Organisation for Economic Co-operation and Development and the Generic Statistical Business Process Model.


full range of current quality concepts, policies and practices, and is forward looking because it takes into

account future actions and activities. The main benefits of having a quality assurance framework in place are:

• It provides a systematic mechanism for facilitating the ongoing identification of quality problems and

possible actions for their resolution. At the same time, it serves to stimulate and maximize the

interaction among staff throughout the organization;

• It gives greater transparency to the processes by which quality is assured and reinforces the image of

the office as a credible provider of good quality statistics;

• It provides a basis for creating and maintaining a quality culture within the organization and contains

reference material that can be helpful for training;

• It supports quality improvements and their maintenance over time;

• It is a mechanism for the exchange of ideas on quality management with other producers of statistics

within the national statistical system and with other national and international statistical organizations.

The process of developing a national quality assurance framework is typically best carried out by a task force

within the national statistical office or the coordinating body, made up of an experienced team of staff from a

variety of areas, for example: programme planning; survey design; survey operations; dissemination;

infrastructure development; and support. The framework development process has intrinsic benefits of its own

since it obliges staff from various disciplines to come together to confront and tackle quality issues and think

through the requirements, agree upon priorities and evaluate the costs and benefits while keeping in mind that

not everything can or should be undertaken.

The process of going through and completing each of the applicable sections of the quality assurance

framework template has benefits as well. As a result of implementing the framework, it can be expected

that the staff will have: become more aware of the various quality concepts, instruments, policies and best

practices and improved the related documentation; gained insights by having carried out or taken part in a

systematic quality assessment (for example a self-assessment, peer review and/or quality audit);

identified the various potential quality problems and formulated improvement actions and priorities;

recognized where a need for additional resources and/or training exists; and established a means for

comparison of the level of quality over time.

Along with the formulation and implementation of a quality assurance framework come significant

challenges. One of the first challenges often encountered - arriving at a common understanding of what

quality actually is – highlights the shift from the typical “old” notion of quality as being synonymous with

accuracy to the more current notion, one in which quality, when referring in particular to statistical outputs,

encompasses many other dimensions such as relevance, timeliness, punctuality, accessibility, clarity,

coherence, comparability, etc.


In countries in which there are multiple producers of statistical information, effective coordination and

communication among all members of the national statistical system is necessary in order to agree on a

common framework and on the commitment towards the harmonization of information, standards, and

other aspects of statistical information production.

Repeatedly mentioned by national statistical agency representatives who have been involved in

introducing quality work in their organizations is the fact that senior management support – long term

sustained support – is crucial to the successful implementation of a quality assurance framework. Another

challenge to be faced is that concrete and practical results need to be demonstrated rather quickly in

order to continue to assure continued support. Since quality assurance work – monitoring, documenting,

standardizing and reporting in particular – is time consuming and labour intensive, often generating

payoffs that are not immediately obvious, staff reluctance to accept an increase in their workload with no

corresponding increase in resources to carry out their “regular” responsibilities has to be overcome. In

addition, to be most effective, this work has to be reviewed, maintained and enhanced over time, which

requires, in addition to resources, a long term commitment not only from management and the quality unit

or team (if one exists), but from all staff at all levels. To obtain this commitment, the effective promotion

and communication of the quality assurance features, benefits and requirements is necessary. This can

be accomplished through the sharing of information and training, both of which should be tailored to the

different levels of staff. Quality has to be considered a core value and needs to become embedded in the

culture of the organization.

1c. Relationship to other statistical agency polici es, strategies and frameworks and evolution over

time

A quality assurance framework is clearly just one of a number of other frameworks, policies and strategies

that typically are in place in statistical agencies. They all should be developed and implemented in an

integrated manner to achieve the agency’s mission and vision statements. The formulation of a quality

assurance framework requires an in-depth and thorough review of those mechanisms most directly

related to quality since the framework’s main focus is on the management of the core statistical functions.

Statistical laws, regulations and acts, codes of practice, and statistical standards, policies and strategies2

will need to be explicitly considered, referenced and made readily available in the process of drawing up a

quality framework.

Other frameworks, policies and strategies that may have a somewhat less direct, but still significant

connection to quality assurance, such as the agency’s multi-year and annual plans, which also make

reference to quality issues, need to be taken into account in the development of a quality assurance

2 For example, policies and strategies for the protection of confidential data; for permitting access to microdata for research purposes; for informing users of the

quality of the data; for publicizing data release, revision and dissemination practices; for addressing the misuses of information; for communicating the agency’s commitment to quality in general; for resource mobilization, etc.).


framework. Ultimately the quality framework will be most effective when it has been built into the agency’s

organizational structure in such a way that the quality practices and procedures become integral parts of

the agency’s other frameworks and processes. Moreover, a national quality assurance framework, like

other frameworks in the agency, should not be fixed over time, but needs to evolve to take account of

changes in priorities, processes, environmental factors, sources, risks, opportunities and strategies.

2. Quality concepts and frameworks

2a. Concepts and terminology

The expert group compiled an online glossary of quality-related terms, the main source of which was the

Statistical Data and Metadata Exchange’s (SDMX) Metadata Common Vocabulary that was developed by

a partnership of international organizations. Since the SDMX Metadata Common Vocabulary is an

agreed global standard, preference was given to the definitions it presents in cases where several

definitions were available for the same terms. In addition to the definition of each term, the NQAF

Glossary includes information whenever possible in the “context” field to provide additional explanations

or other useful information that contributes to the understanding of the concepts. The source of the

definition and a hyperlink to it, when available, are also included. The Glossary is available at

http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx.

The terms included in the NQAF Glossary are primarily those that appear in the Template and these

Guidelines, especially those that are mentioned in the NQAF lines, or directly related to them. The

Glossary’s coverage of terms is not exhaustive, nor is it intended to be. It should be noted that the

Glossary will continue to be updated by UNSD when additional new and relevant information is provided

or identified.

2b. Mapping – of NQAF’s section 3 - to existing fra meworks

A summary table showing the correspondence among the NQAF, the European Statistics Code of

Practice (CoP), the International Monetary Fund’s Data Quality Assessment Framework (DQAF),

Statistics Canada's quality assurance framework, and the Proposal for the Structure of a Regional Code

of Good Statistical Practice for Latin America and the Caribbean is included as Annex 2. The complete

mapping showing the correspondence between the NQAF’s section 3 and the other frameworks is

available online at http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx.

3. Quality assurance guidelines

This section contains the individual guidelines for Section 3 of the Template, broken down into

subsections 3a: Managing the statistical system; 3b: Managing the institutional environment; 3c:

Managing statistical processes; and 3d: Managing statistical outputs.

3. Quality assurance guidelines - NQAF 1: Coordinating the national statistical system 9

3a. Managing the statistical system

NQAF 1: Coordinating the national statistical syst em

Description:

Coordination of the work of the members of the national statistical system is essential for improving and maintaining the quality of official statistics produced by the various statistical agencies.

Elements to be assured:

At the national statistical system level

• Does a law or other formal provision establish the national statistical system, specify the members of the system and designate a coordinating body?

• When the national statistical office has a role in coordinating the national statistical system, is the role legislated or de facto?

• Do the objectives of the coordinating body of the national statistical system include: planning, implementing, coordinating, regulating and evaluating the development, production and dissemination of official statistics and ensuring their quality?

• Do mechanisms exist for facilitating cooperation among the members of the national statistical system in order to improve the performance of the system?

• Are mechanisms in place to facilitate the agreement, among the members of the national statistical system, on priorities for the production of statistics?

• Does the coordinating body of the national statistical system set the methodological guidelines for the production of official statistics, and promote the harmonisation of statistical information as well as the avoidance of duplication of work among the members?

• Does the coordinating body of the national statistical system promote the implementation of standards throughout the system?

• Does the coordinating body of the national statistical system promote the sharing of technical knowledge among the members of the system?

• Does the coordinating body of the national statistical system facilitate the identification of good statistical practices among the members and promote their implementation?

• Do guidelines exist for the exchange, among members of the national statistical system, of unit records or other data?

Supporting mechanisms:

The coordination of the national statistical system is likely to be more effectively managed if the following supporting mechanisms are in place:

• A statistical law or other formal provision that establishes the national statistical system and designates a coordinating body.

• Guidelines, methodological manuals and handbooks on recommended practices.

• Regularly held meetings for members of the system to develop statistical standards and guidelines, exchange technical knowledge, identify good statistical practices, etc. (e.g. committees, working groups, etc.).

• Training courses for members of the system to update knowledge on the contents and application of recommended standards, methodologies, etc.

• Processes for identifying and resolving cases of duplication of efforts in the production of statistics.

3. Quality assurance guidelines - NQAF 1: Coordinating the national statistical system 10

• Arrangements for facilitating regular and timely user-producer consultations and dialogues.

• Processes for the standardized evaluation of the quality of statistical outputs.

• Guidelines on quality management of statistics produced by outsourced agencies.

Selected references: Click here to go to Annex 2, Selected references

3. Quality assurance guidelines - NQAF 2: Managing relationships with data users and data providers 11

NQAF 2: Managing relationships with data users and data providers

Description:

The statistical agencies should build and sustain very good relationships with all of their key stakeholders, including users, data providers, funding agencies, senior government officials, relevant community organizations, and the media.

The user, or recipient of statistical information, transforms it into knowledge needed for decision making or research. As the ultimate client of the statistical agency, the user makes the judgment as to whether its data or services are “fit for purpose”. Delivering quality outputs to the client and obtaining quality feedback are processes that need specific relationship management objectives and supporting processes.

Government departments or other organizations, as providers of administrative data, are essential partners in the provision of statistics that meet the test of fitness for purpose. Administrative data can be very useful to create registers and frames, edit or impute survey data, or validate survey data or outputs. Where the administrative data are of sufficient quality and the data available adequately match the concepts being measured, they can be used instead of a direct survey collection. This in turn creates efficiencies by reducing the size of the survey sample required and also reduces respondent burden. The most important strategy for managing a range of risks with the use of administrative data is the maintenance of ongoing relationships with the data custodians of the source data. These relationships should be established to varying degrees at all levels from the Chief Executive down to operational staff.

Funding agencies need to have a good understanding of the resource pressures facing the statistical agency and the trade-offs that need to be made in matching the high priority demands for statistics with the resources that are likely to be available.

Senior government officials need to understand the importance of good statistics for informed decision-making and the critical importance of their production in accordance with the UN Fundamental Principles of Official statistics.

The media plays a critical role in disseminating statistics to a wide audience. The media can also play a crucial role in framing public opinion about the quality of available statistics and the professional standing of the agency producing them.

Other stakeholders, such as non-government agencies and other community organizations, as users of statistics as well as information providers, can also play a critical role in shaping views on the quality and integrity of official statistics.

Stakeholder expectations are varied and should be explicitly managed at all stages in the statistical production process.


At the agency level

• Has the statistical agency clearly identified all of its stakeholders?

• Is the nature of the relationships between the statistical agency and each of its stakeholders defined and understood by both sides?

• Are processes in place to consult stakeholders on their needs and concerns?

• Are stakeholders kept informed on actions taken to address their needs and concerns?

• Is there a multi-sectoral body that advises the statistical agency in setting overall statistical priorities?

• In regard to users

- Are there subject-specific user committees?


- Are there arrangements in place for periodic high-level discussions with key users?

- Are appropriate strategies in place to service user needs through a combination of print, electronic and other services to ensure users have appropriate access to the statistics they need?

• In regard to data providers

- Does the statistical agency have a provider management policy and/or a provider charter?

- Does the statistical agency have a practice of regularly consulting with provider organizations such as other government departments and industry associations?

- Does the statistical agency have access to records maintained by any government department, corporation, business or organization that could be used for statistical purposes?

- Does the statistical agency have memoranda of understanding or other arrangements with administrative agencies to ensure that by-product administrative data provided to it will be suitable for statistical purposes?

- Does the statistical agency maintain continuing liaison with the providers of administrative records to strengthen the statistical value and usage of the administrative source?

• In regard to the funding agency

- Does the statistical agency have well-documented work plans and budgets that can be shared with the funding agency to ensure mutual understanding of funding requirements and trade-offs?

• In regard to the media

- Does the statistical agency have a strategy to manage media relationships and does it maintain regular contact with the media?

- Does the statistical agency respond, as appropriate, to negative media reporting to ensure fair reporting of its position?

- Are arrangements in place to ensure that the media is able to play a role in disseminating statistics to a wide audience?

• In regard to other stakeholders

- Are specific policies and practices in place to manage relationships with other key stakeholders?

At the programme design stage


- Are processes in place for consulting with user groups when new statistics are developed or existing statistics are reviewed?

- Are processes in place to monitor the relevance and practical utility of existing statistics in meeting their needs?

- Are processes in place for users to advise statistical agencies about their emerging needs and priorities?

- Are mechanisms in place to monitor users’ needs and feed them back into the design process?

- Are users’ priority needs being met and reflected in the work programme of the statistical agency?


- Are processes in place for maintaining close cooperation with the interest groups of those with a duty to provide information, and whose primary concerns have the highest priority?

At the programme implementation stage



- Are user support services available to give prompt assistance to users, by knowledgeable staff, to help them access and interpret the data?

- Is information provided on the methodology of statistical processes and the quality of statistical outputs?


- Are processes in place to assure statistical confidentiality of individuals, businesses or other entities in administrative records, and to ensure that the information will be used for statistical purposes only? At the post-collection evaluation stage

At the post-collection evaluation stage

• Are procedures in place for stakeholders to evaluate whether a statistical product is needed (with respect to its scope, level of detail, cost, etc.)?


The management of relationships with data users and data providers is likely to be more effective if the following supporting mechanisms are in place:

• Appropriate legal mandates and guarantees are provided.

• A cross-cutting statistics advisory committee exists to advise on overall statistical priorities.

• Up-to-date information on data providers and users exists.

• Subject-specific user committees are in place.

• Consultative and “intelligence-gathering” processes and regular stakeholder reviews are in place (e.g., periodic high-level discussions with users; user-producer dialogues; producer-provider dialogues; analysis of media coverage; customer satisfaction surveys; client and stakeholder feedback mechanisms).

• Policies exist for informing users of data quality and methodology, i.e. to inform users of the concepts and methodology used in collecting, processing and analysing data, to inform them about the accuracy of these data, and of any other features that affect their quality or "fitness for use".

• Communication strategies exist that are tailored to specific needs and concerns of stakeholder groups.

• Media relations groups, charged with developing high-level strategies for managing media relations, are established.

• Support services are in place to handle special requests and provide other assistance to users.

• Mechanisms related to managing statistical processes are in place (for example, specialized resource and support centres for certain functions; peer and institutional reviews; active management of response burden; promotion of the value and uses of statistics).


3. Quality assurance guidelines - NQAF 3: Managing statistical standards 14

NQAF 3: Managing statistical standards

Description:

Standards refer to a comprehensive set of statistical concepts and definitions used to achieve uniform treatment of statistical issues within a survey or across surveys, and across time and space. Standards assist in maximising the effectiveness of statistical outputs and the efficiency of the production process in terms of inter-temporal, national and international comparability and coherence (i.e. the capacity for integration) of the statistics.

While comparability and coherence are important for any dataset, they are particularly important where data are obtained from multiple sources and have to be combined or where outputs are used in a wide variety of contexts. For example, the use of standard collection units (e.g. families, households, businesses, etc.) helps the compilation, comparison and dissemination of statistics for these standardised units.

There are basically two broad types of standards – those that are applied to the structure and content of data, and those that are applied to the structure and content of metadata.

Statistical agencies should aim to use consistent names and definitions for populations, statistical units, concepts, variables, and classifications in their statistical programmes/domains.


At the agency level

• Does the agency work towards the development of statistical standards?

• Does the agency actively work with other statistical organizations in developing, reviewing, promoting and implementing statistical standards?

• Does the agency have an organizational unit responsible for taking the lead in the development of statistical standards and for supporting statistical programmes/domains in its efforts to develop standards, where such standards don’t exist or have become outdated? Is this responsibility assigned to staff with the appropriate level of seniority?

• Does the agency monitor the extent to which statistical standards are used by the statistical programmes/domains?

• Are all relevant staff aware of statistical standards and any changes made to them?

• Do statistical standards include a statement regarding the degree to which their application is compulsory?


• Does the process for originating, developing and approving statistical standards involve data users and data providers, including the agency’s own statistical programmes/domains?

• Are agency statistical standards accompanied by a statement of conformity to corresponding international or national standards?

• Are divergences from the corresponding international or national statistical standards documented and explained?

• Are there detailed concordances to corresponding international and national standards?

• Are there detailed concordances to previous statistical standards?


• Does the agency use conceptual frameworks, such as the System of National Accounts, that provide a basis for consolidating statistical information about certain sectors or geographical entities?

3. Quality assurance guidelines - NQAF 3: Managing statistical standards 15

• Does the agency develop integrated statistics programmes/domains that require statistical standards?

• Are statistical programmes/domains held accountable to apply the statistical standards?

• Do statistical programmes/domains have to obtain exemptions from statistical standards if they do not apply them?

• Are plans, including deadlines, for the development and application of new statistical standards communicated to statistical programmes/domains well in advance (even several years)?

• Do statistical programmes/domains, to the extent possible, collect and retain information at the fundamental or most detailed level of each standard classification in order to provide maximum flexibility in aggregation and facilitate retrospective reclassification as needs change?

• Are statistical products accompanied by, or make explicit reference to, readily accessible documentation on the statistical standards used?

• Are periodic reports to senior management prepared on the extent to which statistical standards are used by the statistical programmes/domains?

• Are the statistical standards communicated to all potential data users and the public?


• Are statistical standards regularly reviewed and revised, if necessary, to ensure their quality, notably their relevance, coherence and clarity?


The management of statistical standards is likely to be more effective if the following supporting mechanisms are in place:

• Central organizational units or senior level groups responsible to lead and coordinate the development, implementation, maintenance and use of statistical standards.

• Statistical programmes/domains based on conceptual frameworks or data integration frameworks that rely heavily on statistical standards.

• Active participation of both data users and data providers in the development and approval of statistical standards.

• Active participation with other national and international organizations in the development, review, promotion and implementation of statistical standards (e.g. employees attend workshops, conferences and seminars at the national and international levels on the application of standards, classifications, etc.).

• Correspondence tables for classifications exist and are kept up-to-date and made available to the public with explanatory information.


3. Quality assurance guidelines - NQAF 4: Assuring professional independence 16


NQAF 4: Assuring professional independence

Description:

Statistical agencies should develop, produce and disseminate statistics without any political or other interference or pressure from other government agencies or policy, regulatory or administrative departments and bodies, the private sector or any other persons or entities which may be considered as potential conflicts of interest. Such professional independence and freedom from inappropriate influence ensures the credibility of official statistics. This should apply to national statistical offices and may or may not apply to statistical units within ministries, central banks, etc.


At the agency level:

• Is a law or some other formal provision in force which specifies that statistical agencies are obligated to develop, produce and disseminate statistics without interference from other government agencies or policy, regulatory or administrative departments and bodies, the private sector or any other persons or entities which may be considered as potential conflicts of interest?

• If no law or formal provision exists which explicitly declares the necessity of professional independence, do traditions or cultures of professionalism, historical precedents or conventions exist which are clearly recognized as essential to the credibility of the statistical results of the statistical agencies?

• Are there laws, formal policies or procedures in place for dealing with actual or perceived, or potential, conflicts of interest? Is there a culture within the statistical agency that such issues are dealt with swiftly and effectively?

• Are the rules applied for appointing and dismissing the heads of the statistical agencies based on professional competence, and are they free from political considerations?

• Do the processes in place ensure that the heads of the statistical agencies are of the highest professional calibre and have sufficiently high hierarchical standing to ensure senior level access to policy authorities and administrative public bodies?

• Do the heads of the statistical agencies have exclusive and full control over the decisions on statistical methods, standards and procedures, and on the content and timing of statistical releases?

• Does the responsibility for ensuring that statistics are developed, produced and disseminated in an independent manner rest with the heads of the statistical agencies?

• Are procedures in place for regularly publishing the statistical work programmes and for issuing periodic reports to describe progress made?

• Are procedures in place to ensure that statistical releases are clearly distinguished from political/policy statements and issued separately from them?

• Is there a formal policy or well-established custom entitling statistical agencies to comment publicly on statistical issues, criticisms, misinterpretations and misuses of official statistics?


Professional independence is likely to be more effectively assured if the following supporting mechanisms are in place:

• A national statistical law or other formal policies are in place which specify the statistical agency's independence in terms of: a) Development (activities aiming at setting up, strengthening and improving the statistical methods, standards and procedures used for the production and dissemination of

3. Quality assurance guidelines - NQAF 4: Assuring professional independence 17

statistics as well as at designing new statistics and indicators); b) Production (i.e. the selection of techniques, definitions, methodologies and sources relating to the collection, storage, processing and analysis necessary for compiling statistics); and c) Dissemination (i.e. the content and timing of all forms of dissemination) of statistics and the roles and responsibilities of the statistical agencies.

• The laws, regulations and measures under which the statistical systems operate are made public.

• The procedures to be followed for the appointment and dismissal of heads of the agencies are publicly available.

• A code or declaration on ethics exists (e.g. the International Statistical Institute's Declaration on Professional Ethics) to enable the statistician's individual ethical judgments and decisions to be informed by shared values and experience.

• Appropriate internal and external communication strategies exist that include recognizable logos, designs or formats for statistical agencies’ products to identify them as not being associated with the political or policy bodies.

• Policies or established procedures for preventing and addressing misuse or misinterpretations of official statistics are in place.

• Policies and procedures for dealing with actual or perceived, or potential conflicts of interest are in place.

• Documentation on concepts, sources and methods that describe, among other things, the design decisions and trade-offs made in the development of the statistical product, are available for individual statistical programmes.


3. Quality assurance guidelines - NQAF 5: Assuring impartiality and objectivity 18

NQAF 5: Assuring impartiality and objectivity

Description:

Statistical agencies should develop, produce and disseminate statistics respecting scientific independence and in a manner that is professional, transparent, neutral and unbiased, in which all users are treated equitably.


At the agency level:

• Is there a law or formal provision in force which specifies that statistical agencies should develop, produce and disseminate statistics following professional standards and treat all users in the same way?

• Does the statistical agency follow a declaration or code of conduct or ethics which governs statistical practices (e.g. Code of Practice, declaration on professional ethics, etc.), and if so, is its implementation followed up?

• Do guidelines for assuring impartiality and objectivity exist, and if so, is the implementation of the guidelines followed up?

• Is recruitment and promotion of the staff responsible for the development, production and dissemination of statistical information based on relevant aptitude and expertise in statistics and/or other relevant subject matters?

• Are statistics produced on an objective basis which is determined only by statistical considerations?

• Are sources, concepts, methods, processes and data dissemination paths chosen on the basis of statistical considerations and national and international principles and best practices?

• Does a policy for data dissemination exist and if so, is it made publicly known?

• Is information made available to all users at the same time with no privileged access for governmental representatives?

• In cases where privileged pre-release access is given, is it controlled and publicized?

• Is there a release calendar in place in which dissemination dates and times are pre-announced?

• Are any deviations from the release calendar announced and justified to the users?

• Are major changes in the methodologies and data revisions clearly explained to users?

• Are procedures in place to ensure that statistical releases are clearly distinguished from political/policy statements and issued separately from them?

• Are statistical releases and statements made in press conferences objective and non-partisan?

• In case errors are detected, are they corrected as soon as possible and are users informed about those errors that affected the released data?


Impartiality and objectivity are likely to be more effectively assured if the following supporting mechanisms are in place:

• A law or other formal policies specify the statistical agency's impartiality and objectivity with regard to the development, production and dissemination of statistical information.

• The laws, regulations and measures under which the statistical systems operate are made public.

• A declaration or code of conduct or ethics which governs statistical practices exists (e.g. a code of practice, declaration on professional ethics, etc.).

3. Quality assurance guidelines - NQAF 5: Assuring impartiality and objectivity 19

• An effective human resources system to objectively manage the appointment and promotion of the agency’s staff.

• The choices made for developing, producing and disseminating statistical information are publicly available.

• Guidelines that specify how to ensure impartiality and objectivity in individual statistical programmes are produced and followed up.

• Institutional reviews (e.g. reviews of statistical products and publications by peers and senior management) are performed to ensure that the information produced is impartial and objective.

• The policy for data dissemination is publicly available.

• A release calendar which announces the dates and timing of major statistical releases is prepared and made known in advance to users.

• Information on any major methodological changes, data revisions and correction of errors is prepared and provided to users in a timely manner.

• Policies or established procedures exist for preventing and addressing misuse or misinterpretations of official statistics.

• Policies and procedures exist for dealing with actual or perceived, or potential conflicts of interest.


3. Quality assurance guidelines - NQAF 6: Assuring transparency 20

NQAF 6: Assuring transparency

Description:

The statistical agencies’ statistical policies and practices and the terms and conditions under which their statistics are developed (including the legal basis and purposes for which the data are required), produced, and disseminated (and, if applicable, subsequently revised) should be documented and available to users, survey respondents and the public. Products of statistical agencies/units should be clearly identified as such.


At the agency level

• Are the terms and conditions under which statistics are developed, produced, and disseminated available to the public?

• Is internal government access to statistics prior to their release allowed? If so, is this internal government access publicly disclosed?

• Is there a standard procedure for ensuring that respondents understand the legal basis for a survey and the confidentiality provisions for the data that are collected?

• Are products of the statistical agencies clearly identified as such?

• Is advance notice given of major changes in methodology, source data, and statistical techniques?


Transparency is likely to be more effectively assured if the following supporting mechanisms are in place:

• A law or other formal policy requiring the dissemination of the statistical policies and practices, and outlining the dissemination process. The terms and conditions incorporated in such laws/policies may refer to the relationship of the statistical unit to the larger department or ministry of which it is a part (if relevant), the legal authority to collect data, the requirement to publish data it has collected, the terms of reference for the chief statistician/director, and procedures and processes related to confidentiality of individual responses.

• The compilation of a listing of persons or officials holding designated positions within the government, but outside the agency producing the data, who have pre-release access to the data and the dissemination of the schedule according to which they receive access. This practice is intended to grant full transparency to any pre-release access deemed necessary by the government.

• The existence of a transparent planning process, including a strategic plan which explicitly identifies longer term priorities for public scrutiny.

• Metadata which are necessary for the proper understanding of the statistics and the appropriate uses to which they can be put (such as the underlying concepts and definitions, origins of the data, the variables and classifications used, the methodology of data collection and processing, and indications of the quality of the statistical information) are made available to the public.

• Release calendars and changes in them are communicated to the public.

• Revision policies and practices are communicated to the public.

• Clear attribution of the statistical agency’s products, through, for instance, the use of a logo or other insignia.


3. Quality assurance guidelines - NQAF 7: Assuring statistical confidentiality and security 21

NQAF 7: Assuring statistical confidentiality and s ecurity

Description:

Statistical agencies should guarantee that the privacy of data providers (persons, households, enterprises, administrations and other respondents) will be protected and that the information they provide will be kept confidential, will not be able to be accessed by unauthorized internal or external users, and will be used for statistical purposes only. Statistics shall be considered confidential when they allow statistical units to be identified, either directly or indirectly, thereby disclosing individual information. Examples of purposes that are not exclusively statistical include administrative, legal or tax purposes.

Note: National statistics laws usually spell out the exceptions to the general rule regarding non-release of identifiable unit record data, e.g. where informed consent has been given by a business that its data may be released.


At the agency level

• Is there a law or some other formal provision in force that guarantees the proper management, with regard to privacy and security, of information received from data providers? Are national privacy laws respected?

• Where the statistics law provides for exceptions to the general confidentiality provisions, are clear policies and procedure in place, and are they made public to operationalize the exceptions?

• Are appropriate codes of practice and standards in place to ensure that statistical data about individual respondents remain confidential, and are only released to users in line with statistical legislation and data dissemination policies?

• Do the statistical agencies have a formal data dissemination policy that sets out how statistics are to be disseminated to users and under what circumstances microdata (i.e. statistical information relating to individual respondents) may be made available for research and further analysis?

• Where microdata are to be disseminated, do the statistical agencies have appropriate procedures and processes (e.g. anonymization) in place to ensure that individual respondents cannot be identified from the data?

• Are appropriate penalties provided for statistical staff or other personnel who have been found guilty of activities leading to the release of confidential data?


• Do statistical agencies identify in advance what data and microdata are to be disseminated and where there is a risk that confidential data about individual respondents could be identified?

• Are appropriate procedures identified in advance to ensure that data and microdata are anonymized?


• Are appropriate processes in place to assess the risk that individual respondents can be identified from the public release of statistics or of microdata, and are procedures applied in line with the data dissemination policy to eliminate or minimize this risk?

• Where there is a risk of identification of individuals from the public release of statistics or microdata, and this risk is considered to be above a minimum level, depending on the sensitivity of the data, are the data or microdata then not disseminated?

• Are all procedures that are taken to eliminate or adequately reduce the risk of identification properly documented and made available as part of the metadata related to the statistical dataset?

• Are users made aware that procedures to eliminate the risk of identification have been implemented and that this could lead to a loss of information?

3. Quality assurance guidelines - NQAF 7: Assuring statistical confidentiality and security 22

• Are appropriate physical and information technology security procedures in place to ensure the protection of unit records?


• Do the statistical agencies monitor the use of microdata sets to identify any circumstances in which data confidentiality may be breached, for example, through file matching, and do they take immediate action to redress this situation?


Statistical confidentiality and security are likely to be more effectively assured if the following supporting mechanisms are in place:

• Legal arrangements are in place to protect confidentiality.

• The staff signs legal confidentiality agreements or declarations covering their obligations upon appointment.

• Legally enforceable contracts regarding the use of microdata/public use files exist.

• Policies and procedures and staff training that elaborate the legal arrangements are made known.

• Guidelines and instructions are provided to staff on the protection of statistical confidentiality in the production and dissemination processes.

• Penalties are prescribed against persons who wilfully violate the law by breaching statistical confidentiality.


.

3. Quality assurance guidelines - NQAF 8: Assuring the quality commitment 23

NQAF 8: Assuring the quality commitment

Description:

Statistical agencies should be dedicated to assuring quality in their work, and systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. Processes, staff and facilities should be in place for ensuring that the data produced are commensurate with their quality objectives.


At the agency level

• Does the statistical agency’s policy or message about its commitment to quality in statistics clearly convey and promote the shared concern for quality of all of its staff, and include information about trade-offs affecting the statistical work programme? Is this policy made public?

• Is a culture of continuous improvement promoted that systematically fosters the documentation of methodology and processes and the exchange of good statistical practices, as well as the monitoring, assessment and improvement of the quality of statistical operations?

• Is there a specific person or persons who have been assigned explicit responsibility for the management of quality within the statistical agency?

• Does the statistical agency follow any of the externally recognized processes or activities that focus on quality?

• Are guidelines for implementing quality management defined which: (a) describe the entire statistical process and identify relevant documentation for each stage of production; (b) describe the methods for monitoring the quality of each stage of the statistical production process; and (c) identify the indicators (quality measures) for evaluating the quality of the main stages of production?

• Are the statistical agency’s quality guidelines made available to external users, at least in a summary version?

• Are measures in place for conducting periodic quality reviews of key products to assess adherence to internal guidelines and international standards? Is top management informed of the results in order to define improvement actions?

• Are procedures in place to ensure that the required documentation on quality is regularly updated?

• Are staff training and development programmes in place to ensure that the staff is well aware of the statistical agency’s quality policy and has an understanding as to how quality may be achieved; are the statistical agency’s internal auditors trained in auditing techniques and behavior?

• Does the management of the statistical agency ensure that compiling areas or subject matter units have access to necessary tools and specialized methodological and technical support as needed to help implement their strategies for improving data development, production and dissemination?

• Is benchmarking of key statistical processes with other statistical agencies carried out to identify good practices?


• Is a quality assurance plan or similar mechanism in place that describes the working standards, the formal obligations (such as laws and internal rules) and quality control actions to prevent, monitor and evaluate errors and to control different points at each stage of the statistical process?

• Are procedures in place to monitor the quality of different stages of the statistical production, e.g. according to a quality assurance plan or similar mechanism?

• Are there mechanisms in place to take users’ needs into account?

3. Quality assurance guidelines - NQAF 8: Assuring the quality commitment 24

• Are trade-offs within quality systematically examined?


• Are there mechanisms in place to assure the quality of data collection (including the use of administrative data) and data editing?


• Are metadata and quality indicators or measures prepared and provided to users to help them assess the quality of the released data?

• Are outside experts called in to conduct some of the quality reviews?

• Are mechanisms in place to collect and follow up on users’ reactions and feedback?

• Are user satisfaction surveys implemented regularly, and are their results made public?


The commitment to quality is likely to be more effectively assured if the following supporting mechanisms are in place:

• A written quality policy, declaration or commitment statement is publicly available.

• A staff awareness “campaign” is undertaken to emphasize the statistical agency’s commitment to quality.

• A quality manager, quality committee, unit or group of coaches or advisers is assigned responsibility for quality management.

• Guidelines, methodological manuals and handbooks on recommended practices are made available.

• The use of TQM, ISO 9000, quality initiatives of the European Statistical System, independent evaluations and/or IMF ROSC evaluations is promoted.

• The Generic Statistical Business Process Model is followed for guidance for managing and monitoring the quality of each stage of the statistical development, production and dissemination process.

• Programmes or strategies are in place for regularly carrying out user- and producer-oriented quality reports, self-assessments, and/or audits on statistical operations in order to monitor and report on quality over time.

• External experts conduct quality reviews, e.g. reviews of key statistical domains (for example IMF’s ROSCs) or other reviews such as peer reviews, external audits, and rolling reviews.

• User satisfaction surveys are carried out and reports on the results are made publicly available.

• Work plans, schedules and standard forms or templates are used for facilitating the updating of the documentation on quality in a consistent way.

• The statistical agency’s training programme includes regularly held training courses designed to support its quality policy and to ensure that the internal auditors have proper training in auditing techniques.

• IT staff, methodologists and other specialists (e.g. in questionnaire design) participate in assisting subject matter units; appropriate software is provided.

• Validation techniques are widely promoted and applied.

• Expert group meetings on relevant quality subjects are held regularly.

• User-oriented quality reports are made available to the public.

• Documentation on methods, concepts and definitions is available for all major fields of statistics.


3. Quality assurance guidelines - NQAF 9: Assuring adequacy of resources 25

NQAF 9: Assuring adequacy of resources

Description:

The financial, human, and technological (IT) resources available to statistical agencies should be adequate both in magnitude and quality, and sufficient to meet their needs with regard to the development, production and dissemination of statistics.


At the agency level

• Are the financial and human resources sufficient to implement the statistical work programme?

• Are the available technological resources (hardware, software, etc.) sufficient to support the statistical production process?

• Is resource allocation reviewed on a regular basis?

• Are planning and management principles (e.g. results-based management), aimed at the optimal use of available resources and applied throughout the national statistical system?

• Is a resource mobilization strategy and implementation plan in place?

• Is standardization of statistical production and dissemination pursued as a way to increase efficiency and savings?


• Is the programme feasible given the available resources?

• Does the programme re-use existing methods and tools?

• Does the programme use data from existing sources, where appropriate?


• Are the costs of each stage of the production processed measured?

• Are data processing operations combined with those for other outputs to increase efficiency and savings?


• Are the costs (human and financial) of the statistical production process accurately assessed?

• Is a cost-benefit analysis conducted?


The adequacy of resources is likely to be more effectively assured if the following supporting mechanisms are in place:

• Strategic planning for setting priorities.

• Management accounting standards and techniques.

• Standardization of procedures and tools for statistical production and dissemination in order to increase efficiency.

• User groups for helping to determine priorities when resources are limited.


3. Quality assurance guidelines - NQAF 10: Assuring methodological soundness 26


NQAF 10: Assuring methodological soundness

Description:

In developing and compiling statistics, a statistical agency should use sound statistical methodologies based on internationally agreed standards, guidelines or best practices and consistent with established scientific principles. Effective and efficient statistical procedures should be implemented throughout the statistical production chain.


At the agency level

• Is the overall methodological framework of the statistical agency consistent with international standards, guidelines and good practices?

• If not, are divergences from international standards explained?

• Are procedures in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical agency?

• Are the processes used for the development, collection, compilation and production of statistics well documented and regularly reviewed to assess their efficiency and effectiveness?

• Are measures in place to ensure that the staff is recruited by the statistical agency from the relevant disciplines and has the appropriate qualifications?

• Are training and development programmes in place to ensure that the staff acquires and continuously updates its methodological knowledge?

• Is there cooperation with the scientific community to improve methodology and the effectiveness of the methods implemented, and to promote better tools?

• Are the methodologies of surveys and the use of administrative records evaluated periodically to guarantee high quality statistical outputs?

• Are there management processes in place that allow the senior management of the statistical agency to be assured that sound methodological approaches have been adopted in producing the statistical outputs?

• Do mechanisms exist to ensure methodological soundness and consistency throughout the national statistical system?


• Are the proposed scope, concepts, definitions, classification/sectorization, and basis of recording in accordance with applicable international standards?

• Has explicit consideration been given to overall trade-offs between accuracy, cost, timeliness and provider burden during the programme design stage?

• Has there been explicit consideration of alternative sources of data, including the availability of existing survey data or administrative records, to minimize new data collection?

• Is there adequate justification for each question asked, and has there been appropriate pre-testing of questions and questionnaires in each mode of collection, while also assuring that the set of questions asked is sufficient to achieve the descriptive and analytical aims of the survey?


• Is a systematic approach in place for updating the survey frame to ensure adequate coverage of the target population?

• Within overall trade-offs, has there been proper consideration of sampling and estimation options and their impact on accuracy, timeliness, cost, respondent burden, and data comparability over time and across programmes?

• In respect of use of administrative records, is the population consistent with the statistical output requirements; are the classifications appropriate; are the underlying concepts appropriate; and are the records complete and up to date?

• Are mechanisms in place to facilitate the review, by the statistical agency, of the methodology used by an independent body that may carry out the statistical production process on behalf of the agency; are procedures also in place for the statistical agency to advise the independent body on the methodology to be used?


• Are appropriate instruments of implementation in place, including resource and material plans, the supervisory structure, schedules, operations, procedures and checks, training and the publicity surrounding the collection?

• Are adequate measures in place for encouraging accurate response, following up on non-response, and dealing with missing data?

• Are adequate quality control and quality assurance measures in place at all stages of collection and processing?

• Are there appropriate arrangements in place for internal and external consistency checking of data, with corresponding correction or adjustment strategies?

• Is information on costs and efficiency of operations being assembled to inform future design decisions?

• Is management information available to manage and monitor all aspects of the collection? Such information might include regular reporting and analysis of response rates and completion rates; monitoring refusal and conversion rates; monitoring interviewer and respondent feedback; monitoring of edit failure rates and progress of corrective actions; monitoring the results of quality control procedures during collection and processing; monitoring of expenditures against progress, etc.

• Are contingency plans in place in the event of emerging problems?

• Are assessments done in respect of: coverage of the population against the target population; the sampling error when sampling is being used; non-response rates, or percentages of estimates imputed; or any other serious accuracy or consistency problems with the collection results?

• Are appropriate arrangements in place for post-collection evaluation to take stock of outcomes as compared with design plans, to draw out any issues upon which users should be informed, and to provide feedback for consideration in the planning for future such collections?


• Has there been follow up with users to seek their views on the fitness of the statistical outputs for the purposes for which the data will be used?


Methodological soundness is likely to be more effectively assured if the following supporting mechanisms are in place:

• Standard definitions of concepts, variables and classifications for common subject-matter areas.

• Regular training courses, for staff in production units, on methodologies for statistical production stages (e.g. data collection techniques, data editing, imputation, etc.).


• A unit and/or a group of experts in charge of defining and updating methodologies and providing support to production units.

• Specialized resource and support centres for certain functions (e.g. questionnaire design and testing, seasonal adjustment, data analysis, etc.).

• A software policy that identifies recommended and supported software for key functions.

• The use of specialized staff for subject-matter, methodology, operations and systems to participate in programme design.

• Quality guidelines, methodological manuals and recommended practices handbooks that list considerations and provide guidance in design decisions.

• Internal budgeting and accounting systems so that the real costs of alternative collection approaches can be assessed.

• Appropriate staffing arrangements that facilitate effective respondent conduct and survey processing.

• Standard survey frames for major populations, e.g. the business register, the population register or population survey area sample, the address register, etc.

• Peer and institutional reviews.


3. Quality assurance guidelines - NQAF 11: Assuring cost-effectiveness 29

NQAF 11: Assuring cost-effectiveness

Description:

Statistical agencies should assure that resources are effectively used. They should be able to explain to what extent the set objectives were attained and that the results were achieved at a reasonable cost consistent with the principal purposes for which the statistics will be used.


At the agency level

• Do guidelines for assuring cost-effectiveness exist?

• Does the statistical agency promote and implement standardized solutions that increase effectiveness and efficiency?

• Is the statistical agency’s use of resources monitored both by internal and independent external measures?

• Are sample surveys used instead of censuses when it is appropriate and possible?

• Are the costs of producing the statistics well documented at each stage of statistics production to assess their effectiveness?

• Are the costs of producing the statistics regularly reviewed to assess optimization across the office?

• Are proactive efforts made to improve the statistical potential of administrative data and to limit recourse to direct surveys?

• Are administrative data used instead of sample surveys when it is appropriate and possible?

• Are cost–benefit analyses carried out to determine the appropriate trade-offs in terms of data quality?

• Is the respondent burden managed?

• Are reports on cost-effectiveness made available to the public?


• Is there a clear and documented justification for the specific programme?

• Before contemplating a new data collection, are there mechanisms to review whether current data sources can be utilized with minimal impact on their current purpose and quality?

• Is there an ongoing review process that considers whether a particular programme is still operating in the most cost-effective way to meet its stated requirements?


• Is the productivity potential of information and communications technology being optimized for data collection, processing and dissemination?

• Is every effort made to minimize the reporting burden consistent with the principal purposes for which the statistics will be used?

• Are routine clerical operations (e.g. data capture, coding, validation) automated where possible?


• Is the cost-effectiveness of every statistical survey assessed?

• Do mechanisms exist for assessing whether the outputs produced meet the needs of the key users so as to justify the collection of the data?

3. Quality assurance guidelines - NQAF 11: Assuring cost-effectiveness 30


Cost-effectiveness is likely to be more effectively realized if the following supporting mechanisms are in place:

• Indicators of human and financial resources are monitored centrally and regularly reported to management.

• Accounting systems allow allocation of resources to statistical operations.

• Human resources are evaluated annually in line with office-wide guidelines. The evaluation covers allocation, performance and training needs of the staff.

• Ex-ante cost calculation procedures are available for statistical operations.

• Centralized IT and methodological units provide possibilities for the pooling of resources and investments and the identification of innovation/modernization potential.

• An appropriate IT architecture and strategy exists and is regularly updated.

• IT infrastructure is reviewed regularly.

• Data linking and integration methods are proactively pursued subject to data security considerations.

• Standardization programmes and procedures are defined and implemented in key areas, such as sampling, registers, data collection and data exchange according to the business process model.

• Policies, procedures and tools exist to promote automatic techniques for data capture, data coding and validation.

• The use of automated processing techniques is regularly reviewed.

• Appropriate arrangements (e.g. service level agreements or national legislation) are signed with owners of administrative data collections and regularly updated. The statistical agency seeks to be involved in the design of administrative data collections.

• An assessment of possible administrative data sources is carried out prior to launching any new survey.

• Quality indicators are developed and compiled to improve the use of administrative data for statistical purposes.

• Procedures are in place to measure and manage the respondent burden.

• Staff opinion surveys are conducted regularly.


3. Quality assurance guidelines - NQAF 12: Assuring soundness of implementation 31

NQAF 12: Assuring soundness of implementation

Description:

In order to produce timely, reliable and accurate statistics, a statistical agency should carefully plan the implementation process of its statistical activities based on internationally agreed standards and guidelines and the application of sound and scientific methods. The implementation process refers to all activities which lead to the production of statistics including design and preparations, data collection, data processing (coding, editing, imputation, etc.), assessment and compilation.


At the agency level

• Does the agency have staff selection and training programmes that emphasize the importance of statistics that are fit for purpose?

• Are programme or project management policies and practices in place that emphasize data quality issues and management of risks to data quality?

• Does the agency have practices that build in data quality checkpoints and (as appropriate) sign-offs before proceeding to subsequent stages in the statistical life cycle?

• Does the agency have documented procedures for the design, development, implementation and evaluation of statistical compilations? Do these procedures give adequate attention to data quality issues?

• Are there established practices for consulting with stakeholders, especially users and potential respondents, at all appropriate stages of the statistical life cycle?


• Is the data collection process designed to reduce the respondent burden while increasing response rates, in particular by applying different modes of data collection (for instance a telephone-based interview in a sample survey to complement a self-enumeration process and reduce non-response error)?

• Has the data collection plan taken into consideration different sources of data such as administrative statistics instead of only traditional modes of data collection?

• Is the soundness of the use of administrative data sources ensured?

• Are data collection instruments (mainly questionnaires) designed to minimize coding cost and time?

• Has the data collection plan taken into consideration ethnicity and language of respondents in staff selection, training and development of data collection instruments?

• Does the questionnaire design allow for automated data capture?

• Is the data capture mode designed to improve accuracy and timeliness through integrating, to the extent possible, data capture with data collection or automated data capture rather than the traditional manual mode of capture?

• Are edit rules streamlined in the data collection system in order to validate the data entered and allow for error corrections and quality improvement during the data capture exercise?

• Are follow-up activities planned to collect support information (such as the size of households or establishments, dwelling status, etc.) in order to use the information for non-response adjustment?

• Is there a sound methodology in place for imputation that has been developed based on scientific principles?

• Has the staff directly involved in the implementation process (including interviewers) been consulted before developing training materials?

3. Quality assurance guidelines - NQAF 12: Assuring soundness of implementation 32


• Are training materials and manuals carefully prepared and planned in a way that considers different alternatives for training and established standards for monitoring and improving interviewers’ skills?

• Is the list of respondents’ contact information regularly updated?

• Is the statistical activity properly communicated and advertised in order to increase awareness and trust among respondents?

• Is the staff who is involved in data processing trained and familiar with the principles of confidentiality of data?

• Is the format of the database compatible with different statistical software that is usually used for compilation and data analysis?

• Are data capture and data collection instruments tested and adjusted (if required and possible) prior to the actual field operation or data collection process?

• Are the required unique codes and identifiers created properly for the purpose of record linkage with other sources of information?

• When coding is done through an automated process, is a team of well-trained coders assigned to handle uncoded cases?

• Are proper follow-up procedures planned for filling data gaps and handling inconsistencies?

• Is data editing repeated after each stage of data processing, including imputation?

• When weighting is required, is a weight associated to each sampled unit in the database after the data processing is completed?

• Are activity and cost indicators (survey paradata) produced and properly documented in order to be used in monitoring and managing the current and future collection processes?


Soundness of implementation is likely to be more effectively assured if the following supporting mechanisms are in place:

• Human resources recruitment, training policies and development programmes of the statistical agency put considerable emphasis on the technical competencies as well as on the understanding of the importance and characteristics of quality statistics.

• Risk management policies are in place to identify, analyse and respond to the uncertainties which may arise in different levels of the implementation of a statistical process.

• A strategy for measuring and analysing quality measures including, but not limited to, paradata, data capture and processing errors and non-sampling errors is in place in order to identify causes of errors and to be used by the statistical agency for improving accuracy and timeliness of statistical products and for reporting purposes.

• A strategy is in place for reporting quality measures related to the implementation process such as paradata, data capture and processing errors and non-sampling errors to different levels (managers, implementation personnel, users, stakeholders).

• A system is in place for documentation and for archiving all the documents produced during the implementation process and is accessible for current and later use in order to monitor and evaluate the process, identify lessons learned and utilize for the statistical practices in the future.

• The proper electronic hardware and software is in place and is cost-efficient.


3. Quality assurance guidelines - NQAF 13: Managing the respondent burden 33

NQAF 13: Managing the respondent burden

Description:

Individuals, households or businesses who provide data, upon which statistical products are based, are fundamental contributors to the quality of data and information. The requirement to collect information (user needs) should be balanced against production costs and the burden placed on respondents (supplier costs). Mechanisms to maintain good relationships with individual providers of data and to proactively manage the response burden are essential for improving quality.

This difficult challenge is particularly topical with declining response rates in surveys. This decline lowers quality and increases the cost of surveys. Improving response rates requires a multi-dimensional strategy that addresses the issue of non-response at different stages of the survey process. This includes an assessment of the need to collect the information, the use of data from administrative sources or other surveys, and the use of sound statistical and survey methods to keep the burden to a minimum.


At the agency level

• Is there a respondent relations or respondent burden management programme?

• Does a provider charter exist that spells out the rights and responsibilities of respondents?

• Is there a respondent advocacy position or a respondent relations officer that is clearly separated from the data collection processes?

• Are there guidelines or procedures in place for coordinating and supporting respondent management activities?

• Are there mechanisms in place to promote the value and use of statistics to respondents?


• Are there mechanisms in place to assess the necessity to undertake a new statistical survey?

• Do surveys apply sound methods to reduce or distribute response burden?

• Do surveys apply statistical standards to make it easier to respond to them?

• Are respondents provided with information about: the purpose of the survey (including the expected uses and users of the statistics to be produced from the survey), the authority under which the survey is taken, the collection registration details, the mandatory or voluntary nature of the survey, confidentiality protection, the record linkage plans and the identity of the parties to any agreements to share the information provided by those respondents?


• Are there standard practices to respond to respondents’ requests and complaints?


• Are assessments undertaken to ascertain if there were problematic aspects of the questionnaire design and its implementation?


Management of the respondent burden is likely to be more effective if the following supporting mechanisms are in place:


For assessing the necessity to undertake a new statistical survey:

• Users’ needs and the range of data items involved are assessed against the corresponding respondent burden;

• Statistics suitable for users' needs are sought from some other source;

• A determination is made as to whether the required data can be produced with less response burden by modifying an existing survey rather than instituting a new survey (including the possibility of using a sub-sample of an existing survey);

• A determination is made as to whether an existing survey can be eliminated or reduced in size or content as new data requirements of users indicate a shift in their needs or priorities;

• Any data items that are the same or similar to those collected in another survey are eliminated.

For actively managing the response burden:

• A provider charter exists which spells out the rights and responsibilities of respondents;

• A continuous effort is made to research, develop and implement techniques that reduce the burden on respondents;

• The costs of compliance are systematically taken into account in the design of a new survey or redevelopment of existing surveys;

• The cost of complying with statistical surveys is planned, measured and reported each year;

• The value of administrative data in producing statistics is recognised, and statistical purposes are promoted in the design of administrative systems;

• The burden on respondents is measured and taken into account in regular comprehensive reviews of surveys and their methods.

For applying methods to reduce or distribute the response burden:

• Good quality frames are used;

• Sample surveys instead of censuses are used, as well as advanced sampling techniques to minimize sample sizes to achieve the target level of accuracy;

• Co-ordinated delineation of sampling population or co-ordinated sampling is used that could range from avoiding overlap, i.e. spreading the total burden on many respondents, to maximizing overlap, i.e. concentrating the burden on fewer respondents;

• Multiple modes of collection are offered to respondents;

• Electronic reporting initiatives are introduced, where cost-effective from a respondent perspective;

• Collection of data is done at the most appropriate time of the day or the year;

• Surveys are conducted from central registers or other common frames to better record, assess and control response burden.

For applying statistical standards to make it easier to respond to surveys:

• Standard frameworks, concepts, questions and classifications are used, while respondents are still allowed to easily complete questionnaires from readily available information or bookkeeping records;

• Questionnaires are pre-tested;

• Questionnaires are tested to ensure minimal intrusion on privacy and to respect public sensitivities and overall social acceptability;

• The legal obligation, policies and practices to assure the confidentiality and security of all respondent-provided data are made known.


For promoting the value and uses of statistics:

• Information packages that demonstrate the value of official statistics are provided to respondents;

• Initiatives with community groups and business advocates are undertaken to raise awareness of the value of official statistics;

• Internet-based products are developed that give valuable statistical information to businesses and individuals, and these products are promoted through initiatives with communities and respondents;

• A presence on social media is set up which provides, for example, key population and economic indicators;

• Schools, communities, business associations, etc. are reached out to.


3. Quality assurance guidelines - NQAF 14: Assuring relevance 36


NQAF 14: Assuring relevance

Description:

The relevance of statistical information reflects the degree to which the information meets the current and/or potential or emerging needs or requirements of clients, users, stakeholders, or the audience. Relevance therefore refers to whether the statistics that are needed are produced and whether those that are produced are in fact needed and useful, and shed light on the issues of most importance to users. Relevance also covers methodological soundness, particularly the extent to which the concepts, definitions and classifications correspond to user needs.

Assessing relevance is subjective and depends upon the varying needs of users. The statistical agency’s challenge is to weight and balance the conflicting needs of current and potential users in order to produce statistics that satisfy the most important and priority needs within given resource constraints. Relevance can be seen as having the following three components: completeness; user needs; and user satisfaction.


At the agency level

• Are procedures in place to consult users about the content of the statistical work programme?

• Are strategic goals and work programme plans developed in such a way that judgments are able to be made about competing user needs and are these goals and plans made public on a regular basis?

• Have agreements been laid down with the main users of the statistics (e.g. with respect to what will be supplied by the agency, the product quality of the statistics, the dissemination format, etc.)?

• Has a policy been formulated on the type of statistics that the statistical agency wants to produce and does not want to (or cannot) produce?

• Are procedures in place to prioritise between different users’ needs in the work programme? Are the data on the use of statistics analysed to support priority setting?

• Does the statistical agency have an advisory council to advise on overall statistical priorities?

• Are periodic reviews undertaken of the continuing relevance and cost-effectiveness of individual statistical programmes/domains?

• Is there a thorough understanding of any legislative or regulatory requirements to compile particular statistics?

• Is there a good understanding of the interdependencies between individual statistical programmes/domains?

• Are procedures in place to ensure coordination, harmonisation and full coverage of statistical information produced by the national statistical system?


• Are mechanisms in place to identify users’ needs and to describe how the data relate to their needs?

• Are the users and uses to which they put the statistical products known and regularly tracked?

• Are procedures in place to gather information on potential needs of users of statistics?

• Are the survey objectives set out in written form specifying: the population of interest; the geographic level of detail required; the intended reference period; the frequency and timeliness; the main data items/outputs required; the type of analysis intended to be conducted on the data; and other necessary

3. Quality assurance guidelines - NQAF 14: Assuring relevance 37

quality attributes the statistics need to meet to be consistent with users’ expectations of fitness for purpose?

• Is user satisfaction regularly measured and systematically followed up?


• Are editing and other statistical quality control processes in place to manage operations consistent with the resultant statistics meeting users’ priority needs?

• Are the statistical dissemination products aligned with users needs?

• Are the users informed about known gaps between the measured statistical concept and the user’s concept of interest?

• Are metadata described and made available to the users?


• Are arrangements in place for post-collection evaluations to: take stock of outcomes as compared with user needs; highlight any issues upon which users should be informed; and to provide users ways to give feedback that can be taken into account in the planning for future such collections?

• Are action plans implemented to improve relevance and meet emerging needs?


Relevance is likely to be more effectively assured if the following supporting mechanisms are in place:

• The inclusion in the legislation or some other formal provision of an obligation to consult with the users of the statistics.

• Quality guidelines that list considerations and provide guidance in design decisions.

• Structured and periodic consultation processes (e.g., advisory committees or working groups) with stakeholders and users to review the usefulness of existing statistics and to identify emerging data requirements.

• Participation in statistical meetings and seminars organized by international and regional organizations and by professional organizations.

• Regular programme reviews.


3. Quality assurance guidelines - NQAF 15: Assuring accuracy and reliability 38

NQAF 15: Assuring accuracy and reliability

Description:

Statistical agencies should develop, produce and disseminate statistics that accurately and reliably portray reality. The accuracy of statistical information reflects the degree to which the information correctly describes the phenomena it was designed to measure, i.e. the degree of closeness of estimates to true values. It is usually characterized in terms of estimation of sampling and non-sampling errors. These errors are traditionally decomposed into bias (systematic error) and variance (random error) components, and reflect the major sources of error (e.g. errors linked to sampling, coverage, measurement, non-response and processing). Reliability concerns whether the statistics consistently over time measure the reality that they are designed to represent.


At the agency level

• Are systems for assessing and validating source data, intermediate results and statistical outputs developed and managed?

• Are source data, intermediate results and statistical outputs regularly assessed and validated?

• Do procedures and guidelines for data quality assessment exist, and do they address accuracy issues?

• Are procedures and guidelines available on how to measure and reduce errors?

• Does a revision policy that documents the principles and procedures exist and is it made public?

• Are explanations about the timing, reasons for and nature of revisions made available?


• Is a quality assurance plan in place that describes the quality control actions to prevent, monitor and evaluate non-sampling errors?

• Does the revision policy follow standard and transparent procedures in the context of each survey?


• Do statistical procedures (e.g. compilation, data adjustments and transformations, and statistical analysis) employ internationally recognized statistical techniques?

• Are data sources (e.g. registers) systematically checked, and are the data that are used compared with data from other sources?

• Are results compared with other existing sources of information in order to ensure validity?

• Is periodic quality reporting on accuracy, serving both producer and user perspectives, put in place?

• Are methods and tools for preventing and reducing non-sampling errors in place and used?

• Are sampling and non-sampling errors measured, evaluated and systematically documented?

• Are statistical discrepancies in intermediate data assessed and investigated?

• Are preliminary and revised data clearly identified?

• Is information on the size and direction of revisions for key indicators provided and made public?


• Are errors discovered in published statistics corrected at the earliest possible date and publicized?

• Is an analysis of revisions performed and used to improve the statistical process?

3. Quality assurance guidelines - NQAF 15: Assuring accuracy and reliability 39

• Are the sampling and non-sampling estimates analysed over time and improvement actions taken as a result?

• Is user feedback solicited to assess the fitness for purpose of the statistics?


Accuracy and reliability are likely to be more effectively assured if the following supporting mechanisms are in place:

• Standards, guidelines and handbooks for quality assessments and reports are available.

• A documented revision policy exists.


3. Quality assurance guidelines - NQAF 16: Assuring timeliness and punctuality 40

NQAF 16: Assuring timeliness and punctuality

Description:

Statistical agencies should minimize the delays in making data available. Timeliness refers to how fast - after the reference date or the end of the reference period - the data are released or made available, whether for dissemination or for further processing. Punctuality refers to whether data are delivered on the dates promised, advertised or announced (for example, in an official release calendar).


At the agency level

• Is a release policy, distinguishing between different kinds of statistical outputs (press releases, statistics-specific reports or tables, general publications, etc.) and their corresponding release procedures and timeliness targets defined and published?

• Does the timeliness of the statistical agency’s statistics comply with IMF data dissemination standards or other relevant timeliness targets?

• Are action plans developed and followed if the timeliness targets are not met?

• Is there a published release calendar to announce in advance the dates that statistics (at least the major ones) are to be released?

• Is the release calendar made in consultation with users?

• Are procedures in place to regularly monitor and evaluate the punctuality of every release as per the release calendar?

• Are users informed of any divergences from the advance release calendar?

• Are divergences from pre-announced times published in advance, and a new release time announced with explanations on the reasons for the delays?

• Are user requirements taken into account when the periodicity of the statistics is being decided?

• Are statistics made available to all users at the same time?


• Is explicit consideration given to overall trade-offs between timeliness and other dimensions of quality (e.g. accuracy, cost and respondent burden) during the programme design stage?

• Is consideration regularly given to the possibility and usefulness of releasing preliminary data, while at the same time taking into account the data’s accuracy?

• Are contingency plans in place in the event of emerging problems that could delay the release of data?

• Has an attainable schedule been defined for the production processes?

• Is the maximum acceptable amount of time that can elapse - between the end of the reference period and the availability of the data – specified and known to staff and users?


• Are there agreements on the planned delivery dates with data providers?

• Are procedures in place to ensure the effective and timely flow of data from providers?

• Are follow-up procedures in place to ensure timely receipt of data?

• When preliminary data are released, are they clearly identified as such, and are users provided with appropriate information to be able to assess the quality of the preliminary data?

3. Quality assurance guidelines - NQAF 16: Assuring timeliness and punctuality 41

• Does a published policy exist that describes the revisions for those key outputs that are subject to scheduled revisions?


• Are quality indicators on timeliness and punctuality regularly calculated, monitored, published and followed up?


Timeliness and punctuality are likely to be more effectively assured if the following supporting mechanisms are in place:

• A law or other formal provisions exist that requires the setting of a release calendar.

• The public is informed about the statistics being released via release calendars, which also inform them about how the data can be accessed (e.g. through the Internet or in publications).

• A written release or dissemination policy is publicly available.

• Guidelines are available on how to deal with delays when using administrative data for statistical purposes.

• Respondents are made aware and reminded of the deadlines set for reporting.

• Procedures for consulting with users about periodicity are in place.

• Quality indicators on timeliness and punctuality are regularly calculated, monitored and disseminated.


3. Quality assurance guidelines - NQAF 17: Assuring accessibility and clarity 42

NQAF 17: Assuring accessibility and clarity

Description:

Statistical agencies should ensure that the statistics and metadata they develop, produce and disseminate can be found or obtained without difficulty, are presented clearly and in such a way that they can be understood, are available and accessible to all users on an impartial and equal basis in various convenient formats, and are affordable, if not offered free of charge.

Provision should be made for allowing access to microdata for research purposes, in accordance with an established policy which ensures statistical confidentiality.

Supplementary explanatory information and metadata, which are necessary for the proper understanding of the statistics and the appropriate uses to which they can be put, should be made available by the statistical agencies. This information should normally cover the underlying concepts and definitions, origins of the data, the variables and classifications used, the methodology of data collection and processing, and indications of the quality of the statistical information.


At the agency level

• Are policies and processes in place to ensure that the statistical results are released with readily accessible and up-to-date documentation on concepts, scope, classifications, basis of recording, data sources, compilation methods, statistical techniques, etc. to allow for a better understanding of the data?

• Are statistics and the corresponding metadata presented, and archived, in a form that facilitates proper interpretation and meaningful comparisons?

• Are guidelines that describe the appropriate content and preferred formats and style (layout and clarity of text, tables, and charts) of the agency’s outputs available to authors of statistical publications/databases?

• Are staff training and development programmes in place on writing about statistics (for press releases, publication highlights or other explanatory texts)?

• Is the regular production of up-to-date methodological documents (on concepts, scope, classifications, basis of recording, data sources, compilation methods and statistical techniques), as well as quality reports, part of the work programme of the statistical agency, and are the documents and reports made available to the public?

• Does a data dissemination strategy and policy exist, as well as a clear pricing policy (if applicable) governing the dissemination, and are the policies made public?

• Is modern information and communication technology used for dissemination (i.e. statistical databases and the agency’s website as the main means of dissemination of statistics) in addition to traditional hard copy when appropriate?

• Are users able to generate their own tables in the most appropriate formats (xls, html, etc.)?

• Are the statistics disseminated in ways that facilitate re-dissemination by the media?

• Does the agency consult users on a regular basis to find out about the formats of dissemination that they most prefer?

• Are catalogues of publications and other services made available?

• Is there a well-publicized information or user support service, centre or hotline available for handling requests for data and for providing answers to questions about statistical results?


• Is the public made aware that custom-designed outputs, statistics not routinely disseminated, and longer time series can be provided on request when feasible, and are they instructed how the data can be ordered? Are these outputs made public where possible?

• Is access to microdata allowed for research purposes, subject to specific rules and protocols on statistical confidentiality that are posted on the agency’s website?

• Does the agency control or monitor the access by researchers to microdata by providing them in a secure environment?

• Are remote access facilities also available for accessing microdata, with appropriate controls?

• Are researchers consulted regularly about the effectiveness of the microdata access arrangements?


• Has explicit consideration been given to trade-offs between accessibility and confidentiality during the programme design stage?

• Has a strategy been developed and agreed upon with stakeholders for the release of data, metadata and (possibly) microdata from the data collection?

• Are processes in place to ensure that metadata are documented according to standardized metadata systems, and regularly updated?

• Are procedures in place to ensure that any differences from internationally accepted standards, guidelines, or good practices are consistently annotated?


• Is the mix of printed publications, electronic releases and data available on request considered to be appropriate, given the principal user needs?

• Are statistics presented in a clear and understandable manner?

• Are the explanatory texts that accompany the data reviewed for clarity and readability?

• Are meaningful comparisons included in the publications, when appropriate?

• Are the users informed about the methodology of the statistical processes and the use of administrative data?

• Are different levels of metadata detail made available to users to meet their requirements?

• When preliminary data are released, are they clearly identified as such, and are users provided with appropriate information to be able to assess the quality of the preliminary data?

• Are preliminary and revised data identified?

• Are policies in place for archiving statistics and metadata?


• Are assessments undertaken to ensure that the dissemination arrangements are meeting user needs?

• Are user-oriented quality reports made available for statistical results to keep users informed about the quality of the statistical outputs?


Accessibility and clarity are likely to be more effectively assured if the following supporting mechanisms are in place:

• An easily navigated website for searching for and accessing data and metadata and for facilitating self-tabulations in a variety of formats.


• A written dissemination policy which is publicly available.

• User support or information services for assisting users in placing orders or obtaining answers to questions about data.

• User satisfaction surveys, user focus groups or other user consultation mechanisms to solicit feedback on statistical outputs, dissemination formats, etc.

• A publication catalogue for users.

• Methodological documents (on concepts, scope, classifications, basis of recording, data sources, compilation methods and statistical techniques) available to the public.

• Staff training and development programmes for writing about statistics.

• Regularly produced user-oriented quality reports.

• Archiving procedures for statistics and metadata.


3. Quality assurance guidelines - NQAF 18: Assuring coherence and comparability 45

NQAF 18: Assuring coherence and comparability

Description:

Statistical agencies should develop, produce and disseminate statistics that are consistent internally and comparable over time and are produced using common standards with respect to scope, definitions, classifications and units. It should be possible to combine and make joint use of related data from different sources.


At the agency level

• Are procedures or guidelines in place to ensure and monitor internal coherence (e.g. observance of arithmetic and accounting identities) and consistency?

• Are procedures or guidelines in place to ensure and monitor cross-sectoral coherence and consistency?

• Are statistics kept comparable (availability of time series) over a reasonable period of time?

• Is there a common repository of concepts, definitions and classifications available or do other mechanisms exist to promote coherence and consistency?

• Do common standards exist with regard to definitions, units and classifications in order to enhance the comparability of the statistics?

• Is compliance with international or national standards for statistical production periodically assessed?

• Are deviations from international or national standards made explicit and are users informed about the reasons for such deviations?

• To what extent is it possible to compare statistics derived from different sources or with different periodicities (e.g. monthly, quarterly and yearly)? Are any differences explained and reconciled?

• Is cooperation and the exchange of knowledge between individual statistical programmes/domains promoted?


• Are specific procedures and guidelines for individual statistical programmes/domains available to ensure that outputs obtained from complementary sources are properly combined?

• Are the international and national standards concerning definitions, units and classifications known and followed?

• Is the common repository of concepts, definitions and classifications consulted when designing a new individual statistical programme/domain?

• Have the major related statistics been analysed before designing a new individual statistical programme/domain?


• Are process-specific procedures and guidelines available to ensure that outputs are internally coherent?

• Are changes in methods clearly identified and measured to facilitate reconciliation?

• Are breaks in the series explained and the methods for ensuring reconciliation over a period of time made publicly available?


• Are effects of changes in methodologies on final estimates assessed and is appropriate information provided to users?

3. Quality assurance guidelines - NQAF 18: Assuring coherence and comparability 46

• Are statistical outputs compared with other statistical or administrative sources that provide the same or similar information on the same subject matter, and are divergences identified and explained to users?

• Does quality reporting include a section on the assessment of internal consistency and comparability over time and with other subject matter related statistics?


Coherence and comparability are likely to be more effectively assured if the following supporting mechanisms are in place:

• Internal procedures or guidelines are developed in order to ensure and monitor internal coherence and consistency.

• Statistics are kept comparable (availability of time series) over a reasonable period of time. Significant changes in reality are reflected by appropriate changes to concepts, classifications, definitions and target populations.

• A common repository of concepts, definitions and classifications is set up and regularly updated.

• Coherence and comparability is promoted throughout the statistical agency by promoting the adoption of national or international standards.

• The comparability with other related statistics of a new individual programme is assessed before launching it.

• Specific procedures and guidelines are developed in order to ensure that outputs obtained from complementary sources are comparable and can be properly combined. Compliance is periodically assessed.

• Internal cooperation and exchange of know how is promoted and organizational tools are in place (e.g. intranet fora and working groups).


3. Quality assurance guidelines - NQAF 19: Managing metadata 47

NQAF 19: Managing metadata

Description:

Statistical agencies should provide information covering the underlying concepts, variables and classifications used, the methodology of data collection and processing, and indications of the quality of the statistical information - in general, sufficient information to enable the user to understand all of the attributes of the statistics, including their limitations, for informed decision-making.


At the agency level

• Is the metadata management system of the statistical agency well defined and documented?

• Are procedures or guidelines in place for metadata maintenance and dissemination?

• Are metadata documented according to standards?

• Is a glossary of statistical concepts publicly available?

• Are staff training and development programmes in place on metadata management and related information and documentation systems?

• Is there a systematic way for archiving metadata which also ensures that they are accessible for reuse in the future?


The management of metadata is likely to be more effective if the following supporting mechanisms are in place:

• Participation at international metadata forums.

• Programmes to train staff on metadata.

• International, national or internal standards for metadata documentation, management and archiving.


4. Quality assessment and reporting 48

4. Quality assessment and reporting

The systematic quality assessment of a statistical agency’s statistics, both from the statistical production

process and product point of view, permits the agency to control and evaluate the various statistical sub-

processes such as data collection, editing or weighting and thus be confident that the possible problems

will be detected. The purpose is to have mechanisms in place in order to prevent, reduce and evaluate

problems that may arise during the statistical process and affect the statistical products. Having solid

approaches to quality assessment supports the agency’s claims of being professional and credible as a

producer of high quality data.

The aim of this section is to provide a brief outline of what is involved in the assessment or evaluation of

quality, i.e. the set of information on which the quality assessment is based, the different ways in which

quality assessments can be conducted, how they can contribute to standardisation and to continuous

quality improvement within a statistical agency.

Figure 1 shows Eurostat’s tools and approach to

quality assurance, categorized in three layers. On the

way from layer one to three, information on the quality

of the process is increasingly summarised, making it

more appropriate for managers and users. In some

countries, elements of the first layer3 may be all that

can be undertaken, which nevertheless still contributes

to quality improvements. In other cases, the tools and

methods of layers two and three could also be

envisaged.

4a. Measuring product and process quality - use of quality indicators, quality targets and process

variables and descriptions

In order to assess quality, first of all a clear picture of quality concepts applicable to statistical processes

and products is needed; their definition is a precondition. The generic NQAF template defines product

quality in terms of the following components: relevance, accuracy and reliability, timeliness and

punctuality, coherence and comparability, accessibility and clarity.

Quality indicators are specific and measurable elements of statistical practice that can be used to

characterise the quality of statistics. As simplified and generally quantified measures – calculated

3 In its documents, Eurostat refers to the first layer as the “fundamental package”, the second layer as the “intermediate package” and the third layer as the “advanced package”.


according to clear rules – they intend to characterise a complex phenomenon, i.e. the many different

quality features of the data. Quality indicators measure the quality of statistical processes or products

from several aspects, and for example, can give an indication of both output (e.g. timeliness) and process

quality (e.g. response rates). Some product quality indicators are derived from processes, and are called

process variables.

The quality indicators make the description of a product by quality components more informative and

increase transparency. With them, users can assess the quality of different surveys or the same data in

different periods by looking at the quality indicators; they also increase comparability. With regard to

accuracy, for example, quality indicators can provide a direct measure of the impact of errors on the data

(e.g. sampling variance) or be indirect or proxy measures of non-sampling errors source (e.g. frame

errors) derived as a by-product of process monitoring.

Work on defining and developing the quality indicators can be undertaken by survey managers, data

collection specialists and methodologists. Users should also contribute in this phase so that their needs

are also covered by the indicators. After their definition, quality indicators need to be compiled and

analysed by the statistical agency. When using indicators, it is recommended to include qualitative

statements to help to interpret quality information and to summarise the main effects on the usability of

the statistics. Standard quality indicators used across the statistical agency will help with making

comparisons across domains and in supporting quality initiatives. The statistical agency can also set

levels of requirements for the quality indicators in the form of quality targets. The defined targets can then

serve as a tool for monitoring quality developments over time.

For examples of standard quality and performance indicators that could be used, from the producers'

point of view, for summarising the quality of statistical products in various statistical domains, see the list

of indicators, developed by the Eurostat Expert Group on Quality Indicators, in Annex 3.

Process quality is the degree to which a set of inherent characteristics fulfils process requirements. The

generic NQAF template includes four process quality components: methodological soundness, cost-

effectiveness, soundness of implementation, and respondent burden [NQAF10-13]. (See Annex 1 for the

corresponding components in the CoP, DQAF, Latin America and Caribbean Proposal and StatCan

frameworks).

Process quality can be measured by using process variables and process descriptions. Process variables

give an indication of the quality of the process. The key process variables are the variables which have

the largest effect on product characteristics and vary by product quality component and by type of

process. Examples include resources and time used, response rates and response burden and error

rates (in editing).


Measurement of key process variables is the basis for process management and continuous quality

improvement and also provides input to quality indicators and quality reports. Furthermore, a selection of

key process variables will assume an important role in self-assessments and audits as well as in labelling

and certification. The method consists of using quantitative indicators in order to monitor and assess

processes over time and detect sources of error to improve existing processes. It should therefore be

used for established, i.e. repetitive processes.

Process descriptions are qualitative presentations of the statistical agency's processes, which generally

follow a standard structure and cover all or at least the "critical" processes of the statistical agency. With

the help of standardised process documentation, synergies between the different processes can be

found, further IT developments can be facilitated, the implementation of the agency's strategy and vision

can be supported, further methodological standardisation facilitated and the business continuity assured.

4b. Communicating about quality - quality reports

Communicating about the quality of a statistical process or product can be accomplished through the

preparation of reports that review and explain the characteristics of the process and its products.

Because of the multi-dimensional nature of quality, the quality reports typically examine and describe

quality according to those components or dimensions the agency has used to define its products’ fitness

for purpose, e.g. relevance, accuracy, reliability, timeliness, punctuality, coherence, comparability,

accessibility and clarity.

The reports are meant to convey the necessary information to enable users to assess product quality.

While the main target group of a quality report is the users of the statistics, quality reports are also an

important monitoring tool for producers and managers. In the optimal case, the quality reports are

presented according to a standard reporting structure to facilitate comparability and are based on specific

quality indicators.

4c. Obtaining feedback from users

The next element of the set of information that is needed for quality assessments is user feedback.

Together with the producers of the statistics, users are key stakeholders in the data produced. Therefore,

the statistical agency should regularly consult its users about their needs and perceptions of quality, take

them into account in the quality assessment exercise, and follow up on them, for example through

meetings with them (e.g. focus group discussions) or in a more formalised way by using user satisfaction

surveys. Since the main objective of user surveys is normally to get information on the users’ perceptions

as a basis for improvement actions, the results of them provide valuable inputs to self-assessment and

auditing activities. User satisfaction surveys can take different forms, e.g. using standardised


questionnaires, qualitative interviews or web-based surveys, etc. and the choice will depend on the type

of feedback required and on the resources available.

4d. Conducting assessments; labelling and certifica tion

Based on the information collected by the statistical agency using the tools mentioned in the previous

section, the quality of the processes and products can be evaluated and eventually labelled. Evaluation

can be done in the form of self-assessments, audits or peer reviews. It can be undertaken by internal or

external experts and the timeframe can vary from days to months, depending on the scope. However, the

results are more or less identical: the identification of improvement actions/opportunities in processes and

products.

Self-assessments are comprehensive, systematic and regular reviews of an organization's activities and

results referenced against a model/framework. The choice of the self-assessment tool is a strategic

decision and its scope should be clearly defined. For example, it could be applicable to the whole

institutional environment or simply to the statistical production processes. Oftentimes, self-assessment

checklists are developed to be used for systematic assessment of the quality of the statistical production

processes. (See for example, the European Statistical System’s survey assessment tool, DESAP, in

“Selected references” below).

A quality audit is a systematic, independent and documented process for obtaining quality evidence

concerning the quality of a statistical process and evaluating it objectively to determine the extent to

which policies, procedures and requirements on quality are fulfilled. In contrast to the self-assessments,

audits are always carried out by a third party (internal or external to the organization).

Internal audits are conducted with the purpose of reviewing the quality system in place (policies,

standards, procedures and methods) and the internal objectives. They are led by a team of internal

quality auditors who are not in charge of the process or product under review. External audits are

conducted either by stakeholders or other parties that have an interest in the organization, by an external

and independent auditing organization, or by a suitably-qualified expert.

Peer reviews are a type of external audit which aims to assess a statistical process at a higher level, not

to check conformity with requirements item by item from a detailed checklist. It is therefore often more

informal and less structured than an external audit. Normally peer reviews do not address specific

aspects of data quality, but broader organizational and strategic questions. They are typically systematic

examinations and assessments of the performance of one organization by another, with the ultimate goal

of helping the organization under review to comply with established standards and principles, improve its

policy making and adopt best practices. The assessment is conducted on a non-adversarial basis, and


relies heavily on mutual trust among the organization and assessors involved, as well as their shared

confidence in the process.

The results of the assessment/evaluation phase can then be compared to defined standards and

requirements to help to enhance trust and credibility in official statistics. This is often referred to as the

labelling or certification layer.

Labelling of statistics conveys a message about the extent to which a set of quality standards is met. In

the European Statistical System, labelling means compliance with the European Statistics Code of

Practice. The attachment of a label needs a procedure to guarantee that the message is appropriate and

true. The label as such may be brief, e.g. "official statistics", and in this case it would need to be

accompanied by explanations about its interpretation.

Certification is an activity which assesses whether a particular product, service, process or system (e.g. a

quality management system) complies with requirements defined by an internationally recognised

standard or other formal criteria. It is conducted by an external independent certification body. The result

of the successful certification is the certificate awarded to the organization by the certification body, such

as the International Organization for Standardization. Certification to ISO Standards is an advanced

method/tool of process quality management. It requires the documentation, quality reports, quality

indicators, self-assessment and audit, as mentioned in this chapter.

4e. Assuring continuous quality improvement

By implementing a quality approach following the different processes described above, a statistical

agency can begin to define a framework for continuous quality improvement. If the new information on

quality that becomes available is always fed back into the statistical outputs and statistical production

processes, a cycle of continuous improvement of the quality of the statistics produced can be established

as an integral part of the statistical agency's working practices.

Selected references:

• Eurostat, the Revised European Statistics Code of Practice, Luxembourg, 2011 (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/CoP_October_2011.pdf)

• Eurostat, "Handbook on Data Quality Assessment Methods and Tools (DatQAM)", Eurostat granted project led by FSO Germany with project members: Statistics Sweden, Hungarian Central Statistical Office, State Statistical Institute Berlin-Brandenburg, National Statistical Institute of Portugal, and Statistics Norway, 2007 (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting)

• Eurostat, "European Statistical System Quality Assurance Framework (QAF)”, version 1.0 (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/QAF_version_1.0_EN.pdf)

• Eurostat, "The Eurostat Quality Assurance Framework" (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/QAF%20leaflet.pdfIU)

• Eurostat, "ESS Quality Glossary" (http://ec.europa.eu/eurostat/ramon/coded_files/ESS_Quality_Glossary_2010.pdf)


• Eurostat, "ESS Standard and Handbook for Quality Reports", 2009 edition (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting)

• Eurostat, "Handbook on improving quality by analysis of process variables" produced by ONS-UK, INE Portugal, NSS of Greece and Statistics Sweden, 2004 (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting)

• Eurostat, “DESAP, The European Self Assessment Checklist for Survey Managers”, (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/desap%20G0-LEG-20031010-EN.pdf) and (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting (the electronic version))

5. Quality and other management frameworks 54

5. Quality and other management frameworks

Most operations and functions of a statistical agency have an impact on the quality of the agency’s

information. The management of quality is therefore an integral part of the management of every

programme/domain in the agency, and an important component of the agency’s management as a whole.

It is not a separate management function, but an aspect of the management of the agency that has to be

addressed across all programmes/domains in the same way as, for example, financial management or

human resource management.

Frameworks are used to manage a statistical agency’s quality, human resources, financial resources and

overall performance towards achieving its objectives or mandate. They are an effective way to ensure

that its priorities are clearly articulated and considered as a whole rather than in a piecemeal fashion,

which could result in inconsistent and duplicate activities. Frameworks must not be developed and

implemented in isolation, but rather through close interaction. The purpose of this section is not to

describe each of these frameworks, but to highlight how quality should be present in all of them.

5a. Performance management

The performance management framework encompasses all other frameworks. It is through this

framework that the statistical agency achieves its mandate. This mandate should be stated as simply as

possible; for example, “to ensure that citizens have access to a trusted source of statistics”. The mandate

should be accompanied by clear objectives that explicitly include quality; for example, “providing access

to a trusted source of information can be established only if data are relevant and if users are confident

that the information is of the highest possible quality”. In addition, the extent to which a statistical agency

can fulfill its mandate and related objectives depends on its ability to optimize its management and

operations through organizational efficiency.

All the activities of the agency should be aligned with its mandate and related objectives (e.g. access,

relevance, quality and efficiency). The performance management framework should be enabled by

governance mechanisms, a comprehensive human resource strategy and a framework to address any

risk that may prevent the agency from achieving its objectives.

5b. Resource management

A significant feature of the management of quality is the balancing of quality and quantity objectives

against the constraints of financial and human resources. For instance, quality should not be maximized

at all costs. On the other hand, key agency decisions on increasing, reducing or reallocating resources

should take quality into consideration. Like any public organization or business, a statistical agency

should continuously seek ways of increasing its efficiency by reviewing its business practices, every

process that it uses, how it manages its information, the system that it builds, the way it governs its


programmes/domains and organizes itself. It should strive for greater standardization and harmonization,

while leaving sufficient room for the innovative practices that are often demanded by users. Again, quality

considerations should figure prominently in these reviews and all or some of the resources harvested by

these efficiencies should be reinvested towards improving quality in well selected areas.

Effective human resource management is a key factor in achieving the objectives of the statistical

agency. Efforts should be focused on recruitment, training, career advancement and maintaining a

positive workplace. A comprehensive human resource framework aligned with agency objectives and fully

integrated in the agency’s business model is essential to ensure the availability of qualified and talented

staff.

In addition to technical expertise, agency employees should have knowledge of quality issues, and be

able to develop and implement practices and methods to meet quality objectives. Entry level recruitment

will usually aim at hiring employees that are highly knowledgeable and skilled in specialized areas, such

as economic analysis, sampling or project management. The required knowledge and experience in

quality assurance, especially in the context of a statistical agency, is usually obtained by formal training

and working in the agency. For this reason, quality should be an important element in the agency’s

human resource training, development and promotion strategy.

5c. Ethical standards

Ethical standards, fundamental values and principles should guide the personnel of a statistical agency in

fulfilling their official duties and responsibilities. These principles serve to maintain and enhance public

and user confidence in the integrity of the agency. Ethical standards can be applicable at the national

level to all government employees or are can be specific to each agency or institution responsible for the

production of statistical information.

Employees should be guided in their work and their professional conduct by a balanced set of values,

such as:

• To serve the public interest.

• To serve with competence, excellence, efficiency, objectivity and impartiality.

• To act at all times in such a way to uphold the public trust.

• To demonstrate respect and fairness in dealing with both citizens and fellow employees.

Ethical standards could also include topics, such as confidentiality, conflicts of interest, use of information

for personal advantage, acceptance of gifts and the management of public resources. Furthermore,

ethical standards specific to statistical agencies could refer to international standards and guidelines (e.g.

UN Fundamental Principles and the European Statistics Code of Practice).


New and existing employees could be made aware of the agency’s ethical standards and practices for

good behaviour through training programmes or seminars. The code of conduct should be easily

accessible to all employees (disseminated on the internet or intranet, through publications or regular

meetings). The roles of management and staff with respect to standards for behaviour should be clearly

defined. The agency could have an internal ethics board which meets regularly and cases of non-

compliance could be subject to disciplinary action.

5d. Continuous improvement

Continuous improvement is another foundation of any organization. Several models or frameworks for

continuous improvement have been developed for public administration in several countries: European

Union’s Common Assessment Framework, Canada’s Management Accountability Framework and the

United Kingdom’s Capability Review are a few examples. These frameworks provide a self or peer

assessment framework which is conceptually similar to the major Total Quality Management models, but

which is especially designed for public-sector organizations, taking into account their characteristics.

These frameworks include an assessment of the agency’s performance towards achieving its mandate

and objectives. In the context of a statistical agency, this assessment could include many of the indicators

mentioned in the NQAF guidelines: e.g. user satisfaction; time elapsed between reference date and

release date; or accuracy of key estimates. They would also include other indicators related to the

efficiency use of financial resources or to human resources, e.g. employee satisfaction.

5e. Governance

To achieve its mandate and objectives, a statistical agency needs an effective governance and

management structure, which integrates strategic priority setting and decision making and ensures

accountability. This could take the form of a system of agency-level management committees, or other

entities responsible for consultation and recommending strategic options for programme development

and delivery. The membership of these committees should include senior managers from across the

agency. Leadership should be provided by a senior executive committee, which is responsible for

strategic direction and for corporate-level management and decisions. All significant corporate issues are

reviewed at this highest level, with final decisions rendered by the Head of the statistical agency. The

decision-making infrastructure ensures that decisions are based on what is best for the organization to

achieve its mandate and objectives; it encourages innovation and strategies to improve efficiency; and it

increases capacity by enabling the effective integration of issues and initiatives.

Key corporate-level decisions will rarely attempt to advance on all components of its mandate at the same

time or at the same speed. The trade-offs between key objectives, such as relevance, accessibility,

accuracy, confidentiality and cost-efficiency, could be based on a risk management framework. There are


inherent risks in producing statistical information. Inherent risk is the risk that is linked to an activity by the

nature of the activity and by the very fact that the organization performs this activity. Managers should

identify and categorize all inherent and emerging risks related to quality and the agency’s other

objectives. These risks are mitigated by strategies that have been in place over a number of years, based

on the experience and judgment of managers in the statistical agency. What is left is the residual risk.

This is what the agency is most concerned with, and needs to manage continually as well as external risk

factors that are taken into consideration at the corporate level as part of the framework.

To fully measure the residual risk, managers provide their expert assessment of the probability of the risk

materializing and the subsequent impact. They base their assessments on both quantitative information

and subjective assessments derived from their experience. Experts are also asked to estimate the

composite level of residual risk, or risk exposure, by combining the probability of the risk materializing and

the subsequent impact. In planning the allocation of agency resources, an evaluation of the cost and

benefit of various proposals to further mitigate the most important composite level of residual risks could

be completed. The resource planning process therefore assesses return on investment in risk

management activities towards the achievement of the corporate objectives.

Annex 1 - Part 1: Correspondence between the NQAF and the CoP, DQAF, LAC proposal and StatCan 58

ANNEX 1 – Mapping – of NQAF’s section 3 - to exist ing frameworks

Part 1: Correspondence between the Generic National Quality Assurance Framework Template and the CoP, DQAF, LAC proposal and StatCan

Generic National Quality Assurance Framework Template (NQAF)

European Statistics Code of Practice

(CoP)

International Monetary Fund’s Data Quality Assessment

Framework (DQAF)

Latin America & the Caribbean Regional

Code of Good Statistical Practice

(LAC proposal)

Statistics Canada Quality

Assurance Framework (StatCan)


NQAF1. Coordinating the national statistical system

CoP. 2.1 CoP. 2.2 CoP. 2.3 CoP 3.3 CoP 3.4

DQAF. 0.1.1 DQAF. 0.1.2

LAC. 2.1 LAC. 2.2 LAC. 2.3 LAC. 2.4

LAC. 3.1 LAC. 3.2 LAC. 3.3

CAN 2

NQAF2. Managing relationships with data users and data providers

CoP. 2.3 CoP 7.7 CoP 9.1 CoP 9.2 CoP 9.3 CoP 9.4 CoP 9.5

CoP 9.6 CoP 11.1 CoP 11.2 CoP 11.3 CoP 15.6 CoP 15.7

DQAF. 5.3.1 LAC 2.4 LAC 3.3

CAN 1

NQAF3. Managing statistical standards CoP 7.1 CoP 7.2 CoP 10.4 CoP 14.3 CoP 15.5

DQAF. 3.1.2 CAN 11

3b.Managing the institutional environment

NQAF4. Assuring professional independence

CoP. 1.1 CoP. 1.2 CoP. 1.3 CoP. 1.4

CoP. 1.5 CoP. 1.6 CoP. 1.7 CoP. 1.8

DQAF. 1.1.2 DQAF. 1.1.3

LAC. 1.1 LAC. 1.2 LAC. 1.3 LAC. 1.4

LAC. 1.5 LAC. 1.6 LAC. 1.7

CAN 12

NQAF5. Assuring impartiality and objectivity CoP. 6.1 CoP. 6.2 CoP. 6.3 CoP. 6.4

CoP. 6.5 CoP. 6.6 CoP. 6.7 CoP. 6.8

DQAF. 1.1.1 LAC. 7.1 LAC. 7.3 LAC. 7.4 LAC. 7.5

CAN 12

NQAF6. Assuring transparency CoP. 6.3 CoP. 6.4 CoP. 6.5 CoP. 6.6 CoP. 8.6

DQAF. 1.2.1 DQAF. 1.2.2 DQAF. 1.2.3 DQAF. 1.2.4

LAC. 7.1 LAC. 7.2 LAC. 10.5 LAC. 15.3

CAN12

NQAF7. Assuring statistical confidentiality and security

CoP. 5.1 CoP. 5.2 CoP. 5.3

CoP. 5.4 CoP. 5.5 CoP. 5.6

DQAF. 0.1.3

LAC. 4.1 LAC. 4.2 LAC. 4.3 LAC. 4.4

LAC. 4.5 LAC. 4.6 LAC. 4.7

CAN12

NQAF8. Assuring the quality commitment CoP. 4.1 CoP. 4.2 CoP. 4.3 CoP. 4.4

DQAF. 0.4.1 DQAF. 0.4.2 DQAF. 0.4.3

LAC. 6.1 LAC. 6.2 LAC. 6.3 LAC. 6.4

LAC. 6.5 LAC. 8.1 LAC. 8.2 LAC. 8.3

CAN 12

NQAF9. Assuring adequacy of resources

CoP. 3.1 CoP. 3.2 CoP. 3.3 CoP. 3.4


CAN 12

3c.Managing statistical processes

NQAF10. Assuring methodological soundness CoP. 7.1 CoP. 7.2 CoP. 7.3 CoP. 7.4 CoP. 7.5 CoP. 7.6

CoP. 7.7 CoP 8.2 CoP 8.3 CoP 8.4 CoP. 8.5 CoP. 8.6

DQAF. 2.1 DQAF. 2.2 DQAF. 2.3 DQAF. 2.4

LAC. 9.1 LAC. 9.2 LAC. 9.3 LAC. 9.4 LAC. 9.5

CAN 4 CAN 11

Annex 1 - Part 1: Correspondence between the NQAF and the CoP, DQAF, LAC proposal and StatCan 59

Generic National Quality Assurance Framework Template (NQAF)

European Statistics Code of Practice

(CoP)

International Monetary Fund’s Data Quality Assessment

Framework (DQAF)

Latin America & the Caribbean Regional

Code of Good Statistical Practice

(LAC proposal)

Statistics Canada Quality

Assurance Framework (StatCan)

NQAF11.

Assuring cost-effectiveness CoP. 10.1 CoP. 10.2 CoP. 10.3 CoP. 10.4


NQAF12. Assuring soundness of implementation CoP. 8.1 CoP. 8.2 CoP. 8.3 CoP. 8.4

CoP. 8.5 CoP. 8.7 CoP. 8.8 CoP. 8.9

DQAF. 3.1.1 DQAF. 3.3.1 DQAF. 3.3.2

LAC. 10.1 LAC. 10.2 LAC. 10.3

LAC. 10.4 LAC. 10.5 LAC. 10.6

NQAF13. Managing the respondent burden CoP. 8.7 CoP. 8.8 CoP. 8.9 CoP. 9.1 CoP. 9.2

CoP. 9.3 CoP. 9.4 CoP. 9.5 CoP. 9.6

DQAF. 0.1.4 LAC. 11.1 LAC. 11.2 LAC. 11.3

LAC. 11.4 LAC. 11.5

CAN 10

3d.Managing statistical outputs

NQAF14. Assuring relevance CoP. 11.1 CoP. 11.2 CoP. 11.3


CAN 3

NQAF15. Assuring accuracy and reliability CoP. 8.6 CoP. 12.1 CoP. 12.2 CoP. 12.3



LAC. 10.5 LAC. 14.1 LAC. 14.2 LAC. 14.3

LAC. 14.4 LAC. 14.5

CAN 4

NQAF16. Assuring timeliness and punctuality CoP. 13.1 CoP. 13.2 CoP. 13.3 CoP. 13.4 CoP. 13.5



CAN 5

NQAF17. Assuring accessibility and clarity CoP. 15.1 CoP. 15.2 CoP. 15.3 CoP. 15.4

CoP. 15.5 CoP. 15.6 CoP. 15.7


DQAF. 5.2.1 DQAF. 5.2.2 DQAF. 5.3.2



CAN 6 CAN 7

NQAF18. Assuring coherence and comparability CoP. 14.1 CoP. 14.2 CoP. 14.3 CoP. 14.4 CoP. 14.5

DQAF. 4.2.1 DQAF. 4.2.2 DQAF. 4.2.3

LAC. 16.1 LAC. 16.2 LAC. 16.3 LAC. 16.4

CAN 8

NQAF19. Managing metadata CoP. 15.1 CoP. 15.5

CAN 13

Annex 1 - Part 2: Correspondence between the NQAF and the CoP 60

Part 2: Correspondence between the Generic National Quality Assurance Framework Template and the European Statistics Code of Practice (CoP)

Generic National Quality Assurance Framework Template

(NQAF) European Statistics Code of Practice (CoP)

3a.Managingthe statistical system

NQAF 1: Coordinating the national statistical system

CoP 2.1: The mandate of the statistical authorities to collect information for the development, production and dissemination of European Statistics is specified in law. CoP 2.2: The statistical authorities are allowed by law to use administrative data for statistical purposes. CoP 2.3: On the basis of a legal act, the statistical authorities may compel response to statistical surveys. CoP 3.3: Procedures exist to assess and justify demands for new statistics against their cost. CoP 3.4: Procedures exist to assess the continuing need for all statistics, to see if any can be discontinued or curtailed to free up resources.


CoP 2.3: On the basis of a legal act, the statistical authorities may compel response to statistical surveys. CoP 7.7: Co-operation with the scientific community is organised to improve methodology, the effectiveness of the methods implemented and to promote better tools when feasible. CoP 9.1: The range and detail of European Statistics demands is limited to what is absolutely necessary. CoP 9.2: The reporting burden is spread as widely as possible over survey populations. CoP 9.3: The information sought from businesses is, as far as possible, readily available from their accounts and electronic means are used where possible to facilitate its return. CoP 9.4: Administrative sources are used whenever possible to avoid duplicating requests for information. CoP 9.5: Data sharing within statistical authorities is generalised in order to avoid multiplication of surveys. CoP 9.6: Statistical authorities promote measures that enable the linking of data sources in order to reduce reporting burden. CoP 11.1: Processes are in place to consult users, monitor the relevance and utility of existing statistics in meeting their needs, and consider their emerging needs and priorities. CoP 11.2: Priority needs are being met and reflected in the work programme. CoP 11.3: User satisfaction is monitored on a regular basis and is systematically followed up. CoP 15.6: Users are kept informed about the methodology of statistical processes including the use of administrative data. CoP 15.7: Users are kept informed about the quality of statistical outputs with respect to the quality criteria for European Statistics.


CoP 7.1: The overall methodological framework used for European Statistics follows European and other international standards, guidelines, and good practices. CoP 7.2: Procedures are in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical authority. CoP 10.4: Statistical authorities promote and implement standardized solutions that increase effectiveness and efficiency. CoP 14.3: Statistics are compiled on the basis of common standards with respect to scope, definitions, units and classifications in the different surveys and sources. CoP 15.5: Metadata are documented according to standardised metadata systems.

http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/CoP_October_2011.pdf






CoP 1.1: The independence of the National Statistical Institutes and Eurostat from political and other external interference in developing, producing and disseminating statistics is specified in law and assured for other statistical authorities. CoP 1.2: The heads of the National Statistical Institutes and of Eurostat and, where appropriate, the heads of other statistical authorities have sufficiently high hierarchical standing to ensure senior level access to policy authorities and administrative public bodies. They are of the highest professional calibre. CoP 1.3: The heads of the National Statistical Institutes and of Eurostat and, where appropriate, the heads of other statistical authorities have responsibility for ensuring that statistics are developed, produced and disseminated in an independent manner. CoP 1.4: The heads of the National Statistical Institutes and of Eurostat and, where appropriate, the heads of other statistical authorities have the sole responsibility for deciding on statistical methods, standards and procedures, and on the content and timing of statistical releases. CoP 1.5: The statistical work programmes are published and periodic reports describe progress made. CoP 1.6: Statistical releases are clearly distinguished and issued separately from political/policy statements. CoP 1.7: The National Statistical Institute and Eurostat and, where appropriate, other statistical authorities, comment publicly on statistical issues, including criticisms and misuses of statistics as far as considered suitable. CoP 1.8: The appointment of the heads of the National Statistical Institutes and Eurostat and, where appropriate, of other statistical authorities, is based on professional competence only. The reasons on the basis of which the incumbency can be terminated are specified in the legal framework. These cannot include reasons compromising professional or scientific independence.

NQAF 5:Assuring impartiality and objectivity

CoP 6.1: Statistics are compiled on an objective basis determined by statistical considerations. CoP 6.2: Choices of sources and statistical methods as well as decisions about the dissemination of statistics are informed by statistical considerations. CoP 6.3: Errors discovered in published statistics are corrected at the earliest possible date and publicised. CoP 6.4: Information on the methods and procedures used is publicly available. CoP 6.5: Statistical release dates and times are pre-announced. CoP 6.6: Advance notice is given on major revisions or changes in methodologies. CoP 6.7: All users have equal access to statistical releases at the same time. Any privileged pre-release access to any outside user is limited, controlled and publicised. In the event that leaks occur, pre-release arrangements are revised so as to ensure impartiality. CoP 6.8: Statistical releases and statements made in press conferences are objective and non-partisan.

NQAF 6: Assuring transparency CoP 6.3: Errors discovered in published statistics are corrected at the earliest possible date and publicised. CoP 6.4: Information on the methods and procedures used is publicly available. CoP 6.5: Statistical release dates and times are pre-announced. CoP 6.6: Advance notice is given on major revisions or changes in methodologies. CoP 8.6: Revisions follow standard, well-established and transparent procedures.

NQAF 7: Assuring statistical confidentiality and security

CoP 5.1: Statistical confidentiality is guaranteed in law. CoP 5.2: Staff sign legal confidentiality commitments on appointment. CoP 5.3: Penalties are prescribed for any wilful breaches of statistical confidentiality. CoP 5.4: Guidelines and instructions are provided to staff on the protection of statistical confidentiality in the production and dissemination processes. The confidentiality policy is made known to the public. CoP 5.5: Physical, technological and organisational provisions are in place to protect the security and integrity of statistical databases. CoP 5.6: Strict protocols apply to external users accessing statistical microdata for research purposes.


CoP 4.1: Quality policy is defined and made available to the public. An organizational structure and tools are in place to deal with quality management. CoP 4.2: Procedures are in place to plan and monitor the quality of the statistical production process. CoP 4.3: Product quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics. CoP 4.4: There is a regular and thorough review of the key statistical outputs using also external experts where appropriate.






CoP 3.1: Staff, financial, and computing resources, adequate both in magnitude and in quality, are available to meet current statistical needs. CoP 3.2: The scope, detail and cost of statistics are commensurate with needs. CoP 3.3: Procedures exist to assess and justify demands for new statistics against their cost. CoP 3.4: Procedures exist to assess the continuing need for all statistics, to see if any can be discontinued or curtailed to free up resources.



CoP 7.1: The overall methodological framework used for European Statistics follows European and other international standards, guidelines, and good practices. CoP 7.2: Procedures are in place to ensure that standard concepts, definitions and classifications are consistently applied throughout the statistical authority. CoP 7.3: The business register and the frame for population surveys are regularly evaluated and adjusted if necessary in order to ensure high quality. CoP 7.4: Detailed concordance exists between national classifications systems and the corresponding European systems. CoP 7.5: Graduates in the relevant academic disciplines are recruited. CoP 7.6: Statistical authorities implement a policy of continuous vocational training for their staff. CoP 7.7: Co-operation with the scientific community is organised to improve methodology, the effectiveness of the methods implemented and to promote better tools when feasible. CoP 8.2: In the case of statistical surveys, questionnaires are systematically tested prior to the data collection. CoP 8.3: Survey designs, sample selections and estimation methods are well based and regularly reviewed and revised as required. CoP 8.4: Data collection, data entry, and coding are routinely monitored and revised as required. CoP 8.5: Appropriate editing and imputation methods are used and regularly reviewed, revised or updated as required. CoP 8.6: Revisions follow standard, well-established and transparent procedures.


CoP 10.1: Internal and independent external measures monitor the statistical authority’s use of resources. CoP 10.2: The productivity potential of information and communications technology is being optimised for data collection, processing and dissemination. CoP 10.3: Proactive efforts are made to improve the statistical potential of administrative data and to limit recourse to direct surveys. CoP 10.4: Statistical authorities promote and implement standardized solutions that increase effectiveness and efficiency.


CoP 8.1: When European Statistics are based on administrative data, the definitions and concepts used for administrative purposes are a good approximation to those required for statistical purposes. CoP 8.2: In the case of statistical surveys, questionnaires are systematically tested prior to the data collection. CoP 8.3: Survey designs, sample selections and estimation methods are well based and regularly reviewed and revised as required. CoP 8.4: Data collection, data entry, and coding are routinely monitored and revised as required. CoP 8.5: Appropriate editing and imputation methods are used and regularly reviewed, revised or updated as required. CoP 8.7: Statistical authorities are involved in the design of administrative data in order to make administrative data more suitable for statistical purposes. CoP 8.8: Agreements are made with owners of administrative data which set out their shared commitment to the use of these data for statistical purposes. CoP 8.9: Statistical authorities co-operate with owners of administrative data in assuring data quality.






CoP 8.7: Statistical authorities are involved in the design of administrative data in order to make administrative data more suitable for statistical purposes. CoP 8.8: Agreements are made with owners of administrative data which set out their shared commitment to the use of these data for statistical purposes. CoP 8.9: Statistical authorities co-operate with owners of administrative data in assuring data quality. CoP 9.1: The range and detail of European Statistics demands is limited to what is absolutely necessary. CoP 9.2: The reporting burden is spread as widely as possible over survey populations. CoP 9.3: The information sought from businesses is, as far as possible, readily available from their accounts and electronic means are used where possible to facilitate its return. CoP 9.4: Administrative sources are used whenever possible to avoid duplicating requests for information. CoP 9.5: Data sharing within statistical authorities is generalised in order to avoid multiplication of surveys. CoP 9.6: Statistical authorities promote measures that enable the linking of data sources in order to reduce reporting burden.


NQAF 14: Assuring relevance CoP 11.1: Processes are in place to consult users, monitor the relevance and utility of existing statistics in meeting their needs, and consider their emerging needs and priorities. CoP 11.2: Priority needs are being met and reflected in the work programme. CoP 11.3: User satisfaction is monitored on a regular basis and is systematically followed up.


CoP 8.6: Revisions follow standard, well-established and transparent procedures. CoP 12.1: Source data, intermediate results and statistical outputs are regularly assessed and validated. CoP 12.2: Sampling errors and non-sampling errors are measured and systematically documented according to the European standards. CoP 12.3: Revisions are regularly analysed in order to improve statistical processes.


CoP 13.1: Timeliness meets European and other international release standards. CoP 13.2: A standard daily time for the release of statistics is made public. CoP 13.3: The periodicity of statistics takes into account user requirements as much as possible. CoP 13.4: Divergence from the dissemination time schedule is publicised in advance, explained and a new release date set. CoP 13.5: Preliminary results of acceptable aggregate accuracy can be released when considered useful.


CoP 15.1: Statistics and the corresponding metadata are presented, and archived, in a form that facilitates proper interpretation and meaningful comparisons. CoP 15.2: Dissemination services use modern information and communication technology and, if appropriate, traditional hard copy. CoP 15.3: Custom-designed analyses are provided when feasible and the public is informed. CoP 15.4: Access to microdata is allowed for research purposes and is subject to specific rules or protocols. CoP 15.5: Metadata are documented according to standardised metadata systems. CoP 15.6: Users are kept informed about the methodology of statistical processes including the use of administrative data. CoP 15.7: Users are kept informed about the quality of statistical outputs with respect to the quality criteria for European Statistics.


CoP 14.1: Statistics are internally coherent and consistent (i.e. arithmetic and accounting identities observed). CoP 14.2: Statistics are comparable over a reasonable period of time. CoP 14.3: Statistics are compiled on the basis of common standards with respect to scope, definitions, units and classifications in the different surveys and sources. CoP 14.4: Statistics from the different sources and of different periodicity are compared and reconciled. CoP 14.5: Cross-national comparability of the data is ensured within the European Statistical System through periodical exchanges between the European Statistical System and other statistical systems. Methodological studies are carried out in close co-operation between the Member States and Eurostat.





NQAF 19: Managing metadata CoP 15.1: Statistics and the corresponding metadata are presented, and archived, in a form that facilitates proper interpretation and meaningful comparisons. CoP 15.5: Metadata are documented according to standardised metadata systems.


Annex 1 - Part 3 Correspondence between the NQAF and the DQAF 65

Part 3: Correspondence between the Generic National Quality Assurance Framework Template and the International Monetary Fund’s Data Quality Assessment Framework (DQAF)


(NQAF) International Monetary Fund’s Data Quality Assessment Framework (DQAF)



0.1.1 The responsibility for collecting, processing, and disseminating the statistics is clearly specified. 0.1.2 Data sharing and coordination among data-producing agencies are adequate.


5.3.1 Contact points for each subject field are publicized.


3.1.2 Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required.



1.1.2 Choices of sources and statistical techniques as well as decisions about dissemination are informed solely by statistical considerations. 1.1.3 The appropriate statistical entity is entitled to comment on erroneous interpretation and misuse of statistics.

NQAF5: Assuring impartiality and objectivity

1.1.1 Statistics are produced on an impartial basis.

NQAF 6: Assuring transparency 1.2.1 The terms and conditions under which statistics are collected, processed, and disseminated are available to the public. 1.2.2 Internal governmental access to statistics prior to their release is publicly identified. 1.2.3 Products of statistical agencies/units are clearly identified as such. 1.2.4 Advanced notice is given of major changes in methodology, source data, and statistical techniques.

NQAF7: Assuring statistical confidentiality and security

0.1.3 Individual reporters’ data are to be kept confidential and used for statistical purposes only.

NQAF8: Assuring the quality commitment

0.4.1 Processes are in place to focus on quality. 0.4.2 Processes are in place to monitor the quality of the statistical program. 0.4.3 Processes are in place to deal with quality considerations in planning the statistical program.

NQAF9: Assuring adequacy of resources

0.2.1 Staff, facilities, computing resources, and financing are commensurate with statistical programs.



2.1 Concepts and definitions — Concepts and definitions used are in accord with internationally accepted statistical frameworks. 2.2 Scope — The scope is in accord with internationally accepted standards, guidelines, or good practices. 2.3 Classification/sectorization — Classification and sectorization systems are in accord with internationally accepted standards, guidelines, or good practices. 2.4 Basis for recording — Flows and stocks are valued and recorded according to internationally accepted standards, guidelines, or good practices.


0.2.2 Measures to ensure efficient use of resources are implemented.


3.1.1. Source data are collected from comprehensive data collection programs that take into account country-specific conditions. 3.3.1 Data compilation employs sound statistical techniques to deal with data sources. 3.3.2 Other statistical procedures (e.g., data adjustments and transformations, and statistical analysis) employ sound statistical techniques.


0.1.4 Statistical reporting is ensured through legal mandate and/or measures to encourage response.


NQAF14: Assuring relevance 0.3.1 The relevance and practical utility of existing statistics in meeting users’ needs are monitored.

http://dsbb.imf.org/images/pdfs/dqrs_factsheet.pdf

Annex 1 - Part 3 Correspondence between the NQAF and the DQAF 66


(NQAF) International Monetary Fund’s Data Quality Assessment Framework (DQAF)

NQAF15: Assuring accuracy and reliability

3.2.1 Source data—including censuses, sample surveys, and administrative records— are routinely assessed, e.g., for coverage, sample error, response error, and nonsampling error; the results of the assessments are monitored and made available to guide statistical processes. 3.4.1 Intermediate results are validated against other information where applicable. 3.4.2 Statistical discrepancies in intermediate data are assessed and investigated. 3.4.3 Statistical discrepancies and other potential indicators of problems in statistical outputs are investigated.

NQAF15: Assuring accuracy and reliability

3.5.1 Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes (see also 4.3.3). 4.3.1 Revisions follow a regular and transparent schedule. 4.3.2 Preliminary and/or revised data are clearly identified. 4.3.3 Studies and analyses of revisions are made public (see also 3.5.1).

NQAF16: Assuring timeliness and punctuality

3.1.3 Source data are timely. 4.1.1 Periodicity follows dissemination standards. 4.1.2 Timeliness follows dissemination standards. 5.1.3 Statistics are released on a preannounced schedule.

NQAF17: Assuring accessibility and clarity

5.1.1 Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts). 5.1.2 Dissemination media and format are adequate. 5.1.4 Statistics are made available to all users at the same time. 5.1.5 Statistics not routinely disseminated are made available upon request. 5.2.1 Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences from internationally accepted standards, guidelines, or good practices are annotated. 5.2.2 Levels of detail are adapted to the needs of the intended audience. 5.3.2 Catalogs of publications, documents, and other services, including information on any charges, are widely available.

NQAF18: Assuring coherence and comparability

4.2.1 Statistics are consistent within the dataset. 4.2.2 Statistics are consistent or reconcilable over a reasonable period of time. 4.2.3 Statistics are consistent or reconcilable with those obtained through other data sources and/or statistical frameworks.

NQAF19: Managing metadata

http://dsbb.imf.org/images/pdfs/dqrs_factsheet.pdf

Annex 1 - Part 4 Correspondence between the NQAF and the LAC proposal 67

Part 4: Correspondence between the Generic National Quality Assurance Framework Template and the Latin America and the Caribbean Regional Code o f Good Statistical Practice (LAC proposal)


(NQAF) Latin America and the Caribbean Regional Code of Good Statistical Practice (LAC proposal)



LAC 2.1. The existence of the NSS is specified in national law which designates the managing and coordinating body for the purposes of rationalizing the country's statistical activities. LAC 2.2. The managing and coordinating body lays down methodological guidelines when drawing up statistical plans and programs for the relevant entities of the national statistical system. LAC 2.3. The managing and coordinating body establishes parameters for participation and responsibility for their entities. LAC 2.4. Within the National Statistical System are expert advisory committees with operating regulations that allow better communication in spite of the main users of official statistics. LAC 3.1 The legislation in force grants the National Statistical Office and the other members of the National Statistical Systems a mandate for the collection of information destined for the preparation and dissemination of official statistics. LAC 3.2 The legislation in force specifies that persons and enterprises are obliged to supply information to the National Statistical Office and the other members of National Statistical Systems and provides for penalties for failure to comply with the law. LAC 3.3 The legislation grants the National Statistical Office and the other members of the National Statistical Systems access to and use of administrative registers for producing official statistics.


LAC 2.4. Within the National Statistical System are expert advisory committees with operating regulations that allow better communication in spite of the main users of official statistics. LAC 3.3 The legislation grants the National Statistical Office and the other members of the National Statistical Systems access to and use of administrative registers for producing official statistics.


3b.Managing the institutional environment


LAC 1.1. The legislation in force specifies that the members of the national statistical system must compile and disseminate statistics independently of political influence and other external interference. LAC 1.2. The director of managing and coordinating body has sufficiently high hierarchical standing to ensure that he has senior-level access to political authorities, public organizations and national and international bodies. LAC 1.3. The head of statistical service must be a person of high professional calibre and with expert knowledge of the development and dissemination of official statistics. LAC 1.4. Statistics disseminated by the members of the national statistical system are clearly distinguished from and issued separately from policy statements. LAC 1.5. The National Statistical Office must have a statistics committee made up of a group of experts of professional calibre who advise on the general policy and strategic plans of the national statistical system. LAC 1.6. The head of the National Statistical Office and the heads of the other bodies of the national statistical system are solely responsible for deciding on statistical methods, standards and procedures and on the content and timing of statistical releases. LAC 1.7. When appropriate, the heads of the statistical services of the national statistical system issue public statements on statistical issues, which include criticisms and address misuses of official statistics.

http://www.dane.gov.co/files/noticias/BuenasPracticas_en.pdf





LAC 7.1 Statistical operations and research are implemented using methodologies and processes which are documented, are based on impartiality and transparency, and pursue clearly established aims. LAC 7.3 Selecting information sources, methods, processes, concepts and data dissemination paths is a professional responsibility and is based on national and international principles and best practice taking into account the implications in terms of cost. LAC 7.4 There are guidelines which ensure that all users have access at the same time to statistical releases and there is a timetable in which the date and time of statistical releases are announced in advance. LAC 7.5 The results of statistical operations are presented in an objective and professional manner. The information is made known in an impartial manner which is comprehensible for all users.

NQAF 6: Assuring transparency LAC 7.1 Statistical operations and research are implemented using methodologies and processes which are documented, are based on impartiality and transparency, and pursue clearly established aims. LAC 7.2 The standards, classifications, methods and processes used in the production of statistics (design, collection, processing and release) are documented and made available to the public. LAC 10.5. The revisions follow standard processes and consolidate in accordance with the schedule and comments that may be required. The studies and analysis of revisions are made available to specialized users. LAC 15.3. a specific date and time is laid down for the release of all statistics. Any changes to the dissemination time schedule are made known in advance with explained and a new release date is set.

NQAF 7: Assuring statistical confidentiality and security

LAC 4.1 The legislation specifies that data are confidential and prohibits use for any purpose other than statistical ones (commerce, taxation, judicial investigation, etc.). LAC 4.2 The staff involved in producing official statistics signs a confidentiality undertaking which is enshrined in law and includes penalties for non-compliance. LAC 4.3 There are legal standards and undertakings for the confidentiality of information for the staff involved in the production of official statistics; these are made known to all staff and penalties for non-compliance are stipulated. LAC 4.4 There are protocols for maintaining strict security and integrity of statistical databases. LAC 4.5 Respondents are informed of the main uses and limits to access which apply to the information that they supply for the production of official statistics. LAC 4.6 Access to microdata is subject to strict confidentiality protocols for external users who access them for the purposes of analysis and statistical research. LAC 4.7 The bodies which produce statistics archive any material which could be of historical interest, subject to security, confidentiality and legal obligations. Statistics are thus treated as valuable and irreplaceable given that their value will increase with use and over time.


LAC 6.1 There is a clearly defined policy and quality model which is known to the members of the national statistical system. LAC 6.2 The quality of statistical products is assessed periodically in compliance with internal guidelines and international standards. LAC 6.3 The National Statistical Office and the members of the national statistical system document and make known quality policies and strategic objectives at internal level for the purposes of compliance. LAC 6.4 A culture of constant improvement is promoted and fostered systematically and involves documentation of methodology and processes and exchanges of good statistical practice to assess monitor and improve the quality of statistical operations. LAC 6.5 There are documented processes for assessing and monitoring the quality of the design, production, analysis and dissemination of statistics which produce suitable registers to validate monitoring. LAC 8.1 the National Statistical Office and the other members of the NSS participate in international activities of statistical interest which are supported by international bodies LAC 8.2 the National Statistical Office and the other members of the NSS carry out process of cooperation on knowledge-sharing with international organisations LAC 8.3 there are activities related to international cooperation through the development of different knowledge transfer mechanism






LAC 5.1 Sufficient human, financial, physical and technological resources are available for national production of statistics in order to meet the need for statistical information. LAC 5.2 The scope, details, cost and time for producing official statistics are based on an analysis of the needs for information. LAC 5.3 The National Statistical Office and the other members of the national statistical system have control mechanisms for assessing and justifying requests for new official statistics in relation to their cost. LAC 5.4 Those in charge of producing statistics periodically assess statistical operations to determine whether they should be continued or discontinued or new ones introduced in order to optimize the use of resources.



LAC 9.1 Work is done on coordinating implementation of the concepts, classifications and good practice which follow principles and guidelines accepted nationally and internationally for application in all statistical operations. LAC 9.2 the methodology of surveys and the use of administrative registers are evaluated periodically and adjusted when necessary to guarantee high quality of the products. LAC 9.3 There is detailed concordance between national and international benchmark classifications prepared by the competent bodies. LAC 9.4 At the design stage of statistical operations, flexibility is permitted only in the design of the acquisition and processing instruments for responding to changes in users' needs for information. LAC 9.5 There are academic, interinstitutional and sectoral committees to improve and assess the methodology by means of external quality and efficiency reviews of the methods applied and by promoting adoption of better tools where feasible.


LAC 12.1 Internal and independent external measures are deployed to monitor the efficient use of resources of the bodies producing statistics. LAC 12.2 Proactive efforts are made to improve the statistical potential of administrative records and avoid costly direct surveys. LAC 12.3 National legislation permits members of the NSS to share data on enterprises households and government and to reduce the costs of collection without violating statistical confidentiality. LAC 12.4 Information and communication technologies are used to optimize the processes of producing and disseminating official statistics.


LAC 10.1 T he questionnaires, manuals and computer applications are tested and validated before the process of data collection is started. LAC 10.2 Computerised systems are used in the processes of capturing, encoding and validation of the information and periodic revisions and updates are carried out as provided for. LAC 10.3 The stages in the statistical process are reviewed or updated by statistical operations and research require. LAC 10.4 Appropriate technological resources are used for editing and imputation and are regularly reviewed or updated as provided for by the design. LAC 10.5 The revisions follow standard processes and consolidate in accordance with the schedule and comments that may be required. The studies and analysis of revisions are made available to specialized users. LAC 10.6 The concepts and definitions applied when administrative registers are used for statistical purposes must comply with the parameters required in a quality statistical process.


LAC 11.1 The National Statistical Office has coordinated and systematic procedures for obtaining information from enterprises’ accounts (financial information) and, where possible, electronic means are used to facilitate forwarding of information. LAC 11.2 The scope and detail of the information required from respondents for statistical operations is limited to what is strictly necessary. LAC 11.3 A constant effort is made to use and/or develop appropriate sampling techniques to reduce the burden on respondents. LAC 11.4 Shared use of data between producers of statistics is promoted in order to avoid duplicating requests and production of information. LAC 11.5 Best estimates and approximations are accepted when exact information is not available.





3d.Managing statistical outputs (including metadata)

NQAF 14: Assuring relevance LAC 13.1 The strategic statistics available at national level are based on the priority information needs of the government, enterprises and the public at large, taking into account the available resources. To this end, mechanisms and strategies are available in keeping with statistical plans, sectoral round tables, interinstitutional workshops, surveys on satisfaction and feedback from users. LAC 13.2 There are mechanisms to advise, empower and inform users with regard to new information requirements and priorities and to regularly consult them on the practical use of current statistical operations. LAC 13.3 The National Statistical Systems and the other members of the NSS and the users participate in processes of coordination, analysis and assessment of information requirements and document them via interinstitutional and sectoral committees. LAC 13.4 User satisfaction surveys are conducted periodically.


LAC 10.5 The revisions follow standard processes and consolidate in accordance with the schedule and comments that may be required. The studies and analysis of revisions are made available to specialized users. LAC 14.1 Source data, intermediate results and statistical output are assessed and validated, and compared with other statistical information where necessary. LAC 14.2 Analyzes and document the sampling error, and non-sampling. LAC 14.3 Source data are collected in line with the methodology and design published to guarantee reliability. LAC 14.4 Regular reviews of the statistical process are undertaken to improve documentation thereof. LAC 14.5 Methodologies are updated periodically to comply with quality criteria for producing official statistics and to bring them up to international standards.


LAC 15.1 The timeliness of statistics produced should be reflected in the time between your reference period and their availability, considering the framework of time, ensure that information be useful for different users. LAC 15.2 The periodicity of dissemination of statistical operations is established takes account as far as possible of the requirements, needs of users, international standards and commitments in this area. LAC 15.3 a specific date and time is laid down for the release of all statistics. Any changes to the dissemination time schedule are made known in advance with explained and a new release date is set. LAC 15.4 Any significant error identified in statistics released is corrected and published as soon as possible LAC 15.5 Any substantial updates of the methodology, process, microdata or statistical techniques undertake are announced in advance.


LAC 6.3 The National Statistical Office and the members of the national statistical system document and make known quality policies and strategic objectives at internal level for the purposes of compliance. LAC 7.4 There are guidelines which ensure that all users have access at the same time to statistical releases and there is a timetable in which the date and time of statistical releases are announced in advance. LAC 7.5 The results of statistical operations are presented in an objective and professional manner. The information is made known in an impartial manner which is comprehensible for all users. LAC 17.1 Users are guaranteed free and equal access to official statistics on the basis of clearly established and familiar procedures. LAC 17.2 Statistics are released in such a manner as to enable their contents to be expressed with clarity and precision to users and facilitate proper interpretation and meaningful comparisons. LAC 17.3 The results of statistical output are released in accordance with a timetable for dissemination which is published in advance. LAC 17.4 Statistics are disseminated by means of various media and modern technologies guaranteeing maximum coverage. LAC 17.5 The use of statistics is promoted by preparing and making available didactic material to persons from the press and users in general. LAC 17.6 As far as possible, technical support is offered for the analysis of data at the request of users and in accordance with agreements which are made public. LAC 17.7 Draft documents, special working documents and methodologies are published and advances and progress made are described in reports where feasible.






LAC 16.1 Statistics are internally coherent and reconcilable over a period of time. LAC 16.2 Statistics produced from administrative registers are consistently when it use classifications, definitions and concepts required for a quality statistical process. LAC 16.3 Joint statistical frameworks, definitions, classifications, procedures, indicators, concepts and best practices are promoted and used in all statistical operations in order to increase comparability over time and between sets of data. LAC 16.4 Comparability at national and international level of statistics is promoted.



Annex 1 - Part 5 Correspondence between the NQAF and StatCan QAF 72

Part 5: Correspondence between the Generic National Quality Assurance Framework Template and the Statistics Canada Quality Assurance Framework ( StatCan)

Generic National Quality Assurance Framework

Template (NQAF)

Statistics Canada Quality Assurance Framework as presented in the Statistical Commission document E/CN.3/2010/2



CAN.2 Coordinating the national statistical system – protocols, standards.


CAN.1 Managing user and stakeholder relationships – user satisfaction surveys, feedback mechanisms, councils.


CAN.11 Managing statistical infrastructure – standards, registers, policies



CAN.12 Managing institutional infrastructure -. confidentiality, security, transparency, professional independence, impartiality, objectivity

NQAF5: Assuring impartiality and objectivity


NQAF 6: Assuring transparency CAN.12 Managing institutional infrastructure -. confidentiality, security, transparency, professional independence, impartiality, objectivity

NQAF7: Assuring statistical confidentiality and security


NQAF8: Assuring the quality commitment


NQAF9: Assuring adequacy of resources




CAN.4 Managing accuracy – design, accuracy assessment, quality control, revision policy. CAN.11 Managing statistical infrastructure – standards, registers, policies




CAN.10 Managing provider relationships – response burden measurement and reduction, response rate maintenance.


NQAF 14: Assuring relevance CAN.3 Managing Relevance – program review, planning process, data analysis.


CAN.4 Managing accuracy – design, accuracy assessment, quality control, revision policy.


CAN.5 Managing timeliness and punctuality – advanced release dates, preliminary/final releases.


CAN.6 Managing accessibility – product definition, dissemination practices, search facilities. CAN.7 Managing interpretability/clarity – concepts, sources, methods, informing users of quality.

NQAF18: Assuring coherence and comparability

CAN.8 Managing coherence and comparability – standards, harmonized concepts and methods.

NQAF19: Managing metadata CAN.13 Managing metadata – relating to quality.

http://unstats.un.org/unsd/statcom/doc10/2010-2-NQAF-E.pdf

Annex 2 - Selected references 73

ANNEX 2 – SELECTED REFERENCES for each NQAF line 4

NQAF 1: Coordinating the national statistical syst em

African Development Bank (AfDB), Intersect and Part nership in Statistics for Development in the 21st Century (PARIS21) • Mainstreaming sectoral statistical systems in Africa - A guide to planning a coordinated national statistical system,

Version 1.0 (http://www.paris21.org/sites/default/files/intersect-final-en.pdf)

Australia • Quality Management of Statistical Outputs Produced From Administrative Data, Mar 2011 – Common Issues with

Acquiring Administrative Data: - Legislation (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features5Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

- System issues (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features11Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

- Political issues (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features13Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

Botswana • Coordination of National Statistical Systems and Reporting Mechanisms for MDG Data to International Agencies,

Botswana experience, May 2008 (http://unstats.un.org/unsd/mdg/Resources/Attach/Capacity/Uganda/Uganda08%20Presentations/7%20May/Session%20on%20coordination%20&%20reporting/3%20-%20Botswana%20paper.doc)

Canada • Characteristics of an Effective Statistical System (Dr. I. P. Fellegi)

(http://unstats.un.org/unsd/dnss/docViewer.aspx?docID=190#start) • STATISTICS ACT, 3(a), 3(b), 3(c), 10, 11, 12 (http://www.statcan.gc.ca/about-apercu/act-loi-eng.htm)

China • About the National Bureau of Statistics of China (http://www.stats.gov.cn/english/aboutnbs.htm)

Colombia • Cartillas de difusión de los instrumentos del SEN

(http://www.dane.gov.co/daneweb_V09/index.php?option=com_content&view=article&id=759&Itemid=150) • Circular COINFO 003 de 2010: uso de clasificaciones y nomenclaturas

(http://www.dnp.gov.co/PortalWeb/LinkClick.aspx?fileticket=B4Hwjsapu7o%3d&tabid=1094) • Decreto 262 del 2004- DANE, articulo 1° ( http://www.dane.gov.co/files/acerca/Normatividad/decreto_262.pdf) • Decreto 3851/2006 Artículos 2°, 4°, 5°, 6°, 7° y 8° : Infraestructura Colombiana de Datos

(http://unstats.un.org/unsd/dnss/docs-nqaf/Colombia-d3851.pdf) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,

May 2011 (http://www.dane.gov.co/files/noticias/BuenasPracticas_en.pdf) • Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales

(http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf)

Eurostat • Regulation 223/2009 defines the functioning of the European Statistical System including the coordination role of

NSIs with regard to the Code of Practice (http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-31-09-254/EN/KS-31-09-254-EN.PDF)

France • Official statistical system practices in accordance with the principles of the European Statistics Code of Practice –

Institutional environment: 2- Mandate for data collection (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe2.htm)

• The three institutions of official statistics in France (http://www.insee.fr/fr/publications-et-services/sommaire.asp?codesage=COUHS2011&nivgeo=0)

4 For an alphabetical listing of references by country or organization, see the NQAF website at http://unstats.un.org/unsd/dnss/QualityNQAF/nqaf.aspx.

iliana.vaca-trigo

Line


Italy • Italian Code of Official Statistics (Gazz. Uff. 13/10/2010, n.240) (in Italian)

(http://www.sistan.it/inbreve/Codice_statistico_quarta_edizione.pdf) • Law concerning the National Statistical System and the National Statistical Institute reorganization (Norme sul

Sistema statistico nazionale e sulla riorganizzazione dell'Istituto nazionale di statistica D.Lgs. 06/09/1989, n. 322 (Gazz. Uff. 22 settembre 1989, n. 222) (in Italian) (http://www.sistan.it/norme/322.html)

• Regulation concerning Istat reorganization (Regolamento recante il riordino dell’Istituto nazionale di statistica) DPR 07/09/ 2010 , n. 166 (in Italian) (http://www.sistan.it/norme/dpr166_2010.pdf)

Rwanda • The National Statistical System, National Institute of Statistics of Rwanda (NISR)

(http://statistics.gov.rw/index.php?option=com_content&task=view&id=170&Itemid=221)

South Africa • South African Statistical Quality Assessment Framework (SASQAF), Second edition, Statistics South Africa, 2010,

Section 1.2, prerequisites of quality p. 7 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1.3, prerequisites of quality p. 5 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

Switzerland • Swiss Public Statistics Charter, Feb 2008, and in revision

(http://www.bfs.admin.ch/bfs/portal/en/index/institutionen/oeffentliche_statistik/ethische_prinzipien/charta_2002.html)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 8 - National Co-ordination

(http://unstats.un.org/unsd/goodprac/bpaboutpr.asp?RecId=8) • Handbook of Statistical Organization, Third Edition – The Operation and Organization of a Statistical Agency,

UNSD, 2003 paras. 24 – 28, 50 – 69, 270 – 283, 359 – 361, 391 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)


Australia • ABS Data Quality Framework

(http://www.abs.gov.au/AUSSTATS/[email protected]/Latestproducts/1520.0Main%20Features1May%202009?opendocument&tabname=Summary&prodno=1520.0&issue=May%202009&num=&view=)

Austria • Responsibilities and Principles (http://www.statistik.at/web_en/about_us/responsibilities_and_principles/index.html)

Canada • Statistics Canada Consultations with data users in the context of program reviews (http://www.statcan.gc.ca/about-

apercu/consultations2009-2010-eng.htm) • Statistics Canada Quality Guidelines, Fifth edition, 2009 Chapters 1 (Section 1.3.1) and 15

(http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf) • Statistics Canada’s Quality Assurance Framework, 2002 pp. 5-6 (http://www.statcan.gc.ca/pub/12-586-x/12-586-

x2002001-eng.pdf) • Statistics Canada Training services and workshops (http://www.statcan.gc.ca/ads-annonces/10c0013/training-

formation-eng.htm)

Colombia • Cartillas de difusión de los instrumentos del SEN

(http://www.dane.gov.co/daneweb_V09/index.php?option=com_content&view=article&id=759&Itemid=150) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,



European Central Bank (ECB) • Quality Assurance Procedures within the ECB Statistical Function, Chapter 8

(http://www.ecb.int/pub/pdf/other/ecbstatisticsqualityassuranceprocedure200804en.pdf?b23ee77d0dba0cd6f51d61c053c98fd2)


Finland • Courses and tailored training (http://www.stat.fi/tup/tilauskoulutus/index_en.html)

Italy • Constitution of CNUIS – the National Committee of users of statistical information

South Africa • South African Statistical Quality Assessment Framework (SASQAF), Second edition, Statistics South Africa, 2010

Section 5 accessibility: 5.3 p. 37 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf) • SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 5 accessibility:

5.3 pp. 38 and 5.8 p. 41 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Nations Economic Commission for Europe (UNEC E) • Making Data Meaningful Part 3: A Guide to Communicating with the Media

(http://www.unece.org/stats/documents/writing)

United Nations Statistics Division (UNSD) • Handbook of Statistical Organization, Third Edition – The Operation and Organization of a Statistical Agency,

UNSD, 2003, paras. 159 – 225 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)


Australia • ABS National Statistical Service (NSS) Handbook, Chapter 10 Statistical Infrastructure

(http://www.nss.gov.au/nss/home.nsf/NSS/35BFD39E0E2A8597CA25763F000B622C?opendocument)

Canada • Statistics Canada Policy on Standards (http://www.statcan.gc.ca/about-apercu/policy-politique/standards-normes-

eng.htm)

France • INSEE’s web site : Definitions and methods (http://www.insee.fr/en/methodes/default.asp)

Japan • Master Plan Concerning the Development of Official Statistics, March 13, 2009: Establishment of statistical

standards pp. 14 – 15 (http://www.stat.go.jp/english/index/seido/pdf/2009mp.pdf) • Statistical Act (Act No. 53 of May 23, 2007), Article 27, p. 11 and Article 28, p.12

(http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf)


Section 1 prerequisites of quality: 1.2 p. 7, 1.7 p. 12, Section 4 timeliness: 4.3, p. 32 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1 prerequisites of quality: 1.3 p. 5, Section 4 timeliness: 4.3, p. 36 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)


UNSD, 2003, (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)


Australia, • Australian Bureau of Statistics Data Quality Framework, May 2009 (Cat. no. 1520.0), section on Institutional

Environment (http://www.abs.gov.au/AUSSTATS/[email protected]/Latestproducts/1520.0Main%20Features3May%202009?opendocument&tabname=Summary&prodno=1520.0&issue=May%202009&num=&view=)

Canada • Statistics Canada The Daily releases (http://www.statcan.gc.ca/dai-quo/a-daily-quotidien-eng.htm) • Statistics Canada The Daily release schedule for key economic indicators (http://www.statcan.gc.ca/release-

diffusion/index-eng.htm)


Colombia • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,

May 2011 (http://www.dane.gov.co/files/noticias/BuenasPracticas_en.pdf)

Czech Republic • Statistical Service Act

(http://www.czso.cz/eng/redakce.nsf/i/full_wording_of_act_no_89_1995_coll_on_the_state_statistical_service)

Denmark • Addressing erroneous or misconstrued interpretation of figures by the news media

(http://unstats.un.org/unsd/dnss/docViewer.aspx?docID=168#start)

Eurostat • Regulation 223/2009 defines professional independence (http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-

31-09-254/EN/KS-31-09-254-EN.PDF)


Institutional environment: 1- Professional Independence (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe1.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007, paras. 3.163 – 3.190

(http://dsbb.imf.org/images/pdfs/gddsguide.pdf)


Section 9, integrity: 9.5 p. 54 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf) • SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 9, integrity: 9.2-

9.4 and pp. 59-61 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics -Principle 4 - Prevention of misuse


UNSD, 2003, paras. 5 – 10, 76-111 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)


Canada • Statistics Canada Integrated Business and Human Resources Plan (http://www.statcan.gc.ca/about-

apercu/plan2010-2013/pdf/plan2010-2013-eng.pdf) • Statistics Canada Statistical Methods Research and Development Program... Achievements

(http://www.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=12-206-X&CHROPG=1&lang=eng) • Statistics Canada The Daily release schedule for key economic indicators (http://www.statcan.gc.ca/release-

diffusion/index-eng.htm)

Chile • Policy on Data Dissemination National Statistics Institute (INE-CHILE) (http://unstats.un.org/unsd/dnss/docs-

nqaf/Chile-manual_de_difusion_2009_eng.pdf)

Colombia • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,



Estonia • Dissemination Policy of Statistics Estonia (http://www.stat.ee/dissemination-policy)

Eurostat • Protocol on impartial access for users

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Impartiality%20protocol%20REV2_FINAL_EN.pdf)


France • Official statistical system practices are in accordance with the principles of the European Statistics Code of Practice

– Institutional environment: 6- Impartiality and objectivity (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe6.htm)

Italy • Italian Code of Official Statistics (Gazz. Uff. 13/10/2010, n.240) (in Italian)

(http://www.sistan.it/inbreve/Codice_statistico_quarta_edizione.pdf) • Release calendar (http://en.istat.it/salastampa/appuntamenti/ and http://www.istat.it)

South Africa • South African Statistical Quality Assessment Framework (SASQAF), Second edition, Statistics South Africa, 2010 ,

Section 9, integrity p. 52 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf) • SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 9, integrity 9.4

p. 61 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 1 - Relevance, impartiality and equal access

(http://unstats.un.org/unsd/goodprac/bpaboutpr.asp?RecId=1)

United States • Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by

Federal Agencies, Office of Management and Budget, Executive Office of the President (http://www.whitehouse.gov/omb/fedreg_reproducible/)

NQAF 6: Assuring transparency

Canada • Statistics Canada Corporate Business Plan (http://www.statcan.gc.ca/about-apercu/cbp-pae/pdf/cbp-pea-eng.pdf) • Statistics Canada Departmental Performance Report (http://www.tbs-sct.gc.ca/dpr-rmr/2009-2010/inst/stc/stc-

eng.pdf) • Statistics Canada Plans and Priorities 2011-2012 (http://publications.gc.ca/collections/collection_2011/sct-tbs/BT31-

2-2012-III-68-eng.pdf) • Statistics Canada Publications by Subject (http://cansim2.statcan.gc.ca/cgi-

win/cnsmcgi.pgm?Lang=E&ResultTemplate=/Stu-Etu/Stu-Etu3&ChunkSize=25&AS_Theme=0&ChunkStart=1&AS_Date=.&AS_Ser=.&AS_Auth=.&AS_Srch=&AS_SORT=0&AS_UNIV=3&Version=2&AS_Mode=2)

• Statistics Canada Quality Guidelines, Fifth edition, 2009, Chapter 16 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

• Statistics Canada The Daily release schedule for key economic indicators (http://www.statcan.gc.ca/release-diffusion/index-eng.htm)

Chile • Policy on Data Dissemination of the National Statistics Institute (http://unstats.un.org/unsd/dnss/docs-nqaf/Chile-

manual_de_difusion_2009_eng.pdf)

Colombia • Ley 489 de 1998, artículo 32. Audiencias públicas

(http://www.alcaldiabogota.gov.co/sisjur/normas/Norma1.jsp?i=186) • Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales


Estonia • Dissemination Policy of Statistics Estonia (http://www.stat.ee/dissemination-policy)

Eurostat • Protocol on impartial access for users

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Impartiality%20protocol%20REV2_FINAL_EN.pdf)

France • The full monthly diary and the four-month calendar of major economic indicators

(http://www.insee.fr/en/publics/default.asp?page=presse/presse.htm)


Hungary • HCSO,s Strategy 2009 –2012, Quality Statistics, Methodological developments in order to improve the quality of

statistical products, p. 18) (http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/0_DOCS/hu/strategy_2009_2012.pdf)

International Monetary Fund (IMF) • Data Quality Assessment Framework (http://dsbb.imf.org/Pages/DQRS/DQAF.aspx) • The General Data Dissemination System, Guide for participants and users, IMF, 2007, paras. 3.163 – 3.203

(http://dsbb.imf.org/images/pdfs/gddsguide.pdf) • The Special Data Dissemination Standard, Guide for subscribers and users, IMF, 2007, paras. 7.5 – 7.26

(http://www.imf.org/external/pubs/ft/sdds/guide/2007/eng/sddsguide.pdf)

Italy • Release calendar (http://en.istat.it/salastampa/appuntamenti/ and http://www.istat.it) • Service chart (in Italian) (http://www.istat.it/it/supporto/per-gli-utenti/carta-dei-servizi)

Japan • Statistical Act (Act No. 53 of May 23, 2007), Article 8 p. 5 (http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf) • Master Plan Concerning the Development of Official Statistics, March 13, 2009: Impartiality of statistics p. 34

(http://www.stat.go.jp/english/index/seido/pdf/2009mp.pdf)

Netherlands • Checklist Quality of Statistical Output, Chapter 17 - Remaining quality dimensions pp. 55-57

(http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)

Portugal • Dissemination Policy, Official Statistics, Statistics Portugal, 2008

(http://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=55229723&att_display=n&att_download=y) • Revisions Policy, Statistics Portugal, December 2008

(http://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=70086429&att_display=n&att_download=y)

South Africa • South African Statistical Quality Assessment Framework (SASQAF), Second edition, Statistics South Africa,

2010,Section 8.4, methodological soundness p. 51(http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 8.2, methodological soundness p. 53 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)


UNSD, 2003, paras. 477 – 485 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)



NQAF 7: Assuring statistical confidentiality and s ecurity

Australia • ABS National Statistical Service (NSS) Handbook, Confidentiality and Privacy

(http://www.nss.gov.au/nss/home.nsf/NSS/5D5BDDB294AF7D3DCA25763F0007B7E0?opendocument) • Quality Management of Statistical Outputs Produced From Administrative Data, March 2011 – Common Issues with

Acquiring Administrative Data, Confidentiality and Consent (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features17Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

Canada • Canada's Statistics Act (http://www.statcan.gc.ca/about-apercu/act-loi-eng.htm) • Record linkage at Statistics Canada (http://www.statcan.gc.ca/record-enregistrement/index-eng.htm) • Statistics Canada Definitions, data sources and methods (http://www.statcan.gc.ca/concepts/index-eng.htm) • Statistics Canada Information for survey participants (http://www.statcan.gc.ca/survey-enquete/index-eng.htm) • Statistics Canada Policy on informing survey respondents (http://www.statcan.gc.ca/about-apercu/policy-

politique/info_survey-enquete-eng.htm) • Statistic Canada Privacy Notice (http://www.statcan.gc.ca/reference/privacy-privee-eng.htm)


• Statistics Canada Quality Guidelines, Fifth edition, 2009 Chapters 7 (Section 7.3.3) and 15 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

• Statistics Canada Research Data Centre (http://www.statcan.gc.ca/rdc-cdr/index-eng.htm)

Colombia • Ley 79/93 Articulo 5°, inciso 2°: Reserva estadísti ca.

(http://www.dane.gov.co/files/acerca/Normatividad/Ley79_1993.pdf) • Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-

nqaf/Assurance%20quality%20methodology.pdf) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,



Eurostat • Guidance and references on microdata and confidentiality

(http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/statistical_confidentiality/confidential_data/introduction)

• Handbook on Statistical Disclosure Control (2007) (http://neon.vb.cbs.nl/casc/handbook.htm) • Handbook on Statistical Disclosure Control. Version 1.2. ESSNet SDC (2010)

(http://neon.vb.cbs.nl/casc/.%5CSDC_Handbook.pdf)

France • Guide to statistical confidentiality, INSEE, 2009 (http://www.insee.fr/en/insee-statistique-publique/statistique-

publique/guide.pdf) • Official statistical system practices in accordance with the principles of the European Statistics Code of Practice –

Institutional environment: 5- Statistical Confidentiality (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe5.htm)

International Household Survey Network • Tools and Guidelines on Data Anonymization

(http://www.internationalsurveynetwork.org/HOME/index.php?q=tools/anonymization) • “Dissemination of Microdata Files: Principles, Procedures and Practices”, Olivier Dupriez and Ernie Boyko,

International Household Survey Network Working Paper No. 5, August 2010, Chapter 4, pp 19 – 23 and Chapter 6, pp 26 – 32 (http://www.internationalsurveynetwork.org/home/index.php?q=focus/dissemination-microdata-files-principles-procedures-and-practices)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007, paras. 3.163 – 3.190

(http://dsbb.imf.org/images/pdfs/gddsguide.pdf)

Italy • Handbook on Methodologies and techniques for Statistical Disclosure Control: Istat (2004) Metodologie e tecniche

di tutela della riservatezza nel rilascio di informazione statistica. Metodi e Norme, N. 20 (in Italian) (http://www3.istat.it/dati/catalogo/20040706_00/manuale-tutela_riservatezza.pdf)

Japan • Statistical Act (Act No. 53 of May 23, 2007), Articles 41, 42 and 43, pp. 16 - 18

(http://www.stat.go.jp/english/index/seido/pdf/stlaw.pdf)

Netherlands • Checklist Quality of Statistical Output, Chapter 16 - Confidentiality of a statistic, pp. 53-54

(http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)


Section 1, prerequisite of quality, 1.4, p. 8; Section 5, accessibility, 5.3, p. 37; Section 9, integrity, 9.1, p. 52 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1, prerequisite of quality, 1.4, p.7; Section 5, accessibility, 5.2-5.3, p. 39; Section 9, integrity, 9.1-9.2, p. 59 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Nations Economic Commission for Europe (UNEC E) • Managing Statistical Confidentiality and Microdata Access: Principles and Guidelines of Good Practice

(http://live.unece.org/fileadmin/DAM/stats/publications/Managing.statistical.confidentiality.and.microdata.access.pdf)


• Principles and Guidelines on Confidentiality Aspects of Data Integration Undertaken for Statistical or Related Research Purposes (http://live.unece.org/fileadmin/DAM/stats/publications/Confidentiality_aspects_data_integration.pdf)

• Statistical Confidentiality and Access to Microdata, Proceedings of the Seminar Session of the 2003, Conference of European Statisticians (http://www.unece.org/fileadmin/DAM/stats/publications/statistical.confidentiality.pdf)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics -Principle 6 – Confidentiality


UNSD, 2003, paras. 522-528, 535 – 562 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)






2-2012-III-68-eng.pdf)

Colombia • Decreto 3851/2006 Artículo 3°: Certificación de cal idad.

(http://www.dane.gov.co/files/acerca/Normatividad/d3851.pdf) • Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-

nqaf/Assurance%20quality%20methodology.pdf) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,


(http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf) • Resolución 691 de 2011: proceso de certificación de calidad

(http://www.dane.gov.co/files/acerca/Normatividad/Resolucion691_2011.pdf)

Eurostat • DESAP - The European Self Assessment Checklist for Survey Managers

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/desap%20G0-LEG-20031010-EN.pdf) • Handbook on Data Quality Assessment Methods and Tools (DatQAM Manual)

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20DATA%20QUALITY%20ASSESSMENT%20METHODS%20AND%20TOOLS%20%20I.pdf)

• Handbook on improving quality by analysis of process variables (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20IMPROVING%20QUALITY.pdf)


Institutional environment: 4- Quality commitment (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe4.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007. Quality Dimension of

Disseminated Data paras. 3.155, 3.162 (http://dsbb.imf.org/images/pdfs/gddsguide.pdf)


Section 9 integrity: 9.2 p. 53, Section 1 prerequisite of quality 1.8 p. 13 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 9 integrity: 9.3 p. 62, Section 1 prerequisite of quality 1.8 p. 13 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)


United Nations Economic Commission for Europe (UNEC E) • Generic Statistical Business Process Model (http://www.unece.org/stats/gsbpm)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 2 -Quality management

(http://unstats.un.org/unsd/goodprac/bpcountryfrm.asp?selKeyWord=12)



eng.pdf) • Statistics Canada Integrated Business and Human Resources Plan (http://www.statcan.gc.ca/about-

apercu/plan2010-2013/pdf/plan2010-2013-eng.pdf) • Statistics Canada Plans and Priorities 2011-2012 (http://publications.gc.ca/collections/collection_2011/sct-tbs/BT31-

2-2012-III-68-eng.pdf)

Colombia • Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-

nqaf/Assurance%20quality%20methodology.pdf)


Institutional environment: 3- Adequacy of resources (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe3.htm)

Japan • Master Plan Concerning the Development of Official Statistics, March 13, 2009; Efficient production of statistics and

securing and making effective use of statistical resources p. 5; Ideals for securing and allocating statistical resources and effective use of statistical resources p. 23 (http://www.stat.go.jp/english/index/seido/pdf/2009mp.pdf)


Section 1 prerequisite of quality 1.6, p. 9 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1 prerequisite of quality 1.6, p. 9 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)


Canada • Statistics Canada Survey Methods and Practices (http://www.statcan.gc.ca/pub/12-587-x/12-587-x2003001-

eng.pdf)

Colombia • Guía para la Elaboración de Documentos Metodológicos Estándar de las Operaciones Estadísticas del DANE

(http://unstats.un.org/unsd/dnss/docs-nqaf/Standard%20Guide%20to%20document%20of%20statistical%20operations%20methodologies.pdf)

• Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-nqaf/Assurance%20quality%20methodology.pdf)

• Metodología para el fortalecimiento de registros administrativos (http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos)

• Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean, May 2011 (http://www.dane.gov.co/files/noticias/BuenasPracticas_en.pdf)

• Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales (http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf)

Eurostat • ESS methodological documents

(http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/methodology/ess_methodological_documents)

Finland • Quality Guidelines for Official Statistics, 2nd Revised Edition, 2007 (http://unstats.un.org/unsd/dnss/docs-

nqaf/Finland-qg_2ed_en.pdf)



Statistical procedures: 7- Sound methodology (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe7.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007, Appendix II. International

Guidelines for selected Data Categories paras. 3.155 - 3.162 (http://dsbb.imf.org/images/pdfs/gddsguide.pdf)

Italy • Quality guidelines for statistical processes, Istat (2011)

Mexico • Inventory on International Statistical Standards (ISS) (http://mapserver.inegi.org.mx/estandares/Index1.cfm)

Netherlands • Checklist for the Quality evaluation of Administrative Data Sources

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Checklist%20for%20the%20quality%20evaluation%20of%20administrative%20d.pdf)


Section 8 methodological soundness 8.1 p. 41, (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 8 methodological soundness 8.1 p. 54 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

Spain • AYUDACOD: Ayuda a la Codificación (http://www.ine.es/EX_INICIOAYUDACOD)

United Nations Statistics Division (UNSD) • Methods and classifications website (http://unstats.un.org/unsd/methods.htm) • Fundamental Principles of Official Statistics - Principle 2 – Professionalism



Canada • Statistics Canada Departmental Performance Report (http://www.tbs-sct.gc.ca/dpr-rmr/2009-2010/inst/stc/stc-


2-2012-III-68-eng.pdf) • Treasury Board Management Accountability Framework (http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp) • Treasury Board Policy on the Management of Projects (http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12077)

Colombia • Metodología para el fortalecimiento de registros administrativos

(http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,



Eurostat • Evaluation of activities and user satisfaction in Eurostat

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/evaluation/general_evaluation_results) • Evaluation in the European Commission (http://ec.europa.eu/dgs/secretariat_general/evaluation/index_en.htm)


Statistical procedures: 10- Cost effectiveness (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe10.htm)

Organisation for Economic Co-operation and Developm ent (OECD) • International Standard Cost Model Manual (http://www.oecd.org/dataoecd/32/54/34227698.pdf)



Section 1, prerequisite of quality, 1.6.7, p. 11 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1, prerequisite of quality, 1.6.7, p. 13 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

Ukraine • National Principles of the State Statistical Activity – Principle 13: Cost effectiveness (http://ukrstat.gov.ua/)


Canada • Statistics Canada Quality Assurance Reviews 2008 (http://www.statcan.gc.ca/pub/12-594-x/12-594-x2007001-

eng.htm)


nqaf/Assurance%20quality%20methodology.pdf) • Metodología para el fortalecimiento de registros administrativos

(http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos) • Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales

(http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf) • Resolución 036 del 18 de enero de 2006, por la cual se dispone sobre el funcionamiento de CANDANE

(http://www.dane.gov.co/candane/files/Resolucion_036.pdf)

Eurostat • Code of Practice Compliance Activities

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Compliance_activities_EN_Feb_2011.pdf ) • DESAP - The European Self Assessment Checklist for Survey Managers

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/desap%20G0-LEG-20031010-EN.pdf) • Description of activities to comply with the European Statistics Code of Practice

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/code_of_practice/compliance) • Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical

System, 2006 (http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/documents/Handbook_questionnaire_development_2006.pdf)


Statistical procedures: 8- Appropriate statistical procedures (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe8.htm)

Italian National Statistical Institute (ISTAT), Sta tistics Netherlands (CBS), Swiss Federal Statistica l Office (SFO), Eurostat • Recommended Practices for Editing and Imputation in Cross-Sectional Business Surveys, ISTAT, CBS, SFO,

Eurostat (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/RPM_EDIMBUS.pdf)

Italy • Quality guidelines for statistical processes, Istat (2011)

Netherlands • Checklist for the Quality evaluation of Administrative Data Sources, Statistics Netherlands, 2009

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Checklist%20for%20the%20quality%20evaluation%20of%20administrative%20d.pdf)


Section 8.2, methodological soundness p. 48 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 1 prerequisite of quality 1.6.1 p. 9 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)



Australia • Quality Management of Statistical Outputs Produced From Administrative Data, Mar 2011 - Uses of Administrative Data: Reducing response burden (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features2Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=#Reducing%20respondent%20burden)

- Common Issues with Acquiring Administrative Data Influencing the collection (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features9Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

- Management of Relationships with Data Custodians (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features22Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

Austria • Reducing respondents’ burden

(http://www.statistik.at/web_en/about_us/responsibilities_and_principles/reducing_respondents_burden/index.html)

Canada • Best Practices in Public Opinion Research: Improving Respondent Cooperation for Telephone Surveys

(http://www.tpsgc-pwgsc.gc.ca/rop-por/rapports-reports/telephone/etape-stage-01-eng.html) • Statistics Canada Information for survey participants (http://www.statcan.gc.ca/survey-enquete/index-eng.htm) • Statistics Canada Policy on informing survey respondents (http://www.statcan.gc.ca/about-apercu/policy-

politique/info_survey-enquete-eng.htm)






Eurostat • Handbook for Monitoring and Evaluating Business Survey Response Burden, 2007

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20FOR%20MONITORING%20AND%20EVALUATING%20BUSINESS%20SURVEY%20R.pdf)


Statistical procedures: 9- Non-excessive burden on respondents (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe9.htm)

Italy • Section "Information for respondents" on Istat website (http://www.istat.it/en/support/respondents)

New Zealand • Respondent Load Strategy (http://www.stats.govt.nz/about_us/policies-and-guidelines/respondent-load-

strategy.aspx)


Section 1 prerequisites of quality, section 1.3., p. 8 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 3, paragraph under section 3.2, p. 26; Section 1, paragraph under section 1.3, p. 7; paragraph under section 1.5, p. 8; Section 2, paragraph under section 2.4, p. 18, p. 21 bullet number 5 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

Sweden, Norway and United Kingdom • Developing Methods for Assessing Perceived Response Burden, by Dan Hedlin, Statistics Sweden Trine Dale and

Gustav Haraldsen, Statistics Norway Jacqui Jones, Office for National Statistics, UK, 2005. (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/DEVELOPING%20METHODS%20FOR%20ASSESSING%20PERCEIVED%20RESPONSE%20BURD.pdf)


United Kingdom • National statistics code of practice: protocol on managing respondent load

(http://www.statistics.gov.uk/about_ns/cop/downloads/respondentload.pdf)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 5 - Minimize reporting burden

(http://unstats.un.org/unsd/goodprac/bpcountryfrm.asp?selKeyWord=16)

NQAF 14: Assuring relevance

Australia • Data Quality Online: Hints and Tips - Defining A Data Need

(http://www.nss.gov.au/dataquality/PDFs/DQO_Needs.pdf)

Canada • Statistics Canada Quality Guidelines, Fifth edition, 2009;planning process: Chapters 1, 2, 3, 5 (Sections 5.3.2 and

5.4), 7 (Section 7.4.1);data analysis: Chapter 17;data quality evaluation: Chapter 14 (Section 14.4.2); data dissemination: Chapter 16 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

• Statistics Canada’s Quality Assurance Framework, 2002, Chapter 4, pp. 5-9; program review: pp. 6-7; planning process: pp. 8-9; data analysis: pp. 7-8 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-586-x2002001-eng.pdf)






Eurostat • Evaluation of statistical work programmes

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/evaluation/general_evaluation_results) • Measuring Customer Satisfaction, a methodological guidance 2006

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/CUSTOMER%20SATISFACTION%20SURVEYS_SE_2006_EN_1.pdf)

• ESS Handbook for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-EHQR_FINAL.pdf)

• ESS Standard for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-ESQR_FINAL.pdf)

• Euro-IND Monitoring report (http://epp.eurostat.ec.europa.eu/portal/page/portal/euroindicators/publications/monitoring_report)


Statistical results: 11- Relevance (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe11.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007.Coverage paras. 3.7 – 3.14;

Real sector: National accounts paras. 3.26 – 3.30, 3.62 – 3.63, Production index/ indices paras. 3.65 - 3.67, Price indices paras. 3.71-3.77, Purchasing power parity paras. 3.78 – 3.79, Labor market paras. 3.80 – 3.90; Fiscal sector: Central government operations paras. 3.32 – 3.45, 3.91 – 3.99; Financial sector paras. 3.47 – 3.52, 3.100 – 3.108; External sector paras. 3.54 – 3.60, 3.109 – 3.121; Sociodemographic data paras. 3.122 – 3.150;Data Dimensions of the GDDS Table 3.1 pp. 33 – 38 (http://dsbb.imf.org/images/pdfs/gddsguide.pdf)

Japan • Guideline of "Quality Assurance" of the Official Statistics of Japan (http://www.stat.go.jp/english/index/seido/pdf/qa_gl.pdf ) • Master Plan Concerning the Development of Official Statistics, March 13, 2009, Continuous comprehension and

utilization of statistical need p. 29; Reexaminations and streamlining evaluations of statistics p. 30 (http://www.stat.go.jp/english/index/seido/pdf/2009mp.pdf)


Netherlands • Checklist Quality of Statistical Output, Chapter 14 - Extent of detail of a statistic, Chapter 15 - Completeness of a

statistic, pp. 50-52 (http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)


Section 2: Under relevance p. 14 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 2: Under relevance p. 15 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Kingdom • Guidelines for measuring statistical quality, pp. 14-17, 18, 23, 27, 61, 63-64, 78-79

(http://unstats.un.org/unsd/dnss/docs-nqaf/UK-Guidelines_Subject.pdf)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 1 - Relevance, impartiality and equal access



Australia: • Data Quality Online: Hints and Tips - Defining A Data Need

(http://www.nss.gov.au/dataquality/PDFs/DQO_Needs.pdf) • Quality Management of Statistical Outputs Produced From Administrative Data, Mar 2011 – Uses of Administrative

Data: - Data validation (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features2Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=#Data%20validation)

- Data editing (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features2Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=#Editing)

- Data imputation and substitution (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features2Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=#Imputation%20and%20substitution)

Canada • Statistics Canada Quality Assurance Reviews 2008 (http://www.statcan.gc.ca/pub/12-594-x/12-594-x2007001-

eng.htm) • Statistics Canada Quality Guidelines, Fifth edition, 2009,survey strategy: Chapters 3, 4, 5; data processing:

Chapters 8, 9, 10, 11, 12, 17; compilation: Chapters 6 and 7; accuracy assessment: Section 7.3.2, Chapter 13; quality control: Section 5.3.7, Section 6.3.4; data quality evaluation: Chapter 14 9 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

• Statistics Canada’s Quality Assurance Framework, 2002, Chapter 5, pp. 9-15 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-586-x2002001-eng.pdf)

• Statistics Canada Survey Methods and Practices (http://www.statcan.gc.ca/pub/12-587-x/12-587-x2003001-eng.pdf)

Colombia

• Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-nqaf/Assurance%20quality%20methodology.pdf)

• Metodología para el fortalecimiento de registros administrativos (http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos)

• Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean, May 2011 (http://www.dane.gov.co/files/noticias/BuenasPracticas_en.pdf)

• Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales (http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf)


Eurostat

• DESAP - The European Self Assessment Checklist for Survey Managers (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/desap%20G0-LEG-20031010-EN.pdf)

• ESS Handbook for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-EHQR_FINAL.pdf)

• ESS Standard for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-ESQR_FINAL.pdf)

• European Statistics Code of Practice Self Assessment Questionnaire (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/self%20assesment.pdf)

• Handbook on Data Quality – Assessment Methods and Tools (DatQam Manual) (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20DATA%20QUALITY%20ASSESSMENT%20METHODS%20AND%20TOOLS%20%20I.pdf)

• Handbook of Recommended Practices for Questionnaire Development and Testing in the European Statistical System, Eurostat (2006) (http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/documents/Handbook_questionnaire_development_2006.pdf)

• Methodological guidance (http://epp.eurostat.ec.europa.eu/portal/page/portal/research_methodology/methodology/ess_methodological_documents)

• Monographs of official statistics - "Variance estimation methods in the European Union", European Commission, 2002 edition (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/MOS%20VARIANCE%20ESTIMATION%202002.pdf)

• Recommended Practices for Editing and Imputation in Cross-Sectional Business Surveys, Eurostat (2008) (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/RPM_EDIMBUS.pdf)

France • Guidelines for finding a balance between accuracy and delays in the statistical surveys, INSEE, 2007

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/GUIDELINES_FOR_BALANCE_BETWEEN_ACCURACY_AND_DELAYS.pdf)

• Official statistical system practices in accordance with the principles of the European Statistics Code of Practice – Statistical results: 12- Accuracy and reliability (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe12.htm)

Italy • Development and use of generalized software for survey operation (e.g. editing and imputation, estimation, coding,

data collection and monitoring) (in Italian) (http://www3.istat.it/strumenti/metodi/software/) • Quality guidelines for statistical processes, Istat (2011)

Japan • Guideline of "Quality Assurance" of the Official Statistics of Japan (http://www.stat.go.jp/english/index/seido/pdf/qa_gl.pdf )

Netherlands

• Checklist Quality of Statistical Output, Chapter 6 - Accuracy of data, Chapter 17 - Remaining quality dimensions, pp. 23-30, 55-57 (http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)

Organisation for Economic Co-operation and Developm ent (OECD)/Eurostat • Data and metadata requirements for building a real-time database to perform revisions analysis. Technical Report,

OECD/Eurostat Task Force on “Performing Revisions Analysis for Sub-Annual Economic Statistics”. McKenzie, R. and Gamba, M. (2008) (http://www.oecd.org/dataoecd/47/15/40315408.pdf)

• Interpreting the results of Revision Analyses: Recommended Summary Statistics, Technical Report, OECD/Eurostat Task Force on “Performing Revisions Analysis for Sub-Annual Economic Statistics”. McKenzie, R. and Gamba, M. (2008) (http://www.oecd.org/dataoecd/47/18/40315546.pdf)

Organisation for Economic Co-operation and Developm ent (OECD) • The OECD Project on Revisions Analysis: First Elements for Discussion, OECD Short-term Economic Statistics

Expert Group (STESEG), 27-28 June 2005. Di Fonzo (http://www.oecd.org/dataoecd/55/17/35010765.pdf)

Portugal • Revisions Policy, Statistics Portugal, December 2008

(http://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=70086429&att_display=n&att_download=y)



Section 3: Under accuracy, p. 17(http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf) • SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 3: Under

accuracy, p. 20(http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Kingdom • Guidelines for measuring statistical quality, pp. 17, 19-22, 24-25, 28, 31-49, 51-58, 62, 64-68, 74-76

(http://unstats.un.org/unsd/dnss/docs-nqaf/UK-Guidelines_Subject.pdf) • Reliability and Quality Indicators for National Accounts Aggregates, United Kingdom, 1999

(http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/RELIABILITY%20AND%20QUALITY%20INDICATORS%20FOR%20NATIONAL%20ACCOUNTS.pdf)


UNSD, 2003 paras. 514 – 515 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)



(http://www.nss.gov.au/dataquality/PDFs/DQO_Needs.pdf) • Quality Management of Statistical Outputs Produced From Administrative Data, Mar 2011 – Common Issues with

Acquiring Administrative Data • Timeliness

(http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features12Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

Canada • Statistics Canada Quality Guidelines, Fifth edition, 2009, Chapters 7 (Section 7.4.3), 9, 10, 14 (Section 14.4.1), 16,

18 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf) • Statistics Canada’s Quality Assurance Framework, 2002, Chapter 6, pp. 15-16

(http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-586-x2002001-eng.pdf)






Eurostat • ESS Handbook for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-

EHQR_FINAL.pdf) • ESS Standard for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-

ESQR_FINAL.pdf)


Statistical results: 13- Timeliness and punctuality (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe13.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007; Periodicity: paras. 3.15 –

3.20;Timeliness: paras. 3.21- 3.25;Real sector: National accounts: paras. 3.31, 3.64, Production index/indices: paras. 3.68, Price indices: paras. 3.69 – 3.70, Labor market: paras. 3.81, 3.86, 3.89; Fiscal sector: Central government operations: paras. 3.46, 3.91, 3.97; Financial sector: paras. 3.53, 3.102, 3.103, 3.107, 3.108; External sector: paras. 3.61, 3.12, 3.115, 3.117, 3.119; Data Dimensions of the GDDS, Table 3.1, pp. 33 – 38 (http://dsbb.imf.org/images/pdfs/gddsguide.pdf)



Netherlands • Checklist Quality of Statistical Output, Chapter 10 - Timeliness of the publication, Chapter 11 - Punctuality of the

publication, pp. 40-44 (http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)


Section 4, under timeliness, p. 31 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 4, under timeliness, p. 34 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Kingdom • Guidelines for measuring statistical quality, pp. 25, 29, 69-70 (http://unstats.un.org/unsd/dnss/docs-nqaf/UK-

Guidelines_Subject.pdf)



(http://www.nss.gov.au/dataquality/PDFs/DQO_Needs.pdf)

Canada • Statistics Canada, Policy on Informing Users of Data Quality and Methodology (http://www.statcan.gc.ca/about-

apercu/policy-politique/info_user-usager-eng.htm) • Statistics Canada Publications by Subject

(http://cansim2.statcan.gc.ca/cgi-win/cnsmcgi.pgm?Lang=E&ResultTemplate=/Stu-Etu/Stu-Etu3&ChunkSize=25&AS_Theme=0&ChunkStart=1&AS_Date=.&AS_Ser=.&AS_Auth=.&AS_Srch=&AS_SORT=0&AS_UNIV=3&Version=2&AS_Mode=2)

• Statistics Canada Quality Guidelines, Fifth edition, 2009, Chapters 2, 3, 10, 14 (Sections 14.4.3, 14.4.6), 16, 17, 18 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

• Statistics Canada’s Quality Assurance Framework, 2002, Chapter 7, pp. 16-20 and Chapter 8 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-586-x2002001-eng.pdf)

Colombia • Decreto 3851/2006 Artículo 9: Portal de Difusión (http://www.dane.gov.co/files/acerca/Normatividad/d3851.pdf) • Metodología Aseguramiento de la Calidad de la Información Estadística (http://unstats.un.org/unsd/dnss/docs-





Eurostat • Guidelines relating to quality reporting (http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting) • ESS Handbook for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-


ESQR_FINAL.pdf) • EURO-ESMS Metadata Structure (2009)

(http://epp.eurostat.ec.europa.eu/cache/ITY_SDDS/Annexes/ESMS_Structure.xls)


Statistical results: 15- Accessibility and clarity (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe15.htm)

International Monetary Fund (IMF) • The General Data Dissemination System, Guide for participants and users, IMF, 2007, Quality Dimension of

Disseminated Data, paras. 3.151 3.154 (http://dsbb.imf.org/images/pdfs/gddsguide.pdf)

Italy • Process metadata and quality assurance procedures in place for Istat statistical processes are available to external

users by means of the SIDI/SIQual information system (http://siqual.istat.it/SIQual/lang.do?language=UK)



Netherlands • Checklist Quality of Statistical Output, Chapter 12 - Accessibility of statistics, Chapter 14 - Extent of detail of a

statistic, pp. 48-49, Chapter 17 - Remaining quality dimensions, pp. 45-47, 55-57 (http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)

South Africa • Policy on Informing Users of Data Quality, Statistics South Africa, 2006 (http://unstats.un.org/unsd/dnss/docs-

nqaf/SouthAfrica-Statistics_SA_Data_Quality_Policy.pdf) • South African Statistical Quality Assessment Framework (SASQAF), Second edition, Statistics South Africa, 2010,

Section 5: under accessibility, p. 36, Section 6: under interpretability, p. 40 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 5: under accessibility, p. 38, Section 6:under interpretability, p. 44 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Kingdom • Guidelines for measuring statistical quality, pp. 23, 24, 60-62, 65, 70-72, 76 (http://unstats.un.org/unsd/dnss/docs-

nqaf/UK-Guidelines_Subject.pdf)

United Nations Economic Commission for Europe (UNEC E) • Making Data Meaningful Part 1: A Guide to Writing Stories about Numbers

(http://www.unece.org/stats/documents/writing) • Making Data Meaningful Part 2: A Guide to Presenting Statistics (http://www.unece.org/stats/documents/writing)

United Nations Statistics Division (UNSD) • Fundamental Principles of Official Statistics - Principle 3 – Accountability


UNSD, 2003 paras. 468 -498 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)

United States • Census Bureau Information Quality Guidelines, Objectivity, Informing Users of Data Quality and Methodology

(http://www.census.gov/quality/guidelines/objectivity.html)


Australia - Data Quality Online: Hints and Tips - Defining A Data Need (http://www.nss.gov.au/dataquality/PDFs/DQO_Needs.pdf)

- Quality Management of Statistical Outputs Produced From Administrative Data, Mar 2011 – Common Issues with Acquiring Administrative Data, Stability of Data (http://www.abs.gov.au/ausstats/[email protected]/Latestproducts/1522.0Main%20Features7Mar%202011?opendocument&tabname=Summary&prodno=1522.0&issue=Mar%202011&num=&view=)

Canada - Statistics Canada Survey Methods and Practices (http://www.statcan.gc.ca/pub/12-587-x/12-587-x2003001-eng.pdf) - Statistics Canada Quality Guidelines, Fifth edition, 2009, Chapters 2, 5, 7 (Section 7.4.4), 10, 14 (Section 14.4.5) (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-539-x2009001-eng.pdf)

- Statistics Canada’s Quality Assurance Framework, 2002, Chapter 9, pp. 21-22 (http://unstats.un.org/unsd/dnss/docs-nqaf/Canada-12-586-x2002001-eng.pdf)



(http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos) • Metodología para la estandarización de conceptos (http://unstats.un.org/unsd/dnss/docs-

nqaf/Methodology%20for%20standardization%20of%20concepts%2017.Jun.2010.pdf) • Proposal for the Structure of a Regional Code of Good Statistical Practice for Latin America and the Caribbean,




Eurostat • ESS Handbook for Quality Reports, 2009 edition (http://unstats.un.org/unsd/dnss/docs-nqaf/Eurostat-


ESQR_FINAL.pdf) • Eurostat’s Concepts and Definitions Database (CODED)

(http://ec.europa.eu/eurostat/ramon/nomenclatures/index.cfm?TargetUrl=LST_NOM&StrGroupCode=CONCEPTS&StrLanguageCode=EN)


Statistical results: 14- Coherence and comparability (http://www.insee.fr/en/insee-statistique-publique/default.asp?page=qualite/principe14.htm)

Hungary • HCSO,s Strategy 2009 –2012, Quality Statistics, Methodological developments in order to improve the quality of

statistical products (p. 18) (http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/0_DOCS/hu/strategy_2009_2012.pdf)


Netherlands • Checklist Quality of Statistical Output, Chapter 7 - Coherence of statistics, Chapter 8 - Comparability of data,

Chapter 9 - Numerical consistency of data, pp. 31-39 (http://unstats.un.org/unsd/dnss/docs-nqaf/Netherlands-2009ChecklistQualityofStatisticalOutput.pdf)


Section 7: under comparability and coherence, p. 41 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 7: under comparability and coherence, p. 47 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)

United Kingdom • Guidelines for measuring statistical quality, pp. 18-19, 26, 29, 30, 50, 52, 54, 58-59, 63-64, 73, 77

(http://unstats.un.org/unsd/dnss/docs-nqaf/UK-Guidelines_Subject.pdf)




(http://www.dane.gov.co/files/planificacion/planificacion/metodologia/planes_fortalecimiento_RA.pdf administrativos) • Propuesta de Código Nacional de Buenas Prácticas para las Estadísticas Oficiales

(http://unstats.un.org/unsd/dnss/docs-nqaf/National%20Code%20of%20Practice%2004.Agu.2011.pdf) • Sistema de Metadatos (PAD) (http://190.25.231.249/aplicativos/sen/NADA/)

Eurostat • EURO-ESMS Metadata Structure (2009)

(http://epp.eurostat.ec.europa.eu/cache/ITY_SDDS/Annexes/ESMS_Structure.xls) • Metadata information and links (http://epp.eurostat.ec.europa.eu/portal/page/portal/statistics/metadata)

SDMX (Statistical Data and Metadata eXchange) • SDMX web page (http://sdmx.org/?page_id=6)


Section 6: Under interpretability, p. 40 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_Edition_2.pdf)

• SASQAF Operational Standards and Guidelines, First edition, Statistics South Africa, 2010, Section 6: Under interpretability, p. 44 (http://www.statssa.gov.za/inside_statssa/standardisation/SASQAF_OpsGuidelines_Edition_1.pdf)


United Nations Economic Commission for Europe (UNEC E) • Common Metadata Framework

(http://www1.unece.org/stat/platform/display/metis/The+Common+Metadata+Framework) • Generic Statistical Business Process Model (http://www.unece.org/stats/gsbpm) • Metadata flows within the GSBPM (including contributions from Australia, Norway and Portugal)

(http://www1.unece.org/stat/platform/display/metis/Metadata+Flows+within+the+GSBPM) • METIS Wiki (http://www1.unece.org/stat/platform/display/metis/METIS-wiki)


UNSD, 2003 paras. 470 -476 (http://unstats.un.org/unsd/publication/SeriesF/SeriesF_88E.pdf)

Annex 3 - ESS Quality and Performance Indicators 93

ANNEX 3 – European Statistical System's Quality and Performance Indicators 5 (based on the work of the Eurostat Expert Group on Quality Indicators, the following set of standard quality and performance indicators could be used, from the producers' point of view, for summarising the quality of statistical products in various statistical domains)

Quality component Concept Quality and Performance Indicators

Relevance Completeness R1. Data completeness - rate

Sampling error A1. Sampling error - indicators

A2. Over-coverage - rate

A3. Unit non-response - rate

A4. Item non-response - rate

A5. Imputation - rate

Non-sampling error

A6. Common units - proportion

Accuracy

Revision practice A7. Data revision - average size

T1. Time lag - first results Timeliness

T2. Time lag - final results

Timeliness and Punctuality

Punctuality T3. Punctuality - delivery and publication

Metadata AC1. Metadata - consultations

On-line database AC2. Data tables - consultations

Accessibility and Clarity

Documentation on quality and methodology

AC3. Metadata completeness - rate

Comparability over time CC1. Length of comparable time series Coherence and Comparability Comparability - geographical

(over countries, regions, etc.) CC2. Asymmetry for mirror flows statistics - coefficient

5 More information about the European Statistical System’s Quality and Performance Indicators can be found at http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/quality_reporting and http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/Quality_Performance_Indicators_FINAL_v_1_1.pdf .

guidelines for the template for a generic national quality ...the template for a generic national...

Documents