data quality considerations m&e capacity strengthening workshop, maputo 19 and 20 september 2011...
TRANSCRIPT
Data Quality ConsiderationsM&E Capacity Strengthening Workshop, Maputo19 and 20 September 2011
Arif Rashid, TOPS
Project Implementation
Project activities are implemented in the field. These activities are designed to produce results that are quantifiable.
Data Management System An information system represents these activities by collecting the results that were produced and mapping them to a recording system.
Data Quality: How well the DMS represents the factData Quality: How well the DMS represents the fact
True picture
of the field
True picture
of the field
Data Management
System
Data Management
System
Data Quality
?
Slide # 1
Why Data Quality?
• Program is “evidence-based”
• Data quality Data use
• Accountability
Slide # 2
Conceptual Framework of Data Quality?
Service delivery points
Intermediate aggregation levels(e.g. districts/ regions, etc.)
M&E Unit in the Country Office
Dat
a m
anag
emen
t and
repo
rting
sy
stem
Functional components of Data Management Systems Needed to Ensure Data QualityM&E Structures, Roles and Responsibilities
Indicator definitions and reporting guidelinesData collection and reporting forms/tools
Data management processes
Data quality mechanisms
M&E capacity and system feedback
Dimensions of Data Quality
Accuracy, Completeness, Reliability, Timeliness, Confidentiality, Precision, Integrity
Quality Data
Slide # 3
Dimensions of data quality
• Accuracy/Validity– Accurate data are considered correct. Accurate data
minimize error (e.g., recording or interviewer bias, transcription error, sampling error) to a point of being negligible.
• Reliability– Data generated by a project’s information system are
based on protocols and procedures. The data are objectively verifiable. The data are reliable because they are measured and collected consistently.
Slide # 4
Dimensions of data quality• Precision
– The data have sufficient detail information. For example, an indicator requires the number of individuals who received training on integrated pest management by sex. An information system lacks precision if it is not designed to record the sex of the individual who received training.
• Completeness– Completeness means that an information system from
which the results are derived is appropriately inclusive: it represents the complete list of eligible persons or units and not just a fraction of the list.
Slide # 5
Dimensions of data quality
• Timeliness– Data are timely when they are up-to-date (current),
and when the information is available on time.
• Integrity– Data have integrity when the system used to generate
them are protected from deliberate bias or manipulation for political or personal reasons.
Slide # 6
Dimensions of data quality
• Confidentiality – Confidentiality means that the respondents are
assured that their data will be maintained according to national and/or international standards for data. This means that personal data are not disclosed inappropriately, and that data in hard copy and electronic form are treated with appropriate levels of security (e.g. kept in locked cabinets and in password protected files.
Slide # 7
Data quality Assessments
Slide # 8
Data quality Assessments
• Two dimensions of assessments:1. Assessment of data management and reporting
systems2. Follow-up verification of reported data for key
indicators (spot checks of actual figures)
Slide # 9
Systems assessment toolsM&E structures, functions and capabilities
1 Are key M&E and data-management staff identified with clearly assigned responsibilities?
2 Have the majority of key M&E and data management staff received the required training?
Indicator definitions and reporting guidelines
3 Are there operational indicator definitions meeting relevant standards that are systematically followed by all service points?
4 Has the project clearly documented what is reported to who, and how and when reporting is required?
Data collection and reporting forms/tools
5 Are there standard data-collection and reporting forms that are systematically used?
6 Are data recorded with sufficient precision/detail to measure relevant indicators?
7 Are source documents kept and made available in accordance with a written policy?
Slide # 10
Systems assessment toolsData management processes
Does clear documentation of collection, aggregation and manipulation steps exist? Are data quality challenges identified and are mechanisms in place for addressing them?Are there clearly defined and followed procedures to identify and reconcile discrepancies in reports? Are there clearly defined and followed procedures to periodically verify source data?
M&E capacity and system feedback
Do M&E staff have clear understanding about the roles and how data collection and analysis fits into the overall program quality?Do M&E staff have clear understanding with the PMP, IPTT and M&E Plan? Do M&E staff have required skills in data collection, aggregation, analysis, interpretation and reporting ?Are there clearly defined feedback mechanism to improve data and system quality?
Slide # 11
Schematic of follow-up verification
Slide # 12
M&E system design for data quality
• Appropriate design of M&E system is necessary to comply with both aspects of DQA– Ensure that all dimensions of data quality are
incorporated into M&E design– Ensure that all processes and data management
operations are implemented and fully documented (ensure a comprehensive paper trail to facilitate follow-up verification)
Slide # 13
This presentation was made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of Save the Children and do not necessarily reflect the views of USAID or the United States Government.