Ensuring the quality of Qualitative Data

Presented by

Lorie Broomhall, Ph.D.Senior HIV/AIDS Advisor

Nigeria Monitoring and Evaluation Management ServicesThe Mitchell Group

Abuja, Nigeria

Mini Pop QuizMini Pop QuizPlease indicate T if the statement is true and F if the statement is false:

1. Quantitative methods are more accurate than qualitative methods.

2. Qualitative methods are not as credible as quantitative methods because they cannot be replicated.

3. Quantitative methods are more generalizable than qualitative methods.

4. Qualitative evaluations are more costly and time consuming that quantitative evaluations – less bang for the buck.

5. Quantitative evaluations are less biased than qualitative evaluations

6. Qualitative methods are useful for explaining findings derived from quantitative methods – telling the story – but not the other way around.

7. Qualitative data analysis is not systematic

8. Focus group discussions are the most useful and reliable qualitative evaluation method.

Workshop Objectives

Definition of high quality qualitative data Uses of qualitative data in M&E Quality standards Qualitative method designs Analysis and reporting

High Quality Data Ensure that program and budget decisions are as well

informed as practically possible Support efficient use of resources

Are credible

Meet reasonable standards accuracy, objectivity, and consistency

“In principle, the same quality standards for quantitative data apply to qualitative data”

USAID Functional Series 200 – Programming Policy ADS 203 – Assessing and Learning

High quality can be achieved using any qualitative method (as long as it is used appropriately):

Interviews Mapping/Walk-throughs

Directed observations Participant observation Focus Group discussions Elicitation techniques

Pile sorts Ranking Free lists

High quality data have

Validity (internal and external)

Reliability

Timeliness

Precision

Integrity

Validity

“Data are valid to the extent that they clearly, directly, and adequately represent the result that was intended to be measured.”*

* Quotations in this presentation from USAID ADS 203, and Tips #12

Validity increases with

Clear goals and objectives Sound method design and data

collection plan - triangulation Adequate training and supervision Systematic data analysis Method triangulation

Badly designed data collection instruments

Imprecise definitions Inappropriate sampling techniques Inadequately trained data collectors Transcription error

Validity is compromised by

A Sound Sampling Plan is

Representative: Are all people or groups affected by program represented in the sample? Are they represented in proportion to their size?

Inclusive: Are all the different views and perspectives represented in the sample?

Appropriate: Are the right people being sampled?

Consistent: Is the sampling strategy applied to all sites over the length of the evaluation?

Documented and justified

Examples of sampling strategies for qualitative evaluations: Convenience/purposive sampling Stratified sampling (significant variation)

Random sampling (people, time, location, etc.)

Snowball sampling (social networks)

Reliability

“Data should reflect stable and consistent data collection processes and analysis methods over time.”

Reliability in assured with:

Consistent data collection processes Field supervision and observation Checks for bais and ‘drift’ Data replicability

Precision is improved with: Direct, understandable data collection instruments

Concise and operationalized definitions

Clear, descriptive, objective and decipherable documentation (e.g. field notes, summaries)

Scrutinize!

Frequent checks of data bases and inter-coder reliability

Precision“Data should be sufficiently accurate (precise) to present a fair picture of performance. . .”

Timeliness“Data should be current and available frequently enough to permit management decisions to be made”

IntegrityData should be free from manipulation for professional, political or personal purposes.

Timeliness can be achieved through

Building an appropriate data collection strategy – minimum methods for the maximum return.

Well planned data management/processing

Adequate training and supervision

Effective communication

Efficient analysis

Contingency plans for unforeseen events (strikes)

Maintaining Integrity

Secure data storage: names on folders, electronic files, registers and logs

Supervision – scrutinizers Awareness of potential data

manipulation

Thank youThank you