qualitative methods for health program evaluation chsc 433 module 5/chapter 12 l. michele issel, phd...
TRANSCRIPT
![Page 1: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/1.jpg)
Qualitative Methods for Health Program Evaluation
CHSC 433CHSC 433
Module 5/Chapter 12Module 5/Chapter 12L. Michele Issel, PhD
UIC School of Public Health
![Page 2: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/2.jpg)
“Different kinds of
problems require
different types of
data.” Patton, 1997
![Page 3: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/3.jpg)
Objectives
List the major qualitative designs List at least one pro and con for
each of the major qualitative designs
Provide an outline of how qualitative data analyses are done
![Page 4: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/4.jpg)
Beyond the Paradigm Debate
History of science favored quantitative (empiricism), deductive hypothesis testing, logical postivism
Current science favors understanding based on rigorous methods
![Page 5: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/5.jpg)
Use Qualitative when
Want to minimize research manipulation by studying natural field setting.
Program aims at individual outcomes (so when program aims at common outcomes across individuals, use quantitative methods).
![Page 6: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/6.jpg)
Key Characteristics
The use of non-numeric data, such as narratives, pictures, music
The use of subjective, experiential, naturalistic inquiry to explain phenomena
Use of inductive, iterative analysis Holistic and contextual concerns Pays attention to individual’s
uniqueness
![Page 7: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/7.jpg)
Functions of Qualitative Methods (Adapted from Green & Lewis ‘86)
1. Develop and delineate program elements
2. Booster power of quantitative designs
3. Broad the observational field
4. Analyze processes and cases to understand why or how the program worked
5. Generate a program or intervention theory
6. Use instead of quantitative methods
![Page 8: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/8.jpg)
Underlying Perspectives
Phenomenology- experiences and meanings
Ethnography- cultureCritical analysis- communication and
powerGrounded Theory- discovery of
theory Content Analysis-manifest meanings
in the written word
![Page 9: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/9.jpg)
Perspective --> Question
Phenomenology
Ethnography
Critical analysis
Grounded theory
Content analysis
What does it mean to the person?
What are the norms and values (culture)?
How has power shaped it ?
What are the relationships (theory)?
What themes are in the text?
![Page 10: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/10.jpg)
Major Types of Qualitative Methods
Participant observation Case studies In-depth Interviews Focus groups Open-ended survey questions
![Page 11: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/11.jpg)
Participant Observation
Acting as a member of a group, collect data
Make narrative notes and memos about processes, events, people observed
Use key informants to verify data analysis
![Page 12: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/12.jpg)
Case Studies
Define what is a case (organization, program, person)
Use variety of types of raw data generated by or about the case: memoranda, observations, surveys, interviews, etc
![Page 13: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/13.jpg)
Case Study
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Allows for understanding of context as influence on program or participant
Allows for understanding of context as influence on program or participant
![Page 14: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/14.jpg)
In-depth Interviews
Use open-ended questions with key individuals (participants, key informants)
Use probes to clarify and explore issues or topics alluded to by the respondent or earlier data analysis
Use tape recorder and transcripts of the interviews
![Page 15: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/15.jpg)
In-depth Interviews
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Provides rich insights into personal thoughts, values, meanings, and attributions
Provides rich insights into personal thoughts, values, meanings, and attributions
![Page 16: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/16.jpg)
Focus Groups
Carefully selected group of individuals who participate in guided discussion about a specific topic
Use a facilitator and a recorder
![Page 17: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/17.jpg)
Focus Groups
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Inexpensive given the amount and type of data, get collective views rather than individual views
Inexpensive given the amount and type of data, get collective views rather than individual views
![Page 18: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/18.jpg)
Observations
Non-participatory and Participatory techniques can be used
Need training on what will be observed and how will record the observation
Data collection methods vary:• Audio-visuals recording• Field notes• Logs
![Page 19: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/19.jpg)
Observations
Key Benefits for use in planning and evaluation
Key Challenges for use in planning and evaluation
Can identify sequence of causes and effects, may identify new behaviors or events
Can identify sequence of causes and effects, may identify new behaviors or events
![Page 20: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/20.jpg)
Use open-ended question placed at end of quantitative survey
Unable to use probes for clarification
Handwriting and spelling can make interpretation difficult
Open-ended survey questions
![Page 21: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/21.jpg)
Sampling for Naturalistic Inquiry
Small purposive samples
• Select for a specific characteristic
Theoretical sampling
• Select based on what “ought” to matter
Sample for category saturation
• Select until no new information is
gained from participants
![Page 22: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/22.jpg)
Data Analysis
Coding and interpreting the data
To count or not to count
![Page 23: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/23.jpg)
Coding Terminology
CategoryCategory- classification of concepts in the data
DimensionDimension- implies continuum
PropertyProperty- attributes or characteristics of a category
Constant comparisonConstant comparison- process to develop categories, involves comparing new with existing categories
CodableCodable- unit of data to be categorized
![Page 24: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/24.jpg)
Analysis Procedures
1. Identify codable units of data2. Understand the meaning3. Discover categories4. Name categories5. Discover properties and
dimensions of the categories6. Generate explanation
![Page 25: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/25.jpg)
Scientific Rigor = Trustworthiness (Lincoln & Guba, 1985)
CredibilityCredibility ~ Internal validity
TransferabilityTransferability ~ External validity
DependabilityDependability ~ Reliability
ConfirmabilityConfirmability ~ Objectivity
![Page 26: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/26.jpg)
Credibility
Have confidence in the truth of the findings by:
Invest sufficient time, triangulate Use outsiders for insights (peer
debriefing) Refine working hypotheses with negative
cases Check findings against raw data Use participant feedback
![Page 27: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/27.jpg)
Transferability
Applicability to other contexts and respondents
Provide thick (detailed, comprehensive) descriptions for others to assess possibility of transferability
![Page 28: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/28.jpg)
Dependability
Find same results if repeated the study
Leave a trail that can be followed so that others can see the findings are supported by the data
![Page 29: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/29.jpg)
Confirmability
Findings are from the respondents not the researcher
Leave an Audit Trail (same as for Dependability)
![Page 30: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/30.jpg)
To count or not to count
Number of participants who mentioned a category
Number of times category mentioned throughout the study
Issues…. Neither help with interpretation of meanings, both can misrepresent the scope of the sentiment
![Page 31: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/31.jpg)
From Data to Description
Categories as typologies are rudiments of a theory
Category dimensions and properties as essential
Linkages between categories form theory
![Page 32: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/32.jpg)
Data Presentation
Use descriptions of context to show transferability
Use tables showing category development to show dependability and confirmation
Use participants’ words to show confirmation
Use diagrams of relationships among categories
![Page 33: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/33.jpg)
Realities of Data Analysis
Messy, confusing, repetitive • Iterative category development
Overwhelming quantities of data Conflicting interpretations of data by
peers and participants Manifest versus implied meanings
cloud data analysis Investigator biases
![Page 34: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/34.jpg)
Cost of Data Collection
Interview time• Travel to interviewee• Reading and listening
Transcription time• 1 hour interview: 3 hours transcribing
Data analysis time
![Page 35: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/35.jpg)
Evaluation Caveats
Integrate with quantitative data Use of participant feedback
(credibility) as key to acceptance of findings
Stories are more powerful than numbers and make the numbers more human
![Page 36: Qualitative Methods for Health Program Evaluation CHSC 433 Module 5/Chapter 12 L. Michele Issel, PhD UIC School of Public Health](https://reader035.vdocuments.site/reader035/viewer/2022062718/56649e705503460f94b6ea59/html5/thumbnails/36.jpg)
Qualitative MethodsAcross the Pyramid
Direct Health Care
Services____________________
Enabling Services___________________________
Population-Based Services___________________________________
Infrastructure Services
Each qualitative method has potential usefulness for programs at each level of the Pyramid.