research methods observations interviews case studies surveys quasi experiments
DESCRIPTION
Intervention Target system OutcomesTRANSCRIPT
![Page 1: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/1.jpg)
Research Methods
ObservationsInterviews
Case StudiesSurveys
Quasi Experiments
![Page 2: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/2.jpg)
Classes of Research Methods Experimental Non-Experimental
• Observations • Interviews• Case studies• Surveys• Quasi-Experiments
![Page 3: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/3.jpg)
Intervention
Target system
Outcomes
![Page 4: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/4.jpg)
Approach Comparison
Approach Research Question Control ofBehaviors
Experiments How? Why? Yes
Surveys Who? What? Where?How many? Howmuch?
No
Quasi-Experiments
How? Why? No
Case Studies How? Why? No
Adapted from Yin (1994)
![Page 5: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/5.jpg)
Experiments
![Page 6: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/6.jpg)
Experiments Most common method of evaluation Primary components
• Desired intervention or change • Inputs (what you vary or keep the same)• Outcomes (what you are looking for)
Vary one or more inputs and look for differences in the outcomes.
Randomize
![Page 7: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/7.jpg)
ExperimentsStrengths Can be used to show
cause and effect Quantitative techniques
can be used to show strength of relationships
Accepted across a wide-range of disciplines
Limitations In the “real world” in
may be difficult to have random assignment
May not be able to create realistic conditions in a controlled setting
![Page 8: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/8.jpg)
Observations
![Page 9: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/9.jpg)
Observations Used to capture the ways in which people act and
interact within a particular environment/setting Observations may be either qualitative or
quantitative Researcher will either use detailed field notes, audio
or video recording to capture data Focus may be on general aspects of
behavior/phenomenon for qualitative; for quantitative, typically will focus on a particular aspect of behavior that can be quntified
![Page 10: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/10.jpg)
Strengths Extremely flexible – researcher can shift
focus as new data come to light Good method to study behaviors and other
aspects of a system that may be difficult to quantify
Enables researcher to study the richness and complexity of human behavior
Data may be taken at multiple points in time
![Page 11: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/11.jpg)
Limitations Behavior to be studied must be defined in a
precise, concrete manner to be recognizable Typically takes multiple researchers Researcher’s presence may impact
participants’ behaviors Requires meticulous attention to detail in
planning, data collection, and data analysis Results are limited to the system studied
![Page 12: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/12.jpg)
ReliabilityHow reproducible is the data?
How consistent is the data? Data collection protocol must exist (but may
be modified as research progresses) Data analysis training to assure consistency
between researchers Data analysis reliability assessment if
multiple coders are used
![Page 13: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/13.jpg)
Validity
How well does the data collected address the characteristics of interest?
Face validity Content validity Criterion validity Construct validity
![Page 14: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/14.jpg)
Interviews
![Page 15: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/15.jpg)
Interviews Can produce a great deal of useful
information Can be structured, open-ended, or semi-
structured Can be used for individuals or groups of
individuals (focus groups) IRB – confidentiality and informed
consent must be planned well in advance
![Page 16: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/16.jpg)
Interview tips Make sure interviewees are representative Find a suitable location Take a few minutes to establish rapport Don’t put words in people’s mouths Record responses verbatim Keep your reactions to yourself Use contingency questions to avoid irrelevant
questions
![Page 17: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/17.jpg)
Choosing Interviewees Interviewees must be competent to
answer Interviewees must be willing to answer Questions should be relevant to the
Interviewees
![Page 18: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/18.jpg)
Strengths Rich and complete information Question complexity can be greater than
survey Flexible (particularly if open-ended)
![Page 19: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/19.jpg)
Limitations Can produce information that is not
related to research topic and/or not comparable to other data collected
More expensive than written surveys Time consuming Sensitive questions may not be answered
honestly
![Page 20: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/20.jpg)
Case studies
![Page 21: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/21.jpg)
Case Studies An evaluation method that
• “investigates a contemporary phenomenon within its real-life context,
• especially when the boundaries between phenomenon and context are not clearly evident”
(Yin 1994, p. 13)
![Page 22: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/22.jpg)
Case studies Used to study an individual, program,
event, or organization in depth for a pre-defined period of time
Researcher must know what he/she is looking for
Use multiple sources of data, e.g. interviews, direct observation, document analysis, or surveys
![Page 23: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/23.jpg)
Case StudiesStrengths Can be done after the
fact Valuable technique to
highlight learnings from a project
Data collection methods are relatively straight-forward
Limitations Difficult to generalize
the results to other parts of the organization or to other organizations.
Will not show cause and effect relationships
Time consuming
![Page 24: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/24.jpg)
Quality Digest Case Study Example
Taking SPC to the production line Target System: Organizations that have
front line workers using SPC tools Types of data: Direct observation,
interviews, and documents Findings: Lessons learned, road blocks,
and best practices were highlighted
![Page 25: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/25.jpg)
Surveys
![Page 26: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/26.jpg)
Why use surveys? A method for collecting data to describe
some specific characteristics of individuals or groups of people.
A method for measuring some specific attitudes, perceptions, and beliefs of individuals or groups of people.
![Page 27: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/27.jpg)
Survey design issues Sample selection Sample size Data collection method Questionnaire format Question construction
![Page 28: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/28.jpg)
Surveys Understand how people view a particular
topic without asking every person Good method to quantify otherwise
qualitative information Data may be taken at multiple points in time Use questionnaires or structured interviews
to collect information
Adapted from Creswell (1994)
![Page 29: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/29.jpg)
New Program Evaluation Example
SWE e-mail mentoring program Target System: Program participants Intervention: Mentors/mentees exchange e-mails
about careers in engineering Outcomes:
• Are mentees more likely to consider engineering careers than before?
• Do mentors and mentees feel more positive about themselves?
![Page 30: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/30.jpg)
SurveysStrengths Can be used to create
quantitative measures of “softer” types of data.
Can be done at many points in time if needed
Methods for evaluating the data exist
Limitations Results are limited to
the overall group represented in the survey sample
Does not give you a way to follow up if results are inconclusive.
![Page 31: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/31.jpg)
Quasi-Experiments
![Page 32: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/32.jpg)
Quasi-Experiments Used in social science, education, policy, and
management systems Use when randomization of target system and or
intervention is not possible Multiple-designs exist Primary components
• Study group• Comparison group• Intervention
![Page 33: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/33.jpg)
Quasi-Experimental Approach
![Page 34: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/34.jpg)
Nonequivalent Control Group Example
O1 X O2
O1 O2
![Page 35: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/35.jpg)
Autonomous Team ExampleImplementing teams in a mineral plant Study group: Autonomous teams in a startup
plant Comparison group: Existing shift workers in
the original plant Intervention: Creation of teams Outcomes: Job satisfaction, trust, and
productivity
![Page 36: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/36.jpg)
“Ideal Outcomes”
1.5
2
2.5
3
3.5
4
4.5
T1 T2
Survey at Time T(x)
Scor
e
Study SiteComparison Site
![Page 37: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/37.jpg)
Trends in Opposite Direction
1.5
2
2.5
3
3.5
4
4.5
T1 T2
Survey at Time T(x)
Scor
e
Study SiteComparison Site
![Page 38: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/38.jpg)
Trends with Differing Growth Rates
1.5
2
2.5
3
3.5
4
4.5
T1 T2
Survey at Time T(x)
Scor
e
Irving ParkPalatine
![Page 39: Research Methods Observations Interviews Case Studies Surveys Quasi Experiments](https://reader035.vdocuments.site/reader035/viewer/2022082202/5a4d1ae67f8b9ab059979285/html5/thumbnails/39.jpg)
Quasi-ExperimentsStrengths Can be used in the
“real world” Does not require
changes to existing organization
Designs and techniques for evaluation exist
Limitations Can not be used to
show cause and effect May not be able to
generalize the results Results can potentially
be explained by other factors