![Page 1: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/1.jpg)
Research Design Choices and Causal Inferences
Professor Alexander Settles
![Page 2: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/2.jpg)
Approaches and strategies of research design
• Quantitative Strategies– Exploratory Studies
• In-depth interviews• Focus Group studies
– Descriptive Studies• Survey research• Relationship studies
– Causal Studies• Experimental• Comparative
![Page 3: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/3.jpg)
Approaches and strategies of research design
• Qualitative Strategies– Explanatory Studies
• Case Studies• Ethnographic studies
– Interpretive studies• Grounded theory
– Critical Studies• Action research
![Page 4: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/4.jpg)
Gathering Qualitative Data
• Observation Studies
• Participation Studies
• Interviewing Studies
• Archival studies
• Media Analysis
![Page 5: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/5.jpg)
What Tools Are Used in Designing Research?
![Page 6: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/6.jpg)
Language of Research
Clear conceptualizationof concepts
Shared understandingof concepts
Success of
Research
![Page 7: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/7.jpg)
What Is Research Design?
BlueprintBlueprint
PlanPlan
GuideGuide
FrameworkFramework
![Page 8: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/8.jpg)
MindWriter Project Plan in Gantt chart format
What Tools Are Used in Designing Research?
![Page 9: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/9.jpg)
Design in the Research Process
![Page 10: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/10.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 11: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/11.jpg)
Degree of Question Crystallization
Exploratory Study• Loose structure• Expand understanding• Provide insight• Develop hypotheses
Formal Study• Precise procedures• Begins with
hypotheses• Answers research
questions
![Page 12: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/12.jpg)
Approaches for Exploratory Investigations
• Participant observation• Film, photographs• Psychological testing• Case studies• Ethnography• Expert interviews• Document analysis
![Page 13: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/13.jpg)
Desired Outcomes of Exploratory Studies
• Established range and scope of possible management decisions
• Established major dimensions of research task• Defined a set of subsidiary questions that can
guide research design• Developed hypotheses about possible causes of
management dilemma• Learned which hypotheses can be safely
ignored• Concluded additional research is not needed or
not feasible
![Page 14: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/14.jpg)
Commonly Used Exploratory Techniques
• Secondary Data Analysis
• Experience Surveys
• Focus Groups
![Page 15: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/15.jpg)
Experience Surveys
• What is being done?
• What has been tried in the past with or without success?
• How have things changed?
• Who is involved in the decisions?
• What problem areas can be seen?
• Whom can we count on to assist or participate in the research?
![Page 16: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/16.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 17: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/17.jpg)
Data Collection Method
• Monitoring
• Communication
![Page 18: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/18.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 19: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/19.jpg)
The Time Dimension
• Cross-sectional
• Longitudinal
![Page 20: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/20.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 21: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/21.jpg)
The Topical Scope
Statistical Study• Breadth• Population inferences• Quantitative• Generalizable findings
Case Study• Depth• Detail• Qualitative• Multiple sources of
information
![Page 22: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/22.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 23: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/23.jpg)
The Research Environment
• Field conditions
• Lab conditions
• Simulations
![Page 24: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/24.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 25: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/25.jpg)
Purpose of the Study
Reporting Descriptive
Casual -Explanatory
Causal -Predictive
![Page 26: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/26.jpg)
Descriptive Studies
• Who?
• What?
• How much?
• When?
• Where?
![Page 27: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/27.jpg)
Descriptive Studies
• Description of population characteristics
• Estimates of frequency of characteristics
• Discovery of associations among variables
![Page 28: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/28.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 29: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/29.jpg)
Experimental Effects
Experiment• Study involving the
manipulation or control of one or more variables to determine the effect on another variable
Ex Post Facto Study• After-the-fact report
on what happened to the measured variable
![Page 30: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/30.jpg)
Causation and Experimental Design
• Control/Matching
• Random Assignment
![Page 31: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/31.jpg)
Causal Studies
• Symmetrical
• Reciprocal
• Asymmetrical
![Page 32: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/32.jpg)
Understanding Casual Relationships
Property
Response
Stimulus
Behavior
Disposition
![Page 33: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/33.jpg)
Asymmetrical Casual Relationships
Stimulus-Response
Disposition-Behavior
Property-Behavior
Property-Disposition
![Page 34: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/34.jpg)
Types of Asymmetrical Causal Relationships
• Stimulus-response - An event or change results in a response from some object.– A change in work rules leads to a higher level of worker output.– A change in government economic policy restricts corporate financial decisions.– A price increase results in fewer unit sales.
• Property-disposition - An existing property causes a disposition.– Age and attitudes about saving.– Gender attitudes toward social issues.– Social class and opinions about taxation.
• Disposition-behavior– Opinions about a brand and its purchase.– Job satisfaction and work output.– Moral values and tax cheating.
• Property-behavior -An existing property causes a specific behavior.– Stage of the family life cycle and purchases of furniture.– Social class and family savings patterns.– Age and sports participation.
![Page 35: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/35.jpg)
• Covariance between A and B
• Time order of events
• No other possible causes of B
Evidence of Causality
![Page 36: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/36.jpg)
Descriptors of Research Design
Experimental Effects
Perceptual AwarenessPerceptual Awareness
Research EnvironmentResearch
Environment
DescriptorsDescriptors
Question Crystallization Data Collection
MethodData Collection
Method
Time Dimension
Time Dimension
Topical Scope
Purpose of Study
![Page 37: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/37.jpg)
Participants’ Perceptional Awareness
• No Deviation perceived
• Deviations perceived as unrelated
• Deviations perceived as researcher - induced
![Page 38: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/38.jpg)
Descriptors of Research DesignCategory OptionsThe degree to which the research question has been crystallized
• Exploratory study• Formal study
The method of data collection • Monitoring• Communication Study
The power of the researcher to produce effects in the variables under study
• Experimental• Ex post facto
The purpose of the study • Reporting• Descriptive• Causal-Explanatory• Causal-Predictive
The time dimension • Cross-sectional• Longitudinal
The topical scope—breadth and depth—of the study • Case• Statistical study
The research environment • Field setting• Laboratory research• Simulation
The participants’ perceptional awareness of the research activity
• Actual routine• Modified routine
![Page 39: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/39.jpg)
Readings Review
• Buchanan, D. A. & Bryman, A. (2007). Contextualizing methods choice in organizational research.
1. Choice of methods does not depend exclusively on links to research aims;
2. A method is not merely a technique for snapping reality into focus; choices of method frame the data windows through which phenomena are observed, influencing interpretative schemas and theoretical development.
3. Research competence thus involves addressing coherently the organizational, historical, political, ethical, evidential, and personal factors relevant to an investigation
![Page 40: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/40.jpg)
![Page 41: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/41.jpg)
Organizational Properties
• Negotiated objectives.
• Layered permissions.
• Partisan conclusions.
• Politics of publishing.
![Page 42: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/42.jpg)
Conclusions
• attributes of the organizational research setting or context,
• the research tradition or history relevant to a particular study,
• the inevitable politicization of the organizational researcher’s role,
• constraints imposed by a growing concern with research ethics,
• theoretical and audience-related issues in translating evidence into practice, and
• personal preferences and biases with regard to choice of method.
![Page 43: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/43.jpg)
• Bergh, D. D. & Fairbank, J. F. (2002). Measuring and testing change in strategic management research
• Reliability assumptions of change variables, correlations between the change variable and its initial measure, and selection of unbiased measurement alternatives.
• Found that the typical approach used to measure and test change (as a simple difference between two measures of the same variable) is usually inappropriate and could lead to inaccurate findings and flawed conclusions.
![Page 44: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/44.jpg)
Recommendations
• Researchers to screen their data in terms of: – reliability of component measures– correlation between the component variables– equality of component variable variances; and – correlation with the initial component variable.
![Page 45: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/45.jpg)
Validity and Reliability
Part 2
![Page 46: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/46.jpg)
Validity
• Determines whether the research truly measures that which it was intended to measure or how truthful the research results are.
![Page 47: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/47.jpg)
Reliability
• The extent to which results are consistent over time and an accurate representation of the total population under study is referred to as reliability and if the results of a study can be reproduced under a similar methodology, then the research instrument is considered to be reliable.
![Page 48: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/48.jpg)
Methods to Insure Validity
• Face Validity- This criterion is an assessment of whether a measure appears, on the face of it, to measure the concept it is intended to measure.
• Content validity concerns the extent to which a measure adequately represents all facets of a concept.
• Criterion-related validity applies to instruments than have been developed for usefulness as indicator of specific trait or behavior, either now or in the future.
• Construct Validity - which concerns the extent to which a measure is related to other measures as specified by theory or previous research. Does a measure stack up with other variables the way we expect it to?
![Page 49: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/49.jpg)
Reliability
• Test-Retest Reliability
• Inter-Item Reliability – use of multiple items to measure a single concept.
• Inter-observer Reliability - the extent to which different interviewers or observers using the same measure get equivalent results.
![Page 50: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/50.jpg)
Graphic comparison
![Page 51: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/51.jpg)
How can validity be established?
• Quantitative studies:– measurements, scores, instruments used, research
design
• Qualitative studies:– ways that researchers have devised to establish
credibility: member checking, triangulation, thick description, peer reviews, external audits
![Page 52: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/52.jpg)
How can reliability be established?
• Quantitative studies?– Assumption of repeatability
• Qualitative studies?– Reframe as dependability and confirmability
![Page 53: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/53.jpg)
Reliable
Valid
Not Reliable
Not Valid
Not Valid
Reliability Validity
![Page 54: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/54.jpg)
Methods to insure V&R
• Triangulation – multiple methods or sources
• Controlling for bias– Selection bias– Observer bias– Questionnaire bias– Omitted-variable bias– Confirmation bias
![Page 55: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/55.jpg)
Internal Validity
• High internal validity means that changes in the dependent variable were caused by—not merely related to or correlated with—the treatment.
• High internal validity can be attained only in an ideal true experimental design, or when steps have been taken to eliminate confounding factors or influences.
![Page 56: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/56.jpg)
Internal Validity
Internal validity can be increased by:
• Repeating an experiment many times, and by changing or adding independent variables;
• Switching the control and experimental groups
• Controlling confounding influences.
![Page 57: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/57.jpg)
Internal Validity
Controlled experiments have the virtue of producing results with high internal validity, but they have the liability of low external validity
![Page 58: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/58.jpg)
External Validity
External validity is the extent to which the cause and effect relationship can be generalized to other groups that were not part of the experiment.
![Page 59: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/59.jpg)
External Validity
External validity is critically important in policy and program evaluation research because we would like our research findings to be generalizable. If external validity is low, then the results are not generalizable.
![Page 60: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/60.jpg)
Threats to Validity
History confound —
Any independent variable other than the treatment variable, that occurs between the pre-test and the post-test.
![Page 61: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/61.jpg)
Threats to Validity
Maturation confound —
Refers to the fact that as the experiment is conducted, people are growing older, and are getting more experienced. (Problem in longitudinal studies in particular)
![Page 62: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/62.jpg)
Threats to Validity
Testing confound —
Refers to different responses that an experimental subject may have to questions they are asked over and over again.
![Page 63: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/63.jpg)
Threats to Validity
Instrumentation confound —
Results from changing measurement instruments. (Example: use of different people to collect data, if there is no inter-rate reliability, you have changed the measurement instrument.)
![Page 64: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/64.jpg)
Threats to Validity
Regression to the mean —
Groups tend to increasingly resemble the mean over time.
![Page 65: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/65.jpg)
Threats to Validity
Selection bias confound —
When self-selection by individuals into groups is possible, a selection bias can result.
![Page 66: Research Design Choices and Causal Inferences](https://reader033.vdocuments.site/reader033/viewer/2022051014/552747c7550346e1358b4795/html5/thumbnails/66.jpg)
Threats to Validity
Diffusion of treatment confound —
Occurs when a control group cannot be prevented from receiving the treatment in an experiment.