mixed methods research workshop institute for social research – room 6080 may 16 th, 2012 thomas...

MIXED METHODS RESEARCH WORKSHOPIns t i tu te For Soc ia l Research – Room 6080

May 16 t h , 2012

Thomas N. Templ in , PhDOff ice o f Hea l th Research

Co l l ege o f Nurs ingWayne S ta te Un ive rs i t y

A Statistical Perspective on Mixed Methods

Please do not duplicate or use these slides without the express permission of the author.

Research designs in context

The literature on mixed methods (MM) is large and complex so it is worth taking a look first at what we stand to gain. Put these developments in the context of research

design today. The growth of MM designs over the past few

decades is part of growing trend to make research designs more intelligent, more capable, and more adaptive.

Tom Templin, MCUAAAR workshop 5/2012


Intelligent Uses what we know to learn what we don’t know Example: pretest information

Capable Arrives at the correct population inference Example: all statistical assumptions are met

Adaptive Does not waste resources Example: stopping early



Experimental designs and clinical trials Interim analyses allow multiple looks at the data without

inflating Type 1 error.

Trial can be stopped early for either success or failure (futility).

Sequential multiple assignment randomized trial (SMART) for the development of dynamic treatment regimes. Susan Murphy

Brown, C. H., T. R. Ten Have, et al. (2009). Adaptive Designs for Randomized Trials in Public Health. Annual Review of Public Health. 30: 1-25.



Quasi experimental designs extended the range the interventions we could examine

Cook, T. D. and D. T. Campbell (1979). Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago, Rand McNally.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs: for Generalized Causal Inference.



Causal Modeling

Development of causal modeling theory increased the sophistication of research questions we could address including threats to validity in randomized designs and quasi-experiments

Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66(5).

Pearl, J. (2000). Causality: Models, reasoning, and inference. New York, NY, Cambridge University Press.

Morgan, S. L., & Winship, C. (2007). Counterfactuals and Causal Inference: Methods and Principles for Social Research. New York: Cambridge Univ Press.

Hernan, M., & Robins, J. (2012). Causal Inference: Chapman & Hall/CRC.

Equations plus qualitative information



The Hope for Mixed Methods

Increased credibility of qualitative findings Increased role of qualitative exploration to shape

the direction of the research Increased value of individual differences and

individual variation More intelligent, capable, adaptive

Jennifer Green’s mixed method story


What’s New in Mixed Methods?

We have routinely used open coded items on surveys in convergent (concurrent) designs.

We have always used multi-phased sequential designs in instrument development.

It is not uncommon to include qualitative interviews to determine how participants experience an intervention in embedded designs

We just didn’t use these particular terms



We have routinely used open coded items in survey designs.

We have always used multi-phased designs in instrument development.

It is not uncommon to include qualitative interviews to determine how participants experience an intervention



Mixed methods research design terminology convergent (concurrent or parallel) Sequential (exploratory sequential, explanatory

sequential) Embedded

Terminology to describe how qualitative and

quantitative data can be integrated Merged Connected Embedded



NIH OBSSR “Best Practices for Mixed Methods Research in the Health Sciences Directions for mixed methods research applications to

NIH

Explicit reference to qualitative and quantitative data in the Specific Aims, Significance and Innovation sections

Description of rigorous qualitative and quantitative methodology in the Approach Section



In a concurrent design, investigators often discuss the collection of both types of data (quantitative and qualitative) before discussing the analysis of both types of data (quantitative and qualitative).

Data CollectionQuantitativeQualitative

Data Analysis and InterpretationQuantitativeQualitativeIntegration/Merging Procedures



In a sequential design discuss the collection and analysis of the first type of data (quantitative or qualitative) and then discuss the collection and analysis of the subsequent type of data (qualitative or quantitative).

First Phase (quantitative or qualitative)Data CollectionData Analysis and InterpretationConnecting Procedures (e.g., development of sampling procedures or materials based on the results from the first phase)

Second Phase (qualitative or quantitative)* Data Collection* Data Analysis and Interpretation


MERGING

CONNECTING

EMBEDDING

Three Ways of Integrating Qualitative and Quantitative Data


Integrating qualitative and quantitative data by merging

Merging—1. Qualitative and Quantitative data can be integrated by using words and numbers together. Describing the statistical result and giving examples from the qualitative analysis. In the opening Chapter of Creswell and Plano Clark’s (2011) handbook on mixed methods they give this description of mixed methods as used in everyday life.

Creswell, J. W. & Plano Clark (2011) Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications Inc.



Merging qual and quant data Example 1.“Consider for a moment An Inconvenient Truth, the award-winning documentary on global warming featuring the former U.S. vice president and Nobel Prize winner Al Gore .. In the documentary, Gore narrated both the statistical trends and the stories of his personal journey related to the changing climate and global warming. This documentary brings together both quantitative and qualitative data to tell the story. Also, listen closely to CNN's broadcast reports about hurricanes or about the votes cast in elections. The trends are again supported by the individual stories” (Creswell & Plano Clark, 2011).



Merging ql and qn data-Example 2.

Numerically coded qualitative data can be merged with quantitative data in cross tabulated tables. For example if a qualitative analysis classifies participants in three mutually exclusive classes, say Type A, Type B, and Type C. Then it might be in interest to examine the Type by Gender crosstab.

This crosstab is an example of integration by merging




If the survey contained both Likert type response items and qualitative items with open codes items, it might be of interest to examine the correlation between the Likert items and various recodings of the open ended item.

A specific example is actual report card grade and self reported grade in response to the question, What kinds of grades do you generally get in school?




Short answer questions added to standardized test are coded and factor analyzed with other test items



Mixed method research designs that collect qual and quant data concurrently combine qual and quant by merging as described in the examples above.


Integrating qual and quant data by connecting.

Data are connected when the results of the qual (or quant) in one phase of the study depend in some way on the quant (or qual) data collected in another phase of the study.

Connecting ql and qn--Example 1 Focus groups and experts are used during

Phase 1 to develop questionnaire items for a test that will be administered to the target population in Phase 2.


Integrating qual and quant data by connecting.

Connecting ql and qn--Example 2

In Phase 1, the quantitative analysis ranks students in terms of their reading ability. In Phase 2, the top 5 and the lowest 5 students are selected of in depth interviews about reading attitudes, interests, etc.

Mixed method sequential research designs use

connecting to integrate qual and quant data.


Integrating qual and quant by embedding

Embedding ql and qn—Example 1.

“In this form of integration, a dataset of secondary priority is embedded within a larger, primary design. An example is the collection of supplemental qualitative data about how participants are experiencing an intervention during an experimental trial” (p. 6).


Integrating qual and quant by embedding

Embedding ql and qn—Example 2.“The [researchers] implemented an RCT

study to compare the two treatments in terms of various repeated measure patient outcomes, including pain levels. Embedded within the RCT study, they also gathered qualitative data in the form of audiotapes of the intervention sessions, along with nurse and patient notes, to describe the issues, strategies, and interactions experienced during the intervention. The results provide evaluation of both the outcomes and process of the intervention.” (BPMM, p. 6).


CONVERGENT

SEQUENTIAL

EMBEDDED

Designs for Mixed Methods Research


The OBSSR Best Practices document gives examples of three kinds of mixed methods research designs and notes

that many more are available in the literature.

That is an understatement. Interest in these methods has resulted in a plethora of MM research designs and frameworks for research designs since the Jennifer Greene’s (1989) pioneering review. This can make literature difficult for the novice. Creswell and Plano Clark (2011) listed 15 different

classification schemes. Most used different terminology.

Onwueegbuzie and Collins (2007) noted that a popular handbook of mixed methods contained approximately 35 MM designs.

Jennifer Greene (2007) stated from Maxwell and Loomis, that “the actual diversity in mixed methods studies is far greater than any typology can adequately encompass” (2003), pl. 244). Tom Templin, MCUAAAR workshop

5/2012

Some discontent surrounding mixed method design typologies

In Greene’s mixed method text (2007) she writes, “Muted by the emphasis on design typologies are possible contributions to better understanding that could come from mixes in philosophy, substantive theory, and disciplinary thinking, alongside mixes of differences in personal experience, education, values, and beliefs” (p 15).

Katrin Niglas (2009) noted that in their, “attempts to build more and more exhaustive typologies” … “the terminology in the field: it gets very specific and complicated but remains ambiguous at the same time.

She attempted to deconstruct MM design categories using conventional research design concepts. An interesting approach.

Niglas, Katrin (2009). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 1, 13 - 33.


Convergent (or parallel or concurrent) MM design.

Example from Tony Omwuegbuzie (2007). A study interested in student attitudes toward reading and reading strategies administered a survey containing predetermined questions with Likert type response options and open ended questions to elicit qualitative information about reading strategies.

Since all the information is collected at one time, standard research design terminology would call this a cross sectional design.


Sequential MM design (or explanatory sequential or exploratory sequential)

One data set builds on another. This design involves two distinct interactive phases. Data is collected and analyzed in each phase. The results of Phase 1 inform data collection in Phase 2.

Example 1. Interviews are conducted following the administration of a quality of life instrument to better understand the mechanisms underlying the responses.

What else do we need to specify here?


Sequential MM design (or explanatory sequential or exploratory sequential)

Example 2. Qualitative interviews are conducted to identify risks involved in the treatment of diabetes. A questionnaire is developed and administered to a population of patients.

Typical instrument development design (exploratory sequential); type of data integration is connecting.

See example on connecting.


Embedded MM design

The embedded design can be a variation of a convergent or sequential design but either qual or quant is dominant throughout.

“A prototype would be to conduct an intervention study and to embed qualitative data within the intervention procedures to understand how experimental participants experience the treatment.”


Questions

Seems like the convergent design is the same as a cross sectional design in monomethod research and the sequential design is a phased design approach.


MIXED METHODS IN CONTEXT NEW TERMINOLOGY

NEW BEST PRACTICE GUIDELINES

Conclusions


Mixed methods in context

Think about using mixed methods with advances used in mono method designs

Adaptive experimental designs Causal modeling


New terminology

Terms advanced in the Best Practice Guidelines are probably worth knowing even if not intuitive Data integration terms

Merging Connecting Embedding

Research design Convergent (concurrent or parallel) Sequential (sequential exploratory and sequential

explanatory) Embeded


New best practice guidelines

Guidelines advocate discussion of qual and quant in each part of the application

While qual methods are often used in designs that are not mixed methods designs, mixed methods greatly extends the role of qualitative data by including qual research in study aims, significance, and innovation


Please do not duplicate or use these slides without the express permission of the author.

mixed methods research workshop institute for social research – room 6080 may 16 th, 2012 thomas...

Documents

research designs

randomized designs

adaptive designs

context experimental

context of research

quasiexperimental designs

mcuaaar workshop

social research room