achievingintegrationinmixedmethods designs ...web2.facs.org/orc2014flashdrive/materials/... ·...

Achieving Integration inMixedMethodsDesigns—Principles and PracticesMichael D. Fetters, Leslie A. Curry, and John W. Creswell

Abstract. Mixed methods research offers powerful tools for investigating complexprocesses and systems in health and health care. This article describes integration prin-ciples and practices at three levels in mixed methods research and provides illustrativeexamples. Integration at the study design level occurs through three basic mixedmethod designs—exploratory sequential, explanatory sequential, and convergent—and through four advanced frameworks—multistage, intervention, case study, and par-ticipatory. Integration at the methods level occurs through four approaches. In con-necting, one database links to the other through sampling. With building, one databaseinforms the data collection approach of the other. When merging, the two databasesare brought together for analysis. With embedding, data collection and analysis link atmultiple points. Integration at the interpretation and reporting level occurs throughnarrative, data transformation, and joint display. The fit of integration describes theextent the qualitative and quantitative findings cohere. Understanding these principlesand practices of integration can help health services researchers leverage the strengthsof mixedmethods.Key Words. Qualitative research, survey, sampling, focus groups, biostatisticalmethods, epidemiology, program evaluation, researchmethodology

This article examines key integration principles and practices in mixed meth-ods research. It begins with the role of mixed methods in health servicesresearch and the rationale for integration. Next, a series of principles describehow integration occurs at the study design level, the method level, and theinterpretation and reporting level. After considering the “fit” of integratedqualitative and quantitative data, the article ends with two examples of mixedmethods investigations to illustrate integration practices.

Research Questions and Mixed Methods in Health Services Research

Health services research includes investigation of complex, multilevel pro-cesses, and systems that may require both quantitative and qualitative forms

©Health Research and Educational TrustDOI: 10.1111/1475-6773.12117INTEGRATING MIXED METHODS IN HEALTH SERVICES AND DELIVERYSYSTEM RESEARCH

1

Health Services Research

of data (Creswell, Fetters, and Ivankova 2004; Curry et al. 2013). The natureof the research question drives the choice of methods. Health servicesresearchers use quantitative methodologies to address research questionsabout causality, generalizability, or magnitude of effects. Qualitative method-ologies are applied to research questions to explore why or how a phenome-non occurs, to develop a theory, or to describe the nature of an individual’sexperience. Mixed methods research studies draw upon the strengths of bothquantitative and qualitative approaches and provides an innovative approachfor addressing contemporary issues in health services. As one indication ofthe growing interest in mixed methods research, the Office of Behavioraland Social Sciences at the National Institutes of Health recently developedfor researchers and grant reviewers the first best practices guideline on mixedmethods research from the National Institutes of Health (Creswell et al.2011).

Rationale for Integration

The integration of quantitative and qualitative data can dramatically enhancethe value of mixed methods research (Bryman 2006; Creswell and PlanoClark 2011). Several advantages can accrue from integrating the two formsof data. The qualitative data can be used to assess the validity of quantitativefindings. Quantitative data can also be used to help generate the qualitativesample or explain findings from the qualitative data. Qualitative inquiry caninform development or refinement of quantitative instruments or interven-tions, or generate hypotheses in the qualitative component for testing in thequantitative component (O’Cathain, Murphy, and Nicholl 2010). Althoughthere are many potential gains from data integration, the extent to whichmixed methods studies implement integration remains limited (Bryman2006; Lewin, Glenton, and Oxman 2009). Nevertheless, there are specificapproaches to integrate qualitative and quantitative research procedures anddata (O’Cathain, Murphy, and Nicholl 2010; Creswell and Plano Clark2011). These approaches can be implemented at the design, methods, andinterpretation and reporting levels of research (see Table 1).

Address correspondence to Michael D. Fetters, M.D., M.P.H., M.A., Family Medicine, Universityof Michigan, 1018 Fuller St., Ann Arbor, MI 48104-1213; e-mail: [email protected]. Leslie A.Curry, Ph.D., M.P.H., is with the Yale School of Public Health (Health Policy), New Haven, CT.John W. Creswell, Ph.D., is with Educational Psychology, University of Nebraska-Lincoln, Lin-coln, NE.

2 HSR: Health Services Research

Integration at the Study Design Level

Integration at the design level—the conceptualization of a study—can beaccomplished through three basic designs and four advanced mixed methodsframeworks that incorporate one of the basic designs. Basic designs include (1)exploratory sequential; (2) explanatory sequential; and (3) convergentdesigns. In sequential designs, the intent is to have one phase of the mixedmethods study build on the other, whereas in the convergent designs the intentis to merge the phases in order that the quantitative and qualitative results canbe compared.

In an exploratory sequential design, the researcher first collects and analyzesqualitative data, and these findings inform subsequent quantitative data collec-tion (Onwuegbuzie, Bustamante, and Nelson 2010). For example, Wallaceand colleagues conducted semistructured interviews with medical students,residents, and faculty about computing devices in medical education and usedthe qualitative data to identify key concepts subsequently measured in anonline survey (Wallace, Clark, andWhite 2012).

In an explanatory sequential design, the researcher first collects and ana-lyzes quantitative data, then the findings inform qualitative data collection andanalysis (Ivankova, Creswell, and Stick 2006). For example, Carr exploredthe impact of pain on patient outcomes following surgery by conducting initial

Table 1: Levels of Integration inMixedMethods Research

Integration Level Approaches

Design 3 Basic designsExploratory sequentialExplanatory sequentialConvergent

4 Advanced frameworksMultistageInterventionCase studyParticipatory—Community-based participatory research,and transformative

Methods ConnectingBuildingMergingEmbedding

Interpretation and Reporting Narrative—Weaving, contiguous and stagedData transformationJoint display

Achieving Integration in Mixed Methods Designs 3

surveys about anxiety, depression, and pain that were followed by semistruc-tured interviews to explore further these concepts (Carr 2000).

In a convergent design (sometimes referred to as a concurrent design), thequalitative and quantitative data are collected and analyzed during a similartimeframe. During this timeframe, an interactive approach may be used whereiteratively data collection and analysis drives changes in the data collectionprocedures. For example, initial quantitative findings may influence the focusand kinds of qualitative data that are being collected or vice versa. For exam-ple, in one study Crabtree and colleagues used qualitative findings and quanti-tative findings iteratively in multiple phases such that the data were interactingto inform the final results (Crabtree et al. 2005). In the more common andtechnically simpler variation, qualitative and quantitative data collectionoccurs in parallel and analysis for integration begins well after the data collec-tion process has proceeded or has been completed. Frequently, the two formsof data are analyzed separately and then merged. For example, Saint Arnaultand colleagues conducted multiple surveys using standardized and culturallyadapted instruments as well as ethnographic qualitative interviews to investi-gate how the illness experience, cultural interpretations, and social structuralfactors interact to influence help-seeking among Japanese women (Saint Arna-ult and Fetters 2011).

Advanced frameworks encompass adding to one of the three basicdesigns a larger framework that incorporates the basic design. The largerframework may involve (1) a multistage; (2) an intervention; (3) a case study;or (4) a participatory research framework.

In a multistage mixed methods framework, researchers use multiple stagesof data collection that may include various combinations of exploratorysequential, explanatory sequential, and convergent approaches (Nastasi et al.2007). By definition, such investigations will have multiple stages, definedhere as three or more stages when there is a sequential component, or two ormore stages when there is a convergent component; these differences distin-guishes the multistage framework from the basic mixed methods designs.This type of framework may be used in longitudinal studies focused on eval-uating the design, implementation, and assessment of a program or interven-tion. Krumholz and colleagues have used this design in large-scale outcomesresearch studies (Krumholz, Curry, and Bradley 2011). For example, a studyby their team examining quality of hospital care for patients after heartattacks consisted of three phases: first, a quantitative analysis of risk-stan-dardized mortality rates for patients with heart attacks to identify high andlow performing hospitals; second, a qualitative phase to understand the pro-


cesses, structures, and organizational environments of a purposeful sample oflow and high performers and to generate hypotheses about factors associatedwith performance; and third, primary data collection through surveys of anationally representative sample of hospitals to test these hypotheses quanti-tatively (Curry et al. 2011; Bradley et al. 2012). Ruffin and colleagues con-ducted a multistage mixed methods study to develop and test in arandomized controlled trial (RCT) a website to help users choose a screeningapproach to colorectal cancer. In the first stage, the authors employed a con-vergent design using focus groups and a survey (Ruffin et al. 2009). In thesecond stage, they developed the website based on multiple qualitativeapproaches (Fetters et al. 2004). In the third stage, the authors tested thewebsite in an RCT to assess its effectiveness (Ruffin, Fetters, and Jimbo2007). The multistage framework is the most general framework amongadvanced designs. The additional three frameworks frequently involve multi-ple stages or phases but differ from multistage by having a particular focus.

In an intervention mixed methods framework, the focus is on conducting amixed methods intervention. Qualitative data are collected primarily to sup-port the development of the intervention, to understand contextual factorsduring the intervention that could affect the outcome, and/or explain resultsafter the intervention is completed (Creswell et al. 2009; Lewin, Glenton, andOxman 2009). For example, Plano Clark and colleagues utilized data from apretrial qualitative study to inform the design of a trial developed to comparea low dose and high dose behavioral intervention to improve cancer painmanagement—the trial also included prospective qualitative data collectionduring the trial (PlanoClark et al. 2013). Themethodological approach for inte-grating qualitative data into an intervention pretrial, during the trial, or post-trial is called embedding (see below), and some authors refer to such trials asembedded designs (Creswell et al. 2009; Lewin, Glenton, andOxman 2009).

In a case study framework, both qualitative and quantitative data are col-lected to build a comprehensive understanding of a case, the focus of the study(Yin 1984; Stake 1995). Case study involves intensive and detailed qualitativeand quantitative data collection about the case (Luck, Jackson, and Usher2006). The types of qualitative and quantitative data collected are chosenbased on the nature of the case, feasibility issues, and the research question(s).In one mixed methods case study, Luck and colleagues utilized qualitativedata from participant observation, semistructured interviews, informal fieldinterviews and journaling, and quantitative data about violent events collectedthrough structured observations to understand why nurses under-report vio-lence in the workplace and describe how they handle it (Luck, Jackson, and


Usher 2008). Comparative case studies are an extension of this frameworkand can be formulated in various ways. For example, Crabtree and colleaguesused a comparative case approach to examine the delivery of clinical preven-tive services in family medicine offices (Crabtree et al. 2005).

In a participatory framework, the focus is on involving the voices of the tar-geted population in the research to inform the direction of the research. Oftenresearchers specifically seek to address inequity, health disparities, or a socialinjustice through empowering marginalized or underrepresented populations.The distinguishing feature of a participatory framework is the strong emphasison using mixed methods data collection through combinations of basic mixedmethods designs or even another advanced design, for example, an interven-tion framework such as an RCT. Community-based participatory research (CBPR)is a participatory framework that focuses on social, structural, and physicalenvironmental inequities and engages community members, organizationalrepresentatives, and researchers in all aspects of the research process (Macau-lay et al. 1999; Israel et al. 2001, 2013; Minkler and Wallerstein 2008). In oneCBPR project, Johnson and colleagues used a mixed methods CBPRapproach to collaborate with the Somali community to explore how attitudes,perceptions, and cultural practices such as female genital cutting influencetheir use of reproductive health services—this informed the development ofinterventional programs to improve culturally competent care ( Johnson, Ali,and Shipp 2009). A similar variation involving an emerging participatoryapproach that Mertens refers to as transformative specifically focuses on pro-moting social justice (Mertens 2009, 2012) and has been used with Laotianrefugees (Silka 2009).

Integration at the Methods Level

Creswell and Plano Clark conceptualize integration to occur through linkingthe methods of data collection and analysis (Creswell et al. 2011). Linkingoccurs in several ways: (1) connecting; (2) building; (3) merging; and (4)embedding (Table 2). In a single line of inquiry, integration may occurthrough one or more of these approaches.

Integration through connecting occurs when one type of data links withthe other through the sampling frame. For example, consider a study with a sur-vey and qualitative interviews. The interview participants are selected fromthe population of participants who responded to the survey. Connecting canoccur through sampling regardless of whether the design is explanatorysequential or convergent. That is, if the baseline survey data are analyzed, and


then the participants sampled based on findings from the analysis, then thedesign is explanatory sequential. In contrast, the design is convergent if thedata collection and analyses occur at the same time for the baseline survey andinterviews of all or a subsample of the participants of the survey. A key defin-ing factor in sequential or convergent is how the analysis occurs, eitherthrough building or merging, respectively.

Integration through building occurs when results from one data collec-tion procedure informs the data collection approach of the other procedure, the lat-ter building on the former. Items for inclusion in a survey are built uponpreviously collected qualitative data that generate hypotheses or identify con-structs or language used by research participants. For example, in a projectinvolving the cultural adaptation of the Consumer Assessment of HealthcareProviders and Systems (CAHPS) survey for use in the Arabian Gulf(Hammoud et al. 2012), baseline qualitative interviews identified newdomains of importance such as gender relations, diet, and interpreter use notfound in the existing CAHPS instrument. In addition, phrases participantsused during the interviews informed the wording of individual items.

Integration through merging of data occurs when researchers bring thetwo databases together for analysis and for comparison. Ideally, at the designphase, researchers develop a plan for collecting both forms of data in a waythat will be conducive to merging the databases. For example, if quantitativedata are collected with an instrument with a series of scales, qualitative datacan be collected using parallel or similar questions (Castro et al. 2010).Merging typically occurs after the statistical analysis of the numerical dataand qualitative analysis of the textual data. For example, in a multistagemixed methods study, Tomoaia-Cortisel and colleagues used multiplesources of existing quantitative and qualitative data as well as newly col-lected quantitative and qualitative data (Tomoaia-Cortisel et al. 2013). Theresearchers examined the relationship between quality of care according tokey patient-centered medical home (PCMH) measures, and quantity ofcare using a productivity measure. By merging both scores of quality and

Table 2: Integration throughMethods

Approach Description

Connecting One database links to the other through samplingBuilding One database informs the data collection approach of the otherMerging The two databases are brought together for analysisEmbedding Data collection and analysis link at multiple points


quantity, with qualitative data from interviews, the authors illuminated thedifficulty of achieving highly on both PCMH quality measures and produc-tivity. The authors extended this understanding further by merging staffsatisfaction scores and staff interview data to illustrate the greater work com-plexity but lower satisfaction for staff achieving measures for high-qualitycare (Tomoaia-Cortisel et al. 2013).

Integration through embedding occurs when data collection and analysisare being linked at multiple points and is especially important in interventionaladvanced designs, but it can also occur in other designs. Embedding mayinvolve any combination of connecting, building, or merging, but the hall-mark is recurrently linking qualitative data collection to quantitative datacollection at multiple points. Embedding may occur in the pretrial period,when qualitative (or even a combination of qualitative and quantitative) datacan be used in various ways such as clarifying outcome measures, under-standing contextual factors that could lead to bias and should be controlledfor, or for developing measurement tools to be utilized during the trial. Dur-ing the trial, qualitative data collection can be used to understand contextualfactors that could influence the trial results or provide detailed informationabout the nature of the experience of subjects. Post-trial qualitative data col-lection can be used to explain outliers, debrief subjects or researchers aboutevents or experiences that occurred during the trial, or develop hypothesesabout changes that might be necessary for widespread implementation out-side of a controlled research environment. Such studies require caution toavoid threatening the validity of the trial design. In a site-level controlledtrial of a quality improvement approach for implementing evidence-basedemployment services for patients at specialty mental health clinics, Hamil-ton and colleagues collected semistructured interview data before, during,and after implementation (Hamilton et al. 2013). In another interestingexample, Jaen and colleagues used an embedded approach for evaluatingpractice change in a trial comparing facilitated and self-directed implementa-tion strategies for PCMH. The authors use both embedded quantitative andqualitative evaluation procedures including medical record audit, patientand staff surveys, direct observation, interviews, and text review ( Jaen et al.2010).

Method level integration commonly relates to the type of design used ina study. For example, connecting follows naturally in sequential designs, whilemerging can occur in any design. Embedding generally occurs in an interven-tional design. Thus, the design sets parameters for what methodological inte-gration choices can be made.


Integration at the Interpretation and Reporting Level

Integration of qualitative and quantitative data at the interpretation andreporting level occurs through three approaches: (1) integrating through nar-rative; (2) integrating through data transformation; and (3) integrating throughjoint displays. A variety of strategies have been offered for publishing thatincorporate these approaches (Stange, Crabtree, and Miller 2006; Creswelland Tashakkori 2007).

When integrating through narrative, researchers describe the qualitativeand quantitative findings in a single or series of reports. There are threeapproaches to integration through narrative in research reports. The weavingapproach involves writing both qualitative and quantitative findings togetheron a theme-by-theme or concept-by-concept basis. For example, in their workon vehicle crashes among the elderly, Classen and colleagues used a weavingapproach to integrate results from a national crash dataset and perspectives ofstakeholders to summarize causative factors of vehicle crashes and developempirical guidelines for public health interventions (Classen et al. 2007). Thecontiguous approach to integration involves the presentation of findings within asingle report, but the qualitative and quantitative findings are reported indifferent sections. For example, Carr and colleagues reported survey findingsin the first half of the results section and the qualitative results about contextualfactors in a subsequent part of the report (Carr 2000). In their study of a qualityimprovement approach for implementing evidence-based employmentservices at specialty mental health clinics, Hamilton and colleagues used thisapproach but differ by presenting the qualitative results first and the quantita-tive results second (Hamilton et al. 2013). The staged approach to integrationoften occurs in multistage mixed methods studies when the results of each stepare reported in stages as the data are analyzed and published separately. Forexample, Wilson and colleagues used an intervention mixed methods frame-work involving a clinical trial of usual care, nicotine gum, and gum plus coun-seling on smoking cessation (Wilson et al. 1988). They also used interviews tofind the meaning patients attributed to their stopping smoking (Willms 1991).The authors published the papers separately but in the second publishedpaper, the interview paper, they only briefly mention the original clinical trialpaper.

Integration through data transformation happens in two steps. First, one typeof data must be converted into the other type of data (i.e., qualitative intoquantitative or quantitative into qualitative). Second, the transformed data arethen integrated with the data that have not been transformed. In qualitative


studies, researchers sometimes code the qualitative data and then count thefrequency of codes or domains identified, a process known also as contentanalysis (Krippendorff 2013). Data transformation in the mixed methods con-text refers to transforming the qualitative data into numeric counts and vari-ables using content analysis so that the data can be integrated with aquantitative database. Merging in mixed methods goes beyond content analy-sis by comparing the transformed qualitative data with a quantitative database.Zickmund and colleagues used qualitatively elicited patient views of self trans-formed to a numerical variable, and mortality data to conduct hierarchicalmultivariable logistical modeling (Zickmund et al. 2013).

Researchers have used additional variations. Qualitative data can betransformed to quantitative data, then integrated with illustrative examplesfrom the original qualitative dataset. For example, Ruffin and colleagues trans-formed qualitative responses from focus group data about colorectal cancer(CRC) screening preferences into quantitative variables, and then integratedthese findings with representative quotations from three different constituen-cies (Ruffin et al. 2009). Quantitative data can also be transformed into a quali-tative format that could be used for comparison with qualitatively accesseddata. For example, Pluye and colleagues examined a series of study outcomeswith variable strengths of association that were converted into qualitative levelsand compared across the studies based on patterns found (Pluye et al. 2005).

When integrating through joint displays, researchers integrate the data bybringing the data together through a visual means to draw out new insightsbeyond the information gained from the separate quantitative and qualitativeresults. This can occur through organizing related data in a figure, table,matrix, or graph. In their quality improvement study to enhance colorectalcancer screening in practices, Shaw and colleagues collocated a series of quali-tatively identified factors with CRC screening rates at baseline and 12 monthslater (Shaw et al. 2013).

“Fit” of Data Integration

When using any of these analytical and representation procedures, a potentialquestion of coherence of the quantitative and qualitative findings may occur.The “fit” of data integration refers to coherence of the quantitative and qualita-tive findings. The assessment of fit of integration leads to three possible out-comes. Confirmation occurs when the findings from both types of data confirmthe results of the other. As the two data sources provide similar conclusions,the results have greater credibility. Expansion occurs when the findings from


the two sources of data diverge and expand insights of the phenomenon ofinterest by addressing different aspects of a single phenomenon or by describ-ing complementary aspects of a central phenomenon of interest. For example,quantitative data may speak to the strength of associations while qualitativedata may speak to the nature of those associations. Discordance occurs if thequalitative and quantitative findings are inconsistent, incongruous, contradict,conflict, or disagree with each other. Options for reporting the findings includelooking for potential sources of bias, and examining methodological assump-tions and procedures. Investigators may handle discordant results in differentways such as gathering additional data, re-analyzing existing databases toresolve differences, seeking explanations from theory, or challenging thevalidity of the constructs. Further analysis may occur with the existing data-bases or in follow-up studies. Authors deal with this conundrum by discussingreasons for the conflicting results, identifying potential explanations from the-ory, and laying out future research options (Pluye et al. 2005; Moffatt et al.2006).

Examples Illustrating Integration

Below, two examples of mixed methods illustrate the integration practices.The first study used an exploratory sequential mixed methods design (Curryet al. 2011) and the second used a convergent mixed methods design (Meureret al. 2012).

Example 1. Integration in an Exploratory Sequential Mixed Methods Study—TheSurvival after Acute Myocardial Infarction Study (American College of Cardiology2013). Despite more than a decade of efforts to improve care for patients withacute myocardial infarction (AMI), there remains substantial variation acrosshospitals in mortality rates for patients with AMI (Krumholz et al. 2009;Popescu et al. 2009). Yet the vast majority of this variation remains unex-plained (Bradley et al. 2012), and little is known about how hospitals achievereductions in risk-standardized mortality rates (RSMRs) for patients withAMI. This study sought to understand diverse and complex aspects of AMIcare including hospital structures (e.g., emergency department space), pro-cesses (e.g., emergency response protocols, coordination within hospitalunits), and hospital internal environments (e.g., organizational culture).

Integration through design. An exploratory sequential mixed methodsdesign using both qualitative and quantitative approaches was best suited to


gain a comprehensive understanding of how these features may be related toquality of AMI care as reflected in RSMRs. The 4-year investigation aimed tofirst generate and then empirically test hypotheses concerning hospital-basedefforts that may be associated with lower RSMRs (Figure 1).

Integration through methods. The first phase was a qualitative study of acutecare hospitals in the United States (Curry et al. 2011). Methodological integra-tion occurred through connecting as the 11 hospitals in the purposeful sampleranked in either the top 5 percent or bottom 5 percent of RSMRs for each ofthe two most recent years of data (2005–2006, 2006–2007) from the Centersfor Medicare & Medicaid Services (CMS). The qualitative data from 158 keystaff interviews informed the generation of hypotheses regarding factorspotentially associated with better performance (see Table 3) (Curry et al.2011). These hypotheses were used to build an online quantitative survey thatwas administered in a cross-sectional study of 537 acute care hospitals (91 per-

Aim 1: Generate hypotheses concerning hospital-based efforts that may be associated with RSMRQualitative component connected to CMS national database to identify positivedeviance sample (highest /lowest RSMRs) in 11 hospitals with 158 key hospital staff Analyze data and generate hypotheses

Qualitative paper describes influence of environmental context, eg, organizational values and goals, senior management involvement, staff expertise, communication and coordination among staff, & problem solving and learning (Curry et al. 2011)

8

Aim 2: Test hypotheses and determine hospital efforts that are associated with RSMR Build survey from qualitative data (68 items)Cognitive test and refine survey (n=8)Distribute survey (n=537 hospitals; 91% response)Analyze data and test hypotheses

1

2

4

3

5

6

Qualitative component describes features of high quality discharge processes that may be associated with better hospital care for patients with AMIIn methods, connected to CMS national database for positive deviance sampling per Aim 1Resulting paper illustrates staged integration, and analysis expands qualitative findings by showing comprehensivedischarge processes may reduce RSMR (Cherlin et al. 2013)

Qualitative approach describes the nature of the hospital-emergency services relationships in high performance hospitalsIn methods, connected to the CMS national database using identical positive deviance sampling per Aim 1Resulting paper illustrates staged integration and analysis expands previous findings by showing that high performing hospitals use multifaceted strategies to support collaboration with EMS in AMI care (Landman et al. 2013)

Merge qualitative with quantitativefindings. Resulting paper identifiespredictors of AMI mortality rates (Bradley et al. 2012b)Through weaving narrative, integrate qualitative findings with quantitative findings that confirmfactors influencing RSMRs, and impact on RSMR

RSMR-risk-standardized mortality ratesAMI-acute myocardial infarctionCMS-Center for Medicare and Medicaid Services

Figure 1: Example Illustrating Integration in an Exploratory SequentialMixed Methods Design from the Survival after Acute Myocardial InfarctionStudy


cent response rate) (Curry et al. 2011; Krumholz, Curry, and Bradley 2011;Bradley et al. 2012).

Mixed methods were used to characterize the care practices and pro-cesses in higher performing organizations as well as the organizational envi-ronment where they were implemented. Figure 1 illustrates points in theprocess of integration. In Aim 1, the qualitative component connected with theCMS database in order to identify a positive deviance sample. The investiga-tors conducted a systematic analysis of the qualitative data using a multidisci-plinary team. This provided (point 1, Figure 1) a rich characterization ofprominent themes that distinguished higher-performing from lower-perform-ing hospitals and generated hypotheses regarding factors influencing AMImortality rates (Curry et al. 2011). In Aim 2, the investigators built a 68

Table 3: Examples of How the Qualitative DataWere Used to Build Quanti-tative Survey Items in the Survival after AcuteMyocardial Infarction Study

Domains and Quotations from Qualitative Article Corresponding Survey Item

Broad staff presence and expertise“I started writing my consult note in thephysician progress notes… over the yearsit’s just become the standard… That was away of my breaking into the culture saying,‘This is my note; I want you to read it. It’snot in the nurse’s section. I have someideas… and I’mopen to talking about it.’”(NurseManager, ID #5)

Nurses are comfortable checking withphysicians if they have concerns aboutpatient care. (survey item 65)

Communication and coordination amonggroups“Everyone in this hospital from the housekeeperto the CEO plays a role… The housekeepingneeds to knowwhy it’s important for them to goout and do their job…No one has an insignificantrole in it… So everybody needs to be educated.Everyone.” (Director, Catheterization Laboratory,ID #2)

Clinicians involved in the care of patientswith AMI value each others’ skills andtalents (e.g., physicians’ value nurses’skills and talents and vice-versa). (surveyitem 58)

Problem solving and learning“…the performance improvement team…identifies action steps, the plan is put in place,and then we continue to measure to see if it’sworking or not working… you identify, youintervene, you improve, youmonitor, you tweakand that’s the model that they’ve been using for10 years.” (Director, QualityManagement, ID #4)

After wemake changes to improve AMIcare, we fail to evaluate their effectiveness.(survey item 67)

AMI, acute myocardial infarction; CEO, chief executive officer. Adapted with permission fromBradley, Curry et al., Annals of Internal Medicine, May 1, 2012.


item-survey from the qualitative data. Key concepts from the qualitative data(point 2, Figure 1) were operationalized as quantitative items for inclusion in aweb-based survey in order to test the hypotheses statistically in a nationallyrepresentative sample of hospitals (Bradley et al. 2012). The authors analyzedthe quantitative survey data and then merged the quantitative findings (point 3,Figure 1) and qualitative analysis (point 4, Figure 1) in a single paper. Themerging of the qualitative and quantitative produced a comprehensive, multi-faceted description of factors influencing RSMRs as well as the impact of thesefactors on RSMRs that was presented using a weaving narrative. For example,problem-solving and learning was a prominent theme that differentiatedhigher-performing from lower-performing hospitals. In higher-performinghospitals, adverse events were perceived as opportunities for learning andimprovement, approaches to data feedback were nonpunitive, innovation andcreativity were valued and supported, and new ideas were sought. In the mul-tivariable analysis, having an organizational environment where clinicians areencouraged to creatively solve problems was significantly associated withlower RSMRs (0.84 percentage points). Finally, additional analyses of qualita-tive data examining organizational features related to high-quality dischargeplanning (point 5, Figure 1) (Cherlin et al. 2013), and examining collabora-tions with emergency medical services (point 6, Figure 1) (Landman et al.2013) were also methodologically connected through sampling of high-performing hospitals in the CMS database.

Integration through Interpretation and Reporting. The authors used primarilya staged narrative approach for reporting their results. The process and out-comes of integration of qualitative and quantitative data were primarilydescribed in the quantitative paper (Bradley et al. 2012). The qualitative datainformed the development of domains and concepts for a quantitative survey.Mapping of all survey items to corresponding concepts from the qualitativefindings was reported in a web appendix of the published article. In the pre-sentation of results from the multivariate model, multiple strategies that hadsignificant associations with RSMRs were reported, with a summary of howthese strategies corresponded to five of the six domains from the qualitativecomponent. Quantitative and qualitative findings were synthesized throughnarrative both in the results and discussion using weaving. Key aspects of theorganizational environment included effective communication and collabora-tion among groups, broad staff presence, and expertise. A culture of problemsolving and learning were apparent in the qualitative findings and statisticallyassociated with higher RSMRs in the quantitative findings. Regarding fit, thequantitative findings (Bradley et al. 2012) primarily confirmed the qualitative


findings (Curry et al. 2011). Thus, higher performing hospitals were not distin-guished by specific practices, but instead by organizational environments thatcould foster higher quality care. An accompanying editorial (Davidoff 2012)discusses the complementary relationship between the qualitative and quanti-tative findings, highlighting again the respective purposes of each component.The additional qualitative analyses were published separately (Cherlin et al.2013; Landman et al. 2013) and illustrate staged approach to reporting throughnarrative with ample referencing to the previous studies. This example alsoillustrates expansion of the previously published findings (Stange, Crabtree,andMiller 2006).

Example 2. Integration in a Convergent Mixed Methods Study—The Adaptive DesignsAccelerating Promising Trials into Treatments (ADAPT-IT) Study. The RCT is con-sidered by many trialists to be the gold standard of evidence. Adaptive clinicaltrials (ACTs) have been developed as innovative trials with potential benefitsover traditional trials. However, controversy remains regarding assumptionsmade in ACTs and the validity of results (Berry 2011). Adaptive designs com-prise a spectrum of potential trial design changes (Meurer et al. 2012). A sim-ple adaptation involves early trial termination rules based on statisticalboundaries (Pocock 1977), while a complex adaptation in a dose-finding trialcould identify promising treatments for specific subpopulations and tailorenrollment tomaximize information gained (Yee et al. 2012). The overarchingobjective of ADAPT-IT is “To illustrate and explore how best to use adaptiveclinical trial designs to improve the evaluation of drugs and medical devicesand to use mixed methods to characterize and understand the beliefs, opin-ions, and concerns of key stakeholders during and after the development pro-cess”(Meurer et al. 2012).

Integration through design. One study from the mixed methods evaluationaim of the investigation seeks to describe and compare the beliefs and perspec-tives of key stakeholders in the clinical trial enterprise about potential ethicaladvantages and disadvantages of ACT approaches. A mixed methods conver-gent design was utilized to collect quantitative data through a 22-item ACTsbeliefs survey using questions with a 100-point visual analog scale, and qualita-tive data from unstructured open-response questions on the survey and minifocus group interviews. The scales on the survey instrument assessed beliefsabout ethical advantages and disadvantages of adaptive designs from thepatient, research, and societal perspectives. The qualitative questions on thesurvey and in the interview guides elicited why participants feel there are


advantages or disadvantages to using adaptive designs. A mixed methodsapproach was implemented to elucidate participants’ beliefs, to identify thereasoning behind the beliefs expressed, and to integrate the data together toprovide the broadest possible understanding. Fifty-three individuals partici-pated from the four stakeholder groups: academic clinicians (n = 22); aca-demic biostatisticians (n = 5); consultant biostatisticians (n = 6); and otherstakeholders, including FDA and NIH personnel and patient advocates(n = 20).

Integration through methods. The quantitative and qualitative data werecollected concurrently, and the approach to integration involved merging.With the content of the scales on the survey in mind, the mixed methods teamdeveloped the open-ended responses on the survey and interview questionsfor mini focus groups to parallel visual analog scale (VAS) questions about eth-ical advantages and disadvantages. By making this choice intentionally duringthe design, integration through merging would naturally follow. The researchteam conducted separate analyses of the quantitative and qualitative datain parallel. For the quantitative analytics, the team calculated descriptive statis-tics, mean scores, and standard deviations across the four stakeholder groups.Box plots of the data by group were developed to allow intra- and intergroupcomparisons. For the qualitative analytics, the investigators immersed them-selves in the qualitative database, developed a coding scheme, and conductedthematic searches using the codes. Since the items on the VASs and the ques-tions on the qualitative interview guides were developed in tandem, the codesin the coding scheme were similarly developed based on the items on thescales and the interview questions. As additional themes emerged, codes tocapture these were added. Themethodological procedures facilitated thematicsearches of the text database about perceived ethical advantages and disad-vantages that could be matched and merged with the scaled data on beliefsabout ethical advantages and disadvantages.

Integration through Interpretation and Reporting Procedures. Having orga-nized the quantitative and the qualitative data in a format based on thematicrelevance to allowmerging, higher order integration interpretation was needed.Two approaches were used. First the results from the quantitative and qualita-tive data were integrated using a joint display. As illustrated in Figure 2, theleft provides the participants’ quantitative ratings of their beliefs about the eth-ical advantages as derived from the visual analog scales, with the lowestanchor of 0 signifying definitely not agreeing with the statement and the high-est anchor of 100 signifying definite agreement with the statement. The rightside provides illustrative qualitative data from the free-text responses on the


survey and the mini focus groups. Color matching (see online version) of thebox plots and text responses was devised to help the team match visually thequantitative and qualitative responses from the constituent groups. Multiplesteps in developing the joint display contributed to an interpretation of the data.

In the final report, the quantitative data integration uses a narrativeapproach that describes the quantitative and qualitative results thematically.The specific type of narrative integration isweaving because the results are con-nected to each other thematically, and the qualitative and quantitative dataweave back and forth around similar themes or concepts. The narrative pro-vides intragroup comparisons of the results from the scales about beliefs thatare supported by text from the qualitative database. Each of the six sections ofthe results contain quantitative scores with intergroup comparisons among thefour groups studied, that is, academic researchers, academic biostatisticians,consultant biostatisticians, and “other” stakeholders and quotations from eachgroup.

Regarding the fit of the quantitative and qualitative data, the integrationresulted in an expansion of understanding. The qualitative comments providedinformation about the spectrum of opinions about ethical advantages and dis-

Figure 2: Example of Joint Display Illustrating Integration at the Interpreta-tion and Reporting Level from the ADAPT-IT Project—Potential EthicalAdvantages for Patients When Using Adaptive Clinical Trial Designs


advantages, but the scales in particular were illustrative showing there waspolarization of opinion about these issues among two of the constituencies.

Implications for Practice

This article provides an update on mixed methods designs and principles andpractices for achieving integration at the design, methods, and interpretationand reporting levels. Mixedmethodology offers a new framework for thinkingabout health services research with substantial potential to generate uniqueinsights into multifaceted phenomena related to health care quality, access,and delivery. When research questions would benefit from a mixed methodsapproach, researchers need to make careful choices for integration proce-dures. Due attention to integration at the design, method, and interpretationand reporting levels can enhance the quality of mixed methods health servicesresearch and generate rigorous evidence that matters to patients.

ACKNOWLEDGMENTS

Joint Acknowledgment/Disclosure Statement: At the invitation of Helen I. Meiss-ner, Office of Behavioral and Social Sciences Research, an earlier version ofthis article was presented for the NIH-OBSSR Workshop, “Using MixedMethods to Optimize Dissemination and Implementation of Health Interven-tions,” Natcher Conference Center, NIH Campus, Bethesda, MD, May 3,2012. Beth Ragle assisted with entry of references and formatting. Dr. Fettersacknowledges the other members of the ADAPT-IT project’s Mixed Methodsteam, Laurie J. Legocki, William J. Meurer, and Shirley Frederiksen, for theircontributions to the development of Figure 2.

Disclosures: None.Disclaimers: None.

REFERENCES

American College of Cardiology. 2013. “Surviving MI Intiative” [accessed on May 12,2013]. Available at http://www.cardiosource.org/Science-And-Quality/Quality-Programs/Surviving-MI-Initiative.aspx

Berry, D. A. 2011. “Adaptive Clinical Trials: The Promise and the Caution.” Journal ofClinical Oncology 29 (6): 606–9.


Bradley, E. H., L. A. Curry, E. S. Spatz, J. Herrin, E. J. Cherlin, J. P. Curtis, J. W.Thompson, H. H. Ting, Y. Wang, and H. M. Krumholz. 2012. “Hospital Strate-gies for Reducing Risk-Standardized Mortality Rates in Acute MyocardialInfarction.” Annals of Internal Medicine 156 (9): 618–26.

Bryman, A. 2006. “Integrating Quantitative and Qualitative Research: How Is ItDone?”Qualitative Inquiry 6 (1): 97–113.

Carr, E. 2000. “Exploring the Effect of Postoperative Pain on Patient Outcomes Fol-lowing Surgery.” Acute Pain 3 (4): 183–93.

Castro, F. G., J. G. Kellison, S. J. Boyd, and A. Kopak. 2010. “AMethodology for Con-ducting Integrative Mixed Methods Research and Data Analyses.” Journal ofMixed Methods Research 4 (4): 342–60.

Cherlin, E. J., L. A. Curry, J. W. Thompson, S. R. Greysen, E. Spatz, H. M. Krumholz,and E. H. Bradley. 2013. “Features of High Quality Discharge Planning forPatients Following Acute Myocardial Infarction.” Journal of General Internal Medi-cine 28 (3): 436–43.

Classen, S., E. D. Lopez, S. Winter, K. D. Awadzi, N. Ferree, and C. W. Garvan. 2007.“Population-Based Health Promotion Perspective for Older Driver Safety: Con-ceptual Framework to Intervention Plan.” Clinical Interventions in Aging 2 (4):677–93.

Crabtree, B. F., W. L. Miller, A. F. Tallia, D. J. Cohen, B. DiCicco-Bloom, H. E. McIl-vain, V. A. Aita, J. G. Scott, P. B. Gregory, K. C. Stange, and R. R. McDaniel Jr.2005. “Delivery of Clinical Preventive Services in Family Medicine Offices.”Annals of Family Medicine 3 (5): 430–5.

Creswell, J. W., M. D. Fetters, and N. V. Ivankova. 2004. “Designing a MixedMethodsStudy in Primary Care.” Annals of Family Medicine 2 (1): 7–12.

Creswell, J. W., and V. L. Plano Clark. 2011. Designing and Conducting Mixed MethodsResearch. ThousandOaks, CA: Sage Publications, Inc.

Creswell, J. W., and A. Tashakkori. 2007. “Editorial: Developing Publishable MixedMethodsManuscripts.” Journal of Mixed Methods Research 1 (107): 106–11.

Creswell, J. W., M. D. Fetters, V. L. Plano Clark, and A. Morales. 2009. “Mixed Meth-ods Intervention Trials.” In Mixed Methods Research for Nursing and the HealthSciences, edited by S. Andrew, and E. Halcomb, pp. 161–80. Chichester, WestSussex; Ames, Iowa: Blackwell Publishing.

Creswell, J. W., A. C. Klassen, V. L. Plano Clark, and K. C. Smith, andWorking GroupAssistance. 2011. “Best Practices for Mixed Methods Research in the HealthSciences.” H. I. Meissner, pp. 27. Washington, DC: Office of Behavioral andSocial Sciences Research (OBSSR), National Institutes of Health (NIH).

Curry, L. A., E. Spatz, E. Cherlin, J. W. Thompson, D. Berg, H. H. Ting, C. Decker, H.M. Krumholz, and E. H. Bradley. 2011. “What Distinguishes Top-PerformingHospitals in Acute Myocardial InfarctionMortality Rates? AQualitative Study.”Annals of Internal Medicine 154 (6): 384–90.

Curry, L., H. M. Krumholz, A. O’Cathain, V. Plano Clark, E. Cherlin, and E. H. Brad-ley. 2013. “Mixed Methods in Biomedical and Health Services Research.” Circu-lation 6 (1): 119–23.


Davidoff, F. 2012. “Is Every Defect Really a Treasure?” Annals of Internal Medicine 156(9): 664–5.

Fetters, M. D., N. V. Ivankova, M. T. Ruffin, J. W. Creswell, and D. Power. 2004.“Developing aWeb Site in Primary Care.” Family Medicine 36 (9): 651–9.

Hamilton, A., A. Cohen, D. Glover, F. Whelan, E. Chemerinski, K. P. McNagny, D.Mullins, C. Reist, M. Schubert, and A. S. Young. 2013. “Implementation of Evi-dence-Based Employment Services in Specialty Mental Health: Mixed MethodsResearch.”Health Services Research ???: ???–???.

Hammoud,M.M., M. Elnashar, H. Abdelrahim, A. Khidir, H. A. K. Elliott, A. Killawi,A. I. Padela, A. L. Al Khal, A. Bener, and M. D. Fetters. 2012. “Challenges andOpportunities of US and Arab Collaborations in Health Services Research: ACase Study fromQatar.”Global Journal of Health Science 4 (6): 148–59.

Israel, B. A., A. J. Schulz, E. A. Parker, and A. B. Becker. 2001. “Community-Based Par-ticipatory Research: Policy Recommendations for Promoting a PartnershipApproach in Health Research.” Education for Health 14 (2): 182–97.

Israel, B. A., E. Eng, A. J. Schulz, and E. Parker. 2013.Methods for Community-Based Par-ticipatory Research for Health. San Francisco, CA: Jossey-Bass.

Ivankova, N. V., J. W. Creswell, and S. Stick. 2006. “Using Mixed-Methods SequentialExplanatory Design: FromTheory to Practice.” Field Methods 18 (1): 3–20.

Jaen, C. R., B. F. Crabtree, R. F. Palmer, R. L. Ferrer, P. A. Nutting, W. L. Miller, E. E.Stewart, R. Wood, M. Davila, and K. C. Stange. 2010. “Methods for EvaluatingPractice Change Toward a Patient-Centered Medical Home.” Annals of FamilyMedicine 8 (Suppl 1): S9–20; S92.

Johnson, C. E., S. A. Ali, and M. P. L. Shipp. 2009. “Building Comunity-Based Partici-patory Research Partnerships with a Somali Refugee Community.” AmericanJournal of Preventive Medicine 37 (6S1): S230–6.

Krippendorff, K. 2013. Content Analysis: An Introduction to Its Methodology. 3rd ed. Thou-sandOaks, CA: Sage.

Krumholz, H. M., L. A. Curry, and E. H. Bradley. 2011. “Survival after Acute Myocar-dial Infarction (SAMI) Study: The Design and Implementation of a PositiveDeviance Study.” American Heart Journal 162 (6): 981–7 e9.

Krumholz, H. M., A. R. Merrill, E. M. Schone, G. C. Schreiner, J. Chen, E. H.Bradley, Y. Wang, Z. Lin, B. M. Straube, M. T. Rapp, S. L. Normand, andE. E. Drye. 2009. “Patterns of Hospital Performance in Acute MyocardialInfarction and Heart Failure 30-Day Mortality and Readmission.” Circulation2 (5): 407–13.

Landman, A. B., E. S. Spatz, E. J. Cherlin, H. M. Krumholz, E. H. Bradley, and L. A.Curry. 2013. “Hospital Collaboration with Emergency Medical Services in theCare of Patients with Acute Myocardial Infarction: Perspectives from Key Hos-pital Staff.” Annals of Emergency Medicine 61 (2): 185–95.

Lewin, S., C. Glenton, and A. D. Oxman. 2009. “Use of Qualitative Methods Along-side Randomised Controlled Trials of Complex Healthcare Interventions: Meth-odological Study.” British Medical Journal 339: b3496.

Luck, L., D. Jackson, and K. Usher. 2006. “Case Study: A Bridge across the Para-digms.”Nursing Inquiry 13 (2): 103–9.


———————. 2008. “Innocent or Culpable? Meanings that Emergency Department NursesAscribe to Individual Acts of Violence.” Journal of Clinical Nursing 17 (8): 1071–8.

Macaulay, A. C., L. E. Commanda, W. L. Freeman, N. Gibson, M. L. McCabe, C. M.Robbins, and P. L. Twohig. 1999. “Participatory Research Maximises Commu-nity and Lay Involvement.” British Medical Journal 319 (7212): 774–8.

Mertens, D. M. 2009. Transformative Research and Evaluation. New York: The GuilfordPress.

———————. 2012. “Transformative Mixed Methods: Addressing Inequities.” AmericanBehavioral Scientist 56 (6): 802–13.

Meurer, W. J., R. J. Lewis, D. Tagle, M. Fetters, L. Legocki, S. Berry, J. Connor, V. Dur-kalski, J. Elm, W. Zhao, S. Frederiksen, R. Silbergleit, Y. Palesch, D. A. Berry,and W. G. Barsan. 2012. “An Overview of the Adaptive Designs AcceleratingPromising Trials Into Treatments (ADAPT_IT) Project.” Annals of Emergency Med-icine 60 (4): 451–7.

Minkler, M., and N. Wallerstein. 2008. Community-Based Participatory Research forHealth: From Process to Outcomes. San Francisco, CA: Josey-Bass.

Moffatt, S., M. White, J. Mackintosh, and D. Howel. 2006. “Using Quantitativeand Qualitative Data in Health Services Research—What Happens WhenMixed Methods Findings Conflict?” BMC Health Services Research 6 (28): 1–10.

Nastasi, B. K., J. Hitchcock, S. Sarkar, G. Burkholder, K. Varjas, and A. Jayasena. 2007.“Mixed Methods in Intervention Research: Theory to Adaptation.” Journal ofMixed Methods Research 1 (2): 164–82.

O’Cathain, A., E. Murphy, and J. Nicholl. 2010. “Three Techniques for IntegratingData inMixedMethods Studies.” British Medical Journal 341: c4587.

Onwuegbuzie, A. J., R. M. Bustamante, and J. A. Nelson. 2010. “Mixed Research as aTool for Developing Quantitative Instruments.” Journal of Mixed Methods Research4 (1): 56–78.

Plano Clark, V. L., K. Schumacher, C.West, J. Edrington, L. B. Dunn, A. Harzstark, M.Melisko, M. W. Rabow, P. S. Swift, and C. Miaskowski. 2013. “Practices forEmbedding an Interpretive Qualitative Approach within a Randomized ClinicalTrial.” Journal of Mixed Methods Research 7 (3): 219–42.

Pluye, P., R. M. Grad, L. G. Dunikowski, and R. Stephenson. 2005. “Impact of ClinicalInformation-Retrieval Technology on Physicians: A Literature Review of Quan-titative, Qualitative and Mixed Methods Studies.” International Journal of MedicalInformatics 74 (9): 745–68.

Pocock, S. J. 1977. “Group Sequential Methods in the Design and Analysis of ClinicalTrials.” Biomedics 64 (2): 191–9.

Popescu, I., R. M. Werner, M. S. Vaughan-Sarrazin, and P. Cram. 2009. “Characteris-tics and Outcomes of America’s Lowest-Performing Hospitals: An Analysis ofAcute Myocardial Infarction Hospital Care in the United States.” Circulation 2(3): 221–7.

Ruffin, M. T. I., M. D. Fetters, and M. Jimbo. 2007. “Preference-Based Electronic Deci-sion Aid to Promote Colorectal Cancer Screening: Results of A RandomizedControlled Trial.” Preventive Medicine 45 (4): 267–73.


Ruffin, M. T., J. W. Creswell, M. Jimbo, and M. D. Fetters. 2009. “Factors InfluencingChoices for Colorectal Cancer Screening among Previously Unscreened Africanand Caucasion Americans: Findings from a Mixed Methods Investigation.”Journal of Community Health 34 (2): 79–89.

Saint Arnault, D., and M. D. Fetters. 2011. “RO1 Funding for Mixed MethodsResearch: Lessons Learned from the “Mixed-Method Analysis of JapaneseDepression” Project.” Journal of Mixed Methods Research 5 (4): 309–29.

Scammon, D. L., A. Tomoaia-Cotisel, R. Day, J. Day, J. Kim, N. Waitzman, T. Farrell,andM.Magill. 2013. “Connecting the Dots andMergingMeaning: UsingMixedMethods to Study Primary Care Delivery Transformation.” Health ServicesResearch ???: ???–???. In press.

Shaw, E. K., P. A. Ohman-Strickland, A. Piasecki, S. V. Hudson, J. M. Ferrante, R. R.McDaniel Jr, P. A. Nutting, and B. F. Crabtree. 2013. “Effects of Facilitated TeamMeetings and Learning Collaboratives on Colorectal Cancer Screening Rates inPrimary Care Practices: A Cluster Randomized Trial.” Annals of Family Medicine11 (3): 220–8.

Silka, L. 2009. “Partnership Ethics.” In Handbook of Social Research Ethics, edited by D.M.Mertens, and P. E. Ginsberg, pp. 337–52. ThousandOaks, CA: Sage.

Stake, R. 1995. The Art of Case Study Research. ThousandOaks, CA: Sage.Stange, K. C., B. F. Crabtree, and W. L. Miller. 2006. “Publishing Multimethod

Research.” Annals of Family Medicine 4 (4): 292–4.Wallace, S., M. Clark, and J. White. 2012. “‘It’s On My iPhone’: Attitudes to the Use of

Mobile Computing Devices in Medical Education, A Mixed-Methods Study.”BMJ Open 2 (4): e001099.

Willms, D. G. 1991. “A New Stage, A New Life: Individual Success in Quitting Smok-ing.” Social Science and Medicine 33 (12): 1365–71.

Wilson, D. M., D. W. Taylor, J. R. Gilbert, J. A. Best, E. A. Lindsay, D. G. Willms, andJ. Singer. 1988. “A Randomized Trial of a Family Physician Intervention forSmoking Cessation.” Journal of the American Medical Association 260 (11): 1570–4.

Yee, D., T. Haddad, K. Albain, A. Barker, C. Benz, J. Boughey, M. Buxton, A. J. Chien,A. DeMichele, D. Dilts, A. Elias, P. Haluska, M. Hogarth, A. Hu, N. Hytlon, H.G. Kaplan, G. G. Kelloff, Q. Khan, J. Lang, B. Leyland-Jones, M. Liu, R. Nanda,D. Northfelt, O. I. Olopade, J. Park, B. Parker, D. Parkinson, S. Pearson-White,J. Perlmutter, L. Pusztai, F. Symmans, H. Rugo, D. Tripathy, A. Wallace, D.Wholley, L. Van’t Veer, D. A. Berry, and L. Esserman. 2012. “Adaptive Trials inthe Neoadjuvant Setting: A Model to Safely Tailor Care While AcceleratingDrug Development.” Journal of Clinical Oncology 30 (36): 4584–6; author reply88–9.

Yin, R. K. 1984.Case Study Research: Design and Methods. Beverly Hills, CA: Sage.Zickmund, S., S. Yang, E. P. Mulvey, J. E. Bost, L. A. Shinkunas, and D. R. LaBrecque.

2013. “Predicting Cancer Mortality: Developing a New Cancer Care VariableUsing Mixed Methods and the Quasi-Statistical Approach.” Health ServicesResearch, ???, ???–???.


SUPPORTING INFORMATION

Additional supporting information may be found in the online version of thisarticle:

Appendix SA1: AuthorMatrix.


Shared Decision-Making for Cancer Care Among Racialand Ethnic Minorities: A Systematic Review

To assess decision-

making for cancer treatment

among racial/ethnic minor-

ity patients, we systemati-

cally reviewed and synthe-

sized evidence from studies

of“shareddecision-making,”

“cancer,” and “minority

groups,”usingPubMed,Psy-

cInfo, CINAHL, andEMBASE.

We identified significant

themes that we compared

across studies, refined, and

organized into a conceptual

model.

Five major themes

emerged: treatment deci-

sion-making,patient factors,

family and importantothers,

community, and provider

factors. Thematic data over-

lappedcategories,indicating

that individuals’ preferences

for medical decision-making

cannot be authentically ex-

aminedoutsidethecontextof

family and community.

The shared decision-

making model should be

expanded beyond the tradi-

tionalpatient–physiciandyad

to include other important

stakeholders in the cancer

treatment decision process,

such as family or community

leaders. (AmJPublic Health.

2013;103:e15–e29. doi:10.

2105/AJPH.2013.301631)

Erin L. Mead, MHS, Ardith Z. Doorenbos, RN, PhD, Sara H. Javid, MD, Emily A. Haozous, RN, PhD,Lori Arviso Alvord, MD, David R. Flum, MD, MPH, and Arden M. Morris, MD, MPH

IN SPITE OF DECLINING CANCER

incidence and mortality in theUnited States, minority patientscontinue to have lower 5-yearsurvival rates than non-HispanicWhites.1 Although differences instage at diagnosis may partly ac-count for survival differences,a large proportion of the disparityin survival has been attributed topoorer quality of care, delay inreceipt of treatment, or no receiptof treatment among minority pa-tients.2---8 Specific health systemsor individual providers no doubtcontribute to unnecessary delaysin care. However, patients alsomay have reason to delay or re-fuse treatment, such as a poorpatient---provider relationship ordissatisfaction with the treatmentplan.

Effective patient-centered com-munication and a positive patient---provider relationship have beenassociated with improved treat-ment adherence, survival, andhealth-related quality of lifeamong cancer patients.9,10 Racialand ethnic minority patients are athigher risk than White patientsfor poor communication anda strained patient---provider rela-tionship, which in turn can lead todelayed or nonstandard care.11---13

Incorporating cultural and indi-vidual patient preferences intocancer treatment planningthrough shared decision-making(SDM) may improve adherence totreatment recommendations andreduce delays in care.14

The SDM model emphasizesthe physician’s facilitation of thepatient’s involvement in treatment

decision-making to improve qual-ity of care and patient centered-ness, and thereby improve healthoutcomes. (In health care, “patientcenteredness” includes viewingand treating the patient as a wholeand unique person, rather thansimply focusing on the diseaseprocess.) As opposed to the his-torical paternalistic or informeddecision-making models, the SDMmodel is defined by a 2-way in-formation exchange between thephysician and the patient, fol-lowed by discussion of treatmentpreferences by both parties untilthey reach consensus on a treat-ment decision.15 However, cancerpatients’ preferred level of in-volvement in treatment decision-making may vary. A recent reviewfound mixed effects of the impactof SDM interventions on patients’satisfaction and medication ad-herence.16 Alignment betweenpatients’ preferred and actual de-cisional role may be more impor-tant to patients’ satisfaction thandecisional autonomy.17,18

Taken together, these data sug-gest that the SDM approach maynot be right for every cancer pa-tient. Indeed, the available evi-dence suggests substantial culturalvariation in the cancer treatmentdecision-making process, includ-ing culture-specific differences incommunication style, desiredamount and type of information,and preferred decisional role.15,19

For patients living in a definedsocial and cultural context such asan American Indian/Alaska Na-tive (AI/AN) tribe, cultural con-gruence includes recognition of

the decision-making role playedby other important stakeholdersoutside of the patient---physiciandyad.20,21

To better assess SDM amongracial/ethnic minority cancer pa-tients, we performed a systematicreview of the literature on SDMfor cancer treatment in ethnicminority groups. We createda conceptual model to expand theSDM model and to identify themultilevel determinants of cancertreatment decision-making amongethnic minority groups, which in-clude factors at the level of thepatient, family and significantothers, community, and provider.We summarized the literature ateach level of the model to deter-mine what is currently knownabout cultural variation in cancertreatment decision-making, to ex-pose the relevant gaps in knowl-edge, and to make recommenda-tions for future research.

METHODS

The present systematic litera-ture review followed guidelinesput forth by the Institute of Medi-cine.22 In July 2011, we initiateda search of the published literatureusing PubMed, PsycInfo, CINAHL,and EMBASE, without applyingdate limits. As a first step, wesearched broadly for “shareddecision-making” and “cancer.”Because shared decision-making isa relatively recent concept andtherefore is not a standardizedkeyword search term, we builta search string that captured“shared decision-making”

SYSTEMATIC REVIEW

December 2013, Vol 103, No. 12 | American Journal of Public Health Mead et al. | Peer Reviewed | Systematic Review | e15

individually for each database us-ing the keyword search termsavailable (see appendix, availableas a supplement to this article athttp://www.ajph.org). Next, welimited the search to studies thatwould capture cancer treatmentdecision-making and the impact ofcultural differences on decision-making among racial/ethnic mi-nority populations. “Minoritygroups” were defined as popula-tions who were the racial, ethnic,or cultural minority population inthe country where the study tookplace. As with “shared decision-making,” we developed searchstrategies specific to each databaseto capture the literature on mi-nority populations.

E. L. M. and A. M. M. developedinclusion and exclusion criteria forthe systematic review and for thereview of the search results, usingthe Preferred Reporting Itemsfor Systematic Reviews and Meta-Analyses (PRISMA) schema.23

They excluded studies if theycontained no data about cancer

treatment decision-making, con-tained no data for adult patients,contained no primary data, or didnot report or stratify by cultural,ethnic, or racial minority group.Both quantitative and qualitativestudies were eligible for inclusion.One reviewer assessed 246 ab-stracts (including duplicates) andexcluded 206 on the basis of theexclusion criteria (see the box onthis page) as well as 9 duplicates,leaving 31 eligible abstracts. Anadditional 8 abstracts from thereferences of the eligible abstractsfulfilled criteria and were in-cluded. In the second round ofreview, 2 reviewers independentlyreviewed the 39 full text articles toverify eligibility for inclusion byapplying the inclusion and exclusioncriteria. After discussing discrep-ancies and reaching consensus, thereviewers excluded 16 of thesearticles (PRISMA diagram, Figure 1).

We developed a data abstrac-tion tool for the systematic reviewthat captured detailed data fromboth quantitative and qualitative

studies, including study features,sample information, methods, re-sults, conclusions, and assessmentsof methodological quality. Tworeviewers independently ab-stracted these data points from thefinal articles, and E. L. M. com-bined the results. The reviewersresolved disagreements throughdiscussion until consensus wasreached.

A number of methods exist tosynthesize quantitative and quali-tative evidence from a systematicliterature review, including narra-tive summary, content analysis,and qualitative metasum-mary.24,25 We chose thematicanalysis for the present study be-cause of its organizational andstructural utility for synthesizingquantitative and qualitative evi-dence into coherent themes.24 Weused an inductive approach—thatis, the analysis was directed by thedata content. Six steps of thematicanalysis proceeded as follows.First, 2 reviewers independentlyperformed an in-depth reading ofeach article (familiarization withthe data). Second, they indepen-dently coded data. Third, theysearched the data for significantthemes relating to the role ofshared decision-making in cancertreatment decisions among mi-nority groups. Fourth, the 2 re-viewers discussed, compared, andcontrasted the themes acrossstudies for further refinement until(fifth) they reached consensus onthe final set of data-driven themes,which (sixth) they organized intoa conceptual model.

SDM has been studied in theclinical context of very few cancertypes, and we found no significantdifferences by cancer site in thelimited available evidence for thisstudy. Therefore, we synthesizedfindings for all cancer sites com-bined. We sorted findings andsummarized them by theme and

study type (quantitative vs quali-tative). We report confidence in-tervals for the findings whereavailable. We also describe racial,ethnic, and gender differences,when available.

RESULTS

A final total of 23 articles26---48—11 quantitative studies26---35,48 and12 qualitative studies36---47—ful-filled criteria and were included inthe study (Table 1). Most of the 11quantitative studies focused solelyon breast cancer (n = 8), followedby multiple cancer sites (n = 2)and prostate cancer (n =1). Simi-larly, most of the 12 qualitativestudies focused on breast cancer(n = 9), followed by prostate can-cer (n =1), lung cancer (n =1), andunreported (n =1). Most studieswere conducted in the UnitedStates (n =19). Ten of the quanti-tative studies compared more than1 minority group; African Ameri-cans were the most commonlyrepresented minority (n =14), fol-lowed by Latinas (n = 6), Asians(n =1), and an unspecified “other”(n = 3) minority group. Latinapopulations were frequently di-vided into low-acculturated andhigh-acculturated subgroups, de-fined by language preference (En-glish vs Spanish). By contrast, only4 of the qualitative studies com-pared multiple minority groups.Again, African Americans weremost frequently represented (n =5), followed by Asian (n = 4), La-tina (n = 3), and Jewish (n =1)populations. Within the Asian mi-nority group, Chinese were repre-sented in the most studies, fol-lowed by Punjabis.

Thematic analysis of the quan-titative and qualitative papersrevealed 5 major themes: treat-ment decision-making process,patient factors, factors related tofamily and important others,

Inclusion and Exclusion Criteria for a Systematic Literature

Review on Shared Decision-Making in Cancer Treatment

Among Minority Populations

Inclusion Criteria

d Fit the definition of “shared decision-making.”a

d Reported data on cancer treatment decision-making.

d Collected primary data.

d Included results related to decision-making in a racial/ethnic or cultural

minority group.b

d Published in a peer-reviewed journal.

Exclusion Criteria

d Conducted on decision-making for cancer treatment in children.

d Data not stratified by race/ethnicity or study did not include a minority groupb in

study population.

d Conducted on palliative or end-of-life care in cancer treatment.

a Shared decision-making is defined as a 2-way information exchange between thephysician and the patient followed by discussion of treatment preferences by bothparties until they reach consensus on a treatment decision.15

b A racial/ethnic or cultural minority group is defined as a population that was the racial,ethnic, or cultural minority population in the country where the study took place.

SYSTEMATIC REVIEW

e16 | Systematic Review | Peer Reviewed | Mead et al. American Journal of Public Health | December 2013, Vol 103, No. 12

community factors, and providerfactors. We organized the themesinto a conceptual model of thedecision-making process for can-cer treatment among racial/ethnicminority patients (Figure 2). Sev-eral subthemes emerged, particu-larly among patient factors, whichare described in detail in thissection. Examples of dominantsubthemes include spirituality,

attitudes toward treatment, self-efficacy, acculturation, and advo-cacy (Table 2).

Treatment Decision-Making

Process

The 3 subthemes of treatmentdecision-making process were de-cisional role, decisional regret orsatisfaction, and decisional roleconflict. Among the quantitative

studies, shared and patient-baseddecision-making were most fre-quently used for cancer treatmentdecisions (Table 3). Patients’ de-cisional roles varied among mi-nority groups and non-HispanicWhite patients, but racial differ-ences were not always explicit inthe studies. Adoption of an SDMmodel was reported by 33% to42% of low-acculturated Latina,

high-acculturated Latina, andAfrican American respon-dents.27,30,31,34 Patient-baseddecision-making was reported lessoften among low-acculturatedLatina patients (29%---37%) com-pared with high-acculturated Latinapatients (39%---75%).27,28,33,34

High-acculturated Latina patientswere more likely to report patient-based decision-making than anyother decision-making model,across several studies.27,28,33

Provider-based decision-makingwas less common, reported by only10% to 33% of low-acculturatedLatina patients, 7% to 27% ofhigh-acculturated Latina patients,and 24% to 27% of AfricanAmerican patients.27,28,33,34

Women in all minority groupswere significantly less satisfiedthan White women with their de-cisions and with the decision-making process. Relative totheir White counterparts, low-acculturated Latina patientsreported the highest dissatisfac-tion, with odds ratios ranging from3.6 (95% confidence interval[CI] = 2.9, 6.9]) to 5.5 (95% CI =2.9, 10.5), and highest decisionalregret (odds ratio [OR] = 4.1; 95%CI = 2.2, 8.0), followed by high-acculturated Latina patients (dis-satisfaction OR range = 1.3 [95%CI = 1.0, 1.9] to 3.8 [95% CI =1.6, 5.1]; regret OR = 2.0 [95%CI = 1.2, 3.7]) and African Amer-ican patients (dissatisfaction ORrange = 2.1 [95% CI = 1.3, 3.4] to2.2 [95% CI = 1.7, 3.9]; regretOR= 1.8 [95% CI = 1.1, 3.0]).28,30

Two quantitative studies spe-cifically examined conflict oralignment between patients’ pre-ferred decisional role and theiractual decisional role.28,31 Mostrespondents (69%---93%)reported a match between theiractual and preferred level of de-cisional involvement, althoughlow-acculturated Latina women

Shared Decision-Making and

CancerPubMed 2366

PsycInfo 38 711

CINAHL 683

EMBASE 3562

(with duplicates)

Shared Decision-Making, Cancer,

and Minority GroupsPubMed 106

PsycInfo 10

CINAHL 49

EMBASE 81

(with duplicates)

206 Excluded During Abstract

Review138 Not cancer treatment decision-

making

28 Cancer screening

12 Clinical trial consent

21 Palliative care/end-of-life care

77 Other

9 Cancer treatment decision-making in

children

19 Do not fit the definition of shared

decision-making

43 No primary data collection

24 Does not include or stratify by

cultural, ethnic or racial minority group

8 Other reason (i.e., case study,

conference abstract)

31 Eligible AbstractsPubMed 19

PsycInfo 2

CINAHL 4

EMBASE 6

23 Total Included11 Quantitative studies

12 Qualitative studies

Addition of 8 Abstracts Gathered

From Citations

16 Full Articles Excluded9 Not cancer treatment decision-

making

7 Do not fit the definition of shared

decision-making

1 No primary data collection

5 Does not include or stratify by

cultural, ethnic or racial minority group

Note. Multiple studies fit more than 1 exclusion criteria.

FIGURE 1—Search and selection of published literature on shared decision-making, cancer, and minority

groups through July 2011 in 4 electronic databases.

SYSTEMATIC REVIEW


were more likely to report toolittle involvement. Women whosedecision-making role matchedtheir preferences had the highestsatisfaction with treatment re-ceived, the highest satisfactionwith decision-making process, andthe lowest decisional regret. Bycontrast, women whose decision-making role was less or more thanthey preferred were at increasedrisk of treatment dissatisfaction(OR range =1.5---2.6; P< .005),decision-making process dissatis-faction (OR range = 2.5---3.2;P< .001), and high decisional re-gret (OR range=1.7---2.4; P< .001).31

Qualitative studies indicatedcultural variation in the adoption

of different decision-makingmodels and offered some explan-atory insights (Table 4). High-acculturated Asian Americanwomen preferred a more activedecisional role, believing that ac-tive engagement was necessary toensure that the provider presentedall treatment options.36 A studyof low-acculturated Punjabi womenreported low decisional involve-ment, insufficient informationabout treatments, and a lack of fullengagement in the decision-making process.39 Many (thoughnot all) low-acculturated Chineseand Korean American cancerpatients preferred a provider-based decision-making model.36

Another study concurred thatChinese American womenreported discomfort with the mul-tiple treatment options presented.They felt it indicated a lack ofauthority and expertise and theytried to interpret the “real” rec-ommendation.41 Although 1 studyfound that the majority of Latinaand African American cancer pa-tients preferred a significantamount of provider involvementin decision-making, overall, thesepopulations reported feeling dis-empowered in the decision-making process.42,43,45,46 Theyexpressed the need for more in-formation, empowerment to self-advocate, and physicians’ respectfor patients’ autonomy. In a studyof Haredi Ashkenazi Jewishwomen, respondents reportedthat, although they preferreda rabbi-as-agent model similar tothe provider-based decision-mak-ing model, they struggled with theconsequences and benefits of del-egating their decision to an au-thority figure.38 One study of Af-rican American women withbreast cancer found that lack ofparticipation in decision-makingled many of them to reject che-motherapy.43 Thus, the sense ofdisempowerment from poorlyaligned decision-making prefer-ences and roles could sometimesresult in nonadherence to cancertreatment.

Patient Factors

Patient factors that influencedthe cancer treatment decision-making process were primarilyfocused on attitudes and deeplyheld beliefs, as well as tangibleaspects of a marginalized socialstatus. The dominant subthemeswithin patient factors includedlevel of acculturation or languageissues, spirituality and fatalism,attitudes about treatment and de-cision-making, and self-efficacy.

We investigated socioeconomicstatus (SES) as a potential sub-theme, but the limited literaturedid not enable an in-depth inves-tigation of the effect of SES in-dicators on the decision-makingprocess among racial/ethnicminority patients. One studyof African American patientsreported similar decisional in-volvement by SES status, althoughthe small sample size (n = 26)limited the conclusions that couldbe drawn from the findings.34

Level of acculturation andlanguage issues. Six quantitativestudies included an analytic com-ponent to investigate differencesin decision-making by minoritypatients’ level of acculturation andpreferred language.26---30,33 Fiveof the studies investigateddecision-making differences be-tween low-acculturated and high-acculturated Latina breast cancerpatients.27---30,33 As detailed pre-viously, investigators found statis-tically significant differences indecision-making, decision-makingoutcomes, and decision-makingfactors by acculturation and lan-guage. For example, comparedwith low-acculturated Latina pa-tients, more high-acculturated pa-tients reported patient-based de-cision-making and fewer reportedprovider-based decision-mak-ing.27,28 Low-acculturated pa-tients tended to be least satisfiedwith their decision-making roleand outcome. One potential ex-planatory mechanism for thesedifferences by level of accultura-tion was the need for translation inclinical encounters. In 1 study,only 6% of high-acculturated La-tina patients reported needingtranslation in the clinical encoun-ter, compared with 89% of low-acculturated Latina patients.28

A doctor or staff member trans-lated for 52% of low-acculturatedpatients, and 36% reported

TABLE 1—Characteristics of Articles Included in a Systematic

Literature Review on Shared Decision-Making in Cancer

Treatment Among Minority Populations

Characteristic

Quantitative Studies

(n = 11), No. (%)

Qualitative Studies

(n = 12), No. (%)

Cancer site

Breast 8 (73) 9 (75)

Prostate 1 (9) 1 (8)

Lung 0 1 (8)

Multiple 2 (18) 0

Not reported 0 1 (8)

Study location

United States 10 (91) 9 (75)

Outside United States 1 (9) 3 (25)

Study populationa

Patients 11 (100) 11 (92)

Patients’ family and others 0 3 (25)

Health care providers 1 (9) 3 (25)

Minority race/ethnicity of study populationa

African American 9 (82) 5 (42)

Latina/Latino 6 (55) 3 (25)

Asian 1 (9) 4 (33)

Haredi Ashkenazi Jewish 0 1 (8)

Non-Hispanic White 9 (82) 3 (25)

Other (not specified) 3 (27) 1 (8)

Level of decision-makinga

Patient 11 (100) 12 (100)

Family and others 5 (45) 8 (67)

Community 0 1 (8)

aSome studies had more than 1 subcategory.

SYSTEMATIC REVIEW


translation by family or friends.The final study examined the roleof low acculturation in nondisclo-sure of diagnosis among minoritypatients in Singapore,26 and byextension deferment of decision-making to family members. Notspeaking English was associatedwith 7.6 greater odds (95% CI =1.7, 34.5; P= .009) of nondisclo-sure of diagnosis among Chinese,Malay, and other minority patientsin Singapore.Spirituality and fatalism. A sin-

gle quantitative study34 amongAfrican Americans explored in-trinsic religiosity, defined as thedegree to which participants be-lieve that they have internalized,and live in accordance with, theirreligious principles. In this study,patients who shared decision-making had lower average religios-ity scores than those who reportedphysician-based decision-making(mean score = 13.5 and 14.7 units,respectively; P= .02).

Several qualitative studiesexplicated the role of patient

spirituality or “faith in God” incancer treatment decision-mak-ing.36,40,43,45,47 Spirituality wascited repeatedly as both a copingmechanism and a factor in thedecision-making process. Spiritu-ality was a particularly strongcoping mechanism among AfricanAmericans.40,45 Specifically, spiri-tuality played a strong role inhaving faith in the providers,strengthening the relationshipwith providers, and following theproviders’ recommendations fortreatment. In these studies, prayerwas not viewed as a sufficienttreatment option by itself.40,45 Forseveral minority groups, spiritual-ity may have limited patients’ rolein decision-making by causingthem to put their trust in theirproviders and in their religiousfaith; alternatively, highly spiritualpatients may have had a psycho-social characteristic that also stim-ulated a high level of faith intheir providers. For example, ina study of low-acculturated AsianAmericans, providers reported

that patients were more passivein treatment decision-making.36

According to their providers, thesepatients often viewed their cancerdiagnosis and treatment outcomeas part of “God’s will” and reliedheavily on prayer, which limitedtheir active participation in deci-sion-making.Attitudes about treatment and

decision-making. Three quantita-tive studies explored the impact ofpatients’ attitudes about treatmentand decision-making on the de-cisional process, with mixed find-ings.27,29,34 One study of AfricanAmerican men and women foundthat patients held strong support-ive attitudes for autonomy ininformation seeking, but not au-tonomy in decision-making.34 Pa-tients with higher educationallevels had stronger supportive at-titudes for autonomy in informa-tion seeking and decision-making,but attitudes were similar acrossincome level. Concerns about re-currence, survival, and radiationwere self-reported as the most

significant attitudinal factors influ-encing treatment decision-makingfor Latina, African American,andWhite women.27,29 One studyof breast cancer patients27

found that, compared with low-acculturated Latina and Whitewomen, significantly more AfricanAmerican and high-acculturatedLatina women reported concernsabout recurrence (61.4%---64.1%vs 55.6%---55.8% of patients; Pfor overall trend = .044) and bodyimage (22.3%---25.3% vs 17.2%---19.8%; P for overall trend = .01)as important factors in decision-making. Significantly fewer Whitewomen reported radiation con-cerns as a major decisional influ-ence than low-acculturated Latina,high-acculturated Latina, and Af-rican American women (17.2%vs 26.8%---30.9%; P for overalltrend < .001). Another study29

found that, although concernsabout survival and radiation didnot significantly differ betweengroups, low-acculturated Latinawomen rated concerns aboutappearance (i.e., body image;P £ .001), surgical consequences(P £ .001), and cost (P £ .001)as greater decisional influencesthan did high-acculturated Latinaand White women.

Four qualitative studies ex-plored the theme of patients’ atti-tudes in treatment and treatmentdecision-making in greater detail.Many of the factors or concernsaround decision-making were indirect conflict, thereby heighten-ing patients’ anxiety.36,38,43,44

For example, a study with low-acculturated Korean and ChineseAmerican patients and high-acculturated Asian American pa-tients found that adopting a “posi-tive attitude” played a central rolein treatment and treatmentdecision-making for Korean pa-tients and for highly acculturatedpatients.36 The highly acculturated

TREATMENT DECISION-MAKING PROCESSDecisional role, decisional regret or satisfaction,

decisional role conflict

HEALTH CARE PROVIDER

Preferences and recommendations, communication and information-giving, conflict and cultural

congruence

Collective experience

Norms andvalues

Communication

COMMUNITY

PATIENTSpirituality and

fatalism, attitudes, self-

efficacy,

acculturation

Participationin decision-making

Participationand treatment

Ceding autonomy

Vicarious experiences

FAMILY AND OTHERS

FIGURE 2—A conceptual model of influences on cancer treatment decision-making among racial/ethnic

minority patients.

SYSTEMATIC REVIEW


group defined a positive attitudeas taking an active decisional role,whereas the Korean group viewedit as deferring to the provider indecision-making. Concurrent withthe desire to adopt or displaya positive attitude were concernsabout pain and suffering (Koreanand high-acculturated AsianAmerican patients), public dis-plays of suffering (Koreans), andsurvival (Chinese and high-acculturated Asian Americans). Instudies of Latinawomenwith breastcancer, fear of chemotherapy andside effects was a significant factorin treatment decision-making,44

concurrent with deep fears of re-currence (presumably reduced byuse of chemotherapy).43

Self-efficacy. Two quantitativestudies with similar study popula-tions examined patients’ self-efficacy in communication with

surgeons as a factor in cancertreatment decision-making.32,33 In1 study, White, Latina, and Afri-can American women who feltefficacious in their ability to com-municate with their surgeons hadlower odds of deferring the finaltreatment decision to familymembers (OR = 0.95; 95% CI =0.91, 0.99; P= .009) than womenwho did not feel efficacious.32 Theother study found no associationbetween self-efficacy and cancerknowledge or treatment outcomesacross White, Latina, and AfricanAmerican women.33 Neitherreported the distribution of self-efficacy scores across racial/ethnicgroups. A qualitative study of La-tina breast cancer patients44 in-dicated that self-efficacy in makingdecisions and communicating withproviders was a key theme dis-cussed in the focus groups.

Factors Regarding Family and

Important Others

Family and important othersplayed a role in the cancer treat-ment decision-making process.The dominant subthemes withinthis level of factors included par-ticipation and advocacy in deci-sion-making, impact of participa-tion and advocacy on receipt oftreatment, consequences of cedingdecision-making autonomy to im-portant others, and the experi-ences of important others.Participation and advocacy of

important others in decision-making. Six quantitative studiesexamined the role of family andimportant others in the clinicalencounter in general and decision-making specifically.26---29,33,35

Types of family and importantothers differed by study, includingall companions, spouse or partner,

family, and friends. A generaltrend existed for family and othersplaying an important role for low-acculturated Latina women anda smaller role for African Ameri-can and White women. Twostudies found that most women(70%---79% overall) had a com-panion present at their consulta-tion with the provider.27,28

African American women had alower frequency of accompaniment(71.2%) than either low-accul-turated and high-acculturated La-tina women or White women(77.5%---79.2%; P for overalltrend = .032).27

The importance of companiontypes in decision-making and thedecision role they played differedsignificantly by race/ethnicity.High-acculturated Latina women(59.3%) were more likely toindicate that their spouse or part-ner played an important rolein decision-making than low-acculturated Latina women (54.3%),White women (51.6%), and Af-rican American women (44.5%;P for overall trend = .015).27

Despite the apparent importanceof spouse or partner in decision-making, another study foundthat no high-acculturated Latinawomen and very few low-acculturated Latina women (4.2%)or African American women(3.0%) identified the spouse orpartner as the final treatmentdecision-maker (P for group dif-ferences = .19).33 Family andfriends appeared to play a moresignificant role than spouses orpartners in decision-making,especially among low-acculturatedLatina patients.27,29,33 Low-acculturated Latina women weremuch more likely to report family(75.9%) and friends (45.4%) asimportant in decision-making thanAfrican American women (50.3%and 25.6%, respectively), high-acculturated Latina women

TABLE 2—Organization of References Into Conceptual Model Themes in a Systematic Literature Review

on Shared Decision-Making in Cancer Treatment Among Minority Populations

Theme and Subthemes Quantitative References Qualitative References

Treatment decision-making process

Decisional role 26–28,30–34 36,38–47

Conflict between actual and preferred decisional role 28,31 39,41,43,45,46

Satisfaction or regret with decision-making or decision 28,31,48 39,41,43–46

Patient factors

Level of acculturation and language 26–29,33,48 36,37,41,42,44

Spirituality and fatalism 34 36,40,43–45,47

Attitudes about treatment and decision 27,29,34 36,38,43,45

Self-efficacy 32,33 44

Factors of family and important others

Participation or advocacy of others in decision-making 26–29,33,35

Participation or advocacy of others and receipt of treatment 26–29,33,35

Consequences of ceding decision-making autonomy 37–39,41,42,44,45,47

Vicarious experiences 35 40,43,45

Community factors

Cultural or community norms and values 36–38,40,41,45

Communication with the community 36,37,42,45

Decision-making as a collective experience 37,39,45

Provider factors

Provider preferences and recommendations 27,29,30,35 36,39,41,43,47

Provider communication and information giving 30,32,35 36,39,41–46

Conflict and cultural congruence in the patient–provider relationship 41,42,44

SYSTEMATIC REVIEW


TABLE 3—Summarization of Findings on Shared Decision-Making From the Quantitative Literature

Article Main Outcome Variables Findings

Back and Huak26 Nondisclosure of diagnosis A “family-centered decision-making model” was important, with family-initiated nondisclosure of

diagnosis to patients in 58 of 66 cases. Nondisclosure occurred more often when patients were older

(OR = 7.7; 95% CI = 3.5, 16.8), female (OR = 6.0; 95% CI = 2.7, 13.7), non-English speaking (OR =

7.6; 95% CI = 1.7, 34.5), and in palliative care (OR = 3.2; 95% CI = 1.3, 7.9).

Hawley et al.28 Match between actual and preferred decision-

making role; satisfaction with decision or

decision-making process

Decision-making: 37% of patients reported shared decision-making. Low-acculturated Latina women

were more likely to report surgeon-based decision-making (31% vs 22%–28%; P not reported) and less

likely to report patient-based decision-making (30% vs 33%–41%; P < .1). 93% of patients reported

a match between preferred and actual involvement.

Dissatisfaction and regret: low-acculturated Latina women had the greatest likelihood of dissatisfaction

(OR = 5.5; 95% CI = 2.9, 10.5) and regret (OR = 4.1; 95% CI = 2.2, 8.0). Companion presence was

associated with lower dissatisfaction (OR = 0.65; 95% CI = 0.44, 0.96).

Hawley et al.27 Treatment received Patient–provider decision-making: surgeon-based decision-making was highest (33.1% vs 24.0%–26.7%)

and patient-based decision-making lowest (29.5% vs 33.4%–39.8%) among low-acculturated Latina

women.

Decision-making role of family and important others: African Americans had the significantly lowest

frequency of being accompanied by anyone to a consultation (71.2% vs 77.5%–79.2%; P = .032).

Family role in decision-making was significantly highest among low-acculturated Latina women

(75.9%), followed by African American (50.3%), high-acculturated Latina (49.4%), and White women

(34.1%; P for overall trend < .001).

Accompaniment to the consultation was associated with greater receipt of mastectomy (relative risk =

1.62; 95% CI = 1.14, 2.21).

Kaplan et al.29 Type of treatment received Patient–provider decision-making: more White women reported that the provider indicated radiation as

optional (32.2%) than did high-acculturated Latina (18.7%) and low-acculturated Latina (10.2%)

women.

Provider recommendation and concerns about survival were the most important influences on surgery

decision-making reported by all ethnicities.

Compared with a recommendation that radiation was optional, provider indication that radiation was

necessary was significantly associated with radiation therapy (OR = 8.05; 95% CI = 4.04, 16.03).

Among patients with breast-conserving surgery who did not undergo radiation therapy, low-acculturated

Latina (67%) and high-acculturated Latina (69%) patients more frequently reported lack of provider

recommendation as the reason than White women (57%).

Family influence on decision-making: low-acculturated Latina women rated family influences (P £ .01) asa greater influence on decision-making than other groups.

Katz et al.30 Type of treatment received; treatment delay;

satisfaction with decision or decision-making

process

Low-acculturated Latina women had highest odds of low satisfaction with the decision-making process

(OR = 3.6; 95% CI = 2.9, 6.9), followed by African American women (OR = 2.2; 95% CI = 1.7, 3.9) and

high-acculturated Latina women (OR = 1.3; 95% CI = 1.0, 1.9; P for overall trend < .001).

Katz et al.48 Treatment received Decision-making: 37.1% of all women reported shared decision-making, 41.0% reported patient-based

decision-making, and 21.9% reported provider-based decision-making. Patient-based decision-making

was associated with the highest rate of mastectomy receipt.

Communication with surgeons: ethnic differences in mastectomy receipt were partially related to

information exchange with surgeons. Among women who reported surgeon-based decision-making,

more White women reported a discussion only about breast-conserving surgery (50.7%) than did African

American (31.4%) and “other” women (26.0%; P = .029). Surgeons were more likely to recommend

breast-conserving surgery to White women (51.6%) than to African American (41.7%) and “other”

women (41.0%; P < .001).

Continued

SYSTEMATIC REVIEW


(49.4% and 18.2%), and Whitewomen (34.1% and 13.7%; Pfor overall trend < .001).27

Moreover, significantly morelow-acculturated Latina womenidentified family or friend as thefinal decision-maker (49.3%)than African American women(3.0%), high-acculturated Latinawomen (17.9%), and Whitewomen (2.2%; P for group

differences < .001).33 A study ofAsian patients in Singapore foundthat a “family-centered” decision-making model was most impor-tant to this population.26

Only 1 study adjusted for po-tential confounders to study therole of important stakeholdersoutside the patient---provider dyadin minority patient cancer treat-ment decision-making.33

Adjusting for demographic andclinical factors, patients’ self-efficacy, social support, and pro-viders’ participatory style, theauthors found that both low-acculturated Latina women (OR =7.97; 95% CI = 2.43, 26.20) andhigh-acculturated Latina women(OR = 4.48; 95% CI = 1.09,18.45) were significantly morelikely than White and African

American women to have familymaking the cancer treatment de-cision. Women who felt efficaciousin their ability to communicatewith surgeons (OR = 0.95; 95%CI = 0.91, 0.99) and whosesurgeons had a participatorydecision-making style (OR = 0.98;95% CI = 0.97, 0.997) were lesslikely to have the family as thefinal decision-maker.

TABLE 3—Continued

Lantz et al.31 Satisfaction with treatment; satisfaction with

decision or decision-making process; regret

with decision or decision-making process

Satisfaction with decision-making: most patients were satisfied with the decision-making process

(80.6%). Low satisfaction with decision-making was associated with being African American (OR = 1.56;

P < .001) and being “other” ethnicity (OR = 2.21; P < .001). Women whose decision-making role was

less than or more than their preferred role had increased risk of low satisfaction with the decision-

making process (OR = 3.23 [P < .001] and 2.48 [P < .001], respectively).

Regret with decision-making: increased regret was associated with being African American (OR = 1.82;

P < .001) and being “other” ethnicity (OR = 2.58; P < .001). Women whose decision-making role was less

than or more than their preferred role had increased risk of regret (OR = 2.42 [P < .001] and 1.71

[P < .001], respectively).

Maly et al.33 Type of treatment received; final treatment

decision-maker

Final decision-maker: compared with other ethnicities, low-acculturated Latina women were the least

likely to identify themselves as the final decision-maker (36.6%) and most likely to have family or friend

make the final decision (49.3%).

Family as the decision-maker: low-acculturated Latina (OR = 7.97; 95% CI = 2.43, 26.20) and high-

acculturated Latina (OR = 4.48; 95% CI = 1.09, 18.45) women were significantly more likely than White

and African American women to have family making the treatment decision. Women who felt efficacious

in their ability to communicate with surgeons were less likely to have the family as the final decision-

maker (OR = 0.95; 95% CI = 0.91, 0.99). Women whose surgeons had a participatory decision-making

style were less likely to have the family as the final decision-maker (OR = 0.98; 95% CI = 0.97, 0.997).

Family decision-making was associated with lower odds of breast-conserving surgery (OR = 0.39; 95%

CI = 0.18, 0.85).

Maly et al.32 Breast cancer knowledge; treatment delay; type

of treatment received

Interactive information giving was associated with greater odds of breast cancer knowledge (OR = 1.15;

95% CI = 1.03, 1.27) and breast-conserving surgery (OR = 1.18; 95% CI = 1.05, 1.31) and lower odds

of treatment delay (OR = 0.81; 95% CI = 0.72, 0.91).

Phipps et al.34 Patients’ decision-making role Most patients (42%) reported shared decision-making, followed by patient-based decision-making

(31%). Patients reporting shared decision-making had significantly lower religiosity than those reporting

provider-based decision-making (P = .02) and minimally higher than those reporting patient-based

decision-making (P = .28).

Steenland et al.35 Type of treatment received Provider communication: African American men were more likely to report communication difficulties with

the provider than White men (OR = 3.95; 95% CI = 1.52, 10.30). Men who received no treatment were

more likely to report poor communication with the provider than men who received treatment (OR =

5.77; 95% CI = 1.88, 11.46).

Provider recommendation: provider recommendation was inversely associated with surgery (OR = 0.25;

95% CI = 0.11, 0.60) and no treatment (OR = 0.16; 95% CI = 0.05, 0.52).

Spouse, family, and friend recommendation: spouse recommendation was not associated with any

treatment. Other family or friend recommendation was positively associated with receiving no treatment

(OR = 4.47; 95% CI = 1.45, 13.8) but was not associated with any other treatment choice.

Note. CI = confidence interval; OR = odds ratio.

SYSTEMATIC REVIEW


TABLE 4—Summarization of Findings on Shared Decision-Making From the Qualitative Literature

Article Research Question Main Findings for Decision-Making

Ashing et al.36 What are the cultural and socio-ecological factors that

influence the breast cancer experience of Chinese, Korean,

and “mixed” Asian American women?

Health care professionals’ perspective: Asian American patients (1) exhibited passivity in decision-

making and fatalism toward their cancer; (2) tended to rely on word-of-mouth over provider

recommendations.

Patients’ perspective: less-acculturated patients depended on the provider for treatment decisions

because of the provider’s authority. More acculturated women preferred a more active decisional

role. Attitudes (concern about burdening the family, sexual health, concern for cost of treatment)

influenced treatment decision-making.

Balneaves et al.37 What are the experiences and responses of the family

members of immigrant Punjabi women diagnosed with

breast cancer?

The accepted cultural norm is nondisclosure of a cancer diagnosis to the community and in some

cases to the patient. Family advocated for women in the clinical encounter, had to be assertive

with providers, and helped with language barriers.

Coleman-

Brueckheimer

et al.38

To what extent and in what way do Haredi rabbinical

authorities participate in breast cancer treatment

decision-making?

Involving a rabbi in the decision-making process provided cultural meaning and relieved guilt and

blame, but carried the risk of a binding decision. Involving a culture broker in the decision-making

was considered advantageous because of high levels of trust in the broker, the broker’s experience

interacting with the medical world, and nonbinding decisions, but patients were concerned about

confidentiality and gender role conflicts.

Howard et al.39 What are the experiences of Punjabi women with breast

cancer, as described by their personal narratives?

Many patients reported following the providers’ treatment advice without question. For some

women, family’s decisions took precedence over their own preferences. The women were focused

more on the collective, rather than the individual, experiences of breast cancer. However, they felt

that they were not given enough information and did not feel fully engaged in the decision-making

process.

Jones et al.40 What are the cultural beliefs and attitudes of African American

prostate cancer survivors regarding the use of

complementary alternative medicine?

Spirituality played a significant role in treatment decision-making for patients who used

complementary alternative medicine. Almost all of the men believed that prayer helped to fight

again prostate cancer, but none thought that prayer by itself would be sufficient. They believed

that God works through health care providers. Trust in providers and in their knowledge was very

important for treatment decision-making. Patients reported that changing cultural beliefs have

changed treatment decision-making intergenerationally.

Killoran and Moyer41 What are the cultural factors that influence the treatment

decision-making by Chinese American patients and the

presentation of treatment options by providers?

There was a cultural belief that vanity is dangerous and therefore mastectomy is safer than breast-

conserving therapy, which was considered vain. Family, friends, and community reinforced these

beliefs among the patients, affecting treatment decision-making and regret.

Patient–provider miscommunication: patients felt in conflict with the provider’s decision because

the provider did not respect their choice for mastectomy, their fears, and their beliefs. This

resulted in a loss of faith in provider, dissatisfaction with treatment, or uncertainty regarding

treatment choice, and consideration of nonadherence.

Provider presentation of treatment: triangulation of multiple data sources showed that providers did

not use the mandated handbook on treatment options correctly. Providers conveyed more bias in

the discussion of treatment options than they realized. There was a wide variation in providers’

opinions and presentation of options. There was also wide variation in providers’ views about

Asian American patients and how to present information to them, including using the family as

a guide for nondisclosure. Patients felt uncomfortable with being presented multiple options,

thinking that it indicated a lack of authority and expertise.

Kreling et al.42 What are older breast cancer patients’ attitudes toward

chemotherapy, and what factors influenced their treatment

decision-making?

Most patients reported a provider-based decision-making style, which they preferred. Decision-

making for chemotherapy was promoted by (1) tailored communication, which included family

members; (2) provision of support services for women without support networks; and (3) culturally

sensitive provider communication.

Latina women faced barriers to chemotherapy use that included employment, insurance, and

language issues. They felt little empowerment to seek information. African American women

reported that class and education were important determinants of how providers treated them

and that they had to be advocates for themselves to be treated well. African American women

tended to keep their diagnosis secret, resulting in lack of support from the community.

Continued

SYSTEMATIC REVIEW


Participation and advocacy ofothers and receipt of treatment. Sixquantitative studies examined theimpact of a companion’s presenceand decision-making participation

on treatment-related outcomes,with mixed results.26---29,33,35 Onestudy found that a companion’spresence at the decision-makingclinical encounter was associated

with lower decision dissatisfactionamong cancer patients (OR =0.65; 95% CI = 0.44, 0.96), butnot regret (OR = 0.72; 95% CI =0.43, 1.1).28 A companion’s

presence was positively associatedwith receipt of mastectomy (rela-tive risk [RR] =1.62; 95% CI =1.14, 2.21), whereas a strongspousal decision-making role was

TABLE 4—Continued

Napoles-

Springer et al.43How do understanding of ductal carcinoma in situ diagnosis,

treatment decision-making processes, and satisfaction with

care compare between Latina and White women?

Understanding diagnosis, prognosis, and treatment: White women had a greater understanding of

diagnosis, prognosis, and treatment than Latina women. Low-acculturated Latina women wanted

more information about surgical treatment.

Treatment decision-making process: most White women reported shared decision-making and were

more likely to report patient-based decision-making than Latina women. Latina women reported

equal occurrence of shared decision-making and provider-based decision-making, and were more

likely than White women to report provider-based decision-making. Significant factors affecting

decision-making were friends’ and others’ prior experiences, fear of recurrence, and providers’

recommendation. Less-cited factors included attitudes about the procedure, spiritual beliefs, and

cost. The factors did not differ by ethnicity.

Treatment and decisional role satisfaction: patients were satisfied with their lack of decisional

involvement and with their treatment decisions when providers’ rationale for the decision was

explained clearly and when patients believed themselves to be well informed. Patients were

dissatisfied with their lack of involvement if they did not know the other treatment options or

providers’ rationale. Patients were dissatisfied with their treatment if they perceived that the

providers were unable to commit to a specific treatment recommendation.

Sheppard et al.45 What are the key perceptions, enablers, and nurturers that

influence African American women’s breast cancer adjuvant

therapy decisions?

A common theme was cultural identity and the importance of empowering African American women

with information to self-advocate during the decision-making process. Negative perceptions of

breast cancer treatment were based primarily on the observation of others receiving treatment.

Non–African American providers more often reported their patients as fatalistic. Strong spiritual

beliefs were not a primary decision-making resource for these women.

Patients were less confident in treatment decisions if they reported poor communication with their

providers. Good communication was associated with treatment satisfaction, treatment knowledge,

and use of recommended adjuvant treatment. Lack of decision-making participation, passivity in

the interaction with providers, and limited questioning led several women to reject chemotherapy.

Collectivism, family, the faith community, and others played important roles.

Sheppard et al.44 What are the factors that influence Latinas’’ factors that

influence Latina’ breast cancer treatment decision-making?

Women needed better communication with providers and more information about the disease and

treatment in Spanish. The family played a significant decisional role for Latina women. Factors

that affected treatment decision-making included for following: positive provider–patient

interactions, knowledge of diagnosis and treatment options, self-efficacy in decision-making, low

communication skills or language barriers, cancer fatalism, and cultural incongruence with the

providers. Patients reported not enough time for decision-making.

Williams et al.46 What are the perceptions of communication and decision-

making with providers among African American patients

living with cancer and their families?

Provider–patient communication and quality of life: Participants stressed the importance of effective

provider–patient communication for quality of life, including recognizing the patient’s individuality

and tailoring communication.

Decision-making: effective communication, a sense of control (i.e., desire for shared decision-

making), and providers’ respect for patients’ autonomy was important for effective decision-

making. Respondents noted that not all patients have the capacity to self-advocate because of old

age, poverty, or low education.

Zhang and Siminoff47 What are the major differences of opinion between patients

and family members while making treatment decisions, and

why do they exist?

African American families exerted limited pressure on providers. By contrast, White families

disagreed with providers on many issues, with or without patients’ concurrence. No African

American patients switched doctors. In contrast, 38% of White patients switched doctors and

more than 66% were persuaded by caregivers to do so. African American patients and caregivers

reported following providers’ recommendations closely and relying heavily on their faith in God.

Family discordance in treatment decision-making was common (not stratified by race).

SYSTEMATIC REVIEW


negatively associated with mas-tectomy (RR = 0.53; 95% CI =0.36, 0.78).27 The spousal rolewas not associated with any treat-ment of prostate cancer.35 Inter-actions between others’ decision-making role and race/ethnicitywere not reported in the studies.Consequences of ceding

decision-making autonomy. Eightqualitative studies explored nu-ances and consequences of therole of family and other importantstakeholders outside the patient---provider dyad in decision-making.For racial and ethnic minoritycancer patients, family memberswere important advocates in theclinical encounter and providedsocial and decisional support, fa-cilitated communication with pro-viders, and helped overcome lan-guage barriers.37---39,41,42,44,45

Family members commonly hada substantial impact on treatmentdecision-making and out-comes,37,39,41,42,44,45 and in somecases impeded receipt of treat-ment42,45 or requested nondisclo-sure of diagnosis to the patient.37

In several studies, the treatmentpreferences of family and othersdominated patients’ preferences,specifically with Chinese, Punjabi,and Haredi Ashkenazi pa-tients.38,39,41 For the most part,deferring to others’ decisions fortheir cancer treatment appeared togive patients more satisfactionwith treatment and the decision-making process, whereas opposingothers’ recommendations or pref-erences could lead to decisionalregret.38,39,41 However, a numberof patients also expressed anxietyor regret associated with leavingthe decision up to others, particu-larly when the treatment prefer-ences of the patients and otherswere conflicted.38,41

Vicarious experiences. In addi-tion to directly affecting treat-ment decision-making through

decisional participation and rec-ommendations, family, friends,and other important stakeholdersindirectly influenced patients’cancer treatment decision-makingthrough their personal experi-ences with cancer. In 1 smallquantitative study of prostatecancer treatment among AfricanAmerican men, patients weremore likely to choose brachyther-apy because they had friends whohad received this treatment (OR =2.04; P £ .1).35

Three qualitative studies inves-tigated the theme of others’ cancerexperiences and treatmentdecision-making with AfricanAmerican prostate cancer pa-tients,40 Latina and White breastcancer patients,43 and AfricanAmerican breast cancer pa-tients.45 Prior experiences offriends and family43 as well ascancer survivor mentors45 sub-stantially influenced decision-making about treatments andsometimes engendered a fatalisticattitude toward treatment. Simi-larly, vicarious experiences werecited as reasons for the use ofcomplementary and alternativemedical cancer treatments,40

sometimes without the knowledgeof the biomedical provider.

Community Factors

In addition to main partners,family, friends, and other impor-tant stakeholders, the patients’community played both positiveand negative roles in the cancertreatment decision-making pro-cess. Studies used the term “com-munity” to describe a variety oftypes of groups of individuals, in-cluding the faith community,neighborhoods, cancer survivors,and an unspecified “community.”The dominant subthemes in-cluded cultural and communitynorms and values, communicationwith the community, and

decision-making as a collectiveprocess.Cultural and community norms

and values. One way throughwhich the community influencedthe decision-making process wasthe formation and communicationof collective norms and values,which was explored in 5 qualita-tive studies.37,38,40,41,45 Culturalnorms and values actively shapedwhich treatment to receive forChinese American and Haredibreast cancer patients in 2 stud-ies.38,41 Many Chinese Americanbreast cancer patients reportedcultural pressure to receive modi-fied radical mastectomy instead ofbreast-conserving surgery, even ifthe former procedure was notmedically required, becausebreast-conserving surgery wasseen as “vain” and oncologicallyless safe.41 The desire to adhere toreligious law figured prominentlyin Haredi breast cancer patients’decisions about what treatment toreceive and when to receive it.38

Moreover, rabbis and culturalbrokers served as sources of in-formation for cultural norms andvalues for treatment decision-making and, for some patients,made the treatment decision.Among these women, concernsabout guilt, blame, and confiden-tiality and fears about a bindingand disagreeable treatment deci-sion were important in identifyingwhich, if any, religious authorityfigure they would defer theirtreatment decision-making to.38 Astudy of African American menfound that increased formal edu-cation compared with previousgenerations and changing culturalnorms about medical treatmentresulted in a new preference forallopathic medicine over root andfolk medicine to treat cancer.40

In addition to influencing whichtreatments to receive, culturalnorms shaped disclosure and

others’ involvement. The culturalnorm of collectivism—defined asa social, communal way of livingcharacterized by mutual obliga-tions and expectations withingroups—played a positive rolethrough the support and involve-ment of family and the faith com-munity.45 However, it could alsoplay an adverse role if family in-volvement was negative, leadingto treatment delay and limiteddisclosure of the condition to thefamily.Communication with the

community. Four qualitative stud-ies explored communication withthe community during the cancertreatment decision-making processin minority populations.36,37,42,45

Fear of stigmatization influencedmany patients not to disclose theircancer diagnosis to the community,which resulted in lack of com-munity support.37,45 Many breastcancer patients reported little or nocommunication about their diag-nosis and treatment with theirneighbors36; however, commu-nication with the community ofcancer survivors was viewed asimportant.45

Decision-making as a collectiveexperience. For Punjabi and Afri-can American breast cancer pa-tients, the cancer diagnosis andtreatment decision-making wasviewed as a collective experienceshared with their family, but mostPunjabi and some African Ameri-can patients kept the diagnosis se-cret from the community to avoidstigmatization.37,45 Because of theirdesire to adhere to Jewish law,Haredi breast cancer patientsviewed treatment decision-makingand the treatment experience as acollective experience shared withintheir religious community.38

Provider Factors

Providers’ preferences andrecommendations. Five large

SYSTEMATIC REVIEW


surveys confirm that providers’recommendations were the mostimportant positive influence ontreatment decision-making over-all.29---31,35,48 Conversely, the per-ceived absence of providers’ ad-vocacy of a particular treatmentwas associated with reduced up-take of that treatment. In somestudies, the strength of the pro-vider’s advocacy varied by thepatient’s ethnicity. For example,among breast cancer patients whohad breast-conserving therapybut did not undergo radiationtherapy, 67% of low-acculturatedLatina patients, 69% of high-acculturated Latina patients, and57% of White patients reportedlack of provider recommendationas the reason.29 Compared witha recommendation that radiationwas optional, providers’ indicationthat radiation was necessary wassignificantly associated with re-ceipt of radiation therapy (OR =8.05; 95% CI = 4.04, 16.03).Providers’ indication that it wasunnecessary (or no indication) wassignificantly associated with non-receipt of radiation (OR = 0.09;95% CI = 0.04, 0.19). In anotherlarge survey, racial differences inthe operation type appeared tobe related to perceived commu-nication content.30 Comparedwith Whites, African Americanbreast cancer patients reportedthat surgeons placed less emphasison breast-conserving therapy.These patients in turn were morelikely to opt for mastectomy overbreast-conserving therapy. Nota-bly, abdicating the decision-making role to providers wasalso associated with the highestrates of decisional regret anddissatisfaction.31

Five qualitative studies under-scored the perceived importanceof displaying respect for theproviders’ status as an authorityfigure across racial and ethnic

groups.36,39,41,43,47 In a study ofpatients with advanced lung can-cer, African American patientswere more likely than Whites tofollow providers’ recommenda-tions even when in conflict withthe family.47 The authors inter-preted higher compliance amongminority patients as a sign of re-duced agency and fewer advo-cates within the hospital andhealth care system. Similarly, instudies of Asian women withbreast cancer, patients oftenconsented to the provider-recommended treatment evenwhen it was in conflict with per-sonal or community values.36,41

Provider communication andinformation giving. Provider com-munication permeated nearly allaspects of the decision-makingprocess. Three quantitative studiesreported on dimensions of com-munication that went beyondsimple information giving. Com-pared with White women withbreast cancer, African Americanwomen in a population-basedsurvey reported less optimal en-gagement from their surgeonsabout their desires and expecta-tions for surgery, leading to moredecisional uncertainty.30 Thesefindings were confirmed and ex-tended by another study of White,Latina, and African Americanwomen32 that found that interac-tive information giving on the partof the surgeon was associated withhigher odds of knowledge (OR =1.15; 95% CI = 1.03, 1.27), lowerodds of delay in treatment (OR =0.81; 95% CI = 0.72, 0.91), andhigher odds of breast-conservingtherapy (OR = 1.18; 95% CI =1.05, 1.31). Racial/ethnic differ-ences in knowledge and use ofbreast-conserving therapy disap-peared after adjustment for pro-vider communication style.Finally, a survey of AfricanAmerican and White men with

prostate cancer35 found that Afri-can American men were 4 timesmore likely to report poor com-munication with their providers(OR = 3.95; 95% CI = 1.52,10.30), and this in turn was highlyassociated with nonreceipt oftreatment (OR = 5.77; 95%CI = 1.88, 11.46).

Eight qualitative studies ex-plored the effect of providercommunication on the cancertreatment decision-making pro-cess.36,39,41---46 Across studies, thetheme of needing more providertime, individualized attention,and culturally sensitive communi-cation was commonly iterated.Study participants indicated thatprovider-based decision-makingwas acceptable provided the pro-viders’ rationale was clearlyexplained to them.43---45 Patientswanted informative communica-tion that included both elicitationof their values and guidance fortreatment.

Supportive communicationfrom the provider was as impor-tant for treatment adherence asinformative communication. Ina study comparing Latina andWhite women with ductal carci-noma in situ, Latinas consistentlyreported more distress than Whitewomen, which was alleviatedwhen providers were sensitive totheir emotional needs.43 Partici-pants were dissatisfied when pro-viders did not provide supportivecommunication and highly appre-ciative when it was provided.36

Conflict and cultural congruencein the patient---provider relationship.Three qualitative studies41,42,44

discussed conflict related to thepresence or absence of culturalcongruence (defined as cultural di-versity, awareness, sensitivity, andcompetence) at the level of thehealth care provider.49 In someinstances, the cultural focus wasSES, as in a study of African

American breast cancer patientswho reported that class and edu-cation were important determi-nants in how providers treatedthem.42 They felt that they had toadvocate in the clinical encounterto be treated well by the clinician.In other instances, language bar-riers contributed to culturally in-congruent interactions. For exam-ple, Latina women reported thatthe lack of supportive communica-tion, language barriers, and other-wise incongruent interactionsmadethem fearful of asking questionsand leaving the decision to theprovider.44 Patients who perceivedthat providers did not respect theirchoices, fears, or beliefs lost faithin their providers and considerednonadherence to treatment recom-mendations.41

DISCUSSION

We reviewed a broad array ofstudies examining treatmentdecision-making among racial/ethnic minorities with cancer andfound several commonalitiesacross studies. Common themes fitinto multiple categories, includingtreatment decision-making pro-cess, individual patient factors,factors related to family and im-portant others, community factors,and provider factors. Thematicdata overlapped categories sub-stantially, indicating that individ-uals’ perceptions and preferencesfor medical decision-making can-not be authentically examinedoutside the context of family andcommunity—and that this may beespecially so among ethnic mi-nority patients. In addition, theSDM process is operationalizedduring communication betweenthe patient and the provider. Thus,the important linkages betweenthemes were social support,communication, and culturalcongruence.

SYSTEMATIC REVIEW


Social Support

Social support was generallyprovided by spouse and family butcould also include friends, com-munity members, and members orleaders of a religious community.Although several studies notedthat social support had the poten-tial of reducing individual auton-omy and contributing to nonre-ceipt of treatment, strong socialsupport more commonly resultedin decision-making for more ag-gressive treatment. These findingsare supported across other dis-eases50 and have been expandedby studies indicating that pro-viders are more likely to recom-mend more aggressive treatmentwhen they perceive strong socialsupport.51,52 In fact, the presenceof strong social support was asso-ciated robustly with improvedcancer survival,53 suggesting thatsocial support may be an appro-priate area for intervention in apatient-centered approach to can-cer care.

Communication

Among cancer patients, effec-tive provider communication wasassociated with improved psycho-logical outcomes (e.g., reducedanxiety, improved psychologicaladjustment to diagnosis), quality oflife, and physiological function-ing,14 as well as significantly in-creased adherence to treatmentrecommendations.54 Despite thesebenefits, we found that cancer pa-tients continued to have high ratesof unmet communication needsfrom their providers, and theseunmet needs were amplifiedamong racial/ethnic minority pa-tients. Our findings are supportedby the few studies conducted onprovider communication with in-digenous patients worldwide,which identify significant chal-lenges and a high prevalenceof miscommunication.19,21,55---57

Challenges to communication in-clude different communicationstyles, different medical beliefmodels, language barriers, lack ofprovider skills in cross-culturalcommunication, lack of patients’control, lack of providers’ knowl-edge of indigenous culture andhistory, distrust of providers andthe health care system, lack ofa personal relationship betweenthe provider and patient, and analienating health care environ-ment.21,55---58 Cultural values andnorms among racial/ethnic minor-ity populations may have a signifi-cant impact on patient---providercommunication, which warrantsfurther investigation.59

Cultural Congruence

We found that cultural congru-ence was primarily explored interms of language among quanti-tative studies. Across studies, low-acculturated minority womenreported less participation in andsatisfaction with cancer treatmentdecision-making, which sometimesled to nonadherence to care.The qualitative studies were morelikely to explore nuances of cul-tural competence and to offer ex-amples of how providers mightbridge cultural divides. Given thata culturally congruent approachto care has been associated withimproved communication, less useof costly services (such as clinicvs emergency department), andbetter health outcomes,49 the casefor understanding and strength-ening cultural congruence amongproviders is strong. This is espe-cially so among socially vulnerableminority patients, for whom familyand community relationships arean important source of identityand support.

This systematic review was ini-tially motivated by an inquiry intocancer treatment decision-makingprocesses among AI/AN patients,

but our search was limited by adearth of relevant data. AlthoughAI/AN groups have the lowestcancer rates among racial/ethnicminority populations in the UnitedStates, they also have the highestmortality and lowest 5-year sur-vival rates after diagnosis.1,60,61 Aswith other racial/ethnic minoritygroups, differences in quality oftreatment, delay in receipt oftreatment, and lack of treatmentlikely play a significant role incancer survival disparities,2,3,7,8,62

and enhancing the patient---pro-vider relationship and satisfactionwith the treatment plan may playan important role in addressingthese disparities. In our experi-ence, traditional models of SDMmay be inappropriate to AI/ANcommunities.49,63 The commu-nity and family play significantroles in health care decision-making for AI/AN patients.20,21,55

Moreover, miscommunication be-tween health care providers andindigenous populations is perva-sive and often unrecognized, withnegative consequences for theclinical encounter and treatmentadherence.19 Incorporating cul-tural and individual preferencesinto the treatment decision-making process may substantiallyimprove cross-cultural communi-cation between providers and AI/AN patients. The SDM modelshould therefore be extendedbeyond the traditional patient---physician dyad, especially amongAI/AN cancer patients.

Our review has limitations thatmust be noted. First, as with anysystematic review, our search mayhave been subject to publicationbias. To mitigate this, we usedcomprehensive search terms,searched several large databaseslinked to different disciplines inwhich the topic is studied, andcarefully reviewed the referencelists of included publications.

However, it is possible that im-portant data may not have beenpublished or that papers may havebeen missed. Second, we includedstudies with a broad variety ofracial and ethnic minority patients,which certainly cannot representthe experiences or perspectivesof a single group. However, witha broad search we can understandcommon experiences and themesin cancer treatment decision-making among all minority pa-tients. Furthermore, the findings ofour review are limited by theavailable literature. Most of thestudies included in our reviewwere on breast cancer, based inthe United States, and conductedamong African Americans, whichmay limit the generalizability ofthe findings to other cancer sites,countries, and racial and ethnicminority groups. Because our re-view was motivated by treatmentdecision-making in AI/AN popu-lations, the model needs to beapplied and further refined in thisgroup to enhance utility.

In spite of these limitations, webelieve that the synthesized datapresent a compelling need to ex-pand the SDM model beyond thetraditional patient---provider dyadto other significant stakeholders,including the family, social sup-porters, and the minority commu-nity. Decision aids for treatmentand screening decisions have beenfound to improve the decision-making process,64 and furtherwork is needed to develop suchaids to include important stake-holders outside the patient---providerdyad. An extended, shareddecision-making approach may beparticularly appropriate amongAI/ANs, who are at higher risk fornonadherence to cancer care andfor cancer mortality than anyother ethnic group in the UnitedStates. Although individualAI/ANs may choose not to reside

SYSTEMATIC REVIEW


on reservation lands or not toparticipate in tribal activities, AI/ANs who are more strongly affili-ated with traditional native culturehave been significantly less likelyto be adherent to cancer-screeningregimens.20,65,66 The sensitiveinclusion of family members ortribal leaders in the cancer treat-ment decision process could facil-itate communication and culturalcongruence, and may ultimatelyhave an impact on adherence torecommended cancer care. j

About the AuthorsErin L. Mead is with the Department ofHealth, Behavior, and Society, JohnsHopkins Bloomberg School of PublicHealth, Baltimore, MD. Ardith Z.Doorenbos is with the Department ofBiobehavioral Nursing and Health Systems,University of Washington School ofNursing, Seattle. Sara H. Javid and DavidR. Flum are with the Department ofSurgery, University of Washington Schoolof Medicine, Seattle. Emily A. Haozous iswith the University of New Mexico Collegeof Nursing, Albuquerque. Lori ArvisoAlvord is with the University of ArizonaCollege of Medicine, Tucson. Arden M.Morris is with the Center for HealthOutcomes and Policy, Department ofSurgery, University of Michigan MedicalSchool, Ann Arbor.Correspondence should be sent to Arden

M. Morris, MD, MPH, Associate Professor ofSurgery, Associate Professor of HealthBehavior Health Education, University ofMichigan, 1500 E Medical Center Dr,TC-2920, Ann Arbor, MI 48109-0331(e-mail: [email protected]). Reprints canbe ordered at http://www.ajph.org by clickingthe “Reprints” link.

This article was accepted August 28,2013.

ContributorsE. L. Mead and A. M. Morris conductedthe literature review and the analysis anddrafted the article. A. Z. Doorenbos, S. H.Javid, E. A. Haozous, L. A. Alvord, and D.R. Flum provided essential review andrevisions. All authors contributed to theconceptualization of this review andapproved the final version.

AcknowledgmentsE. L. Mead was supported by a NationalInstitutes of Health National ResearchService Award (T32 CA009314).

These data were presented in a sub-stantially abbreviated poster format at the

Academy Health Annual Research Meet-ing; June 24---26, 2012; Orlando, FL.

Human Participant ProtectionNo protocol approval was required becauseno human participants were involved.

References1. Siegel R, Naishadham D, Jemal A.Cancer statistics, 2013. CA Cancer J Clin.2013;63(1):11---30.

2. Caplan LS, May DS, Richardson LC.Time to diagnosis and treatment of breastcancer: results from the National Breastand Cervical Cancer Early DetectionProgram, 1991---1995. Am J PublicHealth. 2000;90(1):130---134.

3. Fesinmeyer MD, Goulart B, BloughDK, Buchwald D, Ramsey SD. Lung can-cer histology, stage, treatment, and sur-vival in American Indians and AlaskaNatives and whites. Cancer. 2010;116(20):4810---4816.

4. Morris AM, Billingsley KG, HayangaAJ, Matthews B, Baldwin LM, BirkmeyerJD. Residual treatment disparities afteroncology referral for rectal cancer. J NatlCancer Inst. 2008;100(10):738---744.

5. Morris AM, Rhoads KF, Stain SC,Birkmeyer JD. Understanding racial dis-parities in cancer treatment and out-comes. J Am Coll Surg. 2010;211(1):105---113.

6. Revels SL, Morris AM, Reddy RM,Akateh C, Wong SL. Racial disparities inesophageal cancer outcomes. Ann SurgOncol. 2013;20(4):1136---1141.

7. Shavers VL, Brown ML. Racial andethnic disparities in the receipt of cancertreatment. J Natl Cancer Inst. 2002;94(5):334---357.

8. Wilson RT, Adams-Cameron M,Burhansstipanov L, et al. Disparities inbreast cancer treatment among AmericanIndian, Hispanic and non-Hispanic whitewomen enrolled in Medicare. J HealthCare Poor Underserved. 2007;18(3):648---664.

9. Street RL Jr, Voigt B. Patient partic-ipation in deciding breast cancer treat-ment and subsequent quality of life. MedDecis Making. 1997;17(3):298---306.

10. Tattersall MH, Butow PN, ClaytonJM. Insights from cancer patient commu-nication research. Hematol Oncol ClinNorth Am. 2002;16(3):731---743.

11. Ashton CM, Haidet P, Paterniti DA,et al. Racial and ethnic disparities in theuse of health services: bias, preferences,or poor communication? J Gen Intern Med.2003;18(2):146---152.

12. Saha S, Arbelaez JJ, Cooper LA.Patient---physician relationships and racialdisparities in the quality of health care. Am JPublic Health. 2003;93(10):1713---1719.

13. Siminoff LA, Graham GC, GordonNH. Cancer communication patterns andthe influence of patient characteristics:disparities in information-giving and af-fective behaviors. Patient Educ Couns.2006;62(3):355---360.

14. Arora NK. Interacting with cancerpatients: the significance of physicians’communication behavior. Soc Sci Med.2003;57(5):791---806.

15. Charles C, Gafni A, Whelan T.Shared decision-making in the medicalencounter: what does it mean? (or it takesat least two to tango). Soc Sci Med. 1997;44(5):681---692.

16. Joosten E, deWeert G, Sensky T, vander Staak C, de Jong C. Effect of shareddecision-making on therapeutic alliance inaddiction health care. Patient Prefer Ad-herence. 2008;2:277---285.

17. Gattellari M, Butow PN, TattersallMH. Sharing decisions in cancer care. SocSci Med. 2001;52(12):1865---1878.

18. Polacek GN, Ramos MC, Ferrer RL.Breast cancer disparities and decision-making among US women. Patient EducCouns. 2007;65(2):158---165.

19. Cass A, Lowell A, Christie M, et al.Sharing the true stories: improving com-munication between Aboriginal patientsand healthcare workers. Med J Aust.2002;176(10):466---470.

20. Canales MK. Connecting to native-ness: the influence of women’s AmericanIndian identity on their health-care deci-sions. Can J Nurs Res. 2004;36(4):18---44.

21. Kaufert JM, Putsch RW, Lavallee M.End-of-life decision making among Ab-original Canadians: interpretation, medi-ation, and discord in the communicationof “bad news.” J Palliat Care. 1999;15(1):31---38.

22. Institute of Medicine. Finding WhatWorks in Health Care: Standards forSystematic Reviews. Washington, DC: Na-tional Academies Press; 2011.

23. Liberati A, Altman DG, Tetzlaff J,et al. The PRISMA statement for reportingsystematic reviews and meta-analyses ofstudies that evaluate health care inter-ventions: explanation and elaboration.J Clin Epidemiol. 2009;62(10):e1---e34.

24. Dixon-Woods M, Agarwal S, JonesD, Young B, Sutton A. Synthesising qual-itative and quantitative evidence: a re-view of possible methods. J Health ServRes Policy. 2005;10(1):45---53.

25. Sandelowski M, Barroso J, Voils CI.Using qualitative metasummary to syn-thesize qualitative and quantitativedescriptive findings. Res Nurs Health.2007;30(1):99---111.

26. Back MF, Huak CY. Family centereddecision making and non-disclosure ofdiagnosis in a South East Asian oncology

practice. Psychooncology. 2005;14(12):1052---1059.

27. Hawley ST, Griggs JJ, Hamilton AS,et al. Decision involvement and receipt ofmastectomy ethnic minorly and ethnicallydiverse breast cancer patients. J NatlCancer Inst. 2009;101(19):1337---1347.

28. Hawley ST, Janz NK, Hamilton A,et al. Latina patient perspectives aboutinformed treatment decision making forbreast cancer. Patient Educ Couns. 2008;73(2):363---370.

29. Kaplan CP, Nápoles AM, Hwang ES,et al. Selection of treatment among Latinaand non-Latina white women with ductalcarcinoma in situ. J Womens Health(Larchmt). 2011;20(2):215---223.

30. Katz SJ, Lantz PM, Janz NK, et al.Patient involvement in surgery treatmentdecisions for breast cancer. J Clin Oncol.2005;23(24):5526---5533.

31. Lantz PM, Janz NK, Fagerlin A, et al.Satisfaction with surgery outcomes andthe decision process in a population-basedsample of women with breast cancer.Health Serv Res. 2005;40(3):745---767.

32. Maly RC, Leake B, Silliman RA.Breast cancer treatment in older women:impact of the patient---physician interaction.J Am Geriatr Soc. 2004;52(7):1138---1145.

33. Maly RC, Umezawa Y, Ratliff CT,Leake B. Racial/ethnic group differencein treatment decision-making and treat-ment received among older breast carci-noma patients. Cancer. 2006;106(4):957---965.

34. Phipps E, Madison N, Polansky M,Tester W. The importance of patientparticipation in second-line chemother-apy decisions: perspectives of African-American patients. J Natl Med Assoc.2008;100(12):1434---1440.

35. Steenland K, Goodman M, Liff J, et al.The effect of race and rural residence onprostate cancer treatment choice amongmen in Georgia. Urology. 2011;77(3):581---587.

36. Tam Ashing K, Padilla G, Tejero J,Kagawa-Singer M. Understanding thebreast cancer experience of Asian Amer-ican women. Psychooncology. 2003;12(1):38---58.

37. Balneaves LG, Bottorff JL, GrewalSK, Naidu P, Johnson JL, Howard AF.Family support of immigrant Punjabiwomen with breast cancer. Fam Commu-nity Health. 2007;30(1):16---28.

38. Coleman-Brueckheimer K, Spitzer J,Koffman J. Involvement of rabbinic andcommunal authorities in decision-makingby Haredi Jews in the UK with breastcancer: an interpretative phenomenologi-cal analysis. Soc Sci Med. 2009;68(2):323---333.

SYSTEMATIC REVIEW


39. Howard AF, Bottorff JL, BalneavesLG, Grewal SK. Punjabi immigrantwomen’s breast cancer stories. J ImmigrMinor Health. 2007;9(4):269---279.

40. Jones RA, Taylor AG, BourguignonC, et al. Complementary and alternativemedicine modality use and beliefs amongAfrican American prostate cancer survi-vors. Oncol Nurs Forum. 2007;34(2):359---364.

41. Killoran M, Moyer A. Surgical treat-ment preferences in Chinese-Americanwomen with early-stage breast cancer.Psychooncology. 2006;15(11):969---984.

42. Kreling B, Figueiredo MI, SheppardVL, Mandelblatt JS. A qualitative study offactors affecting chemotherapy use inolder women with breast cancer: barriers,promoters, and implications for inter-vention. Psychooncology. 2006;15(12):1065---1076.

43. Nápoles-Springer AM, Livaudais JC,Bloom J, Hwang S, Kaplan CP. Informa-tion exchange and decision making in thetreatment of Latina and white womenwith ductal carcinoma in situ. J PsychosocOncol. 2007;25(4):19---36.

44. Sheppard VB, Figueiredo M, Cañar J,et al. Latina a Latina: developing a breastcancer decision support intervention.Psychooncology. 2008;17(4):383---391.

45. Sheppard VB,Williams KP, HarrisonTM, et al. Development of decision-support intervention for black womenwith breast cancer. Psychooncology.2010;19(1):62---70.

46. Williams SW, Hanson LC, Boyd C,et al. Communication, decision making,and cancer: what African Americans wantphysicians to know. J Palliat Med. 2008;11(9):1221---1226.

47. Zhang AY, Siminoff LA. The role ofthe family in treatment decision makingby patients with cancer. Oncol NursForum. 2003;30(6):1022---1028.

48. Katz SJ, Lantz PM, Paredes Y, et al.Breast cancer treatment experiences ofLatinas in Los Angeles County. Am JPublic Health. 2005;95(12):2225---2230.

49. Schim SM, Doorenbos A, Benkert R,Miller J. Culturally congruent care: puttingthe puzzle together. J Transcult Nurs.2007;18(2):103---110.

50. Waljee AK, Higgins PD, Waljee JF,et al. Perceived and actual quality of lifewith ulcerative colitis: a comparison ofmedically and surgically treated patients.Am J Gastroenterol. 2011;106(4):794---799.

51. Bogart LM, Kelly JA, Catz SL,Sosman JM. Impact of medical and non-medical factors on physician decisionmaking for HIV/AIDS antiretroviraltreatment. J Acquir Immune Defic Syndr.2000;23(5):396---404.

52. Pini TM, Hawley ST, Li Y, Katz SJ,Griggs JJ. The influence of non-clinicalpatient factors on medical oncologists’decisions to recommend breast canceradjuvant chemotherapy. Breast CancerRes Treat. 2012;134(2):867---874.

53. Kroenke CH, Quesenberry C, KwanML, Sweeney C, Castillo A, Caan BJ. Socialnetworks, social support, and burden inrelationships, and mortality after breastcancer diagnosis in the Life After BreastCancer Epidemiology (LACE) study.Breast Cancer Res Treat. 2013;137(1):261---271.

54. Haskard-Zolnierek KB. Communi-cation about patient pain in primary care:development of the Physician---PatientCommunication about Pain scale (PCAP).Patient Educ Couns. 2012;86(1):33---40.

55. Kelly L, Brown JB. Listening tonative patients. Changes in physicians’understanding and behaviour. Can FamPhysician. 2002;48:1645---1652.

56. Shahid S, Finn LD, Thompson SC.Barriers to participation of Aboriginalpeople in cancer care: communication inthe hospital setting.Med J Aust. 2009;190(10):574---579.

57. Towle A, Godolphin W, AlexanderT. Doctor---patient communications inthe Aboriginal community: towards thedevelopment of educational programs.Patient Educ Couns. 2006;62(3):340---346.

58. Barton SS, Anderson N, ThommasenHV. The diabetes experiences of Aborig-inal people living in a rural Canadiancommunity. Aust J Rural Health. 2005;13(4):242---246.

59. Parker PA, Aaron J, Baile WF. Breastcancer: unique communication challengesand strategies to address them. Breast J.2009;15(1):69---75.

60. Espey DK, Wu XC, Swan J, et al.Annual report to the nation on the statusof cancer, 1975---2004, featuring cancerin American Indians and Alaska Natives.Cancer. 2007;110(10):2119---2152.

61. Smith CB, Bonomi M, Packer S,Wisnivesky JP. Disparities in lung cancerstage, treatment and survival amongAmerican Indians and Alaskan Natives.Lung Cancer. 2011;72(2):160---164.

62. Petereit DG, Rogers D, Govern F,et al. Increasing access to clinical cancertrials and emerging technologies for mi-nority populations: the Native AmericanProject. J Clin Oncol. 2004;22(22):4452---4455.

63. Schim SM, Doorenbos AZ. A three-dimensional model of cultural congru-ence: framework for intervention. J SocWork End Life Palliat Care. 2010;6(3---4):256---270.

64. Stacey D, Kryworuchko J, Bennett C,Murray MA, Mullan S, Legare F. Decisioncoaching to prepare patients for making

health decisions: a systematic review ofdecision coaching in trials of patient de-cision AIDS. Med Decis Making. 2012;32(3):E22---E33.

65. Haozous EA, Knobf MT. “All mytears were gone”: suffering and cancerpain in southwest American Indians. JPain Symptom Manage. 2013;45(6):1050---1060.

66. Jemal A, Siegel R, Xu J, Ward E.Cancer statistics, 2010. CA Cancer J Clin.2010;60(5):277---300.

SYSTEMATIC REVIEW


Reproduced with permission of the copyright owner. Further reproduction prohibited withoutpermission.

MethodologicalReporting inQualitative,Quantitative, andMixedMethodsHealthServicesResearchArticlesJennifer P. Wisdom, Mary A. Cavaleri, Anthony J. Onwuegbuzie,and Carla A. Green

Objectives. Methodologically sound mixed methods research can improve ourunderstanding of health services by providing a more comprehensive picture of healthservices than either method can alone. This study describes the frequency of mixedmethods in published health services research and compares the presence of methodo-logical components indicative of rigorous approaches across mixed methods, qualita-tive, and quantitative articles.Data Sources. All empirical articles (n = 1,651) published between 2003 and 2007from four top-ranked health services journals.StudyDesign. All mixedmethods articles (n = 47) and random samples of qualitativeand quantitative articles were evaluated to identify reporting of key components indi-cating rigor for each method, based on accepted standards for evaluating the quality ofresearch reports (e.g., use of p-values in quantitative reports, description of context inqualitative reports, and integration in mixed method reports). We used chi-square teststo evaluate differences between article types for each component.Principal Findings. Mixed methods articles comprised 2.85 percent (n = 47) ofempirical articles, quantitative articles 90.98 percent (n = 1,502), and qualitative arti-cles 6.18 percent (n = 102). There was a statistically significant difference (v2(1)= 12.20, p = .0005, Cramer’s V = 0.09, odds ratio = 1.49 [95% confidence inter-val = 1,27, 1.74]) in the proportion of quantitative methodological components pres-ent in mixed methods compared to quantitative papers (21.94 versus 47.07 percent,respectively) but no statistically significant difference (v2(1) = 0.02, p = .89, Cramer’sV = 0.01) in the proportion of qualitative methodological components in mixed meth-ods compared to qualitative papers (21.34 versus 25.47 percent, respectively).Conclusion. Few published health services research articles use mixed methods. Thefrequency of key methodological components is variable. Suggestions are provided toincrease the transparency of mixed methods studies and the presence of key methodo-logical components in published reports.Key Words. Health services, research methodology, mixed methods, qualitativemethods

©Health Research and Educational TrustDOI: 10.1111/j.1475-6773.2011.01344.xMETHODSARTICLE

721

Health Services Research

As the health services research field continues to evolve, so too does itsmethods. Mixed methods research capitalizes on the strengths of both quali-tative and quantitative methodologies by combining approaches in a singleresearch study to increase the breadth and depth of understanding ( Johnson,Onwuegbuzie, and Turner 2007). Mixed methods can be a better approachto research than either quantitative-only or qualitative-only methods when asingle data source is not sufficient to understand the topic, when results needadditional explanation, exploratory findings need to be generalized, or whenthe complexity of research objectives are best addressed with multiplephases or types of data (Brannen 1992; Creswell and Plano Clark 2011).Rigorous mixed methods approaches require that individual components(qualitative or quantitative) adhere to their respective established standards(Curry, Nembhard, and Bradley 2009; Creswell and Plano Clark 2011).Despite recent guidelines on frameworks for conducting mixed methodsresearch (e.g., Curry, Nembhard, and Bradley 2009; Creswell and PlanoClark 2011), a critical challenge has been ensuring that reports from mixedmethods studies transparently discuss the methodological components inte-gral to the conduct of the studies. Health services researchers and reviewersneed clear guidelines regarding research methodology, including methodo-logical components that should be expected in mixed methods papers toindicate that they are sufficiently rigorous.

Mixed Methods in Health Services Research

Health services research is the study of how social factors, financing systems,organizational structures and processes, health technologies, and personalbehaviors affect access to health care, the quality and cost of health care, andultimately, health and well-being (Lohr and Steinwachs 2002). As a resultof the breadth of topics addressed, health services research draws upon meth-ods and concepts from many fields, including medicine, epidemiological and

Address correspondence to Jennifer P. Wisdom, Ph.D., M.P.H., Psychiatry Department, Colum-bia University and NewYork State Psychiatric Institute, 1051 Riverside Drive Box 100, NewYork,NY 10032; e-mail: [email protected]. Mary A. Cavaleri, Ph.D., L.C.S.W., is with the Psy-chiatry Department at Columbia University and New York State Psychiatric Institute, New York,NY. Anthony J. Onwuegbuzie, Ph.D., is with the Department of Educational Leadership andCounseling at Sam Houston State University, Huntsville, TX. Carla A. Green, Ph.D., M.P.H., iswith the Kaiser Permanente Northwest Center for Health Research, Portland, OR. Portions of thismanuscript were presented at the International Mixed Methods Conference in July 2010 in Balti-more, MD.

722 HSR: Health Services Research 47:2 (April 2012)

economic studies, and the evaluation of services and interventions (Field,Tranquada, and Feasley 1995). Health services researchers increasingly workin interdisciplinary partnerships (e.g., Aboelela et al. 2007) and use innovativemethods, including mixed methods, to more fully understand health servicesphenomena. Mixed methods approaches are also consistent with suggestionsto extend scientific and contextual health knowledge beyond randomized tri-als (Berwick 2005).

Mixed methods research capitalizes on the strengths of both qualitativeand quantitative methodology by combining both components in a singleresearch study to increase breadth and depth of understanding ( Johnson, On-wuegbuzie, and Turner 2007). Qualitative and quantitative methods can beintegrated for different purposes to provide a more comprehensive picture ofhealth services than either method can alone. Mixed methods are appropriatein the following situations: (1) when researchers would like to converge differ-ent methods or use one method to corroborate the findings from anotherabout a single phenomenon (triangulation); (2) when researchers would like touse one method to elaborate, illustrate, enhance, or clarify the results fromanother method (complementarity); (3) when researchers would like to useresults from one method to inform another method, such as in creating a mea-sure (development); (4) when researchers would like to use one method to dis-cover paradoxes and contradictions in findings from another method that cansuggest reframing research questions (initiation); and (5) when researchersseek to expand the breadth and depth of the study by using different methodsfor different research components (expansion) (Greene, Caracelli, and Gra-ham 1989). Bryman (2006) modified and expanded this list to add that mixedmethods can also be useful in obtaining diversity of views, illustrating con-cepts, and developing instruments.

Quantitative and qualitative research can be distinguished by the philo-sophical assumptions brought to the study (e.g., deductive versus inductive),the types of research strategies (e.g., experiments versus case studies), andthe specific research methods used in the study (e.g., structured survey ver-sus observation) (Creswell 2008). Qualitative health services research, forexample, is a method in which the researcher collects textual materialderived from speech or observation and attempts to understand the phenom-enon of interest in terms of the meanings people bring to them (Denzin andLincoln 1994; Shortell 1999; Giacomini and Cook for the Evidence-BasedMedicine Working Group 2000; Malterud 2001; Bradley, Curry, and Devers2007). Certain characteristics are typical of qualitative research, including anaturalistic setting (as opposed to a laboratory), a focus on participants’

Mixed Methods in Health Services Research 723

perspectives and their meaning, the outcome as a process rather than a prod-uct, and data collected as words or images (Padgett 2008).

Guidelines for Conducting Mixed Methods Research

The National Institutes of Health noted the need for rigor in combining quali-tative and quantitative methods to study complex health issues in their recentpublication, Best Practices for Mixed Methods in Health Sciences (Creswell, Klassen,Plano Clark, and Smith for the Office of Behavioral and Social SciencesResearch 2011). There are several frameworks to guide the rigorous conductand evaluation of mixedmethods research (Collins, Onwuegbuzie, and Sutton2006; Curry, Nembhard, and Bradley 2009; Tashakkori and Teddlie 2010;Creswell and Plano Clark 2011). Collectively, these frameworks recommendthat the conduct of mixed method studies—and reports of mixed methodresearch, including peer-reviewed publication—demonstrates explicit ratio-nales for all decisions regarding study design, including the purpose of includ-ing both qualitative and quantitative methods. They specifically advise thateach component (qualitative or quantitative) should be conducted with a levelof rigor in accordance with established principles in its field, and that research-ers be transparent in methodological reporting. For example, sampling designshould be specified as identical, parallel, nested, or mixed (Onwuegbuzie andCollins 2007); the level of mixing methods (fully versus partially) should bedescribed, as should time orientation (sequential or concurrent components ofresearch) and emphasis (equal importance of methodological approaches orone more dominant) (Leech andOnwuegbuzie 2009).

Conducting and evaluating mixed methods research have unique meth-odological challenges, particularly related to rigor. Quantitative studies typi-cally rely on quality criteria such as internal validity, generalizability, andreliability (Campbell 1957; Campbell and Stanley 1963; Messick 1989, 1995;Onwuegbuzie and Daniel 2002, 2004; Onwuegbuzie 2003), whereas qualita-tive studies have roughly comparable quality criteria of credibility, transfer-ability, and dependability (Lincoln and Guba 1985; Guba and Lincoln 1989;Miles and Huberman 1994; Maxwell 2005; Pope and Mays 2006). For exam-ple, questions asked when evaluating a qualitative study might include the fol-lowing: “Were participants relevant to the research question and was theirselection well reasoned?” and “Was the data collection comprehensive enoughto support rich and robust descriptions of the observed events?” (Giacominiand Cook for the Evidence-Based Medicine Working Group 2000). In addi-tion to determining whether methodological approaches unique to qualitative


or quantitative research were employed, an evaluation of a mixed methodsstudy should assess aspects unique to mixed methods, such as how multiplecomponents are integrated and how consistency and discrepancy betweenfindings from each method are managed (Sale and Brazil 2004; O’Cathain,Murphy, and Nicholl 2007). Qualitative, quantitative, and mixed methodolo-gists agree that study procedures should be reported transparently, includingsufficient detail to allow the reader to make inferences about study quality(Lincoln and Guba 1985; Giacomini and Cook for the Evidence-based Medi-cine Working Group 2000; O’Cathain, Murphy, and Nicholl 2007; Arm-strong et al. 2008; Creswell 2008; Curry, Nembhard, and Bradley 2009;Leech et al. 2009; Teddlie and Tashakkori 2009).

Several researchers have proposed specific techniques to assess the over-all methodology of mixed methods research and assess the methodologicalcomponents of the qualitative, quantitative, and mixed portions of the studies(e.g., Pluye et al. 2009; O’Cathain 2010; Tashakkori and Teddlie 2010; Cre-swell and Plano Clark 2011; Leech, Onwuegbuzie, and Combs 2011). Forexample, O’Cathain (2010) assessed quality of mixed methods research byevaluating transparency and clarity in reporting planning, design, data, inter-pretive rigor, inference transferability, reporting quality, synthesizability, andutility. Others have suggested alternativemethods for assessing quality, but cri-teria often are not elucidated or are vague. Further, those frameworks typicallyaddress quality of the study design as opposed to the characteristics provided inthe published article. By contrast, Sale and Brazil (2004) proposed a structuredframework for the evaluation of mixed methods publications by identifying keymethodological components that should be included for both qualitative andquantitative portions of studies. Despite these advances, we found few pub-lished accounts of the rigor of published mixed methods research. Our articlehas three specific research questions: (1) How has the frequency of mixedmethods studies published in health services journals changed over time? (2)How are mixed methods articles being used to elucidate health services? and(3) To what extent do mixed methods reports differ in methodological contentcompared to qualitative-only or quantitative-only articles?

METHOD

This systematic review assessed the frequency of mixed methods publicationsin top health services research journals and compared the frequency of keymethodological components in qualitative, quantitative, and mixed method


studies. We first reviewed articles in health services research journals to deter-mine the prevalence of mixed methods designs and the presence of key meth-odological components. Then, we conducted statistical analyses of trends overtime in the frequency of mixed methods articles and in the presence of keymethodological components of those articles. Because this was an analysis ofpublished data, no ethical oversight was required.

Identification of Mixed Methods Articles

We examined four journals:Health Affairs, Health Services Research,Medical Care,and Milbank Quarterly, which had 5-year impact factors of 2.94–4.71. Journalswere selected by reviewing the Institute for Scientific Information (2007) rank-ings for the top 10 journals in health care sciences and services. Of these 10,we included all journals that focused generally on health services research andexcluded journals with narrower foci (Value in Health, Journal of Health Econom-ics, Journal of Pain and Symptom Management, Statistical Methods in MedicalResearch,Quality and Safety in Health Care, andQuality of Life Research). Although2001 marked a turning point in the proliferation of mixed methods studiespublished in major electronic bibliographic databases such as PubMED (Col-lins, Onwuegbuzie, and Jiao 2007), we chose to examine articles from 2003 to2007 because 2003 marks publication of the first edition of Tashakkori andTeddlie’s landmark Handbook of Mixed Methods in Social and Behavioral Research,which provided the first comprehensive collection of mixed method theory,methodology, and application. Five years represents a sufficient period of timeto examine trends of published articles following the publication of a land-mark methodological work.

We reviewed empirical articles to determine whether each represented aquantitative, qualitative, or mixed methods study. This entailed using all theinformation presented in the abstract and the body of the article to identify theresearch design either as stated or implied by the author(s). We excluded non-empirical articles (book reviews, literature reviews, commentaries and opin-ion articles, letters to the editor, policy statements) and articles from a specialissue of Milbank Quarterly (Volume 83, Number 4) that included only articlespublished between 1932 and 1998.

We classified articles as quantitative if they included (1) a primary goalof testing theories or hypotheses about relationships between/among vari-ables, or (2) quantitative data and methodology, such as hierarchical linearmodeling, multiple regression, or Markov modeling. We classified articles asqualitative if they included either (1) a primary goal of exploring or under-


standing the meaning ascribed to a specific phenomenon or experience, or (2)qualitative data such as observations, unstructured or semi-structured inter-views, or focus group interviews or methodologies such as thematic analysis.Although more complex definitions of mixed method studies exist (e.g., John-son, Onwuegbuzie, and Turner 2007; Creswell and Plano Clark 2011), weclassified articles as mixed methods if they integrated or combined both quan-titative and qualitative methods in a single study (Sale and Brazil 2004). Thisdefinition reflects the general definitions of mixed methods and the lack ofconsensus on a specific definition across all multidisciplinary mixed methodsresearchers.

We used spreadsheets to track classifications, with cells containing arti-cles’ abstracts and our field notes. Two authors read and classified articles inbatches of 50 according to type, conferring as needed until agreement wasachieved (n = 300 articles); the remaining articles (n = 1,351) were eachcoded by one author. For the few articles for which methodology was ambigu-ous (n = 58, 3.5 percent of all empirical articles), classification was resolved inconsultation with a third author. Similar methods have been used in otherevaluations of mixed methods articles (Powell et al. 2008).

Assessments of Articles

We identified all mixed methods articles (n = 47) and equal random samples(n = 47) of quantitative articles (from 1,502 articles) and qualitative articles(from 102 articles) (total n = 141) in the four journals. Random samples ofqualitative and quantitative articles were selected using a random numbergenerator and did not adjust for journal or year. We assessed the frequency ofkey methodological components reported across articles, then compared ratesby article type. The methodological components we focused on were drawnfrom two conceptual frameworks. The first included Sale and Brazil’s (2004)criteria: (1) internal validity for quantitative findings and credibility for quali-tative findings, (2) external validity for quantitative findings and transferabilityor fittingness for qualitative findings, (3) reliability for quantitative findingsand dependability for qualitative findings, and (4) objectivity for quantitativefindings and confirmability for qualitative findings (specific criteria are listedin Table 3). The second was O’Cathain’s transparency criteria for mixedmethods studies (O’Cathain, Murphy, and Nicholl 2007; O’Cathain 2010),which specify that mixed methods studies should state the (1) priority of meth-ods (primarily quantitative, primarily qualitative, or equal priority), (2) pur-pose of mixing methods (e.g., triangulation, complementarity, initiation,


development, or expansion), (3) sequence of methods (qualitative first, quanti-tative first, or simultaneous), and (4) stage of integration of both types of data(e.g., data collection, analysis, interpretation). We assessed four additionalcomponents of mixed methods studies: (1) whether qualitative and quantita-tive components were integrated, (2) whether limitations of design weredetailed, (3) whether areas of consistency between qualitative and quantitativecomponents were elucidated, and (4) whether areas of inconsistency betweencomponents were described.

We assessed components using categories of 0 (not described), 1(described), or not applicable (e.g., for criteria referencing control groups in astudy that had none, or ethical review for a study with no human subjects data)(O’Cathain, Murphy, and Nicholl 2007). We identified only whether the studycontained or did not contain each methodological component and did notattempt to assess quality or appropriateness of each component within thecontext of the study. For example, we assessed whether the publication statedthat missing data were addressed but not whether themethods to address miss-ing data were the best methods for that particular research design. Similar toinitial article classification, two authors read and coded articles to assess pres-ence/absence of each criterion, with any ambiguity resolved in consultationwith a third author.

Quantitative Analyses of Trends and Rigor

Once all articles were coded, we conducted a statistical analysis to determinewhether there were trends over time in the prevalence of mixed methods arti-cles. To assess this, we used linear regression to test the hypothesis that therewould be an increase in the prevalence of the number of mixed methods arti-cles over time. We also conducted chi-square tests to assess differencesbetween mixed methods, qualitative, and quantitative articles on both quanti-tative and qualitative criteria. We tested whether each criterion was present inthe same proportion of quantitative studies as in the quantitative portion ofthe mixed methods studies and in the same proportion of qualitative studies asin the qualitative portion of the mixed methods studies.

RESULTS

In general, coders could easily categorize the type of study. Challenges arosewhen transparency about methods was inadequate (N = 58, 3.5 percent of all


empirical articles). For example, some papers indicated that data from inter-views were included but did not provide details about who was interviewed,what was asked in the interviews, how the interview data were analyzed, orhow the interview data were integrated into the overall study.

Research Question 1: How has the frequency of mixed methods studies publishedin health services journals changed over time?

Table 1 presents a summary of the types of articles published in fourmajor health services research journals from 2003 through 2007. Only 2.85percent (n = 47) of empirical articles were mixed methods studies; 6.18 per-cent (n = 102) of empirical studies represented qualitative research. Quantita-tive research represented 90.98 percent (n = 1,502) of empirical articles. Thejournal containing the highest proportion of empirical studies employing amixed methods design was Milbank Quarterly (8.33 percent), followed byHealth Affairs (6.91 percent), Health Services Research (4.03 percent), andMedicalCare (0.78 percent). Chi-square test showed a significant difference in theseproportions (v2 = 34.67, df = 3, p < .0001).

To detect temporal trends in the frequency of mixed methods researchin the health services literature, articles were collapsed across journal andexamined by publication year. Table 2 presents the frequency of article typefor each of the 5 years. All journals combined published an average of 10.8mixed method articles per year, or 3.27 percent of empirical articles annually.A quadratic trend was seen across the 5 years (R2 = 0.65), indicating a slightincrease in mixed method articles in the first 2 years and then a decrease forthe remaining years.

Table 1: Type and Design of Empirical Articles Published in Health ServicesResearch Journals from 2003 to 2007, Data Presented by Journal

Journal Quant Qual Mixed Total

Health Affairs 305 49 21 37581.33% 13.07% 5.60%

Health Services Research 428 26 17 47190.87% 5.52% 3.61%

Medical Care 751 12 6 76997.66% 1.56% 0.78%

Milbank Quarterly 18 15 3 3650.00% 41.67% 8.33%

Total 1,502 102 47 1,65190.98% 6.18% 2.85%

Note. Mixed, mixedmethod articles; Qual, qualitative articles; Quant, quantitative articles.


Research Question 2: How are mixed methods articles being used to elucidatehealth services research?

Mixed methods articles were categorized into four overlapping catego-ries: Articles on organizational and individual decision making processes(n = 18 studies) combined qualitative interviews with quantitative administra-tive data analyses to assess decision making about processes or impedimentsto processes. Examples include a study of formulary adoption decisions(Dandrove, Hughes, and Shanley 2003) and states’ decisions to reduce Medic-aid and other public program funding (Hoadley, Cunningham, and McHugh2004).

Sixteen articles described outcomes or effects of policies or initiatives bycombining administrative health record or performance data with interviewsof health administrators, providers, or executives. Examples include papersdescribing outcomes of pay-for-performance changes to Medicaid (Felt-Lisk,Gimm, and Peterson 2007; Rosenthal et al. 2007) and hospital patient safetyinitiatives (Devers, Pham, and Liu 2004).

Thirteen measurement development articles employed mixed methodsto create measurement tools to assess, for example, caregiver burden (Cousi-neau et al. 2003), patient activation (Hibbard et al. 2004), and the develop-ment of a Healthcare Effectiveness Data and Information Set (HEDIS)smoking measure (Pbert et al. 2003). These studies typically examined quali-tative data from individual or focus group interviews first to inform creationand testing of a survey.

Table 2: Type and Design of Empirical Articles Published in Four HealthServices Research Journals from 2003 to 2007, Data Presented by Year

Year Quant Qual Mixed Total

2003 260 21 7 28890.28% 7.29% 2.43%

2004 295 18 13 32690.49% 5.52% 3.99%

2005 282 17 8 30791.86% 5.54% 2.61%

2006 321 25 10 35690.17% 7.02% 2.81%

2007 344 21 9 37491.98% 5.61% 2.41%

Total 1,502 102 47 1,65190.98% 6.18% 2.85%

Note. Mixed, mixedmethod articles; Qual, qualitative articles; Quant, quantitative articles.


Articles on experiences and perceptions were the least common cate-gory (n = 8), typically combining surveys and interviews. These includedfamily physicians’ perceptions of the effect of medication samples on their pre-scribing practices (Hall, Tett, and Nissen 2006); caregivers’ experiences of thetermination of home health care for stroke patients (Levine et al. 2006); andconsumer enrollment experiences in the Cash and Counseling program(Schore, Foster, and Phillips 2007).

Only five mixed methods articles (10.64 percent) of the total mixedmethods sample used the terms “mixed method” or “multimethod” in theabstract or text, although four articles (8.51 percent) referred to “qualitativeand quantitative” data.

Research Question 3: Do mixed methods articles report qualitative and quantita-tive methodology differently than methodology is reported in qualitative-only or quanti-tative-only articles?

Table 3 presents a summary of the frequency of key methodologicalcomponents present in quantitative articles, qualitative articles, and mixedmethods articles (each n = 47). For quantitative methodological components(32 items), mixed methods articles (M = 7.02 [21.94 percent], SD = 6.24)averaged statistically significantly fewer (t(92) = �4.50, p < .00001, Cohen’s deffect size = 0.93) components than did quantitative articles (M = 15.06[47.07 percent], SD = 10.53). For qualitative methodological components (35items), mixed methods articles (M = 7.17 [21.34 percent], SD = 6.36) did notaverage a statistically significantly different proportion of components (t(92)= �1.10, p = .14, d = 0.23) than did qualitative articles (M = 8.91 [25.47 per-cent], SD = 8.83). No article met all criteria, and no criterion was met by allarticles. For comparative analyses at a statistical significance level of a = 0.05,power to detect a medium difference (Cohen’s h = 0.50) and a large difference(Cohen’s h = 0.80) was 78 and 99 percent, respectively.

Of quantitative components, mixed methods studies were most likely todescribe sources of data and data collection instruments (61.70 percent of stud-ies), state the purpose/objective of the paper (59.57 percent), state the source ofsubjects (58.70 percent), and define/describe the study population (51.06 per-cent). Most mixed methods studies did not include control and interventiongroups, which excluded related criteria. Quantitative studies tended to containmore key methodological components, with more than 90 percent of studiesdefining outcomemeasures (93.48 percent), defining/describing study popula-tion (91.49 percent), describing statistical procedures (95.74 percent), and stat-ing hypotheses (97.87 percent). Quantitative studies were more likely than thequantitative portion of mixedmethods studies to describe study characteristics


Table3:

Key

Metho

dologicalCom

pone

ntsin

Mixed

Metho

ds,Qua

ntita

tive,

and

Qua

litative

Health

Services

ResearchArticles

Mixed

MethodStudies(n=

47)

Quantita

tiveS

tudies(n

=47

)QualitativeS

tudies(n

=47

)

Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†

Keyquantitativem

ethodologicalcom

ponents

Truthvalue(in

ternalvalid

ity)

Ethicalreview

unde

rtaken

937

119.57

937

119.57

Inform

edconsen

tstated

521

2119.23

538

411.63

Iden

tifying

orcontrolling

fore

xtrane

ous/

confou

ndingvariab

les***

740

014.89

3314

070

.21

Con

fide

ntialityprotected

342

26.67

242

34.55

Com

parabilityof

controltointerven

tion

grou

psatba

selin

e0

047

0.00

836

318.18

Con

trol/com

parisongrou

pstreatedsimila

rly

00

470.00

340

46.98

App

licab

ility

(externa

lvalidity

/gen

eralizab

ility)

Outcomemeasuresd

efine

d7

040

100.00

433

193

.48

Con

trol/com

parisongrou

pde

scribe

d2

045

100.00

1133

325

.00

Datacolle

ctioninstrumen

ts/sou

rceof

data

describe

d***

2918

061.70

461

097.87

Statem

ento

fpurpo

se/objectiv

e**

2819

059

.57

407

085

.11

Source

ofsubjectsstated

(sam

plingfram

e)**

2719

158

.70

416

087.23

Stud

ypo

pulatio

nde

fine

dor

describe

d***

2423

051.06

434

091.49

Source

ofcontrol/compa

risongrou

pstated

11

4550

.00

836

318.18

Selectionof

control/compa

risongrou

pde

scribe

d1

145

50.00

836

318.18

Datagatheringproced

ures

describe

d*23

240

48.94

3314

070

.21

continued


Table3.

Contin

ued

Mixed

MethodStudies(n=

47)

Quantita

tiveS

tudies(n

=47

)QualitativeS

tudies(n

=47

)

Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†

Descriptio

nof

setting

/con

ditio

nsun

derw

hich

datacolle

cted

*22

241

47.83

3215

068

.09

Statistic

alproced

ures

referenced

orde

scribe

d***

1928

040

.43

452

095

.74

Subjectrecruitm

ento

rsam

plingselection

describe

d***

1730

036

.17

3512

074.47

Statem

entabo

utno

nrespo

nden

ts,d

ropo

uts,

deaths

1631

034

.04

2125

145

.65

p-Value

sstated***

1631

034

.04

416

087.23

Bothstatistic

alan

dclinicalsign

ificance

ackn

owledg

ed***

1334

027.66

416

087.23

Stud

yde

sign

stated

explicitly**

1136

023

.40

2621

055

.32

Inclusion/ex

clusioncrite

riastated

explicitly***

1036

121.74

2819

059

.57

Missing

dataad

dressed

1037

021.28

1829

038

.30

Atleaston

ehy

pothesisstated

*10

370

21.28

2324

048

.94

Samplerand

omlyselected

639

213.33

1235

025

.53

Con

fide

nceintervalsg

iven

form

ainresults***

542

010.64

2621

055

.32

Power

calculationprov

ided

146

02.13

740

014.89

Descriptio

nof

interven

tion

02

450.00

736

416

.28

Assessm

ento

foutcomeblinde

d0

047

0.00

241

44.65

Con

sisten

cy(reliability)

Stan

dardizationof

observersd

escribed

344

06.38

740

014.89

Neu

trality

(objectiv

ity)

continued


Table3.

Contin

ued

Mixed

MethodStudies(n=

47)

Quantita

tiveS

tudies(n

=47

)QualitativeS

tudies(n

=47

)

Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†

Statem

ento

fresearche

r’sa

ssum

ptions/

perspe

ctive

542

010.64

443

08.51

Keyqualita

tivem

ethodologicalcom

ponents

Truthvalue(credibility)

Triang

ulationof

qualita

tivesources

2522

053

.19

2720

057.45

Triang

ulationof

qualita

tivemetho

ds16

310

34.04

1334

027.66

Use

ofex

emplars

1334

027.66

1433

029

.79

Ethicalreview

unde

rtaken

1037

021.28

830

921.05

Triang

ulationof

investigators

740

014.89

344

06.38

Inform

edconsen

tstated

641

012.77

335

97.89

Mem

berc

hecks

443

08.51

245

04.26

Con

fide

ntialityprotected

443

08.51

335

97.89

Con

sent

proced

ures

describe

d3

440

6.38

236

95.26

Peer

debriefing

245

04.26

047

00.00

Negativecase

analysis(searching

for

discon

firm

ingeviden

ce)

146

02.13

047

00.00

Triang

ulationof

theo

ry/perspectiv

e0

470

0.00

443

08.51

App

licab

ility

(tran

sferab

ility/fitting

ness)

Statem

ento

fpurpo

se/objectiv

e34

130

72.34

3611

076

.60

Datagatheringproced

ures

describe

d*25

220

53.19

3611

076

.60

Descriptio

nof

stud

ycontex

torsettin

g***

2027

042

.55

389

080

.43

Phen

omen

onof

stud

ystated

1829

038

.30

2423

051.06

Samplingproced

urede

scribe

d18

290

38.30

2223

248

.89

Rationa

leforq

ualitativemetho

ds17

300

36.17

1235

025

.53

continued


Table3.

Contin

ued

Mixed

MethodStudies(n=

47)

Quantita

tiveS

tudies(n

=47

)QualitativeS

tudies(n

=47

)

Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†

Descriptio

nof

participan

ts/informan

ts*

1631

034

.04

2520

255

.56

Statem

ento

fresearchqu

estio

ns15

320

31.91

2126

044

.68

Statem

ento

fhow

setting

was

selected

1532

031.91

3017

063

.04

Dataan

alysisde

scribe

d15

320

31.91

2027

042

.55

Tran

scriptionproced

ures

describe

d11

360

23.40

1328

631.71

Cod

ingtechniqu

esde

scribe

d9

380

19.15

1730

036

.17

Justificatio

nor

ratio

naleforsam

pling

strategy

*8

390

17.02

1827

240

.00

Aud

iotaping

proced

ures

describe

d8

390

17.02

1229

629

.27

Statem

entabo

utno

nrespo

nden

ts,dropo

uts,

deaths

641

012

.77

434

910.53

Descriptio

nof

rawda

ta3

440

6.38

443

08.51

Rationa

lefortraditio

nwith

inqu

alita

tive

metho

ds2

450

4.26

245

04.26

Datacolle

ctionto

saturatio

nspecfiied

245

04.26

144

22.22

Statem

enttha

trefl

exivejourna

ls,log

book

s,no

tesw

erekept

245

04.26

344

06.38

Con

sisten

cy(dep

enda

bility)

Externa

laud

itof

process

047

00.00

047

00.00

Neu

trality

(com

firm

ability)

Externa

laud

itof

data

245

04.26

047

00.00

Bracketingor

epoche

047

00.00

047

00.00

Statem

ento

fresearche

r’sa

ssum

ptions

orpe

rspe

ctive

047

00.00

245

04.26 continued


Table3.

Contin

ued

Mixed

MethodStudies(n=

47)

Quantita

tiveS

tudies(n

=47

)QualitativeS

tudies(n

=47

)

Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†Yes

No

N/A

%with

Com

ponent

†

Keymixed

methodsmethodologicalcom

ponents

Integrationof

qualita

tivean

dqu

antitative

compo

nents

407

—85

.11

Sequ

ence

ofmetho

dsspecified

1037

—27.03

Areas

ofconsistencybe

tweenmetho

dsstated

1235

—25

.53

Areas

ofinconsistencybe

tweenmetho

dsstated

641

—12.77

Stageof

integrationspecified

542

—11.90

Priority

ofmetho

dsspecified

245

—4.44

Purposeof

mixingmetho

dsspecified

245

—4.44

Lim

itatio

nsof

mixed

metho

dsstated

245

—4.26

Note.*p

<.05;

**p<.01;

***p

<.001.

†Percent

with

quality

indicatorisc

alculatedas

n(yes)/n

�n(n/a).


(e.g., study design, subject recruitment), identify or control for confoundingvariables, provide probability values or confidence intervals, state hypotheses,or acknowledgeboth statistical and clinical significance (seeTable 3).

For qualitative methodological components, mixed methods studieswere most likely to state the purpose/objective of the paper (72.34 percent),triangulate qualitative sources (e.g., use both individual and focus group inter-views; 53.19 percent), and describe data-gathering procedures (53.19 percent).More than 50 percent of qualitative studies triangulated qualitative sources(57.45 percent), stated the purpose/objective of the paper (57.45 percent), anddescribed the study setting (80.43 percent), how the setting was selected(63.04 percent), the participants (55.56 percent), and data-gathering proce-dures (76.60 percent). Qualitative studies were more likely than the qualitativeportions of the mixed methods studies to describe the study setting, justify thesampling strategy, participants, and data-gathering procedures.

For criteria regarding method integration, few authors justified the useof mixed methods or clearly described the priority, purpose, and sequence ofmethods, and the stage of integration.Most articles, however, integrated quali-tative and quantitative components (85.11 percent); examination of articlesindicated components were most frequently integrated in the interpretationphase. Across all studies, few articles stated that informed consent wasobtained, ethical review was undertaken, or that subjects’ confidentiality wasprotected.

DISCUSSION

Previous reports indicate mixed methods articles comprised <1 percent ofempirical health articles examined in 2000 (McKibbon andGadd 2004). Sincethen, however, the National Institutes of Health has increased funding formixed methods research, with the proportion of funded research projects upto 5 percent of studies in some institutes (Plano Clark 2010). In the UnitedKingdom, the proportion of funded research that uses mixed methods is at 17percent and continuing to increase (O’Cathain, Murphy, and Nicholl 2007).We found that the use of mixedmethods in articles published in top health ser-vices research journals was generally consistent between 2003 and 2007 atapproximately 3 percent of all empirical articles, lower than would beexpected given the complexity and depth of health services research questionsfor whichmixed methods would be appropriate. The presence of key method-ological components was variable across type of article, but the quantitative


portion of mixed methods articles included consistently fewer methodologicalcomponents than quantitative-only studies and the qualitative portion ofmixed methods articles included about the same proportion of methodologi-cal components as qualitative-only articles. Mixed methods articles also gen-erally did not address the priority, purpose, and sequence of methods or theintegration of methods as suggested by experts in mixed methods (e.g., Cre-swell and Tashakkori 2008; O’Cathain 2010; Creswell and Plano Clark 2011).

Key methodological components that cut across qualitative and quanti-tative methodologies were often missing from mixed methods publications.Descriptions of sample selection and sampling procedures, the study context,and data-gathering procedures are essential aspects of interpreting study find-ings, and mixed methods studies should not be exempt from these basicresearch requirements. Many mixed methods studies did not include the levelof detail that would likely be required for a qualitative or quantitative paper tobe accepted in these high-ranking journals. Further, the studies appeared notto follow available guidance on the structure and components of mixed meth-ods studies that discuss basic quality criteria, data collection strategies, meth-ods of data analysis, procedures for integration of methods, processes ofmaking inferences from text, and recommendations for adequate reporting ofresults (e.g., Giacomini and Cook for the Evidence-based Medicine WorkingGroup 2000; Curry, Nembhard, and Bradley 2009; O’Cathain 2010; Tashakk-ori and Teddlie 2010; Creswell and Plano Clark 2011). In some ways this find-ing is not surprising because guidance on mixed methods standards is stillemerging. We expect that the National Institutes of Health publication, BestPractices for Mixed Methods in Health Sciences (Creswell, Klassen, Plano Clark,and Smith for the Office of Behavioral and Social Science Research) will leadto increased standardization of mixed methods approaches.

Although they reported more key methodological components on aver-age than the mixed methods articles, quantitative articles in this analysis hadsome surprising gaps as well, including low reporting of power analyses, howmissing data were addressed, and descriptions of control/comparison groups.It should be noted, however, that quantitative articles with large sample sizesdo not necessarily need power analyses. With regard to single-method quali-tative articles, low proportions described the study context, coding tech-niques, or data analysis. Few articles with human subjects involvementincluded statements that the research was conducted with ethical oversight,promised confidentiality, or obtained consent. These findings suggest that theissue of poor transparency in reporting methodology is not limited to mixedmethods studies.


Recommendations for Mixed Methods Reporting

The methodological components reported here are not optimal indicators ofthe quality of mixed methods publications; an article could conceivably haveall of these components and yet still be a poor research study. These compo-nents are, however, a useful starting point for a systematic evaluation of therigor of qualitative and quantitative portions of mixed methods studies. Somejournals require inclusion of other criteria (e.g., Consolidated Standards ofReporting Trials 2010) to guide reporting of highly structured methodologies(e.g., randomized clinical trials); it would be useful to examine researchers’and editors’ perspectives on the validity of the methodological components inthis study for mixed method publications. It is difficult, however, to identifymeasurable criteria that capture the breadth of study designs in health ser-vices. Further, determination of what indicators of rigor would be appropriatemight reasonably vary by study design, topic, scope, and even journal, andqualified judgment is required to determine which criteria are appropriate foreach study. These findings suggest mixed methods researchers should provideenough detail on methodology and methodological decisions to allow review-ers to judge quality.

Researchers face challenges writing and publishing mixed methods arti-cles, including communicating with diverse audiences who are familiar withonly one methodological approach (i.e., quantitative research or qualitativeresearch), determining the most appropriate language and terminology to use,complying with journal word counts, and finding appropriate publishing out-lets with reviewers who have expertise in mixed methods research techniquesand who are not biased against mixed methods studies (Leech and On-wuegbuzie 2010; Leech, Onwuegbuzie, and Combs 2011). Our findings sug-gest that Sale and Brazil’s (2004) criteria and existing guidance on conductingmixed methods research (e.g., Collins, Onwuegbuzie, and Sutton 2006;Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011) might be usefulframeworks for health services researchers as they work to improve methodo-logical rigor. Journal editors might also encourage the publication of mixedmethods projects by (1) publishing guidelines for rigor in mixed methods arti-cles (e.g., Sale and Brazil 2004), (2) identifying experienced reviewers who canprovide competent and ethical reviews of mixed methods studies, and (3)requiring transparency of methods for all studies so that (4) rigor and qualitycan be can be assessed to the same extent they are in quantitative studies.These modifications might require (5) some flexibility in word count or allow-ance of online appendices to allow mixed methods researchers to describe


fully and concisely both qualitative and quantitative components, methods forintegrating findings, and appropriate details.

Limitations

In this study, assessment was limited to only published articles. We did notcontact authors to determine specific study activities, and studies may haveincluded methodological components (e.g., consenting) not reported in publi-cations. We assessed only whether publications reported the methodologicalcomponent, but we did not evaluate whether each component was fully andappropriately implemented in the research.

CONCLUSIONS

Mixed methods studies have utility in providing a more comprehensive pic-ture of health services than either method can alone. Researchers who usemixed methods techniques should use rigorous methodologies in their mixedmethods research designs and explicitly report key methodological compo-nents of those designs and methods in published articles. Similarly, journaleditors who publish mixed methods research should provide guidance toreviewers of mixed methods articles to assess the quality of manuscripts, andthey must be prepared to provide adequate space for authors to report the nec-essary methodological information. Frameworks are now available to guideboth the design and evaluation of mixed methods research studies and pub-lished works. Whatever frameworks are used, it is essential that authors whoengage in mixed methods research studies meet two primary goals (developedby the American Educational Research Association 2006): Mixed methodsresearchers should (1) conduct and report research that iswarranted or defensi-ble in terms of documenting evidence, substantiating results, and validatingconclusions; and (2) ensure that the conduct of research is transparent in termsof clarifying the logic underpinning the inquiry.

ACKNOWLEDGMENTS

Joint Acknowledgment/Disclosure Statement: The authors appreciate funding fromthe National Institute on Drug Abuse (K23 DA020487) and comments andfeedback on an earlier draft from the anonymous reviewers, John Creswell,


PhD, Alicia O’Cathain, PhD, Hilary Vidair, PhD, Susan Essock, PhD, and SaShen, PhD. Portions of this manuscript were presented at the InternationalMixedMethods Conference in July 2010 in Baltimore, Maryland.

Disclosures: None.Disclaimers: None.

REFERENCES

Aboelela, S. W., E. Larson, S. Bakken, O. Carrasquillo, A. Formicola, S. A. Glied, J.Haas, and K. M. Gebbie. 2007. “Defining Interdisciplinary Research: Conclu-sions from a Critical Review of the Literature.”Health Services Research 42 (1, part1): 329–46.

American Educational Research Association. 2006. “Standards for Reporting onEmpirical Social Science Research in AERA Publications.” Educational Researcher35 (6): 33–40.

Armstrong, R., E. Waters, L. Moore, E. Riggs, L. G. Cuervo, P. Lumbiganon, and P.Hawe. 2008. “Improving the Reporting of Public Health Intervention Research:Advancing TREND and CONSORT.” Journal of Public Health 30 (1): 103–9.

Berwick, D. 2005. “The John Eisenberg Lecture: Health Services Research as a Citizenin Improvement.”Health Services Research 40 (2): 317–36.

Bradley, E. H., L. A. Curry, and K. J. Devers. 2007. “Qualitative Data Analysis forHealth Services Research: Developing Taxonomy, Themes, and Theory.”HealthServices Research 42 (4): 1758–72.

Brannen, J. (Ed.). 1992.Mixing Methods: Qualitative and Quantitative Research. Aldershot,England: Ashgate.

Bryman, A. 2006. “Integrating Quantitative and Qualitative Research: How Is ItDone?”Qualitative Research 6: 97–113.

Campbell, D. T. 1957. “Factors Relevant to the Validity of Experiments in Social Set-tings.” Psychological Bulletin 54: 297–312.

Campbell, D. T., and J. C. Stanley. 1963. Experimental and Quasi-Experimental Designs forResearch. Chicago, IL: RandMcNally.

Collins, K. M. T., A. J. Onwuegbuzie, and Q. G. Jiao. 2007. “AMixedMethods Investi-gation of Mixed Methods Sampling Designs in Social and Health ScienceResearch.” Journal of Mixed Methods Research 1: 267–94.

Collins, K. M. T., A. J. Onwuegbuzie, and I. L. Sutton. 2006. “A Model Incorporatingthe Rationale and Purpose for Conducting Mixed Methods Research in SpecialEducation and Beyond.” Learning Disabilities: A Contemporary Journal 4: 67–100.

Consolidated Standards of Reporting Trials. 2010. Website [accessed on October 18,2011]. Available at http://www.consort-statement.org/

Cousineau, N., I. McDowell, S. Hotz, and P. Hébert. 2003. “Measuring ChronicPatients’ Feelings of Being a Burden to Their Caregivers.” Medical Care 41: 110–8.


Creswell, J. 2008. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches.ThousandOaks, CA: Sage.

Creswell, J. W., A. C. Klassen, V. L. Plano Clark, and K. C. Smith for the Office ofBehavioral and Social Sciences Research. 2011. Best Practices for Mixed MethodsResearch in the Health Sciences. National Institutes of Health [accessed on October18, 2011]. Available at: http://obssr.od.nih.gov/mixed_methods_research

Creswell, J., and V. Plano Clark. 2011. Designing and Conducting Mixed Methods Research.2nd Edition. ThousandOaks, CA: Sage.

Creswell, J. W., and A. Tashakkori. 2008. “Developing Publishable Mixed MethodsManuscripts.” Journal of Mixed Methods Research 1: 107–11.

Curry, L. A., I. M. Nembhard, and E. H. Bradley. 2009. “Qualitative andMixedMeth-ods Provide Unique Contributions to Outcomes Research.” Circulation 119:1442–52.

Dandrove, D., F. X. Hughes, and M. Shanley. 2003. “Determinants of HMO Formu-lary Adoption Decisions.”Health Services Research 38 (1): 169–90.

Denzin, N. K., and Y. S. Lincoln (Eds.). 1994. Handbook of Qualitative Research. Thou-sandOaks, CA: Sage.

Devers, K. J., H. H. Pham, andG. Liu. 2004. “What Is DrivingHospitals’ Patient-SafetyEfforts?”Health Affairs 23 (2): 103–15.

Felt-Lisk, S., G. Gimm, and S. Peterson. 2007. “Making Pay-For-Performance Work inMedicaid.”Health Affairs 26: w516–27.

Field, M. J., R. E. Tranquada, and J. C. Feasley (Eds.). 1995. Health Services Research:Workforce and Educational Issues. Washington, DC: National Academies Press.

Giacomini, M. K., and D. J. Cook; for the Evidence-based Medicine Working Group.2000. “Users’ Guides to the Medical Literature. XXIII. Qualitative Research inHealth Care A. Are the Results of the Study Valid?” Journal of the American Medi-cal Association 284: 357–62.

Greene, J. C., V. J. Caracelli, and W. F. Graham. 1989. “Toward a Conceptual Frame-work for Mixed-Method Evaluation.” Educational Evaluation and Policy 11: 255–74.

Guba, E. G., and Y. S. Lincoln. 1989. Fourth Generation Evaluation. Newbury Park, CA:Sage.

Hall, K. B., S. E. Tett, and L.M. Nissen. 2006. “Perceptions of the Influence of Prescrip-tion Medicine Samples on Prescribing by Family Physicians.” Medical Care 44(4): 383–7.

Hibbard, J. H., J. Stockard, E. R. Mahoney, and M. Tusler. 2004. “Development of thePatient Activation Measure (PAM): Conceptualizing and Measuring Activationin Patients and Consumers.”Health Services Research 39 (4, part 1): 1005–26.

Hoadley, J. F., P. Cunningham, and M. McHugh. 2004. “Popular Medicaid ProgramsDo Battle with State Budget Pressures: Perspectives from Twelve States.” HealthAffairs 23 (2): 143–54.

Institute for Scientific Information. 2007. “Web of Knowledge Journal Citation Reportfor Health Care Sciences and Services” [accessed on October 18, 2011].Available at http://admin-apps.isiknowledge.com/JCR/JCR?RQ=LIST_SUMMARY_JOURNAL


Johnson, R. B., A. J. Onwuegbuzie, and L. A. Turner. 2007. “Toward a Definition ofMixedMethods Research.” Journal of Mixed Methods Research 2: 112–33.

Leech, N. L., A. B. Dellinger, K. B. Brannigan, and H. Tanaka. 2009. “EvaluatingMixed Research Studies: A Mixed Methods Approach.” Journal of Mixed MethodResearch 4 (1): 17–31.

Leech, N. L., and A. J. Onwuegbuzie. 2009. “ATypology of Mixed Methods ResearchDesigns.”Quality & Quantity: International Journal of Methodology 43: 265–75.

———————. 2010. “Guidelines for Conducting and Reporting Mixed Research in the Fieldof Counseling and Beyond.” Journal of Counseling and Development 89 (1): 61–70.

Leech, N. L., A. J. Onwuegbuzie, and J. P. Combs. 2011. “Writing Publishable MixedResearch Articles: Guidelines for Emerging Scholars in the Health Sciences andBeyond.”Mixed Methods Research in the Health Sciences 5 (1): 7–24.

Levine, C., S. M. Albert, A. Hokenstad, D. E. Halper, A. Y. Hart, and D. A. Gould.2006. “‘This Case Is Closed’: Family Caregivers and the Termination of HomeHealth Care Services for Stroke Patients.”Milbank Quarterly 84 (2): 305–31.

Lincoln, Y. S., and E. G. Guba. 1985.Naturalistic Inquiry. Beverly Hills, CA: Sage.Lohr, K. N., andD.M. Steinwachs. 2002. “Health Services Research: An Evolving Def-

inition of the Field.”Health Services Research 37: 15–7.Malterud, K. 2001. “Qualitative Research: Standards, Challenges, and Guidelines.”

Lancet 358: 483–8.Maxwell, J. A. 2005. Qualitative Research Design: An Interactive Approach. 2nd Edition.

Newbury Park, CA: Sage.McKibbon, K. A., and C. S. Gadd. 2004. “AQuantitative Analysis of Qualitative Stud-

ies in Clinical Journals for the 2000 Publishing Year.” BioMed Central MedicalInformatics and Decision Making 4 (11) [accessed onOctober 18, 2011]. Available athttp://www.biomedcentral.com/1472-6947/4/11

Messick, S. 1989. “Validity.” In Educational Measurement, 3rd Edition, edited by R. L.Linn, pp 13–103. Old Tappan, NJ: Macmillan.

———————. 1995. “Validity of Psychological Assessment: Validation of Inferences from Per-sons’ Responses and Performances as Scientific Inquiry into Score Meaning.”American Psychologist 50: 741–9.

Miles, M., and A. M. Huberman. 1994. Qualitative Data Analysis: An Expanded Source-book. 2nd Edition. ThousandOaks, CA: Sage.

O’Cathain, A. 2010. “Assessing the Quality of Mixed Methods Research: Toward aComprehensive Framework.” In Sage Handbook of Mixed Methods in Social andBehavioral Research, 2nd Edition, edited by A. Tashakkori and C. Teddlie, pp. 531–57. ThousandOaks, CA: Sage.

O’Cathain, A., E. Murphy, and J. Nicholl. 2007. “Integration and Publications as Indi-cators of ‘Yield’ from Mixed Methods Studies.” Journal of Mixed Methods Research1 (2): 147–63.

Onwuegbuzie, A. J. 2003. “Expanding the Framework of Internal and External Valid-ity in Quantitative Research.” Research in the Schools 10 (1): 71–90.

Onwuegbuzie, A. J., and K. M. T. Collins. 2007. “ATypology of Mixed Methods Sam-pling Designs in Social Science Research.”Qualitative Report 12: 281–316.


Onwuegbuzie, A. J., and L. G. Daniel. 2002. “A Framework for Reporting and Inter-preting Internal Consistency Reliability Estimates.” Measurement and Evaluationin Counseling and Development 35: 89–103.

———————. 2004. “Reliability Generalization: The Importance of Considering SampleSpecificity, Confidence Intervals, and Subgroup Differences.” Research in theSchools 11 (1): 61–72.

Padgett, D. 2008. Qualitative Methods in Social Work Research. 2nd Edition. ThousandOaks, CA: Sage.

Pbert, L., N. Vukovic, J. K. Ockene, J. F. Hollis, and K. Riedlinger. 2003. “Developingand Testing New Smoking Measures for the Health Plan Employer Data andInformation Set.”Medical Care 41 (4): 550–9.

Plano Clark, V. L. 2010. “The Adoption and Practice of Mixed Methods: U.S. Trendsin Federally FundedHealth-Related Research.”Qualitative Inquiry 16: 428–40.

Pluye, P., M. Gagnon, F. Griffiths, and J. Johnson-Lafleur. 2009. “A Scoring System forAppraising Mixed Methods Research and Concomitantly Appraising Qualita-tive, Quantitative and Mixed Methods Primary Studies in Mixed StudiesReviews.” International Journal of Nursing Studies 46: 529–46.

Pope, C., and N. Mays. 2006. Qualitative Research in Health Care. 3rd Edition. Malden,MA: Blackwell Publishing.

Powell, H., S. Mihalas, A. J. Onwuegbuzie, S. Suldo, and C. E. Daley. 2008. “MixedMethods Research in School Psychology: A Mixed Methods Investigation ofTrends in the Literature.” Psychology in the Schools 45: 291–309.

Rosenthal, M. B., B. E. Landon, K. Howitt, H. R. Song, and A. M. Epstein. 2007.“Climbing Up the Pay-For-Performance Learning Curve: Where Are The EarlyAdopters Now?”Health Affairs 26 (6): 1674–82.

Sale, J. E. M., and K. Brazil. 2004. “A Strategy to Identify Critical Appraisal Criteria forPrimaryMixed-Method Studies.”Quality and Quantity 38 (4): 351–65.

Schore, J., L. Foster, and B. Phillips. 2007. “Consumer Enrollment and Experiences intheCash andCounseling Program.”Health Services Research 42 (1, part 2): 446–66.

Shortell, S. 1999. “The Emergence of Qualitative Methods in Health ServicesResearch.”Health Services Research 34: 1083–90.

Tashakkori, A., and C. Teddlie (Eds.). 2010. Sage Handbook of Mixed Methods in Social andBehavioral Research. 2nd Edition. ThousandOaks, CA: Sage.

Teddlie, C., and A. Tashakkori. 2009. Foundations of Mixed Methods Research: IntegratingQuantitative and Qualitative Approaches in the Social and Behavioral Sciences. Thou-sandOaks, CA: Sage.


SUPPORTING INFORMATION

Additional supporting information may be found in the online version of thisarticle:

Appendix SA1: AuthorMatrix.

Please note: Wiley-Blackwell is not responsible for the content or func-tionality of any supporting materials supplied by the authors. Any queries(other than missing material) should be directed to the corresponding authorfor the article.


achievingintegrationinmixedmethods designs ...web2.facs.org/orc2014flashdrive/materials/... ·...

Documents