Mixed Research as a Tool forDeveloping QuantitativeInstruments
Anthony J. Onwuegbuzie,1 Rebecca M. Bustamante,1
and Judith A. Nelson1
Abstract
In this methodological article, the authors present a meta-framework—comprising severalframeworks and models, as well as multiple research approaches—that they call an InstrumentDevelopment and Construct Validation (IDCV) process for optimizing the development of quantita-tive instruments. Using mixed research techniques, the IDCV contains 10 phases that detail theprogression from an interdisciplinary review of the literature to the development of the instru-ment to the evaluation of the instrument development and construct validation process andproduct(s). Crossover analyses represent a key mechanism in the IDCV, wherein analysis typesfrom one tradition (e.g., quantitative analysis) are used to analyze data from a different tradition(e.g., qualitative data). Finally, the authors provide a heuristic example of a rigorous way todevelop a Likert-format scale.
Keywords
mixed research, instrument development, construct validation
Researchers have been mixing qualitative and quantitative approaches for decades (Tashakkori
& Teddlie, 1998; Teddlie & Johnson, 2009). Nevertheless, mixed methods research, also known as
mixed research1—wherein ‘‘a researcher or team of researchers combines elements of qualitative
and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data col-
lection, analysis, inference techniques) for the broad purposes of breadth and depth of understand-
ing and corroboration’’ (R. B. Johnson, Onwuegbuzie, & Turner, 2007, p. 123)—represents ‘‘a
new movement, or discourse, or research paradigm (with a growing number of members) that
has arisen in response to the currents of quantitative research and qualitative research’’ (R. B.
Johnson et al., 2007, p. 113). As noted by Denscombe (2008),
Mixed methods research has developed rapidly in recent years. Championed by writers
such as John Creswell, Abbas Tashakkori, Burke Johnson, Anthony Onwuegbuzie,
1Sam Houston State University, Huntsville, TX, USA
Corresponding Author:
Anthony J. Onwuegbuzie, Department of Educational Leadership and Counseling, Box 2119, Sam Houston State
University, Huntsville, TX 77341-2119
Email: [email protected]
Journal of Mixed Methods Research4(1) 56–78
ªThe Author(s) 2010Reprints and permission: http://www.sagepub.com/journalsPermissions.nav
DOI: 10.1177/1558689809355805http://jmmr.sagepub.com
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Jennifer Greene, Charles Teddlie, and David Morgan, the mixed methods approach has
emerged in the last decade as a research movement with a recognized name and distinct
identity. It has evolved to the point where it is ‘‘increasingly articulated, attached to
research practice, and recognized as the third major research approach or research para-
digm’’ (Johnson, Onwuegbuzie, & Turner, 2007, p. 112). (p. 270)
Although mixed research has become popularized, its potential has not yet been fully realized.
Indeed, many researchers do not mix qualitative and quantitative approaches in optimal ways
(see Powell, Mihalas, Onwuegbuzie, Suldo, & Daley, 2008). In an attempt to provide researchers
with a framework for optimizing their mixed research designs, Collins, Onwuegbuzie, and Sut-
ton (2006), in what they called a rationale and purpose model for designing mixed research stud-
ies, conceptualized four rationales for mixing approaches: participant enrichment (i.e., the
mixing of quantitative and qualitative techniques to optimize the sample, such as increasing
the sample size), instrument fidelity (i.e., maximizing the appropriateness and/or utility of the
instruments used, whether quantitative or qualitative), treatment integrity (i.e., mixing quantita-
tive and qualitative techniques to assess the fidelity of interventions, treatments, or programs),
and significance enhancement (i.e., mixing quantitative and qualitative techniques to maximize
researchers’ interpretations of data).
Of the four rationales for mixing qualitative and qualitative approaches, instrument fidelity
most lacks adequate development. Indeed, with very few exceptions (e.g., Collins et al., 2006;
Hitchcock et al., 2005, 2006), scant guidance has been given to help researchers use mixed
research techniques to optimize the development of either qualitative or quantitative instruments.
This lack of guidance likely stems from the perception that the development of instruments oc-
curs within the same methodological tradition, with the exclusive use of qualitative approaches
deemed as being appropriate to develop qualitative instruments (e.g., interview schedules) and
the exclusive use of quantitative approaches deemed as being appropriate to develop quantitative
instruments (e.g., Likert-format scales). For example, with respect to quantitative instrument
development, in their seminal article, Campbell and Fiske (1959) proposed a comprehensive
quantitative framework—which they called a multitrait–multimethod matrix (MTMM) for
assessing construct-related validity using two validity types: convergent validity and discrimi-
nant validity. More specifically, the MTMM was designed
to provide a framework to assess the effects of trait variance (variance attributed to the
intended construct of interest) and method variance (variance attributable to the specific
method of measurement) by examining convergent and discriminant validity. More gener-
ally, the MTMM also provides information about patterns of associations between meth-
ods, and patterns of associations between constructs and possible interactions between
methods and constructs. (Fabrigar & Joelle Estrada, 2007, p. 666)
As noted by R. B. Johnson et al. (2007, pp. 113-114), Campbell and Fiske’s quantitative cross-
validation framework is viewed by many mixed methodologists as ‘‘formalizing the practice of
using multiple research methods . . . for validation purposes,’’ and thereby providing the impetus
for the concept of triangulation. However, as innovative and useful as the MTMM has been as
a quantitative cross-validation tool, it does have some limitations (for a discussion of limitations
of the MTMM, see, e.g., Fabrigar & Joelle Estrada, 2007; see also Teddlie & Johnson, 2009). In
particular, Brewer and Hunter (2006) concluded that MTMM ‘‘warned of over-reliance upon,
and overconfidence in, any single type of research methods’’ (p. xiii). That is, the key assumption
underlying the MTMM is that quantitative techniques are sufficient for the development of quan-
titative instruments. Yet qualitative techniques can be used to enhance the development of quan-
titative instruments and vice versa (Collins et al., 2006). Thus, clearly, more publications are
Onwuegbuzie et al. 57
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
needed that outline explicitly ways of optimizing the development of instruments by mixing
qualitative and quantitative techniques.
Therefore, the purpose of the present methodological article is to provide a mixed research
framework for optimizing the development of a quantitative instrument.2 First, we discuss the
concept of crossover analyses, which involves using one or more analysis types associated
with one tradition (e.g., quantitative analysis) to analyze data associated with a different tradition
(e.g., qualitative data). Second, we deconstruct the concept of instrument fidelity as it pertains to
the construction of quantitative instruments. In particular, we present a framework for develop-
ing and assessing the fidelity of a quantitative instrument via what is termed as areas of validity
evidence and specify the types of crossover analyses that are pertinent for each of the multiple
areas of validity evidence. Third, we describe the 10-phase process for optimizing the develop-
ment of a quantitative instrument. We outline how qualitative and quantitative approaches can be
combined to enhance instrument fidelity at different stages of the development process. Finally,
we provide a brief heuristic example of a rigorous way to develop a Likert-format scale.
Crossover Analyses
Onwuegbuzie and Combs (in press) outlined the concept of crossover analyses. According to
these methodologists, crossover analyses represent the highest form of combining quantitative
and qualitative data analysis techniques—referred to as mixed analysis (Onwuegbuzie, Collins,
& Leech, in press)—because the researcher often has to make Gestalt switches (Kuhn, 1962)
from a qualitative lens to a quantitative lens and vice versa, going back and forth several times
until maximal meaning has been extracted from the data. Moreover, crossover analyses neces-
sitate the researcher to mix or combine a strong use of quantitative and qualitative assumptions
and stances. For example, a researcher might blend a constructivist (analytical) stance (i.e., with
an ontology that multiple contradictory, but equally valid accounts of the same phenomenon can
prevail that represent multiple realities) with a postpositivist (analytical) stance (i.e., with an
ontology that social science research should be objective) by, say, using exploratory factor anal-
ysis to examine the structure of themes that emerged from a qualitative analysis (see Onwueg-
buzie, 2003) as a means of furthering construct validation. Such a strategy would yield
a sequential qualitative–quantitative mixed analysis (Onwuegbuzie & Teddlie, 2003), with the
quantitative analysis phase informing or expanding on the qualitative analyses—or what Greene,
Caracelli, and Graham (1989) refer to as complementarity (i.e., seeking elaboration, illustration,
enhancement, and clarification of the findings from one method with results from the other
method), development (i.e., using the findings from one method to help inform the other
method), or expansion (i.e., expanding the breadth and range of a study by using multiple meth-
ods for different study phases). Alternatively, a researcher might use a thematic analysis to com-
pare emergent themes to factors emerging from an exploratory factor analysis. This analysis
would yield a concurrent mixed analysis with a rationale of triangulation (i.e., comparing find-
ings from quantitative data with qualitative results in hopes of convergence; Greene et al., 1989).
Such analyses have the potential to yield stronger meta-inferences (i.e., involving combining
interpretations of findings stemming from the qualitative and quantitative data analyses into
a coherent whole).
According to Onwuegbuzie and Combs (in press), crossover analyses comprise the following
techniques: integrated data reduction (i.e., reducing the dimensionality of qualitative data/
findings using quantitative analysis and/or quantitative data/findings using quantitative analy-
sis), integrated data display (i.e., visually presenting both qualitative and quantitative results
within the same display), data transformation (i.e., converting quantitative data into data that
can be analyzed qualitatively and/or qualitative data into numerical codes that can be analyzed
58 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
statistically), data correlation (i.e., correlating qualitative data with quantitized/quantitative data
and/or quantitative data with qualitized/qualitative data), data consolidation (i.e., combining or
merging multiple data sets to create new or consolidated codes, variables, or data sets), data com-
parison (i.e., comparing qualitative and quantitative data/findings), data integration (i.e., inte-
grating qualitative and quantitative data/findings either into a coherent whole or two separate
sets of coherent wholes), warranted assertion analysis (i.e., reviewing all qualitative and quan-
titative data to yield meta-inferences), and data importation (i.e., using follow-up findings
from qualitative analysis to inform the quantitative analysis or vice versa). All these crossover
analysis techniques involve some level of abductive logic, which involves moving back and forth
between inductive and deductive logic. These crossover analysis procedures also involve a form
of intersubjectivity (i.e., agreement about reality, ultimately, is socially constructed). Further-
more, these analyses involve the incorporation of both insiders’ (i.e., emic) views and the
researcher-observer’s (i.e., etic) views for instrument development and construct validation
and that the balance between the emic perspectives (i.e., stemming from the participants
involved in the instrument development and/or construct validation) and etic perspectives
(e.g., stemming from extant theories and the researcher’s a priori assumptions) is appropriate
such that quality meta-inferences can be made. This use of abductive logic, intersubjectivity,
and emic–etic perspectives makes the use of mixed research in general and crossover analyses
in particular very appealing for instrument development and construct validation.
Fidelity of Quantitative Instruments
Instrument fidelity includes steps taken by the researcher to develop an instrument and to
maximize its appropriateness and/or utility. Onwuegbuzie, Daniel, and Collins (2009) have pro-
vided a useful framework that can be used both to develop and assess the fidelity of quantitative
instruments. These authors extended Messick’s (1989, 1995) theory of validity by combining his
conceptualization with the traditional notion of validity, thereby yielding a reconceptualization
of validity as presented in Table 1, which they called a meta-validation model. Although viewed
as a unitary concept, Table 1 indicates that content-, criterion-, and construct-related validity can
be subdivided into several areas of evidence. Table 2 delineates the types of crossover analyses
that can be conducted for each validity type. It can be seen from this table that every type of
crossover analysis appears at least once, with data comparison, data integration, and warranted
assertion each appearing on six occasions. That is, these analyses each can be used in the assess-
ment of six validity types.
In this article, we will show that although most of these validity types lend themselves to
quantitative approaches, they can also be addressed via qualitative approaches. For example,
whereas structural validity (see Tables 1 and 2) typically has been addressed using quantitative
analysis of quantitative data (e.g., exploratory factor analysis; see Crocker & Algina, 1986), we
will show how qualitative analysis techniques also can be used to assess structural validity—
yielding a mixed analysis meta-framework for instrument development/fidelity and construct
validation.
Meta-Framework for Instrument Development/Fidelityand Construct Validation
Figure 1 provides an overview of our meta-framework for instrument development/fidelity
and construct validation. This mixed research–based meta-framework was designed to help
instrument developers undergo a rigorous and comprehensive process during instrument
Onwuegbuzie et al. 59
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
development/construct validation. We call our framework the Instrument Development and
Construct Validation (IDCV). This framework consists of 10 interactive phases, as shown in
Figure 1. The 10 phases are the following:
1. Conceptualize the construct of interest
2. Identify and describe behaviors that underlie the construct
3. Develop initial instrument
4. Pilot-test initial instrument
5. Design and field-test revised instrument
Table 1. Areas of Validity Evidence
Validity Type Description
Criterion-relatedConcurrent validity Assesses the extent to which scores on an instrument are related to
scores on another, already established instrument administeredapproximately simultaneously or to a measurement of some othercriterion that is available at the same point in time as the scoreson the instrument of interest
Predictive validity Assesses the extent to which scores on an instrument are related to scoreson another, already established instrument administered in the future orto a measurement of some other criterion that is available at a future pointin time as the scores on the instrument of interest
Content-relatedFace validity Assesses the extent to which the items appear relevant, important,
and interesting to the respondentItem validity Assesses the extent to which the specific items represent measurement
in the intended content areaSampling validity Assesses the extent to which the full set of items sample the total
content areaConstruct-related
Substantive validity Assesses evidence regarding the theoretical and empirical analysisof the knowledge, skills, and processes hypothesized to underlierespondents’ scores
Structural validity Assesses how well the scoring structure of the instrument correspondsto the construct domain
Convergent validity Assesses the extent to which scores yielded from the instrument ofinterest being highly correlated with scores from other instrumentsthat measure the same construct
Discriminant validity Assesses the extent to which scores generated from the instrument ofinterest being slightly but not significantly related to scores frominstruments that measure concepts theoretically and empirically relatedto but not the same as the construct of interest
Divergent validity Assesses the extent to which scores yielded from the instrument ofinterest not being correlated with measures of constructs antitheticalto the construct of interest
Outcome validity Assesses the meaning of scores and the intended and unintendedconsequences of using the instrument
Generalizability Assesses the extent that meaning and use associated with a set of scorescan be generalized to other populations
Source: Reproduced from Collins et al. (2006). Reprinted with kind permission of the Learning DisabilitiesWorldwide.
60 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
6. Validate revised instrument: Quantitative analysis phase
7. Validate revised instrument: Qualitative analysis phase
8. Validate revised instrument: Mixed analysis phase: Qualitative-dominant crossover
analyses
9. Validate revised instrument: Mixed analysis phase: Quantitative-dominant crossover
analyses
10. Evaluate the instrument development/construct evaluation process and product
These 10 stages are blended, allowing for movement among the stages as new findings
emerge. A constant reflection component encourages the researcher to evaluate progress related
to each phase and to identify and address experienced emotions.
Framework for Debriefing the Instrument Developer
In addition to using Onwuegbuzie et al.’s (2009) meta-validation framework, Greene et al.’s
(1989) rationale framework for combining qualitative and quantitative data, and Onwuegbuzie
and Combs’s (in press) typology of cross-analysis strategies at their core, the meta-framework
for instrument development/fidelity and construct validation incorporates Onwuegbuzie, Leech,
and Collins’s (2008) framework for debriefing the researcher. Onwuegbuzie et al. (2008)
recently introduced the concept of debriefing the researcher, wherein the researcher is inter-
viewed by an individual who is not involved directly in the study (e.g., disinterested peer) but
who understands the research construct or topic that is being studied. Onwuegbuzie et al.
(2008) contend that debriefing interview data helps the researcher to evaluate the decisions
made at the various stages of the research process, as well as to reflect on assumptions, biases,
feelings, and perceptions that were present at the beginning of the study and that evolved as the
study progressed. According to Onwuegbuzie et al. (2008), having the researcher explicitly
reveal these elements to the debriefing interviewer helps increase the researcher’s understanding
of the research process as it unfolds, as well as provides an audit trail. We extend this concept of
debriefing to the process of instrument developer/construct validation. In particular, we recom-
mend that the instrument developer(s) be interviewed at all 10 phases of the IDCV process by
Table 2. Crossover Analysis Strategies by Validity Type
Validity Type Type of Crossover Analysis
Criterion relatedConcurrent validity Data correlation, data comparisonPredictive validity Data correlation, data comparison
Content relatedFace validity Integrated data display, warranted assertion, data integrationItem validity Data comparison, warranted assertion, data integrationSampling validity Integrated data display, data comparison, warranted assertion, data integration
Construct relatedSubstantive validity Data consolidation, data comparison, warranted assertion, data integrationStructural validity Integrated data reduction, data transformationConvergent validity Data correlation, data comparisonDiscriminant validity Data correlation, data comparisonDivergent validity Data correlation, data comparisonOutcome validity Data transformation, data consolidation, warranted assertion, data integrationGeneralizability Warranted assertion, data integration, data importation
Onwuegbuzie et al. 61
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
one or more disinterested peers who have expertise in measurement theory and instrument
development.
Phase 1: Conceptualize the Construct of Interest
Phase 1 involves development of the construct of interest. As recommended by Combs, Bus-
tamante, and Onwuegbuzie (in press), an important early goal is for researchers—instrument
developers in this case—to be aware of their own personal belief systems related to three dimen-
sions of belief systems: (a) overall worldview, (b) research philosophy, and (c) discipline-spe-
cific philosophy. Indeed, researchers should strive to understand the role that these three
dimensions have on the instrument development/construct validation process. First, as noted
Phase 2: Identify and Describe Behaviors that Underlie the Construct
Phase 4: Field-test Initial Instrument
Phase 5: Design and Field-test Revised Instrument
Phase 6: Validate Revised Instrument: Quantitative Analysis Phase
Phase 7: Validate Revised Instrument: Qualitative Analysis Phase
Phase 8: Validate Revised Instrument: Mixed Analysis Phase: Qualitative Dominant Cross-Over Analyses
Phase 9: Validate Revised Instrument: Mixed Analysis Phase: Quantitative Dominant Cross-Over Analyses
Phase 10: Evaluating the Instrument Development/Construct Evaluation Process and Product
Phase 1: Conceptualize the Construct of Interest
Phase 3: Develop Initial Instrument
Figure 1. Instrument development and construct validation (IDCV) process
62 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
by R. B. Johnson, Meeker, and Onwuegbuzie (2004), it is essential that researchers are cognizant
of their personal worldviews, namely, the lens through which one sees and interprets the world.
Second, with respect to research philosophy, Onwuegbuzie, Johnson, and Collins (2009) have
provided a framework for examining one’s own research belief systems, particularly as they per-
tain to conducting qualitative, quantitative, and mixed analyses. According to these authors,
many researchers do not realize that their research philosophy (e.g., postpositivism, constructiv-
ism, critical theory) does not prevent them from conducting analyses that are traditionally asso-
ciated with another approach. For example, having a postpositivist orientation does not prevent
a researcher from conducting an array of qualitative analyses. This point is particularly important
because it suggests that the development of a quantitative instrument—traditionally considered
to be an activity that belongs to the postpositivist philosophical stance—can involve both quan-
titative and qualitative analyses. Third, Combs et al. (in press) recommend that instrumental de-
velopers identify their own discipline-specific philosophies. In the field of instrument
development, discipline-specific philosophies could be reframed as construct-specific philoso-
phies. For example, if a researcher was interested in developing a measure of statistics anxiety,
then he/she should explore his beliefs systems and biases surrounding this construct.
One of the primary ways of developing the construct of interest is by conducting an extensive
review of the literature. As part of the literature review, the instrument developer should identify
the relevant theoretical framework(s) (i.e., ‘‘developed by using an established, coherent expla-
nation of certain sorts of phenomena and relationships’’; Lester, 2005, p. 458) and/or conceptual
framework(s) (i.e., ‘‘an argument that the concepts chosen for investigation, and any anticipated
relationships among them, will be appropriate and useful given the research problem under
investigation’’; Lester, 2005, p. 460) from the extant literature (i.e., etic perspectives). A few
mixed research–based models recently have been developed to help instrument developers con-
duct rigorous literature reviews (Combs et al., in press; Onwuegbuzie, Collins, Leech, Dellinger,
& Jiao, in press).
In addition, the instrument developer should consult with a diverse set of local experts (i.e.,
emic perspectives). Where possible, the instrument developer should set up focus group meet-
ings of these local experts to capitalize on the group dynamics and social processes that occur
in focus groups (e.g., Morgan, 1998). We recommend that mixed research techniques are used
to collect and analyze the focus group data (see Onwuegbuzie, Leech, Dickinson, & Zoran, in
press). Also, it is important for the instrument developer to ensure that the voices of key inform-
ants, which include those on whom the instrument will be administered, are heard, with a view to
understanding their cultural milieux. Individual interviews, focus groups, and direct observations
can play an important role here. It is essential that detailed field notes are kept and that an audit
trail is left (see Halpern, 1983). A key goal in Phase 1 is the development of an instrument that
possesses cultural sensitivity (Banks & McGee Banks, 2001), so that, when developed, it will
yield data that are optimally reliable and valid.
Phase 2: Identify and Describe Behaviors That Underlie the Construct
The information from the literature review and the various data from the local experts and key
informants could be used to prepare the instrument developer for Phase 2. In Phase 2, the instru-
ment developer might undergo the grounded theory analytical steps of open coding (i.e., coding
the literature and data extracted from the local experts and key informants by chunking the infor-
mation into smaller segments and then assigning a descriptor, or ‘‘code,’’ for each segment) and
axial coding (i.e., grouping the codes into similar categories; Glaser & Strauss, 1967). Alterna-
tively, some form of ethnographic analysis could be undertaken, comprising Spradley’s (1979)
analysis procedures of domain analysis (i.e., using the relationships between symbols and
Onwuegbuzie et al. 63
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
referents to identify domains), taxonomic analysis (i.e., creating a system of classification that
catalogs the domains into a visual representation to help the researcher understand the relation-
ships among the domains), and componential analysis (i.e., using a visual representation to dis-
cover the differences among the subcomponents of domains). Alternatively still, procedures such
as the Delphi technique might be used. In any case, whatever data collection and data analysis
tools are used, it is essential that a series of rounds is conducted, with each round involving the
collection of qualitative and quantitative data (e.g., interview data, focus group data, observa-
tional data) that are increasingly focused until data saturation is reached (i.e., no new or relevant
information seems to emerge relating to a category, and the category development is well con-
ceptualized and validated; Lincoln & Guba, 1985). Once data saturation is reached, the instru-
ment developer should be in a position to identify the behaviors underlying the construct of
interest. If data saturation has not been reached, the instrument developer might return to Phase
1 to reconceptualize the theory and/or hypotheses, reevaluate any philosophical and research
assumptions held by the instrument developers, and/or collect additional data from the local
experts and key informants.
Phase 3: Develop Initial Instrument
Once all the behaviors have been identified, the instrument developer should be in a position
to start writing items. These items might be written using some type of table of specifications that
links the theory extracted from Phase 1 (etic viewpoint, deductive logic) and the information pro-
vided by the local experts and key informants in Phase 1 and Phase 2 (emic viewpoints, inductive
logic). Such a table of specifications would contain all the identified behaviors (e.g., cognitive,
affective, psychomotor). Where possible, the team of local experts and key informants should
participate in the item-writing process or, at the very least, should be asked to provide feedback
on the items to ensure that they are culturally sensitive, being informed by both emic and etic
perspectives. Although a quantitative instrument is being developed, it is essential that each
item is accompanied by open-ended items that ask the field-testing respondents to assess the
quality of each item and to offer suggestions for improvement. If the feedback given suggests
that the instrument is extremely underdeveloped, then the instrument developer might return
to Phase 2 to reidentify and reconceptualize the behaviors that underlie the construct by collect-
ing additional data from the local experts and/or key informants.
Phase 4: Pilot-Test Initial Instrument
Once the initial instrument has been developed, it is necessary that it is subjected to a field
test. This field test optimally would represent a mixed research study to assess the appropriate-
ness of each item. In particular, each item should be assessed for clarity, esthetics, relevancy,
tone, length of time needed for a response, and, above all, cultural competence. Because sample
sizes at this stage are likely to be smaller than those at the next phase (i.e., Phase 5) when a more
developed version is field-tested, caution typically should be exercised when computing and in-
terpreting score reliability and score validity coefficients. Rather, the focus at this phase should
be more on the content-related validity (i.e., face validity, sampling validity, item validity) and
two elements of construct-related validity (i.e., outcome validity, generalizability) of the initial
instrument (see Table 1) than on the other areas of validity. It is important to note that additional
data beyond the actual instrument itself might be collected. For example, additional individual
interviews and/or focus group interviews might be conducted involving the same or different
local experts or key informants. If the feedback given suggests that the instrument grossly lacks
content-related validity (e.g., the pilot-test participants conclude that the items do not sample the
64 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
total content area and thus has low sampling validity) or lacks cultural relevancy, then the instru-
ment developer might return to Phase 3 to redevelop the instrument.
Phase 5: Design and Field-Test Revised Instrument
After the instrument developer has received qualitative and quantitative data on the initial
instrument, he/she will be in a position to refine or discard problematic items. Once the
initial instrument has been revised according to the responses and feedback provided by the initial
field-test sample, it is then sent out for a more extensive field test. Here, the sample size should be
large enough to justify conducting an exploratory factor analysis and yields adequate (sub)scale
score reliabilities with relatively narrow confidence intervals. Thus, sample size guidelines for
exploratory factor analyses should be followed such as those recommended by researchers who
espouse the importance of an adequate case-to-variable ratio from 5 participants per item on the
quantitative instrument (e.g., Likert-format scale, rating scale) as the bare minimum to at least
10 participants per item (Cattell, 1978; Everitt, 1975). (For excellent discussion of the role of sam-
ple size in exploratory factor analysis, see Hogarty, Hines, Kromrey, Ferron, & Mumford, 2005;
Mundfrom, Shaw, & Ke, 2005.)
In traditional quantitative instrument development, the researcher(s) designs quantitative
items (e.g., Likert-format items, rating scale items) and then field-tests these items. However,
in our IDCV framework, we believe that this standard technique is inadequate because it leads
to the sole reliance of quantitative data to assess the psychometric properties of the instrument.
Rather, we recommend strongly that qualitative-based (i.e., open-ended) items be included with
the quantitative items. (Alternatively, administering the quantitative instrument could be accom-
panied by other forms of qualitative data collection such as individual interviews and focus group
interviews on all or some of the people who completed the quantitative instrument.) For example,
in developing a Likert-format measure of statistics anxiety, we would recommend that the
researcher also collect qualitative data such as by asking participants to describe situations in
which they experience elevated levels of statistics anxiety. These qualitative data then could
be correlated and/or compared with the quantitative responses, not only as a means of enhancing
instrument development but also as a way of validating the construct. However, the biggest role
that the collection of qualitative data can play is eliciting information from participants regarding
their perceptions of the cultural relevance of the underlying quantitative instrument. In any case,
this practice of eliciting qualitative responses in addition to quantitative responses from the same
group of field-test participants would lead to a mixed research sampling design that Onwuegbu-
zie and Collins (2007) referred to as a concurrent design using identical samples.
Phase 6: Validate Revised Instrument: Quantitative Analysis Phase
After collecting the quantitative and qualitative responses, the next phase is to analyze the
quantitative (e.g., Likert-form) data. Here, the major goal is to assess the content-related validity
(i.e., item validity), criterion-related validity (i.e., concurrent validity, predictive validity), and
construct-related validity (i.e., structural validity, convergent validity, discriminant validity,
divergent validity) of the scale. For example, with respect to item validity, different item analysis
indices can be used depending on whether the quantitative measure pertains to the cognitive,
affective, or psychomotor domain. If the cognitive domain is of interest, then item developers
can determine, analyze, and interpret numerous indices (e.g., item difficulty, item discrimina-
tion, point biserial correlation, biserial correlation, difficulty level of an average item, phi coef-
ficient, tetrachoric correlation coefficient, item reliability index, item validity index,
instructional sensitivity, indices of agreement, differential item functioning indices; see Crocker
Onwuegbuzie et al. 65
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
& Algina, 1986). Item developers could decide whether to use classical test theory or item
response theory. With respect to the cognitive domain, concurrent validity and predictive valid-
ity could be assessed via correlational indices, as can the construct-related validity areas of con-
vergent validity, discriminant validity, and divergent validity. Finally, structural validity can be
assessed via an exploratory factor analysis.
With respect to the affective domain, most of the indices that are pertinent for the cognitive
domain also can be used for the affective domain. However, it is exploratory factor analysis that
is a key analysis in Phase 6. For excellent information as to how to conduct exploratory factor
analysis in a rigorous way, we refer the reader to Henson, Capraro, and Capraro (2004), Henson
and Roberts, 2006; Hetzel (1996), Kieffer (1999), Onwuegbuzie and Daniel (2003), Thompson
(2004), and Tinsley and Tinsley (1987).
Phase 7: Validate Revised Instrument: Qualitative Analysis Phase
Phase 7, the next phase, involves the analysis of the qualitative data. Here, the major goal is to
address one or more of Greene et al.’s (1989) five purposes for mixing quantitative and qualita-
tive data (i.e., triangulation, complementarity, development, initiation, and expansion). When
analyzing the qualitative data, instrument developers have many qualitative data analysis tools
at their disposal, including the following: method of constant comparison analysis, keywords-
in-context, word count, classical content analysis, domain analysis, taxonomic analysis, compo-
nential analysis, conversation analysis, discourse analysis, secondary analysis, membership cat-
egorization analysis, narrative analysis, qualitative comparative analysis, semiotics, manifest
content analysis, latent content analysis, text mining, and micro-interlocutor analysis. For a com-
prehensive review of these qualitative data analysis techniques, we refer the reader to Leech and
Onwuegbuzie (2007, 2008).
Phase 8: Validate Revised Instrument: Mixed Analysis Phase:Qualitative-Dominant Crossover Analyses
Phase 8 involves conducting one or more of the nine crossover analyses (i.e., integrated data
reduction, integrated data display, data transformation, data correlation, data consolidation, data
comparison, data integration, warranted assertion analysis, and data importation) to address even
further one or more of Greene et al.’s (1989) five purposes for mixing quantitative and qualitative
data. This phase either involves conducting a qualitative analysis of the quantitative data (i.e.,
scores from quantitative instrument) obtained in Phase 6 or conducting a quantitative analysis
of the qualitative data extracted in Phase 7. With respect to the former, quantitative data can
be transformed to qualitative data via narrative profile formation, wherein narrative descriptions
are constructed from quantitative data. These profiles include the following: modal profiles (i.e.,
detailed narrative descriptions of a group of people based on the most frequently occurring at-
tributes in the group), average profiles (i.e., profiles based on the average of a number of attrib-
utes of the individuals or situations), holistic profiles (i.e., overall impressions of the investigator
regarding the unit of investigation), comparative profiles (i.e., obtained by comparing one unit of
analysis with another, and includes possible similarities/differences between them), and norma-
tive profiles (i.e., similar to narrative profiles but are based on the comparison of an individual or
group with a standard (e.g., normative group) (see Tashakkori & Teddlie, 1998).
In terms of conducting a quantitative analysis of the qualitative data extracted in Phase 7,
numerous strategies exist (see Onwuegbuzie, Collins, & Leech, in press). However, perhaps
the most useful technique is factor-analyzing the themes that emerged in Phase 7. Onwuegbuzie
66 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
(2003) originally proposed the concept of factor-analyzing emergent themes. According to On-
wuegbuzie (2003), a score of ‘‘1’’ is given for a theme if it represents a significant statement or
nonverbal communication that supports endorsement of that theme pertaining to that individual;
otherwise, a score of ‘‘0’’ is given for that theme. That is, for each field-test participant, each
theme is dichotomized either to a score of ‘‘1’’ or a ‘‘0,’’ depending on whether it is represented
by that individual. This dichotomization leads to the formation of what Onwuegbuzie (2003)
referred to as an ‘‘interrespondent matrix’’(i.e., participant × theme matrix) that contains a com-
bination of 0s and 1s (p. 396), which indicates which individuals contribute to each theme that
emerged from Phase 7. Factors that emerge from the exploratory factor analysis of focus group
themes are called ‘‘meta-themes’’ (Onwuegbuzie, 2003, p. 398), which represent themes at
a higher level of abstraction than the original emergent themes. According to Onwuegbuzie
(2003), ‘‘The manner in which the emergent themes cluster within each factor (i.e.,
meta-theme) facilitates identification of the interrelationships among the themes’’ (p. 398).
Thus, factor-analyzing themes that emerge to yield meta-themes represent integrated data reduc-
tion, inasmuch as a quantitative technique (i.e., exploratory factor analysis) is applied to quali-
tative data that represent a reduction of the original qualitative data (i.e., voice of the focus group
participants, nonverbal communication of the focus group participants) via a qualitative analysis
(e.g., method of constant comparison).
Phase 9: Validate Revised Instrument: Mixed Analysis Phase:Quantitative-Dominant Crossover Analyses
Phase 9 involves conducting one or more sets of quantitative-dominant crossover analyses.
Again, the instrument developers have many data analysis tools at their disposal. In particular,
this phase builds on the previous phase. For example, the factor scores derived from the factor
analysis of the emergent themes in Phase 8 could be correlated to the factor scores extracted via
the exploratory factor analysis of the quantitative instrument that was undertaken in Phase 6. As
another example, the themes that emerged in Phase 7 could be correlated to the factors that
emerged in Phase 6, or the meta-themes that emerged in Phase 8 could be correlated to the factors
that emerged in Phase 6.
Phase 10: Evaluate the Instrument Development/ConstructEvaluation Process and Product
Phase 10 involves a comprehensive evaluation both of the product and the process. The prod-
uct, which is the revised instrument, can be evaluated via the findings emerging in Phase 6 to
Phase 9. We suggest use of Onwuegbuzie et al.’s (2008) framework for debriefing the researcher.
This framework can help instrument developers reflect on the IDCV process to discover how
they felt about going through this process, determine which data collection and analytical strat-
egies were useful, and uncover areas for further growth and development of the instrument.
Based on the evaluation data, the instrument developer might need to return to Phase 5 and revise
the instrument further, and repeat Phase 6 to Phase 9, or he/she might even return to Phase 1 to
reconceptualize the theory and/or hypotheses, reevaluate any philosophical and research assump-
tions held by the instrument developers, and/or collect additional data from the local experts and
key informants, before repeating Phase 2 to Phase 9. Thus, the IDCV framework represents an
iterative, cyclical process that promotes more rigor to the process of instrument development and
construct validation.
Onwuegbuzie et al. 67
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
In its most rigorous form, the instrument developer could design and implement a comprehen-
sive, mixed research–based evaluation system. This evaluation system might involve the use of
formal models of program evaluation such as Stufflebeam’s CIPP model, which provides
a framework for the evaluation of the IDCV’s context, input, processes, and products (Stuffle-
beam, 2002). Also, instructional design models such as the systems-based Dick and Carey model
(Dick, Carey, & Carey, 2004) can be used to drive the evaluation process via the applications of
learning theories and principles that inform the process of instrument development and construct
validation.
Heuristic Example
Background Information
To illustrate the role that mixed research can play in developing quantitative instruments in
general and the IDCV in particular, we briefly present the process that Bustamante, Nelson,
and Onwuegbuzie (2009) used to develop an instrument they called the School-Wide Cultural
Competence Observation Checklist (SCCOC). The SCCOC was developed as one of several
assessment tools for use in conducting school culture audits. A culture audit is an approach to
collecting organizational data from multiple sources to determine schoolwide cultural compe-
tence or how well a school’s policies, programs, and practices reflect the needs and experiences
of culturally and linguistically diverse groups in a school and school community (Bustamante,
2006). This process is displayed in Figure 2, which was created by the current authors.
Phase 1: Conceptualizing the construct of interest: Interdisciplinary review of the literature. The ini-
tial phase of SCCOC development (Qualitative; inductive logic) involved an extensive and sys-
tematic review of the academic literature (see Boote & Beile, 2005; Combs et al., in press;
Onwuegbuzie, Collins, Leech, Dellinger, et al., in press) to explore how the notions of individual
and organizational cultural competence had been defined and examined in a variety of academic
disciplines. This was undertaken by first conceptualizing the construct of culture because the
idea of cultural competence essentially is rooted in the concept of culture. This initial step
included a review of relevant studies from the fields of anthropology (Geertz, 1979; Hall,
1976; Kluckhohn & Strodbeck, 1961), sociology (Van Maanen, 1979), intercultural communi-
cation (Brislin, 1993; Casmir, 1999; Hammer, 1989; Ting-Toomey & Oetzel, 2001; Wiseman
& Koester, 1993), cross-cultural psychology (Triandis, 1994), and organizational theory (Hof-
stede, 1980, 2001; Schein, 1992; Trompenaars & Hampden-Turner, 1998). Second, the concepts
of individual and organizational cultural competence as applied to school settings also was
explored by reviewing literature from relevant fields, such as intercultural and cross-cultural
communication (Lustig & Koester, 2002; Lynch & Hanson, 1998; Wiseman & Koester,
1993), multicultural education (Banks, 2002), multicultural competence (Holcomb-McCoy,
2004), cultural proficiency (Lindsey, Roberts, CampbellJones, 2005; Lindsey, Robins, & Terrell,
2003; Salvaggio, 2003), culturally responsive schools (L. S. Johnson, 2003; Ladson-Billings,
1995), inclusive schools (Henze, Katz, Norte, Sather, & Walker, 2002; Riehl, 2000), school
equity audits (Skrla Scheurich, Garcia, & Nolly, 2006), diversity and cross-cultural leadership
in corporate settings (House, Hanges, Javidan, Dorfman, & Gupta, 2004; Kochan et al.,
2003), and cultural competence as defined in human service organizations (Cross, Bazron, Den-
nis, & Isaacs, 1989; Darnell & Kuperminc, 2006; National Center for Cultural Competence,
2005; Sue, 2001; Sue & Constantine, 2005). These interdisciplinary literature reviews served
to identify and define overlapping and deviant themes of similar constructs.
Additionally, in conceptualizing the construct of schoolwide cultural competence, the re-
searchers examined their own belief systems (Combs et al., in press; R. B. Johnson et al.,
68 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
2004). Researcher reflection on our belief systems revealed a tendency to view organization con-
texts through a cultural lens, a commitment to cultural relativism and diversity in learning envi-
ronments, and research belief systems that reflected pragmatic, constructivist, and critical theory
orientations. In Phase 1 of the IDCV model, an examination of researcher belief systems is
described as an important and often overlooked step of instrument development.
Phase 2: Identifying and describing behaviors that underlie the construct: Delphi study. The initial
review of relevant literature informing the construct of schoolwide cultural competence was fol-
lowed by a Delphi study that was conducted with a panel of experts who were practicing school
leaders. Experts completed four rounds of questionnaires until consensus was reached and satu-
ration of themes became evident. These narrative Delphi study results provided some initial ideas
about how cultural competence might be manifested in schools, but specifically in international
Phase 10: Evaluating the instrument development/construct evaluation process and product
Phase 9: Validating revised SCCOC: Quantitative-dominant cross-over mixed analyses
Phase 8: Validating revised SCCOC: Qualitative-dominant cross-over mixed analyses
Phase 7: Validating revised SCCOC: Qualitative Analysis Phase:
Phase 6: Validating revised SCCOC: Quantitative Phase: Exploratory Factor Analysis
Phase 5: Designing and field-testing revised SCCOC: Instrument fidelity
Phase 4: Pilot-testing of Initial SCCOC Instrument
Phase 3: SCCOC Development
Phase 2: Delphi Study
Phase 1: Interdisciplinary Review of the Literature
Figure 2. Instrument development and construct validation (IDCV) process: An heuristic example:School-Wide Cultural Competence Observation Checklist (SCCOC)
Onwuegbuzie et al. 69
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
school settings (Bustamante, 2005). Thus, the Delphi research indicated several potential
domains and indicators of schoolwide cultural competence and proficiency that led to a more
extensive review of the literature. Consequently, the constructs and indicators conceptualized
in this phase were grounded in the relevant literature on cultural competence and formed a basis
for initial instrument design.
Phase 3: Developing initial instrument: SCCOC development. The original SCCOC instrument
contained 43 items and asked observers to provide yes/no responses to statements in several areas
related to potentially culturally competent policies, programs, and practices (Quantitative phase).
Items were developed based on domains identified in the initial Delphi study and were supple-
mented with additional items drawn from the literature on cultural competence and proficiency in
schools.
Phase 4: Pilot-testing of initial instrument. Pretesting for ‘‘face validity’’ was undertaken in focus
groups with graduate students in principal preparation programs who were working in schools at
the time (Qualitative). The participants also were asked to make observations and reflections
using the SCCOC in their actual school environments and to complete a separate open-ended
questionnaire in which they provided specific feedback on the utility and shortcomings of the
original instrument. Based on these pretest data gathered from school leader focus groups, revi-
sions were made to the SCCOC and a second version was created. The second version of the
SCCOC was revised in an unpublished manuscript and contained 33 items (Bustamante & Nel-
son, 2007). The SCCOC continues to be field-tested and revised to ensure instrument relevance
and fidelity.
Phase 5: Designing and field-testing revised instrument: Instrument fidelity. Phase 5 of the SCCOC
instrument fidelity process involved designing an Internet questionnaire to elicit perceptions
from a larger sample of practicing school leaders of the applicability and utility of the SCCOC
in assessing schoolwide cultural competence. Participants were 151 school leaders, comprising
teacher leaders, professional school counselors in training, directors, assistant principals, and
principals. Participants were asked to rate how important they found each checklist item to be
in assessing schoolwide cultural competence (Quantitative) and how they would categorize or
describe each item in their own words (Qualitative). Participants were asked to rate each SCCOC
item on a 4-point Likert-format scale anchored by 1 (strongly agree) and 4 (strongly disagree).
Phase 6: Validating revised instrument: Quantitative analysis phase: Exploratory factor analysis (EFA).The fourth phase (Quantitative phase) of testing for instrument fidelity of the SCCOC involved
use of an EFA to identify the factor structure of the SCCOC. Results revealed two factors that
explained 72.1% of the total variance called Policy (22 items) and Practice (11 items), which
yielded score reliability coefficients of .97 and .89, respectively (Nelson, Bustamante, Wilson,
& Onwuegbuzie, 2008). To check for divergent validity, SCCOC items were compared with
items on occupational health and climate surveys that are typically administered in schools.
Often, school climate and culture erroneously are used interchangeably in academic literature
and practice. However, scholars contend that the two constructs are different. School climate typ-
ically refers more to a temporary, ephemeral situation in a school, perhaps brought on by a school
safety incident or a school event. It may include feelings about the school building or support
staff, school safety, orderliness, and current teacher morale (Freiberg, 1998). School culture,
on the other hand, is characterized by many of the same elements described in seminal anthro-
pological definitions of culture that involve embedded characteristics such as shared values,
beliefs, traditions, and language that are passed on over time (Karpicke & Murphy, 1996; Schein,
1992).
Phase 7: Validating revised instrument: Qualitative analysis phase. In addition to completing both
the 33-item SCCOC and the Importance Scale, participants were asked to comment on the char-
acteristics of the items in a text box that was included on the same Internet survey (Qualitative
70 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
phase). The open-ended responses were analyzed using the method of constant comparison
(Glaser & Strauss, 1967). Four themes emerged from this iterative analysis process that further
informed the fidelity of the SCCOC: (a) policy, (b) practice, (c) school climate and culture, and
(d) barriers to schoolwide cultural competence (i.e., lack of clarity in roles and responsibilities,
resource constraints, limited knowledge of culturally responsive practices) (Bustamante, Nelson,
& Onwuegbuzie, in press).
Phase 8: Validating revised instrument: Mixed analysis phase: Qualitative-dominant crossover analy-ses. Several crossover analyses were used to integrate the qualitative and quantitative findings
(Mixed analysis phase). Specifically, each theme was quantitized (i.e., qualitative data were
transformed to numerical codes that could be analyzed quantitatively; Tashakkori & Teddlie,
1998),3 which allowed the frequency of each theme to be calculated. An EFA was conducted
to ascertain the underlying structure of the four themes. This EFA determined the number of fac-
tors underlying the themes (i.e., meta-themes; Onwuegbuzie, 2003) such that each meta-theme
contained one or more of the emergent themes.4 The EFA yielded two meta-themes comprising:
(a) policy and practice (33.00% of explained variance) and (b) school climate and culture, need
for clarity in implementation responsibility, lack of resources as perceived barriers, and limited
knowledge of culturally responsive instructional practices (29.49% of explained variance).
These meta-themes combined explained 62.5% of the total variance.
Phase 9: Validating revised instrument: Mixed analysis phase: Quantitative-dominant crossover anal-yses. Phase 9 involved the conduct of several crossover analyses that addressed the following
research question: What is the relationship between school leaders’ ratings of the SCCOC and
their perceptions of the SCCOC as a tool for assessing schoolwide cultural competence? The
quantitized themes were correlated to the ratings on the Importance Scale via a series of Bonfer-
roni-adjusted chi-square analyses. The quantitized themes also were correlated (i.e., repeated-
measures analysis of variance, canonical correlation analysis) to the factor scores stemming
from the EFA of the 33-item SCCOC in Phase 4 that yielded a two-factor solution. These anal-
yses yielded interesting relationships that were used to inform further the instrument fidelity of
the SCCOC.
Phase 10: Evaluating the instrument development/construct evaluation process and product. Phase
10 involved a comprehensive evaluation both of the product and the process. Analysis of the
product—namely, the SCCOC—revealed that this instrument appears to have better-than-
adequate face validity, item validity, outcome validity, generalizability, and structural validity.
However, although the quantitative analysis strongly suggested that the SCCOC also had better-
than-adequate sampling validity and substantive validity, the qualitative and mixed (i.e., cross-
over) analyses suggested that these areas of validity are somewhat problematic because the
SCCOC does not measure two of the emergent themes: (a) school climate and culture and (b)
barriers to schoolwide cultural competence. This inconsistency (i.e., initiation; Greene et al.,
1989) has led Bustamante et al. (2009) to return to Phase 5: redesigning and field-testing revised
instrument—including the assessment of concurrent validity and divergent validity—even
though they recognized that the SCCOC appears to have excellent psychometric properties
for measuring the policy and practice dimensions. Thus, the IDCV process allows for a process
approach to reexamining instrument fidelity with the goal of continuous improvement and prac-
tical adaptation to a variety of school contexts and changes over time.
Summary and Conclusions
Results from Bustamante et al.’s (2009) mixed research approach to developing the SCCOC
and verifying its fidelity provided more in-depth information than does a traditional quantitative
validity and reliability study alone would have provided.5 Specifically, two additional themes,
Onwuegbuzie et al. 71
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
three subthemes, and two meta-themes were identified through the method of constant compar-
ison of the collected qualitative data and crossover analysis, respectively, which were not iden-
tified via the traditional exploratory factor analysis of responses to the Likert-format scale. The
mixed data collection and analysis elicited richer information than would have quantitative tech-
niques alone regarding the utility of the SCCOC as one tool in assessing schoolwide cultural
competence as part of an overall culture audit and perceptions of its applicability in actual school
settings by practicing school leaders. Moreover, using mixed research as a way to analyze the
fidelity of the SCCOC complements the culture audit process and its intricacies. Thus, as can
be seen, the IDCV process represents a more comprehensive and rigorous technique for devel-
oping a quantitative instrument and for construct validation. Indeed, the IDCV process represents
an extension—specifically, a mixed research-based extension—of Campbell and Fiske’s (1959)
MTMM because it incorporates all the elements of their cross-validation framework. Further-
more, the IDCV combines quantitative and qualitative data collection and data analysis techni-
ques, yielding an expanded cross-validation process via the use of crossover analysis strategies.
The fluidity of the IDCV process also allows researchers and instrument developers to return to
previous phases in the process to examine any needs for further instrument revisions and to focus
on continuously improving the fidelity and score validity of an instrument.
As noted by Hitchcock et al. (2005), ‘‘what is missing from the literature is a discussion of
how to combine qualitative and quantitative methodologies to investigate a key concern in
any social science endeavor, construct validation’’ (p. 260). We believe that our IDCV process
represents a step in the right direction to filling this void. Thus, we hope that researchers consider
using our IDCV process—or a modification of it—when seeking to develop an instrument and/or
validate a construct.
Table 3. Example of Interrespondent Matrix Containing Four SCCOC-Related Themes Usedto Conduct Mixed Analysis
ID Theme 1 Theme 2 Theme 3 Theme 4
001 1 0 1 1002 0 1 1 1003 0 1 0 1... ..
. ... ..
. ...
..
. ... ..
. ... ..
.
151 0 0 0 1
Note: SCCOC ¼ School-Wide Cultural Competence Observation Checklist. If a study participant listed a characteristic
that was eventually categorized under a particular theme, then a score of ‘‘1’’ would be given to the theme for the
participant’s response; a score of ‘‘0’’ would be given otherwise.
Table 4. Example of How to Use the Interrespondent Matrix to Compute Effect Sizes for Four Participants
ID Theme 1 Theme 2 Theme 3 Theme 4 Theme 5 Theme 6 Total %
001 1 0 1 1 1 0 4 66.7002 0 1 0 1 0 1 3 50.0003 0 0 0 1 0 0 1 16.7004 0 1 1 1 1 1 5 83.3
Total 1 2 2 4 2 2 13% 25.0 50.0 50.0 100.0 50.0 50.0
72 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Declaration of Conflicting Interests
The authors declared no potential conflicts of interests with respect to the authorship and/or publication of
this article.
Funding
The authors received no financial support for the research and/or authorship of this article.
Notes
1. As do several authors (e.g., R. B. Johnson & Onwuegbuzie, 2004), we believe that the phrase mixed
research is more appropriate than is the term mixed methods research because the latter might give
the impression that this research paradigm only involves the mixing of methods. Yet this research par-
adigm involves ‘‘mix[ing] or combin[ing] quantitative and qualitative research techniques, methods,
approaches, concepts or language into a single study’’ (R. B. Johnson & Onwuegbuzie, 2004, p. 17).
2. A mixed research framework also is useful for optimizing the development of a qualitative instrument.
However, it is beyond the scope of the present article. For a discussion of this concept, see Collins et al.
(2006). For an example of this, we refer the reader to R. B. Johnson et al. (2007).
3. Table 3 shows how an interrespondent matrix might look for the 151 participants. Table 4 provides an
example, using four participants, of how the interrespondent matrix is used to compute various effect
sizes. Looking at the row totals and percentages, it can be seen from this table that the fourth participant
(i.e., ID 004) provided comments that contributed to the most themes (i.e., 5/6 ¼ 83.3%), with the third
participant (i.e., ID 003) contributing to the least themes (i.e., 1/6 ¼ 16.7%). Examining the column to-
tals reveals that Theme 4 is the most endorsed theme, with all four participants endorsing this theme.
Thus, the manifest effect size for Theme 4 is 100%. Conversely, the manifest effect size for Theme 1,
the least endorsed theme, is 25.0%. Because sample size (n ¼ 151) was large relative to the number
of themes (i.e., 4 themes)—yielding approximately 37 participants per theme—the interrespondent
matrix was subjected to an exploratory factor analysis (i.e., Phase 8), as well as to an array of general
linear model analyses (e.g., canonical correlation analysis; i.e., Phase 9).
4. As undertaken by Onwuegbuzie et al. (2007), the interrespondent matrix was converted to a matrix of
bivariate associations among the responses pertaining to each of the emergent themes. It is recommended
that these bivariate associations are represented by tetrachoric correlation coefficients because they are
appropriate to use when one is determining the relationship between two (artificial) dichotomous vari-
ables. Furthermore, it should be noted that tetrachoric correlation coefficients are based on the assump-
tion that for each manifest dichotomous variable, there is a normally distributed latent continuous
variable with zero mean and unit variance. Furthermore, it was assumed that the extent to which each
participant contributes to a theme, as indicated by the order in which the significant statements are pre-
sented, represent a normally distributed latent continuous variable. Thus, the matrix of tetrachoric cor-
relation coefficients was the basis of the exploratory factor analysis.
5. The first author recently taught a class in Slovenia containing 24 doctoral students representing numerous
disciplines (e.g., education, political science, agriculture, economics) and many nationalities (e.g., German,
Canadian, Hungarian, British), where he introduced the IDCV process. All the students deemed the IDCV
process to represent a more rigorous approach to instrument development and construct validation than
using quantitative techniques alone (e.g., MTMM). Even more compelling was the fact that when students
were required via a final course examination to use the IDCV framework to design the development of an
instrument in their fields, regardless of discipline or field, the students all submitted designs that were both
comprehensive and rigorous, suggesting that the IDCV framework has far-reaching application.
References
Banks, J. (2002). An introduction to multicultural education (3rd ed.). Needham, MA: Allyn & Bacon.
Onwuegbuzie et al. 73
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Banks, J., & McGee Banks, C. A. (2001). Multicultural education: Issues and perspectives (4th ed.). New
York: Wiley.
Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature
review in research preparation. Educational Researcher, 34(6), 3-15.
Brewer, J., & Hunter, A. (2006). Foundations of multimethod research: Synthesizing styles. Thousand
Oaks, CA: SAGE.
Brislin, R. (1993). Understanding culture’s influence on behavior. New York: Harcourt Brace Jovanovich.
Bustamante, R. M. (2005). Features of cultural proficiency in American International Schools in Latin
America: A Delphi study. Unpublished doctoral dissertation, University of San Diego.
Bustamante, R. M. (2006). The ‘‘culture audit’’: A leadership tool for assessment and strategic planning in
diverse schools and colleges. Connexions. Retrieved March 22, 2009, from http://cnx.org/content/
m13691/latest/
Bustamante, R. M., & Nelson, J. A. (2007). The School-Wide Cultural Competence Observation Checklist.
Unpublished manuscript, Sam Houston State University, Huntsville, Texas.
Bustamante, R. M., Nelson, J. A., & Onwuegbuzie, A. J. (2009). Instrument fidelity and construct validity of
the School-wide Cultural Competence Observation Checklist (SCCOC). Manuscript submitted for
publication.
Bustamante, R. M., Nelson, J. A., & Onwuegbuzie, A. J. (in press). Assessing school-wide cultural com-
petence: Implications for school leader preparation. Educational Administration Quarterly.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-
multimethod matrix. Psychological Bulletin, 56, 81-105.
Casmir, F. (1999). Foundations for the study of intercultural communications based on a third-culture build-
ing model. International Journal of Intercultural Relations, 23, 91-116.
Cattell, R. B. (1978). The scientific use of factor analysis in behavioral and life sciences. New York:
Plenum.
Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and pur-
pose for conducting mixed methods research in special education and beyond. Learning Disabilities: A
Contemporary Journal, 4, 67-100.
Combs, J. P., Bustamante, R. M., & Onwuegbuzie, A. J. (in press). A mixed methods approach to conduct-
ing literature reviews for stress and coping researchers: An interactive literature review process frame-
work. In K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward a broader understanding of
stress and coping: Mixed methods approaches. The Research on Stress and Coping in Education Series
(Vol. 5). Greenway, CT: Information Age.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rine-
hart & Winston.
Cross, T. L., Bazron, B. J., Dennis, K. W., & Isaacs, M. R. (1989). Towards a culturally competent system of
care: Vol. 1. A monograph on effective services for minority children who are severely emotionally dis-
turbed. Washington, DC: Georgetown University, Child Development Center, Child and Adolescent
Services System Program, Technical Assistance Center.
Darnell, A. J., & Kuperminc, G. P. (2006). Organizational cultural competence in mental health service
delivery: A multilevel analysis. Journal of Multicultural Counseling and Development, 34, 194-207.
Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach.
Journal of Mixed Methods Research, 2(3), 270-283.
Dick, W., Carey, L., & Carey, J. O. (2004). The systematic design of instruction (6th ed.). Boston: Allyn &
Bacon.
Everitt, B. S. (1975). Multivariate analysis: The need for data, and other problems. British Journal of Psy-
chiatry, 126, 237-240.
Fabrigar, L. R., & Joelle Estrada, M. (2007). Multitrait multimethod matrix and construct validity. In
N. J. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 666-669). Thousand Oaks, CA:
SAGE.
Freiberg, H. J. (1998). Measuring school climate: Let me count the ways. Educational Leadership, 56,
336-343.
Geertz, C. (1979). The interpretation of culture. New York: Basic Books.
74 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative
research. Chicago: Aldine.
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method
evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.
Hall, E. T. (1976). Beyond culture. Garden City, NY: Doubleday.
Halpern, E. S. (1983). Auditing naturalistic inquiries: The development and application of a model. Unpub-
lished doctoral dissertation, Indiana University.
Hammer, M. (1989). Intercultural communication competence. In M. K. Asante & W. Gudykunst (Eds.),
Handbook of international and intercultural communication (pp. 246-260). Newbury Park, CA: SAGE.
Henson, R. K., Capraro, R. M., & Capraro, M. M. (2004). Reporting practice and use of exploratory factor
analyses in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72.
Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research. Educa-
tional and Psychological Measurement, 66, 393-416.
Henze, R., Katz, A., Norte, E., Sather, S., & Walker, E. (2002). Leading for diversity. Thousand Oaks, CA:
Corwin Press.
Hetzel, R. D. (1996). A primer on factor analysis with comments on patterns of practice and reporting. In
B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 175-206). Greenwich, CT:
JAI Press.
Hitchcock, J. H., Nastasi, B. K., Dai, D. C., Newman, J., Jayasena, A., Bernstein-Moore, R., et al. (2005).
Illustrating a mixed-method approach for identifying and validating culturally specific constructs. Jour-
nal of School Psychology, 43, 259-278.
Hitchcock, J. H., Sarkar, S., Nastasi, B. K., Burkholder, G., Varjas, K., & Jayasena, A. (2006). Validating
culture- and gender-specific constructs: A mixed-method approach to advance assessment procedures in
cross-cultural settings. Journal of Applied School Psychology, 22, 13-33.
Hofstede, G. (1980). Cultures consequences: International differences in work-related values. Beverly
Hills, CA: SAGE.
Hofstede, G. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations
across nations. London: SAGE.
Hogarty, K. Y., Hines, C. V., Kromrey, J. D., Ferron, J. M., & Mumford, K. R. (2005). The quality of factor
solutions in exploratory factor analysis: The influence of sample size, communality, and overdetermi-
nation. Educational and Psychological Measurement, 65, 202-226.
Holcomb-McCoy, C. C. (2004). Assessing the multicultural competence of school counselors: A checklist.
Professional School Counseling, 7, 178-183.
House, R. J., Hanges, P. J., Javidan, M., Dorfman, P. W., & Gupta, V. (2004). Culture, leadership, and or-
ganizations: The GLOBE study of 62 societies. London: SAGE.
Johnson, L. S. (2003). The diversity imperative: Building a culturally responsive school ethos. Intercultural
Education, 14(1), 17-30.
Johnson, R. B., Meeker, K., & Onwuegbuzie, A. J. (2004, April). Development and use of the Philosophical
Beliefs Questionnaire. Paper presented at the annual meeting of the American Educational Research
Association, San Diego, CA.
Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time
has come. Educational Researcher, 33(7), 14-26.
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods
research. Journal of Mixed Methods Research, 1(2), 112-133.
Karpicke, H., & Murphy, M. E. (1996). Productive school culture: Principals working from the inside.
NAASP Bulletin, 80(576), 26-34.
Kieffer, K. M. (1999). An introductory primer on the appropriate use of exploratory and confirmatory factor
analysis. Research in the Schools, 6(2), 75-92.
Kluckhohn, F., & Strodbeck, F. (1961). Variations in value orientations. New York: Row Peterson.
Kochan, T., Bezrukova, K., Ely, R., Jackson, S., Joshi, A., Jehn, K., et al. (2003). The effects of diversity on
business performance: Report of the diversity research network. Human Resource Management, 42, 3-21.
Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
Onwuegbuzie et al. 75
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Ladson-Billings, G. (1995). But that’s just good teaching The case for culturally relevant pedagogy. Amer-
ican Educational Research Journal, 32, 159-165.
Leech, N. L., & Onwuegbuzie, A. J. (2007). An array of qualitative data analysis tools: A call for qualitative
data analysis triangulation. School Psychology Quarterly, 22, 557-584.
Leech, N. L., & Onwuegbuzie, A. J. (2008). Qualitative data analysis: A compendium of techniques and
a framework for selection for school psychology research and beyond. School Psychology Quarterly,
23, 587-604.
Lester, F. K. (2005). On the theoretical, conceptual, and philosophical foundations for research in mathe-
matics education. Zentralblatt fur Didaktik der Mathematik, 37, 457-467.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: SAGE.
Lindsey, R., Roberts, L. M., & CampbellJones, F. (2005). The culturally proficient school: An implemen-
tation guide for school leaders. Thousand Oaks, CA: Corwin Press.
Lindsey, R., Robins, K. N., & Terrell, R. (2003). Cultural proficiency: A manual for school leaders (2nd
ed.). Thousand Oaks, CA: Corwin Press.
Lustig, M. W., & Koester, J. (2002). Intercultural competence: Interpersonal communication across cul-
tures (4th ed.). Boston: Longman.
Lynch, E. W., & Hanson, M. J. (1998). Developing cross-cultural competence: A guide for working with
young children and their families (2nd ed.) Baltimore: Paul H. Brookes.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Old
Tappan, NJ: Macmillan.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons responses
and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
Morgan, D. L. (1998). The focus group guidebook. Thousand Oaks, CA: SAGE.
Mundfrom, D. J., Shaw, D. G., & Ke, T. L. (2005). Minimum sample size recommendations for conducting
factor analyses. International Journal of Testing, 5, 159-168.
National Center for Cultural Competence. (2005). Cultural and linguistic competence: Definitions, frame-
works, and implications. Retrieved May 26, 2009, from www.nccccurricula.info/culturalcompetence.
html
Nelson, J. A., Bustamante, R. M., Wilson, E., & Onwuegbuzie, A. J. (2008). The school-wide cultural com-
petence observation checklist for school counselors: An exploratory factor analysis. Professional School
Counseling, 11, 207-217.
Onwuegbuzie, A. J. (2003). Effect sizes in qualitative research: A prolegomenon. Quality & Quantity:
International Journal of Methodology, 37, 393-409.
Onwuegbuzie, A. J., & Collins, K. M. T. (2007). A typology of mixed methods sampling designs in social
science research. The Qualitative Report, 12, 281-316. Retrieved April 7, 2009, from http://www.no-
va.edu/ssss/QR/QR12-2/onwuegbuzie2.pdf
Onwuegbuzie, A. J., Collins, K. M. T., Leech, N. L., Dellinger, A. B., & Jiao, Q. G. (in press). A meta-
framework for conducting mixed research syntheses for stress and coping researchers and beyond. In
K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward a broader understanding of stress
and coping: Mixed methods approaches. The Research on Stress and Coping in Education Series
(Vol. 5). Greenway, CT: Information Age.
Onwuegbuzie, A. J., Collins, K. M. T., & Leech, N. L. (in press). Mixed research: A step-by-step guide.
New York: Taylor & Francis.
Onwuegbuzie, A. J., & Combs, J. P. (in press). Emergent data analysis techniques in mixed methods
research: A synthesis. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social
and behavioral research (2nd ed.). Thousand Oaks, CA: SAGE.
Onwuegbuzie, A. J., & Daniel, L. G. (2003, February 12). Typology of analytical and interpretational errors
in quantitative and qualitative educational research. Current Issues in Education [On-line], 6(2).
Retrieved April 5, 2009, from http://cie.ed.asu.edu/volume6/number2/
Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. T. (2009). A meta-validation model for assessing the
score validity of student teacher evaluations. Quality & Quantity: International Journal of Methodology,
43, 197-209.
76 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Onwuegbuzie, A. J., Johnson, R. B., & Collins, K. M. T. (2009, April). Toward a philosophy of mixed data
analysis. Paper presented at the annual meeting of the American Educational Research Association, San
Diego, CA.
Onwuegbuzie, A. J., Leech, N. L., & Collins, K. M. T. (2008). Interviewing the interpretive researcher: A
method for addressing the crises of representation, legitimation, and praxis. International Journal of
Qualitative Methods, 7, 1-17.
Onwuegbuzie, A. J., Leech, N. L., Dickinson, W. B., & Zoran, A. G. (in press). Toward more rigor in focus
group research in stress and coping and beyond: A new mixed research framework for collecting and
analyzing focus group data. In K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward
a broader understanding of stress and coping: Mixed methods approaches. The Research on Stress
and Coping in Education Series (Vol. 5). Greenway, CT: Information Age.
Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In
A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp.
351-383). Thousand Oaks, CA: SAGE.
Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M. T., Filer, J. D., Wiedmaier, C. D., & Moore, C. W.
(2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teach-
ing evaluation form using a mixed-methods analysis. American Educational Research Journal, 44, 113-
160.
Powell, H., Mihalas, S., Onwuegbuzie, A. J., Suldo, S., & Daley, C. E. (2008). Mixed methods research in
school psychology: A mixed methods investigation of trends in the literature. Psychology in the Schools,
45, 291-309.
Riehl, C. J. (2000). The principal’s role in creating inclusive schools for diverse students: A review of nor-
mative, empirical, and critical literature on the practice of educational administration. Review of Edu-
cational Research, 70, 55-81.
Salvaggio, K. C. (2003). Perceptions of formal and informal school leaders regarding cultural proficiency
in high-achieving, highly diverse elementary schools. Unpublished doctoral dissertation, University of
LaVerne.
Schein, E. (1992). Organizational culture and leadership (2nd ed.). San Francisco: Jossey-Bass.
Skrla, L., Scheurich, J. J., Garcia, J., & Nolly, G. (2006). Equity audits: A practical leadership tool for
developing equitable and excellent schools. Educational Administration Quarterly, 40, 135-163.
Spradley, J. P. (1979). The ethnographic interview. Fort Worth, TX: Holt, Rinehart & Winston.
Stufflebeam, D. L. (2002). CIPP evaluation model checklist: Fifth installment of the CIPP model. Kalama-
zoo, MI: The Evaluation Center. Retrieved February 4, 2009, from http://www.wmich.edu/evalctr/
checklists/cippchecklist.htm
Sue, D. W. (2001). Multidimensional facets of multicultural competence. The Counseling Psychologist, 29,
790-821.
Sue, D. W., & Constantine, M. G. (2005). Effective multicultural consultation and organizational develop-
ment. In M. G. Constantine & D. W. Sue (Eds.), Strategies for building multicultural competence in
mental health settings (pp. 212-226). Hoboken, NJ: Wiley.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative ap-
proaches. Applied Social Research Methods Series (Vol. 46). Thousand Oaks, CA: SAGE.
Teddlie, C., & Johnson, R. B. (2009). Methodological thought since the 20th century. In C. Teddlie & A
Tashakkori (Eds.), Foundations of mixed methods research: Integrating quantitative and qualitative
techniques in the social and behavioral sciences (pp. 62-82). Thousand Oaks, CA: SAGE.
Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applica-
tions. Washington, DC: American Psychological Association.
Ting-Toomey, S., & Oetzel, J. (2001). Managing intercultural conflict effectively. Thousand Oaks, CA:
SAGE.
Tinsley, H. E. A., & Tinsley, D. J. (1987). Uses of factor analysis in counseling psychology research. Jour-
nal of Counseling Psychology, 34, 414-424.
Triandis, H. (1994). Culture and social behavior. New York: McGraw-Hill.
Trompenaars, F., & Hampden-Turner, C. (1998). Riding the waves of culture: Understanding cultural
diversity in global business (2nd ed.). New York: McGraw-Hill.
Onwuegbuzie et al. 77
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from
Van Maanen, J. (1979). The fact of fiction in organizational ethnography. Administrative Science Quar-
terly, 24, 249-550.
Wiseman, R., & Koester, J. (1993). Intercultural communication competence. Newbury Park, CA: SAGE.
78 Journal of Mixed Methods Research 4(1)
at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from