mixed research as a tool for developing quantitative

Mixed Research as a Tool forDeveloping QuantitativeInstruments

Anthony J. Onwuegbuzie,1 Rebecca M. Bustamante,1

and Judith A. Nelson1

Abstract

In this methodological article, the authors present a meta-framework—comprising severalframeworks and models, as well as multiple research approaches—that they call an InstrumentDevelopment and Construct Validation (IDCV) process for optimizing the development of quantita-tive instruments. Using mixed research techniques, the IDCV contains 10 phases that detail theprogression from an interdisciplinary review of the literature to the development of the instru-ment to the evaluation of the instrument development and construct validation process andproduct(s). Crossover analyses represent a key mechanism in the IDCV, wherein analysis typesfrom one tradition (e.g., quantitative analysis) are used to analyze data from a different tradition(e.g., qualitative data). Finally, the authors provide a heuristic example of a rigorous way todevelop a Likert-format scale.

Keywords

mixed research, instrument development, construct validation

Researchers have been mixing qualitative and quantitative approaches for decades (Tashakkori

& Teddlie, 1998; Teddlie & Johnson, 2009). Nevertheless, mixed methods research, also known as

mixed research1—wherein ‘‘a researcher or team of researchers combines elements of qualitative

and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data col-

lection, analysis, inference techniques) for the broad purposes of breadth and depth of understand-

ing and corroboration’’ (R. B. Johnson, Onwuegbuzie, & Turner, 2007, p. 123)—represents ‘‘a

new movement, or discourse, or research paradigm (with a growing number of members) that

has arisen in response to the currents of quantitative research and qualitative research’’ (R. B.

Johnson et al., 2007, p. 113). As noted by Denscombe (2008),

Mixed methods research has developed rapidly in recent years. Championed by writers

such as John Creswell, Abbas Tashakkori, Burke Johnson, Anthony Onwuegbuzie,

1Sam Houston State University, Huntsville, TX, USA

Corresponding Author:

Anthony J. Onwuegbuzie, Department of Educational Leadership and Counseling, Box 2119, Sam Houston State

University, Huntsville, TX 77341-2119

Email: [email protected]

Journal of Mixed Methods Research4(1) 56–78

ªThe Author(s) 2010Reprints and permission: http://www.sagepub.com/journalsPermissions.nav

DOI: 10.1177/1558689809355805http://jmmr.sagepub.com

at Universitaet St Gallen on April 14, 2016mmr.sagepub.comDownloaded from

http://mmr.sagepub.com/

Jennifer Greene, Charles Teddlie, and David Morgan, the mixed methods approach has

emerged in the last decade as a research movement with a recognized name and distinct

identity. It has evolved to the point where it is ‘‘increasingly articulated, attached to

research practice, and recognized as the third major research approach or research para-

digm’’ (Johnson, Onwuegbuzie, & Turner, 2007, p. 112). (p. 270)

Although mixed research has become popularized, its potential has not yet been fully realized.

Indeed, many researchers do not mix qualitative and quantitative approaches in optimal ways

(see Powell, Mihalas, Onwuegbuzie, Suldo, & Daley, 2008). In an attempt to provide researchers

with a framework for optimizing their mixed research designs, Collins, Onwuegbuzie, and Sut-

ton (2006), in what they called a rationale and purpose model for designing mixed research stud-

ies, conceptualized four rationales for mixing approaches: participant enrichment (i.e., the

mixing of quantitative and qualitative techniques to optimize the sample, such as increasing

the sample size), instrument fidelity (i.e., maximizing the appropriateness and/or utility of the

instruments used, whether quantitative or qualitative), treatment integrity (i.e., mixing quantita-

tive and qualitative techniques to assess the fidelity of interventions, treatments, or programs),

and significance enhancement (i.e., mixing quantitative and qualitative techniques to maximize

researchers’ interpretations of data).

Of the four rationales for mixing qualitative and qualitative approaches, instrument fidelity

most lacks adequate development. Indeed, with very few exceptions (e.g., Collins et al., 2006;

Hitchcock et al., 2005, 2006), scant guidance has been given to help researchers use mixed

research techniques to optimize the development of either qualitative or quantitative instruments.

This lack of guidance likely stems from the perception that the development of instruments oc-

curs within the same methodological tradition, with the exclusive use of qualitative approaches

deemed as being appropriate to develop qualitative instruments (e.g., interview schedules) and

the exclusive use of quantitative approaches deemed as being appropriate to develop quantitative

instruments (e.g., Likert-format scales). For example, with respect to quantitative instrument

development, in their seminal article, Campbell and Fiske (1959) proposed a comprehensive

quantitative framework—which they called a multitrait–multimethod matrix (MTMM) for

assessing construct-related validity using two validity types: convergent validity and discrimi-

nant validity. More specifically, the MTMM was designed

to provide a framework to assess the effects of trait variance (variance attributed to the

intended construct of interest) and method variance (variance attributable to the specific

method of measurement) by examining convergent and discriminant validity. More gener-

ally, the MTMM also provides information about patterns of associations between meth-

ods, and patterns of associations between constructs and possible interactions between

methods and constructs. (Fabrigar & Joelle Estrada, 2007, p. 666)

As noted by R. B. Johnson et al. (2007, pp. 113-114), Campbell and Fiske’s quantitative cross-

validation framework is viewed by many mixed methodologists as ‘‘formalizing the practice of

using multiple research methods . . . for validation purposes,’’ and thereby providing the impetus

for the concept of triangulation. However, as innovative and useful as the MTMM has been as

a quantitative cross-validation tool, it does have some limitations (for a discussion of limitations

of the MTMM, see, e.g., Fabrigar & Joelle Estrada, 2007; see also Teddlie & Johnson, 2009). In

particular, Brewer and Hunter (2006) concluded that MTMM ‘‘warned of over-reliance upon,

and overconfidence in, any single type of research methods’’ (p. xiii). That is, the key assumption

underlying the MTMM is that quantitative techniques are sufficient for the development of quan-

titative instruments. Yet qualitative techniques can be used to enhance the development of quan-

titative instruments and vice versa (Collins et al., 2006). Thus, clearly, more publications are

Onwuegbuzie et al. 57



needed that outline explicitly ways of optimizing the development of instruments by mixing

qualitative and quantitative techniques.

Therefore, the purpose of the present methodological article is to provide a mixed research

framework for optimizing the development of a quantitative instrument.2 First, we discuss the

concept of crossover analyses, which involves using one or more analysis types associated

with one tradition (e.g., quantitative analysis) to analyze data associated with a different tradition

(e.g., qualitative data). Second, we deconstruct the concept of instrument fidelity as it pertains to

the construction of quantitative instruments. In particular, we present a framework for develop-

ing and assessing the fidelity of a quantitative instrument via what is termed as areas of validity

evidence and specify the types of crossover analyses that are pertinent for each of the multiple

areas of validity evidence. Third, we describe the 10-phase process for optimizing the develop-

ment of a quantitative instrument. We outline how qualitative and quantitative approaches can be

combined to enhance instrument fidelity at different stages of the development process. Finally,

we provide a brief heuristic example of a rigorous way to develop a Likert-format scale.

Crossover Analyses

Onwuegbuzie and Combs (in press) outlined the concept of crossover analyses. According to

these methodologists, crossover analyses represent the highest form of combining quantitative

and qualitative data analysis techniques—referred to as mixed analysis (Onwuegbuzie, Collins,

& Leech, in press)—because the researcher often has to make Gestalt switches (Kuhn, 1962)

from a qualitative lens to a quantitative lens and vice versa, going back and forth several times

until maximal meaning has been extracted from the data. Moreover, crossover analyses neces-

sitate the researcher to mix or combine a strong use of quantitative and qualitative assumptions

and stances. For example, a researcher might blend a constructivist (analytical) stance (i.e., with

an ontology that multiple contradictory, but equally valid accounts of the same phenomenon can

prevail that represent multiple realities) with a postpositivist (analytical) stance (i.e., with an

ontology that social science research should be objective) by, say, using exploratory factor anal-

ysis to examine the structure of themes that emerged from a qualitative analysis (see Onwueg-

buzie, 2003) as a means of furthering construct validation. Such a strategy would yield

a sequential qualitative–quantitative mixed analysis (Onwuegbuzie & Teddlie, 2003), with the

quantitative analysis phase informing or expanding on the qualitative analyses—or what Greene,

Caracelli, and Graham (1989) refer to as complementarity (i.e., seeking elaboration, illustration,

enhancement, and clarification of the findings from one method with results from the other

method), development (i.e., using the findings from one method to help inform the other

method), or expansion (i.e., expanding the breadth and range of a study by using multiple meth-

ods for different study phases). Alternatively, a researcher might use a thematic analysis to com-

pare emergent themes to factors emerging from an exploratory factor analysis. This analysis

would yield a concurrent mixed analysis with a rationale of triangulation (i.e., comparing find-

ings from quantitative data with qualitative results in hopes of convergence; Greene et al., 1989).

Such analyses have the potential to yield stronger meta-inferences (i.e., involving combining

interpretations of findings stemming from the qualitative and quantitative data analyses into

a coherent whole).

According to Onwuegbuzie and Combs (in press), crossover analyses comprise the following

techniques: integrated data reduction (i.e., reducing the dimensionality of qualitative data/

findings using quantitative analysis and/or quantitative data/findings using quantitative analy-

sis), integrated data display (i.e., visually presenting both qualitative and quantitative results

within the same display), data transformation (i.e., converting quantitative data into data that

can be analyzed qualitatively and/or qualitative data into numerical codes that can be analyzed

58 Journal of Mixed Methods Research 4(1)



statistically), data correlation (i.e., correlating qualitative data with quantitized/quantitative data

and/or quantitative data with qualitized/qualitative data), data consolidation (i.e., combining or

merging multiple data sets to create new or consolidated codes, variables, or data sets), data com-

parison (i.e., comparing qualitative and quantitative data/findings), data integration (i.e., inte-

grating qualitative and quantitative data/findings either into a coherent whole or two separate

sets of coherent wholes), warranted assertion analysis (i.e., reviewing all qualitative and quan-

titative data to yield meta-inferences), and data importation (i.e., using follow-up findings

from qualitative analysis to inform the quantitative analysis or vice versa). All these crossover

analysis techniques involve some level of abductive logic, which involves moving back and forth

between inductive and deductive logic. These crossover analysis procedures also involve a form

of intersubjectivity (i.e., agreement about reality, ultimately, is socially constructed). Further-

more, these analyses involve the incorporation of both insiders’ (i.e., emic) views and the

researcher-observer’s (i.e., etic) views for instrument development and construct validation

and that the balance between the emic perspectives (i.e., stemming from the participants

involved in the instrument development and/or construct validation) and etic perspectives

(e.g., stemming from extant theories and the researcher’s a priori assumptions) is appropriate

such that quality meta-inferences can be made. This use of abductive logic, intersubjectivity,

and emic–etic perspectives makes the use of mixed research in general and crossover analyses

in particular very appealing for instrument development and construct validation.

Fidelity of Quantitative Instruments

Instrument fidelity includes steps taken by the researcher to develop an instrument and to

maximize its appropriateness and/or utility. Onwuegbuzie, Daniel, and Collins (2009) have pro-

vided a useful framework that can be used both to develop and assess the fidelity of quantitative

instruments. These authors extended Messick’s (1989, 1995) theory of validity by combining his

conceptualization with the traditional notion of validity, thereby yielding a reconceptualization

of validity as presented in Table 1, which they called a meta-validation model. Although viewed

as a unitary concept, Table 1 indicates that content-, criterion-, and construct-related validity can

be subdivided into several areas of evidence. Table 2 delineates the types of crossover analyses

that can be conducted for each validity type. It can be seen from this table that every type of

crossover analysis appears at least once, with data comparison, data integration, and warranted

assertion each appearing on six occasions. That is, these analyses each can be used in the assess-

ment of six validity types.

In this article, we will show that although most of these validity types lend themselves to

quantitative approaches, they can also be addressed via qualitative approaches. For example,

whereas structural validity (see Tables 1 and 2) typically has been addressed using quantitative

analysis of quantitative data (e.g., exploratory factor analysis; see Crocker & Algina, 1986), we

will show how qualitative analysis techniques also can be used to assess structural validity—

yielding a mixed analysis meta-framework for instrument development/fidelity and construct

validation.

Meta-Framework for Instrument Development/Fidelityand Construct Validation

Figure 1 provides an overview of our meta-framework for instrument development/fidelity

and construct validation. This mixed research–based meta-framework was designed to help

instrument developers undergo a rigorous and comprehensive process during instrument




development/construct validation. We call our framework the Instrument Development and

Construct Validation (IDCV). This framework consists of 10 interactive phases, as shown in

Figure 1. The 10 phases are the following:

1. Conceptualize the construct of interest

2. Identify and describe behaviors that underlie the construct

3. Develop initial instrument

4. Pilot-test initial instrument

5. Design and field-test revised instrument

Table 1. Areas of Validity Evidence

Validity Type Description

Criterion-relatedConcurrent validity Assesses the extent to which scores on an instrument are related to

scores on another, already established instrument administeredapproximately simultaneously or to a measurement of some othercriterion that is available at the same point in time as the scoreson the instrument of interest

Predictive validity Assesses the extent to which scores on an instrument are related to scoreson another, already established instrument administered in the future orto a measurement of some other criterion that is available at a future pointin time as the scores on the instrument of interest

Content-relatedFace validity Assesses the extent to which the items appear relevant, important,

and interesting to the respondentItem validity Assesses the extent to which the specific items represent measurement

in the intended content areaSampling validity Assesses the extent to which the full set of items sample the total

content areaConstruct-related

Substantive validity Assesses evidence regarding the theoretical and empirical analysisof the knowledge, skills, and processes hypothesized to underlierespondents’ scores

Structural validity Assesses how well the scoring structure of the instrument correspondsto the construct domain

Convergent validity Assesses the extent to which scores yielded from the instrument ofinterest being highly correlated with scores from other instrumentsthat measure the same construct

Discriminant validity Assesses the extent to which scores generated from the instrument ofinterest being slightly but not significantly related to scores frominstruments that measure concepts theoretically and empirically relatedto but not the same as the construct of interest

Divergent validity Assesses the extent to which scores yielded from the instrument ofinterest not being correlated with measures of constructs antitheticalto the construct of interest

Outcome validity Assesses the meaning of scores and the intended and unintendedconsequences of using the instrument

Generalizability Assesses the extent that meaning and use associated with a set of scorescan be generalized to other populations

Source: Reproduced from Collins et al. (2006). Reprinted with kind permission of the Learning DisabilitiesWorldwide.




6. Validate revised instrument: Quantitative analysis phase

7. Validate revised instrument: Qualitative analysis phase

8. Validate revised instrument: Mixed analysis phase: Qualitative-dominant crossover

analyses

9. Validate revised instrument: Mixed analysis phase: Quantitative-dominant crossover

analyses

10. Evaluate the instrument development/construct evaluation process and product

These 10 stages are blended, allowing for movement among the stages as new findings

emerge. A constant reflection component encourages the researcher to evaluate progress related

to each phase and to identify and address experienced emotions.

Framework for Debriefing the Instrument Developer

In addition to using Onwuegbuzie et al.’s (2009) meta-validation framework, Greene et al.’s

(1989) rationale framework for combining qualitative and quantitative data, and Onwuegbuzie

and Combs’s (in press) typology of cross-analysis strategies at their core, the meta-framework

for instrument development/fidelity and construct validation incorporates Onwuegbuzie, Leech,

and Collins’s (2008) framework for debriefing the researcher. Onwuegbuzie et al. (2008)

recently introduced the concept of debriefing the researcher, wherein the researcher is inter-

viewed by an individual who is not involved directly in the study (e.g., disinterested peer) but

who understands the research construct or topic that is being studied. Onwuegbuzie et al.

(2008) contend that debriefing interview data helps the researcher to evaluate the decisions

made at the various stages of the research process, as well as to reflect on assumptions, biases,

feelings, and perceptions that were present at the beginning of the study and that evolved as the

study progressed. According to Onwuegbuzie et al. (2008), having the researcher explicitly

reveal these elements to the debriefing interviewer helps increase the researcher’s understanding

of the research process as it unfolds, as well as provides an audit trail. We extend this concept of

debriefing to the process of instrument developer/construct validation. In particular, we recom-

mend that the instrument developer(s) be interviewed at all 10 phases of the IDCV process by

Table 2. Crossover Analysis Strategies by Validity Type

Validity Type Type of Crossover Analysis

Criterion relatedConcurrent validity Data correlation, data comparisonPredictive validity Data correlation, data comparison

Content relatedFace validity Integrated data display, warranted assertion, data integrationItem validity Data comparison, warranted assertion, data integrationSampling validity Integrated data display, data comparison, warranted assertion, data integration

Construct relatedSubstantive validity Data consolidation, data comparison, warranted assertion, data integrationStructural validity Integrated data reduction, data transformationConvergent validity Data correlation, data comparisonDiscriminant validity Data correlation, data comparisonDivergent validity Data correlation, data comparisonOutcome validity Data transformation, data consolidation, warranted assertion, data integrationGeneralizability Warranted assertion, data integration, data importation




one or more disinterested peers who have expertise in measurement theory and instrument

development.

Phase 1: Conceptualize the Construct of Interest

Phase 1 involves development of the construct of interest. As recommended by Combs, Bus-

tamante, and Onwuegbuzie (in press), an important early goal is for researchers—instrument

developers in this case—to be aware of their own personal belief systems related to three dimen-

sions of belief systems: (a) overall worldview, (b) research philosophy, and (c) discipline-spe-

cific philosophy. Indeed, researchers should strive to understand the role that these three

dimensions have on the instrument development/construct validation process. First, as noted

Phase 2: Identify and Describe Behaviors that Underlie the Construct

Phase 4: Field-test Initial Instrument

Phase 5: Design and Field-test Revised Instrument

Phase 6: Validate Revised Instrument: Quantitative Analysis Phase

Phase 7: Validate Revised Instrument: Qualitative Analysis Phase

Phase 8: Validate Revised Instrument: Mixed Analysis Phase: Qualitative Dominant Cross-Over Analyses

Phase 9: Validate Revised Instrument: Mixed Analysis Phase: Quantitative Dominant Cross-Over Analyses

Phase 10: Evaluating the Instrument Development/Construct Evaluation Process and Product

Phase 1: Conceptualize the Construct of Interest

Phase 3: Develop Initial Instrument

Figure 1. Instrument development and construct validation (IDCV) process




by R. B. Johnson, Meeker, and Onwuegbuzie (2004), it is essential that researchers are cognizant

of their personal worldviews, namely, the lens through which one sees and interprets the world.

Second, with respect to research philosophy, Onwuegbuzie, Johnson, and Collins (2009) have

provided a framework for examining one’s own research belief systems, particularly as they per-

tain to conducting qualitative, quantitative, and mixed analyses. According to these authors,

many researchers do not realize that their research philosophy (e.g., postpositivism, constructiv-

ism, critical theory) does not prevent them from conducting analyses that are traditionally asso-

ciated with another approach. For example, having a postpositivist orientation does not prevent

a researcher from conducting an array of qualitative analyses. This point is particularly important

because it suggests that the development of a quantitative instrument—traditionally considered

to be an activity that belongs to the postpositivist philosophical stance—can involve both quan-

titative and qualitative analyses. Third, Combs et al. (in press) recommend that instrumental de-

velopers identify their own discipline-specific philosophies. In the field of instrument

development, discipline-specific philosophies could be reframed as construct-specific philoso-

phies. For example, if a researcher was interested in developing a measure of statistics anxiety,

then he/she should explore his beliefs systems and biases surrounding this construct.

One of the primary ways of developing the construct of interest is by conducting an extensive

review of the literature. As part of the literature review, the instrument developer should identify

the relevant theoretical framework(s) (i.e., ‘‘developed by using an established, coherent expla-

nation of certain sorts of phenomena and relationships’’; Lester, 2005, p. 458) and/or conceptual

framework(s) (i.e., ‘‘an argument that the concepts chosen for investigation, and any anticipated

relationships among them, will be appropriate and useful given the research problem under

investigation’’; Lester, 2005, p. 460) from the extant literature (i.e., etic perspectives). A few

mixed research–based models recently have been developed to help instrument developers con-

duct rigorous literature reviews (Combs et al., in press; Onwuegbuzie, Collins, Leech, Dellinger,

& Jiao, in press).

In addition, the instrument developer should consult with a diverse set of local experts (i.e.,

emic perspectives). Where possible, the instrument developer should set up focus group meet-

ings of these local experts to capitalize on the group dynamics and social processes that occur

in focus groups (e.g., Morgan, 1998). We recommend that mixed research techniques are used

to collect and analyze the focus group data (see Onwuegbuzie, Leech, Dickinson, & Zoran, in

press). Also, it is important for the instrument developer to ensure that the voices of key inform-

ants, which include those on whom the instrument will be administered, are heard, with a view to

understanding their cultural milieux. Individual interviews, focus groups, and direct observations

can play an important role here. It is essential that detailed field notes are kept and that an audit

trail is left (see Halpern, 1983). A key goal in Phase 1 is the development of an instrument that

possesses cultural sensitivity (Banks & McGee Banks, 2001), so that, when developed, it will

yield data that are optimally reliable and valid.

Phase 2: Identify and Describe Behaviors That Underlie the Construct

The information from the literature review and the various data from the local experts and key

informants could be used to prepare the instrument developer for Phase 2. In Phase 2, the instru-

ment developer might undergo the grounded theory analytical steps of open coding (i.e., coding

the literature and data extracted from the local experts and key informants by chunking the infor-

mation into smaller segments and then assigning a descriptor, or ‘‘code,’’ for each segment) and

axial coding (i.e., grouping the codes into similar categories; Glaser & Strauss, 1967). Alterna-

tively, some form of ethnographic analysis could be undertaken, comprising Spradley’s (1979)

analysis procedures of domain analysis (i.e., using the relationships between symbols and




referents to identify domains), taxonomic analysis (i.e., creating a system of classification that

catalogs the domains into a visual representation to help the researcher understand the relation-

ships among the domains), and componential analysis (i.e., using a visual representation to dis-

cover the differences among the subcomponents of domains). Alternatively still, procedures such

as the Delphi technique might be used. In any case, whatever data collection and data analysis

tools are used, it is essential that a series of rounds is conducted, with each round involving the

collection of qualitative and quantitative data (e.g., interview data, focus group data, observa-

tional data) that are increasingly focused until data saturation is reached (i.e., no new or relevant

information seems to emerge relating to a category, and the category development is well con-

ceptualized and validated; Lincoln & Guba, 1985). Once data saturation is reached, the instru-

ment developer should be in a position to identify the behaviors underlying the construct of

interest. If data saturation has not been reached, the instrument developer might return to Phase

1 to reconceptualize the theory and/or hypotheses, reevaluate any philosophical and research

assumptions held by the instrument developers, and/or collect additional data from the local

experts and key informants.

Phase 3: Develop Initial Instrument

Once all the behaviors have been identified, the instrument developer should be in a position

to start writing items. These items might be written using some type of table of specifications that

links the theory extracted from Phase 1 (etic viewpoint, deductive logic) and the information pro-

vided by the local experts and key informants in Phase 1 and Phase 2 (emic viewpoints, inductive

logic). Such a table of specifications would contain all the identified behaviors (e.g., cognitive,

affective, psychomotor). Where possible, the team of local experts and key informants should

participate in the item-writing process or, at the very least, should be asked to provide feedback

on the items to ensure that they are culturally sensitive, being informed by both emic and etic

perspectives. Although a quantitative instrument is being developed, it is essential that each

item is accompanied by open-ended items that ask the field-testing respondents to assess the

quality of each item and to offer suggestions for improvement. If the feedback given suggests

that the instrument is extremely underdeveloped, then the instrument developer might return

to Phase 2 to reidentify and reconceptualize the behaviors that underlie the construct by collect-

ing additional data from the local experts and/or key informants.

Phase 4: Pilot-Test Initial Instrument

Once the initial instrument has been developed, it is necessary that it is subjected to a field

test. This field test optimally would represent a mixed research study to assess the appropriate-

ness of each item. In particular, each item should be assessed for clarity, esthetics, relevancy,

tone, length of time needed for a response, and, above all, cultural competence. Because sample

sizes at this stage are likely to be smaller than those at the next phase (i.e., Phase 5) when a more

developed version is field-tested, caution typically should be exercised when computing and in-

terpreting score reliability and score validity coefficients. Rather, the focus at this phase should

be more on the content-related validity (i.e., face validity, sampling validity, item validity) and

two elements of construct-related validity (i.e., outcome validity, generalizability) of the initial

instrument (see Table 1) than on the other areas of validity. It is important to note that additional

data beyond the actual instrument itself might be collected. For example, additional individual

interviews and/or focus group interviews might be conducted involving the same or different

local experts or key informants. If the feedback given suggests that the instrument grossly lacks

content-related validity (e.g., the pilot-test participants conclude that the items do not sample the




total content area and thus has low sampling validity) or lacks cultural relevancy, then the instru-

ment developer might return to Phase 3 to redevelop the instrument.

Phase 5: Design and Field-Test Revised Instrument

After the instrument developer has received qualitative and quantitative data on the initial

instrument, he/she will be in a position to refine or discard problematic items. Once the

initial instrument has been revised according to the responses and feedback provided by the initial

field-test sample, it is then sent out for a more extensive field test. Here, the sample size should be

large enough to justify conducting an exploratory factor analysis and yields adequate (sub)scale

score reliabilities with relatively narrow confidence intervals. Thus, sample size guidelines for

exploratory factor analyses should be followed such as those recommended by researchers who

espouse the importance of an adequate case-to-variable ratio from 5 participants per item on the

quantitative instrument (e.g., Likert-format scale, rating scale) as the bare minimum to at least

10 participants per item (Cattell, 1978; Everitt, 1975). (For excellent discussion of the role of sam-

ple size in exploratory factor analysis, see Hogarty, Hines, Kromrey, Ferron, & Mumford, 2005;

Mundfrom, Shaw, & Ke, 2005.)

In traditional quantitative instrument development, the researcher(s) designs quantitative

items (e.g., Likert-format items, rating scale items) and then field-tests these items. However,

in our IDCV framework, we believe that this standard technique is inadequate because it leads

to the sole reliance of quantitative data to assess the psychometric properties of the instrument.

Rather, we recommend strongly that qualitative-based (i.e., open-ended) items be included with

the quantitative items. (Alternatively, administering the quantitative instrument could be accom-

panied by other forms of qualitative data collection such as individual interviews and focus group

interviews on all or some of the people who completed the quantitative instrument.) For example,

in developing a Likert-format measure of statistics anxiety, we would recommend that the

researcher also collect qualitative data such as by asking participants to describe situations in

which they experience elevated levels of statistics anxiety. These qualitative data then could

be correlated and/or compared with the quantitative responses, not only as a means of enhancing

instrument development but also as a way of validating the construct. However, the biggest role

that the collection of qualitative data can play is eliciting information from participants regarding

their perceptions of the cultural relevance of the underlying quantitative instrument. In any case,

this practice of eliciting qualitative responses in addition to quantitative responses from the same

group of field-test participants would lead to a mixed research sampling design that Onwuegbu-

zie and Collins (2007) referred to as a concurrent design using identical samples.

Phase 6: Validate Revised Instrument: Quantitative Analysis Phase

After collecting the quantitative and qualitative responses, the next phase is to analyze the

quantitative (e.g., Likert-form) data. Here, the major goal is to assess the content-related validity

(i.e., item validity), criterion-related validity (i.e., concurrent validity, predictive validity), and

construct-related validity (i.e., structural validity, convergent validity, discriminant validity,

divergent validity) of the scale. For example, with respect to item validity, different item analysis

indices can be used depending on whether the quantitative measure pertains to the cognitive,

affective, or psychomotor domain. If the cognitive domain is of interest, then item developers

can determine, analyze, and interpret numerous indices (e.g., item difficulty, item discrimina-

tion, point biserial correlation, biserial correlation, difficulty level of an average item, phi coef-

ficient, tetrachoric correlation coefficient, item reliability index, item validity index,

instructional sensitivity, indices of agreement, differential item functioning indices; see Crocker




& Algina, 1986). Item developers could decide whether to use classical test theory or item

response theory. With respect to the cognitive domain, concurrent validity and predictive valid-

ity could be assessed via correlational indices, as can the construct-related validity areas of con-

vergent validity, discriminant validity, and divergent validity. Finally, structural validity can be

assessed via an exploratory factor analysis.

With respect to the affective domain, most of the indices that are pertinent for the cognitive

domain also can be used for the affective domain. However, it is exploratory factor analysis that

is a key analysis in Phase 6. For excellent information as to how to conduct exploratory factor

analysis in a rigorous way, we refer the reader to Henson, Capraro, and Capraro (2004), Henson

and Roberts, 2006; Hetzel (1996), Kieffer (1999), Onwuegbuzie and Daniel (2003), Thompson

(2004), and Tinsley and Tinsley (1987).

Phase 7: Validate Revised Instrument: Qualitative Analysis Phase

Phase 7, the next phase, involves the analysis of the qualitative data. Here, the major goal is to

address one or more of Greene et al.’s (1989) five purposes for mixing quantitative and qualita-

tive data (i.e., triangulation, complementarity, development, initiation, and expansion). When

analyzing the qualitative data, instrument developers have many qualitative data analysis tools

at their disposal, including the following: method of constant comparison analysis, keywords-

in-context, word count, classical content analysis, domain analysis, taxonomic analysis, compo-

nential analysis, conversation analysis, discourse analysis, secondary analysis, membership cat-

egorization analysis, narrative analysis, qualitative comparative analysis, semiotics, manifest

content analysis, latent content analysis, text mining, and micro-interlocutor analysis. For a com-

prehensive review of these qualitative data analysis techniques, we refer the reader to Leech and

Onwuegbuzie (2007, 2008).

Phase 8: Validate Revised Instrument: Mixed Analysis Phase:Qualitative-Dominant Crossover Analyses

Phase 8 involves conducting one or more of the nine crossover analyses (i.e., integrated data

reduction, integrated data display, data transformation, data correlation, data consolidation, data

comparison, data integration, warranted assertion analysis, and data importation) to address even

further one or more of Greene et al.’s (1989) five purposes for mixing quantitative and qualitative

data. This phase either involves conducting a qualitative analysis of the quantitative data (i.e.,

scores from quantitative instrument) obtained in Phase 6 or conducting a quantitative analysis

of the qualitative data extracted in Phase 7. With respect to the former, quantitative data can

be transformed to qualitative data via narrative profile formation, wherein narrative descriptions

are constructed from quantitative data. These profiles include the following: modal profiles (i.e.,

detailed narrative descriptions of a group of people based on the most frequently occurring at-

tributes in the group), average profiles (i.e., profiles based on the average of a number of attrib-

utes of the individuals or situations), holistic profiles (i.e., overall impressions of the investigator

regarding the unit of investigation), comparative profiles (i.e., obtained by comparing one unit of

analysis with another, and includes possible similarities/differences between them), and norma-

tive profiles (i.e., similar to narrative profiles but are based on the comparison of an individual or

group with a standard (e.g., normative group) (see Tashakkori & Teddlie, 1998).

In terms of conducting a quantitative analysis of the qualitative data extracted in Phase 7,

numerous strategies exist (see Onwuegbuzie, Collins, & Leech, in press). However, perhaps

the most useful technique is factor-analyzing the themes that emerged in Phase 7. Onwuegbuzie




(2003) originally proposed the concept of factor-analyzing emergent themes. According to On-

wuegbuzie (2003), a score of ‘‘1’’ is given for a theme if it represents a significant statement or

nonverbal communication that supports endorsement of that theme pertaining to that individual;

otherwise, a score of ‘‘0’’ is given for that theme. That is, for each field-test participant, each

theme is dichotomized either to a score of ‘‘1’’ or a ‘‘0,’’ depending on whether it is represented

by that individual. This dichotomization leads to the formation of what Onwuegbuzie (2003)

referred to as an ‘‘interrespondent matrix’’(i.e., participant × theme matrix) that contains a com-

bination of 0s and 1s (p. 396), which indicates which individuals contribute to each theme that

emerged from Phase 7. Factors that emerge from the exploratory factor analysis of focus group

themes are called ‘‘meta-themes’’ (Onwuegbuzie, 2003, p. 398), which represent themes at

a higher level of abstraction than the original emergent themes. According to Onwuegbuzie

(2003), ‘‘The manner in which the emergent themes cluster within each factor (i.e.,

meta-theme) facilitates identification of the interrelationships among the themes’’ (p. 398).

Thus, factor-analyzing themes that emerge to yield meta-themes represent integrated data reduc-

tion, inasmuch as a quantitative technique (i.e., exploratory factor analysis) is applied to quali-

tative data that represent a reduction of the original qualitative data (i.e., voice of the focus group

participants, nonverbal communication of the focus group participants) via a qualitative analysis

(e.g., method of constant comparison).

Phase 9: Validate Revised Instrument: Mixed Analysis Phase:Quantitative-Dominant Crossover Analyses

Phase 9 involves conducting one or more sets of quantitative-dominant crossover analyses.

Again, the instrument developers have many data analysis tools at their disposal. In particular,

this phase builds on the previous phase. For example, the factor scores derived from the factor

analysis of the emergent themes in Phase 8 could be correlated to the factor scores extracted via

the exploratory factor analysis of the quantitative instrument that was undertaken in Phase 6. As

another example, the themes that emerged in Phase 7 could be correlated to the factors that

emerged in Phase 6, or the meta-themes that emerged in Phase 8 could be correlated to the factors

that emerged in Phase 6.

Phase 10: Evaluate the Instrument Development/ConstructEvaluation Process and Product

Phase 10 involves a comprehensive evaluation both of the product and the process. The prod-

uct, which is the revised instrument, can be evaluated via the findings emerging in Phase 6 to

Phase 9. We suggest use of Onwuegbuzie et al.’s (2008) framework for debriefing the researcher.

This framework can help instrument developers reflect on the IDCV process to discover how

they felt about going through this process, determine which data collection and analytical strat-

egies were useful, and uncover areas for further growth and development of the instrument.

Based on the evaluation data, the instrument developer might need to return to Phase 5 and revise

the instrument further, and repeat Phase 6 to Phase 9, or he/she might even return to Phase 1 to

reconceptualize the theory and/or hypotheses, reevaluate any philosophical and research assump-

tions held by the instrument developers, and/or collect additional data from the local experts and

key informants, before repeating Phase 2 to Phase 9. Thus, the IDCV framework represents an

iterative, cyclical process that promotes more rigor to the process of instrument development and

construct validation.




In its most rigorous form, the instrument developer could design and implement a comprehen-

sive, mixed research–based evaluation system. This evaluation system might involve the use of

formal models of program evaluation such as Stufflebeam’s CIPP model, which provides

a framework for the evaluation of the IDCV’s context, input, processes, and products (Stuffle-

beam, 2002). Also, instructional design models such as the systems-based Dick and Carey model

(Dick, Carey, & Carey, 2004) can be used to drive the evaluation process via the applications of

learning theories and principles that inform the process of instrument development and construct

validation.

Heuristic Example

Background Information

To illustrate the role that mixed research can play in developing quantitative instruments in

general and the IDCV in particular, we briefly present the process that Bustamante, Nelson,

and Onwuegbuzie (2009) used to develop an instrument they called the School-Wide Cultural

Competence Observation Checklist (SCCOC). The SCCOC was developed as one of several

assessment tools for use in conducting school culture audits. A culture audit is an approach to

collecting organizational data from multiple sources to determine schoolwide cultural compe-

tence or how well a school’s policies, programs, and practices reflect the needs and experiences

of culturally and linguistically diverse groups in a school and school community (Bustamante,

2006). This process is displayed in Figure 2, which was created by the current authors.

Phase 1: Conceptualizing the construct of interest: Interdisciplinary review of the literature. The ini-

tial phase of SCCOC development (Qualitative; inductive logic) involved an extensive and sys-

tematic review of the academic literature (see Boote & Beile, 2005; Combs et al., in press;

Onwuegbuzie, Collins, Leech, Dellinger, et al., in press) to explore how the notions of individual

and organizational cultural competence had been defined and examined in a variety of academic

disciplines. This was undertaken by first conceptualizing the construct of culture because the

idea of cultural competence essentially is rooted in the concept of culture. This initial step

included a review of relevant studies from the fields of anthropology (Geertz, 1979; Hall,

1976; Kluckhohn & Strodbeck, 1961), sociology (Van Maanen, 1979), intercultural communi-

cation (Brislin, 1993; Casmir, 1999; Hammer, 1989; Ting-Toomey & Oetzel, 2001; Wiseman

& Koester, 1993), cross-cultural psychology (Triandis, 1994), and organizational theory (Hof-

stede, 1980, 2001; Schein, 1992; Trompenaars & Hampden-Turner, 1998). Second, the concepts

of individual and organizational cultural competence as applied to school settings also was

explored by reviewing literature from relevant fields, such as intercultural and cross-cultural

communication (Lustig & Koester, 2002; Lynch & Hanson, 1998; Wiseman & Koester,

1993), multicultural education (Banks, 2002), multicultural competence (Holcomb-McCoy,

2004), cultural proficiency (Lindsey, Roberts, CampbellJones, 2005; Lindsey, Robins, & Terrell,

2003; Salvaggio, 2003), culturally responsive schools (L. S. Johnson, 2003; Ladson-Billings,

1995), inclusive schools (Henze, Katz, Norte, Sather, & Walker, 2002; Riehl, 2000), school

equity audits (Skrla Scheurich, Garcia, & Nolly, 2006), diversity and cross-cultural leadership

in corporate settings (House, Hanges, Javidan, Dorfman, & Gupta, 2004; Kochan et al.,

2003), and cultural competence as defined in human service organizations (Cross, Bazron, Den-

nis, & Isaacs, 1989; Darnell & Kuperminc, 2006; National Center for Cultural Competence,

2005; Sue, 2001; Sue & Constantine, 2005). These interdisciplinary literature reviews served

to identify and define overlapping and deviant themes of similar constructs.

Additionally, in conceptualizing the construct of schoolwide cultural competence, the re-

searchers examined their own belief systems (Combs et al., in press; R. B. Johnson et al.,




2004). Researcher reflection on our belief systems revealed a tendency to view organization con-

texts through a cultural lens, a commitment to cultural relativism and diversity in learning envi-

ronments, and research belief systems that reflected pragmatic, constructivist, and critical theory

orientations. In Phase 1 of the IDCV model, an examination of researcher belief systems is

described as an important and often overlooked step of instrument development.

Phase 2: Identifying and describing behaviors that underlie the construct: Delphi study. The initial

review of relevant literature informing the construct of schoolwide cultural competence was fol-

lowed by a Delphi study that was conducted with a panel of experts who were practicing school

leaders. Experts completed four rounds of questionnaires until consensus was reached and satu-

ration of themes became evident. These narrative Delphi study results provided some initial ideas

about how cultural competence might be manifested in schools, but specifically in international

Phase 10: Evaluating the instrument development/construct evaluation process and product

Phase 9: Validating revised SCCOC: Quantitative-dominant cross-over mixed analyses

Phase 8: Validating revised SCCOC: Qualitative-dominant cross-over mixed analyses

Phase 7: Validating revised SCCOC: Qualitative Analysis Phase:

Phase 6: Validating revised SCCOC: Quantitative Phase: Exploratory Factor Analysis

Phase 5: Designing and field-testing revised SCCOC: Instrument fidelity

Phase 4: Pilot-testing of Initial SCCOC Instrument

Phase 3: SCCOC Development

Phase 2: Delphi Study

Phase 1: Interdisciplinary Review of the Literature

Figure 2. Instrument development and construct validation (IDCV) process: An heuristic example:School-Wide Cultural Competence Observation Checklist (SCCOC)




school settings (Bustamante, 2005). Thus, the Delphi research indicated several potential

domains and indicators of schoolwide cultural competence and proficiency that led to a more

extensive review of the literature. Consequently, the constructs and indicators conceptualized

in this phase were grounded in the relevant literature on cultural competence and formed a basis

for initial instrument design.

Phase 3: Developing initial instrument: SCCOC development. The original SCCOC instrument

contained 43 items and asked observers to provide yes/no responses to statements in several areas

related to potentially culturally competent policies, programs, and practices (Quantitative phase).

Items were developed based on domains identified in the initial Delphi study and were supple-

mented with additional items drawn from the literature on cultural competence and proficiency in

schools.

Phase 4: Pilot-testing of initial instrument. Pretesting for ‘‘face validity’’ was undertaken in focus

groups with graduate students in principal preparation programs who were working in schools at

the time (Qualitative). The participants also were asked to make observations and reflections

using the SCCOC in their actual school environments and to complete a separate open-ended

questionnaire in which they provided specific feedback on the utility and shortcomings of the

original instrument. Based on these pretest data gathered from school leader focus groups, revi-

sions were made to the SCCOC and a second version was created. The second version of the

SCCOC was revised in an unpublished manuscript and contained 33 items (Bustamante & Nel-

son, 2007). The SCCOC continues to be field-tested and revised to ensure instrument relevance

and fidelity.

Phase 5: Designing and field-testing revised instrument: Instrument fidelity. Phase 5 of the SCCOC

instrument fidelity process involved designing an Internet questionnaire to elicit perceptions

from a larger sample of practicing school leaders of the applicability and utility of the SCCOC

in assessing schoolwide cultural competence. Participants were 151 school leaders, comprising

teacher leaders, professional school counselors in training, directors, assistant principals, and

principals. Participants were asked to rate how important they found each checklist item to be

in assessing schoolwide cultural competence (Quantitative) and how they would categorize or

describe each item in their own words (Qualitative). Participants were asked to rate each SCCOC

item on a 4-point Likert-format scale anchored by 1 (strongly agree) and 4 (strongly disagree).

Phase 6: Validating revised instrument: Quantitative analysis phase: Exploratory factor analysis (EFA).The fourth phase (Quantitative phase) of testing for instrument fidelity of the SCCOC involved

use of an EFA to identify the factor structure of the SCCOC. Results revealed two factors that

explained 72.1% of the total variance called Policy (22 items) and Practice (11 items), which

yielded score reliability coefficients of .97 and .89, respectively (Nelson, Bustamante, Wilson,

& Onwuegbuzie, 2008). To check for divergent validity, SCCOC items were compared with

items on occupational health and climate surveys that are typically administered in schools.

Often, school climate and culture erroneously are used interchangeably in academic literature

and practice. However, scholars contend that the two constructs are different. School climate typ-

ically refers more to a temporary, ephemeral situation in a school, perhaps brought on by a school

safety incident or a school event. It may include feelings about the school building or support

staff, school safety, orderliness, and current teacher morale (Freiberg, 1998). School culture,

on the other hand, is characterized by many of the same elements described in seminal anthro-

pological definitions of culture that involve embedded characteristics such as shared values,

beliefs, traditions, and language that are passed on over time (Karpicke & Murphy, 1996; Schein,

1992).

Phase 7: Validating revised instrument: Qualitative analysis phase. In addition to completing both

the 33-item SCCOC and the Importance Scale, participants were asked to comment on the char-

acteristics of the items in a text box that was included on the same Internet survey (Qualitative




phase). The open-ended responses were analyzed using the method of constant comparison

(Glaser & Strauss, 1967). Four themes emerged from this iterative analysis process that further

informed the fidelity of the SCCOC: (a) policy, (b) practice, (c) school climate and culture, and

(d) barriers to schoolwide cultural competence (i.e., lack of clarity in roles and responsibilities,

resource constraints, limited knowledge of culturally responsive practices) (Bustamante, Nelson,

& Onwuegbuzie, in press).

Phase 8: Validating revised instrument: Mixed analysis phase: Qualitative-dominant crossover analy-ses. Several crossover analyses were used to integrate the qualitative and quantitative findings

(Mixed analysis phase). Specifically, each theme was quantitized (i.e., qualitative data were

transformed to numerical codes that could be analyzed quantitatively; Tashakkori & Teddlie,

1998),3 which allowed the frequency of each theme to be calculated. An EFA was conducted

to ascertain the underlying structure of the four themes. This EFA determined the number of fac-

tors underlying the themes (i.e., meta-themes; Onwuegbuzie, 2003) such that each meta-theme

contained one or more of the emergent themes.4 The EFA yielded two meta-themes comprising:

(a) policy and practice (33.00% of explained variance) and (b) school climate and culture, need

for clarity in implementation responsibility, lack of resources as perceived barriers, and limited

knowledge of culturally responsive instructional practices (29.49% of explained variance).

These meta-themes combined explained 62.5% of the total variance.

Phase 9: Validating revised instrument: Mixed analysis phase: Quantitative-dominant crossover anal-yses. Phase 9 involved the conduct of several crossover analyses that addressed the following

research question: What is the relationship between school leaders’ ratings of the SCCOC and

their perceptions of the SCCOC as a tool for assessing schoolwide cultural competence? The

quantitized themes were correlated to the ratings on the Importance Scale via a series of Bonfer-

roni-adjusted chi-square analyses. The quantitized themes also were correlated (i.e., repeated-

measures analysis of variance, canonical correlation analysis) to the factor scores stemming

from the EFA of the 33-item SCCOC in Phase 4 that yielded a two-factor solution. These anal-

yses yielded interesting relationships that were used to inform further the instrument fidelity of

the SCCOC.

Phase 10: Evaluating the instrument development/construct evaluation process and product. Phase

10 involved a comprehensive evaluation both of the product and the process. Analysis of the

product—namely, the SCCOC—revealed that this instrument appears to have better-than-

adequate face validity, item validity, outcome validity, generalizability, and structural validity.

However, although the quantitative analysis strongly suggested that the SCCOC also had better-

than-adequate sampling validity and substantive validity, the qualitative and mixed (i.e., cross-

over) analyses suggested that these areas of validity are somewhat problematic because the

SCCOC does not measure two of the emergent themes: (a) school climate and culture and (b)

barriers to schoolwide cultural competence. This inconsistency (i.e., initiation; Greene et al.,

1989) has led Bustamante et al. (2009) to return to Phase 5: redesigning and field-testing revised

instrument—including the assessment of concurrent validity and divergent validity—even

though they recognized that the SCCOC appears to have excellent psychometric properties

for measuring the policy and practice dimensions. Thus, the IDCV process allows for a process

approach to reexamining instrument fidelity with the goal of continuous improvement and prac-

tical adaptation to a variety of school contexts and changes over time.

Summary and Conclusions

Results from Bustamante et al.’s (2009) mixed research approach to developing the SCCOC

and verifying its fidelity provided more in-depth information than does a traditional quantitative

validity and reliability study alone would have provided.5 Specifically, two additional themes,




three subthemes, and two meta-themes were identified through the method of constant compar-

ison of the collected qualitative data and crossover analysis, respectively, which were not iden-

tified via the traditional exploratory factor analysis of responses to the Likert-format scale. The

mixed data collection and analysis elicited richer information than would have quantitative tech-

niques alone regarding the utility of the SCCOC as one tool in assessing schoolwide cultural

competence as part of an overall culture audit and perceptions of its applicability in actual school

settings by practicing school leaders. Moreover, using mixed research as a way to analyze the

fidelity of the SCCOC complements the culture audit process and its intricacies. Thus, as can

be seen, the IDCV process represents a more comprehensive and rigorous technique for devel-

oping a quantitative instrument and for construct validation. Indeed, the IDCV process represents

an extension—specifically, a mixed research-based extension—of Campbell and Fiske’s (1959)

MTMM because it incorporates all the elements of their cross-validation framework. Further-

more, the IDCV combines quantitative and qualitative data collection and data analysis techni-

ques, yielding an expanded cross-validation process via the use of crossover analysis strategies.

The fluidity of the IDCV process also allows researchers and instrument developers to return to

previous phases in the process to examine any needs for further instrument revisions and to focus

on continuously improving the fidelity and score validity of an instrument.

As noted by Hitchcock et al. (2005), ‘‘what is missing from the literature is a discussion of

how to combine qualitative and quantitative methodologies to investigate a key concern in

any social science endeavor, construct validation’’ (p. 260). We believe that our IDCV process

represents a step in the right direction to filling this void. Thus, we hope that researchers consider

using our IDCV process—or a modification of it—when seeking to develop an instrument and/or

validate a construct.

Table 3. Example of Interrespondent Matrix Containing Four SCCOC-Related Themes Usedto Conduct Mixed Analysis

ID Theme 1 Theme 2 Theme 3 Theme 4

001 1 0 1 1002 0 1 1 1003 0 1 0 1... ..

. ... ..

. ...

..

. ... ..

. ... ..

.

151 0 0 0 1

Note: SCCOC ¼ School-Wide Cultural Competence Observation Checklist. If a study participant listed a characteristic

that was eventually categorized under a particular theme, then a score of ‘‘1’’ would be given to the theme for the

participant’s response; a score of ‘‘0’’ would be given otherwise.

Table 4. Example of How to Use the Interrespondent Matrix to Compute Effect Sizes for Four Participants

ID Theme 1 Theme 2 Theme 3 Theme 4 Theme 5 Theme 6 Total %

001 1 0 1 1 1 0 4 66.7002 0 1 0 1 0 1 3 50.0003 0 0 0 1 0 0 1 16.7004 0 1 1 1 1 1 5 83.3

Total 1 2 2 4 2 2 13% 25.0 50.0 50.0 100.0 50.0 50.0




Declaration of Conflicting Interests

The authors declared no potential conflicts of interests with respect to the authorship and/or publication of

this article.

Funding

The authors received no financial support for the research and/or authorship of this article.

Notes

1. As do several authors (e.g., R. B. Johnson & Onwuegbuzie, 2004), we believe that the phrase mixed

research is more appropriate than is the term mixed methods research because the latter might give

the impression that this research paradigm only involves the mixing of methods. Yet this research par-

adigm involves ‘‘mix[ing] or combin[ing] quantitative and qualitative research techniques, methods,

approaches, concepts or language into a single study’’ (R. B. Johnson & Onwuegbuzie, 2004, p. 17).

2. A mixed research framework also is useful for optimizing the development of a qualitative instrument.

However, it is beyond the scope of the present article. For a discussion of this concept, see Collins et al.

(2006). For an example of this, we refer the reader to R. B. Johnson et al. (2007).

3. Table 3 shows how an interrespondent matrix might look for the 151 participants. Table 4 provides an

example, using four participants, of how the interrespondent matrix is used to compute various effect

sizes. Looking at the row totals and percentages, it can be seen from this table that the fourth participant

(i.e., ID 004) provided comments that contributed to the most themes (i.e., 5/6 ¼ 83.3%), with the third

participant (i.e., ID 003) contributing to the least themes (i.e., 1/6 ¼ 16.7%). Examining the column to-

tals reveals that Theme 4 is the most endorsed theme, with all four participants endorsing this theme.

Thus, the manifest effect size for Theme 4 is 100%. Conversely, the manifest effect size for Theme 1,

the least endorsed theme, is 25.0%. Because sample size (n ¼ 151) was large relative to the number

of themes (i.e., 4 themes)—yielding approximately 37 participants per theme—the interrespondent

matrix was subjected to an exploratory factor analysis (i.e., Phase 8), as well as to an array of general

linear model analyses (e.g., canonical correlation analysis; i.e., Phase 9).

4. As undertaken by Onwuegbuzie et al. (2007), the interrespondent matrix was converted to a matrix of

bivariate associations among the responses pertaining to each of the emergent themes. It is recommended

that these bivariate associations are represented by tetrachoric correlation coefficients because they are

appropriate to use when one is determining the relationship between two (artificial) dichotomous vari-

ables. Furthermore, it should be noted that tetrachoric correlation coefficients are based on the assump-

tion that for each manifest dichotomous variable, there is a normally distributed latent continuous

variable with zero mean and unit variance. Furthermore, it was assumed that the extent to which each

participant contributes to a theme, as indicated by the order in which the significant statements are pre-

sented, represent a normally distributed latent continuous variable. Thus, the matrix of tetrachoric cor-

relation coefficients was the basis of the exploratory factor analysis.

5. The first author recently taught a class in Slovenia containing 24 doctoral students representing numerous

disciplines (e.g., education, political science, agriculture, economics) and many nationalities (e.g., German,

Canadian, Hungarian, British), where he introduced the IDCV process. All the students deemed the IDCV

process to represent a more rigorous approach to instrument development and construct validation than

using quantitative techniques alone (e.g., MTMM). Even more compelling was the fact that when students

were required via a final course examination to use the IDCV framework to design the development of an

instrument in their fields, regardless of discipline or field, the students all submitted designs that were both

comprehensive and rigorous, suggesting that the IDCV framework has far-reaching application.

References

Banks, J. (2002). An introduction to multicultural education (3rd ed.). Needham, MA: Allyn & Bacon.




Banks, J., & McGee Banks, C. A. (2001). Multicultural education: Issues and perspectives (4th ed.). New

York: Wiley.

Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature

review in research preparation. Educational Researcher, 34(6), 3-15.

Brewer, J., & Hunter, A. (2006). Foundations of multimethod research: Synthesizing styles. Thousand

Oaks, CA: SAGE.

Brislin, R. (1993). Understanding culture’s influence on behavior. New York: Harcourt Brace Jovanovich.

Bustamante, R. M. (2005). Features of cultural proficiency in American International Schools in Latin

America: A Delphi study. Unpublished doctoral dissertation, University of San Diego.

Bustamante, R. M. (2006). The ‘‘culture audit’’: A leadership tool for assessment and strategic planning in

diverse schools and colleges. Connexions. Retrieved March 22, 2009, from http://cnx.org/content/

m13691/latest/

Bustamante, R. M., & Nelson, J. A. (2007). The School-Wide Cultural Competence Observation Checklist.

Unpublished manuscript, Sam Houston State University, Huntsville, Texas.

Bustamante, R. M., Nelson, J. A., & Onwuegbuzie, A. J. (2009). Instrument fidelity and construct validity of

the School-wide Cultural Competence Observation Checklist (SCCOC). Manuscript submitted for

publication.

Bustamante, R. M., Nelson, J. A., & Onwuegbuzie, A. J. (in press). Assessing school-wide cultural com-

petence: Implications for school leader preparation. Educational Administration Quarterly.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-

multimethod matrix. Psychological Bulletin, 56, 81-105.

Casmir, F. (1999). Foundations for the study of intercultural communications based on a third-culture build-

ing model. International Journal of Intercultural Relations, 23, 91-116.

Cattell, R. B. (1978). The scientific use of factor analysis in behavioral and life sciences. New York:

Plenum.

Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and pur-

pose for conducting mixed methods research in special education and beyond. Learning Disabilities: A

Contemporary Journal, 4, 67-100.

Combs, J. P., Bustamante, R. M., & Onwuegbuzie, A. J. (in press). A mixed methods approach to conduct-

ing literature reviews for stress and coping researchers: An interactive literature review process frame-

work. In K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward a broader understanding of

stress and coping: Mixed methods approaches. The Research on Stress and Coping in Education Series

(Vol. 5). Greenway, CT: Information Age.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rine-

hart & Winston.

Cross, T. L., Bazron, B. J., Dennis, K. W., & Isaacs, M. R. (1989). Towards a culturally competent system of

care: Vol. 1. A monograph on effective services for minority children who are severely emotionally dis-

turbed. Washington, DC: Georgetown University, Child Development Center, Child and Adolescent

Services System Program, Technical Assistance Center.

Darnell, A. J., & Kuperminc, G. P. (2006). Organizational cultural competence in mental health service

delivery: A multilevel analysis. Journal of Multicultural Counseling and Development, 34, 194-207.

Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach.

Journal of Mixed Methods Research, 2(3), 270-283.

Dick, W., Carey, L., & Carey, J. O. (2004). The systematic design of instruction (6th ed.). Boston: Allyn &

Bacon.

Everitt, B. S. (1975). Multivariate analysis: The need for data, and other problems. British Journal of Psy-

chiatry, 126, 237-240.

Fabrigar, L. R., & Joelle Estrada, M. (2007). Multitrait multimethod matrix and construct validity. In

N. J. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 666-669). Thousand Oaks, CA:

SAGE.

Freiberg, H. J. (1998). Measuring school climate: Let me count the ways. Educational Leadership, 56,

336-343.

Geertz, C. (1979). The interpretation of culture. New York: Basic Books.




Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative

research. Chicago: Aldine.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method

evaluation designs. Educational Evaluation and Policy Analysis, 11, 255-274.

Hall, E. T. (1976). Beyond culture. Garden City, NY: Doubleday.

Halpern, E. S. (1983). Auditing naturalistic inquiries: The development and application of a model. Unpub-

lished doctoral dissertation, Indiana University.

Hammer, M. (1989). Intercultural communication competence. In M. K. Asante & W. Gudykunst (Eds.),

Handbook of international and intercultural communication (pp. 246-260). Newbury Park, CA: SAGE.

Henson, R. K., Capraro, R. M., & Capraro, M. M. (2004). Reporting practice and use of exploratory factor

analyses in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72.

Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research. Educa-

tional and Psychological Measurement, 66, 393-416.

Henze, R., Katz, A., Norte, E., Sather, S., & Walker, E. (2002). Leading for diversity. Thousand Oaks, CA:

Corwin Press.

Hetzel, R. D. (1996). A primer on factor analysis with comments on patterns of practice and reporting. In

B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 175-206). Greenwich, CT:

JAI Press.

Hitchcock, J. H., Nastasi, B. K., Dai, D. C., Newman, J., Jayasena, A., Bernstein-Moore, R., et al. (2005).

Illustrating a mixed-method approach for identifying and validating culturally specific constructs. Jour-

nal of School Psychology, 43, 259-278.

Hitchcock, J. H., Sarkar, S., Nastasi, B. K., Burkholder, G., Varjas, K., & Jayasena, A. (2006). Validating

culture- and gender-specific constructs: A mixed-method approach to advance assessment procedures in

cross-cultural settings. Journal of Applied School Psychology, 22, 13-33.

Hofstede, G. (1980). Cultures consequences: International differences in work-related values. Beverly

Hills, CA: SAGE.

Hofstede, G. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations

across nations. London: SAGE.

Hogarty, K. Y., Hines, C. V., Kromrey, J. D., Ferron, J. M., & Mumford, K. R. (2005). The quality of factor

solutions in exploratory factor analysis: The influence of sample size, communality, and overdetermi-

nation. Educational and Psychological Measurement, 65, 202-226.

Holcomb-McCoy, C. C. (2004). Assessing the multicultural competence of school counselors: A checklist.

Professional School Counseling, 7, 178-183.

House, R. J., Hanges, P. J., Javidan, M., Dorfman, P. W., & Gupta, V. (2004). Culture, leadership, and or-

ganizations: The GLOBE study of 62 societies. London: SAGE.

Johnson, L. S. (2003). The diversity imperative: Building a culturally responsive school ethos. Intercultural

Education, 14(1), 17-30.

Johnson, R. B., Meeker, K., & Onwuegbuzie, A. J. (2004, April). Development and use of the Philosophical

Beliefs Questionnaire. Paper presented at the annual meeting of the American Educational Research

Association, San Diego, CA.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time

has come. Educational Researcher, 33(7), 14-26.

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods

research. Journal of Mixed Methods Research, 1(2), 112-133.

Karpicke, H., & Murphy, M. E. (1996). Productive school culture: Principals working from the inside.

NAASP Bulletin, 80(576), 26-34.

Kieffer, K. M. (1999). An introductory primer on the appropriate use of exploratory and confirmatory factor

analysis. Research in the Schools, 6(2), 75-92.

Kluckhohn, F., & Strodbeck, F. (1961). Variations in value orientations. New York: Row Peterson.

Kochan, T., Bezrukova, K., Ely, R., Jackson, S., Joshi, A., Jehn, K., et al. (2003). The effects of diversity on

business performance: Report of the diversity research network. Human Resource Management, 42, 3-21.

Kuhn, T. S. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.




Ladson-Billings, G. (1995). But that’s just good teaching The case for culturally relevant pedagogy. Amer-

ican Educational Research Journal, 32, 159-165.

Leech, N. L., & Onwuegbuzie, A. J. (2007). An array of qualitative data analysis tools: A call for qualitative

data analysis triangulation. School Psychology Quarterly, 22, 557-584.

Leech, N. L., & Onwuegbuzie, A. J. (2008). Qualitative data analysis: A compendium of techniques and

a framework for selection for school psychology research and beyond. School Psychology Quarterly,

23, 587-604.

Lester, F. K. (2005). On the theoretical, conceptual, and philosophical foundations for research in mathe-

matics education. Zentralblatt fur Didaktik der Mathematik, 37, 457-467.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: SAGE.

Lindsey, R., Roberts, L. M., & CampbellJones, F. (2005). The culturally proficient school: An implemen-

tation guide for school leaders. Thousand Oaks, CA: Corwin Press.

Lindsey, R., Robins, K. N., & Terrell, R. (2003). Cultural proficiency: A manual for school leaders (2nd

ed.). Thousand Oaks, CA: Corwin Press.

Lustig, M. W., & Koester, J. (2002). Intercultural competence: Interpersonal communication across cul-

tures (4th ed.). Boston: Longman.

Lynch, E. W., & Hanson, M. J. (1998). Developing cross-cultural competence: A guide for working with

young children and their families (2nd ed.) Baltimore: Paul H. Brookes.

Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Old

Tappan, NJ: Macmillan.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons responses

and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

Morgan, D. L. (1998). The focus group guidebook. Thousand Oaks, CA: SAGE.

Mundfrom, D. J., Shaw, D. G., & Ke, T. L. (2005). Minimum sample size recommendations for conducting

factor analyses. International Journal of Testing, 5, 159-168.

National Center for Cultural Competence. (2005). Cultural and linguistic competence: Definitions, frame-

works, and implications. Retrieved May 26, 2009, from www.nccccurricula.info/culturalcompetence.

html

Nelson, J. A., Bustamante, R. M., Wilson, E., & Onwuegbuzie, A. J. (2008). The school-wide cultural com-

petence observation checklist for school counselors: An exploratory factor analysis. Professional School

Counseling, 11, 207-217.

Onwuegbuzie, A. J. (2003). Effect sizes in qualitative research: A prolegomenon. Quality & Quantity:

International Journal of Methodology, 37, 393-409.

Onwuegbuzie, A. J., & Collins, K. M. T. (2007). A typology of mixed methods sampling designs in social

science research. The Qualitative Report, 12, 281-316. Retrieved April 7, 2009, from http://www.no-

va.edu/ssss/QR/QR12-2/onwuegbuzie2.pdf

Onwuegbuzie, A. J., Collins, K. M. T., Leech, N. L., Dellinger, A. B., & Jiao, Q. G. (in press). A meta-

framework for conducting mixed research syntheses for stress and coping researchers and beyond. In

K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward a broader understanding of stress

and coping: Mixed methods approaches. The Research on Stress and Coping in Education Series

(Vol. 5). Greenway, CT: Information Age.

Onwuegbuzie, A. J., Collins, K. M. T., & Leech, N. L. (in press). Mixed research: A step-by-step guide.

New York: Taylor & Francis.

Onwuegbuzie, A. J., & Combs, J. P. (in press). Emergent data analysis techniques in mixed methods

research: A synthesis. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social

and behavioral research (2nd ed.). Thousand Oaks, CA: SAGE.

Onwuegbuzie, A. J., & Daniel, L. G. (2003, February 12). Typology of analytical and interpretational errors

in quantitative and qualitative educational research. Current Issues in Education [On-line], 6(2).

Retrieved April 5, 2009, from http://cie.ed.asu.edu/volume6/number2/

Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. T. (2009). A meta-validation model for assessing the

score validity of student teacher evaluations. Quality & Quantity: International Journal of Methodology,

43, 197-209.




Onwuegbuzie, A. J., Johnson, R. B., & Collins, K. M. T. (2009, April). Toward a philosophy of mixed data

analysis. Paper presented at the annual meeting of the American Educational Research Association, San

Diego, CA.

Onwuegbuzie, A. J., Leech, N. L., & Collins, K. M. T. (2008). Interviewing the interpretive researcher: A

method for addressing the crises of representation, legitimation, and praxis. International Journal of

Qualitative Methods, 7, 1-17.

Onwuegbuzie, A. J., Leech, N. L., Dickinson, W. B., & Zoran, A. G. (in press). Toward more rigor in focus

group research in stress and coping and beyond: A new mixed research framework for collecting and

analyzing focus group data. In K. M. T. Collins, A. J. Onwuegbuzie, & Q. G. Jiao (Eds.), Toward

a broader understanding of stress and coping: Mixed methods approaches. The Research on Stress

and Coping in Education Series (Vol. 5). Greenway, CT: Information Age.

Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In

A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp.

351-383). Thousand Oaks, CA: SAGE.

Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M. T., Filer, J. D., Wiedmaier, C. D., & Moore, C. W.

(2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teach-

ing evaluation form using a mixed-methods analysis. American Educational Research Journal, 44, 113-

160.

Powell, H., Mihalas, S., Onwuegbuzie, A. J., Suldo, S., & Daley, C. E. (2008). Mixed methods research in

school psychology: A mixed methods investigation of trends in the literature. Psychology in the Schools,

45, 291-309.

Riehl, C. J. (2000). The principal’s role in creating inclusive schools for diverse students: A review of nor-

mative, empirical, and critical literature on the practice of educational administration. Review of Edu-

cational Research, 70, 55-81.

Salvaggio, K. C. (2003). Perceptions of formal and informal school leaders regarding cultural proficiency

in high-achieving, highly diverse elementary schools. Unpublished doctoral dissertation, University of

LaVerne.

Schein, E. (1992). Organizational culture and leadership (2nd ed.). San Francisco: Jossey-Bass.

Skrla, L., Scheurich, J. J., Garcia, J., & Nolly, G. (2006). Equity audits: A practical leadership tool for

developing equitable and excellent schools. Educational Administration Quarterly, 40, 135-163.

Spradley, J. P. (1979). The ethnographic interview. Fort Worth, TX: Holt, Rinehart & Winston.

Stufflebeam, D. L. (2002). CIPP evaluation model checklist: Fifth installment of the CIPP model. Kalama-

zoo, MI: The Evaluation Center. Retrieved February 4, 2009, from http://www.wmich.edu/evalctr/

checklists/cippchecklist.htm

Sue, D. W. (2001). Multidimensional facets of multicultural competence. The Counseling Psychologist, 29,

790-821.

Sue, D. W., & Constantine, M. G. (2005). Effective multicultural consultation and organizational develop-

ment. In M. G. Constantine & D. W. Sue (Eds.), Strategies for building multicultural competence in

mental health settings (pp. 212-226). Hoboken, NJ: Wiley.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative ap-

proaches. Applied Social Research Methods Series (Vol. 46). Thousand Oaks, CA: SAGE.

Teddlie, C., & Johnson, R. B. (2009). Methodological thought since the 20th century. In C. Teddlie & A

Tashakkori (Eds.), Foundations of mixed methods research: Integrating quantitative and qualitative

techniques in the social and behavioral sciences (pp. 62-82). Thousand Oaks, CA: SAGE.

Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applica-

tions. Washington, DC: American Psychological Association.

Ting-Toomey, S., & Oetzel, J. (2001). Managing intercultural conflict effectively. Thousand Oaks, CA:

SAGE.

Tinsley, H. E. A., & Tinsley, D. J. (1987). Uses of factor analysis in counseling psychology research. Jour-

nal of Counseling Psychology, 34, 414-424.

Triandis, H. (1994). Culture and social behavior. New York: McGraw-Hill.

Trompenaars, F., & Hampden-Turner, C. (1998). Riding the waves of culture: Understanding cultural

diversity in global business (2nd ed.). New York: McGraw-Hill.




Van Maanen, J. (1979). The fact of fiction in organizational ethnography. Administrative Science Quar-

terly, 24, 249-550.

Wiseman, R., & Koester, J. (1993). Intercultural communication competence. Newbury Park, CA: SAGE.




mixed research as a tool for developing quantitative

Documents