RCAL 1

Validity Threats, Biases & Systematic Error in

International Research

Roger Calantone

2004

Not Complete Without Commentary

RCAL 2

Overarching Empirical Goal Is Cross-cultural Equivalence In:

• Measures & their properties

• Instruments & their administration

• Researcher / experimenter conduct

• Research settings

RCAL 3

We Must Be Able to Set up Controls to Positively Insure

Equivalence.

Failing This We Must Have Specific Measures of These Threats to Validity, So We

Might Control for Them Statistically.

RCAL 4

Bias• Cross-cultural bias is the asymmetry of

measurement across cultures.• Bias can be a function of the instrument

based constructs interacting differentially with the cultures addressed, the instrument itself or both.

• A lack of cross-cultural bias permits measurement equivalence, but cannot guarantee it.

RCAL 5

Equivalence

• Construct equivalence (also referred to as structural equivalence).

• Measurement unit equivalence.

• Scalar equivalence (also referred to as full score comparability).

The above levels of equivalence are listed in increasing order of difficulty.

RCAL 6

Construct Equivalence

• The most basic level of equivalence.• The same construct occurs in each

culture, although it may be measured in a different manner (items or methods).

• (Of course you may not be able to compare scores across cultures unless the constructs are operationalized the same way.).

RCAL 7

Measurement Unit Equivalence

• Constructs are sufficiently similar and well used enough across the cultures so offset mechanism is known or discoverable.

• Simple example is Celsius versus Kelvin scales of temperature.(Simple transform universally).

• Complex example is use of a Swedish IQ test on Turkish youngsters.(A constant disadvantage is known but varies by sample and/or by study).

RCAL 8

Scalar Equivalence

• Hardest to obtain; Rarely demonstrated.• Measurement instrument is on the same

ratio scale in each cultural group. (eg:measure of body length & weight).

• Achievable when scores on an instrument have same interval scale across cultural groups. (Refers to useful characteristic of interval scales => differences on an interval scale are measured at a ratio level).

RCAL 9

Equivalence Must Be Established

• It is not an intrinsic property of a scale or instrument or method.

• There is no evidence that equivalence in one set of cultural comparisons is present across other cultural comparisons.

• Equivalence is a function of the properties of an instrument and the characteristics of the cultural groups involved in the study.

RCAL 10

Claims of Equivalence Can Be Controversial !

• Scalar equivalence is often claimed when only construct equivalence has been established.

• Many times this is based on finding similar item loadings of similar or even identical items, using an EFA on the several cultural groups.

• This is very open to suspicion.

RCAL 11

2 Big Problems Using EFA to Establish Scalar Equivalence

• Measurement units may not be the same across groups. If the scores in one group are doubled, but the correlation pattern remains the same, the cross-cultural differences are affected but the loadings will be unaffected !

• Measurement unit equiv.. May obtain, but a bias that affects all of an instrument’s stimuli in a systematic way cannot be detected by EFA.

RCAL 12

EFA Problems Continued• Both arguments against the use of EFA to

establish scalar equivalence derive from the use of the correlation matrix as input to an EFA.

• Linear transformations, e.g.: (y=ax+b with a>0) impact raw scores & hence x-cultural differences but do not impact correlations. Techniques insensitive to such linear transformations are ill-suited to substantiate scalar equivalence.

RCAL 13

Avoiding EFA

• The simple fix is to design for scalar equivalence from the beginning and to only attempt scalar testing using CFA, which uses the covariance matrix and permits statistical equivalence tests of loadings across cultural groups.

• Unit measure equivalence testing should also employ Optimal Scaling.

RCAL 14

Biases & Their Detection

• Bias is the basic name for all the factors that challenge the validity of cross-cultural comparisons.

• Three types of Bias are commonly recognized: Construct, Method, & Item.

• Bias is always present, but with care, can almost always be mitigated to non-threatening levels through good design.

RCAL 15

TABLE 1 Overview of Types of Bias and Their Most Common Causes

(after van de Vijver & Poortinga, 1997, p. 26)

Type of Bias Source

Construct • incomplete overlap of definitions of the construct across cultures• differential appropriateness of (sub)test content (e.g., skills do not belong to the

repertoire of one of the cultural groups)• poor sampling of all relevant behaviors (e.g., short instruments)• incomplete coverage of the construct (e.g., not all relevant domains are sampled)

Method • differential social desirability• differential response styles such as extremity scoring and acquiescence• differential stimulus familiarity• lack of comparability of samples (e.g.differ in educational background, age, or gender composition)• differences in physical conditions of administration• differential familiarity with response procedures• tester/interviewer effects• communication problems between respondent and interviewer in either cultural group

Item • poor item translation• inadequate item formulation (e.g., complex wording)• item(s) may invoke additional traits or abilities• incidental differences in appropriateness of the item content (e.g., topic of item of educational test not in curriculum in one cultural group)

RCAL 16

TABLE 2 Is Level of Equivalence Affected by Bias?

Level of Equivalence

Construct Measurement Unita Scalara, bType of Bias

Construct bias no no no

Method bias: uniform yes yes no nonuniform yes no no

Item bias: uniform yes yes no nonuniform yes no no

a. The same measurement unit is assumed in each cultural group.b. The same origin is assumed in each cultural group.

RCAL 17

Construct Bias

• Occurs when construct measured is not identical across groups. (appears frequently in comparisons of western & non-western cultures).

• Can be induced by lack of overlap in behaviors associated with the cultures studied. (E.g.. Machiavellianism).

• Also, Construct underrepresentation.

RCAL 18

Construct Bias Fixes

• Emic approaches must precede.

• Decentered approach in conceptualization of the study.

• Convergence approach in conceptualization of the study. This is considerably more difficult if indigenous materials must be developed.

RCAL 19

Method Bias

• Bias arises from characteristics of instrument and/or its administration.

• Differential response styles across cultures. (use of scales/extremes).

• Differential familiarity with stimuli used. • Differential familiarity with response tool:

E.g. draw a picture versus make a wire model.

RCAL 20

Method Bias Avoidance• “Culture-Free” tests.• “Culture-Fair” tests.• Interviewer repetition & rotation.• Systematic variation of stimuli across

cultures & examination of score changes as well as differential learning effects.

• Use of un-timed exercises.

RCAL 21

Item Bias

• Differential appropriateness of item content (asymmetric item sets).

• Inadequate item functioning (e.G. Complex wording).

• Inadequate translation concordance.

• All of these can be mitigated by rigorous pretesting, with post-inquiry debriefing.

RCAL 22

TABLE 3 Four Common Types of Cross-Cultural Studies

Orientation More on

Consideration ofContextual Factors Hypothesis Testing Exploration

No Generalizability Psychological differences Yes Theory-driven External validation

SOURCE: van de Vijver & Leung, 1997..

RCAL 23

Discussion Questions• Can we really design culture-equivalent

instruments? • Can we reduce the bias in our

comparative studies?• Are instrument fixes better than

statistical fixes?• Where does equivalence and bias fit in

the EMIC/ETIC debate?

RCAL 24

References Davis, Harry L., Susan P. Douglas and Alvin J. Silk (1981), “ Measure Uncertainty: A Hidden Threat to Cross-National Marketing Research?” Journal of Marketing, 45(2), 98-109 . Mintu, Alma T., Roger J. Calantone and Jule B. Gassenheimer (1994), “ Towards Improving Cross-Cultural Research: Extending Churchill’s Research Paradigm,” Journal of International Consumer Marketing, Vol. 7(2), 5-23. Mullen, Michael R. (1995), “ Diagnosing Measurement Equivalence in Cross-National Research,” Journal of International Business Studies, Vol. 26, 573-596 . Matt Myers, R. Calantone, T. Page, R. Taylor, (2000), “Assessment of Cross-Cultural Equivalence: An Application and Extension of Multiple Group Causal Models in International Research,” Journal of International Marketing, Vol. 8, No. 4 Salzberger, Thomas, Rudolf Sinkovics & B. Schlegelmich (2001), “Data Equivalence in International Research: A Comparison of Classical Test Theory and Latent Trait Theory Based Approaches”, Australasian Marketing Journal, 7(2). Sekaran, Uma (1983), “Methodological and Theoretical Issues and Advancements in Cross-Cultural Research,” Journal of International Business Studies, 14(2), 61-73 . Singh, Jagdip (1995), “ Measurement Issues in Cross-National Research,” Journal of International Business Studies, vol. 26, 597-619 . Steenkamp J-B E.M. & H. Baumgartner (1998), “Assessing Measurement Invariance in Cross-National Consumer Research”, Journal Of Consumer Research, 25, p. 78ff. Vijier, Van de And Y. H. Poortinga (1997), “ Towards an integrated analysis of bias in cross-cultural assessment”, European Journal of Psychological Assessment, 13, 21-29 .