1

Prepared for:

2009 MSU Symposium on Multicultural Psychology “Conducting Multinational Research Projects in

Organizational Psychology: Challenges and Opportunities”

October 11-12, 2009

Organizational and Methodological Issues in Large-Scale Cross-National Research: UN or NATO Approach?

Juan I. SanchezFlorida Int’l University

Paul E. Spector

University of South Florida

2

Sample Cross-Cultural/Cross-National – CC/CN - Publications

Bonache, J., Sanchez, J. I., & Zarraga-Oberty, C. (in press). The interaction of expatriate pay differential and expatriate inputs on host country nationals’ pay unfairness. International Journal of Human Resources Management.

Sanchez, J. I., Gomez. C., & Wated, G. (2008). A value-based framework for understanding managerial tolerance of bribery in Latin America. Journal of Business Ethics, 83(2), 341-352.

COLLABORATIVE INTERNATIONAL STUDY OF MANAGERIAL STRESS (CISMS):

  Spector, P. E., Cooper, C. L., Sanchez, J. I., O’Driscoll, M., Sparks, K., et al. (2002). Locus of control and well-being at work: How Generalizable Are Western Work Findings? Academy of Management Journal, 45, 453-470.

Spector, P. E., Cooper, C. L., Sanchez, J. I., O’Driscoll, M., Sparks, K., et al. (2001). Do national levels of individualism and internal locus of control relate to well-being: An ecological level international study. Journal of Organizational Behavior, 22(8), 815-832.

Sanchez, J. I., Spector, P. E., & Cooper, C. L. (2000). Adapting to a boundaryless world: A developmental model of the expatriate executive. Academy of Management Executive, 14(2),96-106.

3

CC/CN Research Publications Outlet Number of articles

Academy of Management Executive 1

Academy of Management Journal1 11

Applied Psychology: An Int’l Review1 31

Group and Organization Management 1

Human Resource Management 1

International Journal of Cross-Cultural Mgmt. 1

Int’l Journal of HRM 3

Int’l Journal of Organizational Analysis 1

Int’l Journal of Stress Management1 21

Journal of Business Ethics 2

Journal of Organizational Behavior1 11

Journal of Vocational Behavior1 11

Personnel Psychology1 21

1Large-scale cross-cultural research where n > 5, 000 AND no. of countries > 20

4

Basic research question

Do organizational behavior theories developed primarily in English-speaking countries transfer to other cultural settings?

X Y

Culture-general (etic)?

YX

Culture-specific (emic)?

5

Country rxy

A .06

B .23

C .12

D .39

Variation due to culture?

Country rxy

A .06

B .23

C .12

D .39

Variation due to third variable confounds and statistical artifacts?

Organizational

Methodological

Basic research question (Continued)

6

Organizational choices -> a high degree of control over methodological issues -> adequate progression of research.

Our relatively centralized, goal-oriented approach to the organization of these studies contrasts with a decentralized one (we have tongue-in-cheek labeled these as the “NATO” and the “UN” approach, respectively).

North Atlantic Treaty Organization

United Nations

Basic research question (Continued)

7

Organizational Issues in CC/CN Research (Continued)

Selection of Researchers. Based on our experience with CISMS 1, we decided to classify the participating researchers in CISMS 2 in two groups:

A core group of researchers who participated in the choice of scales to be included in the survey, the design of the study, and the formulation of broad research goals and objectives.

A group of research collaborators who handled primarily data collection in a specific country, and who in several cases went on to conduct within-country studies or smaller scale between-country comparisons.

history of prior collaboration/trust

well-published in English-language journals

8

Pre-study contract. All participating researchers agreed to a “contract” or set of written rules that specified the obligations and rights of each participant.

Most important: Rules regarding authorship.

*Each participating researcher agreed to run their manuscripts by all the members of the core group, and to include their names as co-authors in any resulting manuscripts after incorporating their feedback into the manuscript.

*The order of authorship was determined according to effort and contribution and, when these were similar in magnitude, by alphabetical order.

Organizational Issues in CC/CN Research (Continued)

9

Leadership model. rules for participation and authorship made the study relatively easy to manage.

Core group -> impetus for CISMS: They were instrumental in recruiting other researchers.

All participating researchers were asked to send the data in electronic form to the central team, but in some cases –especially during CISMS 1- they sent the questionnaires and the data were entered by the central team. Instructions regarding data entry were provided to all participating researchers. When issues or questions about the data arose during data entry, the central team contacted the specific researcher.

Organizational Issues in CC/CN Research (Continued)

10

The issues encountered ran the gamut…

* A participating researcher from an Islamic country who, upon learning that the study included data from Israel, decided to withdraw from the study –she alleged her withdrawal was a matter of personal security...

* Endless exchanges and warnings for and against calling Taiwan a “country”… the word “territory” came in handy…

Organizational Issues in CC/CN Research (Continued)

11

Methodological Issues in CC/CN Research

Focus on two types of confounds that may diminish the researchers’ certainty regarding cultural effects:

1. Carefully done translations may not necessarily result in equivalent instruments, despite back-translation (Brislin, 1986).

2. Valid conclusions about culture effects require samples that present minimal differences beyond their differing cultures, but there are numerous factors potentially confounded with culture:

task policieswelfare statesocialized medicineemerging economy…

12

Methodological Issues: Measurement Equivalence

Is measurement equivalence between translated instruments possible?

Two schools of thought…

Nonequivalence is due to poor translation…

Nonequivalence is due to the Sapir-Whorf or Whorfian hypothesis (Werner & Campbell, 1970) -> language is thought to filter the evaluative, connotative, and affective meaning of scale items, thereby preventing comparisons between linguistically different groups.

13

Video-clip:

The German Coast Guardhttp://www.youtube.com/watch?v=8vBn2_ia8zM

14

Methodological Issues: Back-translation

Translating the measure to the target language first, independently translating it back to the source language, and comparing the two versions to ascertain linguistic problems in the translation (Brislin, 1986).

Nevertheless,

back-translations do not necessarily guarantee measurement equivalence. Back-translators share knowledge that may lead them to consider terms as if they were synonyms whereas respondents do not see them that way (e.g., “simpático” in Spanish and “friendly” in English may not always mean the same thing, but translators may see them that way due to their learned semantic connection between the two languages).

back-translations may retain the grammar and idiomatic expressions of the source language, which may be easy to back-translate but may not have the same meaning to monolinguals in the target language. For example, an item developed for a Chinese scale (i.e., “one should not be afraid of the change of heavens”), which measured beliefs of control in China was successfully back-translated from English to Chinese, but it obviously lacks unambiguous meaning for English-speaking monolinguals (Siu & Cooper, 1998).

15

Methodological Issues: Back-translation (Continued)

Differences in item calibration: an individual who agrees with the item, "I am sometimes tense at work", will not necessarily endorse a more extremely worded item like "I sometimes find myself in panic at work." The reason is that, even though both items reflect anxiety, they symbolize different degrees of the anxiety construct.

1. Culture may enhance this effect: A participant from India, for example, might not see an item as representing the exact same degree of the construct than an Australian counterpart.

2. If we cross languages, the problem becomes

worse. Consider an item that reads "I love my job." A linguistically correct Spanish translation of this item will be "Yo amo mi trabajo." However, the verb "amar" (to love) in Spanish is usually confined to people, and it is seldom or never used to refer to things or social constructs like a job.

16

Anchor equivalence: finding equivalent anchors for the

extreme points of the scale is relatively easy, but finding

equivalent anchors for the mid-points can be tricky.

For instance, a suitable Spanish translation of “very much agree” would be “muy de acuerdo,” but translating the mid-anchor “agree slightly” is not straightforward because the exact degree conveyed by the word “slightly” does not have an unequivocal synonym in Spanish…

Example:Consider what it means to be “slightly late” in different cultures...

Methodological Issues: Back-translation (Continued)

17

Back-translation is an “art,” not a science:

Inattention to measurement criteria pervades back-translation procedures.

There are no clear guidelines for deciding when is a back-translated scale sufficiently close to the original version, as this decision is usually left to the translators’ professional judgment.

Methodological Issues: Back-translation (Continued)

18

Methodological Issues: Scale reliability

Reliability: Consider the mean alpha reliability coefficient across 26 scales related to work stress was:

.70 UK

.75 US

.69 Spain Similar level of reliability than the original English scales employed in the U.K. and the U.S.

A LISREL comparison of the var-cov structures of Spector’s (1988) Work Locus of Control Scale between the U.K. and Spanish samples suggested that inter-item relationships differed between the two countries:In Spain, the item “if you know what you want out of a job, you can find a job that gives it to you” had practically null correlations with most of the other scale items (median r = .10).In the U.K. this item had generally larger and statistically significant correlations with other items in the scale (median r = .23). This differential item functioning should not be surprising to anyone familiar with the relatively high levels of unemployment in Spain.

An apparently adequate level of reliability does not necessarily constitute a sign of the equivalence between the original and the translated instrument.

19

Methodological Issues: Construct equivalence

A given construct might manifest itself differently across countries/cultures, so that different items would be needed to capture its unique cultural nuances (Lonner, 1990).

Construct equivalence has received far less attention that measurement equivalence, but it is probably a more important issue…

Example: Stress Researchdirect disagreements among employees might be more likely to be perceived as interpersonal conflict among Chinese for whom group harmony is an important value than among Americans who are less sensitive to direct confrontation.

in an analysis of stressful work incidents, Americans were more likely to have direct conflicts with others, whereas Chinese were more likely to have indirect.

When asked to describe a stressful incident at work, Americans but not Indians reported instances of lack of control and work overload, whereas Indians but not Americans reported instances of lack of structure and equipment/situational constraints (Narayanan, Menon & Spector, 1999). The most often noted stressor for Americans and Indians was opposite—too little control for one and too little direction for the other.

20

Methodological Issues: Response Biases

Culture/national differences in response biases or tendencies (Triandis, 1994b; van de Vijver & Leung, 1997) –e.g., people from some countries preferring extreme responses and others avoiding them.

These tendencies might be more complex than previously thought:

Compared to Americans, Japanese avoided reporting extreme positive feelings, but not extreme negative ones (Iwata, Umesue, Egashira, Hiro, Mizoue, Mishima, and Nagata, 1998).

Differences in response tendencies are NOT equivalent across all scales or even all items within a scale (Iwata et al., 1998).

Finally, we cannot be certain that a given difference can be attibutable to response tendencies just because one group usually scores higher or lower than others. Did the Japanese in the Iwata et al. (1998) study avoid extreme positive reports because of cultural modesty tendencies, or because they experienced less positive affect than Americans?

21

Research designs that analyze/control language effects

Because back-translation does not guarantee the absence of language effects on cross-cultural comparisons, research designs have been employed to control or analyze for them...

(1)between-participant comparisons across languages that hold culture constant, such as for example comparing the English and French versions of a measure using samples from the same nation (e.g., Canadians) (Candell & Hulin, 1987),

(2)between-participant comparisons across cultures holding language constant, such as, for instance, comparing U.S. versus Australian samples (Ryan, Chan, Ployhart, & Slade, 1999),

(3)Use bilinguals in within-participant comparisons of responses in different languages (Hulin, Drasgow, & Komocar, 1982; Katerberg, Hoy, & Smith, 1977; Rybowiak, Garst, Frese, & Batinic, 1999).

22

Each one of these approaches has limitations…

(1)between-participant comparisons across languages that attempt to hold culture constant may not succeed due to self-selection. Take for instance the case of a sample of 1,931 Canadians enlisted in the Canadian Armed Forces used by Candell and Hulin (1987). The 235 French-Canadians who responded in English might have been more acculturated to the patterns of English-speaking Canada than those who chose the French version and, therefore, culture was in part confounded with language in comparing these two groups of French Canadians.

(2)between-participant comparisons across cultures that hold language constant (e.g., Australia vs. U.S.; Brazil vs. Portugal) are still influenced by sample differences in third variables that are often confounded with nation. For instance, a public notary typically enjoys a much higher social status in Europe than in the U.S., and therefore comparing public notaries across countries may yield differences associated with third variable effects rather than with culture. We further elaborate on this issue on the section entitled “Sample Equivalence Between Cultures” included later in this chapter.

(3)Use bilinguals in within-participant comparisons: multilinguals may differ from monolinguals both culturally and linguistically.

Research designs that analyze/control language effects (Continued)

23

Use bilinguals in within-participant comparisons

Bilingualism vs. Biculturalism of Translators.

Although translators may be linguistically competent, their ability to ascertain cultural nuances in the manifestation of stress across cultures may be limited.

Translators should be carefully selected according to not only their linguistic competence, but also the extent to which they are truly “bicultural” individuals familiar with the subtleties inherent in the ways in which individuals in the two cultures express their attitudes and emotions.

A linguistically imperfect translation may provide better psychological equivalence than a linguistically perfect one. Example:

The Spanish term “de pronto,” whose linguistically correct English translation will be “all of a sudden,” is often used in some Latin American countries located in the Andean cordillera to mean “perhaps” or “maybe.” A linguistically competent Spanish-English translator may fail to realize this colloquial usage of the term, hence rendering a technically correct but non-equivalent translation of an item including this term.

24

Check for cultural accommodation or “Whorfian” responding (Bond & Yang, 1982) -> does the same bilingual respondent give different answers to the two versions of the same measure?

Consider order effects in the presentation of the two versions of the same measure. Consider counterbalancing and/or analyzing order of language administration.

Considering controlling (or better yet measuring) language proficiency amongst bilinguals.

Use bilinguals in within-participant comparisons (Continued)

25

Even if the items are successfully translated linguistically, there exists the possibility that the individual items don’t do a good job of reflecting the construct universally –ethnocentric scale development.

A procedure to help eliminate this problem is to enlist researchers in multiple countries to have input into scale development from the beginning.

Multi-Cultural Scale Development Procedures

26

The team goes through a multi-stage procedure for scale development that is more complex than the typical process (Spector, Sanchez, Siu, Salgado, & Ma, 2004):

(1)a clear definition of the construct of interest is discussed and written so everyone has a similar understanding of what the items should reflect.

(2)each team member independently writes a set of scale items.

(3)the items are mixed up and compiled into a questionnaire that is administered to a sample of subjects. This can be done in multiple countries. Item and factor analysis are applied to select the final items.

(4)validation studies should be conducted in multiple countries, although it should be kept in mind that relationships between the construct of interest and other constructs might vary across culturally dissimilar countries.

Multi-Cultural Scale Development Procedures (Continued)

27

One problem in much cross‑cultural stress research is the availability of equivalent participant samples:

We often compare individuals who vary, not only in country, but in demographics such as occupation, income (relative to society) and status. Even within the same occupation, differences in work conditions can masquerade as cultural differences. Suppose we compare physicians in the U.S. versus those in a country with socialized medicine. If we find the Americans are more highly stressed, should we conclude that the U.S. is a more stress-producing culture?

Clearly, with so many potential third-variable effects, we need to be extremely careful to match subjects on as many relevant variables as possible, so that we are able to attribute differences to culture.

Sample Equivalence Between Cultures

28

However, matching participants in all potential third-variables so that one ends up with comparable samples across countries is far-fetched because of wide variation across cultures. For researchers who study employed populations, economic and social factors can make it difficult to identify equivalent samples. Example:In contrast to North America, there are countries where large private corporations are rare. In other countries, state-managed enterprises dominate the economic landscape and, therefore, even matching for industry sector will not rid comparisons of selection bias, because organizations in the same sector (e.g., oil production and distribution) will still differ in meaningful ways like whether they are private or state-run enterprises. Even those who study college students might find differences across countries. In the U.S. a far greater proportion of the general population attends college than in countries where college attendance tends to be limited to the upper echelons of society.

->Thus, statistically controlling for factors representing rival explanations seems necessary even in the best of matched samples.

Sample Equivalence Between Cultures (Continued)

29

One should try to match on variables likely to cause confounded results.

Because matching is often unfeasible ->gather as much information as possible about the samples; this info can be used later on as controls.

We should carefully examine economic (e.g., relative standard of living, economic system, and tax structure), political, and social factors. For research conducted in organizations, data on industry sector, organizational structure, and power structure should be collected. Example:Have samples of American physicians from different settings, such as private practice versus public health facilities. If the Americans, regardless of setting, score higher on the stressor measures than the Chinese, one gains confidence that results might be due to culture. As one compares more and more occupations and continues to find the same results, one begins to establish a pattern of cultural differences.

Sample Equivalence Between Cultures (Continued)

30

In choosing control variables, researchers should be careful about operationalizing such measures in terms of subjective appraisals.

Example 1: A majority of measures of social support rely on subjective appraisals of received support (Viswesvaran, Sanchez, & Fisher, 1999). Reports of social support are influenced by the extent to which societies are collectivistic vs. individualistic (Triandis, 1994) and, therefore, matching samples on the extent to which individuals feel that they had support from supervisors, co-workers, or even family members ignores variations on what various cultures would consider an adequate level of support.

Example 2: Appraisals of perceived workload are likely to differ as a function of culture:

No. of work hrs. Perceived workload

culture

Sample Equivalence Between Cultures (Continued)

31

Conclusion

Globalization will continue to make cross‑cultural comparisons more important than ever…

… and yet total measurement equivalence between instruments employed in cross-cultural comparisons may not be possible (Byrne & Watkins, 2003).

Research procedures and measurement tools that minimize measurement and sampling error in cross‑cultural comparisons are needed to increase our understanding of cultural nuances, which should increase the cross-cultural generalizability of our theories.

32

Thank you!

Contact info:Juan I. Sanchez, Ph.D.Professor and Knight-Ridder Byron Harless Eminent Chair in ManagementDepartment of Management and International BusinessFlorida International UniversityUniversity ParkMiami, FL 33199(305) 348-3307sanchezj@fiu.edu