beyond skin deep: investigating the “who” of the sensitivity review james grand juliya...

BEYOND SKIN DEEP: INVESTIGATING THE “WHO” OF THE SENSITIVITY REVIEW

James GrandJuliya Golubovich

Ann Marie RyanNeal Schmitt

Society for Industrial & Organizational PsychologyAtlanta, GA

April 8, 2010

2

• Both here and in available sources, we have...– Examined why reviews are important– Investigated what current practices are– Discussed how to develop/approach reviews– Considered when to conduct reviews

• Answers to two important questions are missing:– Who should conduct reviews?– Do reviews make a difference on test performance?

What’s Left to Know About Sensitivity Reviews?

<< Main Focus

<< In Progress

3

Current Considerations of the “Who”

• Most commonly observed procedures/suggestions– Formally train reviewers

» Survey of current practices suggests this is rare (33% of reviewers)

– Minority review strategy (Camilli, 1993; Office for Minority Education, 1980)

» Review panel representative of diverse ethnic, gender, and geographic backgrounds (e.g., ACT, 2006; ETS, 2009)

» “Individuals sensitive to differences in cultural groups” (Reckase, 1996)

4

• Assumptions– Primary:

» Demographic representativeness increases likelihood of identifying material biased or offensive towards members of one’s subgroup

– Secondary: » Minorities better calibrated to identify problematic item content as a result of life

experiences» Sociopolitical and legal importance of diverse review panel

A Closer Look atMinority Review Strategies

“Minority review panels...serve important political and social justice goals. For these purposes, it is more appropriate that members of the [sensitivity review] panel represent constituencies pertinent to the use of the test rather than experts on the trait measured by the test.” (cited in Hood & Parker, 1989)

>

5

Evidence of Effectiveness of the Minority Review?

• Currently no evidence exists to directly support whether the demographics of reviewers influence sensitivity judgments– Engelhard, Hansche & Rutledge (1990)

» Neither Black nor White reviewers able to correctly identify which items in a test produce DIF at better than chance levels

» Furthermore, Black reviewers did not identify more biased items nor were they more confident in their judgments than White reviewers

• Individual differences likely affect judgments made by sensitivity reviewers, but has not been examined (Engelhard et al., 1990, Engelhard, Davis, & Hansche, 1999; Plake, 1980; Ramsey, 1993)

6

Research Questions

1. Do demographics influence sensitivity review judgments?

2. What individual differences influence sensitivity review judgments?

7

Hypothesis 6

Individuals who are more socially dominant and more strongly endorse status legitimacy beliefs will be less likely to detect insensitivity

Hypothesis 5

Individuals with greater cultural intelligence, better perspective taking ability, and greater empathic concern will be more likely to detect insensitivity

Hypothesis 4

The effects of gender and ethnicity on the detection of insensitivity will be moderated by gender and ethnic identification

Hypothesis 3

Stigma consciousness, perceived attributions to prejudice, and past experience with discrimination will mediate the relationship between reviewer demographics and detection of insensitivity

Hypothesis 2

Individuals who are more conscious of stigmas, have a greater tendency to make attributions to prejudice, and have had more experiences with discrimination will be more likely to detect insensitivity

Hypothesis 1

Females and non-Whites will be more likely to detect insensitivity than males and Whites

Research Model and Hypotheses

Stereotype-related Characteristics• Gender stigma consciousness• Ethnic stigma consciousness• Perceived attributions to prejudice• Past experience with discrimination

Psychological Characteristics

• Cultural intelligence• Perspective taking• Empathic concern

H1

Gender/Ethnicity

Detection of Insensitivity

H2H3

H4

H5

H6

Gender/Ethnic Identification

• Social dominance orientation• Status legitimacy beliefs


Psychological Characteristics

• Cultural intelligence• Perspective taking• Empathic concern

H1

Gender/Ethnicity


H2H3

H4

H5

H6


• Social dominance orientation• Status legitimacy beliefs

Demographics

Individual Difference Measures

8

Methods• n = 303 student reviewers (26% males, 15% non-White)

recruited to conduct a sensitivity review on a test of general knowledge– Demographics & individual differences collected online prior to

session– Provided brief instructional period on sensitivity reviews prior to

review task– Ratings collected on a 108-item test

9

Review Task: Test Breakdown

Category # Example Item Stem or Response Options

Non-problematic 54 The coach worked long and hard into the night for preparing the team’s strategy for the next game.

Offensive content 7 a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’

Offensive language 9 Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa.

Emotionally provocative content

11 a. Many single women with children choose not to apply for welfareb. According to a national survey, single childless women choose not to have

children because they lack monetary resources.

Portrayal of gender/ racial stereotypes

7 Grace Hopper should be an inspiration to female workers everywhere; not only did she prove that a woman could be highly successful in a field dominated by men, and she was able to do so without special treatment or excessive pleas for equality.

Unequal referrals to men and women

7 The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful.

Vocabulary unfamiliar to a group

7 In order to _________ a mortgage, an individual should periodically pay his or her lender principal and interest.

Content unfamiliar to a group

6 In India, seeing an elephant when one is leaving for a journey is considered ______ because an elephant represents Lord Ganesha, the Indian God who _______ obstacles.

Prob

lem

atic

Item

Cat

egor

ies

(54

item

s)

10

3.7

2.3

Calculating Detection of Insensitivity

Highly Insensitive Moderately Insensitive Possibly Insensitive Not Problematic

1 2 3 4

14Writing a good twenty-page research paper is more difficult than when you have to write two good ten-page papers.

03Those students who sit through her lectures day after day, having been numbed into thinking that history could never be even remotely interesting.

14The coach worked long and hard into the night for preparing the team’s strategy for the next game.

Non

-Pro

blem

atic

11Some religious officials claim that the ancient Egyptian’s history of brutal violence, ritual sacrifices, and worship of non-Christian deities has contributed to the ________ of bloody genocide ravaging Eastern Africa.

04The temperaments of both architects were markedly different; Kevin was reserved and courteous, Joe was ________ and boastful.

4

12a. Some whites believing it’s fashionable to be Indian, stretch the truth about their ancestry, claiming, ‘My grandmother was a Cherokee princess.’

Prob

lem

atic

Accuracy ScoreCorrect?Mean Item

Type RatingRatingItemItem Type

Correctly identified if rating = 1-3

Incorrectly identified if rating = 4

Correctly identified if rating = 4

Incorrectly identified if rating = 1-3

Manipulation Check:Significant difference (d = 1.49, p < .001) between problematic item ratings (M = 3.23) & non-problematic item ratings (M = 3.71)

11

• Hypothesis 1:– No main effect for Gender in ratings or accuracy– Significant main effects for ethnicity, but...

» Non-Whites tended to rate non-problematic items as slightly more insensitive than Whites (d = .19, p < .05) and were less accurate in their judgments overall (d = .36, p < .05)

– Significant interaction (p < .05) between Gender & Ethnicity on problematic item ratings

Results


Gender

Ethnicity

ns

*

Non-white females = White males(Inconsistent with hypothesis)

12

Results• Hypothesis 2:– Individuals more aware of ethnic stigmas tended to rate

problematic items as more insensitive (β = -.19, p < .01)...– ...however, these individuals also tended to rate non-

problematic items as more insensitive (β = -.21, p < .01)


Detection of Insensitivity*

13

• Hypothesis 3:

– Non-Whites tended to exhibit greater awareness of ethnic stigmas, which resulted in a tendency to view both problematic and non-problematic items as more insensitive

Results

Ethnicity Ethnic Stigma Consciousness

Problematic item ratings

Non-problematic item ratings

β = -.24β = -.19

β = -.21***

14

Results• Hypothesis 4:

– Gender/ethnic identification did not moderate the relationship between demographics and sensitivity judgments

Gender/Ethnicity



ns

15

Results• Hypothesis 5:• Hypothesis 6:

– More socially dominant (β = .14, p < .05) and status legitimizing (β = .11, p = .06) individuals tended to rate problematic items as less insensitive

– More socially dominant individuals also tended to produce less accurate sensitivity judgments (β = -.12, p < .05)

Psychological Characteristics• Cultural intelligence• Perspective taking• Empathic concern Detection of

Insensitivity

H5

H6• Social dominance orientation• Status legitimacy beliefs

ns

*

16

Summary of Findings• No support obtained for minority review strategy

– Non-Whites appeared to perceive all items as generally more problematic

• Stereotype-related and psychological individual difference variables explained meaningful variance in sensitivity ratings– Part of the reason non-Whites tended to be slightly over-sensitive to

item content was attributable to their greater awareness of ethnic stigmas

– Individuals who believed in the legitimacy of social hierarchies and the need to earn one’s status in society were less sensitive to problematic content

17

Implications• Practical Implications

– Minority review strategy could potentially increase the rate of false alarms» Problematic as it lengthens development process and time-to-market

estimates– Diversity in the sensitivity review panel can still be beneficial to the

process (Hood & Parker, 1989), but it should not be the end goal

• Research Implications– Some individual difference predictors of sensitivity judgments were

identified, but many more failed– Continued investigation of psychological characteristics, dispositions,

and traits that influence the sensitivity review process

BEYOND SKIN DEEP: INVESTIGATING THE “WHO” OF THE SENSITIVITY REVIEW

James GrandJuliya Golubovich

Ann Marie RyanNeal Schmitt

Society for Industrial & Organizational PsychologyAtlanta, GA

April 8, 2010

beyond skin deep: investigating the “who” of the sensitivity review james grand juliya...

Documents