Update of CSR Diversity Initiatives

ACD Diversity Workgroup Subcommittee on Peer Review

Early Career Reviewer (ECR) Program

Monica A. Basco, Ph.D.

Scientific Review Officer

Coordinator, ECR Program

Exec Sec, Subcommittee on Peer Review

• African American applicants were 10 percentage points less likely to receive NIH research funding compared to Whites

• Suggested explanations:–Bias in peer review–Deficits in applicants’ grant writing abilities

• Applications with strong priority scores were equally likely to be funded regardless of race− This suggests that problems occur at the peer review stage

or earlier

Ginther et al. (2011) Findings

ACD Working Group on Diversity in the Biomedical Research Workforce met throughout 2012 and developed recommendations• Created a permanent Diversity Working Group• Established a Subcommittee on Peer Review to:

− Examine all hypotheses, including the role of unconscious bias, related to disparities in research awards at NIH.

− Provide advice on potential interventions to ensure the fairness of the peer review system.

− Eight scholars with expertise in social science, unconscious bias, stereotyping, and faculty development

NIH Response

• Developed in collaboration with the Subcommittee on Peer Review

• Contest format with cash prizes• Solicit ideas from scientific community and other stakeholders• Two stage judging process

–Technical Evaluation by Subcommittee Members–Selection of Winners by Federal Employees

• $10,000 First Prize• $5,000 Second Prize

America COMPETES Challenges

• How to detect bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.

• Best empirically based idea should:–Be theoretically based and/or hypothesis driven–Propose an experimental design–Be well-grounded in peer reviewed empirical literature–Propose measurement methods–Be implementable–Be related to the Peer Review Process

Challenge #1: New Methods to Detect Bias in Peer Review

• Most Creative Submission should: − Propose novel concepts or translate existing concepts in a

novel way− Challenge existing paradigms− Have potential to be translated for use in an experimental

design− Describe creative ways to apply ideas− Be implementable− Relate to the Peer Review Process

Challenge #1: New Methods to Detect Bias in Peer Review

Best Submission should:− Demonstrate general knowledge of peer review practices− Be grounded in the empirical literature− Be implementable− Have potential to be delivered in a variety of formats− Demonstrate understanding of the training literature− Move theory to practice− Provide evidence that supports the effectiveness of the

approach

Challenge #2: Strategies to Strengthen Fairness and Impartiality in Peer Review

• Text Analysis of Unedited Reviewer Critiques• Word Count Methods• Lexicon of terms

–Achievement (awards, honors) –Ability (skill, ability) –Research (productivity, experiment) –Standout adjectives (exceptional, outstanding)–Grant evaluation words (groundbreaking, meritorious)

Detection of Bias in Reviewer Comments

• Task 1: Validation of Lexicon–Summary Statements–Applicant Gender differences–Applicant Race differences

• Task 2: Test of reviewer Comments by Applicant Race–Test original Lexicon–Revise and Reanalyze – Iterative process

• Task 3: After accumulation of critiques− Test of Reviewer and Applicant Race differences in

evaluation of grant applications

Text Analysis

• If Racial Bias in Grant Reviews Exists:–Reviewers are aware of applicant race/ethnicity–This knowledge influences their ratings of applications

− Anonymized applications from Black applicants will get better scores

Anonymization Experiments – Basic Assumptions

Application Formats

A. Original Application

B. Full Anonymization

C. Partial Anonymization

With Investigator and Institution Information

No applicant or institution information

provided

Applicant and institution information provided after review

of Abstract, Aims, Approach

Anonymizing Experiments – Design Options

Application Formats

A. Original Application

B. Full Anonymization

C. Partial Anonymization

Hypotheses:For White PIs – Differences between A & B are due to Error (i.e., inter-rater reliability)

For Black PIs - Differences are due to Error + Reviewer Bias

Anonymizing Experiments

Application Formats

A. Original Application

B. Full Anonymization

C. Partial Anonymization

If there are differences between A and B

Test for differences in scores between

Full and Partial Anonymization

Anonymizing Experiments

female male Black White Hispanic Non-Hispanic Total Sample0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Percent of ARRA Submissions Funded by Demographic Group

n=22,204 submissions

Re-Review of ARRA Grants

Bias Awareness

The following section includes examples of comments that could be made during review meetings. Please review the following and indicate how often you have heard similar comments in your merit review panels:   Never Occasionally Very frequently 1 2 3 4 5

PI’s ethnicity: “Okay, I’m the lead reviewer for the proposal from [Name of Institution]. I can’t even begin 

pronounce the PI’s name, just look at number .......”

Gender: e.g. “The PI is a talented young woman.  She does a lot of committee work and spends a lot of 

time mentoring her students.  It is not clear that she is making research a priority.”

Faculty Rank: e.g. “The PI is a new assistant professor and the Co-PI is an associate professor. Shouldn’t 

the PI be concentrating on getting tenure?”

Disciplinary Differences: e.g. “I don’t understand the way these educational psychologists write their 

proposals. It needs more tables or charts for an engineer like me to be able to understand what they are 

trying to say.”

Reputation: e.g. “I would have expected more out of this PI; their work is always great, even though this 

proposal is less than their usual effort, I think we should give it a chance.”

Bias Awareness Questionnaire

Mentioning the name of the applicant Mentioning the title of the application Noting that the applicant had

recently completed her K01 training Noting that several of her

publications included her mentor as a co-author

Noting that she has created her own research niche

Voicing concern about her independence

Bringing up the recent article on incarceration among Black male study participants

Dr. Lakeisha Tubman, is a New Investigator from a Historically Black College. In the discussion of her application entitled, “Neuroscience of Engagement of Black Males in Hypertension Management,” a reviewer noted that she has recently completed her K01 training and several of her publications included her mentor as a co-author. While she has created her own research niche, there was some concern voiced about her independence. Another reviewer pointed to a recent article on how incarceration among Black male research participants causes problems with attrition and made the point that these findings raised concerns about the feasibility of the proposed work.

In what way do you think each of the following aspects of this discussion affected reviewers’ scores?

-3 -2 -1 0 +1 +2 +3 Extremely Moderately Somewhat Neutral Somewhat Moderately Extremely Negative Negative Negative Positive Positive Positive Effect Effect Effect Effect Effect Effect

Mentioning that she is from a high research intensive institution

Noting that the applicant has a gap in her publication history

Noting that she has prior NIH funding The Chair’s joke about her

knowledge of the subject matter Making the committee aware of her

recent pregnancy Defense of her two year gap given

the nature of her research Discussion of the high quality work

coming from her department

At a recent study section meeting, an application submitted by Dr. Joanna Fulbright entitled, “Epidemiology of Breast Cancer and Advanced Maternal Age,” was given an average preliminary score of 2. Although she is from a high research intensive university and has had prior NIH funding, her biosketch showed a 2 year gap in her recent publication history. The Chair, who had seen the pregnant applicant present her work at a recent conference, made a good hearted joke about the applicant knowing her subject matter inside and out. During the discussion of her application, a reviewer commented that a two year gap was not unusual given the time it takes to complete epidemiological work. Another reviewer noted the high quality work coming out of the applicant’s department.

In what way do you think each of the following aspects of this discussion affected reviewers’ scores?

-3 -2 -1 0 +1 +2 +3 Extremely Moderately Somewhat Neutral Somewhat Moderately Extremely Negative Negative Negative Positive Positive Positive Effect Effect Effect Effect Effect Effect

Early Career Reviewer Program (ECR)

3134 ECRs have been accepted into the program

1069 have served on at least one study section

30 percent of those who have served are from under-represented groups

Early Career Reviewer Application and Vetting System (EAVS)

ECR video created and disseminated

Outreach webinars for R15 schools

Intervention Development and Testing