update of csr diversity initiatives
Post on 01-Jan-2016
39 Views
Preview:
DESCRIPTION
TRANSCRIPT
Update of CSR Diversity Initiatives
ACD Diversity Workgroup Subcommittee on Peer Review
Early Career Reviewer (ECR) Program
Monica A. Basco, Ph.D.
Scientific Review Officer
Coordinator, ECR Program
Exec Sec, Subcommittee on Peer Review
• African American applicants were 10 percentage points less likely to receive NIH research funding compared to Whites
• Suggested explanations:–Bias in peer review–Deficits in applicants’ grant writing abilities
• Applications with strong priority scores were equally likely to be funded regardless of race− This suggests that problems occur at the peer review stage
or earlier
Ginther et al. (2011) Findings
ACD Working Group on Diversity in the Biomedical Research Workforce met throughout 2012 and developed recommendations• Created a permanent Diversity Working Group• Established a Subcommittee on Peer Review to:
− Examine all hypotheses, including the role of unconscious bias, related to disparities in research awards at NIH.
− Provide advice on potential interventions to ensure the fairness of the peer review system.
− Eight scholars with expertise in social science, unconscious bias, stereotyping, and faculty development
NIH Response
• Developed in collaboration with the Subcommittee on Peer Review
• Contest format with cash prizes• Solicit ideas from scientific community and other stakeholders• Two stage judging process
–Technical Evaluation by Subcommittee Members–Selection of Winners by Federal Employees
• $10,000 First Prize• $5,000 Second Prize
America COMPETES Challenges
• How to detect bias among reviewers due to gender, race/ethnicity, institutional affiliation, area of science, and/or amount of research experience of applicants.
• Best empirically based idea should:–Be theoretically based and/or hypothesis driven–Propose an experimental design–Be well-grounded in peer reviewed empirical literature–Propose measurement methods–Be implementable–Be related to the Peer Review Process
Challenge #1: New Methods to Detect Bias in Peer Review
• Most Creative Submission should: − Propose novel concepts or translate existing concepts in a
novel way− Challenge existing paradigms− Have potential to be translated for use in an experimental
design− Describe creative ways to apply ideas− Be implementable− Relate to the Peer Review Process
Challenge #1: New Methods to Detect Bias in Peer Review
Best Submission should:− Demonstrate general knowledge of peer review practices− Be grounded in the empirical literature− Be implementable− Have potential to be delivered in a variety of formats− Demonstrate understanding of the training literature− Move theory to practice− Provide evidence that supports the effectiveness of the
approach
Challenge #2: Strategies to Strengthen Fairness and Impartiality in Peer Review
• Text Analysis of Unedited Reviewer Critiques• Word Count Methods• Lexicon of terms
–Achievement (awards, honors) –Ability (skill, ability) –Research (productivity, experiment) –Standout adjectives (exceptional, outstanding)–Grant evaluation words (groundbreaking, meritorious)
Detection of Bias in Reviewer Comments
• Task 1: Validation of Lexicon–Summary Statements–Applicant Gender differences–Applicant Race differences
• Task 2: Test of reviewer Comments by Applicant Race–Test original Lexicon–Revise and Reanalyze – Iterative process
• Task 3: After accumulation of critiques− Test of Reviewer and Applicant Race differences in
evaluation of grant applications
Text Analysis
• If Racial Bias in Grant Reviews Exists:–Reviewers are aware of applicant race/ethnicity–This knowledge influences their ratings of applications
− Anonymized applications from Black applicants will get better scores
Anonymization Experiments – Basic Assumptions
Application Formats
A. Original Application
B. Full Anonymization
C. Partial Anonymization
With Investigator and Institution Information
No applicant or institution information
provided
Applicant and institution information provided after review
of Abstract, Aims, Approach
Anonymizing Experiments – Design Options
Application Formats
A. Original Application
B. Full Anonymization
C. Partial Anonymization
Hypotheses:For White PIs – Differences between A & B are due to Error (i.e., inter-rater reliability)
For Black PIs - Differences are due to Error + Reviewer Bias
Anonymizing Experiments
Application Formats
A. Original Application
B. Full Anonymization
C. Partial Anonymization
If there are differences between A and B
Test for differences in scores between
Full and Partial Anonymization
Anonymizing Experiments
female male Black White Hispanic Non-Hispanic Total Sample0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Percent of ARRA Submissions Funded by Demographic Group
n=22,204 submissions
Re-Review of ARRA Grants
Bias Awareness
The following section includes examples of comments that could be made during review meetings. Please review the following and indicate how often you have heard similar comments in your merit review panels: Never Occasionally Very frequently 1 2 3 4 5
PI’s ethnicity: “Okay, I’m the lead reviewer for the proposal from [Name of Institution]. I can’t even begin
pronounce the PI’s name, just look at number .......”
Gender: e.g. “The PI is a talented young woman. She does a lot of committee work and spends a lot of
time mentoring her students. It is not clear that she is making research a priority.”
Faculty Rank: e.g. “The PI is a new assistant professor and the Co-PI is an associate professor. Shouldn’t
the PI be concentrating on getting tenure?”
Disciplinary Differences: e.g. “I don’t understand the way these educational psychologists write their
proposals. It needs more tables or charts for an engineer like me to be able to understand what they are
trying to say.”
Reputation: e.g. “I would have expected more out of this PI; their work is always great, even though this
proposal is less than their usual effort, I think we should give it a chance.”
Bias Awareness Questionnaire
Mentioning the name of the applicant Mentioning the title of the application Noting that the applicant had
recently completed her K01 training Noting that several of her
publications included her mentor as a co-author
Noting that she has created her own research niche
Voicing concern about her independence
Bringing up the recent article on incarceration among Black male study participants
Dr. Lakeisha Tubman, is a New Investigator from a Historically Black College. In the discussion of her application entitled, “Neuroscience of Engagement of Black Males in Hypertension Management,” a reviewer noted that she has recently completed her K01 training and several of her publications included her mentor as a co-author. While she has created her own research niche, there was some concern voiced about her independence. Another reviewer pointed to a recent article on how incarceration among Black male research participants causes problems with attrition and made the point that these findings raised concerns about the feasibility of the proposed work.
In what way do you think each of the following aspects of this discussion affected reviewers’ scores?
-3 -2 -1 0 +1 +2 +3 Extremely Moderately Somewhat Neutral Somewhat Moderately Extremely Negative Negative Negative Positive Positive Positive Effect Effect Effect Effect Effect Effect
Mentioning that she is from a high research intensive institution
Noting that the applicant has a gap in her publication history
Noting that she has prior NIH funding The Chair’s joke about her
knowledge of the subject matter Making the committee aware of her
recent pregnancy Defense of her two year gap given
the nature of her research Discussion of the high quality work
coming from her department
At a recent study section meeting, an application submitted by Dr. Joanna Fulbright entitled, “Epidemiology of Breast Cancer and Advanced Maternal Age,” was given an average preliminary score of 2. Although she is from a high research intensive university and has had prior NIH funding, her biosketch showed a 2 year gap in her recent publication history. The Chair, who had seen the pregnant applicant present her work at a recent conference, made a good hearted joke about the applicant knowing her subject matter inside and out. During the discussion of her application, a reviewer commented that a two year gap was not unusual given the time it takes to complete epidemiological work. Another reviewer noted the high quality work coming out of the applicant’s department.
In what way do you think each of the following aspects of this discussion affected reviewers’ scores?
-3 -2 -1 0 +1 +2 +3 Extremely Moderately Somewhat Neutral Somewhat Moderately Extremely Negative Negative Negative Positive Positive Positive Effect Effect Effect Effect Effect Effect
Early Career Reviewer Program (ECR)
3134 ECRs have been accepted into the program
1069 have served on at least one study section
30 percent of those who have served are from under-represented groups
Early Career Reviewer Application and Vetting System (EAVS)
ECR video created and disseminated
Outreach webinars for R15 schools
Intervention Development and Testing
top related