peer review at nih - albany medical center - nih workshop part ii - peerrevi… · peer review at...
TRANSCRIPT
PEER REVIEW AT NIH
B. Duane Price, Ph.D.
Senior Scientific Review Officer
Immunology Review Branch
National Institute of Allergy and Infectious Diseases (NIAID)
National Institutes of Health (NIH)
15th Annual Upstate New York Immunology Conference
October 22, 2012
Presentation Outline
I. Overview
II. First Level of Review - Scientific Review Groups
A. Peer Review Roles and Meeting Overview
B. Peer Review Criteria and Considerations
C. Scoring
D. Summary Statement
E. Appeals
V. Second Level of Review - Advisory Council/Board
IV. Post-Review
V. Resources
2 http://grants.nih.gov/grants/peer_review_process.htm
I. Overview
NIH policy is intended to ensure that grant
applications submitted to the NIH are evaluated
on the basis of a process that is fair, equitable,
timely, and free of bias.
6
I. Overview
The NIH dual peer review system is mandated by
statute in accordance with section 492 of the
Public Health Service Act and federal regulations
governing "Scientific Peer Review of Research
Grant Applications and Research and
Development Contract Projects" (42 CFR Part
52h).
7
I. Overview (continued)
9
The first level of review is
carried out by a Scientific
Review Group (SRG)
composed primarily of
non-federal scientists who
have expertise in relevant
scientific disciplines and
current research areas.
I. Overview (continued)
10
The second level of review is
performed by Institute and Center
(IC) National Advisory Councils or
Boards. Councils are composed
of both scientific and public
representatives chosen for their
expertise, interest, or activity in
matters related to health and
disease.
I. Overview (continued)
11
Only applications that are favorably recommended
by both the SRG and the Advisory Council may be
recommended for funding.
http://public.csr.nih.gov/Pages/default.aspx
II. First Level of Review – SRG at the Center for Scientific Review
12
http://www.niaid.nih.gov/Pages/default.aspx
II. First Level of Review – SRG at the Funding Institute/Center
13
http://grants.nih.gov/grants/guide/pa-files/PA-11-260.html
II. First Level of Review – Locus of Review Specified in the FOA
14
https://www.federalregister.gov/
II. First Level of Review – All Reviews Announced in the Federal Register
15
II. First Level of Review – Scientific Review Groups
16
Initial peer review meetings are administered by either the
Center for Scientific Review (CSR) or another NIH IC.
The focus of review is specified in the Funding Opportunity
Announcement (FOA).
Peer review meetings are announced in the Federal
Register.
The meetings are closed to the public, although some
meetings may have an open session; the Federal Register
provides the details of each meeting.
II A. Peer Review Roles - SRO
Scientific Review Officer:
Each SRG is led by a Scientific Review Officer (SRO).
The SRO is an extramural staff scientist and the
Designated Federal Official responsible for ensuring that
each application receives an objective and fair initial peer
review, and that all applicable laws, regulations, and
policies are followed.
19
II A. Peer Review Roles - SRO (cont.)
SROs:
Analyze the content and check for completeness.
Document and manage conflicts of interest.
Recruit qualified reviewers.
Assign applications to reviewers.
Attend and oversee administrative and regulatory aspects
of peer review meetings.
Prepare summary statements.
20
II A. SRO manages Conflict of Interest
http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-120.html
Conflict of Interest (COI):
COI in scientific peer review exists when a reviewer has an
interest in an application that is likely to bias his or her
evaluation of it. A reviewer who has a real or apparent COI
with an application, as defined in 42 CFR Part 52h, may not
participate in its review unless a deviation is granted by the
NIH Deputy Director for Extramural Research.
22
II A. SRO manages the appearance of Conflict of Interest
http://grants.nih.gov/grants/guide/notice-files/NOT-OD-11-120.html
Conflict of Interest (COI) - Apparent:
The appearance of COI occurs when a reviewer or close
relative or professional associate of the reviewer has a
financial or other interest in an application that is known to
the reviewer or the government official managing the review,
and this circumstance would cause a reasonable person to
question the reviewer's impartiality if he or she were to
participate in the review.
23
II A. SRO recruits and assigns reviewers
25
SROs recruit qualified reviewers based on scientific and
technical qualifications and other considerations, including:
► Authority in their scientific field
► Dedication to high quality, fair, and objective reviews
► Ability to work collegially in a group setting
► Experience in research grant review
► Balanced representation
II A. Peer Review Roles - Chairperson
Chair:
Serves as moderator of the discussion of scientific and
technical merit of the applications under review.
Is also a peer reviewer for the meeting.
27
II A. Peer Review Roles - Reviewers
Reviewers:
Declare Conflicts of Interest.
Receive access to the grant applications approximately six
weeks prior to the peer review meeting.
Prepare a written critique for each application assigned.
Assign a numerical score to each scored review criterion.
Make recommendations concerning the scientific and
technical merit of applications under review.
Make recommendations concerning additional review criteria,
including protections for human subjects.
Make recommendations concerning budget requests.
29
II A. Peer Review Roles – Other NIH Staff
30
Program Officer (PO)
Grants Management Specialist (GMS)
Support Staff
II A. Peer Review Roles – Other NIH Staff
Other NIH Staff
Federal officials who have need-to-know or pertinent
related responsibilities are permitted to attend closed
review meetings.
NIH IC or other federal staff members wishing to attend an
SRG meeting must have advance approval from the
responsible SRO. These individuals may provide
programmatic or grants management input at the SRO's
discretion.
31
II A. Peer Review Meeting Overview
Peer Review Meeting Procedures:
Applications are reviewed based on established review
criteria.
Assigned reviewers summarize their prepared critiques for
the group.
An open discussion follows.
Final scoring of overall impact scores is conducted by
private ballot.
33
II B. Peer Review Criteria and Considerations
Review Criteria for Grants and Cooperative Agreements:
Overall Impact
► Scored Review Criteria
► Additional Review Criteria
Additional Review Considerations
34
II B. Peer Review Criteria and Considerations
35
Scored Review Criteria:
Significance
Investigator(s)
Innovation
Approach
Environment
II B. Peer Review Criteria and Considerations
Additional Review Criteria:
Protections for Human Subjects
Inclusion of Women, Minorities, and Children
Vertebrate Animals
Biohazards
Resubmission
Renewal
Revision
36
II B. Peer Review Criteria and Considerations
Additional Review Considerations:
Applications from Foreign Organizations
Select Agent
Resource Sharing Plans
Budget and Period Support
37
The scoring system utilizes a 9-point rating scale
(1 = exceptional; 9 = poor).
Score Descriptor Additional Guidance on Strengths/Weaknesses
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
4 Etc…
38
II C. Scoring
II. C. Peer Review Scoring Chart
Impact Score Descriptor Additional Guidance on
Strengths/Weaknesses
High
1 Exceptional Exceptionally strong with essentially no
weaknesses
2 Outstanding Extremely strong with negligible
weaknesses
3 Excellent Very strong with only some minor
weaknesses
Moderate
4 Very Good Strong but with numerous minor
weaknesses
5 Good Strong but with at least one moderate
weakness
6 Satisfactory Some strengths but also some
moderate weaknesses
Low
7 Fair Some strengths but with at least one
major weakness
8 Marginal A few strengths and a few major
weaknesses
9 Poor Very few strengths and numerous major
weaknesses Minor: An easily addressable weakness that does not substantially lessen the impact of the project
Moderate: A weakness that lessens the impact of the project
Major: A weakness that severely limits the impact of the project
Non-numeric score options: NR = Not Recommended for Further Consideration
DF = Deferred, AB – Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed
http://enhancing-peer-review.nih.gov/scoring&reviewchanges.html
40
II C. Scoring (cont.)
Before the SRG meeting, each reviewer and discussant
assigned to an application gives a separate score for each
of the five review criteria (i.e., Significance, Investigator(s),
Innovation, Approach, and Environment).
For all applications, even those not discussed by the full
committee, the individual scores of the assigned reviewers
and discussant(s) for these criteria are reported to the
applicant.
41
III C. Scoring (cont.)
In addition, each reviewer and discussant assigned to an
application gives a preliminary overall impact score for that
application.
The preliminary scores are used to determine which
applications will be discussed in full.
42
II C. Scoring (cont.)
For each application that is discussed at the meeting, a
final impact score is given by each eligible committee
member (without COI) including the assigned reviewers.
Each member's score reflects his/her evaluation of the
overall impact that the project is likely to have on the
research field(s) involved, rather than being a calculation
of the reviewer's scores for each criterion.
Following the discussion, all reviewers rate the overall impact from 1 to 9.
Scores are averaged and rounded to create the overall impact score.
Example: final overall impact score calculation
(1+1+1+2+2+2+3+3+3)/9 * 10 = 20
43
II C. Scoring (cont.)
44
II C. Scoring (cont.)
The final overall impact scores range from 10
(high impact) through 90 (low impact).
The final overall impact score is reported on the
summary statement.
Numerical impact scores are not reported for
applications that are not discussed (ND), which
may be reported as ++ on the face page of the
summary statement.
1.Percentiles are determined by matching an application's overall impact
score against a table of relative rankings containing all scores of
applications assigned to a study section during the three last review
cycles.
NIH calculates percentiles using the following formula:
Percentile = 100/Number of Applications x (relative rank minus 0.5)
(The 0.5 percent is a standard mathematical procedure used for rounding.)
2.These numbers are then rounded up, e.g., a percentile of 10.1
becomes 11, to create a whole number percentile ranging from 1 to 99.
NIH includes not discussed applications in the percentile calculation.
http://www.niaid.nih.gov/researchfunding/grant/strategy/pages/7payline.aspx#c
46
II C. Scoring - Percentile
Percentile Rank:
The percentile rank is based on a ranking of the impact
scores assigned by a peer review committee.
The percentile rank is normally calculated by ordering the
impact score of a particular application against the impact
scores of all applications reviewed in the current and the
preceding two review rounds.
An application that was ranked in the 5th percentile is
considered more meritorious than 95% of the applications
reviewed by that committee.
47
II C. Scoring – Percentile (cont.)
Summary Statement (SS):
The Summary Statement is a combination of the reviewers'
written comments and the SRO's summary of the members'
discussion during the study section meeting.
It includes the recommendations of the study section, a
recommended budget, and administrative notes of special
considerations.
48
II D. Summary Statement
http://www.niaid.nih.gov/researchfunding/grant/Documents/ParrishSS.pdf
II D. Summary Statement (cont.)
49
An appeal… describes a flaw in the review
process for a particular application. It must
display concurrence of the Authorized
Organization Representative (AOR).
56
II D. Appeals
The letter 1) describes a flaw(s) in the review process…, 2)
explains the reasons for the appeal, and 3) is based on one or
more of the following issues…:
Evidence of bias on the part of one or more peer reviewers
COI, on the part of one or more non-federal peer reviewers
Lack of appropriate expertise within the SRG
Factual error(s) made by one or more reviewers that could
have altered the outcome of review substantially
57
II D. Appeals
II D. Appeals (cont.)
The Council may concur:
with the appeal, and recommend that the application be
re-reviewed.
with the SRG’s recommendation and deny the appeal.
58
The Advisory Council/Board of the potential awarding IC
performs the second level of review. Advisory Councils/Boards
are composed of scientists from the extramural research
community and public representatives (NIH Federal Advisory
Committee Information). Members are chosen by the respective
IC and are approved by the Department of Health and Human
Services. For certain committees, members are appointed by the
President of the United States.
59
III. Second Level of Review - Advisory Council
NIH program staff members examine applications, their
overall impact scores, percentile rankings and their summary
statements and consider these against the IC's needs.
Program staff provide a grant-funding plan to the Advisory
Board/Council.
The Advisory Board/Council also considers the IC’s goals and
needs and advises the IC director.
The IC director makes final funding decisions based on staff
and Advisory Council/Board advice.
61
III. Recommendation Process
Fundable Score - Next Steps?
Some of the ICs publish paylines as part of their
funding strategies to guide applicants on their
likelihood of receiving funding. Application scores can
only be compared against the payline for the fiscal
year when the application will be considered for
funding, which is not necessarily the year when it was
submitted. There may be a delay of several months to
determine paylines at the beginning of fiscal years.
62
IV. Post-Review - Funded
http://grants.nih.gov/grants/next_steps.htm
Fundable Score - Next Steps?
If the application is assigned to an IC that does not announce
a payline, the Program Officer listed at the top of the summary
statement may be able to provide guidance on the likelihood of
funding. After the Advisory Council meeting, if an application
results in an award, the applicant will be working closely with
the program officer of the funding Institute or Center on
scientific and programmatic matters and a Grants
Management Officer on budgetary or administrative issues.
The Grants Management Specialist will contact the applicant
to collect information needed to prepare the award.
63
IV. Post-Review - Funded
http://grants.nih.gov/grants/next_steps.htm
Not Funded - Next Steps?
The NIH receives thousands of applications for each application
receipt round. Funding on the first attempt is difficult, but not
impossible. If an application does not result in funding, there
may be an opportunity to respond to the reviewers’ comments
and resubmit the application. Applicants just receiving their
summary statements, should contact the NIH Program Officer
listed at the top of the summary statement to discuss the
review. Also consult the Next Steps page (listed below) for
additional information.
64 http://grants.nih.gov/grants/next_steps.htm
IV. Post-Review – Not Funded
V. Resources
NIH Office of Extramural Research
► http://grants.nih.gov/grants/peer_review_process.htm#Overv
iew
NIH Center for Scientific Review
► http://public.csr.nih.gov/ApplicantResources/Pages/default.a
spx
NIH NIAID Peer Review
► http://www.niaid.nih.gov/researchfunding/grant/pages/review.
aspx
65
For More Information
B. Duane Price, Ph.D.
Senior Scientific Review Officer
Immunology Review Branch
Scientific Review Program
National Institute of Allergy and Infectious Diseases
National Institute of Health
Department of Health and Human Services
6700-B Rockledge Drive
Rm. 3139, MSC 7616
Bethesda, MD 20892-7616
Phone:(301) 451-2592 Fax: (301) 480-2408
CSR assigns applications
to study section (review)
& NIH Institute (funding)
3- 4 reviewers assigned to read and write critiques for each application
Summary Statement and
Priority Score
transmitted to applicant
(Commons) and NIH
extramural staff
Researcher writes and Institution submits application to NIH
Peer Review
Study section composed of 20-30 reviewers – review/discuss applications
*