challenggppes and opportunities in peer review · challenggppes and opportunities in peer review a...

Challenges and Opportunities in Peer Reviewg pp

A Vision for Ensuring Its Strategic National Valueg g

toni scarpa [email protected] scarpa [email protected]

Federal Demonstration PartnershipWashington, DCJanuary 25, 2011

National Institutes of HealthU.S. Department of Health and Human Services

Peer Review at CSR

CSR Activities

Fiscal Applications Applications Number of Number of Number of Year Received Reviewed Study Sections Reviewers SROs

2008 73,000 51,000 1,600 16,000 2402008 73,000 51,000 1,600 16,000 240

2009 113,000 78,000 2,200 39,000 240

2010 88,000 64,000 1,700 17,000 240

This is CSR

September 2009

The Drivers for Change

1st Driver: The NIH Budgeton

$, B

illi

2nd Driver: Number of Applications

100

120

80

100

ds

60

hous

and

40

T

20

01996 1998 2000 2002 2004 2006 2008 2010

R01 and R21 Received for CSR Review12000

10000

12000

8000

6000 R01R21

4000

0

2000

0Feb Jun Oct Feb Jun Oct Feb Jun Oct Feb Jun Oct Feb

2007 2008 2009 2010 2011

Application Received By Month of FY

2011 R i d

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

2009 Received

2010 Received

2011 Received

9472

11336

11323

7123

7509

10184

8636

9886

3706 8319

11290

10532

8866

22327

5209

13003 11903

12014

6521

5216

4650

4835

5961

2006 Received

2007 Received

2008 Received

8943

7928

9459

7052

7103

7304

7751

7764

7679

3677

9220

8736

8106

7924

7171

8166

4338

4070

3689

6018

8313

8211

7748

6238

6188

6886

3679

3516

4102

2866

2003 Received

2004 Received

2005 Received

7513

9341

8903

9038

6998

8001

3508

7417

7954

7677

9133

9018

8319

6841

8079

4499

4635

4250

3626

3556

8117

9239

8766

6532

6523

6627

3698

4083

3995 4290

0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000110,000

2001 Received

2002 Received

5194

6082

4997

5439

2640

3168

5320

6739

5706

6342

3218

3797

3849 5546

7034

5226

6352

2468

2995

Count of Applications

3rd Driver: Reviewer’s Load

12 00

14.00

10.00

12.00

Applications Per Reviewer6.00

8.00

Per Reviewer

2 00

4.00

0.00

2.00

4th Driver: CSR Budget70

50

60

40

50

llion

s

20

30

$ M

il

0

10

2004 2005 2006 2007 2008 20092004 2005 2006 2007 2008 2009

CSR Reviewer Cost Constant $

Cost of Peer Review including travel and small honorarium for 20 000Cost of Peer Review, including travel and small honorarium for 20,000Reviewers is 0.4-0.6% of the funds requested

Annual Savings in Reviewers’ Expense Budget

• Non-refundable tickets with one possible change$17 million$17 million

• 3,000 fewer reviewers$3 million$3 million

• 20% reviews using electronic platforms$6 million$

• One meeting a year on the West Coast$1.8 million

• Replacing CDs with zApp$ 1 million

Enhancing Peer Review

Major Complaints About NIH Peer Review

• The process is too slowp

• There are not enough senior/experienced reviewers

Th f di t bl h i t d f• The process favors predictable research instead of significant, innovative, or transformative research

• The time and effort required to write and review are a heavy burden on applicants and reviewers


1 Addressing Review and Funding for New Investigators1. Addressing Review and Funding for New Investigators

2. Reviewing Transformative Research

3 Funding Promising Research Earlier3. Funding Promising Research Earlier

4. Shortening the Review Time

5 Improving Study Sections Alignment with Science5. Improving Study Sections Alignment with Science

6. Recruiting and Retaining the Best Reviewers

7 Advancing Additional Review Platforms7. Advancing Additional Review Platforms

8. Focusing More on Impact and Significance

9 Saving Reviewer’ s Time and Effort9. Saving Reviewer s Time and Effort

10. Enhancing Peer Review Training

11 Continuously Reviewing the Changes11. Continuously Reviewing the Changes


1. Addressing Review and Funding for New Investigators

• Use different paylines for New Investigators and Early Stage Investigators (Only R01)g ( y )

• Cluster the reviews of New Investigator R01 applications so they are discussed togetherthey are discussed together

Funding New Investigators

Newand Experienced Investigators on R01 Equivalent Grants and New

45.0%6,000

sators

New and Experienced Investigators on R01 Equivalent Grants and New Investigators as a Percentage of All Competing R01 Awardees

(FY 1962 ‐ 2010) preliminary

30 0%

35.0%

40.0%

4 000

5,000

mpe

ting

Awarde

es

t Principal Investiga

20.0%

25.0%

30.0%

3,000

4,000

ercentage of All Co

ced R01 Equivalen t

10.0%

15.0%

1,000

2,000

nvestigators as a Pe

New

and

Experien

0.0%

5.0%

0

1962 1965 1968 1971 1974 1977 1980 1981 1984 1987 1992 1995 1998 2001 2004 2007 2010 est

New

I

Num

ber of

Fiscal Year

Established Investigators New Investigators Percent New

Funding Longevity of NIH Investigators12

10

ort

8

R01

Sup

po

4

6

Life

of N

IH

2

Hal

f L

0

Year of Initial R01 Support


2. Reviewing Transformative Research

• Editorial Board Review (3 Stages)Initial scoring based on innovation and potential science transformation by a small study section of distinguished, y y g ,broad-science reviewers (the editors)Specific science reviewed by appropriate reviewers (subject experts-the editorial board)(subject experts the editorial board)Final ranking by editors in face-to-face meeting


3. Shortening the Review Time

2005

2007

2009

0 2 4 6

Months: Submission to Posting Summary StatementsMonths: Submission to Posting Summary Statements

4 F di th t i i h li


70%

4. Funding the most promising research earlier

60%

70%

40%

50%

Aw

ards

20%

30%

of T

otal

10%

20%

Perc

ent

0%1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Funding the most promising research earlier

60%

70%

40%

50%

Aw

ards

30%

of T

otal

A

10%

20%

Perc

ent o

0%1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

P

5 Improving Study Section Alignment


5. Improving Study Section Alignment

• Input from the community

• Internal IRG reviews

• Open houses

• Advisory Committee

Positional Map of Musculoskeletal Tissue Engineering Study Section

Positional Map of Membrane Biology and Protein Processing Study Section

6 Recruiting the Best Reviewers


6. Recruiting the Best Reviewers

14000

16000

10000

12000

14000

6000

8000

10000

2000

4000

6000

0

000

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Chartered Temporary

Academic Rank of All CSR Reviewers6. Recruiting the Best ReviewersAcademic Rank of ALL CSR Reviewers

70.0%

80.0%

50 0%

60.0%

30 0%

40.0%

50.0%

20.0%

30.0%

0.0%

10.0%

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

PROFESSOR ASSOCIATE PROFESSOR ASSISTANT PROFESSOR

Recruiting the Best ReviewersSome Successful Strategies

• Move a meeting a year to the West Coast

Additi l i l tf• Additional review platforms

• Develop a national registry of volunteer reviewersSearchable database with 5,000 reviewers

• Provide tangible rewards for reviewersNo submission deadlines for chartered members of study sections (effective February 2008)

• Provide flexible time for reviewersChoice of 3 times/year for 4 years or 2 times/year for 6 yearsyears


7. Advancing Additional Review Platforms• Additional Review Platforms Help RecruitingAdditional Review Platforms Help Recruiting

Reviewers

• Electronic Review Modes Reduce Travel

• Electronic ReviewsTelephone Assisted MeetingVid A i t d M tiVideo Assisted MeetingInternet Assisted Meeting (previously AED)

Ad i Additi l R i Pl tfAdvancing Additional Review PlatformsWhat It Looks Like: Video Assisted Meeting

Telepresence Study Sections


8. Focusing More on Impact and Significance and Less on Approach

• Shorten Applications (13 or 7 pages instead of 25 or 12)• Scoring SignificanceScoring Significance• Discussed applications receive additional overall impact score• Training of Reviewers and Chairs


9. Saving Reviewers Time

• Shorter Applications• Bullet Critiques• Additional Review PlatformsAdditional Review Platforms

Template-Based Critiques

• The objective is to write evaluative statements and to avoid summarizing the application

• Comments should be in the form of bullet points or if Co e ts s ou d be t e o o bu et po ts onecessary short narratives

• The entire template is uploaded to IAR to become part of the summary statement. y

1. SignificanceSignificance Please limit text to ¼ pageStrengths•••Weaknesses••••

ScoringImpact Score DescriptorImpact Score Descriptor

1 Exceptional

High Impact 2 Outstanding

3 Excellent

Moderate Impact

4 Very Good

5 Goodp

6 Satisfactory

7 Fair

Low Impact

7 Fair

8 Marginal

9 P9 Poor

ScoringPriority Scores of R01 and R21 Reviewed by CSR

40%

50%

60%Pe

rcen

t

J 2008

10%

20%

30%

Cum

ulat

ive

P June 2008

60%

0%100 150 200 250 300 350 400

Priority Score

30%

40%

50%

60%

ve P

erce

nt

June 2009

0%

10%

20%

30%

Cum

ulat

iv

0%10 20 30 40 50 60 70 80 90

Priority Score

Scores of R01 and R21 Reviewed by CSR

60

70

50

60

cent

30

40

ulat

ive

Perc

Oct/Nov 2009 Meetings

20

30

Cum

u Oct/Nov 2009 Meetings

Feb/March 2010 Meetings

June/July 2010 Meetings

0

10

y g

010 20 30 40 50 60 70 80 90

Overall Impact Score

Order of Review

Why?• Concern of variation of scores during different times of the

meeting. gThe original plan was to recalibrate scores at the end of the meeting

Solution:n:• Recalibrate dynamically by discussing in order of average

preliminary scores from assigned reviewers.

Requirement:Requirement:• Reviewers must participate in entire meeting

10 Enhancing Peer Review Training


10. Enhancing Peer Review Training• CSR and NIH Review Staff

6 face to face training sessions, January 2009g y6 face to face training sessions, April 2009Continuous updating

• ChairsChairs

For Chairs 17 sessions in 2009For Chairs appointed in 2010, 9 sessions so far, 4 more planned

• Reviewers

Training material (Power Point, interactive training, frequently asked questions mock study section videoquestions, mock study section video Senior CSR staff at the first meeting in May-July 2009


11. Continuously Reviewing the Changes

• 12/09 Applicant and Reviewers Survey (64% response)

• 1/10 Advisory Council Survey (291 responses)

5/11 Pl d S Sh t A li ti• 5/11 Planned Survey on Shorter Applications

Key Findings from Reviewers and Councils

• Councils have necessary information to make decisions

• Reviewers like 9 point scoring scaleReviewers like 9 point scoring scale

• Overall impact score is NOT the average of criteria scores

• Approach is most influential criteria score (easiest to assess)

• Clustering works --no difference in scoring ~ ESI/established

• New change—have reviewer write overall impact paragraphg p p g p

Changes in Peer Review

If we want things to stay as they are, things will have to change

The LeopardGi T iGiuseppe Tommasi, Prince of Lampedusa

challenggppes and opportunities in peer review · challenggppes and opportunities in peer review a...

Documents