Revising and Resubmitting: Practical Considerations Based

on the Psychology of Re-Reviews

Marc I. Rosen, M.D.

Most Successful Grants are Revise and Resubmits

Initial or Revised?

Number Application

s

Success Rate

Initial Submission

19,259 8.6%

Revise and Resubmit

5,373 37.2%

http://www.report.nih.gov/success_rates/index.aspx 2012 data for new R01s

Revising and Resubmitting: Lecture Structure

Deciding Whether to Revise and Resubmit

Suggestions for Revising and Resubmitting

ExampleMoral

Review Group Actions

Discussed and Scored with Impact Rating (score and percentile)

Triaged/Not Discussed with no Impact Rating but criteria scores (lower half)

Deferral; Not Recommended; Abstention

Deciding Whether to Resubmit: Consider the Score

Triaged/Not Discussed: How decided? Initial decision to triage at meeting rarely

overturned at meetingHard to completely convert a critic to a gung-ho

boosterCommittee is busy, focuses on close calls

2010-2012 R01s2.3% of unscored new R01s funded on resubmit8.7% of unscored continuation R01s fundedIncludes those not resubmittedhttp://report.nih.gov/FileLink.aspx?rid=880

Reading the Critiques• Read critiques carefully and calmly• Even if you are angry• Assume you got a good-faith, intelligent

review

• Let colleagues and mentors read the reviews for reality testing, support, and input

• Give more weight to comments that• Are in the “Summary”• Are made by more than one reviewer

Deciding Whether to Revise and Resubmit: Get More InformationContact Program Officer

-Ask about study section discussion-Ask about NIH Institute

interest in area?advice?

Talk through reviews with co-investigators and peers

• Reviewers assess your submitted material

• Reviewers are never totally wrong or right

• Extremely competitive process:

• Resubmission is common

• Avoid WYSIATI (what you see is all there is)---other talented people out there

Deciding Whether to Resubmit: Keep Perspective

Deciding Whether to Revise and Resubmit: Prospect TheorySunken Cost Fallacy

• Staying to the end of a boring movie hoping to recoup loss of spent money

• Using a fitness plan even when it’s painful

• It’s a fallacyLoss aversion: It’s not a rejection if you don’t give up

Thinking Fast and Slow (Daniel Kahneman)

Deciding Whether to Revise and Resubmit: Psychology

Overconfidence: Excessive Optimism• Only 5% of U. Chicago MBA students

predict they are in bottom 50%; most predict second decile

• 90% of drivers think they are above average

• Entrepreneurs say success rate for new business is 50% but predict personal success rate of 100%

• Few newlyweds expect to be among 50% who eventually divorce

Nudge, Thaler and Sunstein

Deciding Whether to Revise and Resubmit: Overconfidence?Test of Overconfidence: 90% Confidence Interval for

• Weight of earth in tons• 6.0 X 1024

• Percentage of world’s population who are Native English speakers (per CIA World Factbook 2009)• 4.83%

Reviews that Should NOT Make You Overconfident“This grant addresses an important topic”“Yale has superb facilities for this research”“The investigator is qualified”Only the first reviewer was critical of the

application and the grant was un-scored/poor score

Mild praise and the grant was un-scored/poor score

Meta-Critiques that May Not be Answerable

”There are already a lot of grants in this area”

“Not innovative”“Not significant”“Not exportable”

Consider Alternatives

A smaller grant (R21 instead of R01)

Another funding agencyA substantial change that you can submit as a new grant◦If you can answer critiques revise and

resubmit◦If you cannot answer critiques new grant

Deciding Whether to Revise and Resubmit• Do you have something better to work on for two-plus months?

Revising and Resubmitting: Lecture Structure

Deciding Whether to Revise and Resubmit

Suggestions for Revising and Resubmitting

ExamplesMoral

Consider the Person Behind the Review

◦ Look into research interests of people on the committee

◦ NIH REPORTER search of what committee has funded in the past

◦ Talk to people who have been on the committee

◦ Talk to your project officer

Consider the Person Behind the Review

◦ Reviewers want to avoid cognitive dissonance Cognitive dissonance

Inner drive to hold our attitudes and beliefs in harmony

Drive to avoid dissonance between themExamples

The Fox and the Sour GrapesThe review group that found fault with your grant

◦ So, don’t say the reviewer was wrong

Consider the Person Behind the Review

◦ Even Reviewers Who Change Their Minds Impacted by Anchoring to Prior Score Roulette Wheel Study of Anchoring 1: Volunteers shown rigged roulette wheel that stops at 10 or 65

What % of countries in the U.N. are in Africa?Wheel stops at 10---------25% averageWheel stops at 65---------45% average

http://en.wikipedia.org/wiki/Anchoring; Tversky and Kahneman, 1974

Consider the Reviewer’s Perspective

The reviewer who likes your application has to justify your response to a committee of 12+ very smart people

Make your response easy-to-follow

Use tables for complicated concepts (the reviewer can say to the committee—“He’s got a table laying that out.”)

Consider the Reviewer’s Perspective• If the reviewer likes the

application, he/she is more likely to agree with your justifications

• How juries decide:• They do not weigh the evidence • They do arrive at a narrative that

appears to fit the data• Answer meta-critiques

Revising and Resubmitting:Content Issues

Address any grant-killing meta-questions early in your response emphatically and clearly

Examples of Meta-Critiques

Critique: “They’ll never be able to pull this off – the project is not feasible.”

Answers: -Pilot data-Bring in collaborators who can pull it off

Examples of Meta-Critiques

Critique:“This was written by a slob who just does not know this topic well.”

Answers: -Emphasize how much the application has been cleaned up

-Consider adding expert who would have caught all your mistakes the first time.

Critique:“This was written by Kathy Carroll’s (Stephanie O’Malley’s, Rajita Sinha’s, Marc Potenza’s…) go-fer and is not really an independent application.”

Answer: -Spell out what is yours and what is not

Examples of Meta-Critiques

Examples of Meta-CritiquesCritique:“[zzzz’s] always make a hash out of [yyyy] research– it requires someone in my field of specialization. “

“The application would be strengthened by the involvement of a biostatistician.”

Answer: -Include someone with the recommended expertise

Examples of Meta-Critiques

Critique:”The study design is from hunger.”

Answer: -Table and/or figure justifying and explaining the study design

Revising and Resubmitting:Process IssuesRespond constructively and positively

◦The reviewer is always right (even if not).No more than 1-2 areas of

disagreement, but justify decision thoughtfully and respectfully

If not ready to submit at next deadline, DON’T◦Reviewers generally need a reason to

improve your score

Revising and Resubmitting:General Content Issues

“Thank you for the careful review of our proposal to […] We appreciate the careful, critiques”

Indicate method of highlighting changes (e.g., bold, italics in text)

Main criticism and responseList more minor criticisms (in italics) and

responses“Thank you for reconsidering our application”

Revising and Resubmitting:The Best Responses

Pilot DataRe-analysis of your own dataLiterature

Revising and Resubmitting:Weaker Responses

◦Logic◦Your opinion◦“In my clinical experience…”

Revising and Resubmitting:Don’t, Don’t, Don’t

(Usually) don’t answer questions that were not raised

Don’t malign the review process or the reviewer Don’t spend much effort pointing out that one

reviewer liked what another reviewer critiquedIf the reviewer asks for something that was already

in the application, be humble, e.g. “The information is presented more clearly this time in the methods as follows…”

Don’t get personal (no jokes, personal opinions, etc.)

Revising and Resubmitting:Don’t, Don’t, Don’t

• Don’t repeat every critical word from a review• Summarize criticisms (it was bad enough the first time)

• Don’t over-answer minor criticisms by writing a long essay that makes the criticism seem more major than it is

Revising and Resubmitting: Lecture Structure

Deciding Whether to Revise and Resubmit

Suggestions for Revising and Resubmitting

ExampleMoral

Example of Grant Review

4-year clinical trial to test computer-delivered counseling to improve engagement in work

Grant Review

• Lousy score of 270, 67th percentile• Program officer tells me they liked it, wanted to see it back

• Reviewer response:• Reviewer one liked• Reviewer two mixed• Reviewer three (statistician) gave it terrible score

Reviewer’s Potentially Grant-Killing Responses

“However, no data exists whether veterans would actually use the intervention.”

Summary Statement Recommends “Further conceptually develop and pilot test the internet-based intervention. Provide that data as a part of the proposal.”

Planned Response

• Agree with everything reviewers say and propose three-year, pilot-type, therapy development study to address it

Revising and Resubmitting: Lecture Structure

Deciding Whether to Revise and Resubmit

Suggestions for Revising and Resubmitting

ExampleMoral

Morals

It helps to enjoy the process◦Doing your best◦Advocating for something you believe in

◦Promoting yourselfYour CV lists grants◦No lasting harm from unfunded application

Moral“At the length, truth will out”

◦Shakespeare (Merchant of Venice)

“In the long run, we are all dead.”◦John Maynard Keynes

Thank you