1

CSR INTEGRATED REVIEW GROUP REVIEWS[peer review of peer review]

Don Schneider

National Institutes of HealthU.S. Department of Health and Human Services

November 3, 2008

CSR is committed to peer review of peer review

• Need to respond to changing science• Trace roots to Panel on Scientific

Boundaries for Review (PSBR)• Examine Integrated Review Group-specific

issues• Discuss general issues

2

The PSBR issued a report January 2000 outlining reorganization philosophy and expectations.

• URL - http://cms.csr.nih.gov/NewsandReports/ReorganizationActivitiesChannel/BackgroundUpdatesandTimeline/FinalPhase1Report.htm

• Be monitored continuously: assess peer review system at regular intervals, at the level of the IRG, and globally by the Advisory Committee

3

Working Groups Assess IRGs[Peer review of peer review]

• Peer reviewers report on study section operations

• Feedback from reviewers condensed by staff into a brief 5-10 page white paper report

• Subset of peer reviewers meet as working group to edit/confirm essence of report

• Report is shared with Advisory Committee and used as basis of taking action for improvements

4

IRG Working Groups, Round 1, provided baseline, 2000-2002

• Round 1 working groups – one-cycle visit of each study section, often by single pre-selected reviewer

• Pre-selection probably resulted in perturbations

• Nonetheless, useful information obtained that informed reorganization

• PSBR Reorganization occurred step-wise one IRG at a time from June 2003 to March 2005 5

IRG Working Groups, Round 2, were initiated in 2005 and are currently in progress

• Eliminate perturbation –peer reviewers of peer review selected post-study section meetings

• Solicit input from multiple reviewers each cycle

• Solicit input from reviewers over multiple, at least two, cycles

• Share white papers with Chairs prior to reporting to Advisory Committee

• Provide substantial basis for future actions 6

Schedule for IRG Reviews

July 2008• Brain Disorders & Clinical Neuroscience (BDCN)• Molecular, Cellular & Developmental Neuroscience (MDCN)

September 2008• Integrative, Functional & Cognitive Neuroscience (IFCN)• Emerging Technologies & Training in Neuroscience (ETTN)• Biology of Development & Aging (BDA)

January 2009• Biobehavioral & Biobehavioral Processes (BBBP)• Risk, Prevention & Health Behaviors (RPHB)

March 2009• Healthcare Delivery & Methodologies (HDM)• Epidemiology & Population Sciences (EPS)

7

Cont., Schedule for IRG Reviews

May 2009• Immunology (IMM)• Infectious Diseases & Microbiology (IDM)

July 2009• Oncology 1 – Basic Translational (OBT)• Oncology 2 – Translational Clinical (OTC)

September 2009• Bioengineering Sciences & Technologies (BST)• Surgical Sciences, Biomedical Imaging & Bioengineering

(SBIB)

November 2009• AIDS & Related Research (AARR)• Endocrinology, Metabolism, Nutrition & Reproductive

Sciences (EMNR)8

Cont., Schedule for IRG Reviews

January 2010• Vascular & Hematology (VH)• Cardiovascular & Respiratory Sciences (CVR)

March 2010• Musculoskeletal, Oral & Skin Sciences (MOSS)• Digestive, Kidney & Urological Systems (DKUS)

May 2010

Genes, Genomes & Genetics (GGG)

Biological Chemistry & Macromolecular Biophysics (BCMB)

July 2010• Interdisciplinary Molecular Sciences & Training (IMST)• Cell Biology (CB)

9

Format for IRG Reviews

Late afternoon• Present Peer Review Issues & Current Initiatives (CSR

Director)• Discuss; discuss further over dinner

Morning• Present Working Group White papers• Describe study sections and specific issues/metrics (Chiefs)• Discuss and improve study section guidelines (Chairs)• Forecast future issues & trends in science

Afternoon• Consider the critical role of the study section chair (CSR

Director, in closed session with Chairs)10

IRG specific issues and actions are notedMolecular, Cellular & Developmental Neuroscience

IRG• One study section split for second time• Initial attempt to charter a neurotechnology panel

led to PRAC requested crosscutting look and eventual formation of two study sections, as well as stimulating formation of crosscutting neuroscience IRG and contributing to momentum for new division

11

Cont. IRG-specific issues & actions

Brain Disorders & Clinical Neuroscience IRG• Split two study sections, each into two• Resolve referral issue so that brain tumor

applications focusing on neurological aspects are assigned to BDCN, with those focusing on cancer being assigned to Oncology IRG

12

Cont. IRG-specific issues & actions

Integrative, Functional & Cognitive Neuroscience IRG• No issues & no actions

Biology of Development & Aging IRG• Clustering of International & Cooperative Projects

in BDA, with great breadth of topics – perhaps pilot editorial board review

Emerging Technologies & Training in Neuroscience IRG

• First applications reviewed winter 2008, no revisions/actions yet

• One new study section growing rapidly, on verge of needing to split

13

General Review Issues & Actions

Reviewers• Seniority• Ad hoc/temporary• Telephone• Mail• Registry/College

Scoring• Streamlining/unscoring• Score compression• Scoring consistency throughout meeting

14

IRG Issues & Enhancing Peer Review

Reviewers• Flexible service• Systematic training• Shorter applications and critiques

Scoring• Smaller scale, 7 or perhaps 9 bins• Scores for all applications• Promote score spreading, ranking or order of

review

Emerging scientific trends• Selecting appropriate reviewers• Updating study section guidelines 15

The End