September 17, 2015

Broad PCORI Funding Announcement

Applicant Town Hall

Cycle 2 2015

Agenda

Research Strategy

Patient Engagement and Engagement Plan

Additional sections required for your application

The Merit Review Process

Questions

Submitting Questions:

Submit questions via the

chat function in Meeting

Bridge.

Ask a question via phone

(an operator will standby

to take your questions).

Introductions

Lauren Azar, MHA Senior Program Associate

Improving Healthcare Systems

Suzanne Schrandt, JD Deputy Director

Patient Engagement

Cathy Gurgol, MSProgram Officer

Addressing Disparities

Mari Kimura, PhD Merit Review Officer

Merit Review

Maricon Gardner, CRAContracts Associate, Pre-Award

Contracts Management and Administration

Your letter of intent (LOI) was reviewed and

you have been invited to submit a full

application……..

CONGRATULATIONS!

Letter of Intent and Application

You were invited to submit a full application based on the information provided in the LOI; changes after the LOI require PCORI approval.

Show stoppers include:

Changes to the PI

Changes to the

Institution

Changes to the Study Design

Changes to the Budget/

Period of Performance

Changes to Research Question

Changes to

Specific Aim

Changes to Comparators

Preparing your application for submission –

Research Strategy

First things first……. Get prepared

- Read and review the PCORI Application Guidelines

Document

http://www.pcori.org/sites/default/files/PCORI-PFA-

Cycle 2 2015 Application-Guidelines.pdf

- Re-read the funding announcement for which you

are applying

http://www.pcori.org/funding/opportunities

- Review the PCORI Research Plan Template

- Have a copy of your approved LOI readily

accessible

► Research Strategy

► Dissemination & Implementation Potential

► Replication & Reproducibility of Research

and Data Sharing

► Protection of Human Subjects

► Consortium Contractual Arrangements

► References Cited

► Appendix

Research Plan Template

Research Strategy

Maximum 20 pages in length

Use the Research Plan Template as your guide

- Background

- Significance

- Study Design/Approach

- Project Milestones/Timeline

- Patient Population

- Recruitment Plan

- Estimated Racial/Ethnic and Gender Enrollment Table

- Research Team and Environment

- Engagement Plan

PLEASE provide all the information requested, as outlined in

the template.PageLimit

20

Research Strategy

Two items to note:

- Research questions and specific aims MUST be

the same as in the approved LOI unless the

applicant has explicit and documented approval

from the program to which they will submit.

- While completing the study design/approach

section, applicants should cite PCORI’s

Methodology Standards.http://www.pcori.org/assets/2013/11/PCORI-Methodology-

Report.pdf

Methodology Standards

In any study, methods are critical. PCORI’s Methodology Committee developed Methodology Standards to which patient-

centered CER must adhere

Methodology Standards: 11 Broad Categories

• Formulating Research

Questions

• Patient-Centeredness

• Data Integrity and Rigorous

Analyses

• Preventing/Handling Missing

Data

• Heterogeneity of Treatment

Effects

• Data Networks

• Data Registries

• Adaptive and Bayesian

Trial Designs

• Causal Inference

• Studies of Diagnostic

Tests

• Systematic Reviews

PCORI Monitors Projects

Deliverables:

Measurable and verifiable

outcomes or objects that a

project team must create and

deliver according to the

contract terms

Milestones:

Significant events or

accomplishments within the

project; may have deliverables

associated with them

Example of Milestone Schedule

Dissemination & Implementation

Describe the potential for disseminating and

implementing the results of this research in other

settings.

Describe possible barriers to disseminating and

implementing the results of this research in other

settings.

Describe how you will make study results available to

study participants after you complete your analyses.

PCORI does not expect applicants to disseminate and

implement findings at this time. PageLimit

2

Replication & Reproducibility of Research and

Data Sharing

Describe the ability to reproduce potentially important

findings from this research in other data sets and

populations.

Describe how you will make a complete, cleaned, de-

identified copy of the final data set used in conducting the

final analyses available within 90 days of the end of the

final year of funding, or your data-sharing plan, including

the method by which you will make this data set available, if

requested.

Propose a budget to cover costs of your data-sharing plan,

if requested. PageLimit

2

Describe the protection of human subjects who will be

involved in your research.

Refer to NIH standards for research involving human

subjects

PageLimit

5

Protection of Human Subjects

Study Participant Protection

Provide a Data and Safety Monitoring Plan that operates under the auspices of your institution

Assure that key personnel are educated on human subjects protections

Assure appropriate informed consent

Establish procedures to minimize risks to participants

Establish procedures to protect privacy and maintain confidentiality

If you anticipate seeking waiver of individual informed consent, provide the rationale

Refer to NIH standards for research involving human subjects

Describe the proposed research projects that will be

performed by subcontracted organizations; explain the

strengths that these partners bring to the overall project.

PageLimit

5

Consortium Contractual Arrangement

References Cited

Following scholarly citation practice, list the source

material cited in the Research Plan.

PageLimit

10

Appendix

PCORI applications may include an appendix for

additional materials the investigators think may be useful

Examples of additional materials are:

– Survey instruments

– Papers and publications from members of the

research team; however, reviewers will not be

required to include the appendices in the review and

assessment of the project

PageLimit

10

Recruitment

Discuss past experiences with recruitment of the target

population

Provide preliminary evidence of the potential for

successful recruitment

Consider barriers to recruitment – and how you plan to

overcome them

Strategies for successful recruitment

– Engaged clinical sites

– Clinical advocates

– Proactive, experienced research coordinator

Application Common Mistakes

Inadequate support for the research question based on

systematic reviews or identified gaps in clinical

guidelines

Studies of efficacy, not comparative effectiveness

Low likelihood of study impact

Limited, inappropriate, or non-specific patient-centered

outcomes

Inappropriate use or explanation of usual care as a

comparator

Missing key aspects of scientific design, such as well-

justified power calculations

Overly optimistic timeframe for enrollment -- failure to

think through complexities of recruitment and enrollment

Lack of support for key statements in the Research Plan

Inadequately justified budgets

Application Common Mistakes

Addressing engagement in

your application

Patient-Centeredness; do the question(s) and outcomes

matter and are they important to patients and their

caregivers?

Patient and Stakeholder Engagement; are patients and

other stakeholders involved in meaningful ways

throughout the research process?

Remember; patient partners are not the same as patient

subjects or participants

Remember: there are eight stakeholder categories in

addition to patients and caregivers

Patient-Centeredness vs. Patient and

Stakeholder Engagement

The Engagement Rubric

Planning the Study

Conducting the Study

Disseminating the Study Results

PCOR Engagement Principles

Potential activities include but are not limited to;

Identifying the research question and outcomes

Drafting study protocol

Examples;

“Mental Health” study name and materials amended by

partners

Diabetes study; clinician partners amended study design

to more accurately reflect real-world practice settings

Planning the Study

Potential activities include but are not limited to;

Recruitment and retention of study participants

Data collection

Data analysis

Examples;

Patient partners in cancer imaging studies amended

inclusion criteria to be more reflective realistic patient

actions

Parent partners in pediatric psychiatry study are delivering

components of the intervention

Conducting the study

Potential activities include but are not limited to;

Co-authoring publications such as journal articles

Co-presenting at conferences

Using social media outlets

Examples;

Neurology study results are being shared at a neurology

patient advocacy organization conference

Patient partners are co-authoring journal articles on a pain

study

Disseminating study results

Six Engagement Principles

- Partnership

- Co-learning

- Reciprocal relationships

- Trust

- Transparency

- Honesty

Embodying the Engagement Principles

Real-World Examples

- Providing financial compensation to patient and

stakeholder partners

- Providing training opportunities for patient,

researcher, and stakeholder partners

- Scheduling and locating meetings at mutually

accessible times and locations

Embodying the Engagement Principles

(Cont’d)

Addressing engagement

“Show your work”

Provide letters of support

Budget appropriately

New this cycle

Public posting of patient and stakeholder partner names

If a proposal is selected for funding, these names will be included

with the name of the PI and the public abstract on our website

By providing names of individuals or organizations, you are

indicating that you have received any appropriate authorizations or

permission to do so

If you have patient or stakeholder partners who wish to remain

anonymous, please contact pfa@pcori.org for guidance on how to

proceed

Compensation Framework

Engagement Resources

PCORI’s “Engagement Rubric”http://www.pcori.org/sites/default/files/PCORI-Engagement-Rubric-

with-Table.pdf

Sample Engagement Planshttp://www.pcori.org/sites/default/files/PCORI-Sample-Engagement-

Plans.pdf

PCORI Compensation Framework

http://www.pcori.org/sites/default/files/PCORI-Compensation-

Framework-for-Engaged-Research-Partners.pdf

Engagement in Research website page

http://www.pcori.org/content/engagement-research

PCORI’s Methodology Standards PC-1 to PC-4http://www.pcori.org/assets/PCORI-Methodology-Standards1.pdf

Other sections required for your application

pcori.org/apply

PCORI Online: Application

► PI and Contact Information

► Project Information

► Key Personnel

► Milestones

► Templates and Uploads

► Save

Public Abstract

Project

Information

Technical Abstract

Project Narratives

Key Personnel

► PI and Contact Information

► Project Information

► Key Personnel

► Milestones

► Templates and Uploads

► Save and Review

Milestones

► PI and Contact Information

► Project Information

► Key Personnel

► Milestones

► Templates and Uploads

► Save and Review

People and Places Template - Biosketch

You may use the NIH biosketch or PCORI’s format

Biosketches are required for all key personnel

List all partners within the Key Personnel section

Patient/Stakeholder Biosketch

PageLimit

5 Per person

People and Places Template –

Project / Performance Site(s)

Demonstrate that the proposed facilities have

the appropriate resources required to conduct

the project to plan, within budget, and on time.

Provide a description of the facilities that will be

used during the project, including capacity,

capability, characteristics, proximity, and

availability to the project

PageLimit

15Professional

Profile/Biosketch

Letters of Support

Save all letters of support as single PDF file prior to uploading

to the PCORI Online System.

All letters of support should be addressed to the PI and

demonstrate the commitment of key personnel and supporting

organizations to your proposed project.

Letters of support should clearly reflect the substantive

involvement and material contribution to be provided by the

signatory parties, and are meant to substantiate the

commitment of collaboration of all forms.

Letters of support should be organized in the following

manner:

- Letters of organizational support

- Letters of collaboration

- Letters confirming access to patient populations, data sets, and

additional resources

Budget Templates: Overview

Three budget sections must be submitted

as part of the online application process:

Detailed Budget Budget Summary Budget

Justification

NOTE:

A detailed budget is needed for each year of the

program. Complete each budget section for the

prime applicant and any/each subcontractor.

Detailed Budget

• Personnel

• Consultant

• Equipment

• Supplies

• Travel

• Other Expenses

• Consortium/Contractual Direct Costs

• Consortium/Contractual Indirect Costs

• Prime Indirect Costs

Budget Justification

Narrative that fully supports and explains the basis for the

information in the Budget Detail

– Provide sufficient detail to understand the basis for costs, the

reason that the costs are necessary, and an explanation for

major cost variances

– Use the budget template to tell PCORI why the costs are

reasonable for the work to be performed

Breakdown of costs proposed for each consortia or contractor

Must specify any other sources of funding that are anticipated to

support the proposed research project

Provide quotes, indirect cost rate letter, fringe benefit policy

Allowable and Unallowable Costs

44

NOTE Review Appendix 3 of Application Guidelines for updates

on allowability of patient and clinical care costs.

Justifying Commonly Denied Items

You must do the following:

• Provide a detailed explanation for why the purchase of

the requested item(s) is required for project completion.

• Include a statement verifying the requested item(s) is

not currently available for PI use.

• Include a statement that the requested item(s) will be purchased in accordance to applicable cost principles.

NOTEItems purchased under PCORI funded projects are

not to be used as incentives to recruit or retain

graduate students or other personnel.

Templates and Uploads

► PI and Contact Information

► Project Information

► Key Personnel

► Milestones

► Templates and Uploads

► Save and Review

Common Application Errors

Using the wrong browser, access PCORI Online via Chrome

or Safari browsers

Not entering information into all required fields in the system

Having multiple people working on the application at the

same time

Having the incorrect file extension, only PDF files can be

uploaded

Not choosing the correct document type from the drop-down

menu

AO is unable to view the application

Tips for Success

• Adhere to the PFA and Application Guidelines for the funding

cycle you are applying to

• Talk to a Program Officer if you have questions

• Start and submit early

• Download PCORI’s Online User Manual for Submitting an

Online Application

• Ensure that all team members can see the application in the

system (check during the LOI stage)

• Inform your AO of your intent to submit

• Submit the completed application before the due date or on

it by 5:00 PM ET

Resources

Refer to the funding opportunities page in our Funding

Center (http://www.pcori.org/funding/opportunities) for

the following resources:

PFA and Application Guidelines

PCORI Online User Manuals

Sample Engagement Plans

General Applicant FAQs: http://bit.ly/applicant_faqs

PCORI Online: https://pcori.fluxx.io/

Research Methodology: http://www.pcori.org/node/4020

What happens to your application

after you submit it?

Administrative Screening

Applicants must follow administrative requirements set in

PCORI’s Application Guidelines.

► Exceeding page limits, budget, or time

limitations

► Not using PCORI’s required templates

► Submitting incomplete sections or applications

Missing

the Mark

Programmatic Screening

► Study deviates from approved LOI

► Study includes cost-effectiveness analysis

► Study is not responsive to PFA and/or does NOT

address the PFA’s “Research Areas of Interest”

► Study is not comparative

Missing

the Mark

PFA

Responsiveness

PCORI Mission

Advance Through

to Merit Review

Merit Review

Building an Inclusive Merit Review

• Panels include 3 reviewer types to bring diverse

perspectives to the merit review process.

• Each application is reviewed by 2 scientists,1 patient,

and 1 other stakeholder.

• The panel chair facilitates discussion and promotes a

culture of mutual respect and understanding among

reviewer types.

PatientsOther Stakeholders

Scientists

Merit Review Criteria

Criterion #1: Impact of the condition on the health of individuals and population

Criterion #2: Potential for the study to improve healthcare and outcomes

Criterion #3: Technical merit

Criterion #4: Patient-centeredness

Criterion #5: Patient and other stakeholder engagement

Patient and Other

Stakeholder Reviewers

Scientist Reviewers

Scoring Range

Range Score Descriptor Characteristics

High

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

Medium

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weakness

Low

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

A score of 1 indicates an

exceptionally strong application.

A score of 9 indicates an application with

serious and substantive weaknesses.

How Are Applications Reviewed?

Impact of the condition on the health of

individuals/populations

Potential for the study to improve

healthcare and outcomes

Technical merit

Patient-centeredness

Patient and stakeholder engagement

Reviewers evaluate applications using five

criteria:

PCORI’s Board of

Governors makes

funding decisions

based on merit review

and staff

recommendations.

Impact of the condition on the health of

individuals and populations

Is the condition or disease associated with a significant

burden in the US population, in terms of prevalence,

mortality, morbidity, individual suffering, or loss of

productivity?

Alternatively, does the condition or disease impose a

significant burden on a smaller number of people who have

a rare disease?

Does the proposal include a particular emphasis on patients

with one or more chronic condition?

Potential for the study to improve

healthcare and outcomes

Does the research question address a critical gap in current knowledge as noted in systematic reviews, guideline development efforts, or previous research prioritizations?

Has it been identified as important by patient, caregiver, or clinician groups?

Do wide variations in practice patterns suggest current clinical uncertainty?

Is the research novel or innovative in its methods or approach, in the population being studied, or in the intervention being evaluated in ways that make it likely to improve care?

Do preliminary studies indicate potential for a sizeable benefit of the intervention relative to current practice? How likely is it that positive findings could be disseminated and implemented quickly, resulting in improvements in practice and patient outcomes?

Technical Merit

Does the proposal delineate a clear conceptual

framework/theory/model that anchors the background literature and

informs the design, key variables, and relationships being tested?

Are the comparison interventions realistic options that exist in current

practice?

Are sample size and power estimates presented that are based on

realistic and careful evaluations of the anticipated effect size?

Is the project timeline realistic, including specific scientific and

engagement milestones?

Does the research team have the necessary expertise to conduct the

project?

Is the organizational structure and are the described resources

appropriate to carry out the project?

Is there a diverse study population with respect to age, gender, race,

ethnicity, and clinical status, appropriate for the proposed research?

Patient-centeredness

Is the research focused on questions that affect outcomes of

interest to patients and their caregivers?

Does the research address one or more of the key questions

mentioned in PCORI’s definition of patient-centered

outcomes research?

– “Given my personal characteristics, conditions, and preferences, what

should I expect will happen to me?”

– “What are my options, and what are the potential benefits and harms of

those options?”

– “What can I do to improve the outcomes that are most important to me?”

– “How can clinicians and the care delivery systems they work in help me

make the best decisions about my health and health care?”

Patient and stakeholder engagement

Are patients and stakeholders engaged in:

- Formulating research questions

- Defining essential characteristics of study participants, comparators, and

outcomes

- Identifying and selecting outcomes that the population of interest notices

and cares about (e.g., survival function, symptoms, health-related quality of

life) and that inform decision making relevant to the research topic

- Monitoring study conduct and progress

- Designing/suggesting plans for dissemination and implementation activities

Are the roles and the decision making authority of all research

partners clearly stated?

Does the proposal demonstrate the principles of reciprocal

relationships, co-learning, partnership, trust, transparency, and

honesty?

Multi-phase process for review

Preliminary (online) review

In-Person review

Post-Panel review (PCORI program staff)

Submission and Key Dates

What When

Application Deadline November 3, 2015 by 5:00pm

ET

Merit Review Dates February 2016

Awards Announced April 2016

Earliest Start Date June 2016

Where can I find help?

Visit pcori.org/apply

Application Guidelines

FAQs

PCORI Online User Manuals

Sample Engagement Plans

Schedule a Call with a Program Officer

Submit a request at pcori.org/content/research-inquiry

Call 202-627-1884 ( programmatic inquiries)

E-mail sciencequestions@pcori.org

Contact our Helpdesk

E-mail pfa@pcori.org

Call 202-627-1885 ( administrative and

technical inquiries)

Q&A

Ask a question via the chat function in Meeting

Bridge.

Ask a question via phone (an operator will

standby to take your questions).

If we are unable to address your question during this time, e-mail the

Helpdesk at pfa@pcori.org.