Securing External Federal FundingJanice F. Almasi, Ph.D.

Carol Lee Robertson Endowed Professor of Literacy

University of Kentucky

Janice.almasi@uky.edu

Institute of Education Sciences2 Grant Competitions per year

Janice F. Almasi, Ph.D., University of Kentucky

Current Funding Opportunities14 Long-term Programs of Research

Drag picture to placeholder or click icon to add

Be InformedSubscribe to Newsflash at ies.ed.gov/newsflash

Janice F. Almasi, Ph.D., University of Kentucky

IES Research Goals

Goal 1: Identification Identifying programs and practices associated with

better educational outcomes (secondary data analysis) Goal 2: Development Projects

Developing educational interventions Goal 3: Efficacy and Replication Projects

Determine if fully-developed interventions are effective Goal 4: Scale-Up Goal 5: Measurement Projects

Janice F. Almasi, Ph.D., University of Kentucky

Prior to Peer Review Meeting

Triage identifies top 25 applications

Reviewers read, rate about 8 applications

Reviewers check for COIs

Each application assigned to at least 2 reviewers

Janice F. Almasi, Ph.D., University of Kentucky

Criteria

Significance Research Plan Personnel Resources

Janice F. Almasi, Ph.D., University of Kentucky

Review Criterion Ratings(M

ore

Weak

ness

es

than

St

reng

ths)

(Bala

nce

of

Stre

ngth

s and

We

akne

sses

)

(Mor

e St

reng

ths t

han

Weak

ness

es)

1

Poor Excellent

765432

Janice F. Almasi, Ph.D., University of Kentucky

What Reviewers Look ForWhere Applications Tend to be Weak

Janice F. Almasi, Ph.D., University of Kentucky

Significance

Goal 1• Theoretical and

empirical rationale for study and practical importance of the intervention (e.g., program, practice) that will be examined

Goals 2 and 3• Describe (a) the

intervention (e.g., features, components) and the logic model for the intervention, (b) theoretical and empirical support for intervention, and (c) practical importance of the intervention

Janice F. Almasi, Ph.D., University of Kentucky

Significance in Goals 2 and 3

Context for Proposed Interventions Provide context for the proposed intervention by including

data on, or reviewing research describing, the attributes of typical existing practices.

Identify shortcomings of current practice and how they contribute to the rationale for the proposed intervention.

Provide context for understanding how much of a change the proposed intervention is intended to achieve.

Janice F. Almasi, Ph.D., University of Kentucky

Significance in Goals 2 and 3 Intervention, Theory of Change, Empirical/Theoretical Rationale

Clearly describe the intervention Clearly describe the theory of change for the intervention

How do the features or components of the intervention relate to each other temporally (or operationally), pedagogically, and theoretically (e.g., why A leads to B)?

Provide a strong theoretical and empirical justification for the design and sequencing of the features or components of the intervention. Enables evaluation of: Relation between the intervention and its theoretical and

empirical foundation (e.g., is the proposed intervention a reasonable operationalization of the theory?)

Relation between the intervention and the outcome measures (e.g., do the proposed measures tap the constructs that the intervention is intended to address?)

Include a Logic Model

Janice F. Almasi, Ph.D., University of Kentucky

Significance in Goals 2 and 3

Practical Importance of Intervention When the proposed intervention is fully developed will it

have the potential to improve student outcomes in educationally meaningful increments, if it were implemented over the course of a semester or school year?

Would the proposed intervention be both affordable for and easily implemented by schools (e.g., not involve major adjustments to normal school schedules)?

Janice F. Almasi, Ph.D., University of KentuckyResearch Plan: Goal 2

Sample• Samples and settings used to

assess feasibility of intervention and for pilot data assessing promise of intervention

Iterative Development Process• Revision• Implementation• Observation• Revision

How do you define“operating as intended?”

What data will begathered to determine

how intervention isoperating?

How will the datagathered be used

to revise the intervention?

What criteria will be used to

determine if the intervention operates

as intended?

Janice F. Almasi, Ph.D., University of KentuckyResearch Plan:Goal 2

Feasibility of Implementation• Goal is a fully developed intervention• Data that addresses feasibility of

implementing in small sample of authentic education settings

• Promise of intervention in terms of outcomesPilot Study

• Pilot data on outcome measures progressing in right direction

• Pilot data demonstrates implementation of intervention is associated with behaviors consistent with theory of change

• No more than 30% of funds• Data should not be a test of efficacy

Janice F. Almasi, Ph.D., University of KentuckyResearch Plan:Goal 2

Measures• Clearly describe procedures for

gathering data to refine and revise the intervention and provide insight into feasibility and usability of proposed intervention• What needs to be observed?• How will observations be

gathered?• Clearly describe measures that will

be used (and reliability and validity if appropriate)

Janice F. Almasi, Ph.D., University of KentuckyReearch Plan:Goal 3

Research Questions• Pose clear, concise hypotheses

or research questionsSample• Define sample to be selected• Define sampling procedures

(including justification for inclusion and exclusion)

• Strategies to be used to reduce attrition

Janice F. Almasi, Ph.D., University of KentuckyReearch Plan:Goal 3

Research Design• Provide detail!• How will threats to internal/external validity be

addressed?• Studies using random assignment are preferred

where feasible• What is unit of randomization and what

procedures will be used to make assignments to conditions?

Power• What power is needed to detect a reasonably

expected and minimally important effect?• How was effect size calculated?• If clusters are randomly assigned to treatment

conditions be sure to include intraclass correlation and anticipated effect size in power analysis

Janice F. Almasi, Ph.D., University of KentuckyResearch Plan:Goal 3

Measures• Justify appropriateness of measures• Are measures of practical interest to

educators and not overly aligned with intervention?

• Include reliability and validity informationFidelity of Implementation• How will implementation be documented and

measured?• How will factors associated with fidelity be

identified and assessed?• How will fidelity data be incorporated into

analyses of impact?• How do conditions in the school setting affect

fidelity of implentation?

Janice F. Almasi, Ph.D., University of KentuckyResearch Plan:Goal 3

Comparison Group• How does comparison group compare to intervention

on critical features of intervention?• Using a “business-as-usual” comparison is

acceptable but explain why using it is acceptable• How will contamination be avoided?Mediating and Moderating Variables• Observational, survey, or qualitative methods are

encouraged to help identify factors that may explain the effect or lack of effect of intervention

Data Analysis• Quantitative: Specify statistical procedures and

include formulas where appropriate• Qualitative: Specific methods used to index,

summarize, and interpret data should be identified• Relation between hypotheses, measures, and

independent and dependent variables should be clear

Janice F. Almasi, Ph.D., University of Kentucky

Personnel

What role will each individual have in the project? What qualifications, training, and experience do key

personnel possess? How will qualifications be used on the research? Are key personnel dedicating sufficient time to

competently implement proposed research?

Janice F. Almasi, Ph.D., University of Kentucky

Resources

Are the resources adequate to support the proposed activities in terms of: Facilities Equipment Supplies Institutional Support for Managing/Directing Grants and

Supporting Scholarship

Have partners shown support for implementation and support of the project?

Janice F. Almasi, Ph.D., University of Kentucky

Scientist Reviewer Critiques

A brief description of the overall application. Identify each application’s key strengths and

weaknesses in each of the evaluation areas and prepare critical, evaluative comments.

Integrated summary of the overall assessment of the application, including the main strengths and weaknesses of the application.

Janice F. Almasi, Ph.D., University of Kentucky

Overall Score

Overall Score Range Adjectival Equivalent1.0 - 1.5 Outstanding1.6 - 2.0 Excellent2.1 - 2.5 Very Good2.6 - 3.0 Good3.1 - 4.0 Fair4.1 - 5.0 Poor

Janice F. Almasi, Ph.D., University of Kentucky

Peer Review Meeting Process

3. Notetaker

Summarizes the discussion orally and in writing

2. Full PanelDiscusses the application, asks questions, and offers additional

critiqueDiscusses the budget

1. Assigned Scientist Reviewers Share Overall ScoresReviewer 1 Summarizes the

application and its strengths and weaknesses in each

Reviewer 2 elaborates on areas of agreement or disagreement

Janice F. Almasi, Ph.D., University of Kentucky

Peer Review Meeting Process

6. Assigned Scientist ReviewersMay edit/revise their original written critiques

based on panel discussion

5. Full PanelPrivately assigns criteria scores and overall scores

4. Assigned Scientist ReviewersAdjust initial recommended criteria scores and overall scores