OFFICE OF ENGLISH LANGUAGE ACQUISITION NATIONAL PROFESSIONAL DEVELOPMENT PROGRAM (NPD)

2017 NPD Grant Competition Webinar 3: Evidence and Evaluation

April 6, 2017

ATTENTION!

To hear the presentation you must dial into the conference line: – Dial: 1-800-369-2136 – Code: 6210871

You cannot hear or fully participate in the webinar unless you call the conference line

During the presentation all phone lines will be placed on mute to

avoid feedback during the presentation

2

DISCLAIMER This webinar training series serves to provide technical

assistance to applicants in preparing their applications and to respond to applicant questions. The purpose of the webinars is to review the competition priorities, requirements, and selection criteria with potential applicants, as well as for Department staff to answer questions about the Notice Inviting Applications (NIA) for this competition.

Please refer to the NIA published in the Federal Register for additional information. Application instructions can be found in the Application package located in

grants.gov (search CFDA 84.365) or the NPD applicant information website http://www2.ed.gov/programs/nfdp/applicant.html

Do not rely solely on the information shared in the webinars.

3

WEBINARS We offered a total of three pre-application webinars:

• March 15, 2017 – Webinar #1 • March 22, 2017 – Webinar #2 • April 6, 2017 – Webinar #3

– This is the third and final pre-application webinar for the 2017 NPD competition.

Please use the “chat box” located on the right side of the screen to submit questions. If time permits, your questions will be answered during the presentation and we will respond to others at the end of the presentation or via Frequently Asked Questions (FAQs) on the NPD website.

A link to the PowerPoint slides will be posted at: http://www2.ed.gov/programs/nfdp/applicant.html.

4

AGENDA

NPD Program Priorities – Absolute – Competitive – Invitational

NPD Program Evidence Criteria – “Entry” and “Exit” Evidence

Competitive Preference Priority 1 – Moderate Evidence of Effectiveness

Project Evaluation – Meeting What Works Clearinghouse (WWC) Standards

Resources and Reminders

5

PRIORITIES

Absolute Priority: Providing Professional Development to Improve Instruction for English Learners (ELs)

Competitive Preference Priority (CPP) 1: Moderate Evidence of Effectiveness (0 or 5 points)

Competitive Preference Priority (CPP) 2: Improving Parent, Family, and Community Engagement (0 to 5 points)

Invitational Priority 1: Dual Language Approaches Invitational Priority 2: Supporting the Early Learning

Workforce to serve ELs

ABSOLUTE PRIORITY

Absolute Priority: For FY 2017 and any subsequent year in which we make awards from the list of unfunded applicants from this competition, this priority is an absolute priority. Under 34 CFR 75.105(c)(3) we consider only applications that meet this priority.

Under this priority we provide funding to projects that provide professional development activities that will improve classroom instruction for ELs and assist educational personnel working with ELs to meet high professional standards, including standards for certification and licensure as teachers who work in language instruction educational programs or serve ELs.

7

COMPETITIVE PREFERENCE PRIORITY 1 MODERATE EVIDENCE OF EFFECTIVENESS

Applicants seeking to address CPP 1 should identify up to two citations for studies that meet the definition of Moderate Evidence of Effectiveness.

Cited studies may include those already listed in the Department’s WWC Reviewed Studies Database (see https://ies.ed.gov/ncee/wwc/ReviewedStudies).

Cited studies must be included in both the application narrative

section of the application and the application abstract.

NOTE: The Department will review the studies cited by the applicants to determine if they meet the requirements for, Competitive Preference Priority 1, Moderate Evidence of Effectiveness.

8

COMPETITIVE PREFERENCE PRIORITY 2 IMPROVING PARENT, FAMILY AND COMMUNITY ENGAGEMENT

Up to an additional 5 points may be awarded to applicants who address Competitive Preference Priority 2.

This would be for projects that are designed to

improve student outcomes through one or more of the following activities:

9

COMPETITIVE PREFERENCE PRIORITY 2 (CPP2) IMPROVING PARENT, FAMILY AND COMMUNITY ENGAGEMENT

CPP 2: (a) Developing and implementing Systemic Initiatives to improve Parent and Family

Engagement by expanding and enhancing the skills, strategies, and knowledge (including techniques or use of technological tools needed to effectively communicate, advocate, support, and make informed decisions about the student’s education) of parents and families.

(b) Providing professional development that enhances the skills and competencies of school or program leaders, principals, teachers, practitioners, or other administrative and support staff to build meaningful relationships with students’ parents or families through Systemic Initiatives that may also support students’ learning at home.

(c) Implementing initiatives that improve Community Engagement, the relationships between parents or families and school or program staff by cultivating Sustained Partnerships.

10

INVITATIONAL PRIORITY 1 DUAL LANGUAGE APPROACHES

We encourage applicants to propose projects to improve educator preparation and professional learning for dual language implementation models to support effective instruction for ELs. In particular, we encourage such approaches to take into account the unique needs of recently arrived limited English proficient students, immigrant children and youth, and Native American students, who are members of federally recognized Indian tribes.

11

INVITATIONAL PRIORITY 2 SUPPORTING THE EARLY LEARNING WORKFORCE TO SERVE ELS

We encourage applicants to propose projects that improve the quality and effectiveness of the early learning workforce, including administrators, so that they have the necessary knowledge, skills, and abilities to improve ELs’ cognitive, health, social-emotional, and dual language development. Early learning programs are designed to improve early learning and development outcomes across one or more of the Essential Domains of School Readiness for children from birth through third grade (or for any age group within this range). Further, we encourage applicants to include in such projects these foundational professional learning domains for educators at all levels of teaching including secondary preparation.

12

AGENDA

NPD Program Priorities – Absolute – Competitive – Invitational

NPD Program Evidence Criteria – “Entry” and “Exit” Criteria

Competitive Preference Priority 1 – Moderate Evidence of Effectiveness

Project Evaluation – WWC with reservations

Resources and Reminders

13

DISTINCTION BETWEEN ENTRY AND EXIT EVIDENCE

Encourage the use of evidence to increase the effectiveness of projects funded by the National Professional Development program (“entry” criteria—to encourage applications for evidence-BASED projects)

– Competitive Preference Priority 1: Moderate Evidence of Effectiveness

Develop evidence base of effective professional

development practices for educators of ELs (“exit” criteria—to encourage applications for evidence-BUILDING projects)

– Project Evaluation (Section D of selection criteria)

14

AGENDA

NPD Program Priorities – Absolute – Competitive – Invitational

NPD Program Evidence Criteria – “Entry” and “Exit” Evidence

Competitive Preference Priority 1 – Moderate Evidence of Effectiveness

Project Evaluation – Meeting WWC Standards

Resources and Reminders 15

A STUDY PROVIDING MODERATE EVIDENCE OF EFFECTIVENESS…

(cf. Education Department General Administration Regulations, Title 34 of CFR, Part 77)

1. Is a quasi-experimental design study (QED) or an experimental design study (e.g., RCT) that Meets WWC Standards With or Without Reservations [if the study Meets WWC Standards With Reservations, it also needs to include a large sample* (350+ students, or 50+ clusters of at least 10 students/each) and a multi-site sample* (> 1 LEA, locality, or state)]

2. Has overlap with the populations or settings proposed for the intervention

3. Shows a statistically significant favorable impact with no statistically significant and overriding unfavorable impacts in that study or other studies reviewed & reported by the WWC

*NOTE: Two or more studies can cumulatively meet the large sample & multi-site sample requirements, as long as the other requirements are met

16

CPP 1: MODERATE EVIDENCE OF EFFECTIVENESS

To receive points for Competitive Preference Priority (CPP) 1, an application for an NPD grant must be supported by moderate evidence of effectiveness

An applicant should identify up to two study citations to be

reviewed against WWC standards for the purpose of meeting the Competitive Preference Priority

An applicant should clearly identify these citations in both

the application narrative and the application abstract. The Department will not review a study citation that an applicant fails to clearly identify for the evidence review.

17

HOW TO IDENTIFY RELEVANT STUDIES OPTION 1 - USING THE IES PRACTICE GUIDE

Applicants can reference studies in the IES Practice Guide, Teaching Academic and Literacy to English Learners in Elementary and Middle School http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=19

18

HOW TO IDENTIFY RELEVANT STUDIES USING THE IES PRACTICE GUIDE (CONTINUED)

Using the Practice Guide to identify studies that will meet the moderate evidence priority is a multi-step process.

First, go to Appendix D (pgs. 80-103) of the guide for a listing of all studies that provide evidence for the recommendations in the guide.

Note: whether the “Domain Effect Size” is statistically significant; RCT vs. QED.

19

HOW TO IDENTIFY RELEVANT STUDIES USING THE IES PRACTICE GUIDE (CONTINUED) After you have identified a study, you will need to locate it in the WWC

Reviews of Individual Studies database to see the study’s What Works Clearinghouse rating: https://ies.ed.gov/ncee/wwc/ReviewedStudies

Note: This database will provide you with the study rating, but you may need to examine the practice guide tables in Appendix D and/or the original study to determine whether the study meets the other Moderate Evidence requirements.

20

HOW TO IDENTIFY RELEVANT STUDIES OPTION 2 – PROPOSE ANOTHER STUDY

Applicants can also propose to use studies that are not included in the Teaching Academic and Literacy to English Learners in Elementary and Middle School Practice Guide, provided that studies meet the requirements of Moderate Evidence

21

22

OVERVIEW: WHAT KINDS OF STUDIES QUALIFY UNDER MODERATE EVIDENCE?

ORIGINAL STUDIES OF THE EFFECTIVENESS OF INTERVENTIONS

IN Publicly-available original research studies Peer-reviewed or not Full written technical document (report, article), final version of the results Studies by developers of their own product or independent evaluators

OUT Newspaper articles Literature reviews, briefs, summaries, syntheses Secondary research (re-analysis of others’ data) Implementation studies Correlational studies Descriptive studies of baselines and trend lines

PROCESS FOR ASSESSING RESEARCH EVIDENCE

The WWC reviews studies using a relevant topic-specific protocol where possible.

– Review Protocols guide all WWC reviews https://ies.ed.gov/ncee/wwc/Protocols

– For more information on the WWC standards and review process, please see the “Handbooks and Other Resources” section of the WWC’s website: https://ies.ed.gov/ncee/wwc/Handbooks

23

RESPONSIBILITY FOR THE “ENTRY” EVIDENCE REVIEWS

The What Works ClearinghouseTM (WWC) conducts the review of “entry” evidence for CPP 1.

– ED staff sends evidence citations to WWC contractor to conduct reviews.

ED staff working with the WWC team provide the WWC reviews, as well as information about the intervention, sample, and setting for each study reviewed to OELA.

OELA makes a determination about the relevance of

the intervention, sample, and setting of the study compared to what the applicant has proposed.

24

EVIDENCE CITATION SUBMISSIONS

ED will limit reviews to studies explicitly cited in the application narrative and application abstract for CPP1

Applicants must ensure evidence is available to ED from publicly available sources and provide links or other guidance indicating where it is available.

– Or an applicant may include copies of evidence in Section 6: Other Forms.

Include only citations that are relevant to the intervention being proposed.

For evidence that will go through WWC review, include only citations that are primary analyses of the effect of the intervention being proposed.

25

AGENDA

NPD Program Priorities – Absolute – Competitive – Invitational

NPD Program Evidence Criteria – “Entry” and “Exit” Evidence

Competitive Preference Priority 1 – Moderate Evidence of Effectiveness Project Evaluation

– WWC with reservations Resources and Reminders

26

SELECTION CRITERIA QUALITY OF THE PROJECT EVALUATION (UP TO 20 POINTS)

27

The Secretary considers the quality of the evaluation to be conducted of the proposed project. In determining the quality of the evaluation, the Secretary considers the following factors: (1) The extent to which the methods of evaluation are thorough,

feasible, and appropriate to the goals, objectives, and outcomes of the proposed project.

(2) The extent to which the methods of evaluation will, if well implemented, produce evidence about the project’s effectiveness that would meet the What Works Clearinghouse Evidence Standards with reservations.

(3) The extent to which the methods of evaluation will provide performance feedback and permit periodic assessment of progress toward achieving intended outcomes.

PROJECT EVALUATION

Plan for design and execution of the evaluation must be able to meet WWC standards with reservations

Implications – must have: – Definition of the intervention – Student outcome(s) or other relevant outcome(s) – Strategy for assigning individuals to the intervention

group or to a control/comparison group – Plans to

Collect data on intervention and outcomes Analyze data statistically Report on findings

28

AGENDA

NPD Program Priorities – Absolute – Competitive – Invitational

NPD Program Evidence Criteria – “Entry” and “Exit” Evidence

Competitive Preference Priority 1 – Moderate Evidence of Effectiveness

Program Evaluation – WWC with reservations

Resources and Reminders

29

RESOURCES FOR DESIGNING PROJECT EVALUATIONS TO MEET WWC STANDARDS

Technical Assistance Materials for Conducting Rigorous Impact Evaluations http://ies.ed.gov/ncee/projects/evaluationTA.asp

What Works Clearinghouse Handbooks http://ies.ed.gov/ncee/wwc/Handbooks

WWC webinars on Designing Strong Studies and QEDs http://ies.ed.gov/ncee/wwc/Multimedia/18 http://ies.ed.gov/ncee/wwc/Multimedia/23

Additional Resources on the Design of QEDs http://www.dir-online.com/wp-content/uploads/2015/11/Designing-and-Conducting-Strong-Quasi-Experiments-in-Education-Version-2.pdf

Free, IES-supported Software for Analyzing Impacts using RCTs or QEDs https://www.rct-yes.com/

30

DATES TO REMEMBER

Feb. 21, 2017 - application period opened

Pre-Application webinars March 15, 2017 – Webinar #1 March 22, 2017 – Webinar #2 April 6, 2017 – Webinar #3

April 24, 2017 - application period closes August 2017 - awards announced* Grant period begins after awards are announced * Dates are tentative and subject to change.

31

APPLICATIONS DUE APRIL 24, 2017

IMPORTANT DATE

Questions?

THANK YOU