Learning from the

National Evaluation of

i3: Challenges,

responses, and future

plans

i3 Learning Community Meeting

September 11-12, 2014

pg

What We Have Learned

A grant program/agency can incentivize high-quality evaluations

Across the first 3 cohorts, the majority of the studies have the potential to meet NEi3 standards with or without reservations

– 89% of all grants have potential to meet standards

– 83% of development grants using RCTs, QED (twice as many QEDs)

– 100% of validation and scale-up grants using RCTs or QEDs (about half and half)

National Evaluation of i3 1

pg

What We Have Learned

At the same time, this emphasis and the success of helping evaluators design strong studies does not eliminate risks that can undermine what can be learned

– Low power: 17 development grants (as designed), 3 validation grants (as implemented)

– Serious confounds: 6 development grants, 2 validation grants, 1 scale-up grant (1 of multiple studies)

– Only studying part of the full i3-supported intervention or sample: 2 development grants, 2 validation grants, 1 scale-up grant

– Dropping down in rigor of design: 6 development grants proposed QEDs but ended up implementing promise evaluations

National Evaluation of i3 2

pg

What We Have Learned

Institutionalizing rewards for prior evidence has increased attention on/efforts to implement strong studies

– OII and other federal agencies are increasingly focused on evidence—as rationale for funding and as a result of funding

– OII/IES are increasingly clear about the evidence requirements for winning another i3 grant—bar has been set higher

In earlier meeting, this link to evidence cited as one definition for the “innovation” in Investing in Innovation

National Evaluation of i3 3

pg

What We Have Learned

A tiered evidence program such as i3 faces issues

about appropriate design standards for lower level

grants

– Many of the development grants are developing,

investigating innovative approaches

– Guidance on rigorous designs may be misaligned with stage

of intervention

• Focus on implementation may be more aligned?

• How to weigh innovation versus need for believable evidence

National Evaluation of i3 4

pg

What We Can Learn

Effectiveness of each individual

intervention/approach at improving student

achievement

– Which interventions have been shown to be

“effective” and “promising”

This is the smallest ‘grain size’

5 National Evaluation of i3

pg

What Else Can We Learn

Can we learn something more systematic about

educational interventions, can we go beyond “one-off”

evaluations

Is there a way to group interventions and summarize

findings within groups

– WWC has moved away from topic area summaries and

towards intervention reports

– Dangers of summarizing effects of groups of i3 interventions

6 National Evaluation of i3

pg

What Else We Can Learn

Group the i3 interventions by

–Absolute Priority?

–Outcome domains?

– Intervention components?

7 National Evaluation of i3

pg

Group by broad policy area/goal

National Evaluation of i3 8

Author, Affiliation, i3 role;

FY2010-2013 i3 Grants by Absolute Priority

pg

Group by intended outcome?

National Evaluation of i3 9

Author, Affiliation, i3 role;

80

43

18

7

9

8

7

5

4

5

5

3

2

4

4

4

4

4

4

11

4

6

3

1

1

2

1

1

1

1

1

1

0 20 40 60 80 100

ELA achievement

Math achievement

Science achievement

Social studies achievement

Classroom/school climate

General Achievement

Attitudes / beliefs / self-efficacy / motivation

Attendance

Understanding of content / pedagogical content knowledge

Instructional quality or effectiveness

College/career readiness

High school graduation

Retention

Engagement / attentiveness / involvement

Use and expertise of assessment, monitoring, measures of achievement

Collaboration among students/teachers/admin/staff

Quality of leadership

Language/Literacy development

Cognitive development/ Executive functioning Meet NEi3 Criteria With or Without Reservations

Provide Evidence of Promise

Will not meet NEi3 criteria for grant type

*nearly all grants are conducting multiple contrasts, some across multiple domains

pg

Group by approach to affecting change?

National Evaluation of i3 10

Author, Affiliation, i3 role;

78%

57%

41%

27%

24%

18%

18%

16%

16%

8%

8%

6%

6%

4%

2%

Educator / School Leader Professional Development

Curriculum / Instructional materials / Instructional tools / Coursework

Support from Admin / leadership

Learning communities (PLCs, Online networks, etc)

Coaching / Mentoring of Educator / School Leader

Data Collection and analysis; monitoring and/or reporting of data

Selection, recruitment, hiring, and/or placement of Ss/Ts/Others

College / Career Readiness

Parental/community involvement, engagement, support, and training

System of student assessment

Teacher Assessment/Evaluation

Schedule / Calendar

Student Mentoring

Student Case Management

Environmental / Physical structure

0% 20% 40% 60% 80% 100%

*Many grants utilize multiple key components

pg

Revisiting Original Goal of Learning Community

Trying to identify site-level contextual factors that help

us understand variation in effects across sites

– Site-level variation can be studied only in evaluations where each site has

an effect (i.e., each site has treatment and comparison units)

Grantees selected for this learning community based on having

evaluation designs that would allow for studying site-level

variation

The field has not moved far in measuring common contextual

factors

National Evaluation of i3 11