Researchers Without Borders Webinar 4

A Framework and Suite of Instruments for Examining Fidelity of Implementation

Jeanne CenturyCenter for Elementary Mathematics and Science Education

University of ChicagoJuly 7, 2010

This work has been supported in part by the National Science Foundation Project ESI-0628052

Why Measure Fidelity of Implementation (FOI)?

Determine if programs are effective

Determine which parts of programs are effective

Determine why and under what conditions parts of programs are effective

What is actually happening?

NSF Project Challenges

Instruments that could be used across multiple programs

A suite of instruments that could be used alone or in combination depending on resources and goals

Provide detailed and specific information about FOI

Five Dimensions Approach Dane and Schneider (1998)

39/162 studies measured integrity In the 39 studies, the authors defined “integrity” along one or more of five dimensions They are: 1) adherence; 2) exposure; 3) quality of delivery; 4) participant responsiveness and

5) program differentiation

Critical Components Approach

Hall and Hord (1987), Bond et al. 2000; Huntley, 2005; Sabelli & Dede (2001) and others

Some called them model dimensions, fidelity criteria, critical parts

Structure and Process ApproachMowbray et al. (1987), Wang et al. (1984)

Structure (composition of the intervention)Process (human interactions that take place during delivery)

FOI Approaches…

Combining Frameworks Approach

Ruiz Primo (2005) – matrix of 5 dimensions and structure/processLynch and O’Donnell (2005); Lastica and O’Donnell (2007) – matrix of 5 dimensions and structure/process

CEMSE Approach

combine critical component approach and structure process approachshow relationships to 5 dimensions approach

use combination of approaches to create categories of critical components and a framework that can apply to multiple interventions

Questions?

FOIThe extent to which an enacted program is consistent with the intended program model

Critical ComponentsThe elements of a program model that are essential to its implementation

FOI OperationalizedThe extent to which the critical components of a program are present when the program is enacted

Definitions

Written Materials

Developers

Critical Component Identificationtheory…

Users

ChallengesTheories aren’t always explicitCritical components very in “grain size”The “it” can vary, depending on how the intervention is framed

Critical Component Identificationevidence…

FOI of Instructional Materials

Categories of Critical Components

Structural Instructional

Procedural Educative Pedagogical Student Engagement

FOI Framework

FOI of Instructional Materials

Categories of Critical Components

Structural Instructional

Procedural Educative Pedagogical Student Engagement

Inclusion of all essential segments

within a lesson

Use of class structures

Content background information

Lesson Notes

Teacher facilitation of group work

Teacher facilitation of student autonomy

Students contribute to small group work

Students demonstrate autonomy

FOI Framework and Critical Components

FOI of Instructional Materials

Categories of Critical Components

Structural

Procedural Educative

Time 1. Duration of unit 2. Time spent on instruction

Order 3. Lesson order 4. Order of segments and activities within lesson

Inclusion 5. Inclusion of all essential segments within a lesson 6. Inclusion of all essential lessons

Essential Program Elements 7. Materials presence 8. Writing structures 9. Readings 10. Assessments and assessment tools 11. Content of lesson (procedures, facts, concepts, processes) 12. Use of class structures 13. Use of instructional delivery formats

Non-Essential Program Elements* 14. Projects* 15. Extensions* 16. Additional Resources* 17. Homework*

1. Content background information2. Pedagogy background information3. National standards and benchmarks information*4. Lesson notes

FOI Framework and Critical Components

FOI of Instructional Materials

Categories of Critical Components

Instructional

Pedagogical Student Engagement

Facilitating student engagement with others1. Teacher facilitation of group work2. Teacher facilitation of student discussion

Facilitating student engagement with content3. Teacher facilitation of students doing intellectually challenging

work4. Teacher emphasis on types of content

Facilitating student role as learner5. Teacher facilitation of student autonomy6. Teacher facilitation of students taking risks7. Teacher facilitation of student interest

Pedagogical strategies14. Teacher facilitation of materials, manipulatives and tools15. Teacher use of assessment to inform instruction16. Teacher use of differentiation

Students engage with others1. Students contribute to small group work2. Students engage in discussion

Students engage with content3. Students do potentially intellectually challenging work

Students develop role as learner4. Students demonstrate autonomy5. Students take risks

Students use the materials6. Students do/complete essential activities7. Students do/complete optional or non-essential activities

FOI Framework and Critical Components

Questions?

FOI of Instructional Materials

Categories of Critical Components

Structural Instructional

Procedural Educative PedagogicalStudent

Engagement

Categories of Differentiation

MathematicsAnd Science

Science Only

Mathematics Only

Program Specific

FOI Framework and Differentiation

FOI of Instructional Materials (adherence)

Categories of Critical Components

Structural (structure) Instructional (process)

Procedural (exposure;

dosage)Educative

Pedagogical (quality)

Student Engagement (participant

responsiveness)

Categories of Differentiation (differentiation)

Common Across Interventions

Unique to Interventions

CEMSE Framework Aligned with (Others’ Approaches)

Questions?

InstrumentsTeacher QuestionnaireTeacher Instructional Observation ProtocolTeacher Interview ProtocolTeacher Instructional LogTeacher Attitude QuestionnaireSchool Leader QuestionnaireSchool Leader Interview ProtocolSchool-wide Observation Protocol

Structural Procedural

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

TeacherQuestionnaire

X X X X X X X X X X X X X X X X

School LeaderQuestionnaire

X X

Teacher Interview Protocol

X X X X X X X

School Leader Interview Protocol

X X

Classroom Observation Protocol

X X X X X X X X X X X X

School Wide Observation Protocol

X X X X X X

Teacher Log X X X X X X X X X X X X

First Teacher Log X X X X X

Teacher Attitude Questionnaire

X

Sample Instrument Construct Matrix

FOI of Instructional Materials

Categories of Critical Components

Structural Instructional

Procedural Educative Pedagogical Student Engagement

Categories of Differentiation

Common

Program A

Program B

Program C

Impact of a single programCompare two or more programsCompare programs to “business as usual”Compare two versions of a program

Uses of the Framework and Measures

Developers

U.S. Satellite, Inc. – additional elements

EvaluatorsSchool–based change model – Lindblom High SchoolProgrammatic model – E2SP

Researchers

Trailblazers – look at a critical component in more depth

Programs Science and Technology for ChildrenFOSSSEPUPIESEveryday Mathematics

Science Companion

Jeanne CenturyCenter for Elementary Mathematics and Science Education

University of Chicagojcentury@uchicago.edu

www.researcherswithoutborders.org

FOI Supplement Work Researchers Without Borders

Resources FAQs; Research; Tools and Instruments

Learning Webinars; Collaboration

Community Development www.researcherswithoutborders.org;

This work has been supported in part by the National Science Foundation Project ESI-0628052