Studying Fidelity of Implementation (FOI):

How FOI influenced SCALE-uP’s Theory of Action for Middle School Science

Curriculum Materials

*SCALE-uP = Scaling up Curriculum for Achievement Learning and Equity Project, a partnership between George Washington University

and Montgomery County Public Schools**

Sharon Lynch, PICo-PIs: Curtis Pyke, Joel Kuipers, Michael Szesze** & Bonnie Hansen-

Grafton**

http://www.gwu.edu/~scale-up/

Prepared for Researchers Without Borders Webinar, May 26, 2010

FOI Research Group

• Carol O’Donnell • Suzanne Merchlinski & MCPS evaluation staff• Bonnie Hansen-Grafton • Joelle Lastica• Vasuki Rethinam• Bill Watson• Rob Ochsendorf• Liz Hatchuel• Annie HansenWith special thanks to MCPS middle school science

teachers who participated in this study

Disclaimer

• Indebted to the Interagency Educational Research Initiative (IERI) administered by the NSF, for research funding for SCALE-uP (7 year research program).

• I am currently working at NSF as a Program Director in EHR/DRL ( return to my position as a professor at GWU in September).

• However, the ideas and opinions discussed here are entirely my own and in no way represent the those of NSF.

SCALE-uP/FOI Webinar:Cautionary Tale w/ Happy Ending • Background of SCALE-uP and initial Theory of Action• Year 0 Pilot Study: Curriculum modifications are

tricky business!• Year 1+: Comparison groups are (incredibly) handy in

developing FOI instruments and understanding the study context

• Year 2 & 3: Comparison group and FOI evidence are crucial for credible evidence of effectiveness

• Year 4: Putting it all together: How the Theory of Action was changed by FOI evidence: FOI as “process” and “structure” constructs for both teacher and student

Background for SCALE-uP and Initial Theory of Action

• In 1990’s, AAAS Project 2061 developed a Curriculum Analysis to identify curriculum materials likely to help students learn a target idea (benchmark/standard).

• Curriculum Analysis relied on experts’ judgment of written curriculum materials.

• Two parts:

--Focused, accurate, coherent content on a standard/benchmark

--Instructional strategies contained in written curriculum materials

Project 2061 Instructional Strategies for curriculum materials

1. Convey sense of purpose 2. Address student ideas and

misconceptions3. Promote engagement with relevant

phenomena4. Developing, using scientific ideas 5. Encourage student thinking6. Encourage assessment of progress7. Creating positive learning environment:

curiosity, all students

AAAS. Project 2061.

Background for SCALE-uP

• Project 2061 Curriculum Analysis had located only 2 acceptable curriculum units in middle school science.

• Units had been field-tested with small numbers of students (no comparison groups).

• Note. More mathematics curriculum materials had acceptable ratings and were field-tested and studied and scaled.

Background for SCALE-uP and Research Questions

If science curriculum materials having Project 2061 attributes were studied in a series of large (N = ~ 2000) quasi-experiments using carefully matched comparison groups:

• Would they be effective?• Would they be equitable? • Would there be a relationship between fidelity

of implementation to a unit and student outcomes?

• Could the materials be scaled-up in this large school district?

• How did they function in classroom (video-ethnography)?

SCALE-uP’s Interventions: 3 Science Units with Coherent

Focused on Target Ideas• State of Michigan’s Chemistry That Applies

(CTA) focuses on conservation of matter. 8th graders, unit ~ 6 weeks long.

• GEMS Lawrence Hall of Science Real Reasons for the Season (Seasons) focuses on the reasons for the Earth’s seasons. 7th graders, unit, ~ 3 weeks.

• ARIES Harvard Smithsonian Motion and Forces (M&F) focuses on portions of Newton’s Laws. 6th graders, unit ~ 6 weeks long.

Curriculum Analysis: Instructional Strategies

Instructional Category Chemistry That

Applies

ARIESMotion & Forces

GEMSSeasons

Macmillan/McGraw Hill

I. Identifying a Sense of Purpose

Conveying Unit Purpose ○ ○ NR ◔ Conveying lesson/activity purpose ◕ ◒ ● ◔ Justifying lesson/activity sequence ◒ ◒ ◒ ○II. Taking Account of Student Ideas

Attending to prerequisite knowledge and skills ◒ ○ ○ ○ Alerting teacher to commonly held ideas ◒ ○ NR ○Assisting teacher in identifying own students’ ideas ◒ ◒ ○ ○Addressing commonly held ideas ◒ ○ ◒ ○

● =Excellent, ◕=Very Good, ◒=Satisfactory, ◔=Fair ○=Poor

Instructional Category Chemistry That Applies

ARIES

Motion & Forces

GEMS

Seasons

Macmillan/

McGraw Hill

III. Engaging Students with Relevant Phenomena

Providing a variety of phenomena ● ● ○ ○ Providing vivid experiences ● ● ◒ ○ IV. Developing and Using Scientific Ideas

Introducing terms meaningfully ● ◒ ◒ ◔ Representing ideas effectively ◒ ◒ ● ○Demonstrating use of knowledge ◕ ○ ◒ ○Providing practice ● ○ ○ ○V. Promoting Student Thinking about Phenomena, Experiences, and Knowledge

Encouraging students to explain their ideas ● ◒ ○ ○Guiding student interpretation and reasoning ● ○ ● ○Encouraging students to think about what they’ve learned ○ ○ ○ ○

● =Excellent, ◕=Very Good, ◒=Satisfactory, ◔=Fair ○=Poor

SCALE-uP’s Outcome Measures

• Curriculum-independent measure for each unit focusing on the unit’s target idea.

• Assessments had good psychometric properties and were developed using a Project 2061 assessment system.

• Multiple choice and constructed response items designed to be maximally accessible to students of varied language skills.

Questions?

Background for SCALE-uP and Initial Theory of Action, c. 2001

• Curriculum units highly rated on Curriculum Analysis could be effective overall because: – each focused coherently on one big idea/standard/

benchmark– each had a carefully planned sequence of activities,

and – each contained identified instructional strategies

leading students to construct understanding of one target idea/benchmark/standard.

Big Question: Would they be equitable?Assumption: “Business as usual” comparison

classrooms would be less focused, rely more on textbooks and worksheets, and provide less time for guided inquiry and lab work.

“Typical” Theory of Action

TeacherCurriculum Materials

Student Outcomes

Fidelity of implementationMowbray, Holter, Teague & Bybee, 2003

• “Fidelity of implementation is the extent to which the delivery of an intervention adheres to the original program theory behind its development; it confirms that the implementation of the independent variable in outcome research occurred as planned…

• …(and involves) the dynamic nature of fidelity criteria, appropriate validation and statistical analysis methods, the inclusion of structure and process criteria in fidelity assessment and the role of program theory in deciding on the balance between adaptation versus exact replication of model programs.”

SCALE-uP Theory of Action c. 2001

TeacherCurriculum Materials

Student Outcomes

Teacher FOIInstructionalStrategies

SCALE-uP Theory of Actionc. 2001

• Teachers would need to locate and enact the instructional strategies embedded in the curriculum unit (identified via the Project 2061 Curriculum Analysis).

• Doing this well would be to “implement with fidelity”.• Hypothesis: The better the implementation of the

unit’s embedded instructional strategies in a classroom, the higher the student outcomes.

• All SCALE-uP needed to do to create a classroom observation instrument that captured teachers’ implementation of the strategies.

• This would become Instructional Strategies Classroom Observation Protocol (more on this later).

• Simple!

Questions?

On to Fidelity of Implementation (FOI)!

Year 0 (Pilot Study) Results: Chemistry That Applies (CTA) with Modifications Encouraged

• CTA’s results showed effect sizes = .52.• Results disaggregated for subgroups of students showed

that no students disadvantaged by CTA. • Teachers were asked to modify CTA for diverse learners

and record modifications.• Virtually none did! This raised questions about whether

to modify CTA in future. Teacher beliefs about the unit were mixed.

• Co-PI Szesze wanted to be sure units were unambiguously effective, or not.

• Decision to “implement with fidelity” in the future studies . • Teachers and researchers drew up fidelity guidelines

together.

Lesson Learned :Modifying Curriculum Unit is Tricky Business!

• If an intervention does not have solid

evidence of efficacy/effectiveness, then modifications muddy the water; CTA and the other units were “unproven” interventions.

• Therefore, their critical components could not be “known”, but assumed.

• In retrospect, this was a good decision for SCALE-uP; capturing modifications while trying to establish if a unit “worked” would not be credible in this school district context.

Iterative Process of Identifying Critical Attributes, Measuring Attributes, and

Looking for Relationships between FOI and Outcomes

Identify CriticalAttributes of Intervention

Outcomes

Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3),315-340.

FOI measureReconsider

CriticalAttributes

Questions?

On to Year 1

Replication of CTA

Building the first FOI Instrument

Yr. 1 Replication of CTA and Develop ISCOP/FOI Process

• Given Theory of Action (the more teachers implemented Project 2061 instructional strategies embedded in curriculum units, the higher student outcomes), we began to develop an instrument that could capture fidelity to identified Project 2061 instructional strategies.

• A “generic” instrument for all 3 units studied, but the units had been carefully vetted and had much in common.

• ISCOP (Instructional Strategies Classroom Observation Instrument), the first FOI measure was born, and developed over the next 4 years.

Effect Sizes: CTA (Year 1)

Years 1, 2, 3, 4: Test ISCOP in Treatment and

Comparison Classrooms• Was the ISCOP capturing fidelity to instructional

strategies in a way that discriminated between Treatment and Comparison classrooms?

• ISCOP did not discriminate very well, so it was refined and refined and refined.

• Was ISCOP a “bad” instrument? Or were Treatment and Comparison classrooms similar for Instructional Strategies?

• If Treatment and Comparison classrooms were similar in strategies, should the Theory of Action be revisited?

Lesson Learned: Comparison Classrooms Invaluable as Counterfactuals

• ISCOP data suggested that more complexity than assumed in the Theory of Action.

• Measuring FOI relying solely with ISCOP might not answer FOI research question if comparison classroom data were taken into consideration.

• Generic measures of instructional strategies/FOI process are notoriously hard to develop—valid and reliable?

Questions?

On to Years 2 and 3 with a new unit, Real Reasons for the Seasons

(Seasons)

Year 2 and 3 Results: Comparison group outscores

Seasons group: How to account for this?

• Seasons studied in 7th grade classrooms (N ~ 2000 students and ~ 40 classrooms).

• Year 2 and 3 results: Comparison group has higher outcome than Seasons on curriculum independent measure (ES = -.36, -.18, respectively).

Comparison Classrooms Invaluable as Counterfactuals

• Comparison classroom teachers surveyed and interviewed.

• Data showed that Comparison classrooms were:

-Non-traditional-Variety of curriculum materials used,

including other inquiry units-Focused on the target idea -Equal Duration of Treatment and

Comparison units-Comparison and Treatment teacher

characteristics similar.

Seasons Replication in Year 3: Focus on FOI

• Use ISCOP in Treatment and Comparison classrooms.

• Develop a new Lesson Flow FOI process measure that gauged Teacher, Student-Group or Individual Centeredness of classrooms based on hunch that Student-Group Centeredness was important to student construction of science ideas in groups.

• Videotaped a Seasons and Comparison classroom.

• Interviewed and surveyed Seasons and Comparison Teachers again.

Lesson Flow Classroom Observation Instrument

Lesson Flow Classroom Observation Protocol (LFCOP)

Effect Sizes: Seasons (Year 3)Overall ES = -.18

Year 3 FOI Results• Only 3 items on ISCOP showed significant

differences between Seasons and Comparison classrooms, and 2 of 3 were observed more often in Comparison classrooms.

• Lesson Flow: Seasons classrooms were more teacher-centered than Comparison:

Teacher-CenterednessSeasons = 71 % of timeComparison = 58 % of time

• Video-data backed this up.

• Emerging Conjecture: Students need time in to work and talk in groups to develop their ideas and Seasons allowed less time for this.

SCALE-uP Theory of Action c. 2006

TeacherCurriculum Materials

Student Outcomes

Teacher FOIInstructionalStrategies

Student FOIStudent GroupCenteredness

Changing Theory of Action and Approach to FOI c. 2006

• Focusing FOI entirely on teacher in a guided inquiry unit may be simplistic.

• Student agency implicated in FOI constructs.• But Lesson Flow is controlled by the teacher

who provides students with the time/space to work in groups; students can choose to do the science work, or not.

• Lesson Flow is not Time On Task.

Seasons Unit and FOI:The Research Re-visited

• Seasons was designed to be a supplementary unit.

• Developers seemed not to understand the implications of this when agreement was made to study effectiveness/FOI.

• SCALE-uP researchers did not appreciate the differences in philosophy of Seasons with research design.

• Seasons was a bad match for this study.• Revisions made to Seasons based upon

feedback from MCPS teachers.

Questions?

On to Year 4 and Motion and Forces(M&F)

Note this is the third curriculum unit studied, not to be confused with the first two, and was

implemented in 6th grade classrooms.

M&F Results in Years 2 and 3:Underwhelming

• Year 2 ES = +.10• Year 3 ES = -.06• FOI had not been emphasized, so there was

little information to explain lackluster outcomes.

• However, we learned that students had not been issued M&F student Journals in Years 2 and 3. Was this an important, overlooked FOI issue?

Year 4 Replication of M&F: Focus on FOI

• Replicated quasi-experiment in 10 new schools (N ~ 2000 students).

• Teachers asked to focus on FOI.

• Wonderful teachers who understood the study goal: study the impact of M&F.

FOI in Year 4 with M&F

FOI included:• ISCOP (Process FOI)• Lesson Flow (Process FOI)• Adherence to unit (ACOP), a new instrument

that measured close adherence to M&F’s structure (Structure FOI)

• Teacher Interviews/Surveys• Teacher Logs • Student Journal Entries, # of responses to

Journal (Structure FOI)• Student survey about self-reporting use of

instructional strategies (Process FOI)

Overall Results for M&F in Year 4

• Student Level results using traditional ANOVA, ES = .23

• Classroom Level results using HLM,

ES = .56 (Rethinam, Lynch, & Pyke, 2008)

Effect Sizes for subgroups of students: M&F (Year 4)

ISCOP Strategies: Means and Correlations with Outcomes for M&F

Instructional Criterion Mean (Scale = 0 - 3) Correlation with Outcomes

Treatment Comparison Treatment Comparison

Identifying a Sense of Purpose

Conveying lesson/activity purpose 0.65 0.58 .57** -.28

Justifying lesson/activity purpose .46 .58 .56** -.40

Taking Account of Student Ideas

Assisting teacher in identifying own students’ ideas

2.40 1.38 -.05 -.52**

Engaging Students with Relevant Phenomena

Providing variety of phenomena 1.71 1.50 .44* -.23

Providing vivid experiences 2.75 2.46 .13 .08

Developing and Using Scientific Ideas

Introducing terms meaningfully 2.48 2.38 .60** -.09

Representing ideas effectively 1.46 1.13 .16 -.01

Promoting Student Thinking about Phenomena, Experiences, Knowledge

Encouraging students to explain their ideas

2.13 2.25 .62** -.20

ISCOP• This “generic” instrument for measuring FOI

instructional strategies did not obviously distinguish between M&F and Comparison classrooms.

• But some items were correlated with student outcomes for M&F classrooms.

• O’Donnell dissertation elegantly teased out which instructional strategies seemed to matter for higher student outcomes.

• ISCOP needs more work on validity and reliability--OR SCALE-uP Theory of Change needs refinement—Project 2061 Curriculum Analysis needs refinement?

TeacherIndividual

Group

TeacherGroup

Individual

M&F Comparison

Lesson Flow for Instruction for units on Motion and Force

Lesson Flow

• M&F classrooms provided more time for students to work in groups and individually than Comparison classrooms.

• Students construct meaning in groups, consistent with notion of community of practice and situated cognition.

Adhering to M&F Lesson Components (ACOP)

• M&F teachers adhered to M&F unit > 80% of time.

• ACOP did not predict outcomes because the range was narrow; this was a good thing for this study because high fidelity in this measure of “structure” is credible and strong—good face validity for FOI, and highly reliable—the unit was well-implemented.

• Teachers adhered to the unit for this study, to ascertain impact of M&F.

• Thank you teachers.

Results: Teacher FOI Structure

Adherence Classroom Observations Protocol ACOP Item

n %Not implemented

%Somewhat implemented

%Mostly implemented

%Fully implemented

Teacher provides students with materials

30 -- -- 53.3 46.7

Teacher provides opportunity to write responses

30 -- -- 10.0 90.0

Teacher discuss with students responses to questions

30 13.3 3.3 3.3 80.0

Teacher provides opportunity to set up equipment

30 10.0 -- 33.3 56.7

Teacher provides opportunity to record and/or discuss predictions

8 12.5 -- -- 87.5

Teacher provides opportunity to conduct investigation

30 -- 3.3 6.7 90.0

Teacher provides opportunity to reflect and write

20 15.0 -- -- 85.0

Interpreting results and discussion in writing

30 -- -- 13.3 86.7

Interpreting results and discussion verbally

30 13.3 6.7 20.0 60.0

Teacher Guide Additions

24 41.7 41.7 8.3 8.3

Adhering to sequence of activities

30 -- 3.3 46.7 50.0

Student Journals

• M&F students completed their journal responses ~ 80% of time.

• Rates of journal question completion predicted classroom outcomes.

Unpublished Year 4 HLM Results for M&F: ISCOP, ACOP, Lesson Flow,

Student Journals• HLM analysis found one classroom-level

factor that predicted student outcomes:

--Amount of Student-Group-Centeredness (Lesson Flow)

• and one student level factor

--Student Journal Completion • Thanks to Dr. Jaewa Choi of GWU for this data analysis.

SCALE-uP Theory of Action c. 2010

TeacherCurriculum Materials

Student Outcomes

TeacherInstructionalStrategies

Teacher Adherence to Lesson Structure

Student GroupCenteredInstruction

Student Adherenceto Lessons

ISCOP ACOP

Lesson Flow Student Journal

ProcessFOI

ProcessFOI

StructureFOI

StructureFOI

SCALE-uP Theory of Action c. 2010

TeacherCurriculum Materials

Student Outcomes

TeacherInstructionalStrategies

Teacher Adherence to Lesson Structure

Student GroupCenteredInstruction

Student Adherenceto Lessons

ISCOP ACOP

Lesson Flow Student Journal

SCALE-uP Theory of Action c. 2010

Teacher

Curriculum Materials

Student Outcomes

TeacherInstructionalStrategies

Teacher Adherence to Lesson Structure

Student GroupCenteredInstruction

Student Adherenceto Lessons

ISCOP ACOP

Lesson Flow Student Journal

Students

SCALE-uP Theory of Action c. 2010

Teacher

Curriculum Materials

Student Outcomes

TeacherInstructionalStrategies

Teacher Adherence to Lesson Structure

Student GroupCenteredInstruction

Student Adherenceto Lessons

ISCOP

ACOP

Lesson Flow

Student Journal

Students

Questions?

Summary Findings• Score Card: CTA and M&F seemed to be effective

and equitable in this school system, Seasons did not seem to be effective.

• Theory of Action: Changed from looking at one relatively “generic” measure of teacher process fidelity to multiple measures of FOI adding student FOI and measures of FOI structure, and developing a better Theory of Action.

• Lesson Flow and Student Journal response best predicted student outcomes; ISCOP analysis illuminated most potent instructional strategies for M&F.

FOI Lessons Learned• Studying curriculum adaptations for an

“unproven” curriculum is a slippery slope because critical ingredients are unknown.

• Study of FOI in Comparison groups provides important counterfactual.

• This study shows potential for both teacher and student FOI, as well as process and structural approaches.

• Study of FOI should reveal more about critical components of an intervention if measures of each component shows a positive relationship with student outcomes.

The End

Thanks to Researchers Without Borders for sponsoring this Webinar.

References• American Association for the Advancement of Science (AAAS). (2003). Project 2061 middle grades

science textbooks: A Benchmarks-based evaluation. Retrieved June 1, 2004, from http://www.project2061.org/tools/textbook/mgsci/index.htm.

• Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research Theory and Practice, 18(2), 237-256.

• Harvard-Smithsonian Center for Astrophysics. (2001). ARIES: Exploring motion and forces: Speed, acceleration, and friction. Watertown, MA: Charlesbridge Publishing

• Kesidou, S., & Roseman, J.E. (2002). How well do middle school science programs measure up? Findings from Project 2061’s curriculum review. Journal of Research in Science Teaching, 39(6), p. 522-549

• Lastica, J.R., & O'Donnell, C.L. (2007, April). Considering the role of fidelity of implementation in science education research: Fidelity as teacher and student adherence to structure. In C. O'Donnell (Chair), Analyzing the relationship between Fidelity of Implementation (FOI) and student outcomes in a quasi-experiment. Symposium conducted at the Annual Meeting of the American Educational Research Association, Chicago, IL.

Lawrence Hall of Science. (2000).GEMS: The real reasons for seasons—Sun-Earth connections. Berkeley: The Regents of the University of California.

• Lynch, S. (2000). Equity and science education reform. Mahwah, NJ: Lawrence Erlbaum and Associates.

• Lynch, S., Kuipers, J.C., Pyke, C., & Szesze, M. (2005). Examining the effects of a highly rated science curriculum unit on diverse students: Results from a planning grant. Journal of Research in Science Teaching, 42(8), 912-946.

• Lynch, S. & O’Donnell, C. (2005, April). The evolving definition, measurement, and conceptualization of fidelity of implementation in scale-up of highly rated science curriculum units in diverse middle schools. In S. Lynch (Chair), The role of fidelity of implementation in quasi-experimental and experimental research designs: Applications in four studies of innovative science curriculum materials and diverse student populations. Symposium conducted at the Annual Meeting of the American Educational Researchers Association, Montreal, Canada.

• Lynch, S., O'Donnell, C., Hatchuel, E., & Rethinam, V. (2007, April). A model predicting student outcomes in middle school science classrooms implementing a highly-rated science curriculum unit. Paper presented at the Annual Meeting of the National Association for Research in Science Teaching, New Orleans, LA.

• Lynch, S., O'Donnell, C., Hatchuel, E., Rethinam, V., Merchlinsky, S., & Watson, W. (2006, April). What’s up with the Comparison group?: How large quasi-experimental study of highly rated science curriculum units came to grips with unexpected results. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

• Lynch, S., Taymans, J. Watson, W., Ochsendorf, R., Pyke, C. & Szesze, M. (2007). Effectiveness of a highly-rated science curriculum unit for students with disabilities in general education classrooms. Exceptional Children, 73(2), 202-223.

• Merchlinsky, S. & Hansen-Grafton, B. (2007, April). Considering the role of Fidelity of Implementation (FOI) in science education research: Evaluation and science specialists' role in collecting FOI data in a large school district. In C.L.

• Michigan Department of Education. (1993). Chemistry That Applies. The State of Michigan.

• Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3),315-340.

• O’Donnell, C. L. (2007). Fidelity of implementation to instructional strategies as a moderator of curriculum unit effectiveness in a large-scale middle school science experiment. Dissertation Abstracts International, 68(08). (UMI No. AAT 3276564)

• O'Donnell, C.L., Lynch, S., Lastica, J., & Merchlinsky, S. (2007, April). Analyzing the relationship between Fidelity of Implementation (FOI) and student outcomes in a quasi-experiment. Symposium conducted at the Annual Meeting of the American Educational Research Association, Chicago, IL.

• O'Donnell, C.L., Lynch, S., Watson, W., & Rethinam, V. (2007, April). Teacher and student Fidelity of Implementation (FOI) to process: Quality of delivery and student responsiveness and relationships to classroom achievement. In C.L. O'Donnell (Chair), Analyzing the relationship between Fidelity of Implementation (FOI) and student outcomes in a quasi-experiment. Symposium conducted at the Annual Meeting of the American Educational Research Association, Chicago, IL.

• O'Donnell, C. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78(1), 33-84.

• O'Donnell, C., & Lynch, S. (2008, March). Fidelity of implementation to instructional strategies as a moderator of science curriculum unit effectiveness. Paper presented at the 2008 annual meeting of the American Educational Research Association, New York, NY.

• Rethinam, V., Pyke, C., & Lynch, S. (2008). Using multilevel analyses to study the effectiveness of science curriculum materials. Evaluation and Research in Education, 21(1), 18-42

• Rethinam, V., Pyke, C., & Lynch, S. (in preparation.). Using Multilevel Analyses to Study Individual and Classroom Factors in Science Curriculum Effectiveness.

• Songer, N. B., & Gotwals, A. W. (2005, April). Fidelity of implementation in three sequential curricular units. In S. Lynch (Chair), “Fidelity of implementation” in implementation and scale-up research designs: Applications from four studies of innovative science curriculum materials and diverse populations. Symposium conducted at the meeting of the Annual Meeting of the American Educational Research Association. Montreal, Canada.

• Stern, L. & Ahlgren, A. (2002). Analysis of students’ assessments in middle school curriculum materials: Aiming precisely at benchmarks and standards. Journal of Research in Science Teaching, 39, 889-910.

• Watson, W., Lynch, S., Rethinam, V., & O’Donnell, C. (2006, April). Development of an instrument to measure student responsiveness to implementation of science curriculum materials. Paper given at the annual meeting of the National Association for Research in Science Teaching, April, 2006 , San Francisco.

• Further acknowledgments of important contributions to this work on fidelity of implementation made by Rob Ochsendorf, Aiyita Ruiz-Primo, Doug Clement, Okhee Lee, Bruce Ward, Carolyn Walton, Theron Blakeslee, Andy Anderson, Phyllis Blumenfeld.