the impact of a state systemic initiative on u.s. science teachers and students

DOUGLAS HUFFMAN and FRANCIS LAWRENZ

THE IMPACT OF A STATE SYSTEMIC INITIATIVEON U.S. SCIENCE TEACHERS AND STUDENTS

ABSTRACT. This study investigated the extent to which a State Systemic Initiative (SSI),a National Science Foundation program designed to improve science education across anentire state, implemented in the United States, could reform science education. Impactsthat were measured included teachers’ instructional practices, professional community,influence of the SSI on school policy, external influences on science instruction, and familyinvolvement. In addition, students’ views of instructional practice, school community andfamily involvement were measured. A retrospective comparative design was used to collectsurvey data from 46 middle schools; 23 that had significant amounts of contact with theSSI and 23 matched schools that had little to no contact with the SSI. The results suggestedthere were important differences favoring schools whose science teachers had participatedin the SSI. Included were differences in the use of standards-based instruction, and externalinfluences on science instruction teachers’ influence on policy. No differences between thetwo groups were found for professional community or family involvement. For students,significant differences were only found for access to standards-based instruction. Resultsimply that SSIs can help change specific aspects of the system, but broader impacts aremore difficult to achieve.

KEY WORDS: professional development, reform, science education

INTRODUCTION

As in many countries around the world, the United States is currentlyengaged in an effort to reform science education. With funding from thefederal government numerous new programs were created in an attempt toreform aspects of educational systems. Included were systemic initiativesat both the state and local level, and initiatives with a focus on urban orrural areas. One such program that was funded by the National ScienceFoundation was the State Systemic Initiatives (SSIs) program. The SSIswere designed to broaden the impact, accelerate the pace and increase theeffectiveness of improvements in K-12 science and mathematics educationthroughout an entire state. These initiatives followed closely on the heelsof the development of national standards for science and mathematics inthe United States and were grounded in the belief that significant changewould be most likely in an entire system that was supportive of change.From the National Science Foundation (NSF) point of view the goal was

International Journal of Science and Mathematics Education 1: 357–377, 2003.© 2004 National Science Council, Taiwan. Printed in the Netherlands.

358 DOUGLAS HUFFMAN AND FRANCIS LAWRENZ

to produce an environment where all the components of the educationalsystem in an entire state were coordinated to produce change (Anderson,2002). The expectation was that teacher enhancement efforts, curriculumdevelopment and implementation, teacher preparation programs, informalscience and mathematics education institutions, business and industry, col-leges and universities and state and local policies would all come togetherto achieve a common goal of elevating the teaching and learning of scienceand mathematics for all students. Given this background, the purpose ofthis study was to examine the impact of a state systemic initiative on alarge group of 8th grade science teachers and their students. Specifically,this study sought to determine the extent to which a state systemic initia-tive could impact teachers’ instructional practices, school policy, externalrelations, and family involvement.

THEORETICAL BACKGROUND

While the goals of SSIs are certainly worthy, achieving these goals isclearly a difficult task. Systemic reform is an extremely complex processand it is not clear whether or not systemic reform is even possible. AsFullan and Miles (1992) pointed out, traditional approaches to reform ineducation have historically been unsuccessful. They go on to state thatalthough systemic reform appears to be both more efficient and more ef-fective, there is little empirical evidence to support it. A recent review ofthe statewide systemic initiatives funded by the NSF indicated that whilethere were some positive impacts from the SSIs, the impacts were almostalways uneven and differentially affected some schools, teachers, and stu-dents more than others (Anderson, 2002). In addition, not one of the overtwenty-five SSIs throughout the United States was able to “go to scale”and achieve comprehensive reform.

Research suggests we still know very little about how to drive an ex-ternally developed reform initiative into the classroom (Fullan & Stiegel-bauer, 1991; Scheerens, 1992). Most research on comprehensive schoolimprovement efforts suggests that teaching and learning practices havechanged only modestly despite concerted effort at reform (Tyack & Cuban,1995; Elmore, 1995). Although the school improvement literature has as-sumed that the most appropriate unit for change is the school itself (VanVelzen et al., 1984), the focus of what is to be improved must be within theclassroom, where students and teachers create the most powerful learningcommunities. In order to obtain classroom effects in a school, there must besupportive conditions throughout the organization and in the local environ-ment. This view assumes that the school and other contextual conditions do

IMPACT OF A STATE SYSTEMIC INITIATIVE 359

not typically have a strong direct effect on student learning, but largely anindirect effect by enabling improvements in the functioning of classrooms.

Previous studies of SSIs have defined six drivers of systemic reform(SRI International, 1998). The first four drivers are process drivers, whilethe last two are output drivers. For the purposes of this study, the focus wason the four process drivers. This study incorporates the process drivers anda theoretical framework of school change into a model to help explain theimpact of a systemic reform effort on science teachers and their students.The SSI drivers are:

1. Implementation of comprehensive, standards-based curriculum and in-struction, including student assessment.

2. Development of a coherent, consistent set of policies that support theprovision of high quality mathematics and science education for allstudents.

3. Convergence of the usage of all resources that are designed for orthat reasonably could be used to support science and mathematics ed-ucation, including fiscal, intellectual, material, curricular, and extra-curricular.

4. Broad-based support from parents, policy-makers, institutions of highereducation, business and industry, foundations, and other segments ofthe community for the goals and collective value of the program.

5. Accumulation of a broad and deep array of evidence that the programis enhancing student achievement.

6. Improvement in the achievement of all students, including those histor-ically underserved.

In this study we placed the above SSI drivers in a broader contextof what is known about how to effect improvements in student achieve-ment (Louis & Miles, 1900). The broader context included both classroomand school level factors. At the classroom level, broad patterns of effec-tive instruction include elements that Newmann and Associates (1996)summarized in their dimensions of authentic pedagogy. Newmann and As-sociates incorporated findings from both the effective teaching research,and models of constructivist teaching to define authentic pedagogy. Au-thentic pedagogy includes: instruction focused on higher order thinking,which involves manipulating information and ideas by synthesizing, gen-eralizing, explaining, hypothesizing or generating conclusions; instructionfocused on deep knowledge addressing central ideas of a topic or disci-pline; engaging students in substantive conversation which includes ex-tended exchanges with the teachers and/or peers about subject matter; andconnecting the content of the classroom to the world beyond the class-


room so students make connections between knowledge and both publicor personal experience.

Reform also requires professional communities that are focused on stu-dent learning (Kruse, Louis & Bryk, 1995; Bryk, Camburn & Louis, 1996).School capacity for the development of professional communities is de-pendent on a number of factors, including (1) the availability of meaningfulstandards for student development and achievement; (2) the availabilityof research knowledge about practices and structures that will improvestudents’ achievements, and (3) the development of stronger collabora-tive relationships within the school. Reforming schools demands differentforms of leadership (Murphy & Louis, 1994; Leithwood, 1994).

Decentralized, facilitative leadership is particularly important in creat-ing the consensus that characterizes a highly reliable organization that doesnot accept failure (Stringfield, 1995; Louis & Kruse, 1995). Another factoris a supportive structure including cultural features such as an orientationto innovation and problem solving, adequate professional development, theeffective use of time to permit teachers to work in innovative ways bothin and outside the classroom and interdependent teaching roles (Louis,Marks & Kruse, 1996; Stringfield, 1995). In addition, there is a need toconfront the structural barriers to increased classroom performance, in-cluding traditional schedules, and limited teacher and administrator con-tact with each other. Finally reform requires that relationships be forgedwith significant local constituencies, including parents, district and localleaders.

The above school reform factors form the core components of a frame-work with which the SSI drivers were merged for this study. Although theschool reform paradigm was developed largely outside the research thatled to the identification of the SSI drivers, it is not surprising that the twoare so similar. The SSI drivers are defined in terms of policy-manipulabledimensions – e.g., levers that are amenable to intervention by state andnational policy. The school reform factors, on the other hand, have emergedprimarily as a result of studying “reform in situ” – research focusing onschools that are making concerted efforts to change in the context of stateand national policies. Table I shows the overlap of the four “top down” SSIinput drivers and the “bottom up” school reform factors. The “XX” in thetable represents places where the SSI drivers and the school reform factorsoverlap. The areas of overlap formed a framework that was used to guidethe design of this research study. Specifically, the areas of overlap wereused to define the key variables to be measured in this study and to helpin the design of teacher and student surveys. For example, the overlap be-tween the SSI driver on “convergence of resources” and the school reform


TABLE I

Overlap of the SSI drivers and school reform factors

SSI drivers

School reformfactors

Standards-basedcurriculumand instruction

Coherentpolicies

Convergence ofresources

Support fromconstituents

Knowledgebase focusedon coreconcepts

XX

Effective ped-agogy

XX

Knowledgeutilizationprocess

XX

Professionalcommunity

XX

Expandedleadership

XX

Local ac-countabilitystandards

XX

Linkages withcommunity

XX

factors on “professional community” and “expanded leadership” formedkey conceptual components of the teacher survey.

METHODS

Research Design

The authors of this study were independently funded by the National Sci-ence Foundation to conduct a retrospective study of the impact of SSIs onteachers and students. We were not part of any SSI implementation, whichallowed us to provide an external perspective on the program. The SSIexamined in this study worked intensively with some schools more thanothers creating a natural ex-post facto comparison design. In this design,the views of teachers and students from schools with high involvement in


the SSI were compared to those with little or no involvement in the SSI.The treatment group included 8th grade science teachers in schools witha high level of contact with the SSI. High level of contact was definedas a school where at least 50% of the science teachers participated in theprofessional development activities offered by the SSI. The comparisongroup included 8th grade science teachers in matched schools where thescience teachers had no or very low contact with the SSI. No or low contactwas defined as schools where less than 20% of the science teachers wereinvolved in the SSI and where the remaining percentage of teachers wereinvolved in SSI activities for less than one day.

While it would have been ideal to have a control group of teachers withno contact at all, that was not possible given that all schools engaged insome type of professional development. Using a comparison group withlittle or no contact with the SSI was viewed as a conservative choice be-cause it reduced the likelihood of making a type I error. In other words, ifwe had only examined extreme groups of high contact versus no contactteachers it would have increased the likelihood of finding a significantdifference. By using more realistic groups that naturally occurred in theschools we increased the confidence that difference between groups aretrue differences.

Sample

The treatment and comparison schools were matched according to de-mographic characteristics, including type of geographic location (rural,suburban, urban), size of school, and socioeconomic and ethnic charac-teristics of the students. This matching was conducted by local liaisonpersons in the state familiar with the schools and who assisted the out-of-state research team. Chi-square tests indicated that the two groups were notsignificantly different on the matching criteria. Data were collected from23 matched pairs of schools. Within each school, data were collected fromall the 8th grade science students and teachers. This resulted in data from94 science teachers and the 1,393 students in their classes. The treatmentgroup included 49 science teachers and 737 students, while the comparisongroup included 45 science teachers and 656 students. The local liaison per-sons were essential in gaining the cooperation of schools. They contactedthe schools and obtained permission from the principals and teachers toparticipate in the study. Surveys were then mailed to schools for comple-tion by teachers and students. We obtained an eighty-six percent responserate from teachers. Teachers administered the surveys to all of their scienceclasses and we obtained over a ninety percent response rate from studentsin the classes.


The high contact schools in this study were selected by the local li-aisons as schools representative of schools that have significantly engagedwith the state systemic initiative. Originally, the goal was to find 30 highcontact schools and 30 matched low contact schools. Invitations to par-ticipate in this study were sent to 30 high contact schools, but severaldeclined to participate. For each high contact school that participated, lo-cal liaisons helped the research team identify a nearby school with sim-ilar demographic characteristics. In the end we were able to secure theparticipation of 46 schools; 23 with high contact and 23 with no to lowcontact.

The schools were located throughout the state and in rural, suburban,and urban districts. Teachers in this state are some of the lowest paid inthe country. In addition, the state suffers from a large number of teacherswho teach out of their own subject or who do not have backgrounds inscience. Demographic characteristics of the teachers in this study are in-cluded in Table II. Statistical tests indicated that the high contact teachershad more years of teaching experience and more advanced degrees thanthe low contact teachers. Given this difference, both of these variableswere used as covariates in analyses in order to help control for the in-fluence of these variables on results. This state also suffers from one of the

TABLE II

Demographic characteristics of science teachers

High contact No/low

science contact

teachers science

(percent) teachers

(percent)

Highest degree

Bachelors 57.2 81.8

Masters or Ph.D. 42.8 18.2

Years teaching subject

1 year 12.2 20.5

2 years 6.1 18.2

3–5 years 10.1 22.7

6–10 years 30.6 18.2

11–20 years 26.5 11.4

More than 20 years 14.3 9.1


TABLE III

Demographic characteristics of students

Students of Students of

high contact no/low

teachers contact

(percent) teachers

(percent)

Female 55.2 48.5

Male 44.8 51.5

Typical grades in school

As 11.6 11.2

As & Bs 32.7 29.3

Bs 3.8 6.7

Bs & Cs 27.5 26.3

Cs 8.1 7.8

Cs & Ds 12.7 14.2

Ds or lower 3.4 4.5

Speak English at home

Always or almost always 90.9 91.9

Sometimes 5.8 6.4

Almost never 3.3 1.7

Caucasian 64.9 65.9

African-American 22.5 21.0

Hispanic/Latino 2.5 2.2

American Indian 1.5 2.4

Asian 1.5 1.7

highest poverty rates in the nation with over 50% of all students receivingfree or reduced-price lunches. In addition, 8th grade student NAEP scoresin both science and mathematics are among the lowest in the country.Demographic characteristics of the students are included in Table III.

The SSI Program

The SSI program studied in this state began in 1991 and was funded by theNational Science Foundation for eight years along with matching fundsfrom the state department of education. The grant was made to the state de-partment of education, and all SSI activities were administered by the state


department of education. The goal of the SSI was to improve mathematicsand science education through professional development opportunities, en-hance the way the subject matter was taught in the classroom, and increasethe use of standards-based curriculum, instruction and assessment. The SSIprogram itself was designed to impact many aspects of the educationalsystem; however, one of the unique features of this SSI was its focus onprofessional development. The goal was to help impact teachers throughprofessional development as a means of systemic reform. A wide varietyof professional development workshops were offered throughout the state.From 1999–2001 the SSI offered 63 professional development workshops.Throughout the state, more than 7,000 teachers from 66 school districtswere impacted. The SSI workshops were sponsored by the state depart-ment of education, but they were offered by university faculty. Teacherscould pick and choose among a wide variety of different offerings withvaried length and duration. The workshops ranged from short three tofive day workshops, to longer summer workshops with extended follow-up throughout the school year. For example, in one workshop for biology,chemistry, and physical science teachers the focus was on collecting andanalyzing water collected from a local watershed. The teachers attended aninitial ten day summer workshop, followed by five additional days duringthe academic school year. Given the voluntary nature of teachers’ partici-pation, the engagement of teachers varied quite widely, which meant thatsome teachers engaged in more professional development than others.

To measure the extent to which teachers had either high or low con-tact with the SSI, survey questions were completed by teachers regardingthe type and duration of their professional development. These items pro-vided evidence about the extent to which high contact teachers actuallyhad high contact with the SSI, and the extent to which teachers had lowor no contact. The survey included five questions about different types ofprofessional development and had a Cronbach’s alpha of .84. The differenttypes of professional development are included in Table IV and are basedupon the recommendations made by Loucks-Horsley, Hewson, Love andStiles (1998) for science and mathematics professional development. Foreach type of professional development teachers were asked to indicate howlong they engaged in the activities. The choices were not at all, less thantwo weeks, two to four weeks, four to six weeks, or continuing contact forsix months or more. An overall mean score on the five items indicated thatthe teachers with high SSI contact engaged in significantly more profes-sional development than the no/low contact teachers F(1, 94) = 7.227,p = .009. When one examines the individual questions it is apparent thata larger percentage of high contact teachers engaged in more long-term,


TABLE IV

Percent of high and no/low contact teachers by types of professional development

Type of professional Duration High contact No/low contact

development teachers teachers

(percent) (percent)

1. Immersion: immersion None 55% 71%

into solving scientific < 2 weeks 26% 20%

problems or experiences 2–4 weeks 2% 0%

in the day-to-day work of 4–6 weeks 6% 7%

scientists. Continuing 11% 2%

2. Curriculum or instruction None 15% 20%

implementation: learning, < 2 weeks 27% 49%

using and refining use of 2–4 weeks 21% 12%

instructional materials in 4–6 weeks 10% 7%

the classroom. Continuing 27% 12%

3. Curriculum or instruction None 29% 26%

development: creating < 2 weeks 17% 40%

new instructional 2–4 weeks 17% 9%

materials and strategies to 4–6 weeks 10% 16%

better meet the learning Continuing 27% 9%

needs of students.

4. Examining practice: case None 35% 44%

discussions of classroom < 2 weeks 33% 42%

scenarios or examining 2–4 weeks 13% 14%

student work and scoring 4–6 weeks 10% 0%

assessments. Continuing 8% 0%

5. Collaborative work: None 33% 42%

study groups, coaching, < 2 weeks 21% 37%

mentoring or classroom 2–4 weeks 15% 7%

observation and feedback. 4–6 weeks 6% 12%

Continuing 25% 2%

continuing professional development. This fits with recent recommenda-tions on professional development and the need for teacher to engage inlong-term rather than short-term activities (Loucks-Horsley et al., 1998).

Instrumentation

The instruments used in this study were designed to measure the differentcomponents of the framework described previously (see Table V). Sur-


TABLE V

Teacher survey scales and reliabilities

Curriculum & instruction scale (alpha = .87)

How often do students: (rarely or never, once a month, once a week, 2–3 times a week,daily)

1. participate in student-led discussions,

2. participate in discussions to deepen science understanding,

3. make formal presentations to the class,

4. read from a science textbook in class,

5. read other science related material,

6. work on solving real-world problems,

7. share ideas or solve problems with each other in small groups,

8. engage in hands-on science activities,

9. follow prescribed steps in an activity or investigation,

10. record, represent and/or analyze data,

11. complete worksheets that emphasize mastery of essential skills,

12. prepare written science reports of at least three pages,

13. write reflections in a notebook or journal,

14. describe what they know about a topic before it is taught,

15. use community resources in the classroom,

16. use calculators or computers to solve science problems,

17. document and evaluate their own science work,

18. design or implement their own investigation,

19. design objects within constraints.

Professional community scale (alpha = .90)

In this school: (strongly disagree, disagree, no opinion, agree, strongly agree)

1. Goals and priorities are clear.

2. There is a great deal of cooperate effort.

3. Teachers are continually learning and seeking new ideas.

4. Most of my colleagues share my beliefs and values about the central mission ofthe school.

5. I am encouraged to experiment with my teaching.

6. Staff are involved in making decisions which directly affect me.

7. Teachers in this school have a shared vision of effective science instruction.

8. I feel accepted and respected as a colleague.

veys were developed for both 8th grade science students and 8th gradeteachers. The instruments included items from existing national sourcessuch as TIMSS, Newmann’s surveys of reformed schools (Newmann &


TABLE V

(Continued).

Influence on school policy (alpha = .84)

How much influence do teachers in your school have over school policy in the areas below?(no influence, some influence, a great deal of influence)

1. determining the content of in-service programs

2. establishing school curriculum

3. determining the school’s schedule

4. hiring new professional personnel

5. planning school budgets

6. determining specific professional and teaching assignments

7. selecting textbook and other instructional materials

External influence on science (alpha = .83)

Please rate the impact of the following on your science instruction (substantial negativeimpact, slight negative, no clear impact, slight positive impact, substantial positive impact)

1. state and/or district policies, requirements or standards

2. local school board

3. district/school structures for recognizing and rewarding teachers

4. an externally sponsored school reform initiative in which your school participates

5. professional organizations

6. faculty at colleges or universities

7. available instructional materials

8. local organizations, institutions, and/or businesses

Family involvement (alpha = .75)

How many of your students’ parents do the following?

(none, few, some, about half, almost all)

1. attend parent-teacher conferences

2. express their support for the use of investigative approaches to science

3. participate in reflective discussions related to curriculum and instruction

Associates, 1996), and Horizon Research, Inc.’s local systemic changeproject surveys (Horizon Research Incorporated, 1997). To ensure qualityinstruments the initial drafts of the surveys were written, pilot tested andrevised by a team of university level science and mathematics educators.In addition, a measurement specialist from the university was consultedon psychometric properties of the items. Two different pilot tests wereconducted with classroom teachers and their students. First, a talk-aloudpilot study was done with a small group of teachers and students who were


not part of an SSI to determine if the survey questions were understood byteachers and students as intended. A second pilot test was done with stu-dents and teachers located in an SSI state. Over thirty teachers completedthe survey, and over a hundred of their students. The data from the pilottests were used to check individual items for appropriate mean scores andranges as well as to conduct reliability analyses.

The teacher survey included six scales: curriculum and instruction, pro-fessional community, teachers’ influence on policy, external influence onscience, and family involvement. Using Cronbach’s alpha as a measureof reliability, all the scales had acceptable reliabilities ranging from .75to .90 (see Table V for teacher survey scales and reliabilities). The cur-riculum and instruction scale focused on questions about the extent towhich teachers used the type of instructional techniques advocated in theNational Science Education Standards. The second scale focused on pro-fessional community and included questions about teachers’ shared normsand values, collaboration among teachers, shared decision-making, open-ness to innovation, and shared vision among the teaching staff. The policyscale included questions about teachers’ influence on policies such as theschool schedule, hiring practices, the curriculum, instructional strategies,and school budget issues. The fourth scale focused on external factors andthe extent to which they influence science instruction. Included were ques-tions about the impact of professional organizations, faculty at collegesand universities, externally sponsored initiatives, the school board, andstate and district policies. Finally, the family involvement scale includedquestions about support from parents, influence of parents, and their in-volvement in school. The student survey was designed to match the teachersurvey as much as possible (see Table VI). It included three reliable scales:standards-based curriculum and instruction, school community, and familyinvolvement.

RESULTS

Mean scores on each of the scales were compared for the high andno/low contact teachers and their students using an analysis of covari-ance with teachers’ years of experience and educational background en-tered as covariates. The high SSI contact teachers indicated they providedmore standards-based curriculum and instruction in their classes, and sawa greater external influence on science instruction than the teachers inschools with little or no involvement in the SSI. No differences were foundbetween the two groups of teachers in influence over shool policyprofessional community or family involvement (see Table VII). Effect


TABLE VI

Student survey scales and reliabilities

Curriculum and instruction (alpha = .79)

How often do you: (rarely or never, once a month, once a week, 2–3 times a week, daily)

1. participate in student-led discussions

2. participate in discussions to deepen science understanding

3. make formal presentations to the class

4. read from a science textbook in class

5. read other science related material

6. work on solving real-world problems

7. share ideas or solve problems with each other in small groups

8. engage in hands-on science activities

9. follow prescribed steps in an activity or investigation

10. record, represent and/or analyze data

11. complete worksheets that have you practice essential skills

12. prepare written science reports of at least three pages

13. write reflections in a notebook or journal

14. describe what you know about a topic before it is taught

15. use community resources in the classroom

16. use calculators or computers to solve science problems

17. document and evaluate your own science work

18. think about what a problem means and different ways it might be solved

19. design or implement your own investigation

Family involvement (alpha = .79)

How often do your parents do the following: (none, few, some, about half, almost all)

1. attend parent-teacher conferences

2. express their support for the use of investigative approaches to science

3. express their support for traditional textbook-based approaches to science instruction

School community (alpha = .75)

How much do you agree with the following statements: (strongly disagree, disagree, agree,strongly agree)

1. students get along well with teachers

2. the teaching is good

3. teachers are interested in students

4. most of my teachers really listen to what I have to say

5. I feel comfortable voicing my opinion in my science class


TABLE VII

Teacher survey comparisons for high SSI contact and no/low SSI contact teachers

Scale High No/low F p-value Effect size

contact contact (γ )

(n = 49) (n = 45)

Mean SD Mean SD

Curriculum and 2.78 .66 2.44 .64 4.51 .036∗ .52

instruction

External influence on 2.48 .78 1.96 .73 9.60 .003∗ .68

science

Teacher influence on 3.91 .66 3.55 .61 3.45 .067 .57

school policy

Professional 4.10 .69 3.86 .71 1.31 .256 .34

community

Family involvement 2.59 .97 2.34 .75 .43 .512 .29

sizes were also calculated to help interpret the relative magnitude of thedifferences. Cohen (1997) defined three levels of effect size that are basedupon standard deviation units. According to Cohen, large is defined as.80 or more, medium is defined as .50 to .80, and small is defined asbetween .20 and .50. Using these definitions, the significant differencesare considered medium in size.

On the curriculum and instruction scale the mean score for high contactteachers indicated that teachers used the methods slightly less than once aweek. No/low contact teachers indicated they used these techniques some-where between once a month and once a week. High contact teachers’influence on policy was in the “some” range, while no/low contact teach-ers were in between “some” and “no influence.” The impact of externalinfluence on science instruction was “slightly positive” for high contactteachers, and between “no impact” and “slightly positive” for no/low con-tact teachers. For professional community, both groups of teachers hadrelatively high ratings that fell in the “agree” range, suggesting that bothgroups had positive professional communities. Finally, on the family in-volvement scale both high and no/low contact teachers indicated that only a“few” to “some” of their parents provided support for science by attendingconferences or supporting standards-based curriculum use.

For students the only significant difference was on the curriculum andinstruction scale. The students of high SSI contact teachers indicated they


TABLE VIII

Student survey comparisons for students of high and no/low SSI contact teachers

Scale High No/low F p-value Effect size

contact contact (γ )

(n = 737) (n = 656)

Mean SD Mean SD

Curriculum and 2.54 .59 2.42 .59 5.90 .015∗ .20

instruction

School community 2.68 .45 2.65 .51 .457 .491 .06

Family involvement 1.97 .49 1.95 .47 .687 .407 .04

experienced more standards based curriculum and instruction in theirclasses. No differences, however, were found for extent of family involve-ment or views of the school community. The effect size calculation sug-gests the difference on curriculum and instruction scale was small in mag-nitude (see Table VIII).

In terms of the relative differences on these scales, the curriculum andinstruction rating by the students of high contact teachers indicated thatthey experienced standards-based methods between once a month and oncea week, while students of no/low contact teachers indicated it was closer toonce a month. In terms of school community, students of both no/low andhigh contact teachers were leaning towards ‘agree’ on statements aboutthe extent to which a supportive community existed in their schools. Onthe family involvement scale both groups indicated that on average only afew of their parents indicated support for science.

DISCUSSION

The United States has made a large investment in systemic reform; how-ever, it is still not clear whether or not such reform is entirely successful.Systemic reform is clearly quite challenging, and to help meet this chal-lenge, the field is in need of more information about the extent to whichsystemic reform can actually be achieved. It is especially important toprovide information about the extent to which specific goals of SSIs canbe achieved, especially given the lack of empirical evidence regardingsystemic reform (Fullan & Miles, 1992).

One of the key findings from the present study is that both high SSIcontact teachers and their students reported using more standards-based in-


structional techniques as defined by the National Science Education Stan-dards (NRC, 1996). This included instructional techniques such as par-ticipating in discussions to deepen understandings, working on solvingreal-world problems, engaging in more hands-on activities, and record-ing, representing and analyzing data. This is an important finding for sys-temic reform and suggests that systemic reform efforts are associated withstandards-based instructional practice. It may be that teachers who en-gaged in SSI sponsored professional development activities tended to havemore standards-based classroom environments, or it may be that the SSIactually had an impact on teachers’ instructional practice. This is a positivefinding for professional development given that short-term professionaldevelopment does not always have such an impact on classroom practice(Loucks-Horsley et al., 1998). This finding is also supported by previousresearch. In a comprehensive study using U.S. national longitudinal datafrom over one thousand teachers, Desimone, Porter, Garet Yoon and Bir-man (2002) found that high quality professional development, includingcollective participation, long term contact, active learning, coherence andcontent focus can have a positive influence on teachers’ classroom practice.

It is also interesting that teachers in high SSI contact schools reportedthat external groups had more influence over science instruction in theirschools. The teachers in high contact schools reported that state and districtpolicies, local school districts, externally sponsored projects, professionalorganizations, universities and faculty, and local businesses had a slightlypositive impact on science instruction in their schools, whereas teachersin low contact schools did not consider these influences as positive. Thisis a significant finding given the importance of building relationships withexternal organizations and the critical role they can play in sustaining re-form. These results are encouraging for reform efforts and partnershipsthat attempt to create a larger role for the standards and attempt to havelocal organizations, and universities collaborate with science teachers.

No significant difference between groups was found for teachers’ in-fluence on school policy. Both high and low contact teachers reportedrelatively low levels of influence on school policies including influenceover in-service programs, curriculum, hiring personnel, and budgets. Theseresults suggest that the systemic initiative is not associated with teacherinvolvement in school policy issues. One of the goals of systemic reformis to engage teachers in school policies and practices that go beyond theclassroom. The goal is to actually help teachers engage in the larger schoolcommunity – something that is critical to creating and sustaining reform.However, in this study it does not appear that either group of teachers hadmuch influence on school policies. This is unfortunate given the positive


impact of teachers’ influence on school policy reported in the literature.Decentralized, facilitative leadership, which includes teachers having moreauthority in schools, has been shown to be particularly important in creat-ing the consensus that characterizes a highly reliable organization (String-field, 1995; Kruse et al., 1995). Given the low levels of influence that allteachers reported in this study, it is important for future systemic reforminitiatives to carefully consider who they might engage teachers in policy.

Another factor that was not significantly different was professionalcommunity. Students and teachers in both groups reported relatively highlevels of professional community in their schools such as cooperative ef-fort, teachers learning and seeking new ideas, shared beliefs and values,involvement in decision making, an orientation to innovation and problemsolving, and teachers working in innovative ways both in and outside theclassroom (Louis et al., 1996; Stringfield, 1995). Professional communityhas been shown to be an important factor in systemic reform, and profes-sional development has been shown to enhance professional community(Youngs & King, 2000). McLaughlin, Talbert, Kahne and Powell (1990)found that professional communities provided the support necessary forteachers to make changes in their practice, sustain professional learningand enact high quality instruction. Louis et al. (1996) reported that ad-ministrative support, respect, and openness to innovation were positivelyrelated to professional community. Furthermore professional communitywhich includes strong collaborative relationships has been shown to bedirectly associated with higher achievement (Kruse et al., 1995; Louis &Marks, 1998). It is encouraging to report that both groups of schools hadhigh ratings on professional community, given its importance in reform.However, these results suggest that systemic reform efforts may not need tofocus as much effort on professional community as much as other factorsgiven that schools with little or no SSI contact were also able to createpositive professional communities.

Finally, no differences were found for family involvement. Both groupsof teachers and their students reported that family involvement was rel-atively low. This is discouraging given that family involvement has beenshown to be an important variable. Research has shown that positive fam-ily involvement is associated with student achievement (Bogenschneider,1997; Epstein & Dauber, 1991). One of the factors necessary for effectiveschool restructuring is a clear focus on external support (Newmann & As-sociates, 1996). Systemic reform relies upon support outside the classroomand school and family involvement is an important part of creating support.It appears that other methods are needed to positively impact on family


involvement and SSI’s attempting to increase family involvement shouldfocus more effort on this goal.

The results of this study provide important information for the designof future reform initiatives. The SSI program in the United States wasdesigned to focus on wide variety of issues such as policy, curriculum,resources, and external support, improved classroom instructional prac-tice, and student achievement. It is difficult to affect all of these issuessimultaneously through one initiative. From a design perspective, the SSIprogram was organized with the view that the state, district, and schoolare the important units of change. The school improvement literature hastaken the perspective that the school itself is the most important unit ofchange (Van Velzen et al., 1984); however, to create reform we wouldargue that the focus of what is to be improved must be within the class-room, where students and teachers can create the most direct change. Thedesign of SSIs tends to focus on the system, with less emphasis on teachersand how they interact with students. In all fairness, the SSI studied inthis state focused its resources on professional development for teachers,and the results of this study suggest that the teachers who participated inSSI activities used more standards-based instructional practices in theirclasses; however, external factors such as policy, professional community,and family involvement were not found to be significantly different. Theseresults bring up questions of design and the best way to make an impactof a wide variety of factors. It may be impossible for reform to truly besystemic and impact such a wide range of variables without careful designand consideration of the best ways to achieve each goal. Future large scalereform efforts should carefully consider issues of allocation of resourcesas they attempt to design programs to have the greatest impact.

CONCLUDING REMARKS

Even though many countries around the world have national curricula andcan mandate national changes, reform is still a great challenge. As Fullan& Miles (1992) point out, the cardinal rule of reform is that it is alwayslocally implemented by everyday teachers, principals, parents, and stu-dents (p. 752). This is not to diminish the important role that countries,states, district and schools can play in reform; but these large institutionsare not necessarily sufficient to sustain systemic reform in the classroom.It is clear we need to better understand how to create and sustain reform,especially given that in the U.S. not one of the twenty-five SSIs that werefunded by the National Science Foundation was able to go to scale andachieve comprehensive reform (Anderson, 2002). Perhaps comprehensive


systemic reform is an unrealistic goal, but it does suggest we need to betterunderstand systemic reform and use research to help design future reformsefforts. The results of the present study suggest that a SSI is positivelyassociated with instruction and external relations, but not related to policy,professional community, and family involvement. Hopefully this informa-tion will be used to help improve systemic reform and create more effectivemodels of systemic change in the U.S. and other countries. It is critical thatwe understand better the impact of systemic reform in order to help to trulyreform science education for all students.

REFERENCES

Anderson, B. (2002). Evaluating systemic reform: Evaluation needs, practices, and chal-lenges. In J. Altschuld & D. Kumar (Eds.), Evaluation of science and technologyeducation at the dawn of the new millennium. New York: Kluwer Academic.

Bogenschneider, K. (1997). Parental involvement in adolescent schooling: A proximalprocess with transcontextual validity. Journal of Marriage and Family, 59(3), 718–733.

Bryk, A., Camburn, E. & Louis, K.S. (1996). Promoting school improvement through pro-fessional communities: An analysis of Chicago elementary schools. Paper presented atthe annual meeting of the American Educational Research Association, New York.

Cohen, J. (1977). Statistical power analysis for behavioral sciences (Rev. ed.). New York:Academic Press.

Desimone, L.M., Porter, A.C., Garet, M.S., Yoon, K.S. & Birman, B.F. (2002). Effects ofprofessional development on teachers’ instruction: Results of a three-year longitudinalstudy. Educational Evaluation and Policy Analysis, 24(2), 81–112.

Elmore, R.F. (1995). Structural reform and educational practice. Educational Researcher,24(9), 23–26.

Epstein, J.L. & Dauber, S.L. (1991). School programs and teacher practices of parent in-volvement in inner-city elementary and middle schools. The Elementary School Journal,91(3), 289–305.

Fullan, M. & Miles, M. (1992). Getting reform right: What works and what doesn’t. PhiDelta Kappan, 73(10), 745–752.

Fullan, M. & Stiegelbauer, S. (1991). The new meaning of educational change. New York:Teachers College Press.

Horizon Research Incorporated (1997). Local systemic change core evaluation datacollection manual. Chapel Hill, NC: Horizon Research Incorporated.

Kruse, S., Louis, K.S. & Bryk, A. (1995). A framework for analyzing professional com-munity. In K. Louis, S. Kruse & Associates (Eds.), Professionalism and community:Strategies for reforming urban schools. Thousand Oaks, CA: Corwin Press.

Leithwood, K. (1994). Leadership for school reform. Educational Administration Quar-terly, 4, 498–518.

Loucks-Horsley, S., Hewson, P.W., Love, N. & Stiles, K.E. (1998). Designing professionaldevelopment for teachers of science and mathematics. Thousand Oaks, CA: CorwinPress.

Louis, K.S. & Kruse, S.D. (1995). Professionalism and community: Perspectives onreforming urban schools. Thousand Oaks, CA: Corwin.


Louis, K.S. & Marks, H.M. (1998). Does professional community affect the classroom?Teachers work and student experiences in restructuring schools. American Journal ofEducation, 106(4), 532–575.

Louis, K.S., Marks, H.M. & Kruse, S.D. (1996). Teachers’ professional community inrestructuring schools. American Educational Research Journal, 33(4), 757–798.

Louis, K. S. & Miles, M. (1990). Improving the urban high school: What works and why.New York: Teacher’s College Press.

McLaughlin, M.W., Talbert, J., Kahne, J. & Powell, J. (1990). Constructing a personalizedschool environment. Phi Delta Kappan, 72, 230–235.

Murphy, J. & Louis, K.S. (Eds.) (1994). Reshaping the principalship: Insights fromtransformational reform efforts. Thousand Oaks, CA: Corwin.

National Research Council (1996). National Science Education Standards. Washington,DC: National Academy Press.

Newmann, F. & Associates (1996). Authentic achievement: Restructuring schools forintellectual quality. San Francisco: Jossey Bass.

Scheerens, J. (1992). Effective schooling. London: Cassell.SRI International (1998). A report on the evaluation of the National Science Foundation’s

statewide systemic initiatives program. Menlo Park, CA: Author.Stringfield, S. (1995). Attempting to enhance students’ learning through innovative

programs: The case for schools evolving into high reliability organizations. SchoolEffectiveness and School Improvement, 6, 67–96.

Tyack, D. & Cuban, L. (1995). Tinkering toward utopia: A century of public school reform.Cambridge, MA: Harvard University Press.

Van Velzen, W., Miles, M., Hamayer, U., Ekholm, M. & Robin, D. (1984). Making schoolimprovement work. Leuven, Belgium: Acco Press.

Youngs, P. & King, M.B. (2000, April). Professional development that addresses profes-sional community in urban elementary schools. Paper presented at the annual meetingof the American Educational Research Association, New Orleans, LA (ERIC DocumentReproduction Service No. ED 449 266).

Douglas HuffmanScience Education,Department of Teaching and Leadership,University of Kansas,347 Joseph R. Pearson Hall,122 West Campus Road,Lawrence, KS 66045,U.S.A.E-mail:[email protected]

Francis LawrenzUniversity of Minnesota

the impact of a state systemic initiative on u.s. science teachers and students

Documents