school level conditions affecting the effectiveness of instruction

This article was downloaded by: [University of Kiel]On: 25 October 2014, At: 04:20Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T3JH, UK

School Effectiveness andSchool Improvement: AnInternational Journal ofResearch, Policy and PracticePublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/nses20

School Level ConditionsAffecting the Effectiveness ofInstructionBert P.M. Creemers a & Gerry J. Reezigt aa GION, University of GroningenPublished online: 03 Aug 2006.

To cite this article: Bert P.M. Creemers & Gerry J. Reezigt (1996) School LevelConditions Affecting the Effectiveness of Instruction, School Effectiveness andSchool Improvement: An International Journal of Research, Policy and Practice, 7:3,197-228, DOI: 10.1080/0924345960070301

To link to this article: http://dx.doi.org/10.1080/0924345960070301

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all theinformation (the “Content”) contained in the publications on our platform.However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness,or suitability for any purpose of the Content. Any opinions and viewsexpressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of theContent should not be relied upon and should be independently verifiedwith primary sources of information. Taylor and Francis shall not be liablefor any losses, actions, claims, proceedings, demands, costs, expenses,damages, and other liabilities whatsoever or howsoever caused arising

http://www.tandfonline.com/loi/nses20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/0924345960070301

http://dx.doi.org/10.1080/0924345960070301

directly or indirectly in connection with, in relation to or arising out of theuse of the Content.

This article may be used for research, teaching, and private studypurposes. Any substantial or systematic reproduction, redistribution,reselling, loan, sub-licensing, systematic supply, or distribution in any formto anyone is expressly forbidden. Terms & Conditions of access and use canbe found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

http://www.tandfonline.com/page/terms-and-conditions

School Effectiveness and School Improvement 0924-3453/96/0703-0197$12.001996, Vol. 7, No. 3, pp. 197-228 © Swets & Zeitlinger

School Level Conditions Affecting the Effectiveness ofInstruction

Bert P.M. Creemers and Gerry J. ReezigtGION, University of Groningen

ABSTRACT

In this article, the current status of school level factors, as they appear in research reviewsand in school effectiveness models is criticised both from a theoretical and from anempirical perspective. School level factors are often related to student achievement with-out taking into account the classroom level, where teaching and learning primarily takeplace. As a theoretical alternative, an overview of school level factors that enhance thequality of instruction, time for learning and opportunity to learn at the classroom level isprovided. These factors are supposed to contribute to the explanation of differences instudent achievement. Some empirical support for these ideas is found in educationalresearch studies that have made use of three-level analyses.

INTRODUCTION

School effectiveness research started as a reaction to the disappointingresults of educational research with respect to the influence of schoolsand teachers. Carefully designed research with respect to teacher behav-iour in classrooms and the structure and organisation of schools could notavoid the disappointing conclusion that student characteristics (such asabilities and social background) accounted for major proportions of vari-ance in student outcomes, even though research into the relationship be-tween teacher behaviour and student outcomes proved that teacher behav-iour could account for a small proportion of the variance in student out-comes.

Correspondence: GION, Groningen Institute for Educational Research, University of Gro-ningen, P.O. Box 1286, 9701 BG Groningen, The Netherlands. Tel.: (31) 50 636635636660. Fax: (31) 50 636670. E-mail: [email protected].

Manuscript submitted: December 15, 1994Accepted for publication: May 14, 1996

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

198 B.P.M. CREEMERS AND G.J. REEZIGT

School effectiveness research was a reaction to the common feelingthat schools and teachers did not matter. Initially, no distinction wasmade between the different levels within schools, such as departmentsand classrooms, in order to make the point as strong as possible: schoolsmatter. The factors connected with educational effectiveness were a com-bination of all kinds of factors, as can be seen in an analysis of the so-called five-factor model and other models (Creemers & Scheerens, 1991;Scheerens, 1992). Later, the loose theoretical framework of school effec-tiveness was criticised and as a result more emphasis was given to thedevelopment of a conceptual framework that contains more levels in ad-dition to the student level. At least two levels: the classroom level withinthe school and the school level were discerned. Other levels, for examplethe departmental level in the school organisation can also be discerned aswell as levels immediately above the school, such as the school board.

In the effectiveness models developed later on (Scheerens, 1989; Creem-ers, 1991; Slater & Teddlie, 1992; Stringfield & Slavin, 1992) a relation-ship between the classroom and the school level can be found. The educa-tional processes - instruction and learning - take place at the classroomlevel and the other levels are supposed to provide the conditions forinstruction at the classroom level. This conceptual framework, however,did not stop the search for all kinds of correlates of effectiveness, and thelist of possible correlates for effectiveness continued to grow. As a resultof the methodology used, these correlates can be found at all levels, theclassroom, the school, the school board, the district, and the nationallevel. One of the criticisms against the correlates is that it is not clear howthe relationships between these correlates might be, and what goes on atthe classroom level.

Research including both the school level and the classroom level showedthat the classroom level was important, quite often even more so than theschool level (see, for example, Scheerens et al., 1989). Also, theory-oriented publications stressed the predominance of the classroom level.This implies that in school effectiveness research attention for the schoollevel is diminishing whereas the classroom level gets more attention thanbefore. In fact, we could be back at the stage of the pre-school-effective-ness-movement: teachers and classrooms make a difference, albeit quitesmall.

However, even when learning and teaching take place primarily at theclassroom level, educational effectiveness means more than outcomes injust one class. Educational effectiveness has to do with outcomes andeffectiveness in successive classes in a school and in the educationalsystem as a whole. The classroom level alone cannot guarantee adequate

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

SCHOOL LEVEL CONDITIONS 199

transitions between different classes. Effectiveness of education there-fore is more than effectiveness at the classroom level. This raises thefollowing questions:1. Which unique contribution, in terms of proportions of variance ac-

counted for, can be expected from the school level when it is analysedsimultaneously with the classroom level?

2. Taking the classroom level into account, which school factors can besupposed to contribute to educational effectiveness, and what is therelationship between factors at the school level and factors at the class-room level?

To answer these questions, we will deal with the factors at the schoollevel discovered in research, and the supposed effects of these factors onoutcomes (in the next section). In doing so, for the sake of clarity we willfocus on general effects of factors on cognitive achievement criteria,which means that issues of differential effects of factors and the potentialconsistency of factors across different types of outcomes will not betouched upon. The research results can be criticised on theoretical andempirical grounds (see the section on theoretical and empirical consider-ations). A theoretical basis for school level factors, connecting the schoollevel and the classroom level, will be outlined (see the section entitled "Atheoretical basis for school factors"). Finally, some examples of researchinto school and classroom effectiveness are discussed (see the section onthree level research on school level factors) and guidelines for futureresearch are given (see the summary and conclusion).

THE CURRENT STATUS OF SCHOOL FACTORS IN SCHOOLEFFECTIVENESS THEORY AND RESEARCH

School effectiveness research has yielded impressive numbers of schoolfactors related to student achievement. Two types of factors can be dis-cerned: school factors that are summed up in reviews of research (sup-posed to be empirically and directly related to achievement), and schoolfactors that are included in school effectiveness models (presumed toinfluence classroom processes and therefore indirectly related to studentachievement).

Reviews of ResearchThe school factors in research reviews are mostly derived from case studyresearch and correlational research with an aggregated index of studentachievement as the criterion for effectiveness. The school factors repre-

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


sent a wide variety of characteristics, ranging from very global (e.g.,school demographics; Wang, Haertel, & Walberg, 1993) to very specific(e.g., communication with the school adviser; Spade, Vanfossen, & Jones,1985). The main categories of school factors, mentioned by several re-viewers such as Purkey and Smith (1983), Mortimore, Sammons, Stoll,Lewis, and Ecob (1988), Levine and Lezotte (1990), Scheerens (1992),and Wang et al. (1993) are the following:1. orderly environment/school climate,2. consensus and cooperation between teachers,3. focus on basic skills/learning time,4. monitoring of student progress/evaluation,5. school educational/administrative leadership,6. policy on parental involvement,7. high expectations.Many authors, especially Levine and Lezotte (1990), mention subcatego-ries (such as support of teachers, acquisition of resources, utilisation ofinstructional support personnel as refinements of the main category ofschool leadership), or additional school factors (such as teacher experi-ence, changes in staff, or urbanisation level of the school environment.(Scheerens, 1992). These subcategories and additional school factors arenot so consistently reported as the main categories mentioned above. Themain categories, with the exception of consensus and cooperation be-tween teachers and policy on parental involvement, reflect the five fac-tors suggested by Edmonds (1979). This finding might indicate the ro-bustness of these factors. However, the similarity of categories may alsobe attributed to reviewers who continuate to study a set of literaturewhich .has been reviewed before, thereby creating a self-repetition offindings. It may also point at a research artefact: because these factorswere supposed to influence student achievement, they were studied overand over again, and indeed were found to influence student achievementfrom time to time. The factors therefore are certainly empirically support-ed, but other factors, that have not been studied so often, might also beimportant.

School Effectiveness ModelsNot all of the models that are cited in the literature on educational effec-tiveness specify school level factors. Relatively early models (e.g., Slavin,1987; Walberg, 1984) are restricted to the student level and the teacher/classroom level. More recently, several models for educational effective-ness that do specify school level factors were proposed as a contributionto theory development (Scheerens, 1992; Slater & Teddlie, 1992; String-

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Stringfield & Slavin, 1992 Scheerens, 1992 Slater & Teddlie, 1992

- meaningful goals

- attention to academicfunctioningin all classes (leadership)

- coordination of curriculaand instruction acrossclasses, programs, andgrades

- recruitment and develop-ment of staff

- efficient organising ofschool functioning

School Level Factors

- achievement-orientedpolicy

- educational leadership

- consensus/cooperativeplanning

- quality of school curricula- orderly atmosphere- evaluative potential

- administrativeappropriateness

management (influencingschool structure)

leadership (influencingschool culture)

Levels in the Model

studentteacher/classroomschoolcontext

- studentteacher/classroom- school- context

- studentteacher/classroom- school

Basic Assumptions about the Role of School Level Factors

indirect effects on studentachievement bymanagerial andorganisational processesthat influence qualityand appropriatenessof instruction, incentives,and time (qait) at theclassroom level

- indirect effects onstudent achievement;school factors asconditions (reinforcers/buffers) for classroomlevel processes

- indirect effects on studentachievement; schoolfactors interact withclassroom level factors;effects of interactionsexplain changingeffectiveness

Criteria for the Inclusion of School Level Factors

- theory on school learning - eclectic: organisational, - organisational theorieseconomical, politicaltheories

- empirical evidence - empirical evidence - empirical evidence(Louisiana studies) (reviews) (Louisiana Studies)

Fig. 1. School level factors in recent educational effectiveness models.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


field & Slavin, 1992). Although the school level factors described inthese models are partly selected on the basis of their empirical support(often postulated as direct relationships with student achievement), theyare mainly included in the models because of their presumed influence onclassroom processes (and therefore their indirect relationships to studentachievement). Figure 1 shows the school level factors in these three mod-els and the assumptions that led to their inclusion.

Although the models differ with regard to the exact number of levelsincluded, the assumptions about interactions between levels, and the the-ories used for the selection of school factors, the selected school levelfactors in the models are not essentially different in nature. They are alsonot so very different from the school factors in research reviews. Forexample, the factors 'attention to daily academic functioning in all class-es' (Stringfield & Teddlie, 1991) and 'recruitment and development ofstaff are elements of the main category of leadership in the review byLevine and Lezotte (1990). The only factor that is often mentioned inreviews but not included in the models is a school policy on parentalinvolvement. Factors in the models that are not clearly discerned in re-search reviews are: 8. coordination of curricula and instruction acrossclasses, programs and grades (Stringfield & Slavin, 1992); and 9. qualityof school curricula (Scheerens, 1992). The school factors in researchreviews and in effectiveness models are largely the same. This suggeststhat there is some general agreement about the effectiveness of a set of, intotal, 9 school factors.

The Size of Effects of School FactorsThe general agreement about the set of school factors does not necessar-ily reflect the actual importance of school factors for student achieve-ment as indicated by the size of their effects. Information about the sizeof effects is necessary to judge the relative importance of isolated fac-tors. Despite the rich literature on school effects, this kind of factualinformation is not easy to obtain. Research reviews not always mentionthe size of correlations; they often include school factors merely becausethey were studied in a lot of projects and then were repeatedly found tohave some kind of positive influence on an achievement indicator, butnot on behalf of their effect sizes. For example, Levine and Lezotte donot give any information about the size (or the average size) of correla-tions between school factors and student achievement. Their factors areoften derived from small-scale case study research, which does not allowfor the computation of correlations or effect sizes (Levine & Lezotte,1990).

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Sometimes correlations, regression results or results of two-level re-search of individual studies are reported. For example, Fraser, Walberg,Welch, & Hattie (1987) mention an average correlation of school charac-teristics (such as aims and policy) and student achievement of. 12; Schee-rens (1992) mentions correlations ranging from .02 (heterogeneity ofclassrooms) to .79 (high expectations). According to Wang et al. (1993),the most important school factor is school culture (average T score of 53),followed by teacher/administrator decision making (7=48), a policy onparental involvement (T=46), and school demographics (7=41). The schoolfactor with the smallest impact on student achievement is school policyand organisation (T=37). Teddlie and Stringfield (1993) find correlationsof school factors ranging from .28 (parental support) to .49 (perception ofschool success by the principal). They compare their findings with corre-lations found by Brookover, Beady, Flood, Schweitzer, & Wisenbaker(1979), and they report some stability with regard to the size of correla-tions. These kinds of comparisons, the same group of factors in one studycompared to the same group of factors in another study, are rather rareand because of the different combinations of factors in different studies,it is difficult to find out which factors are stable. Also, correlations ofisolated factors with student achievement may hide problems of multicol-linearity. Such problems are somewhat smaller when regression tech-niques or two-level analyses are used, but even then it is difficult toassess the relative importance of school factors because of the differencesin school factors represented in several studies, and because of the differ-ences in indicators of achievement (student scores on standardised tests,exam results, attained educational level).

Generally it is assumed, at least in the Western hemisphere, that 10 to20 per cent of variance in student achievement can be attributed to schoolfactors after correction for student intake measures such as aptitude orsocial class (Bosker & Scheerens, 1994; Mortimore et al., 1988; Scheer-ens, 1992). It is important to note that these percentages only set theboundaries for the potential impact of school factors. The question as towhich school factors are most important for student achievement still islargely unanswered.

THEORETICAL AND EMPIRICAL CONSIDERATIONS

Taking the set of 9 school factors discerned in research reviews and ineducational effectiveness models as a point of departure, critical remarkscan be made from a theoretical perspective and from an empirical per-spective.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Theoretical PerspectiveTheoretical considerations refer to the theoretical basis of school factors,the conceptual range of school factors, the mutual relationships betweenschool factors, and the relationships between school factors and class-room level factors.

The Theoretical Basis of School FactorsThe five factors suggested by Edmonds have dominated school effective-ness research for years, although initially the five factors were neitherbased on a specific theory nor abundantly supported by research. Empiri-cal support has grown over the years, but still the factors are not firmlyembedded in a set of theoretical concepts that can explain the differencesin student achievement as far as these are caused by schools. The educa-tional effectiveness models developed so far study school factors fromvarying theoretical perspectives (learning theory, eclectic approach, or-ganisational theory), but the actual factors in the models were partlyincluded because of their empirically supported status. This explains thesimilarities of the models, despite the different theories that they arebased upon. The translation of concepts from organisational and econom-ical theories to the educational domain, as suggested by Scheerens (1992),has not yet led to a new set of concepts or a different view on the impor-tance of the traditional school factors.

The Conceptual Range of the School Factorsthe lack of a theoretical basis explains the wide conceptual variety ofschool factors. Even when only the main categories of factors in researchreviews and models are considered, the factors represent quite differentdimensions. There are factors referring to the behavior of persons in theorganisation (mainly the principal), to persons related to the school (par-ents), to material conditions (quality of school curricula), to organisation-al elements (recruitment of staff), to school culture (school climate, highexpectations), and to educational aspects (monitoring and evaluation).Although all of these dimensions may be important for school effective-ness, it is necessary to arrange them according to theoretical notions tomake their importance for student achievement easier to understand.

The Mutual Relationships Between School FactorsBecause of the variety in dimensions, the relationships between schoolfactors are not very clear. Some factors are partly overlapping (e.g., mon-itoring/evaluation and achievement-oriented policy), some factors are causedby or at least influenced by other factors (e.g., an orderly environment by

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


educational leadership), and some factors seem to be conditional for eachother (e.g., high expectations and a focus on basic skills/learning time).More clarity about these relationships is needed to interpret effects ofschool factors correctly.

The Relationships of School Factors with Classroom Level FactorsSome school level factors, for example focus on basic skills, monitoring/evaluation, and high expectations, refer to processes that seem to be merelyessential at the classroom level. When teachers are not focusing on basicskills, it is hard to imagine what the special contribution of the schoollevel could be in this respect and what the concept might actually mean atthe school level. Such factors derived from classroom level processes aresomewhat problematic because it is not clear if they refer to more than amean score of teacher behaviour indicators.

School factors are generally described as conditional for classroomfactors (Scheerens, 1992; Stringfield & Slavin, 1992), or interacting withclassroom level factors (Slater & Teddlie, 1992) but the relationships ofschool factors with classroom level factors are not delineated in detail inthe models. In general, one block of school level factors is depicted asinfluencing one block of classroom level factors. It is not clear whichsubgroups of school factors are supposed to influence which subgroups ofclassroom level factors and in what way they are supposed to do so(Bosker & Scheerens, 1994).

Empirical PerspectiveThe empirical considerations about school factors concern the lack ofinformation about their influence on student achievement controlled forthe influence of classroom factors. Research in the field of school effec-tiveness is often criticised, but despite all these criticisms the school fac-tors have obtained a status which is not sufficiently supported by empiricalfindings, probably because they appeal to practitioners and school improv-ers as important or essential factors. Whether school effectiveness find-ings can really be attributed to school level factors or are confounded withclassroom level factors, and therefore should be attributed to some extentto classroom level factors is not a regular topic for discussion.

However, the design of most studies, i.e., school factors directly relat-ed to student achievement without controlling for the influence of class-room factors leaves questions on the validity of school level findingsunanswered. Because of the separate research traditions in the fields ofschool and classroom effectiveness, up till now very few studies havecollected data at both levels. According to Teddlie and Stringfield (1993),

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


only nine studies on school and teacher effects in a total of 328 that werereviewed in the 1980s involved both levels. And even when studies havecollected data both at the school and the teacher level, school effects andclassroom effects were often analysed separately instead of simultane-ously (Brandsma, 1993; Mortimore et al., 1988; Van der Werf & Gulde-mond, 1994). Although school factors are theoretically thought to influ-ence student achievement indirectly through classroom factors, researchconcentrated on direct effects and as a consequence may have under- oroverestimated the actual influence of school factors.

Factual information on the relative impact of school and classroomfactors is provided by Wang et al. (1993). In their overview of the influ-ence of six theoretical constructs on student learning (the average T scoreset at 50), school factors show an average score of 45. They are surpassedsubsequently by student characteristics (T score 55), classroom practices(T score 53), home and community educational contexts (T score 51), andthe design and delivery of curriculum and instruction (T score 47). Theonly construct ranging lower than school factors is state and district gov-ernance and organisation (71 score 35). Teddlie and Stringfield (1993)performed regression analyses with mean student achievement as the de-pendent variable and 17 factors derived from student, teacher and schoollevel data as the predictors. Each of the 17 factors initially correlatedsignificantly with achievement, but the regression procedure showed thatin the end only 3 factors, all of them at the student level, contributedsignificantly to the measurement model in the end. Similar results werefound in a two-level HLM reanalysis. Student and classroom level factorsare potentially more powerful predictors of student achievement thanschool level factors are.

Estimating the Impact of School Level FactorsThe degree of inaccuracy in the estimation of the impact of school levelfactors ideally can be established by performing two-level (school and stu-dent) and three-level (school, classroom, and student) analyses on an iden-tical dataset and then comparing the estimations of the proportions of schoollevel variance. Even more ideally, achievement measurements over severalpoints in time would be included which would lead to a four-level analysis(time points nested within students within classrooms within schools). Four-level analyses, although they can be performed by some multilevel packag-es have not been published so far. Even three-level analyses are hardlyavailable in the educational effectiveness literature, and it is even harder tofind comparisons between two-level and three-level estimations, as describedabove. Some studies do offer information on the proportions of variance at

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


the school level when estimated simultaneously with the classroom level ina two-level analysis and when estimated separately in a three-level analy-sis. For example, the proportion of school level variance in a simultaneousestimation was 15 for language achievement and 24 for mathematics achieve-ment (Reezigt, 1993). When the classroom level was introduced in analy-ses of roughly the same dataset, the proportions of school level variancewere reduced to 13 and 15, respectively, while the proportions of class-room level variance amounted to 6 and 11 (Bosker, 1991). So, the separateestimation of the classroom level and the school level variance shows otherresults than does the simultaneous estimation.

In the IEA-studies on mathematics achievement in secondary educa-tion (Scheerens et al., 1989), the mean proportion of variance at the schoollevel when estimated simultaneously with the classroom level is 42 (8countries, range 8-64). The classroom and the school level variance wereestimated separately in 9 countries. The mean proportions are 33 for theclassroom level (range 17-46) and 7 for the school level (range 0-15).These analyses point at an overestimation of the school level variancewhen the classroom level is not analysed separately. The figures men-tioned so far only concern unconditional models. It is still not clear whichspecific factors might account for the variance at the different levels.

To overcome the theoretical and empirical considerations describedabove, firstly school factors have to be related to classroom level factorson the basis of a theory that can explain differences in student achieve-ment by the selection of factors at different educational levels and thatrationally relates factors both within and across levels to each other.Secondly, empirical support for the school factors that are selected ac-cording to this procedure might be found in research that explicitly takesthe hierarchical structure of education into account. The following sec-tions will deal with these topics.

A THEORETICAL BASIS FOR SCHOOL FACTORS

In the previous sections we have seen that research on school effective-ness revealed a lot of effective school factors and present research goeson detecting even more factors that are connected in one way or anotherto student outcomes. Research can be criticised for methodological rea-sons but also from a theoretical point of view. Recently, based on a pleafor a theoretical foundation of school effectiveness research, several the-oretical models were developed that take into account the difference be-tween the classroom level, where instruction and learning take place, and

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


the levels above the classroom level, where the conditions that contributeto effectiveness at the classroom level should be provided (Creemers1994; Scheerens 1992; Slater & Teddlie, 1992; Stringfield & Slavin,1992). The models are mostly based upon learning theories, as far as theclassroom level processes are concerned. These theories explain howlearning can take place and include instructional factors. The most impor-tant elements of learning theories however are student level factors likesocial background, aptitudes, and motivation.

Instructional Theories as Basis for School Effectiveness ModelsTheories about effective education that start at the classroom level focuson the instructional elements of learning theories. In fact these theories,taking into account the background characteristics at the student level, tryto explain how the instructional factors can contribute to the outcomes ofeducation, or more precisely how differences in educational outcomescan be explained by differences in instruction at the classroom level.These theories emphasize instructional factors that perhaps could bechanged. In addition to the student background characteristics, instruc-tional theories take into account elements or components of instruction atthe classroom level such as the methods used at the classroom level andother learning materials, the learning environment and especially the teacher.Based on the distinction between different components of instruction atthe classroom level, one can discover correlates for effectiveness in re-search, and in fact the correlates for effectiveness from past research canbe re-arranged in a conceptual framework. Taking theories about learningas a point of departure, it is possible to re-arrange these correlates withrespect to learning outcomes and to develop a theory and a programmefor educational research accordingly.

In various instructional theories developed in the past (Bloom, 1976;Car-roll, 1963; Cooley & Lohnes, 1976; Glaser, 1976; Harnischfeger & Wi-ley, 1976; for an overview see Walberg, 1986; Warries & Pieters, 1992),the theoretical constructs are almost the same. The theories and the theo-retical frameworks often make a distinction between quality of instruc-tion, time on task and opportunity to learn. Like Carroll (1963), they taketime and opportunity together. However, one could argue that time itselfis somewhat different from what happens within the time framework. Forthat reason we would like to make a distinction between the time on taskon the one hand and the opportunity to learn on the other.

Time and opportunity can be discerned in several categories. Harnisch-feger and Wiley (1976), for example, distinguish seven categories oftime. The same holds even more for quality of instruction, which can be

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


outlined in quite a lot of different elements, especially when quality isdistinguished for different components within the instructional processsuch as curricula, grouping procedures, and teacher behaviour. Each ofthese three components of instruction can contribute to the quality ofinstruction and to put it differently, can have characteristics that are allconnected with effectiveness of education at the classroom level. Thesecomponents can have a direct influence on learning outcomes, but theycan also influence time and opportunity: the quality of instruction caninfluence learning outcomes directly and also have an indirect influenceby generating time for learning and enlarging the opportunity to learn.

When we make a distinction between instruction and learning, it isobvious that there is a difference between what is offered to students andthe actual use students make of that offer. That holds both for time and foropportunity. The quality of instruction can influence the use that studentsmake of time and opportunity as well as the amount they need before theymaster the objectives of education. With respect to time, this distinctionbetween offered time and used time is obvious in terms of allocated andengaged time. With respect to opportunities, it is more rare, but thisdistinction can provide a good instrument to explain differences in stu-dent outcomes.

More or less elaborated, at the instructional level theories and theoret-ical frameworks are available based on learning theories, especially thelearning theory developed by Carroll (1963). These theories focus oninstruction at the classroom level. Characteristics at this level are catego-rised in overarching categories that contribute to the outcomes of instruc-tion. These overarching categories are quality of instruction, time, andopportunity. This leads to the categorisation of correlates for effectiveinstruction found so far, such as evaluation, high expectations, emphasison cognitive learning, and grouping procedures in these overarching cat-egories (see, for example, Creemers, 1991; Scheerens, 1989; Stringfield& Slavin, 1992; Teddlie & Stringfield, 1993).

The Conditional Role of the School LevelWe take as point of departure that learning and especially differences inlearning outcomes have to be explained by the primary processes at theclassroom level. The primary processes directly influence time on taskand opportunities to learn used by students and indirectly student achieve-ment. It can not be expected that the school level directly contributes totime on task and opportunities used by students or to student achieve-ment. The school, as said before, can, however, provide conditions forquality of instruction, time for learning and opportunity to learn at the

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


school level

classroom level

Quality/educationalQuality/organisationalTimeOpportunity

>f

Quality of instruction- curriculum

grouping proceduresteacher behaviour

>1

Time for learningOpportunity to learn

>

student level

1

Time on taskOpportunities used

Motivation

tAptitudesSocial background

— *

Studentachievement

tj

Fig. 2. The conditional role of the school level.

classroom level. We have to keep in mind, however, that in the empiricalprocess with respect to the influence of the school level conditions, theycan be absorbed by the classroom level. It is postulated that at the schoollevel the conditional factors can be related to the overarching categoriesmentioned earlier: quality, time, and opportunity. The conditional role ofthe school level is depicted in Figure 2.

Quality, Time, and Opportunity at the School LevelFigure 2 shows that school level factors are supposed to directly influ-ence the quality of instruction, time for learning and opportunity to learn

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


at the classroom level. Their influence on student achievement is mediat-ed by time on task and opportunities used at the student level. Schoollevel factors are categorised within the same conceptual notions of quali-ty, time and opportunity as the classroom level factors. All specific schoollevel factors are supposed to be related to either quality or time or oppor-tunity.

The school level includes the organisation of the school (teachers,students, parents), but also the educational system above the classroomlevel. This relates to the curriculum of the school, such as the text-books and the time schedule. This suggests a distinction in the schoollevel as an organisational and an educational system. Both systems arerelated to each other, but the first can create and sustain the situation inwhich education takes place to some extent, comparable with what man-agement of the classroom 'does' for instruction. The factors at the schoollevel are conditions for what goes on at the classroom level. Conditionscan be clear, such as the curriculum, or less apparent, like the structure ofthe organisation. These also can affect the instructional process by, forexample, influencing what happens between principals and teachers(see, for example, Rosenholz, 1989; Stoll, 1992; Teddlie & Stringfield,1993).

A distinction has been made between educational and organisationalaspects of the quality of instruction, although both are interrelated. Withrespect to the educational aspects, the rules and agreements in the schoolconcerning the instructional process at the classroom level are of utmostimportance, especially those related to curricular materials, grouping pro-cedures and teacher behaviour. It is important to choose curricular mate-rials for specific grade levels that are also used in other grade levels, or atleast to ensure that they have the same structure. There should be a policyat the school level which defines educational goals that have to be achievedin classrooms. This does not imply a wide range of goals that may beimpossible to achieve by schools and teachers, but realistic goals that canactually be achieved in education and that can give guidance to teacherbehaviour. When a specific grouping procedure is chosen, it is importantto implement the same grouping procedure in several grade levels be-cause effectiveness is also enhanced by constancy. There should be apolicy at the school level to guide the continuity of grouping proceduresacross grade levels. Schools should have an evaluation policy that directsactivities at classroom and student level by means of a student monitoringsystem. Schools can promote the testing of students and stimulate teach-ers to give information about test results to students, to take correctivemeasures, to monitor student progress and to act as necessary based on

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


the student progress, providing opportunities for rehearsal, correctivematerials and remedial teaching.

The organisational aspects of quality at the school level deal with theintervision (mutual supervision) of teachers and the professionalisationof teachers and principals. These aspects not only refer to the structure ofthe organisation, but also to the way teachers and principals cooperate topromote the effectiveness of instruction at the classroom level and of theschool as a whole. In this respect, the school principal as an 'instructionalleader', who takes responsibility for the conditions mentioned above canbe an important conditional variable (Smith & Andrews, 1989). Teamconsensus about the mission of the school and the way to fullfill thismission through shared values will clearly support the activities of indi-vidual teachers and result in continuity (see Sashkin & Walberg, 1993).This can create a school culture in principals, teachers, students and par-ents that promotes and supports effectiveness (Cheng, 1993). In this re-spect, the influence of the school board should also be mentioned. Schoolboards, like principals, can support school effectiveness and the effec-tiveness of classroom instruction. When they confirm school curriculumdocuments, they should attend to the elements mentioned above. Theyshould strive for effective education when deciding about personnel, forexample when they recruit new teachers or make decisions about careerplanning. School boards that are more attentive in this respect can in-crease effectiveness at the classroom level in the end, through the meansof school principals, school teams and individual teachers (Hofman, 1993).

Conditions for time at the school level are connected with the timeschedule. For all grade levels, this schedule spells out how much timeshould be devoted to different subjects. Apart from that, it is important tokeep track of the time schedule. In less effective schools, a lot of sched-uled time is wasted, because there is no system to control classrooms inthis matter.

The time available for learning can be expanded by a homework poli-cy. In this respect good relations with the home environment can contrib-ute to effectiveness. When homework assignments are well controlledand structured, and adequate feedback is given, such assignments canexpand effective learning time outside the school. Good contacts betweenschools and parents and agreements about school policies and activitiesmay lead to effective use of time spent on homework. When parentsexpect their children to achieve goals set by the school, the effectivenessof education increases.

Measures at the school level can maintain an orderly atmosphere thatfacilitates teaching and learning at the classroom level. Learning time is

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


increased by an orderly and quiet classroom climate. At the school levelsuch a climate can be fostered; therefore, it is necessary to establishorder, a quiet atmosphere and regularity, and to support teachers trying toachieve an orderly climate in their classes.

At the school level, conditions can be created that contribute to theopportunity to learn at the classroom level, which is especially important,at the school level, between classes and grade levels within the school. Atthe school level, the opportunity to learn is provided by the developmentand availability of documents such as a formal curriculum, a school workingplan and an activity plan on what has to be done to pursue the goals of thecurriculum and the working plan. In this document, the school manage-ment team can explain its vision about education and make clear howeffectiveness will be pursued in the school. Schools should have a policythat increases effectiveness. Schools should feel responsible for studentachievement: it is their 'mission' to contribute to achievement. A schoolpolicy based on these principles can yield important effects. It has alreadybeen mentioned that school working plans are not very effective, but theireffects might be improved by a stronger relationship to instructional ef-fectiveness at the classroom level. But not only the document is impor-tant; it is also important to use the document and to 'control' what hap-pens in the schools and classrooms with respect to the school curriculum.This is especially important for transitions between classes and grades.Therefore, there should be rules and agreements about how to implementthe curriculum at the school level.

In summary, the following conditions at the school level can be describedfor quality of instruction with respect to the educational aspects:- rules and agreements about all aspects of classroom instruction, espe-

cially curricular materials, grouping procedures and teacher behaviour,and the consistency between them;

- an evaluation policy and a system at school level to check on pupilachievement, to prevent learning problems or to correct problems at anearly stage. This includes regular testing, remedial teaching, studentcounselling and homework assistance.

With respect to the organisational aspects at the school level importantconditions for quality of instruction are:- a school policy on intervision and supervision of teachers, heads of

departments and school principals by higher-ranking persons, and aschool policy to correct and further professionalise teachers who donot live up to the school standards;

- a school culture inducing and supporting effectiveness.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Conditions for time at the school level are:- the development and provision of a time schedule for subjects and

topics;- rules and agreements about time use, including the school policy on

homework, pupil absenteeism and cancellation of lessons;- the maintenance of an orderly and quiet atmosphere in the school.Conditions for the opportunity to learn at the school level are:- the development and availability of a curriculum, school working plan

or activity plan;- consensus about the mission of the school;- rules and agreements about how to proceed and how to follow the

curriculum, especially with respect to transition from one class to an-other or from one grade to another (Creemers et al., 1992).

It is not our purpose to give an overview of all kinds of school factors.This has been done before (Levine & Lezotte, 1990). For the Internation-al School Effectiveness Research Programme, Reynolds (1993) provideda possible list of factors. The majority of the factors in these lists arecomparable with the ones mentioned above, that were derived accordingto the conceptual notions of quality, time and opportunity. Resources arenot regarded as a separate category of school factors. It is not expectedthat just providing additional finance and resources to schools is likely toimprove them (Cohn & Geske, 1990; Walberg, 1992). Resources shouldbe defined in such a way that their relationship to effectiveness is clari-fied, i.e. by describing their influence on curricular materials, teacherbehaviour and other factors that support education.

The Importance of Formal CriteriaResearch on effective education at the classroom level shows that isolat-ed components or effective elements of individual components do notresult in strong effects on student achievement. Good curricula need teacherswho can make adequate use of them and who will show effective instruc-tional behaviour. The same holds for grouping procedures. Integration ofcomponents is necessary to achieve substantial effects. An integratedapproach for education at the classroom level with an even higher level ofintegration than direct instruction or mastery learning is necessary. In thisintegrated approach, the educational components of curricula, groupingprocedures, and teacher behaviour are adapted to each other. To achievetime and opportunity to learn, the three components in general shouldhave the same effective characteristics. This is called consistency of theeffective characteristics, which can cause synergistic effects which influ-ence the effects of instruction at the classroom level positively.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Like consistency at the classroom level, there are formal criteria crite-ria for effectiveness at the school level, especially when the longitudinalaspect of effectiveness is at stake. Students move from one grade level toanother, from one class to another and from one school subject to another.Formal criteria can not be seen immediately in schools, but we can con-clude that they exist based on the fact that the same factors are seen acrossinstructional components, subjects, grades, and classes. They are impor-tant conditions for effectiveness and continuing instructional settings withinthe school. These formal criteria are consistency, cohesion, constancy,and control (Creemers, 1991, 1994; Reynolds & Creemers, 1992; Rey-nolds, 1993; Stringfield & Slavin, 1992):- Consistency: at the school level, conditions for effective instruction

related to curricular materials, grouping procedures and teacher behav-iour should be in line with each other.

- Cohesion: all members of the school team show consistency of effec-tiveness characteristics. In this way, effective instruction between classescan be guaranteed.

- Constancy: effective instruction is provided during the total schoolcareer of students. Too often students are confronted with differences

Characteristics of Quality, Time and Opportunity Formal Criteria

Quality/educational rules and agreements about classroom Consistencyinstructionevaluation policy/evaluation system

Quality/organisational policy on intervision, supervision,professionalisationschool culture inducing effectiveness

Cohesion

Time time schedulerules and agreements about time useorderly and quiet atmosphere

Constancy

Opportunity school curriculumconsensus about missionrules and agreements about how toimplement the school curriculum

Control

Fig. 3. School level factors and formal criteria.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


in instruction between teachers and grade levels. Constancy means thatconsistency and cohesion are guaranteed over longer periods of time.

- Control: control refers not only to the evaluation of student achieve-ment and teacher behaviour, but also to an orderly and quiet schoolclimate, which is necessary to achieve results. Control also refers toteachers holding each other esponsible for effective instruction.

Although we restrict ourselves to the school level, the theory about edu-cational effectiveness outlined in this section can be expanded above theschool level to the contextual level, taking into account the same ele-ments of quality, time, and opportunity, and the formal criteria for effec-tiveness.

The school level factors that are supposed to be conditional for theclassroom level and the formal criteria at the school level are summarisedin Figure 3.

THREE-LEVEL RESEARCH ON SCHOOL LEVEL FACTORS

The conditional role of school level factors needs empirical support byresearch that explicitly takes into account the hierarchical structure ofeducation (students nested within classrooms nested within schools). Three-level designs can provide the best evidence in this respect. Althoughsoftware for three-level modeling has been available for some time now,very few applications can be found in educational research as yet (seeAppendix A for further information). Beside the unfamiliarity of educa-tional researchers with this technique, which explains the lack of rean-alyses of existing three-level datasets, budgetarial constraints are hinder-ing the collection of new data at three levels. A certain minimum (e.g.,25) units of analysis at the highest level is required to prevent robustnessproblems. With two classrooms per school, a three-level study then com-prises 50 units at the classroom level and, assuming an average number of25 students per classroom, 1250 units at the student level.

Seven studies on three-level analyses concerning the student, class-room and school level were tracked representing research from the Unit-ed States, the Netherlands, and several countries involved in the IEA-studies (see Appendix B for information about features of these studies).The studies of Scheerens et al. (1989) and Luyten (1994) are partly over-lapping in their datasets, as are the studies by Bosker et al. (1990) andBosker and Scheerens (1994).

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


The Allocation of Variance across the Classroom and the School LevelIdeally, the studies would perform four steps in the multilevel analyses(Bryk & Raudenbush, 1992) to provide information on:

- the allocation of variance across the three levels in a fully uncondition-al model, to get a preliminary indication of the relative importance ofthe school level.

- the allocation of variance after the specification of predictors at thestudent level (student background indicators, pretest measures), to findout what the relative importance of the school level in explaining dif-ferences between students that cannot be attributed to student charac-teristics (the 'net' effects of schools and classrooms).

- the allocation of variance after the specification of classroom and schoolpredictors, to find out which proportions of variance at different levelsare accounted for by classroom and school characteristics.

- the allocation of variance after the specification of fixed and randomeffects as well as cross-level interactions. This shows if classroom andschool characteristics have the same effects on different kinds of stu-dents and if student, classroom, and school characteristics actually dointeract.

However, the studies generally only provide data on the first and thesecond step. Table 1 shows the proportions of variance at the classroomand the school level in an unconditional model, after the specification ofstudent background characteristics and after the specification of studentbackground characteristics as well as pretest measures.

In interpreting the proportions of variance controlled for student back-ground characteristics, it should be noted that the numbers of studentbackground predictors vary in the selected studies. Therefore, some pro-portions are much more heavily controlled for student level influencesthan others. The studies by Bosker (1991) specify the socio-economicstatus (SES) as measured by the educational as well as the occupationallevel of both parents, ethnic background, gender, and aptitude (verbal andperformal intelligence). Van der Tuin (1993) specifies age, aptitude, gen-der, and advice for secondary education of the elementary school princi-pal for the analyses with math achievement as the dependent variable andage, educational support of the father, and advice for the analyses withlanguage achievement as the dependent variable. Both studies also pro-vide proportions of variance after the additional specification of a pretestscore, taken one year (Bosker) or two years (Van der Tuin) earlier thanthe posttest. In the IEA-data reanalysed by Scheerens et al. (1989), SESas measured by the occupational level of the student's father is the only

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Table 1. Proportions of Variance at the Classroom and the School Level.

unconditional students background

model* background** and pretest***

class school class school class school

Elementary SchoolStringfield & Teddlie (1989)- achievement 1 2 1 3 - - - -

Bosker(1991)- math (different grades) 11 15 7 9 6 5- language (different grades) 6 13 5 3 4 1

- math (same grade) 5 22 3 13 4 8- language (same grade) 0 14 0 8 0 4

Secondary School

Scheerens et al., 1989- math

Canada 18 9 17 9 -Finland 45 0 41 0 -France 17 6 16 5 -Israel 22 10 21 8 -Luxemburg 29 15 29 15 -New Zealand 45 1 42 0 -Scotland 34 12 31 5 -Sweden 45 0 45 0 - -United States 46 10 45 9 - -

Bosker et al., 1990- math 6 11 4 11 -

VanderTuin, 1993- math 15 19 - 2 9- language 10 12 - 2 2

Luyten, 1994- math

USA/1 st subsample 61 0 - - -USA/2nd subsample 30 26 - -Sweden 44 0 - -

Bosker & Scheerens, 1994- math 2 30 2 30 -

Note. Unconditional model; **after the specification of student background characteris-tics; ***after the specification of student background characteristics as well as pretestmeasures.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


predictor. The studies by Bosker et al. (1990) and Bosker and Scheerens(1994) specify SES, gender, and ethnic background as predictors at thestudent level.

In elementary education the proportions of variance in an uncondition-al model outscore the proportions of classroom variance. In the IEA-studies of Scheerens et al. (1989) and Luyten (1994) on secondary educa-tion, the proportions of classroom variance in an unconditional modelconsequently outscore the proportions of school level variance. However,in the Dutch studies of Bosker (1990), Van der Tuin (1993) and Boskerand Scheerens (1994) the school level is consequently dominating theclassroom level, although the differences in the proportions of variancebetween both levels are much smaller than the IEA-data suggest.

The relative importance of the classroom and the school level remainsthe same after the specification of student background characteristics,with the exception of the findings of Bosker (1991) and Van der Tuin(1993). In the study of Bosker on elementary schools, the school levelloses its initial predominance when student predictors are taken into ac-count, in particular when pretest scores are included in the analyses. Inthe study of Van der Tuin the difference between school level and class-room level proportions of variance in language achievement disappearswhen background characteristics and pretest scores are taken into ac-count.

The main conclusion to be drawn from Table 1 is that some but not allfindings lend support to the conditional role of the school level for theclassroom level. However, the exact proportions of classroom level vari-ance vary from one study to another. Because of this variation, statementsabout the importance of the school level can only be tentative. In addi-tion, the stability of proportions of school level variance in two relativelylarge subsamples (Luyten, 1994: 58 schools in each subsample) is notvery high. These findings make it even harder to make statements aboutthe potential impact of the school level.

Specification of Classroom and School Level PredictorsUnfortunately, the two studies concerning elementary schools (String-field & Teddlie, 1991; Bosker, 1991) have not specified classroom andschool level predictors. The study by Luyten (1994) only concerns schoolsize, which does not have any effect on student achievement. The otherstudies perform analyses involving varying sets of classroom and/or schoollevel predictors. All of these studies concern math achievement (becauseof the very low porportion of school level variance in language achieve-ment after the specification of student background characteristics and

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

Scheerens et al., 1989 Bosker et al., 1990 VanderTuin, 1993 Bosker & Scheerens, 1994

School Level Factors Related to Quality, Time, and Opportunity

(none)

urbanisationn female math teachersn male math teachersfrequency of math teachermeetings

time for learning (time spentkeeping order/teaching)opportunity to learn (testitems covered)

evaluation (use of tests/published vs.teacher made)high expectations (estimationof number of high achievers)

- teacher experience- class size

- rules and agreements(teacher behaviour)

- rules and agreements(teacher behaviour)

- evaluation policy- school culture inducing

effectiveness

Other School Level Factors

- school size (n of classes)- type of school (compre-

hensive/categorical)

- tracking- parental involvement

Classroom Level Factors Related to Quality, Time, and Opportunity

- time for learning (time (none)spent on instruction)

- opportunity to learn(curriculum coverage)

- evaluation (ways of assess-ment and use of information)

- emphasis on cognitivelearning

Other Classroom Level Factors

- teacher satisfaction (none)

rules and agreements(teacher behaviour)evaluation policyconsensus about mission

curricular trackmean SES% of boys

time for learning (timespent on instruction)opportunity to learn(curriculum coverage)

evaluation (ways of assessmentand use of information)emphasis on cognitive learninggrouping procedure (degreeof individualisation)

teacher satisfactionteacher autonomy

Fig. 4. Classroom level and school level factors in three-level studies.

toK)O

CO

yso73tnratn73

>

DO

73W

o

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


pretest scores, Van der Tuin only performed further analyses on the mathachievement data). The predictors are not always operationalisations ofclassroom and school factors related to the categories of quality, time,and opportunity. Figure 4 gives an overview of the classroom and schoollevel predictors that were studied.

Figure 4 shows that the three-level studies cover the classroom levelfactors better (at least by proxies) than the school level factors related toquality, time, and opportunity. As far as the school level is concerned,more attention is paid to school demographics such as school size andmean SES than to factors that are related to quality, time and opportunity.Scheerens et al. (1989) performed three-level analyses on the IEA-data of9 countries. Classroom and school predictors vary from country to coun-try because predictors had to show a significant relationship with achieve-ment in order to be specified. In most countries, the set of predictorssucceeded better in accounting for classroom level variance (up to 41 %)than school level variance (up to 7 %), which is understandable becausein most countries only classroom level predictors were specified.

Bosker et al. (1990) find no main effects of school factors when theyare analysed together with classroom factors. However, when cross-levelinteractions are specified, they find significant effects of rules and agree-ments at the school level as well as a significant interaction of rules andagreements with the classroom factor time for learning: time for learningis especially important in schools with a low level of rules.

Van der Tuin (1993) did not include classroom level factors in thethree-level analyses. She reports positive effects on student achievementof the school factors rules and agreements about teacher behaviour andevaluation policy, and a negative effect of school culture inducing effec-tiveness (operationalised as the extent to which schools show activities toprevent dropout of students). Tracking has a negative effect on mathachievement, parental involvement a positive effect. All school level fac-tors together account for a major proportion of school level variance,although 2 per cent (out of 9 %) is left unexplained. Van der Tuin did notspecify cross-level interactions.

In the analyses by Bosker and Scheerens (1994), none of the schoolfactors except for curricular track show main effects on achievement.Again, the specification of cross-level interactions suggests that the influ-ence of school level factors is mainly mediated by classroom level fac-tors: significant interactions are found for rules and agreements (with theclassroom factor grouping procedure), consensus (with emphasis on cog-nitive learning and time for learning) and evaluation policy (with empha-sis on cognitive learning, time for learning, and teacher satisfaction).

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


The school factors investigated in the four studies cover only a verysmall part of the school factors related to quality, time, and opportunity(see Figure 3). For this reason, conclusions on the supposed importanceof the school factors cannot be drawn at this stage of research. It shouldbe noted that the absence of main effects in some studies does not auto-matically imply that school factors are not important for student achieve-ment: effects of school factors sometimes only appear when interactionswith classroom factors are specified. The assumption that school factorsare important, and that they are mainly important because of the way theyare related to classroom factors, is supported to some extent by some ofthe findings of the three-level analyses.

SUMMARY AND CONCLUSION

For a long time, school effectiveness research concentrated on findingrelationships between school level factors and indicators of student achieve-ment. Most studies did not investigate factors at the classroom level.Research into the relationships of all kinds of classroom factors wentlargely unnoticed by the school effectiveness research community, al-though recently this situation is changing. Because of the absence ofclassroom level factors in most school effectiveness studies, the well-known lists of effective school level correlates can be challenged withregard to their actual empirical status. The effect sizes now attributed toschool factors, often based on correlations with student achievement orsignificant effects on student achievement as shown in two-level analy-ses, might be reduced when classroom level factors are taken into ac-count.

The lists of effective school correlates can also be challenged from atheoretical point of view. Until now, school factors were hardly integrat-ed in theories on learning and instruction, because these theories onlydealt with factors at the student and the classroom level. Therefore, itseems difficult to define on theoretical grounds which school level corre-lates are essential for student achievement.

In this article, a theoretical basis for school level factors was outlined.Starting on the basis of the key concepts of theories on learning and instruc-tion, i.e. quality of instruction, time for learning and opportunity to learn,factors at the school level were described that are supposed to contribute tostudent achievement because of their influence on classroom level factors.When quality of instruction, time for learning and opportunity to learn aredefined as the essential classroom level factors for student achievement,

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


school level factors should be defined according to the same criteria. Thisimplies that school factors are only supposed to enhance student achieve-ment when it is possible to relate them conceptually to quality, time, andopportunity. The selection of school factors along these lines does not re-sult in a totally new set of correlates, but in a conceptual re-ordering offactors. It is expected that the school factors, selected and ordered in thisway, will better be able to explain differences in student achievement, be-cause of their clear relationship with classroom level factors.

To study the impact of school level factors on student achievement onthe basis of the ideas described above, research should take classroomfactors into account. Because of the separate research traditions in schooland classroom effectiveness research, there are not very many studies thatallow for a simultaneous analysis of the impact of school level and class-room level factors. Some of the studies that do offer these possibilitiesunfortunately have not as yet been analysed optimally. Although data atthe student level, the classroom level and the school level are availablesometimes, analyses were mostly performed on two levels (classroom-student or school-student). Because of the relatively recent developmentsin multilevel techniques there are only a few examples of three-levelanalyses in which student achievement is the dependent variable andfactors at both the classroom and the school level are specified. Theresults concerning the relative proportions of classroom and school levelvariance, computed in unconditional models, are somewhat contradicto-ry. Some studies offer proportions of variance that are much more heavilycontrolled for student background characteristics than others, which makescomparisons very difficult. Moreover, most three-level datasets have con-centrated on demographic school factors or rather loosely operationalisedschool factors. Even though some of the results of research mentioned inthis article seem to support the ideas about which school factors should beconsidered important and about the way they influence student achieve-ment, it is far too early to make definite statements in this respect. To findout whether the concepts of quality, time and opportunity are meaningfulconcepts for the school level as they have shown to be for the classroomlevel more research is necessary. Reanalyses of existing datasets is arelatively easy way to obtain useful results. However, in order to opera-tionalise the key concepts of quality, time, and opportunity at the schoollevel adequately, new research studies should also be undertaken.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


APPENDIX A: SELECTION OF THREE-LEVEL STUDIES

To track examples of three-level studies in educational research, the fol-lowing books and journals were checked:- proceedings of the ICSEI-meetings in London (Reynolds, Creemers &

Peters, 1989), Rotterdam (Creemers, Peters & Reynolds, 1989) andJerusalem (Bashi & Sass, 1992) and papers from the meetings in lateryears,

- volumes of the International Journal of Educational Research on schooleffectiveness (Creemers & Scheerens, 1989; Bosker, Creemers & Schee-rens, 1994),

- volumes of School Effectiveness and School Improvement (since Janu-ary 1990),

- volumes of Review of Educational Research since 1990,- volumes of the American Educational Research Journal since 1990,- volumes of the Multilevel Modelling Newsletter (since October 1989),- volumes of the Dutch research journal Tijdschrift voor Onderwijsweten-

schappen on school effectiveness (Van den Eeden & Meijnen, 1990;Creemers & Scheerens, 1991; Creemers, 1994),

- references to three-level modeling in recent books on multilevel mode-ling (i.e. Raudenbush & Willms, 1991; Bryk & Raudenbush, 1992).

Also, some examples of three-level research were tracked by means ofpersonal communications with leading researchers in the field of educa-tional effectiveness.

Three-level analyses are still very uncommon. For example, in the 17issues of School Effectiveness and School Improvement that were pub-lished since 1990, with three or four publications per issue, eight exam-ples of two-level research were found but only three examples of three-level modeling. Moreover, not all three-level analyses are performed withstudent achievement as the dependent variable (i.e. Hill, Holmes-Smith,& Rowe, 1993) or actually offer information on the relevant levels in thiscontext: student, classroom, and school. Some publications are aboutmodeling individual growth within the organisational context of class-rooms and/or schools (i.e. Bryk & Raudenbush, 1992). Other publicationsconcern levels such as classes, teachers and schools (Raudenbush, Row-an, & Cheong, 1993), items, teachers and schools (Raudenbush, Rowan,& Kang, 1991), repeated measurements at teacher level, teachers andschools (Raudenbush, Rowan, & Kang, 1994), students, ability levels andschools (Nuttall, Goldstein, Prosser, & Rasbash, 1989), students, cooper-ative groups and classrooms (Terwel, Van den Eeden, and Mooij, 1992),teachers, schools and school support services (Stokking, 1993), students,schools and Education Authorities (Willms, 1987).

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

APPENDIX B: FEATURES OF THREE-LEVEL STUDIES

authors/year of publication level of education dependent variables n of students, classrooms, schools

1. Scheerens et al.,1989

2. Stringfield & Teddlie,1989

3. Bosker, et al.,1990

4. Bosker, 1991

5. VanderTuin, 1993

6. Bosker & Scheerens,1994

7. Luyten, 1994

secondary (grade 2) math achievement

elementary achievement test


elementary (grade 6/8) math achievementlanguage achievement

secondary (grade 3) math achievementlanguage achievement



secondary (grade 1)

varying per country(17 countries)

5400 students, 250 teachers,76 schools


a. 4658 students, 300classes, 150 schoolsb. 485 students, 48 classes,24 schools

1298 students (math), 1332students (language), 90 classes,26 schools


a. 4507 students, 211classes, 116 schools (USA)b. 3500 students, 182classes, 95 schools (Sweden)

1/1

noorra

to

noo

o

on

to

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


REFERENCES

Bashi, J., & Sass, Z. (1992). School effectiveness and improvement. Jerusalem: MagnesPress.

Bloom, B.S. (1976). Human characteristics and school learning. New York: McGrawHill.

Bosker, R.J. (1991). De consistentie vam schooleffecten in het basisonderwijs (The con-sistency of school effects in primary education). Tijdschrift voor Onderwijsresearch,16, 206-218.

Bosker, R.J., Creemers, B.P.M., & Scheerens, J. (1994) (Eds). Conceptual and methodo-logical advances in educational effectiveness research. International Journal ofEducational Research, 21.

Bosker, R.J., Kremers, E.J.J., & Lugthart, E. (1990). School and instruction effects onmathematics achievement. School Effectiveness and School Improvement, I, 233-248.

Bosker, R.J., & Scheerens, J. (1994). Alternative models of school effectiveness put to thetest. International Journal of Educational Research, 21, 159-181.

Brandsma, H.P. (1993). Basisschoolkenmerken en de kwaliteit van het onderwijs. [Pri-mary school characteristics and the quality of education]. Groningen: RION.

Brookover, W.B., Beady, C.H., Flood, P.K., Schweitzer, J., & Wisenbaker, J. (1979).Schools, social systems and student achievement: Schools can make a difference.New York: Bergin.

Bryk, A.S., & Raudenbush, S.W. (1992). Hierarchical linear models: Applications anddata analysis methods. London: Sage.

Carroll, J.B. (1963). A model of school learning. Teachers College Record, 64, 723-733.Cheng, Y.C. (1993). Profiles of organizational culture and effective schools. School Ef-

fectiveness and School Improvement, 4, 85-110.Cohn, E., & Geske, T.G. (1990). The economics of education. Oxford: Pergamon.Cooley, W.W., & Lohnes, P.R. (1976). Evaluation research in education. New York:

Wiley.Creemers, B.P.M. (1991). Effectieve instructie [Effective instruction]. Den Haag: SVO.Creemers, B.P.M. (1994). The effective classroom. London: Cassell.Creemers, B.P.M. (1994) (Ed.) Contributions to educational effectiveness. Tijdschrift

voor Onderwijsresearch, 19.Creemers, B.P.M., Reezigt, G.J., & Van der Werf, M.P.C. (1992). Development and

testing of a model for school learning. Groningen: RION.Creemers, B.P.M., Peters, T., & Reynolds, D. (1989). School effectiveness and school

improvement. Amsterdam: Swets & Zeitlinger.Creemers, B.P.M., & Scheerens, J. (1989) (Eds.). Developments in school effectiveness

research. International Journal of Educational Research, 13.Creemers, B.P.M., & Scheerens, J. (1991). School- en instructie-effectiviteit [School and

instructional effectiveness]. Tijdschrift voor Onderwijsresearch, 16.Edmonds, R.R. (1979). Effective schools for the urban poor. Educational Leadership, 37,

15-24.Fraser, B.J., Walberg, H.J., Welch, W.W., & Hattie, J.A. (1987). Syntheses of educational

productivity research. International Journal of Educational Research, 11, 145-252.

Glaser, R. (1976). Components of a psychology of instruction: Toward a science ofdesign. Review of Educational Research, 46, 1-24.

Harnischfeger, A., & Wiley, D.E. (1976). The teaching learning process in elementaryschools: A synoptic view. Curriculum Inquiry, 6, 5-43.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Hill, P.W., Holmes-Smith, P., & Rowe, K.J. (1993). School and teacher effectiveness inVictoria: Key findings from phase 1 of the Victorian Quality Schools Project.Melbourne: Institute of Education.

Hofman, R.H. (1993). Effectief schoolbestuur [Effective schoolboards]. Groningen: RION.Levine, D.U., & Lezotte, L.W. (1990). Unusually effective schools. Madison: National

Center for Effective Schools Research and Development.Luyten, H. (1994). School size effects on achievement in secondary education: Evidence

from the Netherlands, Sweden and the USA. School Effectiveness and SchoolImprovement, 5, 75-99.

Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Ecob, R. (1988). School matters.Somerset: Open Books.

Nuttall, D.L., Goldstein, H., Prosser, R., & Rasbash, J. (1989). Differential school effec-tiveness. International Journal of Educational Research, 13, 769-776.

Purkey, S.C., & Smith, M.S. (1983). Effective schools: A review. Elementary SchoolJournal, 83, 427-452.

Raudenbush, S.W., Rowan, B., & Cheong, Y.F. (1993). Higher order instructional goalsin secondary schools: Class, teacher, and school influences. American Educa-tional Research Journal, 30, 523-553.

Raudenbush, S.W., Rowan, B., & Kang, S.J. (1991). A multilevel, multivariate model forstudying school climate with estimation via the EM algorithm and application toU.S. high school data. Journal of Educational Statistics, 16, 295-330.

Raudenbush, S.W., Rowan, B., & Kang, S.J. (1994). Methods for studying stability, sus-tained change, and temporal instability in school climate. Michigan State Uni-versity: College of Education.

Raudenbush, S.W., & Willms, J.D. (1991). Schools, classrooms, and pupils. San Diego:Harcourt Brace Jovanovich.

Reezigt, G.J. (1993). Effecten van differentiatie op de basisschool [Effects of grouping inprimary education]. Groningen: RION.

Reynolds, D. (1993). Conceptualising and measuring the school level. Paper presented atthe Annual Meeting of the American Educational Research Association, Atlanta,GA.

Reynolds, D., & Creemers, B.P.M. (1992). The International School Effectiveness Re-search Program: An outline. Cardiff: University of Wales.

Reynolds, D., Creemers, B.P.M., & Peters. T. (1994). (Eds.), School effectiveness andschool improvement. Groningen/Cardiff: RION/University of Wales.

Rosenholz, S.J. (1989). Teachers' workplace: The social organization of schools. NewYork: Longman.

Sashkin, M., & Walberg, H.J. (1993). Educational leadership and school culture. Berkeley:McCutchan.

Scheerens, J. (1989). Wat maakt scholen effectief [What makes schools effective]. DenHaag: SVO.

Scheerens, J. (1992). Effective schooling: Theory, research and practice. London: Cassell.Scheerens, J., Vermeulen, C.J., & Pelgrum, W.J. (1989). Generalizibility of instructional

and school effectiveness indicators across nations. International Journal of Edu-cational Research, 13, 789-799.

Slater, R.O., & Teddlie, C. (1992). Toward a theory of school effectiveness and leader-ship. School Effectiveness and School Improvement, 3, 247-257.

Slavin, R.E. (1987). A theory of school and classroom organization. Educational Psy-chologist, 22, 89-108.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4


Smith, W.F., & Andrews, R.L. (1989). Instructional leadership: How principals make adifference. Alexandria: Association for Supervision and Curriculum Develop-ment.

Stokking, K.M. (1993). Regressie-analyse, variantie-analyse en multiniveau-analyse vanverschillen in het basisonderwijs [Regression analysis, analysis of variance andmultilevel analysis of primary school differences]. Utrecht: ISOR.

Stoll, L. (1992). Teacher growth in the effective school. In M. Fullan & A. Hargreaves(Eds.), Teacher development and educational change (pp. 104-122). London:Falmer.

Spade, J.Z., Vanfossen, B.E., & Jones, E.D. (1985). Effective schools: Characteristics ofschools which predict mathematics and science performance. Paper presented atthe Annual Meeting of the American Educational Research Association, Chicago,IL.

Stringfield, S.C., & Slavin, R.E. (1992). A hierarchical longitudinal model for elementaryschool effects. In B.P.M. Creemers & G.J. Reezigt (Eds.), Evaluation of educa-tional effectiveness (pp. 35-69). Groningen: ICO.

Stringfield, S.C., & Teddlie, C. (1991). Observers as predictors of schools' multiyearoutlier status on achievement tests. Elementary School Journal, 91, 357-376.

Teddlie, C , & Stringfield, S.C. (1993). Schools make a difference: Lessons learned froma 10-year study of school effects. New York: Teachers College Press.

Terwel, J., Van den Eeden, P., & Mooij, T. (1992). Effects of group composition andinteraction in cooperative groups on the learning of mathematics. Paper pre-sented at the Convention of the International Association for the Study of Coop-eration in Education, Utrecht, The Netherlands.

Van den Eeden, P., & Meijnen, G.W. (1990). Onderwijsonderzoek vanuit multi-niveauperspectief [Educational research from a multilevel perspective]. Tijdschrift voorOnderwijsresearch, 15, 259-273.

Van der Tuin, A.C. (1993). Schoolkenmerken en effectiviteit van scholengemeenschappen[School characteristics and the effectiveness of comprehensive schools]. Groningen:Rijksuniversiteit Groningen, Vakgroep Sociologic.

Van der Werf, M.P.C., & Guldemond, H. (1994). De tweede meting van de OVB-evaluatie:School- en klaskenmerken in schooljaar 1990/1991 [The second measurement ofthe evaluation of the Educational Priority Programma: School and classroomcharacteristics in 1990/1991]. Groningen: RION.

Walberg, H.J. (1984). Improving the productivity of America's schools. EducationalLeadership, 41, 19-27.

Walberg, H.J. (1986). Syntheses of research on teaching. In M.C. Wittrock (Ed.), Hand-book of research on teaching (pp. 214-229). New York: MacMillan.

Walberg, H.J. (1992). Educational indicators for educational progress. In B.P.M. Creemers& W.Th.J.G. Hoeben (Eds.), Indicatoren van onderwijseffectiviteit [Indicators ofeducational effectiveness (pp. 7-27)]. Groningen: ICO.

Wang, M.C., Haertel, G.D., & Walberg, H.J. (1993). Toward a knowledge base for schoollearning. Review of Educational Research, 63, 249-295.

Warries, E., & Pieters, J.M. (1992). Inleiding instructietheorie [An introduction in in-structional theory]. Amsterdam: Swets & Zeitlinger.

Willms, J.D. (1987). Differences between Scottish Education Authorities in their exami-nation attainment. Oxford Review of Education, 13, 211-232.

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 0

4:20

25

Oct

ober

201

4

school level conditions affecting the effectiveness of instruction

Documents