document resume ed 203 729 · document resume. he 013 710. scriven, michael the evaluation of...

ED 203 729

AUTHORTITLE

INSTITUTION

SPONS AGENCYPUB DATENOTEAVAILABLE FPOM

EDRS PRICEDESCRIPTORS

DOCUMENT RESUME

HE 013 710

Scriven, MichaelThe Evaluation of College Teaching. ProfessionalDevelopment Occasional Paper Number 3.National Council of States on Inservice Education,Syracuse, N.Y.Department of Education, Washington, D.C.(80122p.National Dissemination Center, Syracuse UniversitySchool of Education, 123 Huntington Hall, Syracuse,

NY 13210.

MF01 Plus Postage. PC Not Available from EDRS.*College Faculty: *College Instruction; *EvaluationCriteria: *Evaluation Methods: Faculty Development:*Faculty Evaluation: Grading: Higher Education:Student Evaluation of Teacher Performance: *TeacherEffectiveness: Testing

ABSTRACTPerspectives on the evaluation of college teaching

are considered. Some of the conditions that are needed generallyinclude the following: the existence of an evaluatiOn system that has

a process for evaluating research and service, a clear definition of

the difference between the evaluation of worth and the evaluation ofmerit, and a separate support system to assist faculty in improvementefforts. The following are not recommended for evaluating teaching:classroom visits by colleagues, focusing only on measuring the amountlearned, and checking on the relative number of further courses inthe same subject matter in which the instructor's former studentsenroll. It is suggested that the key component in the evaluationprocess is the student questionnaire, which should allow for anoverall iudgment of the merit of the instructor as an instructor.

Other possible questions concern ethical or professional obligationsof the instructor and components of instruction, such as ratings ofthe text, quizzes, and the grading system. It is suggested thatinstructional materials, tests, student grading, and the student's

role in grading be examined in terms of comprehensiveness,

correctness, and ency. Other ways that teaching can be evaluated

include assessmen actual learning gains and facultyself-development and self-improvement efforts. The use of facultyevaluation in a faculty development process, integrating anddisseminating the results of faculty evaluations, and the keyethical/legal constraints on data gathering are also addressed.

(SW1

***********************************************************************Reproductions supplied by EDRS are the best that can be made

from the original document.***********************************************************************

a

rofessionalevelopment

OccasionalPaper

"PERMISSION TO REPRODUCE THISMATERIAL IN MICROFICHE ONLYHAS BEEN GRANTED BY

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC).-

The Evaluationof

College Teaching

Michael Scriven

NationalCouncilof States onInser dice Education

The Evaluationof

College Teaching

NATIONAL COUNCIL OF STATESON INSERVICE EDUCATION

COUNCIL CHAIRSRonald Hall ChairmanLouis Morelli Past Chairman

PROJECT OFFICIALSJames F. Collins Project.Director

'Syrac.ise UniversityCarol Lewis Assistant DirectorR. Neal Appleby Administrative

Al:vivant. Syracuse UniversityTheodore Andrews Program

Coordinator. Newsletter EditorHelen Hartle ... Interstate Certification

ProjectDiane Jones .... NCS!E Project Officer.

Teacher CorpsJames Steffensen Associate Director,

Teacher Corps Development BranchJohn Minor ... Director. Teacher Corps

StateAlabamaAlaskaArizonaArkansasCalifornia

ColoradoConnecticutDistrict ColumbiaDela wa teFloridaGeorgiaHawaiiIdahoIllinoisIndianaIowaKansasKentuckyMaineMarylandMassachusettsMichiganMinnesotaMississippiMissouri

MEMBERSOrganization RepresentativeAACTE David ImigAASA Robert SnyderAFT Pat WeilerATE Robert StevensonNASDTEC James Nohara

NEA . Roy EdelfeltTeacher Centers Allen Schmieder

RepresentativeBilly Campbell

J. Kelly TonsmeireFred Sughrue

Austin HannerRichard MastainWilliams Webster

M.D. SpurlinSigmund Abeles

Joan BrownErvin Marsh

Louis Morelli'William Leach

Eugene YamamotoA.D. Luke

William Gillies'Ronald A. BoydOrrin NearhoofMary Martin

Shirley DuffDavid C. TiltonIrving Herrick

Thomas O'ConnorPaula Brictson'

Judith WainRussell Crider

P.J. Newell

StateMontanaNebraskaNevadaNew HampshireNew JerseyNew MexicoNcw YorkNorth CarolinaNorth DakotaOhioOklahomaOregonPennsylvaniaRhode IslandSouth DakotaTennessee

N r mo ntVirginiaWashingtonWest VirginiaWisconsinWyoming

RepresentativeJohn Voorhis

Gerald Sughroue*LaMar LeFevre

Gary BarkerDaniel KunzAlan Morgan

Vincent Gazzetta'Elizabeth Koontz

Lowell JensenNancy Eberhart

John FolksRichard S. Jones

Ronald HallEdward L. Dainbruch

James HansenJohn Gaines

James L. KiddLaMar AllredHenry Bissex

Everett B. HowertonLillian Cady

Nicholas Hobar'Robert Skeway

Thomas L. Morris

'Member of the Executive Board

4

The Evaluationof

College Teaching

,tlichael ScrivenDirector. Evaluation his:fluteUniversity of San Francisco

NationalCouncil of States onInser vice Education

The preparation of this publication was supported in part by funds made available

by the United States Department of Education through Teacher Corps to Syracuse

University Office of Sponsored Programs. National Council of States on Inservice

Education Project. However, the opinions expressed herein are those of the author

and are not to be considered ,as representing the opinions of the United States

Government. the National Council of States on Inservice Education, any of the States

which are members of the Council or of Syracuse University School of Education.

distributed hi.:National Dissemination CenterSyracuse University School of Education

123 Huntington HallSyracuse. N.Y. 13210

Library of Congress Cataloging in Publication Data

Scriven. Michael.The evaluation of college teaching.(Professional development occasional paper: 3)

Includes bibliographical references.1. College teachers, Rating of United States.

1. National Council of States on Inservice Education. 11. Title. Ill. Series.

LB2333.S37 378'.I222

I\

80-26644

The EvaluationOf

College Teachingtfichael Seri% en

I. IntroductionI he normal procedures for evaluating

college teaching are so shoddy at the intel-lectual and the practical level. that it is

hardly surprising that teaching is rarely re-warded in an appropriate way. That theseprocedures arc intellectually slope} is a re-flection on both the scholars and the admin-istrators of the academy, and that their me-thods of application are also slipshod tellsus something more about the trues aloe sys-tem of the academy which is not to itscredit. Had colleges spent one percent of thetime on this problem that they have spenton internally-funded research in the sciencesor the humanities, we would now haves'omething to be proud of instead of aSource of shame. What follows has only thestatus of preliminary notes about the prob-lems and their solution and easy ways toimprove it greatly. I have tried to indicatethe reasons why even these preliminary con-siderations clearly entail the rejection of es-sentially all existing systems.

11. Contextual PrerequisitesIt is not only pointless but improper to

proceed with a system for evaluating teach-ing that is supposed to operate well and eth-ically regardless of the administrative andlegal context. The following are some of theconditions that must be present in thatcontext.

A. The evaluation of teaching must bepart of a system which also has an appro-priate proceSx for evaluating research andservice, and a specific commitment to theirrelative weighting. Otherwise one is playingin a game for which only a few of the ruleshave been stated. It is notable that, for ex-ample, the University of Texas' system andthe University of California at Berkeley'ssystem have no such commitment to therelative worth of teaching. The fact thatUCB requiresand enforces the require-mentthat data on teaching, including stu-dent evaluations, must be included in a

dossier before it will be considered for per-sonnel decisions seems to show a valuing ofteaching. It does not. The data may showthat the teacher is a bad teacher, vet the"data requirement" does not result in a pemalty: and if it showsi he is a good teacher.no determinate benefit results. Specific rela-tive weightings of relevant criteria of fac-ulty assessment is the only non-vacuousprocedure. and not incidentally the onlyequitable one. It is hardly surprising that inthe two systems mentioned (and most oth-ers) the standards \ ary by campus anddepartment.

B. You must have a system of adminis-trator evaluation in place in order to avoidthe entirely justifiable resistance of the"serfs" to being evaluated by the "folk in thecastle" who are above such things them-selves. Administrator evaluation, as it is

commonly done at universities and colleges.is a truly remarkable phenomenon in that itsucceeds in being even worse than the evalu-ation of faculty, and it occurs in even fewercases.

C. The institution must have clearly un-derstood and defined the difference betweenthe evaluation of worth and the evaluationof merit, with respect to the evaluation ofteaching in particular. (The distinction alsoapplies to the evaluation of research andservice.) There are a half dozen areas wherethe impact of this distinction is crucial, butI'll just mention two. Suppose that you havea teacher whose teaching is superlative byevery reasonable standard, and who is alsoproducing substantial quantities of abso-lutely first-rate research, while renderingimpressive service to the profession, thecampus and the community. If you are not,as an institution, absolutely clear that inspite of all this you may have to deny tenureto the teacher, you do not understand thedifference between worth and merit. Thedecision whether to make an initial ap-

Professional Developmentpointment, the decision on granting tenure,and the decision on pushing for earlyretirement by the various means availablefor that, should all depend upon the north

a faculty member to the institutionand not merely upon the faculty member'sprofessional merit. The worth to the institu-tion is essentially connected with such issuesas the income generated by the student en-rollment that the instructor produces, thespecial services to the institution's generalmission that are provided by the instructor(e.g. a uniquely talented Latin scholar orteacher at a Jesuit institution, a womanmathematician in an otherwise all-male de-partment that is anxious to recruit womenas mathematics majors etc). Neither privatenor public institutions can today afford tobe awarding tenure on merit alone as if theenrollments were irrelevant, but many ofthem are locked into a system of faculty eval-uation which makes it impossible to deviatefrom merit considerations even in extremecases (except at the appointment decision).If worth is to be given any weighting at all,as it should be, then the exact size and limitof this weighting must be as carefully de-fined as possible in order that one does notget into obvious abuses such as firing fac-ulty members whose political activitieslead to the loss of large donations by alumsor legislatures. the sort of abuse which re-sults from thinking that "worth" means fi-nancial worth. it's not that worth doesn'thave anything to do with financial worth (asin the enrollment example), it's just that italso has something to do with the ;ntegrityof the mission of the school and there is noworth left in an institution that abandonsappropriate professional standards inhigher education in order to achieve finan-cial worth.

D. There should be an independent sup-, pori system of some kind available to fac-

ulty ,o assist them in the effort to improve,so that the system of faculty evaluation isnot merely punitive. and not seen as merelypunitive (in which case it is unlikely to heviable). The system of independent supportfor improvement must be independent ofthe department chairs. the deans and the

peers in order to avoid the disincentive touse it that would result if instructors knowthat administrators will know or can find

out when it is being used by a particular in-structor. Hence the consultations must be

absolutely confidential as well as profes-sionally based. Finding somebody who canprovide the appropriate kind of assistance isvery difficult because there aren't many whomight he said to be qualified in the firstplace, and a large slice of that group is muchtoo inclined to think that there is one tritesolution to the problem of how best toteach. The cost of providing this supportsystem is low. One professional plus onesecretary/assistant per 10.000 students cando this and also a number of other jobs thatshould be done on any campus such askeeping up with the research and practice inthe field of faculty evaluation and teachingresearch, maintaining a small consulting li-brary for faculty, improving the studentquestionnaire and managing its administra-tion and data synthesis phases. This in-volves research-reading competence: but thehelping role that works with the primadonna type of faculty is not something thatprima donna types of teaching researchersare necessarily used to, so a nice balance ofskills is required. It would be foolhardy tomake appointments to this job for morethan a trial period at first.

E. The background system must be onewhich has consistent and apps opriate prac-tice's, not just consistent and appropriaterhetoric. A typical example of how not toachieve consistent practice is to have a sys-tem in which the quality of teaching is saidto be important (which involves, amongstother things, the quality of the assessmentof student work) while in fact departmentsdre issued new positions and replacementsfor retiringor departing -old appoint-ments on the basis of enrollments. Whilethis is idmirable in the sense that it reflectsan attention to a reasonable considerationthat a decade or two ago was almost totallydisregarded, it is absurd in that in manycontexts it amounts to rewarding depart-ments for -arr~st other unattractivethings --easy griliggIg, which is inconsistentwith the rhetoric of respect for high stand-ards. Another example is the use of a stu-dent questionnaire which attends to the ex-tent to which instructors get to know thenames and personalities of individual stu-dents, provide ample office hours etc.. all of

The Evaluation of College Teaching

which arc results easily achieved fur smallclasses but not fur large classes; and then tocomplain about declining enrollments.which this approach reinforces. Consistentsystems arc hard to set'uti,. but the inconsis-tencies in systems are extremely expensive.

F, A reasonable modUs vivendi with stu-dent government must he worked out formutual benefit from student evaluation.(See sectionVI of this paper.)

III. The Definition of Good TeachingThe best teaching is not that which pro-

duces the must learning. No definitions inthe literature avoid obvious counter-examples, The following definition avoidsthe so-far-identified counter-egtinples, butpossibly has some of its owh. The idealteacher is one who has the maximal possibleinfluence towards beneficial learning on thepart of his or her students, subject to threeconditions; I) the teaching process used isethical, 2) the curriculum coverage and pro-cess is consistent with what has been prom-ised. 3) the teaching process and its foresee-able effects are consistent with appropriateinstitutional and professional goals andobligations.

Perhaps we can best illustrate what is in-tended by the three qualifications with someexamples of what they exclude. I) Unethicalprocesses including not only cruel and un-usual punishment but also those processeswhich are completely non-generalizable e.g.putting so much pressure on the studentsthat they abandon their work for otherclasses in order to do the work for this one.It is often argued on virak a priori groundsthat the use of a reward system such as atoken economy or grades is unethical; infact, the failure to use such a "reward sys-tem" is both unethical and unprofessional.It is unethical because it fails to inform thestudents about their progress or competenceand/or it fails to show that the institutioncares about quality work as opposed to min-imally competent work etc.: and it is unpro-fessional because it keeps important infor-mation about the student's progress fromthe student and the student's advisors, par-ents and potential employers. It alsohappens to provide a legitimate incentivefor many students, and not to use it is thuspoor pedagogy unless you have direct evi-

9

Bence for a better approach. (Grading caneven he arranged so as to make inter-student competitiveness impossible, whileretaining the feedback required in strivingfor excellence.) So, even if grading reducedlearning, it would have to ex done and doneproperly. But --properly managed itshould increase learning.

2) Certain contracts are made by instruc-tors and institutions, both explicitly andimplicitly, to which they frequently do notadhere. Such contracts are involved inpromises as to what a course will cover.made either in a catalogue, the departmen-tal handouts, the faculty handbook, the un-icn contract, the class handouts or in thelanguage of the opening presentation inclass. Apart from the simple sin of mislead-ing advertising, much of the hierarchicalstructure of sequential curricula has beenmade laughable by the failure of instructorsto adhere to these standards. Maximizinglearning cannot he given precedence overthese obligations.

3) It would usually be appropriate, if themaximization of learning were the only ob-ligation of an instructor, to adjust the levelof instruction to the class average so thatmore people would he able to benefit fromit. But there are institutional and profes-sional commitments that transcend thiscommitment.' For example. if one is facedwith a class of medical students who willgraduate at the end of this year. and if one'sobligation is to instruct them in professionalprocedures that they will be practicing uponan 'unsuspecting public a year from now.and if it is clear that most of them are so faroff the pace that no instruction within thetime available can get those students up toa level of competence, then it is profession-ally obligatory to focus the instruction uponthose few who can be brought up to the ap-propriate standard and flunk the rest if theydon't make it. Other obligations that areimportant and must be considered veryseriously here include the problems of pro-viding compensatory justice for minoritiesand women (which may mean focusingmore effort there than shows up in maxim-izing classroom learning), and providingknowledge that is going to be expected bythe instructor of a higher level course etc.

3

Professional DevelopmentSome assorted notes that hear on the

preceding points include the following.First, the "learning" referred to in the defi-nition can certainly cover attitudes, in therare cases where it is possible to justify theclaim that teaching a certain attitude is anappropriate part of an instructor's obliga-tions. Teaching the scientific attitude mightbe a case in point, as might be teaching themotivation to learn. It mast also be remem-bered that teaching can often be done andoften is done to a very important extentoutside the classroom; hence evaluating itmust involve looking for and at this out-of-classroom teaching.

Perhaps the most important feature ofthis definition of teaching is that it does notidentify good teaching with the productionof good learning. Considerations such asthose mentioned preempt such a flakeequation (one that I rather fancied in myearlier thinking atrout this subject): butthere is yet .another type of example thatshould be mentioned. If you have a numberof students in the class who, for various rea-sons not connected with one's own perfor-mance, are not working hard enough to keepup with the class, one should not be down-graded as a teacher for the failure of thesestudents to learn. A teacher's task is only toprovide the best possible environment, notto guarantee that the results will be effectiveno matter how little effort is made by thestudents. It follows that one must never useon a student questionnaire the question:"How much did you learn from this class(that you think was valuable to yOtij?" orcognates of that question. One must insteaduse questions such as "How well do youthink the instructor taught the course?" It ispartly for this reason that the answers tostudent questionnaires are not to be re-garded as intrinsically inferior to studies ofthe learning gains of students in class. Theyare more valid in one respect, in that theyaddress the correct question.

A second incorrect definition of goodteaching which has some supporters in-volves the identification of teaching with thetransfer of learning from the teacher to thestudent. But of course -as has often beenpointed outinspiring the student to seeklearning elsewhere may be the best ap-proach to maximizing learning even in the

4

short run and more likely in the long run.

IV. now Not to Evaluate Teaching ,

A. It will he clear from the preceding dis-cussion that any way of evaluating teachingt hat .tintp/rr consists in measuring theamount learned is oversimplified and can he

extremely unfair to some teachers. It's alsoclear that evaluating a teacher's worth onlyor merit only will on occasions be entirelyinappropriate. Let us turn quicklynumber of other errors that have 0-widely incorporated into quasi-systems forevaluating teachers.

B. Classroom Visits. Using classroomvisits by colleagues (or administrators or"experts") to evaluate teaching is not justincorrect, it is a disgrace. Some of the rea-sons for this conclusion follow. First, thevisit itself alters the teaching, so that the vis-itor is not looking at a random sample. Sec-ond, the number of visits is too small to bean accurate sample from which to general-lie, even if it were a random sample, Third,the visitor is not devoid of independent per-sonal prejudices in favor of or against theteacher, arising from the fact that the visitoris normally an administrator or colleague ofthe teacher and in his/ her other role is in-volved in adversary proceedings, alliances,etc. with the teachers. Fourth, even ifnone of the preceding.conditions make thewhole affair ridiculous, which each of themdoes independently, there is nothing thatcould he observed in the classroom (apartfrom the most bizarre special cases) whichcan possibly be used as a basis for an infer-ence about the merit of the teaching. Thatthis is so follows inexorably from the resultsof the enormous number of studies on styleresearch that have been done and sumthar-.lied on various occasions, the sum total ofthese resulting in the conclusion that thereare/ no style indicators that can he said toreliably correlate with learning now or inthe future by the students in the classroom.Fifth, regardless of the fact that no obsera-Lions of teaching style can legitimately heused as a basis for inference to the merit ofthe teaching. the visitor normally heliesesthe contrary. This is often because the is-

nor has his or her own .preferences as to acertain style. or has many years of expe-

1 0


nowt: in teaching this same ty pe of A:outse.and consequently belies e' that not doing itthis way Or in ,ine'or two other ways thatarc approseLl is doing it badly. I lieseiprejudices are totally without loundation.to the hest of our knowledge. and shouldnot he allowed to come into the es Ammonof teaching at all. (Among the rare esccp-lions are the possibility that what theteacher is saying is known to he lake 13% thevisitor, and so grossly false as to constitutean impossibly .chicle for teaching the truth.that the visitor observes racist or sexist orother immoral practices by the teacher, ortam the visitor ohsers es a total lack of class-room discipline to a degree which cannotpossibly he reconciled with the continuanceof a learning process of any kind, sincenone of these nets has ever been recordedon any of the classroom visiting sheets ofthe many thousands that I hase either in-spected directly or of which I !lase seensummaries, this cannot he taken seriously asa reason for making classroom visits part ofteacher evaluation.)

Ultimately, the problem about the visitoris the lack of similarity between the visitor'shead and the student's bead. They arc gen-erally separated by several decades in theirlearning maturity, thus they may have sub-stantially different vocabularies and con-ceptual repertoires, and they certainly lackmuch cultural similarity; because of this. thevisitor is not a good indicator of how muchlearning is going on in the heads of the stu-dents. Since the secondary indicators ofteaching style (i.e. everything besides em-pathy) turn out to he invalid, this leaves thevisitor--or should leave the visitor-- -up theproverbial gum tree. I need hardly stress thefact that in spite of these staggering objec-tions, the method of classroom visitation isthe universal method whereby teachers inthe elementary and high school are evalu-ated, and is--depressingly enoughbeingquite steadily implemented at the post-secondary level, on the grounds that it rep-resents an improvement. No variation ofthis is of any value whatsoever; visits by ex-perts are no better, visits by peers are nobetter than visits by administrators. with re-spect to the personnel evaluation protest.There is a very limited role in which visits by.a consultant are defensible in the effort to

11

pi Os ide, help to impose teaching dwadi Ai ii wn that dn. neon une o t CPA .1///tug e"illt I his is supply because the con-sultant mt he able to suggest someoptions not opiitmi/ alternatk es. but iustpmithh ones. Hut it's usually unneeessarto has e the (costly) classroom visit a taperecording or 'nen a serhal report by theteacher plus the student es aluation forms ismore than enough basis for consulting rec-ommendations in most eases.

C. I caching is usually evaluated withoutany serious attempt at es :dilating thequality of the content of the course. Thisis the "methods-madness" that has madesehoids of education a latighingstock in theintellectual community and increasinglyin the total community. for many years.From what we lime said about the defini-tion of good teaching, it is clear that onecannot make one's judgment of teachingmerit entirely on the basis of the content ofwhat is learned by the students. but to do sois much better than making the judgmenton the basis of merely inspecting what ispresented to students. Both should be con-,idered, but with an emphasis upon studentperforroance. Here is the one place wherepeer evaluation of a limited kind isappropriate -but evaluation of materialsnot process and even here it is better touse people from another institution (but inthe same subject-matter area), eliminatingcosts by trading services in kind with thatinstitution. Such an arrangement -with itssocial pressure tends to do much more toimprove standards than in-house materialsevaluation does anyway, apart from in-creased validity.

D. Another popularly acclaimed thoughinfrequently employed measure of teachingsuccess is checking on the relative numberof further courses in the same subject matterin which "graduates-of the instructor beingevaluated enroll. This is an unethical indica-tor and its use cannot be countenanced bycentral administration because it is non-generalizable. That is. one can only scorepoints on this dimension by stealing, buyingor seducing students from other depart-ments. It's exactly like the process of gettingmore work out of the students by having

5

Professional Development

them go. e up On their homework for othercourses. It's a great indicator if you treat thewelfare of the discipline ai the .stimimemhomun, rather than the welfare of the insti-tution or of higher education or of the hu-manities or of the society; or of the student(with a minor qualification).

E. Most student questionnaires are im-proper bases for faculty evaluation. becausethey generally involve ratings on'.4tyle orratings ttn non-generalizable indexes etcBut the way in which the data from them issynthesized is also a problem. If you onlyuse mean ratings, then you overlook theimportant case of the instructors who aretremendously successful with a sub-groupof the class. Although their mean score maybe no different from that of another instruc-tor who receives weak ratings from every-body in the class, their teaching potential isobviously different and a sensitive adminis-tration should be making some provisionsto tap such a promising source of inspira-tion in a more appropriate situation thanthe particular class that resulted in'a lowoverall score. Or the consultant may be ableto work out a way to generalize from thispromising basis to the other students. Sim-ilarly, averaging the means from coursestaught at different levels may be unfair togood teachers of introductory (or graduate)courses.

There are a number of other ways not todo the job that are widely used, but let'sturn our attention to at least a preliminarysketch of how the job might be doneproperly.V. How Teaching Should Be Evaluated

A. The key component in the evaluationprocess is the student questionnaire. but thepiece of paper itself is only part of the story.The support 'system has got to provide meth-ods for administering this that are bullet-proof against complaints about the possibil-ity of selective return and prompting. Agood straightforward approach is to havethe central administration staff (a cheaperfallback system is using the, departmentalsecretaries) take the questionnaires out toeach class, have the professor leave theroom for a few minutes, provide a brief ex-planation of the process and how the resultsare to be used (possibly in writing). encour-age questions. and pass out questionnaires

6

to be filled out by every met»ber of theclass. One should get a 99 percent returnrate from those present.. Where knowingthat the questionnaire will he distributed onthat date is an incentive, the date fordistribution should be set in advance if onewants most of those enrolled in the class toattend. (This does weaken one's defensesagainst certain types of preparation of thestudents by the professor, but this can hecontrolled by asking about it on the form soit is a less worrying problem than low atten-dance rates.) "Absentee ballots" are not agood idea, logistically or for inferentialpurposes. One can get that kind of responserate, and tolerance from the faculty fortaking time out of their classes, if the wholeprocess takes less than about 5 minutes.This is the first of two compelling reasonsfor using a veer Elton questionnaire.

The second is that there is not the slight-est justification for using a long one. Theusual questionnaires with twenty to seventy,items on them are fishing expeditions of anentirely improper kind. Unless you can sayof a particular question that it is demon-strable that an answer of a particular kindindicatils merit or demerit in "(welting. thenit has no place on the personnel evaluationquestionnaire. Other and more detailedquestions may be used in a questionnairedesigned by the instructor with the assist-ance of the consultant (or perhaps the otherway around), when an instructor wants to

.get specific feedback on some specific effortor style venture in which he or she is inter-ested. But no such commitments can be legit-imated for general evaluative purposes andno such questions should ever be on a formwhich is seen by anybody except the instruc-tors and consultants of their choice. Ifplaced on a form along with legitimatequeStions they will be likely to biaS the re-sponse of somebody who favors or disfa-vors the style that they uncover. and suchbiases are illegitimate. (They would be cer-tain to seriously affect the legitimacy of anypersonnel decisions that were appealed incourt. for example; but the ethical point ismore serious than that.)

Since one cannot help but be worried bythe contamination of instructor ratings byirrelevant personal lactors and by factors

k\_connected with li .-or dislike of the course\

12

The Evaluation of College Tzaching

itself, one should make some effort to si-phon off those other considerations. A sim-ple way is to stress tho contrast betweenthese considerations in the introduction tothe basic judgmental question on the ques-tionnaire, i.e. between liking atal dislikingthe instructor as a person and thinking wellof him as an instructor; and between likingand -disliking the course content (of a re-quired course) and thinking that the instruc-tor did or did not do a good job of teachingit. Another approach to siphoning wouldconsist in asking specifically for I) evalu-ation of the instructor as a person, and 2)of the course itself, then asking for 3) anevaluation of the job done by the instructorin teaching this course. The course evalua-tion couid then he torn off the form andsent to the chair and/or dean for courseevaluation; the personal evaluation could hetorn off and thrown away or given toinstructor if she or he requested it; and dnewould then use the rest of the form for per-sonnel evaluation.

The crucial question in the rest of theform is a request for the overall judgment ofthe merit of the instructor as an instructor.There are some other possible questions onsuch a form. of a kind not often encoun-tered. They should come before the requestsfor an overall rating since they are likely todepress it. (and increase its validity) which isfine because the main problem with the usualresults is that the scores are too high toallow adequate headroom in which excii-tionally good teaching can distinguish itself.These other (possible) questions are of twokinds. The first concerns matters that can bedescribed as ethical or professional obliga-tions of the instructor and can he phrased ineither a positive or a negat;ve way, or --better alternately. They may concern suchmatters as the match between the pre-announced content of the course and the ac-tual content: the match between the contentof the course and the content or the exami-nations: the extent to, which the reasons forgrades were explicitly and adequately justi-fied, the extent to which the possibility ofappealing a grade was explained: the fre-quency with which scheduled class and of-fice hours were met: the use of racist or sex-ist or other bigoted remarks or materialsand so on. This is the so-called "BlackMarks list-1w hen phrased negatively) and it

is surprising the extent to which it remindsinstructors as well as students of the min-imum professional obligations of the in-structor. obligations that can rather easilybe discharged and which one constantlyfinds are an underlying cause of dissatisfac-tion and had holistic ratings.

The second kind of legitimate questionconsists in a listing 01:the components of in-struction for independent "micro-assessment" i.e. "please rate the catalog des-cription, text quizzes grading system. classhand outs sections or labs or fieldworketc.". I now feel that a form with thesecomponent-evaluation questions PIP'S themacro-question, plus an optional requestfor suggested improvements in the form andprocess of administering (using it) is the bestbasic form. though one can reduce the effortby using only the macro-evaluation, goingto component analysis only when the in-structor requests it or is seriously and regu-larly below the mean for comparablecourses. At that point. where remedial ac-tion is necessary the components evaluationis an obvious help. When dcrucial person-nel decision has to he made. I think that the"prolessionality" questionnaire provides auseful and valid supplement to informationfrom the other two. Such data should he ac-companied by comments on it from theinstructor.

B. There must he careful examination ofthe quality and professionality of contentand process; the three qualities here are currency. correctness. and comprehensiveness.Ratings must be made on the basis of asample of I) the materials provided. 2) thetexts required and recommended. 3' the ex-ams. 4) the term paper topics. 5) the studentperformances on the preceding. 6) thestructor's performance in grading studentwork. 7) the instructor's performance in jus-tifying the g'ade-and providing other help-ful feedback'

r

C. Actual, learning, gains. These arc onlygoing to be useful if one can in advancespecify and justify the comparisons,that willbe used. By themselves. they have no legiti-mate interpretation except. perhaps. wherethey are negative. In cases of multiplytaught sections for an instructional course.the comparison between sections can be ex-tremely useful. In cases where there are

137

Professional Developmentnational norms they can sometimes he use-ful. And with respect to instructional im-provcment, one can run a comparisonagainst one's own,previous performanceand 11 -over whether slight or large changesin a; ..ych or text. turn out to have signifi-cant results.

The . main aim of the comparisons in-volved is to try to determine what could rea-sonably have been taught to the students:it's always fairly straightforward to find outwhat was actually taught, but that is evalua-tively useless without some sense of whatwas possible. If allocation of students be-tween the afternoon sections of a large introclass is random, then one can indeed dis-cover something quite important about this.if one is pftpared to look for patterns thatextend across a couple of years. Shorter pe-riods of study are likely to yield results dueto unrecognized idiosyncracies of the par-ticular classroom, time, groups present etc.

D. Professional Development Dossier(Self-evaluation and Self-improvement in-formation.) Although the results of self-improvement should show up eventually inimproved performance on the above scales.it's worth including it directly since doing soencourages faculty efforts in this direction.hence speed up the improvement; and it alsoimproves the validity of decisions that haveto be made under a time constraint. Underthis heading we expect the submission of arationale for each course's methods andcoverage (where the latter is the instructor'soption), evidence about professional devel-opment activity such as readings, courses.workshops, consulting aimed to improveteaching, and most importantly. a descrip-tion of planned and performed experimentson the individual instructor's own teachingapproach. A basic list of courses taught, en-vironments, and grades; and of committees(instructional, departmental, school and'College) should be included here preferably'as hard -copy output from the institutionaldata base.

The preceding list of four dimen,iiinscovers the four crucial types of data thoishould be gathered. There are ways in whichthese can be ramified usefully and withoutsignificant extra cost. For example. feed-back from the students should be supple-

8

mented by feedback from teaching assis-tants or teaching aides where these areiimployed, and where confidentiality of theresponses can be preserved (as is usually thecase). I have sometimes previously includeda separate ethical component or a "justicedimension" (as I then called it). but I havehere amalgamated it into "professional con-tent and process."

For faculty self-development purposes,the questionnaire can of cour:;e be ex-panded to include an open-end question inwhich there is a call for identifying particu-larly good and particularly bad features ofthe instruction, but I usually keep these re-sponses down in volume (because it's sohard to simplify large numbers of these re-sponses) by requesting that they be pro-vided only when either the top, or the bot-tom two scoring categories is used. (i.e. onlywhen an A or a D or an F rating of the in-structor is given by the student). This alsoimproves their utility, because it avoids theprobability of interpreting e.g. favorable re-sponses with long lists of criticismsattached.

In addition to the ratings of the instructordone by students during the class, there is

another type of student rating that is excep-tionally valuable, at least when done spo-radically, and that is the "exit" interviewdone at graduation. It falls midway betweenthe in-class rating and the alum rating; andis'vastly better than the alum rating becausethe rate of return can be around 100 percentinstead of around 30 percent (rendering thelatter essentially useless for personnel deci-sions), memories are more reliable at thatpoint, it refers to a more recent version ofthe courses, and causal inferences are morelikely to be valid at that point. Exit inter-views.have a quite disproportionately largeproportion

anythe students, but who do not

ashow up in ny other search.

Notice that the approach outlined heredoes a great deal to protect faculty from ,ivosources of injustice that are partici rlypervasive in current systems. The first is thekind of injustice that makes it impossiblefor the teacher of a generally unpopular butabsolutely essential prerequisite course e.g.calculus for architect students, to scorereally well compared to the norms against

14

The Evaluation of College Teac.',ing

which she or he will usually he judged. Wemake allowances in the above system for atleast three ways in which such an instructorwill be able to emerge as first-rate. Theother kind of injustice relates to the individ-uals who put an enormous amount of workinto developing a course, keeping it up todate, and making it as effective a teachingdevice as they possibly can, at the expenseof lots of happy friendly humanistic inter-changes and socializing. When an evalua-tion system is issued which does not pay agreat deal of attention to the content of thecourse, and which throws in irrelevant re-quests to have students rate the course on"touchy-feely" dimensions, these efforts arelargely unrewarded.

Although peers one kind of expertmust be used for evaluation content, theyare usually not competent to handle all thatfalls under B. although they should he. For ..

there is one kind of usually hidden behai,'iorthat should be picked up by an evaluationsystem, and which does require a differentkind of expertise to assess, and this is thetechnical competence of the instructor inpedagogy e.g. in writing reasonable exami-nation questions that are unambiguOus, in-volve adequate coverage of the intendedtarget area, are not overcued. etc. This kindof quality is rather easily picked up bysomebody with good skills in test construc-tion. and such a person should routinely re-view samples of the tests of those instructorsthat are up for (or a year from) particularlyimportant personnel decisions like tenure.

Another point.4t which such experts havesome relevance ctincerns the way in whichexams are graded or marked. There are pro-fessionally required standards. here. withwhich virtually no faculty members at uni-

\ versifies have the sliL.Iluest familiarity. As a'remedy for this, administrators shouldrequest that as a normal part of the pro-

cess of talking about self improvement,the instructor fill out a form indicating howpapers are in fact graded. The kind of grad-ing that is legitimate involves at least thefollowing requirements: the papers aregraded "blind" e.g. by having the studentswrite their names inside the back cover of ablue book or in any of a dozen other morefoolproof ways: the papers are graded ques-

tion by question, not paper by paper (this isto avoid the known large halo.effect that re-sults from having read a good, or bad firstquestion by the same author just beforereading the second question). When one hasgraded all the nth questions in a given set ofpapers one must then go back and grade thefirst six to ten papers again in order to seewhether one's grading standards haveslipped, upwards or downwards (as theyoften do); the pile of papers should beshuffled after each question is graded sothat different students stand the brunt ofone's fatigue (or initial optimism) for differ-ent questions; and se on. With multiplechoice exams,. rn my of the preceding re-quirements are otiose; but the requirementfor technical proficiency in posing them be-comes enormously more important and issomething that is beyond the competence ofnearly all academic instructors at t' e mo-ment. (It's illuminating to talk to people atthe Educational Testing Service about thedifficulty of finding individuals who canwrite good multiple choice items. I think onecan tie in these practices with those requiredby the "Black Marks List" and call them theProfessionality Dimension (E). But they canalso he left buried in A and B.

Much of the preceding refers to summa -tive evaluation for personnel decisions. Forthat purpose, it is quite important to giveout the class questionnaires at about thesame time in the term for all instructors.What is the best time? Faculty at USF arguethat the tenth through the twelfth weeks ofclass (semester system) would be aboutright; not too near to the beginning to give areading on the basis of inadequate expe-rience, not too near to the end so as to inter-fere with the intensive review period for theexaminations and the occasional drop in at-tendance while students stay at home. Get-ting all the classes visited within this period,is logistically feasible for a relatively smallcampus, but would involve substantial diffi-culties in a large one. But validity considera-tions suggest that administration of question-naires after final grades have been seen ispreferable and this also increases headroomon the distribution. See section VI for moreon this approach.

One could turn to the use of senior de-partmental secretaries to distribute and col -

y5 9

Professional Developmentlect questionnaires. but there is still acredibility problem about that. It is essen-tial not 'to use graduate students or stu-dents from the same campus. for similar rea-sons. The system of having each class electa student who will do the job and take thematerials over to the central administrativeoffices has worked quite well at some insti-tutions. But it does not provide a personwho can answer questions and exhibit in-side knowlecipv: about the importance of theprocess and its results. Under no circum-stances can one go to mail ballots or "dropit in the nearest collection box" ballots,. be-cause the response rate deteriorates se-riously, and it's hard to justify validity withunder 85% return rates especially if varia-ble. One must select the best procedure interms of the accessible resources at a partic-ular site.

Similarly, although it is rather easy to useoptically scannable cards. this gets one intothe business of providing the appropriatemarking pencils. which is to be avoided atall costs. It's 'better to use straightforwardforms and have them keypunched; it turnsout that this approachrather surpris-inglyis less expensivl. The computer pro-gram for combining the data on the type ofquestions that we have been talking about is

of course.extremely simple and can providea variety of interesting readouts. After in-vestigating variations in small and largeclasses, required and optional classes, upperand lower division classes and differencesbetween fields, one usually finds that mostof these differences are not sufficiently im-portant to be worth reporting separately, al-though one can run them through the ma-chine for a check every time the system, is

operating. One should also have the compu-ter automatically flag performances that arc

either standard deviation above or oneand two standard deviations below the av-

.erages, not because a standard deviationmeans something in terms of traditionalsign'5cance with the kind of skewed distri-bution you get here, but because it's a con-venient and quite appropriate flag. If onedoes not use a set of preliminary questions

on components or professionality. one willprobably have to set the upwards flag athalf or two-thirds of a standard deviation inorder :o get any success ',,-;CS at all.

10

How often should this process be run'? Inmy view, every second course or every thirdcourse given. stratified by upper and lowerdivision and graduate categories, is a goodcompromise. Moreover, it cuts the person-nel and time requirements for distributingand collecting the forms by a half to two-thirds, besides giving faculty a chance to dosomething about an unsatisfactory readoutbefore the next evalUation is upon them.

There are certain by-products of this ac-tivity that are of some, interest to ad-ministrators. For example. one can get avery good readout on the actual teachingload of the faculty. in terms of numbers ofclasses and in terms of numbers of live bod-ies; it turns out that there are sometimesa startling number 4"phantom" classes i.e.classes that are not in fact meeting althoughthey haven't been cancelled. usually becausethere weren't enough people to jtk.:ify ascheduled class meeting and the instructordidn't want to convey to the chair or deanthe fact that her or his load had partlyevaporated. A comparison of the number ofgrades awarded with the number present atthe time of the administration's question-naire will give an attendance rate ratiowhich is also something that bears watchingfor face;:, members who are trying to im-prove their performance. The number pres-ent for the evaluation should not be printedout on the summary sheets without a refer-ence to the number absent, if the registrar'scomputer can be brought to produce a cur-rent figure on that, Printing out some indexof bi-modal distributions is alsq easy, butshould in general be reserved for the situa-tion where someone is looking for help.

VI. How to Use Faculty Evaluation in aFaculty Development ProcessMost of the preceding refers fairly specif-

ically to summative evaluation: it is time forus to say something more specific aboutformative evaluation. In general, there is noneed at all for this to involve the kind ofrigorous supervision of questionnaire ad-ministration we've been talking about todate. Nor need it be done at any particularscheduled time. In my classes I have fre-quently used the following eccentric proce-dure. Evaluation forms are distributed to

its


everybody that comes into the class themoment they walk in the door on the first orsecond or third day while the tourists arestill shopping around. The forms requestthe student to turn them in by putting thermon the front desk, or by giving them toanother student to turn in. if they decide notto continue with the course after even oneor two sessions. The reason for this is thatone wants to know whether there was some-thing misleading about the advertising.needs-irrelevant about the content or offputting about the early presentations, andthis is the best possible way to pick is up.Perhaps there was something about the waythat one was outlining the proposed exami-nation and assignment process that seemedparticularly formidable. One won't pickthese objections up from people that left be-cause of them and hence aren't there towardthe end when the usual forms are distrib-uted. Indeed, as you reflect upon the usualprocess. you see that it is heavily biased infavor of the instructor because of the highlyself-selected population that stays around.One is losing much of the critical and nega-tive feedback, and I personally find that Ilearn a good deal from it, though I won'tpretend that I like reading all of it.

With respect to students who stay around.one next requeqs that all of them should feelfree to submit anonymous suggestions andcriticism at any time throughout the term.One needs to make very careful and specificarrangements about this, to preserve theiranonymity and to encourage them to seethat one is going to do something with theresults: e.g. issue feedback forms and pass acollection box .around once a week. Afterthe midsemester, one requests that a fullcomponent analysis form be submitted withperhaps a half dozen questions on it relatedto how fair the exams and grades were, thecoverage to date. the treatment of studentsraising questions in class etc. Now one hasthe chance to show that this input really isuseful, by discussing it and taking whatever.steps seem best to improve the class in thelight of it. This is also a very good time todemonstrate that some of the criticisms aremutually contradictory and hence that itisn't easy to satisfy everybody. Next oneapplies to the faculty Senate or the approp-

crate administrator for permission to give anearly final. The final is given during the lastclass (or two classes) and is corrected intime to be returned at the time originallyscheduled for the official final in finalsweek. Attendance at the session is required.as it would be fo'r a final, upon penalty ofreceiving an incomplete in the course. andpossibly a grade penalty. At that last ses-sion. the exam questions are gone over infront of the class, an answer key with"model answers" is handed out. a sheet ofactual and unsatisfactory answers that illus-trate certain types of mistakes are handed outwith the appropriate commentary on it, anopportunity is provided to raise objectionsto the grade with the instructor and perhapswith teaching assistants, and then the stu-dents are required to hand in a final evalua-tion form on the class. The students whohad been optimistically supposing that theywere going to get an A and, in the light ofthis, had given the instructor quite a goodgrade in the tenth. eleventh or twelfth week.are now suddenly confronted with realityand their evaluation of the instructor isoften affected. Too badbut one cannotexpect them to provide as good a judgmentof the course prior to receiving a grade for itas subsequently. Moreover this arrange-ment- -apart from improving the evidentialbasis of the student evaluation and henceprobably the validity of their ratingsmakes the exams part of the learning pro-cess, not an arrow shot into the air and fal-ling to earth one knows not where exceptfor a grade. Improvement and effort shouldbe directed by the most accurate evaluationpossible. whether or not the institution usesthe hest systetn. That's the first point aboutformative evaluation (evaluation fordevelopment.)

In the course of formative evaluation. it isentirely appropriate to a'sk questions of thestudents about style, where one is-striving toachieve a particular style. But it is evenmore appropriate (because of known valid-ity) to ask some microquestions to identifyproblem areas such as the text. handouts.quizzes. mid-term assignments. gradingprOcesS, treatment of questioners. availabil-ity in office and after class,' and final exams.There are no style assumptions lying behindthe belief that it is better to perform well in

17


each of these areas. Whereas it is certainlypossible to have all instructors ask every-body about these matters, i.e. build theminto the general form, the consequent in-crease in the load on the computer is notonly costly but less effective because therigid time for collection of the data does notprovide one the opportunity to see whetherimprovements have been effectuated thatseem to work well with the class. A one-shotfeedback is never as useful formatively as asequence of feedbacks.

If an instructor is getting bad ratingsover that do not seem to be explicable interms f. for example, the peculiarity thatthe course is required for non-majors, I.-cf'

'she or he should certainly go to the 1, -td ofcomponents analysis form that I have justmentioned. Professional consulting at thispoint will often suggest a number of ways toimprove performance. Only if all of thatfails should one consider going to an analy-sis of style and a discussion of a,ernativepossible approaches to that. because the re-sults of style research make this a last resortand there are still other non-style reasons toworry about first, such as the giving of in-discriminate A's, which may better serve toprecipitate a request or requirement that theinstructor alter a particular teaching processbecause it's unprofessional.

I think that most people who have therequisite content knowledge are capable ofbecoming rather good instructors; but Idon't think that many of them do so, and Idon't think' that all of the faculty--eventhose with tenureeither could or will be-

come or remain reasonable insif uctors.Hence the evaluation system needs to beone which provides a solid basis for pro-ceeding first to put pressure on an unsatis-factory instructor to upgrade performance,and second to remove them either from the

faculty or from the instructional faculty ifthe first result is not possible. We are nowmoving away from the legal model of the"master-slave" relationship in colleges andtowards the legal model of "just cause" fordismissal, as a result of the affirmative ac-tion legislation and the gradual develop-ment of an orientation toward collectivebargaining. So previous fears about viola-tions of academic freedom, which were themost important basis for tenure, are some-what less of a worry and can certainly be

12

taken care of via protective contracts.whether, formal or informal. The residualinfamy of tenure, due to the number of peo-ple who are "burn,r!fi out but kept on-by it,stands as a proernatiura of lack of respon-sibility that is becoming increasingly prom-inent as the hard times for higher educationdevelop. We can't afford to continue thatway, and we can't move any other waywithout a rock solid process for evaluatingteaching and/or improving it. which mustbe the court of first resort.

VII. Integrating and Disseminating theResults of Faculty EvaluationsWhile the results of formative feedback

go to the instructor alone, or to some con-sultant that is chosen by the instructor, thesumma'ive evaluations from students go tothe responsible first level administrator.They must be combined with the informa-tion in the files, direct measurement oflearning gains (if available) and appropriatecomparisons, ratings by teaching assistants.the quality ratings made by topic experts ofthe content and by process experts of theexamining and certain other professionalprocesses. In addition, considerations ofwort? must be brought in for the approp-riate decisions. The integrative process is avery tricky one and all the preceding workmay be wasted if bias can slip in here as itusually can and does. Without getting intothe minutiae of the topic, it is not easy togive a complete outline of what should .happen at this point. But the ideal towardwhich one should strive is clear e- .iugh: theintegrative process should-be a simple linearweighting and summing procedure. In thisrespect it should be the same as the proce-dure for combining the results of the teach-ing evaluation with the results of researchand service evaluations. Extreme rigidity inthe process is far better (although not neces-sary) than allowing the department chair orthe dean to do a holistic "seat of the pants"synthesis of the data at this point, a synthe-sis which is a principal way for bias to comein.

There are of course a variety of situationsin which One does not want to use equalweighting for all these components, but itshould be a matter of extreme opennesshow the,actual weights are determined and

s


what they are; and where it is possible toarrange for individual variations in the"contract."

What we have sketched above is a ratherincomplete version of what is only a fairb.complicated procedure. It's a perfectlycomprehensible procedure and can readilybe made a rather equitable one. One of theresults to be avoided at all costs is thepressure to go with different evaluativeapproaches by different departments orschools. It may be a useful political pro-cedure to start off with that as a possibil-ity in order to get people on boardwho would rather die than admit that theirteaching process in a lab or a clinic hasanything in common with the teaching pro-cess in a philosophy seminar or a basic sta-tistics course, but the real truth of the mat-ter is that any such view is completenonsense. There is nothing about any of theprocedures recommended here that will notwork perfectly well for any course offeredon a university campus, from the school ofmusic to molecular biology. One flay wantto add an extra question or two here, dropone or two there, especially from the forma-tive evaluation procedures. any r:rhapseven from the "Black Marks List." tiut theseare very minor changes and the simplestway is to put all of the questions on all ofthe forms that are generally distributed,with a "not applicable" option for the stu-dents. The backup use of other forms, withreference to style, where necessary or de-sired, may certainly be made as idiosyn-cratic as the instructor pleases, and the de-partments may have a preferred version ofthese. But those forms should not get intothe personnel review process even at the de-partmental level, without both legal andethical expert opinion.

VII. The Key Ethical/Legal ConstraintData - Gathering

The underlying reason for taking such afirm line about; this point is not merely theabsence of any scientific evidence that con-Iects particular approaches to teachi-ngwithv..ccessful results: it is a much graver Mat-ter. The error in racism, or sexism, does notlie in the empirical falsehood of the claime.g. that the crime rate amongst blacks ishigher than it is amongst whites, or that

management experience is much less likelyto be present in a woman than it is in a man:both these and many other similar generali-zations are true. But their truth fails to jus-tify the appeal to them in order to makea personnel decision adverse to a black orwoman candidate. And their truth does notjustify such an appeal, not because ithappens to be illegal, but because it happensto be scientifically indefensible. Generaliza-tions like this are very much less reliable aspredictors than inferences made from trackrecords of the individual candidate. It is ad-ditionally and ethically unquestionable thatany use of such generalizations would leadto self-fulfilling and socially undesirable re-sults, so on that additional ground theymust be abhorred. But one does not needthe additional ground in order to see that,no.matter what the results of research onteaching styles ever turns out to be, one willnever he able to use those results for makingdecisions about individual instructors be-cause to do so would he a case of guilt byassociation. (One may well ask what thepoint of research on teaching styles is, if itcan never be used in this way. The bestanswer is that it can increase our repertoireof promising options.)

A final point on the ethical side. I do notthink that it is ethical, and I suspect it maynot even be legal, to deny the students whogenerate the key evaluation judgments inthis whole process the opportunity to see thesummarized results. One might certainlyargue against it with respect to some detailedwritten-in remarks, because those cannotproperly be used without balancing themagainst all the rest, anddoing that in a sys-tematic way is something that students maynot be in a position to.do: but the holisticresponses and their distribution is some-thing one cannot properly withhold fromthe students. In addition to questions ofpropriety, there is the further thoroughlyunattractive possibility' that the students,frustrated in their attempt to get these, willset up their own evaluation system and onewill then either have to cooperate with it.which increases the chance of questionnaireoverload with a reduction in response rateand validity. as well as a loss of class time: ortake the extremely unattractive position thatstudents are not allowed to poll their peers in

1913


order to evaluate instructors.Some compromises will be necessary be-

cause the kind of evaluation of instructorsthat students are interested in doing is asdifferent from the kind that we have beentalking about as formative is from summa-tive. Students are interested in such ques-tions as the assignment-mode, the cost ofthe textbooks and whether they are reallynecessary, and certain teaching style vari-ables which can well be an appropriate basisfor the student with a certain learning styleto use in selecting in or out of a particularclass. But compromises of this kind shouldbe made and can be made in the interest of acampus-community approach. Student mo-rale is not well served by leaving them out ofthis process, and in the long run faculty eval-uation and faculty improvement suffersfrom not having input from these studentpoints of view.

VII. ConclusionPersonnel evaluation in general is a diffi-

cult field and one that is usually done withshocking incompetence ds one can readilydiscover by studying the forms used by the'White House, or by large corporations orthe Armed Services. If it is to be improved,presumably the. academies shouldbe the source of the leadership. It is they af-ter all, who do most of the relevant researchand interpretation. In the key matter of theevaluation of teaching at the college levelthey have shown a disgraceful disinclinationto get into it, and in recent years when thatdistaste has partly evaporated, a disgracefullack of capacity to do it in a defensible way,Perhaps the above remarks will prove suffi-ciently irritating to lead to improvement,even if they are not themselves accepted assuggesting appropriate improvement.

A Partial Bibliography Relatedto Clinical Supervision

Alfonso, Robert J.; Firth, Gerald R.; andNeville, Richard. F. Instructional Supervi-sion. A Behavior System. Boston: Allyn andBacon, 1975.

Beegle, Charles W. and Brandt, Richard M.Observational Methods in the Classroom.Washington, D.C.: Association for Super -visior. And Curriculum Development, 1973.

Blumberg, Arthur and Cusick, Philip."Supervisor-Teacher Interaction: An Anal-

. 14

ysis of Verbal Behavior." i:ihriution. Vol.91. 1970.

Boyar'. Norman J. and Copeland. Willis D.in.structional Supervision Training Pro-gram. Columbus, Ohio: Charles E. MerrillPublishing Company, 1978.

Brammer, Lawrence M. The Helping Rela-tionship: Process and Skills. EnglewoodCliffs. New Jersey: Prentice-Hall. 1973.

Cogan, Morris L. Clinical Supervision.Boston: Houghton Mifflin Company. 1971Goldhammer. Robert. Clinical Sup:Special .tiler /toss for Supervision ofTeachers. New York: Holt. Rinehart. andWinston, 1969.

Hyman, Ronald T. School Administrator'sHandbook of Teacher Supervision andEvaluation Methods. Englewood Cliffs, NewJersey: Prentice-Hall. 1975.

Koran. J.J. "Supervision: An Attempt toModify Behavior." Educational Leadership.Vol. 26, May, 1969.

Kysilka, Marcella L. "Research in ReviewClinical Supervision: A Review of theResearch ." Ethical imull Leadership. Vol.35, April 1978.

Reavis, Charles. Cliaical Supervision: ATimely Approach." Educational Leader-ship. Vol. 33, February 197.

The Supervisory Proms: Helping Teachersto Improve Instruction - Videotape,Washington, D.C.: Association for Super-vision and Curriculum Development, 1978.

Some Other Documents Providing infor-mation Related to Teacher Evaluation

Evaluating Teacher IffectivenessERICAbstracts. Eugene, Oregon: ERIC Clear-

, inghouse on Educational Management,University of Oregon. 1971. EvaluatingTeacher Performance. Arlington, Virginia:Educational Research Service, Inc. 1978.Evaluating Teachers for ProfessionalGrowth: Current Trends in School Policiesand Practices. Atlington, Virginia: NationalSchool Public Relations Association, 1974.Evaluating Teaching Performance. Wa-shington, D.C.: Educational Research Ser-viCe, American Association of School Ad-ministrators, 1969.

20

NATIONAL DISSEMINATION CENTERPROFESSIONAL DEVELOPMENT PUBLICATIONS

PUBLICATIONS ORDER FORMTitleNo. Amt.

Perspectives on Preservice and Inservice Education $4.00

Louis RubinThe Role of Itte State Education Agencies in Inservice Education 53.00

Wendell AllenInservice Education: State Scenes 56.50

National Council of States on Inserv.,:e EducationSources & Resources: An Annotated Ribtiography on Insercicc Education $8.50

National Coo' of States on Inseecice Education

Inservice Priority of the "Ms S4.50

Margo I :nson

Pro, I+! i.eadership for Inservice Education 54:50

,ccedings of the 1979 NCSIE National WorkshopState-Level Teacher Performame Evaluation Policies $2.00

Lou M. Carey (NCSIE Occasional Paper NumberMerging Resources of the IHE and the LEA for the Improvement 52.00

of Inter-Institutional ProgramsDwight W. Allen

andThe Competency Centered Field Based Early Childhood

Certification ProjectGordon E. Fade (NCSIE Occasional Paper Number 2)

The Evaluation of College Teaching 52.00

Scriven (NCSIE Occasional Paper Number 31

Expanding the Implications of Public Law 94-142 S2.00

Maynard Reynolds (NCSIE Occasional Paper Number. 4)Issues in Inservice Education: State Action on Inservice Education $4.00

National Council of States on Inservice EducationInvolvement: A Study of Shared Governance 54.00

Bruce R. Joyce

Toward Meeting the Needs of the Beginning Teacher 54.50Kenneth R. Howey and Richard H. Bents. Editors

Sal of Five ISTE Reports 520.00

or

ISTE Report I .---Issues to Face 53.00Bruce Joyce. Ken ,Howey. and Sam Yarger

ISTE Report IIinterviews: Perceptions of Professionals and Policy 55.00Makers. Bruce Joyce. Kathleen McNair. Richard Dial. and

Michael McKibbenISTE Report III --The Literature on Inservice Teacher Education: 54.00

An Analytic Review. Alexander Nicholson and Bruce Joyce

ISTF Report IV Creative Authority and Collaboration: A Collection of Position 55.00Papers. Sam Yarger et. al.

ISTE. Report V -Culturdl Pluralism and Social Change 55.00

Richard Brandt et. al. ,Meeting the Special Needs of Students in Regular Classrooms $3.00

James F. Collins and Joseph A. Mercurio. EditorsCriteria for Describing and Assessing Competency Based Program, 54.00

J. Bruce Burke. John H. Hansen. W. Robert Houston.Charles Johnson

A General Catalog of Teaching Skills 59.00Richard Turner. Editor

A Catalogue of Concepts in the Pedagogical Domain of Teacher Edtication $4.00

Bryce D. Hudgins. Project DirectorCompetency Assessment. Research and Evaluation 53.00

W. Robert Houston. EditorClosing the Knowledge Gap 53.00

H. Del Schalock

21

1Resources for Pertormance-liased I'd iwation & Supplement A S31111

W Robert RousionSocial Studies leacher Competencies S2 00

Minnesota Department of Education

AssessmentS2 00

I heudore F. Andres% s. Editor

To order publications from the National Dissemination Center, check t he titles sou require. indicating number ut copies

needed. and return this lorm to the National Dissemination (enter, Sracuse l nis e , 123 Huntington Hall, Syracuse,

New York 13210.An orders not accompanied by a formal purchase order must include a check or mines order made pioable to t he

National Dissemination Center.

MAILING AND HANDLING CHARGE:U.S. and Canada:for one publication $0.60

for more than one publication $1.00

FOREIGN:surface mail $0.50 publicationair mail $2.00 publication

Payment enclosed_ Purchase Order No _ Amount $

NAME

ADDRESS

ZI P

16 22