DOCUMENT RESUME

ED 384 533 SE 056 718

AUTHOR Tourtillott, Leeann; Britt, Peggy

TITLE Evaluating Environmental Education Materials. TheEnvironmental Education Toolbox--EE Toolbox WorkshopResource Manual.

INSTITUTION Michigan Univ., Ann Arbor. School of NaturalResources and Environment.; National Consortium forEnvironmental Education and Training.

SPONS AGENCY Environmental Protection Agency, Washington, D. C.

REPORT NO ISBN-1-884782-10-8

PUB DATE 94

CONTRACT EPA-G-NT901935-01-0

NOTE 55p.

PUB TYPE Guides Non-Classroom Use (055)

EDRS PRICE MF01/PC03 Plus Postage.

DESCRIPTORS Educational Resources; Elementary SecondaryEducation; *Environmental Education; EvaluationCriteria; *Evaluation Methods; Inservice TeacherEducation; *Instructional Material Evaluation;Learning Activities; Workshops

ABSTRACTThis manual was written for Lth environmental

educators who want to evaluate the environmental education (EE)

instructional materials they use or recommend to teachers, andinservice providers who want to help teachers evaluate EE materialsin workshop settings. This information is provided to expandknowledge and expertise, facilitating informal yet thoroughevaluations, and to help educators instruct others to do the same.The user is guided through an evaluation process and provided with

tips, suggestions, criteria, worksheets, and workshop activities.

Sections cover evaluation basics, evaluation criteria, developing an

evaluation strategy, formal evaluation projects, workshop activities,

and activity masters. A resources section contains a brief glossary

of evaluation terms and 20 references to evaluated EE materials,

evaluation tools, and additional readings. The manual is not intended

to direct the development of formal evaluation projects that requirestatistically significant resulLs. It is a practical guide, onedesigned to help users simply but effectively examine EEinstructional materials to determine whether they meet their needs.

(LZ)

****************1:k*****************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

*

***********************************************************************

ToolboxWorkshop Resource. MdnuJI

U & DEPARTMENT Of EDUCATIONOnce ot Educatoonel ShmeanA and laddessenonl

ED AT1ONAL RESOURCES INFORMATIONCENTER (ERIC)

rs document has been reproduced asCervoci from tn gmwoorn M oroor000non

ongmohng it0 Motor cnandes have been made 10 0MIXOrit

fOrrrOduChOn oualsty

Points of wow" of opniOns stated in th400CYmint .10 not OSIIIIHY frbOnnflot ("COIOEM p0OltiOn Or 1)01...y

EE ToolboxWorkshop Resource Manual

EvaluatingEnvironmentalEducationMaterialsLeeann TourtillortDepartment,1 InvInmmental

And Planntru;

'()Itotrz,! Mate t'mverstry

Peggy BrittNational (:onsortiunt

nI;;U

I ducatrun and

I'mversztv of .112l:wan

This protect has been funded by theUnited States Environmental ProtectionAgency under assistance agreement numberEPAGNT901935.01.0 to the Universityof Michigan School of Natural Resourcesand Environment.

3

NCEET

School of NaturalResources and Environment

University of MichiganAnn Arbor, MI 48109.1115

313 998-6726

Workshop Resource ManualManaging Editors:

Martha C. MonroeDavid CappaertPeggy Britt

ISBN 1 884782 10 8

The contents of this documentCO not necessarily reflect theviews and potion of theJotted States EnvironmentalProle Ction Agency. nor doesmention of trade names orcOrnmercial produCts corratote erdorsement or recommendatiOn for use

Copyright 1994 by theRegents of the University

Michigan

Educators may PhOtocopvthese materials for thenon commercial purpc se

educational advancement

Printed on paper containing1104, rer yr .ed h Per 70"

r,.st consumer fiber US Cbasea one

Production Coordinator'oan Wolf

Designed by Margaret Re-vveset in 'aabon'Iews Gothic and Franidol.othic Number 2

4

Introduction 2

Evaluation BasicsA Few Terms 4

A Few Key Concepts 5

Comparing Direct vs. Indirect Evaluation 5

Evaluation Crir riaEnvironmental Education Goats and Objectives 7

Pedagogy 8

Presentation and Design 9

A Bit About Bias 9

0 Developing Your Evaluation StrategyStep 1. Quick Review of Materials 11

Step 2. Prepare Your Evaluation Tool 12

Step 3. Compile Results 14

Step 4. Evaluate the Evaluation 15

411 A Look at Formal Evaluation ProjectsThe Expert F.eview 17

The Group Review 17

The PreTest/PostTest Review 18

0 What to Do in a WorkshopApproach A: Provide Evaluated Materials . . . 20

Approach B: Develop Criteria 20

Approach C: Do a Mini Evaluation 21

411) Activities and Masters 23

(ID Resources 44

.

This unit is written for both environmental educators wl)i) minttO et,aluate the environmental education instructional Materials

. the\; use Or rec-ommend tU tegicliers,.ind inseivice ploviders who

want to help teachers evaluate PI: materials in umkshop settings.,I his information zoill ex /)an(/ their knowlelge.,incl expertise,.lasilitating informal vet.thorough entluations, and hel pthemnistruct others to do the Sa 111C. 11 illtiOtinees inr /)ol tant baCkglintni

infomation, then guides n'ailers through an evalua-tionpa)ces.s,providing tips, suggestions. criteria. wori;socets, and t(?inks1)opactivities. \k'(' intend to hell) the following VI.: professionals:

leac-her ecluc,itors can Informally et,illuate the materials thin.

provide or recommend to leachers..11AT ,zir also in ernexcellent position. to help teache;rs..learn about the evaluation

proc-ss. In the zvorkshop idcLis Lind activities sections. teacher

educators will find truys to imiocluce the central concepts ()I

Nonfornzal environmental educators will lind strategics to.1.04) thenl predict how well a given set orenviroinnenLtieducation instructional matenals meets their needs.

Curriculuni specialists can use this unit to develop a..vsteinatic approach /01' reviewing f.l..111.st177ctional inatenals

in their district. 'this approach helps C11.5'1111' that

e / /CC-1;('C' tIllif meet state. loc al. or other specilic.educ,rtional

objectives.

I his manual is not intended to tined the development of4.,

- tonna! entluation pnnects that 11'41Ifi1e Statlst1Ca111' 611111/1(' lilt

results. it_is a lulu-tied' guide, one designed to hell) users simply

but e.vainine malenals lo determine

u'llether ;Yal their-n(e(l.s.

4

Steve and Anita Take Stock

"I was nervous as we planned this workshop but the teachers really seemed to learn a lot!"exclaimed Steve, a resource specialist. He and Anita, a district environmentaleducation coordinator, were meeting to discuss their recent biodiversity workshop.

"They seemed to have a good time too, but let's see how they evaluated us," Anitaproposed as she began looking through the surveys teachers filled out at the endof the workshop. Anita pointed to a few of the completed evaluations, noting,"Several teachers did say they were overwhelmed with the volume of materials."

Looking over her shoulder at the evaluation forms, Steve said, "I wonder if we mightbe wasting some trees. Can teachers really use all those materials? Some of themtook so much stuff, they had to make two trips to their car!"

"I remember when I was a teacher," Anita recalled. "Most of the resources I got at work-shops turned out to be dust collectors. I just didn't have time to sort through allthat stuff to figure out which were worth using. I still have that problem!"

She and Steve decided it would be better to provide teachers with only the best resources.They agreed it made sense to evaluate the resources they provide, just as theyalways evaluate the success of their workshops.

"It seems like a big task," said Steve. "I've read some articles in the Journal of Environ-mental Education (JEE), and all of them stressed being very systematic aboutevaluations, developing a rating scale, and setting up a committee of teachers.It seems like we should get someone at the university to do it."

"I don't think we need to be that technical about it for our own personal use," repliedAnita. "I'm sure we can combine our knowledge of biodiversity and educationwith some of the information from those JEE articles and come up with anevaluation strategy that works for us and our workshop participants."

"I don't know, Anita. Do you think we'll be able to predict teachers' needs?" Steve asked.

Anita thought for a moment. "I have an idea," she said. "What if we build some timeinto our workshop for teacher:: to learn about a couple of evaluation scales.Then, using those scales as examples, they can select or create their own criteria,scales, and evaluation methods."

"Great idea!" Steve added. "We can figure it out and then pass those skills on to teachersand they can improve on our work. That way they can apply evaluation to alltheir EE materials." Anita and Steve agreed to do some research on their own.They planned to meet again to evaluate their materials and begin developinga workshop session on evaluation.

7Evaluating Environmental Education Materials

Introductionma

NE OF THE REASONS TEACHERS ATTEND INSERVICE PROGRAMS iS to

find new instructional resources. As inservice providers, we respondto this need by distributing instructional materials and handouts.Walk into any environmental education (EE) teacher workshop and

you will see the walls covered with posters and tables piled high withsupplemental resources containing content background, activities, and newteaching tips. EE conferences have rooms full of exhibits displaying new activityguides and other instructional materials.

Often teachers have no trouble finding EE materials, rather formany it is a daunting task to choose from all that areavailable. Teachers oftenhave neither the time nor the confidence to sort through and choose the bestinstructional materials for incorporating EE into their curriculum. You can helpteachers increase the quality of their lessons by providing simple and effectiveways to evaluate EE instructional materials. Furthermore, evaluation canfamiliarize teachers with environmental education concepts. Teachers canmake better choices to meet any numb,r of objectives and individual needs,such as:

meet state and local learning objectives

address environmental education goals

convey accurate information

include relevant context for various socio-economic groups

relate age-appropriate concepts

adapt to different teaching styles

accommodate the different learning styles of their students

select interesting and easy-to-use materials

fit individual constraints, such as time or money.

All educators can gain the skills and knowledge to evaluate environmentaleducation materials. We will provide you with background information,simple tools, relevant examples, and activities. You can use these to evaluatc thematerials you present in workshops and to lead activities that build teachers'evaluation skills.

.VorAshof.) Resource Manual

This unit focuses on evaluation of written environmental educationinstructional material, most specifically activity guides. However, the criteriaand techniques discussed here can be adapted for use with many types ofinstructional material such as video, software, and texts.

The sections in this unit focus on the following steps:

1:valuati;;n BasicsBackground information puts the evaluationprocess in a larger context, helping you tobetter understand evaluation procedures,guidelines, and criteria. Included in SectionI are basic terminology and key concepts.A more complete glossary of terms is pro-vided on page 44.

Evaluation CriteriaThe heart of any evaluation effort is thecriteria used to rate the instructional mate-rials. There is no single answer to whatmakes a good activity guide. However, thereare several aspects that every EE accivityguide should include. Section 2 providesa framework for developing criteria andsuggests specific criteria relevant to EE.I ;se this information to generate a list

f criteria that meets your specific needs.

Developing Your Evaluation StrategyJust as no single set of criteria will meetall evaluation needs, no one evaluationstrategy applies '-.) all EE activity guides.Selecting a strategy can be a confusingtask. This section (pages I 1-15) demysti-fies evaluation strategies by providing bothguidelines for modifying existing tools andstep-by-step procedures for developingyour own.

. Look at Formal Evaluation ProjectsJust because an activity guide har, been for-mally evaluated and approved by a partic-ular organization does not guarantee itwill satisfy your needs. The guidelines andreal-life examples in Section 4 can helpyou evaluate methodologies. You can alsouse this section to understand some ofyour options if you arc interested in devel-oping a more formal evaluation project.

of, t,,,,,,,,t.)1 L.

Most instructional materialsmay be useful even if they donot rate well on all criteria. Itis best to focus on strengths,weaknesses, and usefulnessin your evaluations.

What To Do in a WorkshopTeachers are more likely to evaluate materi-als if they receive hands-on experience inyour workshops. Section 5 provides ideas,tips, and sample agendas to help you incor-porate evaluation topics into your work-shops. Several strategies are provided tohelp you choose an approach that willwork best in your workshops.

Workshop Activities and Sample lbolsSection 6 complements the previoussection by matching the workshop ideaswith specific activities. Each activity isdesigned so you can use or modify it foryour particular workshop.

ResourcesThis document may leave you wantingmore information, ideas, and details. TheResource Section provides a bibliographyof evaluation projects and additionalreadings.

9

Evaluation Basics

IKE ANY OTHER DISCIPLINE, evaluation has unique terms and con-cepts. This section presents some of the basic tenets of evaluation andI. environmental education. Understanding this background will help

4 prepare you to adapt evaluation criteria and methods (described insections II and III) without compromising effectiveness.

A Few TermsTerms such as"curriculum," "instructionalmaterial," and "activity guide" are oftenused interchangeably. Perhaps they havevery specific meanings for educators inyour area. To avoid confusion in this docu-ment, we use these terms very specifically.

A curriculum is a complete course of studycarried out over the semester or entire schoolyear to meet a specific set of learning objec-tives. The learning objectives may be estab-lished by teachers, school districts, stateguidelines, or national standards.

Instructional materials are resources thatteachers use to address the curriculum.Such resources may include texts, activityguides, videos, and other materials designedspecifically for classroom use, as well asnewspaper articles, movies, comic books,and other sources of information on envi-ronmental concepts or issues.

Activity guides are one type of writteninstructional material that often containbackground information, student activities,and suggested lesson plans. Many activityguides cover specific learning objectivesand are designed as supplements, form aunit, or make up a course.

Evaluation of instructional materials isthe systematic application of criteria tojudge whether the materials will meet theuser's goals.

These distinctions are intended to make onepoint very clear: most instructional mate-rials should not be evaluated as ifthey will make up an entire curriculum.As part of a larger picture, they may beuseful even if they do not rate well on allcriteria. Hence, this unit focuses on identi-fying the strengths, weakness, and useful-ness of written EE instructional materialsfor the specific needs of teachers.

Another set of terms is used to describemethodology. To help you meet yourgoals of developing a simple and effectivestrategy, the following sections outlinean informal approach with the followingcharacteristics:

Am:chair Review: Emphasis is on anindirect evaluation of materials (beforeyou use them) to select those that best meetyour needs.

Teacher usefulness: These strategiesinclude criteria based on goals of EE andhow well the materials meet specific needsof teachers (vs. the goals of the authors).

Limited scope: The criteria and strategiesin this unit are best suited to evaluateEE activity guides. Some criteria may notbe appropriate if you are evaluating othertypes of instructional resources.

Single or Comparative: Criteria canbe applied to one activity guide or usedto compare several guides.

A Few Key ConceptsYou may have an intuitive sense of whatworks, or know which type of activities areusually popular with students. Yet, if askedto provide an "evaluation," you may notknow how to support your intuition.Is it enough to know whether activitiesexcite students? Is helping them learn oneor two concepts sufficient? You may findit difficult to define all of the factors thatmake an activity work with students.

One way to evaluate EE activity guidesis to think about your goals as a teachereducator. Your role is to motivate teachersand help them develop a good knowledgebase, skills, and a desire to bring EE totheir classrooms. In measuring the successof your workshop, you may gauge whetherthey became more motivated, knowledge-able, skilled, and active environmentaleducators. Likewise, evaluating EE activityguides considers many of the same typesof elements: reactions, learning, behavior,and results.'

Reaction Evaluation measures the users'immediate positive or negative responseto the learning experience. This is themost common level of evaluation. An arm-chair review may predict how students willrespond to the material.

Know lecke Evaluation measures whetherusing the activities with students increasestheir knowledge and awareness of environ-mental issues and influences attitudes.In an armchair evaluation you can assessthe quality of the content and if teachingmethods meet your students needs.

Behavior Eniltiation measures whetherdie learning experience alters studentbehaviors. Your armchair review coulddetermine whether the materials providethe knowledge and skills the learners needto be able to engage in new behaviors.

Results Liviziation measures whether thebehavioral changes have an overall impacton the environment or students' lives.

It is very hard to measure whether an impacton students' lives is due to the instructionalmaterials or any of a million other poten-tial factors. In fact, evaluating for results isoutside the scope of most informal evalua-tions and will not he discussed in this unit.Let's explore how you may achieve reac-tion, knowledge, and behavior evaluationin an armchair setting.

Comparing Directvs. Indirect EvaluationA direct evaluation of EE activity guides iseasy to envision. Teachers can gauge studentresponse find use a variety of assessmenttools to find out whether students learned.In the context of an informal armchairreview, you can only gauge your responsesto the material. However, you can still adddepth to your evaluations by focusing onthe likelihood that the activity guide willelicit a positive response, help studentslearn, and change behaviors.

Direct evaluation usually yields a stron-ger certainty, but it is difficult to evaluatemore than a few activities. With an indirectreview you can critique many activity guidesin one sitting. The two scenarios in thebox on the next page illustrate how youmight indirectly predict which guides willbe useful.

ke .;%7A,

VerI.11111rTV!..!

Examples of Direct Evaluation and Indirect Review

Layton and Miguel have each received a new activity guide on recycling.Layton, a high school teacher, decides to evaluate it by using it with hisclass. Miguel, a resource specialist with a local governmental agency, doesan armchair evaluation before using it in his next workshop.

Layton's Direct Evaluation

Reaction: Layton noted students'general level of enthusiasm, atten-tion span, how many questionsstudents asked and how hard it wasto encourage discussion.

Knowledge: Layton started his cl:: ,ssession by having students write ashort essay about the impact of solidwaste in their lives. He comparedthe level of knowledge in the essaysto test scores from a post-lesson quiz.

Behavior: Students asked permissionto set up a classroom recycling project.Layton noted whether the unit pro-vided them with the motivation andskills to accomplish this task.

-. I.

Miguel's Indirect Evaluation

Reaction: Miguel listed the attributesof fun learning experiences. Hereviewed the activity guide for suchfactors as providing a variety oflearning situations, relevancy tostudents in his area, and hands-onactivities.

Knowledge: Miguel reviewed theactivity guide to determine if keyconcepts were present, how wellthey were addressed from a varietyof perspectives, and whether prob-lems and solutions were presentedso students may identify appropri-ate roles for themselves.

Behavior: Miguel looked to seewhether activities presented infor-mation on a broad range of feasibleaction strategies and included activ-ities directed at building skills stu-dents would need to engage in suchactions.

12r;eroi.,

Evaluation Critzria

RITERIA ARE THE HEART Of any evaluation effort. Now that you havea sense of .mhat an armchair evaluation is, it's time to start definingthe specifics. The number and type of criteria you use and the methods

11/44 used to apply them will all effect your results. These specifics makeeach evaluation effort unique.

There are different categories of criteria you may consider for yourevaluation. Educators tend to focus on the quality and accuracy of content.Learning styles and presentation of the material are also important consider-ations. Many EE evaluation efforts measure arrainst the goals of environmentaleducation Often teachers use state or local education objectives to define thecontent goals. This section provides a framework for developing criteria basedon three overlapping aspects: environmental education goals, pedagogy, andmaterials design. Together, they can create a well-rounded, thoughtful evalu-ation. After you identify relevant criteria, Section III will help you place themin a usable format.

As you read this section, remember that the focus is on EE activityguides; there will be categories or specific criteria that do not apply to othertypes of instructional materials. For example, you would not judge a text onclassification of trees against criteria for involving students in action strategies.

Environmental EducationGoals and ObjectivesThe ultimate goal of environmental educa-tion is to establish an environmentally lit- Objectivescrate and involved citizenry.= To meet this of Environmental Educationgoal, it is important that students not onlylearn environmental concepts but also be Awareness

able to define environmental issues within Knowledgea social context, understand multiple per-spectives on an issue, and understand their Attitudesroles in both environmental problems andsolutions. (For a more in-depth discussion Skills

on the definition and goals of EE, refer to Participation"Defining Environmental Education" inthe Workshop Resource Manual.) from the Tbilisi Declaration, Unesco Conference, 1977

1.3

Environmental Awareness and KnowledgeTwo important factors that you can assessin an armchair review are accuracy andbreadth of content. Teachers not familiarwith an environmental topic area mayneed help in developing content specificcriteria. You can start by narrowing thetopic into 5 10 key concepts and developcriteria that address these concepts. Youmay wish to team up with environmentalspecialists for this step. Some importantcriteria might be:

terms are defined

environmental concepts are setin a social as well as an ecologicalcontext

ecosystems include individuals,the local community, and the globalconununity

environmental issues relateto specific environmental concepts

information contains multipleperspectives (or bias is declared)

concepts are relevant to learners' life

depth of information is relevantfor the intended developmental level.

AttitudesThere's no way around it, environmentalissues and the solutions we choose are value-laden. Attitudes about the environment arcbased on individual and social value systems.It is important for an EE activity guide tohelp students define and understand theirown values as well as understand and valueother people's perspectives. Students ideallycome away from a lesson feeling positiveabout participating in environmental issues.Considerations of attitudes include:

activities encourage a sense of personalstake and responsibility

materials express a hopeful outlook

materials contain positive role modelsfor discussion and action

activities encourage students to clarifytheir own values

environmental topics are covered froma variety of perspectives, and no one per-

spective is predominant.

Skills and ParticipationStudents are more likely to become activein environmental solutions if they areaware of the range of possible actions, areable to evaluate them for appropriateness,and have the skills necessary to implementchosen action strategies. Successful EEinstructional materials address action strat-egies in several ways:

environmental problems are presentedwith a range of possible solutions

the material compels students toconsider the implications of possiblesolutions

activities help students gain newproblem solving skills

action strategies focus on win-winsolutions, whenever possible

activities provide students withcritical reasoning and problemsolving experience

both individual and community actionstrategies are presented

activities help students gain communica-tion skills, group skills, leadership skills,and other interpersonal skills necessaryto become involved in solutions.

PedagogyPedagogy, or how the material is taught,is an important consideration. Good envi-ronmental education is often said to begood education in that it emphasizes inter-disciplinary, experiential, relevant learningand focuses on preparing students to par-ticipate in their community. Because envi-ronmental education is as much a processas a content area, many pedagogical crite-ria overlap with the environmental educa-tion criteria.

a variety of cultural perspectivesis included

content excludes gender, race, age,and other biases (for more informationon bias, see page 9)

different learning styles are included(for example, hands-on, student-centered, research, lecture, and othermethods)

14,^%."0 t )n(1,j1

- activities allow students to build fromprevious knowledge

- materials offer students the opportunityto practice critical thinking processes(such as hypothesis making, collectingand organizing data, analyzing, drawingconclusions, and problem solving)

activities facilitate and encouragecooperative learning strategies

- teaching methods are appropriate to theguide's goals

- content pertains to state or local learn-ing objectives

materials can he readily integrated intoestablished curricula

- appropriate student assessment methodsare included

- learning opportunities are conductedin a variety of settings

lessons include suggestions for extend-ing or adapting the learning experience

- the materials can be easily adaptedto varied learning environments, suchas large and small classes, mixed-levelclasses, or rural and urban settings.

Presentation and DesignIf a guide has illegible type, few graphics, abinding that disintegrates, or an exorbitantprice, teachers may never discover that thecontent is good. Evaluating effectiveness ofmaterial design takes into account such ele-ments as text and illustration quality, andthe guide's organization.

- the goals and objectives of the materialsare clearly stated

teacher instructions are clear

- guide is easy to use and understand

handouts and other masters areincluded and are easy to duplicate

- the writing is clear and interesting

- illustrations are appropriateand engaging

- required activity materi ils are suppliedor easily obtained.

I t"Ir POMViltdi L.111Cdt Jr!

A Bit About BiasTeachers should address the issue of biaswhen evaluating resources. Each of us hasour own biases developed through ourexperiences. In addition to our personalbiases, we are confronted with many formsof bias every dayin textbooks, posters,newspapers, and on television. Industryand environmental groups are most oftenaccused of presenting slanted information,

Freshkills, Staten Island is theworld's largest landfill. It serves7.5 million people and won't befilled for another 10 years.Thirty -five similar landfills couldserve the whole United Statesand occupy 0.005% of the landin the lower 48 states. It wouldtake more than 7,000 centuriesfor the United States to be buriedin its own solid waste.

NYC Department of Sanitation

Although this is factual, one can detectbias. Where 35 enormous landfillswill be placed, the cost of truckingwaste from one state to another; andthe wisdom of burying resourcesat current rates aren't mentioned.

but almost all forms of informational mediacontain some type of bias. Many are verysubtle and merely a matter of how "facts"are determined and presented. Who paysfor the material often indicates what slant't might take. For example, textbooks tendto include the same material, edition afteredition, because that is what teachers andcurriculum committees expect and there-fore will buy. The bias is toward statusquo and tradition, determined by sales.Other forms are more obvious, such aslanguage that portrays leaders, scientists,or active individuals as men, or illustrationsof domestic roles portrayed by women.

15

It is important, therefore, for educatorsto acknowledge that we do not live in aworld free of bias and to facilitate stu-dents' ability to recognize and address biasin their lives.

Recognizing bias is the first step toapproaching issues (especially value-ladenenvironmental issues) with an open per-spective. V,. "ich have our own bias: com-paring different materials will help identifyit. Activity 5: "What About Bias" (page43) is designed to help teachers see bias ininstructional materials and discuss how toovercome it.

Some types of bias relate to:

race

culture

age

gender

political views

economic status

different stances (e.g. industrialistvs. environmentalist).

Identification of bias is a critical step inevaluating materials. You may not want toeliminate all biased material. Such materialcan help students understand what bias is,that they are affected by bias, and that theywould, therefore, gain from identifyingtheir own biases. Educators can deal withbias in a variety of ways:

not use the material

accept the bias as is

balance the bias with other materials

point out the bias to students and havethem discuss and analyze it

help students develop critical thinkingskills to identify bias on their own

modify materials to reduce bias.

16

N)rkshop Resource Manudi

Developing Your Evaluation Strategy

Hoar Do YOU BEGIN to deal with all these criteria and how do youkeep track of them when you are reviewing activity guides? Indeveloping your strategy you will prioritize criteria, choose ques-tions that address these criteria, devise a way to apply the criteria

that is meaningful, and keep track of your impressions so that you can compareseveral activity guides.

Teachers will benefit most from a streamlined evaluation process andsimple tools. This section, along with the evaluation tools included in SectionVI, will allow you to develop an evaluation strategy that meets your own andteachers' needs. Section V provides several ideas for helping teachers person-alize this evaluation strategy in a workshop setting.

For ease of understanding we have presented the evaluation process asfour steps: Step 1, Quick Review of Materials (p. 11), Step 2, Prepare YourEvaluation Tool (p. 12), Step 3, Compile Results (p. 14), and Step 4, Evaluatethe Evaluation (p. 15). You will probably go back and forth between steps toassure your strategy meets your goals. Keep in mind that you don't have toreinvent the wheel. Use this section to help you modify existing tools to bestsuit your needs. (Three samples are provided in Masters 3, 4, and 5.)

Step 1Quick Review of MaterialsThis first step is a fast way to get aninitial impression of the materials youwill be evaluating. You will most likelynotice prevailing presentation styles,glaring biases or omissions, and age-appropriateness concerns. A quickreview is not necessary but is usefulin the following ways:

To categorize materials into contentareas, age levels, and other categories.One set of criteria may not he appropri-ate for all types of materials; you maywish to refine your criteria for eachcategory.

To reduce the number of materialsincluded in your evaluation. Once youhave categorized materials, you maywant to eliminate some categories fromyour evaluation. For example, you maydecide to evaluate only those materialsrelevant to middle school social studiesor to water quality issues. Be careful notto discard items because they are notfamiliar or do not match your particularstyle and viewpoint.

. t "NO 1 stcr.

To review the material with an eyetoward deciding how to apply criteria.For example, you may find that allof the materials fit basic EE goals butnot in the same way. This would leadyou to ask how well each of the goalsare met.

You can use the questions that follow tomake your initial cuts. Questions of thistype can most often he answered with asimple "yes" or "no" and still providegood direction about whether to take acloser look at the material.

Does the activity guide focuson a desired content area?

Are activities age-appropriate?

Are the activities appropriate for thelearning setting (for example, outdoorcenter, classroom, factory visit)?

Is the guide readily available to teachersin your area?

17

Picking StrategySometime after they discussed developing an evaluation session for theirnext workshop, Anita and Steve gathered their teacher resource materials.

"No wonder teachers don't use all the resources we give them. No onehas enough time to go through all these materials!" Steve sighed as hepushed aside a stack of activity manuals. "I know some of these are greatresources only because I've used them. We need to have some system thatwill help teachers make choices without having to test each one."

Anita glanced at Steve', stasic of resources. "OK, it's obvious we don'tneed some of these niatrffials for a biodiversity workshop. This one doesn'taddress biodiversity and here's another that only has ac..vities for early ele-mentary children. Let's weed out the ones that don't fit with the workshop."

Anita and Steve began sorting through the materials selecting only thosethat pertained to biodiversity. After a few moments Steve noted that he hadfound quite a few resources addressing biodiversity but one was a comicbook, two were videos, several were activity guides, and some were simplyinformational pamphlets. Using the same criteria to evaluate such variedmedia would be difficult. They decided to narrow their search to activitymanuals that dealt with biodiversity.

Step 2Prepare Your Evaluation Tool

There are two ways to approach thisstep: use an existing evaluation tool ordevelop your own. While it may be moreexpedient to use an existing tool, you willwant to modify it to meet your specificneeds. Activity 3, (page 28) providesseveral sample tools. Use the ideas underapproach A as a guide to modifying thesetools for your use. Developing your ownevaluation tool may seem like a dauntingtask. Approach B breaks this task downinto substeps to make it more manageable.Remember, the shorter and simpler yourtool, the easier it will be to use

No matter which approach you take,remember that the easier the tool is to use,the more likely your results will be easyto understand. Here are some general rulesof thumb when modifying or developingan evaluation tool:

Format so that the page is easy to read;guidelines are easy to follow

Make it easy to tally answers

Have questions relate to one criterion

Keep it short

Do not use double negatives

Do not ask two questions at once.

A. Modifying Existing ToolsMost evaluation tools were developedfor evaluation projects that meet the needsof a large and varied audience. The criteriamay be broad and the rating scheme com-plex. Before using them, consider adaptingthem to reflect the specific needs of theteachers in your area. If you are a bituncertain about your teachers' concerns,you could meet with a few teachers or talkto a district curriculum coordinator. In ateacher workshop you may want to brain-storm what teachers need from EE materi-als (see Activity 1). To modify existingtools, you may cut, change, or add ques-tions, and modify how questions are rated.Here are some guidelines on modifyingboth criteria and rating schemes.

181.1 nual

Both went back to work. In a short time Steve and Anita reduced theirstack to a more manageable size. Anita noted, "We could weed out afew more if we identified resources that deal with the appropriate age level.Let's select only those that are designed for grades 4.6 or grades 7-9."

"We could be even more specific and categorize them by learning setting,like outdoor, classroom, or nature center," Steve suggested.

This time Anita and Steve found their system was getting too complicated.Few of the resources met all the criteria. They decided upon the followingcriteria to make their initial cuts:

activity manual

contain activities on biodiversity

designed for grades 4.6 or 7.9

When they finished sorting and piling, Steve felt a sense of accomplish-ment. "I think we've saved ourselves a lot of time. Now that we've reducedthe numbers, we can devote more time to evaluating them," he said.

Anita pulled out a sheet of paper. "You're right! Let's figure out what kindof things we'll look for as we evaluate."

Narrow you./ scope: Balancing thorough-ness and simplicity is crucial to success.Start by eliminating criteria that do notrelate ro your aiiiterit ,-,.;:k..erns. Thenprioritize the ::,iteria and focus on whatvviii. help teachers incorporate EE intotheir lessons.

Adding Questions: You may find thatyou wish to add questions that apply tolocal education mandates or goals. Whenadding questions, frame them similarlyto existing questions so you may use thesame ranking process. To make tallyingresponses easier, keep the ideal solutionpositive. Similarly, on a rating scale of oneto five, always make five the best or theworst.

Mattri,11.,

Changing Criteria: Rather than eliminatea criterion, consider modifying the way itis applied to make it more relevant. Thereare several ways to ask a question aboutthe same criteria. For example, you couldchange a general question about learningstyles, "Are many types of learning situa-tions used?" to fit a specific learning styleneed: "How many lessons incorporatestudent-directed learning?" You may wishto refine the questions to better relate tothe materials you are evaluating or teachersneeds. Use the same guidelines as aboveto write good questions.

Vlodifying the rating system: Many toolsare designed to give statistically reliabledata. Therefore, the rating proceduremay he more complex than you need.You may not even need a rating ystemunless you wish to compare among severalactivity guides. Instead you could makenotes about the strengths and weaknesses.

19

B. C.ating Your Own ToolTo develop your own evaluation tool, youmust first create a survey instrument outof the criteria you have chosen. Then youfigurc Jut how the answers to those ques-

tions will help you compare activity guides.

These two concepts go hand in hand.

Converting Criteria: The criteria sets in

Section II will help you think about thecriteria that are important to you. To turn

a generic criterion into an evaluation ques-tion, think about what you really want

to know and how teachers will use theinformation. Questions can be asked to

give yes/no answers, to help choose a num-

ber on an answer scale, or to give a narra-

tive response. For example, the criterion,"Activities encourage students to clarify

their own values" may be converted intoseveral questions: "Do activities guide stu-

dents in identifying value positions held by

themselves and others?" "To what extentdo activities allow students to form an opin-

ion?" or "To what extent are studentsencouraged to defend or challenge theiropinions?" Framing the criteria will take

some practice and you may wish to do soin conjunction with the next step.

Developing a Rating System: The ques-

tions must help you evaluate the materials.What will each answer mean? You can use

anything from a simple checklist to a com-

plicated ranking procedure. The following

common methods may be appropriate foryour use:

Cheek of`which criteria are met.This works best with yes/no questions.

Each activity can be assigned a score

and the scores averaged or addedtogether to give each guide a total score.

If some questions are more importantthan others, think about weighting somequestions in a summary score.

Use a narrative response style for thecriteria. This will provide more informa-

tion, but it is difficult when used to

compare many activity guides. One way

to present information is to group simi-lar responses together or to develop a

short annotation for each activity guide.If more than one person is evaluating,

you can indicate how often similarresponses were given.

The most common type of tool providesa range of responses for each criteria.In this method, criteria measure howwell the guide meets each criteria, ratherthan a yes/no type of response. Forexample, "On a scale of 1-5, whereI = not at all and 5 = very well, howwell does the activity guide incorporatedifferent learning styles?"

Most new evaluation tools will have a fewkinks to be worked out. We recommendthat you pilot test your tool on at least twoactivity guides. See if each criterion can beaddressed the way it is asked. If not, try tophrase it differently. Also, make sure the in-formation generated by each criterion is

useful in determining wb;:cher the activity

guide meets your goals and those of EE.

Step 3Compile Results

If you are evaluating ma:crials simply foryour own use, you may not need to com-pile the results into a publication or report.You could simply keep the rating sheetwith the activity guide or jot down noteson the strengths and weaknesses. If youdo wish to compile the results fc.r others

there are several approaches:

List only the materials that have met

a minimum number of criteria. Includea list of the criteria so that teacherscan determine whether the materialshave been rated on items that meet

their needs.

Present the ratings in a matrix thatshow how each guide met the criteria.This allows teachers to select materialsthat rate high on criteria that are impor-

tant for them.

Write an annotation for each guide thatshares the strengths and weaknesses. If

you compile results for others, be sureto explain your evaluation methodsand criteria.

20,Voy,,t,h;g) PeS lurce

Table 1: Sample Format for Compiling Results

Water, Precious Water Overall Score = 4This collection of lesson plans is a wellorganized series of laboratoryhands-on activities.

Report Card

Teaching and Learning A. Grades K 6

Presentation and Organization A Disciplines:Math, Science

Water Content. B+

Teaching and Learning: The labs emphasize the scientific processof investigation and offer many opportunities for inquiry. Suggestionsfor extended activities are included. Most teachers will want to choosefrom the wide variety.

Presentation and Organization: The directions to teachers are clearlywritten and contain well-defined objectives. The work pages are well-illustrated, uncluttered, and easily duplicated. The book's binding is alittle weak. No bibliography.

Water Content: The broad range of topics cover water awareness,the water cycle, water quality, water conservation, treatment anddistribution, and the properties of water. However, there is nothingon water resource problems in California.

Teacher's Thoughts: Students and teachers both will enjoyworking through these lessons which emphasize the quantitativenature of science.

Adapted from Compendium for Water Resources, California Department of Educations

Step 4Evaluate the EvaluationThink of your evaluation technique as anevolving process. The more you reflecton the results, the more you will increaseyour skills and produce better results. Toevaluate your evaluation, make sure thatyou have met the needs of teachers who areselecting EE activity guides. Indirect evalu-ation can not, however, guarantee that thematerials will work with students. When-ever possible maintain records of teacherfeedback on their experiences with the ma-terials in the classroom. Comparing this in-formation to your evaluation results willhelp you refine your criteria and methodsthe next time you evaluate materials.

Guidelines and Reminders:

Aim to identify strengths andweaknesses of the materials.

Establish a systematic procedure,follow, and document it.

Include a Not Applicable optionin your questions.

Pilot test your tool.

Check with teachers about theirreactions and advice.

Document your methods.

21

A Look at Formal Evaluation Projects

F YOU ARE USING someone else's evaluation results, you may wish tocritique their findings. First, examine their work closely to make surethe evaluation is relevant to your situation. Find out how the data weregathered. If this information is not apparent or detailed enough, contact

the authors.This section gives an overview of three common evaluation methods,

described in order of increasing complexity. Each is illustrated with a realapplication to help you critically review the evaluation results, understand thestrengths and weaknesses of each approach, and use the results of each methodappropriately. As you review these evaluation strategies, be sure to note themethods used, built-in assumptions, and the format used to report results.These and the factors listed below affect how applicable the results are to yourspecific needs:

Evaluators: Do evaluators have relevant experience? Were they familiarwith the activity guide's specific content and EE in general? Do they haveclassroom experience? Do they represent the teachers who might usethe evaluation results?

Criteria: Were they chosen to meet specific audience needs or generalaudience needs? Are criteria meaningful to you? Are criteria that areimportant to you given a high priority?

Results: Do the results provide enough information to allow you tomake an informed decision? Do the recommendations take intoaccount the context in which the guides will be used?

Be wary of many "evaluations" that include no critical analysis. Theyare usually reviews that list materials in categories, but without ratings. If youdo an ERIC (Educational Resources Information Center) search for evalua-tions of EE instructional materials, you will find the majority are reviews andformative evaluation projects. Comparative, quantitative evaluations that rateEE activity guides are fairly rare.

22

.Vorkshon Resource Manual

The Expert ReviewThe Expert Review method uses one ormore select people to conduct an armchairreview of instructional materials. Theexpert is most often an impartial externalevaluator who has experience with the cri-teria. The expert must have both environ-mental education and content expertise.

This approach offers several advantages.An external expert may provide an objec-tive iook at materials that are producedinternally. An expert review can provide ncomparative evaluation of many resourcesin a short amount of time. This approachalso offers a relatively low cost alternativeto developing staff expertise in EE andevaluation. When using the recommenda-tions from this type of review look for:

Bias The biggest disadvantage to theexpert review model is the limited per-spective one or a few persons can bringto the task. You run the risk of a heavilybiased evaluation.

Qualifications of Experts It may hedifficult to find one person with both con-tent and educational expertise.You maywish to research the qualifications of theexperts if they are not listed.

Support for Recommendations Thismethod yields an educator's interpretationof what will be instructive for students.Actually using the materials may yield dif-ferent results. You may wish to find first-hand knowledge of some of the materialsto double check the recommendations ofthe expert.

The Pacific Gas and ElectricTeacher's Resource Guide,Sacramento, California '

Pacific Gas and Electric (PG&E) recog-nized that the utility needed to evaluatethe materials they disseminate to educa-tors before ordering new materials andprinting another catalog. Since they didnot have an education specialist on staff,the company opted to condr:t an evalua-tion by contracting with an EE specialistfamiliar with energy issues. The expertthey contacted recently compiled a com-puterized collection of more than 300

nvironmpntal L 11;; Jt,cr

energy education resources and had writ-ten an energy education activity packetfor grades 4-6.6 The expert spent threedays evaluating the utility's library ofmaterials based on criteria used in otherevaluation projects. As a result of thiseffort, the expert recommended whichmaterials should be rewritten, expunged,reordered, or added to the collection.Based on these recommendations PG&Erewrote its teachers' guide.

The Group ReviewThe Group Review is an expansion of theexpert review. It uses a group of contentexperts, educators, and/or environmentaleducators. Their results can be tallied andstatistically analyzed to say with qualifiedcertainty that some materials are bettersuited to the chosen goals than others.Ideally, the evaluators represent a specificaudience, provide expertise across thediversity of materials, and provide a geo-graphical, cultural, and gender mix. Thisapproach is also an indirect evaluationalthough some of the evaluators may havefirst-hand classroom experience with thematerials. Things to look for when review-ing this type of evaluation project include:

Evaluators Do they represent the spec-trum of teachers that come to your work-shops? Does the mix of evaluators givethe breadth of expertise that represents therange of materials evaluated?

Number of Evaluators If there are notenough evaluators and responses don'tagree, it will not be possible to interpretdata with reliability. Resulting data couldhe misleading. The number of evaluatorsshould he noted along with the degree ofagreement between evaluators (often dis-played as a standard deviation).

Methods It is important that the evalua-tion tool was tested to assure that all eval-uators were evaluating in a similar fashion,interpreting criteria the same, and that allevaluations were accurately recorded andcompiled.

Results Are the criteria reported with theresults? How were the criteria determined?

23

ss

'I.'''.

Educatint; Yount: People .tboutGuide to Goals ,nul Resources

The United States Department of Agricul-ture Extension Service gathered a reviewteam of water education specialists to eval-

uate water education materials. Sixteenpeople from state and private agenciescombined their efforts to (among othergoals) summarize information about watereducation materials. The project focused

on nonformal education materials. Theresulting guide was developed to helpdevelopment of Cooperative Extensioneducation programs and materials.

This Water Quality Initiative Team sup-plied copies of instructional materials andrecommended other resources. They alsohelped identify water education goals andtopics. It was determined that due to thevolume of water education materials it

would be expedient to develop a short-term,initial project. The objective was to reviewand classify a selection of available materi-als as a basis for understanding what wasmissing and needed.

An initial selection for materialsto review was based on three criteria.Material was included in the review if it:

addressed one or more of the study'sgeneral or specific goals

presented a planned educationexperience

improved representation of regionalwater concerns, varied water topicsor environmental education goals.

The chosen materials were evaluatedagainst eight water topic criteria, five

EE goals criteria, and five format criteria.The intent was to see whether the watertopics, EE goals, and preferred format werepresent. The results were compiled to showoverall strengths and weaknesses in eachinstructional material. A summary of each

material provided information that willhelp educators select materials. The final

report presents information in four ways:

an alphabetized listing of materialsthat provides a brief description of the

material,

a listing of availability by stateor region,

a listing of materials from nationalorganizations or nationally availablesources, and

a matrix that shows review results.

The Pre-Test/Post-Test Review

Instructional materials sometimes fallshort of expectations. Directly evaluatingthe impacts of the activities is the mostaccurate way to determine whether thematerials meet the learning objectives.The pre/post approach is the only onedescribed here that evaluates materialsthrough direct interaction with students.This model dictates that students be

given a test before they have a plannedlearning experience with the activities tobe evaluated. After the lessens, studentstake a post-test, which is similar to thefirst test. Presumably an improved scoremeans that the learning experience wasa success. A control group completes thepre-test and post-test at the same timeas the treatment group to assure that theinstructional activity, and not anotherlearning situation, was the source of theimproved scores. Any increase in the con-trol group's post-test score results froma factor other than the instruction; thisincrease should be considered when com-puting the treatment groups gain.

Teachers chosen to evaluate the activi-ties are randomly assigned to treatmentand control groups. Activities are random-ly assigned to help increase accuracy of theevaluation results. The pre/post test meth-od is not usually used to compare amongactivity guides: it is most often used tomeasure the effectiveness of a single setof activities.

When reviewing the results of this

type of evaluation, be sure to consider thefollowing:

Test Construction In using a pre-test,you run the risk of alerting the studentsto the important concepts or simply raisingtheir curiosity about the new material.The tests should he similar but not exactor students may improve due to practice.

Biases in Teaching Teachers who arefamiliar with the activities could do a bet-ter job at teaching than ones that are not.Activities should be randomly assignedso teachers get a mix of familiar and newactivities.

BEST COPY AVAILABLE

24ManU,I

Significance of Learning Success is typi-cally defined as a better post-test score,based on the activities that are conducted.This will not guarantee that all activitiesare equally successful, or that broad goalsfor the materials are met that are notaddressed in these activities.

Project Learning Tree.Envimnmental Education Pre K-8.1ctivity Guide'

Pre/Post testing was the primary methodused to do a summative evaluation of the1994 Project Learning Tree (PLT) ActivityGuide. PLT contracted with the NorthAmerican Association for EnvironmentalEducation (NAAEE) to evaluate the effec-tiveness of the new activities. The projectentailed several professionals and a budgetof over $50,000.

Pre-tests were given in 120 classroomsin the grade-level groups of Pre K-1, 2-3,4-6, and 7-8. (Pre K -1 students performedtasks and their teachers reported resultson computerized answer sheets.) The testswere designed to measure students' knowl-edge and attitudes about one of severalthemes, including diversity, interrelation-ships, patterns, structure and scale, andsystems. About two days after administer-ing the pre-test, teachers in 60 classroomsused between three and seven randomlyassigned activities that addressed one ofthe five strands. Within a week or two,students in all 120 classrooms took thesame test again. All of the test data froma single classroom yielded a single scorefor that class. All the classes from the samegrade level that experienced activities ad-dressing the same PLT theme were aver-aged into one score. Teachers were alsoasked to keep a log, noting anything thatseemed significant as the evaluation pro-gressed. Logs are not always included inthis evaluation model, but can providesome significant and helpful information.This qualitative information helps the re-searchers interpret results more acct' -atelyand can provide new insights into the use-fulness of the materials.

nor(,,,n..,.t it f :,., !Ai Wiwi,

What to Do in a Workshop

0 NE OF THE BEST WAYS to build teachers' skill is to have them developa survey tool and use it to evaluate some materials. If you do nothave the luxury of a few extra hours, even a short overview onevaluation can he a valuable addition to any EE workshop. Below

are three scenarios to help you plan your workshops. The activities areprovided in Section VI.

Approach AProvide Evaluated Materialsand CriteriaProviding evaluated EE resources willreduce the effort teachers spend on select-ing instructional materials. Although thismethod often requires more of your timeand work before the workshop, it is oneof the simplest workshop strategies. Ifyou familiarize participants with the goals,criteria, and results of evaluations, theycan use this information to review otherinstructional materials.

You can either use resources that havebeen evaluated or evaluate the ones youhave. If you present materials that havebeen evaluated make sure the criteria,methods, and results are relevant to yourteachers. Be sure to provide and discussthese elements. If you evaluate the materi-als, consider using one of the evaluationtools supplied in Activity 3, page 28.(Sections II and III can help you modifythese tools.) When using this method toplan a workshop, consider the followingtips:

- Before the workshop, identify some ofthe teachers' key needs for resources andthe areas of EE they want to address.Use these needs to modify the evaluationtools.

Reserve time in the workshop for teach-ers to describe and discuss the resourcesthat have worked for them.

- Consider providing workshop partici-pants with the actual evaluation toolthat was used to evaluate the workshopmaterials. With the tool available, theyan decide which criteria are most im-

portant to them.

Example Agenda

I. Distribute handout listing evaluatedEE instructional materials

II. Overview of evaluation method

A. Criteria used to evaluate

B. Results of evaluation

III. Teachers share favorite materials

Approach BDeveloping CriteriaThe workshop strategy described inapproach A puts you in the position ofdoing most of the work. Although thisstrategy saves time for teachers, empower-ing them with a hands-on evaluation expe-rience may increase their commitment toevaluating ocher EE instructional materi-als. This approach builds teachers' interestin evaluation and helps them understandhow to best choose criteria that meet theirneeds. You might consider the followingsuggestions:

- Build a rationale for evaluating EEmaterials. Using Activity I on page 24,help teachers think about why theywould bother evaluating activity guides.Pair this activity with one of the twoideas below to give teachers the experi-ence of developing their own evaluationcriteria.

Introduce the topic with a short overview of evaluation and the goals ofF.E. You can provide them with ashort vet sion of the information onpages 4 7. After this discussion,have teachers brainstorm and prioritizea list of criteria. You may augmentthis list with Master 2.

Workshop Kesource Manual

Alternately, get teachers thinkingabout evaluating criteria by havingthem review an activity guide and askthem for their opinions. Help them seehow their "gut reaction" can help gen-erate criteria. Have teachers work ingroups to develop criteria for EEgoals, pedagogy, and presentation. Youmay refer them to Section 11. Activity 2details this strategy.

Example Agenda

I. Build a rationale (Activity 1)

II. Overview of evaluation concepts

A. Levels of evaluation

B. Indirect vs. Direct

Ill. Develop criteria (Activity 2)

IV. Teachers share their results

V. Teachers share their favorite materials:explain why they work

Approach CDo a Mini EvaluationOne of the best ways to prepare teachersto evaluate potential resources is to havethem evaluate an activity or a guide duringa workshop. While this approach is time-consuming, it provides teachers with theskills and practice necessary to becomeproficient evaluators. Given teachers'schedules, this may he the only time theyactually get to evaluate some of the mate-rials you give them.

Building on the activities suggested inthe approach above, take teachers to thenext step. Activities 3 and 4 provide strat-egies for leading teachers through an eval-uation process. As you develop theworkshop, consider these ideas:

1 ,1 warily ,onmolt ii taL. at co Mdtertdis

Give participants an opportunity todevelop their own criteria. You canstreamline these strategies by havingteachers review the EE guides whileasking: How will students react tothe activities? What will the activitieshelp them learn? How will the activityguides affect student behaviors andparticipation in environmental issues?

Divide into groups and have eachgroup evaluate the same activity guidebut with different tools. Discuss thestrengths and weaknesses of the tools.

ff One of the most preciousgifts you can give to teachersis time to plan how to use thenew information it,dy acquirefrom an inservice workshop."Margaret PennockTeacher, Mast Landing School. Freeport.Maine

You can use this feedback to mod-ify the tools.

Divide teachers into groups. Eachgroup can evaluate a different guidewith the best tool. Compile the resultsfrom group reports.

Provide time for teachers to talk aboutwhich resources have worked well forthem.

27

Example Agenda

I. Build a rationale for evaluation (Activity 1)

II. Brainstorm the goals of EE instructionalmaterials

III. Develop criteria (Activity 2 )

IV. Evaluate an activity or activity guide(Activ,'.y 3)

A. Groups do same activity guide

B. Groups ao different activity guides

V. Compile results

VI. Discuss how to adapt tools

VII.Teachers share their favorite materials

SummaryThere is no shortage of EE materials fromwhich to choose. Only you can reallydetermine which instructional materialsbest meet your needs and the needs of theteachers you serve. Helping you makethose decisions accurately and quickly isa major goal of this unit. We have lookedat some of the general principles of evalua-tion so you might create evaluation strate-gies that yield accurate and usefulinformation. In this process we have dis-cussed selecting criteria and how to bestapply them to the materials you wish toevaluate. Interpreting your results andevaluating your process are the final stepsof this procedure. We also hope you gainedsome skills to interpret the result of themore formal evaluation efforts you will en-counter.

If you find this systematic evaluationprocess valuable, you will want to shareit with the teachers with whom you work.The background information, the work-shop guidelines, and the activities providedwill help you encourage teachers to criti-cally evaluate the instructional materialsthey have and will encounter in the future.Ideally, we will all become informed con-sumers and disseminators of high-qualityFE instructional materials.

26t I ,

Activities and Masters

THESE WORKSHOP ACTIVITIES AND SAMPI,ETOOLS will help you design

a short segment or develop an entire workshop that focuses on eval-uating EE materials. Using Section V as a guide, you can build theseactivities into a module that fits your participants' needs.

Activities

The first two activities help develop an overview of the topic of evaluation bydefining the goals of evaluating materials and what it means to evaluate.

0 Building a RationaleThis activity helps participants justifyan armchair review of EE materials.Teachers discuss benefits and goalsfor evaluating instructional materials.

0 What is Evaluation?To introduce teachers to the conceptof evaluation, this activity asksteachers to develop a framework forevaluating EE activity guides. Theoverview of EE goals and evaluationcriteria categories will help teachersuse evaluation tools and resultsof evaluation efforts effectively. Twomasters for overheads are included.

Immersing teachers in an evaluation project will help them gain both a betterunderstanding of the process of evaluation and confidence in their ability toeffectively evaluate new materials. Activities 3, 4, and 5 will also leave teacherswith a set of strategies and tools that suit their particular needs.

()Trying Out Some ToolsUsing three sample questionnaires,participants critique and modify aninstrument to help identify materialsthat meet their needs.

0 Learning to LookTeachers will compare activities thatcontain similar content but differentteaching methods. Participants willdiscuss aspects of pedagogy that willaffect evaluations of these types ofinstructional materials.

..1,U,Ibor f r,,,,vorirvnt.1 1 44.itcrials

()What About Bias?Teachers discuss the inevitabilityof bias, then practice discoveringit in materials. Five strategies fordealing with bias are provided.

29

Activity 0 Building a Rationale

This activity helps

participants justifyan armchair review

of EE materials.

Teachers discuss

benefits and theirgoals for evaluatinginstructional materials.

ObjectivesTo help teachers identify goals fortheir evaluation.

Outline

Set the stage by asking teachers to imaginethat they will be given a complete set ofnew resources for the school year. Althoughthey are not familiar with these EE materi-als, they are all good teachers and knowwhat works in the cIassroom. They will beusing their expertise to evaluate these newmaterials, and to determine how to usethem. But first, they should discuss whatthey will be looking for.

1

Divide participants into smaller groupsof three to five. Ask them to appoint afacilitator, recorder, and timekeeper. Giveeach group a piece of newsprint and afew markers.

2

Give the groups 15 minutes to come upwith a list of goals for EE instructionalmaterials and record them on paper. Youmight find these questions help stimulategood discussion:

What do you want teaching materialsto accomplish?

What do you look for in a pieceof instructional material?

What makes activities a successwith students?

What do you need from new materialsto help you teach well?

MaterialsMarkers, tape.

Time

30 minutes

3

Have each group post their report on thewall. Invite all participants to walk aroundthe room viewing each others' goals.

4Conclude by leading a large group discus-sion on the similarities and differencesof each group's goals. Invite participantsto think about the value of taking timeto evaluate materials by asking them thefollowing questions:

What are the basic goals of evaluation?

How can evaluation help you meet yourteaching goals?

How can you use this exercise to helpyou think about evaluation?

30 aksnoo 'ource Manua/

Activity 4 What is Evaluation?

To introduce teachers

the c.;;nceot of

.,valuation, this

.ictivity asks teachers

develop a frame-

.:OrK for evaluating

EE activity guides.

An overview of EE

oais aria evaluation.riteria cateories

heir) teachers

.rise evaluation tools

and the results of

evaluation efforts

more effective/y

ObjectivesTo introduce teachers to the conceptof evaluation; to provide a frameworkfor evaluating EE activity guides.

Outline

Divide participants into groups.of threeor four. Thu may group teachers randomly,or by grade level or content specialty.Explain to the teachers that tr--v will eval-uate activity guides.

1

Give each group copies of an activityguide. NOTE: It you wish to have teach-ers bring their favorite activity guides toevaluate, ask them to work in pairs, usingone guide between them.

2

Give the groups 10 minutes to quicklyreview the guide. Have teachers indepen-dently list their impressions of the activityguide, both good and had. Teachers thenshare their impressions with their groupor partner. Each group should recordtheir reactions to the material.

3

Have each group report their results.Record their comments on a newsprintor overhead. Lead a discussion usingthe following questions:

How were their results different? Why?

On what grounds do they base theirreaction to the material?

If they had more time, what else wouldthey consider?

311 sJiudtaw I In ironmenta I t ducdt ion Mite, ,)

MaterialsA set of activity guides for each groupof three to four participants, news-print, markers, and overhead Masters1 and 2, if you wish.

Time

30 minutes

Using overhead Master 1 and the back-ground on pages 4 5, explain the levelsof evaluation. Identify which oftheir reactions relate to each level.

4

Using background from pages 7 9 andoverhead Master 2, provide an overviewthe three categories of criteria: EE goals,Pedagogy, and Presentation and Design.Discuss what criteria the groups used. Youcan stimulate the discussion by asking:

What questions did you ask yourself?

On which aspects of the material didyou focus?

5

Have each group put their evaluationcriteria into the categories. Have themdiscuss any gaps they see and developadditional criteria to fill these gaps.Have them review the activity guide asecond time to rate how it meets theirnew criteria set.

6Conclude by asking teachers to discuss:

How is evaluation different fromjudging on opinion alone?

How can they apply this knowledgeto looking at new EE materials in thefuture?

Master n Levels of Evaluation

Reaction Evaluation

measures immediate response

to the learning experience (students)

to the materials (teachers)

Learning Evaluation

looks for changes in:

knowledge

awareness of environmental issues

attitudes

Behavior Evaluation

looks for changes in:

students' skillsstudent behaviors

Results Evaluation

measures impact on:

the environment

students' lives

32Li..unnmpl,i I .

k rpt) lit%rut r Minna

Master Criteria Categories

Goals of EE

Awareness

Knowledge

Attitudes

Skills

Participation

Pedagogy

Teaching and Learninginterdisciplinaryage appropriatecritical thinkingcooperative learningassessment

Bias

Integration into Curriculum

Presentation and Design

Format

Writing

Appearance

Organization

331..1 t . 1111'" o'r00,1).0 V V 'WWI

ti

Activity 0 Trying Out Some Tools

h.7 sarnaie

:L.E.,st:cnnaires,

crThaue

'ems ana modify.:3:niment to

identify

-3ter,ais that meet-oecs.

ObjectivesTo help participants recognize theimportance of how evaluation ques-tions are worded and the length ofevaluation tools. To give participantspractice in using several evaluationtools. To have participants modifyan instrument to meet their evaluationneeds.

Outline

Participants will practice using differentevaluation tools to evaluate the same pieceof instructional material. This will givethem insight into how different criteria,questions, and rating strategies can affectthe results. They will use this insight toadapt the tools for their use.

1

Distribute a copy of each sample tool andeither one activity or a complete activityguide. Ask each team to evaluate theirmaterial with each of the three tools. Havethem discuss among themselves which toolthey liked best and why.

2When teams finish their reviews, lead adiscussion with the following questions:

Did the evaluations result in equivalentinformation?

If nor, how did they vary?

On which criteria arc the answersmost varied? Why?

What questions did they have aboutusing the tools?

Which tool did they like best and why?

MaterialsOne activity guide for each group orcopies of activities. Copies of each ofthe sample evaluation tools providedin Masters 3, 4, and 5.

TimeOne to two hours with optionalactivities

3(optional)

Have each group list the characteristicsof the tools they liked best. Ask them toconsider the types of questions, ratingstrategy, case of use and how it related totheir teaching needs. Using these qualities,have the group modify one of the toolsor develop a new one based on their needs.

4(optional)

Have each group share their new tool,presenting the characteristics, typesof questions, and comparison strategyto the whole group. If possible, makecopies of each new tool for all groups.

34Irkshoo Resource Manual

Master El The EE Curriculum Inventory Tool

Directions: Indicate how well the inventoried materials will helpstudents to know about or accomplish the statements for each goal level.Circle the appropriate number for each statement according to the scalebelow. Compute the Goal Level Average as directed below.

Evaluation Scale: Not at all Poorly Slightly Fairly Well Very Well

0 1 2 3 4

Goal Level 1: Ecological Foundations (Knowledge)

1. Populations: The nature/behavior of natural populations(birth, growth, change & its effects, death, extinction, etc.)

2. Natural Communities: Structure behavior, interactionof individuals and populations in a particular place (forest,desert, sea, pond, etc., food chain, food web, predation, etc.)

3. Ecosystems: Structure. interaction, and effects betweenliving things and the nonliving environment (natural change,succession, adaptation, etc.)

4. Energy & Chemical Cycles: How energy, chemicals, andpollutants go through a natural system (water, oxygen,energy, carbon, etc.)

5. Balance of Nature: The tendency of natural systems to establish balance over time when disturbed naturally or by humanpollution.

6. Humans as a Component of Natural Systems: The effectof human behavior on natural systems. (Development oftechnology, effects of people's attitudes, and values.)

Ecological Foundations Total: Add all circled numbers.

Ecological Foundations Average: Divide by 6.

35

Rating

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

i .aluatirt. itniner,t,11 Hut ,thon 1,0,_rwis 0,1i7e rky,, ;, ,urt,. VdnuJ,

Master Ei Page 2

Goal Level 2: Issues and Values (Knowledge) Rating

7. Identifies the difference or discrepancy between an environ-mental problem and issue. (Environmental problems maybe natural or human-caused while issues are the conflictsarising from how people view the problem).

8. Communicate the way people's culture affects the environ.ment (politics. society, religion. etc.).

9. Communicate the way individual people affect the environ.ment (individual actions and behavior).

10. Identifies (clarifies) one's own attitudes and values towardan issue or the environment.

11. Communicates that there may be more than one wayto resolve an environmental issue.

12. Communicates that the difficulty in resolving environmentalissues stems from the different attitudes and values of thepeople involved in them.

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

0 1 2 3 4

Issues and Values Total Add all circled numbers.

Rating

Issues and Values Average Divide by 6.

Goal Level 3: Investigation and Evaluation (Skills, participation)

13. Uses problem-solving skills to identify specific environmental 0 1 2 3 4problems and related issues.

14. Identifies the people (actors) involved in an environmental 0 1 2 3 4issue.

15. Identifies the beliefs and values that explain people's 0 1 2 3 4positions on an environmental issue.

16. Examines a variety of environmental issues and their 0 1 2 3 4solutions.

17. Identifies the ecological costs and benefits of designated 0 1 2 3 4solutions to environmental problems and issues.

18. Identifies the human costs and 'benefits of designated 0 1 2 3 4solutions to environmental prob ems and issues.

Investigation and Evaluation Total: Add all circled numbers.

Investigation and Evaluation Average: Divide by 6.

36. Putlt,'W itOn rkstv.0 Mantur,

" )4eMaster El

Goal Level 4: Environmental Action(Knowledge, skills, participation)

Rating

19. Communicates the need for responsible citizen action 0 1 2 3 4to resolve environmental issues.

20. Communicates that there are various levels of environmental 0 1 2 3 4action (individuals, groups, organizations).

21. Identifies examples of environmental action categories 0 1 2 3 4(persuasion, consumerism, political action, legal action,ecomanagement).

22. Examines scenarios/case studies that would allow the 0 1 2 3 4student to apply knowledge of environmental action andin choosing responsible environmental action.

23. Identifies the human and ecological costs and benefits 0 1 2 3 4to identified environmental actions.

24. Takes individual and/or group action on an environmental 0 1 2 3 4problem they have identified and analyzed.

Environmental Action Total: Add all numbers circled

Rating

Environmental Action Average: Divide by 6.

Goal Level 5: Teaching Methods and Practice

25. Do the materials provide specific training in the use 0 1 2 3 4of problem solving skills such as observing, inferring,predicting, classifying, hypothesizing?

26. Do the materials provide opportunities to apply problem 0 1 2 3 4solving skills with simulated models, real lift, situations,scenarios, etc.?

27. Are the materials educationally appropriate for the intended 0 1 2 3 4student users (reading level, conceptual and skill levels, etc.)?

28. Would the materials be attractive and appealing enough 0 1 2 3 4to gain students' interest?

29. Do the materials contain goals and measurable objectives? 0 1 2 3 4

30. Do the materials provide sufficient suggestions or mecha- 0 1 2 3 4nisms for evaluation?

31. Is the information provided for students and teachers 0 1 2 3 4accurate?

32. Do the materials provide sufficient background information 0 1 2 3 4for the teacher?

33. Do the materials provide clear directions and suggestions 0 1 2 3 4for teaching?

Teaching Methods Total: _Add all numbers circled

Teaching Methods Average: Divide by 9.

37.1 uat ot' '1""n, "' t'ot.) R.out( c

Master

4

3

2

1

Page 4

Summary Data

Evaluator's Name

Title of Curriculum Materials

Date

Author

Publisher

Summary Graph

Directions: You may present the results of your evaluation on thegraph below (with the averages on pages 1. 2, and 3.) Severalpeople may also choose to combine and average their scores foreach curriculum so that a group score can be graphed. Shade thebar that corresponds to each goal.

0

Ecological ssues nvestigation Environmental Teaching

Foundations and Values Evaluation Action Methods

Average Average Average Average Average

Adapted and used with permIsston..I.R. Garde lla. 1993 Hghlann Heirhts. KY:School of Education. Northern Kentucky University

tl.aluatinF f%irnnmental Eu:icatfon Mater fat,:

Copyright Date

Resource k4,4nua,

Master ri Development andEE Resources Tool

Resource:

Rate As: "1" Meets criterion"2" Some attention to criterion"3" No attention to criterion"NA" Not Applicable

Knowledge CriteriaResources should contain essential elements of core concepts.

Development Issues

Sustainable DevelopmentPresents the need for balance between economic growth andsocial development in the interests of sustainable develop-ment and overcoming poverty and social injustice.

Environment and DevelopmentPresents the importance of ecological balance against theeffects of environmental destruction and the need to balanceeconomic growth and conservation through a process ofsustainable development. Presents the importance of biodi-versity sustainability.

Rights and DevelopmentPresents the individual rights and responsibilities as citizensof local, national, and global communities. Supports the needto eliminate discrimination and other threats to human rightsfrom various sources. Presents the rights of all species asessential for biodiversity and sustainability.

Green AidPresents aid projects that promote economic development inan environmentally responsible manner. Presents benefits toboth donors and recipients.

Global PerspectivePresents the interdependence of global economic, social and po-litical systems. A selected global perspective reflects the princi-ples of economic justice, human rights, sustainable developmentand peace.

Comments:

39

Rating

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

, ..,ourte .'.111thP

Master

Critical Skills CriteriaResources should help participants challenge bias, support ratio-nal decision making, examine solutions and prospects for change.

Balance Issues

ComprehensivenessIncludes a wide range of social-cultural viewpoints. Sensitiveto explanations of social structure. Can distinguish betweendescription and exlanation.

StereotypingPresents a range of roles and human characteristics for anygroup. In-depth portrayals of sub-groups include accuratepresentations of the group culture, from the point of view ofthat group. Sub.groups portrayed active participants in theircommunity in a variety of decision making situations.

Misrepresentation and ValidityDoes not present unsubstantiated 'factual' claims as 'truths.'Sources are verifiable, accurate, and up-to-date. Does notinclude misleading implications.

TechnologyDoes not imply that the world's major problems can besolved by further development of industry and technology.Identifies technology as part of the technical-social complexof productive forces. Technological solutions to social issuesare seen in their political, economic, and social context.

InterdependencePresents an issue in its connection with the wider ecological.economic, political, and social environment. Presents individ-uals and communities as linked.

Decision MakingPresents conditions for rational decision-making method.Emphasizes inquiry process of hypothesis and generalization.cooperative learning, critical reflection, dissemination,andinvolvement and action.

Problem SolvingEvaluates the relevance, validity, and implications of alterna-tive solutions. Presents the interests and values underlyingvarious solutions. Supports a multi-disciplinary approach.

Comments:

40mivallt)r t nornnt-nr.?. I : Tit, f, t NI

Rating

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

1 2 3 NA

rkEnnn key,urce P.1.1nual

Master Page 3

RatingAttitudes CriteriaResources should help promote understanding and solidarityamong all peoples.

1 2 3 NA

EmpathySensitive to the viewpoints and feelings of other people,particularly those belonging to groups. cultures, and nationsdifferent from our own.

1 2 3 NA

RespectSupports the developriient of selfrespect and respectfor others, particularly those with national, cultural, andfamily background different from our own.

1 2 3 NA

ParticipationSupports solidan4y with victims of injustice in one's own andother societies. Supports participation in political decision-making at local, national. and international levels. Supportssolutions of major world issues through cooperation at levelsof individuals. organizations. and nations.

1 2 3 NA

Comments:

Values Criteria RatingResources should help participants develop a commitment toworking with and for others in the interests of equality, equity,and sustainability.

1 2 3 NA

Social JusticeExamines inequalities of wealth and power. Presents effortsto redress them through development, aid, trade, and equity.

1 2 3 NA

Equality of PeoplesPresents the principles of equality and democracy whichunderlie a belief in the equality of all peoples, as the rightof all peoples to life, freedom, and self-determination.

1 2 3 NA

Ecological SustainabilityPresents biodiversity as a right of all living species and as abasic principle of sustainability. Presents intra-generationaland inter.generational equity as a right of present and futuregenerations and as a basic principle of sustainability.

1 2 3 NA

Presents the satisfaction of essential human needs and aspir-ations as a priority and as a basic tenet of development.

Comments:

Used with permission and adapted from "Analyzing Education Resources" by Zita Unger, 1992 In Tgachine for,Sustainable World: Environmental and Development Education Project for Teacher Education. John Fien. editor.Brisbane. Australia: Griffith University Press.

41Puatalt! t ,,%onrionf,i, f 1u dUOn N'aturtaP,

Master el California Compendium Tool

The following evaluation tool used by the teachers to rate materialswas adapted from the California Science Framework, Chapter 8,"Instructional Materials Criteria."

a b c d e=not applicableexcellent poor

A. General Content

1. Are ideas expressed through unifying themes and big ideas. without facts?

2. Is content interdisciplinary?

Are students challenged to utilize higherlevel thinking processes(i.e., inferring, relating, and applying)?

4. Are ideas presented logically and connected through the curriculum?

5. Is depth of understanding emphasized (rather than encyclopedic breadth)?

6. Are historical, ethical, cultural, geographic, economic, andsociopolitical relationships addressed?

7. Are knowledge and learning shown to be connected to students' lives and society?

3.

nn

I I

I

I I

I I

B. Presentation

1. Are instructional materials clearly and engagingly written with the main conceptswell articulated?

2. Are the roles of environmental ethics, citizenship, and stewardship explored?

3. Do lessons promote respect and caring for the environment,yet be non.dogmatic and open to inquiry and differences of opinion?

4. Are personal and societal values and conflicting points of view explored in context?

5. Are instructional materials easy for students to use and understand?

6. Is learning made accessible to all students?

7. Are writing and concepts developmentally appropriate for thedesignated grade, yet sensitive to individual differences in educationalexperience and learning mode?

8. Is environmental responsibility modeled in design, underlyingphilosophy, and suggested activities by the lessons and materials(e.g. using recycled materials and properly disposing of wastes)?

9. Are there clear linkages presented between communities of all levels("thinking globally, acting locally")?

10. Are vocabulary words defined in context and not obscuring learning goals?

11. Is the layout of instructional materials interesting and appealing?

BEST COPY AVAILABLE

42 ,VnrOc(hool;.0S91),.

Master 13

C. Pedagogy

1. Does almost half of the curriculum have students engage in active learning?

2. Do students construct knowledge through research, discussion,and application to gain conceptual understanding?

3. Are evaluation devices included and appropriate? (Highest pointsfor authentic, performance based assessment devices.)

Are instructional materials sensitive to social, economic, and cultural diversity?

Do lessons encourage students to develop awareness, knowledge.and strategies for responsible action?

Are group/cooperative learning strategies used?

Is intergenerational responsibility, linking today's actionswith future consequences. implicit in instruction?

4.

5.

LJ 6.

7.

HI

11

I I

:

D. Presentation

1. Are instructions for the teacher clear and concise?

2. Are lesson objectives and outcomes clear and appropriate?

3. Are materials easily integrated into an established curriculum?

4. Is background information for the teacher adequate and accurate?

5. Can the materials be adapted to varied learning environments(large or small classes, of mixed levels, from rural or urban settings)?

6. Are consumable instructional materials of good quality, easily duplicatedfor student use, and in sufficient quantity to support the objectives?

7. Are equipment and materials listed and reasonably accessible?

8. Are a variety of instructional strategies, expanded learningenvironments. and resources suggested in the curriculum's design?

9. Is the time required to complete each lesson indicated?

10. Do the materials clearly list the subject discipline(s) integratedinto each lesson?

Used and adapted with permission by the California Department of Education from theEnvironmental Education Compendium for Water Resources, The California Departmentof Education. The California Department of Water Resources: Sonoma State University.March 1992.

43

Activity 0 Learning to Look

ObjectivesTo introduce teachers to the power ofpedagogy and how it will effect theirevaluation.

Preparation

This activity suggests teachers comparetwo EE activities for students. One pairis provided, though you may substituteanother from your own materials. If so,look for activities that are designed forthe same age level, cover the same content,and have similar learning objectives, butuse different teaching methods to reachthe goals. For example, the activities onMasters 6 and 7 look at the value of farm-lands but use slightly different approaches."Down on the Farm" uses simulation anddiscussion. "Vanishing Farms" uses simu-lation and role play.

Other potential sets of activities are:

"Water, Water Everywhere" from EarthMatters, "How Wet is Our Planet" fromAquatic WILD, and "Eighty-six Gallonsper Day" from Living Lightly on the Planet:

"Bye Bye Birdie" from Earth Matters byZPG and "Here Today, Gone Tomorrow"from Project WILD:

"Watch on Wetlands" from Project Learn-ing Tree and "The Great Swamp Debate"in Nature Scope's Wading into Wetlands.

MaterialsActivities from Master 6 and 7 ortwo activities that cover the sametopic. Optional: Copies of Master3, 4, and 5.

Time40 minutes

Outline

Start a discussion among teachers aboutpedagogy and the design of instructionalmaterials. Ask them to think about howthese concepts affect the way we evaluatean activity or activity guide. Explain thata single activity rarely incorporates all thegoals in the instructional guide. Duplicatethe activities so each participant receivesa pair.

1

Divide the participants into teams of fourto five. Hand out copies of Masters 6 and7 to everyone. These activities have similarcontent but different objectives. Challengethe teams to critique both activities, notingthe teaching methods, student outcomes,and ultimate objectives. You may wish tohave the teams use one or more of the eval-uation tools from Masters 3, 4, or 5.

2

Lead a discussion about the differencein the two materials using the followingquestions:

Would each activity meet yourteaching objectives? Why or why not?

What makes a good learningexperience?

What criteria would you use toevaluate each activity?

What criteria ought you use for thepedagogical aspect of the materials?

44 .; irkcn_,11 Roso,.r(o Manudi

Master Down On The Farm

Objectives:

Students will:

Measure and mark off an area of land which represents the amount of land requiredto feed them for one year.

Investigate the conflict over private vs. public control of farmland resulting fromagricultural zoning.

Materials:

Student information sheet provided: a long tape measure (at least 71 yards long)or, for a handy measuring device.use a yardstick, four small nails, 71 yards of kitestring, and a hammer. You will also need four stakes marked with brightly coloredpaint or material.

Advance Preparation:To make the measuring device, pound two small nails into each end of a yardstick.Tie one end of the string to the nails at one end of the stick.

Overview:It takes approximately 1.2 acres of land to feed one person each year. This includesgrazing land for dairy and beef cattle as well as cropland to grow grains, fruits, and vege-tables. Between 1967 and 1973 the amount of cropland and potential cropland converted to urban uses was 875.000 acres per year. This rate of cropland conversion is causingsome concern in that we may be presenting future generations with a shortage of pro.ductive land. In an effort to stem the tide of cropland conversion, many communities aredeveloping land use plans which include zoning laws to preserve farmland. The conflictthat these laws create will be debated by our students as they investigate this issue.

1. Ask students how much land they think would be required to feed each of them for aone-year period. Round the 1.2 acre figure off to one acre and tale yourstudents outto measure an acre of land on your school grounds. An acre is 4,840 square yards(4,047 square meters). A square 69 yards, 20.5 inches on each side is an acre. (If theland you have to work with is not large enough, measure a one-half acre area. A halfacre square is 49 yards, 7 inches on each side.)

If you use the measuring device suggested above, have students wrap the stringaround the yardstick. Each loop around the length of the yardstick equals two yardsof string. Have them wind the string until they have 69 yards, 20.5 inches and cut thestring. Then use the string like a tape measure to mark off the four corners of theacre. Have some students hammer a stake into the ground at each corner.

2. Discuss the figures below that indicate what is happening to our individual shareof cropland on a global basis.

Year Cropland per person1650 4 acres

1975 1 acres

2135 1/4 acres (projected)

The decline of arable land per person is projected to decrease as the world populationcontinues to expand. Most of the land suitable for farming is now in use. Only 11 per-cent of the world land is arable. The rest is too steep or cold or too dry to supportagriculture.

liudtt L' L Imeot t ; . ;It on ..11ton.n.

45,,) 4.9 I. fiCS(,111,C Mdri, r

Master 6

Reprinted with permissionfrom Lime lightly on MCPlanet, through arrange-ment with Schlitz Audubon Center of the NationalAudubon Society, 1111 E.Brown Deer Road. Milwau-kee, WI 53217

.age 2

3. Either read the information on the student sheet or distribute copies for studentsto read on their own. Discuss the issue of agricultural zoning and its infringementon individual property owner's right to sell their land as they choose. Review thediscussion questions on the student's sheet and have students express their ownpoints of view on the issue.

Extensions:Write the following statement on the board: "Asphalt is the land's last crop." (The state-ment was made by M. Rupert Cutler, former United States Assistant Secretary of Agri-culture. in reference to the problem of vanishing farmland.) Have students use this asan introductory sentence in an essay expressing their views on the subject of farmlandconversion.

To help your students understand the special nature of topsoil, have them plant cornsleds in a container of topsoil and in a container of subsoil. Subsoil is the layer of nutri-ent-poor soil to be found under humus-rich topsoil. It provides a poor growing mediumand will help students to realize that not all !and is suitable for growing food.

Student InformationHow would you react to this situation if it were taking place in your community?

Voices are rising along with tempers in the crowded room in City Hall. There is a publichearing being held to give citizens a chance to express their views about the proposednew zoning laws. Zoning laws set aside areas of land for specific uses. When a communi-ty has zoning laws, some areas are zoned for homes, others for industry, and other ar-eas are zoned for open space. In this community new zoning laws will restrict the usesof farmland. There is a hot debate taking place between people who are divided in theiropinions.

The proposed zoning would require that farmland be sold in tracts no smaller than 40acres and that the lands remain as open space or farmland.

The community is near a growing city. The trend has been for farmland to be sold anddivided into subdivisions and shopping malls. In many cases, it was more profitable forfarmers to sell their land to developers than to continue to farm. Many farmers are an-gered over the proposed denial of their right to sell their own land for uses other thanfarming. They believe that people should have the right to do what they want with theirown land.

Other people in the room are concerned about the community's loss of farmland. Theyview the land as a valuable, irreplaceable resource. They argue that farmland should notbe viewed as something to be bought or sold, but as a resource to be preserved for fu-ture generations They believe that preservation of farmland is a social obligation. Thisleads them to believe that the public, in this instance, has the right to exercise controlover these valuable lands. The debate rages on.... What do you think?

Discussion Questions

How would you react to the proposed zoning restrictions if you were a farmer?A housing developer?

Do you believe that the public, in the form of government, has the right to imposerestrictions on individual landowners? State reasons for you beliefs.

Should land use decisions we make today consider the needs of generations whichwill follow us? Why or why not?

46

,.1111,1t+rs 71( "t i f .

Master Ei Vanishing Farms

Objective:Students will gain an understanding of the pressures to develop open space and farms.

Summary:Students will role-play farmer, crops, farm animals and the different types of intereststhat also desire land.

Time:45 minutes

Student Grouping:Whole class

Materials:10.30 foot piece of string with ends tied together to make a circle. (Longer string forlarger groups)

Background Information:Every year three million acres of farmland are lost to some sort of development. If thiscontinues, eventually there will not be enough land to grow food. Localities have begun tomake ordinances and laws that facilitate maintaining agricultural land. The issue of private rights versus the common good is brought into play in these situations.

Marin County, California, Agriculture Facts:The debate over land use is very real to Marin residents. For years there have been effortsto preserve county agricultural lands. Many people have been involved, particularly sincethe 1970's, in making it feasible for farmers to keep farming in spite of the skyrocketingprices of land in the area. The Williamson Act, passed in 1965 and amended in 1970,allows for reduced property taxes for agricultural lands within certain areas. MALT, theMarin Agricultural Land Trust, purchases agricultural easements, restricting future usesof property to agriculture and recreation. Without these types of provisions, much moredevelopment would be inevitable.

Preparation:1. This is a very simple activity. You Just need to be familiar with the exercise so it will

flow smoothly in class.

2. Acquire your string and decide where you will lay it out.

Procedure:1. Place the string on the ground in a circle.

2. Assign one student the role of "Farmer."

3. The farmer lists the crops and livestock he would need in order to feed and clothethe local people. As the farmer names the things on his list (like corn, wheat, cows,sheep, fruit) a student is selected to step into the circle and represent the commoditynamed.

4. One by one the rest of the students become people who also want to use the farm.land: for example, a road builder, someone developing a shopping center, someonedigging for water, a school builder, a hotel owner. These people move to the edgeof the farmland, pick up the string and move it inward, making the circle smaller.

47..',)',,,,111,,..c.P. C.

Master

Used with permission andadapted from "VanishingFarms" by LeeannTourtillott and Ellen Rata.1992. In Food forThought: AgriculturalClassroom Activities forGrowing Minds, Universityof California CooperativeExtension, Mann Agricultural Literacy Project.

Page 2

5. Developers continue until the farm circle is too small to hold all the farmers livestockand crops. There has been too much development in the area and thus the farm hasvanished.

6. Discuss the activity with the students.

Questions for Discussion:How does development pressure affect food production?

Would this affect prices?

What can all those developers do?

What can farmers do?

How does development and loss of farmland affect the families in the community?

Extensions:Have a farmer and developer come to the class and talk about their ideal planfor land use in the area.

Find some oid pictures of the school area (check with a local historical societyor local paper) and compare the open space then and now.

How does Marin Agricultural Land Trust work?Marin Agricultural Land Trust (MALT) seeks to permanently protect Marin's agriculturallands by purchasing agricultural easements from ranchers.

An easement is a legal property right and can address many of the ways in which landcan be used. An agricultural easement is one which allow the land to be used for agricul-tural purposes. MALT buys agricultural easements and then permanently assures thatthey are not sold off to allow subdivision of the land for development.

Agricultural easements are always sold willingly by ranchers to allow them to get cashfor farming operations, to make improvements, or meet other needs. The farmer sellingthe agricultural easement still owns the property, and the land is guaranteed to stay inagricultural production.

MALT was formed by a coalition of ranchers and environmentalists in 1980 andis considered a national model for agricultural land preservation.

: . nuatinn t nonnmental j (I aton Mater:a, :art..1.' 46 Nnrkshon b'woun e Manual

Activity 0 What About Bias?

reacners discuss the

nevitabilitv of bias

9nd practice c9scov-

t,rin2 it in materials.

ObjectivesTo enable teachers to detect biasin teaching materials and help themto develop strategies that overcomebiases in EE instructional materials.

Outline

Begin by explaining that all materials havesome bias toward the author's, producer's,or funder's viewpoints. Whether from in-dustry, textbook companies, government,or environmental organizations, most ma-terial includes a particular bias.

1

Have teachers brainstorm the types of biasthat they have detected in instructionalmaterials. Some possible answers are: gen-der, age, culture, resource use vs. preserva-tion.

2

Divide the participants into small groupsand distribute Masters 6 and 7 to each.Ask the groups to list examples of wherethe author's bias comes through.

3

Have groups share their insights. Notewhether the groups came up with the samebiases and examples. If there are differenc-es, ask the group to consider how theirpersonal biases affected their perception ofthe materials.

4q1 Iluat , nmr,,,kr at,on Mater,a,,,

MaterialsHandouts made from Masters 6and 7.

Time30 minutes

4Ask how teachers might approach materi-als with bias. Add to their list any of thefollowing:

Not use the material

Accept bias as is

Balance this with other materialsof an opposite bias

Have students discover the biasas a learning experience

Modify materials to reduce bias

Have each group discuss what they mightdo with these activities given the bias theyfound. Record all strategies on an over-head or newsprint.

5(Optional)

Ask participants to generate questions foran evaluation tool that would address theirconcerns about bias.

7. tOCOK oef!nfoons

nay oe neioluf to

:;r0v10iPZ: aLldft,C11.3;

these

T.7ese Germ,-

ons refer t.o

j170

:-:rms cc-Kr:Jai !IIt.net.

Resources

Glossary

Evaluation Tool/Instrument The test,questionnaire, or checklist used to measurethe performance of the instructional mate-rials to be evaluated.

Formative Evaluation Evaluation doneduring development of instructional mate-rials for the purpose of rewriting andimproving the quality of the final product.

Measurement Provides quantitative de-scriptions of the characteristics of avariable.

Qualitative Data Non-numerical infor-mation often obtained through open-end-ed questions, observations, or interviews.

Quantitative Data Numerical informa-tion from an evaluation. This informationcan be manipulated and may be reportedas an average, rank, median, mean, orstandard deviation.

Reliability For our uses this refers to theconsistent nature of the evaluation tools'performance. Does the tool yield consis-tent results under similar circumstances?The degree of consistency determines thereliability of the tool.

Significance The odds that observed dif-ferences or similarities are due to the"real" influence of variables and notchance.

Standard Deviation A statistical functionthat serves as measurement of the degreeof agreement between responses for a spe-cific criterion on a given piece of material.

Summative Evaluation An evaluationdone to examine the end result of a projector activity. Materials developers may con-duct a summative evaluation to measurewhether their product does what it wasintended.

Validity An estimate of the degree towhich an evaluation tool measures whatit is designed to measure.

Variables All the factors that could affectresults of an evaluation project. For exam-ple, the evaluator may tire with a lengthytool and therefore rush through the evalu-ation of the last few pieces of instructionalmaterial.

Evaluated EE Activity Guides

The Curricula and Compendia Project,Six Volumes: Energy Resources, WaterResources, Integrated Waste Management,Air Quality, Human Communities, andNatural Communities. California Depart-ment of Education, 1992-95. Sacramento,CA. These compendia contain the resultsof evaluations of K-12 environmentaleducation curricula based on presentation,pedagogy, teacher usability, general con-tent, and subject-specific content. Eachcompendium includes a "report card,"evaluator comments, a brief description,and sample pages. Available from:

Bureau of Publications, Sales UnitCalifornia Department of EducationP.O. Box 271Sacramento, CA 95812-0271

© (916) 445-1260Fax: 916 323-0823

"Curriculum Materials for EE, "D.L.Haurv,J.E. Heimlich, and M. Norton, edited byR.J. Wilke. In Environmental EducationTeacher Resource Book, A Practical Guidefor Environmental Education, 1993.Milwood, NY: Kraus International Publi-cations. A selected, annotated listing ofreadily available materials for use in envi-ronmental education. Included are curricu-lum guides, supplementary materials,activities, nonprint media, modules, etc.),program frameworks, and resource materi-als. Available from:

Kraus International Publications358 Saw Mill River Rd.Milkwood, NY 10546

t (914) 762-2200

Educating Young People About Water.A Guide to Goals and Resources withan Emphasis on Non formal and SchoolEnrichment Settings. Elaine Andrews,1992. Madison, WI: University of Wiscon-sin. This guide was developed by the WaterCurriculum Needs Assessment Projectto help curriculum coordinators select and

t ' ' ,It r ,1

develop water quality training programsand curricula. Includes summaries of 63reviewed curricula. Available from:

Elaine AndrewsUniversity of WisconsinEnvironmental Resources Center216 Agriculture Hall1450 Linden DriveMadison, WI 53706

t (608) 262-6968

Energy and the Environment: A Teacher'sResource Guide. Pacific Gas and ElectricCompany, 1991. San Francisco, CA.A catalog of energy education resourcesfor elementary, secondary and highereducation. Available from:

Pacific Gas and ElectricEnergy and Environment Department77 Beale, 28th FloorSan Francisco, CA 94106

.40 (415) 973-4361

"Evaluation of Selected RecyclingCurricula: Educating the Green Citizen,"S. Boerschig and R. DeYoung. In Journalof Environmental Education, 24:17-22.1993. Evaluation of 14 solid waste curric-ula. Provides a ratings matrix of evaluatedmaterials and recommendations for devel-opment of new materials.

Evaluation Tools"Analyzing Educational Resources," ZitaUnger, edited by John Fien. In Teaching fora Sustainable World: Environmental andDevelopment Education Project for Teach-er Education. 1992. Brisbane, Australia:Griffith University Press. This workshopoutline includes guidelines for group dis-cussions, overhead masters, readings, andan evaluation instrument to help partici-pants become engaged in evaluation issues,particularly bringing to light the range ofideologies and viewpoints that are embed-ded in resources. Available from:

Dr. John FienFaculty of Environmental SciencesGriffith UniversityNathan, BrisbaneAustralia 4111fax: 6 1 -7-875 -7459

51

"A Checklist for Reviewing EnvironmentalEducation Programs," Fred C. Niedermey-er. In Journal of Environmental Educa-tion, 23 (2):46-50. 1992. Enumerates, in achecklist form, the characteristics of well-developed, effective instructional programsto provide criteria for their review.

Environmental Education CurriculumInventory. J.R. Gardella, 1993. HighlandHeights, KY: School of Education, North-ern Kentucky University. Includes twoforms that use accepted goals of environ-mental education in a checklist formatwith a data summary device to identify theemphasis of the evaluated curricular mate-rials. Available from:

North Kentucky UniversityEducation DepartmentHighland Heights, KY 41099

© (606) 572-5624

"Evaluating Sponsored Materials," MaryE. Haas. In How To Do It in the SocialStudies Classroom Series.1985.Washington, D.C.: National Council forthe Social Studies. Two evaluation formsare provided to carefully evaluate spon-sored materials and guest speakers. Avail-able from:

The National Councilfor the Social Studies

3501 Newark St., NWWashington. DC 20016

© (202) 966-7840

Additional ReadingsIf you arc interested in more informationon evaluation or want to do a formalevaluation project we recommend theseresources.

"The CIPP Model for Program Evalua-tion," D.L. Stufflebeam, edited by G.F.Madaus, M. Scriven, & D.L. Stufflebeam.In Evaluation Models. 1993. Boston: Klu-wer-Nijhoff. The CIPP approach is basedon the view that the most important pur-pose of evaluation is not to prove but toimprove. Evaluation is seen as a tool bywhich to help make programs work betterfor the people they are intended to serve.Available from:

Kluwer Nijhoff Publications101 Phillip DriveNorwell, MA 02061

© (617) 871-6600

Criteria for Evaluating InstructionalProducts. James L. Moseley, 1987. ERICDocument Number: 281-870. This five-page checklist presents a rating scale ap-proach to evaluation instructionalproducts and educational media. Fifteencategories of evaluation criteria are includ-ed. Available from:

ERIC555 New Jersey Ave., NWWashington, DC 20208-5720(202) 219-2088

ti (800) 538-3742

"Evaluating Environmental Education inthe Context of a Junior High School StateStudies Program," Dean Bennett. InJournal of Environmental Education,13(4): 13-18. 1982. Describes the MaineStudies Curriculum, a state-validated dem-onstration project taught as part of theregular school curriculum. Also describesthe curriculum evaluation procedure andresults.

52rk h;;) Prsobke Manuds

1114.1.)tr.? w 7". . ,

"Evaluation of Environmental EducationPrograms at the Elementary and SecondarySchool Levels," Bernard J. Lucko, John EDisenger, and Robert E. Roth. In Journalof Environmental Education, I3 (4 ): 7-12.1982. Provides an overview of recent andcurrent activity in environmental educa-tion, particularly as it relates to programevaluation in elementary and secondaryschools. Also includes major steps in con-ducting an evaluation based on the goalsof environmental education.

"Evaluating Instructional Materials," TimWentling and William E. Piland. In LocalLeader Guide IX: Locally Directed Evalu-ation Handbook. Second Edition.1982.Springfield, IL: Illinois State Board of Edu-cation. One of 12 guides that have beendeveloped to facilitate evaluation by andfor local education agency personnel in Illi-nois. Covers the evaluation of both printand audiovisual instructional materials.Available from:

Curriculum PublicationsClearing House

Western Illinois UniversityHorrabin Hall #46Macomb, IL 61455(800) 322-3905

"Evaluation of Natural Resource Educa-tion Materials: Implications for the Re-source Manager," Gerri Pomerantz. InJournal of Environmental Education, 22(2): 16-23. 1990. An analysis of elementa-ry school and natural resource lessons thatfocus on ecological principles, resourcemanagement issues, and analytical skill de-velopment. The conclusion is that veryfew of the lesson materials help to developcritical thinking skills and behaviors neces-sary for environmental action.

Experimental and Quasi-ExperimentalDesigns for Research. Donald T. Campbelland Julian C. Stanley, 1963. Chicago, IL:Rand McNally College Publishing Co.Examines the validity of 16 experimentaldesigns against 12 common threats to val-id inference. Available from:

Rand McNally College Publishing Co.444 N. Michigan Ave.Chicago, IL 60611

r (312) 321-1751

nrilentai I du. Mar ,

Procedures for Developing an Environ-mental Education Curriculum. HaroldHungerford and Robert Ben Peyton,1986. EE Series 22. Paris, France: UnitedNations Educational,Scientific and Cultural Organization. ERICdocument: ED 277 569. This publicationis an update of a 1980 document andrelates trends in environmental educationto curriculum development. Availablefrom:

UnescoEnvironmental Education Unit7 place de Fontenoy75352 Paris 07 SPFRANCE

A Scope and Sequence for EnergyEducation Grades K-12. Sonoma StateUniversity, Environmental Studies andPlanning Department, 1989. A conceptualmatrix for evaluating criteria related toenergy education. Available from:

Sonoma State UniversityEnvironmental Studies and PlanningDepartment1801 East Copati Ave.Rohnert Park, CA 94928

Et (707) 664-2306

Standards for Evaluation of InstructionalMaterials with Respect to Social Content.California State Department of Education,1986. Overview of the California "Educa-tion Code" standards, which are intendedto help dispel negative stereotypes by em-phasizing people in diverse, positive, andcontributing roles. Available from:

California State Departmentof Education

P.O. Box 271Sacramento, CA 95802-0271

t (916) 653-7026

vv

A Qualitative Study of How OneTeacher Selects Instructional Materials.John Elliot, 1986. ERIC document: ED269-381. This case study explores waysin which a primary grade teacher selects in-structional materials. Areas coveredunder the study include: the text as a setcurriculum, reflections on the teacher'schoices of instructional materials, thesource of the teacher's instructional ideas,the teacher as "scrounger," instructionalmaterials and parent involvement, and in-structional materials as outside experience.Available from:

ERIC555 New Jersey Ave., NWWashington, DC 20208-5720(202) 219-2088

6 (800) 538-3742

"What is the Purpose of Evaluation?"George T. O'Hearn. In Journalof Environmental Education, 13 (4): 1-3.1982. Provides a broad, working definitionof environmental education, suggestingthat such broad definitions necessitate theidentification of a few measurable objec-tives as "indicators" of an environmentaleducation program's impact. Discusses thepurpose of and methods for evaluation.

EN ONOTES

1. Kirkpatrick, Donald. "Evaluation," Trainingand Development Handbook edited byRobert L. Craig, third edition. New York:McGraw-Hill, 1987.

2. United Nations Educational, Scientific, andCultural Organization. 19-8. Final Report,Intergovernmental Conference on Environ-mental Education. Organized by Unesco inCooperation with LINER Tbilisi, Georgia,USSR, 14-26 October 19, Paris: UnescoED/MD/49.

3. Ibid

4. (..aliforma Department of liducanon, Califor-nia Department of Water Resources andSonoma State University. LovinumzentalEducation: Lompendium for Water Resource.5.Sacramento, ( A, 1992.

i. Pacific Gas and Electric Company. Energyand the Environment: A Teachers ResourceGuide. San Francisco. CA. 1991.

6. Tourfillott, Leeann. Conserve and Renew,Energy Education Activities for Grades 4 - 6.Rohnert Park, CA: Sonoma State University,1990.

Andrews, Elaine. Educating Yining Peopleabout Miter: A Guide to Goals andResources with an Emphasis on Nonformaland School Enrichment Settings. Madison.WI: University of Wisconsin, 1992.

8. As of this printing Project Learning Tree hasnot yet published the results of their evalua-tion. For more information on this cvalua-non project contact Kathy \lcGlauflin, Direc-tor, Project learning Tree, 1111 19th StreetNW, Suite 800, Washington. l)(. 20036.(2021463-2468.

BEST COPY AVAILABLE