an evaluation of the sustainable schools project ssp...an evaluation of the sustainable schools...

An Evaluation of

The Sustainable Schools Project 2003-2004

Prepared for Shelburne Farms

The Vermont Education for Sustainability Project and

The Place-based Education Evaluation Collaborative

Prepared by: Program Evaluation & Educational Research (PEER) Associates

September 8, 2004

The Sustainable Schools Project is part of the Place-based Education Evaluation Collaborative

(PEEC), a unique partnership of organizations whose aim is to strengthen and deepen the practice and evaluation of place-based education initiatives.

PEEC programs (and organizations) include the CO-SEED Project (Antioch New England Institute); the Community Mapping Program (the Orton Family Foundation, and Vermont

Institute of Natural Science); the Sustainable Schools Project (Shelburne Farms, and the Vermont Education for Sustainability Project); and A Forest for Every Classroom Project

(Shelburne Farms, The Northeast Natural Resource Center of the National Wildlife Federation, The Marsh Billings Rockefeller National Historical Park, The Conservation Study Institute,

and Green Mountain National Forest). In addition, the Upper Valley Community Foundation provides funding and support for

several of these programs through its Wellborn Ecology Fund, as well as financial, administrative and staff support for collaborative evaluation and research efforts.

ACKNOWLEDGEMENTS Special thanks to:

• Dr. Jennifer Jewiss of the University of Vermont Department of Education and

the Vermont Research Partnership; • Dr. Tom Marcinkowski of the Florida Institute of Technology; • Dr. Michael Murphy of Harvard Medical School;

for their comments and suggestions through review of PEEC work at various stages. In September 2003, these advisors reviewed and commented on evaluation plans. In July 2004, they reviewed and commented on drafts of the final evaluation reports. Michael Duffin’s work on this evaluation report was partially funded through a fellowship from the National Network for Environmental Management Studies (NNEMS) program of the U.S. Environmental Protection Agency. Thank you. Michael Duffin also wishes to thank the Craiglow Center for Applied Research and Policy of Antioch New England Institute for administrative and institutional support. Finally, we wish to extend a particular thanks to the individual teachers, students, community members and SSP staff who so graciously participated in this evaluation.

TABLE OF CONTENTS

EXECUTIVE SUMMARY..................................................................................................... II

INTRODUCTION.................................................................................................................... 2

SUSTAINABLE SCHOOLS PROJECT OVERVIEW ....................................................................... 2 Sustainable Schools Project Goals .................................................................................... 2

SSP AND THE PLACE-BASED EDUCATION EVALUATION COLLABORATIVE........................... 3 Place-Based Education Theory of Change ........................................................................ 4 Members of the Collaborative............................................................................................ 6 External Evaluation Team ................................................................................................. 6

SSP AND THE EDUCATIONAL RESEARCH LITERATURE .......................................................... 8 Need for Place-Based Education ....................................................................................... 8 Connecting School and Community ................................................................................... 9 Student Achievement ........................................................................................................ 11 Educational Reform ......................................................................................................... 15 Diffusion of Innovations................................................................................................... 17 Measuring Results............................................................................................................ 21

EVALUATION METHODS ................................................................................................. 24

EVALUATION QUESTIONS.................................................................................................... 24 PROCESS AND INSTRUMENTS............................................................................................... 25

Literacy Process and Instruments.................................................................................... 27 Survey Administration Process ........................................................................................ 30

DATA ANALYSIS .................................................................................................................. 30 Analysis of Interview Data and Documents ..................................................................... 30 Analysis of Literacy Data................................................................................................. 31 Analysis of Survey Data ................................................................................................... 31

FINDINGS AND DISCUSSION ........................................................................................... 42

TEACHERS’ CURRICULUM PLANNING: INCREASED COHERENCE AND INTEGRATION ........ 42 Using Essential Questions ............................................................................................... 43 Interrelated Units of Study ............................................................................................... 43 Connecting Specials and Classrooms .............................................................................. 45

STUDENT AWARENESS OF AND ENGAGEMENT IN THE COMMUNITY................................. 47 Grades 4/5 Team Tackles Citizenship Projects ............................................................... 47 Younger Grades Explore the Community ........................................................................ 49 Students’ Sensitivity Grows with Community Engagement ............................................. 50

OTHER STUDENT OUTCOMES .............................................................................................. 51 Linking Health and Sustainability.................................................................................... 51 SSP Impacts Student Learning Process ........................................................................... 52 Students Take Sustainability Messages Home ................................................................. 53

SSP PROCESS STRENGTHS.................................................................................................... 54 SSP Reaches a Schoolwide Audience............................................................................... 54

Composting Takes Hold ................................................................................................... 54 Skilled and Effective SSP Staff Promote Change............................................................. 56 Providing Resources and Teaching Classes .................................................................... 56 Second Summer Workshop Embeds Changes in Practice................................................ 57 Individualization and Flexibility of the SSP Model ......................................................... 57 SSP has Staying Power .................................................................................................... 59

EVIDENCE OF SSP’S STAYING POWER ................................................................................. 59 A Wide Network of Support.............................................................................................. 59 Continuation of Established Relationships ...................................................................... 62 Documenting SSP Work ................................................................................................... 63

SSP’S PROCESS CHALLENGES.............................................................................................. 64 Time and Scheduling Can Be Barriers ............................................................................ 64 Language of Sustainability Can Be a Deterrent .............................................................. 66 Some Subjects Seen as Outliers to Sustainability Work................................................... 67 Other Concerns ................................................................................................................ 68

INTEGRATING LITERACY AND SUSTAINABILITY.................................................................. 70 Project List ....................................................................................................................... 70 Champlain Literacy Assessment Questionnaire............................................................... 72 Literacy/Sustainability Study Group Reflections ............................................................. 73 Quasi-experiment ............................................................................................................. 74 Overall Synthesis of Literacy Integration ........................................................................ 78

PRESENTATION OF EDUCATOR AND STUDENT SURVEY DATA: BARNES AND CHAMPLAIN82 Educator Survey Results .................................................................................................. 82 Student Survey Results ..................................................................................................... 87

CONCLUSIONS .................................................................................................................... 92

IMPLICATIONS FOR PRACTICE .................................................................................... 94

LITERATURE CITED.......................................................................................................... 96

APPENDIX ........................................................................................................................... 102

APPENDIX A: SSP LOGIC MODEL ..................................................................................... 103 APPENDIX A: SSP LOGIC MODEL ..................................................................................... 103 APPENDIX B: SSP EVALUATION OVERVIEW 2003-04 ....................................................... 104 APPENDIX C: INTERVIEW GUIDE ....................................................................................... 106 APPENDIX D: EDUCATOR SURVEY .................................................................................... 108 APPENDIX E: STUDENT SURVEY ........................................................................................ 111 APPENDIX F: MONTHLY STAFF OBSERVATION RECORD .................................................. 116 APPENDIX G: SSP MONTHLY PROCESS WATCHER FORM ................................................ 118 APPENDIX H: LETTER TO TEACHERS................................................................................. 120 APPENDIX I: DIRECTIONS FOR TEACHERS ADMINISTERING SSP SURVEYS TO STUDENTS . 121 APPENDIX J: SSP SURVEY ADMINISTRATION PROCEDURES ............................................. 122 APPENDIX K: CHAMPLAIN LITERACY ASSESSMENT QUESTIONNAIRE............................. 124 APPENDIX L: PRE-ASSESSMENT QUESTIONS FOR LITERACY/SUSTAINABILITY STUDY GROUP125 APPENDIX M: LITERACY TEXT ASSESSMENT RUBRICS ....................................................... 126 APPENDIX N: SAMPLE PAGE FROM INFORMATIONAL TEXT PROJECT GUIDE................... 128

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates ii

EXECUTIVE SUMMARY INTRODUCTION The Sustainable Schools Project (SSP) is a partnership program of Shelburne Farms and Vermont Education for Sustainability (VT EFS). The project uses sustainability as the integrating context on a school-wide level, seeking to connect science, literacy, ecology, and community. SSP focused its first school year (2002-2003) piloting its program in one urban elementary school in Burlington, Vermont. In this, its second year, SSP continued its intensive work with the same elementary school while beginning to make inroads into its next elementary school site on the other side of Burlington. Full-fledged involvement with the second site will take place in the third year of programming. The primary goal of SSP is to use sustainability as an integrating context for strengthening a whole school’s curriculum improvement efforts, teacher leadership, community partnerships, and campus ecology. SSP strives to help a school become more connected and collaborative internally, as well as more connected with the school’s local community and natural environment. SSP staff describe the desired outcome as creating a more connected, coherent learning experience that inspires teachers to collaborate, and inspires students to engage with their school, community, and environment.

The SSP Logic Model (see Appendix A) further describes the program’s short and long term outcomes. The evaluation team worked with key program stakeholders to develop this year’s evaluation plans by using the SSP Logic Model as the anchor. SSP’s sponsoring organizations are among the founding members of the Place-based Education Evaluation Collaborative (PEEC), recently formed to evaluate the individual programs of its members and also to lay the groundwork for broader research into the effectiveness of place-based education.

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates iii

METHODS The first cycle of program evaluation sought to understand the successes and challenges of SSP’s process of program development and implementation, and to measure the degree to which its short-term projected outcomes were attained. The second year of evaluation continued to monitor progress at Champlain Elementary School, observing program impacts on faculty and students, investigating the integration of literacy and sustainability, and assessing the program’s staying power. The current evaluation cycle also involved gathering baseline data for the newer SSP school, Barnes Elementary. While this is primarily a formative evaluation, the report does provide evidence of positive outcomes as well as suggestions for program improvements. The evaluation is intended to be useful to program managers, funders and interested stakeholders, and to contribute to the field of place-based education. Based on the SSP Logic Model, and the objectives of the program stakeholders, the evaluation team developed the questions listed in the Figure M1. After SSP staff reviewed and approved the questions, the evaluators designed appropriate research instruments to explore questions.

Figure M1. Evaluation Questions for Sustainable Schools Project 2003-2004 Area of focus Evaluation Questions

Champlain Elementary

• What evidence is there that SSP work is and will be sustained in a school after year one and two?

• In what ways is teacher practice changing and sustained as a result of involvement with SSP?

• As the model develops, what are the strengths and challenges faced by SSP staff and participants?

• What effects does SSP work have on students’ awareness of places and people in their community?

• What effects does SSP work have on students’ engagement in their community?

Literacy at Champlain

• How is the presence of SSP helping teachers meet existing literacy goals and requirements?

• In what ways do the data from standardized test scores, locally scored portfolios and teacher perceptions triangulate to suggest effects of SSP on student literacy?

Barnes School • What baseline data will help us evaluate Barnes in future years in terms of process and outcomes?

The evaluation utilized multiple methods. Semi-structured interviews and focus groups were conducted with teachers, administrators, parents, community members, and SSP

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates iv

staff members. Students were interviewed informally during site visits. Classroom and school observations, internal reflection instruments, student academic achievement data and educator and student surveys served to triangulate the interview data. Figure M2 summarizes the data collection methods and instruments used in this evaluation:

Figure M2. Sustainable Schools Data Sources for Evaluation 2003-04

Instrument Type and Number Administered

Interviews

9 Classroom Teachers 3 Specials Teachers 1 School Administrator 1 Community Partner 1 Parent/PTO Member 4 SSP staff 19 interviews total +/- 20 informal interviews with students

Written Surveys

11 Barnes teachers (Jan – Feb 2004) 53 Barnes students, gr. 4, 5 (Dec ’03 – Jan ‘04) 82 Champlain students, gr. 4,5 (Nov ‘03) 13 Champlain teachers (June ’04) 80 Champlain students (June ’04)

Staff Reflections Forms 4 (Oct, Nov, Feb/Mar/Apr, Jan) Process Watcher Forms 4 (Oct, Nov, Dec, Jan) Project List (of activities that integrate sustainability and literacy) 1 (compiled June, ’04)

Literacy Assessment Questionnaire 13 teachers (October, ’03, June, ’04) Literacy/Sustainability Study Group Reflections

• Written by 5 teachers (September, ’03); • Discussed/dictated by 4 teachers (June, ’04)

Quasi-experimental Assessment of Student Literacy Skills

13 first grade and 4 third grade students (April - June, ’04)

Observations

• School building, school yard • School wide Sustainability Exposition • Students presenting projects, composting • SSP staff leading classroom lesson

Document Review

• Project fliers, brochures, Logic Model • Student work samples • Notes from SSP team planning session • Newsletters, Community Works Journal article • Literacy/sustainability study group materials • Sustainability & Informational Text Project

Guide

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates v

The evaluation team collected both qualitative and quantitative data about the integration of literacy and sustainability at Champlain Elementary. Qualitative interview data were gathered as part of semi-structured interviews that also sought data about other aspects of SSP. Quantitative data was gathered using a series of four locally designed and implemented assessments:

1) A compilation of two lists: one that showed all the major activities within Champlain elementary that integrated sustainability and literacy during the 2002-03 school year; and a second list that showed similar activities that occurred during the 2003-04 school year

2) A simple, one page questionnaire about the use of non-fiction resources administered to the entire teaching staff.

3) A two page questionnaire consisting of open-ended questions about literacy teaching practices.

4) A quasi-experimental assessment of student literacy skills was designed, piloted, and administered to a representative random sample of students.

FINDINGS This year’s evaluation of SSP focused on both process and outcome elements. In particular, we monitored the impacts of the program on teachers and students at Champlain Elementary School, now in its second year with SSP. Teacher Outcomes One of SSP’s primary strategies is to help teachers plan and implement sustainability-related curricula through the summer institute and ongoing coaching and support during the school year. There has been ample evidence this year that such practices are flourishing and becoming more embedded in the school’s operations. In addition to an overall sense that teachers were thinking differently about the curriculum planning process as a whole, there were three specific areas in which teachers described changes in their teaching practice. Teachers are increasingly:

• Using essential questions to plan and teach their students throughout the year

• Creating interdisciplinary units of study that weave together various disciplines, often using concepts of sustainability as an integrating theme

• Connecting their curricula with that of other classrooms, including special subjects such as art, physical education, and music, and across the grade levels.

“SSP brought an awareness to our social studies and science curriculum, and made us sit down and think about the scope and sequence of where we started in kindergarten and what we hope fifth graders will leave our school with. That helped us with our teaching and taking a look at our school.” -Kindergarten teacher

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates vi

Student Outcomes Through SSP work, students were exposed to diverse new places in the community, and--equally importantly--they began to develop a deeper understanding of places they already knew. This exposure came on various levels: a greater awareness of schoolyard flora and fauna, increased knowledge of the different groups that make up a neighborhood, and heightened insight into the ways that a business fits into the local economy. Three key areas are highlighted in the report to demonstrate the impact of SSP on student awareness of and engagement in their community:

• The grades 4/5 team immersed itself in comprehensive citizenship projects that had beneficial effects on the community and for the young learners.

• Younger grades also explored the community, including forming relationships with elders and developing a more fine-tuned sense of local geography.

• Teachers reported that students’ sensitivity was growing as they engaged more deeply with the community and developed a sense of involvement and feelings of responsibility.

Furthermore, three other areas emerged as outcomes related to student involvement in SSP:

• Through their school work, students demonstrated that they were linking issues of personal and community health to the topic of sustainability.

• Teachers reported that because of the level of student engagement in sustainability-related work, students are increasingly

“I learned a lot about the vital concepts of sustainability over the three years - not just ideas about how to teach it to kids but for my own personal awareness. The knowledge & support are always high quality. You will be sorely missed. Keep up this work- not only does it have a direct impact on communities, but it teaches kids to be empowered citizens.”

- Educator survey respondent

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates vii

making connections across units of study, and are increasingly directing their own learning processes.

• Students are making an increased personal commitment to sustainable practices at school, and they are taking sustainability messages, such as composting food waste, into their homes.

Process Strengths An overwhelming portion of the interview data and document review comprising this evaluation pointed toward program strengths. The most salient of these are:

• SSP reaches a schoolwide audience • Composting takes hold • Skilled and effective SSP staff promote change • Providing resources and teaching classes • Second summer workshop embeds changes in practice • Individualization and flexibility of the SSP model • SSP has staying power

A crucial measure of the success of a program such as SSP is the likelihood that the program will be continued over time, or have “staying power,” within the site. Three key areas emerged as clear evidence of the staying power of SSP, defined more specifically as its potential to continue at Champlain School beyond the first two program implementation years:

Figure F1. Examples of School/Community Cooperation Note: In addition to students venturing out into the community, numerous community members have come into the school on an ongoing basis to share their talents, skills and stories, and to help students develop their own skills. • Mayor came to school on numerous occasions including the Sustainability

Exposition day • Community partners and former teachers taught afterschool programs • Burlington Neighborhoods Project worked with 4/5 on community

improvement grants • Abenaki Grandmother came in to work with the school • A local university student helped work on a garbology unit. • A first grade teacher worked with the Senior Center to develop a

multigenerational unit. • Students met with a local doctor to talk about health choices. • Students did a mapping project with a staff member from Lake Champlain

Maritime Museum • Burlington Electric Department installed solar panels

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates viii

• A wide network of support among the school administration, parents, members of

the community, and the teachers. • A commitment to continuing relationships that were established through SSP,

such as those fostered between community members and teachers. • An effort to document SSP-related projects, such as the living machine, on site.

Process Challenges Unequivocal praise for the SSP experience at Champlain Elementary School and clear evidence of positive outcomes strongly indicate that the model is a success. Nonetheless, as can be expected with any program, a pattern of challenges emerged in the data. The challenges are broken down into the following three areas:

• Time and scheduling can be barriers • The language of sustainability can be a deterrent • Some subjects seen as outliers to sustainability work

Integrating Literacy and Sustainability This section of the report explores data collected from four different assessments of various aspects of SSP’s efforts to integrate literacy and sustainability, and then synthesizes that data with relevant evaluation interview data. A comparison between the past two school years reveals a marked increase in the amount of effort put into literacy at Champlain as a result of SSP. Not only did the amount of activity increase from 2002-03 to 2003-04 (especially in terms of more work with specific grade levels), but the type of activities seems to have taken on a new dimension as well. Clearly, sustainability concepts are being successfully integrated with literacy. This presents SSP with an opportunity to articulate just how much integration needs to happen in order to call their efforts a success, and how that might be accurately measured. Champlain Literacy Assessment Questionnaire One way that SSP’s strategic focus on the use of informational (non-fiction) texts showed up was through a simple, one-page, pre-post questionnaire given to all teachers at Champlain elementary school. The most prominent pattern in the findings is a seeming increase for every one of the (loosely) scaled items. This provides one tangible level of affirmation that SSP’s efforts to integrate sustainability and literacy are having positive results.

“Everything becomes that much more relevant when it's interrelated. I think we've always looked for that but [sustainability] is a perfect vehicle for doing it.”

-Second grade teacher

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates ix

Literacy/Sustainability Study Group Reflections The literacy/sustainability study group was an intentional component of the SSP at Champlain School. Five teachers participated in this study group that met 11 times for a total of 30 hours from September through December 2003. The purpose of the group was to build skills in integrating literacy into classroom work and to develop units to support such integration. In order to capture more detailed evaluation data from this study group, SSP staff designed a brief assessment, and had them write out their responses at the beginning of the school year and then discuss the questions as a group at the end of the school year. Participants echoed the clear intention and focus of the SSP staff and district literacy specialist in their various presentations of the importance and utility of non-fiction, informational texts. Participants also expressed the poignant tension between the additional demands and commitment required by SSP-type efforts and the benefits that can accrue as a result. Quasi-experiment SSP piloted a direct assessment of student literacy skills during the 2003-04 school year. The most striking element of the data is that all students demonstrated improvement from pre- to post-, except those who had achieved at or near the highest possible score on the pre-assessment. Assuming the integrity of the scoring process, it can be confidently asserted that the integrated sustainability/literacy lessons that took place between the pre- and post-assessments led to improved student skills in the first two out of the three questions/skill areas covered by the assessment rubric. This pilot quasi-experimental investigation could well become a replicable model for similar

“Their talent as a sustainability staff--to be able to see what could be done with the people in this building and this community -- that was key…. I think it has just taken roots of its own…”

-Specials teacher

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates x

investigations of other PEEC or place-based programs if some of the research design and administration procedures were to be slightly more refined and systematized. Presentation of Educator and Student Survey Data: Barnes and Champlain Schools The main thrust of the survey-related sections of this report is to present the findings of the effort to pilot a dose-response measurement strategy. Thus, this report takes an educative approach to presenting statistical findings. We omit some of the more descriptive statistical representations (e.g. simple bar graphs) in favor of putting more focused effort into helping the reader understand the meaning and power of the inferential statistics that undergird this dose-response measurement strategy. Several potentially useful findings have emerged. A very strong correlation was found between amount of exposure to SSP (i.e. “dose”) and responses to survey items about teacher practice. Other positive, substantial, and statistically significant results were found for student time spent outdoors and student attachment to place. Several other outcomes showed little relationship to SSP exposure or were not statistically significant. Since the primary focus of the quantitative part of this evaluation was to pilot test the dose-response measurement strategy itself, there is still more to be learned from analyzing the data more exhaustively.

CONCLUSIONS The SSP does not require that a specific topic, method, or activity be taught in a school. Rather, SSP encourages people to think about connections and to see the web that weaves together topics (e.g. art and writing), places (e.g. classrooms and neighborhoods), and people (e.g. students and local workers).

Figure S2. Teacher Practice (overall module) From SSP educator surveys, 2003-2004

ë ë ë ë ë ë ë ë ë ë ë ë ë ë ë ë

0.0 1.0 2.0 3.0 4.0

Dosage compositeScale: No exposure to SSP = 0,

Very high SSP exposure and implementation = 4

í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of s

core

s fr

om it

ems

abou

t

teac

her u

se o

f loc

al re

sour

ces

and

teac

her m

etho

ds/c

raft

(tea

prac

mea

)Sc

ale:

Stro

ngly

Dis

agre

e =

1, to

Str

ongl

y A

gree

= 4

Scal

e: Tw

ice p

er y

ear

or le

ss =

1, t

o O

nce a

wee

k or

mor

e = 4

YY

Y Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

The best fit multiple regressionline above shows that 42% ofthe variability in survey responseis predicted by dose of SSP. Theresult is statistically significant.

R2= .42, p = .001, n = 24.

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates xi

There is a parallel conceptual and structural evolution of the program taking place. As SSP’s sustainability-related content takes hold and begins to create a curricular web, so too the students, teachers, staff, and others involved in the process find themselves in an evolving framework of interconnectivity that has many benefits for the school and community alike. A specific example of this is the impact of SSP’s literacy efforts. By focusing on integrating sustainability broadly into as many aspects of the literacy curriculum as possible, SSP itself is becoming more broadly integrated into the school and district. In the broadest sense, what is happening with SSP is an expansion of perspective as participants begin to see the interconnectivity of all things. More specifically, at Champlain Elementary School, the SSP model’s “testing grounds,” teachers are learning how all elements of their curriculum are or can be related, and are gaining awareness of how the curriculum is linked between grades and between special subjects, like art and music, and classrooms. They are refining their skills in curriculum planning built upon a new foundation of essential questions and sustainability. Moreover, students are seeing that their learning process extends well beyond the classroom, to their neighborhoods, and to other students, the schoolyard, their parents, and other community adults. IMPLICATIONS FOR PRACTICE Evaluation provides the opportunity for program growth, reflection and refinement. Interpretation of the data used in this year’s evaluation points to a number of recommendations for program development, also called implications for practice. The following list offers a snapshot of the more detailed and extensive implications for practice provided in the report.

• Acknowledge from the start that the process of change itself does indeed require an investment of time, but that there is ample evidence that it is worth it in the longer term.

• Allow participants to explore and define the language pertinent to the project—

particularly “sustainability” and “sustainable,”--and to select the best way to represent the concept for their school and community.

“The kids have told me that they feel invested in the health of their neighborhoods. They are proud of their community improvement projects and understand the importance of maintaining clean, safe neighborhoods.”

-SSP community partner

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates xii

• Funding should be sought to allow for expansion of the summer institutes to include a greater number of days and greater variety of participants.

• Include all teachers equally from

the beginning of program implementation.

• It would be valuable to have a

school liaison who can assist a school in its third year as the site transitions away from heavy involvement of SSP staff.

• Assure that the social equity element of education for sustainability is addressed in

as much depth as the environmental integrity and economic prosperity elements.

• Assure that all program staff feel adequately prepared to implement the area of focus, such as coaching teachers on literacy work or diversity issues.

• Consider building upon the quasi-experimental assessment exercise as a way to

eventually provide a solid body of evidence about SSP’s direct influence on student achievement.

• The first round of analysis of the survey data collected in 2003-04 reveals some

compelling confirming evidence of program success. Further exploration of the data is warranted.

• Continue to explore the benefits of a deeper collaboration and sharing of

responsibility between program staff and evaluators.

“The improvements to the grounds here and the gardens here-- certainly planting and growing and looking at cycles of life and habitat--those will continue too.”

-School Administrator, commenting on aspects of SSP that will have staying

power at Champlain

SSP Program Evaluation Report 2003-04 EXECUTIVE SUMMARY PEER Associates xiii

SSP Program Evaluation Report 2003-04 PEER Associates 2

Figure C1. Goals and Strategies of the Sustainable Schools Project

Goals:

• Improve student learning • Increase student engagement in

their communities Strategies:

• Curriculum connections • Community partnerships • Collaboration • Campus ecology

INTRODUCTION SUSTAINABLE SCHOOLS PROJECT OVERVIEW The Sustainable Schools Project (SSP) is a partnership program of Shelburne Farms and Vermont Education for Sustainability (VT EFS). The project uses sustainability as the integrating context on a school-wide level, seeking to connect science, literacy, ecology, and community. SSP focused its first school year (2002-2003) piloting its program in one urban elementary school in Burlington, Vermont. In this, its second year, SSP continued its intensive work with the same elementary school while beginning to make inroads into its next elementary school site on the other side of Burlington. Full-fledged involvement with the second site will take place in the third year of programming. SSP emerged out of the VT EFS initiative which has provided professional development training for more than 30 schools in Vermont and successfully added sustainability and sense of place standards to Vermont’s Curriculum Frameworks. The EFS project works to promote the three goals of the sustainability movement--environmental integrity, economic prosperity, and social equity—by promoting an understanding of the interconnectedness between the environment, economy, and society. VT EFS holds the position that sustainability education can strengthen student learning, teacher instructional practice and community development throughout K-12 education. SSP is an attempt to build on the type of support VT EFS has provided individual teachers by working with whole schools.

Sustainable Schools Project Goals SSP embraces all of the goals and purposes of VT EFS, but the primary goal of SSP is to use sustainability as an integrating context for strengthening a whole school’s curriculum improvement efforts, teacher leadership, community partnerships, and campus ecology. SSP strives to help a school become more connected and collaborative internally, as well as more connected with the school’s local community and natural environment. SSP staff describe the desired outcome as creating a more connected, coherent learning experience that inspires teachers to collaborate, and inspires students to engage with their school, community, and environment.


The SSP Logic Model (see Appendix A) further describes the program’s short and long term outcomes. The evaluation team worked with key program stakeholders to develop this year’s evaluation plans by using the SSP Logic Model as the anchor. SSP staff work closely with school personnel and community members to tailor a program appropriate to the site. At their first site, Champlain Elementary School, SSP staff implemented a two-day summer institute for teachers in summer 2002, and have since provided on-going support to the school in the form of:

• attending grade level meetings and assisting teachers with curriculum planning • attending and facilitating staff meetings that focused on sustainability • assisting teachers in the classroom working with small and large groups of

students, including modeling teaching techniques and co-teaching • providing teachers with curriculum and content resources (such as books, posters,

hands-on project materials, etc.) • facilitating connections between classrooms and community and parent volunteers • working with the after school program coordinator to infuse the sustainability

theme into after school activities • organizing evening family education events on the focus areas of the school’s

sustainability work • providing a second summer institute in 2003 • facilitating teacher internships through the Linking Learning to Life program • meeting with teachers and administrators individually to coordinate SSP efforts.

Through these forms of support provided by SSP, teachers are exposed to new content for their teaching, and given the opportunity to discuss new ideas about how to link subjects to the local community and resource base. Teachers are guided through an innovative curriculum planning process, introduced to resource specialists, and have the opportunity to explore local community places to use in their teaching. While the opportunity to conduct a whole school summer workshop at SSP’s second elementary school did not materialize, SSP staff led five afterschool sessions and two a two-day summer institute. The other forms of support provided by the program will be in play at SSP’s second site, Lawrence Barnes Elementary School, in the coming year.

SSP AND THE PLACE-BASED EDUCATION EVALUATION COLLABORATIVE In October 2001, several New England foundations and educational organizations came together to explore how they might collectively strengthen the evaluation of their place-based environmental education programs. They each sensed that their organizations could be doing more and better evaluation of their programs by working together than by working independently. The group decided to form an evaluation collaborative to evaluate their individual programs and also to lay the


groundwork for broader research into the effectiveness of place-based education. SSP’s sponsoring organizations are among the founding members of this recently formed group, called the Place-based Education Evaluation Collaborative (PEEC). PEEC has three main purposes. It serves as a learning organization for program developers, fueling internal growth and program development for the individual organizations. PEEC also aims to identify, develop, and disseminate evaluation techniques, tools, and approaches that can be applied to other place-based education providers, thereby promoting better evaluation practice in the field. Finally, as a long-range goal, the collaborative intends to contribute to the research base underlying the field of place-based education and school change. The goals of the four collaborating projects vary somewhat but common themes are:

• enhanced community/school connections

• increased understanding of and connection to the local place

• increased understanding of ecological concepts

• enhanced stewardship behavior • improvement of the local

environment • improvement of school yard habitat

and use as teaching space • increased civic participation.

All four programs focus on linking the school curriculum to the local community. Two programs work with whole schools and their local communities (e.g. through in-service days, staff meeting integration, community-school forums, and other ongoing activities), and two work primarily with individual teachers through institutes, curriculum development, follow-up support, etc. All four programs work with teachers and communities over the course of at least 12 months, and in some cases for over three or more years.

Place-Based Education Theory of Change PEEC members tend to conceive of their programs as unique, locally appropriate, and adaptable instances of an overall philosophy of place-based education. During the Fall of 2003, members of the collaborative worked to create the following generalized theory of change (Figure C2) to visually articulate many the commonalities underlying the logic of the four individual PEEC programs. Program staff then used this working model to portray their programs in the larger context of place-based education to both


internal and external stakeholders. This working theory of change also informed both general and specific aspects of individual and cross-program evaluation plans. It is interesting to note that this model emerged in partial response to the conceptual limitations of the linear, more lock-step process suggested in the proposed theory of change that emerged in the 2002-03 PEEC evaluation reports. While this newer model more accurately captures the iterative, adaptive, interdependent systems nature of place-based education program as they are implemented in local contexts, it may be too general and abstract for some uses. Thus, as of the writing of this report, PEEC members are working on developing a “PEEC Cross-Program logic model” to complement the theory of change depicted below with a representation that might be both more detailed and more simplified. The process of developing these various models and theories embodies a central tension in place-based education: simultaneous commitment to broad overarching concepts and locally unique details.

Figure C2. PEEC Working Theory of Change for place-based education


A more detailed PowerPoint presentation of the components of this theory of change is available on the PEEC web site at http://www.peecworks.org/PEEC/PEEC_Reports/.

Members of the Collaborative PEEC is comprised of the following “member” organizations and programs, with the Upper Valley Community Foundation (Hanover, NH) operating as the fiscal agent and umbrella organization, contributing directly to PEEC’s work from its Wellborn Ecology Fund. The business of PEEC is carried out primarily by the individuals noted in the sidebar, who meet face to face several times per year and carry out extensive phone and email conversations between PEEC meetings.

PEEC programs • The CO-SEED project, sponsored

by Antioch New England Institute (Keene, NH),

• The Sustainable Schools Project, sponsored by Shelburne Farms and the Vermont Education for Sustainability Project (Shelburne, VT)

• The Community Mapping Program, sponsored by the Orton Family Foundation (Rutland, VT) and Vermont Institute of Natural Science (Woodstock, VT)

• A Forest for Every Classroom project, sponsored by a partnership between Shelburne Farms (Shelburne, VT), Marsh-Billings-Rockefeller National Historical Park (Woodstock, VT), Green Mountain National Forest (Rutland, VT), the Northeast Office of the National Wildlife Federation (Montpelier, VT), the Conservation Study Institute (Woodstock, VT), and the Northern Forest Center (offices in NH, ME, VT)

External Evaluation Team All evaluation reports prepared for PEEC were generated by a team of evaluators operating as PEER Associates under the supervision of principal investigators Amy Powers and Michael Duffin. Both Amy and Michael are former employees of collaborating organizations of PEEC who have shifted their careers from program

Figure C3. PEEC individual members • Megan Camp, Shelburne Farms • Delia Clark, Antioch New England

Institute • Bo Hoppin, Antioch New England

Institute • Nora Mitchell, Conservation Study

Institute • Kevin Peterson, New Hampshire

Charitable Foundation • Bill Roper, Orton Family Foundation • David Sobel, Antioch New England

Graduate School • Liz Soper, Forest for Every

Classroom Program • Ned Swanberg, Vermont Institute of

Natural Science • Erica Zimmerman, Vermont

Education for Sustainability Project


delivery to program evaluation in recent years. Neither of them has ever been employed directly by the particular programs involved in PEEC. Their level of previous familiarity with the evaluated programs and their staff was generally agreed to be an asset to the more participatory approach that PEEC desired. During the course of most of the evaluation efforts contained in this report, Amy operated as a private consultant and Michael was an employee of Antioch New England Institute but working under a different Center than the one that houses the CO-SEED project. In the summer of 2004, PEER Associates was formerly incorporated as a private consulting firm, and Michael ceased being an employee of Antioch New England Institute. PEER Associates is committed to using a multiple-methods, utilization-focused, participatory evaluation process. It is our intention to help organizations better understand their programs and to help them to improve their programs based on evidence of program functioning and outcomes. We also intend to help organizations build their own capacity to reflect on and internally evaluate programs and to help to improve the evaluablility of programs. Other evaluators from PEER Associates who contribute to all reports include Dr. George Tremblay, Andrew Powers, and various Graduate Research Assistants. George is core faculty in the Clinical Psychology doctoral program at Antioch New England Graduate School, serves as the Director of Research for that program, and was brought onto PEER Associates specifically for his expertise in quantitative analysis. Andrew Powers serves as both Research Associate and Administrative Director for PEER Associates. For this particular report, Amy Powers acted as the Principal Investigator, guiding the bulk of the evaluation activities and serving as overall editor of the report document. Andrew Powers provided much support during the analysis phase, and Michael Duffin directed most of the literacy-related evaluation activities for this project. George Tremblay worked closely with Michael to support all aspects of the survey portions of this evaluation. Graduate Research Assistant Ben Skolnik transcribed many of the interviews. All core employees of PEER Associates provided editorial input during the reporting phase. The following measures were taken to mitigate the potential for researcher bias:

• Three nationally respected researchers were retained as advisors for the evaluation, all of whom provided interim consultation on research design as well as reviewing final reports.

• All interviews were recorded and transcribed either fully or partially. • Field notes for interviews and observations were generally typed up within

twenty-four hours or less after completing on-site work. • The faculty of Michael Duffin’s Ph.D. program were consulted on specific aspects

of research design and analysis.


• The issue of potential researcher bias was publicly and explicitly explored and discussed multiple times to the satisfaction of all members of PEEC.

• The PEER Associates team was intentionally constructed to represent complementary positivist and constructivist epistemological biases.

• Interview guides were developed and followed, and generally accepted methods for coding interview data were employed.

• Every effort was made to maintain high standards for methodological rigor.

SSP AND THE EDUCATIONAL RESEARCH LITERATURE The Sustainable Schools Project is one of numerous educational programs emerging throughout the country that use local ecological and cultural resources as a basis for learning. Schools, nature centers, government agencies, and non-profit organizations are working to bring about educational reform by intentionally connecting schools to their communities (Chin, 2001; Smith & Williams, 1999; Stone, 2001), often referring to their approach as “place-based education. In order to give a flavor for the rich educational context that the Sustainable Schools Project and companion programs in PEEC are working in, this section of the report touches very briefly on research and theoretical literature in the following areas:

• Need for Place-Based Education • Connecting School and Community • Student Achievement • Educational Reform • Diffusion of Innovations • Measuring Results

Place-based education roots learning about abstract systems in the concrete experiences of the schoolyard and community. However, the term place-based education is often used interchangeably with a number of other, similar terms: community-based learning, service-learning, sustainability education, project-based learning. Each of these terms refers to an explicit connection between the school and the community in which the school resides. A broader hope is to “tear down school walls” such that the community becomes integral to all facets of student learning—the school is open and inviting to the community and the community not only welcomes, but inspires and participates in student learning. A more detailed literature review of the evolution of place-based education and service-learning is available on the PEEC web site (Plumb, 2003).

Need for Place-Based Education This report is not the place for a litany of statistics about declining ozone and biodiversity, or increasing pollution and material consumption. Yet it warrants note

“Place-based education is the process of using the local community and environment as a starting point to teach concepts in language arts, mathematics, social studies, science and other subjects across the curriculum.”

-Sobel, 2004


that general awareness of things like the 30% decline from 1970 to 1995 in the “Living Planet Index”1, (and subsequent 3% annual drop) are clearly a driving force behind current efforts to help young people learn about and care for the environments in which they live (figures from Wilson, 2002). From Rachel Carson’s “Silent Spring” in 1962, through the first Earth Day in 1970, and continuing on into the present century, the call for environmental education has been growing steadily. A 1996 EPA report assessing the implementation of National Environmental Education Act of 1990 listed as its first recommendation to make environmental education a priority across the country (EPA, 1996). After surveying 1,500 adults every year since 1991, the NEETF/Roper report concluded that “95% of American adults (96% of parents) think environmental education should be taught in the schools and 90% believe that people in the workplace and in other places in adult society should receive environmental education too” (Coyle, 2004, p. 4). This same study goes on to report that “while the weight of the research shows that the simplest forms of environmental knowledge are widespread, real comprehension of more complex environmental subjects is very limited within the public” (Ibid, p. 7). Another interesting finding is that “more children (83%) get environmental information from the media than from any other source” (Ibid, p. 7). What can schools do to address some of these broad concerns? Place-based education is one response to this question. Rather than being a totally new approach, place-based education might be more accurately seen as an extension and refinement of environmental education. The key difference is that place-based education focuses on all aspects of the local environment by including local culture, history, social/political issues, and the built environment in its purview, as opposed to focusing more exclusively on the non-human natural world.

Connecting School and Community In place-based education, the community becomes the curriculum. One of the key ideas is that when one has developed an attachment to one’s place and the skills to act upon that attachment, an individual will become a more active participant in his or her community. This is sometimes referred to as civic engagement. When levels of civic engagement and participation increase in a community, social capital--the invisible web of relationship--is said to broaden and deepen. According to Robert Putnam (2001), author of Bowling Alone, social capital refers to features of social organization such as networks, norms, and social trust that facilitate coordination and cooperation for mutual benefit. An intensification of social capital then leads, in the long run, to healthier communities, both natural and social. This construct is an essential part of the theory of change embedded in the SSP logic model and the PEEC working theory of change for place-based edcuation.

1 Distilled from databases of the World Bank and United Nations Development and Environment Program by the World Wide Fund for Nature.


If fostering a sense of place and teaching action skills are the first steps toward the desired change, then programmatically it is the job of projects like SSP to find the most appropriate leverage points in a system (or community) to initiate these first steps. Implicit in the SSP mission is that educational intervention is an essential way to make change at the community level. A program evaluation conducted by the Harvard Graduate School of Education for the Rural Trust (1999) provides case studies of schools and communities throughout rural America that have been transformed by grounding students’ education in the local community and intentionally moving away from didactic approaches to standardized schooling. The evaluation concludes that as schools and communities work together to design curricular goals and strategies, students’ academic achievement improves, their interest in their community increases, teachers are more satisfied with their profession, and community members are more connected to the schools and to students. Getting parents involved in the school can be a good first step. Several studies show that involving parents in regular day to day activities of the school such as helping with classroom lessons can help parents connect to the values and educational concerns of the school (Comer, 1984, 1998; Epstein, 1991; Epstein & Becker, 1982; Paulsen, 1994a, 1994b; reported in Marzano, 2003). An extensive review of research literature on community organizing for schools conducted by the Harvard Family Research Project concludes that “poor school performance, high dropout rates, lack of qualified teachers, and inadequate facilities demand new forms of parent engagement to hold schools accountable. Community organizing offers one strategy to engage parents to effect system change” (Lopez, 2003, p. 2). The suggested strategies include paying special attention to the roles, relationships, and locus of power of parents. Also developing parent leadership, mobilizing collective power, and building social capital. Such efforts can require significant investment and commitment, but the payoff can be healthy policy and system changes, stronger home-school connections, improved school climate, and increased student achievement (Ibid.). Other studies show direct evidence that parent and community involvement in school can lower absenteeism, truancy, and dropout rates (Bucknam, 1976; reported in Marzano, 2003), and spark parental interest in school governance decisions (Stallworth & Williams, 1982; reported in Marzano, 2003). Many studies of the effectiveness of service-learning have been sponsored by the Corporation for National Service. These studies demonstrate powerful linkages between grounding the learning experience in the local context, enhanced student participation in community matters, and increased student engagement in their academic studies. In particular, service-learning experiences have been shown to promote a “pro-social, active conception of citizenship” in students (Chi, 2002, p. vi)


when implemented consistently and intensively including opportunities for analysis of and reflection on the service experience and regular opportunities for teachers and students to engage in dialogue. Connecting schools and communities is not an approach that is limited to the United States, either. Roger Hart describes several successful programs from England, Scotland, Columbia, Italy, Brazil, Nicaragua, the Philippines, and elsewhere (Hart, 1997). Denmark has a particularly strong movement toward collaboration between schools and communities around environmental and health concerns (Carlsson, 2004; Jensen et al., 2000). These references describe just a few of the many international examples of projects that share key design and implementation characteristics with the PEEC programs. All of the programs involved in the Place-based Education Evaluation Collaborative, including the Sustainable Schools Project, make extensive use of community members as a core part of their program logic and implementation. Existing research suggests that this is effort well spent.

Student Achievement Many of the goals that place-based education strives to achieve are notably lofty and difficult to convincingly measure. Student academic achievement is an example of this. It is challenging to establish a compelling, direct causal link between a student looking for insects in a local stream or interviewing community elders and that student’s scores on standardized tests. There is, however, a noteworthy body of research that suggests that student engagement in learning or motivation can function logically as a measurable proxy for student academic achievement. The first part of this section explores that idea more fully. The second part of this section looks at some of the more general factors that are associated with improved student academic achievement.

Engagement, motivation, and achievement A very thorough and lucidly written review of the educational research of the last thirty-five years opens chapter 15 with the following statement:

The link between student motivation and achievement is straightforward. If students are motivated to learn the content in a given subject, their achievement in that subject will most likely be good (Marzano, 2003, p. 144).

In the next five pages, Robert Marzano cites over 40 different studies that collectively lay out the evidence that supports these opening lines. Included in his argument are references to several quantitative studies that show correlations between motivation and achievement ranging from .19 to .63, and effect sizes that range from two-thirds to one and two-thirds standard deviations of improved achievement (Schiefele,


Figure C4. Excerpt from: The Generally Accepted Principles of Teaching and

Learning and their Implications for Local Education Support Systems

• All children do not learn in the same ways or at the same

pace. (Good instruction provides students instructional choices and multiple ways to engage with content to help them take ownership of their learning and demonstrate competence.)

• Learning is active. It requires effort and resilience on the

part of the student as well as interaction (Good instruction promotes this interaction by maximizing opportunities for students to engage in their learning, rather than passively absorb information with teachers, texts, materials, and/or other learners.)

• Learning depends on a foundation of factual knowledge,

the understanding of concepts in context, and the organization of facts and concepts so that they can be retrieved and applied.

• Learning is not limited to school. It can happen

anywhere. (Good instruction incorporates children’s out-of-school experiences in school with lessons that have value beyond school and is connected as much as possible to settings in the community that enhance learning for children and adults both inside and outside of school.)

*Created by School Communities that Work: A National Task Force on the Future of Urban Districts (June 2002)

Krapp, & Winteler, 1992; Schiefele & Krapp, 1996; Geisler-Brenstein & Schmeck, 1996; Tobias, 1994; Bloom, 1976; Steinkamp & Maehr, 1983; Willingham, Pollack, & Lewis, 2002; reported in Marzano, 2003). These are very impressive findings individually, and taken as a whole they make a fairly compelling case. Please refer directly to Marzano’s book, What works in schools: Translating research into action (2003), for a fully detailed argument of the research-based connection between motivation and achievement (and for several other useful, well-documented insights into the educational process as well.) This body of evidence holds an important implication for PEEC programs. If PEEC programs can present a case to support the claim that their programs increase student engagement and motivation in the learning process, then it is logically reasonable to connect that claim to the above mentioned body of evidence about student motivation to suggest that PEEC programs are likely to positively influence student academic achievement.

Other factors that support student achievement Some environmental education research literature advises that a conservation ethic and responsible behavior must begin with early, sustained exposure coupled with action strategies and behavioral practice (Hungerford & Volk, 1990). One of the more prominent and dramatic studies, entitled Closing the Achievement Gap (Lieberman & Hoody, 1998) shows broad reaching positive effects of curricula in over 40 schools nationwide that is grounded in the local environment. This 1998 study by the State Environmental Education Roundtable demonstrated that when the environment is used as an integrating context (EIC), student achievement and in-school behaviors improve.


There is a very strong body of general educational research evidence that suggests almost unequivocally that individual teachers make a difference in student achievement (Wright, Horn, & Sanders, 1997; Pedersen, Faucher, & Eaton, 1978; Marzano, 2003, pp. 71-105). This provides support for a key strategy embedded in both the SSP logic model and the PEEC working theory of change for place-based education (and the program evaluation strategies that follow from them), i.e. that school educators serve as the first level of change. It is primarily from this leverage point that students might be reached. Additional audiences for the PEEC programs include community members, school administrators, and students, but focusing on changing teacher practice is the place to start. One compelling set of research that supports the notion that individual teachers make a difference comes from an ongoing series of studies working with the Tennessee Value-Added Assessment System (TVAAS). In technical terms “TVAAS has been designed to use statistical mixed-model methodologies to conduct multivariate, longitudinal analyses of student achievement to make estimates of school, class size, teacher, and other effects” (Wright, Horn, & Sanders, 1997, p. 57). In other words, they use high powered math to track the way that student test scores go up or down based upon which teacher they have, and they look at large numbers of students over long periods of time. A relatively recent study using the TVAAS (Sanders & Rivers, 1996) found that: • Three straight years of most-effective teachers gives kids a 50-percentile point

advantage on students who spend three straight years with least-effective teachers. • The effects of teachers on student achievement are both additive and cumulative

with little evidence of compensatory effects. • As teacher effectiveness increases, lower achieving students are the first to benefit.

The top quintile of teachers tend to reach students of all achievement levels. • Students of different ethnicities respond equivalently within the same quintile of

teacher effectiveness. A very recent study of 92 elementary and middle school teachers in and around Chatanooga, TN (Public Education Foundation, 2002) is perhaps more directly applicable to place-based education efforts. This report found that the classrooms of the most effective teachers tended to be similar in the following ways: • Student work could be found everywhere, inside the classroom, out the door and,

in some cases, down the hall. • The teachers did not stand still and lecture; they covered every part of the room

and monitored every activity that took place. • Multiple small group activities were often found in their classrooms, with the

traditional arrangement of desks in rows practically non-existent.


• Students in their classes were at ease asking questions and commenting on statements made by teachers and other students.

• Expectations for the students were clearly stated and exemplars of previous years’ assignments were shown to students as models of what to produce.

• The organization of the rooms and the lessons was clearly evident. Materials were easily accessible when needed and no class time was wasted from lack of preparation.

So, if PEEC programs can demonstrate an increase in student motivation toward school work as a result of their programs, and they can also demonstrate that their programs lead to individual teachers changing their practice to be more like the characteristics described above, then the claim for a link between PEEC programs and student academic achievement becomes quite strong. Furthermore, the Rural School and Community Trust recently released a report about their place-based education efforts in Louisiana (Emekauwa, 2004). This report documents improvements in state wide assessment test scores over several years for fourth graders in selected schools and districts with place-based education programs in place as compared to statewide averages. While the results raise some interesting and provocative questions and may well suggest a testable correlation between place-based education training and student academic achievement, the strong claims implied in this report warrant more extensive documentation of context and methodology and deeper exploration of competing explanations for these test score trends in order to improve confidence in the assertions left implicit in the report. This report does, however, add another layer of incremental credibility to the increasingly strong claim that place-based education can positively impact student academic achievement. As the teaching strategies that are consistent with place-based education methods are increasingly shown through research to generally lead in the direction of increased student academic achievement, a new opportunity for researchers and program staff emerges. Namely, to systematically explore the ways that specific teaching practices lead to specific types of student achievement. One example of this type of research is a recent doctoral dissertation that combined an experimental research design with in depth qualitative interviews to investigate the effects of GIS (Geographic Information Systems) instruction on the environmental knowledge of middle school students. The study concluded that GIS can be effectively used for a wide range of classroom instruction contexts. Perhaps even more interestingly, the study also concluded that

…using GIS may aid students in constructing concepts and promoting understanding of environmental content, problem solving, experimental design and data analysis, and communicating findings to others. Using GIS in classroom instruction may be a way of incorporating spatial learning in schools (Hagevik, 2003, abstract).


This piece of research certainly strengthens the claims of the viability of the particular PEEC program that uses GIS as one of its main tools for implementing place-based education. As more place-based education research and evaluation takes on this kind of targeted focus, the case for the overall effectiveness of place-based approaches will grow stronger.

Educational Reform The body of research and theoretical literature on the topic of educational reform is vast. This section of the report touches very briefly on two concepts that may provide some useful insights for the programs involved in PEEC. First, there is a categorization scheme that has been used extensively in the educational research literature to simply classify the various factors that affect student achievement (Marzano, 2003). School-level factors have to do with school wide administrative, cultural, and/or policy decisions, initiatives, and influences. Teacher-level factors are the decisions and behaviors that individual classroom teachers have choice to directly affect. Student-level factors have to do with the unique characteristics that individual students bring to school, such as background, intelligence, and motivation. Which of these three factors has the biggest influence on student achievement? Just nine years after Sputnik, a landmark study involving 640,000 students and entitled Equality in Educational Opportunity (but more commonly referred to as the “Coleman report”) made the shocking assertion that student-level factors accounted for 90% of the variance in student achievement (Coleman, Campbell, Hobson, McPartland, Mood, Weinfield, & York, 1966). This report led some to believe that schools really didn’t make that much of a difference, so why bother, because the die is already cast by the background that a student is born into and lives in at home. A more optimistic synthesis of 10 high visibility studies (Marzano, 2000), however, suggests that schools account for 20% of the variance in student achievement, i.e. more than twice that suggested by the Coleman report. Of the 20% of influence that can be attributed to schools, about 13% comes from teacher-level factors, and 7% come from school-level factors (Bosker, 1992; Luyten, 1994; Madaus et al., 1979; Marzano, 2000; Stringfield & Teddlie, 1989; reported in Marzano, 2003). Perhaps most interestingly, if one reinterprets these statistics in terms of the percentage of students who do or do not “fail”, then a defensible case can be made that “schools that are highly effective produce results that almost entirely overcome the effects of student background” (Marzano, 2003, p. 7). It is not immediately clear where the effects of place-based education programs show up in this 3-level classification system. One could argue that place-based education might be classified as a “community-level” factor. This would suggest that its impact on students would be even less than the 7% level ascribed to schools, since impact on


students seems to decrease geometrically as the factors become further and further removed from the individual student unit of analysis. On the other hand, one could also argue that by bridging the worlds of school, home, and community, the effects of place-based education might show up as part of the 80% contribution due to student-level factors. When a student becomes actively involved in community environmental and/or social issues, would this show up in these statistical computations as a glowing, high leverage piece of the student-level factor pie, or as a muted, marginally influential piece of a diffuse community-level factor? The answer does not flow directly from the educational research. This dilemma may also be symptomatic of the general difficulty that place-based education has fitting into simplified classification schemes because of its interdependent, flexible, highly contextualized philosophy. In any case, the convenient three level categorization scheme of factors affecting students (i.e. school-level, teacher-level, and student-level) provides a potentially useful conceptual framework for place-based education proponents to talk about the program outcomes and impacts they seek to influence. In fact, the programs in PEEC have already adopted language that speaks to this categorization scheme. The Sustainable Schools Project and CO-SEED are “whole school change” (i.e. school-level) models, whereas the Community Mapping Program and a Forest For Every Classroom project are framed more as “professional development” (i.e. teacher-level) models. The second concept from the educational reform literature that warrants mention is the growing influence of systems thinking terminology and conceptual frameworks. Perhaps the best illustrative example of this is a small 1993 book by Michael Fullan called Change Forces. He talks about the complexity of the school change business in terms of paradoxes that only begin to make sense when one looks at the system as a whole and see interrelationships, processes, and feedback loops instead of linear cause and effect chains and snapshots. These ideas are not so different from those that PEEC members are drawing on when they cast their working theory of change for place-based education in terms of iterative feedback loops and the stocks and flows of various types of capital. Analyzing school reform from this global perspective leads Michael Fullan to the conclusion that teacher education is the highest leverage way to help schools (and the students within them, and the communities around them) increasingly act as “learning organizations.” This echoes much of the educational research that suggests that teacher-level factors are big in the lives of students. This also lends theoretical support to the notion that PEEC programs ought to focus their efforts on teacher practice change.


Diffusion of Innovations The term “diffusion of innovations” refers to the process by which a new idea or technology becomes increasingly used by a specified group of people. A tiny sampling of the list of fads, trends, policies, and revolutions whose key elements can be described by the diffusion of innovations process includes things like: the popularity of Sesame Street, Hush Puppies, or body piercing; increasing use of computers, the internet, and cell phones; use of citrus to control scurvy in the British navy; use of hybrid corn in Iowa; Paul Revere’s midnight ride; or the dismantling of apartheid in South Africa, just to name a few. Considering the spread of effect of place-based education programs within a given school (or within the field of education in general) as another case of the general diffusion of innovations process has both descriptive and prescriptive power. Analysis of both qualitative and quantitative data from evaluations of PEEC programs in 2002-03 and 2003-04 lend immediate support to the notion that participants in PEEC programs can be fairly accurately (if somewhat loosely) described in terms of “adopter categories.” Viewing the design and implementation of PEEC programs through the lens of diffusion of innovations theory and research could potentially help speed up and deepen program impacts as well as inform decisions about how to most efficiently use limited financial and time resources. The three main ideas summarized throughout this section of the report represent only a few of the many interesting ideas contained in three very different books about diffusion of innovations: a 500+ page scholarly review of over 5,200 publications aptly titled Diffusion of Innovations (Rogers, 2003); a general synthesis and extension of the key concepts, engagingly written for general audiences and called The Tipping Point (Gladwell, 2002); and an extremely practitioner-oriented application of the ideas to a specific context entitled Crossing the Chasm: Marketing and Selling High-tech Products to Mainstream Customers (Moore, 1999). The first main idea is that people respond differently to new ideas and technologies based upon individual psychological and/or demographic characteristics. When faced with the uncertainty inherent in considering the adoption of a new technology or way of doing things, people tend to fall into one of the “adopter categories” described in Figure C5 below. The distribution of people in a given population tends to follow a normal, bell-shaped pattern with the early and late majority categories each comprising about a third of the population, and the innovators, early adopters, and laggards collectively making up the remaining third of the population. This general idea of adopter categories leads directly to perhaps the most important overall prescription for those planning to create a change. Whether it’s a place-based education program or a fashion fad, one should intentionally target their


implementation/ marketing strategy based upon the core wants and other unique characteristics of each adopter category.

Figure C5. Adopter Categories (Moore, 1999, p. 17) The second main idea is that for successful innovations the rate of adoption through time in a given population tends to follow a fairly predictable S-shaped curve pattern. Diffusion proceeds very slowly at first, then reaches a “critical mass”, “tipping point”, or “take-off” period of rapid spread, then levels off at some more “permanent” level of adoption. Conceptual frameworks associated with epidemics and contagiousness are often applied to this S-shaped diffusion pattern. Rogers (2003) describes the tipping point as typically happening when the adoption rate is between 10-20% of the target population. Gladwell (2002) notes the “Rule of 150” (p. 175) which purports that innovations tend to tip after a sub-group of about 150 people in the larger social group have adopted.

Figure C6. General Diffusion Curve (Rogers, 2003, p. 11)


The third main idea is that diffusion is a highly subjective social phenomenon, meaning that word of mouth, reputation, and context are critical factors affecting the rate and depth of adoption of new practices. This is where some of the most useful applications of the theory come from, because it is about how the diffusion actually happens. The sustainability of a place-based education project (in an educational context) or the making of large profits (in an entrepreneurial business context) is all about breaking into the early majority and then successfully transitioning through into the late majority. Here are four things to think about: • Change tends to be discontinuous. Despite the charts above, graphs of real change

over time tend to look more like staircases than hockey sticks. This is largely because change happens relatively easily within adopter categories in which people tend to share the same interests, concerns, and networks, but moving between adopter categories is far more difficult. The biggest gap is the “chasm” between early adopters and early majority because the pragmatic early majority tends not to trust the judgment of the visionary early adopters. Many innovations fail to tip because they fall into this chasm before establishing a hold in the early majority. Moore (1999) suggests that the D-Day invasion of Normandy provides an effective analogy for strategically crossing the chasm. Place-based education supporters may, however, prefer to think in terms of a more nature-oriented analogy such as protecting endangered spotted owls, California gnatcatchers, or coho salmon. By focusing political advocacy and ecological restoration resources on these single keystone species, efforts will hopefully lead to preserving of habitat for the bulk of other species (i.e. the "early" and "late" majority) that depend on the same habitat. The lesson from this category is to focus all of one’s resources on a strategic “beach head” or “keystone” in the early majority, i.e. a very tightly defined sub-segment that has many connections within the larger early majority category.

• Specific types of people tend to make an innovation tip. These are the

networkers who know and are known widely (but not necessarily deeply) within many different sub-groups. These are the opinion leaders who are esteemed because they embody the implicit cultural and group norms and so become key reference points for others in their identity group. The rare people with extraordinary depth of content knowledge and the enthusiasm to share it can also be key ingredients in the recipe for successful diffusion. Strategically, these people tend to be good targets for a “D-Day” or “keystone species” type invasion into the early majority.

• Context matters. How an innovation is perceived has a major impact on how

likely it is to be adopted. The two most influential perceived attributes of innovations are “relative advantage” (i.e. the extent to which the new idea is thought to be better than the old way of doing things), and “compatibility” with


existing values, cultural norms, and past experiences of the potential adopter. Further, when interpreting human events and behavior we tend to reach for “dispositional” rather than “contextual” explanations, overestimating the contribution of individual character traits and underestimating the influence of situation and context. This is sometimes referred to as Fundamental Attribution Error (Gladwell, 2002, p. 160). The bottom line here is to remember that diffusion is a social process, and is dependent on group social interactions.

• “Re-invention” tends to be a good thing. When members of a target population

have the ability to change, adapt, and otherwise influence the new idea itself during the process of adopting it as their own, innovations tend to diffuse more rapidly and are more likely to be sustained.

Figure C7. Adopter Category Summary Chart

Adopter category

Descriptors from Moore, 1999 from Rogers, 2003

Core wants Strategies for working with them

Inno

vato

r

Enthusiast Venturesome

• Straight facts, truth, no tricks

• Be first

• Don’t expect immediate “profits” • Look for ones who can garner R&D

support by virtue of being close to the “big boss”

Earl

y ad

opte

r

Visionary Respect

• Breakthrough technologies

• Pursue a dream • Project orientation

• Maintain frequent contact • Manage unrealistic expectations • Chunk innovations into discreet

products or phases

Earl

y m

ajor

ity

Pragmatist Deliberate

• Incremental, predictable, measurable progress

• D-Day analogy • Keystone species • Focus, focus, focus effort on strategic

networkers and opinion leaders

Late

m

ajor

ity

Conservative Skeptical

• Smooth, easy change

• Discount prices

• Work the bugs out first • Plan for a customer service orientation

Lagg

ard

Skeptic Traditional • Keep status quo

• Actively listen for “Emperor’s New Clothes” phenomena (e.g. the Amish v. modern agribusiness)

• Otherwise try to neutralize influence There are a few additional ideas that warrant mention at this point. Diffusion of innovations research has been critiqued for having a pro-innovation bias that too often assumes the perspective of the change agency rather than the individual adopter. Thus it is wise to remember that almost all innovations have undesirable,


indirect, and unanticipated consequences. Additionally, innovators should be encouraged to pay attention to and mitigate for ways that the general nature of innovations can tend to widen gaps between haves and have-nots. Lastly, there could well be rich theoretical and practical grounds for combining elements of diffusion of change models with other psychologically oriented and well-tested models of human behavior such as the stages of change model developed to understand smoking cessation (see Prochaska, 1992).

Measuring Results In April 2003 the CoEvolution Institute published an important report entitled Measuring Results (Schneider & Cheslock, 2003). This review of research literature focuses on the impacts of non-formal programs in environmental education, museums, social marketing, and health programs. These four domains all seek sustained behavior change in program participants and so have some valuable lessons to share with each other. One of the main findings of Measuring Results is that actually measuring the results of complex human behavior in response to these non-formal programs is a difficult task in and of itself. In the field of environmental education in particular, the authors note a “weak link between theory and practice” (Ibid., p. 26). Interestingly, the theory of change that has perhaps the longest tradition in the field of EE (i.e. that knowledge about the environment leads to positive attitudes about the environment which then leads to pro-environmental behavior, or KAB for short) is not very convincingly supported by the research literature. In summing up the findings of the four behavior change domains as a whole, they note:

The social science nature of evaluation and the focus on human behavior have made for a lack of systematic analysis, which is attributed at least in part to the necessary reliance on self-reported data. Tracking people’s adoption of positive behavior or retention of what they have learned is easier in some case than it is in others. Follow-up is inconsistent and longitudinal analyses are rare…[but] useful and often worth the effort and cost (Ibid., p. 134).

To help strengthen the collective body of evidence for the impact of behavior change programs, the authors recommend “…systematizing evaluation strategies across the field[s]” (Ibid., p. 133) and greater dissemination of measurement strategies and findings. They also recommend rigorous articulation of program goals and mission, and the use of multiple-method research strategies and design. The work of the Place-based Education Evaluation Collaborative could well be a leading example of embodying all of these recommendations.


Beyond the “measurement challenge,2” the Measuring Results report notes some exciting lessons learned about effective behavior change strategies that emerged from their review of the research literature. For environmental education in particular,

People need to know why and how to act in environmentally responsible ways. Effective programs train participants for specific behaviors. In addition…prompts or triggers [e.g. goal setting, commitment strategies, personal reminders, information feedback systems, role modeling] increase the frequency of desirable behaviors and decrease the frequency of undesirable ones (Ibid., p. 46, emphasis in original).

The summary of cross-domain lessons learned echoed the importance of targeting specific behaviors in EE and added two other recommendations. First, programs should tailor interventions to the “individual characteristics and agendas” of the specific program participant audience. Second, programs should directly address the feelings and emotions of participants in order to “instill positive attitudes toward specific actions,” help participants believe that those actions will make a difference, and help them “believe in their own abilities to engage in action” (Ibid., p. 130-131). Summary of literature review Place-based education is still an emerging field. It is not separate from the general awareness of ecological issues that is increasingly a part of public discourse. Programs like those in PEEC are still connected to the longer and stronger tradition of environmental education, even as they work to identify place-based education as a distinct approach. And of course, all PEEC programs operate (or at least strive to operate) in the very heart of the thing called educational reform. It seems that the burden of proof for the efficacy of a place-based education approach lies with the programs themselves. PEEC is putting forth substantial effort to continually and rigorously evaluate their programs and actively seek theoretical and research-based bridges to other fields of study, including areas like psychology, behavior change, and diffusion of innovations. This effort to bridge gaps between far ranging but fundamentally connected arenas is characteristic of place-based education’s philosophical commitment to bridging the global and the local.

2 PEEC evaluators and members have taken to naming “The Measurement Challenge” as a kind of shorthand for all the effort (and opportunity) that is involved in doing this in depth exploration of program outcomes.


EVALUATION METHODS Evaluation of SSP began with the program’s inception at Champlain Elementary School in Summer 2002. SSP staff invited systematic program evaluation from the start of its project in order to better understand the successes and challenges of their process of program development and implementation, and in order to measure the degree to which SSP’s short-term projected outcomes were attained. The second year of evaluation focused on continuing to monitor progress at Champlain Elementary School, assessing the program’s staying power, investigating the integration of literacy and sustainability, and monitoring program impacts on faculty and students. This year’s evaluation also involved gathering baseline data for the newer SSP school, Barnes Elementary. While this is primarily a formative evaluation, the report does provide evidence of positive outcomes as well as suggestions for program improvements, and is intended to be useful to program managers, funders and interested stakeholders, and to contribute to the field of place-based education. Throughout the evaluation process SSP staff played an active role in structuring and contributing to the evaluation process in order to increase the likelihood that evaluation processes and products will be both appropriate and useful for stakeholders. In addition to the evaluators’ interviews, observations and surveys, a “process watcher” role was established among the SSP participants, and SSP staff discussed and recorded monthly reflections about their work at both schools. For the literacy component of this evaluation, the SSP staff played an extremely large role in the implementation of evaluation activities, including data analysis, and especially for data collection. The evaluation team retained strict control of the discussion of findings and conclusions presented in the report as well as overall integrity of the research design, but much credit needs to be shared with the SSP staff for the extensive work they performed in carrying out the literacy component of the evaluation. All of the processes described in this paragraph were designed to build SSP’s internal capacity for evaluation by encouraging staff to actively participate in ongoing documentation, monitoring, data collection, analysis, and reflection.

EVALUATION QUESTIONS The evaluation team consulted the goals and expected outcomes outlined in the SSP Logic Model (see Appendix A) and met with program stakeholders in order to develop the evaluation questions listed in the Figure M1. After SSP staff reviewed and approved the questions, evaluators designed appropriate research instruments to support those questions. (See Appendix for Evaluation Overview 2003-2004 and Instrument templates.)


Figure M1. Evaluation Questions for Sustainable Schools Project 2003-2004 Area of focus Evaluation Questions

Champlain Elementary

• What evidence is there that SSP work is and will be sustained in a school after year one and two?


• As the model develops, what are the strengths and challenges faced by SSP staff and participants?

• What effects does SSP work have on students’ awareness of places and people in their community?


Literacy at Champlain

• How is the presence of SSP helping teachers meet existing literacy goals and requirements?

• In what ways do the data from standardized test scores, locally scored portfolios and teacher perceptions triangulate to suggest effects of SSP on student literacy?

Barnes School • What baseline data will help us evaluate Barnes in future years in terms

of process and outcomes?

PROCESS AND INSTRUMENTS The evaluation utilized multiple methods. Semi-structured interviews and focus groups were conducted with teachers, administrators, parents, community members, and SSP staff members. Students were interviewed informally during site visits. Classroom and school observations, internal reflection instruments, document review, and teacher and student surveys served to triangulate the interview data. Open-ended interview questions are particularly useful in program evaluation because they encourage engaging interactions that help us understand both the process and the outcomes of a program, including what participants know and like about the program, how they have been affected by the program, and what they think should be different (Monroe, 2002). An interview guide was developed that was specific enough to adequately encompass the evaluation questions but flexible enough to meet the stakeholders’ level of participation in SSP activities. See Appendix C for interview guides. Most interviews were audio recorded and transcribed, and one interviewee who was not available for personal interview answered the interview questions on paper. Figure M2 summarizes the data collection methods and instruments used in this evaluation:


Figure M2. Sustainable Schools Data Sources for Evaluation 2003-04

Instrument Type and Number Administered

Interviews

9 Classroom Teachers 3 Specials Teachers 1 School Administrator 1 Community Partner 1 Parent/PTO Member 4 SSP staff 19 interviews total +/- 20 students (informal)

Written Surveys

11 Barnes teachers (Jan – Feb 2004) 53 Barnes students, gr. 4, 5 (Dec ’03 – Jan ‘04) 82 Champlain students, gr. 4,5 (Nov ‘03) 13 Champlain teachers (June ’04) 80 Champlain students (June ’04)

Staff Reflections Forms 4 (Oct, Nov, Feb/Mar/Apr, Jan) Process Watcher Forms 4 (Oct, Nov, Dec, Jan) Project List (of activities that integrate sustainability and literacy) 1 (compiled June, ’04)

Literacy Assessment Questionnaire 13 teachers (October, ’03, June, ’04) Literacy/Sustainability Study Group Reflections

• Written by 5 teachers (September, ’03); • Discussed/dictated by 4 teachers (June, ’04)

Quasi-experimental Assessment of Student Literacy Skills

13 first grade and 4 third grade students (April - June, ’04)

Observations

• School building, school yard • School wide Sustainability Exposition • Students presenting their projects • SSP staff leading classroom lesson • Students composting

Document Review

• Project fliers, brochures, Logic Model • Student work samples • Notes from SSP team planning session • Newsletters (SSP, Champlain Elementary,

Healthy Neighborhoods/Healthy Kids) • Champlain Elementary School Newsletter • Community Works Journal article • Literacy/sustainability study group materials • Sustainability and Informational Text

Project Guide


Literacy Process and Instruments The evaluation team collected both qualitative and quantitative data about the integration of literacy and sustainability at Champlain Elementary. Qualitative interview data was gathered as part of semi-structured interviews that also sought data about other aspects of SSP. A key SSP stakeholder expressed a particular need for quantitative data about literacy, and this greatly informed the evaluation design. The initial plan called for using reading and writing scores from statewide standardized tests to compare various combinations of second, fourth, and fifth grade scores from SSP schools with appropriate district and/or state averages. Also included in the initial plan were suggestions to analyze student writing portfolio grades/scores given by teachers as part of regular classroom assessment. SSP program staff, in consultation with stakeholders and evaluators, subsequently determined that data from a series of four locally designed and implemented assessments (described in detail below) would provide more useful evaluation findings. This decision was primarily based on: 1) the challenge of extracting previous year’s scores and grades for selected students; and 2) the need to isolate effects and interventions related to SSP as distinct from the district-wide literacy improvement efforts. The first three assessments were fairly straightforward. The first was simply the compilation of two lists: one that showed all the major activities within Champlain elementary that integrated sustainability and literacy during the 2002-03 school year; and a second list that showed similar activities that occurred during the 2003-04 school year. These were collected from teachers’ year-end staff meeting discussions with the district literacy specialist, Jane Miller. The second assessment involved administering to the entire teaching staff a simple, one page questionnaire about the use of non-fiction resources. This instrument was designed by SSP program staff and included four questions answered using a graphic frequency scale, and two open-ended questions (see Appendix K). It was administered as a “pre-“ on October 14, 2003 and as a “post-“ on June 18, 2004. The third of these assessments was a two page questionnaire, also designed by SSP program staff (see Appendix L), consisting of open-ended questions about literacy teaching practices. This was administered as a “pre-“ on September 16, 2003 and as a “post-” by discussion on June 1 and 8, 2004 to the literacy/sustainability study group. The literacy/sustainability study group was a specific part of the SSP at Champlain. Five teachers participated in this study group that met eleven times for a total of 30 hours from September through December 2003. The purpose of the group was to build skills in integrating literacy into classroom work and to develop units to support such integration. Teachers selected science, social studies and service-learning as the content areas in which to focus their sustainability integration efforts. SSP project staff and district staff worked to support and evaluate progress toward this goal.


Quasi-experimental assessment of student literacy skills The fourth quantitative literacy assessment was more involved and is described in detail below.

Background The district literacy specialist and SSP program staff together cited the following four reasons for choosing to focus their literacy instructional support on the use of informational (or non-fiction) texts. These texts are seen to be:

1) the most “important” reading genre for life-long learning; 2) critical reading genre for middle grades (e.g. textbooks); 3) more interesting to many students (especially to boys, who often make up the

majority of students performing below standard in reading); 4) typically less often taught and less commonly included in classroom libraries.

Based on this rationale, they planned and implemented the following activities related to using informational texts to integrate sustainability into literacy education:

• Creating and distributing Informational Text Lesson Guides and Booksets that

specifically focused on key teaching/learning strategies for using non-fiction, informational texts about sustainability related topics. (These were introduced to the teaching staff at Champlain Elementary and two lessons were piloted. Full implementation is planned for 2004-05).

• Developing curriculum support for sustainability units, and directly assisting study group participants in designing their own units.

• Assigning readings and facilitating discussion with study group teachers. • Co-teaching with study group participants (and others) in order to refine

teaching strategies and skills. • Piloting an assessment exercise in the classrooms of the first and third grade

teachers who participated in the study group.

Designing the assessment exercise Piloting the Informational Text Reading Assessment constituted the fourth piece of SSP’s literacy evaluation strategy for the 2003-04 school year. These steps were followed:

1. Teachers, SSP staff, and the district literacy specialist collaborated to identify the key skills to be assessed. In each case, the skills targeted (i) one of the conceptual skills identified as fundamental to understanding sustainability (and therefore part of that grade’s focus in the articulated K-5 Sustainability curriculum framework); (ii) a key reading comprehension strategy targeted by the district’s literacy program; and (iii) a key informational “text feature” essential to successful reading.


2. SSP staff and literacy specialist developed criteria for making sure the identified skills were developmentally appropriate to the grade level.

3. SSP staff and literacy specialist designed grade- and unit-specific assessment tasks for the sustainability units.

4. SSP staff and literacy specialist developed a three-point rubric based on the results of steps b) and c) above, then reviewed/revised the rubric in consultation with classroom teachers (see Appendix M).

5. SSP staff and literacy specialist met with the evaluator to design the sampling and administration procedure described below.

Design notes • Informed consent from parents was not sought since the whole procedure was

so similar to regular classroom assessment processes. • This was not a true experiment in the most technical sense because there is no

“control” group that did not receive the “treatment” of the lesson.

Sampling procedure Teachers of two first grade classrooms provided class lists that grouped their students into high-, middle-, and low-performing readers. Two or three students from each category of reader from each classroom were randomly selected from these lists by choosing those students present in the room on the first day. Thus, the group of 13 first graders comprised a somewhat stratified random sample which is fairly representative of the 42 students that make up the whole first grade at Champlain Elementary. A similar selection process was attempted for the third grade but scheduling problems and one teacher’s decision to withdraw from the process resulted in a sample of only four third grade students. Analysis for the third grade sample cannot be reasonably claimed to be representative of the third grade population at the school. The combined sample of 17 students was then divided among two raters (the district reading specialist and SSP program staff) for administration as described below.

Administration procedure Working with individual students (as opposed to small groups) in semi-private spaces in the school’s learning center (outside the classroom), each rater read the selected text to the student and then asked the student the three questions. Student responses were then rated according to the rubric. Students were assessed before and after the classroom teacher and district literacy specialist delivered the lessons related to informational texts from the units designed by district reading specialist. Each assessment session lasted approximately 8-12 minutes. The pre-tests were administered approximately one month before the lessons, the lessons lasted 30


minutes and the post-tests were administered four to seven days after the lessons. This investigation took place on several dates between April and June, 2004.

Survey Administration Process PEER Associates encourages program staff to be involved in the administration of evaluation instruments to the greatest reasonable extent. Written guidelines for survey administration are provided to SSP staff. Evaluators oversee program staff as they download, copy, distribute, confidentially collect, and remit all survey instruments. For a detailed description of the survey administration process, see Appendix J. By the end of the evaluation period, the opportunity existed for most surveys administered by PEER to be taken by participants in an on-line format, rather than as a hard copy piece of paper. Many of these can be previewed by following the links at http://www.peecworks.org/PEEC/PEEC_Inst/. However, SSP staff and evaluators decided together that the logistics would be easier for the SSP surveys to continue to be administered in paper format during 2003-04.

DATA ANALYSIS Below are summaries of the various techniques and systems used for each type of data that was collected for this evaluation.

Analysis of Interview Data and Documents After fieldwork was complete, descriptive observation data, additional documents, and transcribed interviews were coded to illuminate key emergent issues and answer the evaluation questions (Miles & Huberman, 1994). The most prevalent themes emerging from the data were analyzed and are synthesized into this report. Specifically, the evaluators used the following protocol for analyzing the data:

1) Wait until the vast majority of data has been collected. 2) Read through all data (making minimal notes) for the purpose of clarifying

the context and getting a holistic impression of the data set. 3) Create an initial list of 5-20 themes that seem to reflect the data.

Note: When more than one individual is involved in the data collection, analysis or write up, each person must go through steps 1-3.

4) Code all data according to the theme list, while remaining very open to the emergence of new themes, sub-themes, and meta-themes, including three categories: Irrelevant, Probably Irrelevant, Need to Confer.

5) As the remaining data is collected, code it according to the theme list. 6) Look within the data from each theme, sub-theme and meta-theme and

recode as necessary to establish clarity and coherence within each level. 7) Generate an outline of the findings and discussions section of the report based

upon the final theme list.


8) Write up the narrative based upon the outline, pulling in data from documents and transcripts to support as appropriate.

9) Each person involved in the above steps agrees that the final analysis is consistent with their experience. Note: If a Research Assistant is involved in the process, then the primary author checks their work at each step.

In the case of SSP data, all of these steps, including those in the notes, were employed. Additionally, NVIVO software for qualitative analysis was used to code and sort the data.

Analysis of Literacy Data Responses to the open-ended questions on written pre-post instruments were analyzed together with the portions of interview transcripts that were relevant to the question of literacy integration, using the same protocol as noted above for analyzing interview data. Scores from the quasi-experimental assessment of student literacy skills were compiled and reported using descriptive statistical techniques. While the design of this investigation was robust in many ways (e.g. stratified random sampling, and informal checks of inter-rater reliability), the sample size is small enough that more formal inferential null hypothesis statistical tests were not warranted. The findings from both qualitative and quantitative sections of the investigation into literacy were woven in with the non-literacy components of the evaluation to form the Conclusions and Implications for Practice sections of this report.

Analysis of Survey Data

Pilot measurement strategy Following the 2002-03 evaluation cycle, the members of the Place-Based Education Evaluation Collaborative wished to strengthen the quantitative elements of their evaluation strategy. Emerging from the September 2003 PEEC meeting, the top priority for the 2003-04 PEEC cross-program research agenda was to develop a way to measure the extent to which outcomes of interest varied consistently with degree of program exposure. By demonstrating the effectiveness of this pilot measurement strategy in 2003-04, PEEC has created a very solid methodological foundation for on-going evaluation in years to come. The allocation of intensive effort this year to measurement challenges was made possible through the pooled resources of PEEC programs, aided by the support provided indirectly by the EPA through an evaluation team member’s doctoral fellowship.


The evaluation team settled on a “dose-response” approach, adapted from concepts in the field of behavioral psychology (Strosahl et al.,1998). From this frame, the question becomes: Do participants exposed to a higher “dose” of a program systematically report higher levels of behaviors and attitudes that the program is trying to impact? "To the extent that relevant outcomes vary with dose, assertion of program impact becomes increasingly credible." A primary benefit of this dose-response measurement strategy for PEEC is that evidence of program impact can be assessed using survey data collected at any point before, during, or after program completion, provided there is enough variation of dose among program participants. This sidesteps numerous conceptual and logistical obstacles to pursuing strict pre- and post-program measurement efforts, and also allows a relatively standardized set of surveys to be more easily used across the diverse program designs in PEEC. The main thrust of all the survey-related sections of this report is to present the findings of the effort to pilot a dose-response measurement strategy. Thus, this report takes an educative approach to presenting statistical findings. We omit some of the more descriptive statistical representations (e.g. simple bar graphs) in favor of putting more focused effort into helping the reader understand the meaning and power of the inferential statistics that undergird this dose-response measurement strategy. It will substantially increase the long term value of this pilot measurement strategy effort if key program staff have a solid understanding of the significance of the statistical foundation upon which it is based. There are, however, additional questions that could be explored with the existing data, beyond the scope of the current report. It is possible that PEEC programs might individually or collectively find it useful to more exhaustively analyze some of the survey data already collected, in order to produce a supplementary “quantitative only” evaluation report or pamphlet targeting specific decision-making needs of priority stakeholders. Another consequence of this focus on piloting a measurement strategy is that some of the most important evaluation “findings” are actually presented here in the “methods” section of the report for the sake of coherence of the narrative. Testing the method was the finding.

Instrument design strategy Indices and Modules Much of 2003-04 was spent developing and refining a set of survey modules that were standardized for use across all PEEC programs. Design elements were drawn from previously implemented surveys from several PEEC programs, as well as from individual program logic models and the PEEC cross program theory of change. This standardization process allows for much larger samples and more varied comparison groups over longer time periods. Further, it is hoped that the place-based education


philosophy that is shared by all PEEC programs and designed into the fabric of these surveys will make the instruments transferable/modifiable for use in other place-based education programs. All survey instruments are available for free distribution (citation requested) on the PEEC web site. The design process began by grouping items from existing surveys into broad conceptual categories, and adding additional categories to the list where needed. This resulted in a list of eight core ideas for potential inclusion across all PEEC surveys. We call these “modules.” Each module was then broken down into two to five constituent “indices,” each intended to capture a distinct element of the idea represented by the module. Similarly, two or more “items” (i.e. individual survey questions) were developed for each index, taking care to use items from existing surveys whenever possible in order to maximize the possibility of comparing current and future results to previously collected data. Finally, because changes in teacher practice are at the heart of many PEEC interventions, we grouped several modules together to form, at the highest level in this scheme, an “overall module” construct representing all survey elements that reflect teacher practices targeted by the PEEC program. Several design guidelines were used in the construction of survey items. Most modules have one negatively worded item as a way to help keep respondents focused on the meaning of the response options. Extreme wording was avoided while constructing the phrasing for items. Response scales for all Likert items were standardized with four interval points (either agreement or frequency) in order to oblige respondents to choose either the top half or the bottom half of the scale (or else the systematically included option for “n/a, don’t know”). From what might be regarded as a "test bank" of survey components, program-specific surveys were then compiled by choosing the highest priority modules and indices for each program and audience (i.e. community member/partner, educator, or student). Drafts were compiled by evaluators and shared with program staff, with each instrument going through multiple rounds of editing and/or field testing (sometimes more than dozen!). Many of the surveys were converted into on-line versions through the commercial software package, Survey Monkey. By 2004-05 the evaluation team intends to have on-line administration options for all survey versions. There are three important reasons for using this indices and modules design approach. The first is that the survey design begins with relatively broad questions or concepts. Although it proceeds to more intricate items, the "answers" we're interested in are usually not focused on the item level, and we want our analytic strategy to be consistent with our primary interests. The second reason is that each time one conducts a statistical test, there is a small probability of reaching an erroneous conclusion on the basis of that test (think of the "margin of error" reported with polling results). That probability of error compounds with multiple tests, and for this reason it is common


practice to minimize the number of tests by reducing the number of "scores" one has to work with--a process referred to as data reduction. Conducting our analyses at the level of indices instead of individual survey items reduces the risk that we will be attempting to extract meaningful interpretations from what are, in fact, chance arrangements of the data. Finally, the third reason for this approach is to stabilize the construct being analyzed. Individual survey items can be modified slightly without having to change the index. This was crucial during 2003-04 because different versions of surveys needed to be administered (even within the same PEEC program) as a consequence of the iterative design process. This also allows for the refinement and evolution of the wording of survey items. Finally, it is hoped that this indices and modules design approach increases the transferability of surveys to other place-based education programs. The resulting indices and modules are summarized in Figure M3 below. See Appendices D and E for samples of surveys used for SSP this year. A complete list of all survey items used for PEEC surveys is available on the PEEC web site. It was conceded at the beginning of the design phase that any measurement of the “dose” of a program would be necessarily imprecise. PEEC programs are not simple training seminars that are over and done with in two hours. Rather, they are multi-faceted, highly adapted to local conditions, take place over extended periods of time, and intentionally rely on informal diffusion throughout the school and community in which they take place. These factors make it difficult to definitively determine even who is an official “program participant.” Still, any measure of dose would likely be more accurate than no measure of dose, and so the number of hours of direct, personal contact with formal program elements was taken as the starting target for measurement. Comprehensive lists of the core activities for each PEEC program were developed, and survey respondents were asked to estimate the number of times that they had participated in each activity over the years of their involvement with the program (see item D1 on the survey in Appendix D). Separately, program staff generated estimates of the average duration of a typical example of each activity in the list. From this information it was possible to calculate a rough estimate of the total number of hours of participation. Additional multiple choice items were created which asked participants for overall estimates of: the number of hours of participation (item D2); the frequency with which they were implementing the program in their classrooms (item D3); the extent to which the program is embedded in their curriculum (item D4); and the amount of effort they expended on this program relative to other aspects of their teaching (item D5). During the Spring of 2003 we realized that it was also critical to find out the amount of training participants had received in place-based education prior to involvement with PEEC programs, and so an additional question to that effect (item D12) was added for later versions of the surveys.


Figure M3. Summary of PEEC Survey Modules and Indices


Analysis strategy for Likert scale data Assumptions First, it is important to reiterate that the entire dose-response measurement strategy is built upon the assumption that program dose is, in fact, a latent construct that can sensibly and accurately describe the experience of an individual participant in one of these PEEC programs. If one does not accept this assumption, then one should disregard all reported survey results involving inferential statistics. Second, there is debate in the academic community about the validity of applying parametric statistical tests to data from Likert scale type sources (see Newton & Rudestam, 1999, pp. 179-187). We are convinced by the side that argues that such tests are appropriate due to the assumption that the latent constructs being measured are continuous in nature, despite being manifested in the data as ordered categories. Third, because the primary goal was to pilot a measurement strategy, a broadly exploratory approach was taken during the first stages of data analysis. In order to protect against over-interpretation of correlations between dose and program outcomes at multiple levels of analysis (i.e. item level, index level, module level, and overall module level), or the critique of going on a “fishing expedition” for affirming results, stricter criteria for statistical significance were assumed prior to analysis. The Bonferoni correction stipulates that for aggregate categories of analysis, the p value statistical significance threshold should be divided by the number of component constructs. In our case, this means that the p value threshold for modules and overall modules is closer to .01 than the typical .05. The final and most general assumption is that deep methodological rigor in statistical analysis was extremely important for PEEC. In the highly politicized arena of education, quantitative evaluation and research findings are at a premium, even though many consumers of the findings may not be substantially trained in critically examining the math and philosophy underlying statistical presentations of data. Thus, the evaluation team took the position that PEEC members (i.e. our clients) need to be entirely confident that any statistical evaluation findings they present to stakeholders are thoroughly defensible on methodological grounds. Tools Since the number of surveys collected for PEEC during the 2003-2004 evaluation cycle was fairly large (i.e. over 1600 total, including 239 for SSP), most of the survey data entry was outsourced to professional data entry sub-contractors or completed by graduate research assistants for PEER. In all cases, data entry specialists were provided with a detailed data entry protocol and database (SPSS) template files that were already configured for the specific surveys. Responses to open-ended questions were typed into MS Word documents and analyzed separately.


Additionally, a journal was kept that documents the key junctures in the exploratory process of coming to a viable analysis strategy. Survey analysis process findings In short, the dose-response measurement strategy tested this year has generated defensible inferences about program effects. Refinements are certainly warranted, but overall the pilot test can be considered to have been “successful.” The paragraphs that follow describe process findings for the measurement of dose at the teacher, student, and school levels. The measurement of dose at the teacher level was the first and most intensely investigated element of the strategy to be pursued. In the first step, a data set was constructed that included educator data from three of the four PEEC programs3. Bivariate correlations between each of the potential dosage items (D1-D5), and program outcome indices were analyzed4. As new data sets became available from all PEEC programs, interpretative hypotheses were continually generated and tested on the aggregate data sets and also on individual program data sets. Eventually, items D2 (participant estimate of overall hours of exposure to formal program elements) and D5 (amount of effort put into the program relative to other teaching activities) were excluded from subsequent analyses, due to low and/or highly variable correlations with other data. A composite dose score was then calculated by scaling the raw score for item D15 and averaging that result with the average of the scores for item D3 (frequency of program activities in the classroom curriculum) and item D4 (extent to which the program is embedded in the year-long curriculum). The resulting dose composite score was a number between 0 and 4.0. This dose scale score was then successfully cross-validated with various combinations of the original data set and with all new data sets as they became available. Measurement of dose at the student level was more problematic. Many of the student survey instruments contained student versions of items D3-D5 with the intent of corroborating educator reports of program implementation. Averaging the scores on these items into a student dose variable did, indeed, reveal statistically significant correlations between dose and selected program outcomes (which were typically higher than those obtained by simply assigning students the same dose

3 Administration of the educator survey from the fourth PEEC program and subsequent data entry were not yet complete. These data became available within two weeks and were immediately incorporated into the analysis. 4 …by calculating Pearson correlation coefficients and also by systematically viewing vicariate scatter plots and frequency distributions for outliers or other confounding anomalies. 5 Item D1 generated an estimate of the total number of hours of exposure to formal program elements. The raw scores ranged from 0 to over 500. Assigning a cumulative value of .5 for each 1/3 of a standard deviation in the raw score range generated a scale of 0-4 that is compatible with the scale of items D3 and D4.


composite score as their teachers), but this approach was considered problematic on conceptual grounds. We realized that this measure of student dose is not likely to be independent of student engagement. Students who were engaged in the program are likely to systematically remember more and/or overestimate the amount that the program was a part of the experienced curriculum. We thus settled for assigning students with the composite dose score of their teacher whenever that data was available. This raised three important implications:

1. Does it even make sense to conceptualize dose as varying among students in a given classroom? This question needs to be explored further with program staff to determine whether or not to continue efforts to find a suitable student-level dose measure other than that of the student’s teacher.

2. Since student dose is determined to be rationally derived from the dose of

their teacher, it becomes essential that student and educator surveys be administered at the same time. There were several cases in 2003-04 where this did not happen. For example, the dose-response statistical analysis could not be run for the situations in which the student surveys were administered in the spring while the educator surveys were administered in the fall.

3. It is critical that educators put their names on the surveys. Otherwise there is

no way to assign a dose to students.

Perhaps the most far-reaching results of PEEC’s pilot of a dose-response measurement strategy comes from looking at dose at the school level.6 It appears that, at least for whole school change model programs within PEEC, teacher level dose begins to be superceded by a school level dose after a program has been involved with that school for a year or more. It seems that as programs become an accepted part of the school culture and norms, duration of the program at the school becomes a more accurate predictor of program outcomes for an individual participant than the individual’s own teacher-level dose. This argument is discussed more fully in the PEEC 2003-04 Cross-Program Evaluation Report (PEER, 2004) and provides striking support for the educational impact model proposed by Marzano (2000, 2003) that was discussed in a previous section of this report. The “response” side of the pilot dose-response measurement strategy findings was considerably more straightforward than the “dose” side. Only one potential challenge with using the above described indices and modules system for measuring program outcomes was found. Educator responses to many of the questions tended to be skewed toward the “agreement” side of the scale. This “ceiling effect” imposes limits on both

6 This general approach was loosely tested in the 2002-03 evaluation of CO-SEED, a program in PEEC, with inconclusive results (Powers & Duffin, 2003, p. 61).


the power of statistical analyses with this data, and relatedly, on the sensitivity of these measures to program effects. Some suggestions for identifying and interpreting this ceiling effect are provided in the next section. One implication of skewed distributions for the dose-measurement strategy is that simple averages (means) may not accurately describe the central tendency of responses, so we have reported medians in addition to means. Another implication is that it may be advisable to explore ways of rewording many of the survey items to push the average response lower on the agreement scale. The distribution of student responses, however, was almost perfectly normal (i.e. a bell-shaped curve) for all indices and modules across all programs. This supports a claim for a relatively high degree of instrument and/or construct validity for the student survey items. Survey administration process findings On-line and paper administration of surveys each have their advantages and disadvantages. Paper administration tended to work better in situations where surveys were administered at a school-wide staff meeting or in-service day. As availability and comfort with internet access increase, this advantage may slowly erode over time. Additionally, data entry for on-line survey administration becomes incrementally less expensive in terms of both time and money as each sample becomes larger than a dozen or so. More importantly, the sample of educator and student surveys collected across PEEC was large enough in 2003-04 that continued increase in sample size will not necessarily lead to increases in statistical power or significance. This is especially true for the larger programs in PEEC (i.e. CO-SEED and CMP), and less true for the smaller programs (i.e. SSP and FFEC). With each succeeding year of evaluation efforts, a cost-benefit analysis of survey administration is likely to favor on-line administration to a randomly or purposefully sampled set of participants instead of paper administration to all participants. This does not, however, take into account the less tangible benefit of having participants fill out surveys as a way to engage them with the intended outcomes of the program. As mentioned above, one of the primary advantages of this dose-response measurement strategy is that the measurement event can take place at any time in the program cycle. This provides an excellent opportunity to avoid administering surveys during the end of the school year rush in April and May. We recommend that programs work to create a culture and expectation of a regular time of year (e.g. mid- or late winter) as the “survey season.”

Notes for interpreting tables and graphs The “ceiling effect” mentioned above is an example of a phenomenon that can be interpreted from the data tables presented. If responses are skewed toward the


“strongly agree” or “once a month or more” ends of the scale, the mean will be higher than the median. The mean (indicated by the letter X) is the same thing as the arithmetic average. This is the measure of central tendency that is often easier to understand since it is more common in public discourse. The median (indicated by the letter M) is the number that divides the top half of responses from the bottom half of responses. Inferential statistics deduce mathematical patterns in a given data set and then use that pattern to predict dependent variables (marked on the vertical axis of a graph) from given independent variables (marked on the horizontal axis). In the context of this report, that translates as: ‘given the responses that people actually made on these SSP surveys, if a teacher had a given score of such and such for dose, then they would most likely to have a score of such and such for this particular index or module.’ The multiple regression line on the scatterplot graphs in this report represents that statistical prediction. If the regression line slopes from lower left to upper right, it is essentially saying that the higher the dose of the program a participant has, the higher they are likely to rate themselves on the intended program outcomes. Similarly, a lower dose (e.g. dose = 0, which is equivalent to the typical “pre-test” situation), predicts that a participant will report lower scores on desirable program outcomes. Sloping lines on graphs in this report can be broadly interpreted as evidence that the program is likely to be contributing to the desired program outcomes. For clarity’s sake, graphs are only shown if they meet additional tests for “statistical significance,” which is a fancy way of saying that it is highly unlikely that observed results are due to chance only. The slope of the regression line represents the strength of the effect of the program. Steeper slopes suggest stronger effects of the program. This is represented in the data tables in this report by the variable R2 which is directly interpretable as “percent of variance.” For example, if R2 = .2 for the overall teacher practice module, that means that 20% of the variance in teacher practice can be predicted by the extent to which participants have been exposed to the PEEC program, i.e. their dose. But what percent variance constitutes a large effect for PEEC programs? The literature review section of this report provides a couple of benchmarks for comparison. The Coleman report claimed that schooling accounted for only 10% of the variance in student achievement (Coleman et al., 1966). Marzano claims that that number is actually closer to 20%, with 13% deriving from teacher-level factors, and 7% attributable to school-level factors (Marzano, 2003). Studies asserting that motivation predicts student academic achievement show correlations (which are directly comparable to percent variance or R2) values ranging from .19 to a whopping .63 (or 19% to 63%). As a final benchmark for this somewhat arcane (to the lay reader) statistic of percent variance or R2, consider “Abelson’s paradox”


which applies this statistical method to the everyday world of professional baseball. “The percentage of variance in any single batting performance explained by batting skill is about one third of 1%” (Abelson, 1985, p. 131). Further, and perhaps more revealing for PEEC,

". . . the attitude toward explained variance ought to be conditional on the degree to which the effects of the explanatory factor cumulate in practice. Some examples of potentially cumulative processes are educational interventions, the persuasive effects of advertising, and repeated decisions by ideologically similar policy makers. In such cases, it is quite possible that small variance contributions of independent variables in single-shot studies grossly understate the variance contribution in the long run" (Ibid., p. 133, emphasis added).

Essentially, Abelson is arguing that small statistical effects can indicate very important outcomes in educational contexts. In sum, if 10% (or R2=.10) of a teacher’s attitude or behavior as reported on a PEEC survey can be attributed to that PEEC program (especially given that there are so many other factors at play) this could defensibly be interpreted that the program is almost certainly having some noteworthy effect. Correlations of .30 (or 30%) likely represent large effects. This section closes with two important caveats. First, the number values of the dependent variables (i.e. program outcomes, on the vertical axis) shown in any graphs in this report should not be interpreted as corresponding to any direct physical reality. In actuality, they are averages of averages for groups of self-reported responses. Statistics do enable us to identify systematic patterns in the data which can be broadly connected to observed phenomena, and which can be very helpful in making important programmatic decisions. But it is not defensible to assert that a “3.5” on, say, the student academic achievement index, corresponds directly to a certain test score or any other isolatable empirical event. Second, we encourage the reader to heed the old admonition that “correlation does not equal causation.” The statistical data presented here (and in almost any other piece of social science research with statistics) are correlational. They say that one thing is more likely to be observed when another thing is also present. That is it. The notion of what constitutes “causation” has been, and continues to be, hotly debated in academic and political circles. In a recent paper, Michael Scriven, a very well-respected educational researcher/evaluator notes that, “This concept [causation] has had a stormy history, from which it has not yet emerged” (Scriven, 2004, p. 1). Generally speaking, and for program evaluation in particular, this evaluation team proposes that “validity” is in the eye of the stakeholder.


FINDINGS AND DISCUSSION This year’s evaluation of SSP focused on both process and outcome elements. In particular, we monitored the impacts of the program on teachers and students at Champlain Elementary School, now in its second year with SSP. Eight sections of findings follow, detailing the major areas of focus in the evaluation as well as emergent themes:

• Teachers’ Curriculum Planning: Increased Coherence And Integration • Student Awareness of and Engagement in Community

o Citizenship Projects, Community Exploration, Student Sensitivity • Other Student Outcomes

o Health and Sustainability, Student Learning, Taking Sustainability Home • SSP Process Strengths • Evidence Of SSP’s Staying Power • SSP’s Process Challenges • Teachers Integrate Literacy And Sustainability • Presentation Of Educator and Student Survey Data: Barnes And Champlain

Schools

TEACHERS’ CURRICULUM PLANNING: INCREASED COHERENCE AND INTEGRATION One of SSP’s primary strategies is to help teachers plan and implement sustainability-related curricula through the summer institute and ongoing coaching and support during the school year. The intention is not only to work with individual teachers in isolation, but to help them put their curriculum into a schoolwide context so they can see how underlying concepts of sustainability can build from grade to grade. There has been ample evidence this year that such practices are flourishing and becoming more embedded in the school’s operations. Comments in support of the type of planning that SSP is cultivating in the school were reiterated across the grade levels, including kindergarten. In addition to an overall sense that teachers were thinking differently about the curriculum planning process as a whole, there were three specific areas in which teachers described changes in their teaching practice. Teachers are increasingly:

• Using essential questions • Creating interdisciplinary units • Connecting specials and classrooms

Evaluation Question: In what way is teacher practice

changing and sustained as a result of involvement with SSP?


According to SSP staff, these changes as among the biggest impacts SSP has had at Champlain. Teachers remarked that the major impact has been on the way they think about curriculum development as much as on the topics they teach or how they teach. Significantly, a number of teachers reported that the practice of developing a curriculum map has given them a new understanding of what is happening at grade levels throughout the school. This heightened awareness appears to have contributed to a more unified experience for students, as described below.

Using Essential Questions A key element of the SSP approach to curriculum planning is the use of essential questions as a planning tool. According to SSP staff, “the essential question idea is really a meaningful, long term thing that can bring everybody together.” Last year’s evaluation reported that teachers were beginning to use essential questions more, and this year teachers spoke even more highly of the benefits of this approach. Two teachers specifically reported that because of SSP they will think of their curriculum differently and not start with activities but, rather, with the “essential questions and enduring understandings.” A second grade teacher had expressed some initial hesitation about organizing her curriculum under one unifying theme, worrying that it would focus the kids too much and she would not be able to appeal to their interests in a variety of science topics. What she found, however, was that under the sustainability umbrella and using her essential question about cycles she “could teach those sub themes, integrating them in very nicely.” In the eyes of one administrator:

…this faculty is looking at the curriculum and grade level expectations and framing it in terms of an essential question and a theme. That process has been a stimulating one and I can see with this faculty that that’s sustainable.

Interrelated Units of Study Rather than teaching topics in isolation, SSP staff have worked to assist teachers in integrating their teaching. A first grade teacher recalled how different her teaching was before she shifted to an interdisciplinary focus:

“SSP brought an awareness to our social studies and science curriculum, and made us sit down and think about the scope and sequence of where we started in kindergarten and what we hope fifth graders will leave our school with. That helped us with our teaching and taking a look at our school.” -Kindergarten teacher

“[SSP teachers] have evolved in their thinking about what they are teaching.” - Community Works Journal (Spring 2004


I remember a time when we were doing many different units. We would do oceans and dinosaurs and space, you know all the kind of typical world-around- us stuff, but it's bigger than that now.

A second grade teacher referred to her own evolution toward a bigger picture:

…it's given us an opportunity to realize how interrelated everything is and how we can start with a big question, an essential question, or start with one idea and everything connects, and I think that's been the best thing for me is how interconnected everything is.

In concert with linking the curriculum across the grade levels, teachers were found to be doing more to link together the different units of study. Indeed, more teachers were putting lessons into the greater context of sustainability, thereby making the whole curriculum more interdisciplinary. One teacher found that, due to growing pressures in the educational arena to cover more material, it had become essential to bring all the subjects together, simply to fit in all the requirements.

Everything we do revolves around a theme; we pull in the math and the writing, whether it's your science journal or a published book. We pull in artwork. It happens in second grade, definitely. And it will continue to happen because it's really the only approach that will work, the interdisciplinary approach. There's so much that's put on schools now that you can't teach things in isolation. You won't get to everything, so everything has to be interconnected.

This teacher explicitly stated that, while she feared at first that the school focus on sustainability would add work to her plate, she found that in reality the sustainability umbrella aided her in meeting requirements which she realized she would not be able to meet without integrating subjects under a common theme. One example of interrelating different subjects was highlighted at the Sustainability Exposition, where third grade students shared the math word problems that they had written in a lesson linking nature, literacy, and math. See sidebar.

• “If there are 8 red-eyed tree frogs, 4 bullfrogs, 7 green frogs, how many frogs would there be in the pond altogether?”

• “We were observing the pond. We found that

25% of the fish were goldfish, 50% were perch, and 25% were trout. If we had a fishing pole, what kind of fish do you think we’d be more likely to catch today?”

-Third grade students’ math word problems

which they had created in a lesson linking nature, literacy, and math


Connecting Specials and Classrooms Similar to the way in which teachers have been relating different units of study within the home classrooms, there is evidence that specials (including art, afterschool program, physical education and music) are increasingly connecting to classroom studies, creating a more coherent learning experience for the students. While the physical education teacher seemed to connect sustainability activities with those of the regular classroom teachers during year one of SSP, such integration was not apparent in the practice of other specials teachers. This shift toward a more universal integration is a positive indicator of SSP’s incremental schoolwide spread in year two. In the case of the afterschool program, sustainability became a means of “recruiting” new teachers to teach programs. The afterschool program coordinator reported that teachers who had not previously been a part of the afterschool program were now accepting the offer to present sustainability themes. She reported that if she “…can just hit on the right things to start the teachers’ interest…” then they get excited about participating. For certain teachers, sustainability was what piqued their interest. The coordinator noted that as a result of SSP she had formed new collaborative relationships with teachers, and was very positive about getting them to come back in future years. It is worth noting that about ten of the approximately 60 afterschool programs this year had sustainability themes. This is partially attributable to the fact that SSP provided an AmeriCorps Vista member who assisted with infusing sustainability themes into the afterschool program. Another significant reason for the increased infusion of sustainability into the afterschool program was that the afterschool program coordinator said she had “tried to shift over to the sustainability mode.” She noted that this was a dramatic change:


During my first two years here we didn't have the sustainability project at all and we had no classes that were, in any way, even closely related to sustainability. I think it's expanded what I've been able to do in the afterschool program having that available.

The art teacher found that SSP had helped her link to the classrooms, and felt that she had worked “fairly collaboratively” on a few occasions. She stated that it was “…really valuable for us to have connections between the classroom and the special.” One exemplary project she conducted with the whole school involved students making drawings of a local animal species and doing research on some aspect of its natural history. Students’ final products were impressive displays combining art and science. Even though the art teacher expressed frustration over administrative issues such as limited time and resources, a large number of students each week, and being split between two schools, she felt that sustainability “…was something that felt like it could fit into my curriculum. I did it, I enjoyed it, and the kids did a good job on it too.”

SSP staff reported that the music teacher is enthusiastically incorporating sustainability into her curriculum, and the physical education teacher explained that there was a connection between teaching about health and self-care and taking care of one’s community. She also underlined the importance of having a schoolwide organizing principle integrating the learning experience for students:

…I want to teach about responsibility--like being responsible as a community member and being responsible as a member of society as a whole…. [SSP] has really made it easier to integrate since it’s talked about throughout the school.

In addition, this teacher expressed allegiance to the sustainability

theme, something she had come to realize she had been teaching all along even “before [she] knew what it meant.”


STUDENT AWARENESS OF AND ENGAGEMENT IN THE COMMUNITY Through SSP work, students were exposed to diverse new places in the community, and--equally importantly--they began to develop a deeper understanding of places they already knew. This exposure came on various levels: a greater awareness of schoolyard flora and fauna, increased knowledge of the different groups that make up a neighborhood, or heightened insight into the ways that a business fits into the local economy. There were three key areas which highlighted the impact of SSP on student awareness of and engagement in their community:

• Grades 4/5 Team Tackles Citizenship Projects • Younger Grades Explore the Community • Students’ Sensitivity Grows with Community Engagement

Grades 4/5 Team Tackles Citizenship Projects In particular, the grades 4/5 team engaged in the Healthy Neighborhoods/Healthy Kids Project in conjunction with their community partner, the Vermont Forum on Sprawl. Students divided into teams whose projects included sprucing up neighborhood parks, raising money for a basketball court, cleaning up a ravine near the school, and organizing a neighborhood block party. Over the course of these projects, students presented at Neighborhood Planning Association meetings and City Council meetings. They also met with the Mayor,

“The kids have told me that they feel invested in the health of their neighborhoods. They are proud of their community improvement projects and understand the importance of maintaining clean, safe neighborhoods.”

-SSP Community Partner


members of the Parks and Recreation Department, and members of the Public Works Department. A grades 4/5 teacher reported that students were “…taking an active role in citizenship and learning how to access city government, services, and people that make decisions which impact their lives. They're learning how to make a difference as a citizen.” At the year-end, day-long Sustainability Exposition, the students were available to talk about what they had done in their groups in the Healthy Neighborhoods/Healthy Kids Project. When questioned about the impacts of their projects and what they had learned, they demonstrated knowledge about the concepts of sustainability, and more importantly a sense that the value of their work went far beyond the academic realm. Students who cleaned up the brook near the school explained that the brook was part of the watershed feeding Lake Champlain. Recognizing that Lake Champlain is the source of their drinking water, students had become dismayed at the level of pollution evident in the brook. Consequently, they had produced a brochure containing ideas about keeping the lake clean. It is significant to note that the last paragraph in the brochure reads, “You may think we’re just trying to help animals, and we are, but we’re also trying to help people.” As simple a statement as this is, it is a good example of the sort of big-picture thinking that sustainability education intends to impart. The students are beginning to see themselves and their neighbors as part of the ecosystem. When asked if they thought the lake would be clean someday, their response demonstrated both ownership of the issue and empowerment to act: “I think that we can make a difference.” In the SSP

Figure F1. Examples of School/Community Cooperation

Note: In addition to students venturing out into the community, numerous community members have come into the school on an ongoing basis to share their talents, skills and stories, and to help students develop their own skills. • Mayor came to school on numerous

occasions including the Sustainability Exposition day

• Community partners and former teachers taught afterschool programs

• Burlington Neighborhoods Project worked with 4/5 on community improvement grants

• Abenaki Grandmother came in to work with the school

• A local university student helped work on a garbology unit.

• A first grade teacher worked with the Senior Center to develop a multigenerational unit.

• Students met with a local doctor to talk about health choices.

• Students did a mapping project with a staff member from Lake Champlain Maritime Museum

• Burlington Electric Department installed solar panels


“Community work is more fun than you’d think. I’d do it again. But not on my own because we needed to have adults help us plan things and get paint and stuff.”

-4/5 Student

newsletter, a staff member highlighted the importance of this type of result for the students:

Knowledge and connection to a place do not make an engaged citizen, however. In addition, students need to feel confident in their ability to make a difference with this knowledge.

Students who worked on improving neighborhood parks made various connections between sustainability and their projects. Some commented on how they had improved the social and environmental quality of life in their neighborhood. They explained that they had “…made the park more attractive for the community.” Acknowledging that keeping benches well painted will save time and resources over the long run, one student explained, “Sustainability is about making something last…. So we made the park a better place that will last longer.”

Younger Grades Explore the Community Complex, community-focused projects tend to be adopted by older grade levels because the students are more developmentally prepared for such tasks, and because it can be easier to take them off the school grounds (i.e. younger students need booster seats, older students walk faster). While these types of projects are more likely to generate community acknowledgement and create acclaim for the program, it should be recognized that the intention of SSP is also to create the building blocks of citizenship at the lower grade levels. A second grade teacher suggested that a sense of community and stewardship “…starts small with the school grounds and fans out to the earth in general.” Teachers throughout the school offered examples of students learning about new local places and beginning to think differently about their community. A first grade teacher described how her students began to develop not only a better grasp of local geography, but also the desire to demonstrate that understanding:

At the beginning of the year the students come in with this vague ’yeah, we went to Grandma’s’. And when I ask, ‘Oh, where is that?’, they say, ‘I don't know.’ So I begin telling them, ‘Well, your job today is to go home and find out.’ After hearing that conversation a few times kids begin getting savvy and coming in saying, ‘I went to my cousin’s graduation. It was at Middlebury College.’ And I pull out a little Vermont map, and they find Middlebury on the map.


When the kindergarteners learned about finding animal tracks, they took the lesson home right away. A kindergarten teacher reported that, following the tracking exploration, “…students came back into school on Monday all excited. Even in their backyards they found rabbit tracks.” Because they had explored the topic outside in the schoolyard, she explained, students were able to apply the learning back in their own neighborhoods.

The afterschool program coordinator described how she integrated community engagement into athletic activities. Students who participated in an afterschool running club learned the local geography on foot. In addition to

discovering new places, by considering their geography more closely, students were able to put into context places they had visited before but had been unable to locate prior to participating in the club. The afterschool program coordinator was genuinely surprised at how few places in the community the students were familiar with. She cited a local trailhead, quarry, a town forest, and several cultural sites.

Students’ Sensitivity Grows with Community Engagement An increased sensitivity to the needs of others also seemed to be developing. As one teacher reported, “What I have seen since we started the sustainability [focus] is that kids are really being patient with other kids and really trying to include them in the group…. [They are] caring about each other and caring about how they work together.” She cited examples of the students helping each other with work, and going out of their way to make sure they could get to afterschool activities. It was also noteworthy that in their community improvement project, one group chose to work on improving the playground slides for younger kids and the basketball court for older kids. They clearly were not focused on their own needs. Finally, since the first grade teacher was chosen to participate in an internship sponsored by SSP and Linking Learning to Life, she was able to take first graders on an adventure to the community center for elderly citizens. The young students had the chance to perform for and with the elders, exchange tales, and explore a new facet of their community. This exposure, their teacher reported, sparked a new sensitivity in the first graders. When the teacher


off-handedly used the term “old person,” a student piped right up. “They’re elders,” said the first grader. At Champlain Elementary School, students’ awareness of and engagement with the community can hardly be separated. The hands-on nature of the SSP work was such that students often gained awareness through engagement, which, as one administrator suggests, generates perhaps the most lasting change in consciousness.

The whole mission these past two years has been to really connect children to their communities, to their environments, the sense of advocacy, the sense of being involved, responsibility… feeling like they have a voice, feeling like they’re connected to their community, feeling like people care, really feeling successful.

She also explained how feeling successful came with the ability to accomplish “genuine tasks” and how these endeavors were true learning experiences.

Day in and day out they may forget what went on in that reading group or that math group, but they’re going to remember the project where they went out and improved their neighborhood, or worked on demonstrating the effectiveness of wind power, or building bookcases for [a local shelter]. All of these different projects have connected them to adults, to the missions of different organizations, to their communities and helped them to develop a critical eye.

OTHER STUDENT OUTCOMES Three other areas emerged as outcomes related to student involvement in SSP:

• Linking health and sustainability • SSP impacts student learning process • Students take sustainability messages home

Linking Health and Sustainability SSP’s definition of sustainability includes improving personal and societal quality of life for current and future generations. Given the current public and governmental outcry about such topics as obesity, smoking, and prescription drugs, it is fair to say that health is a major social concern of the time. It follows logically that health indeed falls under the umbrella of sustainability. While not a major schoolwide focus, health topics were integrated into the curriculum, especially in the grades 4/5 Healthy Neighborhoods/Healthy Kids Project. One student working on a park improvement

“The healthy child who feels validated and successful in the community is going to be successful in the school.”

-School Administrator


project made a connection to health suggesting that, “If we fix [the park] up, more people might go outside and that way they might not be so overweight.” An SSP staff member suggested further connections between sustainability and health:

Because they’re focused on health, kids are now walking to school, thinking about health. It’s a way to build momentum toward better understanding of place and sustainability. When kids are outdoors more, walking to school, for example, then they start to notice things about their place—they know when the red wing black bird returns to Vermont.

Teachers, students and SSP staff at Champlain Elementary School have begun to demonstrate that fostering a healthier society may be akin to fostering a more sustainable one.

SSP Impacts Student Learning Process Many examples of student learning outcomes also surfaced during the evaluation process. As often seems to be reported with hands-on, project-based work, one teacher said that his special education students were more involved and more successful. Because of the diverse tasks offered by some of the sustainability-related projects, his students were better able to work to their strengths. If the students did not want to work on writing the report, they could contribute artwork, or poster design, or “go outside and work.”

Figure F2. Evidence of Health-Sustainability Connections in the

Curriculum • 4/5 students are keeping

food/exercise journals • Greater awareness of where

locally grown food is available • No Idling program • Walk to School Day • Healthy Neighborhoods/Healthy

Kids Project


A second grade teacher felt that because she was implementing a well-planned, themed curriculum the students themselves were generating many of the ideas to guide the learning process. She stated that because of her own newfound flexibility within the umbrella of her essential question, students felt an increased sense of ownership, and thus the learning process “literally takes off on its own.” She cited examples of students directing the course of study based on their own curiosity. One example she offered was of students studying water. When they asked, “what would happen if we put salt in this water?” she decided to follow their line of inquiry and initiated a unit of study on salt water. She further described how her students seemed to be making more connections between learnings that were months apart in ways that she “had not seen before.” For instance, she noted that, because the theme of water was threaded throughout the curriculum, students more readily made connections between concepts when they saw the water theme surfacing throughout the school year. One administrator summarized the connection between community involvement and academic success. She said, “When children are involved in projects that are of great interest to them and have some meaning, then that motivates them more to learn and write, and to read.”

Students Take Sustainability Messages Home Apart from the program’s effects on students, many interviewees reported a spread of effect from children to parents. One teacher described how some of her students returned to a local site with their parents after having visited them on a field trip. The Legacy Card project, in which students on the 4/5 team visited and learned about local businesses, also “heightened awareness for parents about the sustainability of local businesses,” reported another teacher. Students brought other SSP inspired lessons home to their families as well, lessons


ranging from animal tracking to composting. For example, a parent from one household that already engaged in composting recounted, “My kids come home and say, ‘We’re not composting enough. We need to do more.’” Improved at-home recycling efforts were also mentioned, as well as family discussions about food. A parent of a child on the grades 4/5 team who studied food systems last year reported that, "…every time we put something on the table now we've got to talk about where it came from."

SSP PROCESS STRENGTHS The analysis of interview data and the review of program documents provided strong evidence of SSP’s many strengths. The most salient of these strengths are highlighted here:

• SSP Reaches a Schoolwide Audience • Composting takes hold • Skilled and effective SSP staff promote change • Providing resources and teaching classes • Second summer workshop embeds changes in practice • Individualization and flexibility of the SSP model • SSP has staying power

SSP Reaches a Schoolwide Audience New ideas cannot thrive in isolation, and the very structure of SSP embodies that principle. There was abundant confirmation of the program’s influence schoolwide. Sustainability themes are being taught at every grade level, in the specials, and in the afterschool program. Teachers report that they are collaborating with one another more, and parents and other community partners have a growing presence in the school. The school building has been the focus of environmental health improvements and energy programs, and recycling and composting programs have been thriving. The bulk of this evaluation report highlights the evidence that SSP has achieved its desired schoolwide status.

Composting Takes Hold Though there are different elements of sustainability evident throughout the school, composting is one arena that SSP staff made an effort to include everyone in uniformly. Composting bins were set up in the lunchroom, some classrooms composted snack waste, and SSP staff offered every classroom a composting lesson. Since composting was a new program introduced by SSP, many interviewees discussed its merits. Teachers were excited about and thankful for the composting lessons brought to their classrooms, and in one case a teacher taught SSP staff a new composting song after composting lessons. Another teacher was enthusiastic about creating compost pledges and displayed in the classroom the students’ pledges and knowledge about compostable items.


During an observation of students in the lunchroom, it was noted that every student who brought up their tray after finishing lunch deposited items in both trash and recycling bins, most of the time accurately. No student simply dumped the tray’s contents into the garbage can. Two specials teachers also remarked on this new behavior:

The biggest impact [of SSP] I've noticed is in the lunchroom. We have the recycling, we have the compost bin, and we have the trash. And there's actually kids assigned to monitor that and make sure people throw the right trash in the right place.

I'll be in there asking ‘which bin do I throw it in?’ And the kids can tell me which bin to throw it in, which I think is really neat.

Composting also proved to be easy for students to engage in at home, especially after a workshop was held that included the parents. Remembering this workshop, one parent said “Our last composting workshop was kids and parents together, which was great. And the kids knew it all. I think it was an education for the parents, but it’s the kids that are bringing this stuff home” Another parent reiterated this sentiment and thought that “having parents and kids learn about these concepts together is great.” An administrator commented that the composting effort improved her collaboration with some parents and the Parent Teacher Organization (PTO). “Trash is bringing us together,” she concluded. On a larger scale, she added that due to Champlain’s success, composting would be operationalized at a district level, and she was committed to seeing it stick within Champlain, because “once you start something like composting you can’t really go back because it's not a good message for children; they think it's phony then.”


In a district-wide compost meeting, the successes of SSP at Champlain did not go unnoticed. A teacher from another elementary school said, “I'm jealous of Champlain because they have sustainability to unify them.”

Skilled and Effective SSP Staff Promote Change It became eminently clear over the course of the interviews that the talents and personal qualities of the SSP staff played a significant role in the success of program implementation. Akin to the way that SSP attempts to instill sustainability as an underlying theme permeating the curriculum, SSP staff seemed to infuse themselves throughout the school, offering their assistance wherever it was appropriate or requested. It is notable that feedback about the staff was uniformly positive. See sidebar for examples of participant comments about the strengths of SSP staff.

Providing Resources and Teaching Classes Because of SSP’s strong input of human resources (four staff members comprise roughly the equivalent of 1.5 full time people,) SSP staff were able to make themselves available in numerous ways, including providing teaching resources and teaching students directly. Both types of involvement were cited by a range of teachers as being very helpful. One teacher stated, “We have this wonderful science curriculum but we're given absolutely nothing to teach it.” She then described the tremendous benefit of having the SSP staff supply background information, written materials, and teaching tools. Particularly with SSP’s strong focus on literacy, teachers found it helpful that the staff brought numerous books to the school for them to use with their classes. Other teachers were very appreciative of the lessons SSP offered students both in the classroom and schoolyard. Several teachers affirmed that it was “…nice to see someone

Figure F3. Sample of Comments about SSP staff• They’re here any time you need them. • They are impeccable in their professionalism,

intelligence, enthusiasm, and creativity. • They’re skilled at facilitation. • They have become an integral part of us. • They have amazing coordination skills and

educational skills. • They bring us expertise in the environment,

healthy food, and learning systems. • They were very flexible. • They're very visible. • They work together really well. • They worked very well with young children. • They’ve just been tremendously helpful. • It's been a gift. We're fortunate.

o Q: There were four staff members this year. Could this have been done with just half of them?

o A: No. Only half as well.


else modeling the lesson,” and “…important for the children to have other adults in the classroom.” Some noted that they valued the staff’s knowledge of the subjects, and one teacher mentioned learning quite a bit herself about a particular subject. During one classroom observation by the evaluator, it was clear that students were highly engaged in the hands-on watershed pollution lesson that an SSP staff member was providing, and that the classroom teacher was fully present to the lesson, offering additional comments about the content area for the children, passing out materials, and assisting with behavior management. All of these roles are elements of team-teaching which SSP strives for when providing lessons directly to students.

Second Summer Workshop Embeds Changes in Practice Though only two teachers explicitly discussed it, it is worth mentioning that the second summer workshop was described as extremely useful to those teachers who had been lukewarm about the program during the first summer institute. One teacher summarized it this way:

I really liked the summer workshop that we went to this year. That was necessary, and I left there feeling excited and much more comfortable about things…. It was a nice team building and school community building activity. That was time well spent, an opportunity to kind of pull things together.

For those teachers who are slower to adopt a new innovation, the idea of offering a summer gathering after one year of warming up to the idea appeared to be highly beneficial.

Individualization and Flexibility of the SSP Model Many people talked about the significance of the breadth of the sustainability umbrella. Rather than being tight and limiting, sustainability was seen as an easy principle under which to organize many topics. Teachers talked about how this focus was more far-reaching than other themes they have encountered, allowing them to individualize their curriculum. As a second grade teacher noted “…everything becomes that much more


relevant when it's interrelated. I think we've always looked for that but this is a perfect vehicle for doing it.” With the help of skilled SSP staff, many teachers were able to see that their individual teaching styles as well as many of the topics and subject areas they were accustomed to teaching in the past could be worked into the sustainability umbrella. They appreciated this flexibility. A second grade teacher offered the following:

The feeling I got [from SSP staff] was, ‘Well, you're kind of doing this anyway. Now you're just doing it with a little more awareness.’ And this is what helped us tremendously, and that's why we've embraced it. We don't see it as an added burden. We see it as ‘Hey, we're doing in anyway, but we can do it more effectively.’ For example, the one big change we've made is the overriding year-long theme. And it really hasn't changed what we've done a whole lot, but it has changed the outcome. [SSP staff] have been really, really helpful, tremendously helpful, with promoting that feeling that this doesn't have to be a separate thing; it should be woven throughout your curriculum. They've been very helpful in finding ways to do that.

An administrator effectively summarized the flexibility that is inherent in SSP:

One of the strengths of this project is probably one of the strengths of a good teacher in a classroom, that there is a lot of individualization. Adult learners are maybe not as varied as student learners, but [SSP staff] really tap into the learning style, the teaching style, and the personal interests of the teachers. Instead of being force fit, one-size-fits-all, they have the personnel to engage in dialogue with the clusters of teachers, even the individual teachers. That way they can say, ‘What are your hopes and dreams for your class? If you could envision something, what would that be?’ and then help them get there.

Figure F4. Expressions of Outside Interest in SSP

• Cross-fertilization with other PEEC programs: CO-SEED, FFEC

• Community Works Journal is publishing a series of articles on the program, and program coordinator is a contributing editor for the Journal

• Teachers in Pennsylvania • Project Learning Tree in NH • Compost group in county • AmeriCorps*Vista applicants • UN Decade acknowledgement • Growing Farms, Growing Minds

project is a spin-off of SSP, potentially bringing a similar model to the entire Burlington district.


SSP has Staying Power It could be argued that the most important strength of a program designed to create change is its ability to sustain itself. Three key areas of evidence indicating the potential for SSP to create sustained change at Champlain –- a wide network of support, continuation of established relationships, and documenting SSP work – are also considerable strengths of the program. Each of these aspects of SSP’s apparent staying power at Champlain is detailed in the section below.

EVIDENCE OF SSP’S STAYING POWER Three key areas emerged as clear evidence of the potential for SSP to continue beyond the first two program implementation years. These are discussed below:

• A wide network of support o Administration o Parents o Community o Teachers

• Continuation of established relationships • Documentation of SSP work

A Wide Network of Support The attempt to implement change often falls on a few committed individuals with boundless enthusiasm. In the case of SSP quite the opposite is true as staff enjoyed wide support and involvement from a network of parents, administrators, the community, and teachers in particular.

A Supportive Administration One administrator was supportive of SSP for a number of reasons, a primary one being that sustainability is part of the Vermont standards and thus will always need to be

taught to some degree. She also felt that sustainability was an “…easy umbrella to look for funding under,” and potentially it could encompass many different topics. She seemed to believe that having a broad unifying theme throughout the curriculum was much better that having it “…look like we’re jumping from thing to thing.” Also of note was that sustainability had been

a topic of several district-wide meetings, and that the district’s superintendent has been “keenly interested” in the SSP. One topic in particular that began at Champlain but generated interest and action beyond the one school was a composting endeavor.

“The improvements to the grounds here and the gardens here-- certainly planting and growing and looking at cycles of life and habitat--those will continue too.”

-School Administrator, commenting on aspects of SSP that will have

staying power at Champlain


Because of the example set at Champlain, the new bids that have been put out for trash removal for the district include the separation of compost. On the topic of her commitment to promoting sustainable practices such as composting, a Champlain administrator noted that sustainability is not just something that can be “taught” to children; it must be modeled by adults as well.

Supportive Parents After an initial period of learning what the program is about, a PTO member affirmed that the PTO readily supports the school’s sustainability umbrella. With SSP, the PTO this year hosted a series of three evening presentations on sustainability. A PTO member reported that these events, which occurred during the normal PTO meeting time, were more heavily attended than the usual meetings. Furthermore, an administrator reported that “there have been a number of parents participating in the projects” within the school. She also mentioned that some school building environmental quality improvements had taken place that were tied into SSP, and she was pleased to note that “…our parents’ understanding is that this is a school that cares about health in many different ways.” A grades 4/5 teacher reported that there had been “an incredible amount of involvement” from parents.

A Supportive Community Diverse examples of support for SSP from the community were also evident. SSP staff described how the city of Burlington has been using the language of sustainability for the past five years in diverse arenas. In December of 2003 the City Council and the mayor’s office passed a resolution to support education for sustainability and sustainable schools. It seems clear that the city government is an extension of the people, as Burlington is widely regarded as a progressive community. Various interviewees suggested that the people of Burlington would lend their support to a program like SSP. Likewise, through involvement with SSP during the two years of the program, the business community in Burlington has showed itself to be equally receptive and supportive. (See section below “Continuation of established relationships.”) SSP also seemed to be making some waves in the educational community, since other schools in the area are beginning to express interest in becoming “Sustainable Schools.”

Supportive Teachers Undoubtedly, one of the most significant assurances that a program will continue to be implemented in a school or district is teacher buy-in. Buy-in is indicated by a number of characteristics, including enthusiasm, commitment, and personal belief in the value of the program. Numerous examples of these indicators were reported across the grade levels.


At the start of the project’s first year, several teachers were not immediately enthusiastic about SSP and expressed concerns that it would be just one more thing to squeeze onto an already full plate. Over time, however, they began to embrace the project. Several comments illustrate this clearly:

• “When we first started, I saw it as ‘oh boy, something else to do.’ Over the last couple of years I now see how it fits and how it is a vehicle for weaving things together. Everything can fit under that umbrella, and I don't think we can even do all of this anymore without making it connect.” (Second grade teacher)

• “After the first year, I was very skeptical. I thought, ‘OK, we'll see what's going to

happen.’ And, once again, it has evolved. Their talent as a sustainability staff-- to be able to see what could be done with the people in this building and this community -- that was key…. I think it has just taken roots of its own…” (Specials teacher)

Now that the program has been in place for two years, the program has become entrenched in the school vocabulary and in the way school personnel do business. This is illustrated by one specials teacher’s comment:

I don't necessarily know that I can remember a time when that word wasn't used every day. I mean, that first year we didn't want to hear that word ever again: sustainability. ‘[We would say to the SSP staff member], we're going to kill you if you say that word one more time!’ But now I have a hard time remembering what it was like before.

For some, the reasons for continuing with the sustainability framework are based on personal conviction. A specials teacher, for instance, recognized that for the initiative to take hold school wide, she needed to do her part:

I incorporated [sustainability] because I feel like this is something that is definitely important, and I think it needs to be school wide. I want to work with the classroom teachers and back up what they’re doing.

And for a grades 4/5 teacher, his values dictated his commitment to being part of the sustainability effort:

What difference does it make if we're teaching our regular curriculum if we're destroying the planet or if there's a tremendous inequity in the world…. I think if we don't teach about sustainability, what’s the point? What's the point of everything if we're not going to be a sustainable, civil society?


Others knew that it would be harder to maintain the focus without having the SSP staff readily available, but sounded as though they were going to stay the course. When the question of continuing to teach about sustainability was raised, one conversation transpired like this:

Teacher A: “We’ll definitely continue it, it's just not going to be as easy without their help. They were great, if we needed something they were there with it. It’s been great work; I'd hate to see it discontinued.”

Teacher B: “And we know where they'll be.”

Teacher A: “They can't get rid of us that easy.”

Finally, one of the most significant indicators of staying power was described in detail in a previous section of this report. The fact that teachers within the Champlain Elementary School have made substantive changes in their teaching practice is a solid indicator that change within the school has great potential to be lasting.

Continuation of Established Relationships Over the course of two years of program implementation, many relationships have been forged between students, teachers, and various community partners. Throughout the evaluation data, there is evidence that different teachers and partners were looking forward to continuing those relationships. The afterschool program coordinator hoped to encourage the return of two new community partners who led programs in the past year. She felt that since she had established the relationships this year, in future years setting up the connections will be easier. She explained “I’ll just have to say, ‘you came here last year; can you come back now?’” Similarly, the community partner who worked on the Healthy Neighborhoods/ Healthy Kids Project was very enthusiastic about her involvement with SSP and the students at Champlain Elementary. She expressed confidence that the Healthy Neighborhoods/Healthy Kids Project will be recreated at Barnes School next year, and remarked that this spread of effect would only serve to deepen the effects at Champlain as well:

If that is the case, sustainability will be integrated further into the Champlain School curriculum since the Champlain teachers and students would like to do a teacher

“I never thought it would be this successful. I wish I could do this all the time.” -Community Partner, referring to her SSP work

with students and teachers at Champlain Elementary School


and student exchange with Barnes, helping the Barnes kids and teachers implement a similar program in the north end of Burlington.

The Legacy Card project, which began with the 4/5 team in year one of SSP and continued into the second year, involved a number of local businesses, who were reported to have expressed interest in continued participation. “This is a great project,” affirmed one business owner. The presence of SSP at Champlain school opened the door for other sustainability-related initiatives. For instance, the electric department installed a small, solar-powered electric system on the school building and has made plans to work with teachers to integrate solar concepts into the curriculum. In the meantime, SSP staff helped some teachers integrate the solar concept into their curriculum, using essential questions and other means of infusing the solar project into their underlying sustainability theme. The electric department was also involved in the “No Idling” campaign taken on by the grades 4/5 team in an effort to reduce pollution in front of the school building. Students created informational signs asking school visitors not to idle their cars near the building, and ultimately the school was awarded a clean air award in Spring 2004. Although the presence of SSP staff on site at the school helped to foment many of these relationships, all interviewees were optimistic that the relationships will live well beyond the direct involvement of SSP staff.

Documenting SSP Work In addition to documenting curriculum, two additional sets of documentation were undertaken this year in order to improve the chances that SSP projects would have staying power in year three and beyond. One teacher created a manual for the living machine, and an SSP staff member created a manual for the afterschool program. Most interviewees cited the living machine as an important resource that created great excitement among the majority of students. One teacher said, “I think kids have really


taken that on as their thing, and what they like to show their parents when they bring them to school.” Many grade levels reported using the living machine in their studies. Although most interviewees felt that the living machine would be a lasting feature of the school, there was some concern that it was overly complicated and labor intensive. In order to facilitate the ongoing use and maintenance of the living machine, a third grade teacher collaborated with SSP staff and a community partner to develop a comprehensive management plan. The hope was that such a document might be of use to other schools interested in building one. The afterschool program, as described above, proved to be a successful venue for teaching about sustainability. Because the SSP staff was fortunate to have an AmeriCorps Vista member for year two of the program, there was staff time available to develop an afterschool program manual for Champlain. The manual will also be available for use as a resource for the Barnes School afterschool program.

SSP’S PROCESS CHALLENGES Unequivocal praise for the SSP experience at Champlain Elementary School and clear evidence of positive outcomes strongly indicate that the model is a success. Nonetheless, as can be expected with any program, a pattern of challenges emerged in the data and are reported here. The challenges are broken down into the following four areas:

• Time and scheduling can be barriers • Language of sustainability can be a deterrent • Some subjects are seen as outliers to sustainability work • Other concerns

Time and Scheduling Can Be Barriers As is so often the case in education, time is a limiting factor. Almost inevitably there seem to be more good ideas than there is time to discuss, plan, or implement them. A lack of time is at the root of many of the barriers cited by both SSP staff and classroom teachers. One grades 4/5 teacher described the limited time challenge as


“…an ongoing process. We struggle every year with something new that comes up and something old we must leave out, and we have to figure out how to get it all together.” A kindergarten teacher echoed this sentiment, adding that “especially when things are new it's hard to take chances.” Despite SSP staff working hard to help teachers understand that sustainability did not need to be treated as an additional topic requiring more time, there were still obstacles to overcome. These challenges are discussed in greater detail in the infusion section below. One third grade teacher enthusiastically assumed many roles, but eventually became somewhat overwhelmed. As a result, some program documentation on which she was working was delayed and the absence of this data could lead to some difficulties in making the most effective use of her work. Other teachers who found themselves involved in several of the program elements, such as a study group or internship, found that one activity robbed time from other important aspects of their practice, such as devoting time to curriculum development with a co-teacher. An SSP staff member wished she had more opportunity for classroom teaching, but planning and preparation time kept her too busy. Communication issues arose as barriers to progress, and often lack of time and scheduling conflicts were at their source. SSP staff intended to do more modeling of SSP/community collaboration but found difficulty in coordinating their schedules to implement that plan as much as they would have liked. Similar reasons limited the time the staff felt they needed for essential internal communications. The afterschool program coordinator was frustrated that she was never able to attend afterschool staff meetings with SSP staff and teachers. She felt that she would have “gotten more involved in [the program] and therefore have a better understanding” and been able to collaborate much more with teachers had she been able to participate in meetings. This was a concern which was expressed in year one of the SSP evaluation and is an inherent barrier to developing greater collaboration between teachers with vastly different schedules. A final barrier related to time constraints was that periodic school vacations in February, March, and April made it difficult for SSP staff to coordinate with teachers on long-term projects or planning.


Language of Sustainability Can Be a Deterrent Both SSP staff and an administrator raised some concerns regarding misconceptions about the word “sustainability.” As a term that recently has come into somewhat heavy use, sustainability has not found a clear working definition among the general public. Some consider it to be a buzzword that inherently denotes a pro-environment and anti-business position and as such can be perceived as controversial. SSP staff observed that at the beginning of the program a number of teachers had a knee-jerk reaction when they heard the word “sustainability.” A parent is reported to have said, “I don’t like this word.” The staff explained that the parent’s objection was simply to the word, that she did in fact support the goals of the project. Resistance to change is normal, but when the language associated with the desired change is seen as being “loaded,” it can generate conflict among certain groups. Under those circumstances, laying the groundwork for careful communication is essential. In Burlington it was assumed that the word had been used enough by city agencies and talked about enough in a wide variety of circles that the population was accustomed to it. Additional concerns were raised, however, about whether the name SSP might be a barrier for entry into some future communities with which SSP might choose to work, since more rural areas may not be as likely to embrace new political buzzwords or progressive agendas. One administrator, while wholeheartedly endorsing the program, expressed some ambivalence about the language. “I never know quite how to use the word ‘sustainability’.... Something is sustainable but then that word “sustainability” sounds a little odd.” She then pointed out that it was a good idea to keep the term in operation, nonetheless, because it is the same language referenced in the standards, and it is a useful and broad idea under


which many themes can be organized. Similar to SSP staff’s experience, she found that it just needs to be properly defined. One final note of concern about the language was raised by the afterschool program coordinator, who said that she had some difficulty in getting students to sign up for some of the sustainability related programs. She interpreted this resistance to the students’ immediate association of sustainability with academics and their desire, by the end of the day, simply to do something different. After they participated in the program, however, they enjoyed themselves. She felt confident that once these new afterschool programs have established themselves they would be greeted warmly by students.

Some Subjects Seen as Outliers to Sustainability Work The intention of SSP is to introduce sustainability as a thread that can be woven through the curriculum. However, some teachers still expressed the belief that sustainability was yet another topic to add on, or that sustainability themes began to dominate certain subjects at the expense of some of the other standards. The afterschool program coordinator did not feel that sustainability had “been totally successfully integrated yet.” She felt that some topics were easily brought under the sustainability umbrella, but that others were more challenging to bring in. In the latter case, she thought that teachers tried to incorporate sustainability in response to “a pressure to do it because it's part of the school,” but that this motivation made it harder for them to teach some of the requirements of a given subject. A grades 4/5 teacher talked about similar difficulties in his class. Using an example of specific content that students are required to learn about the American Revolution, he described his teaching challenge:

I'm having a hard enough time just figuring out how I'm going to teach the social studies in a way that's meaningful -- that's not just me standing up here lecturing, that's going to have an interesting essential question, and that's going to have an interesting culminating task. To tie it all into the umbrella of sustainability on top of all that feels really hard.

It should be noted that he is a teacher who is particularly committed to the ideas and practices of SSP, and one who evidently gives a great deal of thought to his teaching. SSP staff suggested that in order to most effectively plan curriculum through the sustainability lens, a certain mindset needs to be in place. They believe that it is possible to cultivate this mindset within a school, but that a solid week long institute is needed at the outset. In the case of Champlain, the latter had been financially prohibitive, and the institute was conducted within a two day time period.


Other Concerns The following challenges were not mentioned as frequently as those above, but because they were referenced by participants, they are presented for future consideration.

Barriers SSP staff encounter • As in year one, SSP staff expressed some frustration that, at times, classroom

teachers focus on other activities while SSP staff are teaching their students. Since the intention is team-teaching and modeling particular lessons and teaching techniques, if teachers are not fully attentive, the value of this activity is diminished. Conversely, many teachers noted that they are planning to utilize the techniques and lessons they have seen SSP staff modeling once SSP staff are not available as a resource.

• One staff member did not feel sufficiently prepared to offer coaching to teachers

on literacy, despite this being a program focus. She did not have experience teaching particular forms of writing such as personal essays or business letters.

• The staff felt that greater attention to teacher reflection would be a useful

component of SSP, but there was little opportunity for it.

• Several staff members questioned whether the diversity and equity aspects of sustainability were being addressed as fully as they should be in the curriculum. Of the three “E’s,” it seemed that a heavier emphasis was placed on Environment and Economy than on Equity.

Inclusion • One specials teacher pointed out that she would like to have been invited to the

summer workshop. That not having been the case, she did not feel included in the start-up of the whole school initiative and was subsequently given the message by other teachers that she was expected to participate in the school wide initiative.

Questioning Staying Power • One teacher pointed out that some of the “glue” for the program came from

having staff meetings which frequently focused on sustainability issues. She expressed concern that without the SSP staff present in the school, this key means of communicating about sustainability would slip away. “I think if it's not discussed at a regular meeting there's going to be less emphasis placed on it, and so I think it's going to have a tendency to just slip through the cracks, because it won't be brought back to the center every couple of weeks,” she said


• Two interviewees indicated that the living machine might be the first program element to “go by the wayside” since it required “a tremendous amount of work” and had “gotten a little frenetic at times.” They appreciated the educational value of the living machine, but questioned whether starting off with such a long-term, labor-intensive project was the most effective way to introduce the program. It should be noted that, by contrast, many interviewees referenced the strong appeal of the living machine, both for academic uses and for its community building features, such as being a focal gathering place for school visitors, such as parents and siblings.


INTEGRATING LITERACY AND SUSTAINABILITY This section explores data collected from four different assessments of various aspects of SSP’s effort to integrate literacy and sustainability, and then synthesizes that data with relevant evaluation interview data. The final part of this section explores ways in which the collective body of data can begin to answer the two key literacy questions from the 2003-04 SSP evaluation plan.

Project List The side by side comparison between school years shown in Figure 5 reveals a marked increase in the amount of effort put into literacy at Champlain as a result of SSP. Not only did the amount of activity increase from 2002-03 to 2003-04 (especially in terms of more work with specific grade levels), but the type of activities seems to have taken on a new dimension as well. For instance, four of the twelve activities listed for the 2003-04 school year are evidence of SSP interacting with the Champlain staff as a whole group, and five of the activities reference direct involvement from the district learning specialist. These seem to be indications of SSP as a program becoming more deeply integrated into the fabric of the school system. In all the data collected for this evaluation, there was nothing to suggest resistance to SSP operating at this higher level of prominence (or deeper level of integration) in the school system. In a discussion with evaluators, SSP staff noted a potential weakness in this project list. Simply listing activities does not reflect the depth to which the integration is really happening, and so, taken by itself, the list may well under represent the extent to which sustainability and literacy efforts are truly being integrated at the day to day level. Building on this idea, the SSP staff person wrote:

It appears difficult to ask teachers to generate self-reflective data about their implementation of teaching strategies. Teachers have effectively identified


needs for the future and new topics of interest, but once a topic has been introduced, it tends to be named as “implemented” regardless of the level or extent of implementation.

Clearly, sustainability concepts are being successfully integrated with literacy (and also with other aspects of the curriculum at Champlain). This leaves SSP with the fortunate problem (or opportunity!) of now needing to articulate just how much integration needs to happen in order to call their efforts a success, and how that might be accurately measured.

Figure F5. Activities that are examples of integrating literacy and sustainability at Champlain Elementary School

2002-03 School Year 2003-04 School Year 1. SSP staff were

invited to district’s literacy institute.

2. SSP staff led and/or suggested reading/ writing activities when co-planning grades’ science/social studies units.

3. Grades 4/5 half-year service-learning project included report – designed by SSP staff.

4. Invited SSP staff to suggest topics on which to order informational texts.

1. District literacy specialist and SSP presented informational text literacy lessons at staff meeting.

2. District literacy specialist co-taught informational text lessons with first and third grades.

3. District literacy specialist and SSP wrote draft informational text literacy lessons and guide to be implemented in 2004-05.

4. Five teachers participated in study group led by SSP staff on integrating literacy.

5. The staff as a whole focused on teaching with informational texts and on creating leveled libraries for their classrooms using district, grant, and SSP funds.

6. The staff as a whole reflected a greater cohesion and coordination of literacy and sustainability instruction and higher levels of student “talk” and performance.

7. SSP led an evening workshop for families on literacy, informational texts and journaling. One teacher participated. The district literacy specialist presented an overview of the literacy program – the first time ever presented to families, and reported by SSP staff, school administrator and literacy specialist, not teachers.

8. First and third grade teachers wrote science and service-learning units that purposefully integrated info text reading and scaffolded writing into “sustainability.”

9. First grade unit incorporated scaffolded comprehension and writing into year-long science/social studies “sustainability” unit.

10. Third grade integrated scaffolded report writing into main science unit – consulting with SSP staff and district literacy specialist, inviting them to co-teach one class.

11. SSP was invited to participate in grades 4/5 year-long service-learning unit designed by teachers – including persuasive letter and report, and teacher meetings on selecting informational text book sets for primary grades.

12. SSP was invited to co-teach and incorporate literacy at district elementary science camp, led by Champlain & Barnes teachers.


Champlain Literacy Assessment Questionnaire One way that SSP’s strategic focus on the use of informational (non-fiction) texts showed up was through a simple, one-page, pre-post questionnaire given to all teachers at Champlain elementary school. The questionnaire (see Appendix K) was designed by SSP staff and administered as a “pre-“ to 13 teachers in October 2003, and as a “post-“ to 8 teachers in June 2004. Since the design of the questionnaire was fairly informal and does not lend itself to systematic quantitative analysis, one SSP staff member provided written documentation of her informal analysis. These findings are presented in Figure F6 below. The most prominent feature of the findings is a seeming increase for every one of the (loosely) scaled items. This provides one tangible level of affirmation that SSP’s efforts to integrate sustainability and literacy are having positive results. Further, it appears that teachers may be slightly better able to integrate literacy into sustainability than the other way around (i.e. than integrating sustainability into literacy efforts). This phenomenon may be worth testing through more in depth investigation in subsequent evaluations. If it turns out to be a systematic trend, then that could influence the strategies and priorities of future efforts to integrate sustainability and literacy.

Figure F6. SSP staff analysis of Champlain Literacy Assessment

Items from one-page “Champlain Literacy Assessment” questionnaire

Informal analysis reported by SSP staff after comparing written responses from

pre- and post-assessments 1. How often have you used non-fiction

in literacy lessons with your students? (mark a point on the line)

“General increase from rarely to several times/month (average).”

2. How much have you used non-fiction compared to fiction in your literacy lessons? (mark a point on the line)

“Small average increase from ‘much less’ to between ‘much less’ and ‘as much.’”

3. How much have you integrated literacy into your sustainability units? (mark a point on the line)

“Large general increase – from ‘once in a while’ in pre to all reporting ‘most of the time/significantly’ [in post].”


Items from one-page “Champlain Literacy Assessment” questionnaire

Informal analysis reported by SSP staff after comparing written responses from

pre- and post-assessments 4. How much have you integrated

sustainability topics into your literacy work? (mark a point on the line)

“Moderate general increase. Five of eight report increase from ‘once in a while’ to ‘most of the time.’”

5. What are the benefits you see of using non-fiction (informational texts)? (open response)

“A range of reasons clearly expressed in posts that capture the range of reasons supported by literature.”

6. What are your barriers to using more non-fiction (informational texts)? (open response)

“Only barrier significantly reported is availability of high-quality books which is ‘being addressed.’”

Literacy/Sustainability Study Group Reflections The literacy/sustainability study group was an intentional component of the SSP at Champlain School. Five teachers participated in this study group that met 11 times for a total of 30 hours from September through December 2003. The purpose of the group was to build skills in integrating literacy into classroom work and to develop units to support such integration. Teachers selected science, social studies and service-learning as the content areas on which to focus their sustainability integration efforts. Perhaps the strongest piece of evaluation data resulting from this study group is simply documenting that it did, indeed, exist! It is a substantial accomplishment to bring together more than a third of the school’s teaching staff to meet regularly to develop their skills in a specific area such as integrating sustainability and literacy. Perhaps this is evidence of what the school administrator was talking about in her evaluation interview when she said:

I think that the merging of our literacy efforts and our Sustainable Schools efforts is really going to be a sustainable piece…the commitment that we have…it’s not necessarily policy but it’s kind of curriculum philosophy.

Interpreting the existence of the literacy/sustainability study group in light of the literature on diffusion of innovations7 suggests that integrating literacy and sustainability may well have already reached the “early majority.” If this is the case, the next step is to bring the “late majority” on board by working out many of the bumps and kinks so their adoption of the innovation can proceed more smoothly. The Sustainability and Informational Text Project Guide (see Appendix N) created by the district literacy specialist and SSP this year could be exactly what is needed to really make these new practices stick. 7 See page 17 of this report for a discussion of diffusion of innovations theory and research.


In order to capture more detailed evaluation data from this study group, SSP staff designed a brief assessment (seven open-ended questions, see Appendix L), and had them write out their responses at the beginning of the school year and then discuss the questions as a group at the end of the school year. Several of the quotes from the “post” assessment discussion expressed a sense of focus that emerged over the year. One teacher said: “The units we did … focus[ed] on specific skills we need to teach, that are on the report card but that we haven’t paid attention to before.” Another commented: “It helped make our planning and literature and writing choices more purposeful.” Yet another noted: “This was kind of what we already did but it made our focus tighter.” This echoes the clear intention and focus of the SSP staff and district literacy specialist in their various presentations of the importance and utility of non-fiction, informational texts. A final comment from this year-end study group discussion seems to express the poignant tension between the additional demands and commitment required by SSP-type efforts and the benefits that can accrue as a result. In reference to the task of writing up new units that integrate sustainability and literacy, the teacher said:

Any time you have an agenda to follow, you have to fit [new topics/projects] in, and that’s hard. But it goes with what we have to teach and making the connections was really easy.

Quasi-experiment SSP piloted a direct assessment of student literacy skills during the 2003-04 school year. The student scores are presented in Figure F7 below. Clearly, the most striking element of the data is that all students demonstrated improvement from pre- to post-, except those who had achieved at or near the highest possible score on the pre-assessment. Assuming the integrity of the scoring process, it can be confidently asserted that the integrated sustainability/literacy lessons that took place between the pre- and post-assessments led to improved student skills in the first two out of the three questions/skill areas covered by the assessment rubric (see Appendix M). Question 1 was about sustainability concepts, and question 2 was about reading and thinking strategies. That assertion cannot be made about question 3 (key informational text feature) since all but one student scored at the highest level for this skill area on the pre-assessment. Question 3 should be made more difficult in future iterations of the scoring rubrics. In addition, the lack of a suitable control group that did not receive instruction with the sustainability-oriented lessons precludes inferences about the success of this approach compared to other, perhaps more traditional, ways of teaching. This is not a deficiency of any of the participants, but is simply a consequence of the quasi-experimental design.


Figure F7. Scores from pilot quasi-experimental assessment of student literacy Rubric Scores

# Student Grade/ Classroom

Student Reading Level

(High, Medium, Low)

Pre Q1, Q2, Q3

Post Q1, Q2, Q3

Change in cumulative

score

1 First/Teacher A Low 1, 1, 3 2, 3, 3 +3 2 First/Teacher A Medium 2, 1, 1 3, 3, 3 +5 3 First/Teacher A High 3, 3, 3 3, 3, 3 0 (9) 4 First/Teacher A High 2, 1, 3 3, 3, 3 +3 5 First/Teacher A Medium 2+, 3, 3 3, 3-, 3 +1 First/Teacher A High 3, 3, 3 3, 3, 3 0 (9) 7 First/Teacher A Low 2, 1, 3 3, 3, 3 +3 8 First/Teacher B Low 2, 1+, 3 2, 2-, 3 +1 9 First/Teacher B High 2+, 3-, 3 3, 3, 3 +1 10 First/Teacher B Low 2, 1, 3 3, 3, 3 +3 11 First/Teacher B Medium 3, 2, 3- 3, 2, 3 0 (8) 12 First/Teacher B Medium 3, 1, 3 3, 3, 3 +2 13 First/Teacher B High 2, 3, 3 3, 3, 3 +1 14 Third/Teacher Not available 3, 3, 3 3, 3, 3 0 (9) 15 Third/Teacher Not available 1, 1, 3 3, 1, 2 +1 16 Third/Teacher Not available 3, 2, 3 3, 3, 3 +1 17 Third/Teacher Not available 1, 3, 3 3, 3, 3 +2

It is important to remember that this quasi-experimental investigation was a pilot. Thus, it is no surprise that the richest learning comes from reflecting on the design and implementation process. The SSP staff member had much to say when asked by the evaluators to reflect intently on what we collectively learned while piloting this study. Her responses are presented here verbatim (numbered by the evaluators for clearer reference.)

1) In order to provide a test of inter-rater reliability, it was suggested that a second rater observe some of the sessions and make their own rating of student skills using the same rubric. However, there was not sufficient time or staff to do this. Instead, the two raters both rated the first student and discussed their observations to “calibrate” [our] observations.

2) One third grade teacher’s unanticipated withdrawal from the study created the potential for a control group [and thus a true experimental design], but it happened too late to be pursued this year.

3) The assessment administration, particularly of the third grade, was hampered by frequent school schedule changes, [my] need for better organization systems for collecting the resulting data, and other logistical challenges.


4) The effectiveness of the assessment task in revealing skill levels was limited by the challenges of selecting a text that was complex enough to include all the criteria but simple enough to be accessible to all students and readable in a short period of time.

5) While the results of the assessment task are limited this year, the process of developing the task and the actual exercise and rubric have already been shown to be of high interest for professional development on assessment and of high value for future curriculum development. Few teachers in Burlington schools have time or experience in developing performance- or standards-based assessments. Indeed, the literacy program is just beginning to focus on assessment. These rubrics and tasks will be used as examples in future work by the district literacy program and by SSP. The evaluation and SSP team should develop data collection methods to make use of teacher- and specialist-administered rubrics in future evaluations.

6) The SSP staff members have repeatedly found it very challenging to administer multiple surveys and especially post-assessments to teachers at the end of the school year. Burn-out, exhaustion and time shortages all impede such data collection. Staff discussion was very useful in 2003 but less revealing in 2004, perhaps because of fatigue with the subject matter of literacy as the meeting followed a district-wide literacy institute and preceded vacation. The evaluation and SSP teams should carefully plan the administration of “posts” for the future to gather more robust results.

It is quite common in successful program evaluations that “the evaluation becomes part of the programmatic intervention to the extent that the way it is conducted supports and reinforces accomplishing desired program goals” (Patton, 1997). Another aspect of this same idea is the common saying “what gets measured gets done.” Reflection number 5) above is a clear and strong example of these two sentiments. In fact, it is quite likely that the benefits of developing the assessment rubrics and having them accessible for future classroom use substantially outweigh the benefits of having the numeric results of the quasi-experimental investigation that the rubrics were developed for. At the same it is important to realize that designing, implementing, and analyzing this quasi-experiment took significant staff and evaluator time. Before extending this pilot into a more regular part of the SSP or school action plan, it would be advisable to consider the following points:

• Resources. Are there sufficient time and money resources to invest in this effort? Be sure to plan for enough resources to allow teaching staff and/or SSP program staff enough time to really dive in to the rubric creation process. That is where some of biggest benefits lie.


• Scheduling. Avoid conducting such an exercise in May or June if at all possible. Conducting the assessment in late fall or winter is likely to reduce the complications arising from the end of school rush.

• Stakeholders. Are high priority stakeholders willing to accept the value of integration of sustainability into the core of a school’s curriculum at face value? If so, then it may be sufficient to implement this assessment with relatively few modifications and simply celebrate the benefits accruing from the process of thinking about and creating the assessment rubrics. If those stakeholders require a higher level of statistical validity, then it is crucial to implement the following research design refinements.

• Research Design, point A. Consider making the extra investment to turn this into a randomized control trial (RCT), also called a “true” experiment. The essential criteria here is to have a clear distinction between “control” group(s) and “treatment” group(s), and for students to have an equal chance of being in either a control or the treatment group. Even a small sample of student participants (e.g. 15-20) could possibly generate statistically significant or important results if the investigation is designed and implemented well. The logistics and politics of this could be tricky, though.

One good solution would be to have a group of teachers participate in the study, all of whom are relatively equally committed to sustainability. Choose two topics and develop assessment rubrics. For each topic, have some teachers deliver a sustainability-oriented lesson (i.e. that uses the sustainability integration plans and materials developed through SSP), and some teachers deliver a traditional lesson (i.e. that does NOT use the sustainability integration plans and materials). Assess each lesson with the same rubric. Have each teacher deliver at least one lesson of each type (sustainability AND traditional). This design keeps any one teacher from being “stuck” with the potential benefit or loss of being exclusively in the control group or exclusively in the treatment group.

Another solution would be to find some teachers who are less interested in using the sustainability-oriented plans and materials but are willing to have their students assessed with the same rubric. Use students working with these teachers as the control group. This could be problematic in terms of selection bias if students do not have an equal (i.e. random) chance of being in either a control or treatment group, but it might still be worth the effort if the other solution above is not feasible.

• Research Design, point B. Create a more systematic check of inter-rater reliability. Also, a simple step in this regard that should be done in any case is to fold the rubric score sheets in half after the pre-assessment or put the pre- and post-assessments on the front and back of the same page. This will help keep the post- rater from being influenced by seeing the scores from the pre-assessment.


One final comment on this pilot quasi-experimental investigation is that it could well become a replicable model for similar investigations of other PEEC or place-based programs if some of the research design and administration procedures were to be slightly more refined and systematized.

Overall Synthesis of Literacy Integration This section looks across data from the general evaluation interviews and the four literacy-specific assessments to try to answer the two literacy oriented evaluation questions that were articulated in the fall of 2003. One existing goal for teachers is simply to help their students learn to read and write well. The quasi-experimental assessment showed clear evidence of at least one way that students are, indeed, gaining skills in literacy. There were also a number of comments from interviews that directly address this issue. Below are quotes from a teacher, an SSP staff person, and a community partner.

The quality of writing, for my kids, I’ve seen improvements in the persuasive letters. Granted, the mechanics sometimes were still weak, but sometimes just the quality of ideas, they were able to just express, it seemed like their voice was more in the piece. The persuasive was just an easy tie-in to the sustainability unit. (grades 4/5 teacher) …parents were talking to me about how their daughter took her journal home and was writing in it every day. And her daughter actually brought it, then, to an after school program that we were doing on Stories of the Land and showed me what she had written in her journal. (SSP staff)

Literacy and sustainability worked together when we integrated the students’ unit on business letter writing with writing letters to the Director of the Department of Public Service…about the improvements they would like to see in their communities. (community partner)

It is important to not give the impression that everyone thought that sustainability and literacy are or should be completely integrated with each other all the time. There were a handful of comments that echoed the sentiments of a third grade teacher who said: “I can see part of it being integrated, not all of it. Not all of your reading and writing program is going to be connected to sustainability but I can see parts of it where it would fit nicely, where it wouldn't be forced.”

Evaluation Question: How is the presence of SSP helping teachers meet existing literacy goals

and requirements?


Another measure of teacher “goals and requirements” is the level of alignment with the school administration. Evidence of the deep involvement and commitment of the district reading specialist showed up in nearly every type of data collected for this report. There was a sense of rapport and respect associated with this as well. One SSP staff person said:

I think [the district specialist] knows a lot more about the project now … [but] I want to be careful because they've done so much that's been so great before, without [SSP]. [The district specialist] has been a tremendous asset to us, first and foremost….

Perhaps the administrative support is most concisely summed up by the following school administrator comment:

We have more of… a healthy diet of reading in school. I think that that’s been a bigger picture change.

Triangulation in a research context means the extent to which different sources of data support similar interpretations. Even though the specific data sources named in the original framing of the evaluation question above were replaced by a new, more feasible suite of data sources, it is still useful to see if there were any particular insights that tended to be reinforced by multiple data sources. The support for and from the district reading specialist (mentioned above) is one such example. Another closely related example is the apparent success of the strategy to focus the sustainability and literacy integration on a specific, well-reasoned genre of literacy activities, namely, use of informational, non-fiction texts. This was a prominent feature of the project list, it was the focus of much of the work of the literacy/sustainability study group, it was embedded in the design of the quasi-experimental assessment of student skills, and showed up in the written comments on questionnaires. A number of teachers also mentioned it during their evaluation interviews. For instance:

…it’s really sort of a stepping stone, and it’s just getting underway and people are just figuring out some strategies around best practices to teach using informational texts and getting materials together… (3rd grade teacher)

It is hard to imagine SSP’s literacy efforts having been as successful as they were without such a targeted, focused approach.

Evaluation Question: In what ways do the data from standardized

test scores, locally scored portfolios and teacher perceptions triangulate to suggest

effects of SSP on student literacy?


The other theme that jumps out in stronger relief when viewed specifically through the lens of triangulation is the importance of just having access to good books to use with the students. A Kindergarten teacher said in an interview that “…[SSP staff] brought books in on [sustainability] to read to children, and everything we do in science and math, we write about it. So there's always literacy tied to what we're doing…” The improved availability to “leveled” books was also mentioned by several teachers in the various questionnaires that were administered (often as a barrier that was thankfully in the process of being overcome). A grade 4/5 teacher said plainly: “…every time we wanted books or literacy that we've wanted to tie in with [sustainability] we've gone to them, and they've been able to give us books to do.” The school administrator commented fairly glowingly that:

…there have been so many titles of books that [SSP staff person] has brought to numerous faculty meetings and gatherings, they’ve been laid out so you can look at them and [district specialist] has looked at them and compiled, using informational text outlines of certain titles. So, it’s been very stimulating because it’s provided a thematic structure by which teachers can look at titles and then look at them to make sure they have them across levels and that they have a wide variety for all readers.

In some ways this may be a fairly mundane observation, i.e. that having access to good books is a critical part of any effort to connect sustainability to literacy. But perhaps it is also a good reminder of how much of an elementary school teacher’s life is spent just trying to get basic foundations in place. The message may share some important elemental aspect with the core message of sustainability, namely to take care of the basics like clean air, clean water, and healthy ecosystems.


PRESENTATION OF EDUCATOR AND STUDENT SURVEY DATA: BARNES AND CHAMPLAIN As discussed in detail in the methods section of this report, the primary purpose of evaluation survey efforts during 2003-04 was to pilot test a dose-response. This section presents data tables and scatterplot correlation graphs from the educator and student versions of the survey that both demonstrate key features of the measurement strategy and also show some compelling evidence of early success of the SSP model.

Educator Survey Results Table S1 summarizes the descriptive and inferential findings for two sets of educator surveys: 11 teachers from Barnes Elementary, administered January and February 2004, and representing 70% of the Barnes teaching staff; and 13 teachers from Champlain Elementary, administered June 2004, and representing nearly the entire teaching staff of Champlain. In general, S1 shows that most of the modules and indices are significantly correlated with an educator’s overall dose of SSP. An exhaustive exploration of the rich possibilities for interpretation and presentation of this data is beyond the scope of this report, but the following three scatterplot correlation graphs say a lot by themselves.

Table S1. Summary of Data for 2003-2004 SSP Educator Surveys, Relating SSP Dose to SSP-Related Outcomes

Variable (items included) N

_ X M SD R2 p F df

Dose composite (calculated from = d1,d3,d4) 24 1.0 .83 .92 - - - - Other place-based ed. training (item = d12) 12 2.0 2 1.0 - - - - Overall teacher practice

(overall module=p1,p2,p3,p4,p5,p6,l1,l2,l3,l4,l5,l6) 24 2.8 2.9 .72 .42 .001** 15.7 22

Use of local resources (L module = l1,l2,l3,l4,l5,l6) 24 2.5 2.4 1.0 .27 .010* 8.0 22

Use of local places (llp index = l1,l4) 23 2.6 3.0 1.1 .17 .05* 4.2 21 Use of local people (llpeop index = l2,l5) 22 2.7 2.8 .97 .22 .03* 5.8 20 Service learning (lsl index = l3,l6) 23 2.2 1.5 1.2 .38 .002** 13.1 21

“I learned a lot about the vital concepts of sustainability over the three years - not just ideas about how to teach it to kids but for my own personal awareness. The knowledge & support are always high quality. You will be sorely missed. Keep up this work- not only does it have a direct impact on communities, but it teaches kids to be empowered citizens.”




_ X M SD R2 p F df

Improving teacher craft (P module = p1,p2,p3,p4,p5,p6) 24 3.3 3.3 .53 .32 .004** 10.4 22

Meeting curricular goals (pcg index = p1,p4) 18 3.3 3.3 .57 .14 .13 2.5 16 Teacher collaboration (ptc index = p2,p5) 24 3.1 3.5 .70 .25 .01** 7.4 22 Teacher engagement/growth (pteg index = p3,p6) 22 3.5 3.5 .46 .23 .02* 6.0 20 Reports of student performance

(X module = x1,x2,x3,x5,x7,x9,x10,x11,x12) 24 3.0 3.1 .55 .30 .006* 9.3 22

Student engagement in learning (xsel index = x1,x5,x12) 24 3.2 3.2 .75 .02 .52 .44 22 Student academic achievement (xsaa index = x2,x10,x11) 24 3.2 3.0 .50 .12 .10 2.9 22 Student civic engagement (xsce index = x3,x7) 23 3.3 3.5 .75 .33 .004** 10.3 21 Student stewardship behavior (xsbb index = x4,x8) 23 2.7 3.0 .85 .31 .006** 9.5 21 Student test scores (xts index = sq. root of x9*x10) 6 2.5 2.4 .44 .35 .22 2.1 4 Helps students w/ learning challenges (item = x11) 13 3.4 3 .65 .02 .69 .16 11 Reports of whole school improvement

(W module = w1,w2,w3,w4) 24 2.9 3.3 .71 .28 .008** 8.4 22

School culture, people (wpeop index = w1,w3) 24 3.1 3.5 .74 .28 .008** 8.6 22 Environmental quality (wenv index = w2,w4) 23 2.8 3.0 .77 .19 .04* 4.9 21 Perceptions of community improvement

(Y module = y3,y4,y5,y6,y7,y8,y9,y10) 13 3.6 3.5 .24 .00 .83 .05 11

Community civic engagement (yce index = y3,y6) 13 3.7 3.5 .25 .09 .31 1.1 11 Community environmental quality (yeq index = y4,y7) 13 3.5 3.5 .38 .00 .95 .01 11 Community planning/decision process (ypdm index = y5,y8) 13 3.6 3.5 .30 .02 .62 .26 11 General community quality (ygen index = y3,y4,y5) 13 3.4 3.3 .43 .00 .91 .01 11 Program adds value to community (ypav index = y6,y7,y8,y9) 13 3.7 4 .35 .00 .97 .00 11 Personal efficacy/involvement in com. (item = y10) 11 3.4 3 .51 .15 .23 1.6 9 SSP-specific teacher practice (stp index = s2,s3,s4,s5,s6,s7,s8) 24 2.7 3.0 1.0 .24 .016* 6.9 22 SSP- organize curriculum around “big ideas” (item = s2) 20 3.1 4 1.2 .02 .54 .39 18 SSP- activities focus on environment (item = s3) 21 2.8 3 1.2 .29 .011* 7.9 19 SSP- activities focus on community (item = s4) 24 2.4 2 1.1 .13 .08 3.3 22 SSP- activities focus on econ./equity (item = s5) 17 2.4 2 1.1 .13 .16 2.2 15 SSP- activities focus on “stewardship” (item = s6) 23 2.7 3 1.1 .26 .013* 7.4 21

SSP- literacy lessons support science (item = s7) 19 3.3 3 .8 .19 .07 3.9 17 SSP- science, etc. supports literacy (item = s8) 17 3.1 3 1.1 .00 .95 .00 15

N = sample size; X = mean; M = median; SD = standard deviation; R2 = % of outcome variability accounted for by dose composite; p = statistical significance test, threshold < .05/(# of component indices);

* = significant at p < .05; ** = significant at p < .01; F = regression test; df = degrees of freedom Figure S2 below shows the relationship between total composite dose of teacher exposure to SSP and self rated impact on teaching practice. The first of three key


points to make about this Figure S2 is that the effect size it shows is very large8. The idea that an external program could account for well more than a third of the variance in scores about teacher practice in a school (that obviously has many other

simultaneous factors in play) is fairly remarkable. Before making this claim with full confidence, however, all other competing explanations would need to be convincingly ruled out. One such possible explanation is that the difference in demographics between Champlain and Barnes elementary schools could be affecting these results. A main reason the correlation (i.e. the slope of the

regression line) is so strong is because the educators from Barnes tended to respond with low estimates on the use of local resources and other teacher practice-related items on the survey, and the educators from Champlain tended to respond with high estimates on the same items. The survey was administered at Barnes just as SSP was getting started at that school, so their SSP dose was very low (i.e. nearly everyone was “0”). The survey was administered at Champlain near the end of SSP’s second year in the school, so their SSP dose was relatively high. Thus the strong correlation:

8 Please see the methods section of this report for a full discussion of effect sizes and other important notes for interpreting these graphs.

“This has been a great project. Our kids have been involved with both community and neighborhood projects. They are involved and care about what happens to others.”


Figure S2. Teacher Practice (overall module) From SSP educator surveys, 2003-2004


0.0 1.0 2.0 3.0 4.0



í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of s

core

s fr

om it

ems

abou

t

teac

her u

se o

f loc

al re

sour

ces

and

teac

her m

etho

ds/c

raft

(tea

prac

mea

)Sc

ale:

Stro

ngly

Dis

agre

e =

1, to

Str

ongl

y A

gree

= 4

Scal

e: Tw

ice p

er y

ear

or le

ss =

1, t

o O

nce a

wee

k or

mor

e = 4

YY

Y Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

The best fit multiple regressionline above shows that 42% ofthe variability in survey responseis predicted by dose of SSP. Theresult is statistically significant.

R2= .42, p = .001, n = 24.


lower dose lower outcomes; higher dose higher outcomes. The question becomes: ‘Are the lower outcome scores from Barnes due to the fact that haven’t had the benefit of SSP yet, or are they due to some other factor, e.g. being in a more inner city setting as opposed to the more suburban setting of Champlain, or some other factor inherent to that school?’ This year’s evaluation does not have enough data to conclusively rule out these possible alternative interpretations. Administering the educator surveys again at Barnes toward the latter part of next year could go a long way toward clarifying this issue. The second key point about the graph is also a possible competing explanation for this very large effect size. It is likely that training in other place-based education programs other than SSP is contributing to the higher scores from Champlain. A question was added to the survey between when Barnes educators took the survey and when Champlain educators did. When a sequential regression is run on the data from Champlain only in order to “control” for the influence of prior place-based education training, the effect size for the overall teacher practice module is less than half of that shown in the graph above. The sample size is too small, however, to make that calculation statistically significant. This should temper (but by no means eliminate) SSP’s claims of large effect size for effects on teacher practice. The third and final key point about the graph of teacher practice survey results is that the dose-response measurement strategy made it possible to combine survey results from Barnes and Champlain in a meaningful way to show highly statistically significant results. Even though the surveys were administered to the whole teaching staff at each school, the schools only has about a dozen teachers each. By themselves, the samples are too small to generate statistical significance. But in combination, the results are highly statistically significant, meaning that it is extremely unlikely (a fraction of a percent chance) that the same results could have happened just by chance. Had the measurement strategy been limited to a strict pre-post, matching sets approach, we would have been hard pressed to extract statistically significant findings at the end of the first year. Evaluation efforts for 2004-05 can specifically seek to rule out the possible competing interpretations described above. In the meantime, the finding from this current year of educators surveys provides strong quantitative evidence that SSP may be having a substantial effect on teacher practice. Figure S3 below shows the relationship between the dose of educator exposure and educators’ ratings of the degree to which they believe that SSP adds value to the community. Although a quick look at the flat regression line in the figure suggests no relationship a more careful reading suggests a different interpretation. Looking

“I am really excited about what this program can do for us here at Barnes!” - Educator survey respondent


only at the inferential tests of multiple regression and statistical significance (i.e. the line on the graph), this index would suggest that nothing at all is going on since it is

no higher for teachers who received high doses than those who received lower doses. However, looking at the ‘simpler’ data of means and medians (suggested by the dots on the graph) shows that the lowest score was from a respondent who “tended to agree” on every item, and that over half the participants “strongly agreed” to every item in the index. The clear message from this is that all survey respondents reported that SSP adds value to the community. This is an important finding, in that the very consistency of acclaim

for the program limited our ability to detect a statistical correlation with dose. This graph also provides a good lesson on the importance of looking at the sample size. The graph only represents 13 respondents (n=13), whereas many of the other lines in the data table above report nearly two dozen respondents. As it turns out, the graph represents only the educators from Champlain, since the Barnes version of the survey did not have that module of questions on it.

“We encourage our second grade students to feel empowered to be stewards of the environment (starting with school and reaching out to the community). We want them to understand the interrelationships among different components of society.”


Figure S3. SSP Adds Value To Community (index) From SSP educator surveys, 2003-2004


0.0 1.0 2.0 3.0 4.0



í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of r

espo

nses

to it

ems

abou

t SSP

help

ing

the

com

mun

ity m

eet e

nviro

nmen

tal,

prob

lem

sol

ving

, and

eng

agem

ent n

eeds

(ypa

v)Sc

ale:

Stro

ngly

Dis

agre

e =

1, to

Str

ongl

y A

gree

= 4

Y

Y

Y

Y YY

YY

Y

YYYY

The best fit multiple regressionline above shows that 0% ofthe variability in survey responseis predicted by dose of SSP. Theresult is not statistically significantbecause there is no correlation.

R2= .00, p = .970, n = 13. However,the high response scores make thisan example of a "ceiling effect."


Figure S4 displays an analysis of responses from a single survey item. The fact that statistical significance was achieved is clear evidence that the correlation pattern is fairly strong. It is intriguing that this result occurred on a single survey item with a relatively small sample (n=21), which serves as a good reminder that larger sample sizes do not necessarily lead to increased statistical significance. This graph presents compelling evidence that the presence of SSP can lead to an increasing prominence of environmental themes in a school’s curriculum9. The graph cannot, by itself, say to what extent that is occurring. But this survey result is highly consistent with data from qualitative interviews and the investigation into the integration of literacy and sustainability at Champlain, and so that very much strengthens the claim.

Student Survey Results Table S5 summarizes the descriptive and inferential findings for two sets of fourth and fifth grade student surveys: 53 students from Barnes Elementary, administered Dec 2003 – Jan 2004, and representing nearly all of the student population for those grades; and 80 students from Champlain Elementary, administered June 2004, again representing nearly all of the collective fourth and fifth grade student population from that school. Like the table for educator survey data, Table S5 shows highly statistically significant correlations between SSP dose and outcomes of interest, though the size of the effects is generally smaller in the student data than in the educator data. 9 This evidence also supports the argument for an emergent school-level dose factor. The argument is discussed in detail in the PEEC 2003-2004 Cross-program report (PEEC, 2004).

Figure S4. Focus On Environment (item)From SSP educator surveys, 2003-2004


0.0 1.0 2.0 3.0 4.0



í

í

í

í

í

í

í

í

í

í

í

í

1

2

3

4

Raw

scor

es o

n ite

m "L

earn

ing

activ

ities

focu

s

on th

e en

viro

nmen

t/nat

ural

reso

urce

s" (S

3)Sc

ale:

Twic

e per

yea

r or

less

= 1

, to

Onc

e a w

eek

or m

ore =

4

Y

YY

Y YY YYY Y

Y

Y

Y

YY

Y Y

YY Y YThe best fit multiple regressionline above shows that 29% of thevariability in survey responseis predicted by dose of SSP. Theresult is statistically significant.

R2= .29, p = .011, n = 21.


Table S5. Summary of 2003-2004 Student Survey Data Relating the SSP Dose of a Student’s Teacher and Student Reports of SSP-related Outcomes


_ X M SD R2 p F df

Dose composite (calculated from = d1,d3,d4) 112 1.4 2.3 1.3 - - - - Use of local resources

(L module = l1,l2,l4,l5)128 3.1 3.3 .69 .07 .005** 8.2 105

Use of local places (llp index = l1,l4) 118 2.9 3.0 .87 .00 .54 .38 96 Use of local people (llpeop index = l2,l5) 125 3.4 3.5 .74 .12 .000** 13.6 102 Student engagement-civic, learning (C module=c1,c2.c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14,c15)

133 3.3 3.4 .48 .10 .001** 11.9 110

Connection to community (ccc index = c1,c2,c7) 132 3.3 3.3 .58 .06 .010** 6.8 109 Self-efficacy (cse index = c5,c8) 128 3.5 3.5 .67 .09 .002** 10.1 106 Social responsibility (csr index = c3,c4,c6,c9) 133 3.6 3.8 .45 .05 .013* 6.3 110 Enthusiasm for learning(cel index = c10,c11,c12,c15) 132 2.9 3.0 .72 .01 .36 .85 109 Stewardship behavior (csb index = c13,c14) 128 3.1 3.5 .89 .26 .000** 37.0 105 SSP helps me get better grades (item = x10) 111 3.2 3 .86 .00 .81 .06 92 Satisfied with quality of school (item = x6) 124 3.76 4 .56 .04 .03* 4.7 102 School is environmentally healthy (item = w2) 128 3.6 4 .65 .04 .03* 4.7 105 Community is environmentally healthy (item = y4) 121 3.2 3 .80 .04 .06 3.6 100 Student attachment to place

(N module = n1,n2,n3,n4,n5,n6,n7,n8,l1,l2,l4,l5,c1,c2,c7)133 3.1 3.3 .48 .14 .000** 17.7 110

Learning through local resources (nlr index=l1,l2,l4,l5) 128 3.1 3.3 .69 .07 .005** 8.2 105 Time spent outdoors (nto index = n1,n2,n6) 129 2.9 3.0 .80 .29 .000** 45.6 106 Understanding of place (nup index = n3,n5,n7) 125 3.1 3.3 .74 .01 .34 .90 102 Overall affect about place (noa index=n4,n8,c1,c2,c7) 133 3.3 3.4 .56 .04 .03* 4.8 110

N = sample size; X = mean; M = median; SD = standard deviation; R2 = % of outcome variability accounted for by dose composite; p = statistical significance test, threshold < .05/(# of component indices);

* = significant at p < .05; ** = significant at p < .01; F = regression test; df = degrees of freedom Overall, this set of survey data provides further affirmation of the viability of the pilot dose-response measurement strategy. Figure S6 below shows the relationship between the SSP dose of a student’s teacher and student reports of measures about their own attachment to the place where they live. This bodes well for SSP and PEEC since “attachment to place” is perhaps the core idea of place-based education. Figure S6 represents a particularly robust calculation because the sample size is fairly large (n=133 students), and also because the module consists of so many (15) different items. That is why there is virtually zero chance that the surveys responses could have ended up in this configuration by chance. Something systematic is going on here. To the extent that other competing interpretations can be eliminated (similar to the discussion of teacher practice


change above), then SSP is well justified in claiming that this program helps kids feel and act more connected to the community in which they live. Not all of the evaluation findings are positive. Figure S7 shows that students whose teachers are exposed to SSP are not necessarily more engaged with the overall learning process. This means that SSP cannot defensibly rely on the logic chain connecting programs to student academic achievement via the intermediate step of establishing evidence of motivation to learn. That is not to say that SSP does not affect academic achievement. It is simply to say that a claim for a positive effect in this one

area will need to rest on evidence other than these particular surveys. This finding could perhaps be interpreted in light of the developmental needs of children. One of the unique aspects of SSP is that is focused on the early elementary grades. Both Champlain and Barnes are K-5 schools. It may be exactly right that SSP demonstrates stronger affects in terms of student engagement with their physical surroundings than any connection to academic achievement, which becomes increasingly more important in the upper grades. The philosophical underpinnings of

Figure S6. Student Attachment to Place (module)From SSP student surveys, 2003-2004


0.0 1.0 2.0 3.0 4.0

Dosage compositeScale: Teacher had no exposure to SSP = 0,

Teacher had very high SSP exposure and implementation = 4

í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of s

core

s fr

om it

ems

abou

t loc

al

lear

ning

, tim

e ou

tdoo

rs, u

nder

stan

ding

of

plac

e, co

nnec

tion

to c

omm

unity

(nm

ean)

Scal

e: St

rong

ly D

isag

ree

= 1,

to S

tron

gly

Agr

ee =

4

Scal

e: Tw

ice p

er y

ear

or le

ss =

1, t

o O

nce a

wee

k or

mor

e = 4

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

YYYYY

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

YYY

Y

Y

YY

YYYYY

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YYY

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

YYY

Y

YY

Y

The best fit multiple regressionline above shows that 14% of thevariability in survey responseis predicted by dose of SSP. Theresult is statistically significant.

R2= .14, p = .00, n = 133.

Figure S7. Student Enthusiasm for Learning (index) From SSP student surveys, 2003-2004


0.0 1.0 2.0 3.0 4.0



í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of s

core

s fr

om it

ems

abou

t stu

dent

atti

tude

s

abou

t bei

ng in

scho

ol, w

orki

ng o

n sc

hool

topi

cs a

t

hom

e, a

nd e

njoy

men

t of l

earn

ing (

cel)

Scal

e: St

rong

ly D

isag

ree

= 1,

to S

tron

gly

Agr

ee =

4

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

YY

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YYY

The best fit multiple regressionline above shows that 1% of thevariability in survey responseis predicted by dose of SSP.

R2= .01, p = .36, n = 132.


place-based education place a strong emphasis on developmental appropriateness. One of the individual members of PEEC, David Sobel, suggests in his books that the appropriate scale of activity for children grows as they grow. Early elementary school children can learn about place at the scale of their homes and their classrooms. Older elementary children are most appropriately engaged at the scale of the schoolyard and neighborhood. Middle school is the time to focus on making forays into the larger community, and high school is the best time to engage with issues of global scope (Sobel, 1998). Figure S9 shows that students at Champlain are reporting more time spent engaging in first hand personal contact with the natural world. Again, a percent variance of 29% is very large in this context. The correlation between student time outdoors and involvement with SSP is quite strong. Perhaps, like the teacher practice graph shown earlier, this difference can be attributed to different demographics

between Barnes and Champlain. That is a potential area to explore more deeply in future evaluations. Earlier in this report we mentioned the theoretical and empirical evidence for focusing on teacher-level factors first. However, Figures 22 and 24 show that changes in student behavior and attitude can, indeed, begin to manifest quite clearly within the first couple of years of a program intervention. In sum, even though testing the dose-response measurement strategy, was the primary focus of

Figure S9. Student Time Spent Outdoors (index) From SSP student surveys , 2003-2004


0.0 1.0 2.0 3.0 4.0



í

í

í

í

í

í

í

í

í

í

í

í

1.0

2.0

3.0

4.0

Ave

rage

of s

core

s fr

om it

ems

abou

t

freq

uenc

y of

vis

its to

out

door

are

as

vers

us ti

me

spen

t in

build

ings

and

aut

os(n

to)

Scal

e: Tw

ice p

er y

ear

or le

ss =

1, t

o O

nce a

wee

k or

mor

e = 4

Y

Y

Y

Y

Y

YY

YY

Y

Y

YY

YY

Y

Y

Y

Y

YY

YY

Y

Y

YY

Y

Y

Y

Y

YY

Y

Y

Y

YYY

Y

YY

YY

Y

Y

YYY

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

YY

Y

Y

YY

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

Y

The best fit multiple regressionline above shows that 29% of thevariability in survey responseis predicted by dose of SSP. Theresult is statistically significant.

R2= .29, p = .000, n = 129.

Figure S8. Sample of comments Barnes student surveys

“I think Lawrence Barnes is the best school to learn at! This school is awesome.” “I love to read and write.” “I think it would be a better place if nobody would fight.” “We should stay safe.”


the quantitative evaluation efforts this year, several potentially useful findings have emerged. There is still more to be learned from this data, should the opportunity present itself to drill both deeper and broader into the data.

Figure S10. Sample of comments Champlain student surveys “I had a lot of fun fixing up our neighborhoods.”

“I think that I learned about sustainable schools was that it was fun.”

“I think it would be a better place if nobody would fight.”

“That to keep a community together you need at least have a gathering every other month or more often.”

“I think this project made me think about our environment more. It made me care more about it.”

“Finishing it was long and hard.”

“I think that learning about sustainability was important because when we get older we well know how important it is to be sustainable.”


CONCLUSIONS The SSP is not about demanding that a specific topic, method, or activity be taught in a school. It is about encouraging people to think about connections and to see the web that weaves together topics (e.g. art and writing), places (e.g. classrooms and neighborhoods), and people (e.g. students and local workers). There is a parallel conceptual and structural evolution of the program taking place. As SSP’s sustainability-related content takes hold and begins to create a curricular web, so too the students, teachers, staff, and others involved in the process find themselves in an evolving framework of interconnectivity that has many benefits for the school and community alike. A specific example of this is the impact of SSP’s literacy efforts. By focusing on integrating sustainability broadly into as many aspects of the literacy curriculum as possible, SSP itself is becoming more broadly integrated into the school and district. In the broadest sense, what is happening with SSP is an expansion of perspective as participants begin to see the interconnectivity of all things. More specifically, at Champlain Elementary School, the SSP model’s “testing grounds,” teachers are learning how all elements of their curriculum are or can be related, and are gaining awareness of how the curriculum is linked between grades and between specials and classrooms. They are refining their skills in curriculum planning built upon a new foundation of essential questions and sustainability. Moreover, students are seeing that their learning process extends well beyond the classroom, to their neighborhoods, and to other students, the schoolyard, their parents, and other community adults.


IMPLICATIONS FOR PRACTICE Interpretation of the data used in this year’s evaluation points to a number of recommendations for program development, also called implications for practice.

• Time investment: While SSP work is not necessarily an “add-on” layer in the school day, SSP staff and teachers concede that taking on any new initiative does take more time and commitment than simply doing what one has done in the past. To adequately prepare teachers for the level of commitment involved--and the level of reward--SSP staff should continue to validate the notion that time investment is required, and that there is ample evidence that it is worth it in the longer term. It could be valuable, for instance, to provide teachers with testimonials about the longer term “pay out” of their initial time investment. Several Champlain School teachers expressly offered to be contacted by newer SSP teachers to assure them that the steep learning curve was worth the climb.

• Language: Allow participants to explore and define the language pertinent to the

project—particularly “sustainability” and “sustainable”, and to select the best way to represent the concept for their school and community. If participants feel a sense of ownership of the terms used to describe their school initiative, they will be less likely to consider it jargon and better able to promote the concept when connecting with community members, explaining their class work to parents or communicating with fellow teachers.

• Institutes: Several hindrances were attributed to two elements of the first summer

institute—namely, its short duration and limited participants. Given that the two summer institutes are such key program elements—the first institute for building the program’s foundation and getting participants excited about the work, and the second for reinvigorating and planning for staying power--we recommend that funding be sought to allow for expansion of the summer institutes to include a greater number of days and greater variety of participants.

• Inclusion: To avoid feelings of “outsider status” by any school personnel and to

promote school wide impacts and staying power, make explicit attempts to include all teachers equally from the beginning of program implementation.

• Transition staff: This year’s addition of an AmeriCorps member as part of the SSP

staff was highly regarded and seemed to contribute a considerable value in terms of quantity and quality of work. This position might also serve a valuable function as a school liaison who can assist a school in its third year as the site transitions away from heavy involvement of SSP staff.


• Maintenance plans: It is important to assure from the start that large projects such as the Living Machine have designated caretakers and a concrete, long term plan in place. One teacher’s internship creating a maintenance plan for the Living Machine is one such example. Designating a leader and developing such a plan early on in a project may help all involved to feel confident that the project will not be abandoned.

• Reflection: Build in time for reflection on the part of participating teachers. This

year’s study group is an involved example of doing this. Brief weekly check-ins at staff meetings is another way to promote reflection on the project.

• All Three E’s: Assure that the social equity element of education for sustainability

is addressed on par with the environmental integrity and economic prosperity elements. SSP has laudably included this element by building relationships between its project sites and the Peace and Justice Center, for instance, and could seek out more connections like this as well as encouraging teachers to see the links between what they teach and how it links to diversity and equity.

• Staff preparation: As the project takes on new areas of focus based on new

communities or different funding sources, assure that all program staff feel adequately prepared to implement the area of focus, such as coaching teachers on literacy work or diversity issues.

• Literacy-related implications: Consider building the quasi-experimental assessment

exercise as a way to eventually provide a solid body of evidence about SSP’s direct influence on student achievement. This type of curriculum sensitive measure may prove to have more decision relevance than analysis of standardized test scores. (See Literacy section above for detailed recommendations.)

• Surveys: The first round of analysis of the survey data collected in 2003-04

reveals some compelling confirming evidence of program success. Further exploration of the data is warranted. Survey administration efforts for 2004-05 should be strategically targeted to complement the existing data set. More detailed demographic information may allow a more thorough interpretation of differential results between Barnes and Champlain.

• Evaluation implications: Continue to explore the benefits of a deeper collaboration

and sharing of responsibility between program staff and evaluators, as was done this year with the literacy component of the evaluation. This could become a model for other PEEC programs, and also could greatly expand the internal capacity of SSP to integrate a useful evaluation mindset more deeply into the program philosophy and activity, thus providing for the most efficient use of limited financial resources.


LITERATURE CITED Abelson, R.P. (1985). A variance explanation paradox: When a little is a lot. Psychological

Bulletin, 97, 129-133.

Barton, M. L & Jordan, D. L. (2001). Teaching Reading in Science: A Supplement to Teaching Reading in the Content Areas, Teacher’s Manual (2nd Edition). Mid–Continent Research for Education and Learning, Colorado.

Bloom, B. S. (1976). Human characteristics and school learning. McGraw-Hill, New York.

Bosker, R. J. (1992). The stability and consistency of school effects in primary education. Enschede, University of Twente, The Netherlands.

Bucknam, R. B. (1976). The impact of EBoE: An evaluator’s viewpoint. Illinois Career Education Journal, 33(3), 32-36.

Carlsson, M., Sanders, D. (2004). Perspectives on collaborative projects between schools and external actors. Conference presentation. Retrieved May 19, 2004 from http://www.nfer.ac.uk/research/papers/perspectives.doc

Chi, B. (2000). Service-learning as “Citizenship” Education: The Promise and the Puzzles. Corporation for National Service, Washington, D.C. Retrieved August 1, 2004 from http://www.nationalservice.org/research/fellows_reports/2000/chi.pdf

Chin, J. (2001). All of a Place: Connecting Schools, Youth and Community through Place-based Education. Paper presented at the Funders Forum on Environment and Education, San Francisco, CA.

Cirillo, J. (2004). Understanding the World’s Interconnectedness: Stories from a

Sustainable School. Community Works Journal, 6:2.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfield, F. D., & York, R. L. (1966). Equality of educational opportunity. U.S. Government Printing Office, Washington, DC.

Comer, J.P. (1984). Home-school relationships as they affect the academic success of children. Education and Urban Society, 16(3), 323-337.

Comer, J.P. (1988). Educating poor minority children. Scientific American, 259(5), 42-48.


Coyle, K. (2004). Understanding environmental literacy in America: And making it a reality: What Ten Years of NEETF/Roper research and related studies tell us about how to achieve environmental literacy in America. National Environmental Education & Training Foundation, Washington, D.C. Retrieved August 1, 2004 from http://neetf.org/roper/ELR.pdf

Emekauwa, E. (2004). They remember what they touch: The impact of place-based learning in East Feliciana parish. Rural School and Community Trust. Retrieved August 1, 2004 from http://www.ruraledu.org/docs/Louisiana-East_Feliciana.pdf

Epstein, J. (1991). Effects on student achievement of teachers’ practices of parent involvement. In S. B. Silver (Ed.), Advances in reading/language research: Vol. 5. Literacy through family, community and school interaction (pp. 261-276). JAI Press, Greenwich, CT.

Epstein, J., & Becker, H. (1982). Teacher practices of parent involvement. Elementary School Journal, 83, 103-113.

Fullan, M. (1993). Change forces: Probing the depths of educational reform. Flamer, London.

Geisler-Brenstein, E., & Schmeck, R. R. (1996). The revised inventory of learning processes: A multifaceted perspective on individual differences in learning. In M. Brenham Y F. J. R. C. Chichi (Eds.), Alternatives in assessment of achievements, learning process, and prior knowledge (pp. 284-317). Lower, Boston, MA.

Generally accepted principles of teaching and learning: And their implications for local education support systems. (2002). Annenberg Institute for School Reform, Brown University. Retrieved August 1, 2004 from http://www.schoolcommunities.org/images/GAPTL.pdf

Gladwell, M. (2002). The tipping point: How little things can make a big difference. Little, Brown & Co., Boston, New York, London.

Hagevik, R. A. (2003). The effects of online instruction using geographic information systems to foster inquiry learning of middle school teachers and science students. Ph.D. dissertation, North Caroline State University. Retrieved August 1, 2004, from http://www.lib.ncsu.edu/theses/available/etd-11032003-235227/unrestricted/etd.pdf

Hart, R. (1997). Children’s participation: The theory and practice of involving young citizens in community development and environmental care. Earth scan, London.

Harvey, S. (2002). Nonfiction Inquiry: Using Real Reading and Writing to Explore the World in Language Arts, 80:1). National Council of Teachers of English.


Hungerford, H. R., & Volk, T. (1990). Changing learner behavior through Environmental education. The Journal of Environmental Education, 21(3), p. 8-12.

Jensen, B. B., Shack, K., & Mimosa, V. (Ends). (2000). Critical environmental and health education research issues and challenges. Danish University of Education.

Liebermann, J., & Hoody, L. (1998). Closing the achievement gap: Using the environment as an integrating context for learning. State Education and Environment Roundtable, San Diego, CA.

Living and learning in rural schools and communities: a report to the Annenburg Rural Challenge. (1999a). Cambridge, MA: Harvard Graduate School of Education.

Living and learning in rural schools and communities: lessons from the field. (1999b). Cambridge, MA: Harvard Graduate School of Education.

Lopez, M. (2003). Transforming Schools Through Community Organizing: A Research Review. Harvard Family Research Project, Harvard, MA. Retrieved August 1, 2004 from http://www.gse.harvard.edu/hfrp/projects/fine/resources/research/lopez.html

Loveland, E. (2002). Connecting communities and classrooms. Rural Roots Newsletter(3).

Luyten, H. (1994). School effects: Stability and malleability. Enschede, University of

Twente, The Netherlands.

Madaus, G. F., Callaghan, T., Krakow, E. A., & King, D. (1979). The sensitivity of measures of school effectiveness. Harvard Educational Review, 49(2), 207-230.

Marzano, R. J. (2000). A new era of school reform: Going where the research takes us. Mid-continent Research for Education and Learning, Aurora, CO.

_________. (2003). What works in schools: Translating research into action. Association for Supervision and Curriculum Development, Alexandria, VA.

Miles, M. B. and A. M. Huberman. (1994). An expanded sourcebook: Qualitative data analysis (2nd ed.). California: Sage.

Monroe, M. C. (2002). Evaluation's friendly voice: the structured open-ended interview. Applied Environmental Education and Communication, 1, 101-106.

Moore, G. A. (1999). Crossing the chasm: Marketing and selling high-tech products to mainstream customers. Harper Collins, New York.


National Environmental Education Advisory Council. (1996). Report assessing environmental education in the United States and the implementation of the National Environmental Education Act of 1990. US EPA. Retrieved August 1, 2004 from http://www.epa.gov/enviroed/pdf/report.pdf

Newton, R. A. & Rudestam, K. E. (1999). Your statistical consultant: Answers to your data analysis questions. Sage, Thousand Oaks, CA.

Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd edition). Sage, Thousand Oaks.

Paulsen, S. E. (1994a). Parenting style and parental involvement: Relations with adolescent achievement. Mid-Western Educational Researcher, 7, 6-11.

________ (1994b). Relations of parenting style and parental involvement with ninth-grade students’ achievement. Journal of Early Adolescence, 14, 250-267.

Pedersen, E., Faucher, T. A., & Eaton, W. W. (1978). A new perspective on the effect of first grade teachers on children’s subsequent adult status. Harvard Education Review, Vol. 48, No. 1. President and Fellows of Harvard College, Harvard, MA.

PEER Associates. (2004). Place-based Education Evaluation Collaborative: Cross-program evaluation report, 2003-2004. Retrieved October 1, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/

Place-based Education Evaluation Collaborative (PEEC) website: http://www.peecworks.org

Plumb, S. (2003). The potential power of place in education: Literature review. Master’s Thesis, University of Vermont. Retrieved August 1, 2004 from http://www.peecworks.org/PEEC/PEEC_Research/0009D4DB-007EA7AB.0/Plumb%20CMP%20Thesis%20Lit%20Review.pdf

Powers, A. L. & Duffin, M. T. (2003). An evaluation of the Community-School Environmental Education Program: 2002-2003. Retrieved August 1, 2004 from http://www.peecworks.org/PEEC/PEEC_Reports/S0004B2DD-0004B2DE.1/02-03%20CO-SEED%20Eval%20web.pdf

Prochaska, J. O., Decremented, C. C., & Norcross, J. C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47(9), 1102-1114.

Public Education Foundation. (2002). Highly Effective Teacher Study. Chatanooga, TN. Retrieved August 1, 2004 from http://www.pefchattanooga.org/www/docs/5.261


Putnam, R. (2001). Bowling Alone: The Collapse and Revival of American Community. Touchstone, New York.

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press, New York.

Sanders, W., & Rivers, J. (1996). Cumulative and residual effects of teachers on future student achievement. University of Tennessee Value-Added Research and Assessment Center, Knoxville, TN. Retrieved August 1, 2004 from http://www.heartland.org/pdf/21803a.pdf

Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A meta-analysis of research. In K. A. Reengineer, S. Hide, & A. Krapp (Eds.), The role of interest in learning and development, pp. 183-212. Erlbaum, Hillsdale, NJ.

Schiefele, U., & Krapp, A. (1996). Topics of interest and free recall of expository text. Learning and Individual Differences, 8(2), 141-160.

Schneider, B. & Cheslock, N. (2003). Measuring results: Gaining insight on behavior change strategies and evaluation methods from environmental education, museum, health, and social marketing programs. CoEvolution Institute, San Francisco, CA.

Scriven, M. (2004). Causation. Retrieve August 1, 2004 from http://www.arts.auckland.ac.nz/FileGet.cfm?ID=7745A664%2D64CE%2D48D2%2DA07B%2D39D663B23FF3

Smith, G. A., & Williams, D. R. (1999). Ecological Education in Action: on Weaving Education, Culture and the Environment. State University of New York Press, Albany, NY.

Sobel, D. (2004). Place-based education: Connecting Classrooms & Communities. Orion Society,

Great Barrington, MA.

_______(1998). Mapmaking with children: Sense of place education for the elementary years. Heinemann, Portsmouth, NH.

Stallworth, N. T., & Williams, D. L. Jr. (1982). Executive summary of the final report: A survey of parents regarding parent involvement in the schools. Southwest Educational Development Laboratory, Austin, TX.

Steinkamp, M. W., & Maehr, M. L. (1983). Affect, ability, and science achievement: A quantitative synthesis of correlational research. Review of Educational Research, 53(3), 369-396.

Stringfield, S., & Teddlie, C. (1989). The first three phases of the Louisiana school effectiveness study. In B. P. M. Creamers, T. Peters, & D. Reynolds (Eds.), School


effectiveness and school improvement: Proceedings of the second international congress, Rotterdam (pp. 281-294). Sweets & Jetliner, Lisa.

Strosahl, K. D. (1998). Assessing the field effectiveness of acceptance and commitment therapy: An example of the manipulated training research model. Behavior Therapy, 29, 35-64.

Tobias, S. (1994). Interest, prior knowledge and learning. Review of Educational Research, 64(1), 37-43.

Willingham, W. W., Pollack, J. M., & Lewis, C. (2002). Grades and test scores: Accounting for observed differences. Journal of Educational Measurement, 39(1), 1-37.

Wilson, E.O. (2002). The future of life. Vintage, New York. Wright, S. P., Horn, S., & Sanders, W. (1997). Teacher and classroom context effects on

student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 11, 57-67. Lower Academic Publishers, Boston, MA.


APPENDIX APPENDIX A: SSP LOGIC MODEL.................................................................................................. 103 APPENDIX B: SSP EVALUATION OVERVIEW 2003-04................................................................... 104 APPENDIX C: INTERVIEW GUIDE................................................................................................... 106 APPENDIX D: EDUCATOR SURVEY ................................................................................................ 108 APPENDIX E: STUDENT SURVEY .................................................................................................... 111 APPENDIX F: MONTHLY STAFF OBSERVATION RECORD ............................................................. 116 APPENDIX G: SSP MONTHLY PROCESS WATCHER FORM ........................................................... 118 APPENDIX H: LETTER TO TEACHERS............................................................................................. 120 APPENDIX I: DIRECTIONS FOR TEACHERS ADMINISTERING SSP SURVEYS TO STUDENTS........... 121 APPENDIX J: SSP SURVEY ADMINISTRATION PROCEDURES ........................................................ 122 APPENDIX K: CHAMPLAIN LITERACY ASSESSMENT QUESTIONNAIRE ....................................... 124 APPENDIX L: PRE-ASSESSMENT QUESTIONS FOR LITERACY/SUSTAINABILITY STUDY GROUP.... 125 APPENDIX M: LITERACY TEXT ASSESSMENT RUBRICS .................................................................. 126 APPENDIX N: SAMPLE PAGE FROM INFORMATIONAL TEXT PROJECT GUIDE............................. 128


APPENDIX A: SSP LOGIC MODEL Hypothesis Resources/Inputs Activities/

Strategies Short Term Outcome Long Term

Outcomes

Using Sustainability aan Integrating Context will improve student learnin& community engagement, teacher leadership & parent/ community involvement.

Shelburne Farms: science, agriculture & environmental education resources; project direction; & partnerships VT EFS: project coordination;curriculum design; place-based & embedded proof. development models; networof EFS/place teachers & organizations

School District: Administrators’, specialists’ & teachers’ planning & inservice time Local community’s sustainability initiatives/ network: information & coordination of community partners Research: best practicesfor professional development; service learning; & place-basededucation Funding: private; local & national governmentin-kind Evaluation: Shared learning of PEEC

Teachers’ collaborative planning of curriculum, stewardship projects & community partnerships Concept-based curriculum units/courses related to sustainability for all students. Workshops on sustainabilitissues related to campus & community. Community partners’ ongoing collaboration withteachers, supporting place-based & sustainability education School/Community Stewardship projects as opportunities for school-wide interaction, improvedsustainability practices, & service-learning. Integration of SSP strategie& other professional development/student learning priorities Teacher study groups on developing curricula/projects, expectations/assessment ofstudent learning

Teachers coordinate &collaborate more on curriculum planning & instruction. Teachers & students demonstrate increaseunderstanding of sustainability. Teachers include morcommunity-based &schoolyard learningopportunities in curriculum. More teachers engaged in more extended communitypartnerships. Increased student engagement in community Teachers & students demonstrate more enthusiasm for teaching & learning. More parents involvein school & community projects Increased attention tosustainability practicein school building & grounds.

More decision-making & actionsthat lead to sustainable communities More integration of place-based EFS in curriculum Ongoing school-community engagement More schools engaged in EFS Systemic school improvement


APPENDIX B: SSP EVALUATION OVERVIEW 2003-04 Sustainable Schools Project Evaluation Methods 2003-2004 September 16, 2003 Amy Powers, Michael Duffin, George Tremblay Project evaluation will:

• Evaluate effectiveness of the SSP model in terms of process (program implementation)

• Evaluate effectiveness of the SSP model in terms of outcomes (results) • Provide useful information for project coordinators and funders to assist with

program development, justification and refinement

Evaluators’ Roles Meet with project staff to develop evaluation plan, and make modifications as

needed Data collection including site visits to schools, interviews, photo documentation Data analysis Report writing (to be completed by August 31, 2004) Provide planning and/or recommendations for Year Three evaluation (as

appropriate) Areas of Focus: Amy will focus on areas 1 and 3 below; Michael will focus on area 2.

Amy is contact person for SSP. Estimated number of days to complete SSP evaluation products: 30

SSP Staff Roles in evaluation process Develop Year Two evaluation plan with evaluators Provide input throughout the year via meetings, phone and/or email on evaluation

direction and appropriateness of instruments Liaison between evaluators and school (e.g. setting up interview schedule) Collect and share observation notes, project documentation, photos with evaluators

as designated Assist in administration of surveys, including data entry (organizational capacity of

Shelburne Farms) Provide data such as test scores as designated Provide incentives for participant participation in evaluation process Meet with evaluators to consider needs assessment for Year Three


Evaluation Strategies for SSP Evaluation 2003-2004 Area of focus Strategies Timing Evaluation Questions

a) Re-establish in-school process-watcher(s)—SSP staff (JC, TT) checks in with process watchers and provide written record to evaluators monthly

Begin Fall ‘03

b) Monthly monitoring/observations/reflections by SSP staff provided to evaluators (EZ will record at monthlmeeting)

Begin Fall ‘03

c) Teacher post-post survey (slightly modified from ‘02-03)

May ‘04

d) Teacher/school staff/community partner/SSP staff interviews (1-2 day site visit)

April ‘04

• What evidence is there that SSP work is anwill be sustained in a school after year one and two?


• As the model develops, what are the strengths and challenges faced by SSP staffand participants?

1. SSP One- Champlain Elementary (Amy, evaluatiocontact)

e) Student survey 4/5 team (cross-project survey modulemodified from CO-SEED’s) to measure community engagement/stewardship

September ’03 &May ‘04

• What effects does SSP work have on students’ awareness of places and people intheir community?


f) Analysis of standardized test scores (if they are providedto us). Suggested comparison between SSP One, SSP Two, district and state reading and writing scores for second, 4th and 5th grade.

When available (probably July ’04)

g) Analysis of student writing portfolio scores (if they are provided to us). Suggested comparison between SSP One, SSP Two for selected grade(s), with a focus on writing genre.

“

h) Teacher survey or interviews (as d. above; appropriateinterviewees will include questions about literacy)

April ‘04

2. Literacy (Michael, evaluation contact)

i) Use data from f. and g. above as SSP two baseline

• How is the presence of SSP helping teachermeet existing literacy goals and requirements?

• In what ways do the data from standardize

test scores, locally scored portfolios and teacher perceptions triangulate to suggest effects of SSP on student literacy?

j) Teacher/staff pre-surveys ( as c. above) Fall ‘03 k) Establish in-school process-watchers (as above) Fall ‘03

3. SSP Two- Barnes School (Amy, evil contact)

l) Student survey 4/5 grade (as e. above) (To serve as control group for Champlain’s 4/5 survey as well)

Sept ’03,May ‘04

• Gather baseline data for process and outcome questions


APPENDIX C: INTERVIEW GUIDE SSP Evaluation 2003-04 Interview guide for School Personnel and Community Partners May 2004 April 19, 2004 Note: the content of these questions is drawn from SSP Evaluation Overview evaluation questions

Focus on understanding what’s working with SSP, how and why it’s working, and what could be improved

Confidentiality Permission to record

Questions for School Personnel (educators, administrators): General program satisfaction

1. (open-ended, see what emerges) In your eyes, what 2-3 things stand out as evidence of SSP success this year? (whether in your own classroom or in the school as a whole)

2. What has been most challenging about having SSP as part of your school’s focus?

Teacher Practice

3. As a result of working with SSP, would you say you have changed the content or style of your teaching? In what ways? (Check in about: use of outdoors, school yard, community partners, sustainability or place themes)

4. What is it about the program that helped you change your practice? (one on one

coaching, summer workshops 2002-03, study group, modeling, campus ecology resources?) What things have been most helpful?

5. Has SSP presence in the school fostered any change in the level of collaborating

you do with other teachers? (how about other teachers? Within and across grade levels?)

6. Three areas of emphasis for SSP are: unit development, integrating literacy and

outdoor teaching. Have these areas of emphasis been helpful to you? Are there other areas of focus that would be more useful to you?

Student outcomes


7. In what ways, if any, have you seen sustainability-related work affecting your students? Are they more aware of places in their community? People in the community?

8. As a result of SSP work, would you say that students’ level of engagement in

their community has changed in any ways?

9. Has your own level of civic engagement changed since SSP involvement in the school?

Literacy

10. SSP emphasized literacy this year. To what extent did your literacy and sustainability work this year either support each other or get in the way of each other?

Program Staying Power

11. In your view, is the sustainability theme as prevalent this year as last school year? (or has it become more prevalent? Less prevalent?)

12. What evidence have you seen that the integration of sustainability into the

curriculum and school life will continue at Champlain beyond this year? a. In your view, what aspects of the program are most likely to have ‘staying

power’? b. What aspects of the program are most likely to disappear?

Specific follow-ups for each interviewee extracted from monthly reports:


APPENDIX D: EDUCATOR SURVEY


APPENDIX E: STUDENT SURVEY


Recorder: Meeting date: Date form submitted:

APPENDIX F: MONTHLY STAFF OBSERVATION RECORD SSP Evaluation 2003-2004 To be used at one SSP staff meeting each month The purpose of this form is to provide the evaluation team with monthly insights into SSP work at various sites, and to encourage reflective practice on the part of SSP staff. Throughout each month, please try to keep notes (mental, paper or otherwise) that will jar responses to these questions. If you don’t have a response to a particular question in a given month, leave it blank. Since this form can be useful in prompting follow-up in interviews, please record names when possible (i.e. teachers, administrators, community members; students can be referred to as “third graders in Ms. Smith’s class”.) Teacher practices 1. What evidence did you see this month of the teachers utilizing resources (human or

otherwise) from “outside” the school? (You might comment on the type of interactions you saw, types and/or depth of relationships forming and evidence that these relationships have staying power.)

2. What evidence do you see that literacy is being integrated into sustainability units? 3. What evidence do you see that teachers are using essential questions in their

teaching, either specifically or generally? 4. What evidence did you see/hear of sustainability concepts, topics, or resources

being used to enhance curriculum or learning? 5. What other observations have you made about changes in teacher practice? (i.e.

enhanced collaboration; enhanced dialogue about sustainability; when you’re teaching are teachers taking notes, demonstrating enthusiasm, participating?)

Schoolwide efforts 6. What evidence do you see that the school is sustaining efforts at school and

schoolyard improvements? (e.g. Living Machine, Schoolyard habitat, etc.) Process comments 7. What barriers, frustrations or challenges having to do with the SSP model are you

facing? What challenges do you notice school personnel, students or community partners facing? (example: “college students not showing up for their intended roles…”)

8. In what aspects of the SSP process did you see a particular strength or success?

(example: “second grade teachers very receptive to team meeting facilitation…”)


Other 9. Any other observations, evidence of SSP impacts you would like to share? 10. Any quotes or anecdotes (positive or negative) from teachers or students related to

the sustainability theme? 11. When possible, please take photos and gather documentation of student work or

teacher practice change. Feel free to attach any additional evidence of SSP in action—teacher or student work, photos, additional reflections sheets.

Please return this reflection sheet monthly to Amy Powers. Thanks!

([email protected]) 836 Snipe Ireland Road Richmond, VT 05477


APPENDIX G: SSP MONTHLY PROCESS WATCHER FORM Name (Who is completing this form?): School Name: Month of Record: Date Submitted (by 5th of month): Goals • To encourage reflective practice on the part of those involved with SSP. • To utilize the teachers’ and SSP staff’s insider’s view to keep an on-going record of SSP’s

impacts at individual sites. • To build up a pattern of information that we can use to improve the implementation and

outcomes of SSP and that ELCs can use to improve their work. Directions • Submit this form monthly to Amy Powers, by the 5th of the month. You might prefer to fill it out

as you observe things during the month, or to do it at one sitting at the end of the month. It may be sent by email or post.

• Consider each area each month and fill in those where you have something to say. It is fine to leave some areas blank if that is not where you have observed changes. Don’t get bogged down by figuring out which category something goes into.

• Clarification of terms: Who: The role and name (optional) of the individual or group on whom the impact was observed, for example, “Nick Brown, Conservation Commission Chair” or “Jane Emerson’s 5th grade class” Impact: The trend you have observed, for example “increased student participation in local government” or “higher engagement in written work”. Evidence: The specific thing you observed, for example, “implementation of weekly team planning meetings” or “the outdoor learning lab is now a regular item on the conservation commission’s agenda” or quotes such as “he said to me: “I’m never going back. When I see the way kids’ eyes light up at this stuff, I’m convinced it’s worth the effort.” Whenever possible, please attempt to quantify the trend. For instance: “Five miles of trails cleared by students.” or “Three new teachers approached me for ideas about community projects this month.” Key Question: This month, how has SSP affected the students, teachers, school, environment and community?

What impacts have you observed on…? Who Impact/Evidence Students Student learning/acad’c achievement Student understanding of sustainability, ah-has, struggles, ideas

Stewardship behavior (for school or comm..): decision-making, ideas, actions

Community engagement EXAMPLE

Three third graders

These students presented to NPA and PTO about butterfly gardenplanting EXAMPLE

Community engagement


What impacts have you observed on…? Who Impact/Evidence (involvement/participation) Other: Teacher Practice

Curriculum planning (integration, coordination, use of Esq., concepts)

Integration/coordination of school improvement initiatives (EFS & literacy, etc.)

Teacher collaboration or coordination– with colleagues/staff

Teacher collaboration with community resources, partnerships, co-teaching, planning, staying power

Teacher enthusiasm, interest; understanding of sustainability, interest in SSP

Other Campus sustainability practices Ideas, planning, decision-making School policies, campus practices and procedures

School building or schoolyard Other: Community Partnerships Vitality Changes in community interaction with school

Local orgs, business, gov’t: partnerships with schools, attitudes toward youth, interest in sustainability

Community partners’ ability to work with school/teachers

Parent involvement with school, teachers, community partners

Other: Other notes/ SSP process Barriers to progress Facilitators of progress Lessons learned Observations related to program “staying power”

Other Attach another sheet if more space is needed.

Please return to: [email protected]; Amy Powers, 836 Snipe Ireland Rd. Richmond, VT 05477 (p: 802-434-4257)


APPENDIX H: LETTER TO TEACHERS Fall 2003 Dear Teacher, The Sustainable Schools Project (SSP) is continuing to evaluate how the program affects schools, teachers and students, and to refine the program based on this information. Many of you were very helpful in your participation in the evaluation process last year, and we appreciate your on-going involvement. With this letter you will find several items: • A letter to be sent home with students for their parents. This is a passive consent

form for participation in the evaluation procedures. Parents do not need to complete a form if they are comfortable with their child being part of any evaluation activities. If they do not wish to grant such permission, they are asked to contact you. We would then ask that you not administer a survey to those children and also contact us so that we can make any other necessary arrangements to insure that parental wishes are honored.

• A sheet of instructions guiding you through the process of administering the surveys

to students in a systematic way. • A survey for students to complete. At this time, we are administering PRE surveys,

and hope that you will have students complete the surveys before or as close to the start of SSP-related work as possible. You will be asked to re-administer a similar survey to students this coming Spring and/or next year.

• An envelope in which to return the surveys. The sooner you can return these to us,

the better. Thanks in advance for your promptness! All evaluation materials and responses are kept private, though photos are often used in evaluation reports. If you have any questions about SSP you may contact Erica Zimmerman at Shelburne Farms at 985-8686. If you have any questions about the evaluation, feel free to call Amy Powers at 434-4257. Thank you very much for your cooperation. Sincerely, Amy Powers Michael Duffin Program Evaluation & Educational Research Associates


APPENDIX I: DIRECTIONS FOR TEACHERS ADMINISTERING SSP SURVEYS TO STUDENTS Teachers: Thank you for helping us to better understand the workings and impacts of the Sustainable Schools Project. The students’ input is very important. Consistent administration of these surveys will assure that the information they provide is usable in our research. If different classes approach the survey in different ways, we will have results that are not comparable. Please follow these steps when administering the enclosed surveys to your students. 1. Please administer the survey in the classroom (rather than sending it home.) 2. Please assure that students are not talking to one another or sharing answers. 3. For younger students, it may be necessary to read each question aloud as they

choose an answer. If so, please use the questions as they are on the sheet without providing examples. Simple clarifications of terms may be necessary.

4. Please allow a maximum of 30 minutes for completion of the survey. (It is unlikely

to take that long, however.) 5. There is a sample permission slip attached if your school deems it important to

receive parental permissions for this kind of program evaluation process.

To read aloud to students: 1. Your participation will help people understand how you feel about your school

and community and what kinds of things you are learning. 2. This is not a test. There are no right or wrong answers. You do not need to write

your name on the survey. 3. Please circle one number for each question. Notice there are different types of

questions. First you are asked to choose among several options, then you are asked to write your ideas in your own words. Later you are asked how strongly you disagree or agree.

4. Please do not leave any answers blank. Choose the answer that best matches how you feel.

5. You will be asked to complete a follow up to this survey at a later date. 6. Your ideas are valuable. Thank you for participating.

If you have any questions, concerns or feedback about this please feel free to call or email SSP’s program evaluator, Amy Powers, at 802-434-4257 [email protected].

Thanks again for taking the time to assist with this research.


APPENDIX J: SSP SURVEY ADMINISTRATION PROCEDURES Spring 2004 1. How to access the most current version of the surveys.

1. Go to PEEC website http://cee.schoolsgogreen.org/PEEC/ 2. Click on Research/Evaluation Tools 3. Click on SSP surveys available on-line 4. Print surveys from the PDF files: please do not alter the surveys in any way. If

you have suggestions, let us know. 5. After May 17 there will also be an on-line option for teachers/staff to fill out the

surveys if they prefer that method. (They will not need to print anything or attach anything to email.)

2. Administration of Champlain Educator/Staff surveys

Timing: I recommend you administer these no later than the week of May 24 (possibly earlier) so that you can do follow up during the school year.

Best option: administer the survey at a staff meeting so you have a captive audience. Try to also administer it to as many school personnel as possible—administrator, nurse, secretary, etc.

Evaluation Explanation: It will help if you frame the evaluation to them when distributing this. Why it is important, why they are being asked to do multiple things for multiple years, how SSP is part of PEEC, etc.

Confidential Collection: Collect them confidentially (i.e. provide an envelope at the meeting that people put the survey into and that you will seal immediately). Have a master list of staff on the outside of the envelope so that people can check off their names as they return surveys. This will allow you to keep track of who you haven't gotten them from. (They are asked to put their names on the last page of the surveys. Assure them that SSP staff will not be looking at the surveys.)

Other option (less preferable): distribute them in mailboxes and provide individual envelopes for folks to put them into before handing them in. Again, keep track of who has turned them in without looking at the surveys.

**Follow-up: Either way these are distributed, it will also be important to follow up with those from whom you haven't received surveys (whether they were at the meeting and didn’t turn it in or whatever). Here are some dates guidelines (I'm making up staff meeting date):

o May 18: distribute surveys to those at staff meeting and put in boxes of those not present

o One week later: Send around a reminder to everyone who has not returned surveys (including giving them the web link so they can access the survey themselves if preferred)


o One week after that: Distribute another survey with a note asking them to please complete and return the survey. (again, include reference to the web link)

Return Surveys: You can return all surveys to Amy in person or stick them in the mail to:

Michael Duffin 272 Eaton Road Swanzey, NH 2003446-3614

3. Administration of surveys to Champlain grades 4/5 students: Please follow the above general framework (confidentiality, timing, etc. with the following modifications

1. Give 4/5 teachers the printed surveys (same website procedure to access it) and a deadline.

2. You may wish to administer them yourselves in the classrooms if the teachers will go for that.

3. Follow up until you have all the sets returned, please!


APPENDIX K: CHAMPLAIN LITERACY ASSESSMENT QUESTIONNAIRE Champlain Literacy Pre-assessment Grade level:

Name (optional) 1) How often have you used non-fiction in literacy lessons with your students? (mark a point on the line) 3 times/week one time/week several times/month rarely 2) How much have you used non-fiction compared to fiction in your literacy lessons? (mark a point on the line) Much less than fiction As much as fiction Much more than fiction 3) How much have you integrated literacy into your sustainability units? Most of the time/significantly Once in a while Not at all 4) How much have you integrated sustainability topics into your literacy work? Most of the time/significantly Once in a while Not at all 5) What are the benefits you see of using non-fiction (informational texts)? 6) What are your barriers to using more non-fiction (informational texts)?


APPENDIX L: PRE-ASSESSMENT QUESTIONS FOR LITERACY/SUSTAINABILITY STUDY GROUP

1) Looking back at last year, in what ways did your literacy and Sustainable Schools work come together? In what ways did they help each other? In what ways did they obstruct each other or just not help each other?

2) What literacy teaching strategies or goals do you especially want to work

on this year?

3) How much have you used non-fiction with your students?

4) Which writing genre is most challenging for you to teach? Why?

5) In what ways do you imagine you will bring together literacy and sustainability work this year?

6) What do you want your students to know, understand and construct through your sustainability curriculum this year?

7) How will you know if you’ve been successful with your students with

your sustainability curriculum this year?


APPENDIX M: LITERACY TEXT ASSESSMENT RUBRICS Champlain Literacy/Sustainability Integration

FIRST GRADE INFO TEXT ASSESSMENT (prompts and rubrics) Spring 2004

Student’s Name Date Q1) Conceptual Schema: Change/Cycle: Sequencing Task: Provide laminated cards of the tree in the four different seasons, the branch/bud in the four seasons, and the four names of seasons. What happens to the tree and the branch in winter, spring, summer and fall? Use these cards to show me. Level/Criteria

3 2 1

Q1 Understands how the sequence of seasons affects living things (e.g. a tree)

Matches all cards and verbalizes their relationships

Matches most cards and identifies them, but without explaining relationships

Makes few matches and shows misunderstanding of the sequence

Q2) Parts of a whole – Use of photographs/close-ups Task: Show pages 6 and 7. How do these two pages go together? [Back-up: Why did the author put this page next to that page? Tell me more. Tell me other reasons.] Q2 Understands the use of close-ups to show parts of a whole

Describes all smaller pictures as parts of the bigger picture. Discusses the connection to the specific season being illustrated.

Describes some smaller pictures as parts of the bigger picture. Accurately names the pictures.

Describes pictures in isolation from each other. May have misunderstandings.

Q3) Directionality – position of sentences/captions Task: Show pages 4-5. How do you read these pages? OR I’m going to read you this page. Where will I start? Where will I go next? [Back-up: Which sentences go with which pictures?] Q3 Understands the intended direction of how to read text and its relationship to the pictures or graphics

Points to all text and pictures in an appropriate direction from start to end, as intended by the author. Accurately moves from top to bottom, left to right.

Describes a logical sequence which doesn’t match author’s intentions of top to bottom, left to right.

Points to some of the sequence of the page.


Sustainable Schools Project 2004 3RD GRADE INFO TEXT READING ASSESSMENT May 2004 Prompts Q1 Conceptual Schema: Life Cycle/Habitat/Interdependence Frogs grow and change. Describe the habitat they need as you tell about each stage. (supporting prompt: tell me more…) Q2 Parts of a whole – Use of subheadings to show parts of the whole concept/topic Q2a) The subheading is missing for one section. What should we name that section? Why does that make sense? Q2b) How do all these subheadings go together? Q3) Text Feature – captions Match a picture to each section of the text, and match a caption to each picture. Explain why they go together. Rubric Student’s name Date Level/Criteria 3 2 1 Q1 Understands how life cycle changes relate to habitat conditions

Accurately describes at least two stages of the frog’s life cycle, matched with each stage’s habitat conditions

Describes one or more stages of the life cycle with one or more habitats matched

Describes stages and habitats without any connection or shows no understanding

Q2 Understands the use of subheadings to show parts of the whole (lifecycle)

Names the subheadings and gives creative explanations of headings

Names the subheadings and explains how subheadings relate to the main idea (the title)

Explains parts of the headings but doesn’t explain/understand the relationship of the main idea

Q3 Understands the use of captions to add supporting facts/details

Matches captions appropriately to the text and explains how content of the caption fits with the focus of that section.

Matches all the captions but gives incomplete explanations (with at least one accurate explanation)

Has inaccuracies or misunderstandings in matching. No explanations of focus.


APPENDIX N: SAMPLE PAGE FROM INFORMATIONAL TEXT PROJECT GUIDE