city year whole school whole child fy10 evaluation report · whole school whole child fy10...

City Year Whole School Whole Child

FY10 Evaluation Report

Prepared by

Brett Consulting Group with the

City Year WSWC Evaluation Team

December 2010

Brett Consulting Group 103 Raymond Ave.

Somerville, MA 02144

City Year WSWC FY10 Evaluation Report Table of Contents

0. Introduction …………………………………………………………………….. 1 I. Overview of the WSWC Evaluation Initiative .……………………………….. 1

A. Overview of WSWC and the Evaluation Process …………………………....... 1 B. FY10 Evaluation Objectives ………………………………………………….. 2 C. FY10 Evaluation Questions ………….………………………………………. 3 D. FY10 Evaluation Products…………..………………………………………... 3 E. The FY10 Evaluation Team…………………………………………………… 4

II. Outcomes, Indicators, and Implementation Measures ……………………… 4

A. Overview …………………………………………………………………….. 4 B. FY10 Student-Level Literacy Data: Measures, Accomplishments, Challenges, and Learnings.................................................................................... 6 C. FY10 Student-Level Attendance Data: Measures, Accomplishments, Challenge, and Learnings...................................................................................... 7 D. FY10 Behavior Test Team Pilot……………………………………………… 7 E. Indicators/Metrics for Measuring Implementation ………………………….... 9

III. WSWC High School Theory of Change ……………………………………… 10

A. Overview and Process ………………………………………………………... 10 B. The High School TOC…………………………………………… ………….. 10

IV. Methods ………………………………………………………………………… 12

A. Overview ……………………………………………………………………... 12 B. Existing Tools……………………………………………………………….... 12 C. New Tools for FY10 ………………………………………………………… 13 D. Summary of Methods and Potential Learning ……………………………….... 13 E. Plan for Analysis ……………………………………………………………… 15 F. Grades 9 Student Survey Pilot…..……………………………………………... 16

V. Preliminary Findings …………………………………………………………… 21

A. Overview and Response Rates ………………………………………………… 21 B. WSWC Inverventions/Participation……...…………………………………….. 21 C. Corps Member Preparation and Support………………………………………. 25 D. Corps Member and Team Performance……………………………………….. 28 E. Perceived Impacts of City Year…..……………………………………………... 30 F. Student Level Outcomes………………………………………………………... 34

VI. Sharing and Using Data and Findings ………………………………………… 41

VII. Summary of Findings, Discussion, and Recommendations …………………. 43 A. What We Learned about WSWC ………………………………………………. 43 B. Recommendations/Actions Steps ……………………………………………… 51

VIII. Conclusions ……………………………………………………………………… 53

Appendices Appendix A: City Year WSWC High School Theory of Change Appendix B: City Year Student Survey, Grade 9

City Year Whole School Whole Child (WSWC) FY10 Evaluation Report I

Prepared by Brett Consulting Group and The City Year WSWC Evaluation Team

December, 2010 0. Introduction This report describes accomplishments and results for City Year’s evaluation related to its Whole School Whole Child (WSWC) model in FY10. The overall goal of the evaluation work was to continue to test, understand, and strengthen the model to assure that it is replicable and scalable. In FY10, we had three foci: 1) to continue to build and refine City Year’s internal capacity for evaluation of WSWC by revising, developing, and implementing measures, tools, and methods for collecting and sharing quality information from a variety of stakeholders, 2) to ramp up collection of individual course performance data for students given intensive support from City Year in various school districts with different practices/policies, to more systematically collect attendance data for individual students, and to begin exploring ways to understand student behavior as related to City Year’s interventions; and 3) to sharpen the differentiation in our data collection between the three age levels: elementary (especially grades 3-5), middle grades (6-8), and high school (grade 9). This report is organized as follows: I. Overview of the WSWC Evaluation Initiative II. Outcomes, Indicators, and Implementation Measures III. WSWC High School Theory of Change IV. Methods V. Preliminary Findings VI. Sharing and Using Data and Findings VII. Summary of Findings, Discussion, and Recommendations VIII. Conclusions I. Overview of the WSWC Evaluation Initiative A. Overview of WSWC and the Evaluation Process In FY08, City Year established Whole School, Whole Child (WSWC), a more standardized model of school service to address the academic, social, and emotional needs of children, both individually and collectively in their school environments. Following research highlighting the importance of four indicators that measured as early as the sixth grade could predict whether or not students would graduate from high school (Nield, Balfanz, and Herzog, 2007)1, City Year began focusing its WSWC efforts on getting students back “on track” by helping them to improve their attendance, behavior, and course performance in English and Math (the “ABCs”). Accordingly, the model is tiered according to how much intended interaction students have with City Year. For example, students who may benefit most from individual interventions receive one-on-one or small group tutoring and/or behavior coaching (Tier 2); others are in classrooms, homerooms, or after-school programs

1 Nield, R.C., Balfanz, R., & Herzog, L. An early warning system. (2007). Educational Leadership, 65:2, pp.28-33.

City Year FY10 WSWC Evaluation Report, p. 1

where City Year corps members provide more generalized assistance (Tier 1); at a minimum, students in WSWC schools are beneficiaries of City Year’s school-wide efforts, such as assemblies, lunch programs, family engagement, and school physical improvement projects. We expected that student level outcomes would be best realized when the intervention is more targeted, intensive, and developmentally appropriate for the different grade levels. Evaluation of both implementation and outcomes in a deep and meaningful way, by these grade levels, is crucial to the success of these efforts. The FY10 WSWC Evaluation Plan built on the FY08 and FY09 goals of testing, understanding, and strengthening the model, with added foci of differentiating between the elementary and middle school grades and early exploration of evaluation for the grade 9 program, which was still in development. We continued to define and refine metrics (for both outcomes and performance) for the “ABCs” (attendance, behavior, course performance in English and mathematics), obtain data from a variety of stakeholders, and analyze it for factors that appear to contribute to or hinder success while also making sure that our City Year sites were developing the capacity for strong evaluation. B. FY10 Evaluation Objectives Specifically, the objectives for FY10 WSWC evaluation were:

• Continue to understand available metrics and determine ways to collect unique and often confidential student or class level data from school districts on literacy as well as increase the number of teams providing beginning, middle, and end of year literacy data.

• Develop standardized measures across a variety of instruments for assessing student progress and find ways of interpreting disparate standards.

• Develop methods/metrics for attendance and behavior initiatives, including ways to standardize these, and pilot these across the Network.

• Complete the adaptation of our Theory of Change (TOC) for use in the high school grades. • Adapt teacher and service partner instruments for use at the middle (grades 6-8) and high school (grade 9)

levels. • Develop and pilot a high school student survey. • Conduct complete administration and new scaling of the grade 6-8 student surveys. • Create corps members mid-year survey focused on training and preparation. • Analyze data according to school level (i.e., elementary, middle, high). • Continue to revise/create summarizing documents for team and site use in order to increase usage

of data to inform service. • Deepen exploration/analysis of data collected across instruments. • Collect targeted information (both oral and written) to uncover challenges in implementation of

WSWC strategies. • Continue to increase the quality of information collected by sites (higher response rates, more

thorough and accurate information).


C. FY10 Evaluation Questions As in FY08 and FY09, a set of key questions guided the evaluation work in FY10. These were:

1. Metrics. What are the available metrics to describe literacy, attendance, and behavior initiatives? How can these be standardized across schools/districts for comparison purposes? What challenges remain in terms of data collection—access, numbers, quality?

2. Interventions/Participation. What do the key interventions both in-school and after-school

look like at each school level (grades K-5, grades 6-8, grade 9)? How many students are receiving these interventions and at what dosage levels? Are sites meeting their numeric targets at the different school levels both for the in-school and after-school components? Are they accomplishing what they set out to accomplish qualitatively? What are the barriers to meeting targets? What happened in the behavior initiative pilot?

3. Preparation. How well-trained, prepared, and supported are corps members for their different

roles (e.g., literacy training, attendance improvement) in WSWC schools at the different levels? What aspects of training and support are most related to corps members feeling prepared at the different levels?

4. Performance. What is the quality of corps member/team performance as assessed by

different stakeholders at the different grade levels?

5. Perceived impacts. What is the extent of perceived impacts/added value of City Year’s work at the student, classroom, and school levels at the different grade levels?

6. Student level outcomes. What student level outcomes are being realized (i.e., literacy for Grades

3-5, attendance for 6-9, the “C’s” for Grades 3-5 and 6-8)? How do outcomes different according to dosage/exposure to City Year? What is the added value of after-school for grades 3-5?

7. Factors related to differential perceived impacts/actual outcomes. What factors are

related to differential perceived impacts and actual outcomes by teams? (In a separate forthcoming report)

D. FY10 Products In addition to this report, the WSWC Evaluation Plan yielded a set of products in FY10 to provide the basis for future programming as well as future evaluation.

• A Theory of Change map for high school (see Appendix A) • Adaptations of City Year teacher and service partner instruments (mid-year and end-of-year)

for grades 6-9. Revisions of grade K-5 versions. • A new mid-year Corps Member Survey and revisions to the start-of-year and end-of-year

surveys. • A student survey for Grade 9 and a pilot test of that instrument (Appendix B for instrument,

analysis in Chapter IV)


• A refined pilot set of network-wide metrics for measuring implementation and outcomes, especially at the student level (Chapter II)

• A more sophisticated version of the Cross-Instrument Database to help us explore relationships among variables from different instruments administered at different time points over the year. (Note: We have not yet conducted analyses using this database.)

E. The FY10 Evaluation Team We accomplished our objectives through weekly hour and a half meetings of a cross-departmental evaluation team, with participation by the following:: External Consultant

• Belle Brett, Ed.D., of Brett Consulting Group (BCG), with assistance from Kathryn Race who carried out specific aspects of the work, including analyses of student survey data.

National Leadership • Stephanie Wu, Senior VP of Program Design and Evaluation • Jason Waite, Senior Director of Impact Development • Bethiel Girma, Natonal Manager of Behavior Initiatives • Rachael Alexander, National Manager of Literacy Initiatives • Justin Burton, Product Operations Manager and Coach • Marc Morgan, National Director of Secondary School Initiatives • Michelle Regan, National Director of Middle School Initiatives

The entire Evaluation Department • Gretchen Biesecker, Ph.D., Senior Director of Evaluation • Dannalea D’Amante, Evaluation Analyst • Ashley Kurth, WSWC Evaluation Analyst • Tavia Lewis, Evaluation Analyst and Coordinator

II. Outcomes, Indicators, and Implementation Measures Evaluation Question 1. Metrics. What are the available metrics to describe literacy, attendance, and behavior initiatives? How can these be standardized across schools/districts for comparison purposes? What challenges remain in terms of data collection—access, numbers, quality? A. Overview In FY010, City Year undertook a deeper exploration of ways to collect student level data in literacy, attendance, and behavior for Tier 2 students. As well, the team refined existing outcome and implementation measures in stakeholder surveys, added a corps member mid-year survey, developed and piloted Grade 9 student survey, and implemented pre and post administrations of the Grade 3-5 Student Survey, and the Grade 6-8 Student Survey. Table 1 provides an updated summary of FY10 measures and metrics by level of intervention. In Section IV, we will discuss further the tools/methods we used for collecting data about outcomes.


Table 1. FY10 Indicators & Metrics for Measuring Selected Student/School Outcomes by Level of Intervention

Outcome/construct Indicator/Metric Source(s)

Tier 2: REGULAR 1:1 OR SMALL GROUP ASSISTANCE2 Course Performance

Grade level/academic performance

Varies

School records

Test performance Standardized test scores School/district records Perceived impact, as a group, of CY on students receiving Tier 2 assistance: academic performance, motivation to learn, active engagement in learning, time spent on learning tasks, confidence in ability to learn, and study habits (grade K-9), ability to access resources, goal orientation (grades 6-9)

Six items for grades K-5 Eight items for grade 6-9

Teacher Surveys, mid-yr (MY/end-of-year (EOY)

Attendance Attendance

School attendance

School records

Perceived impact of CY on attendance of Tier 2 students as a group

One item Teacher Surveys, MY/EOY

Behavior Behavior

Behavior grades Official conduct record Referrals to office

School records Trackers (school staff and corps members)

Perceived impact on CY on behavior of Tier 2 students as a group

One item Teacher Surveys, MY/EOY

Tier 1: CLASSROOM ASSISTANCE AND/OR AFTER SCHOOL PARTICIPATION Perceived performance in English and in Math (self-report)

One item each Grades 6-8 Survey (post) Grade 9 (pilot)

Number of times received reward for academic performance/honor roll (self-report)

One item Grades 6-8 Survey (post)

Capable and committed learner Scale of 13 items Scale of 12 items Scale of 8 items

Grades 3-5 Student Survey, pre/post Grades 6-8 Survey, pre/post Grade 9 Student Survey, pilot (post)

School connectedness Scale of 11 items Scale of 6 items Scale of 7 items

Grades 3-5 Student Survey, pre/post Grades 6-8 Student Survey, pre/post Grade 9 Student Survey, pilot (post)

Number of detentions/suspensions (self-report) Number of times skipped school/classes, been asked to leave class

One item each Three items

Grades 6-8 Student Survey, post Grade 9 Student Survey, pilot

Community mindedness (relationships) Scale of 8 items Scale of 6 items

Grades 6-8 Student Survey, pre/post Grade 9 Student Survey, pilot

Perceived overall impact of City Year

Scale of 8 items Grades 3-5 Student Survey, post Grades 6-8 Student Survey, post Grade 9 Student Survey, pilot

Perceived impact of CY on students in class Perceived impact of CY on homeroom

Scale of 6 items Scale of 4 items

Teacher Surveys, MY/EOY Grades 6-9 Teacher Surveys, MY/EOY

Perceived impact of CY on students in CY’s after school programs

Scale of 10 items Service Partner Survey (SP), EOY

WHOLE SCHOOL School Climate Scale of 5 items

Scale of 6 items Grades 6-8 Student Survey, pre/post Grade 9 Student Survey (pilot)

Perceived CY impact on teacher Scale of 5 items Teacher Surveys, MY/EOY Perceived CY Impact on School Climate Scale of 6 items (teachers),

Scale of 14 items (SPs) SP/Teacher, EOY Surveys


B. FY10 Student-Level Literacy Data: Accomplishments, Challenges, and Learnings Overview During the 2009/2010 school year, all WSWC elementary and middle/high school teams were required to submit student level data three times a year (fall, winter and spring) to City Year headquarters. Elementary teams submitted literacy progress monitoring assessment data (e.g., DIBELS, Measures of Academic Progress, etc.) for any students in grades 1-5 who were in corps member assigned classrooms and/or received targeted literacy support (Tier 2). Accomplishments: Data Collection and Analysis

• 12 of 14 sites submitted EOY data. • Data for 1,691 students were analyzed and included as part of the network results. • Data from 10 different assessments were analyzed. • Sites seem to be more confident with the collection of literacy assessment data – more data

savvy. • Headquarters and a few sites worked through FERPA issues with particular districts as

encountered. • A couple of sites established relationships with districts that enabled them to have access to

district data systems in order to pull data for CY students in a more timely fashion. Data Collection Challenges/Learnings

Challenges What We Learned/Next Steps Did not have 100% participation from sites at all three time periods.

City Year will establish a School Operations Group whose members can collaborate with the Evaluation and Impact team members to monitor sites’ challenges in collecting student level data from school partners.

FERPA issues surfaced (e.g., CY Los Angeles). In collaboration with City Year’s School Relations Group need to create resources to support sites when they encounter FERPA issues.

Benchmarks not provided for all assessment data submitted (and if submitted an overall lack of confidence that the ones provided are national benchmarks as opposed to local benchmarks).

Still need to work on identifying benchmarks for some assessments; continue to research these and possibly reach out to a literacy expert for assistance.

Could not link dosage information from cyIMPACT to student outcome data.

Need to determine how to link output and outcome data. In FY11 sites will provide cyIMPACT-generated ID numbers for all Tier 2 students, enabling CY to connect these two data points.


C. FY10 Student-Level Attendance Data: Accomplishments, Challenges, and Learnings Overview FY10 was the first year all WSWC middle/high school teams were charged with the task of submitting student level attendance data. Middle/high school teams submitted formal school record attendance data for any students in grades 6-9 who received attendance monitoring and coaching support. Accomplishments: Data Collection and Analysis

• 6 of 8 sites submitted EOY data. • Data for over 1,600 students were analyzed and included in network results.

Data Collection Challenges/Learnings

Challenges What We Learned/Next Steps Most of the students for whom data were collected and submitted were on-track in attendance at the start of year. Therefore, opportunity to show growth from start to end of year was limited. Sites had a hard time collecting students’ prior year average daily attendance data across all grades 6-9, but particularly for transition grades (e.g., grade 6 and grade 9).

Selection of students for CY attendance Tier 2 interventions would be more appropriate if based on previous year attendance data if available. Next steps: examine challenges around and provide sites with additional guidance for collecting prior year average daily attendance data, in order to establish focus lists that serve the right students and to measure City Year’s impact on attendance.

Based on some sites’ large sample sizes, it was unclear whether data submitted was for focus list students only.

Need for clarifying for sites the importance of separating out data for focus list students.

Could not link dosage information from cyIMPACT to student outcome data

Need for creating a method for linking output data from cyIMPACT and student outcome data. In FY11 sites will provide cyIMPACT-generated ID numbers for all Tier 2 students, enabling CY to connect these two data points.

D. FY10 Behavior Test Team Pilot Overview of Intervention For this initial year of the school culture initiative, 14 teams serving in elementary and middle schools in Chicago, Detroit, New York, Miami and Washington, DC tested activities that address student behavior and school culture more broadly. These teams have contributed immensely to our organizational learning on how to best support positive student behavior and school cultures. These activities included:

• Morning Greeting – City Year actively supports attendance performance with students of any achievement level through a variety of measures- from supporting all students in the assigned homeroom (in addition to focus list students) to leading positive school climate efforts and appreciation/recognition programs for student attendance. City Year’s Morning Greeting is an opportunity to foster positive school culture and climate by ensuring that students begin their day with positive peer and adult interaction, while also fostering positive associations and connection to their school.


• Positive Behavior Support – As each corps member provides academic services to a classroom,

he or she also encourages positive, productive class behavior by supporting the teachers’ classroom management efforts. Corps members also encourage positive peer interaction during class transitions, lunch, recess and during after school hours. In addition, many teams support behavior recognition programs via event planning and student recommendations. Students are awarded for: 1) acting in a way that supports the school’s values, 2) resolving conflicts, 3) being a good friend, and 4) showing compassion as a citizen. Students are recommended for this status by faculty, staff and City Year corps members.

• Behavior Coaching Interventions – Corps members provide targeted behavioral interventions to

students on their focus list with below benchmark behavior. These targeted behavioral interventions can include the “Lunch Buddies” program, 50 Acts of Leadership, and/or a school partner’s existing program. All behavior support and coaching activities help to reinforce the importance of community among students. Although these activities are primarily intended to support students who struggle with behavior, they are intentionally structured to include other students and prevent any stigma associated with participation.

Kinds of Assessment Measures

• Service partner and teacher mid-year and end-of-year surveys contain items about the impact of corps members on student behavior, especially in relation to other students, as well as on school climate. These responses provide data about perceived impacts of City Year’s behavior interventions overall (not at the individual student level.)

• Student surveys, administered at the start-of-year and again near the end-of-year, contain items about student attitudes and behaviors, especially in relation to other students, along with self-report data about amount of exposure to City Year. Pre and post-surveys are linked by student generated IDs but cannot be linked back to specific students at this stage.

• Teams were asked to gain access to centrally collected student level behavior data or to create ways of collecting such data. In Washington DC a tracker and system to collect the following behavior data was created in coordination with the school: --Daily behavior grades (tracked by school personnel) --Trips to the office (tracked by school personnel) --Acts of Leadership (tracked by corps members) --Time spent in behavior coaching activities (tracked by corps members)

Accomplishments: Data Collection and Analysis

• Through monthly calls with Program Managers, reporting, and school observation, City Year’s National Manager of Behavior Initiatives, Bethiel Girma, monitored progress and collected feedback from all 14 test teams. From this work, City Year has begun to identify key learnings and promising practices that will inform program design for FY11.

• Selected items from service partner, teacher, and student surveys have been analyzed for behavior pilot teams and compared with teams that are not part of the pilot.

• The DC tracker described above can serve as a possible model for other sites.


Data Collection Challenges and Next Steps

Challenges What We Learned/Next Steps Lack of centrally collected behavior data

--We learned that some teams may be able to collaborate with school partners to create a school- wide behavior data tracking system (e.g., CYDC team helping to create behavior grading system). --We created the Student Selection Rubric to help teams select students for behavior coaching groups in the absence of centrally collected behavior data. --We are also piloting the use of the teacher scores from the Student Selection Rubric as student level data for teams without access to school based behavior data. --We integrated data collection and evaluation expectations in the MOU process for FY11- sites were instructed to choose schools that have a history of collecting behavior data.

Lack of corps member ability to access and track behavior data, when available, for many reasons, including: --Late start in schools (didn’t set up systems to track and collect behavior data) --Focus on gathering literacy and attendance data --Lack of school standardization of behavior management system --Teacher difference in reporting behavior issues.

--Created more resources (e.g., trackers, ops guides, etc) for FY11 and rolled them out to sites earlier in the year. --Integrated behavior prototype data collection deadlines in the MOU signed by each site as well as in the WSWC evaluation calendar.

Some schools have no formal way of measuring and tracking student behavior

See item #1.

Teams that struggled to collect other indicator data (literacy and attendance) also struggled to collect behavior data, suggesting challenges with setting up the appropriate systems and/or service partner support issues

As we continue with the prototype we will learn more about why the challenges exist and how we can address them.

E. Indicators/Metrics for Measuring Implementation Paying attention to the nature, extent, and quality of implementation of the WSWC model allows City Year to understand what factors may be contributing to results and to strengthen its service as a result. Table 2 summarizes the key indicators we used to help CY assess implementation of its service.


Table 2. Indicators/Metrics for Measuring Implementation

Aspect of Implementation Indicators Source Extent of intervention Numbers of students tutored, in CY

after-school programs cyIMPACT

Understanding of WSWC Scales of items SP & Teacher Surveys, MY Assessment of preparedness of Corps Members (CM)/Teams

CM ratings of training & preparation SP & Teacher ratings of CM/Team preparedness

CM Survey, MY SP & Teacher Surveys, MY/EOY

School/teacher support for team’s work

Ratings related to school support for CY

Teacher Surveys, MY/EOY

Team Leader (TL) leadership & communications with school/teachers

Ratings related to communications with TL, to overall leadership by TL Scale of items about TL leadership

SP & Teacher Surveys, MY CM Survey, EOY

Activities performed compared to baseline requirements

Checklist-yes/partially/no

Team Goals/cyIMPACT SP & Teacher Surveys, MY/EOY

CM/Team Performance Scales of items

Teacher Survey, MY SP Surveys, MY/EOY

Team practices Scale of items CM Survey, EOY Satisfaction with CMs & CY Satisfaction with working with

CMs/CY; quality of work SP & Teacher Surveys, MY/EOY

III. WSWC High School Theory of Change A. Overview and Process The Theory of Change (TOC) maps are designed to guide developments in both the program itself as well as evaluation, by linking WSWC’s program elements and strategies with its assumptions about how change occurs to produce desired outcomes. We created the original elementary grades TOC for WSWC in FY08 and the middle grades version in FY09. Finally, in FY10, we produced a TOC for City Year’s high school work. The whole evaluation team initially contributed ideas, especially regarding developmental issues that differentiated high schoolers from middle schoolers. Then a sub-team with the key input of Marc Morgan and Michelle Regan completed the task of creating the high school TOC. This project occurred as the range of City Year interventions for the high school years was being developed. Thus, the model was subsequently refined to reflect updates. B. The High School TOC The resulting High School TOC is currently more detailed than its elementary counterpart. (See Appendix A.) However, as the high school WSWC program achieves greater clarity, we hope to simplify it. The following summarizes what we kept the same and what we changed.

• As with our Elementary and Middle School TOCs, the model portrays WSWC in the High School as a City Year- defined program with a focus on student success. We have maintained the foundation of the model in which City Year capitalizes on its strengths to help students achieve the four Cs (capable and committed learners, who are connected to school and are


community-minded) and get on track in relation to the ABCs (attendance, behavior, and course performance in English and mathematics), with the ultimate goals of increasing the number of students who complete school, are college ready/job competitive, and have strong civic identities.

• We kept the general framework of the original model with its problems/needs, CY activities, core strategies, first line outcomes, interim outcomes, and longer term outcomes. As with the other two TOCs, the High School TOC remains student-centered and strategy-focused. As we have noted previously, highlighting core strategies is especially useful for multi-site programs that might use different activities, but common strategies, to achieve the same ends.

• The eight student needs (representing City Year’s values of “spirit, discipline, purpose, and pride,” both for student learning engagement and for school/community connectedness) were all refined to reflect the increasing developmental need for students to be more proactive in their learning, apply self-management skills, see the role of learning in their futures, develop their own identities, value both peers and adults, understand long term consequences of actions, and understand their roles within the broader community.

• City Year activities were similar to those of the middle grades although are subject to change as the model develops. These include the various types of academic support, strategies for improving attendance and behavior, and out-of-school time and service activities.

• Core strategies across activities were also similar to those of the middle grades, with a strong focus on youth voice, peer-to-peer relationships, connections to students’ interests and goals. Maintained across the grade levels were the emphasis on creating trusting relationships with corps members, coaching for growth/positive development, modeling positive values, opportunities for student contributions, and engagement of other resources. One change from the Middle School TOC was a sharpening of the strategy for peer relationships to read: “proactive peer systems for creating a positive and safe school environment.”

• Outcomes are presented as hierarchical—that is, some outcomes are important antecedents to other outcomes, but many outcomes are, in fact, highly interactive.

• Related to school/academic success, first line outcomes have students applying self-management skills and good study habits, seeing the relevance of learning, developing and taking ownership of goals, feeling confident about their ideas, experiencing both academic and non-academic success, and knowing how to access support. There is a much greater emphasis on longer term goal setting and on proactivity than in the Middle School TOC.

• Outcomes related to other aspects of student development and behavior include appreciating and respecting differences, understanding the long-term consequences of behavior, developing mature relationships with peers, collaborating to improve their school and community. Compared to the Middle School TOC, there is more emphasis on fuller development of identity and leadership skills.

• Longer term outcomes. Finally, the High School TOC is distinguished by the idea that students will “contribute to building a strong community and creating change through cooperation, respect, and taking responsibility.”

As with the Elementary and Middle School TOCs, the High School TOC has provided a shared framework to ensure alignment of goals and activities across the network.


IV. Methods A. Overview To answer the six implementation and impact questions posed in section I-C, we proposed a multi-faceted plan for data collection. Because our work was not a summative study, but rather a formative one, relying on site, team, corps member, and school participation for its success, we continued to be concerned with quality data collection from stakeholders in WSWC schools. As always, we emphasized appropriate administration of instruments, high response rates, and thoroughness and candor of information. Each tool has explicit instructions about how it is to be administered, including information about timelines and follow-up reminders to respondents. Instructions are available at two levels: an Operations Guide for Program Managers, and a Facilitator’s Guide for those doing the actual administration. Regular conference calls encouraged site sharing of effective ways to comply with expectations. In this section, we will describe the tools we used; their sample, administration, timing, and areas of inquiry addressed; our plan for analysis; and the tools/methods for each FY10 evaluation question. In addition, we will provide an overview of the initial analysis from our pilot Grades 9 Student Survey conducted in spring 2010. B. Existing Tools (Including Those That Were Revised) Team Goal Setting. This instrument asks teams for overall goals and targets prior to the start of their service. Goals-to-actuals can be measured at the end-of-the year. cyIMPACT Database: The City Year enterprise-wide database (cyIMPACT) allows for a standardization of data about outputs of service, including attendance at events and participation in ongoing enrollment-based programs. Although outputs are only the beginning of the story, they provide important information about depth and breadth of service. cyIMPACTallows for the ability to look at results closer to real-time for better support of goal management for teams across the WSWC network. Service Partner Surveys: Both mid-year and end-of-year surveys for service partners (principals, school liaisons, and after-school coordinators) were greatly revised in both FY08 and FY09. The focus on the mid-year survey is on understanding of City Year’s role and its relationship to school goals, assessment of team performance, and assessment of school climate factors. At the end-of-year, service partners rate the perceived impact on school climate as well as their overall satisfaction with City Year. Service partners have an opportunity to add open-ended examples of City Year’s impact as well as provide additional comments. Service partners also offer their assessment of City Year after school programs. In FY10, mid-year and end-of-year versions were developed for grades 6-9 as well as revised for the elementary grades (K-5). Teacher Surveys: The focus of this survey is teacher perceptions of corps members who work closely with them and students from their classes. Data are collected on corps member activities and time associated with the classroom; teacher support for corps members; corps member preparation and performance; assessment of corps member impact on classrooms, students, and the teachers themselves; assessment of school climate; and overall satisfaction. At the end of the year the focus is on perceived impact on students, classrooms, and school climate. As with the Service Partner


Surveys, teachers can provide open-ended data. The large sample size gives us an opportunity to examine relationship among factors. In FY10, mid-year and end-of-year versions were developed for grades 6-9 as well as revised for the elementary grades (K-5). Student Survey (grade 3-5 version, grade 6-8 version): The Student Survey is a two-sided instrument that focuses on student perceptions of their own learning commitment and capacity, their feelings about school and their own behavior at school, and their perceptions of City Year and its impact on them. The grades 3-5 version forms three high reliability scales: Learning Engagement, Connectedness to School, and Connectedness to City Year. The grades 6-8 version forms five scales: Learning Engagement, Connectedness to School, Community-Mindedness, School Climate, and Connectedness to City Year. Both versions of the survey also ask students to identify how much exposure they have with City Year and in what ways. In addition, the grades 6-8 version asks students to assess their performance in English and math and provide frequencies for detentions, suspensions, and times on the honor roll. The surveys are administered to classrooms of students where a corps member provides tutoring and other kinds of assistance. Surveys for students in grades 3-5 are also administered in after-school programs led by City Year, and for grades 6-8 in homerooms where a City Year corps member is present. Corps Member End-of Year Survey: This on-line survey is administered at the end of the corps member year and emphasizes perceived impact of service on the corps member, perceived impact on the students and school they served, and their team experience. Student Baseline and Performance Data: Schools across the City Year network use data to support instructional improvements although approaches vary widely. Team Leaders and local Service Directors work with school partners to collect student data for those students receiving targeted interventions. FY09 was the first time many sites requested data so early in the school year. In FY10, teams were expected to collect data at three time points for both literacy and attendance. C. New Tools for FY10 Corps Member Mid-Year Survey. The mid-year survey focuses on preparation, training, and support for service. Student Survey for Grade 9 (pilot). This survey was an adaptation of the Grades 6-8 Student Survey. It reflected the High School TOC, greater sophistication of students of this age, and their enhanced language ability. See Section F for a discussion of the pilot results and Appendix C for the instrument itself. D. Summary of Methods and Potential Learning Table 3 provides a summary of methods.


Table 3. Summary of Required WSWC Evaluation Tools for FY10

Tool (Responsibility)

Delivery Mode & Timing

Target Respondents

Relevant Areas of Inquiry What will we learn?

Team Goal Setting

Spreadsheet Pre-service

WSWC Point People / Team Leaders

Process and Outcome targets against which teams can be measured

How do teams intend to fulfill baselines and what are their performance targets?

cyIMPACT Database (Teams & Sites)

Database Ongoing

Teams # of students receiving participating in after-school (Literacy data not available for FY10 due to data collection challenges)

What is the reach of WSWC in the after-school?

Service Partner Surveys (Teams & Sites)—Gr. K-5 & Gr. 6-12

On-line (paper available) Mid-Year (MY) End-of-Year (EOY)

School Principals School Liaisons After School Directors

Understanding of WSWC; Team performance; Impact of CY on school climate; Satisfaction with service Suggestions for improvement

How well do SPs understand the model & their school’s roles? How do SPs perceive the impact & the quality of CY implementation at the school level?

Teacher Surveys (Teams & Sites)—Gr. K-5, Gr. 6-12

Paper or on-line MY EOY

Teachers with CY working in classroom.

Understanding of WSWC; CM performance Impact on classrooms, student learning and behavior, teacher Perceived impact on school

How do teachers perceive the WSWC model and their own roles, CM impact and quality of CM implementation?

Student Surveys (Teams & Sites) Gr 3-5 version, Gr. 6-8 version

Paper Pre and Post

Students at WSWC schools, in classrooms or homerooms with CMs, and after-school led by CMs

Capacity for & commitment to learning, Connectedness to school, Community-mindedness, Perceptions of City Year

How do student outcomes vary according to the intensity of their CY exposure? How do students feel about City Year and its perceived impact on them?

Corps Member Survey (Sites)

On-line MY EOY

Corps Members Assessment of preparation, training & support, TL leadership, team experiences Assessment of CM impact

How well prepared are CMs for different aspects of service? What is the quality/level of site support & training? To what extent do corps members believe they have an impact on students/schools?

Student Data Varies by location Varies by location

School Liaisons / Principals / District officials/Teachers

Data types and their comparability across various markets

What data are available for CY and corps members’ use in improving service delivery? What do our preliminary student outcome data tell us about CY’s intervention in terms of literacy, attendance, and behavior?


E. Plan for Analysis

• Target numbers from Team Goal Setting for the after-school were compared with actual numbers in cyIMPACT.

• Quantitative data from surveys were analyzed using SPSS • Descriptive statistics were compiled for selected variables (frequencies, means, standard

deviations) • Scales were created for groups of items in the Teacher Surveys, Service Partner Surveys, and

Corps Member Surveys • Scale analysis for the Grades 6-8 Student Pre-Survey was confirmed. • Pre- and post-student survey data for grades 3-5 were compared, incorporating grade,

gender, and level of exposure to City Year. Pre- and post-student survey data for grades 6-8 were also compared, looking at the same variables, with the addition of student reports of grades and behavior.

• Selected qualitative data were coded for themes. • Literacy data were collected from different instruments were standardized and progress in

literacy compared across time points. Data from grades 6-8 attendance initiatives were analyzed with respect to students’ movement on- or off-track. Initial behavior data were examined more for viability as measures.

• Selected variables from a variety of the above instruments were summarized at the team level (into the Cross-Instrument Database) and will be examined using regression and best case and worst case scenarios.

• Each tool was reviewed for its future utility and revised as needed. Table 4 matches the evaluation questions with the methods used in our analyses to answer these questions.


Table 4. Match of Evaluation Questions and Methods

WSWC Evaluation Question Tools/Methods 1. Metrics. What are the available metrics to describe literacy, attendance, and behavior initiatives? How can these be standardized across schools/districts for comparison purposes? What challenges remain in terms of data collection—access, numbers, quality?

Research data available through the different WSWC districts. Create methods of standardizing. Informally assess challenges.

2. Interventions/Participation. What do the key interventions both in-school and after-school look like at each school level (elementary, middle, high)? How many students are receiving these interventions and at what dosage levels? Are sites meeting their numeric targets at the different school levels both for the in-school and after-school components? Are they accomplishing what they set out to accomplish qualitatively? What are the barriers to meeting targets? What happened in the behavior initiative pilot?

Team Goal Setting cyIMPACT CM EOY Survey Service Partner MY /EOY Surveys Teacher MY/EOY Surveys Behavior pilot

3. Preparation. How well-trained, prepared, and supported are corps members for their different roles (e.g., literacy training, attendance improvement) in WSWC schools at the different levels? What aspects of training and support are most related to corps members feeling prepared at the different levels?

Service Partner MY /EOY Surveys Teacher MY/EOY Surveys CM MY/EOY Surveys

4. Performance. What is the quality of corps member/team performance as assessed by different stakeholders at the different grade levels?

Service Partner MY/EOY Surveys Teacher MY/EOY Surveys

5. Perceived impacts. What is the extent of perceived impacts/added value of City Year’s work at the student, classroom, school level at the different grade levels?

Service Partner MY/EOY Surveys Teacher MY/EOY Surveys CM EOY Survey Student Surveys (grades 3-5 pre and post, 6-8 pre and post, grade 9 pilot)

6. Student level outcomes. What student level outcomes are being realized (i.e., literacy for Grades 3-5, attendance for 6-9, the “C’s” for Grades 3-5 and 6-8)? How do outcomes different according to dosage/exposure to City Year? What is the added value of after- school for grades 3-5?

School records (will vary by district) Student Surveys (grades 3-5, grades 6-9), pre and post

7. Factors related to differential perceived impacts/actual outcomes. What factors are related to differential perceived impacts and actual outcomes by teams?

Available data, Cross Instrument Database

F. Grade 9 Student Survey Pilot Overview In spring 2010, City Year developed and launched a pilot study for a Grade 9 Student Survey. Patterned after the Grades 6-8 Student Survey and based on the new High School Theory of Change (see Chapter III), the purpose of the survey is to find out about student attitudes and behaviors related to their studies, school, and relationships with others as well as their assessment of City Year. As with the other surveys, our intention was to develop a survey that could be administered in the fall as a pre-survey and with some modifications, in the spring as a post-survey. The purpose of the pilot was to find out how well the survey worked. Because these surveys were collected in Spring 2010 from a limited and non-representative sample (without corresponding “pre” data), our analyses are preliminary. The smaller size of the sample precluded developing any scale scores at this time.


Methods The tool. The pilot survey consisted of four sections with 27 items about learning engagement (8 items), school connectedness (7 items), relationships with others (6 items), and school climate (6 items). Additionally, the core survey asked students about the amount of help they received from City Year (7 items) and the extent of City Year’s helpfulness (11 items), their assessment of their grades in English/Language Arts and Math, the frequency of their school-related behavior (8 items relating to attendance, detentions, suspensions, honor roll, and other awards), their agreement with a set of beliefs and attitudes (around making a difference, plans for graduating from high school, preparation for 10th grade, and feelings about City Year), and background information, including a question to rate the honesty with which students answered the survey. In all there were 68 items in the core survey (not including five additional items to gauge their interest in possible volunteer opportunities). Sample/Administration. Sites were provided paper versions of the Grade 9 Student Survey to administer to students with whom they worked either in class or homeroom. Philadelphia (six schools) and San Antonio (one school) participated in the pilot. Analysis. We reviewed descriptive statistics for each item for distribution as well as amount of missing data We also constructed scales based on our theory of how items should be grouped and tested the reliability of these scales (using Cronbach’s alpha).3 To help look at the more complex relationships between the various factors and gender, grade, exposure to City Year services, and perceived school climate, we conducted a series of stepwise regression analyses. Results Respondents Total number of responses: A total of 170 usable surveys were received from grade 9 students. Sites: Philadelphia returned 93 (54.7%) usable surveys, and San Antonio, 77 (45.3%). Gender: Slightly more than half of the survey respondents were girls (52.9%, n=90), and 45.9% (n=78) were boys. Two did not give gender. Age: Mean age was 15.04 (SD 0.67), with 18.2% (n=31) age 14, 60% (n=102) age 15, 18.8% (n=32) age 16, and 1.8% (n=3) age 17. Two did not give age. English as main language spoken at home: Two-thirds (67.6%, n=115) reported that English was the primary language spoken at home, and 28.2% (n=48) said that it was not. Seven did not respond. City Heroes: Only 8.8% (n=15) said that they participated in City Heroes. 11.2% (n=19) were not sure.

3 The sample size was not large enough to conduct a full scale analysis. In any case, it is preferable to conduct scale analyses prior to program implementation.


Honesty in completing survey: Most respondents (85.9%, n=146) said they were honest most or all of the time when completing the survey. Only a few (7.1%, n=12) said they were hardly honest or honest only some of the time. The rest (22.4%, n=38) said they were honest about half the time. Two did not respond. City Year exposure: Almost two-fifths (n=64) of respondents reported receiving help from CY in English/language arts at least one or two times a week; 56.1% (n=92) of responding students reported receiving help from CY for mathematics this often as did 47.9% (n=80) in another subject, (Table 5.) As well, 35.6% (n=58) received help regarding attendance issues, 57.6% (n=95) on homework or assignments, 47.6% (n=78) on behavior, and 55.3% (n=89) on preparing for college or career. Most students reported some formal association with City Year at least once or twice a week, with a mean of 3.38 (SD 2.57) different kinds of help received at least once or twice a week from City Year. Thus, students reported receiving the most help with math, homework, and preparation for college or career.

Table 5. Percent/Frequencies of Different Kinds of Exposure to City Year Type of help from CY (number of respondents)

Never/ hardly ever

% (n)

Once in a while % (n)

1 -2 times a week % (n)

3 or more times a week % (n)

English/LA (165) 32.7% (54) 28.5% (47) 20.6% (34) 18.2% (30) Math (164) 25.0% (41) 18.9% (31) 23.2% (38) 32.9% (54) Another subject (167) 25.7% (43) 26.3% (44) 25.7% (43) 22.2% (37) Attendance (163) 37.4% (61) 27.0% (44) 20.9% (34) 14.7% (24) Homework/assignments (165) 18.8% (31) 23.6% (39) 25.5% (42) 32.1% (53) Behavior (166) 23.8% (39) 28.7% (47) 25.0% (41) 22.6% (37) Preparing for college or career (161)

24.2% (39) 20.5% (33) 26.1% (42) 29.2% (47)

Self-reported performance in English/Language Arts and Mathematics: Two-thirds of students reported that they were getting As or Bs in English/Language Arts (65.6%, n=107) Almost as many (63.2%, n=100) said they were getting As or Bs in mathematics. Only a small number reported receiving mostly Ds or Fs in either subject. (Table 6.) Table 6. Frequencies of Self-Reported Performance in English/Language Arts and Mathematics and

Percent Receiving Help from CY at Least 1-2x a Week, by Grade Received Self-reported performance in… (number of respondents)

Mostly As % (n)

Mostly Bs % (n)

Mostly Cs % (n)

Mostly Ds or Fs % (n)

English/Language Arts (163) 20.2% (33) 45.4% (74) 28.2% (46) 6.1% (10) % receiving help from CY at least once or twice a week

29.0% 37.0% 47.8% 44.4%

Mathematics (158) 25.9% (41) 37.3% (59) 27.2% (43) 9.5% (15) % receiving help from CY at least once or twice a week

43.9% 63.8% 57.1% 50.0%

Self-reported performance by Amount of Help from City Year: More students with Bs in English/Language Arts received help from City Year in this subject area at least once or twice a


week than those who received any other grade (43.5% of all of those who said they received help at least once or twice a week), but within a grade category, a larger percentage of those who said they got mostly Cs, received help (47.8% of those who reported Cs) than within other grade categories. Only 29% of those who said they got As reported receiving City Year help in English/Language Arts. In mathematics, more B students (43.5%) got help from City Year than students with any other grade in math. Also, within grade groupings, a larger percentage of B students (63.8%) received help from City Year in math than students in other grade groupings. Those who had As reported receiving the least help, but even then 43.9% of those with As said they received help in math at least once or twice a week from City Year. Attendance and Lateness: Almost half of students (45.5%, n=76) reported being late to school at least three times in the last month, but less than 17% reported skipping school, skipping out on classes during the school day, or being asked to leave a class by a teacher. (Table 7.) These three behaviors were somewhat correlated with each other. Lateness was also correlated with these behaviors, but less so. Approximately two-thirds of students reported never skipping school or skipping out on class in the last month. About three-fifths reported not being asked to leave class by a teacher.

Table 7. Percent/Frequencies of Self-Reported Attendance and Lateness Times in past month— (number of respondents)

Never 1-2 times 3-5 times 6 or more times

Skipped school (166) 68.7% (114) 14.5% (24) 6.6% (11) 10.2% (17) Been late to school (167) 29.9% (50) 24.6% (41) 15.6% (26) 29.9% (50) Skipped out on classes during school day (165)

65.5% (108) 18.8% (31) 6.1% (10) 9.7% (16)

Been asked to leave class by a teacher (167)

61.7% (103) 21.6% (36) 7.2% (12) 9.6% (16)

Detentions, suspension, honor roll, recognition for positive behavior/attendance: Students were also asked about the frequency of detentions, suspensions, appearance on the honor roll (or receipt of an award for grades), and recognition for positive behavior or attendance during the current year. (Table 8.) More than half of students reported at least one detention (56.3%, n=94), but only 27.7% (n=46) responded that they had been suspended one or more times. Just under half (48.5%, n=79) reported that they had been on the honor roll or received an award for grades, and just over three-fifths (62.8%, n=103) said they had been recognized for positive behavior or attendance at least once. Almost one fifth (19.5%, n=32) reported that they had been recognized six or more times during the year.


Table 8. Frequencies of Self-Reported Number of Detentions, Suspensions, and Times on Honor Roll, Times Recognized for Positive Behavior or Attendance

Never 1-2 times 3-5 times 6 or more times Detentions 43.7% (73) 27.5% (46) 10.2% (17) 18.6% (31) Suspensions 72.3% (120) 19.3% (32) 3.6% (6) 4.8% (8) Honor Roll or award for grades

51.5% (84) 27.6% (45) 12.3% (20) 8.6% (14)

Recognition for positive behavior or attendance

37.2% (61) 27.4% (45) 15.9% (26) 19.5% (32)

Factor analysis. Analysis of the data yielded scales with acceptable reliability based on Cronbach’s Alpha. (Table 9.) However, as mentioned, we were not able to conduct a full scale analysis due to sample size.

Table 9. Alphas, Number of Items, and Sample Items for Each Grade 9 Survey Factor Factors/Scales Alpha # of

items Sample Items

Learning Engagement .84 8 When I work hard in school, I succeed. I come to class with my assignments finished. I enjoy learning new things in class.

School Connectedness .87 7 I enjoy coming to school. I feel like adults in my school respect me. I care about my school.

Relationships .85 6 I get along well with people who are different from me. I stand up for someone who is being picked on. I think about how my behavior affects others.

School Climate .80 6 My school feels like a safe place. Adults in my school treat students fairly. Students in my school are respectful of one another.

Helpfulness of City Year .91 11 How much has City Year helped you-- --Understand why learning is important --Understand how your behavior can affect your grades --Organize yourself better for studying

Overall scale scores. Learning Engagement had the highest overall mean and the least amount of variation (SD). The School Climate scale (About Your School) had the lowest mean. (Table 10.)

Table 10. Overall Mean Scores for Each Grade 9 Survey Factor Factors/Scales N Mean Standard Deviation (SD) Learning Engagement 158 4.06 0.63 School Connectedness 156 3.36 0.92 Relationships 163 3.66 0.86 School Climate 156 3.08 0.85 Finally, the mean total Helpfulness of City Year score (based on 11 items, with a 4-point scale, ranging from 1-“not at all” to 4-“a lot,” for a total possible score of 44) was 34.49 (SD 7.77), or a standardized mean of 3.14


V. Preliminary Findings A. Overview and Response Rates In Sections II, we answered our first evaluation question (on outcomes and metrics). In this section, we share some findings that address each of the other evaluation questions.4 As shown in Table 11, response rates for all tools in FY10 were at least 74%. Response rates were comparable to FY09.

Table 11. Response Rates for WSWC Instruments Administered in FY10

WSWC Instruments Total

distributed#

received Response Rate

FY10 Grades 3-8 Pre Student Surveys* 115 110 87% Mid-Year Corps Member Survey 1495 1476 99% Mid-Year Teacher Survey 1428 1053 74% Mid-Year Service Partner Survey 370 282 76% End-of-Year Teacher Survey 1308 1036 79% End-of-Year Service Partner Survey 347 276 80% End-of-Year Corps Member Survey 1448 1441 99% Grades 3-8 Post Student Surveys* 115 94 82% *Based on total # of WSWC teams serving grades 3-8

B. WSWC Interventions/Participation Evaluation Question. Interventions/Participation. What do the key interventions both in-school and after-school look like at each school level (grades K-5, grades 6-8, grade 9)? How many students are receiving these interventions and at what dosage levels? Are sites meeting their numeric targets at the different school levels both for the in-school and after-school components? Are they accomplishing what they set out to accomplish qualitatively? What are the barriers to meeting targets? What happened in the behavior initiative pilot? Corps Member and Team School-based Activities At the end of year we asked service partners (principals, school liaisons and/or after-school coordinators) to indicate which activities corps members performed for their school or program. Table 12 shows their responses by overall grade level (elementary and middle/high), but is not a perfect one-to-one match with teams, as in a number of cases, schools had more than one respondent, and some schools may have had no respondents. In-school academic tutoring, the core activities of WSWC, was the most commonly reported WSWC activity, with over 90% of service partner respondents naming it at both grade levels. At the elementary level, Starfish Corps was the next most often reported City Year activity at 80%, followed by after school academic or homework support at 74%. Only three-fifths said that corps members provided whole class academic support. At the middle and high school levels, whole class academic support along with after school academic or homework support were the next most

4 Results are included only when they apply to an evaluation question.


common activities reported after in school academic tutoring, with 86% and 87% respectively. Attendance programming was named by only 40.5%. A greater percentage of middle/high school teachers (63%) named positive behavior support as a corps member activity compared with elementary teachers (52%). At both grade levels, parent and family engagement were mentioned by about half of respondents, but supporting transition or non-classroom times was named by more than two-third of elementary level respondents and three-quarters of middle/high school respondents.

Table 12. WSWC Activities as Reported by Principals, School Liaisons, and After School Coordinators

Activity Elementary % (n)

N=163 Middle/High School % (n)

N=116 In school academic tutoring 90.2% (147) 92.2% (107) Whole class academic support 59.5% (97) 86.2% (100)

Positive behavior programming 52.1% (85) 62.9% (73) Report card conferencing n/a 26.7% (31) Attendance programming n/a 40.5% (47) Starfish Corps 80.4% (131) n/a After school academic or homework support 74.2% (121) 87.1% (101) After school enrichment programming (non (Starfish corps)

32.5% (53) 75.9% (88)

In-school extra-curricular clubs/activities or enrichment activities

41.1% (67) 62.9% (73)

Career/college focused events n/a 31.0% (36) Parent and family engagement 50.9% (83) 48.3% (56) Capacity building projects 31.9% (52) 49.1% (57) Supporting transition or non-classroom times 68.1% (111) 77.6% (90) Physical/community service projects or service learning programs

63.2% (103) 61.2% (71)

Teachers were asked about the activities corps members performed for their class(es) and/or homerooms (for grades 6-12). (Table 13.) Corps members were also asked to report the school-based activities in which they participated. The most common activity reported by teachers in grades K-5 was literacy/English/language arts (ELA) one-on-one or small group tutoring. For corps members working with students in grades K-5, the most commonly reported activities were homework assistance, after school-other enrichment, and literacy/English/language arts one-on-one or small group tutoring, with at least 87% of corps members doing each of these. (Teachers were not asked about after-school activities.) Teacher and corps member responses appeared to be quite correlated, with the exception of whole class academic support, which corps members were much more likely to say they were doing (67% compared with 47%). Mean number of activities out of the 11 listed on the corps member surveys was 6.17 (SD (2.13) for those serving grades K-5. The profile for those serving grades 6 and above is quite different, at least from the corps member perspective. Teachers reported that corps members most often did whole class academic support, followed by behavior coaching, with just over half of teachers saying that corps members did the former, and just under half, the latter. Corps members concurred that their most common in-school activity was whole class academic support although, as with those working at the elementary level, a somewhat larger percentage of corps members (73%) than teachers reported doing this activity. In addition, three-quarters of corps members said they did literacy tutoring, compared with only half of


teachers, and two-thirds reported doing math tutoring compared with fewer than two-fifths of teachers. However, among English teachers only (n=128), 86.7% reported that corps members did one-on-one or small group tutoring in ELA; and among math teachers only (n=108), 92.6% said that corps members did one-on-one or small group tutoring. More than half of corps members in this age group (52.6%) said they did both ELA and math tutoring. The largest discrepancy between corps members and teachers was in whole class (or homeroom) positive behavior support. Seventy-one percent of corps members indicated they did this activity compared with only 36% of teachers saying that corps members did so. Just a third of corps members said they did attendance coaching (similar to what teachers thought). Corps members working with students in grade 6 and above were most likely to say they provided homework assistance (86%). Mean number of activities out of the 11 listed on the corps member surveys was 6.88 (SD (2.66) for those serving grades 6 and higher.

Table 13. WSWC Activities as Reported by Teachers and Corps Members Grades K-5 Grades 6-12 Activity Teachers

% (n) N=587

Corps Members % (n)

N=684*

Teachers % (n)

N=449

Corps Members % (n)

N=578* Literacy/English/LA one-on-one or small group tutoring (tier 2)

86.7% (509) 87.7% (600) 49.2% (221) 74.4% (430)

Math one-one-one or small group tutoring (tier 2)

50.6% (297) 46.1% (315) 37.9% (170) 65.6% (379)

Whole class academic support* 47.4% (278) 67.3% (460) 53.5% (240) 80.4% (465) Whole class (or homeroom) positive behavior support

51.8% (304) 58.8% (402) 36.1% (162) 70.8% (409)

Report card conferencing n/a 3.7% (25) 17.1% (77) 35.1% (203) Attendance coaching (tier 2) 8.5% (50) 12.4% (85) 27.2% (122) 32.0% (185) Formal mentoring or behavior coaching

37.5% (220) 43.4% (297) 48.6% (218) 53.3% (308)

Service learning/community service projects

30.3% (178) 42.0% (287) 27.6% (124) 45.7% (264)

After school academic support n/a 78.4% (536) n/a 79.8% (461) After school—other enrichment, including Startfish Corps, After School Heroes

n/a 88.7% (607) n/a 65.6% (379)

Homework assistance n/a 88.7% (607) n/a 85.6% (495) *from midyear surveys Team Goals-to-Actuals Prior to the start of service, each team created goals for the number of students they would serve in different in- and after-school programs (e.g., 1:1 or small group literacy and Starfish Corps or other after school programs). At the end of year, these goals were compared to their actual numbers in order to better understand how many teams met, exceeded or did not meet their goals and why. This process was introduced in FY09 but still proved to be problematic in FY10. We were, however, able to identify a number of challenges that we have addressed for FY11 to improve the quality and accuracy of the goals-to-actuals data for this year. We are not reporting goals-to-actuals for literacy here because of the concern we have about the reliability of those data.


Challenges Plan for FY11

Teams were generally not provided the focus lists for students in the classrooms where they served as early in the school year as they had hoped. Therefore, many teams enrolled all of the students in their classrooms into programs with the intention of only entering participation data for those that ended up on their focus lists. This greatly affected the actuals for number of students enrolled in different programs, as well as calculations for average hours per student.

Sites have received more in-depth trainings around program design as well as how to translate programming into data entry for cyIMPACT. Sites have been trained to enroll only those students that are on focus lists even if those lists are not available at the start of service in schools. Further, we at CY HQ have identified processes for removing students from analyses that have zero sessions and zero hours logged, in the event any teams improperly enroll students that do not actually receive any programming.

We were not able to see that sites were over-enrolling students into the database until later in the year when reports from cyIMPACT became available. Additionally, it was difficult to identify teams that may not have been up-to-date throughout the year with their data entry.

We will be providing monthly team goals-to-actuals reports to teams starting at the beginning of the year and throughout the year in order to provide opportunities to identify and address any data entry issues early on. We have also developed a new report available to all teams this year that allows selection of different time periods to view data entry activity during those selected time periods. This allows sites to see if a team has not entered data for a given week, month etc.

The importance of accurate data entry into cyIMPACT became more critical this year with the introduction of new AmeriCorps reporting requirements.

To meet these requirements, we will be linking dosage data from cyIMPACT to student level outcomes for each student. Reporting requires looking at outcomes only for students that reach certain specified participation thresholds (or minimum dosage), which we will extract from cyIMPACT. Sites know that accurate data in cyIMPACT will result in their ability to accurately report their AmeriCorps results.

Of the 128 school-based teams with after-school programming, all but eight (one each from eight sites) provided information about their goals and their numbers in their after-school programs, for a total response rate of 94%. (Table 14.) For the 13 sites with Starfish Corps or After School Heroes, eight met or exceeded their goals, three additional sites reached 75% of their target, one reached 37%, and one, none. For homework help, of the 17 sites for whom this activity was applicable, only four reached their goals. An additional two sites met 75% or more of their target numbers, two sites met 50% to 74%, two sites met 25% to 49%, two sites met 10% to 24%, and five sets did not record any students as receiving homework help. Finally, for the 17 sites with after-school programs other than Starfish or After School Heroes, 12 met or exceeded their goals and the remainder reached at least 75% of their target number. Thus, overall sites and teams did best in projecting and meeting their target numbers for general after-school programs, and least well in projecting and meeting numbers for homework help.


Table 14 . After-School and Starfish Corps Goals-to-Actuals by Site Site

# of Teams After School/# Teams Reporting

Goals to Actuals, After-School

Goals to Actuals, Homework Help

Goals to Actuals, Starfish Corps or After School Heroes

Boston 10/10 144% 33% 81% Chicago 12/12 139% 10% 128% Cleveland 5/4 202% 123% 254% Columbia 3/2 97% 57% N/A Columbus 3/2 80% 30% 37% Detroit 3/2 99% 0% 78% Little Rock 2/2 120% 120% 117% Los Angeles 12/12 126% 0% 106% Louisiana** 4/3 89% N/A 217% Miami 8/7 122% 95% 105% New Hampshire 3/3 118% 55% 0% New York 20/20 138% 0% 92% Philadelphia 22/22 76% 23% N/A Rhode Island 4/3 238% 0% N/A San Antonio 4/3 161% 77% N/A San Jose/Silicon 4/4 214% 0% 441% Seattle/King Cty. 2/2 284% 164% 112% Washington DC 7/7 N/A 373% N/A C. Corps Member Preparation and Support Evaluation Question: Preparation. How well-trained, prepared, and supported are corps members for their different roles (e.g., literacy training, attendance improvement) in WSWC schools at the different levels? What aspects of training and support are most related to corps members feeling prepared at the different levels? Teacher/Service Partner Assessment of CM/Team Preparation Teachers, service partners, and corps members were all asked to rate their agreement on the statement: “Corps members are well-prepared for the academic work they do with my students [teachers]/in our school/[service partners] program.” (Table 15.) Both teachers and service partners generally consider their corps members to be well prepared for the academic work they do, with little difference between mid-year and end-of-year or between service partners and teachers. Teachers serving grades 6-12 showed a slight drop between mid-year and end-of-year. Means were slightly higher than in FY09.


Table 15. Teacher and Service Partner Assessment of Corps Member/Team Academic Preparation, Mid-Year and End-of Year, by Grade Level*

Elementary Middle/High

Mid-year End-of-Year Mid-Year End-of-Year

N Mean (SD)

N Mean (SD)

N Mean (SD)

N Mean (SD)

Teachers 608 4.28 (0.79)

575 4.21 (0.94)

465 4.26 (0.79)

417 4.12 (0.86)

Service Partners 164 4.33 (0.64)

160 4.31 (0.75)

115 4.31 (0.79)

112 4.28 (0.67)

*based on one item, with a 1-5 “strongly disagree” to “strongly agree” scale. Corps Member Assessment of Preparation and Training Corps members were asked to rate specific aspects of their preparation. (Table 16.) They felt best about their preparation for after school and homework assistance. Both corps members working with the elementary grades and those working with grades 6 or higher felt well prepared for whole classroom academic support. However, the elementary group felt better prepared than did the middle/high group for literacy tutoring, but the middle/high corps members felt better prepared for math tutoring. Corps members serving grades 6 or higher were somewhat less comfortable than those working with grades K-5 regarding whole class or homeroom behavior support, with only half of the former group feeling “quite” or “very” prepared. Corps members serving grades 6 and higher felt least prepared for attendance coaching, with just over one-third feeling “quite” or “very” prepared at mid-year. They reported being only slightly more prepared for report card conferencing.

Table 16. Corps Member Assessment of Their Preparation, by Grade Level Grades K-5 Grades 6-12 Area of preparation N Mean (SD) % very or

quite prepared

N Mean* % very or quite

prepared One-on-one/small group tutoring in literacy (tier2) 620 4.01 (0.88) 77.0% 538 3.84 (0.98) 68.6%

One-on-one/small group tutoring in math (tier 2) 497 3.53 (1.14) 57.1% 541 3.70 (1.13) 63.6%

Whole classroom academic support 533 3.95 (0.99) 73.1% 552 3.89 (1.03) 69.3%

Whole class and/or homeroom behavior support 535 3.75 (1.01) 62.8% 547 3.45 (1.10) 50.3%

Report card conferencing n/a n/a n/a 442 3.14 (1.29) 43.2% Attendance coaching (tier 2) n/a n/a n/a 466 3.05 (1.19) 36.5% Formal mentoring or behavior coaching 488 3.73 (1.03) 63.7% 528 3.70 (1.07) 65.0%

After school 642 4.12 (0.81) 81.5% 594 4.01 (0.90) 76.3% Homework assistance 639 4.23 (0.81) 85.3% 598 4.16 (0.88) 79.7% Corps members were also asked about the different kinds of training they received for their school-based service. (Table 17.) Corps members working at both overall grade levels reported that they most often received training related to school-based service from their sites, with more than three-fifths reporting receiving such training two or more times by the mid-year. About half said that they had not received any trainings from their school or district, nor been observed and received related


coaching by teachers or other school staff. However, most corps members had been observed and received related coaching from their SCM or PM at least once, with the majority receiving this kind of help at least two times by the mid-year. Finally, more than three-fifths of corps members reported reviewing student performance data with teachers or other school staff at least once by mid-year, with about two-fifths having done so two or more times.

Table 17. Frequency of Different Kinds of Training Related to School-Based Service Grades K-5 % (n) Grades 6-12 % (n) Type of Training Never

Once 2 or more

times Never Once 2 or more

times Training or professional development sessions by school or district

50.6% (333)

21.3% (140)

28.1% (185)

53.2% (328)

19.8% (122)

27.0% (167)

Training related to school-based service offered by CY site

17.2% (113)

17.8% (117)

65.0% (428)

22.6% (139)

16.1% (99)

61.3% (377)

Observation and related coaching by SCM or PM

15.7% (102)

21.5% (140)

62.8% (409)

22.8% (140)

22.7% (139)

54.5% (334)

Observation and related coaching by teachers or other school staff

45.9% (300)

24.3% (159)

29.7% (194)

51.1% (315)

25.1% (155)

23.8% (147)

Review of student performance data with teacher or other school staff

37.2% (243)

23.6% (154)

39.2% (256)

37.8% (233)

24.5% (151)

37.8% (233)

Corps Member Assessment of Support from Their Teacher(s) At the mid-year, corps members rated their agreement with five items related to support from their teachers, including “has established clear expectations with me for my work in his/her classroom,” “meets regularly with me to review my students’ performance,” and “gives me increasingly greater responsibilities.” The mean response for both grade level groups was somewhat neutral, but corps members working at both these grade levels somewhat agreed by the end of the year that their teachers had integrated them into their classrooms. (Table 18.)

Table 18. Corps Member Assessment of Support from Schools and City Year, Mid-Year and End-of-Year

Grades K-5 Grades 6-12 Time of year N Mean* SD N Mean* SD Mid-Year 533 3.14 0.92 487 2.98 0.85 End-of-Year: Teacher has integrated me into classroom

619 3.70 1.18 582 3.64 1.13

*All items used a 1-5 “strongly disagree” to “strongly agree” scale. Cronbach’s alpha for the mid-year scale of five items: .87 Teacher involvement with City Year At mid-year, teachers were asked to respond to four items about their involvement with corps members. At the end-of-year, they responded to two items. (Table 19.) K-5 teachers were slightly more involved than Grade 6-12 teachers at mid-year, but the difference is negligible at end-of-year.


Table 19. Teacher Involvement with City Year, Mid-Year and End-of-Year, by Grade Level Mid-Year* End-of-Year** Source of Data N Mean SD N Mean SD K-5 Teachers 587 3.65 0.91 545 3.10 1.26 Gr 6+ Teachers 458 3.46 0.84 416 3.03 1.17 *Teacher Mid-Year items used a 1-5 “strongly disagree” to “strongly agree” scale. Mean is based on four items from the Mid-Year Survey, including “My corps member(s) and I have established clear expectations for their work with my classroom,” and “My corps member(s) and I meet regularly to review their performance.” Cronbach’s alpha: for K-5: .90; for grades 6-12: .87 **Teacher End-of-Year uses the same scale but is based on only two items related to meeting with corps members regularly. D. Corps Member and Team Performance Evaluation Question: Performance. What is the quality of corps member/team performance as assessed by different stakeholders at the different grade levels? Planning and Communications with School At the mid-year, service partners were asked nine items about their understanding of City Year’s work, the adequacy of the planning process with City Year, and the strength of City Year’s communications with them on different aspects of their work. Using a 1-5 scale (strongly disagree to strongly agree), 157 service partners from Grade K-5 schools rated their overall agreement on the nine items as 4.39 (SD 0.54) and 111 service partners from Grade 6-12 schools rated their agreement as 4.29 (SD 0.66). Overall Corps Member/Team Performance At the mid-year both teachers and service partners were asked a number of items relating to the quality of their corps members’/teams’ performance. Service partners were asked the same questions at the end of the year as well. (Table 20.) Both teachers and service partners agreed that their corps members/teams were performing well overall. Service partners in schools serving middle and high school grades had slightly more favorable views of corps member performance than did their teachers.

Table 20. Teacher and Service Partner Mean Ratings of Corps Member/Team Performance, Mid-Year and End-of-Year*


Mid-year End-of-Year Mid-Year End-of-Year

N Mean (SD)

N Mean (SD)

N Mean (SD)

N Mean (SD)

Teachers** 582 4.41 (0.65)

n/a n/a 439 4.33 (0.64)

n/a n/a

Service Partners*** 155 4.50 (0.49)

154 4.42 (0.65)

111 4.58 (0.49)

107 4.57 (0.48)

*All items used a 1-5 “strongly disagree” to “strongly agree” scale. **Grade K=5 teacher means are based on ten items, including “My corps members handle student behavior management issues well,” “work well with my students,” and “Set high expectations for students.” Cronbach’s alpha: .96. Grade 6-12 teacher means are based on the same ten items plus an additional item, “My corps members have the emotional maturity to work with students close to their own age.” Cronbach’s alpha: .96 ***Principal means are based on nine items, of which six overlap with the teacher survey. Items include “Corps members conduct themselves professionally and appropriately,” “work well with our students,” and “convey through their words and actions that it is possible to make a difference.” Cronbach’s alpha for elementary SPs: .93; for middle/high: .94


Team Leader Performance Teacher and service partner assessments of team leader communications with them were generally favorable, with service partner assessments somewhat higher than teachers at both grade levels. In addition, we asked service partners for their agreement with a statement about how well the team leader managed the team. Agreement level was similar to that of the communications items (mean of 4.54, SD 0.66 for the elementary level, and mean of 4.68, SD 0.59, for the middle and high school levels). (Table 21.)

Table 21. Teacher and Service Partner Assessment of Team Leader Communications, Mid-Year, by Grade Level


N Mean* SD N Mean* SD Teachers 612 3.98 0.85 461 3.84 0.93 Service Partners 165 4.71 0.48 116 4.68 0.58 *All items used a 1-5 “strongly disagree” to “strongly agree” scale. Cronbach’s alpha for K-5 teachers: .85, for Grade 6-12 teachers: .89; for elementary SPs: .85; for middle/high SPs: .88 Satisfaction with City Year Both teachers and service partners were asked about their satisfaction with the quality of service provided by corps members/teams and the overall experience of working with City Year. (Table 22.) Teachers were also asked about their satisfaction with the overall impact of City Year on their students. Finally, both groups were asked about the likelihood of recommending City Year to someone else who serves in their position in another school. Overall, both teachers and service partners are very satisfied, with satisfaction by service partner higher than in FY09. About two-thirds of service partners at the elementary school level (65.2%, n=105) and 72.4% (n=84) at the middle and high school levels said they were “very satisfied” with the quality of service provided by their City Year teams. End-of-year teacher “very satisfied” ratings for quality of service provided by their corps members were 57.9% (n=334) for grade K-5 teachers and 59.1% (n=260) for grade 6-12 teachers.

Table 22. Teacher and Service Partner Satisfaction with City Year, End-of-Year, by Grade Level

Elementary Middle/High Teachers Service Partners Teachers Service Partners

N Mean (SD)

N Mean (SD)

N Mean (SD)

N Mean (SD)

The quality of service provided by your City Year corps members/team

577 4.44* (0.80)

161 4.57* (0.67)

440 4.48* (0.75)

116 4.66* (0.60)

The overall experience of having City Year in your school/working with City Year

577 4.53* (0.69)

161 4.62* (0.58)

435 4.50* (0.70)

116 4.77* (0.50)

The overall impact of City Year on your students

578 4.42 (0.75)

n/a n/a 436 4.44 (0.73)

n/a n/a

The likelihood that you would recommend City Year to someone else who serves in your position at another school

567 8.94** (1.69)

157 9.43** (1.25)

432 8.96** (1.64)

115 9.36** (1.20)

*1-5 “strongly dissatisfied” to “strongly satisfied” scale. **1-10, “extremely unlikely” to “extremely likely”


Comparison with Last Year’s Team Service partners were also asked to compare the performance of this year’s team with the City Year team they had in FY09, if applicable. (Table 23.) The majority considered the service provided this year to be better, with more than two-thirds of elementary school service partners and about three-quarters of middle/high school service partners rating it better or a lot better. For elementary schools, only 12 individuals (10.1%) thought that the quality of service provided was worse as did 10 (8.4%), the overall experience of working with the team. For middle/high schools, these numbers were much lower, with only two individuals saying that these aspects were worse.

Table 23. Service Partner Comparison with Last Year’s Team, End-of-Year, by Grade Level Elementary Middle/High Aspects of Satisfaction for Comparison

N Mean (SD)

% better or a lot better

N Mean (SD)

% better or a lot better

The quality of service provided by the team 119 3.98*

(1.07) 68.1% 59 4.22* (0.91) 74.5%

The overall experience of working with City Year 119 4.04*

(1.08) 66.4% 59 4.31* (0.95) 72.9%

*1-5 “a lot worse” to “a lot better” scale. After School Implementation At the mid-year, Grade K-5 service partners were given four items regarding implementation of City Year’s after-school programs. (Table 24.) Service partners agreed that City Year’s after-school programs aligned well with their own student learning goals and that students enjoyed the programs. Answers were more spread out regarding the quality of City Year instruction in the after-school and corps members’ handling of student behavior management issues, but respondents generally agreed that these two aspects were well done. Responses were very similar to those from FY09. Table 24. Grade K-5 Service Partner Ratings of Quality of After-School Implementation, Mid-Year

Performance Aspects N Mean* SD The content of the City Year program aligns well with the student learning goals of our school/program 150 4.21 0.71

The quality of the instruction is high. 146 3.95 0.76 City Year corps members handle student behavior management issues well. 148 3.84 0.89 Students enjoy participating in the program 150 4.44 0.65 *All items used a 1-5 “strongly disagree” to “strongly agree” scale. E. Perceived Impacts of City Year Evaluation Question: Perceived impacts. What is the extent of perceived impacts/added value of City Year’s work at the student, classroom, and school levels at the different grade levels? Perceived School Climate Impacts CY’s school climate scale includes such items as, “Inside, our school has a physically welcoming appearance,” “Adults in our school help students to examine the relationship between actions and consequences, “Many parents and community members participate in our school (e.g., come to events, volunteer), and “Our school regularly creates ways to promote pride in the school community.” Because we do not expect ratings to change much in a one year period, we asked


various stakeholder groups to assess the impact of City Year on selected aspects of school climate at the end of the year. The teacher scale for both grade levels included six items (down from 10 items in last year’s teacher survey); the elementary service partner scale, 13 items; and the middle and high school service partner scale, 14 items. Thus, the teacher and service partner scales are not comparable. Elementary service partners gave their schools the highest overall school climate ratings, and grade 6-12 teachers, the lowest. Both groups generally agreed that their overall school climate was positive. End-of-year assessments of City Year impact were similar across groups, with the mean at “moderate amount of impact.” (Table 25.) On a sub-group of four items related directly to the impact of City Year on student behavior, the mean for K-5 service partners was 3.00 (SD 0.67), and for Middle/High school service partners, 2.96 (SD 0.71), slightly lower than the overall mean for City Year impact on school climate.

Table 25. Teacher and Service Partner Assessment of School Climate at Mid-Year and Perceived Impact of City Year on School Climate at End-of-Year, by Grade Level

Assessment of School Climate

(Mid Year) CY Impact on School Climate

(End of Year) Stakeholder Group N Mean* SD N Mean** SD Grade K-5 Teachers 546 3.74 0.63 559 3.17 0.64 Elementary Service Partners 136 3.97 0.55 145 3.10 0.59 Grade 6+ Teachers 413 3.43 0.67 404 3.19 0.64 Middle/High Service Partners 102 3.75 0.54 106 3.13 0.58 *All items in the mid-year surveys used a 1-5 “strongly disagree” to “strongly agree” scale. Cronbach’s alpha—for K-5 Teachers: .94; for Grade 6-12 Teachers: .94; for elementary SPs: .94; for middle/high SPs: .91. ** Based on a 4-point scale, from “no impact”=1 to “a great deal of impact”=4. Cronbach’s Alpha—for K-5 teachers: .87; for grade 6-12 teachers: .85; for elementary SPs: .93; for middle/high SPs: .93 Teacher Perceived Impacts on Focus List Students Teachers were asked a set of six questions regarding their perceived impact by their corps members on the academic behaviors, attitudes and performance of the students they tutored consistently one-on-one or in small groups. In addition, they answered one item each on corps member impact on targeted students’ attendance and behavior. (Table 26.) Again, agreement levels at the mid-year and end-of-year were high, as were agreement levels between the two grade levels. Overall, teachers agreed that their corps members did improve the learning behaviors and attitudes as well as the behavior of focus list students. They were slightly less likely to agree that corps members had an impact on attendance.


Table 26. Teacher Perceived Impact of Corps Members on Focus List Students, Mid-Year and End-of-Year, by Grade Level

Grades K-5 Grades 6-12

Mid-year End-of-Year Mid-Year End-of-Year Area of Impact N Mean

(SD) N Mean

(SD) N Mean

(SD) N Mean

(SD) Impact on Student Learning*

596 4.21 (0.68)

541 4.17 (0.73)

422 4.11 (0.63)

385 4.11 (0.67)

Impact on Attendance 598 3.74 (0.95)

544 3.59 (0.98)

426 3.85 (0.87)

399 3.81 (0.94)

Impact on Behavior 600 4.03 (0.87)

547 3.90 (0.95)

428 4.07 (0.79)

407 4.05 (0.86)

*Items included student motivation to learn, active engagement in learning, time spent on learning tasks, confidence in ability to learn, study habits, and academic performance. Cronbach’s alpha for K-5: .96, and for Gr6+, .93. Teacher Perceived Impact on Classrooms and Homerooms All teachers were given six items at the mid-year and end-of-year on which to indicate their level of agreement about corps member impact on their classrooms. (Table 27.) Items included “My corps members have helped foster a positive environment for learning,” “My corps members have helped increase my students’ enjoyment of school,” and “My corps members have improved the overall discipline and order in the classroom.” Mean level of agreement for both grade levels did not change much between mid-year and end-of-year, with an average level of agreement of “4” on a five point agreement scale, suggesting that teachers believed that corps members had an impact on their classrooms. Corps members serving in middle and high schools may also spend time in homerooms. Teachers with corps members in their homerooms on a regular bases answered four items about corps member impact on student attendance, punctuality, behavior (and order in the classroom), and focus/readiness at the start of the day. Mid-year and end-of-year assessments for these items were slightly lower than for impact on classrooms.

Table 27. Teacher Perceived Impact of Corps Members on Classroom, Mid-Year and End-of-Year, by Grade Level

Grades K-5 Grades 6-12 Mid-year End-of-Year Mid-Year End-of-Year

Area of Impact N Mean (SD)

N Mean (SD)

N Mean (SD)

N Mean (SD)

Impact on Classrooms 555 4.14 (0.73)

502 4.11 (0.78)

439 4.06 (0.68)

391 4.19 (0.68)

Impact on Homerooms n/a n/a n/a n/a 324 3.69 (0.81)

243 3.75 (0.80)

* All items used a 1-5 “strongly disagree” to “strongly agree” scale. For K-5, Cronbach’s alpha for classroom: .94, for grades 6-12: .92; for grades 6-12 homeroom: .90. Corps Member Assessment of Their Own Impact on Students At the end-of-year, corps members were asked to rate the extent of their impact on 15 aspects of students’ lives in schools. From these items, we created five informal scales (not through rigorous factor analysis but by grouping similar items according to our Theory of Change): overall learning (including study habits, performance, engagement, and confidence), reading (enjoyment, ability,


confidence), math (ability and confidence), behavior (student respect for each other, behavior and order in classrooms, and student conflicts), and school connectedness (enjoyment of school, pride in school). One item was on its own: student attendance. We also computed an overall impact score although it should be noted that if corps members were not responsible for implementing a particular activity, they probably answered “not at all” in regard to extent of impact. For grades K-5, corps members reported the greatest impact on student reading, with a mean of over 3.0 (on a 4-point impact scale), followed by student learning. Attendance, which is not an elementary school initiative, had the lowest rated impact. For grades 6 to 12, corps members rated overall student learning (including study habits) higher, with a mean of 2.77, followed by math (2.69). The range of means on these various scales was smaller for those serving grades 6-12 than for those serving grades K-5. In terms of individual items that went into the scales, those serving grades K-5 were most likely to report increasing student confidence in their reading ability (mean of 3.17), and those serving grades 6 and higher were most likely to report having an impact on student’s academic performance (2.98). (Table 28.)

Table 28. Corps Member Assessment of Their Impact on Students

Grades K-5 Grades 6-12 Area of Impact N Mean* SD N Mean* SD Student Learning (Alpha: .84) 613 2.89 0.64 584 2.77 0.68 Reading (Alpha: .88) 619 3.09 0.72 583 2.51 0.83 Math (Alpha: .96) 614 2.47 0.97 585 2.69 0.97 Behavior (Alpha: .83) 611 2.71 0.75 580 2.66 0.75 School Connectedness (Alpha: .70) 614 2.78 0.76 578 2.60 0.83

Attendance (1 item) 626 2.31 0.92 591 2.46 0.89 Overall impact (Alpha: .92) 577 2.78 0.59 550 2.64 0.64 *All items used a 1-4 scale: “not at all,” “a little,” “somewhat,” and “a great deal.” Impacts on Teachers At the mid-year and end-of-year, teachers were asked to rate their agreement on five items related to the impact of their corps members on their own work and relationships with students. (Table 29.) Teachers generally agreed that corps members had had an impact on such areas as helping them to “Feel supported in their work,” allowing them to “improve the quality of the contact” they had with students,” and giving them “more time for planning.” Mid-year and end-of-year assessments were similar as were assessments by teachers at both grade levels. Overall agreement on this scale was slightly less than for perceived impact on classrooms.

Table 29. Perceived Impacts of Corps Members on Teachers, Mid-Year and End-of-Year, by Grade Level

Mid-Year End-of-Year Stakeholder Group N Mean* SD N Mean** SD Grade K-5 Teachers 599 3.91 0.84 559 3.82 0.90 Grade 6-12 Teachers** 455 3.89 0.76 404 3.93 0.74 * All items used a 1-5 “strongly disagree” to “strongly agree” scale. Cronbach’s alpha for K-5:.90, for Grade 6-12:.86. **At the mid-year, Grade 6-12 teachers had four items. An additional item on helping the teacher to differentiate the instruction was asked, but appeared in a different section, with a “not applicable” option. At the end-of-year it was included in the teacher impact scale.


AfterSchool Impacts Service partners rated City Year’s impact on their after-school programs. (Table 30.) Four items concerned improvement related to learning (active engagement in learning, time spent on learning tasks, fostering a positive environment for learning, and strengthening academic performance); at the elementary level, five items concerned City Year impact on student behavior, social-emotional learning, and service/social issues; the middle and high school levels included the same five items plus “helped to develop youth as leaders.” Principals were generally enthusiastic about City Year’s impact on both areas.

Table 30. Service Partner Perceived Impacts on After-School Student Outcomes,

End-of-Year, by Grade Level Elementary Middle/High City Year Impacts* N Mean* SD N Mean** SD Improvement related to learning** 142 4.22 0.65 103 4.26 0.60

Other afterschool impact*** 142 4.11 0.64 102 4.16 0.66 *All items used a 1-5 “strongly disagree” to “strongly agree” scale. **Cronbach’s alpha—for elementary: .93; for middle/high: .91. ***Cronbach’s alpha—for elementary: .92; for middle/high: .92 F. Student Level Outcomes Evaluation Question: S udent level outcomes. What student level outcomes are being realized (i.e., literacy for Grades 3-5, attendance for 6-9, the “C’s” for Grades 3-5 and 6-8)? How do outcomes different according to dosage/exposure to City Year? What is the added value of after- school for grades 3-5?

t

Elementary Literacy Data Twelve of 14 sites offering literacy tutoring to elementary school students submitted literacy data. Data from 10 of the 14 sites, representing 10 different assessments and 1691 grade 1-5 students who received tutoring for at least two time periods (e.g., start-of-year and end-of-year) were analyzed and included in the network results. Of the two sites’ data not included, one had data that were hard to interpret (New York), and for the other it was unclear which students received City Year tutoring for at least two time periods (Detroit). A total of 47 teams were represented in the overall results. DIBELS was the most common assessment used (n=3). City Year analyzed these data in three different ways: 1) The percentage of students demonstrating raw score improvement from start-of-year to end-of- year 2) Whether the site had met its original literacy goal in terms of percent improved—80% of students will demonstrate improved performance on literacy assessments from start-of-year to end- of-year 3) The percentage of students who improved at least one level (i.e., what percentage of students either moved from off-track to sliding or on-track or moved from sliding to on-track) according to the established benchmarks of each assessment administered. Results indicate that overall, of the students who received literacy tutoring for at least two time periods, 90% (1515 out of 1691) demonstrated improvement on literacy assessments from start of


year to end of year. Site success rates varied from 44% (Cleveland) to 100% (San Jose/Silicon Valley), with eight of the 11 sites achieving a success rate of 90% or greater, and these same sites all achieving their goal for the year. The percentage of students moving up at least one level were somewhat lower, ranging from 8% (Seattle) to 58% (San Jose/Silicon Valley), with an overall average of 25% (331 out of 1313). (Table 31.)

Table 31. Literacy Results by Site: Percent Improved, Goal Met, Percent Positively Changing Tracks

Site* # teams Grades N of

students % (n)

Improved Goal Met

(Y/N) % (n)

Improved At Least 1 Level

Chicago 9 1-5 344 97% (332) Y 13% (40/312)Cleveland 4 3-5 78 44% (34) N 13% (7/56) Columbia 2 1-3 82 98% (80) Y 19% (11/58) Columbus 3 1-4 33 91% (30) Y 52% (15/29) Little Rock 2 1-5 114 96% (110) Y 14% (15/105)Los Angeles 8 1-5 282 90% (253) Y 13% (25/190)Miami 7 3-5 254 78% (198) N 39% (59/151)San Jose/SV 4 1-5 104 100% (104) Y 58% (56/97) Seattle/KC 1 1-5 39 79% (31) N 8% (3/39) WashingtonDC 7 1-5 361 95% (343) Y 36%

(100/276) Total** 47 1-5 1691 90% (1515) Y 25%

(331/1313) *Only sites serving in elementary school settings and conducting literacy tutoring support are included in this table. Grade 6 to 9 Attendance Data Six of eight sites offering attendance support for grades 6 to 8 provided start-of-year and end-of-year average daily attendance (ADA) data for a total of 24 teams. (Cleveland and New Hampshire did not provide data.) (Table 32.) Of the 1605 students for whom data were collected, 79% (n=1274) were already “on-track” with regards to attendance at the start-of-year (i.e., greater than 90% ADA), leaving only 14% (n=219) of the total sample who were “sliding” (i.e., less than 90% ADA) and 7% (n=112) who were “off-track (i.e., less than 80% ADA). Thus, improvements were calculated only for these 21% who were sliding or off-track. The overall goal for this intervention for Diplomas Now schools was for 50% or greater of off-track students to move up at least one level, but no specific goal was set in this pilot year for other WSWC middle school level students. Overall, the network achieved a success rate of 54% of off-track students improving at least one level/no longer off-track Three of the six sites achieved this goal, with Philadelphia having the greatest success at 92% (n=22).


Table 32. Attendance Results by Site: Changes from Off-Track and Sliding at Start-of-Year* Site #

teams Grades Total N N off

track at start-of-year (SOY)

% (n) off-track at SOY, no longer off-track**

Goal Met (Y/N)

N sliding or off-track at SOY

% (n) sliding/off track at SOY, on track at EOY***

Louisiana 2 6-8 111 5 80% (4) Y 23 57% (13) New York 6 6-8 608 67 40% (27) N 162 49% (80) Philadelphia 9 6-8 322 24 92% (22) Y 64 81% (52) Rhode Island 3 6-8 365 13 46% (6) N 59 32% (19) San Antonio 3 6-8 26 0 n/a n/a 3 67% (2) San Jose/SV 1 6-8 173 3 33% (1) N 20 75% (15) Total 24 6-8 1605 112 54% (60) Y 331 55% (181) *Only students identified as receiving support SOY to EOY and/or students that had complete datasets at both time points were included in this table. Not all sites provided enrollment and exit dates on their trackers. **The goal was 50% of off-track students will achieve an 80% or higher average daily attendance. ***No longer off-track is <80% ADA. Sliding is 80-89% ADA. Student Survey, Grades 3-5 In FY10, as in FY09, we collected individual student survey data, both near the beginning of the school year and near the end, from a large sample of students served by City Year in grades 3 to 5. Previously, we established that there were three factors with high reliability (Cronbach’s alpha) represented by the statements in the survey: Learning Engagement (.82), Connectedness to School (.92), and Connectedness to City Year, including perceived impact of City Year (.92). We were most interested in examining the relationship to these factors of exposure to City Year through corps member presence in classrooms helping students, personal help from City Year with in-class work, personal help from corps members with their homework, and attendance at a City Year after-school program. There were high correlations between the first two kinds of exposure (.468, p<.001) and between the last two (.599, p<.001), but these two groups have very low or even negative correlation with each other. Thus, we did not create an overall exposure measure for each student as we did last year when we used different measures. The matched sample of pre- and post-surveys consisted of 1146 students (30.9% of the pre-survey sample) representing 16 City Year sites. Fifty-three percent (53.5%, 613) of these were girls, and 46.5% (533) were boys. Grade distributions were as follows: 3rd grade, 29.9% (343); 4th grade, 33.9% (379); and 5th grade 37% (424). Ratings between the pre-survey and post-survey declined for all three factors as they did in FY09. However, it should be noted that overall means were generally very high (over 4 for all factors on the pre-survey). Post ratings for Connectedness to City Year factor were higher than for the other two factors, and ratings for Learning Engagement were higher than for Connectedness to School. Declines were significant (p<.001) for all three factors. In terms of gender, differences between pre and post ratings were significant only for City Year Connectedness, with girls declining significantly more than boys (p=.04). Differences between pre and post ratings were not significant by grade level. (Table 33.)


Table 33. Grade 3-5 Means of Learning Engagement, Connectedness to School, and Connectedness to City Year by Gender and Grade

Learning Engagement

Connectedness to School Connectedness to City

Year Variable

Pre-rating Post-rating Pre-rating Post-rating Pre-rating Post-rating

Gender Boys 4.02 3.92 4.08 3.86 4.54 4.42 Girls 4.23 4.10 4.19 3.90 4.68 4.50 Grade Third 4.17 4.07 4.18 3.99 4.70 4.59 Fourth 4.16 4.01 4.14 3.84 4.66 4.52 Fifth 4.09 3.99 4.10 3.85 4.52 4.31 Overall 4.14 4.02 4.14 3.89 4.62 4.46 Exposure to City Year More than 70% of students reported that City Year spent time in their classes helping students at least 3-4 times a week. More than half (57%) reported attending a City Year after-school program at least 1-2 days a week, with 42% saying that they attended 3-4 times a week. A greater percentage of students indicated that City Year helped them with their in-class work (77%) than with their homework (61%). (Table 34.)

Table 34. Exposure to City Year, Grade 3-5 Student Survey Respondents Area of Exposure to City Year

N (1146)

Not very often/not at

all % (n)

1 to 2 times a week % (n)

3 to 4 times a week % (n)

How often does City Year spend time in your class helping kids?

1103 13.4% (148) 14.7% (162) 71.9% (793)

How often does City Year help you with your in-class work?

1097 22.7% (249) 25.7% (282) 51.6% (566)

How often does City Year help you with your homework?

1098 39.3% (431) 21.5% (236) 39.3% (431)

How often do you attend a City Year after school program?

1102 42.6% (470) 15.2% (168) 42.1% (464)

*based on 3 point scale where 1 is “not very often/not at all, 2 is “1-2 times a week” and 3 is “3-4 times a week.” Examining the factors scores by exposure to City Year and controlling for grade, we find that greater exposure to City Year is generally related to more favorable differences between pre and post factor scores. In no case was grade a significant factor. However, exposure does not explain much of the variance. Learning Engagement: Frequency of City Year spending time in class and of City Year helping the respondent in class were each positively related to Learning Engagement (p=.001, and p=.003, respectively). School Connectedness: On its own, only frequency of City Year helping with in-class work was positively related to School Connectedness (p=.049), but the more exposure through help with in-class work plus homework was even more positively related to School Connectedness (p=.008).


City Year Connectedness: Each type of exposure to City Year was positively related to City Year Connectedness (time in class, help with in-class work, help with homework, p<.001, attendance in after school, p=.004)), with the combination of in-class work and homework help especially positive (p<.001). Student Survey, Grades 6-8 We also conducted a full administration of our Grade 6-8 version of the survey. In this version, we had established that there were five scales with high reliability: City Year Connectedness, Learning Engagement, School Connectedness, Relationships with Others, School Climate, and City Year Connectedness. In addition, we collected self-report data on amount of exposure to City Year through help in English/Language Arts, help in math, or attendance at a City Year after-school program; self-reported academic performance in English/Language Arts and mathematics; and self-reported suspensions and detentions, awards for academic performance; and awards for good attendance or positive behavior. The matched sample of pre- and post-surveys consisted of 608 students, or 31.4% of the pre-surveys received (n=1936). Fifty-one percent (51%) of these were girls, and 49%, boys. Grade distributions were as follows: 6th grade, 48%; 7th grade, 30%; and 8th, grade 21%. Grade distribution among those completing pre-surveys was more even. As with the Grade 3-5 Survey, overall factor scores declined between pre and post. However, we were more interested in the relationship of exposure to City Year to changes in each of the factor scores as well as to self-reported change in academic performance and number of detentions or suspensions. A summary of these results by factor are presented below City Year Connectedness

• Positive attitudes toward City Year increased with exposure to English/language arts help. [Based on CY English/language arts help means: Never = 28.18, Once in awhile = 30.17, and One or more times a week = 31.31; F(2, 483) = 5.41, p <.01].

• Positive attitudes toward City Year increased with involvement in CY’s after-school program. [Based on CY after-school attendance means: Never = 29.37, Once in awhile or more frequent = 31.17; F(1, 483) = 5.87, p <.02]. But there was a grade by CY after-school interaction, with those not participating showing a decrease in attitudes from 6th to 8th grades; and those students participating in the after-school program showing an increase in attitudes from 6th to 8th grades but these attitudes flattened for 7th and 8th grades [F(2, 483) = 3.95, p <.02] (see Table 1).

• Positive attitudes toward City Year increased with exposure to math help. [Based on math means: Never = 29.23, Once in awhile = 29.89, and One or more times a week = 31.77; F(2,,

486) = 4.57, p <.02]. Learning Engagement

• No differences were found between attitudes toward their studies (Learning Engagement) based on gender, grade, exposure to English/language arts help, math help, or involvement in the after-school program.


School Connectedeness • Students expressed more positive attitudes toward their school (School Connectedness) as

exposure to English/language arts increased [Never = 21.42, Once in awhile = 21.68, and One or more times a week = 22.89; F(2, 515) = 4.56, p <.02].

• No differences were found between these attitudes by gender, grade, exposure to the after-school program, or help with math.

Relationships with Others

• No differences between attitudes toward Relationships with Others were found based on exposure to English/language arts help, math help, or involvement in the after-school program.

• But a grade by after-school interaction was found suggesting that positive Relationships with Others were evident for 7th graders who participated in the after-school program as compared to 7th graders who did not. [F(2, 478) = 4.31, p < .02]. This relationship was not evident for 6th or 8th graders.

• A gender by grade interaction was found. That is, Relationships with Others tended to increase for boys over 6th through 8th grades, whereas Relationships with Others for girls tended to remain about the same across these grade levels [F(2, 478) = 5.51, p <.01].

School Climate • Positive school climate attitudes were associated with more English/language arts help [Never = 15.35, Once in awhile = 15.49, and One or more times a week = 16.51; F(2, 494) = 3.77, p <.03]. • No differences between School Climate attitudes and grade level were found. • No overall differences between School Climate attitudes and math help were found. But a

gender by math help interaction was found. For boys, School Climate attitudes were most positive when they had had moderate help in math (i.e., Once in awhile); whereas girls reported the most positive attitudes toward School Climate when their help in math was one or more times a week [F(2, 496) = 4.44, p <.02].

• No differences in School Climate attitudes were related to exposure to the after-school program but a gender by after-school program interaction was found. That is, boys expressed a more positive attitude toward their school overall (School Climate) when they had had at least some exposure or participation in the after-school program. For girls, change in School Climate attitudes was not evident, based on participation in the City Year after-school program. [F(1, 496) = 5.63, p <.02].

Self-reported Academic Performance

• More students indicated they were doing “fine” in English/language arts when they had also received CY in-classroom help in this subject one or more times a week; and, students reported less often that they were doing “not so good/very poorly” in English language arts when they also received CY in-classroom help this frequently [χ2 (4, 591) = 16.78, p < .002].

• No difference was found for self-reported math performance and math help. • After-school participation was not related to self-reported math performance.


Self-reported Detentions and Suspensions • Exposure to English/language arts help, math help, or participation in the after-school

program was not related to the number of detentions and suspensions that students reported during the school year.

Self-reported Awards for Positive Behavior/Attendance or Academic Performance

• Exposure to English/language arts help, math help, or participation in the after-school program was not related to the number of positive behavior or good attendance awards that students reported that they received during the school year or to awards for academic performance.

Student Survey, Grades 9 (Pilot post data, only) For the Grade 9 Student Survey, we explored some of the simple relationships among exposure to City Year; gender; primary language; self-reported grades in English and mathematics; behaviors (i.e., suspensions, detentions, removal from class); awards for academic performance, attendance or behavior; assessed helpfulness of City Year; and the various factor scales. (Most of the factors were the same as for the Grade 6-8 version, though based on slightly different items. However, we created a separate City Year Helpfulness scale, which used different answer options than the other factors.) Caution should be taken when reviewing these results because of the small sample size (n=170 usable surveys from seven schools at two CY sites) and the lack of a pre-survey. In addition, no causality is implied.

• In general, there were few differences by gender in terms of the scaled items. Boys reported receiving more help from City Year than did girls (p=.01) and also skipping school or classes or being asked to leave class more frequently (p=.015.) Mean grades for both English/Language Arts and Math were 2.80 (about a B-), with no significant gender differences.

• Students whose primary language at home was not English rated School Climate higher than those

whose primary language was English (p=.035).

• Extent of help received by City Year was significantly positively correlated with assessment of helpfulness of City Year (.657, p<.001). Extent of help received was also positively correlated with a student’s frequency of skipping school or classes or being asked to leave class more frequently (.198, p=.016). This result may be due to the fact that City Year is helping more students whose behavior is already negative.

• Assessment of helpfulness of City Year was positively correlated with Learning Engagement (.351,

p<.001), School Connectedness (.216, p=009), Relationships with Others (.258, p=.001), and School Climate (.284, p=.001). However, it is difficult to draw conclusions from these correlations in this one survey, as they may be related to students having an overall positive relationship with school.

• Grades in English/Language Arts and Mathematics were significantly positively related to

Learning Engagement (.350 and .312 respectively, p<.001), to making honor roll (or equivalent) (.320 and .414 respectively, p<.001) and to recognition of positive behavior or attendance (.228, p=.004 and .344, p<.001 respectively).


• Making honor roll (or receiving an award for grades) and receiving recognition for positive behavior or attendance were highly correlated (.586, p<.001).

• Receiving recognition for positive behavior or attendance was positively correlated with School

Connectedness (.170, p=.037).

• Skipping school or classes or being asked to leave class is negatively correlated with extent of positive Relationships with Others (-.172, p=.03), but unpredictably, positively correlated with making honor roll or receiving an award for grades (p=.189, p=.017). Skipping school, etc. is not related to self-report of receiving good grades. These relationships may be a possible indication of improvement in grades among those with negative behavior whom City Year has assisted, but it is too early to tell.

VI. Sharing and Using Data and Findings A. Overview The WSWC Evaluation Plan calls for teams and sites to be engaged with data in an ongoing way. Because a key goal of this project is learning, City Year regularly built in opportunities to discuss and interpret data at the team, site, and Headquarters levels. These opportunities took place during normally scheduled times, such as Service Director conference calls and City Year trainings as well as at specially scheduled events. Survey data analyzed by site were distributed to all sites in a timely manner. In addition, key results from various instruments were distributed in user friendly slide format for use with internal or external audiences. Two formal lengthy sessions, one held after the mid-year and one after the end-of-year data were processed, allowed Headquarters staff (the ProServe team) to grapple with the meaning of the quantitative results from WSWC evaluation initiatives. Overall, evaluation is being used in more sophisticated ways at the local and national levels. In particular, the evaluation process has created a dialogue around the need for success measures and organizational goals. In addition to regular reporting to funders, evaluation data help City Year to strengthen its service and corps member recruiting, inform and engage broader audiences in City Year’s work, and provide evidence of effectiveness. Some specific examples include: Training for staff and corps members

• In training corps members about in-class time. • For the literacy leadership session at Academy. • In general, to provide the why behind trainings and documents distributed to sites/corps

members. • In the Mid-Year ProServe Retreat to examine and reflect on student level data in relation to

students being on or off track. • For a data-informed decision making session at academy for all Program Managers and

Team Leaders. • In trainings and retreats with Executive Directors of City Year sites


Supporting other evaluation initiatives at City Year • For the Einhorn behavior initiative, to prepare David Osher and his team for their work on

social-emotional learning and City Year. • For research consultants at RFA so that they could conduct their evaluation work with the

teams they were studying. Disseminating results internally

• One-page displays (as predecessor of progress reports) from teacher, service partner, and cyIMPACT data for use at a City Year Program Director/Service Director conference as well as a LEAD Program Manager conference.

• To provide the structure for the monthly progress reports developed for use in FY11. Setting targets and creating common expectations for service

• To set for the first time as an organization meaningful national data targets around dosage for FY11 using FY10 results.

• To develop success measures embedded in the site Memos of Understanding (MOU) and tying together practice and evaluation.

• In the behavior initiative, expectations for FY11 regarding preparation in the MOUs. • By Regional Directors to set targets for their regions for the year.

Enhancing service at the site level

• For conversations with service partners about creating an appropriate list of focus group students for literacy.

• To help corps members see where they can make improvements (e.g., in Los Angeles, where literacy data were used in this way and also worked into trainings).

Sharing with experts and other audiences

• In preparation for the thought leader meeting in literacy. • For presentations to other evaluators at the annual meeting of the American Evaluation

Association. • To develop new collateral for sharing at the Summit meeting (Jim Balfanz and Michael

Brown used these data in their talks to focus the audience on City Year impact). • To share City Year experiences in student data level collection with the Corporation for

National Service as they considered piloting common performance measures. Understanding the impact of different kinds of corps members

• By the recruitment department to understand why a particular target around numbers of corps members whose highest education level is high school was not met—1) how do prospective corps members learn about City Year, and 2) how corps members at different education levels may or may not be achieving desired results with students.

• To understand more about the impact of corps members who were born and raised in the area/city in which they are serving as a corps member as compared to corps members who were not.


Enhancing reporting and applications to funders • To boost reporting to AmeriCorps. • In the i3 application for the Diplomas Now collaboration, showing the required level of

evidence. VII. Summary of Findings, Discussion, and Recommendations A. What We Learned about WSWC Overview In this section we first summarize and briefly comment on findings related to the major student level outcomes: performance in literacy/English/language arts and mathematics, attendance, and behavior. Findings will be organized around the key question areas we posed relating to implementation and impact of WSWC: metrics, interventions and participation, corps member preparation, perceived impacts, and student level outcomes. Finally, we will present other pertinent findings for WSWC. Course Performance in Literacy/English/Language Arts and Mathematics Metrics

• During the 2009/2010 school year, all WSWC elementary and middle/high school teams were required to submit student level data three times a year (fall, winter and spring) to City Year headquarters. Twelve of 14 sites submitted EOY data. Data from ten different kinds of assessment were analyzed for almost 1700 students.

• City Year is working to address several challenges related to data collection in this area. 1) establishment of a School Operations group to work with Evaluation and Impact team members to monitor issues in data collection, 2) resources to support sites when they encounter FERPA issues, 3) identification of benchmarks for some assessments, 4) linking output and outcome data through cyIMPACT-generated ID numbers for all Tier 2 students, 5) better training for sites around data entry (and the need for accuracy) in cyIMPACT.

Interventions/participation

• 88% of corps members serving grades K-5 and three-quarters of those serving grades 6-12 reported that they were involved in one-on-one or small group tutoring in literacy/English/LA (ELA). Among those serving grades 3-5, that figure is slightly lower: 84.5%.

• Total literacy enrollment was 10,241, representing two-thirds of all students served by City Year in school.

• 46% of corps members serving grades K-5 reported conducting tier 2 tutoring in mathematics to students as did two-thirds of those serving grades 6-12.

• Teacher perception of corps member tutoring in ELA and mathematics was similar to what corps members reported for grades K-5.

• Among English teachers in grades 6-12, 87% said that corps members conducted one-on-one or small group tutoring in ELA; among math teachers at these grade levels, 93% reported that corps members conducted one-on-one or small group tutoring with their students.


Preparation • Corps members generally felt well prepared for their one-on-one and small group work in

literacy, with 77% of those serving grades K-5 feeling very or quite prepared and 69% of those serving grades 6-12. In fact, corps members felt more prepared for literacy work at the K-5 level than for any other aspect of their school-based work except for after-school and homework assistance.

• Two-thirds (67%) of corps members who reported conducting tier 2 tutoring in mathematics in grades K-5 felt at least “quite prepared,” but less prepared than they did for most of their other areas of work with students. A slightly higher percentage of those doing tier 2 math tutoring in grades 6-9 (73%) felt at least “quite prepared.”

• Teachers at both the grade K-5 and 6-12 levels believed that corps members are well prepared for the academic work they do with students (mean of 4.21 and 4.12 respectively, on a 5-point Likert agreement scale, where 5 is “strongly agree”).

• Most corps members had received observation and related coaching from their Senior Corps Member of Program Manager at least once by the mid-year, with the majority receiving observation and coaching more than once.

• Review of student-based performance data with teachers or other school staff happens as often at both grade levels, with at least three-fifths of corps members reporting that this happened at least once by the mid-year.

Perceived impacts

• At the end-of-year, teachers rated corps member impact on focus list student learning as 4.17 for grades K-5, and 4.11 for grades 6-12 (on a 5-point Likert agreement scale, where 5 is “strongly agree”).

• From self-reported impact of 15 aspects of students’ lives in schools, we created seven subscales. Corps members working with students in grades K-5 rated their impact on student reading as the highest: 3.09 on a 4-point impact scale, where 4 is a “great deal of impact,” followed by student learning overall (2.89). Those serving grades 6-9 rated their overall impact on student learning as highest (2.77).

• Corps members serving those in grades K-5 rated their impact on mathematics learning as somewhat lower (2.47). However, those serving grades 6-12 gave their impact on mathematics (2.69) their second highest rating, right after student learning overall and somewhat higher than their self-reported impact on student reading (2.51).

Student level outcomes

• Ten sites representing 47 teams provided literacy data that met City Year’s standard of offering one-on-one or small group tutoring at least two time periods a week.

• 90% of students tutored at least two time periods a week by corps members improved in literacy over the course of the year, with site success varying from 44% to 100%.

• Eight of the 11 sites achieved a success rate of 90% or greater, thus meeting their goal for the year.

• One-quarter of students moved up at least one track (from off-track to sliding or on-track or from sliding to on-track), ranging from 8% to 58% across sites.


Discussion: English/literacy/Language Arts (ELA). Collecting student data on literacy is still a challenge that City Year HQ is working to help sites address. Literacy data will have even more value when dosage can be connected to outcomes through cyIMPACT as it will be in FY11. Training and accountability for data entry will become that much more important at the site level. Some corps members do still not fully understand that they should be working one-on-one or in small groups for tutoring students, but the numbers who say they do this should increase as the official Memo of Understanding with school partners includes this language. A pleasant surprise this year was the somewhat higher ratings both teachers and corps members gave to corps member preparation for the academic component of their work with students, suggesting that City Year’s training has been more thorough and effective. These ratings may go up even further as City Year’s Manager of Literacy Initiatives and National Literacy Training Manager help sites to obtain local help and create standardized trainings. In addition, observations and related coaching by SCMs or PMs appear to have become somewhat standard practice, reinforced in the future by expectations set at City Year’s Summer Academy. Regarding impact, corps members both see and believe the impact they have on their assigned children’s K-5 reading (not just improved reading but also attitudes). At the grade 6-12 level, corps members rate their impact higher on the more slightly diffuse concept of “student learning.” Given the greater emphasis on other City Year interventions at these grade levels, this finding makes sense for FY10. It is encouraging to see that teachers also believe that City Year is making a difference on focus list student learning. Site success in moving the needle on literacy varies, with most students in most sites showing improvement. However, the larger goal of helping students change tracks (off to sliding, or sliding to on) is less common within this one year period. Following students over time will not be easy but may be important if the ultimate goal of contributing to the reduction of the dropout rate is to be reached. Overall, it is clear that City Year’s increased attention to training, support, and documentation regarding literacy is paying off. Mathematics. A focus on mathematics is relatively new at City Year and has not yet been fully reflected in some of the instruments, especially those for teachers. Thus, we have less information on this aspect of corps member work than on some others. We note that corps members working with elementary students are much less likely to tutor them in mathematics than are those working with grade 6 and above. An encouraging sign is that corps members working with grade 6 and above students are generally feeling prepared for their work in this area. However, a narrow focus on mathematics or ELA is probably not sufficient to promote student success. City Year will need to be purposeful in helping students improve their overall study habits and goal-setting skills so that students can tackle longer-term projects. New funding will create additional math program development in FY11. Student attendance. Metrics

• FY10 was the first year all WSWC middle/high school teams were charged with the task of submitting formal student level attendance data for any students in grades 6-9 who received attendance monitoring and coaching support. Six of eight sites submitted end-of-year data. Data for over 1600 students were analyzed.


• Challenges included: 1) difficulty in collecting previous year attendance data that resulted in selection of inappropriate students for intervention (i.e., students who were on track at the start of year), 2) lack of clarity as to whether data from sites was for focus list students only, and 3) inability to link dosage with student outcomes. Additional support for sites around data collection and the use of cyIMPACT-generated ID numbers for tier 2 students as mentioned in the course performance section will help to address these issues.

Intervention/participation

• Only two-fifths of service partners in middle and high schools were aware that City Year conducted attendance programming.

• An even smaller percentage of corps members in middle and high schools reported doing attendance coaching (32%) even though this is supposed to be a core intervention.

Preparation

• Corps members serving grades 6-12 felt least prepared for their attendance coaching compared to other core interventions they carry out with students, with only 37% feeling “very” or “quite prepared.”

Perceived impact

• Although attendance coaching is not a core City Year intervention at the primary grade level, teachers in grades K-5 generally agreed that City Year had an impact on the attendance of focus list students.

• Teachers in grades 6-12, where attendance coaching is a core intervention, were even more in agreement that City Year had an impact on attendance of focus list students.

• Grade 6-12 teachers with corps members in their homerooms also agreed that City Year helped to increase overall attendance of students, with over half agreeing.

• Of the seven areas of corps member self-reported impact on students, attendance received the lowest means, both for corps members serving grades K-5 (2.31 on the 4-point scale) and grades 6-12 (2.46)

Student level data

• Four-fifths of the grade 6-9 students for whom attendance data were collected were already on-track at the start of year.

• Of the one-fifth who were sliding or off-track, the overall success rate (moving one track rung) was 54%, with three of six sites achieving their goal of 50% of students moving one track rung.

• In the 9th Grade Student Survey Pilot, 65% of students believed that City Year had “somewhat” or “a lot” helped them to improved their attendance.

Discussion. In FY10, the attendance initiative was somewhat new. The main observation about attendance coaching is that the wrong students were targeted. By encouraging sites to review prior year attendance data, City Year will create a greater likelihood that students who need help in this regard will be selected in the future. However, access to attendance data may remain a problem. Messaging to sites, corps members, and schools about attendance coaching and programming also needs improving. In FY10, service partners and teachers did clearly not understand City Year’s role


and activities vis a vis attendance, such as calls home. Further, only one-third of corps members working with students in grade 6 or higher said they were doing attendance coaching. It is understandable that lack of clarity at the site level would carry over to service partners. As City Year’s programming sharpens in the attendance arena and is reflected in the MOUs, it is hoped that schools can be appropriately supportive to assure the initiative’s success. Despite these issues and that deliberate attendance coaching may not have been happening to any great extent, teachers did still perceive that City Year had an impact on the attendance of focus list students in grades 6-12 and to a lesser extent on grade K-5 students, where attendance coaching was not a formal initiative. Ninth graders in the Student Survey pilot, also credited City Year for help with their attendance. Behavior Metrics

• In this pilot year, metrics were focused on data in service partner, teacher, corps member and student surveys on perceived impacts of City Year’s interventions on behavior concerns, such as relationships with other students, as well as school and learning climate. None of these can be connected to specific individual students although student pre- and post-surveys can be linked to one another through student generated IDs.

• Teams were asked to gain access to centrally collected student level behavior data (such as behavior grades, trips to office) or to create ways of collecting such data. Washington DC developed a special tracker that also focused on the positive—e.g., student acts of leadership.

Intervention/participation

• Just over half of K-5 Service Partners and more than three-fifths of grade 6-12 Service Partners reported that City Year did positive behavior programming in their schools.

• Among teachers, fewer then two-fifths of K-5 teachers and about half of grade 6-12 teachers believed that City Year did behavior coaching with their students. The numbers were almost the reverse regarding whole class and/or homeroom positive behavior support.

• For K-5, percentage of corps members reporting that they conducted these same behavior interventions with students are similar to, though slightly higher, than those of teachers, with 43% saying they did formal mentoring or behavior coaching, and 59%, whole class positive behavior support. In contrast, corps members working with grade 6 and higher students were much more likely than were teachers to say they did whole class positive behavior support (71% compared with 36%). About half of teachers and corps members at these grade levels indicated that corps members did formal mentoring or behavior coaching.

Preparation

• Corps members serving grades K-5 and grades 6-12 reported feeling equivalently prepared for their formal mentoring and behavior coaching work (63% and 65% respectively feeling “very” or “quite prepared.”). Those serving grades K-5 felt more prepared than those serving older students for conducting whole class and/or homeroom behavior support (63% and 50% respectively).


Perceived impacts • Teachers generally agreed that corps members working with focus list students had an

impact on those students’ behavior, with 71% of grade K-5 teacher agreeing and 80% of grade 6-12 teachers.

• 69% of K-5 teachers agreed that corps members helped “improve overall behavior and order in the classroom” as did 79% of grade 6-12 teachers. Further, 72% of grade 6-12 teachers with corps member who were present in their homerooms agreed that with that statement. In addition, more than three-quarters of teachers with corps members in their classrooms agreed that corps members “helped increase my students’ respectfulness to each other” in their classrooms. Finally, more than three-quarters also believed that City Year had at least a moderate impact on the respect that students exhibited to adults in their schools.

• The mean of corps member self-reported impact on students based on three items (student respect for each other, behavior and order in the classroom, and reduction of number of student conflicts) was 2.71 for grades K-5 and 2.66 for grades 6-12 (based on a 4-point impact scale). Three-fifths of corps members thought they had had at least “somewhat” of an impact on behavior and order in the classroom, with no difference between grade levels.

• More than three-quarters of service partners believe that City Year has had at least “somewhat” of an impact on student respect to peers and ability to generate peaceful solutions to conflicts. They were almost as positive regarding City Year’s impact on student respect towards adults and on the number of conflicts that occur between students. Results of K-5 services partners and those serving grades 6 and higher were very similar.

Student level data

• On the grade 6-8 surveys, amount of exposure to City Year was not related to ratings on “Relationships with Others.”

• On the 9th Grade Student Survey Pilot, four-fifths of students believed that City Year had “somewhat” or “a lot” helped them to understand how their behavior could affect their grades.

Discussion: At the elementary level corps member and teacher perceptions of corps member activity regarding behavior support and coaching are similar. Teachers at the middle and high school level also appear to be aware of City Year’s work in behavior coaching but much less aware about corps member contributions to whole class behavior support. Corps member feelings of preparation for behavior management have greatly improved from FY09 and show the impact of training in this area. In the coming year, there will be additional training for behavior test teams, and it will be interesting to see if there are differences between these teams and non-test teams in the extent to which corps member feel prepared. However, even without a special behavior focus for most teams in FY10, teachers still perceived that corps members had an impact on the behavior of focus list students. Does the additional attention that students on the academic focus list receive carry over into their general behavior? By necessity do corps members tutoring students in literacy or math also need to spend time on behavior issues? One big challenge in measuring future impact of City Year’s work on student behavior through its coaching initiative is in finding suitable measures. Not all schools give behavior grades nor track office referrals or detentions. In addition, standardizing those measures that do exist across schools


will be much more difficult than for attendance. After-School Intervention/participation

• Four-fifths of grade K-5 service partners reported that City Year had a Starfish program in their schools. In addition, three-quarters of grade K-5 service partners said that City Year provided after school academic or homework support to students and one-third, after-school enrichment programming. Percents for these latter two activities were higher for grades 6-12, with 87% of service partners stating that City Year provided after school academic or homework support, and 76%, after school enrichment programming. Corps member reports of their activities for this age group were slightly lower, at 80% and 66% respectively.

• With 120 out of 128 teams providing data about their after-school programs. For the 13 sites with Starfish Corps or After School Heroes, eight met or exceeded their goals, and all but two met 75% of their target. For homework help, only four of 17 reached their goals, with an additional two sites meeting 75% of their target goal. Finally, for the 17 sites with after-school programs other than Starfish or After School Heroes, 12 met or exceeded their goals and the remainder reached at least 75% of their target number. Thus, overall sites and teams did best in projecting and meeting their target numbers for general after-school programs, and least well in projecting and meeting numbers for homework help.

Preparation

• Corps members reported being prepared for their after school work, with 82% of those serving grades K-5 and 76% of those serving grades saying they were “quite” or “very prepared” for this work. In fact, corps members feel more prepared for this work than they do their other school responsibilities.

Performance

• Grade K-5 service partners gave teams good ratings on four aspects of performance and implementation of the City Year after school programming for grades K-5, with highest ratings (over 4.0 on a 5-point agreement scale) for content alignment of the program with the school’s goals and for student enjoyment.

Perceived impacts

• Service partners credited City Year with impacts in the after-school related to improvement in learning (active engagement in learning, time spent on learning tasks, fostering a positive environment for learning, and strengthening academic performance) and behavior/ SEL/social issues, with means of over 4.0 (on a 5-point agreement scale) for each of these two scales at both the elementary and middle/high school levels. Out of a list of 12 school climate factors, service partners at both the grade K-5 and 6-12 levels rated City Year’s impact on “school capacity to meet after school needs” as highest.

Student level outcomes

• Boys in grades 6-8 expressed a more positive attitude toward their school overall (School Climate) when they had had at least some exposure or participation in the after-school program.


• Positive “Relationships with Others” were evident for 7th graders who participated in the after-school program as compared to 7th graders who did not.

• School Connectedness and Learning Engagement were not affected by exposure to City Year’s after-school program for students either from grades 3-5 or 6-8.

Discussion: Although sites are doing well meeting their Starfish and After-School Heroes targets, they are doing less well meeting their goals for other kinds of after-school programs, especially those that provide homework help. Nevertheless, corps member self-reported preparation for after-school is higher than their preparation for most other areas of their work, and service partners give teams high performance ratings as well as high impact ratings. It is possible that service partners are less familiar with the nature and extent of team after-school activity than they are with other City Year roles in their schools and that City Year may need to do more in working with schools to live up to its potential in this area. Corps Member Performance & Service Partner Satisfaction

• Both teachers and service partners were asked to rate corps members’ performance at mid-year. Based on a series of items related to performance, mean ratings by both groups were high. Service partners were given the same items at the end-of-year; their ratings differed little from the mid-year. Service partners gave somewhat higher ratings to team leader communications than did teachers.

• Comparing this year’s team to last year’s (for schools that had a team in the previous year), the majority of service partners considered the quality of service of this year’s team as well as the overall experience of working with City Year to be better.

• Both service partners and teachers gave high satisfaction ratings for the quality of service provided by City Year and the overall experience of having City Year in their schools, with little difference between the grade levels in mean ratings. However, middle/high school service partners were most likely to say they were “very satisfied” with the quality of service.

Other Impacts Overall School Climate

• Service partners and teachers believed that City Year had at least a moderate impact on school climate, based on the mean ratings of a set of items about school climate. Mean ratings across the two levels (grades K-5, grades 6-12) and two kinds of stakeholders (service partners and teachers) were very similar, with teacher ratings only slightly higher than those of service partners. However, teacher assessments of their school climate at mid-year were lower than the assessments of service partners at both grade levels.

• At the end-of-year, when examining individual items in the school climate impact scale, teachers believed that City Year had had the most impact on the “number of clubs and extra-curricular activities for students.” As mentioned, service partners chose “Our school capacity to meet after school needs.” Both groups felt that City Year had the least impact on “The number of parents and community members who participate in our school.”

• For boys in grades 6-8, attitudes towards their school climate were most positive when they had had moderate help in math (i.e., once in awhile); whereas girls reported the most positive attitudes toward school climate when their help in math was one or more times a week


Impact on Teachers • At the mid-year and end-of-year, teachers were asked to rate their agreement on five items

related to the impact of their corps members on their own work and relationships with students. Teachers generally agreed that corps members had had an impact on such areas as helping them to “Feel supported in their work,” allowing them to “improve the quality of the contact” they had with students,” and giving them “more time for planning.”

• Mid-year and end-of-year assessments were similar as were assessments by the two grade levels. Overall agreement on this scale was slightly less than for perceived impact on students.

Other Student Impacts

• For students in grades 3-5, frequency of City Year spending time in class and of City Year helping the respondent in class were each positively related to Learning Engagement.

• In addition, for this age group, frequency of City Year helping with in-class work was positively related to School Connectedness, but the more exposure through help with in-class work plus homework was even more positively related to School Connectedness.

• In grades 6-8, students expressed more positive attitudes toward their school (School Connectedness) as exposure to English/language arts increased, but amount (or kinds) of exposure to City Year did not make a difference to Learning Engagement.

B. Recommendations/Actions Steps Overview In this section, we will provide recommendations/action steps that are suggested by the data to improve planning, preparation, implementation, and evaluation, leading, we hope, to enhanced outcomes and impacts, especially in the targeted areas of course performance in English and mathematics, attendance, and behavior. Recommendations are organized by these target areas and framed in terms of actions that may need to be taken by different City Year stakeholders. However, schools also need to take some responsibility for providing the appropriate support to ensure team success. Therefore, one key to effectiveness is strong relationships with service partners. In many instances, City Year has already acted to remedy problems cited below, such as creating a school relations department. Course Performance in ELA and Mathematics Overall, City Year has made great strides in preparing corps members for their academic work with students. Teachers seem to better understand corps member roles and to support them in their work.

• Continue to provide focused training and ongoing support to corps members in the areas of literacy and mathematics. Provide adequate training to corps members serving the elementary grades as well as to those working with the middle school grades. In addition, continue to provide more generalized training for corps members to help students improve their overall study habits (i.e., teach their students to “fish.”) (HQ and sites)

• Ensure that sites understand and carry out their WSWC requirement in grades 3-5 to provide one-on-one or small group ELA tutoring. Fifteen percent of corps members working with these grades say they currently do not do this. (HQ and sites)


• Provide additional support to program managers and team leaders in sites where student gains are not as frequently achieved to find out where issues/challenges exist. (HQ and sites)

• Explore working with the same students in consecutive years to assure that any gains are maintained and amplified. (Sites)

• Update surveys where needed to reflect greater attention to mathematics and to differentiate mathematics tier 2 tutoring from ELA tutoring. (HQ)

• Continue to support sites in working through their data collection challenges and in giving staff time to collect and enter accurate data on students receiving tier 2 tutoring. (HQ and sites)

Attendance Attendance is a relatively new initiative in the middle grades, and understanding about this initiative is still not clear to stakeholders on all sides. Corps members feel less positive about the impact they are having on attendance compared with other areas of impact.

• Ensure that sites serving the middle school grades fully understand their roles in helping students with attendance issues, not just the interventions themselves but the selection of appropriate students. New policies and systems relating to identifying students should help improve this latter aspect, but sites may continue to need guidance in this area. (HQ)

• Provide adequate and ongoing training and support for corps members doing attendance coaching. The majority of corps members serving students in grades 6-12 did not feel adequately prepared to carry out this intervention. (Sites)

• Work closely with service partners on the attendance initiative. In FY10, service partners did not appear to be aware that attendance coaching was supposed to happen (and indeed, it may not have been happening). Their support is critical. In addition, teams will need to mesh their practices with existing school policies and practices to be effective. (Teams)

• Adequately inform teachers about attendance related activities, such as calls home. (Teams) • Continue to address challenges around data collection and access to data. Improved service

partner understanding of this initiative may help in allowing better access. (HQ and sites.) Behavior Although the Behavior Initiative was in its pilot year in FY10, behavior management has been a concern since the creation of the WSWC program. Teachers and service partners do report that corps members have had a positive impact on student behavior, and in the 9th grade student survey pilot, students reported that City Year had helped understand the impact of their behavior on their academic work. As the initiative develops, a few areas need some attention.

• Clarify corps member roles in relation to whole class (and homeroom) positive behavior management, especially in the middle school, where corps members were much more likely to say they did whole class positive behavior support than their teachers were. Be sure that these roles are represented in MOUs. (HQ , sites and teams)

• Ensure that teachers understand, approve of, provide input and feedback to, and support corps members in this kind of work. Corps members can learn much from experienced teachers but only if they are willing partners. (Sites and teams)

• Continue to provide appropriate training for the different age groups, and differentiate between behavior support for individual students and behavior support for larger groups of students, as this latter area appears to be more of a challenge especially to those serving


those in the middle grades. Training needs to embrace the larger social-emotional learning domain.

• Encourage and support the regular use of observations of corps members both by Program Managers/Team Leaders and even peers (using the SEL Observation Protocol).

• Focus on ways to measure positive behavior, not just absence of negative behavior. In addition to using existing school measures of behavior (grades, office referrals), develop or adapt measures that can be applied in a more standardized way across schools and districts, perhaps engaging teachers in the use of these measures.

VIII. Conclusions To evaluate WSWC in FY10, City Year was able to able to build on the foundation of the previous year. In particular, we developed a Theory of Change for high school as well as piloted a student survey for grade 9, conducted complete pre- and post-survey administrations of the Grade 3-5 and Grade 6-8 Student Surveys; collected literacy data from 12 out of 14 sites; began collecting attendance data; developed middle/high school versions of our teacher and service partner mid-year and end-of-year surveys; developed a mid-year corps member survey; and strengthened use of data across the Network. Overall, our data indicate that City Year is having an important impact on the students and schools it serves. In the coming year, we hope to strengthen our ability to access and collect individual level data on literacy, attendance, and behavior; improve the capacity of cyIMPACT to link dosage data with student outcomes; and develop a longer term overarching plan for use of our various tools and instruments. Our goal is to help City Year continue to foster a culture that asks questions, seeks data to drive decisions, and employs high standards in collecting and interpreting these data in order to provide effective service. The ultimate aim of these efforts is to increase students’ chances of success in school.


Appendix A: City Year High School

Theory of Change

STUDENTS ARE NOT ENGAGED IN THEIR LEARNING

NEED TO DEVELOPTHEIR OWN

IDENTITIES & TAKEOWNERSHIP OF

THEIR GOALS(PRIDE)

NEED TOUNDERSTAND THE

ROLE OFLEARNING TO

THEIR FUTURES(PURPOSE)

NEED TOUNDERSTAND THEIRIMPORTANCE IN THE

BROADERCOMMUNITY

(PRIDE)

NEED TO FEEL THEYCAN MEANINGFULLY

PARTICIPATE IN THEIRSCHOOL & COMMUNITY

(PURPOSE)

NEED TO APPLYSELF-

MANAGEMENTSKILLS TO THEIR

LEARNING(DISCIPLINE)

Consistent,trusting, &productive

relationshipswith near peers

Recognition ofstudent strengths

& coaching forpositive

development

Proactive peersystems for

creating a positive& safe schoolenvironment

Opportunities for& encouragementof student voice

& leadership

NEED TO DEMONSTRATEUNDERSTANDING OF

LONG TERMCONSEQUENCES OF

BEHAVIOR(DISCIPLINE)

Opportunities forservice learning andfor students to make

contributions toschool & community

Connectionsto real world& students’interests &

goals

Modeling &reinforcement of CY

values: inclusion,diversity, teamwork,

work ethic

NEED TO BEPROACTIVE IN

THEIR LEARNING &GOAL FORMATION

(SPIRIT OFLEARNING)

NEED TO VALUE THEROLES OF BOTH

PEERS & ADULTS INTHEIR LIVES(SPIRIT OF

COMMUNITY)

STUDENTS ARE NOT CONNECTED TO THEIR SCHOOL OR COMMUNITYPROBLEM

STUDENTNEEDS INCONTEXT

OF CYCULTURE

CORESTRATEGIES

ACROSSACTIVITIES

GROWTH-ORIENTED(Capable & Committed Learners)

SCHOOLCONNECTED

PERFORMANCE IN ENGLISH & MATH IMPROVED ATTENDANCE IMPROVED BEHAVIOR

COMMUNITY-MINDED

HIGH SCHOOL GRADUATION STRONG CIVIC IDENTITYCOLLEGE READY & JOB COMPETITIVE

Engagement ofadditional resourcesto support studentsuccess (family,

corporate sponsors)

Apply self-managementskills & good study habits

PROMOTION TO THE NEXT GRADELEVEL

ACTIVE PARTICIPATION IN SCHOOLACTIVITIES

As MEASURED BY

Students actively participate inCY sponsored activities

FIRST LINEOUTCOMES

ULTIMATE OUTCOMES

FULL PARTICIPATION IN COMMUNITY-ORIENTED EVENTS

CITY YEAR, WHOLESCHOOL WHOLE CHILDTHEORY OF CHANGE

High School Versionv.2(2), 7-22-09

DATA &THEORY

DRIVEN CYACTIVITIES

CITY YEAR CLASSROOMS (C)Group Support

WHOLE SCHOOL (W)Positive School Climate and After-School Opportunities

STUDENTS ON FOCUS LIST (F)Individualized/Targeted Support

Families and otherstakeholders support school

& City Year efforts

Students develop long-term goals

Students developproactive relationshipswith corps members

Grounded inunderstandingof adolescent/young adult

development

Use of “viral”and other

peeroriented

approaches

See that learning isrelevant to their lives

Feel more confidentabout their ideas

Students collaboratewith each other to

improve their school

Students see near peers asrole models for behavior &

values

Identificationof student

needs, withteachers

1:1 & Small GroupTutoring (T) in English &

Math throughout day

GeneralClassroomSupport (C)

Attendance IncentivePrograms (F)

Greetings atHomeroom Door

(C) (F)

Support for SchoolPositive Reinforcement

Systems (C)(W)

Out of School Time &Service Opportunities (W)

ASMEASURED

BYStudents have less than

80% attendance rate Students have unacceptable behavior indicatorsStudents are off-track in theirperformance in English

Students are off-track in theirperformance in Math

Believe that school is a safeplace where they can develop

interests & express themselves

Feel hopeful that they cansucceed if they put in the effort

& demonstrate that effort

Understand the importance ofand seek out learning as the

foundation for life long success

Believe that they can make adifference as part of a group

Experience academicsuccess

Feel a greater sense ofconfidence & self-worth

Appreciate & respectothers’ differences

Experience success in non-academic subjects

Help to improveschool & community

Develop theirown identities

Feel like family & others careabout their success at school

Develop mature relationshipswith their peers

Structured PositiveReinforcement &

Intervention (T) (C) (W)

Students understand thelong-term consequences of

their behavior

Contribute to building a strong communityand creating change through cooperation,

respect, & taking responsibility

INTERIMOUTCOMES

The C’s

Homework Helpin/out of School

(T) (C) (W)

Take ownership oftheir goals

Know how toaccess support

Are proactive inrelation to their goals

Developleadership skills

Appendix B: City Year

Grade 9 Student Survey

1

SPRING 2010 CITY YEAR GRADE 9 STUDENT SURVEY

Read each item carefully, including the answer choices. After you have read the item, CIRCLE the one best answer for each. Choose only ONE response for each item. Be as honest as you can. Your answers will be kept private. Keep on working until you have completed your survey. If you don’t understand something, ask for help. THANK YOU!

I. YOU AND YOUR STUDIES Almost always Often Some-

times Not

often Hardly ever

1. When I work hard in school, I succeed. 5 4 3 2 1 2. I come to my classes with my assignments finished. 5 4 3 2 1 3. I know what I am good at. 5 4 3 2 1 4. I enjoy learning new things in class. 5 4 3 2 1

5. I work hard on my schoolwork even when it is difficult. 5 4 3 2 1 6. I pay attention in class. 5 4 3 2 1 7. I make an effort to do my best in school. 5 4 3 2 1 8. I believe what I learn in school is important for my future. 5 4 3 2 1

II. YOU AND YOUR SCHOOL Almost always Often Some-

times Not

often Hardly ever

9. I enjoy coming to school. 5 4 3 2 1 10. I feel like my teachers care about me. 5 4 3 2 1 11. I feel like the adults at my school respect me. 5 4 3 2 1 12. I feel accepted by other students outside my circle of friends. 5 4 3 2 1 13. I like my school. 5 4 3 2 1 14. I feel good when I am at school. 5 4 3 2 1 15. I care about my school. 5 4 3 2 1

III. YOU AND OTHERS Almost always Often Some-

times Not

often Hardly ever

16. I get along well with people who are different from me. 5 4 3 2 1 17. I try to help others even if I don’t know them well. 5 4 3 2 1 18. I am respectful of others even when I don’t agree with them. 5 4 3 2 1 19. I stand up for someone who is being picked on. 5 4 3 2 1 20. I resolve conflicts with other students peacefully. 5 4 3 2 1 21. I think about how my behavior affects others. 5 4 3 2 1

IV. ABOUT YOUR SCHOOL OVERALL Almost always Often Some-

times Not

often Hardly ever

22. My school feels like a safe place. 5 4 3 2 1 23. Adults in my school treat students fairly. 5 4 3 2 1 24. Adults in my school respect student opinions. 5 4 3 2 1 25. Students in my school are respectful of one another. 5 4 3 2 1 26. Students are rewarded for positive behavior. 5 4 3 2 1 27. Students show respect to adults in my school. 5 4 3 2 1

Grade 9 Student Survey – Spring FY10 2

V. YOUR GENERAL THOUGHTS

28. How much do you agree with the following statements? Strongly Disagree Agree Disagree Strongly

Agree a. It is my responsibility to help make my school a better place. 4 3 2 1 b. I believe I can make a positive difference in my school. 4 3 2 1 c. I plan on graduating from high school. 4 3 2 1 d. By the end of the school year, I feel like I will be prepared to enter 10th grade. 4 3 2 1 e. There is at least one City Year corps member that I trust. 4 3 2 1 f. I respect City Year for the way they help others. 4 3 2 1

VI. CITY YEAR

29. During the school day (not after school), how often does someone from City Year--

3 or more times a week

1-2 times a week

Once in awhile

Never/ hardly ever

a. Tutor you in English/Language arts (by yourself or in a small group) 4 3 2 1 b. Tutor you in math (either by yourself or in a small group) 4 3 2 1 c. Tutor you in another subject (not English or math) 4 3 2 1 d. Talk to you about your attendance 4 3 2 1 e. Help you with your homework or assignments 4 3 2 1 f. Talk to you about how to behave 4 3 2 1 g. Talk to you about preparing for college or a career 4 3 2 1

30. How much has City Year helped you- A lot Somewhat A little Not at all

a. Understand why learning is important 4 3 2 1 b. Believe you can succeed 4 3 2 1 c. Understand what you need to do to be promoted to the next grade 4 3 2 1 d. Understand better what is being taught in your classes 4 3 2 1 e. Understand how your behavior can affect your grades 4 3 2 1

f. Complete more of your homework and assignments 4 3 2 1 g. Become more interested in school 4 3 2 1 h. Improve your attendance 4 3 2 1 i. Organize yourself better for studying 4 3 2 1 j. Work out a problem you had with a teacher 4 3 2 1 k. See the value of helping others 4 3 2 1

VII. ABOUT YOUR SCHOOLWORK AND BEHAVIOR

31. In the current school year, how are your grades in the following? Mostly As Mostly Bs Mostly Cs Mostly Ds or Fs

a. English/Language Arts 4 3 2 1 b. Math 4 3 2 1

32. In the PAST MONTH, how many times have you--- 6 or more times 3-5 times 1-2 times Never

a. Skipped school 4 3 2 1 b. Been late to school 4 3 2 1 c. Skipped out on classes during the school day 4 3 2 1 d. Been asked to leave class by a teacher 4 3 2 1

33. In the CURRENT SCHOOL YEAR, how many times have you— 6 or more times 3-5 times 1-2 times Never

a. Had detention 4 3 2 1 b. Been suspended 4 3 2 1 c. Made honor roll or received an award from your school for your grades 4 3 2 1 d. Been recognized for positive behavior or attendance 4 3 2 1

Grade 9 Student Survey – Spring FY10 3

VIII. YOUR FEEDBACK

34. How interested would you be in--- Very interested

Somewhat interested

A little interested

Not interested

a. Attending an after-school drop-in program to help you with your schoolwork 4 3 2 1 b. Doing service projects, such as painting a mural or planting a garden 4 3 2 1 c. Helping younger students, such as doing tutoring 4 3 2 1 d. Learning skills, such as how to resolve conflicts, during your advisory/ homeroom period 4 3 2 1

35. To what extent does your school provide volunteer opportunities for students? Check one response.

□ Not at all

□ Some

□ A lot

□ I don’t know. IX. ABOUT YOU Check one response for each question.

36. Your gender: □ Male □ Female

37. Your age: □ 12 □ 13 □ 14 □ 15 □ 16 □ 17

38. Your grade: □ 9th □ 10th □ 11th □ 12th

39. Is English the main language spoken in your home? □ No □ Yes

40. Do you currently participate in the City Heroes program on Saturdays? □ No □ Yes □ I don’t know 41. About how many hours a day do you spend on your homework and school projects to be completed outside of class?

□ None

□ Less than 1 hour

□ 1-2 hours

□ 3-4 hours

□ More than 4 hours 42. How often were you honest in filling out this survey?

□ Hardly at all

□ Some of the time

□ About half of the time

□ Most of the time

□ All of the time

city year whole school whole child fy10 evaluation report · whole school whole child fy10...

Documents