Evaluation of Xtreme Reading Implementation

Sean MurphyNorth East Leadership Academy

Cohort 1Spring 2012, NCSU

Problem of PracticeSTRATEGIES OBJECTIVES ULTIMATE GOAL

A: Train teachers on the13 instructional stages used year long as coreinstructional strategies

Teachers know and useresearch-basedstrategies effectively

D: Provide on-going support for Xtremereading teachers andstudents

Teachers differentiateinstruction

Students use literacystrategies proficiently

At-risk students are more confident and successful readers.

Teachers know how to reach struggling learners

C: Train teachers to usePossible Selves strategy

Students collaborateduring class

B: Train teachers onresearch based literacystrategies through XtremeReading Program

STRATEGIES Evaluation Questions

A: The 13 instructionalstages provided aninstructional routine ofpedagogic practice(the “how”).

D: On-going support forXtreme Reading teachersand students includedcoaching, planningassistance and modeling.

For the purpose of the project,results were insignificant .

C: Possible Selveswas a motivationalstrategy.

On how many strategies wereteachers trained?B: Xtreme Reading

included 8 research-basedliteracy strategiesexplicitly taught tostudents (the “what”).

How many hours were spent trainingteachers on 13 stages of instruction?

How many hours were spentsupporting teachers on the 13 stagesof instruction?

Did the teachers think the session wasof high quality?

How many professional developmenttraining sessions were given?

To what extent did teachers think thetrainings were useful for their instruction?

How many hours of support wereprovided for the strategies?

How supported did teachers feelduring the process?

Training agendas

Support logs

Professional Development Questionnaire

Likert Scale

Focus Group

Training agendas

Support logs

Professional Development Questionnaire

Data

Objectives Evaluation Questions

1: Teachers can explicitlyteach literacy strategiesto students (the “what”).

4: Students use literacy strategies proficiently

How often do students collaborate during class?

3: Students collaborateduring class, as laid out inthe 13 instructional stages(the “how”).

How often do teachers differentiate instruction?

2: Teachers differentiateinstruction, as laid out inthe 13 Instructional Stages(the “how”).

How confident do teachers feel using research-based strategies?

How often do teachers use research-based strategies?

How many students used thestrategies proficiently?

Professional Development Questionnaire

Learning Strategies Walk Through Rubric

Teacher Self-Report Survey

Pre-Post Assessments for Each Strategy

Retrospective Pre/Post Student Survey

Data

Results• According to training agendas, teachers received approximately 4

hours of training on the 13 stages of instruction at the beginning of the year.

• According to the support logs, teachers received 7 hours of support

specifically regarding teacher practices (13 stages of instruction). Teachers did not receive support for 13 stages of instruction (as a whole model) after initial training. However, there was specific support for differentiation and student collaboration. For the former teaching strategy, I provided 3 hours of support (mainly through directive support and modeling) and curriculum coaches provided .5 hours. Teachers received 1 hour of support on student collaboration from myself and 2.5 hours of support from coaches.

A: The 13 instructional stages provided an instructional routine of pedagogically practice (the “how”)

Results• Teachers were trained on a total of 3 strategies during the first

semester of implementation, conducted over 4 training sessions. Teachers received a total of 12 hours of training each during the Fall Semester. Teachers were trained on 2 strategies from January through March in 3 training sessions, totaling 8 hours for each teacher.

• Over the course of the semester teachers were trained on literacy strategies. Their reaction was mixed. During the first session, 1 out of 3 teachers strongly agreed that training was useful for their instruction, 2/3 disagreed. For the second session, 1/3 strongly agreed and 2/3 agreed. With the third session, all teachers strongly disagreed that it was useful. All agreed the fourth session was useful.

B: Xtreme Reading included 8 research-based literacy strategies explicitly taught to students (the “what”)

Results

Results

• Questionnaires showed that 3/3 teachers felt confident and prepared to teach the Possible Selves strategy after training.

• According to interviews, teachers were introduced and discussed how to use Possible Selves strategies in the classroom. Sharing of Possible Selves lessons was facilitated by administration during weekly PLC sessions. Only 1 lesson was shared, student artifacts were collected but not reviewed.

C: Possible Selves was a motivational strategy.

Results• Support logs showed administration provided 26 hours of support either individually or with

Xtreme teachers as a PLC, over the course of the school year. Support came in the form of coaching, modeling and facilitating planning. University of Kansas coaches provided 6 hours. Support included coaching and facilitating planning.

• According to self-report likert scales, 3 out of 3 teachers strongly agreed that administrators prepared them for implementing strategies. They also strongly agreed that administration supplied adequate on-going support to effectively teach strategies. However, the feelings about curriculum coaches were less enthusiastic. 2 teachers agreed that the coaches prepared them for implementing strategies and 1 disagreed. Regarding the on-going support from curriculum coaches, 1 teacher felt neutral about the level of support and 2 felt inadequately supported.

• In a 30-minute focus group, teachers voiced negative feelings toward support 18 times and positive feelings toward support 6 times. It was stated that more on-going support and coaching was needed after initial training. Specifically, teachers felt “neglected,” “abandoned,” and “used” by the curriculum coaches after training. They generally agreed that administration provided on-going support (4 of 6 positive comments were directed toward administration).

D: On-going support for Xtreme reading teachers and students included coaching, planning assistance and modeling.

Results• Based on survey data, 66% of teachers they strongly

agreed that they were confident teaching three out of four of the strategies they received training. They were least confident teaching the Word ID strategy. In comments, teachers said this training was ineffective because it was conducted virtually through Face Time link.

• 26 copies of the Learning Strategies Walk Through rubric showed that teachers were using research-based strategies on 12 occasions; 4 in January, 6 in February, 2 in March.

1: Teachers can explicitly teach literacy strategies to students (the “what”)

Results

• According to 26 copies of the Learning Strategies Walk Through Rubric, teachers differentiated instruction 18 times; 2 in January, 6 in February and 10 in March. Differentiation consisted of purposeful organization of group work, leveled texts, task variety and individual support.

2: Teachers differentiate instruction, as laid out in the 13 Instructional Stages (the “how”)

Results• According to a retrospective pre/post survey, Xtreme Reading students said

the number of times they collaborated in the classroom increased from the first days of school to the middle of spring. Specifically, in response to a prompt asking, “how often did you collaborate the beginning of the school year,” 33% responded “about once a week”, 29% responded “once or twice a week” and 14% collaborated “everyday.” In the same survey, a prompt asked student, “how often do you collaborate now?” 20% of students marked that they collaborated “once a week,” 42% reported “once or twice a week” and 25% said they collaborated everyday.

• In 26 copies of the Learning Strategies Walk Through Rubric taken from December-March, students were collaborating 18 times. Collaboration consisted of student paired-reading 8 times and collaboration on projects 10 times. 2 in January, 6 in February, 10 in March.

3: Students collaborate during class, as laid out in the 13 instructional stages (the “how”)

Results

Results

Recommendations: Strategies• Provide More Spiraled Support for 13 Stages of Instruction: One major lesson learned, in reflection, was the necessity

for clarifying the need for instructional change. One of the reasons Xtreme Reading was chosen as an intervention program was that it required differentiated and collaborative work. At the time of initial training it was assumed that this was already a routine, to some extent, in the classrooms. Thus, teachers did not have ample training on effective pedagogy, as presented in the 13 instructional stages routine. Informal and formal observations showed that South Johnston teachers relied heavily on direct instruction and assessment. This was true both before and after the program, proving the need to measure the amount of differentiated instruction and collaborative work. In preparing for next year’s implementation, much consideration and focused support should be given on instructional change.

• Offer more support. Three major changes should be made in supporting methods: 1) Frontload support, 2) Offer more ongoing support, 3) Make support more directive. First, after consulting with the KU coaches, it is clear that amount of support should be more frontloaded next year, with consistent visits and updates during the first quarter. Second, while the amount of support might decrease in the second semester, open and regular dialogue should persist throughout the implementation (including analysis of data collected). The third point will be discussed in more detail below. At this point, though, it is worth mentioning that support should be more rooted in the materials and include specific follow-up about how instruction can be improved.

• Manage the Transition. In hindsight, it is clear that the transition from should have been planned and managed more effectively. Many of the teachers made it clear in support sessions that they were unfamiliar with specific strategies. They voiced apprehension with teaching reading strategies explicitly and had a hard time letting go of traditional instructional practices (predominately lecture and test). This could be expected, but plans should have including pathways forward, including methods for celebrating accomplishments and acknowledging areas for improvement.

Recommendations: Objectives• Utilize the program’s resources. This was alluded to above in the “offer more directive support”

bullet. The Xtreme Reading program has a vast amount of resources, including lesson plans, pacing outlines, and formative assessments that can inform instruction. Yet, almost all of the support we offered (especially in the beginning of the year) did not put these materials to use. On a regular basis, administration and/or instructional coaches should sit down with the teachers and the materials. This can be used to ground conversation. Teachers should reflect on how closely their planning and pedagogy follow the guidelines. Of course, teachers should have the autonomy to make modifications; however, they must know what the standard looks like and try adhering to the program as it is designed.

• Make Goals Clearer and More Prominent: In retrospect, the outcomes for the program should’ve been framed in both teacher and student terms. We were trying to change instructional habits as much as content. However, this is a more delicate process, as former teaching habits were already in place, demanding attention to the transition. What’s more, the entire instructional cycle should have been emphasized, so that teachers were constantly reminded how differentiated teaching and collaborative learning fit into the framework for teaching. Once the goals were stated more clearly, systems should be in place to regularly monitor progress. The walk through form could have been used more consistently with teachers, anchoring coaching conversations.

Evaluation Reflections• Don’t ask so many questions. To be embarrassingly honest, I didn’t mean

for this PowerPoint to be this long. However, by asking so many questions, I left myself with little choice. I remember Dr. Corn saying, “Keep it simple, you’ll have to collect data on everything you ask.” Well, I guess we learn from experience.

• Stop and smell the data. I waited too long to really start analyzing the data I was collecting. If I had started earlier, I would’ve noticed a pattern (content not changing, pedagogy changing) earlier on. This could have shaped student interventions and teacher support.

• Sense of efficacy from understanding the process. In general, I didn’t have the slightest idea about how to conduct an program evaluation. This projects helped me take the long-view. By seeing how the entire cycle is completed, I’ll be more prepared every step of the way, from planning to reflection.