copyright © los angeles unified school district 2011 copyright © tulsa public schools 2011...

Copyright © Los Angeles Unified School District 2011Copyright © Tulsa Public Schools 2011Copyright © Los Angeles Unified School District 2011

Teacher Development and Support System

Feedback from SIG schools

July, 2011

Copyright © Los Angeles Unified School District 2011

Context for feedback • As part of developing the teacher evaluation and support system for

SIG schools, the district has been soliciting feedback from participants on various aspects of the system

• Approximately 650 teachers and staff participated in this work– 75% completion rate as of June 24

• The feedback is still in the process of being gathered and analyzed; those areas with the most feedback to date have been analyzed

• This feedback will inform the further development of the measures and the piloting of the evaluation system in school year 2011-12


The Multiple Measuresof Teacher Effectiveness

Differentiated Compensation & Recognition

Development & Support

Self-Reviews

Individual Growth Plans

Observation of PracticeObserving teaching & review artifacts of practice (e.g. lesson and unit plans, student

work)

By Administrators&

By Teachers

Stakeholder FeedbackParent Surveys

Student Surveys

Contributions to School

CommunityTBD

Contributions to Student Learning

OutcomesAcademic Growth over

Time

Analysis of Supplemental Close-Ended and Open-

Ended Assessments

Teaching & Learning FrameworkA common foundation for effective teaching

Multiple

Measures

Mul

tiple

M

easu

res




Self-Reviews

Individual Growth Plans

Observation of PracticeObserving teaching & review artifacts of practice (e.g. lesson and unit plans, student

work)

By Administrators&

By Teachers

Stakeholder FeedbackParent Surveys

Student Surveys


CommunityTBD


OutcomesAcademic Growth over

Time

Analysis of Supplemental Close-Ended and Open-

Ended Assessments


Multiple

Measures

Mul

tiple

M

easu

res

Key questions for teachers’ feedback

How is effective teaching best measured?

How are decisions related to promotion,

compensation, and intervention aligned to support teacher

effectiveness?

How is effective teaching best supported?

How are changes to teacher supportand development best implemented?




Observation of Practice

Stakeholder Feedback


Community


Outcomes


Multiple

Measures

Mul

tiple

M

easu

res

Activities that informedeach component of the system

Teaching & Learning Framework (TLF): Workshop I & IITeaching & Learning Framework: Book Study GroupTotal Effectiveness Results: Small GroupTotal Effectiveness Results: Individual FeedbackListening Sessions

Stakeholder Input: Survey Feedback

Total Effectiveness Results: Small Group & Individual

Listening Sessions

Total Effectiveness Results: Small Group & Individual

AGT: School Level Results

TER: Small Group & Individual

Listening Sessions

TLF: Workshop IIListening Sessions

Implementation

Diff Comp: Small Group

Listening Sessions


FEEDBACK ON OBSERVATION OF PRACTICE


What are the benefits of the Teaching and Learning Framework?

Listening Sessions:•Virtually all teachers seemed to like the TLF and praised it for providing “common language” for all. Comments included:

– “really good guide on how to be a good teacher”– “not McStandards”– “good foundation”– “job description” that includes “all aspects” of being a teacher.

•In two separate schools, teachers expressed their pleasure that the new framework seemed to be a move away from evaluation based on “red flags” such as “standards on the board.” Book Study:•Predominantly, participants felt that the TLF will benefit teachers as a roadmap and guide for reflection and self review. •Participants also felt that the TLF will promote collaboration between colleagues by providing a common language and clear expectations.


What are the concerns with the Teaching and Learning Framework?

Listening Sessions:•The most prevalent concern was principals’ capacity to use the framework to evaluate teachers due to the time, knowledge and skill required by such a complex and rigorous tool. They acknowledged the demands that the TLF placed on evaluators •Several other concerns were mentioned, though less frequently:

– Concerns about favoritism (real or perceived) – Potential for distorted reports or poor interpretation of the framework’s

intention – Time required to complete might lead to falsifying the document (as has been

known to happen with the STULL evaluation).

Book Study:•Given the complexity and size of the TLF, participants were concerned about the implementation and whether the district will provide enough time and support to teachers and administrators.


Perceptions about the Teaching and Learning Framework Workshops

Workshop 1:Over 90% of participants agreed or strongly agreed that

• They learned useful information about the TLF• Colleagues were interested in the workshop content• They were interested in the workshop content

Workshop 2:•87% of participants agreed or strongly agreed that they learned useful information about the TLF self review process and lesson design template.•88% of participants agreed or strongly agreed that they were interested in the workshop content.•90% of participants agreed or strongly agreed that their colleagues were interested in the workshop content


Feedback on the TLF lesson design process and template

Workshop 2:•Benefits of the lesson design template (link):

– considers all components of teaching, very detailed– can be used as a guide to organize lesson– pushes evaluator to look at various aspects– lets teacher know what's being observed

•Concerns about the lesson design template (link):– the design template is overwhelming and will take too much time to complete– the observers, especially the administrators, will not understand the design

template and adhere to the observation protocol– teachers will have difficult meeting the expectations of the design template


Total Effectiveness Result: Teacher Results:•On average, participants felt that Observation of Practice should count towards 55% of the Total Effectiveness Result, with 35% coming from Administrator Observation and 20% coming from Teacher Leader Observation (link).•80% and 62% of responses wanted Administrator Observation and Teacher Leader Observation, respectively, to count towards teacher evaluations (link).

How would you weight Observation of Practice towards a Total Effectiveness Result?


Total Effectiveness Result: Teacher Results:

Benefits of and concerns with using Observation of Practice as part of teacher evaluations

Category Benefits Concerns

Administrator Observations

• Experienced leaders who manage instruction in the school

• May not have subject matter or instructional expertise

Teacher Leader Observations

• Bring content and pedagogical expertise

• Come from the classroom with understanding of it

• If from same building, could create tension


FEEDBACK ON STAKEHOLDER SURVEYS


Feedback on Stakeholder SurveysStakeholder Input Small Group Activity:•Participants felt the pilot student survey was clear and organized, had a good coverage of topics, and would be useful for informing teaching practice•Many also felt the survey was too lengthy and could be subject to student bias

Listening Session:•Few participants mentioned the role of surveys. Of those who did, most thought they would be useful and had the potential to engage others (students and parents) in the work of school improvement. One teacher thought that disgruntled students would use them as a means to get back at strict teachers.


How would you weight Stakeholder Input towards a Total Effectiveness Result?

Total Effectiveness Result: Teacher Results:•On average, participants felt that Stakeholder Input: Student Surveys should count towards 5% of the Total Effectiveness Result (link), with 51% wanting to use it as feedback only, and 14% not count all (link).• The benefit of Stakeholder Surveys: provides opportunity for

learners to weigh-in (link).• The concerns: subjective, could conflict with discipline and grading

efforts (link).


FEEDBACK ON CONTRIBUTIONS TO SCHOOL COMMUNITY


How would you weight Contributions to School Community towards a Total Effectiveness Result?

Contributions to School Community Activity•Unfortunately, due to time constraints, we were not able to conduct the activity for this component.

Total Effectiveness Result Small Group Activity:•On average, participants felt that Contributions to School Community would count towards 8% of the Total Effectiveness Result, with 33% wanting to use it as feedback only, and 21% not count all.•The benefit of using this measure: supports collaboration (link)•The concern: depends on definition of measure (link)


FEEDBACK ON CONTRIBUTIONS TO STUDENT OUTCOMES


What are the benefits of using Academic Growth over Time (AGT) as a measure of teacher effectiveness?

AGT Small Group Activity:•Gives teachers motivation and direction to improve•Can be used as benchmark and comparison to other schools•Focus on improvement and not just achievement

Total Effectiveness Result Small Group Activity:•Provides objective measure of student learning•Supports educators to seek best practice to improve growth•Supports collaboration within school

Listening Sessions:•Some teachers appeared to look forward to the opportunity to demonstrate their effectiveness via test scores


What concerns do you have about AGT?

AGT Small Group Activity:•External factors/variables not factored into score•Incomplete/limited results by subject, grade level, subgroups•Accuracy, Validity, Reliability of AGT

Total Effectiveness Result Small Group Activity:•Could focus efforts too narrowly on test prep•Test scores are due to many variables•Can be outside of individual teacher’s control

Listening Sessions:•“…participants named as a fear the possibility that test scores would come to dominate the evaluation process…”


How would you weight AGT towards aTotal Effectiveness Result?

• 66% and 53% of participants supported the inclusion of individual and school AGT, respectively, in the TER score (link).

• On average, student outcomes (individual and school AGT) weighted 28% of Total Effectiveness, second to observations (link).

• Variance shown in how much AGT should count towards TER score (link).

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups; blank answers excluded and answers adjusted to total 100%.


FEEDBACK ON DEVELOPMENT AND SUPPORT


How should we support effective teaching?

Support teachers through FEEDBACK•While overall teachers liked the framework as a document, how the document is used was equally important:

– Use “as a tool, not a weapon” – Use as an “appraisal,” as a step in developing teachers not, “punitively.”

•They wanted to know that the process would be “sufficient” and “fair,” providing multiple rounds of constructive feedback and allowing for opportunities to improve.

Support teachers through CULTURE OF COLLABORATION•The dominant theme among all groups was the importance of collaboration. Collaboration was indicated as a way to benefit both individual teachers and their school overall. •A few teachers noted that collaboration is not necessarily intrinsic and that it must be developed. Several more expressed concerns that the EE work has the potential to be divisive, if it becomes competitive rather than collaborative.

Source: SIG Listening Small Group Sessions 2011; 17 sessions; 76 participants.



Support teachers through PROFESSIONAL DEVELOPMENT•Not surprisingly, teachers repeatedly noted the importance of high quality professional development. Several specified that it should be specifically linked to an element of the framework, including opportunities to learn further about the framework itself and to strategize with peers about putting it into action•Several teachers noted the importance of “real” time – not just the minutes left over during their weekly meeting after the school logistics and announcements are completed.

Support teachers through FLEXIBILITY•In numerous sessions, teachers spoke about the flexibility inherent in the TLF. They appreciated that the framework appears to focus on the quality of instruction rather than adherence to a particular way of teaching. •Some fear that the TLF will be misinterpreted as “one and only one way to implement.”




Support teachers through SELF-ACCOUNTABILITY•In nearly every listening session, teachers talked about their hopes for their own growth as teachers, many very eloquently. Many appreciated the role that the TLF could play in encouraging “self-reflection,” “accountability for self” and “personal growth.” Interestingly, most of these comments were made prior to participating in the second TLF workshop that included the self-review process.



FEEDBACK ON DIFFERENTIATED COMPENSATION AND RECOGNITION


What kinds of incentives should be used and on what basis?

• Monetary incentives were, by far, the most frequently proposed form of compensation, followed by career pathways (link).

• Academic Growth was the most frequently mentioned standard of measurement for receiving incentives (link).

• Most respondents preferred at least 1 incentive (link).


How should the differentiated compensation system be structured?

• Majority of respondents preferred fixed incentives, such as bonuses (link).

• Respondents preferred individual and school-wide incentives over group (team/department) incentives (link).

• Variance shown in the weighting of individual, versus group versus school-wide incentives (link).

• Respondents tended to weight individual incentives more than group and school incentives (link and reasons for weighting: individual, group, school).


How should the differentiated compensation system be structured?

• Majority of respondents say teachers who are more effective should be compensated more. However, there was concern about divisiveness and less collaboration (link).

• Respondents also felt that multiple indicators, and not just CSTs and graduation rates, should be used as a measure of achievement (link).


FEEDBACK ON IMPLEMENTATION


What additional factors should we consider in implementation?

Student Achievement•In every single listening session, teachers named improved student learning and achievement as a hoped for outcome from the Educator Effectiveness work.

Communal Responsibility •Teachers spoke about the role that other factors play into successful teaching and learning. One asserted that “academic success of a student is not solely the responsibility of one teacher [but rather] a collection of things.”•Several teachers believed that peer observations and collaborative work focused on the framework could be used to hold each other accountable and to encourage everyone to “do a little extra” thereby raising expectations for all. •Some teachers stressed the role of collaboration among school staff in the form of “collective accountability for students with each other.” Others talked about seeing parents and students as “pieces of the puzzle” and discussed how to engage parents as partners in their children’s success.




Student Accountability•Several groups of teachers spoke about the need for student accountability as well as teacher accountability.

“Teacher Bashing”•In most sessions, teachers expressed concerns about the trend toward “teacher bashing” and fears that the “entire burden” for student and school success would be placed on them. One teacher (who incidentally likes the TLF and expressed strong support for its use) noted that there is an “underlying attitude that teachers are not working hard enough.”

Turnover•In about half of the sessions, teachers talked about the disruptive and demoralizing effects of turnover. While concerns about turnover appeared repeatedly, some thought this work would increase it, while others thought it would lead to a more stable teaching force.




Alignment•Aligning TLF with Beginning Teacher Support and Assessment (BTSA) standards, or finding a way to use TLF in its place, would simplify the work for beginning teachers and those who work with them.

Organization and Communication•The nature of the pilot phase (i.e. compressed timeline to complete 50-hours of professional development, school calendars that were established prior to pilot phase, and inconsistent information pathways) led to teachers’ concerns about organization and communication. This created an “air of anxiety” in many schools. Teachers stressed repeatedly that “transparency,” “good communication” and “consistency” would significantly improve the implementation of the EE work.

“Stick With It” •In all schools, there were teachers who expressed the need to persist with the Educator Effectiveness work beyond the 50-hour commitment. Many noted that that it requires substantial learning and will take time to succeed. They expressed hopes that their colleagues will “step up.”



APPENDIX: ANALYSES OF FEEDBACK BY ACTIVITY


Year 1 SIG Activities:Teacher Effectiveness

Click the hyperlinks below to jump to any of the activities:

1.Teaching & Learning Framework: Workshop I—Introduction to the Framework2.Teaching & Learning Framework: Workshop II—Application of the Framework3.Teaching & Learning Framework: Book Study Group Activity4.Academic Growth over Time: School Level Results Overview5.Total Effectiveness Results: Small Group—Teacher Results6.Total Effectiveness Results: Individual Feedback Form7.Differentiated Compensation: Small Group8.Stakeholder Input: Survey Feedback—Small Group Activity or Focus Group9.Listening Sessions


Teaching & Learning Framework: Workshop 1

• The first Teaching & Learning Workshop introduced participants to the Teaching and Learning Framework and rubric.

• Participants filled out feedback forms at the end of their session, 495 responses in total.

Click for last viewed slide

Click forActivities List


Feedback on Workshop 1 Structure• Overall, over 90% of the participants Agreed or Strongly

Agreed that• The sessions were well organized• Activities were well-paced and appropriately varied• Presenters had strong knowledge• They learned useful information about the TLF• Colleagues were interested in the content• They were interested in the content

• Overall, 75-77% of participants Agreed or Strongly Agreed that• They have a clear understanding of how the session relates to

their SIG work back at school• They understand the next steps related to the SIG work



Source: SIG TLF Workshop 1 feedback forms 2011; N= 239; Blank answers excluded.


Feedback on Workshop 1 Content• Overall, Wisdom of Practice and Memorable Moment were

the most engaging for the teachers (79-80% selecting highly engaging or engaging)

• Give One-Get One, Evidence for Standards 4 & 5, and Reflective Writing were the least engaging (60%, 54%, and 55% selecting highly engaging or engaging, respectively)

• As a whole, participants had a clearer understanding of “Establishing a Culture for Learning” (Q18), the difference between Developing and Effective (Q19), and Cognitive Engagement (Q21).

• Less than 50% of participants selected the correct answer for “Ineffective Practice” (Q20) and 21st Century Skills (Q22).





Feedback from Participants

• About 20% of the surveys had comments• Participants were most confused about applying the

framework to their teaching, understanding the purpose of the framework, implementing the framework at the classroom, schools, and district level given the complexity, and the next steps in the SIG process

• Participants thought it would be most helpful to review and study the content on their own or with peers, see real-life examples of the framework being used in the classroom, and receiving more PD like this one.





Workshop 1 Participant Make-Up• Over half were middle school teachers, 15% were elementary

teachers, and 19% were high school teachers.• 2% were principals• Compared to the High School teachers, Elementary teachers

had a more favorable opinion about the workshop structure (on average +8% response rate for agree and strongly agree) and content (on average +13% response rate for engaging and highly engaging).

• High school teachers and middle school teachers had similar ratings of the workshop

• Compared to the Elementary school teachers, principals had a more favorable opinion about the structure but a less favorable opinion about the content.





Teaching & Learning Framework: Workshop 2

• The second Teaching & Learning Workshop showed participants how to apply the Teaching and Learning Framework and rubric to self review and lesson study.

• 239 feedback forms inputted to date with approximately 400 remaining




Feedback on Workshop 2 Structure• Overall, over 90% of the participants Agreed or Strongly Agreed that

• The sessions were well organized• Activities were well-paced and appropriately varied• Presenters had strong knowledge• Colleagues were interested in the content

• Overall, 82-88% of participants Agreed or Strongly Agreed that• They learned useful information about the Self-Review process• They have a clear understanding of how the session relates to their

SIG work back at school• They learned useful information about the Lesson Design Template• They were interested in the content that was discussed in the

workshop





Feedback on Workshop 2 Content

• Reflecting on My Instruction and Completing the Self Review were most engaging (77% selecting highly engaging or engaging)

• Reviewing the TLF, Examining a Sample Lesson Plan Using the Lesson Design Template, and Applying the Lesson Design Template to My Own Content were the least engaging (71%, 71%, and 68% selecting highly engaging or engaging, respectively)





Feedback on using the Lesson Design Template when preparing for a formal classroom observation by administrator or another

teacher



Benefits Frequency of Responses

Considers all components of teaching, very detailed 38Can be used as a guide to organize lesson 22Pushes evaluator to look at various aspects; lets teacher know what's being observed 14

Generates more knowledge and understanding of the lesson 9Ensures clear expectations 9Demonstrates preparedness 8Promotes self reflection and thoughtfulness 7Uses a common language for greater uniformity and consistency 7Considers students' needs 5Generally beneficial 4




teacher



Concerns Frequency of Responses

Too much time, too long, too detailed 99Difficult to use, complex wording, confusing format 15Observers not having enough training or understanding of context 11Meeting every category in the template 8Stifles creativity, flexibility 5Subject to bias 5Challenge of students not reflected in template 4Too difficult for daily planning or multiple preps 4Teacher resistance 3




teacher



Suggestions Frequency of Responses

Use design template for a unit instead of lesson 3Use design template in grade level planning 2Reformat template so it is easier to use 2Do not use design template judgmentally 1Need extra prep time in order to use design template 1



Teaching & Learning Framework: Book Study

• Participants formed book study groups to read and discuss Charlotte Danielson’s Enhancing Professional Practice, culminating in written reflections about the Teaching and Learning Framework and the SIG educator effectiveness work in general.

• 631 people participated of which 100 reflections were sampled





• Predominantly, participants felt that the Teaching and Learning Framework (TLF) will benefit teachers as a roadmap and guide for reflection and self review.

• Participants also felt that the TLF will promote collaboration between colleagues by providing a common language and clear expectations.

• Given the complexity and size of the TLF, participants were concerned about the implementation and whether the district will provide enough time and support to teachers and administrators.







Categories/Themes from Book Reflections Frequency of Responses

Roadmap, guide, an opportunity to reflect 35Concerned about implementation 21Encourages collaboration within school 18Common language, clear expectations 17Framework is detailed and thorough 10Useful to teachers of all levels 8Need time to implement 7Need support to implement 6Other factors not considered in framework (student, environment, community, etc.) 5

Framework susceptible to subjectivity and bias 5


General comments about the TLF, evaluations, and educator effectiveness



…being a BTSA support provider...I wonder if BTSA will eventually take on this framework.[The framework] is up to date on current trends and beliefs.Danielson identifies the need for a uniform evaluation tool used across education programs, school districts, and school sites. I found this to be a very salient point. In my five years of teaching I have been evaluated under four different frameworks. What evaluators were looking for was often unclear. Having a single framework would have simplified my work as a beginning teacher. However, I found this framework to be very cumbersome...For the framework to be a useful evaluation tool, evaluators need to provide an easily readable document with clearly identified areas of focus. I like the idea of a framework for teaching. I believe that the BTSA framework (adopted by the State) would be helpful to novice teachers and teachers that have been at it for a while. I think the big concern of teachers is the use of the framework as a summative and formative evaluation that will affect people’s incomes and positions at work. Personally, I was inspired by the descriptions of the distinguished teaching and paid special attention to the information from chapter 4 where details about demonstration of effectiveness. I aspire to the highest levels of performance in my profession and was glad to see that I have been involved in many of the actions that were listed in that chapter. The framework is also independent of any particular teaching methodology. There are many options for the teacher. It is up to the teacher to find out what works best in the classroom.

There should not by any secrets in what we are trying to teach the students and how we can address anything that is not working

Throughout this experience, I have come to value the ideas of my colleagues and I have begun to allow for more personal reflection. The only concern I have pertains to the actual implementation of the LAUSD framework.

We learn not from our experience but from our thinking about the experience. It is the thinking that matters.

Looking back through Danielson's book, I would have to agree with her that [the framework] could be used to guide professional growth as well as act as a rubric for evaluative purposes. As with all fair assessment processes and evaluative measures, the tool used must be provided upfront...the biggest challenge apparent to me is teacher buy-in. The overall framework has many uses. It is not only useful to new teachers, but to veteran teachers who are looking to improve their overall teaching. It is basically a road map that guides us through our teaching journey.


General comments about the benefits of the TLF



If an entire staff can become immersed in this level of practice and commitment to improve, then the results that they produce would drive the continuing desire to grow and create a cycle of success that could be maintained. LAUSD has finally recognized the need for a Framework and the importance of it for teachers. The Teacher effectiveness framework is essential in incorporating a positive working environment at school sites.

The levels of performance will structure profession dialogue as teachers collaborate on what level they are at and discuss strategies to help themselves and each other reach the higher levels. Some teachers will possibly excel at different categories. This will be beneficial as they collaborate as they can share their strengths and weaknesses. These are invaluable for personal reflection as they are easy to understand and state implicitly what needs to be done to raise yourself to the next level.

The most powerful use of the framework was for reflection and self assessment.These frameworks can help evaluators in honestly and objectively evaluating new teachers and more experienced staff. While personality issues and politics sometimes interfere with objective evaluations, it is hoped that evaluators will take into consideration the overwhelmingly stressful and demanding scenarios of struggling new teachers. They need assistance with many of the domains covered in chapter 4. They need support in many ways, and timely and helpful, not judgmental feedback on their performance in the classroom. While I am not performing at an optimal level in all that I think I could or should, I can see where and can compare my current standing against my desired level of performance should be aimed. This is an opportunity that I am thankful for. At this stage of my career, the knowledge and information from this reading, makes for a timely reminder of where and what to set for myself as goals for the sake of improvement.the framework for teaching can be used for supervision and evaluation; it is valuable because it provides a common language and clear expectations on what good teaching isAs an evaluative tool, I felt that the framework is a good to benefit both teachers and administrators. The book does a good job of putting forth a common language for communication among stakeholders in the typical school system. While it could be seen as an attempt to put more pressure on teachers in a very austere environment, the truth is that it places the educational system on the path towards recovery. One cannot say enough about what this portends for the future of higher education and the quality of tomorrow’s workforce on the long run.If the Framework can be applied effectively, then it seems as if it will be able to convey not only to administrators and other teachers, but to the community, all of the effort that teachers put into their profession in order to become effective teachers. using the protocols of the pre and post observation worksheets will help new teachers understand what we are looking for. Basically, information from the framework means that there are no mysteries or secrets. We agree that Danielson’s Evaluation system is designed to promote conversations, feedbacks, and will allow the teachers to self reflect and self assess their practice. The teacher’s engagement in the process is meaningful, beneficial, and will promote teacher learning and professional growth. The rubric designed for each domain and component allows the teachers to self assess and evaluate and also since each component is divided to different elements it can easily point to the area(s) of need for growth and improvement.

While reading the book, I felt myself reflecting on my own teaching process, to try and figure out where I fit into the different rubrics for evaluations. After finishing the book, I found myself more aware of areas that I am successful in and could be considered proficient in, but more importantly I found areas that could use improvement. Through reading this book and the rubrics, not only have I found areas to improve in, but also I have figured out strategies to improve my own teachings.

…as a guide, the Framework will certainly make teaching, assessment, reflection, self-assessment, and communication part of the tools that all educators can use to improve their teaching and provide a better learning environment for their students.


Concerns with the TLF and teacher evaluation



I fear that the framework will be another fad that we are studying, but will never get to implement.

I have found that this book has no earth shattering information. It is what every professional teaching book says is "good teaching" in a rubric form. I think what has been done with it is dangerous. I[t] will be used to classify many teachers as "unsatisfactory" by people who don't understand teaching.Teachers are concerned with the lack of flexibility presented by the district in regards to having one way to show that students are aware of grade level expectations (i.e. posting objects on the board daily), pacing plans, and 5 to 10 minute classroom visitations by administrators that are used to evaluate teacher effectivenessThe supposed “gotcha” safety feature of the framework doesn’t exist. An instructor can still get their fingers chopped off by an accidental inference from an observer. The framework merely asks that observers infer within according to the variables and context of a classroom, which to my mind requires that an observer can understand and appreciate the context and variables in the first place. That requires experience in the classroom.There are two basic flaws. Observation of typical behaviors during an evaluation, and second [whether] the rater is objective and rational in defining what is observed.To adopt this rubric in the STULL process would amount to maintaining the status quo, for the conversation between teacher and administrator would remain basically the same: a conversation about meeting the standards of the teaching profession. Even though the Danielson framework may make some contributions in that regard, it will still fall within the purview of the conversation, nothing more...However, if teachers were allowed to select which aspect of their practice they want to develop and choose the process by which to improve, many would select [the] video option.

As the discussion ended, there were some ideas that Danielson’s framework left us questionable. For example, some respondents indicated that collaboration from peers, within and across grades and subjects, is a critical factor of planning was absent from the framework. The designing of student assessment with district and school wide common assessment needs more justification. Unlike the LAUSD framework Danielson’s components and elements were very explicable. The level of performance indicators is great to include because it explains the particulars of how effective are you and what can be improved to meet the criteria of effective teaching.


General comments about the implementation of the TLF and teacher evaluation



…a team of individuals should assess the teachers at not only 1 time but many times through the year…

It is our hope that an expanded selection process of those charged with evaluating the specialists’ effectiveness, which would include educational leaders, as well as members of the specialist positions being evaluated and other educational stakeholders, will accompany the adoption of these frameworks.

Openness and honesty would be necessary, but peer evaluations would add depth to the evaluations. Among learning community members that are self-selected, trust can help us to accept constructive criticism.

We would like to know if supervisors or administrators are taught how to tactfully deal with teachers of different age, culture, sexual preference, etc? There is a high turnover of administrators at our school and they come with a wide variety of operational polices, beliefs, and values. The same goes true with teachers and thus there is a melting pot of professionals all attempting to achieve the school mission and improve student test scores. Is there a better method of mating a supervisor with a teacher other than rating them by batches based on their last name? Could it be possible that an administrator that has more experience and expertise rate a more seasoned teacher? Would it be more feasible to have novice teachers rated by more senior or experienced teachers and have those senior teachers report to an administrator as the reviewing authority and sign off on the evaluations? Could we have a two-tiered rating scheme instead of just one? This would definitely cut down the workload on an already overworked administrator and empower the more seasoned teacher to take more ownership and responsibility of the school he or she teaches in. Will there be opportunities where teachers can visit classrooms within the school campus and or other school campuses to experience and observe other classrooms?


Academic Growth over Time

• Participants were introduced to the Academic Growth over Time School Report through a webinar and then asked to analyze their school report.

• 155 small group responses, representing 562 participants




What are the benefits of using Academic Growth over Time (AGT) as a measure of teacher effectiveness?

Benefits Frequency of Responses

Gives teachers motivation and direction to improve 77

Can be used as benchmark and comparison to other schools 28

Focus on improvement and not just achievement 21

Provides subgroup data 18

Brings more focus on using data to improve student outcomes 12

Controls for external factors 11

No benefits/relevance to student outcomes 2

Source: SIG Academic Growth over Time: School Results small group activity 2011; N= 155 small groups; Blank answers excluded.




What concerns do you have about AGT?

Concerns Frequency of Responses

External factors/variables not factored into score 53Incomplete/limited results by subject, grade level, subgroups 36Accuracy, Validity, Reliability of AGT 18Limitations of source data (heavy reliance on CSTs) 13Doesn't indicate how to improve 9Difficult to understand 4A single measure of performance 3Doesn't involve all stakeholders 3





What questions do you have about AGT?

Questions Frequency of Responses

How will AGT be used? 41How was AGT score created? 38Can we have more training? 12





Comments about AGT“This data is certainly helpful in monitoring the effects we are having in getting our students caught up to other schools in LAUSD. We like that this “does not mean a lowering of expectations for any grouping of students addressed by a control variable”. This gives a very clear picture of whether the school is doing its job of getting students up to speed despite some environmental challenges. Without this data, we think it becomes very easy to blame everything on our kids’ home life and pass the responsibility from ourselves. This data is at least meant to isolate our effect on our students. The question then becomes, how effective and accurate is it at isolating the data. Our main concern is that measures of growth are still based solely on CST scores. We have grave concerns about the validity of the results of this one test. Our concerns about the CST (or any high stakes testing model) are numerous and echoed throughout academia. We acknowledge that this is the situation that we are confronted with and AGT is a superior lens through which to view the results of the test than just comparing teacher effectiveness on a completely level field. Our other concern is the effectiveness of the isolation of data as mentioned in question #1. We see the 9 control variables presented but don’t know yet how well they work. We are especially concerned because students advance to the next grade level, whether or not the students’ ability advances from year to year. If a student has not adequately learned addition one year, how could they be expected to “grow” under 8th grade standards of algebra? Perhaps the student advanced 2 grade levels under a teacher, learning addition, subtraction, multiplication, division, etc. but did not get to algebra. That school is now being judged as not adequately meeting the standards of growth for a student who probably actually advanced two years under their teacher...”



Comments about AGT“AGT is an invaluable tool to utilize in assessing the effectiveness of schools as it is useful in tracking the progress or growth of individual students over time, rather than the percentage of students that meet an absolute target or standard. This allows for true comparisons of effectiveness. Additionally, AGT enables us to identify, study, and share the practices of schools who are achieving effective results. These results are based upon a common standardized measure of growth, so that when we compare growth in one school to another, we know that we are using a term with one definition.

Although AGT models are obviously beneficial, they also represent a narrow way to assess the effectiveness of educators and schools. The results should be considered within the context of other metrics – both qualitative and quantitative. Like all measures related to student learning, this approach is subject to limitations, which is why (i) it is important to look at multiple indicators and results and (ii) this approach should be used as one amongst multiple measures. Further, the effectiveness of this analysis is lost if it is not linked to ways to improve practice or to learn from those with clear positive effects on student learning.

Further, AGT models are complex statistical models, which can be easily misread or misinterpreted. Moreover, there are factors external to our schools and classrooms that can impact student learning rates, that may not be reflected in AGT models. Additionally, it only evaluates how students perform on a standardized test, which measures only a small part of what happens in a classroom during the academic year.

Will LAUSD be offering Learning Zone classes to supplement the AGT primer?

The AGT model follows the premise that students can be fairly evaluated by how they perform on a standardized test, but such tests only measure a fraction of what happens in a classroom during the academic year. Although the AGT model is constructed to evaluate a school's growth, it may well be a way to link student scores to teacher evaluations.”



Comments about AGT“Our group thinks that the data may be helpful for us as teachers because it allows for us to examine our student’s data and make predictions on their expected growth for that academic year. Furthermore, if we examine our students more closely we should be able to compare our student’s actual achievement growth with their predicted achievement growth. Moreover, the school can benefit from the AGT because the results are suppose account for the environmental variables therefore making the AGT more reliable, fair, and accurate when calculating student growth.

The concerns that we have about AGT is how does it truly differs from any other testing model or tool used. It appears that many teachers when analyzing student data utilize the student’s previous performance on standardized test. This means we as teachers are already taking into account the student’s previous performance or academic ability. A second concern that is relevant to our group which was not mentioned in environmental variables are the students previous teachers or schools that have contributed towards the students success or failure in achieving a predicted growth. For example, a student who attends a low API school, and had received their instruction predominantly from substitute teachers versus a student who doesn’t have these environmental variables, how do they account for these factors when creating a fair result. The video uses two gardeners as an analogy to demonstrate AGT and it states the difference between Gardner A and Gardner B are their strategies, however the strategies were never fully discussed what was discussed was the uncontrolled variables such as soil richness, rain, etc. Factors which neither Gardner can truly control. Whereas the strategies that were not examined in regards to how often the soil was turned, cutting dead leaves off, talking to our tree, etc. AGT didn’t seem to focus on the controlled strategies, which is what we as classroom teachers provide our students.

A wish that has not been included is actually walking us through the AGT process with real students and teachers or a panel. A visual presentation could have been more useful. Second, making the material more personal to my particular department in which I teach in such as Special Education, Math, Science, ELA, and History. I don’t think AGT is explained as easily as it could be for your teacher who just wants to know or see how to apply this to their work.”



Total Effectiveness Result:Teacher Results

• Participants were asked to read articles about teacher evaluation systems and then determine the best weighting for each measure of teacher effectiveness towards a Total Effectiveness Result.





Total Effectiveness Result:Teacher Results

• Greatest support for observations (esp. by administrators) followed by student outcomes (esp. at individual level)

• On average, observations weighted 55% of total effectiveness and student outcomes 28%

• For nearly all measures, the distribution of teacher responses reflects the full range, from “Not Count” to 100% of the weighting– Observations had highest weightings with lowest indication of

“feedback-only” or “not counting”– Stakeholder Input received low weightings, with many teachers

indicating that the surveys should be feedback-only or not count

• Teachers’ qualitative comments illuminated both benefits and concerns of particular measures and their inclusion in the total effectiveness result Click for last

viewed slideClick for

Activities List


On average, teachers indicated observations should count for the majority (55%) with student outcomes

(28%) accounting for the next largest component

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups; blank answers excluded and answers adjusted to total 100%.




Overall, most teachers supported the inclusion of observation and student outcomes measures; fewer

supported other measures

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups; blank answers excluded.




Distribution of teacher perspectives across measures shows variance

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups, blank answers excluded.




Teachers’ comments illuminate benefits and concerns of the measures under consideration


Overall • Provide balance, stability, and accuracy

• Provide more avenues for feedback and support

• Reduce isolation

• N/A

Administrator Observations

• Experienced leaders who manage instruction in the school

• May not have subject matter or instructional expertise

Teacher Leader Observations

• Bring content and pedagogical expertise

• Come from the classroom with understanding of it

• If from same building, could create tension

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups.

EXAMPLES




Teachers’ comments illuminate benefits and concerns of the measures under consideration (cont.)


Individual Academic Growth Over Time

• Provides objective measure of student learning

• Supports educators to seek best practice to improve growth

• Could focus efforts too narrowly on test prep

• Test scores are due to many variables

School-Level Academic Growth Over Time

• Supports collaboration within school

• Can be outside of individual teacher’s control

Contributions to School Community

• Supports collaboration • Depends on definition of measure

Student Survey • Provides opportunity for learners to weigh-in

• Subjective• Could conflict with discipline

and grading efforts

Parent Survey • Can provide helpful feedback

• Subjective• May not know classroom

environment or instruction

Source: SIG Total Effectiveness Results: Teacher Results small group activity 2011; N= 133 small groups.

EXAMPLES




More teachers thought more measures should count when giving small group feedback as compared to individual feedback

Source: SIG Total Effectiveness Results: Teacher Results small group and individual activity 2011; 133 small groups; N= 333 individuals.

INDIVIDUALMeasure % Count % Feedback Only % Not Count % blank

Administrator Observation 80% 15% 1% 4%Teacher Leader Observation 62% 32% 2% 3%Parent/Guardian Survey 28% 56% 12% 4%Student Survey 31% 51% 14% 4%Contributions to School Community 42% 33% 21% 4%Individual Academic Growth over Time 66% 24% 6% 4%School Level Academic Growth over Time 53% 34% 8% 5%

DIFFERENCE (IND - SG)Measure % Count % Feedback Only % Not Count % blank

Administrator Observation -4% 6% 0% -1%Teacher Leader Observation -7% 9% 0% -2%Parent/Guardian Survey -8% 5% 4% -1%Student Survey -10% 3% 8% -1%Contributions to School Community -4% 6% 0% -2%Individual Academic Growth over Time -1% 7% -2% -4%School Level Academic Growth over Time 1% 9% -8% -2%

SMALL GROUPMeasure % Count % Feedback Only % Not Count % blank

Administrator Observation 85% 9% 1% 5%Teacher Leader Observation 69% 23% 2% 5%Parent/Guardian Survey 36% 51% 8% 5%Student Survey 41% 48% 6% 5%Contributions to School Community 46% 27% 21% 6%Individual Academic Growth over Time 68% 17% 8% 8%School Level Academic Growth over Time 52% 25% 16% 8%


Variation in weightings for teachers* between individual feedback and small group feedback concentrated on surveys

*Teachers with AGT results.Source: SIG Total Effectiveness Results: Teacher Results small group and individual activity 2011; 133 small groups; N= 333 individuals.

INDIVIDUALMeasure 0% 1% <-> 10% 11% <-> 25% 26% <-> 50% >50%

Administrator Observation 13% 10% 20% 37% 20%Teacher Leader Observation 33% 11% 21% 30% 6%Parent/Guardian Survey 70% 21% 7% 2% 0%Student Survey 66% 21% 8% 4% 1%Contributions to School Community 55% 19% 20% 6% 1%Individual Academic Growth over Time 30% 16% 30% 23% 0%School Level Academic Growth over Time 44% 19% 24% 11% 1%

SMALL GROUPMeasure 0% 1% <-> 10% 11% <-> 25% 26% <-> 50% >50%

Administrator Observation 11% 7% 29% 43% 11%Teacher Leader Observation 25% 16% 28% 29% 2%Parent/Guardian Survey 55% 34% 9% 1% 2%Student Survey 52% 35% 11% 1% 2%Contributions to School Community 50% 27% 18% 4% 2%Individual Academic Growth over Time 25% 25% 31% 17% 3%School Level Academic Growth over Time 39% 30% 24% 6% 1%

DIFFERENCE (IND - SG)Measure 0% 1% <-> 10% 11% <-> 25% 26% <-> 50% >50%

Administrator Observation 2% 3% -8% -6% 9%Teacher Leader Observation 8% -5% -7% 1% 3%Parent/Guardian Survey 16% -12% -2% 1% -2%Student Survey 15% -14% -3% 3% -1%Contributions to School Community 5% -8% 2% 2% -1%Individual Academic Growth over Time 5% -8% -1% 6% -3%School Level Academic Growth over Time 5% -11% 0% 5% 1%


Differentiated Compensation

• Participants were asked to select and read articles about differentiated compensation and incentives, and then suggest the type of incentives they would like to have, if any





“Monetary incentives” and “career pathways” were the most frequently proposed incentives

Type of Incentive Frequency Of Response

Monetary 128

Career Pathways 24

Recognition/Awards 13

Equipment/Supplies/School Programs 8

Education 5

Perks (parking, class coverage, conference period) 5

Time Off/Sabbatical 4

Subsidized Home Loans 2

No Incentives 2

Source: SIG Differentiated Compensation small group activity 2011; 132 small groups; blank answers excluded.




“Academic Growth” was the most frequently mentioned standard of measurement for

receiving incentives

Standard of Measurement for Receiving Incentives Frequency of Response

Growth in Student Achievement 73

Observation of Practice/Stakeholder Feedback 31

Student Achievement 27

Level of Professional Training 20

Added Responsibilities 17

Combat Pay 15

Attendance (teacher and/or student) 9





Most respondents preferredat least 1 incentive

Number of Incentives Mentioned Frequency of Response

0 incentives 12

1 incentive 55

2 incentives 31

3 incentives 10

4 incentives 24





A majority of respondents prefer fixed monetary incentives, such as bonuses

Type of Monetary Incentive Frequency of Response

Fixed Amount 71

Percentage of Salary 23

Scaled to Performance 13





Individual and school incentives were weighted more heavily than group incentives, with individual and

school incentives sharing equal weight

Type of IncentiveMean Weighting as a

Percent of Total Incentive Package

Median Weighting as a Percent of Total

Incentive Package

Individual Incentives 41% 40%

Group Incentives 20% 20%

School-wide Incentives 39% 25%





Variance shown in the relative amount of compensation going towards individual, group, and/or school incentives





Respondents tended to weight individual incentives higher than group and school incentives

Type of Incentive Number of responses that weight the following incentives higher than the others

Individual Incentives 63

Group Incentives 6

School-wide Incentives 40

Evenly Distributed 21

No Incentives 2





Reasons for usingindividual incentives

• Teachers have the greatest impact on student outcomes

• Motivation and retention of teachers• Group rewards may dilute incentive• Cannot control what happens in other

classrooms; don’t want to be penalized by the performance of ineffective teachers





Reasons for usinggroup (team/department) incentives

• Promotes team building• Share common goals





Reasons for usingschool-wide incentives

• Promotes collaboration and team building• Avoids division and envy• Prevents favoritism and cheating• Everyone at the school site contributes to student

outcomes, not just teachers• Not all teachers have the same type of students;

some students are harder to teach than others• Some teachers do not have test scores





Should everyone have a chance at getting the reward (greater chance, less amount), or should there be a scaled system that gives more money for making a greater difference in student outcomes (less chance, greater amount)?

Shared or Competitive Awards? Frequency Of Responses

Top performers awarded more 71

Everyone has a chance to receive reward based on growth targets 37

Everyone shares the same amount whether based on a sliding scale or reaching target 25

No monetary awards 5





Should the incentive system use just CSTs and graduation rates as indicators of

achievement, or other indicators as well?

Measures of Achievement Frequency of Responses

Multiple indicators 85CSTs 26Graduation rates 9No CSTs 8CAHSEE 5Grades 5Student attendance 5Growth in student achievement 5Periodic assessments 3Student behavior 2PSAT 2





Stakeholder Survey Feedback: Small Group Activity

• Participants gave feedback on the Stakeholder Surveys (Student and Staff) that were piloted at the SIG schools

• Additional feedback on the surveys has been collected by the researchers, who will present their findings at a later time.





General comments about student survey

General Comments Frequency

Lengthy, needs to be shortened or broken into smaller pieces 42

Clear, easy to follow, organized 41

Good coverage of topics 31

Useful 17

Subject to bias 12

Needs open ended questions 6

Too many answer choices 4

no names on forms 4

Feedback only 3

Need questions about how students feel about themselves 3

Source: SIG Stakeholder Survey small group activity 2011; N= 111 small groups; Blank answers excluded.




General comments about student survey (cont.)

General Comments Frequency

Needs to take into account different instructional methods 2

No accommodations for sped students 2

more departments/teachers need survey 2

Students may feel uncomfortable, confidentiality 2

Questions about administrators 2

some questions not pertinent to grade level 1

need questions about impact of environment on students 1

Spanish version 1

Parental involvement 1

Online administration 1





Recommendations for improving student survey

Recommendations Frequency

Include open-ended questions 6

Reduce the number of answer choices 4

Keep names off of forms/administer survey online 5

Include questions about how students feel about themselves 3

Need to take into account different instructional methods 2

Make accommodations for special education students 2

Survey more departments and teachers 2

Include questions about administrators 2

Make questions more pertinent to grade level 1

Include questions about impact of environment on students 1

Create a Spanish version 1

Include questions about parental involvement 1





Listening Sessions

HILLARY JOHNSON, ED.D. Educational & Instructional Consulting

TO: Nader Delnavaz, Noah Bookman FROM: Hillary Johnson CC: Drew Furedi, Paul Hsu, Sarah Figueroa, Joe Doctor RE: SIG Educator Effectiveness Listening Sessions – Key Findings DATE: June 21, 2011

OVERVIEW

At each school participating in the School Improvement Grant (SIG), I conducted one or more “listening sessions,” or small group conversations, in which the primary purpose is to listen. These groups served as an opportunity to gather in-person feedback on the Educator Effectiveness (EE) portion of the SIG grant as a supplement to the feedback submitted online. The sessions were optional and volunteer teachers’ participation was not counted towards the 50-hours of professional development required as part of the SIG grant. The listening sessions had three purposes:

1. Provide opportunity for teachers to talk & Supporting All Employees (SAE) representative to listen 2. Increase the degree of trust regarding SAE and the new frameworks process 3. Gather additional feedback regarding SAE tools.

DATA COLLECTION

ARRANGING THE SESSIONS During my March 2011 visits to each of the SIG schools with Paul Hsu, we discussed the possibilities of holding listening sessions at their school sites with school principals and in most cases, chapter chairs. At that point, we conceived of the listening sessions as drop-in, similar to open office hours. Prior to the listening sessions, we redefined the purpose of listening sessions to something akin to focus groups. In order to schedule sessions, I included the following in my email communications to principals and chapter chairs:

SIG Listening Sessions

Who? Teachers who wish to participate. It’s entirely optional.

What? An opportunity to provide in-person feedback.

How? I will guide the sessions by asking questions to elicit teachers’ hopes, fears and concerns about the Teacher/School Leader Effectiveness work. In addition, I may share with them some trends that have emerged in the feedback submitted by teachers across the SIG schools, and ask for their opinion, or to elaborate on the idea.

To access the full report, end slideshow (if you haven’t already) and double-click the image to the right



copyright © los angeles unified school district 2011 copyright © tulsa public schools 2011...

Documents

individual feedback

teachers feedback

support system feedback

school community contributions

effective teaching best

school year

teacher support

teacher development