Gathering Feedback for TeachingCombining High-Quality Observations with

Student Surveys and Achievement Gains

Closing the effectiveness gap

2

Progressing beyond “The Plateau”

3

Ensuring Reliable & Trustworthy ObservationsEnsuring Reliable & Trustworthy Observations

The First of Three QuestionsHow many distinct levels of teacher performance do you think an evaluation system should recognize?

5

The Second of Three QuestionsWhat performance levels would you assign to what fraction of your district’s teachers on the following competencies?A.Classroom Management (time, behavior, materials)

B.Goals & Tasks (clear, appropriate, rigorous, interesting)

C.Supporting Student Understanding (content depth, feedback, questioning and discussion, instructional dialogue)

6

The Third of Three QuestionsHow closely associated are teaching behaviors with student outcomes (academic growth over time)?

7

Stud

ent P

erfo

rman

ce

Teacher’s Classroom Observation Score

highly

moderately

weakly

8

Validation

Teachers with Higher Observation Scores Had Students Who Learned More

Observation Score DistributionsFramework for Teaching

9

PLATO Prime, CLASS and MQI Lite

10

Observation Score Distributions UTeach Observation Protocol

11

12

13

14

Measure Predictive power ReliabilityPotential for

Diagnostic Insight

Value-added

Student survey

Observation

Measures have different strengths …and weaknesses

15

16

The Importance of Multiple Measures

Compared to What?

Compared to MA Degrees and Years of Experience, the Combined Measure Identifies Larger Differences

17

… on state tests

Compared to What?

…and on low stakes assessments

18

Compared to What?

…as well as on student-reported outcomes.

19

The Measures of Effective Teaching ProjectThe Measures of Effective Teaching Project

Two school years: 2009–10 and 2010–11 >100,000 students Grades 4–8: ELA and Math High School: ELA I, Algebra I and Biology

Participating Teachers

Research PartnersResearch Partners

22

The MET Project is ultimately a research project. Nonetheless, participants frequently tell us they have grown professionally as a result of their involvement. Below is a sampling of comments we received.

From Teachers:•“The video-taping is what really drew me in, I wanted to see not only what I’m doing but what are my students doing. I thought I had a pretty good grasp of what I was doing as a teacher, but it is eye opening … I honestly felt like this is one of the best things that I have ever done to help me grow professionally. And my kids really benefited from it, so it was very exciting.”

•"With the videos, you get to see yourself in a different way. Actually you never really get to see yourself until you see a video of yourself. I changed immediately certain things that I did that I didn't like.”

•“I realized I learned more about who I actually was as a teacher by looking at the video. I learned of the things that I do that I think that I’m great at I was not so great at after all.”

•“Even the things I did well, I thought, ok that's pretty good, why do I do that, and where could I put that to make it go farther. So it was a two-way road, seeing what you do well, and seeing the things that have become habits that you don't even think about anymore."From Raters:•“Being a rater has been a positive experience for me. I find myself ‘watching’ my own teaching more and am more aware of the things I should be doing more of in my classroom.”•“I have to say, that as a teacher, even the training has helped me refine my work in the classroom. How wonderful!”•“I have loved observing teachers, reflecting on my own teaching and that of the teachers teaching in my school.”

What the Participants Said….

MET Extension: A Library of Teaching Practice

Additional Data Collection•Subset of 360 MET Teachers (disproportionately highly effective)•50 lessons taped (18,000 total lessons)•100% teacher & parental consent (allowing for broader public use)•Cheaper cameras with potential for scale

Library of Practice•Searchable database (tagged by Common Core standards, teaching practices, etc.)•Tagging to be done in partnership with schools of education (w/ teachers-in-training)

Potential Uses:•Rater training, certification, and calibration•School districts – professional development•Teacher training institutions – teaching methods•Observation instrument developers – validation of new & existing tools

MET Logical SequenceMET Logical Sequence

24

Measures predict

Measures predict

Measures fairly reflect teacherMeasures fairly reflect teacher

Measures combine

Measures combine

Effective Teaching Index

Effective Teaching Index

Teaching Effectiveness Dashboard

Teaching Effectiveness Dashboard

Research Use

Measures reliable

Measures reliable

Measures improve

effectiveness

Measures improve

effectiveness

Measures stable under pressure

Measures stable under pressure

Measures communicated

effectively

Measures communicated

effectively

Validation EngineValidation Engine

System picks observation rubric & trains raters

Raters score MET videos of instruction

Software provides analysis of:•Rater consistency•Rubric’s relation to student learning

26

Four Steps

Four Steps to High-Quality Classroom Observations

27

Step 1: Define ExpectationsFramework for Teaching (Danielson)

28

Four Steps

Uns

atisf

acto

ry

Yes/no Questions, posed in rapid succession, teacher asks all questions, same few students participate.

Basic

Some questions ask for student explanations, uneven attempts to engage all students.

Profi

cien

t

Most questions ask for explanation, discussion develops/teacher steps aside, all students participate.

Adva

nced

All questions high quality, students initiate some questions, students engage other students.

Step 2: Ensure Accuracy of Observers

29

Four Steps

Step 3: Monitor Reliability

30

Four Steps

31

Four Steps

Multiple ObservationsLeads to Higher Reliability

NOTES: The numbers inside each circle are estimates of the percentage of total variance in FFT observation scores attributable to consistent aspects of teachers’ practice when one to four lessons were observed, each by a different observer. The total area of each circle represents the total variance in scores. These

estimates are based on trained observers with no prior exposure to the teachers’ students, watching digital videos. Reliabilities will differ in practice. See the research paper, Table 11, for reliabilities of other instruments.

32

Students with Most Effective Teachers Learn More in School

Student Perceptions

33

CareCare

ControlControl

ClarifyClarify

ChallengeChallenge

CaptivateCaptivate

ConferConfer

TestPrepTestPrep

ConsolidateConsolidate

CareCare

• My teacher makes me feel that s/he really cares about me

• My teacher seems to know if something is bothering me

• My teacher really tries to understand how students feel about things

• My teacher makes me feel that s/he really cares about me

• My teacher seems to know if something is bothering me

• My teacher really tries to understand how students feel about things

ControlControl

• Students in this class treat the teacher with respect

• My classmates behave the way the teacher wants them to

• Our class stays busy and doesn’t waste time

• Students in this class treat the teacher with respect

• My classmates behave the way the teacher wants them to

• Our class stays busy and doesn’t waste time

ClarifyClarify

• If you don’t understand something , my teacher explains it a different way.

• My teacher knows when the class understands, and when we do not.

• My teacher has several good ways to explain each topic that we cover in the class.

• If you don’t understand something , my teacher explains it a different way.

• My teacher knows when the class understands, and when we do not.

• My teacher has several good ways to explain each topic that we cover in the class.

ChallengeChallenge

• My teacher asks students to explain more about the answers they give.

• My teacher doesn’t let people give up when the work gets hard.

• In this class, we learn to correct our mistakes.

• My teacher asks students to explain more about the answers they give.

• My teacher doesn’t let people give up when the work gets hard.

• In this class, we learn to correct our mistakes.

CaptivateCaptivate

• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class

• My teacher makes learning enjoyable• My teacher makes learning interesting• I like the way we learn in this class

ConferConfer

• My teacher wants us to share our thoughts• Students get to decide how activities are done in

this class

• My teacher wants us to share our thoughts• Students get to decide how activities are done in

this class

ConsolidateConsolidate

• My teacher takes the time to summarize what we learn each day

• The comments that I get on my work in this class help me understand how to improve

• My teacher takes the time to summarize what we learn each day

• The comments that I get on my work in this class help me understand how to improve

TestPrepTestPrep

• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of

time in our class

• I have learned a lot this year about [the state test]• Getting ready for [the state ] test takes a lot of

time in our class

Student Perceptions Top 5 Correlations

34

Survey StatementSurvey Statement CategoryCategoryRankRank

11

22

33

44

55

• Students in this class treat the teacher with respect• Students in this class treat the teacher with respect

• My classmates behave the way my teacher wants them to• My classmates behave the way my teacher wants them to

ControlControl

ControlControl

ControlControl• Our class stays busy and doesn’t waste time• Our class stays busy and doesn’t waste time

ChallengeChallenge• In this class, we learn a lot every day• In this class, we learn a lot every day

ChallengeChallenge• In this class, we learn to correct our mistakes• In this class, we learn to correct our mistakes

3333 • I have learned a lot this year about [the state test]• I have learned a lot this year about [the state test] Test PrepTest Prep

3434 • Getting ready for [the state test] takes a lot of time in our class• Getting ready for [the state test] takes a lot of time in our class Test PrepTest Prep

Dynamic Trio

Combining Measures Improved Reliabilityas well as Predictive Power

35

Note: For the equally weighted combination, we assigned a weight of .33 to each of the three measures. The criterion weights were chosen to maximize ability to predict a teacher’s value-added with other students. The next MET report will explore different weighting schemes.

FFT alone

Tripod alone

VA alone

Combined(Equal Weights)

Combined(Criterion Weights)

.05

.1.1

5.2

.25

Diff

ere

nce

in M

ath

VA

(T

op

25

% v

s. B

ott

om

25

%)

0 .1 .2 .3 .4 .5 .6 .7Reliability

Note: Table 16 of the research report. Reliability based on one course section, 2 observations.

The Reliability and Predictive Power of Measures of Teaching: