building expertise 2016 june 15-17, 2016 annual ...€¦ · –building leadership- district view/...

BUILDING EXPERTISE 2016Annual International Marzano Conference

June 15-17, 2016

©2016 Learning Sciences International 1

The Marzano Model: How Much Does

It Really Matter?

Dr. Lindsey Basileo and Dr. Robert Marzano

Learning Objectives

1. Understand how your district or school

compares to others who implement the

Marzano Teacher Evaluation Model

2. Understand the relationship between teacher

observation scores and value-added metrics

(VAM)

3. Understand which elements are more highly

correlated to VAM

iObservation User?

• Open Explorer Internet Browser

– Do NOT use Chrome

• Log into your iObservation Account to follow

along

• www.effectiveeducators.com/login/auth

• Organizational/Principals Users

– Reports

– Search for Dashboards

– Building Leadership- District View/ Building

Leadership

http://www.effectiveeducators.com/login/auth


June 15-17, 2016


iObservation User?

1. Organization/building

2. Start Date = 08/01/2014

3. End Date= 07/31/2015

4. Select the Domain 1 Forms that Apply to that School YearIf more than one version select V3 or V2

5. Select Domains 2, 3, 4 Forms

6. Compared Start Date = 08/01/2013

7. Compared End Date= 07/31/2014

8. Select the Domain 1 Forms that Apply to that School YearIf more than one version select V3 or V2

9. Select Domains 2, 3, 4 Forms for Comparison Date Range

10. Evaluative Option: Count toward annual evaluation

11. Click Create

BACKGROUND

iObservation™

Technology Platform

• Supports schools and educational initiatives in

– 11 countries

– 3 provinces in Canada

– 43 states in the United States

• Serves over 420 school districts

– 5,500 school buildings

– 311,000 teachers

– 4 million students

iObservation collects millions of records

each year on observations


June 15-17, 2016


iObservation Customers

Across the Nation

There are iObservation customers in 43 states

iObservation Customers

Across the Nation

• Florida has the largest

number of customers that

use iObservation

• 40% of the districts in

Florida implement the

Marzano Teacher

Evaluation Model

THE DATA


June 15-17, 2016


Observation Data

• Three years of data were analyzed in the

State of Florida to asses scoring trends

SCORING FREQUENCY

Evaluative Score Distribution By

Lesson Segment

The Lesson Segment, Addressing Content, was observed most

often in the 2014-15 and 2013-14 school years


June 15-17, 2016


Evaluative Distribution By

Design Question

Design Questions 2 and 5 were observed most often

Distribution of Elements Observed

Addressing Content

12%

11%

13%12%

7%6%

7%

5%

7%

3%

5%

2% 2%

4%

1% 1% 1% 1%

14%

13%

11%

10%9%

7%

5% 5% 5%

4% 4% 4%

2% 2% 2%1% 1% 1%

0%

2%

4%

6%

8%

10%

12%

14%

16%

2012-13 2013-14 2014-15

Florida Distribution

Addressing Content (Design Questions 2, 3 & 4)

Providing resources and guidance for cognitively complex tasks consistently had the highest score

Element Count Freq. Avg. Count Freq. Avg. Count Freq. Avg.

Helping Students Practice Skills, Strategies & Processes 73,949 12% 3.11 94,559 14% 3.12 82,979 14% 3.09

Reviewing Content 68,165 11% 3.04 89,049 13% 3.04 76,889 13% 3.00

Identifying Critical Content 82,705 13% 3.07 73,558 11% 3.09 69,098 11% 3.01

Chunking Content into Digestible Bites 77,543 12% 3.07 68,285 10% 3.11 60,982 10% 3.07

Organizing Students to Practice & Deepen Knowledge 43,541 7% 3.12 60,548 9% 3.13 55,825 9% 3.10

Helping Students Record & Represent Knowledge 41,036 6% 3.06 40,344 6% 3.09 43,578 7% 3.08

Organizing Students to Interact with New Content 47,620 7% 3.09 37,497 6% 3.13 33,696 5% 3.08

Helping Students Process New Content 35,238 5% 3.02 27,721 4% 3.03 32,038 5% 3.01

Previewing New Content 47,136 7% 3.02 39,136 6% 3.04 31,637 5% 3.02

Helping Students Examine Similarities & Differences 17,558 3% 3.09 25,138 4% 3.09 23,541 4% 3.09

Helping Students Elaborate on New Content 32,630 5% 3.06 28,334 4% 3.06 23,310 4% 3.06

Helping Students Examine Their Reasoning 12,868 2% 3.02 18,607 3% 3.03 22,650 4% 3.07

Helping Students Reflect on Learning 14,778 2% 3.00 14,080 2% 3.02 12,544 2% 3.06

Providing Resources & Guidance for Cognitively Complex

Tasks22,474 4% 3.22 21,731 3% 3.27 12,237 2% 3.29

Helping Students Revise Knowledge 6,155 1% 3.07 9,756 1% 3.05 12,156 2% 3.08

Engaging Students in Cognitively Complex Tasks 6,604 1% 3.25 9,533 1% 3.21 7,104 1% 3.27

Using Homework 6,577 1% 3.05 7,876 1% 3.09 6,819 1% 3.10

Organizing Students for Cognitively Complex Tasks 5,135 1% 3.17 7,107 1% 3.21 6,350 1% 3.27


2012-13


2013-14


2014-15


June 15-17, 2016


Turn & Talk:

What are some similarities and

differences that you see?

OBSERVER

PERFORMANCE

Observation Count for Teachers in

Domain 1

The majority of teachers received 3 observations in the 2014-15 school year


June 15-17, 2016


Number of Scored Elements for

Teachers ─ Domain 1

The majority of teachers had 20 elements scored in the 2014-15 school year

Turn & Talk:

Do you have the same number of

observations and/or scored

elements?

SCORE DISTRIBUTION


June 15-17, 2016


Score Distribution in Domain 1

The majority of scores were at the applying level for all three years

Average Teacher Score in Domain 1

The average score for teachers was 3.07 in 2012-13, 3.10 in 2013-14 and

3.09 in 2014-15

14,529

4 0 1 5 6 5 7 8 16 21 43 23 40 73 74 97104

143184

231370

395458 770

9081,228

1,8152,188

3,731

5,587

10,075

7,308

5,076

4,502

3,3352,717

2,193

1,658

1,5291,287

1,197

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

0.0

0

0.1

0

0.2

0

0.3

0

0.4

0

0.5

0

0.6

0

0.7

0

0.8

0

0.9

0

1.0

0

1.1

0

1.2

0

1.3

0

1.4

0

1.5

0

1.6

0

1.7

0

1.8

0

1.9

0

2.0

0

2.1

0

2.2

0

2.3

0

2.4

0

2.5

0

2.6

0

2.7

0

2.8

0

2.9

0

3.0

0

3.1

0

3.2

0

3.3

0

3.4

0

3.5

0

3.6

0

3.7

0

3.8

0

3.9

0

4.0

0

2012-13 2013-14 2014-15

Turn & Talk:

How does your score

distribution and average

teacher score compare?


June 15-17, 2016


DO TEACHERS WITH

HIGHER OBSERVATION

SCORES HAVE HIGHER

VAM SCORES?

Matching Procedure

• Teachers were matched to the VAM scores

by first and last name, building and district

School Year% of Teachers

with VAM

Count of

Teachers with

VAM

Total

Teachers

2014-15 20.6% 12,248 59,412

2013-14 21.2% 12,379 58,520

2012-13 21.2% 13,316 62,742

Valued-Added Metrics

• Florida Department of Education’s Aggregated

VAM Score (1 Year)

– Standard score

– Student weighted average of the Teachers VAM

estimates divided by the average student growth for

that grade and subject

– 3 years of data used (2014-15, 2013-14 and 2012-13)


June 15-17, 2016


Correlation Results

Average Observation Score &

Aggregated 1 Year VAM

• Positive and small correlations were found for 2014-15 and 2013-14

• These compares with a 2013-14 pilot study where observers and teachers

were trained on the model

Observation

ScoreRead VAM

Math

VAM

Combined

VAM

Observation

Score1.00 .173** .226** .199**

N 59,412 9,888 6,624 12,248

Observation

Score1.00 .150** .208** .186**

N 58,520 10,245 6,750 12,379

Observation

Score1.00 .145** .185** .173**

N 62,742 10,727 7,192 13,316

2013-14

2014-15

2012-13

Observation

Score1.00 .168 .444** .239*

N 249 61 40 75

PCS

2013-14

Correlation Coefficients

Corrected for Measurement Error

• Once corrected for measurement error,

coefficients increased in size

Read VAM Math VAMCombined

VAM

2014-15 .232 .303 .267

2013-14 .201 .279 .250

2012-13 .195 .248 .232

PCS

2013-14.225 .596 .321

Correlation Results:

Compared with Pilot Findings

• Out of the 26 assessments that had a control

group match, 21 showed positive and significant

growth for students at treatment schools (p < .10)

• Independent sample t-tests showed that students

who attended treatment schools had

significantly increased growth scores (.32

standard deviations above prediction) compared to

students at control schools, accounting for student

characteristics (Hedges’ g= .35)

• Fixed effects models showed similar results as

independent sample t-tests


June 15-17, 2016


Turn & Talk:

How could you improve your

correlations in your data?

DO TEACHERS’

OBSERVATION SCORES

PREDICT THEIR VALUE-

ADDED SCORE?

OLS Regression Model

• OLS regression was used to assess whether

teachers’ observation score predicted their VAM

controlling for teacher characteristics

• Teacher characteristics included:

– Education

– Gender

– Race

– Ethnicity


June 15-17, 2016


Teacher Observation Score Predicting

Aggregated Year 1 VAM

• The teacher

observation score

was the largest

predictor in the

model (β=.161)

• This held true

across subjects

and years

Variable Name β SE p-value

Constant - .078 .000

Education .024 .013 .043

Black -.056 .022 .000

Hispanic .000 .027 .978

Female .032 .024 .006

Observation Score .161 .021 .000

R2

N

2014-15 VAM Combined

.034

12,248

WHICH ELEMENTS ARE

MORE HIGHLY

CORRELATED TO VAM?

Correlations Between Average

Teacher Element Scores &

Aggregated VAM

• Each teacher had an average score per element

they were observed on and these were correlated

with Aggregated VAM estimates

• Correlation coefficients between average element

score and combined VAM were significant all years

Min. Max. Avg.

2014-15 .061 .200 .136

2013-14 .064 .186 .122

2012-13 .060 .169 .118

Avg. .062 .185 .125


June 15-17, 2016


Highest VAM Correlations

The table shows the largest correlation coefficients by element

using the combined VAM from 2014-15

2014-15Read

VAM

Math

VAM

Combined

VAM

Applying Consequences for Lack of Adherence to Rules and Procedures*** .171 .224 .200

Helping Students Practice Skills, Strategies, and Processes*** .149 .182 .173

Organizing Students to Interact with New Content .161 .159 .172

Organizing Students to Practice and Deepen Knowledge** .151 .190 .172

Identifying Critical Content** .154 .161 .164

Probing Incorrect Answers*** .146 .169 .164

Maintaining a Lively Pace*** .144 .160 .160

Helping Students Record and Represent Knowledge .139 .172 .159

Establishing Classroom Routines*** .134 .191 .158

Noticing When Students are Not Engaged*** .127 .190 .153

Tracking Student Progress** .133 .174 .153

Demonstrating Withitness*** .132 .170 .152

Engaging Students in Cognitively Complex Tasks** .131 .127 .150

* indicates a high correlation in 2013-14; ** indicates a high correlation in 2012-13; ***indicates high correlations for both years

Lowest VAM Correlations

Red indicates that the element did not have a significant

correlation

2014-15Read

VAM

Math

VAM

Combined

VAM

Using Verbal and Nonverbal Behaviors that Indicate Affection for Students** .118 .103 .115

Providing Opportunities for Students to Talk** .111 .097 .113

Using Academic Games*** .077 .152 .109

Acknowledging Adherence to Rules and Procedures** .071 .154 .102

Demonstrating Intensity and Enthusiasm** .083 .119 .097

Demonstrating Value and Respect for Low Expectancy Students*** .068 .141 .095

Celebrating Success* .078 .108 .085

Using Friendly Controversy*** .065 .150 .085

Displaying Objectivity and Control*** .059 .125 .075

Understanding Students Interests and Backgrounds*** .047 .101 .061

* indicates a low correlation in 2013-14; ** indicates a low correlation in 2012-13; ***indicates low correlations for both years

Turn & Talk:

How could you use

this information at your

district or school?


June 15-17, 2016


MarzanoCenter.com

LearningSciences.com

1.877.411.7114

building expertise 2016 june 15-17, 2016 annual ...€¦ · –building leadership- district view/...

Documents