Milwaukee Mathematics Partnership

Program EvaluationMTL MeetingNovember 6,7 2006

November 6-7, 2006 2

MMP Goals

Comprehensive Math Framework

Distributed Leadership

Teacher Learning Continuum

Student Learning Continuum

November 6-7, 2006 3

MMP Core Partners

University of Wisconsin—Milwaukee

Milwaukee Public Schools

Milwaukee Area Technical College

November 6-7, 2006 4

Evaluation Goals

Help the MMP better serve its constituents and improve its effectiveness

Serve the broader mathematics education community through documentation and dissemination of MMP activities

November 6-7, 2006 5

Evaluation Logic ModelStudent

Achievement

Teacher Content& Pedagogical

Knowledge

Math FacultyInvolvement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

NewCourses

DistrictBuy-in

MPA Ownership

MATCBuy-In

UWMBuy-In

ClassroomPractice

MMPActivities

ProximalOutcomes

DistalOutcomes

November 6-7, 2006 6

2005-06 Evaluation Activities

MMP Online Survey MTS Survey Learning Team Observations Classroom Observations Assessment of Teacher MKT Social Network Analysis MPS Data Mining

November 6-7, 2006 7

Presentation Overview

Part I: District Wide Analysis

Part II: School Case Studies

November 6-7, 2006 8

Part I: District Wide Indicators

StudentAchievement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

November 6-7, 2006 9

Part I Activities

MMP Survey Designed to measure differences in the quantity and quality of MMP related activities

MTS Survey Designed to measure how well MTS perceived school to be, in terms of meeting the goals of the MMP

MKT Assessment Designed to assess teachers’ mathematical knowledge for teaching

WKCE Tests Designed to assess students’ mathematical content knowledge

November 6-7, 2006 10

Respondents

Number of MPS Respondents by Role in the MMP

Academic Year

2004-05 2005-06

Math Teacher Leader 124 140

Learning Team Member & Mathematics Teacher

167 284

LT Member (Administrative) 80 171

Math Teacher Only 676 1340

Literacy Coach 47 94

Total 1094 2029

November 6-7, 2006 11

Research Questions

1. Validity of MMP Survey2. Change in MMP Impact3. Characteristics of math-focused

schools4. MMP Impact on student achievement5. Characteristics of quality

learning teams6. Characteristics of quality MTLs

November 6-7, 2006 12

Research Question #2

Has the perceived impact or focus of the MMP changed since last year?

Analytical Approach Dependent t-tests conducted at the school level for all school level variables obtained in both administrations of the MMP Survey

November 6-7, 2006 13

Results – Trends in Impact of MMP

This year statistically significant increases

MTLs reported discussing mathematics with others at their school (t(90) = 12.06, p < .001)

Teachers reported engaging in activities designed to align their curriculum to standards (t(111) = 8.53, p < .001)

Teachers reported engaging in activities designed around CABS or student work samples (t(106) = 7.04, p < .001)

November 6-7, 2006 14

Research Question #3

What variables characterize schools that are more focused on increasing student achievement in mathematics?

Analytical Approach Stepwise multiple regression

November 6-7, 2006 15

Results – Characteristics of Schools with Greater Math Focus

68% of variability in schools’ overall self-reported focus on mathematics could be explained by differences in: Teachers reported working together to improve content and pedagogical knowledge (b = .46, t = 6.7, p < .001)

Teachers reported consistent instructional practices used at their school (b = .14, t = 2.4, p = .018)

Teachers perceived the Learning Team to be supportive of efforts to improve math teaching and learning (b = .38, t = 5.6, p <.001)

November 6-7, 2006 16

Research Question #4

What variables help to explain differences in the percentage of students classified as proficient in mathematics?

Analytical Approach Stepwise multiple regressionscontrolling for previousachievement and SES

November 6-7, 2006 17

Results – Impact of MMP on Increasing Student Achievement

Schools with a stronger focus on increasing student achievement in mathematics are have a higher percentage of students proficient in mathematics, after controlling for SES and previous achievement (b = .26, t = 3.7, p =.001)

An additional 7% of variability in student proficiency rates explained by the addition of this predictor

November 6-7, 2006 18

Research Question #5

What variables characterize Learning Teams that are perceived to be more helpful in terms of increasing student achievement in mathematics?

Analytical Approach Stepwise multiple regression

November 6-7, 2006 19

Results – Characteristics of Supportive Learning Teams

64% of variability in schools’ overall perception of the level of support provided by the Learning Team could be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b = .41, t = 5.5, p < .001)

Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .18, t = 2.4, p = .021)

Teachers perceived the MTL to be supportive of efforts to improve mathematics teaching and learning (b = .46, t = 5.9, p <.001)

November 6-7, 2006 20

Research Question #6

What variables characterize Math Teacher Leaders that are perceived to be more helpful, in terms of increasing student achievement in mathematics?

Analytical Approach Stepwise multiple regression

November 6-7, 2006 21

Results – Characteristics of Supportive Math Teacher Leaders

42% of variability in schools’ overall perception of the level of support provided by the MTL be explained by differences in: Teachers reported working together on improvement activities designed around CABS or student work samples (b = .38, t = 4.5, p < .001)

Teachers reported a greater alignment between their school’s adopted curriculum and standards/learning targets (b = .26, t = 3.0, p = .004)

MTLs perceived themselves as being supported by others at their school (b = .27, t = 3.2, p = .002)

November 6-7, 2006 22

Conclusions

MTSs in general have a strong sense of what is going on with school leadership, but less awareness about activity at the classroom level.

MMP efforts are being felt beyond the learning team and MTL by school staff

MMP activities are helping schools become more focused on increasing student achievement in mathematics

Schools that are more focused have increased the proportion of students proficient in mathematics

Adoption of MMP-related principles is reported to be

related to supportive learning teams and to supportive Math Teacher Leaders

November 6-7, 2006 23

Part II: Case Study Schools

StudentAchievement

Teacher Content& Pedagogical

Knowledge

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

Collaboration

November 6-7, 2006 24

Eleven Case Study Schools

Schools were diverse in terms of

Type Geography Student demographics

November 6-7, 2006 25

Case Study Data Collection

22 learning team observations—2 in each school

44 classroom observations—4 in each school; 2 teachers observed 2 times each

MKT Assessment for math teachers

SNA Survey for matheducation ‘stakeholders’

November 6-7, 2006 26

Results of Learning Team Observations

Team Functioning

LeadershipParticipation

Organization & StructureResults

Overall Functioning

MMP Issues

Math VisionIntegration

Math LeadershipMMP Work

Overall MMP

Overall, strongest areas were participation & mathematics leadership

Biggest areas for improvement were mathvision & results

November 6-7, 2006 27

Characteristics of Hi-Lo Scoring Learning Teams—Team Functioning

Distributed leadership Positional authority is

less important Multiple views are

represented and heard Multiple segments of the

school are represented Written agenda, note

taker, facilitator Explicit action items Participants have hi

knowledge and skill levels

Principal does all the talking

A few individuals dominate the discussion

No agenda or team is easily distracted from the agenda

Little follow-through on assignments

No clear action items

Hi Lo

November 6-7, 2006 28

Characteristics of Hi-Lo Scoring Learning Teams—MMP Issues

Consistent curriculum Math is addressed

alongside and in combination with other subjects

Coherent within grades and (at times) across grades

MTL clearly in charge with respect to math

Attention to CABS; reference to MMP courses; reviewing student work

Variation in curriculum Math not addressed at

the meeting No clear math leader—

i.e., hard to tell who the MTL is

Confusion about the MMP and CMF

Too much non-academic housekeeping

School climate isthe priority

Hi Lo

November 6-7, 2006 29

Results of Classroom Observations

General Practice

Articulating Math TaskFormative Assessment

Overall

ComprehensiveMath Framework

UnderstandingComputingApplicationReasoning

Engagement

Overall, strongest areas were articulating the mathtask & understanding

Biggest areas for improvement were useof formative assessment & engagement

November 6-7, 2006 30

Characteristics of Hi-Lo Scoring Classroom Performance—General

Math task within the lesson was easy to identify

Math task was discrete and level-appropriate

Encouraging self-assessment and peer-assessment

Establish criteria for proficiency

Promoting problem solving and independent thinking

Math task was to complex or obscure

Only feedback provided was if answer was correct

Little teacher involvement in the lesson

Feedback focuses on student behavior

Hi Lo

November 6-7, 2006 31

Characteristics of Hi-Lo Scoring Classroom Performance—CMF

Student explanations sought

Computation is presented as a means to an end

Problem solving was emphasized

Students had to justify solutions

Lessons are made relevant by using everyday things like money or time and seeking examples from students’ lives

Close ended questions are emphasized

Only one way to solve problems presented

Minimal time allowed to share solutions

Students not accountable for responding to questions

Problems not presentedin context

Hi Lo

November 6-7, 2006 32

Results of MKT Assessment

Number &Operations

43 item assessment addressed 3 content areas:

Algebra Geometry

OverallScore

& &

November 6-7, 2006 33

Results of MKT Assessment

Average IRT Scores

nNumber & Operations Algebra Geometry Total

Lo 12 -0.78 -0.60 -0.65 -0.59

HI 32 0.12 0.31 0.20 0.22

Mean -0.36 -0.13 -0.32 -0.31

Median -0.38 -0.09 -0.39 -0.44

SD 0.70 0.77 0.79 0.75

November 6-7, 2006 34

Social Network Analysis

Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics

Statistical analysis focused on1. Network and in-school density2. Importance of MTL and MTS

November 6-7, 2006 35

Metric n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

Lo 13 31 3.9% 7.6% 3.33 1.14

Hi 32 91 11.4% 31.1% 28.24 18.52

Mean 21.1 54.0 6.7% 17.6% 13.81 5.31

SD 6.8 17.6 2.6% 9.6% 7.2 4.9

Median 19 48 6.2% 15.4% 13.07 3.75

Overall SNA Results

Density—a perfect score is 100% where everyonenames everyone else

In-Degree scores are relative measures

November 6-7, 2006 36

Example Network

November 6-7, 2006 37

Example Network

November 6-7, 2006 38

Report Card Indicators

19 indicators in 7 domains based on in-school data collection, online surveys, and MPS data

1. MTS Assessment2. Collaboration3. Learning Teams4. Classroom Practice5. Professional Development6. Teacher MKT7. Student Achievement

November 6-7, 2006 39

Report Card Results

StudentAchievement

Teacher Content& Pedagogical

Knowledge

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

WKCEMean % Proficient = 44%

Overall rating = 3.5Gap MTL v. other teacher = .2Teacher Engagement = 3.2

Overall IRT = -0.34Algebra IRT = -0.18

Team Functioning = 3.5MMP Principles = 3.6LT Quality = 3.1

PD Hrs. = 17.8Facilitation Hrs. = 1.0PD Quality = 3.1

Network density = 6.7% / School density = 17.6%MTL Role = 13.8 / MTS Role = 5.3

SR MTL Engagement = 4.4 / MTS Quality = 3.0

MTS Assessment = 38.3 of 55

November 6-7, 2006 40

Student Achievement & In-School Network Density

November 6-7, 2006 41

Student Achievement & Learning Team MMP

November 6-7, 2006 42

Student Achievement &Professional Development

November 6-7, 2006 43

Conclusions

MMP is advancing concepts, ideas, & principles that can help schools improve student achievement results in math.

Schools that score well with regards to MMP-related metrics have higher student achievement.

Learning team adoption of MMP ideas and dense in-school communication networks are predictors of high student achievement

November 6-7, 2006 44

Conclusions

At the same time… Some MPS schools are lagging behind in terms of adopting MMP ideas.

These schools perform do not score as well on MMP metrics, which is consistent with student achievement results.

We know that other factors—prior year student achievement and SES—are stronger predictors

November 6-7, 2006 45

Evaluation Next Steps

District Wide Analysis Continue online survey & data mining Improve ability to link student and teacher data

working with MPS

Case Study Schools1. Recruit case study schools Nov2. Plan observations Nov-Dec3. Observations Round 1 Jan-Feb4. Observations Round 2 March-April5. MKT Assessment May6. SNA Survey May