Measuring the Effect of the Measuring the Effect of the Milwaukee Mathematics Milwaukee Mathematics Partnership on Student Partnership on Student

AchievementAchievementCindy M. WalkerJacqueline K. GoszDeAnn HuinkerUniversity of Wisconsin - Milwaukee

Structure of MMPMath TeachingSpecialist (MTS)

Learning Team(LT)

Literacy Coach (LC)

Math TeacherLeader (MTL)

Math Teacher

Concerns

• Difficulty measuring impact of MMP on student achievement in mathematics– Much of the effect of the MMP is not under

the direct control of the MMP– There are uncontrollable variables at the

school level that must be taken into consideration

• Finding an appropriate statistical model– Status-based models– Value-added Assessment (VAA) models

Status-Based Growth Models

• Rely upon cohort-to-cohort comparison to assess improvement in achievement

• Compare unadjusted average student performance on standardized tests across successive cohorts

• Inherent shortcomings– Based on a non-equivalent group case

study design– Do not take into consideration pre-

existing differences between students

VAA Growth Models

• Consider the growth of students from one year to the next through repeated measures analyses

• Use longitudinal data to compare a students’ progress to his/her past academic achievements

• Provides some control over the individual differences between students because students serve as their own controls

Purposes

1) To determine if reported differences in how the model of professional development is being implemented by the MMP are related to differences in student achievement gains in mathematics

2) To compare and contrast status-based and VAA models in terms of assessing the impact of the effectiveness of the MMP

3) To address issues specific to the MMP that have been encountered in attempting to implement a system of accountability

Methodology – Participants

• Schools– 92 of the 143 (64%) elementary & middle

schools that participated in the MMP during its 1st year were considered in this study.

• Students – 3rd – 8th graders who participated in the district’s standardized testing – VAA

• Only students who had test scores for both the 2003-4 and 2004-5 school years, did not change schools, and did not repeat/skip a grade could be considered

• 14,259 total students

– Status-based• 27,117 students for the 2003-4 school year• 27,079 students for the 2004-5 school year

Methodology - Instrumentation• Self-report instrument administered to MTLs

1) How often MTLs meet with their Math Specialist and others in their building;

2) How satisfied MTLs feel with the amount of time they have to discuss issues centered around mathematics with their Math Specialist and others in their building;

3) How supported MTLs feel in their role as a MTL by their Math Specialist and others in their building;

4) The MTLs perceptions of the functioning of school’s Learning Team in terms of the general climate of the meetings as well as the overall cohesion in accomplishing goals related to their mathematics curriculum.

• MTL Tracking variables– Number of MMP meetings attended and whether or

not a vision statement and/or action plan was turned in

Methodology – Instrumentation• Mathematics scale scores from the

November administration of the Terra Nova (grades 3, 5-7) and WKCE (grades 4 and 8) for 2003-4 and 2004-5.– Both tests published by CTB-McGraw Hill– Mathematics scale scores from Terra

Nova and WKCE shown to be highly correlated by CTB and therefore, according to the test publishers, it is appropriate to directly compare the results of the WKCE and Terra Nova

• DV = school level difference score representing growth in student mathematics achievement• Weighted average of individual difference scores was

calculated to obtain an overall measure of growth for each school

• Weighted average used because expected growth for each student is not equal to zero ((Dj-i) ≠ 0)

I = the lowest tested grade level served by a particular schoolj = the highest grade level served by a particular schooln(j - i) = the # of students with scale scores for both grades i

and j = the average of individual difference scores for students from grade j to grade I, and

N = total number of students used to calculate the overall growth

Dependent Variables - VAA

N

Dn ijij

j

Ii)()( −−

−∑

)( ijD −

Dependent Variables – Status based• DV = school level difference score

representing growth in student mathematics achievement• The difference in average scale scores for

each grade between the 2004-5 and 2003-4 school years was calculated and then the average difference across all grade levels was calculated for an overall measure of growth for each school

• Simple average used because the expected difference for each grade level is zero ((D2-1) ≠ 0)

• Does the degree in which schools’ learning teams embrace the vision of the MMP influence student growth in mathematics achievement test scores?

– Stepwise regression analysis used with five independent variables with four interaction terms• Measure of the frequency in which the MTL met with

the LT and had useful discussions about how to help students succeed in mathematics

• Climate of the LT LT collaboration

• Vision Plan submission Action Plan Submission

– Analyses exploratory in nature

Evaluation Question #1

Evaluation Question #2

• Does the level of satisfaction with time spent discussing mathematics, as reported by MTLs, and frequency of MTL attendance at district-wide MMP meetings influence student growth in mathematics achievement test scores?

– Stepwise regression analyses used with two independent variables with one interaction term• Number of district-wide MMP meetings attended by

the MTL• Satisfaction the MTL felt with the amount of time

spent discussing mathematics with others in their school

• The interaction of the above two variables

– Analysis exploratory in nature

Evaluation Questions #3 and #43) Is there a difference in student growth

in mathematics for schools for which MTLs report feeling a greater ability to accomplish goals regarding mathematics?

4) Is there a difference in student growth in mathematics for schools in which MTLs feel more supported by other math teachers in their school?

• Each used a one-way ANOVA with three levels on the DV

Results

• Very little statistically significant results were found– This is not surprising given that the

analyses conducted were based only on approximately six months of MMP activity

• Evaluation Question #1 – degree in which LTs embrace vision of the MMP– VAA – no IVs were significant enough to be

entered into the regression model– Status based – LT collaboration found to be

a significant predictor of student growth in math (F(1,88) = 5.06, p=.027, = -2.23, R2 = .05)

Results (con’t)• Evaluation Question #2 – MTL level of

satisfaction with time spent math and frequency of MTL attendance at MMP meetings– VAA and Status based – MTL attendance at

district-wide MMP meetings found to be a statistically significant predicator of overall school level growth in math• VAA: F(1,88) = 4.17, p=.044, = .98, R2 = .05

• Status based: F(1,90) = 6.37, p=.011, = 1.31, R2 = .07

– Encouraging that attendance seems to play a role in increasing student achievement

Results (con’t)

• Evaluation Questions #3 and #4 – MTLs feeling of ability to accomplish goals and perception of support from other math teachers– No statistically significant results

were found for either evaluation question for either accountability system

Discussion

• Statistical analyses did not detect many relationships between survey responses (IVs) and overall school level growth in mathematics– Not surprising given that the data

collected from the survey was obtained only six months after the inception of MMP activities

– Encouraging that a small positive effect on overall growth in student achievement in mathematics was found in terms of MTL attendance at MMP meetings

Discussion

• Differences found between the models highlight major flaws of using a status based growth model– Namely, the status based model

utilizes different cohorts of students to ascertain a measure of change in achievement

Discussion• Data exemplifying the major flaw of status based growth modelsMeans and SD of Difference Scores Obtained from VAA and Status Based Accountability Systems

VAA Mean SD Status Based

Mean SD

Overall 16.27 28.30 Overall -2.55 7.39

4th - 3rd Gr 23.29 33.30 3rd Gr -3.05 14.77

5th - 4th Gr 8.37 26.55 4th Gr -5.10 10.77

6th - 5th Gr 11.69 26.38 5th Gr -5.59 11.52

7th - 6th Gr 15.17 26.01 6th Gr 0.37 9.83

8th - 7th Gr 20.69 25.58 7th Gr -3.66 12.03

8th Gr 7.91 11.79

Discussion

• Support for VAA models– Use of longitudinal data to estimate

the impact of large-scale educational reform on student achievement•Compares student’s progress to his/her

past academic achievement rather than to a generic cohort of students

•Offers a picture of the rate at which a student is learning

•By tracking individual student’s progress from one year to the next, VAA models provide a control for pre-existing student differences

Discussion

• Practical disadvantages of VAA models– Require an extensive database of

longitudinal data •Student achievement scores•Student characteristic variables•Means of linking students to educational reform activities

– Requires reliable and valid methods of quantifying educational reform endeavors and outcomes

Discussion

• Theoretical concerns with using VAA– Model specification– Viability of using the model to claim

causal effects of educational reform on student learning

– Inclusion of covariates and possible confounding variables

– Use of achievement test data as the primary outcome measure

Discussion

• Theoretical concerns (con’t)– Effect of having missing data that is NOT

completely missing at random (CMAR)•CMAR data – there are not discernable

features that characterize the students who have missed a test from those who have not

•However, students who are more likely to miss tests often have certain characteristics that distinguish them from the population of students who take the tests

•Data that is NOT missing at random cannot be ignored because it introduces bias into the estimates of school-level effects

Discussion

• Theoretical concerns (con’t)– How best to take into account the

time at which student achievement tests are administered•Unclear how to best model growth in

achievement so that the growth is accurately distributed among the schools responsible for that growth

•Especially salient when school level variables are collected at a different point in time as when students are tested

Discussion

• Theoretical concerns (con’t)– How best to take into account the time at

which student achievement tests are administered is a major issue in trying to model how variability at the school level, which is related to the efforts of MMP, is having an impact on increasing student achievement in mathematics.

MMP CONUNDRUM: While the efforts of the MMP start in September and end in August, the school year runs from September to June, students are tested in November, and data is collected from teachers, learning team members, and MTLs in June.

Discussion• Theoretical concerns (con’t)

– Using growth as a DV implies the scale of measurement is constant across years• Vertical scaling assumes each test is measuring

a common trait so that the scales have identical meanings

• However, as students progress from one grade to the next, their curriculum progresses in content and difficulty and so tests designed for different grade levels may not allow for a measure of true growth

– Standardized achievement tests are subject to measurement error and the amount of error varies depending on student’s level of achievement

Discussion

• Problems encountered in MMP evaluation– Currently only school level variables

can be utilized– Lack of means to link teacher

responses to student achievement• Issues associated with teacher

confidentiality

– Lack of participation in data collection efforts•Likely that modeling classroom effects

would be statistically biased due to the magnitude of missing responses

Future Direction

• Conduct more refined analyses using data collected at the end of year two– On-line survey made available to all

MPS participants in the MMP that may have an effect on increasing student achievement in mathematics including•Learning Team Members•Literacy Coaches•Math Teacher Leaders•Math Teachers

Future Direction

• Data collected at the end of year two– Six primary areas were evaluated

1.Professional Development2.School/District Functioning3.Learning Team Functioning4.Classroom Practice5.Role of the MTL6.Role of the LC

Future Direction

• Data collected at the end of year two– Professional Development (PD)– the

extent to which those surveyed participated in, and were influenced by, PD activities supported by the MMP•Both formal and informal professional

development activities were considered•PD items were administered to all

participants

Future Direction

• Data collected at the end of year two– School/District Functioning

• The extent to which those surveyed feel involved in helping shape their school’s mathematics program

• Participant’s perception regarding the degree of alignment between the school and district goals

• Participant’s perception regarding the level of support provided by the Math Teaching Specialists and Learning Targets

• Items were administered to all participants

Future Direction

• Data collected at the end of year two– Learning Team Functioning

•The extent to which the Learning Team is functioning as a cohesive unit

•The degree to which the Learning Team is emphasizing school improvement efforts in mathematics in their work

• Items administered to all MTLs and all Learning Team members

Future Direction

• Data collected at the end of year two– Classroom Practice

•The extent to which math teachers engage their students in interactive vs. didactic instruction

•The amount of time math teachers devoted to each of the learning targets in MPS

• Items administered to all participants that reported teaching math in the classroom

Future Direction

• Data collected at the end of year two– Role of Math Teacher Leader (MTL)

•The extent to which MTLs met with others in their school to discuss the teaching and learning of mathematics

•The level of support MTLs felt was provided to them by others in their school and the MTSs

• Items administered only to MTLs

– Role of the Literacy Coach (LC)•The extent to which the LC collaborated with

and supported the MTL at their school• Items only administered to the LCs