Has NCLB Improved Teacher and Has NCLB Improved Teacher and Teaching Quality for Disadvantaged Teaching Quality for Disadvantaged

Students?Students?

Laura M. DesimoneLaura M. Desimone Thomas M. Smith Thomas M. Smith Vanderbilt University Vanderbilt University David Frisvold David Frisvold University of Michigan University of Michigan

Presented at the Presented at the Society for Research on Educational Society for Research on Educational Effectiveness Effectiveness conference, Lansdowne, Virginia, December 10-conference, Lansdowne, Virginia, December 10-12, 2006.12, 2006.

Research QuestionsResearch Questions

1.What were the gaps in teacher and teaching quality for students in poverty compared to their more advantaged peers in 2000, and to what extent did those gaps narrow by 2003?

2. Are improvements in teacher quality and/or the narrowing of teacher quality gaps associated with state implementation of NCLB?

Why is Teacher and Why is Teacher and Teaching Quality Teaching Quality

Important?Important?

More Experience &Content Knowledge

Improved Instruction

Certification

Increased Student Learning

Teacher Quality Teaching Quality

State Implementation of State Implementation of NCLBNCLB

NCLB requires states to take steps to NCLB requires states to take steps to ensure that poor and minority children “are ensure that poor and minority children “are not taught at higher rates than other not taught at higher rates than other children by inexperienced, unqualified, or children by inexperienced, unqualified, or out-of-field teachers.”out-of-field teachers.”

Time period: 2000-2003Time period: 2000-2003

Target: 2005-2006Target: 2005-2006

Conceptual Framework: Conceptual Framework: Policy Attributes TheoryPolicy Attributes Theory

Stability

Power Authority

Specificity

Consistency

Policy

DataData

•State Policy Database

•2000 National NAEP

•2000 and 2003 State NAEP

MeasuresMeasures

InstructionInstruction -Conceptual Emphasis -Conceptual Emphasis -Conceptual Strategies -Conceptual Strategies -Procedural Teaching -Procedural Teaching

Full vs. partial certificationFull vs. partial certification

Inexperienced Teacher: 2 Years or FewerInexperienced Teacher: 2 Years or Fewer

BA or higher in mathematicsBA or higher in mathematics

RQ1: RQ1: What were the gaps in teacher What were the gaps in teacher and teaching quality for students and teaching quality for students in poverty compared to their more in poverty compared to their more advantaged peers in 2000, and to advantaged peers in 2000, and to what extent did those gaps narrow what extent did those gaps narrow by 2003? by 2003?

•Mean comparisons

•2000 & 2003 national and by state and free lunch status

Comparing Disadvantaged to Advantaged Students on Three Indicators of Teacher Quality

0

20

40

60

80

100

2000 2003 2000 2003 2000 2003

Certification Math Major +

Perc

ent o

f Tea

cher

s

Disadvantaged Students Advantaged Students

Inexperienced Teacher

Comparing Advantaged Students in High-Poverty Schools to Disadvantaged Students in Low-Poverty Schools

0

20

40

60

80

100

2000 2003 2000 2003

Certification Math Major +

Per

cen

t of

Stu

den

ts W

hos

e T

each

ers

Hav

e C

erti

fica

tion

or

Mat

h+

Deg

ree

Disadv. Students in low-poverty Schools Adv. Students in high-poverty schools

Summary of State-by-State Mean Summary of State-by-State Mean Comparisons on Teacher Quality IndicatorsComparisons on Teacher Quality Indicators

Number of states where… Regular Certification

BA Degree or Higher in Mathematics

New Teacher

Disadvantaged students were better off in 2003 than 2000 in this category 7 0 8

Disadvantaged students were worse off in 2003 than 2000 in this category 7 36 3

There was a teacher quality gap in 2000 15 19 0 Advantaged students were better off than disadvantaged students by greater than 5% in 2000 3 8 0

Disadvantaged students were better off than advantaged students by greater than 5% in 2000 0 0 0

There was a teacher quality gap in 2003 16 16 1 Advantaged students were better off than disadvantaged students by greater than 5% in 2003 5 5 0

The teacher quality gap increased from 2000 to 2003 3 2 2 The teacher quality gap increase was >=5% 0 0 0 The teacher quality gap decreased from 2000 to 2003 6 10 4

RQ2: Are improvements in teacher quality RQ2: Are improvements in teacher quality and/or the narrowing of teacher quality and/or the narrowing of teacher quality

gaps associated with state gaps associated with state implementation of NCLB?implementation of NCLB?

Do states that are stronger on the Do states that are stronger on the policy attributes have higher policy attributes have higher teacher quality in 2000 than states teacher quality in 2000 than states with weaker policy attributes? with weaker policy attributes?

Are changes in state policies Are changes in state policies between 2000 and 2003 associated between 2000 and 2003 associated with increases in teacher quality?with increases in teacher quality?

Was the implementation of policies Was the implementation of policies between 2000 and 2003 associated between 2000 and 2003 associated with a reduction in poverty gaps in with a reduction in poverty gaps in teacher quality?teacher quality?

State Policy DataState Policy Data

Compiled from existing national data Compiled from existing national data sources sources – Education Week’s Quality Counts Education Week’s Quality Counts – American Federation of Teachers’ report on American Federation of Teachers’ report on

states states – Key State PoliciesKey State Policies by the Council of Chief State by the Council of Chief State

School Officers School Officers – Report cards on standards by the Thomas B. Report cards on standards by the Thomas B.

Fordham Foundation Fordham Foundation Chose measures available in both 2000 Chose measures available in both 2000

and 2003and 2003

State Policy State Policy MeasuresMeasures

Alignment of standards and assessments Alignment of standards and assessments (consistency) (consistency)

NCLB: NCLB: “State assessments shall be aligned “State assessments shall be aligned with the State’s challenging academic with the State’s challenging academic content and student academic achievement content and student academic achievement standards” [(C)(ii), p.115 STAT. 1450].standards” [(C)(ii), p.115 STAT. 1450].

Our measure: Our measure: state used criterion-referenced state used criterion-referenced assessments in middle school mathematics assessments in middle school mathematics that had undergone an external alignment that had undergone an external alignment review in 2000 and in 2003.review in 2000 and in 2003.

State Policy State Policy MeasuresMeasures

Clear and detailed standards (specificity) Clear and detailed standards (specificity)

NCLB: NCLB: “challenging academic standards shall include “challenging academic standards shall include challenging academic content standards in academic subjects challenging academic content standards in academic subjects that specify what children are expected to know and be able that specify what children are expected to know and be able to do and contain coherent and rigorous content” [SEC. 1111; to do and contain coherent and rigorous content” [SEC. 1111; p. 115 STAT. 1445] p. 115 STAT. 1445]

Our MeasureOur Measure: : state had “clear and specific standards” in state had “clear and specific standards” in middle school mathematics. middle school mathematics.

State Policy MeasuresState Policy MeasuresProviding professional development resources Providing professional development resources (authority1) (authority1)

NCLB: NCLB: To improve the academic achievement To improve the academic achievement of the disadvantage can be accomplished by of the disadvantage can be accomplished by “significantly elevating the quality of “significantly elevating the quality of instruction by providing staff in participating instruction by providing staff in participating schools with substantial opportunities for schools with substantial opportunities for professional development” [SEC. 1001; p.115 professional development” [SEC. 1001; p.115 STAT. 1440];STAT. 1440];

Our measureOur measure: : state provides assistance to state provides assistance to low-performing schools low-performing schools

State Policy MeasuresState Policy MeasuresProviding resources to low-achieving schools Providing resources to low-achieving schools (authority2) (authority2)

NCLB: NCLB: To improve the academic achievement To improve the academic achievement of the disadvantage can be accomplished by of the disadvantage can be accomplished by “distributing and targeting resources “distributing and targeting resources sufficiently to make a difference to local sufficiently to make a difference to local educational agencies and schools where educational agencies and schools where needs are greatest” [SEC. 1001; p.115 STAT. needs are greatest” [SEC. 1001; p.115 STAT. 1440];1440];

Our measureOur measure: : state provides professional state provides professional development resourcesdevelopment resources

State Policy MeasuresState Policy MeasuresRanking schools (power1) and number of possible Ranking schools (power1) and number of possible sanctions (power2) sanctions (power2)

NCLB: NCLB: “…“…a local educational agency shall a local educational agency shall identify for school improvement any identify for school improvement any elementary school or secondary school served elementary school or secondary school served under this part that fails, for 2 consecutive under this part that fails, for 2 consecutive years, to make adequate yearly progress” [(b)years, to make adequate yearly progress” [(b)(1)(A), p. 115 STAT. 1479].(1)(A), p. 115 STAT. 1479].

Our measuresOur measures: :

(1) (1) state assigns ratings to all schools or state assigns ratings to all schools or identifies low-performing schoolsidentifies low-performing schools(2)(2) number of possible sanctions number of possible sanctions

:::::::::<

<

< ..

.

.

.

.

.

..

.

.

<

.

.

.

.

PhoenixPhoenixPhoenixPhoenixPhoenixPhoenixPhoenixPhoenixPhoenix

WashingtonWashingtonWashingtonWashingtonWashingtonWashingtonWashingtonWashingtonWashingtonIndianapolisIndianapolisIndianapolisIndianapolisIndianapolisIndianapolisIndianapolisIndianapolisIndianapolis

BostonBostonBostonBostonBostonBostonBostonBostonBoston

ChicagoChicagoChicagoChicagoChicagoChicagoChicagoChicagoChicago

DallasDallasDallasDallasDallasDallasDallasDallasDallas

DetroitDetroitDetroitDetroitDetroitDetroitDetroitDetroitDetroit

HoustonHoustonHoustonHoustonHoustonHoustonHoustonHoustonHouston

JacksonvilleJacksonvilleJacksonvilleJacksonvilleJacksonvilleJacksonvilleJacksonvilleJacksonvilleJacksonville

Los AngelesLos AngelesLos AngelesLos AngelesLos AngelesLos AngelesLos AngelesLos AngelesLos Angeles

MemphisMemphisMemphisMemphisMemphisMemphisMemphisMemphisMemphis

New YorkNew YorkNew YorkNew YorkNew YorkNew YorkNew YorkNew YorkNew York

San DiegoSan DiegoSan DiegoSan DiegoSan DiegoSan DiegoSan DiegoSan DiegoSan Diego

San FranciscoSan FranciscoSan FranciscoSan FranciscoSan FranciscoSan FranciscoSan FranciscoSan FranciscoSan Francisco

Ranges for Power scoreMeans

0.93 to 1.25 (10)0.68 to 0.93 (14)0.47 to 0.68 (16)0.22 to 0.47 (10)

Power

Do States With Stronger Policy Attributes Have Do States With Stronger Policy Attributes Have Higher Teacher Quality In 2000 Than States With Higher Teacher Quality In 2000 Than States With

Weaker Policy Attributes?Weaker Policy Attributes?

Three-level hierarchical linear model on the NAEP Three-level hierarchical linear model on the NAEP 2000 national sample:2000 national sample:

Where Where QQijsijs = inexperienced teacher (2 or fewer = inexperienced teacher (2 or fewer years of experience), certification status, level of years of experience), certification status, level of preparedness to teach different mathematics preparedness to teach different mathematics topics, whether or not the teacher has a degree topics, whether or not the teacher has a degree in mathematics, as well as teachers’ use of in mathematics, as well as teachers’ use of different instructional strategies, including different instructional strategies, including conceptual emphasis, conceptual strategies, and conceptual emphasis, conceptual strategies, and procedural teaching. procedural teaching.

000 010 001%ijs js s s js ijsQ FreeLunch POLICY u r

Conceptual Strategies Conceptual Emphasis

Procedural Teaching Self-reported Preparedness to Teach Mathematics Topics

Inexperienced Teacher

Regular Certification Mathematics Major or Higher

Beta SE P value

Beta SE P value

Beta SE P value

Beta SE P value

Beta SE P value

Beta SE P value

Beta SE P value

Level 1 (teacher)

Intercept 46.21 2.31 0.00 46.29 2.42 0.00 48.65 3.24 0.00 45.59 1.92 0.00 -1.72 0.56 0.01 2.78 0.48 0.00 -0.93 0.40 0.03 Level 2 (school)

Free lunch -0.33 0.08 0.00 -0.25 0.07 0.00 -0.15 0.11 0.17 -0.28 0.06 0.00 1.06 0.02 0.01 0.96 0.03 0.21 1.00 0.03 0.90 *Consistency 0.12 0.11 0.22 0.10 0.05 0.07 0.10 0.13 0.47 0.09 0.07 0.25 0.97 0.03 0.36 0.97 0.02 0.35 1.00 0.02 0.99 *Specificity 0.09 0.07 0.22 0.09 0.05 0.09 -0.005 0.11 0.97 0.09 0.05 0.08 0.96 0.02 0.08 1.04 0.03 0.15 0.97 0.02 0.14 *Authority2 0.09 0.03 0.01 0.10 0.04 0.01 0.05 0.03 0.14 0.05 0.03 0.07 0.99 0.01 0.57 1.00 0.01 0.68 0.98 0.01 0.17 *Power 2 0.03 0.02 0.07 -0.01 0.02 0.65 0.03 0.01 0.03 0.04 0.01 0.00 1.00 0.00 0.52 1.00 0.01 0.66 1.01 0.00 0.00 Level 3 (state)

Consistency 1.50 1.31 0.26 1.59 1.51 0.30 -1.26 1.73 0.47 1.46 0.89 0.11 0.92 0.29 0.80 0.42 0.38 0.03 1.03 0.33 0.92 Specificity 0.98 1.69 0.56 -1.34 1.72 0.44 -0.45 2.65 0.87 0.60 1.40 0.67 2.17 0.45 0.10 0.81 0.48 0.66 0.91 0.41 0.82 Authority2 1.81 1.25 0.16 3.51 1.38 0.02 2.37 1.16 0.05 2.20 1.10 0.05 0.80 0.31 0.48 2.00 0.51 0.18 1.04 0.23 0.83 Power 2 0.23 0.79 0.77 -0.10 0.74 0.90 -0.02 0.64 0.97 0.18 0.77 0.82 0.92 0.22 0.73 0.63 0.39 0.25 1.04 0.15 0.78 Variance Components

Level 1 Variance

62.62 61.23 84.14 59.30 0.58 0.15 0.76

Level 2 Variance

14.19 17.10 12.67 11.34 1.77 27.05 0.99

Df 284 284 284 284 284 284 284 Chi-Square 624.07 591.98 492.04 554.02 472.58 -

552.32 3981.3

7

P value 0.00 0.00 0.00 0.00 0.00 > 0.5 0.00 Level 3 Variance

8.24 9.84 5.68 4.38 0.52 4.05 0.19

Df 37 37 37 37 37 37 37 Chi-Square 126.12 144.98 105.14 82.35 67.41 192.61 73.70 P value 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Cross-sectional Relationship Cross-sectional Relationship Between Policy and Between Policy and

Teacher/Teaching Quality in High-Teacher/Teaching Quality in High-Poverty SchoolsPoverty Schools

Conceptual Teaching

Conceptual Emphasis

Procedural Teaching

Self-reported Preparedness to Teach Mathematics Topics

Inexperienced Teacher

Regular Certification

Mathematics Major or Higher

Aligned Standards and Assessments (Consistency)

Clear and Detailed Standards (Specificity)

Offering Professional Development Resources (Authority 2)

+ +

Number of Possible Sanctions (Power 2)

+ + +

Changes in state policy Changes in state policy between 2000 and 2003between 2000 and 2003

11 states adopted measures to assist low-11 states adopted measures to assist low-performing schools (authority1)performing schools (authority1)

9 states started or increased the resources they 9 states started or increased the resources they gave to professional development (authority2)gave to professional development (authority2)

23 states began ranking schools according to 23 states began ranking schools according to achievement results (power)achievement results (power)

Most states had clear and detailed standards as Most states had clear and detailed standards as early as 2000, but a handful of states early as 2000, but a handful of states implemented them between 2000 and 2003 implemented them between 2000 and 2003 (specificity) (specificity)

Most states conducted an alignment review in Most states conducted an alignment review in 2000; four states conducted an alignment of 2000; four states conducted an alignment of their standards and assessments in 2003 their standards and assessments in 2003 (consistency)(consistency)

To examine the relationships between To examine the relationships between change in state policies and change in change in state policies and change in

teacher quality, while controlling for change teacher quality, while controlling for change in state-level poverty, we estimated the in state-level poverty, we estimated the

following model:following model:

Where Where QQstst represents average teacher represents average teacher quality in state quality in state ss at time at time tt ( (tt = 2000, 2003), = 2000, 2003), POLICYPOLICY represents each of the five state represents each of the five state policies in the analysis (Power, Consistency, policies in the analysis (Power, Consistency, Specificity, Authority1, and Authority2) Specificity, Authority1, and Authority2)

Also, we ran models separately for high and Also, we ran models separately for high and low poverty schoolslow poverty schools

stsststst FRLPOLICYQ

Change in State Policy from 2000 to 2003 Associated with Change in Teacher Quality from 2000 to 2003

Aligned standards and assessments (consistency) (+)Clear and detailed standards (specificity)Providing assistance to low-performing schools (authority1)Offering professional development resources (authority2)Ranking low-performing schools (power1)Number of sanctions imposed (power2Free lunch

Change in State Policy from 2000 to 2003 Associated with Change in Teacher Quality from 2000 to 2003for High Poverty Schools

Aligned standards and assessments (consistency)Clear and detailed standards (specificity)Providing assistance to low-performing schools (authority1)Offering professional development resources (authority2)Ranking low-performing schools (power1)Number of sanctions imposed (power2)Free lunch

Inexperienced Regular Mathematics Teacher Certification Major or Higher

(+) (-)

Teacher Certification Major or Higher Inexperienced Regular Mathematics

+

-

Was the implementation of policies between 2000 Was the implementation of policies between 2000 and 2003 associated with a reduction of poverty gaps and 2003 associated with a reduction of poverty gaps in teacher quality?in teacher quality?

where where QQijstijst represents the characteristics of teacher represents the characteristics of teacher ii in in school school jj in state in state ss at time t at time t

2003 *2003

*2003*

ijst st jst t st

jst s ijst

Q POLICY FRL POLICY

FRL POLICY

Change in Teacher Quality Associated with Change in State Policy from 2000 to 2003

Inexperienced Teacher

Regular Certification

Mathematics Major or Higher

Consistency Specificity - Authority1 + Authority 2 Power1 Power2 Year 2003 - - Consistency*2003 Specificity*2003 + Authority1*2003 Authority 2*2003 Power1*2003 Power2*2003 - Free Lunch - - Free Lunch *Consistency*2003 Free Lunch*Specificity*2003 - Free Lunch*Authority1*2003 Free Lunch*Authority 2*2003 Free Lunch*Power1*2003 + Free Lunch*Power2*2003

Predicted percentage of teachers certified by power (whether a state ranks schools) and student poverty level

0.7

0.75

0.8

0.85

0.9

0.95

1

No Power, Low Poverty Power, Low Poverty No Power, High Poverty Power, High Poverty

2000 2003

Predicted percentage of teachers certified by specificity (clear and specific standards) and student poverty level

0.7

0.75

0.8

0.85

0.9

0.95

1

No Specificity, Low Poverty Specificity, Low Poverty No Specificity, High Poverty Specificity, High Poverty

2000 2003

Summary: Poverty Gaps and Summary: Poverty Gaps and How They Have ChangedHow They Have Changed

Small poverty gaps exist and didn’t Small poverty gaps exist and didn’t change much from 2000 to 2003change much from 2000 to 2003

Disadvantaged students in Disadvantaged students in

advantaged schools are worse off advantaged schools are worse off than their advantaged peers in than their advantaged peers in disadvantaged schoolsdisadvantaged schools

Summary: Associations of Summary: Associations of Teacher Quality with NCLB-Teacher Quality with NCLB-

Related PolicyRelated Policy The relationship between policy attributes and teacher The relationship between policy attributes and teacher

quality is not much different for advantaged and quality is not much different for advantaged and disadvantaged students.disadvantaged students.

States that provided professional development States that provided professional development resources (authority) had students whose teachers resources (authority) had students whose teachers were more likely to use conceptual teaching were more likely to use conceptual teaching strategies; state with more sanctions had students strategies; state with more sanctions had students whose teachers used more procedural instruction and whose teachers used more procedural instruction and had higher content knowledge.had higher content knowledge.

Implementation of certain policies was weakly Implementation of certain policies was weakly associated with improvements in teacher quality in associated with improvements in teacher quality in several cases, but generally did not reduce the several cases, but generally did not reduce the relationship between poverty and teacher quality.relationship between poverty and teacher quality.– Power mitigated it, specificity made it worsePower mitigated it, specificity made it worse

Challenges to Studying Policy Challenges to Studying Policy Effects on Teacher QualityEffects on Teacher Quality

Complex nature of interactions of state Complex nature of interactions of state and local policy (direct and indirect effects)and local policy (direct and indirect effects)

Implementation of simultaneous multiple Implementation of simultaneous multiple policy leverspolicy levers

Time ordering: policy may first reflect reaction Time ordering: policy may first reflect reaction to a problem, and only several years later to a problem, and only several years later would positive results be expected; how long would positive results be expected; how long does it take for policy to affect trends in does it take for policy to affect trends in teacher quality?teacher quality?

Real change vs. random fluctuation in short Real change vs. random fluctuation in short time span (e.g., 3 years)time span (e.g., 3 years)

Quality of measuresQuality of measures

SignificanceSignificance

Teacher and teaching quality affect student Teacher and teaching quality affect student outcomesoutcomes

Currently we are implementing multiple Currently we are implementing multiple policy levers to improve teaching: (1) merit policy levers to improve teaching: (1) merit pay, (2) professional development pay, (2) professional development (mentoring, coaching, induction), (3) school (mentoring, coaching, induction), (3) school and curriculum reforms, (4) NCLB and curriculum reforms, (4) NCLB requirements, (5) teacher preparation requirements, (5) teacher preparation reforms (6) recruitmentreforms (6) recruitment

What policies work to improve teaching, retain What policies work to improve teaching, retain

good teachers, and recruit potentially good good teachers, and recruit potentially good teachers? teachers?

Use student achievement as a measure of Use student achievement as a measure of effectivenesseffectiveness

Ask how teachers are changing in response to Ask how teachers are changing in response to policy leverspolicy levers

Ground studies in theory about policy and teacher Ground studies in theory about policy and teacher behaviorbehavior

Directly compare effects of 2 or 3 key policy Directly compare effects of 2 or 3 key policy levers, e.g., preparation, incentives, levers, e.g., preparation, incentives, mentoring/coaching/professional developmentmentoring/coaching/professional development

Future Questions and How We Might Pursue Them