Copyright © 2010 National Comprehensive Center for Teacher Quality. All rights reserved.

TQ Research & Policy Brief: TQ Research & Policy Brief:

Challenges in Evaluating Special Challenges in Evaluating Special Education Teachers and English Education Teachers and English Language Learner SpecialistsLanguage Learner Specialists

Lynn Holdheide and Dan Reschly, Ph.D. Vanderbilt University

July 20, 2010

OSEP Project Director’s Conference

www.tqsource.org

About the National Comprehensive Center for

Teacher Quality The National Comprehensive Center for Teacher

Quality (TQ Center) is a federally funded partnership whose mission is to help regional comprehensive centers and states carry out the teacher quality mandates of the Elementary and Secondary Education Act (ESEA) as reauthorized by the No Child Left Behind Act.

Partners:• Learning Point Associates• Vanderbilt University• ETS

2

www.tqsource.org

Today’s Goals

Seeks to build the capacity of participants to• Articulate the challenges identified with

evaluating special education teachers through value-added and other measures of teacher evaluation.

• Actively participate in the creation or redesign of teacher evaluation models that support the development of strong, valid and reliable teacher evaluation policies and practices that recognize and promote the unique contribution of special education teachers.

3

www.tqsource.org

The Purpose

Identify the specific challenges in evaluating this population of teachers.

Determine the current status of state policy and practice.

Identify promising evaluation practices and instruments.

Provide guidance and policy recommendations to districts and states.

4

www.tqsource.org

The Inquiry

Review of policy/literature

Survey inquiry Series of interviews

with state- and district-level practitioners and researchers

Data collection period: December 2009–April 2010

5

• Designed in collaboration with Council for Exceptional Children (CEC) and national experts

• State and local survey

• Respondent pool: state and local directors (identified within CEC’s Council of Administrators of special education listserve)

1,143 total

respondents

www.tqsource.org

Modification of Evaluation Processes for Special Educators

0%5%

10%15%20%25%30%

State Local Total

Percentage of Administrators

Who Report an Allowance in Modification for Special Educator

Among the local administrators,

reported that contractual agreement prevented modification in the evaluation process.

6

81%

www.tqsource.org

Opinions Regarding Special Education Teacher Evaluation

7

Strongly Agree or Agree

84%

92%

32%

www.tqsource.org

Evidence-Based Practices

Meeting the needs of “diverse” learners may not attend to the following:

• Special skills (individualized education program [IEP] facilitation, collaboration, secondary transition, social and behavioral interventions, compliance with legal mandates)

• Evidence-based instructional methods (direct/explicit instruction, scientifically based reading instruction, learning strategy instruction)

8

www.tqsource.org

Multiple Measures

15%

20%

27%

37%

44%

62%

93%

0% 20% 40% 60% 80% 100%

Parent/Family Teacher Evaluations

Other

Student Teacher Evaluations

Standardized Achievement Test

Criterion Referenced/CBM

Teacher Portfolio

Self-Report Measures

Classroom Artifacts

Goal-Driven Professional Development

Observation Protocols

9

Use more than one measure.

95+%

Survey inquired about current practice.

Respondents indicated value-added models in future evaluation efforts.

www.tqsource.org

Practical Example: District of Columbia IMPACT

Individual Teacher Value-Added Scores

Non-Value-Added Achievement Teaching and Learning Framework Commitment to the School School Value-Added Scores Core Professionalism IEP Quality Plan IEP Timeliness

10

Special Education

10%

50%

15%

5%

5%

15%

www.tqsource.org

Opinions Regarding Use of Student Achievement for Special

Educators

11

Strongly Agree or Agree

73%

60%

21%

www.tqsource.org

Student Growth MeasuresPractical Examples

Austin Independent School District, Texas• Student Learning Objectives

One is targeted toward classroom performance. One is targeted toward particular skills or subgroups

of students.

Norwell Public Schools, Massachusetts• Progress on the IEP is factored into evaluation

of special educators.

Both districts are heavily dependent on teacher training and support.

12

www.tqsource.org

Observation Protocols

13

Use the same observation instrument as that of general

education teachers.

Use a modified or different observation instrument.

85%

12%

Align to the state’s

professional teaching standards.

26% Didn’t know.

51%

“Our evaluation tool was developed in the district

over 40 years ago.”

“Our current evaluation system is outdated and

applied to nothing.”

www.tqsource.org

Observation ProtocolPractical Example

Alabama Department of Education’s Professional Education Personnel Evaluation Program• Slightly modified for the following:

Specialty area systems (speech paths, library specialist) Teachers of students with significant cognitive

disabilities

• Competencies added in certain areas (e.g., classroom is expanded to include community settings, and academic content is expanded to include functional life skills.)

14

www.tqsource.org

Require training for evaluators.

Require specialized training.

Expert Opinions Regarding Evaluators

15

Strongly Agree or Agree

77%

61%

12%

60%

www.tqsource.org

Practical Examples

Toledo’s Peer Assistance and Review• School-based teams evaluate.

Norwell Public Schools, Massachusetts• All teachers are evaluated using the same

instrument.

• Two formative assessments are conducted: One with principal One with special education administrator

• Each evaluator focuses on expertise areas.

• Both work collaboratively to develop summative evaluation.

16

www.tqsource.org

Opinions Regarding Attribution in Coteaching Setting

17

Strongly Agree or Agree

13%

85%

75%

www.tqsource.org

Policy and PracticeRecommendations

Include special education administrators when revamping/designing evaluation frameworks.

Identify a common framework that defines effective teaching for all teachers, differentiating for special educators as appropriate.

Integrate evidence-based practices for students with disabilities into evaluation models.

18

www.tqsource.org

Policy and PracticeRecommendations

Improve data quality.

In addition to─or, in some situations, in the absence of─appropriate standardized assessment data, incorporate other reliable evidence of teachers’ contributions to student learning into the teacher evaluation system, such as progress toward accomplishing IEP objectives and student learning objectives across broad academic and behavioral domains.

19

www.tqsource.org

Policy and PracticeRecommendations

Ensure that evaluator training includes explicit training for evaluators of special educators and/or consider establishing a model of peer-to-peer observations or a model in which evaluators are matched to specific disciplines.

Collaborate with teacher preparation programs to ensure that evidence-based practices are incorporated into teacher preparation coursework and professional development activities.

20

www.tqsource.org

Evidence-based Practices in Special Education and Practice

Scientifically based instruction ESEA (2002) and IDEA (2004)

IES criteria and evolution to evidence-based practices

Research supported evidence-based practices in special education• ABA and its many variations/application

• Direct instruction, big D and little d, reading and mathematics

• Formative assessments with instructional decision making

• Learning strategies

21

www.tqsource.org22

What Works? Research Foundations

From Meta-AnalysisTreatment Effect

Size• Applied Behavior Analysis + 1.00• Formative Evaluation: Curriculum-Based

Measurement+Graphing+Decision Rules+Reinforcement + 1.00

• Explicit Instruction and PS + .70 to 1.50

• Comprehension Strategies + 1.00 • Mathematics Interventions +.60 to

1.10• Writing Interventions +.50 to .85• Matching instruction to

learning styles? 0.00

Sources: Kavale, 2005

Note: These effect sizes are stable across cultural groups.

www.tqsource.org

Scientifically Based Instruction in Reading

Reading Curricula content (Snow, Burns, & Griffin, 1998)• Phonemic Awareness• Phonics• Fluency• Vocabulary• Comprehension

Problem of teacher preparation Vanderbilt University/TQ Center innovation

configurations: reading, classroom behavior, inclusive services, learning strategies (Reschly, Smartt, & Oliver, 2007)

23

PLUS

• Direct, systematic instruction

• Universal screening and formative evaluation

www.tqsource.org

Learning to Read: The Great Debate (Chall, 1967)

Research review 1900–1965

Early reading, K–3 Code versus meaning

emphasis Phonics or whole word Code superior,

especially for struggling readers

Lamented the generally poor preparation of teachers to teach reading

24

www.tqsource.org

National Council on Teacher Quality: Reading Components

Taught Well

25

Percentage

Number of Components

11%

N=8 7%

N=5

11%

N=8

13%

N=9

43%

N=31

Source: Walsh, Glaser, and Wilcox, 2006

www.tqsource.org

NCES Reading Report Card 2009: Categories

< Basic: Less than partial mastery of prerequisite knowledge and skills fundamental to proficient work at the grade level

Basic: Partial mastery of …Proficient: Solid academic performance and

demonstrated competency over challenging subject matter

Advanced: Superior performance

Source: National Center for Education Statistics, 2009

26

www.tqsource.org2727

Reading 2009 Grade 4

0102030405060708090

100

Black Hispanic Native American

White Asian/ Pacific

Islander

Students With

Disabilities

53 52 48

23 21

66

32 3230

3631

22

13 1417

3131

10

2 2 510 17

2

< Basic

Basic

Proficiency

Advanced

Percentage

Source: National Center for Education Statistics, 2009, Table A-12

www.tqsource.org

Preparation of Special Education Teachers in Scientifically Based Reading Instruction

in 27 Institutions of Higher Education (IHEs)

28

www.tqsource.org

Reading Course Syllabi: Projects

Explain your philosophy of literacy.Develop bulletin board to motivate

children to read.Produce journal explaining personal

experience in learning to read.Analyze the social justice implications

of literacy.

29

www.tqsource.org30

TQ Research & Policy Brief

Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality.

30

http://www.tqsource.org/publications/June2007Brief.pdf

www.tqsource.org

Current Special Education Practice With High-Incidence Disabilities

Review of special education case files for the state of Washington for a court case

Randomly selected 900 special education students• Ten districts were represented.• All students were in special education for at least 12

months.• Case files varied from 50 to 1,100 pages.• The review evaluated individualized education

programs (IEPs) using checklist for required components and evidence of formative evaluation.

• How many graphs?

31

www.tqsource.org

Review of Special Education Case Files: Results

Little evidence of systematic, direct instruction or behavior interventions using problem solving

Assessment and formative evaluation nearly nonexistent (11 of 870 cases had graphs.)

Lots of test protocols documenting weaknessesLittle objective evidence of positive outcomes

(i.e., benefits of special education are largely undocumented in high incidence.)

No assessment of progress

32

www.tqsource.org

National Mathematics Advisory Panel (2008) Report Key

FindingsConceptual understanding, computational and

procedural fluency, and problem solving skills are equally important and mutually reinforce each other.

Students should develop immediate recall of arithmetic facts to free the “working memory” for solving more complex problems.

Teachers’ regular use of formative assessments can improve student learning in mathematics.

33

www.tqsource.org

National Mathematics Advisory Panel (2008) Report Key

FindingsExplicit instruction for students who

struggle with mathematics is effective in increasing student learning.

Teachers should understand how to provide clear models for solving a problem type using an array of examples, offer opportunities for extensive practice, encourage students to “think aloud,” and give specific feedback.

34

www.tqsource.org

Summary: Teacher Preparation and Practice

Insufficient use of evidence-based practices in teacher preparation/comprehensive professional development and practice

TQ Center use of evidence-based innovation configurations to address these issues (See the TQ Center Special Education Resource List.)

Improved implementation of evidence-based principles leading to improved outcomes

Major Challenge: Narrowing the gap between what is known about evidence-based instruction and teacher preparation and special education practice

35

www.tqsource.org

References

Chall, J. S. (1967). Learning to read: The great debate. New York: McGraw-Hill.

Kavale, K. (2005). Effective intervention for students with SLD: The nature of special education. Learning Disabilities, 13(4), 127–138.

National Center for Education Statistics. (2009). The nation’s report card: Reading 2009─National Assessment of Educational Progress at Grades 4 and 8 (NCES 2010-458). Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved July 1, 2010, from http://nces.ed.gov/nationsreportcard/pdf/main2009/2010458.pdf

National Mathematics Advisory Panel. (2008). The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education. Retrieved July 1, 2010, from http://www.ed.gov/about/bdscomm/list/mathpanel/report/final-report.pdf

36

www.tqsource.org

References

Reschly, D. J., Smartt, S. M., & Oliver, R. M. (2007). Innovation configurations to improve teacher preparation in reading, behavior management, and inclusive practices. In C. A. Dwyer (Ed.), America’s challenge: Effective teachers for at-risk schools and students (pp. 23–45). Washington, DC: National Comprehensive Center for Teacher Quality.

Smartt, S. M., & Reschly, D. J. (2007). Barriers to the preparation of highly qualified teachers in reading (TQ Research & Policy Brief). Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved July 1, 2010, from http://www.tqsource.org/publications/June2007Brief.pdf

Snow, C. E., Burns, M. S., Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press.

Walsh, K. Glaser, D., & Wilcox, D. D. (2006). What education schools aren’t teaching about reading and what elementary teachers aren’t learning. Washington, DC: National Council on Teacher Quality. Retrieved July 1, 2010, from http://www.nctq.org/p/docs/nctq_reading_study_app.pdf

37

www.tqsource.org38

Lynn Holdheide, Vanderbilt UniversityP: 615-322-8150E-Mail: lynn.holdheide@vanderbilt.edu

Dan Reschly, Vanderbilt UniversityP: 615-322-8169E-Mail: dan.reschly@vanderbilt.edu