ASSESSING THE CONTINUED EFFICACY OF

AUNIVERSITY CENTER MODEL OF TEACHER

PREPARATION PROGRAM

Diana LysKristen Cuthrell

Laura Bilbro-Berry

2

PURPOSE OF THE STUDY Builds upon

Strand 2 – Demonstrating Effectiveness and Relevance by engaging in the process of continuous program improvement.

Prior research which noted the need for more valid and reliable assessments upon which to base program pathway comparisons.

As a new teacher performance assessment – edTPA – is implemented, do candidates in different program pathways continue to have comparable outcomes?

3

ANTECEDENT RESEARCH-UNIVERSITY CENTER MODEL Boyd, Lankford, Loeb and Wyckoff (2005)–

geography of first employment to hometown and/or college

Lorenzo (2005) – co-location models provide level access

Grady (2005) and Vaughan (2006) – barriers to community college transfer

Locklear (2007) and Locklear et al (2009) – university center models as comparable preparation programs

Bill and Melinda Gates Foundation; With Their Whole Lives Ahead of Them (2009) – reasons for leaving college early; lack of persistence

4

ANTECEDENT RESEARCH:TEACHER PERFORMANCE ASSESSMENTS When assessing preservice teachers, it is important to

evaluate their knowledge and skills, student learning, professional dispositions, and reflective practices (Darling-Hammond, 2006).

Performance assessments provide documentation of the teachers performance, note progress toward reaching the program goals, and dissect the program’s strengths and (Darling-Hammond, 2006).

Portfolio assessments are one type of performance-based assessment, used for formative, summative, and predictive assessment (Bannink, 2009).

Portfolio assessment may be beneficial for certain aspects of the teaching certification process, such as documentation of planning and examples of instruction, but may not be valid for the assessment of teacher competencies (Yao, Thomas, Nickens, Downing, Burkett, & Lamson, 2008).

5

TRADITIONAL ELEMENTARY EDUCATION PROGRAM OVERVIEW Includes face to face coursework in pedagogy,

knowledge, and skills . Majority of junior and senior level coursework

includes practica experiences; many supervised by faculty.

Spirally instruction is woven throughout program courses in observational skills, planning for diverse learners, research based instructional strategies, Common Core curriculum, classroom assessment, and differentiation.

Emphasis on classroom management occurs at the senior level.

Candidates participate in a year long internship: Senior 1 (1 day a week) and Senior 2 (5 days a week)

6

UNIVERSITY CENTERPROGRAM OVERVIEW Five hub site community colleges, each with

several spoke site CCs creating regional consortia

Each hub site has an IHE employee that works full-time on the CC campus, recruiting, advising, marketing in the region

Cohort model used; 14 current cohorts exist On-line delivery; same program taught by

same faculty Part-time delivery model requires 3 ½ years

to finish “2” 443 graduates; 77% teaching within N.C.;

95% teaching in rural eastern N.C.

7

IMPLEMENTING EDTPA IN A LARGE ELEMENTARY EDUCATION PROGRAMComparing Program Pathways

8

BRIEFLY, WHAT IS EDTPA? Capstone, summative performance assessment

portfolioLinks theory to practiceIncludes 3-4 tasks requiring candidates to plan,

instruct, and assessCandidates must video record themselves teaching

lessons they plan for a specific group of students. Nationally validated instrument developed at Stanford

University as a measure of teaching proficiency at the pre-candidate level

Results are summative for candidate and formative for programs

Currently over 25 states and 180 teacher preparation programs have adopted or considering adoption of the edTPA

9

HOW IS EDTPA SCORED? 2012 TPA Field Test Handbooks Evaluators rate candidates’ performances on

planning, instruction, assessment, analyzing teaching, and academic language in 12 rubrics.

Each item based on a 5 point scale: 5= stellar candidate4= solve foundation, knowledge & skills3= acceptable levels to begin teaching2= some skills, more practiced needed1= struggle candidate not ready

10

HOW WERE CANDIDATES PREPARED FOR EDTPA? Revised teacher education curricula was

aligned with: Common Core State Standards 21st Century Skills

ISLES modules developed as part of TQP Grant Curriculum Reform ISLES 3 aligns with edTPA Task2

Instructional Coaching Support for Candidates in Partner Districts

11

SAMPLE AND DATA ANALYSIS

12

STUDY SAMPLE AND METHODS Utilized preexisting integrated assessment system

databases on candidate performance, competence and descriptive characteristics.

JMP Pro 9 provided the ability to compare the two sections using matching student ID analysis.

Sample included 132 elementary education degree completers (74 Non-WPE and 58 WPE) Fall 2012 semester .

Dataset included teacher performance data: edTPA assessment scores, test scores, GPA, internship grades, and demographic data.

90 participants were randomly selected for analysis, 45 WPE and 45 WPE candidates.

14

AGE CHARACTERISTICS

15

ETHNICITYCHARACTERISTICS

16

FINAL INTERNSHIP GRADE DIFFERENTIALS

A A- B B+ C0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

WPE NON-WPE

GPA Compari-son

* WPE: 3.75

* Non-WPE: 3.310

18

ACADEMIC CHARACTERISTICS

N % of Total M SD Range Min Max N % ot Total M SD Range Min Max

Test ScoresACT Math 1 11.38% 19 0 19 19 7 88.62% 21.143 4.880 11 16 27ECU-Math Placement Math Lab 0 0ECU-Math Placement Orientation 0 20 100.00% 17.042 3.651 12 12 24Elem Ed Instr Pract App (5015) 44 54.45% 176.12 8.70 38.33 155.67 194 37 45.55% 175.23 10.36 37.67 154.33 192Fundamental Subj: CK (5511) 1 100.00% 158 0 158 158 0SAT Mathematics 9 19.53% 471.11 133.36 360 330 690 34 80.47% 513.82 60.05 220 390 610Sp Ed Core K Mild Mod GC(5543) 1 100.00% 174 0 174 174 0

Grades and Test Score DifferentialsWPE Non-WPE

19

EDTPA RUBRIC SCORES Assessed candidate’s performance in each

content area by comparison analysis of each rubric score.

Evaluation includes mean, standard deviation, and percent total.

Significant results: Rubric 8- Assessment: Using feedback to

guide further learning (WPE, M=3.60. Non-WPE, M=3.36)

Rubric 12-Academic Language: Developing student’s academic language and literacy (WPE, M=3.56. Non-WPE, M=3.26)

20

WPE V NON-WPE – ALL RUBRICS

1 2 3 4 5 6 7 8 9 10 11 12 AVG3.000

3.100

3.200

3.300

3.400

3.500

3.600

Student Performance Assessment as Determined in the Evaluation of edTPA Rubric Scores

Mea

n

edTPA Rubric

WPE NON-WPE

22

RUBRIC 12- ACADEMIC LANGUAGE: DEVELOPMENT STUDENTS’ ACADEMIC LANGUAGE – ALL CANDIDATES

WPE (n=45) Non-WPE (n=45)0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

2.2%8.9%

26.7%

42.2%

66.7%

44.4%12345

t-Test Results (95% CI)d-.311df-88.00P-value-.027χ² -5.149

Competence Scale

24

WPE V NON-WPE, WITH “NOT MET” REMOVED Further analysis shifted from program

pathway comparison to proficiency of candidates. To focus on proficiency, candidates who were not proficient on the edTPA were removed from the analysis.

Excluded 3 “not met” candidates to prevent skew data. WPE: N=44Non-WPE: N=43

Significant results: Rubric 8, 10, & 12

25

RUBRIC 8- ASSESSMENT: USING ASSESSMENT TO INFORM INSTRUCTION-PROFICIENT CANDIDATES

Competence Scale

t-Test Results (95% CI)d-.0263df-84.49P-value-.048χ² -2.813

26

RUBRIC 10- ACADEMIC LANGUAGE: UNDERSTANDING STUDENTS’ LANGUAGE DEVELOPMENT AND ASSOCIATED LANGUAGE DEMANDS- PROFICIENT CANDIDATES

t-Test Results (95% CI)d-.225df-76.75P-value-.062χ² -3.718

Competence Scale

27

RUBRIC 12- ACADEMIC LANGUAGE: DEVELOPMENT STUDENTS’ ACADEMIC LANGUAGE-PROFICIENT CANDIDATES

t-Test Results (95% CI)d-.034df-83.58P-value-.0017χ² -5.639

28

IMPLICATIONS FOR OTHER PROGRAMS

29

ISSUES TO CONSIDER WHEN ADDRESSING PROGRAM PATHWAYS Impact of online delivery

Feasibility of supervised practica? Level of support from faculty

How do we provide supports to DE students that are provided in face to face forums?

Candidate developmentAre DE students more effective

independent learners in working through edTPA handbooks than nonDE candidates? If so, what traits could be utilized/taught in face to face instruction?

30

ISSUES TO CONSIDER WHEN UTILIZING EDTPA DATA Program gateways

Should we have stronger gateways in Junior level classes that are aligned with edTPA?

Candidates CharacteristicsWhat is occurring in which WPE candidates

outperform nonWPE candidates on Academic Language rubrics?

Individual Rubric Scores vs. Total AverageWhat considerations should be made when

using edTPA scores?How will this analysis change with shift to

Operational Handbooks with 15 rubrics?

31

OUR NEXT STEPS

32

WHAT’S NEXT?

1. Future research should continue to study efficacy of the model with larger population samples.

2. Future research should investigate the validity and reliability of our performance measures and assessments.

3. Future research should expand to address other recruitment and retention factors that may influence enrollments.

33

QUESTIONS?

34

CONTACT INFORMATIONMs. Laura Bilbro-Berry252-328-1123bilbroberryl@ecu.edu

Dr. Diana B. Lys252-328-2037lysd@ecu.edu

For a copy of this PowerPoint presentation, please email Dr. Diana Lys

Dr. Kristen Cuthrell252-328-5748cuthrellma@ecu.edu