an implementer’s perspective: an evaluation and compensation pilot

AN IMPLEMENTER’S PERSPECTIVE:

AN EVALUATION AND COMPENSATION PILOT

October 17, 2012Richard Bowman, Ph.D.

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Context• Reform efforts

• Teacher Evaluation Task Force (TETF)• Race To The Top (RTTT)• Planned legislation• Prior legislation• Shared urgency

• Four schools• SIG application

• “Funds have been set aside to support performance pay.”• $1,000 per teacher assigned to the school. (~ $300K)

• Strategic Data Project Data Fellows• Highly trained and capable personnel pressed into service

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

The Proposal• Performance compensation• Teacher evaluation• Pilot program• Multiple measures

• Student Surveys• Observations• School Value Added• Individual Value Added• PLC Student Learning Goals (SLOs)• Individual Student Learning Goals (SLOs)

• In-depth research

The Proposal at 50,000 feet

The Proposal at 10,000 feet

Student Surveyson the ground

I wonder who translated

this?

Student Surveyson the ground

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Proposal Results

Most Valued: Student Surveys

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Union Leader Buy-in (Negotiations)• Voluntary participation• Student Learning Goals (SLOs)• Hold-harmless evaluation• Bonus, not pay• Percentages• Ferguson’s research

Teacher Buy-in• Emphasize shared background• Time – personal time spent• Focus on concerns

• Negative consequences• Time taken

• Talk context• Responsiveness• Transparency

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Logistics - Survey• One friendly Denver SDP fellow• Two local SDP fellows• Four schools• Eight survey versions• 16 survey days• 93 participants• ~20,000 student surveys

• Blank photocopied surveys• Rough class count stuffed in envelopes• Envelopes sorted by period and teacher

• ~20,000 student barcode labels• Cut by guillotine and bagged by class period

Logistics - Software• Excel (Report production)

• Lookup tables• VB Macros

• RDBMail (Outlook Integration)• PDF generation

• Google Docs/Drive (Collaboration)• Stata (Data analysis and management)• Remark Office OMR (Optical recognition)

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Student Surveyson the ground

Metrics• Survey

• Average responses assuming a 1 - 5 valued scale.• Average for each of the “Seven Cs”• Composite average of the “Seven Cs”

• Reporting• Average for each of the “Seven Cs” and composite

• Individual• School-wide• Pilot

• Histogram of the composite• Top two relative strengths and weaknesses• No item-level detail

• Evaluation• Thirds of composite score within school

Metrics• Value-added Model

• Includes student, teacher, and school covariates• 1-year estimates for both teachers and schools

• Best Linear Unbiased Predictor (BLUP) aka. Bayesian or shrunken estimates• Relative to (normalized) district-wide student test scores

• By year, standard deviations• Random effects model estimated by REML• Estimates produced four times, based on SCA and SBA

• Reporting• Limited numbers

• Concern about meaningfulness of numbers• Conversion to months of learning

• Confidence level• Visual• Range of months of learning in text

• Histogram• School estimate and teacher estimates

• Evaluation• Three possible ratings

• Significantly above, significantly below, and not significantly different than average• 95% significance

Metrics• Student Learning Goals

• Teacher developed and measured and reported• Individual goals and PLC goals

• “Reach” goal – goal that would be a stretch to achieve• “Expected” goal – goal expected to be achieved

• Evaluation• Three possible ratings

• Met Reach Goal, Met Expected Goal, Did not meet either goal

Metrics• Observations

• Administrator evaluates teachers based on three domains of Danielson’s FFT

• Reported electronically• Ratings converted into a four-point scale and averaged

• Evaluation• Teachers reported whether or not observation happened

• Context• Proposal• Results• Buy-in• Logistics• Metrics• Issues

Observation Data

Issues• Training and Professional Development• Observation

• Principal time• No consequences• Length

• Student Learning Goals• Ensuring rigor and relevance• Professional development

• Surveys• Special Populations• Proctoring

• Value-added• Understanding, fairness, and use

• School-wide evaluations• Universally disliked

• Joint or shared evaluations• Largely disliked

Issues• How can this information inform instruction?• Evaluation Metrics

• Selection bias due to volunteers• Relative classifications

• With or without base year, creates problems• Absolute classification

Thank you!

Questions or Comments?