The Power of Reliable Rubrics Promoting Equitable and Effective Assessment

Practices through Collaboratively Designed Rubrics.

Whitney Bortz, Ph.D. whitneyelaine8@gmail.com

Radford University, CEHD Nov 2015

Objectives of the Session

• Experience how rubrics can improve the assessment process

• Encounter a model for collaboratively writing tools, introducing tools to faculty and testing reliability using a data management system

• Consider how rubric assessment can be improved in our own institutions

Questions

• How many of you work directly with teacher preparation programs at your institution?

• How many of you play a key role in preparing for CAEP or SPA (Specialized Professional Association) accreditation?

• How many of your teacher preparation programs regularly use detailed rubrics to validate common assessments?

• What about outside of teacher preparation?

Context

• School of Teacher Education and Leadership (STEL) at Radford University

• Formerly accredited by NCATE (National Council for the Accreditation of Teacher Education)

• Now working toward accreditation from Council for Accreditation of Educator Preparation (CAEP)

• 13 Licensure areas • CAEP requires common assessments across all initial

licensure programs that demonstrate candidate performance on the Interstate New Teachers Assessment and Support Consortium (InTASC) standards

The Challenge

• STEL has 13 initial licensure programs • 4 – 6 Advanced programs • CAEP requires that each assessment uses a detailed

rubric/scoring guide • Rubric should have 3 – 5 rating levels with clear,

distinguishable descriptions of performance in each level on each indicator

• Creating such rubrics requires much time and effort (Reddy & Andrade, 2010)

• Should be valid, reliable and fair (Andrade, 2005) (Also, required by CAEP)

Valid and Reliable Rubrics

• Validity – Does the assessment tool measure what it intends to measure?

• Reliability – Will the use of the assessment tool produce consistent results across instances?

• Why? Judgments are inherently subjective (Turbow, 2015)

Rate the painting below on a scale of 1 (bad) to 4 (great).

Reflection

• Did the rubric help you confidently rate the painting? Explain.

• How might the rubric have been helpful

to the student artist?

Benefits of Rubrics

• Positively influence the learning process (Jonsson and Svingby, 2007)

• Play a key role in formative assessment (Sadler, 1989) • Assist in the feedback process, specifically (Hattie &

Timperley, 2007; Shute, 2008) • Can increase the quality of student performance

(Petkov and Petkova, 2006) • Can also help identify programmatic areas for

improvement (Song, 2006; Andrade, 2005; Powell, 2001)

• Others?

Formation of the Rubric Writing Team

• Director of STEL, Director of Field Experience, Director of Assessment, Six Faculty Members in Teacher Preparation

• Interdisciplinary • Disciplinary Experts • Hand-selected

Planning and preparation

– Internal grant funding – Established goals and tasks – Collaboratively wrote grant proposal – Established roles and expectations – Training

What makes a good rubric?

According to CAEP (Chepko, 2014)… • Appropriate - Alignment to standards • Definable - clear, agreed-upon meaning • Observable – quality of performance can be

perceived • Distinct from one another – each level defines

distinct levels of candidate performance • Complete – all criteria together describes the

whole of the learning outcome

Rubric Writing Tips

• Start from the middle and work out (Chepko, 2014; Tomei, 2014)

• Changes in levels – Additive – each levels adds more advanced behaviors – Qualitative – the quality of the behavior changes in each

level • The lowest level should still contain a description of

what the rater could expect to see rather than simply the “absence” of something.

• Align the rubric and the assignment guide, if applicable

Common Issues

• Double- or multiple-barrelled criteria or descriptors in one row

• Use of subjective language • Performance descriptors do not encompass all possible

scenarios • Overlapping performance descriptors • Use of technical language that may be interpreted

differently across students and/or raters • Others?

The Writing Process

1. Split into three groups of two to three 2. Draft the three assessment tools 3. Send to whole group for feedback 4. Meeting with a Consultant 5. Groups switch assessments for further editing 6. Whole group review and commenting 7. Final edits

Launching the Rubrics

1. Roll out to Faculty 1. Accompanied by a guidance document 2. Presented as a “suite” of assessments

2. Requesting Feedback 3. Inter-rater reliability exercise 4. Validity

Inter-rater Reliability Process

Select Student Work

Score Independently

Share ratings

Discuss and Approach Consensus (Turbow,

2015)

Change assessment

tool, as needed

Inter-rater Reliability Exercises • Two sample lesson plans (Early Childhood and Middle

School Math) • Rated independently • Shared ratings (see handout) • The Delphi Technique - until consensus was reached

(Hsu & Sandford, 2007) • Consensus often required changes to the rubric and/or

to the assignment guide • Correlation – Spearman’s Rho (see handout)

Next Steps

• Cohen’s Kappa Coefficient: Inter-observer agreement?

• Next steps: Repeat with two new student artifacts using the modified rubric

• Invite other stakeholders to test the tools

Validity (Allen & Knight, 2009)

Focus: 1. Insure the academic concepts and dimensions are

learning skills students need (employer data and professional standards)

2. Insure these are professionally anchored (standards and experts)

Validity (Allen & Knight, 2009)

1. Develop learning objectives 2. Identify a sample of work 3. Develop an evaluation rubric 4. Longitudinally test student learning against professional

assessments 5. Identify problems with sub-optimal learning

performance 6. Improve the construct validity for the rubric 7. Determine the ability of the rubric to differentiate

submissions 8. Improve the reliability of the rubric with data analysis

to further validate the rubric

Professional Learning • Community of Practice (Wenger,

1998, p. 5) • Wenger describes individual

development as the negotiation of meanings through practice

• Negotiation of – Expectation – Meaning – Priority – Performance

Learning

Community learning as belonging

Identity learning as becoming

Meaning learning as experience

Practice learning as

doing

Discussion and Sharing

• Describe rubric use at your institution?

– How are rubrics created? – How are they used? – What benefits have you seen?

• What are some areas of challenge or areas for growth?

Resources

• http://www.introductiontorubrics.com/samples.html • https://www.aacu.org/value-rubrics • http://manoa.hawaii.edu/assessment/howto/rubrics.htm

References

• Allen, S. & Knight, J. (2009). A Method for collaboratively developing and validating a rubric. International Journal for the Scholarship of Teaching and Learning, 3(2), 1 – 17. Retrieved from: http://www.georgiasouthern.edu/ijsotl

• Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53(1), 27–31.

• Chepko, S. (2014, September). Quality Assessment Workshop. Workshop presented at the annual meeting of the Council for Accreditation of Educator Preparation, Washington D.C.

• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.

• Hsu, C. & Sandford, B. (2007). The Delphi Technique: Making sense of concensus. Practical Assessment, Research & Evaluation,12 (10), 1 – 8.

• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.

• Petkov, D., & Petkova, O. (2006). Development of scoring rubrics for IS projects as an assessment tool. Issues in Informing Science and Information Technology, 3, 499–510.

References

• Powell, T. A. (2001). Improving assessment and evaluation methods in film and television production courses (Unpublished doctoral dissertation). Capella University, Minneapolis.

• Song, K. H. (2006). A conceptual model of assessing teaching performance and intellectual development of teacher candidates: A pilot study in the US. Teaching in Higher Education, 11(2), 175–190.

• Tomei, L. (2014, October). Rubric Design Webinar, Part II. Presented online on behalf of LiveText.

• Turbow, D. (2015, February). Introduction to Rubric Norming. Presented online on behalf of LiveText.

• Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Retrieved from

http://books.google.com/books?hl=en&lr=&id=heBZpgYUKdAC&oi=fnd&pg=PR11&dq=wenger+1998+&ots=kelf2kew5d&sig=6o4HfPARHoP2zkgas2Gy7DdCx3s