HOW TO NORM RUBRICS

Ursula Waln, Director of Student Learning Assessment

Central New Mexico Community College

WHAT IS A RUBRIC?

A rubric is a scoring guide such as: A checklist A rating scale A matrix or list containing descriptions of student

work characteristics at different levels of sophistication, a.k.a. a descriptive rubric• Holistic (containing performance levels but not

separating out criteria)• Analytic (providing descriptions for each criterion at

each performance level)

Checklist

Sanitized hands

Verified the patient’s fasting status

Asked about latex sensitivity

Selected appropriate gloves and tourniquet

Assembled necessary supplies

Positioned the patient properly

Descriptive Rubric  Beginning

(1)Developin

g(2)

Proficient(3)

Consideration of Diverse

Points of View

Wholly dismisses or disparages points of view that diverge from own worldview

Identifies valid components of differing perspectives but responds in accordance with own worldview without reflection

Analyzes the complexity and validity of differing perspectives and re-evaluates own perspectives in light of alternative worldviews

Rating Scale  Beg

inning(1)

Developing(2)

Proficient(3)

Developed key ideasAddressed important detailsOrganized information logicallyUsed proper writing mechanics

OBJECTIVE VS. SUBJECTIVE SCORING

And Selecting the Right Tool

With the exception of checklists, rubrics are used to lend a level of objectivity to evaluation that is inherently subjective Checklists are for use when the demonstration of learning either

is or is not present, with no in-between degrees of manifestation• Checklists do not require norming

Rating scales are the most subjective because they rely on the scorer’s interpretation of the performance-level headings

Descriptive rubrics can essentially eliminate subjectivity by clearly identifying indicators of distinct levels of performance

WHY NORM A RUBRIC?

Rubric norming is called for when more than one person will be scoring students’ work and the results will be aggregated for assessment of student learning To develop shared understanding of the outcome(s)

assessed To discover any need for editing of the rubric To develop scoring consistency among raters• Minimize the variable of individual expectations regarding rigor• Minimize potential for differences in interpretation of criteria

tied to identification of performance levels

NORMING RUBRICS OR NORMING RATERS?

Norming rating scales = developing consensus among raters What the different performance levels are intended to capture What level of rigor should be applied in distinguishing the levels All who will be raters should be involved in the norming session(s)

Norming descriptive rubrics = perfecting the rubric The better written the rubric, the less possibility it allows for

differences in interpretation• Identify and fix gray areas and ambiguities in the rubric • Reduce need for scorers to conform to a group standard for interpretation

WHAT YOU’LL NEED

A rubric A facilitator Some work samples Student learning outcome statements to which

the rubric is tied An outline of the steps of the norming process Raters (the faculty who will be doing the scoring)

SELECTING WORK SAMPLES

Real student work or mock-up samples If using real student work, redact any identifying

information

Select samples that demonstrate different performance levels Plan to have 1 to 3 for each of 2 to 4 scoring sessions

• Determine number based on the time and complexity of scoring

If the rubric contains multiple criteria, select samples that display differing levels of performance on differing criteria (i.e., samples that are neither all good nor all bad)

OUTLINE OF THE STEPS

I. Orientation

II. Discuss levels and rating criteria/thought processes

III. Score the samples

IV. Compare scores

V. Discuss (and possibly modify the rubric)

VI.Repeat above two steps as needed until consensus is reached

ORIENTING THE RATERS TO THE PROCESS COMPONENTS

The student learning outcome(s) being assessed The purpose of the norming session The rubric itself

How it came to be Its intended alignment to the SLO(s) Its intended use

An outline of the process

DISCUSSING LEVELS & CRITERIA

Part of the Orientation

Model the thought processes involved in using the rubric

Entertain perspectives regarding: The number of performance levels and their

headings The construct validity of the criteria that have been

identified Perceived distinctions between performance levels Perceptions regarding how the faculty think most of

the student work will be scored and the discriminative value of the rubric

SCORING THE SAMPLES

All raters score the same 1 - 3 samples concurrently

Raters refrain from discussing the works and/or their scores during the scoring sessions

Each time, samples provide a range of skill demonstration

Start with the most straight-forward samples and work up to those that require more refined decision-making

COMPARING SCORES

Look for consistencies and inconsistencies Confirm and summarize the rationale behind

consistencies Ask raters to articulate the rationale behind

inconsistencies Review the scoring criteria Encourage discussion

RECONCILING INCONSISTENT SCORES

Descriptive rubrics: Can the criteria or descriptions be revised in a way that produces agreement? Strive for natural delineations in performance that

reflect discernable steps or benchmarks in the development of proficiency

Rating scales: Can the criteria be revised in a way that reduces the opportunity for rater bias Strive for consensus or at least democratic agreement

IF THE PROCESS GETS STUCK

Part of Reconciliation

If disagreement persists, change course Consider breaking to calculate percentages of

agreement• May help raters calibrate their judgments by seeing how

their scores compare to those of other raters• Calculate the percentage of agreement between pairs of

raters and then calculate the mean of the percentages overall (shown on next slide)

Consider asking for all to commit to a democratically established convention for scoring in this particular context• Those who are the outliers agree to disagree but

nonetheless concede to score in accordance with the majority’s bias for the sake of consistency

PERCENTAGE OF AGREEMENT

How to Calculate It

Raters Compared

Number of Agreements

Total Items Percentage of Agreement

1 & 2 3 7 14%

1 & 3 6 7 86%

1 & 4 6 7 86%

2 & 3 4 7 57%

2 & 4 4 7 57%

3 & 4 7 7 100%

Overall 30 42 71%

FINAL NOTES

Rubric norming is a means for creating solidarity among the faculty regarding what the shared goals are (as reflected in the student learning outcome statements) and what student performance looks like when the outcomes are partially versus wholly achieved.

Rubric norming is most effective when assessment is steeped in an ethos of inquiry, scholarly analysis, and civil academic discourse that encourages faculty participation in decision-making.