using rubrics for evaluating student learning. purpose to review the development of rubrics for the...
TRANSCRIPT
Using Rubrics for Evaluating Student Learning
Purpose
To review the development of rubrics for the purpose of assessment
To share an example of how a rubric can be developed for use in assessment
To show how rubric assessment can be quantified and shared
To encourage use of this method as a form of direct assessment of student learning
Why Rubric Assessment?
Rubrics…Provide standardized information about student learning on specifically defined student learning objectivesEmphasize department/program controlCan link to other programs (i.e., general education program objectives)Form of direct assessment of student learningCan be directly linked to program objectivesResults are more easily reported
Defining Assignment To Be Assessed
Should be a critical assignment that includes several student learning/program objectivesFaculty should agree with the utility of the assessmentShould allow for simple measurement (i.e., descriptive statistics, etc.)Some additional comparisons could be made (i.e., comparisons between freshman and senior performance, evening and day courses, on-line and traditional courses, etc.)
Match Assignments With Program Objectives
Rubric elements should match with student learning objectivesFaculty should play significant role in developing and implementing rubricCritical question: Is rubric comprehensive enough to assess several objectives?Caution: Do not “shoehorn” a rubric by including categories that are difficult to assess in the first place. It is fine if all objectives are not measured by one instrument.When developing a rubric, it is always (therefore) a good idea to make sure that a program is assessing as much as it can, given the amount of effort it may take to develop and then implement the rubric on a regular cycle. A fair amount of work is involved, but it is worth it!
Brief Case Study
Case: A program has a number of learning objectives it needs to assess. It has chosen a research project students must complete during their senior year. The assignment requires students to put forward a point of view, and support the argument by critically evaluating competing points of view, use an effective research method, and apply appropriate content knowledge specific to their subject area.The faculty feel that they are clear on the purposes of the assignment, and what each purpose means, so it collaboratively creates a rubric to evaluate these projects.
Example
Grading Rubric for GE420
Target Performance
Satisfactory Performance
Unsatisfactory Performance
Use of statistical knowledge (10 points)
Paper contains accurate use of descriptive and inferential
statistics to make argument. (9-10 points)
Paper uses descriptive statistics to make argument,
and contains only a few errors. (6-8 points)
Paper either uses no statistics to make its argument, or contains major errors in
descriptive and/or inferential statistics that detract from the
paper’s argument (5 points or less)
Critical evaluation of opposing points
of view (20 points)
Paper addresses opposing points of view, recognizing
strengths and weaknesses with accuracy
(18-20 points)
Paper addresses opposing point of view recognizing strengths and weaknesses;
there are some inaccuracies (12 to 17 points)
Paper does not recognize opposing points of view, or addresses only one point of view; there are numerous
inaccuracies (11 points or less)
Advocacy of own point of view (20 points)
Paper has a clear thesis that is advocated consistently
throughout (18-20 points)
Paper has a thesis that is advocated within the paper
(12 to 17 points)
Paper does not contain a clear thesis,
(11 points or less)
Content knowledge (40 points)
Paper contains no factual or logical errors; use of factual
information arises from a clear consideration of multiple
sources (36-40 points)
Paper contains only a few factual errors; use of factual
information arises from multiple sources.
(24 through 35 points)
Paper contains numerous major factual errors; use of
factual information arises from a limited number of sources
(23 points or less)
Research methodologies
(10 points)
Paper contains comprehensive literature review that clearly
leads to identification of appropriate research
methodology; research methodology addresses
research task and is effectively implemented (9-10 points)
Paper contains literature review that leads to
identification of research methodology; research methodology addresses
research task with only a few concerns about effective
implementation (6-8 points)
Paper contains literature review that is incomplete and
unbalanced; research methodology is faulty and ineffectively implemented
(5 points or less)
Categories chosen according to student learning/program objectives of specific department or program. Each category is assigned different values (or weights) in order to arrive at final score. This is not necessary if the purpose of the rubric is to examine each individual category, however.
Columns chosen according to degree. In this hypothetical case, the faculty chose “target,” “satisfactory,” and “unsatisfactory.” When describing what these different degrees mean, it is important to discuss what is ideal, what is satisfactory, and what is unsatisfactory. When filling in each cell in a rubric, it may be best to define the extremes first—target and unsatisfactory—and then fill in the satisfactory cell (Harder and Harper, ISU Assessment Workshop, April 2004)
When filling in each cell, care must be taken to be very specific about expectations. When discussing this, faculty and evaluators may want to discuss the meaning of key words, avoiding vague wording and sentences that may contradict. For example, the wording within each of these cells may be unclear. 2-3 faculty members may want to review it to clarify some of the terms.
Choosing How and When To Use Rubrics
When using a rubric as a method of assessing department/program goals, one may want to perform the following actions:
Require the use of the rubric in all course sections in which the assignment is required
Capstone courses can be used in this respectRequires cooperation among all faculty teaching the courseA department may use selected courses (not all sections), although it is important to know which courses are not being included to anticipate any bias that might result from such an arrangement.
Choose a random sampleCollect samples from all students and store in a central locationSelect a random sample from this collectionDefine periodicity (every year, every two years, three?)Select evaluators (faculty or outside qualified evaluators)
Pre-test the rubric• Test rubric for inconsistencies by evaluating
a couple of assignments first• Later test for inter-rater reliability
Use of Rubric Data
Final totals can be used to observe what grades students have receivedSub-category totals are most useful because they assess student learning on each objective (especially if each row in rubric corresponds with a program/student learning objective)
Example
In the following example, a random sample of thirty final papers was chosen and graded according to the rubric developed in a prior slide. Totals were inputted on an excel spreadsheet. Percentages were then calculated by dividing the average number of points for each category by the total number of assignments graded (in this case, 30). This presents a percentage between 0 and 1, allowing for comparison among the categories.
Rubric Results
Rubric Averages by Sub-Category and Total
0.750.81 0.83
0.88
0.73
0.83
0.500.550.600.650.700.750.800.850.900.95
Use
of
Sta
tistic
alK
now
ledg
e
Crit
ical
Eva
luat
ion
Adv
ocac
y
Con
tent
Kno
wle
dge
Use
of
Res
earc
hM
etho
dolo
gies
TOTA
LA
VE
RA
GE
Categories
Per
cent
age
Rubric categories are included on a graph (a table is also fine, of course). Categories can be compared by taking the average for each category and dividing by the total number of responses. The differences among the categories can then be discussed by faculty in a faculty meeting. For example, in this group of histograms, faculty might conclude that students appear to perform well on content knowledge, advocacy, and critical evaluation, but may need to improve in respect to using statistical knowledge and use of research methodologies. Of course, as always, it is also helpful to use additional forms of assessment to “triangulate” this finding.
A total average is included here to compare with each sub-category in case there is interest.
Demonstrating Value Added
It may be a good idea to compare freshman (or entering student) performance with that of seniors. One can perform statistical tests to see if there is improvement between students at entry in a program and senior studentsAssumes that entering and exiting students complete similar assignments, requiring a good deal of coordinationExcellent method of demonstrating impact of a program on student learning
Conclusion
Rubric-based assessment requires a good deal of coordination among facultyRequires specific identification of student learning goals and how they are to be measured. Despite high coordination costs, very valuable method of direct assessmentOnce implemented, comparatively easy to implement down the road assuming consistent leadership and implementation.As always, any assessment has little value until findings are discussed with faculty and results incorporated into some result, such as the program review process, enhancement of program offerings, changes in teaching emphases, etc.
Contact Information
Dr. Sean McKitrick, Assistant Provost for Curriculum, Instruction, and Assessment
(607) 777-2150