aligment of assessment with learning outcomes...use assessment tasks that also contain that verb,...
TRANSCRIPT
Aligment of assessment with
learning outcomes
Jadwiga Mirecka
Department of Medical Education
Jagiellonian Unversity Medical College
Learning Outcomes
• For the level of education
• For disciplines (eg.UK
Benchmark Statements,
Tuning projects)
• For the institution (graduate
outcomes)
• For the program
• For the module
• For the teaching unitAfter M.F.Frankowicz
A good balance of learning outcomes
Typical learning outcomes in higher education
• Knowledge, comprehension, application, analysis,
synthesis, evaluation, etc. (Bloom)
• Problem solving
• Working alone and in teams; personal and interpersonal
skills
• Communications; “information literacy”.
Which of these are included in your courses? Are all of
them assessed?
Seminario Internacional SCT, Pucón, Chile 31 Aug 2011
Why should we assess students’ progress?
Passed?
Failed?
66
TeacherPerspectives: Learning Outcomes Teaching Activities Assessment
StudentPerspectives: Assessment Learning Activities Outcomes
The relationship between Learning Outcomes and
assessment as seen by teacher and students
Biggs (2003)
Drivers of Outcomes Assessment
• A revolution in education: the student centered learning (teachers go from being the “sage on
the stage” to the “guide on the side”)
• Higher Education’s response to theTotal Quality Management movement ( Review, assess, improve … review,assess, improve …
• Historical role of accreditation in ensuring quality
“Constructive Alignment”
Constructive
• The students construct understanding for themselves through learning activities. “Teaching is simply a catalyst for learning” (Biggs).
Alignment
• Alignment refers to what the teacher does in helping to support the learning activities to achieve the learning outcomes.
Aligning the assessment with the learning outcomesmeans that students know how their achievements will be measured.
Seminario Internacional SCT, Pucón,
Chile 31 Aug 2011
(Biggs, 2005)
Stages in the design of constructively aligned
teaching and assessment
1. Describe the intended learning outcome in the form of a
verb (activity), its object (the content) and specify a
standard the students are to attain.
2. Create a teaching-learning environment that address
that verb and are likely to bring about the intended
outcome.
3. Use assessment tasks that also contain that verb, thus
enabling you to judge with the help of rubrics if and how
well they have achieved the intended learning outcomes
4. Transform these judgments into standard grading
criteria.
(Biggs 2007)
Choosing the form of assessment
• What will the student’s work /performance tell me about
their level of competence on the targeted learning
objectives?
• How will my assessment of their work help guide
students’ practice and improve the quality of their work?
• How will the assessment outcomes for the class guide
my teaching practice?
Example of Matching the Assessment
to the Learning Outcome
Learning outcomes
1. Demonstrate good
presentation skills.
2. Formulate food product
3. Identify an area for
research
4. Identify signs of
financial crisis
Assessment?a) Multiple choice
questions
b) Prepare a 1000-word research proposal
c) Lab-based project
d) Make a presentation to peers
Seminario Internacional SCT, Pucón,
Chile 31 Aug 2011
Constructively aligned program
Course LOs
Course LOs
Course LOs
Program LOs
Courses are
coordinated to
assure steady
development of
the learning
outcomes leading
to achievement of
the intended
program-level
learning
outcomes
Mapping Program-level LOs to Teaching,
Learning and Assessment
Programe
Learning
Outcomes
Degree
Level
Expectations
Teaching
Learning
Activities
Assessment
and
Evidence
Looking for Evidence that each program-level Learning
Outcome has been achieved. It is better not to assess
individual outcomes but focus on complex competencies
Some assessment methods are more suitable for single LOs
in courses, others for program-level LOs (portfolios and
capstone projects)
• Program outcomes are more global but should match a
specific domain
• Course outcomes are more specific
• Assessment of learning outcomes at the course level
can be aggregated (by learning outcome) to the level of
the program
Program vs. Course Outcomes
Aligning assessments with LOs.
• increases the probability that we will provide students
with the opportunities to learn and practice the
knowledge and skills that will be required on the various
assessments we design
• “good grades” are more likely to translate into “good
learning”
Assessment of LOs achievement
- essentially in the category YES/NO
- for all (representative) LOs if licensing is
considered
- for selected LOs (but students do not know which
of them will be chosen)
Types of assessment
• Formative vs Summative
• Criterion-referenced vs Norm-referenced
• Direct vs Indirect
• Holistic vs Atomistic
• Objective vs Subjective
• Quantitative vs Qualitative
• Longitudinal vs Transversal
• Divergent vs. Convergent Assessment
• Process Assessment vs Product Assessment
Norm- referenced Assessment
“Competitive assessment”
Compares students’ achievement with those of his /her
classmates
Student X has answered exam questions better than 25% of her colleagues , which means that she was better prepared
for 100MCQs
60 mean
Norm-referenced assessment
Mean result +/- 2 standard deviations
Can be also done by looking on “breaks” in distribution
“curve” assessment
failed v.good
Norm-referenced assessment
Can be used for students’ selection, comparison
Is not suitable for assessment of learning outcomes
achievements
Its results can hide very low level of students preparation
Is de-motivating for learning
Criterion-referenced assessment
“Assessment of competence”
Is based on standards established before exam
Student X has answered correctly 70% of questions, which means that she
achieved the required level of knowledge
Criterion-referenced assessment
• Reflects achievements on the required level
• Promotes group cooperation
• Gives information regarding efficiency of training
Criteria are best expressed in a numerical way:
• % of correct answers
• Number of points to be collected etc.
Structure of the Observed Learning
Outcome (SOLO)
• SOLO is a means of classifying learning outcomes in
terms of their complexity
• At first we pick up only one or few aspects of the task
(unistructural), then several aspects but they are
unrelated (multistructural), then we learn how to
integrate them into a whole (relational), and finally, we
are able to generalised that whole to as yet untaught
applications (extended abstract)
• This approach is helpful in implementing constructive
alignment
Assessment of complex achievements
• Alternative assessment applies to any and all assessments that differ from the multiple-choice, timed, one-shot approaches that characterize most standardized and many classroom assessments.
• Authentic assessment applies to assessments whichengage students in applying knowledge and skills in the same way they are used in the "real world" outside of school.
• Performance assessment is an assessment in whichstudents are given opportunities to demonstrate their understanding and to apply knowledge, skills, and habits of mind in a variety of contexts.
Authentic Assessment
Three factors determine the authenticity of an assessment:
• the task (possibly interdisciplinary.
• the context, (as realistic as possible including time available, access to resources, group work)
• the evaluation criteria (similar to those used for evaluation of professionals
Wiggins (1990),
Performance assessment
Requires students to demonstrate that they have mastered
specific skills and competencies by performing or
producing something
• designing and carrying out experiments;
• writing essays, reports, poems
• using equipment or a technique
• building models
• playing musical instruments;
Performance criteria must define the attribute(s) being
evaluated and also develop a performance continuum.
Multi source assessment (360-degree
feedback )• is a method of systematically collecting
opinions about an individual's performance from a wide range of
coworkers
• is utilized by organizations to solicit information from a variety of workplace sources on an employee's work-related behavior and/or performance
• In medicine feedback can be collected from: patients, secretary, senior doctors
Portfolios:
• A student portfolio is a systematic collection of student work and related material that depicts a student's activities, accomplishments, and achievements in one or more school subjects.
• Portfolios are very useful for summative assessment of attitides and behaviors
• Portfolio may contain various types of student work such as:
-essays;
-reports from laboratories
-research projects
-volunteering activities
-experience abroad
Self-assessment
„Some people are highly competent but are unaware of it,„
Gruppen says. "I am less concerned with those individuals than with the students who are poor self-assessors and are doing poorly. They are the ones most at risk for making dangerous mistakes out of overconfidence."
`
T. DeANGELIS
Can be performed in different forms:
• Peer ranking (each member ranks all others on one or
more factors)
• Peer nomination ( indicating a member highest on a
particular characteristics or dimension)
• Peer rating (rating each other using the scale)
• Voting with cards given to chosen classmates
Advantages of self and peer assessment
• develops learner confidence in
assessing/ marking peers (through
practice).
• develops learner ability to self-
evaluate and reflect.
• saves teachers time
• makes feedback faster
• students can see tests not as
punishments but as useful feedback
Principles of students assessment from the
perspective of Learning Outcomes
•Assess all the outcomes
•Use assessment methods adequate for outcomes
•Combine formative and summative assessment
•Use criterion based assessment
•Standardize methods of assessment
Principles of students assessment from the
perspective of Learning Outcomes cont.
• Do not compensate lack of achievement in some
LOs by excellent achievement in others
• Consider external benchmarking
• Decide about % representation of LOs in the
exam
- Assess important aspects
- Use multiple methods of assessment
- You can assess several LOs in one exam/task
Assessment of students’
achievements vs assessment of
learning outcomes
Achievements
of studentsLearning
outcomes
On the level of
institution /course/
On the level of an
individual person
Without gradesWith grades (sum of achievements)
Typical Grade Book
Summary
• Analyzing the row data results in a
summary of individual student
performance.
• Analysis of the column focuses on
learning outcomes and can identify
strengths and weaknesses of the course
which are not apparent when looking at
student summary data only.
Student Learning Outcomes-Competences
Student A B C D E F Student
performance
1 1 0 1 1 1 1 84%
2 0 1 1 1 1 1 84%
3 1 1 1 1 0 1 84%
4 0 0 1 1 1 1 67%
5 1 1 1 1 1 1 100%
6 1 0 1 1 1 1 84%
7 1 1 0 1 1 1 84%
8 1 0 1 1 1 1 84%
9 1 1 1 1 1 1 100%
10 1 1 1 1 1 1 100%
Outcome 80% 60% 90% 100% 90% 100%
Grading vs. Assessment
• The goal of grading is to evaluate individual students’ learning and performance. Grades may incorporate criteria – such as attendance, participation, and effort –that are not direct measures of learning.
• Assessment goes beyond grading by systematically examining patterns of student learning across courses and programs and using this information to improve educational practices.
Grading vs. assessment of LOs
100%
%0
LO achievement level
Grades indicate achievement ABOVE the required level.
Student’s
achievement
Grading
• Evaluates the quality of a student’s work;
• Communicates with the student, as well as employers,
graduate schools, and others;
• Motivates how the students study, what
• Organizes to mark transitions,
Criteria for grading:
• A (5) - very good – student knows very well
• B (4) - good - student knows well
• C (3) - satisfactory - student has satisfactory
knowledge
• F (2) - failed - student does not know enough
Criteria for grading
1. A when student knows 95-100%
2. B when student knows 83-94%
3. C when student knows 60-82%
4. F when student knows less than 60%
Using grades for meaurement of LOs
1. Grades should distinguish between indicators of
learning outcomes and indicators of other
behaviors.
2. Grades should be based on clearly articulated
criteria that are consistently applied.
3. Separate grades should be used for the major
components of knowledge and skills so that evidence
of students’ specific areas of strength and weakness
could be identified.
Traditional grading of the course
Requirement Points
Attendance and participation 50
Homework 100
Exams (4 at 100 points) 400
Presentation 100
Final Exam 200
TOTAL 850
K. McClendon,E. Eckert
Outcomes-based Grading Scale
Points Grade
44-50 A (superior or good on all outcomes)
37-43 B (good or adequate on all outcomes)
30-36 C (adequate all outcomes)
15-29 D (developing)
-0-14 F (non-existent or minimal)
K. McClendon,E. Eckert
Grades
• Course grades are generally calculated by averaging the results of various measures.
• Specific strengths and weaknesses are masked by the process of averaging.
• One student with a grade of "C" might perform very differently from another student with a grade of "C." And the letter grade by itself fails to suggest ways for students to improve their performance. What does a "C" grade tell a teacher or, more important, a student about how that student can improve?
Grading individual LOs
LO Identify
sol.Ex1
15pt
Identify
sol.
HW2
20pt
Impl.
sol.Ex2
10pt
Impl.
sol. HW3
10pt
Total
Impl.sol
35pt
Total
ident.sol
20 pt
Total
score 55
pt
Stud.1 10 15 6 9 25 15 40=C
Stud.2 14 18 10 8 32 18 50=B
Stud.3 14 17 5 5 31 10 41=C
Class
average
12,67 16,67 7 7,33 29,34 14,33
Maintaining Grading Consistency in Multi-
sectioned Courses
• agree your grading policies, standards, and criteria
• have teaching assistants grade assignments for students
not in their section
• have each section of an exam graded by only one
teaching assistant
• have each section of an exam graded by only one
teaching assistant
•
••
•
••
Designing the Grading System
• Determine weight of each outcome.
• Determine relationship between points and performance
levels.
• Determine cutoff for a passing grade, points needed for
higher grades.
K. McClendon,E. Eckert
But remember “no grade or grading system is immutably
right by some eternal standard”
Seven Common (Mis)Perceptions About
Outcomes Assessment
• We’re doing just fine without it. (Assessment is medicine only for the sick.)
• We’re already doing it. (Assessment is just old wine in new bottles.)
• We’re far too busy to do it. (Assessment is an “administrivial” burden.)
• The most important things we do can’t/shouldn’t be measured. (Assessment is too reductive and quantitative.)
• We’d need more staff and lots more money to do assessment. (Assessment is too complex and expensive.)
• They’ll use the results against us. (Assessment is a trick or a Trojan horse.)
• No one will care about or use what we find out. (Assessment is a waste of time.)
„it is better to measure in unperfected way
something important than to measure
perfectly something which is irrelevant”
Hamdorf i Hall 2001