tcj ensuring the alignment of assessment with learning outcomes
TRANSCRIPT
Ensuring the Alignment of Assessment
with
Learning Outcomes
Working Smarter not Harder
Dr. Michele Pinnock
From which angle are we viewing the outcome?
• Are your learning outcomes attainable?• What are we valuing process or product?• Are you compromising the validity and
reliability of the assessment • Is your expert knowledge; prior learning
experience [when I was in College] syndrome; your knowledge of the standards and expectations of the real world all clouding your role as assessor?
CONCERNS
• Students complain that they are being over-assessed
• Students complain about not receiving meaningful feedback
• Teachers complain about the ‘ marking’; we continue to over-assess and overworked ourselves
Valuing vs Crediting
• What are you valuing and what is being credited by your course?
• Are we valuing high order thinking?• We say we value completing assignments of
time but do we credit such?
How valid are our assessments?
• Are we testing what we say we would test?• Is our test mirroring the emphasis of our instruction?• Are we asking enough questions to make informed
judgments about student performance?• Are we able to accurately predict future
performance of our students accurately?
What are we prepared to do about this situation?
The Message from the Assessment
• Assessment sends clear messages to students regarding what is valued in a course.
• Students who perceive that the assessment will test memorization are more likely to adopt surface approach to learning (Scouller, 1998)
• Ensure that the message from assessment is in accordance with what is valued by the lecturer and the course.
Owning Intended Learning Outcomes
• We must clearly communicate to our students the intended learning outcomes so they can own them and take responsibility for achieving them
• Students are rarely concerned about learning outcomes they are interested in what is being valued by the assessment.
• Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates.
• Students take their cues from what is assessed rather than from what lecturers assert is important
George Brown et al ,
How do we get students to learn what we intend for them to learn?
• Constructive Alignment is the answer.
?
Constructive Alignment
• This is a conscious effort to provide the learner with a clearly specified goal, a well designed learning activity or activities that are appropriate for the task, and well designed assessment criteria for giving feedback to the learner.
Aligning learning outcomes, learning and teaching activities and the assessment.
Adapted from Biggs(1999) p 27
Constructive Alignment• Encourages clarity in the design of the
curriculum, • Ensures that both students and teachers are
cognizant of what is being valued and the weighting being ascribed to such
• Offers transparency in the links between learning and assessment.
• Encourages institutions to strive towards quality assurance and enhancement.
Think about Assessment at the beginning of the Learning Process
According to Biggs(1995) in aligned courses, • objectives are usually clear, functioning at the
high order level; • teaching methods usually elicit from students
those learning activities that are likely to achieve those objectives; and
• our assessment confirms that students are in fact learning what our objectives say they should learn (p.11)
Unaligned Course
Teacher Intentions
Student
Activity
Exam
Assessment
e.g.- explain- relate- prove- apply
e.g.- memorize- describe
"Dealing with the test"
Aligned Course
Teacher Intentions
Student
Activity
Exam
Assessmente.g.- explain- relate- prove- apply
e.g.- explain- relate- prove- apply
Assessment influences Learning
• Assessment methods and requirements probably have a greater influence on how and what students learn than any other single factor (Boud,1995).
• How we go about assessing students can make significant impact on how well they achieve in their studies
• Poor assessment design will lead students to ascribe to behaviours that are counter-productive to learning
Steps Towards Constructive Alignment
Consider assessment at the beginning of the course • Teachers need to be clear on attainment targets
for their learners• Understand, explore the assessment possibilities
for measuring these targets and then selecting the most relevant
• Understand the process that will cause students to attain these targets
Designing Intended Learning Outcomes (ILO)
• Bloom’s Taxonomy• Structure of Observed Learning Outcomes
(SOLO) Taxonomy helps to map levels of understanding that can be built into the intended learning outcomes and to create the assessment rubrics.
Valuing competences
Have the student do something,and then measure product and/or process
'SOLO' = Structure of the Observed Learning Outcome
Objective !
To learn to: analyze systems for... explain cause/effects... prove properties of... compare methods of... ...
[ Competence := knowledge + capacity to act upon it ]
'SOLO'
SOLO 1 no understanding irrelevant information misses point
SOLO 2
SOLO 3
SOLO 4
SOLO 5
"pre-structural"
to identify to do procedure to recite
"uni-structural"
to classify to combine to enumerate
"multi-structural"
to relate to compare to analyze
"relational"
to generalize to hypothesize to theorize
"extended abstract"
dept
h(q
ualit
ative
leve
ls)
surf
ace
(qua
ntita
tive
leve
ls)
SOLO • to theorize• to generalize• to hypothesize• to predict• to analyze• to relate• to compare• to explain causes• to describe• to combine• to classify• to perform algorithm• to do simple procedure• to define• to identify• to recite
extendedabstract
relational
multi structural
SOLO 5
SOLO 4
SOLO 3
R
R2
R3
R1
R
xx
Graphic Illustration Legend
immediately relevant aspects – given! related or hypothetical – not given! irrellevant or inappropriate student responsexR
R
SOLO 2
uni structural
R
x R'
R''
x
RX
Designing Student Activities• It is imperative that congruence exists between
the – intended learning outcomes; intended learning outcomes; – the learning activities and the learning activities and – the assessment taskthe assessment task
• If we value level 2 & 3 of the SOLO - Surface Understanding will be attained
• Valuing level 4 & 5 will result in Deep Understanding
SOLO (verbs)
"extended abstract"
"relational"
"multi structural" &"uni structural"
to theorize to hypothesize to generalize to critize to predict
to apply theory (to 'near' problems) to reason about (reach conclusion) to explain (cause-effect) to explain (similarities-differences) to explain (strengths-weaknesses)
to describe to structure at collate to combine to classify to perform algorithm
to apply theory (to 'distant' problems) to put-into-perspective to reflect to judge to discuss
SOLO 2+3
SOLO 4
SOLO 5
to analyze to argue to relate to compare to integrate
to enumerate to paraphrase to do simple procedure to define to identify / name to recite
Copied from http://www.cmu.edu/teaching/assessment/basics/alignment.html
TYPE OF LEARNING OBJECTIVE EXAMPLES OF APPROPRIATE ASSESSMENTS
RecallRecognizeIdentify
Objective test items such as fill-in-the-blank, matching, labeling, or multiple-choice questions that require students to:•recall or recognize terms, facts, and concepts
InterpretExemplifyClassifySummarizeInferCompareExplain
Activities such as papers, exams, problem sets, class discussions, or concept maps that require students to:•summarize readings, films, or speeches•compare and contrast two or more theories, events, or processes•classify or categorize cases, elements, or events using established criteria•paraphrase documents or speeches•find or identify examples or illustrations of a concept or principle
ApplyExecuteImplement
Activities such as problem sets, performances, labs, prototyping, or simulations that require students to:•use procedures to solve or complete familiar or unfamiliar tasks•determine which procedure(s) are most appropriate for a given task
TYPE OF LEARNING OBJECTIVE EXAMPLES OF APPROPRIATE ASSESSMENTS
AnalyzeDifferentiateOrganizeAttribute
Activities such as case studies, critiques, labs, papers, projects, debates, or concept maps that require students to:•discriminate or select relevant and irrelevant parts•determine how elements function together•determine bias, values, or underlying intent in presented material
EvaluateCheckCritiqueAssess
Activities such as journals, diaries, critiques, problem sets, product reviews, or studies that require students to:•test, monitor, judge, or critique readings, performances, or products against established criteria or standards
CreateGeneratePlanProduceDesign
Activities such as research projects, musical compositions, performances, essays, business plans, website designs, or set designs that require students to:•make, build, design or generate something new
This table does not list all possible examples of appropriate assessments.
Authentic Assessment• Authentic assessments require students to be
effective performers with acquired knowledge.• Traditional tests tend to reveal only whether the
student can recognize, recall or "plug in" what was learned most times out of context.
• Authentic assessments present the student with the full array of tasks that mirror the priorities and challenges found in reality.
Assessing Learning in the Information Era
• With this information age – students need to learn how not just to access, but evaluate and use appropriate information to solve real problems/issues
• Let’s get them to make value judgments about the validity and reliability of the information
• Remember that student learning does not just depend on what we teach
• Spend more time and resources on assessment-
"The Learning Pyramid"Average
retention rate
5%
10%
20%
30%
50%
75%
80%
Lecture
Reading
Audiovisual
Demonstration
Discussion group
Practice by doing
Teaching others
[Kilde: NTL Institute for Applied Behavioral Science, Bethel, Maine]
passive
activestudent
level
of
enga
gem
ent
Biggs (2003) Concept Map showing relationships within the Curriculum Design Process.
Concerns
• Incidental learning outcomes need to be identified and if of value incorporated as intended learning outcomes for the next course offering
• Constructive alignment cannot be achieved or maintained in an institutional system that does not allow frequent review and modification by teachers
Putting it all TogetherThere are four stages in a constructively aligned curriculum
(Biggs & Tang, 2007, pp. 54-55).
1. Describe the intended learning outcomes in the form of a verb (learning activity), its object (the content), and specify the context and a standard the students are to attain.
2. Create a learning environment using teaching/learning activities that address that verb and therefore are likely to bring about the intended outcomes.
3. Use assessment tasks that also contain that verb, thus enabling you to judge with the help of rubrics if and how well students’ performances meet the criteria.
4. Transform these judgments into standard grading criteria.
Examine carefully Goals/ Rationale
of your course
Operationalize the goals into
intended learning outcomes
(ILO)
Choose carefully the Assessment
That measures these Outcomes
Choose carefully the forms of teaching
That focus on these outcomes
Do we possess a vision of Evaluation
Unless the purpose is perceived to be significant, the procedures are clearly understood, and the results are perceived to be useful and relevant, the individuals whose performances are being assessed will not do their best and will not facilitate the assessment process.
Learners can resist even high stake test if they are deemed meaningless and unmanageable.
MEANINGFUL ASSESSMENT
• Achieves specific goals or purposes that are significant to all especially students
• Clear procedure, criteria, rubrics that are understood by ALL stakeholders
• Produce results that provide clear directions for improved learning and instruction
Manageable Assessment
Describes assessment that provides useful information on performance given time and other resources.
• Planning and organizing the assessment• Collecting / analyzing assessment data• Recording and communicating the results of
the assessment to stakeholders
Aligning Coursework
• Coursework as grades or part of the learning process?
• Are we engaged in formative assessment?
Proportions of Formative Assessment in Classrooms
No Formative
assessment
Token Formative
Assessment
Moderate Formative
Assessment
Near-Total Formative
Assessment
Class Activities Formative Assessment
Formative Assessment
• Ongoing• Aids learning– Teachers feed information back to students in ways that
enable the student to learn better, or
– Students can engage in a similar, self-reflective process. • Helps students understand the rules of the game • Helps in the continuous monitoring of the quality of
the work as both students and teacher strive towards attaining desired learning outcomes
Formative Assessment
• Aids instruction – using evidence of student’s mastery status to make adjustments to instruction if the evidence suggests these adjustments are warranted
Formative assessment make teachers teach better and learners learn better
Formative assessment represents evidence-based instructional decision making
Steps for Establishing Formative Assessment That Solicits Classroom Climate Shifts
1. Distribute classroom climate guidelines2. Seek trust constantly and nurture it seriously3. Model and reinforce appropriate conduct4. Solicit students’ advice on classroom climate5. Assess students’ relevant affective status
Concerns/ Dissonance
• Formative assessment though it elicits deeper understanding among learners it will not necessarily improve student’s scores on the standardized examinations
How well do teachers manage assessment ?
• Many assessment procedures are labour intensive and time consuming on the part of teachers
• Teachers spend 15 -20 minutes outside of school grading essay assignments.
Swain & Swain (1999)
VALIDITYThe accuracy of a test to test what it is suppose to
test.• Number of items• The types of items• The time allotted for the completion of the
assessment• Quality of task instructions• The weighing of each assessment in relation to the
course objective and the course in its entirety
RELIABILITY
This speaks to the consistency of the assessment in determining student performance.
If assessment is not reliable then decisions made based on such will lead to problems.
Issues associated with some frequently used test formats
GROUP ASSESSMENT• Deciding on group members• Assessing the input of each members process vs. product• Assessing whether all members possess the
knowledge/ skill/ attitude being examined
Formative assessment aids learning by generating feedback information that is of benefit to students and to teachers.
Feedback on performance, in class or on assignments, enables students
• to restructure their understanding /skills and • build more powerful ideas and capabilities.
Conditions necessary for students to benefit from feedback
According to Sadler (1989):
a) possess a concept of the goal/standard or reference level being aimed for
b) compare the actual (or current) level of performance with that goal or standard
c) engage in appropriate action which leads to some closure of the gap
Issues with Feedback
• Students not clear on learning outcomes• Students not sure what standards/
expectations look like• Students not sure about what is actually
necessary to help students close the gap. Example of a comment -‘this essay is not
sufficiently analytical’)
Good feedback practice:
1. Facilitates the development of self-assessment (reflection) in learning. 2. Encourages teacher and peer dialogue around learning. 3. Helps clarify what good performance is (goals, criteria, expected
standards). 4. Provides opportunities to close the gap between current and desired
performance. 5. Delivers high quality information to students about their learning. 6. Encourages positive motivational beliefs and self-esteem . 7. Provides information to teachers that can be used to help shape the
teaching.
FEEDBACK now Feed-forward
• Feedback is the most significant factor in student progress
• For feedback to be effective students must be able to apply the comments they receive to improve their chances of success with the following assessment
• High level of feedback results in high quality student outcomes and so expectations are achieved
Feedback
• Comparing students is not of importance because our objectives/ criterion are written in terms of individual mastery of a course
• Forms of feedback- checklists/ rubrics/ codes/ regular conferencing
feedback
• How can we get students to read the feedback and do something with it, react to it, respond to it, use it as a feed-forward – something that will make the next assignment better.
• We need to design ways of giving feedback much faster, maybe emailing
ASSESSMENT INSTRUMENTS
Checklists Rating Scales Rubrics
• Tools used to measure performance on assessments.
RUBRICS
Rubrics are a scoring scale consisting of
a set of criteria that describe what expectations are
being assessed/evaluated and
descriptions of levels of quality used to evaluate
students work or to guide students to desired
performance levels.
Rubrics
• Communicate expectations and aid instruction• Indicates quality and quantity of student
learning • Should be given with the assessment task • allow students the ability to assess their own
work
Sample of a Maths Rubric4 3 2 1
Demonstrates a thorough understanding of the main concepts.
Demonstrates an understanding of the main concepts.
Demonstrates a partial understanding of the main concepts.
Demonstrates little understanding of the main concepts.
Very capably and independently selects the most efficient algorithm and solutions are accurate.
Performs algorithms accurately and is usually accurate.
With some assistance, performs algorithms with some accuracy present.
Uses and performs simple algorithms with some accuracy present. Assistance is usually required.
Independently applies the steps required with a very high degree of accuracy.
Applies the steps of a problem and is usually accurate.
Some effectiveness is evident when following and applying the steps of a problem.
Often forgets the steps in a problem, some accuracy noted some of the time.
Thorough analysis of the problem with accurate solutions.
Analysis of the problem is evident, considerable accuracy.
Analyzes the problem with some success, accuracy needs to improve.
Very little evidence of analysis. Some educated guesses. Accuracy is weak.
Assessment Instruments List of
Assessment tools
http://course1.winona.edu/shatfield/air/rubrics.htm
http://academic.scranton.edu/department/assessment/ksom/
http://flightline.highline.edu/socc/ToolsResources/Tools/samplerubrics.htm
Written Papers http://www.fordham.edu/halsall/med/rubric.html
Class Participation
Critical thinking
Class Discussion http://www.landmark-project.com/classweb/tools/printable.php?rbrc_id=64683
Journal Entry http://course1.winona.edu/shatfield/air/Journal%20Entry%20Rubric.pdf
Class Debate http://www.winona.edu/AIR/documents/classdebate.pdf
Oral Communication
Problem solving
Presentation Checklist
http://www.sasked.gov.sk.ca/docs/comm20/pg061.pdf
Selecting Assessment Strategies
• Ensure that the strategy does mirror the objectives• Are competencies and skills being adequately
measured?• At the tertiary level we rely heavily on three main
assessment strategies – class presentations, essays and examination. Recently group work has been added to the list
Alternate Assessment• Portfolios where a complexity of meaningful tasks
geared towards enhancing learning are included. Please note that for portfolios assessment is the secondary purpose
• Simulations; • Work experience • Projects – Product? Process? Or both product and
process• Debates• Displays
• Concern – external examiners – need to validate grades and thus need proof of assessment.
Answer • design your assessment framework prior
to the course; videotape; and even invite them to play a passive role while you assess
Designing Assessments that Capture Student Interest and
Promote Learning • Using the Document Camera• Discussion Board• Weblogs• Chatrooms• Making movies/digital stories• Podcasts
Consider all the factors prior to…
• Emailing of assignments- implications- who will print; format etc
• Digital form of assignment – copy and paste etc
• Availability of resources/ materials• Guarding against plagiarism
Enhancing Reliability ..• Frequency and duration of assignments• More assessment tasks lead to increase in reliability
– greater consistency• More and thus shorter assessment items – students
will get more guidance on their performance which often leads to less anxiety HOWEVER In the real world students will have to face problems that are multifaceted and so the assessment would not be accurately measuring their ability to solve such problems
• Sometimes in a bid to ensuring reliability validity is sacrificed
• Using more than one scorer increases reliability• Defining marking scheme
Good Assessment Principles
• Use a range of assessment tasks to ensure balance of coverage and depth
• Validity achieved when assessment items measure the kind of knowledge desired – Relevance
• Emphasis of instruction is in tandem with the emphasis of assessment - Balance
• Assessment tasks are pitched at the levels outlined in the learning outcomes
Evaluation as a Tool for Empowering our Students
• Ensure that all students have an equal opportunity to achieve to the best of their ability– Consider the Learner - Test Anxiety; workload– Assessment of, as and for Learning
Integrating Assessment• Look at the assessment • How do assessment pieces relate to other pieces
over the semester• Are students being asked to do a similar piece of
work in another course – integration within disciplines (departments) and with other disciplines (departments)
• Look at the weightings• Who will be marking – set your schedule• What type of feedback will be given and when
Managing the measures
• Using excel
Online Assessment
• http://quizstar.4teachers.org
• http://quizstar.4teachers.org/instructor/class_mgr_ov.jsp?pl=cm&cl=cm_ov&qsts=1345707814404
Moving Forward…
• Constructive alignment cannot be achieved or maintained f we are not actively engaged in systematic and frequent reviews and modification of course and programme offerings
• We need to ensure that we use a modest number of extraordinarily important curricular aims
• We must be committed to analyzing and compiling reports for each outcome / course
Conclusion• Let’s stop marking • Let’s move towards ensuring that there is a
seamless transition between assessment and learning
• Ensure that assessment is a meaningfully integrated into the learning process
• Let’s ensure that opportunities for meaningful and timely feedback is intertwined with instruction
Let’s Work Smarter not Harder
• I thank you
References• Biggs, J. (1999). Teaching for Quality Learning at University,
(SRHE and Open University Press, Buckingham)• Biggs, J. (2003). Aligning Teaching and Assessment to
Curriculum Objectives, (Imaginative Curriculum Project, LTSN Generic Centre)
• Biggs, J and Tang C. (2007). Teaching for Quality Learning at University, (McGraw-Hill and Open University Press, Maidenhead)
• Sadler, D. (1989) Formative assessment and the design of instructional systems Instructional Science 18, 119-144.