Assessment:Assessment:Achieving improved efficiency, Achieving improved efficiency,

effectiveness, educational integrity, effectiveness, educational integrity, equity and ethical practiceequity and ethical practice

Assoc Prof Janice OrrellAssoc Prof Janice Orrell

Education Consultant Education Consultant

What is assessment?What is assessment?DifficultSubjectivity ComplexAre comments right?Deciding if students have engaged with the course and met learning outcomesSource of incomeGrading and rankingClarify standardsLearn about the studentsFormative – engage and develop understandingFeedbackExams- quality controlCredentialing – capacity as a professionMarking – Assessment; differentClinician – information gathering about capacity for understanding of learning outcomes – trigger intervention

Why do we do it?Why do we do it?

Purpose – defines what it is

Process – integral to learning and teaching to guide and support learning (not just feedback) in light of the ‘orderly progression’

OverviewOverview What is Assessment?What is Assessment? What are some of the problems and What are some of the problems and

challenges associated with assessment?challenges associated with assessment? Designing assessment for greater Designing assessment for greater

efficiencyefficiency, , effectivenesseffectiveness, , educational educational integrityintegrity, , equityequity and and ethical practiceethical practice

Feedback, the elusive element.Feedback, the elusive element. Grading!Grading!

Assessment is diversely regarded Assessment is diversely regarded as…..as…..

Marking, Correcting and GradingMarking, Correcting and Grading CredentialingCredentialing Sorting the …. from the …….Sorting the …. from the ……. Directing learningDirecting learning Diagnosing learningDiagnosing learning Feedback on learningFeedback on learning Feedback on teachingFeedback on teaching Opportunity for critical reflectionOpportunity for critical reflection

In reality…Assessment is…In reality…Assessment is…

A A postscriptpostscript in teaching and curriculum in teaching and curriculum designdesign

An activity to Test An activity to Test what is knownwhat is known PivotalPivotal in learning in learning A common practice A common practice routine routine and and ritualritual Time consumingTime consuming AA negotiatednegotiated exchangeexchange (product for grades)(product for grades) High-stakes - high risk –high fallibilityHigh-stakes - high risk –high fallibility

Assessment practice is surrounded Assessment practice is surrounded by uncertainty by uncertainty (Allen 1998)(Allen 1998)

Academics design tasks, award grades and Academics design tasks, award grades and provide feedback that,provide feedback that,

They feel comfortable with,They feel comfortable with, They believe the student will feel is fair,They believe the student will feel is fair, Can with stand robust scrutiny by colleagues.Can with stand robust scrutiny by colleagues.

They make decisions that lead to preferred and They make decisions that lead to preferred and expected outcomes using their experience to expected outcomes using their experience to predict the best and worst outcomes!predict the best and worst outcomes!

Assessment is the Assessment is the HE Triple Blindfolded High Jump! HE Triple Blindfolded High Jump!

Much assessment is grounded in:Much assessment is grounded in:

Implicit University and academic disciplinary Implicit University and academic disciplinary traditionstraditions

Tacit Tacit wisdom of experiencewisdom of experience (Schulman: 1989) (Schulman: 1989)Source of assessment “expertise” is its own Source of assessment “expertise” is its own

practice. (practice. (Practice Making PracticePractice Making Practice; Britzman, 2003); Britzman, 2003)

Privacy Privacy and not open to scrutiny because of and not open to scrutiny because of academic freedomacademic freedom

EpisodicEpisodic routines and rituals routines and rituals

Assessment is high stakes for us Assessment is high stakes for us allall

Students are defined by their gradesStudents are defined by their grades Academic worth is often measured as Academic worth is often measured as

capacity to gradecapacity to grade Institutions depend on assessment Institutions depend on assessment

outcomes to assure the quality or its outcomes to assure the quality or its programmes and graduatesprogrammes and graduates

Professions and Society use them as Professions and Society use them as code to recruit new staff.code to recruit new staff.

Conservatism is evident in much of Conservatism is evident in much of respected higher education respected higher education

assessment practiceassessment practice

Longing for Longing for past glories past glories

Efforts focus Efforts focus on inventing on inventing better mouse better mouse trapstraps

Conservative responses to the Conservative responses to the challengeschallenges

Seek assessment Seek assessment processes that are:processes that are:

Fixed, non-negotiable Fixed, non-negotiable

Within known Within known boundariesboundaries

Predictable Predictable

SafeSafe

ExaminationsExaminations

measure measure knowledge that knowledge that is known &is known &

merely useful merely useful

instead ofinstead of

really usefulreally useful

““Good practice” in assessment Good practice” in assessment underpins fairnessunderpins fairness

Appropriate tasks that encourage Appropriate tasks that encourage learninglearning

Clear descriptions of grading criteria & Clear descriptions of grading criteria & grade related performance standardsgrade related performance standards

Timely feedback that linked to the Timely feedback that linked to the assessment criteria assessment criteria

Decision-making guided by posted Decision-making guided by posted criteria and standards when grading criteria and standards when grading

(adapted from Carroll 2004)(adapted from Carroll 2004)

Designing the Assessment TaskDesigning the Assessment Task Efficiency. Efficiency. Is the work of staff and students on the Is the work of staff and students on the

assessment task the best use of time and effort or just assessment task the best use of time and effort or just helping to spin out a grade?helping to spin out a grade?

Effectiveness: Effectiveness: Does the assessment process account for the diverse ways in which learning occurs and produce the best most enduring learning::

Assessment of learning? (postscript)Assessment of learning? (postscript) Assessment as learning? (pivot)Assessment as learning? (pivot)

Educational integrity: Educational integrity: Is there curriculum alignment?Is there curriculum alignment? Equity: Equity: Is success possible for all students?Is success possible for all students? Ethical practice: Ethical practice: Are students educated about academic educated about academic

integrity and does the design minimize opportunities for integrity and does the design minimize opportunities for plagiarism, cheating collusion or academic fraud?plagiarism, cheating collusion or academic fraud?

Assessment Assessment ChoicesChoices

Formative or summative

Grade or no grade

Teach to the test or encourage creativityWeekly work or end of semester exam

Individual or group work

Essays or authentic projects

Who grades…student, peer, profession or teacher

Open or closed book exams

Who will grade? University or professional practitioners?

Set or negotiated tasks?Assess participation in class or not?

How many pieces of assessment is enough?

Seen or unseen exams

Should presentation impact on grades?

Should all students have to do the same tasks?

How important are “hand-in” dates?

How much feedback?

What attributes would you like to What attributes would you like to engender in your graduates?engender in your graduates?

Knowledgeable disciplinary specialists?Knowledgeable disciplinary specialists? Ethical, efficient & trust worthy Ethical, efficient & trust worthy

professionals?professionals? Collaborative, reliable colleagues and team Collaborative, reliable colleagues and team

players?players? Community & corporate citizens?Community & corporate citizens? Effective problem solvers?Effective problem solvers? Creative & critical reasoning thinkers?Creative & critical reasoning thinkers? Effective verbal & written communicatorsEffective verbal & written communicators? ?

To what extent do your current To what extent do your current assessments encourage this?assessments encourage this?

Expert capabilities

Professional Behaviour

Disciplinary expertiseKnowledge Skills

Dispositions

Assessment as Mapping Learning Assessment as Mapping Learning for the Curriculum?for the Curriculum?

Workplace Context

Professional Action

Professional dispositions

Knowledge Skills

You asked about current practices You asked about current practices and policies…and policies…

Length of assignment per credit point?Length of assignment per credit point?

How to avoid over-assessing?How to avoid over-assessing? compliance / motivated to learn.compliance / motivated to learn. Measures or Indicators (How will they best Measures or Indicators (How will they best

demonstrate the learning you are seeking?demonstrate the learning you are seeking?

Integrating more practical and problem-based Integrating more practical and problem-based assignmentsassignments

How can you make it assessment as learning?How can you make it assessment as learning?

What designs best produce desired What designs best produce desired learning and graduate capabilities?learning and graduate capabilities?

Authentic Tasks: Authentic Tasks: Assessment as LearningAssessment as Learning Eg. Library search Eg. Library search (critical selection & team (critical selection & team

activity)activity)

Forensic role play Forensic role play (investigation/problem(investigation/problem solving)solving)

Observation Observation (noticing)(noticing)

Case presentations Case presentations (theory/practice application)(theory/practice application)

Debates and position statement Debates and position statement (evidence-based, (evidence-based, opinion formation)opinion formation)

Mastery learning Mastery learning (Skill development)(Skill development)

+ FEEDBACK!+ FEEDBACK!

Feedback Feedback

Does it make a difference?Does it make a difference? Students always want moreStudents always want more Do they use it to critically reflect on Do they use it to critically reflect on

their performance?their performance? Does the typical feedback provided really Does the typical feedback provided really

inspire critical self reflection?inspire critical self reflection?

Spelling?

Where is the intro?

Analysis?

Link?

Meaning?

Tense?

Evidence?

Reference?

?

??

??? ?

!5/10

OrOr

Great!

Well done!

Excellent

I like it!

√

√√

10/10

Quality of feedbackQuality of feedback

Much is code or crypticMuch is code or cryptic Few students are clear about what they Few students are clear about what they

need to do to improveneed to do to improve Good students get least feedbackGood students get least feedback Little is contingent up the feedbackLittle is contingent up the feedback Student critical reflection and uptake on Student critical reflection and uptake on

feedback is largely optional and feedback is largely optional and dependent on personal dispositiondependent on personal disposition..

Assessment & Feedback for Assessment & Feedback for LearningLearning

1.1. Design assessment to Design assessment to INCLUDE the act of INCLUDE the act of feedback and critical reflectionfeedback and critical reflection from the from the startstart!!

2.2. Feedback has Feedback has CONSEQUENCIAL CONSEQUENCIAL expectations expectations ((Utilisation of feedback an Utilisation of feedback an essential aspect of the grading process!)essential aspect of the grading process!)

3.3. Ensure that there is a climate of Ensure that there is a climate of collaborative critical reflection by using peer collaborative critical reflection by using peer and self review (assessment NOT grading)and self review (assessment NOT grading)

Good practice in feedback:Good practice in feedback: Delivers high quality information to correct & enhance Delivers high quality information to correct & enhance

learning and teaching.learning and teaching. Makes grades contingent upon students acting on the Makes grades contingent upon students acting on the

information.information. Encourages positive motivational belief and self esteem.Encourages positive motivational belief and self esteem. Facilitates student’s capacity for critical reflect and self Facilitates student’s capacity for critical reflect and self

assessment.assessment. Encourages dialogue about learning between teachers & Encourages dialogue about learning between teachers &

students.students. Clarifies what is a good performance of both criteria and Clarifies what is a good performance of both criteria and

standards.standards. Opportunity to close the gap between current & desired Opportunity to close the gap between current & desired

performance.performance.

What is NOT FeedbackWhat is NOT Feedback

NOTNOT editing. Require them to do so and hand editing. Require them to do so and hand back in.back in.

NOTNOT spell checking. Require students to do so and spell checking. Require students to do so and hand back in.hand back in.

NOTNOT grammar checking grammar checking

Point out these issues Point out these issues ONCE ONCE and require students and require students to undertake some activity to improve and report to undertake some activity to improve and report back to you once it has been done. back to you once it has been done.

What can be feedback?What can be feedback? Test at the beginning of semester on what needs to be Test at the beginning of semester on what needs to be

known at the end known at the end Regular (weekly) independent, on-line, ‘low stakes’ tests for Regular (weekly) independent, on-line, ‘low stakes’ tests for

students to completely so that they can assess the students to completely so that they can assess the accuracy of their understanding of key concepts.accuracy of their understanding of key concepts.

Weekly worked examples on problems at the beginning or Weekly worked examples on problems at the beginning or end of lectures and ask students to compare their (or end of lectures and ask students to compare their (or peers) work with the example peers) work with the example

Require students to nominate the kind of feedback they are Require students to nominate the kind of feedback they are seeking seeking

Require students to seek peer feedback & to indicate how Require students to seek peer feedback & to indicate how they have acted on the peer advice.they have acted on the peer advice.

Provide generic one page handouts summarising how the Provide generic one page handouts summarising how the class has performed overall and describe the qualities of class has performed overall and describe the qualities of performances that attracted specific grades.performances that attracted specific grades.

Use lecturesUse lecturesAsk students to complete and hand up:Ask students to complete and hand up: ‘‘Minute papers’ in which they outline their current Minute papers’ in which they outline their current

understanding or a core conceptunderstanding or a core concept ‘‘Application papers’ in which they apply a theoretical Application papers’ in which they apply a theoretical

concept to an everyday instance. concept to an everyday instance. ‘‘Five main points’ summary of the lecture.Five main points’ summary of the lecture.

Collect and quickly analyse student responses. (You could Collect and quickly analyse student responses. (You could do 50% if the class is large or get a tutor to analyse do 50% if the class is large or get a tutor to analyse them. them.

Provide (i) on-line feedback or (ii) feedback at the next Provide (i) on-line feedback or (ii) feedback at the next week’s lecture (5 minutes at the beginning, it gets the week’s lecture (5 minutes at the beginning, it gets the students there on time) to the class on your assessment students there on time) to the class on your assessment or common misunderstandings.or common misunderstandings.

Strategies Strategies Use a Use a grading rubric grading rubric to give students feedback to give students feedback

on where they sit in a progression of on where they sit in a progression of performance standards (say 4-5 levels) on range performance standards (say 4-5 levels) on range of attributes (criteria) of attributes (criteria)

Invite students to establish so personal learning Invite students to establish so personal learning goals for their next assignment based on this goals for their next assignment based on this feedback.feedback.

Invite students to self-assess using the rubric Invite students to self-assess using the rubric Discuss with students the differences in your Discuss with students the differences in your

assessment and theirs on the different attributes assessment and theirs on the different attributes in the rubric.in the rubric.

Grading! Grading!

Marking!Marking!

A Test! A Test! True or False!True or False!

Is assessment as simple as Is assessment as simple as giving a grade?giving a grade?

248248X46X46

------------------14481448992992

--------------------1136811368----------------

What grade out of 10 What grade out of 10 would you give a would you give a student for this?student for this?

What feedback would What feedback would you give to justify the you give to justify the grade?grade?

Grade & feedback from other Grade & feedback from other assessors!assessors!

9 9 A simple carrying error, otherwise it’s correct.A simple carrying error, otherwise it’s correct. 8 Simple error deduct 28 Simple error deduct 2 7 Three process & layout, errors deduct 1 mark each.7 Three process & layout, errors deduct 1 mark each. 6 Should have known the process6 Should have known the process 5 Knows the process, but can’t reward carelessness5 Knows the process, but can’t reward carelessness 4 Three errors deduct 2 marks each.4 Three errors deduct 2 marks each. 3 Three processes correct 1 mark each3 Three processes correct 1 mark each 2 Careless work but partly correct2 Careless work but partly correct 1 For trying1 For trying 0 Wrong! No marks!0 Wrong! No marks!

Interpretation& grading is influenced Interpretation& grading is influenced by by non-student performance factorsnon-student performance factors

Quality of other student’s work, Quality of other student’s work, Assessors’ personalityAssessors’ personality Assessors’ experiences and beliefs Assessors’ experiences and beliefs Knowledge of the student and classroom Knowledge of the student and classroom

eventsevents

Surface featuresSurface features Introductory paragraph,Introductory paragraph, Presentation factorsPresentation factors

Grading rubric standardsGrading rubric standards

LevelLevel BiggsBiggs PerryPerry QualitiesQualities

1.Developing1.Developing Uni-structuralUni-structural DualisticDualistic Limited Limited knowledgeknowledge

2.Functional2.Functional Multi-structuralMulti-structural MultiplisticMultiplistic Knowledge no Knowledge no structure structure

3.Proficient3.Proficient RelationalRelational Relative Relative commitmentcommitment

Analysis & Analysis & applicationapplication

4.Advanced4.Advanced Extended Extended AbstractAbstract

Limited Limited commitmentcommitment

MetacognitiveMetacognitive

& abstract& abstract

What is assessment?What is assessment?

Not optimal, dynamic – changed as use to standardsIncreasing complexity – need to look at whole programScaffolded learning for studentsFormative – Summative; Passion; integrated across the program

Testing understanding, rather than what they remember

Assessment can undermine learning – rituals of memorisation

Why do we do it?Why do we do it?