workplace-based assessment (wba) – beyond the numbers

Workplace-based Assessment (WBA) – Beyond the numbers

James KwanAssociate Professor, Emergency Medicine and Medical Education

Acknowledgements

• A/Prof Lukas Kairitis• Dr Karuna Keat• Professor Jenny Reath• Professor Wendy Hu• Members Year 3/5 Committee

Outline

• Competence vs. Performance• Overview WBA at UWS• Assessor judgements on performance• Delivering Feedback

Definitions• Competence

– “Ability to do something successfully” (Oxford Dictionaries, 2012)

• Competence in medicine– Ability to perform a specific clinical task– Requires integration of competencies– Measurable in terms of observable behaviour– Context specific– Process of growth and development

Knows

Shows how

Knows how

Does

Prof

essi

onal

aut

henti

city

Prof

essi

onal

aut

henti

city

StandardisedAssessment(mostly established)

Non- standardised assessment(emerging)

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Miller’s pyramid

Goals of WBA

• Make a judgement on a student’s performance of a specific task in the clinical setting

• Provide guidance to the student through effective feedback to improve future performance of the task

Examples

• Encounter based– Case Discussion (Long case in the workplace)– Mini-CEX (direct observation clinical/procedural

skill)

• Global– Clinical Attachment Assessment– Multi-Source Feedback (360)– Shift Feedback Form

Year 3 Mini-CEX

• Student (or assessor) selects a patient • Student performs a focused clinical task e.g.

history taking, physical examination, performing a procedural skill, counseling a patient

• Assessor directly observes the encounter• Assessor rates the student’s performance in a

range of domains and provides feedback• No longer than 30 min including feedback

Year 3 Mini-CEX

Challenges• Competence is developmental vs. normative (not

satisfactory, borderline, satisfactory, good and excellent)

• Assessors do not read the back page – use their own internal rubric

• Reluctance to rate a student as “not satisfactory” – “Failure to fail”

• Feedback is often limited and not as helpful as it could be

• A “tick box” exercise resulting in students chasing a number

Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ, 45(6), 560-569.

Construct aligned scales

• Judgement on extent to which a supervisor would trust a trainee (or student) to perform a task independently

• Descriptors reflect the need for close or more distant supervision

• Different contexts will have a different way of construing “independence”

• Greater reliability with fewer assessments– 3 vs. 6 Mini-CEXs required to achieve a

Generalisability coefficient 0.7

Domains of assessment

Construct aligned Mini-CEX

Providing effective feedback

Effective Feedback

• Characteristics• Perceived need for change• Technique• Creating an action plan• Coaching

Delivering Feedback

• Enable learner reaction• Encourage self assessment• Provide an assessment of strengths and

weaknesses• Develop an action plan• Document the feedback

Year 5 Emergency Medicine

Shift Feedback Form

Summary

• Competence is developmental• Construct aligned scales may help align both

assessor judgements on performance and the development of competence by the student

• Delivering effective feedback is an important part of the assessment process to guide student learning– Action plan and coaching

Questions?

j.kwan@uws.edu.au