ESCalate Research Project

‘Using reflective dialogue to assess professional learning’

website http://escalate.ac.uk/6333Ruth Pilkington RMHPilkington@uclan.ac.uk

Formal Learning

Non-formal Learning

Induction or Toolkit Week UK PSF

Courses I lead, e.g.

University HE Teaching Toolkit

Standard Descriptor One

PG Cert LTHE or another PG Certificate

Standard Descriptor Two

PG Diploma & CPD awards

MEd Standard Descriptor Three

NEW HE STAFF

EXPERIENCED HE STAFF

Informal, on-the-job development

Professional Dialogue Assessment Process

A CPD FRAMEWORK FOR UCLan showing formal and informal learning routes to professional development and accreditation

The Uclan Approach

Formal Courses• Toolkit• PGCert LTHE• Certificates/other

PGCerts• PGDiploma• MEd• EdD

Informal Process• Experienced• Discipline-specific• Reflective• Developmental• Discursive• Meaning-making

= Professional Dialogue

Informal Learning Route:The ‘Professional Dialogue’

• Normally takes place between an experienced ‘other’ and the ‘student’.

• It involves a 3-stage process:– an initial set-up dialogue, – an exploratory dialogic process, and – the assessed dialogue.

• Stage 3 is recorded and involves a second assessor to act as moderator or second marker.

• In Stage 3, one assessor comes from the subject or ‘field’ and acts as mentor through the preparation phase of dialogue, the other may have a broader, objective, generic teaching and learning perspective

Set up Dialogue:high Mentor input; low Mentee ownership

Dialogue 1:high Mentor leadership; increased Mentee ownership

Dialogue 2:shared ownership; increased Mentee input and determination

Assessed Dialogue:celebration of Mentee role and practice

Assessment and Reward: the Professional Dialogue showing Timeline, Input and Process

Mentee Participant

Mentor

Adapted and adopted by Ruth Pilkington from the process outlined by Brockbank and McGill (2007)

Benefits

• Ethos of process• Peer-led, supportive, celebratory, affirmative• Status to teaching and learning; space• Sharing and exchange• Communication medium: reliability• Rigour of process• Can work with teams – culture change• Meaning-making, reflective, subject and individual focus• Not ideal for new staff or directed development needs

The Professional Dialogue Model• Applies a model by Brockbank & McGill (2007). They outline specific conditions

supporting a collaborative approach to process-oriented, active, peer-supported, reflective learning: – Ascertain a reflective dialogue has taken place (ideally with others);– Establish evidence of learner’s participation in dialogue;– Identify evidence of developmental process over time, regardless of the start / end

point; – Ascertain evidence that process review has taken place, enabling students to take

away understanding of the learning process and replicate it elsewhere. (p194)• Boud & Falchikov (2007) use a cognitive model of coaching particularly relevant to

peer assessment of professional learning called ‘cognitive apprenticeship ‘. • Cognitive apprenticeship focuses on dialogue over an indefinite period using a

structured process of ‘modelling, scaffolding, fading and coaching’ (Collins et al 1991:2, cited Boud & Falchikov, 2007:130).

• Coaching forms a thread that runs throughout ensuring that the collegial relationship is built on trust, confidentiality and empowerment.

• Meta-cognitive apprenticeship skills are developed in partnership with the experienced ‘other’ who initially models behaviours, scaffolds professional learning using reflective processes, encourages interrogation and engagement with self assessment, and then gradually fades out as ‘expert’.

Areas of Activity1. Design and planning of learning activities and/or programmes of study2. Teaching and/or supporting student learning3. Assessment and giving feedback to learners4. Developing effective environments and student support and guidance5. Integration of scholarship, research and professional activities with teaching and supporting learning6. Evaluation of practice and continuing professional development

Core Knowledge1. The subject material2. Appropriate methods for teaching and learning in the subject area and at the level of the academic programme3. How students learn, both generally and in the subject4. The use of appropriate learning technologies5. Methods for evaluating the effectiveness of teaching6. The implications of quality assurance and

enhancement for professional practice

Values Statements for Professional Standards a) respect for individual learners b) commitment to incorporating the process and outcomes of relevant research, scholarship and/or professional practice c) commitment to development of learning communities d) commitment to encouraging participation in higher education, acknowledging diversity and promoting equality of opportunity e) commitment to continuing professional development and evaluation of practice

The UK Professional Standards Framework

Project Process• Four UK Institutions• Professional Dialogues used

1. To assess academic leadership (Standard Descriptor 3)2. To assess new lecturers wanting to meet Standard Descriptor 13. To assess experienced lecturers seeking Standard Descriptor 2 (core lecturer role)

• Dialogues recorded and transcribed• Analysis of themes emerging (Tag Cloud, thematic identification, reflective

cyclical review). attentiveness to issues – a personal response, discussion with colleagues – strengthen and derive codes

– Structure of dialogues and management of interaction; facilitation and support; – Evidence of reflection and evidence of meeting Standard Descriptors– Phatics , use of small talk– Rigour of dialogue as assessment tool– Judgement and power; performance and control– Storytelling, narrative of meaning, cosntruction

• Reflection by team and review of data, experience• Interview data collected from participants and assessors on process

Group Activity

• Explore within groups how this might work within your context

• Examine examples provided to discuss issues of dialogue within the process

• Can you use the model for your assessment?– (Share outlines from institutions and documents)

• What would be the issues for you in terms of assessing professional learning?

AFTERNOON SESSION

Issues and OutcomesPower, JudgmentTraining and Support for Assessors

Review of Dialogues

• Modelling of dialogue• Emergence of issues of power• Analysis of different types of dialogue; ways

that power emerges• Evidence of judgment – a complex thing

Use of Questions

• Open Qs / Direct Closed Qs / Open Qs– Redirect issues or focus

• Closed questions move things on; focus attention

• Open Questions probe and explore• Affirmatory and Summative statements

Dialogue – power and parity• Power (assessor): scene setting, introductions;

interruptions, questioning, probing & direction, judgment • Power (assessee): taking ownership by managing

exploration, redirecting questions, pursuing own line of thought, own judgments

• Reflecting back at assessee; affirmation; noise; comments on issues; sharing own thoughts; empathy – support; meta-cognitory unpacking

• Equality: shared exchanges; shared feelings; asides, exploratory detours

Enhancement components

• Reflection: description, thinking things through, exploring own experience, reasons behind, recognition of own values

• Storytelling: evidence, authenticity, entry to reflection, role and identity

• Relationship building: sharing, humour, use of phatics, empathy

• Meaning making: exploration, explanation, interpretation, reflecting back, meta-analysis, positioning within outcomes

Dialogue - Types

• Parity, Equality • Assessor control

Incr. Input by assessor

Incr. Input by assessee

Incr. Control from assessor

Incr. Control by assessee

monologue

interrogation

interview

presentationdialogue

supported monologue

interview

Dialogue Cycles

QUESTION

PROBE

REVIEW

NEW QUESTION

SHARE

1

SET CONTEXT

TOPIC

AFFIRM RESPONSE

PROMPT/ PROBE

NEW TOPIC

JUDGMENT

EXAMPLE

DESCRIBE

LINK MADE

CONCLUSION

REPEAT CYCLESSUMMARISE

GAP CHECKFINAL STAGE

REFLECT

REVIEW

FEED BACK

INTRODUCTION 2

Institution A

• Preparatory dialogue and final dialogue

• 2nd final dialogue• Peer dialogue• Video of final dialogue

by deaf studies

• Mentoring process• Co-construction• Story telling• Turn-taking• Power and

management• Open v. closed

questions• Judgment

Assessment, Reflection and Professional Judgement within Dialogue – an exciting outcome

ICEBERG

Dialogue

An Interesting ‘Sub-text’

•Assessment of professional knowledge•Judgment•Assessment of values•Use of reflective models to assess professional learning

Judgment

• Expertise and experience of assessor• Comfort and familiarity with outcomes (UK PSF) and

field (L&T), and appreciation of subject (of assessee)• Familiarity with setting: evidence, locus, complexity,

authenticity – areas of activity• Reading body language, examples, stories, language –

truth, reliability, values• Contextualising: alignment to outcomes; tacit v. explicit;

hidden text• Gap analysis: Is it enough? What more is needed?

Use of Dialogue

• Purpose of Dialogue• Position of Dialogue• Process• Training of Assessor, Participant

Brockbank A & McGill I 2007 Facilitating Reflective Learning in HE SRHE/OUP 2nd Ed.Bowen Clewley L ‘Assessing against competency standards in the workplace’ in 207-227 in

Arguelles & Gonczi (2000)Eraut M 1997 Professional Learning and CompetenceGhaye T & Lillyman S 2006 Learning Journals and Critical Incidents RP for HE Professionals 2nd Ed

Quay Books LondonO’Donovan B, Price M and Rust C (2004) ‘Know what I mean? Enhancing Students understanding

of assessment standards and criteria’ in Teaching in HE Vol 9, No 3, July 2004Knight P & Yorke M (2003) Assessment Learning and Employability SRHE/OUPBoud D and Falchikov N Eds Rethinking assessment in HE 2007 RoutledgeBryan C & Clegg K 2006 Innovative Assessment in HE RoutledgeGoodfellow R & Lea MR (2007) Challenging E-learning in the University: a literacies perspective

SRHE / OUPLarrivee, Barbara(2008)'Development of a tool to assess teachers' level of reflective practice',

Reflective Practice,9:3,341 — 360 Moon J (2004) A Handbook of Reflective and Experiential Learning: Theory & Practice Routledge

FalmerPedler M (1996), Action Learning for Managers, The Learning Company ProjectShulman LS ‘Knowledge and Teaching’ pp61-77 in Leach J, Moon B (eds) (1999) Learners and

Pedagogy Sage/ Paul Chapman pubs