cmdn research symposium ajjawi final
TRANSCRIPT
Centre for Medical Education, Tay Park House, 484 Perth Road, Dundee DD2 1LR, Scotland, UK
Dr Rola Ajjawi
interACT: Interactive Assessment and Collaboration via Technology
Sean McAleer Assessment Advisor
Grant Murray Learning Technologist
Karen BartonProject Officer
David WalkerLearning Technology
Advisor
Natalie Laffertye-learning Advisor
Rola AjjawiProject Director
Susie Schofield Project Manager
The Project Team
http://www.psy.gla.ac.uk/~simon/DIALOGUE.html
A critique of monologic
feedback1-6
• Lack of learner engagement with feedback
• Lack of understanding of feedback (and acting on)
• Transmitted feedback creates dependency on teacher
• Not utilising self-evaluation or peer-feedback
• High teacher effort — low efficiency
• Lack of a shared context for assessment between teacher and learner
• Reduced staff satisfaction as evidence of feedforward not seen
Theoretical shift in conceptions of feedback that:
is a dialogue not a product to be delivered (relational rather than transmissive)7
develops the students’ capacity to make evaluative judgements about their own and others’ work1,2
serves the function of progressively enabling students to better monitor, evaluate and regulate their own learning, independently of the teacher3
Aims & Research questionsAim We sought to re-engineer our assessment and feedback
processes in line with good practice principles from the literature and the use of technology
Research questions How can technology be used to promote feedback dialogue in
online distance learning? What are tutors’ and students’ current conceptions of
feedback and barriers to change? What is the impact of dialogic feedback processes on students’
self-assessment ability, on tutors’ feedback profiles and key stakeholder satisfaction?
Research approach Action research8
Data collection methods
Student and staff interviews
Questionnaire
Content analysis of cover pages and wikis
Quantitative data about engagement rates and time to progress through modules
Routinely collected data from module evaluations, external examiner reports
Feedback audit
Equivalent 2
chaptersEquivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
PG Certificate in Medical Education
Equivalent 2
chaptersEquivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chaptersEquivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chaptersEquivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Equivalent 2
chapters
Revised PG Certificate in Medical Education
Chapters 2 4 6 8 10
Chapters 2 4 6 8 10
Chapters 2 4 6 8 10
Chapters 2 4 6 8 10
Formative assessment
Medium stakes assessment
High stakes assessment
©Mark Russell 2010University of Hertfordshire
ESCAPE project
Student engagement interACT launched 30th April 2012
100% completion of the cover page
Wiki engagement varies from 65-20%
85.2%(n=46) of the students who responded to the survey thought that the instructions provided about the assignment submission process were clear
0
5
10
15
20
25
30
35
40
Valuable for my own learning
Valuable for my own teaching
Promoted dialogue about
feedback
Promoted Self Evaluation
Beneficial to future
assignment tasks
Nu
mb
er o
f St
ud
ents
Disagree/Strongly Disagree
Not sure
Agree/Strongly Agree
Majority agree the cover page is valuable (CP)
Ability to request feedback about specific issues
Allows last review of work before submission and focuses the self assessment
Gives you a chance to tell the tutor what your concerns about the assignment are 'up front‘
It is a chance to think critically about how you approached the assignment and the assignment assessment criteria
There is an element of expectation that there will be some dialogue
Aspects I would like feedback on: Ideas of not preparing too rigidly in order to be flexible
within sessions – practical advice would be welcomed!
As the first essay I have written in nearly 20 years, I would like to know whether the standard overall was acceptable
Please advise me how I can enter a specific page number in a reference when using Endnote
Feedback on whether my peers have had similar thoughts for their own teaching, or other ideas that have been commonly developed would be beneficial in case I have not thought or considered them
any part of it
How did previous feedback inform this assignment
It made me realise that instead of focusing on a single or a few key teaching principles, I focused on many of them without going into much detail
Feedback that my writing style was agreeable was reassuring. I appreciated knowing my use of literature was valid so have tried to continue applying the literature to my work.
I tried to be careful to define and reference jargon
It was really helpful in writing present assignment
Which aspect(s) of your assignment would you specifically like feedback on?
Student comment (CP): I originally had a paragraph around learning principles and theories relating to small group teaching. I unfortunately had to delete it due to the word restriction. Is that something I should have included?
Tutor feedback (CP): Would you if you were the new tutor have appreciated this?
Student comment (wiki): I do think the tutor would have appreciated a small paragraph around educational principles and theories to help put small group teaching into context. I think that 5 pages though is plenty for the tutor to read, anymore than this and they may not have time to read it all. I have also learned how to feedback to the tutor based on the feedback given.
Which aspect(s) of your assignment would you specifically like feedback on?
Student comment: My choice of the Angoff method for this exam. I heavily considered the borderline group method, but decided against it as there was relatively little evidence for its use in exams with smaller examinee numbers such as this one. Thanks.
Tutor feedback: Yes, when borderline group method is applied to an exam with a small group of candidates there is a risk of not finding any borderline candidates! However, if you use borderline regression method, in which marks of all candidates are considered for setting the pass mark, this risk can be overcome.
Conclusions & implications Creating assessment and feedback dialogue in online
distance learning is possible
Majority of students find it valuable for their learning and that it promotes self-evaluation and dialogue
Feedback audit has been a useful tool for faculty development initiatives and measuring change in quantity and quality of feedback
Challenges have included streamlining the process, dealing with tokenism, and improving the quality and timeliness of the feedback
Research &Development
Continue to refine and streamline the process
More work is needed to help students better understand the pedagogic rationale and how to optimise their use of feedback
Embed into programmatic assessment approach through introduction of a personal development plan and patchwork/capstone assessment
Future research needs to explore how the online environment mediates learning from feedback
Project websitehttp://blog.dundee.ac.uk/interact
Contact DetailsRola Ajjawi
Centre for Medical Education
University of Dundee
Email: [email protected]
Twitter: @r_ajjawi
http://blog.dundee.ac.uk/interact/
http://youtu.be/S5bBFEbXDD0
Karen Barton
Centre for Medical Education
University of Dundee
Email: [email protected]
References1. Sadler, D. R. (2010) Beyond feedback: developing student capability in complex
appraisal. Assessment & Evaluation in Higher Education, 35:5, 535-550.2. Boud, D., & Associates. (2010). Assessment 2010: Seven propositions for
assessment reform in higher education. Sydney: Australian Learning and Teaching Council
3. Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218.
4. Nicol, D. (2012). Assessment and feedback - in the hands of the student [Online]. JISC. Available: http://jiscdesignstudio.pbworks.com/w/page/50118521/Assessment%20and%20feedback%20-%20in%20the%20hands%20of%20the%20student [Accessed 01/02/12].
5. Brown, E. & Glover, C. (2006) Evaluating written feedback. in: B. C. & K. Klegg(Eds) Innovative assessment in higher education. London, Routledge), 81-91.
6. Hattie, J. & Timperley, H. (2007) The Power of Feedback. Review of Educational Research, 77, 81-112.
7. Ajjawi, R. (2012). Going beyond ‘received and understood’ as a way of conceptualising feedback. Medical Education, 46(10), 1018-1019.
8. Carr, W. & Kemmis, S. 1986. Becoming critical: education, knowledge, and action research, Lewes, Falmer Press.