building a bridge between theory and practice in formative e-assessment
DESCRIPTION
invited talk at e-Assessment in Practice, November 12, Cranfiled http://talks.yishaymor.org/e-assessmentinpractice2009TRANSCRIPT
![Page 1: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/1.jpg)
Building a Bridge Between Theory and
Practice in Formative e-Assessment
e-Assessment in e-Assessment in Practice,
12 November, Cranfield
![Page 2: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/2.jpg)
Scoping a Vision for Formative e-Assessment
• Commissioned by JISC
• June 2008 to January 2009
• Norbert Pachler, Harvey Mellar, Caroline Daly, Yishay Mor, Dylan Wiliam, Emmanouil AgianniotakisEmmanouil Agianniotakis
• http://feasst.wlecentre.ac.uk/
2Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 3: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/3.jpg)
Methodology
Desk researchDesk research
Literature review
Comparing frameworks
Five Practical Enquiry Days
Combination of collaborative reflection, report back Combination of collaborative reflection, report back from team, and guest plenaries
Launch day, three Pattern workshops, developers' day
3Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 4: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/4.jpg)
What is formative e-assessment?
Formative?
‘practice’ assessment /serial summative assessment
synonymous with learning
Alternative view:
The e?
The use of digital means to support
Alternative view:Formative features of assessment, which are afforded by specific features of digital media 4
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 5: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/5.jpg)
OK, so what do we mean by Formative Assessment?
“An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the decisions that would have been made in the absence of that evidence”
(Dylan Wiliam)
5Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 6: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/6.jpg)
Formative = feedback + moments of contingency
"... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as improvement, and activate students as instructional resources for one another."
(Leahy, Lyon, Thompson, and Wiliam 2005)
6Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 7: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/7.jpg)
Participatory Methodology for Practical Design Patterns
• Problem• Problem
– Acceleration → need for effective protocols for sharing of design knowledge
• Context
– interdisciplinary communities of practitioners
7
– interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice.
Son, this was my
dad's mobile.
I want you to
have it.Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 8: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/8.jpg)
Problem: The Design
Divide
the gap between those who the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*)
8
the critical resource is not the capacity to produce, but the knowledge to do it right.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 9: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/9.jpg)
Solution...(in architecture)
9Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 10: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/10.jpg)
What is a pattern?
• At is simplest, it is a
– Generalised solution to a problem– Generalised solution to a problem
– Follows a specific structure
Problem Solution
What are we trying to achieve / solve?
Cookbook: ingredients, procedure, expected
outcomes
10
C o n t e x t
Problem Solution
When, Where, Who
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 11: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/11.jpg)
A burda pattern..
Season: Fall For: Women Garment Type: Dress Style: Classic, Evening Wear, RomanticMaterial: Taffeta
“if I copy a dress, I can only create the same dress. If I have a pattern, I
11
the same dress. If I have a pattern, I can create many dresses” (Yim Ping LENDEN)
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 12: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/12.jpg)
As for software
12
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 13: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/13.jpg)
In learning / e-learning..
13Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 14: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/14.jpg)
The Participatory Pattern Workshops Methodology
14Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 15: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/15.jpg)
Collaborative reflection workshop
Facilitate on-going design-level conversation between designers and practitioners designers and practitioners involved in diverse aspects of the problem domain.
Open, trusting and convivial.
15
Open, trusting and convivial.
And at the same time
Critical, focused and output-directed.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 16: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/16.jpg)
Collaborative reflection workshop
16Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 17: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/17.jpg)
Case Stories Workshop
Engender collaborative reflection among practitioners by a practitioners by a structured process of sharing stories of successful practice.
17Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 18: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/18.jpg)
Pattern Mining Workshop
Shift from anecdotes to transferable design transferable design knowledge by identifying commonalities across case stories, and capturing them in a semi-structured form.
18Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 19: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/19.jpg)
Future Scenarios Workshop
Validate design patterns by applying them to by applying them to novel real problems in real contexts.
19Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 20: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/20.jpg)
A few cases
• Creature of the week• Creature of the week
• CoMo
• Post 16 String Comparison
• Open Mentor
• ...• ...
20Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 21: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/21.jpg)
Creature of the week (Judy Robertson)
SituationSituation
large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life.
Task
− Engage and motivate the students
− show examples of good work which others could − show examples of good work which others could learn from
− show students their work is valued.
− build a sense of community.
http://purl.org/planet/Cases/creatureoftheweek21
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 22: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/22.jpg)
22Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 23: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/23.jpg)
CoMo (Niall Winters, Yishay Mor)
SituationSituation
− Royal Vet College.
− Hospital rotations as part of their training.
Task
− Allow students to capture critical incidents in text − Allow students to capture critical incidents in text and image.
− Support sharing of clinical experiences and co-reflection.
http://purl.org/planet/Cases/CoMo23
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 24: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/24.jpg)
24Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 25: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/25.jpg)
Post 16 string comparison (Aliy Fowler)
SituationSituation
− Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students.
− Sixth Form level, grammatical consolidation and whole-sentence translation.
Task
− Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.
http://purl.org/planet/Cases/Post16stringcomparison
25Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 26: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/26.jpg)
Solution
A bespoke string (sequence) comparator was A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.
26Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 27: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/27.jpg)
Open mentor (Denise Whitelock)
http://purl.org/planet/Cases/OpenMentor27
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 28: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/28.jpg)
A few patterns..
• Try Once, Refine Once
• Feedback on Feedback
• Classroom display
28Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 29: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/29.jpg)
Try Once, Try Once, Refine Once
(Aliy Fowler)
http://pul.org/planet/Patterns/TryOnceRefineOnce
29Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 30: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/30.jpg)
Problem
Lack of immediate feedback for students Lack of immediate feedback for students leads to fossilisation of errors and misconceptions
providing immediate feedback in an iterative fashion can also hinder iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.
30Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 31: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/31.jpg)
Context
Class sizeClass size
− Large (30-300)
Content
− Skills \ facts
Mode of instructionMode of instruction
− Blended / on-line. Computer tested.
31Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 32: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/32.jpg)
SolutionSolution
32
![Page 33: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/33.jpg)
Feedback on on
Feedback
(Linda McGuigan) McGuigan)
http://purl.org/planet/Patterns/FeedbackonFeedback33
![Page 34: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/34.jpg)
Good feedback should -
�Alert learners to their weaknesses.�Diagnose the causes and dynamics of these.
Problem
�Diagnose the causes and dynamics of these.�Include operational suggestions to improve the learning experience.�Address socio-emotive factors.
Tutors know this, but are pressed for time. Or not aware of their feedback time. Or not aware of their feedback strategies
Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching 34
![Page 35: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/35.jpg)
Context
Large scale, technology supported, graded coursescourses
− many tutors instructing many students.
Feedback is mediated by technology that allows it to be captured and processed in real time
Topic of study is subject to both grading and formative feedback.formative feedback.
35Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 36: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/36.jpg)
Solution
Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms.
In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as tutors with immediate analysis of their feedback, as well as long-term aggregates.
36Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 37: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/37.jpg)
Classroom Display
http://purl.org/planet/Patterns/Classroomdisplay 37Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 38: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/38.jpg)
ProblemUsing learner Using learner
generated content..generated content..
�Rewards participation.�Relates to learner's personal experiences.�Window on student conceptions.
� Need to collate works in a single easy to access location.� Learners uncomfortable about presenting their work in public� Legal or other restrictions on sharing work.
38
![Page 39: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/39.jpg)
Context
Class size:
− Small / medium (6-60) − Small / medium (6-60)
Mode of instruction:
− Blended (preferable)
Time frame
− Continuous, over a period− Continuous, over a period
Pedagogy
− Involves construction / media production
39Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 40: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/40.jpg)
Solution
40Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 41: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/41.jpg)
Augmented domain map
41
![Page 42: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/42.jpg)
Example scenario
When using Try Once Refine OnceTry Once Refine Once, there is a risk that high-achievers do not receive feedback.that high-achievers do not receive feedback.
So -
Use Showcase Learning Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors.
Use Feedback on FeedbackFeedback on Feedback to alert tutors to the problem.
42Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 43: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/43.jpg)
Conclusions
Tip of the iceberg
Practitioners (educational / software) Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios
Collaborative elicitation of patterns from cases could be a potent form of professional development.development.
43Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
![Page 44: Building a Bridge Between Theory and Practice in Formative e-Assessment](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b4d0854a7959be0f8b45f8/html5/thumbnails/44.jpg)
Thank youThe pattern language network project:
http://patternlanguagenetwork.org
The learning patterns project:
http://lp.noe-kaleidoscope.org/
The formative e-assessment project:
http://feasst.wlecentre.ac.uk/
Yishay Mor
44
This presentation:http://talks.yishaymor.org/e-assessmentinpractice2009http://www.slideshare.net/yish/eain
Yishay Mor
people.lkl.ac.uk/yishay
@yishaym
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009