participatory design of a learning analytics enhancement e-portfolio: the human factor eelco...
TRANSCRIPT
WORKPLACED–B
ASED E–A
SSESSMENT TECHNOLOGY
FOR COMPETENCY–B
ASED HIG
HER MULT
IPROFESSIO
NAL EDUCATIO
N
Participatory Design of a Learning Analytics enhancement E-portfolio: The Human Factor
Eelco Schreurs1, Atta Badii2, Annemarie Camp1, Inge Duimel-Peeters1, Ovidiu Serban2, Daniel Thiemert2, Jaime Costa2, Marieke van der Schaaf3, Jeroen Donkers1, Geraldine Clarebout1
1
2
3
INTRODUCTION
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
2
Workplace learning
Different learning opportunities for
each learner
Encountered cases
Feedback
Same competency level to be achieved
Importance of “good” assessment instruments
INTRODUCTION
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
3
Importance of “good” assessment instruments
Measure required competency level
Considering variety of learning
experiences
(Electronic) Portfolio
• Work completed• Feedback• Progress made• Reflections
=> Longitudinal and multi-faceted information (van Tartwijk, Driessen, van der Vleuten & Stokking, 2007; Sengler & Kanthan, 2012)
www.blendspace.com
INTRODUCTION
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
4
Succes factors
Mentor
Part of institutional assessment procedure
Flexible format
Threads
Availability of time
User friendeliness
(Driessen, van Tartwijk, van der Vleuten and Wass, 2007)
Learning analytics?
adaptivePersona
-lized
(Greller & Drachsler, 2012, p. 44)
LEARNING ANALYTICS WITHIN THE WATCHME PROJECT
Learning analytics as (personal) probabilistic learning models
Based on all information in electronic portfolio
Computerized processing of large amount of data
BUT: also human factor; initiated through interface
(Dron & Anderson, 2009)
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
5
RESEARCH QUESTION
Starting point:
Human-centred design approach usability of e-portfolio
Research Question
What are the needs and objectives of the different stakeholders (‘the human factor’) in relation to a learning analytics enhanced electronic portfolio assessment?
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
6
METHOD
Participants
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
7
Participatory meeting
Anaesthesiology Veterinary Medicine Teacher Education Team Members
1 2 1 3 14 (10 technical, 4 educational)
2 4 1 1 3 (technical)
3 7 (5 physicians, 2 trainees)
- - 9 (technical)
METHOD
DesignParticipatory design strategy (Berns, 2004; Könings, Brand-Gruwel & van Merriënboer, 2010)
Three participatory meetings between May - September 20141st participatory meeting
visualise the landscape in which the WATCHME-project and its stakeholders will exist.
2nd participatory meetingvisualise the landscape for each domain (veterinary medicine, anaesthesiology and teacher education)
3rd participatory meeting=> ideas and requirements for just-in-time feedback=> ideas and requirements for the visualization module=> ideas for building a student model
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
8
RESULTSFirst participatory meeting:
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
9
Technical stake-holders
Develop new FB modules
Integration of different systems
and models
Creation of cross-domain system
Managerial stake-holders
Comparedifferent groups
Hospitals: higher quality,
safety of care,
transparancy
Curriculum improve-
ment
RESULTSFirst participatory meeting:
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
10
Info from different sources / observers
Trainers / assessors
Easy and fast access
to info
360° insight
Trainees
Trans-parancy
Structured feedback conver-sations
Safeguards confidentia-
lity & anonymity
History of learner => focussed
FB
Just-in-time
feedbackLongitu-
dinal monitori
ng
Anywhere and anytime!Just-In-Time feedback
RESULTSSecond participatory meeting:1. Distinction needed between supervisors and assessors
• Supervisors: • Analyze group of students of same cohort • Development of individual student compared to group• Accessible through personal computer
• Assessors:• Quick overview on smartphone• Elaborate overviews on personal computer• Need for notification when assessment needed
2. Definition of different categories of JIT-feedback• reminders/notifications• content feedback• longitudinal feedback
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
11
RESULTSSecond participatory meeting:3. feedback in a quantitative (scores) and qualitative way (narrative feedback)
needed
4. Insight into the learning progress• Over longer period about students’ performance on a general level• Short term for performance for particular activities
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
12
RESULTS
Third participatory meeting1. Focus on JIT-feedback
• Preference for narrative feedback• Possibility to tag narrative feedback• For numeric feedback: comparison with peers or European standard• Draw attention on difference score assessor and self-assessment• Alert when they miss milestone or deadline
2. Rules for not well-performing students• Number of mini-clinical evaluation exercises (mini-CEX) is dropping.• Poor reflective skills: poor narratives, small sentences.• Pattern in recent assessments.• Poor performers seek supervisors who do not give feedback or only
positive feedback (feedback avoidance)
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
13
DISCUSSION AND CONCLUSION
• Participatory meetings allowed to grasp the requirements and expectations of the stakeholders
• System will need to provide personalized feedback
• System will need to be adaptive with respect to: • Device used• Professional domain• Role of the user
• Shortcomings:• Not all target groups participated => convenience sample• Data analysis => time pressure
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
14
ACKNOWLEDGMENTS
• European commission: grant agreement n°: 619349 : ‘WATCHME Workplace-based e-assessment technology for competency-based higher multi-professional learning
• Suzanne Schut (for acting as an interim coördinator)
E A R L I C O N F E R E N C E - L I M A S S O L ( C Y P R U S ) – A U G U S T 2 0 1 5
15