research and deployment of analytics in learning settings
DESCRIPTION
TRANSCRIPT
Research and Deployment of Analytics in Learning Settings
Katrien Verbert���
PAWS Meeting 9 April 2012 School of Information Sciences, University of Pittsburgh
Human-Computer Interaction
prof. Erik Duval
“Flexible Interaction between people and information”
Awareness & Sense-making
Computer Graphics
Language Intelligence & Information Retrieval
prof. Phil Dutré
prof. Sien Moens
http://hci.cs.kuleuven.be/
more focus on interaction...
tracking traces
Rescuetime Rabbit- eclipse plugin
Blogs
tracking traces
Rescuetime Rabbit- eclipse plugin
Blogs
tracking traces
www.role-project.eu
Duval, Erik. Attention please! Learning analytics for visualization and recommendation, Proceedings of LAK11: 1st International Conference on Learning Analytics and Knowledge, pages 9-17, ACM (2011)
objectives
• self-monitoring for learners
• awareness for teachers
• learning resource use and recommendations
• part of Learning Analytics research [ACM LAK conf., Siemens 2011,
Duval 2011]
overview
• Student Activity Meter
• Step Up!
• Recommender systems for learning
• Future research plans
Student activity meter (SAM): demo.
http://ariadne.cs.kuleuven.be/monitorwidget-rwtheval/ or http://bit.ly/I8AYV1
Design Based Research Methodology
• Rapid prototyping
• Evaluate Ideas in short iteration cycles of Design, Implementation
& Evaluation
• Focus on Usefulness & Usability
• Think-aloud evaluations, SUS (System Usability Scale) surveys,
usability lab, ...
Iteration one • usability and user satisfaction evaluation
• 12 CS students, using a -based
time tracker
• 2 evaluation sessions:
• task based interview with think aloud (after
1 week of tracking)
• user satisfaction (SUS & MSDT) (after 1
month)
User satisfaction
• average SUS score: 73%
iteration two
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7
CS TA’s.
• An online survey about usefulness, teacher issues and how the tool can resolve these.
• on average: 40 mins are spent using SAM.
CGIAR CASE STUDY
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?!✔
✔
✔ ?!
✔
?!?!
✔
✔ ✔ ?!?!✔
?!
✔
demographics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher needs,
use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% wants it
iteration three
• open course on learning and knowledge
analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who
also teach)
• almost identical survey to CGIAR case.
LAK CASE STUDY
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗ ✔
✔
✔
?!
?!
?!
✔
?!
✔ ?!
✔ ?!
✗
✗ ?!
?!
ideas from experts
1
2
3
4
5 detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
demographics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher
needs, use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% want it
III. 12
participants
assessing teacher
needs, expert feedback, use & usefulness
re-orderable parallel
coordinates with
histograms
most addressed needs are indecisive
• provides awareness and feedback • many uses • 66% want it • recomm. can be useful
Iteration four
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course
planner
• richer data set: tracking from programming
environment
• qualitative study using a structured face-2-face
interview ���
with 25 open questions.
USER SATISFACTION
• average SUS score: 69,69%
all: want to continue using it 9/11: give it to students
demo-graphics
evaluation goal
design changes
negative positive
I. 12 CS
students
usability, satisfaction, preliminary usefulness
1st iteration small usability
issues
• ↑learnability • ↓errors • good satisfaction • usefulness positive
II. 19
teachers & TA’s
assessing teacher needs, use & usefulness
help function resource
recomm. not useful
• provides awareness • all vis. useful • many uses • 90% want it
III. 12
participants
assessing teacher needs, expert
feedback, use & usefulness
re-orderable PC with
histograms
most addressed needs are indecisive
• provides awareness and feedback • many uses • 66% want it • recomm. can be useful
IV. 11
teachers & TA’s
use, usefulness & satisfaction
filter & search, icons, zooming in line chart,
editing PC axes
conflicting visions of
students doing well or at risk
• provides time overview • provides course overview • PC assist with detecting problems • many uses & insights • 100% want it
conclusion
• SAM enables to find a wide variety of ���
new insights
• a better course overview
• understanding student time spending
• almost all participants want to continue
using SAM
26
Santos Odriozola, Jose Luis; Govaerts, Sten; Verbert, Katrien; Duval, Erik Goal-oriented visualizations of activity tracking: a case study with engineering students, Proceedings of LAK12: 2nd
International Conference on Learning Analytics and Knowledge, pages 10, ACM (to appear)
Human-Computer Interaction Course
http://bit.ly/I7hfbe
usage
User satisfaction
• average SUS score: 77%
Nikos Manouselis, Hendrik Drachsler, Katrien Verbert and Erik Duval. Recommender Systems for Learning. SpringerBriefs in Computer Science, 90 pages, Springer US (to appear).
http://bit.ly/A4CwZU
challenges
• Evaluation
• Data sets
• Context
• User interfaces
EVALUATION & DATA SETS
Verbert, Katrien; Drachsler, Hendrik; Manouselis, Nikos; Wolpers, Martin; Vuorikari, Riina; Duval, Erik. Dataset-driven research for improving TEL recommender systems, LAK11:1st International
Conference on Learning Analytics and Knowledge, pages 44-53 (2011)
http://bit.ly/acBKsp
how to achieve objectives
• Setting up a website / maintain TELeurope group community
• Set up a open data repository for sharing educational datasets and
related researches outcomes
• Organizing annual workshop and SI
• Organizing a data competition like in TREC
dataTEL challenge & dataTEL cafe event
• a call for TEL datasets
• eight data sets submitted
http://bit.ly/ieqmWW
http://dev.mendeley.com/datachallenge/
Mendeley APOSDLE ReMashed Organic.edunet
Mace Melt
Collection period 1 year 3 months 2 years 9 months 3 years 6 months
Users 200.000 6 140 1.000 1.148 98
Items 1.857.912 163 96.000 11.000 12.000 1.923
Activities 4.848.725 1.500 23.264 920 461.982 16.353
reads + + - - + -
tags - (+) + + + +
ratings (+) - + + + +
downloads + + - - + +
search - + - - + -
collaborations - + - - - -
tasks/goals - + + - - -
sequence - + - - - -
competence - + - - + -
time - - - - + +
Sam
Ian
Neil
A
B
C
high correlation
User-based CF
Sam
Ian
Neil
A
B
C
high correlation
Item-based CF
similarity measures
• Cosine similarity
• Pearson correlation
• Tanimoto or extended Jaccard coefficient
similarity measures
MAE of item-based collaborative filtering based on different similarity metrics
algorithms
MAE of user-based, item-based and slope-one collaborative filtering
CONTEXT
Verbert, Katrien; Manouselis, Nikos; Ochoa, Xavier ; Wolpers, Martin; Drachsler, Hendrik; Bosnic, Ivana; Duval, Erik. Context-aware recommender systems for learning: a survey and future challenges, IEEE
Transactions on Learning Technologies, 20 pages (Accepted)
data dimensions
challenges
• context acquisition
• standardized representation of contextual data
• evaluation
• user interfaces
VISUALIZING THE RATIONALE OF RECOMMENDATIONS���
Visualizing recommendations
adapted from Keim et al. 2008
objectives
• Address cold start issues
• Justification and trust
• Richer interaction capabilities
examples
Klerkx and Duval 2009
O'Donovan et al. 2010
Suggestions welcome!
References
• Duval, E. (2011). Attention please!: learning analytics for visualization and recommendation. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, (pp. 9-17), ACM.
• D. Keim, G. Andrienko, J.-D. Fekete, C. Go ̈rg, J. Kohlhammer, and G. Melanc ̧on. Visual Analytics: Definition, Process, and Challenges. In A. Kerren, J. Stasko, J.-D. Fekete, and C. North, editors, Information Visualization, volume 4950 of Lecture Notes in Computer Science, pages 154–175. Springer Berlin / Heidelberg, 2008
• J. Klerkx and E. Duval. Visualising social bookmarks. Journal of Digital Information, 10(2):1–40, 2009
• J. O'Donovan, B. Gretarsson, S.Bostandjiev, C. Hall, and T. Hollerer. SmallWorlds: Visualizing Social Recommendations. In G. Melançon, T. Munzner, and D. Weiskopf (eds) Eurographics/ IEEE-VGTC Symposium on Visualization 2010, Volume 29 (2010), Number 3, 10 pages
• Siemens, G. & Gasevic, D. (eds) (2011). Proceedings of the 1st conference on Learning Analytics and Knowledge 2011. ACM.