Download - uts-learning-futures-learning-analytics
LEARNING ANALYTICS — an essential tool for learning in the future
uts.edu.au UTS CRICOS PROVIDER CODE: 00099F
Simon Buckingham Shum Professor of Learning Informatics Director, Connected Intelligence Centre twitter @sbuckshum #UTSfuturelearn
cic.uts.edu.au
3
can we tell from your digital profile if you’re learning?
4
Every educational product now comes with a “dashboard”…
5
I was speaking at this event
http://codeactsineducation.wordpress.com
6
I asked Siri to find the conference website…
“Find code acts in education”
7
“Find code acts in education”
I asked Siri to find the conference website…
h,ps://twi,er.com
/Wisw
ijzer2/status/414055472451575808
8
“Note: check the huge difference between
knowing and measuring…”
Introducing ‘Joe’
9
Visualising “attainment” and “progress”
10
‘Joe’ may be developing curiosity, resilience, and collaboration skills, but these are left invisible
CRITICAL: accounting tools are not neutral
Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp. 12-13
“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure”
12
sta:s:cal analysis of educa:onal data is hardly new
13
14
Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers
Macro: region/state/national/international
15
Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers
Macro: region/state/national/international
16
Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers
Meso: institution-wide
Macro: region/state/national/international
17
NSW DEC • CESE Datahub (+ many comparable open data initiatives)
https://data.cese.nsw.gov.au
18
UK RAISEonline data: schools live or die by this
19
Learning Analytics: Data explosion at the micro level enriching, and enriched by, the meso+macro levels
Learning Analytics UNESCO Policy Briefing — http://iite.unesco.org/publications/3214711
Micro: individual student progress
Meso: institution-wide
Macro: region/state/national/international
outcome
Towards rapid formative feedback
researchers / educators / instrucGonal designers administrators / leaders / policymakers
intent
20
outcome Faster feedback loops could enable more rapid adaptation by students and teachers, and of learning resources and designs
researchers / educators / instrucGonal designers administrators / leaders / policymakers
intent
21
Towards rapid formative feedback
22 http://cdn.nmc.org/media/2014-nmc-horizon-report-k12-EN.pdf
23 http://redarchive.nmc.org/horizon-project/horizon-reports http://www.open.ac.uk/blogs/innovating
24 http://solaresearch.org/Policy_Strategy_Analytics.pdf http://tech.ed.gov/files/2013/02/Expanding-Evidence-Approaches.pdf
25
Ed-Tech startups and venture capital
26 https://www.edsurge.com/n/2014-10-06-diving-into-data-analytics-tools-in-k-12
http://techcrunch.com/2014/02/19/google-capital-invests-40m-in-learning-analytics-firm-renaissance-learning-at-1b-valuation
Ed-Tech startups and venture capital
27 http://learncapital.com
“LearnCapital is a venture capital firm focused exclusively on funding entrepreneurs with a vision for better and smarter
learning.”
Hang on: algorithm designers in ed-tech startups are shaping the learning experience?…
28 h,p://reinventors.net/roundtables/strategies-‐for-‐organizaGonal-‐change
Candace Thille (Stanford)
“Why would universities outsource their core
competency to ed-tech vendors?”
the core ques:on in slow mo)on
29
30
can we tell from your digital profile if you’re learning?
31
can we tell from your digital profile if you’re learning?
Who?
32
can we tell from your digital profile if you’re learning?
Who? How? With what confidence? After what kinds of training?
33
can we tell from your digital profile if you’re learning?
Who? How? With what confidence? After what kinds of training?
Sourcing which data, with what integrity?
34
can we tell from your digital profile if you’re learning?
Who? How? With what confidence? After what kinds of training?
Sourcing which data, with what integrity?
What kind of learning? What kind of teaching?
35
what kinds of learners are we trying to create?
– this should drive our analytics
Let’s hear from the UTS graduates of 2026
36 http://learningemergence.net/2014/04/07/learning-dispositions-authentic-inquiry-in-a-primary-school
The Knowledge – Agency Window
Pre-‐scribed curriculum
Open-‐ended Enquiry
Teacher Directed Learning
Self-‐directed Learning
Teacher-‐directed enquiry
Student led-‐enquiry
Content/Expert led teaching
Student-‐led revision
38 Guy Claxton: Constant change is here to stay: why schooling will always be about the future. UK ESRC Futures Meeting, May 2011. http://www.slideshare.net/edfutures/guy-claxton-esrc-futures-may11
We worry about disengaged low achievers like Joe — but we need to worry about Emily too...
The Knowledge – Agency Window
Pre-‐scribed curriculum
Open-‐ended Enquiry
Teacher Directed Learning Self-‐directed Learning
Teacher-‐directed enquiry
Student led-‐enquiry
Content/Expert led teaching
Student-‐led revision
The Knowledge – Agency Window
Pre-‐scribed curriculum
Open-‐ended Enquiry
Teacher Directed Learning Self-‐directed Learning
Teacher-‐directed enquiry
Student led-‐enquiry
Content/Expert led teaching
Student-‐led revision
The Knowledge – Agency Window
Pre-‐scribed curriculum
Open-‐ended Enquiry
Teacher Directed Learning Self-‐directed Learning
Teacher-‐directed enquiry
Student led-‐enquiry
Content/Expert led teaching
Student-‐led revision
Pre-‐scribed curriculum
Open-‐ended Enquiry
Teacher Directed Learning Self-‐directed Learning
Teacher-‐directed enquiry
Student led-‐enquiry
Content/Expert led teaching
Student-‐led revision
So our analytics depends on what space we’re in
Authenticity Agency Identity
43
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
44
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
46
More efficient learning for mastering core skills
Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 14. http://jime.open.ac.uk/article/2008-14/352
“In this study, results showed that OLI-Statistics students [blended learning] learned a full semester’s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester.”
“We’re looking at the profiles of what it means to be effective in the 21st century. […] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger.”
“Dispositions are now at least as important as Knowledge and Skills. …They cannot be taught. They can only be cultivated.”
John Seely Brown
47
US Dept. of Educ. http://reimaginingeducation.org conference (May 28, 2013) Dispositions clip: http://www.c-spanvideo.org/clip/4457327 Whole talk: http://www.c-spanvideo.org/program/SecD
Learning analytics for this?
“It’s more than knowledge and skills. For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity.”
Larry Rosenstock
LearningREimagined project: http://learning-reimagined.com Larry Rosenstock: http://audioboo.fm/boos/1669375-50-seconds-of-larry-rosenstock-ceo-of-hightechhigh-on-how-he-would-re-imagine-learning
Learning analytics for this?
48
Survey-based analytics for learning dispositions (Ruth Deakin Crick, Univ. Bristol / UTS:CIC Visiting Professor)
Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.
49
50
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Survey-based analytics for learning dispositions
Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.
Rapid Visual Feedback to SGmulate Self-‐Directed Change
A framework for a coaching conversaGon
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Taking responsibility for my own learning over time through defining my purposes, understanding and managing my feelings, knowing how I go about learning & planning my learning journey carefully.
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Making connections between what I already know & new information & experience. Making meaning by linking my story, my new learning & my purpose.
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Using my intuition & imagination to generate new ideas & knowledge. Taking risks & playing with ideas and artefacts to arrive at new solutions.
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Wanting to get beneath the surface & find out more. Always wondering why and how.
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Being part of a learning community at work, at home, in education & in my
social networks. Knowing I have social resources to draw on when I
need them
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Being able to work with others, to collaborate and
co-generate new ideas and artefacts. Being able
to listen and contribute productively to a team.
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
Having the optimism & hope that I can learn & achieve over
time. Having a growth mindset; believing I can generate my
own new knowledge for what I need to achieve
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
1 Dependent Closed Openness to learning
An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to persist & manage any self-doubt.
A necessary pre-requisite for developing resilience in learning
Mindful Agency
Sense making
Creativity
Curiosity Belonging
Collaboration
Hope and optimism
From self-report to behavioural analytics for learning dispositions?
Social network patterns, teamwork effectiveness and
initiation of relationships?
Questioning, arguing and search behaviours reveal
intrinsic curiosity and epistemic commitments?
Tagging/sharing/blogging/social patterns reveal how you see connections between ideas?
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
Relatively simple: • How much/when library and
other resources are accessed • Annotations on texts, images,
movies • Video replays and rewinds • Learner-shared resources
Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)
What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254
Does the way you search online reveal what you think
counts as trustworthy knowledge?
Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)
Dimensions of Epistemic Belief
Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute, to tentative and evolving
Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts
Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self
Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise
Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226
64
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
65
Are fieldwork students competent?
In the future, how might we track teamworking? (U. Sydney)
R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.
Automatic student tracking from multiple digital tabletops in classroom Analyse the students’ activity traces for significant patterns Visualise on a teacher’s dashboard
In the future, how might we track teamworking?
R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.
Learning Technology
KMi, OU
AI & ArgumentaGon
Learning DisposiGons
Human-‐Centred InformaGcs
Learning AnalyGcs
SemanGc ScienGfic Publishing
Dialogue / Issue / Argument Mapping
Social learning analytics — quantifying “professional identity”
Visual analytics to filter a social learning network by topic and type of social tie
Schreurs B, Teplovs C, Ferguson R, De Laat M and Buckingham Shum S. (2013) Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891
Visual analytics to filter a social learning network by topic and type of social tie
Schreurs B, Teplovs C, Ferguson R, De Laat M and Buckingham Shum S. (2013) Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
What epistemic contributions are learners making in the community?
71
Rebecca is playing the role of broker,
connecting different peers’ contributions in
meaningful ways We now have the basis for recommending that
you engage with people NOT like you…
72
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
Real time traffic-lights for students based on predictive model (Purdue University Signals)
73 Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x
Validate a statistical model from: • ACT or SAT score • Overall grade-point average • LMS usage composite • LMS assessment composite • LMS assignment composite • LMS calendar composite
Predicted 66%-80% of struggling students who needed help
Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen.
In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges
Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on.
1 2 3
45
Prototyping a social learning analytics dashboard
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910
74
75
Analytics for the UTS learning model?
http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches
ELLI Spider widget
Teacher’s dashboard for EnquiryBlogger
The hallmarks of educated writing/critical thinking Relevance Understanding & Knowledge Structure & Organisation Linguistic Accuracy Illustrations Referencing
Argumentation
78
Rhetorical functions of metadiscourse identified by the Xerox Incremental Parser (XIP)
BACKGROUND KNOWLEDGE
“Recent studies indicate …”
“… the previously proposed …”
“… is universally accepted ... “
NOVELTY
... new insights provide direct evidence ...
... we suggest a new ... approach ...
... results define a novel role ...
OPEN QUESTION … little is known … … role … has been elusive
Current data is insufficient …
GENERALIZING ... emerging as a promising approach Our understanding ... has grown exponentially ... ... growing recognition of the importance ...
CONTRASTING IDEAS … unorthodox view resolves … paradoxes …
In contrast with previous hypotheses ...
... inconsistent with past findings ...
SIGNIFICANCE studies ... have provided important advances
Knowledge ... is crucial for ... understanding valuable information ... from studies
SURPRISE We have recently observed ... surprisingly We have identified ... unusual
The recent discovery ... suggests intriguing roles
SUMMARIZING The goal of this study ... Here, we show ...
Altogether, our results ... indicate
79
XIP rhetorical parser applied to student writing
Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391
80
XIP rhetorical parser applied to student writing
Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391
CONTRAST
SUMMARY & CONTRIBUTION
81
82
to conclude…
h,p://th02.deviantart.n
et/fs71/PR
E/f/2011/119/c/0/rabbit_shadow
_by_plasGcx76-‐d3
f5qtd.jpg
83
Learners will leave an increasingly rich “digital shadow”
Learning Analytics use this to provide rapid formative feedback (but this is of course an imperfect proxy for real experience)
84
Now the shadow can talk back!
Let’s design shadows that learners + teachers value