uts-learning-futures-learning-analytics

LEARNING ANALYTICS — an essential tool for learning in the future

uts.edu.au UTS CRICOS PROVIDER CODE: 00099F

Simon Buckingham Shum Professor of Learning Informatics Director, Connected Intelligence Centre twitter @sbuckshum #UTSfuturelearn

cic.uts.edu.au  

3

can  we  tell  from  your  digital  profile  if  you’re  learning?  

4

Every educational product now comes with a “dashboard”…

5

I was speaking at this event

http://codeactsineducation.wordpress.com

6

I asked Siri to find the conference website…

“Find code acts in education”

7

“Find code acts in education”

I asked Siri to find the conference website…

h,ps://twi,er.com

/Wisw

ijzer2/status/414055472451575808  

8

“Note:  check  the  huge  difference  between  

knowing  and  measuring…”  

Introducing  ‘Joe’  

9

Visualising “attainment” and “progress”

10

‘Joe’ may be developing curiosity, resilience, and collaboration skills, but these are left invisible

CRITICAL: accounting tools are not neutral

Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp. 12-13

“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure”

12

sta:s:cal  analysis  of  educa:onal  data    is  hardly  new  

13

14

Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers

Macro: region/state/national/international

15

Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers

Macro: region/state/national/international

16

Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers

Meso: institution-wide

Macro: region/state/national/international

17

NSW DEC • CESE Datahub (+ many comparable open data initiatives)

https://data.cese.nsw.gov.au

18

UK RAISEonline data: schools live or die by this

19

Learning Analytics: Data explosion at the micro level enriching, and enriched by, the meso+macro levels

Learning Analytics UNESCO Policy Briefing — http://iite.unesco.org/publications/3214711

Micro: individual student progress

Meso: institution-wide

Macro: region/state/national/international

outcome  

Towards rapid formative feedback

researchers  /  educators  /  instrucGonal  designers  administrators  /  leaders  /  policymakers  

intent  

20

outcome  Faster feedback loops could enable more rapid adaptation by students and teachers, and of learning resources and designs

researchers  /  educators  /  instrucGonal  designers  administrators  /  leaders  /  policymakers  

intent  

21

Towards rapid formative feedback

22 http://cdn.nmc.org/media/2014-nmc-horizon-report-k12-EN.pdf

23 http://redarchive.nmc.org/horizon-project/horizon-reports http://www.open.ac.uk/blogs/innovating

24 http://solaresearch.org/Policy_Strategy_Analytics.pdf http://tech.ed.gov/files/2013/02/Expanding-Evidence-Approaches.pdf

25

Ed-Tech startups and venture capital

26 https://www.edsurge.com/n/2014-10-06-diving-into-data-analytics-tools-in-k-12

http://techcrunch.com/2014/02/19/google-capital-invests-40m-in-learning-analytics-firm-renaissance-learning-at-1b-valuation

Ed-Tech startups and venture capital

27 http://learncapital.com

“LearnCapital is a venture capital firm focused exclusively on funding entrepreneurs with a vision for better and smarter

learning.”

Hang on: algorithm designers in ed-tech startups are shaping the learning experience?…

28 h,p://reinventors.net/roundtables/strategies-­‐for-­‐organizaGonal-­‐change  

Candace Thille (Stanford)

“Why would universities outsource their core

competency to ed-tech vendors?”

the  core  ques:on    in  slow  mo)on  

29

30

can  we  tell  from  your  digital  profile  if  you’re  learning?  

31

can  we  tell  from  your  digital  profile  if  you’re  learning?  

Who?

32

can  we  tell  from  your  digital  profile  if  you’re  learning?  

Who? How? With what confidence? After what kinds of training?

33

can  we  tell  from  your  digital  profile  if  you’re  learning?  

Who? How? With what confidence? After what kinds of training?

Sourcing which data, with what integrity?

34

can  we  tell  from  your  digital  profile  if  you’re  learning?  

Who? How? With what confidence? After what kinds of training?

Sourcing which data, with what integrity?

What kind of learning? What kind of teaching?

35

what kinds of learners are we trying to create?

– this should drive our analytics

Let’s hear from the UTS graduates of 2026

36 http://learningemergence.net/2014/04/07/learning-dispositions-authentic-inquiry-in-a-primary-school

The  Knowledge  –  Agency  Window      

Pre-­‐scribed  curriculum    

Open-­‐ended  Enquiry    

Teacher  Directed  Learning    

Self-­‐directed  Learning    

Teacher-­‐directed  enquiry      

 Student  led-­‐enquiry      

Content/Expert  led  teaching  

Student-­‐led  revision  

38 Guy Claxton: Constant change is here to stay: why schooling will always be about the future. UK ESRC Futures Meeting, May 2011. http://www.slideshare.net/edfutures/guy-claxton-esrc-futures-may11

We worry about disengaged low achievers like Joe — but we need to worry about Emily too...

The  Knowledge  –  Agency  Window      

Pre-­‐scribed  curriculum    

Open-­‐ended  Enquiry    

Teacher  Directed  Learning     Self-­‐directed  Learning    

Teacher-­‐directed  enquiry      

 Student  led-­‐enquiry      

Content/Expert  led  teaching  

Student-­‐led  revision  

The  Knowledge  –  Agency  Window      

Pre-­‐scribed  curriculum    

Open-­‐ended  Enquiry    

Teacher  Directed  Learning     Self-­‐directed  Learning    

Teacher-­‐directed  enquiry      

 Student  led-­‐enquiry      

Content/Expert  led  teaching  

Student-­‐led  revision  

The  Knowledge  –  Agency  Window      

Pre-­‐scribed  curriculum    

Open-­‐ended  Enquiry    

Teacher  Directed  Learning     Self-­‐directed  Learning    

Teacher-­‐directed  enquiry      

 Student  led-­‐enquiry      

Content/Expert  led  teaching  

Student-­‐led  revision  

Pre-­‐scribed  curriculum    

Open-­‐ended  Enquiry    

Teacher  Directed  Learning     Self-­‐directed  Learning    

Teacher-­‐directed  enquiry      

 Student  led-­‐enquiry      

Content/Expert  led  teaching  

Student-­‐led  revision  

So our analytics depends on what space we’re in

Authenticity Agency Identity

43

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

44

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

46

More efficient learning for mastering core skills

Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 14. http://jime.open.ac.uk/article/2008-14/352

“In this study, results showed that OLI-Statistics students [blended learning] learned a full semester’s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester.”

“We’re looking at the profiles of what it means to be effective in the 21st century. […] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger.”

“Dispositions are now at least as important as Knowledge and Skills. …They cannot be taught. They can only be cultivated.”

John Seely Brown

47

US Dept. of Educ. http://reimaginingeducation.org conference (May 28, 2013) Dispositions clip: http://www.c-spanvideo.org/clip/4457327 Whole talk: http://www.c-spanvideo.org/program/SecD

Learning analytics for this?

“It’s more than knowledge and skills. For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity.”

Larry Rosenstock

LearningREimagined project: http://learning-reimagined.com Larry Rosenstock: http://audioboo.fm/boos/1669375-50-seconds-of-larry-rosenstock-ceo-of-hightechhigh-on-how-he-would-re-imagine-learning

Learning analytics for this?

48

Survey-based analytics for learning dispositions (Ruth Deakin Crick, Univ. Bristol / UTS:CIC Visiting Professor)

Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.

49

50

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Survey-based analytics for learning dispositions

Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.

Rapid  Visual  Feedback  to  SGmulate  Self-­‐Directed  Change    

A  framework  for  a  coaching  conversaGon    

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Taking responsibility for my own learning over time through defining my purposes, understanding and managing my feelings, knowing how I go about learning & planning my learning journey carefully.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Making connections between what I already know & new information & experience. Making meaning by linking my story, my new learning & my purpose.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Using my intuition & imagination to generate new ideas & knowledge. Taking risks & playing with ideas and artefacts to arrive at new solutions.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Wanting to get beneath the surface & find out more. Always wondering why and how.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Being part of a learning community at work, at home, in education & in my

social networks. Knowing I have social resources to draw on when I

need them

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Being able to work with others, to collaborate and

co-generate new ideas and artefacts. Being able

to listen and contribute productively to a team.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Having the optimism & hope that I can learn & achieve over

time. Having a growth mindset; believing I can generate my

own new knowledge for what I need to achieve

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

1 Dependent Closed Openness to learning

An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to persist & manage any self-doubt.

A necessary pre-requisite for developing resilience in learning

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

From self-report to behavioural analytics for learning dispositions?

Social network patterns, teamwork effectiveness and

initiation of relationships?

Questioning, arguing and search behaviours reveal

intrinsic curiosity and epistemic commitments?

Tagging/sharing/blogging/social patterns reveal how you see connections between ideas?

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

Relatively simple: •  How much/when library and

other resources are accessed •  Annotations on texts, images,

movies •  Video replays and rewinds •  Learner-shared resources

Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)

What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254

Does the way you search online reveal what you think

counts as trustworthy knowledge?

Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)

Dimensions of Epistemic Belief

Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute, to tentative and evolving

Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts

Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self

Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise

Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226

64

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

65

Are fieldwork students competent?

In the future, how might we track teamworking? (U. Sydney)

R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.

Automatic student tracking from multiple digital tabletops in classroom Analyse the students’ activity traces for significant patterns Visualise on a teacher’s dashboard

In the future, how might we track teamworking?

R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.

Learning  Technology  

KMi,  OU  

AI  &  ArgumentaGon  

Learning  DisposiGons  

Human-­‐Centred    InformaGcs  

Learning  AnalyGcs  

SemanGc  ScienGfic    Publishing  

Dialogue  /  Issue  /  Argument  Mapping  

Social learning analytics — quantifying “professional identity”

Visual analytics to filter a social learning network by topic and type of social tie

Schreurs B, Teplovs C, Ferguson R, De Laat M and Buckingham Shum S. (2013) Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891

Visual analytics to filter a social learning network by topic and type of social tie

Schreurs B, Teplovs C, Ferguson R, De Laat M and Buckingham Shum S. (2013) Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829

What epistemic contributions are learners making in the community?

71

Rebecca is playing the role of broker,

connecting different peers’ contributions in

meaningful ways We now have the basis for recommending that

you engage with people NOT like you…

72

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

Real time traffic-lights for students based on predictive model (Purdue University Signals)

73 Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x

Validate a statistical model from: •  ACT or SAT score •  Overall grade-point average •  LMS usage composite •  LMS assessment composite •  LMS assignment composite •  LMS calendar composite

Predicted 66%-80% of struggling students who needed help

Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen.

In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges

Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on.

1 2 3

45

Prototyping a social learning analytics dashboard

Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910

74

75

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

ELLI Spider widget

Teacher’s dashboard for EnquiryBlogger

The hallmarks of educated writing/critical thinking Relevance Understanding & Knowledge Structure & Organisation Linguistic Accuracy Illustrations Referencing

Argumentation

78

Rhetorical functions of metadiscourse identified by the Xerox Incremental Parser (XIP)

BACKGROUND KNOWLEDGE

“Recent studies indicate …”

“… the previously proposed …”

“… is universally accepted ... “

NOVELTY

... new insights provide direct evidence ...

... we suggest a new ... approach ...

... results define a novel role ...

OPEN QUESTION … little is known … … role … has been elusive

Current data is insufficient …

GENERALIZING ... emerging as a promising approach Our understanding ... has grown exponentially ... ... growing recognition of the importance ...

CONTRASTING IDEAS … unorthodox view resolves … paradoxes …

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE studies ... have provided important advances

Knowledge ... is crucial for ... understanding valuable information ... from studies

SURPRISE We have recently observed ... surprisingly We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARIZING The goal of this study ... Here, we show ...

Altogether, our results ... indicate

79

XIP rhetorical parser applied to student writing

Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391

80

XIP rhetorical parser applied to student writing

Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391

CONTRAST

SUMMARY & CONTRIBUTION

81

82

to conclude…

h,p://th02.deviantart.n

et/fs71/PR

E/f/2011/119/c/0/rabbit_shadow

_by_plasGcx76-­‐d3

f5qtd.jpg  

83

Learners will leave an increasingly rich “digital shadow”

Learning Analytics use this to provide rapid formative feedback (but this is of course an imperfect proxy for real experience)

84

Now the shadow can talk back!

Let’s design shadows that learners + teachers value