uts-learning-futures-learning-analytics

LEARNING ANALYTICS — an essential tool for learning in the future

uts.edu.au UTS CRICOS PROVIDER CODE: 00099F

Simon Buckingham Shum Professor of Learning Informatics Director, Connected Intelligence Centre twitter @sbuckshum #UTSfuturelearn

cic.uts.edu.au

3

can we tell from your digital profile if you’re learning?

4

Every educational product now comes with a “dashboard”…

5

I was speaking at this event

http://codeactsineducation.wordpress.com

6

I asked Siri to find the conference website…

“Find code acts in education”

7

“Find code acts in education”

I asked Siri to find the conference website…

h,ps://twi,er.com

/Wisw

ijzer2/status/414055472451575808

8

“Note: check the huge difference between

knowing and measuring…”

Introducing ‘Joe’

9

Visualising “attainment” and “progress”

10

‘Joe’ may be developing curiosity, resilience, and collaboration skills, but these are left invisible

CRITICAL: accounting tools are not neutral

Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life. Sage, London. pp. 12-13

“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure”

12

sta:s:cal analysis of educa:onal data is hardly new

13

14

Institutional/Academic Data and Analytics: current focus on summative data and trends for senior educational leaders and policymakers

Macro: region/state/national/international

15



16


Meso: institution-wide


17

NSW DEC • CESE Datahub (+ many comparable open data initiatives)

https://data.cese.nsw.gov.au

18

UK RAISEonline data: schools live or die by this

19

Learning Analytics: Data explosion at the micro level enriching, and enriched by, the meso+macro levels

Learning Analytics UNESCO Policy Briefing — http://iite.unesco.org/publications/3214711

Micro: individual student progress

Meso: institution-wide


outcome

Towards rapid formative feedback

researchers / educators / instrucGonal designers administrators / leaders / policymakers

intent

20

outcome Faster feedback loops could enable more rapid adaptation by students and teachers, and of learning resources and designs

researchers / educators / instrucGonal designers administrators / leaders / policymakers

intent

21

Towards rapid formative feedback

22 http://cdn.nmc.org/media/2014-nmc-horizon-report-k12-EN.pdf

23 http://redarchive.nmc.org/horizon-project/horizon-reports http://www.open.ac.uk/blogs/innovating

24 http://solaresearch.org/Policy_Strategy_Analytics.pdf http://tech.ed.gov/files/2013/02/Expanding-Evidence-Approaches.pdf

Ed-Tech startups and venture capital

26 https://www.edsurge.com/n/2014-10-06-diving-into-data-analytics-tools-in-k-12

http://techcrunch.com/2014/02/19/google-capital-invests-40m-in-learning-analytics-firm-renaissance-learning-at-1b-valuation

Ed-Tech startups and venture capital

27 http://learncapital.com

“LearnCapital is a venture capital firm focused exclusively on funding entrepreneurs with a vision for better and smarter

learning.”

Hang on: algorithm designers in ed-tech startups are shaping the learning experience?…

28 h,p://reinventors.net/roundtables/strategies-‐for-‐organizaGonal-‐change

Candace Thille (Stanford)

“Why would universities outsource their core

competency to ed-tech vendors?”

the core ques:on in slow mo)on

29

30


31


Who?

32


Who? How? With what confidence? After what kinds of training?

33



Sourcing which data, with what integrity?

34



Sourcing which data, with what integrity?

What kind of learning? What kind of teaching?

35

what kinds of learners are we trying to create?

– this should drive our analytics

Let’s hear from the UTS graduates of 2026

36 http://learningemergence.net/2014/04/07/learning-dispositions-authentic-inquiry-in-a-primary-school

The Knowledge – Agency Window

Pre-‐scribed curriculum

Open-‐ended Enquiry

Teacher Directed Learning

Self-‐directed Learning

Teacher-‐directed enquiry

Student led-‐enquiry

Content/Expert led teaching

Student-‐led revision

38 Guy Claxton: Constant change is here to stay: why schooling will always be about the future. UK ESRC Futures Meeting, May 2011. http://www.slideshare.net/edfutures/guy-claxton-esrc-futures-may11

We worry about disengaged low achievers like Joe — but we need to worry about Emily too...

The Knowledge – Agency Window



Teacher Directed Learning Self-‐directed Learning







Teacher Directed Learning Self-‐directed Learning





So our analytics depends on what space we’re in

Authenticity Agency Identity

43

Analytics for the UTS learning model?

http://www.uts.edu.au/research-and-teaching/teaching-and-learning/learning2014/new-approaches

44



46

More efficient learning for mastering core skills

Lovett M, Meyer O and Thille C. (2008) The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education 14. http://jime.open.ac.uk/article/2008-14/352

“In this study, results showed that OLI-Statistics students [blended learning] learned a full semester’s worth of material in half as much time and performed as well or better than students learning from traditional instruction over a full semester.”

“We’re looking at the profiles of what it means to be effective in the 21st century. […] Resilience will be the defining concept. When challenged and bent, you learn and bounce back stronger.”

“Dispositions are now at least as important as Knowledge and Skills. …They cannot be taught. They can only be cultivated.”

John Seely Brown

47

US Dept. of Educ. http://reimaginingeducation.org conference (May 28, 2013) Dispositions clip: http://www.c-spanvideo.org/clip/4457327 Whole talk: http://www.c-spanvideo.org/program/SecD

Learning analytics for this?

“It’s more than knowledge and skills. For the innovation economy, dispositions come into play: readiness to collaborate; attention to multiple perspectives; initiative; persistence; curiosity.”

Larry Rosenstock

LearningREimagined project: http://learning-reimagined.com Larry Rosenstock: http://audioboo.fm/boos/1669375-50-seconds-of-larry-rosenstock-ceo-of-hightechhigh-on-how-he-would-re-imagine-learning

Learning analytics for this?

48

Survey-based analytics for learning dispositions (Ruth Deakin Crick, Univ. Bristol / UTS:CIC Visiting Professor)

Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.

49

50

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Survey-based analytics for learning dispositions

Deakin Crick et al (In Press). Developing Resilient Agency in Learning: the Crick Learning for Resilient Agency Profile.

Rapid Visual Feedback to SGmulate Self-‐Directed Change

A framework for a coaching conversaGon

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Taking responsibility for my own learning over time through defining my purposes, understanding and managing my feelings, knowing how I go about learning & planning my learning journey carefully.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Making connections between what I already know & new information & experience. Making meaning by linking my story, my new learning & my purpose.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Using my intuition & imagination to generate new ideas & knowledge. Taking risks & playing with ideas and artefacts to arrive at new solutions.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Wanting to get beneath the surface & find out more. Always wondering why and how.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Being part of a learning community at work, at home, in education & in my

social networks. Knowing I have social resources to draw on when I

need them

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Being able to work with others, to collaborate and

co-generate new ideas and artefacts. Being able

to listen and contribute productively to a team.

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

Having the optimism & hope that I can learn & achieve over

time. Having a growth mindset; believing I can generate my

own new knowledge for what I need to achieve

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

1 Dependent Closed Openness to learning

An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to persist & manage any self-doubt.

A necessary pre-requisite for developing resilience in learning

Mindful Agency

Sense making

Creativity

Curiosity Belonging

Collaboration

Hope and optimism

From self-report to behavioural analytics for learning dispositions?

Social network patterns, teamwork effectiveness and

initiation of relationships?

Questioning, arguing and search behaviours reveal

intrinsic curiosity and epistemic commitments?

Tagging/sharing/blogging/social patterns reveal how you see connections between ideas?



Relatively simple: •  How much/when library and

other resources are accessed •  Annotations on texts, images,

movies •  Video replays and rewinds •  Learner-shared resources

Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)

What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254

Does the way you search online reveal what you think

counts as trustworthy knowledge?

Shifts in epistemic commitments? (Simon Knight, KMi Open U. UK)

Dimensions of Epistemic Belief

Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute, to tentative and evolving

Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts

Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self

Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise

Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226

64



65

Are fieldwork students competent?

In the future, how might we track teamworking? (U. Sydney)

R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.

Automatic student tracking from multiple digital tabletops in classroom Analyse the students’ activity traces for significant patterns Visualise on a teacher’s dashboard

In the future, how might we track teamworking?

R. Martinez, K. Yacef, J. Kay, and B. Schwendimann. An interactive teacher’s dashboard for monitoring multiple groups in a multi-tabletop learning environment. Proceedings of Intelligent Tutoring Systems, pages 482-492. Springer, 2012.

Learning Technology

KMi, OU

AI & ArgumentaGon

Learning DisposiGons

Human-‐Centred InformaGcs

Learning AnalyGcs

SemanGc ScienGfic Publishing

Dialogue / Issue / Argument Mapping

Social learning analytics — quantifying “professional identity”

Visual analytics to filter a social learning network by topic and type of social tie

Schreurs B, Teplovs C, Ferguson R, De Laat M and Buckingham Shum S. (2013) Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829

What epistemic contributions are learners making in the community?

71

Rebecca is playing the role of broker,

connecting different peers’ contributions in

meaningful ways We now have the basis for recommending that

you engage with people NOT like you…

72



Real time traffic-lights for students based on predictive model (Purdue University Signals)

73 Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x

Validate a statistical model from: •  ACT or SAT score •  Overall grade-point average •  LMS usage composite •  LMS assessment composite •  LMS assignment composite •  LMS calendar composite

Predicted 66%-80% of struggling students who needed help

Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen.

In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges

Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on.

1 2 3

45

Prototyping a social learning analytics dashboard

Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910

74

75



ELLI Spider widget

Teacher’s dashboard for EnquiryBlogger

The hallmarks of educated writing/critical thinking Relevance Understanding & Knowledge Structure & Organisation Linguistic Accuracy Illustrations Referencing

Argumentation

78

Rhetorical functions of metadiscourse identified by the Xerox Incremental Parser (XIP)

BACKGROUND KNOWLEDGE

“Recent studies indicate …”

“… the previously proposed …”

“… is universally accepted ... “

NOVELTY

... new insights provide direct evidence ...

... we suggest a new ... approach ...

... results define a novel role ...

OPEN QUESTION … little is known … … role … has been elusive

Current data is insufficient …

GENERALIZING ... emerging as a promising approach Our understanding ... has grown exponentially ... ... growing recognition of the importance ...

CONTRASTING IDEAS … unorthodox view resolves … paradoxes …

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE studies ... have provided important advances

Knowledge ... is crucial for ... understanding valuable information ... from studies

SURPRISE We have recently observed ... surprisingly We have identified ... unusual

The recent discovery ... suggests intriguing roles

SUMMARIZING The goal of this study ... Here, we show ...

Altogether, our results ... indicate

79

XIP rhetorical parser applied to student writing

Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391

80

XIP rhetorical parser applied to student writing

Simsek, D., S. Buckingham Shum, Á. Sándor, A. De Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, at 3rd International Conference on Learning Analytics & Knowledge. Leuven, BE (Apr. 8-12, 2013). http://oro.open.ac.uk/37391

CONTRAST

SUMMARY & CONTRIBUTION

81

82

to conclude…

h,p://th02.deviantart.n

et/fs71/PR

E/f/2011/119/c/0/rabbit_shadow

_by_plasGcx76-‐d3

f5qtd.jpg

83

Learners will leave an increasingly rich “digital shadow”

Learning Analytics use this to provide rapid formative feedback (but this is of course an imperfect proxy for real experience)

84

Now the shadow can talk back!

Let’s design shadows that learners + teachers value

uts-learning-futures-learning-analytics

Education

summative data

institutionalacademic

learning analytics

data explosion

onal data

policymakers macro

reports http

event http