Our Learning Analytics are Our Pedagogy

Simon Buckingham Shum Knowledge Media Institute, The Open University UK simon.buckinghamshum.net

Keynote Address, Expanding Horizons 2012, Macquarie University http://staff.mq.edu.au/teaching/workshops_programs/expanding_horizons

1

@sbskmi

learning objective:

walk out with

better questions than you can ask right now

2

Urgent need: quality dialogue between analytics stakeholders, to accelerate invention innovation

3

www.SoLAResearch.org Follow: @SoLAResearch

The global demand for learning

4 http://www.col.org/resources/speeches/2012presentations/Pages/2012-02-01.aspx

Implications for assessment and

feedback at massive scale?

John Daniel

is education poised to become a data-driven enterprise and science

5

?

Possibly 90% of the digital data we have today was generated in the last 2 years

6

Volume outstrips old infrastructure

Variety Internet of things, e-business transactions, environmental sensors, social media, audio, video, mobile…

Velocity The speed of data access and analysis is exploding

A quantitative shift on this scale is in fact a qualitative shift, requiring

new ways of thinking about societal phenomena

edX: “this is big data, giving us the chance to ask big questions about learning”

7

Will the tomorrow’s educational researcher be

as helpless without an analytics infrastructure, as

a geneticist without genome databases, or a physicist without CERN?

Lifelogging: explosion of data capture and sharing about personal activities

8

http://www.mirror-project.eu

http://quantifiedself.com/guide

Educational Data Mining research community

Learning Analytics research community

different levels of analytic

11

‘Learning Analytics’ and ‘Academic Analytics’

Long, P. and Siemens, G. (2011), Penetrating the fog: analytics in learning and education. Educause Review Online, 46, 5, pp.31-40. http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education 12

Macro/Meso/Micro Learning Analytics

Macro: region/state/national/international

Macro/Meso/Micro Learning Analytics

Meso: institution-wide

Macro: region/state/national/international

Micro: individual user actions

(and hence cohort)

Macro/Meso/Micro Learning Analytics

Meso: institution-wide

Macro: region/state/national/international

Will institutions be dazzled by the dashboards, or know what

questions to ask at each level?

Macro/Meso/Micro Learning Analytics

Macro: region/state/national/international

US states are getting the infrastructure in place

17

dataqualitycampaign.org

Shared Learning Collaborative

18

http://slcedu.org

National league tables for English schools

19

Macro/Meso/Micro Learning Analytics

Meso: institution-wide

Analytics-savvy Leaders are the future?

21 Parr-Rud, O. (2012). Drive Your Business with Predictive Analytics. SAS White Paper http://www.sas.com/reg/gen/corp/1800392

Business Intelligence companies see an education market opening up

22 http://www.sas.com/industry/education/highered

These are pedagogically agnostic: they seek to optimize operational

efficiency whatever the sector

These may make pedagogical assumptions: how will learning

design and assessment regimes shape the analytics they offer?

Business Intelligence companies see an education market opening up

23

…but do they know anything about the roles that language plays in

learning and knowledge construction?

BI+HigherEd communities of practice

24

Business Intelligence

≠ Learning Analytics

Micro: individual user actions

(and hence cohort)

Macro/Meso/Micro Learning Analytics

Analytics in your VLE: Blackboard: feedback to students

27

http://www.blackboard.com/Platforms/Analytics/Overview.aspx

Purdue University Signals: real time traffic-lights for students based on predictive model

28

Premise: academic success is defined as a function of aptitude (as measured by standardized test scores and similar information) and effort (as measured by participation within the online learning environment). Using factor analysis and logistic regression, a model was tested to predict student success based on:

•  ACT or SAT score •  Overall grade-point average •  CMS usage composite •  CMS assessment composite •  CMS assignment composite •  CMS calendar composite

Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x

Predicted 66%-80% of struggling students who needed help

Desire2Learn visual analytics & predictive models which can be interrogated on different dimensions

29

http://www.desire2learn.com/products/analytics

Desire2Learn visual analytics & predictive models which can be interrogated on different dimensions

30

http://www.desire2learn.com/products/analytics

Socrato: train for SATs

31 http://www.socrato.com

Khan Academy: more data to teachers, finer-grained feedback to students

32 http://www.thegatesnotes.com/Topics/Education/Sal-Khan-Analytics-Khan-Academy

33

https://grockit.com/research

Adaptive platforms generate fine-grained analytics

Adaptive platforms generate fine-grained analytics

http://knewton.com

Adaptive platforms generate fine-grained analytics

http://oli.cmu.edu

The VLE—BI convergence

36

Micro: individual user actions

(and hence cohort)

Hard distinctions between Learning + Academic analytics may dissolve

Meso: institution-wide

Macro: region/state/national/international

Aggregation of user traces enriches meso + macro analytics with finer-grained process data

…as they get joined up, each level enriches the others

Micro: individual user actions

(and hence cohort)

Hard distinctions between Learning + Academic analytics may dissolve

Meso: institution-wide

Macro: region/state/national/international

Aggregation of user traces enriches meso + macro analytics with finer-grained process data

Breadth + depth from macro + meso levels add power to

micro analytics

…as they get joined up, each level enriches the others

…so everybody’s happy?

dawn of a new data-driven enterprise + science?

39

wrong.

a very healthy debate is brewing…

40

data (indeed technology)

is not neutral

data does not wholly ‘speak for itself’

41

Measurement tools are not neutral

“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure”

Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life

Sage, London. pp. 12-13

Beyond big data hubris

1.  Automating Research Changes the Definition of Knowledge

2.  Claims to Objectivity and Accuracy are Misleading 3.  Bigger Data are Not Always Better Data 4.  Not All Data Are Equivalent 5.  Just Because it is Accessible Doesn’t Make it

Ethical 6.  Limited Access to Big Data Creates New Digital

Divides

boyd, d. and Crawford, K. (2001). Six Provocations for Big Data. Presented to: A Decade in Internet Time: Symposium on the

Dynamics of the Internet and Society, Oxford Internet Institute, Sept. 21, 2011.

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

Analytics provide maps = systematic ways of distorting reality

“A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.”

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling

and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM: New York.

Eprint: http://oro.open.ac.uk/32823

context

context

context

45

Analytics in English schools RAISEonline platform: cohort visualization

46

Will our analytics reflect the progress that ‘Joe’ has made on so many other fronts – but not his SATs?

47

Will our analytics reflect the progress that ‘Joe’ has made on so many other fronts – but not his SATs?

48

?

Video conferencing analytics OU KMi’s Flashmeeting

49

Video conference spoken foreign language tutorials Se

ssio

n

AV Chat AV Chat

2

3

Mentor 1 Mentor 2

Video conferencing analytics OU KMi’s Flashmeeting

50

Video conference spoken foreign language tutorials Se

ssio

n

AV Chat AV Chat

1

2

3

Mentor 1 Mentor 2

Video conferencing analytics OU KMi’s Flashmeeting

51

Video conference spoken foreign language tutorials Se

ssio

n

AV Chat AV Chat

1

2

3

Mentor 1 Mentor 2

— which mentor would you want to have?...

Video conferencing analytics OU KMi’s Flashmeeting

52

Video conference spoken foreign language tutorials Se

ssio

n

AV Chat AV Chat

1

2

3

Mentor 1 Mentor 2

— which mentor would you want to have?...

Mentor 1 is doing the best job: at this introductory

level, students need intensive input and

flounder if left

course completion is only one proxy for good learning

and what’s easy to

measure isn’t always what’s most important

53

The Wal-Martification of education?

54 http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college http://lak12.wikispaces.com/Recordings

The Wal-Martification of education?

55 http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college http://lak12.wikispaces.com/Recordings

“What counts as data, how do you get it, and what does it

actually mean?”

“The basic question is not what can we measure? The basic question is

what does a good education look like?

Big questions.

“data narrowness” “instrumental learning”

“students with no curiosity”

Course completion as a proxy for learning

56

http://annezelenka.com/2012/05/05/but-what-about-learning

Course completion as a proxy for learning

57

http://annezelenka.com/2012/05/05/but-what-about-learning

NOTE TO SELVES: We are the HigherEd market who make it

worthwhile for major vendors to design analytics focused on

maximising course completion

PS: HEIs may feel that they are trapped by external expectations

and requirements. Systems thinking required…

let’s just pretend that learning analytics took seriously the revolution going on outside the

university front door…

we would need to devise

learning analytics for this?...

58

“We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.”

“Shift Happens” http://shifthappens.wikispaces.com

59

Learning analytics for this?

Learning analytics for this?

“While employers continue to demand high academic standards, they also now want more. They want people who can adapt, see connections, innovate, communicate and work with others. This is true in many areas of work. The new knowledge-based economies in particular will increasingly depend on these abilities. Many businesses are paying for courses to promote creative abilities, to teach the skills and attitudes that are now essential for economic success…”

All our Futures: Creativity, culture & education, May 1999

60

“Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.”

John Dewey, 1933

61

Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Heath and Co, Boston, 1933

Learning analytics for this?

Learning analytics for this?

“The test of successful education is not the amount of knowledge that pupils take away from school, but their appetite to know and their capacity to learn.”

Sir Richard Livingstone, 1941

62

Expert-led enquiry

Student-led enquiry

Expert-led teaching

Student-led revision

Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge

Teacher agency Student agency

Repetition, Abstraction Acquisition

Authenticity Agency Identity

Teaching as learning design

The Knowledge-Agency Window

Learning analytics for this?

Learning analytics for this?

64

Creativity, Culture and Education (2009) Changing Young Lives 2012. Newcastle: CCE. http://www.creativitycultureeducation.org/changing-young-lives-2012

Musicality ≠ Musical Reproduction

65

In those early days the children were taught from the start to develop their own voice, whether literally singing, or through the

instrument they played. They were not taught music, but musicality. Central to this tuition were the partimenti, many

pages of detailed music notes which pose many questions, but leave the pupil to find the solutions. The music is not a literal transcript, which the musician reads and reproduces.

The partimenti establish, at the start, a set of rules and then pose a set of conflicts for the musician to resolve, in their own way.

http://bit.ly/U1vkNf

consider assessment for learning

(not summative assessment for

grading pupils, teachers, institutions or nations)

66

Assessment for Learning

67

http://assessment-reform-group.org

Assessment for Learning

68

http://assessment-reform-group.org

Assessment for Learning

69

http://assessment-reform-group.org

To what extent could automated

feedback be designed and evaluated with

emotional impact in mind?

Assessment for Learning

70

http://assessment-reform-group.org

Can analytics identify proxies

for such advanced qualities?

Assessment for Learning

71

http://assessment-reform-group.org

Do analytics provide constructive next

steps?

Assessment for Learning

72

http://assessment-reform-group.org

How do we provide analytics feedback

that does not disempower and de-motivate struggling

learners?

analytics for…

dispositions discourse

social networks

73 See SoLAR Storm: Social Learning Analytics symposium http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium

Social Learning Analytics

§  Analytics focused on social learning theories, practices and platforms, e.g.

§  Discourse analytics: beyond quantitative summaries of online writing, to qualitative analysis

§  Social network analytics: visualizing effective social ties for collective learning

§  Dispositional analytics: measuring students’ readiness to engage in lifelong, lifewide learning

Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM Press. Eprint: http://oro.open.ac.uk/32910

Buckingham Shum, S. and Ferguson, R., Social Learning Analytics. Educational Technology & Society (Special Issue on Learning & Knowledge Analytics, Eds. G. Siemens & D. Gašević), 15, 3, (2012), 3-26. http://www.ifets.info Open Access Eprint: http://oro.open.ac.uk/34092

-60

-40

-20

0

20

40

60

80

9:2

8 9

:32

9:3

6

9:4

0

9:4

1

9:4

6

9:5

0

9:5

3

9:5

6

10:

00

10:

05

10:

07

10:

07

10:

09

10:

13

10:

17

10:

23

10:

27

10:

31

10:

35

10:

40

10:

45

10:

52

10:

55

11:

04

11:

08

11:

11

11:

17

11:

20

11:

24

11:

26

11:

28

11:

31

11:

32

11:

35

11:

36

11:

38

11:

39

11:

41

11:

44

11:

46

11:

48

11:

52

11:

54

12:

00

12:

03

12:

04

12:

05

Average Exploratory

Discourse analytics on webinar textchat

Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny here!

See you! bye for now! bye, and thank you Bye all for now

Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak…

Discourse analytics on webinar textchat

-100

-50

0

50

100

9:2

8 9

:40

9

:50

1

0:00

1

0:07

1

0:17

1

0:31

1

0:45

1

1:04

1

1:17

1

1:26

1

1:32

1

1:38

1

1:44

1

1:52

1

2:03

Averag

Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

Classified as “exploratory

talk”

(more substantive for learning)

“non-exploratory”

Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations?

Discourse Network Analytics = Concept Network + Social Network Analytics

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

KMi’s Cohere: a web deliberation platform enabling semantic social network and discourse network analytics

Rebecca is playing the role of broker,

connecting 2 peers’ contributions in meaningful ways

Analytics for C21 learning skills

Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823

Different social network patterns

in different contexts may

load onto Learning

Relationships

Questioning and challenging may load onto Critical

Curiosity

Sharing relevant resources from other contexts may load onto

Meaning Making

Repeated attempts to pass

an online test may load onto

Resilience

LearningEmergence.net: embedding dispositional analytics into practice + tools EnquiryBlogger: Wordpress plugins for reflective learning journals

80

Discourse analysis (Xerox Incremental Parser)

BACKGROUND KNOWLEDGE:

Recent studies indicate …

… the previously proposed …

… is universally accepted ...

NOVELTY:

... new insights provide direct evidence ...

... we suggest a new ... approach ...

... results define a novel role ...

OPEN QUESTION: … little is known … … role … has been elusive

Current data is insufficient …

GENERALIZING: ... emerging as a promising approach Our understanding ... has grown exponentially ... ... growing recognition of the

importance ...

CONRASTING IDEAS: … unorthodox view resolves … paradoxes …

In contrast with previous hypotheses ...

... inconsistent with past findings ...

SIGNIFICANCE: studies ... have provided important advances

Knowledge ... is crucial for ... understanding

valuable information ... from studies

SURPRISE: We have recently observed ... surprisingly

We have identified ... unusual The recent discovery ... suggests intriguing roles

SUMMARIZING: The goal of this study ... Here, we show ...

Altogether, our results ... indicate

Detection of salient sentences in scholarly reports, based on the rhetorical signals authors use:

Ágnes Sándor & OLnet Project: http://olnet.org/node/512

De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

Human and machine analysis of a text for key contributions

19 sentences annotated 22 sentences annotated 11 sentences same as human annotation

71 sentences annotated 59 sentences annotated 42 sentences same as human annotation

Document 1

Document 2

http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

Closing thoughts

83

84

“The basic question is not what can we measure?

The basic question is what does a good education look like?

Big questions. (Gardner Campbell)

Do we value what we can measure, or measure what we really value? And just because this is tough to do, doesn’t mean we don’t do it.

(Guy Claxton, BBC Radio 4 Education Debate, Nov. 2012)

85

Our analytics promote values, pedagogy and assessment regimes.

Are we clear which master

our analytics serve? Are we happy to be judged by them?

http://simon.buckinghamshum.net http://twitter.com/sbskmi