Claudia Hauff

Web Information Systems, TU Delft

Large-scale Learning Analytics

It’s data that is on the Web … Web data … lets find the Web Information Systems people!

✤ 40+ MOOCs ✤ 1+ Million enrollments ✤ From primary school to PhD level ✤ Lots of user data (click logs)

Our goals

Data

Knowledge

Application to learning

Gain actionable insights into learner behaviours at scale. a. Data Science b. Big data processing

Increase our knowledge about learners by looking beyond the learning platform a. Web data analytics

Design technology interventions that enable adaptive learning at scale. a. Web data analytics b. Human-centered design c. Learning technologies

Learner profiling beyond the MOOC platform

ACM WebScience 2016

Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform: Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.

Whythis research?

Learner

Before the MOOC

NOTHING

Engagement, retention, …

During the MOOC

NOTHING

After the MOOC

Howto solve the problem?

We propose:

a deeper understanding about learnerscan be gained by exploring their traces on the Social Web.

Whatresearch questions?

1 On what Social Web platforms can a significant fraction of MOOC learners be identified?

Are learners who demonstrate specific traitson the Social Web drawn to certain types of MOOCs? 2

To what extent do Social Web platforms enable us to observe (specific) user attributes

that are relevant to the online learning experience? 3

Learner identificationacross Social Web platforms

edX learners

Email Login name Full name+ +

1. Explicit Matching

Profile images & links

Identification via emails

2. Direct Matching

Identification via profile links from Step 1

3. Fuzzy Matching

Search learners by their login & full names

Compare: 1. profile link2. profile image3. login & full names

Social Web platformsinvolved in our work

Matching resultsfor 18 DelftX MOOCs

Lowest Highest Overall

Gravatar 4.37% 23.49% 7.81%

Twitter 4.99% 17.58% 7.78%

Linkedin 3.90% 11.05% 5.89%

StackExchange 1.23% 21.91% 4.58%

GitHub 3.43% 41.93% 10.92%

Matching resultsfor 18 DelftX MOOCs

Lowest Highest Overall

Gravatar 4.37% 23.49% 7.81%

Twitter 4.99% 17.58% 7.78%

Linkedin 3.90% 11.05% 5.89%

StackExchange 1.23% 21.91% 4.58%

GitHub 3.43% 41.93% 10.92%

On average, 5% of learners can be identified on globally popular Social Web platforms.

Learners on

Linkedin- Using job titles & skills to characterise learners

Spreadsheet MOOC- Software Engineer- Business Analyst- …

Design Approach MOOC- Co founder- UX designer- …

Learners onStackExchange

- Functional Programming learners in StackOverflow

- To what extent do learners change their question/answering behaviour during and after a MOOC?

Take-homeMessages

On average, 5% of learners from 18 DelftX MOOCscan be identified on 5 globally popular Social Web platforms. 1

Learners with specific traits prefer different types of MOOCs.2

Learners’ post-course behaviour can be investigated by using their external Social Web traces.3

Learning Transfer: does it take place?

Best Paper Nominee at ACM Learning At Scale 2016

An Investigation into the Uptake of Functional Programming in Practice

Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.

Whatis learning transfer?

Learning transfer is the application of knowledge or skills gained in a learning environment to another context.

Whydo we care?

Learning transfer is a more important measure of learning in MOOCs than retention, success or engagement.

FP101x

@flickr:christiaan_008

Course programming language: Haskell

Run as a typical video-lecture based MOOC

Assessment: 288 Multiple Choice questions

Introduction to Functional Programming

37,485 learners registered.41% engaged with the course. 5% completed the course.33% were active on GitHub (1.1M events).

Whatdid we do?

FP101xlogs surveys coding

activities

3 months 2.5 years + 0.5 years

+ +

email address

Are changes made in a functional language?

GitHub

10+ million registered users

hosting, collaboration and organisation

the most popular social coding platform

founded in 2007long-term

large-scale

detailed

detailed logs

code changes

project meta-data

A sanity check

Are “GitHub learners” different? GitHub

learnersNon-GitHub

learners

#Learners 12,415 25,070

Completion rate 7.71% 4.03%

Avg. time watching videos 49.1 min 27.7 min

Avg. #questions attempted 31.3 17.5

Avg. accuracy of learners’ answers 23.4% 12.9%

GitHub learners are more engaged than non-GitHub learners and exhibit higher levels of knowledge.

Are “Expert learners” different? Expert GitHub

learnersNovice GitHub

learners

#Learners 1,721 10,694

Completion rate 15% 6.5%

Avg. time watching videos 78.6 min 44.4 min

Avg. #questions attempted 57.9 27.0

Avg. accuracy of learners’ answers 38.0% 21.1%

Expert learners are more engaged than Novice learners and exhibit higher levels of knowledge.

To what extent do engaged learners exhibit learning transfer?

5-10% >30%10-30%<5%

To what extent do engaged learners exhibit learning transfer?

5-10%

Which type of learner is more likely to display learning transfer?

flickr@ConalGallagher

Intrinsically motivated Extrinsically motivated

Which type of learner is more likely to display learning transfer?

flickr@ConalGallagher

Intrinsically motivated

Which type of learner is more likely to display learning transfer?

Experienced Inexperienced

Which type of learner is more likely to display learning transfer?

Experienced

Which type of learner is more likely to display learning transfer?

High-spacinglearning routine

Low-spacinglearning routine

Which type of learner is more likely to display learning transfer?

High-spacinglearning routine

Learners who transfer quickly move to Scala

FP101x

Conclusions

Most transfer learning findings from the classroom hold.

The observed transfer rate is low: 8.5%.

Learners quickly moved on after the course to industrially-relevant functional languages.

@flickr:torsten-reuschling

From Learners to Earners: Enabling MOOC Learners to Apply their Skills and Earn Money in an

Online Market Place

IEEE Transactions on Learning Technologies

Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in

an Online Market Place, IEEE Transactions on Learning Technologies.

WhatMOOCs aim to educate the world. Most successful learners are already highly educated. Learners from developing countries are underrepresented.

is the problem?

Whatis the problem?

EX101x: Data Analysis to the MAX()

HowPay learner at scale: recommend tasks from online market places to learners that are relevant to the course material.

can we tackle it?

Howcan we tackle it?

What1) To what extent do online market

places contain relevant tasks? 2) Are learners able to solve

real-world tasks with high quality?

do we need to look at?

Setup1) Weekly spreadsheet “bonus

exercises” drawn from Upwork (manually checked) in EX101x

2) Accuracy check 3) Quality check (code smells)

Howare learners doing?

Good accuracy & quality.

Built a workingrecommender.

Deployed in a MOOC by the end of October.

Our goals one more time…

Data

Knowledge

Application to learning

Gain actionable insights into learner behaviours at scale.

Increase our knowledge about learners by looking beyond the learning platform

Design technology interventions that enable adaptive learning at scale.

MOOCs are vital to bring higher education to the world. Lots of unexplored potential. Plenty of data. Many users.http://bit.ly/lambda-lab

Overall …