learning analytics 101

June 10-15, 2012

Growing Community; Growing Possibilities

Learning Analytics 101

Steve Lonn, University of MichiganJosh Baron, Marist College

2012 Jasig Sakai Conference 2

1. What is Learning Analytics (LA)?

2. Current LA work in Higher Education

3. Data available in Sakai CLE & OAE

4. Big Questions to Ponder

5. Q & A

Slides Available: slideshare.net/stevelonn/

Agenda


BIG Data

“...datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.”

Manyika et al. (2011)


Big Data in Higher Education


Big Data in Higher Education

Analytics:

An overarching concept that is defined as data-driven

decision making

van Barneveld, Arnold, & Campbell, 2012adapted from Ravishanker


Analytics at Your Institution RIGHT NOW


Analytics at Your Institution RIGHT NOW

Business / Academic Analytics:

A process for providing higher education institutions with

the data necessary to support operational and financial

decision making

van Barneveld, Arnold, & Campbell, 2012adapted from Goldstein and Katz

8

evidenceframework.org/big-data/

Educational Data Mining

Learning Analytics

Bienkowski, Feng, & Means, 2012◦ SRI International

2012 Jasig Sakai Conference

Dept. of Education Issue Brief

http://www.ed.gov/edblogs/technology/research/




Generally emphasizes reduction into small, easily analyzable components◦ Can be then adapted to student by software

◦ Siemens and Baker, 2012

Predicting future learning behavior

Domain models for content / sequences

Software-provided pedagogical supports

Computational models that incorporate student, domain, and pedagogy



Example: Cognitive Tutors Pittsburgh Advanced Cognitive Tutor Center Carnegie

MellonUniversity


http://ctat.pact.cs.cmu.edu


Educational Data Mining:

A process for analyzing data collected during

teaching and learning to test learning theories and

inform educational practice

Bienkowski, Feng, & Means, 2012


Understand entire systems and support human decision making

Applies known methods & models◦ answer questions about learning and

organizational learning systems

Tailored responses◦ adapted instructional content, specific

interventions, providing specific feedback

Learning Analytics


Learning Analytics:

The use of analytic techniques to help target instructional, curricular, and support resources to support

the achievement of specific learning goals through

applications that directly influence educational practice

van Barneveld, Arnold, & Campbell, 2012adapted from Bach


Predictive Analytics◦ uncover relationships and patterns◦ can be used to predict behavior and events

Visual Data Analytics◦ discovering and understanding patterns in large

datasets via visual interpretation

Additional Terms


Terms are not ExclusiveTerm Definition Level of

Focus

Analytics An overarching concept that is defined as data-driven decision making

All levels

Academic Analytics

A process for providing higher education institutions with the data necessary to support operational and financial decision making

Institution


A process for analyzing data collected during teaching and learning to test learning theories and inform educational practice

Department / Instructor / Learner

Learning Analytics

The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals through applications that directly influence educational practice

Department / Instructor / Learner


Examples of LA ProjectsWho’s been working in this space in Higher Education?

17

Purdue University’s Course Signals

◦ College-wide learning analytics approach

University of Michigan’s E2Coach

◦ Course-specific learning analytics approach

UMBC’s “Check My Activity” Tool

◦ Student-centered learning analytics approach


Three Different Approaches…

18

Built predictive model using data from…◦ LMS – Events (login, content, discuss.) & gradebook

◦ SIS – Aptitude (SAT/ACT, GPA) & demographic data

Leverage model to create Early-alert

system◦ Identify students at risk to not complete the course

◦ Deploy intervention to increase chances of success

Systems automates intervention process◦ Students get “traffic light” alert in LMS

◦ Messages are posted to student that

suggest corrective action (practice tests)



19

Impact on course grades and retention◦ Students in courses using Course Signals…

scored up to 26% more A or B grades up to 12% fewer C's; up to 17% fewer D's and F‘s

Ellucian product that integrates w/Blackboard

Open Academic Analytics Initiative (OAAI)◦ Creating a similar Sakai-based OS solution2012 Jasig Sakai Conference


Arnold & Pistilli, 2012 - LAK

20

Focused specifically on introductory Physics Uses data from…

◦ Pre-course survey: academic info,

learner’s goals, psycho-social factors

◦ Performance: Exams, Web HW, Sakai

Michigan Tailoring System (MTS)◦ OS tool designed for highly customized messaging

◦ Used in health sciences for behavior change

◦ Messaging based on input from many sources


University of Michigan’s E2Coach

“…to say to each what we would say if we could sit down with them for a

personal chat.”


Example MTS Message


UMBC found that students earning D/F’s use Bb 39% lessthen higher grade achievers◦ Not suggesting cause and effect◦ Goal is to model higher achiever

behavior Provides data directly student

◦ Compare LMS use to class averages◦ Can also compare averages usage

data to grade outcomes Feedback has been positive

UMBC’s Check My Activity Tool


Student Success Plan – Sinclair CC◦ Holistic case-management system◦ Connects faculty, advisors, counselors, & students◦ Jasig Incubation Project

STAR Academic Journey – U of Hawaii◦ Online advising and degree attainment system

SNAPP – UBC/Wollongong◦ Visualize networks of

interaction resulting fromdiscussion forum posts andreplies

A Few Others Projects…

24

Papers and Articles on Purdue’s Course Signals http://www.itap.purdue.edu/learning/research/

Michigan’s Expert Electronic Coaching http://

sitemaker.umich.edu/ecoach/home

UMBC’s Check My Activity Tool http://

www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/Vi

deoDemoofUMBCsCheckMyActivit/219113

Student Success Plan http://

www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolu

m/TheStudentSuccessPlanCaseManag/242785

STAR Academy Journey http://

net.educause.edu/ir/library/pdf/pub7203cs7.pdf

SNAPP http://research.uow.edu.au/learningnetworks/seeing/snapp 2012 Jasig Sakai Conference

More information at…

http://www.itap.purdue.edu/learning/research/

http://www.itap.purdue.edu/learning/research/

http://sitemaker.umich.edu/ecoach/home

http://sitemaker.umich.edu/ecoach/home

http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/VideoDemoofUMBCsCheckMyActivit/219113



http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/TheStudentSuccessPlanCaseManag/242785



http://net.educause.edu/ir/library/pdf/pub7203cs7.pdf



http://research.uow.edu.au/learningnetworks/seeing/snapp




Data Available in SakaiWhat can we know in CLE and OAE products?


User-level data stored as “events”

List of events available on Confluence◦ Search for “event table description”

CLE Data Overview

sakai_session SESSION_ID SESSION_USER SESSION_IP SESSION_USER_AGENT SESSION_START SESSION_END SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME

sakai_eventEVENT_IDEVENT_DATEEVENTREFSESSION_ID

EVENT_CODECONTEXT


Site-level data stored in separate tables

CLE Data Overview

sakai_siteCUSTOM_PAGE_ORDEREDSITE_IDTITLETYPESHORT_DESCDESCRIPTIONICON_URLINFO_URLSKINPUBLISHEDJOINABLEPUBVIEWJOIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDONIS_SPECIALIS_USER

sakai_realmREALM_KEYREALM_IDPROVIDER_IDMAINTAIN_ROLECREATEDBYMODIFIEDBYCREATEDONMODIFIEDON

realm_id like '/site/' || site_id


F04

W05 F0

5W

06 F06

W07 F0

7W

08 F08

W09 F0

9W

10 F10

W11 F1

1W

120

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

Project Sites Logarithmic (Project Sites)Course Sites Logarithmic (Course Sites)Max Users Logarithmic (Max Users)Logarithmic (Max Users)

Thanks to John Leasia

Site Creation & Concurrent Users


Overall Tool Usage

43%

25%

18%

3%

3% 2

%1% 1% 1%

Presence Web Content ResourcesAttachments Test Center AssignmentsSyllabus Forums GradebookDrop Box Evaluations MessagesPreferences Basic LTI Site Info / SetupRealms Announcements ChatCalendar Digest ChecklistiTunes U Help ModulesEmail Archive Course Eval Help PodcastsWiki Polls NewsSign Up Library Materials OSPDiscussion Engin Honor Code UbookLibrary Help Super User DB HelpSearch Page Order Global Alert


Instructors Using / Not Using Sakai


Dental School


Subjects Using the Wiki Tool

Sum of RevisionsBITENGRNURSIOESIENGLISHRCHUMSAAASEECSRCLANGCOMPPSYCH

Count of Course Sites

SIENGLISHBITEECSPSYCHCOMPMODGREEKNURSNRELING


Summary information about site visits, tool activity, and resource activity

Site Stats Tool


Site Stats: Activity Detail

35

User-level data available via “activity feeds”◦ follows a “push and publication” model rather than

a “store and query” model (CLE is store & query)

◦ Activity is both highly specific: individual interactions between users, content, contexts…

◦ …and more general: user interaction everywhere rather than only within a single course context.

What new questions will we ask?◦ Interesting activity can happen with external

capabilities: CLE tools, LTI tools, widgets. How will we ensure this data is captured?

Analytics in OAE


Many thanks to Nate Angell for OAE slides


OAE activity stream design: feeds

FYI: Designs are still in draft form.

37

OAE activity stream design: user



38

OAE activity stream design: content




Activity (OAE) & Grades (CLE): Week 1

Student success in OAE & CLE

Developed by the Kaleidoscope Project in collaboration with rSmart.


Activity (OAE) & Grades (CLE): Week 7




Activity (OAE) & Grades (CLE): Animation




Tools / services to support analytics initiatives◦ Ways to connect different silos of data◦ Methods to connect back to CLE / OAE

LTI? Web services? Others?

OAE improvements over CLE approach to user data◦ What data is most relevant for analytics?◦ What displays and/or data are most useful to help

learners?

Developer Questions to Ponder


Some Big LA Questions: Research & Ethics

Josh Baron, Marist College


Data Mining vs. Learning Science Approaches◦ Do we build predictive models from large data sets or

from our understanding of learning sciences?◦ Is both the right answer? How does that work?

Challenges of Scaling LA Across Higher Ed◦ Does each institution have to build its own model?

How “portable” are predictive models?

◦ Do we need an open standard for LA? Could LIS and LTI play a role?

How can LA be used to assist ALL students?◦ Michigan’s E2Coach system is a good example

Big LA Research Questions


“The obligation of knowing” – John Campbell◦ If we have the data and tools to improve student

success, are we obligated to use them? Consider This > If a student has a 13% chance of

passing a course, should they be dropped? 3%? Who owns the data, the student?

Institution?◦ Should students be allowed to “opt out”?

Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?

What do we reveal to students? Instructors?

Consider This > If we tell a student in week three they have a 9% chance of passing, what will they do? Will instructors begin to “profile” students?

Big LA Ethical Questions


Resources & Conference Sessions

Connect with Learning Analytics communities


http://www.solaresearch.org/

Learning Analytics & Knowledge Conferences (LAK)

STORM – initiative to help fund research projects

FLARE – regional practitioner conference ◦ Purdue University, Oct 1-3, 2012




Symposium on Learning Analytics at Michigan

http://sitemaker.umich.edu/slam/

15 speakers (12 UM, 3 external)

Videos & slides available from all speakers

http://sitemaker.umich.edu/slam/


Analytics in Higher Education: Establishing a Common Language◦ Van Barneveld, Arnold, Campbell, 2012◦ http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405

Analytics to Literacies: Emergent Learning Analytics to evaluate new literacies◦ Dawson, 2011- http://blogs.ubc.ca/newliteracies/files/2011/12/Dawson.pdf

Learning Analytics: Definitions, Process Potential◦ Elias, 2011◦ http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

The State of Learning Analytics in 2012: A Review and Future Challenges◦ Ferguson, 2012 - http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf

Academic analytics: A new tool for a new era. ◦ Campbell, Deblois, & Oblinger (2007). Educause Review, 42(4), 40-57. ◦ http://net.educause.edu/ir/library/pdf/ERM0742.pdf

Mining LMS data to develop an "early warning system" for educators: A proof of concept. ◦ Macfadyen & Dawson (2010) - Computers & Education, 54(2), 588-599.

Classroom walls that talk: Using online course activity data of successful students to raise self- awareness of underperforming peers. ◦ Fritz, 2011 - Internet and Higher Education, 14(2), 89-97.

Publications

http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405

http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405

http://blogs.ubc.ca/newliteracies/files/2011/12/Dawson.pdf



http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf

http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf

http://net.educause.edu/ir/library/pdf/ERM0742.pdf



50

Wednesday, 13 June◦ Learning Analytics: A Panel Debate on the Merits,

Methodologies, and Related Issues (1:15pm)

◦ Learning Analytics at Michigan: Designing Displays for Advisors, Instructors, and Students (2:30pm)

◦ BOF for Learning Analytics: Current and Planned Projects and Tools (3:45pm)

Thursday, 14 June◦ Creating an Open Ecosystem for Learner Analytics

(10:15am) Open Academic Analytics Initiative (OAAI) https://confluence.sakaiproject.org/x/8aWCB


Related Conference Sessions

https://confluence.sakaiproject.org/x/8aWCB

https://confluence.sakaiproject.org/x/8aWCB


Steve Lonn◦ [email protected] @stevelonn

Josh Baron◦ [email protected] @joshbaron

Questions?

mailto:[email protected]

mailto:[email protected]

learning analytics 101

Education

data available

sakai cle oae4

events visual data analytics

analytics la

predictive analytics

educational practicebienkowski

business academic analytics

supporthuman decision