April 13, 2023 1

Open Academic Analytics Initiative (OAAI)Josh Baron

Sandeep Jayaprakash

April 13, 2023 OAAI 2

PROJECT OVERVIEW

Open Academic Analytics Initiative

• EDUCAUSE Next Generation Learning Challenges (NGLC)

• Funded by Bill and Melinda Gates Foundations

• $250,000 over a 15 month period• Goal: Leverage Big Data concepts to

create an open-source academic early alert system.

Student Attitude Data (SATs, current GPA, etc.)

Student Demographic Data (Age, gender, etc.)

Sakai Event Log Data

Sakai Gradebook Data

Predictive ModelScoring

Identifies students “at risk” to not complete

course

SIS

Dat

aLM

S D

ata

OAAI Early Alert System Overview

Intervention Deployed“Awareness” or Online

Academic Support Environment (OASE)

“Creating an Open Academic Early Alert System”

Academic Alert Report (AAR)

Model DevelopedUsing Historical Data

Online academic support environment (OASE)

• OER Content• Self-Assessments• Learning Skills -

Flat World Knowledge

• Learning Support Facilitation & Mentoring

OAAI Goals and milestones

• Build “open ecosystem” for Learning analytics– Sakai Collaboration and Learning Environment

• Secure data capture process for extracting LMS data

– Pentaho Business Intelligence Suite• Open-source data mining, integration, analysis and

reporting tools

– OAAI Predictive Model released under open license• Predictive Modeling Markup Language (PMML)

• Researching learning analytics scaling factors – How “portable” are predictive models?– What intervention strategies are most effective?

April 13, 2023 OAAI 7

DEVELOPING AND DEPLOYING OAAI

OAAI Architecture

Predictive Modeling using Marist Data

Pentaho Kettle Data Integration• Training Dataset – Marist Fall 2010 & Spring

2011 (7344 records) Testing Dataset – Marist Fall 2011 (5101 records )

• Extractions were joined, cleaned, recoded, and powerful predictors were derived to produce an input data file for each student- course combination.

April 13, 2023 10

Prediction Metrics

Predictive Modeling using Marist Data

Pentaho WEKA 3.7 and IBM SPSS Modeler 14.2• Generate 10 different training datasets by varying random

seeds• Balance each training dataset using sampling techniques. • Train a predictive model(Logistic Regression, SVM/SMO, J48

decision Trees) for each balanced training dataset 10 datasets x 3 algorithms = 30 models

• Score the testing dataset(Marist Fall 2011) for each student-course combination

• Measure predictive performance of classifiers Accuracy, Recall, Specificity and Precision.

• Produce summary measures (mean and standard error)

Predictive Modeling using Marist Data

Predictive Performance on Marist Data

AAR Project Site

Faculty Folder

Dropbox Tool

Academic Alert Report

(AAR)

Student Identification

Key (SIK)

Gradebook Data Extract

Pentaho[data processing,

scoring and reporting]Academic

Alert Report (AAR)

Specific Sakai Course Site

Messages ToolIdentified Student

Online Academic Support Environment

(OASE)

AAR transferred from Marist into a Project Site for faculty at each institutions Sakai system

Faculty message ͞9identified͞:

students through the class Course Site

Open Academic Analytic Initiative Workflow for Academic Alert

Reports (AAR) and deployment of intervention strategies

/͞Awareness Intervention 1͞

Student Aptitude and Demographic Data Extract (SIS)

Sakai Event Log Data Extract

A sub-folder for each course/section used to organize the

AAR and course SIK

Faculty notified when new AA is posted

and access their Dropbox to review AAR

The Sakai Dropbox tool

is used to provide each

faculty with a private folder

Running Pilots at Partner Institutions

Academic Alert Reports (AARs)

April 13, 2023 OAAI 16

RESEARCH FINDINGS: PORTABILITY

Predictive Performance in partner institutions

April 13, 2023 18

Portability Analysis

• The models developed at one academic context are scalable to other academic contexts.

• The evaluation accuracies start at 65 % at the first wave and the accuracies improves to 75% - 80% with more availability of data in the subsequent waves.

• Pilot Evaluation results show that recall and specificity completion values are just around 10% lower when compared to Marist results.

• Gradebook (CMS data) and CUM_GPA have been very important predictors. Followed by LMS metrics & SAT scores

• Evidence of good portability in institutions collecting such data.

April 13, 2023 20

RESEARCH FINDINGS: INTERVENTIONS

April 13, 2023 21

• Analysis showed a statistically significant positive impact on final course grades– No difference

between treatment groups

• Saw larger impact in spring then fall

• Similar trend amount low income students

Intervention Research Findings: Final Course Grades

Awareness OASE Control50

60

70

80

90

100

Fin

al G

rade

(%

)

Mean Final Grade for "at Risk" Students

April 13, 2023 22

• Student in intervention groups were statistically more likely to “master the content” then those in controls.– Content Mastery =

Grade of C or better

• Similar for low income students.

Intervention Research Findings Content Mastery

Yes No Yes No0

200

400

600

800

1000

Content Mastery for "at Risk" Students

Control Intervention

Fre

quen

cy

April 13, 2023 23

• Students in intervention groups withdrew more frequently than controls

• Possibly due to students avoiding withdrawal penalties.

• Consistent with findings from Purdue University

Intervention Research Findings Withdrawals

Yes No Yes No0

200

400

600

800

1000

Withdrawal rates for "at Risk" Students

Control Intervention

Freq

uenc

y

Instructor Feedback

"Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills.

P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention!

Future Research Interests

• Factors that impact on intervention effectiveness– Intervention Immunity – Students who do not

respond to first intervention tend to never respond

– Student Engagement – How can we increase the level of engagement between students and help resources?

• Can predictive models be customized for specific delivery methods and programs/subjects?

• Can Learning Analytics identify “at risk” students who would otherwise not be identified?

Questions

ReferenceOAAI Sakai confluence Wiki pagehttps://confluence.sakaiproject.org/pages/viewpage.action?pageId=75671025

ContactJosh Baron - Senior Academic Technology Officer

josh.baron@Marist.edu

Sandeep Jayaprakash - Learning Analytics Specialistsandeep.jayaprakash1@Marist.edu