parameterized exercises in java programming: using knowledge structure for performance prediction

Parameterized Exercises in Java Programming: Using Knowledge

Structure for Performance Prediction

Shaghayegh Sahebi (Sherry)1, Yun Huang1, and Peter Brusilovsky1,2

1 Intelligent Systems Program, University of Pittsburgh2 School of Information Sciences, University of Pittsburgh

Parameterized Exercises in Java Programming 2Shaghayegh Sahebi (Sherry)

QuizJET

• Programming questions– Java problems

• Can be designed with parameterized exercises– One question with multiple parameter sets– Can be repeated multiple times by one student

• Authoring tool for Java questions– Create and modify questions– Indexing service to define concepts inside the question


QuizJET: Sample Question

Each question is generated from a template, and students can try multiple attempts.

Students give values for specified variable, or give the output of the code.

A question for practicing skill nested loops


QuizJET: Sample Parameterized Question


Some Stats


Aggregate (MasteryGrids

Services)

Aggregate

UM2

Other (content specific)

PAWSUM Services

Content apps

Server side apps(Apache Tomcat)

Databases (MySQL)

Client interfaceMasteryGrids Interface

Content popout iframe

QuizJet

WebEx

SQLKnot

(a)GUI calls MG

services

direct link

services calls

(b)Aggregate uses

UM services

cbum login

Overall view of the architectureMastery Grids


Mastery Grids

• Students can choose what question to solve– Using social navigation support

• Adding guidance to the question – Use the whole set of data to develop personalized

guidance– Predict how likely the problem will be solved– Avoid too simple and too complex problems


Predicting Students’ Performance (PSP) in Parameterized Questions

• Predicting the student’s capability to perform an educational task

• Assumption: the student can learn by practicing over time by repeating– Time sequence modeling effect on PSP • Will present at the Problem Solving & Strategies session

on Monday

– Knowledge structure effect on PSP• Today’s talk


Traditional PSP: Ignoring Multidimensionality of the Data

• Questions related to topics, concepts, or skills – many dimensions in the data– Structure in the data (knowledge structure)

• Traditional methods: mostly consider student’s past performance– Only consider correct/incorrect attempts of students

(ignoring the multidimensionality of the data) – Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)


New Approaches to PSP: Including Multidimensionality of Data

• Considering knowledge structure in PSP– Feature-Aware Knowledge Tracing (FAST) [González-Brenes

et al., ‘13]

– Our suggestion: Tensor Factorization Methods


Our Goal

• Study the effect of knowledge structure modeling in PSP for parameterized questions

• Compare five approaches:– Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)– Feature-Aware Knowledge Tracing (FAST)– 3D and 4D Tensor Factorization (3D-BPTF, 4D-

BPTF)


BKT

• Markov Model with two states

• No knowledge structure: Only one type of knowledge component

• Guess, slip, learning, and initial knowledge parameters

Knowledge Structure


PFA

• Regression model

• No knowledge structure

Knowledge Structure


• Extension of BKT• Can include knowledge structure as regression

variables

Knowledge Structure

FAST


Tensor Factorization

• Tensors: n-dimensional arrays

• Used in collaborative-filtering recommender systems– Estimates each tensor as the sum of multiple rank-1

tensors

• Can be extended to as many dimensions – Can include the data structure– Each dimension of the data ≈ one dimension of the

tensor


3D-Tensor Factorization (3D-BPTF)

• Used successfully in PSP for traditional PSP• No knowledge structure• We use Bayesian Probabilistic Tensor

Factorization Model (3D-BPTF) [Xiong et al., 2010]

Stud

ents

Time

Questions/ topics

…

Knowledge Structure


4D-Tensor Factorization (4D-BPTF)

• Used for the first time in PSP• Adds knowledge structure modeling• Can be extended to more dimensions if

needed

Stud

ents

Topics

QuestionsSt

uden

tsTo

pics

Questions

Stud

ents

Topics

Questions

Attempt 1 Attempt 2 Attempt 3

Knowledge Structure


Data

• From QuizJET system• Java programming questions• Six semesters• 166 students• 103 questions• 69.04% majority class (successes)


Study Setup

• Here, each topic can have multiple questions and each question is related to one topic– Two dimensions: questions and topics

• Study 1: traditional approach – Question as knowledge unit

• Study 2: considering knowledge structure– Topic added as knowledge unit

• 5-Fold cross validation– 80% of students in train data, rest in test data– User-stratified


Study 1: Question as knowledge unit

Study 1: comparing traditional approaches(no knowledge structure)


Accuracy of all models: very close to each other

FAST with no additional

parameters

BKT PFA 3D-BPTF73

73.2

73.4

73.6

73.8

74

74.2

74.4

74.6

74.8

Accuracy of Traditional Models



BKT overestimates student’s performance


parameters

BKT PFA 3D-BPTF0

200

400

600

800

1000

1200

1400

False Positive



FAST underestimates student’s performance


parameters

BKT PFA 3D-BPTF0

100

200

300

400

500

600

700

800

False Negative



FAST Predicts Success Better

FAST with no addi-

tional pa-rameters

BKT PFA 3D-BPTF0

10

20

30

40

50

60

70

80

90

Majority PrecisionMinority Precision

BKT predicts failure better



Study 2: Topic as additional knowledge unit

Study 2: comparing approaches including knowledge structure


FAST and 4D-BPTF more accurate than other approaches

FAST 4D-BPTF BKT PFA 3D-BPTF66

67

68

69

70

71

72

73

74

75

76

Accuracy of Approaches with Additional Knowledge Structure



3D-BPTF overestimates students’ performance


200

400

600

800

1000

1200

1400

1600

1800

False Positive



4D-BPTF underestimates students’ performance


100

200

300

400

500

600

False Negative



4D-BPTF and FAST Predict Success Better


10

20

30

40

50

60

70

80

90

100

Majority PrecisionMinority Precision


3D-BPTF predicts failure better


FAST and BPTF improved using knowledge structure


67

68

69

70

71

72

73

74

75

76

Question as KC (No Structure)Topic as KC (with Question Structure)

3D-BPTF

Accu

racy


Traditional models declined with topic as KC


67

68

69

70

71

72

73

74

75

76

Question as KC (No Structure)Topic as KC (with Question Structure)Ac

cura

cy


Conclusions

• Accuracy in predicting students performance depends on the input of the method– When ignoring the topic of questions as KCs, all

models perform similarly– When including topic information, in addition to

the question information, the methods that can leverage it perform better


Conclusions

• Adding the extra topic data in the methods that cannot model this information decreases the method’s accuracy

• Knowledge structure can add to the accuracy of PSP in parameterized questions

• Tensor factorization methods are as good, or better than the pioneers PSP methods


Future Work

• Include additional structure into tensor factorization using more dimensions

• Use of other collaborative filtering methods for PSP

• Test on other programming courses


Thank You!


Implementation

• EM algorithm for BKT and set the initial parameters as follows: p(L0) = 0:5 , p(G) = 0:2 , p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use

• the implementation of logistic regression in WEKA [3]. For BPTF and BPMF,

• we utilize the Matlab code prepared by Xiong et. al. We experimented with different latent space dimensions for BPTF and BPMF (5, 10, 20 and 30) and chose the best one, which has the latent space dimension of 10


Matrix Factorization (BPMF)

• From collaborative filtering • Used successfully in PSP for static questions• No attempt sequence modeling• We use Bayesian Probabilistic Matrix

Factorization (BPMF)

1 0 0 01 1 0 10 0 1 10 0 0 1St

uden

ts

Questions/ topics

0.9

0

1.5

0.4

0 1.4

0 0.9

Stud

ents

KCs

0.8

0.5

0 0.3

0 0 0.5

0.8

KCs

Questions/ topics


Data Stats



Accuracy of all models is very close to each other



BKT over estimates the student’s performance



FAST tends to predict more failures for the students



If FAST predicts a success for a student and if BKT predicts a failure for students, their prediction is more likely to be true compared to the other methods.


Study 2: Topic as knowledge unit

FAST and 4D-BPTF perform significantly better than all other approaches


Study 2: Topic as knowledge unit

BKT and PFA perform similarly to their results in Study 1 and 3D-BPTF on topics is slightly weaker than 3D-BPTF on questions in terms of accuracy.


Mastery Grids

• Visual, interactive, adaptive E-learning platform– Multi-facet social comparison– Multi-type learning materials support– Social navigation– Personalized guidance

• Integration with other systems with little set up and modification

parameterized exercises in java programming: using knowledge structure for performance prediction

Data & Analytics

shaghayegh sahebi sherry

knowledge structure

e knowledge structure

knowledge structure

j jkcs knowledge structure

data structure

attempts of students

t timesteps t