CS539: Project 3

Zach Pardos

Assistments Online Dataset

Math question response data from 592 students.

1,143 math question attributes {correct, incorrect}

Average of 200 questions answered per student (lots of missing values)

Class: MCAS SCORE {0-29}

Assistments Online Dataset

Skill models: 1, 5, 39, 106

Assistments Online Dataset

How well can ANNs fit the dataset with only 1, 5, 39 or 106 hidden nodes?• Default Weka values used for ANN training

• Epochs: 500

• Learning: 0.3

• Momentum: 0.2

• No validation set

• Training-set for testing

Assistments Online Dataset

Results for training-set testing:• With 1 Hidden Node:

• Correctly Classified Instances 77

• Incorrectly Classified Instances 515

• Relative absolute error 95.5309 %

• With 5 Hidden Nodes:• Correctly Classified Instances 220

• Incorrectly Classified Instances 372

• Relative absolute error 77.8246 %

Assistments Online Dataset

Results for training-set testing:• With 39 Hidden Nodes:

• Correctly Classified Instances 590

• Incorrectly Classified Instances 2

• Relative absolute error 3.2983 %

• With 106 Hidden Nodes:• Correctly Classified Instances 587

• Incorrectly Classified Instances 5

• Relative absolute error 2.8975 %

Assistment Online Dataset

Conclusion: 39 and 106 models predict very well.

How well can ANNs generalize and predict instances they haven’t trained on?

Next up: 10-fold cross validation

Assistment Online Dataset

0102030405060708090

100

absolute error %

1 5 39 106

hidden nodes

training-set

10-fold validation

Assistment Online Dataset

Conclusions:• ANNs very good at fitting data

• Not as good at predicting unseen cases

• Possible that more nodes are required to properly generalize (more CPU!)