learning cognitive models...geoff gordon—pslc summer school—apr 2010 it’s not easy student...
TRANSCRIPT
![Page 1: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/1.jpg)
LEARNING COGNITIVE
MODELS
Geoff Gordon
Joint work w/ Ajit Singh, Hao Cen, John Stamper, Ken Koedinger
![Page 2: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/2.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
COGNITIVE MODEL
3(2X – 5) = 9
6X – 15 = 9 2X – 5 = 3 6X – 5 = 9
IF GOAL IS SOLVE A(BX+C) = DTHEN REWRITE AS ABX + AC = D
IF GOAL IS SOLVE A(BX+C) = DTHEN REWRITE AS ABX + C = D
IF GOAL IS SOLVE A(BX+C) = DTHEN REWRITE AS BX+C = D/A
Example benefit of cognitive model: Model Tracingfollow students through individual approaches to problem ⇒ context-sensitive instruction
SKILLS BUG
Solves or simulates solving problems in the many ways students can
2
![Page 3: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/3.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
COGNITIVE MODEL
STUDENTHAS:
KC1: 80%KC2: 72%KC3: 11%KC4: 34%…
STEPREQUIRES:
KC1: 2.32KC2: 0KC3: 1.07KC4: 0…
E.g., James L. E.g., find the area of region B
3
Solves or simulates solving problems in the many ways students can
RESULTE.G.:‣ right answer‣ forgot to carry‣ needed 2 hints…
![Page 4: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/4.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
COGNITIVE MODEL
STUDENTHAS:
KC1: 80%KC2: 72%KC3: 11%KC4: 34%…
STEPREQUIRES:
KC1: 2.32KC2: 0KC3: 1.07KC4: 0…
E.g., James L. E.g., find the area of region B
CAUSE
3
Solves or simulates solving problems in the many ways students can
RESULTE.G.:‣ right answer‣ forgot to carry‣ needed 2 hints…
![Page 5: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/5.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
COGNITIVE MODEL
STUDENTHAS:
KC1: 80%KC2: 72%KC3: 11%KC4: 34%…
STEPREQUIRES:
KC1: 2.32KC2: 0KC3: 1.07KC4: 0…
E.g., James L. E.g., find the area of region B
CAUSE
LEARNS
KC1: 87%
3
Solves or simulates solving problems in the many ways students can
RESULTE.G.:‣ right answer‣ forgot to carry‣ needed 2 hints…
![Page 6: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/6.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
SKILLS × ITEMS: THE Q MATRIX
4
Representing Knowledge Components
as factors of items
! Problem: How to represent KC model?
! Solution: Q-Matrix (Tatsuoka, 1983)Items X Knowledge Components (KCs)
" Single KC item = when a row has one 1! 2*8 above
" Multi-KC item = when a row has many 1’s! 2*8 – 3
Item | Skills: Add Sub Mul Div
2*8 0 0 1 0
2*8 - 3 0 1 1 0
What good is a Q matrix? Can predict
student accuracy on items not previously
seen, based on KCs involved
Qkj = does step k need skill j?
Simplest representation of a cognitive model
Fancier: e.g., skills ⟷ rules in a production system
![Page 7: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/7.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
GETTING THE MODEL RIGHT!
Cognitive model determines instruction
Through instructional decisions like problem selection, hints, …
A correct model is one that is consistent with student behavior, predicting task difficulty and transfer between instruction and test
Cognitive models are discovered not designed
![Page 8: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/8.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
GETTING THE MODEL RIGHT!
Cognitive model determines instruction
Through instructional decisions like problem selection, hints, …
A correct model is one that is consistent with student behavior, predicting task difficulty and transfer between instruction and test
Cognitive models are discovered not designed
should be
⇒ Huge data mining opportunity
![Page 9: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/9.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
IT’S NOT EASY
Student models are a key bottleneck in cognitive tutor authoring and performance
~80 hrs (and up!) to hand-develop model for 1 hr content
result may be too simple, not rigorously verified
Machine learning, computational modeling, and data mining can help us discover detailed, accurate models of how students learn
New models; data-driven revision of existing models
6
![Page 10: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/10.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
BUT IT’S WORTH IT
We have demonstrated improvements in learning from these more accurate models
E.G., Salden et al [2009]: adapting #examples vs. #problems yielded better transfer, same time spent
E.g., Cen et al [2007]:12% less time to learn 6 geometry units (same retention) using tutor w/ better model
Results can transfer beyond PSLC
7
![Page 11: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/11.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICAL METHODS FOR LEARNING COGNITIVE MODELS
Data → learn a better model:
improved parameters
refine list of skills (e.g., split a KC)
discover completely new skills
8
more accurate sequencing
better coverage
aid problem design
⇒⇒⇒
STUDENT
HAS:KC1: 80%KC2: 72%KC3: 11%KC4: 34%…
STEP
REQUIRES:KC1: 2.32KC2: 0KC3: 1.07KC4: 0…
E.g., James L. E.g., find the area of region B
CAUSE
LEARNS
KC1: 87%
RESULT
E.G.:!right answer! forgot to carry!needed 2 hints…
![Page 12: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/12.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RAW DATA
Student ID Step ID Correct? Skills
1 1 0 DECLARE_PARAM
1 2 1 WHILE_LOOP
1 3 1 DECLARE_PARAM
2 1 0 DECLARE_PARAM
2 4 1 PREFIX_OP
… … … …
9
![Page 13: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/13.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RAW DATA
Student ID Step ID Correct? Skills
1 1 0 DECLARE_PARAM
1 2 1 WHILE_LOOP
1 3 1 DECLARE_PARAM
2 1 0 DECLARE_PARAM
2 4 1 PREFIX_OP
… … … …AUTOMATICALLY
RECORDEDFROM Q MATRIX
9
![Page 14: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/14.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RAW DATA
Student ID Step ID Correct? Skills
1 1 0 DECLARE_PARAM
1 2 1 WHILE_LOOP
1 3 1 DECLARE_PARAM
2 1 0 DECLARE_PARAM
2 4 1 PREFIX_OP
… … … …AUTOMATICALLY
RECORDEDFROM Q MATRIX
Opp
1
1
2
1
1
COMPUTED
9
![Page 15: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/15.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RAW DATA
Student ID Step ID Correct? Skills
1 1 0 DECLARE_PARAM
1 2 1 WHILE_LOOP
1 3 1 DECLARE_PARAM
2 1 0 DECLARE_PARAM
2 4 1 PREFIX_OP
… … … …AUTOMATICALLY
RECORDEDFROM Q MATRIX
Opp
1
1
2
1
1
COMPUTED
9
E.g., KDD Cup data: 1,000s of students, 1,000,000s of steps
![Page 16: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/16.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICAL METHODS FOR LEARNING COGNITIVE MODELS
Model P(correct | features of student and step)
10
P (Yi = 1 | Xi1, Xi2, . . .)i = 1, 2, . . . , N
Yi ! {0, 1}Xij ! R ← may be arbitrary real, but often binary
![Page 17: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/17.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
INDICATOR FEATURES
11
Student ID Step ID Correct? Skills
1 1 0 DECLARE_PARAM
1 2 1 WHILE_LOOP
1 3 1 DECLARE_PARAM
2 1 0 DECLARE_PARAM
2 4 1 PREFIX_OP
Yi
0
1
1
0
1
Xi1 Xi2 Xi3 Xi4 Xi5 Xi6 Xi7 Xi8 Xi9
1 0 1 0 0 0 1 0 0
1 0 0 1 0 0 0 1 0
1 0 0 0 1 0 1 0 0
0 1 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 1
i
1
2
3
4
5Student ID Step ID Skill
i
1
2
3
4
5
![Page 18: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/18.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICS OF COGNITIVE MODELS: LOGISTIC REGRESSION
Write pi = P(Yi = 1 | features of student and step for example i)
12
!! " !"
"#$
"#%
"#&
"#'
"#!
"#(
"#)
"#*
"#+
$
,-
.-
logpi
1! pi= !0 +
d!
j=1
!jXij " zi
!j # R
Logistic regression
model
![Page 19: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/19.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICS OF COGNITIVE MODELS: ADDITIVE FACTOR MODEL
Additive Factor Model (Draney et al., 1995)
Logistic regression for P(correct answer | student & skill info)
13
i: student j: step k: skill
![Page 20: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/20.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICS OF COGNITIVE MODELS: ADDITIVE FACTOR MODEL
Additive Factor Model (Draney et al., 1995)
Logistic regression for P(correct answer | student & skill info)
CORRECT?
OPP COUNT
STEP J USES SKILL K
13
i: student j: step k: skill
![Page 21: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/21.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STATISTICS OF COGNITIVE MODELS: ADDITIVE FACTOR MODEL
Additive Factor Model (Draney et al., 1995)
Logistic regression for P(correct answer | student & skill info)
CORRECT?
OPP COUNT
STEP J USES SKILL K
PARAMETERS
θ = student mean–β = skill initial difficulty γ = skill learning rate
13
i: student j: step k: skill
![Page 22: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/22.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
VISIBLE MANIFESTATION: THE LEARNING CURVE
Good models ⇒ smooth, decreasing curves
Good models ⇒ accurate predictions
14
0 5 10 15 20 250
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Opportunity Count
Assis
tance S
core
(H
ints
+ E
rrors
)
observed
predicted
![Page 23: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/23.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
USING LEARNING CURVES TO EVALUATE A COGNITIVE MODEL
15
“Good” learning curveModel appears to be
predicting well
Problematic learning curveModel fails to predict
performance
Corbett, Anderson, O’Brien (1995)
141210864200.0
0.1
0.2
0.3
0.4
0.5
Opportunity to Apply Rule (Required Exercises)
Err
or R
ate
![Page 24: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/24.jpg)
Geoff Gordon—Latent Factor Models—July, 2010
Modify cognitive model Blips occur when a new, unmodeled latent skill appears Split skill into two new skills
With new model, tutor can treat these skills separately
![Page 25: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/25.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AUTOMATED DETECTION OF “BLIPS” IN LEARNING CURVES
We identified a latent factor by manually examining learning curves
Problem:
Requires lots of up-front time from expert to define skills
Can potentially discover automatically that skills are wrong, but can’t fix automatically
Can we automate the process of finding latent factors?
increase repeatability, reduce bias, reduce human expert time
will still need human judgement to connect the identified latents to properties of the problems
17
![Page 26: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/26.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 27: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/27.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 28: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/28.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 29: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/29.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 30: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/30.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 31: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/31.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
AFM RESULTS
“Original”: proposed by domain experts
“Unique step,” “Geometry”: maximal or minimal splitting
“Unique step” yields IRT model
“Textbook”: discovered by an automated model-search technique
“DecompArithDiam”: we discovered by manual search (best)
made possible by visualization and analysis tools in DataShop
18
KC Models KCs BIC RMSEDecompArithDiam 13 5613 0.401Textbook 10 5678 0.405Original 15 5762 0.409Geometry 1 6039 0.427Unique_step 132 7182 0.403
![Page 32: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/32.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
CAN WE GENERALIZE AFM?
i,j,k = student, item, skill
p = probability correct
θ = student overall performance
β = skill easiness / difficulty
Q = item × skill matrix
γ = skill practice slope
T = number of practice opportunities
Tik γkΘ
β0
Q
1
×
AFM:
19
![Page 33: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/33.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
CAN WE GENERALIZE AFM?
i,j,k = student, item, skill
p = probability correct
θ = student overall performance
β = skill easiness / difficulty
Q = item × skill matrix
γ = skill practice slope
T = number of practice opportunities
Tik γkΘ
β0
Q
1
×
AFM:
19
Relax
con
stra
ints
on
U, V
UV
![Page 34: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/34.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
HIERARCHICAL LOGISTIC PRINCIPAL COMPONENTS ANALYSIS
Y
V
U
U: student latent factorsV: item latent factorsY: observed performanceR: shared prior for student latentsS: shared prior for item latents
n students
m itemsk latent factors
k latent factors
observed unobserved
R
S
student factoritem factor
log!
pij
1! pij
"=
#
k
UikVjk
20
![Page 35: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/35.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
PCA IS A WIDELY USED AND SUCCESSFUL MODEL
Y1
Y2
Y3
.
.
.YN
USE
RS
MOVIES
21
![Page 36: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/36.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
PCA IS A WIDELY USED AND SUCCESSFUL MODEL
Y1
Y2
Y3
.
.
.YN
USE
RS
MOVIES
EACH ENTRY: HOW MANY STARS DOES USER I GIVE TO MOVIE J?
4
21
![Page 37: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/37.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RESULT OF FACTORING
U1
U2
U3
.
.
.UN
V1 … VK
USE
RS
MOVIESBASIS WEIGHTS
BA
SIS
VEC
TOR
S
Low-d basis = latent variables
Basis vectors represent latent properties of movies, e.g., “is a comedy”
22
![Page 38: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/38.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
Y1
Y2
Y3
.
.
.YN
IN OUR CASE: STUDENT-ITEM DATAST
UD
ENT
S
ITEMS IN TUTOR
23
![Page 39: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/39.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
Y1
Y2
Y3
.
.
.YN
IN OUR CASE: STUDENT-ITEM DATAST
UD
ENT
S
ITEMS IN TUTOR
EACH ENTRY: DOES STUDENT I GET ITEM J RIGHT?
1
23
![Page 40: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/40.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RESULT OF FACTORING
U1
U2
U3
.
.
.UN
V1 … VK
STU
DEN
TS
ITEMSBASIS WEIGHTS
BA
SIS
VEC
TOR
S
Basis vectors are candidate “eigenskills”
Weights are students’ knowledge levels
24
![Page 41: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/41.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
PCA VS LOGISTIC PCA
Ordinary PCA: linear, Gaussian
Logistic PCA: can handle conjunctive, disjunctive skills
P(C
OR
REC
T)
SKILL 1 SKILL 2
NONLINEARITY: CONJUNCTIVE SKILLS
25
![Page 42: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/42.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
BAYESIAN INFERENCE
Ordinary PCA yields maximum-likelihood estimate
Good, right?
sadly, the usual reasons to want the MLE don’t apply here
e.g., consistency: variance and bias of estimates of U and V do not approach 0 (unless #items/student and #students/item → ∞)
Result: MLE is typically far too confident of itself
26
![Page 43: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/43.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
TOO CERTAIN: EXAMPLE
!"# !"$ !"% & &"' &"# &"$ &"%!%
!(
!$
!)
!#
!*
!'
! " # $ %!&'(
&
&'(
&'"
&'$
&')
*
*'(
Learned coefficients (e.g., a column of V)
Predictions
27
![Page 44: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/44.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RESULT: “FOLD-IN PROBLEM”
Nonsensical results when trying to apply learned model to a new student or item
Similar to overfitting problem in supervised learning: confident-but-wrong parameters do not generalize to new examples
Unlike overfitting, fold-in problem doesn’t necessarily go away with more data
28
![Page 45: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/45.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
EXPERIMENTAL COMPARISONGEOMETRY AREA 1996-1997 DATA
Geometry tutor: 139 items presented to 59 students
On average, each student tested on 60 items
0 = WRONG1 = RIGHT
29
![Page 46: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/46.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
RESULTS: HOLD-OUT ERROR
Embedding dimension is k = 15, except PCA+AFM where k = 1
Credit for logistic PCA:
Ajit Singh0.28
0.3
0.32
0.34
0.36
0.38
0.4
0.42
RM
SE
Bayes logistic PCA
Bayes PCA
AFM: textbook
Baseline: mean
30
Non-Bayesian PCA/LPCA performs at about chance level in similar
problems
![Page 47: LEARNING COGNITIVE MODELS...Geoff Gordon—PSLC summer school—Apr 2010 IT’S NOT EASY Student models are a key bottleneck in cognitive tutor authoring and performance ~80 hrs (and](https://reader033.vdocuments.site/reader033/viewer/2022050401/5f7f04fc12485b013f6da1df/html5/thumbnails/47.jpg)
Geoff Gordon—PSLC summer school—Apr 2010
STILL MISSING
A way to include time in PCA
A way to encourage interpretable latent models
A way to take advantage of partial prior knowledge of model
31