Detecting Diligence with Online Behaviors on Intelligent Tutoring Systems
Steven DangMichael YudelsonKenneth R. Koedinger
Novice –> Expertise
2
Importance of Diligence
Ideal Reality
3
Research QuestionCan we develop a model for measuring diligence at scale using only naturalistic online behaviors?
4
The Model
Defining DiligenceWorking on academic tasks which are beneficial in the long-run but tedious in the moment, especially in comparison to more enjoyable, less effortful activities
6
Academic Diligence Task (ADT)
7
DoMath
PlayGamesor
WatchVideos
Academic Diligence Task (ADT)
8
PlayGamesor
WatchVideos
Academic Diligence Task (ADT)
9
DoMath
Personality to Behavior to Learning Outcomes
ITS
Productivity#ofproblems
Time-on-taskTimeworkingmathproblems
ADTConscientious/Self-Control
MathGrade
StandardizedTestScore
On-timeGraduation
4-yearCollegeEnrollment
DiligenceMeasures
Personality
Outcomes
10
Academic Diligence Task (ADT)
Ydiligence = f(Xtot, Xprod)
Ydiligence – Diligence Xtot – Total time workingXprod – Total problems completed
11
Diligence during online learning
12
PracticeTutor
Problems
Goofoffwith
Friends
Differences in the Task
13
AcademicDiligenceTask Adaptive OnlineLearning OldFeature
ProposedFeature
Differences in the Task
14
AcademicDiligenceTask Adaptive OnlineLearning OldFeature
ProposedFeature
timeworkedingiventimewindow
Studentsmaychoosetoworkmoreorlesstime Xtot Xtot
Differences in the Task
15
AcademicDiligenceTask Adaptive OnlineLearning OldFeature
ProposedFeature
% timeworkedinconstanttimewindow
Studentsmaychoosetoworkmoreorlesstime Xtot Xtot
Problemdifficultyislowanduniform
Varyingproblemdifficultyandadaptivenumberofexercises
Xprod Xwr
Differences in the Task
16
AcademicDiligenceTask Adaptive OnlineLearning OldFeature
ProposedFeature
% timeworkedinconstanttimewindow
Studentsmaychoosetoworkmoreorlesstime Xtot Xtot
Problemdifficultyislowanduniform
Varyingproblemdifficultyandadaptivenumberofexercises
Xprod Xwr
Cognitiveprocessisprimarilyrecallofwellknowninformation
Cognitiveprocessesinvolvemakingsenseofnewinformationandintegratingwithpriorknowledge
- Xprior
A Naturalistic Diligence Model
Ydiligence = β0Xtot + β1Xwr + β2Xprior + ε
Ydiligence – Diligence Xtot – Total time in systemXwr – Work-rate (Steps completed / Total time in system)Xprior – Prior knowledge/abilityε – Residual measurement error
17
Calculating model weights
Ydiligence ~ Ygrade
Ygrade ~ β0Xtot + β1Xwr + β2Xprior + ε
1. Calculate behavior using all student data
2. Fit the model using student outcome measure
3. Approximate YDiligence as the fitted value
18
The Dataset
Carnegie Learning CT DataWestern PA Junior High School– 426 students– 7th-9th Grades in Pre-Algebra– Full year of log data (~4M transactions)
Demographic Measures– Gender, Ethnicity, Free/Reduced Lunch
(SES)– Grade from Prior year in Math, each quarter
of study year, and End of Year Grade
20
Carnegie Learning ITS Data• Survey Measures (collected in Sept.)– Effort Regulation – Achievement Goals
• Mastery Orientation• Performance Orientation• Performance Avoidance
– Theory of Intelligence – Self-Efficacy – Interest in Math
21
Results
Convergent/Divergent Validity
Convergent/Divergent Validity
SurveyMeasure Correlation(p-value)EffortRegulation
TheoryofIntelligence
MasteryOrientation
PerformanceOrientation
PerformanceAvoidance
MathInterest
Self-Efficacy
24
(*)– Significantcorrelation (-)– Non-significantcorrelation
Convergent/Divergent Validity
SurveyMeasure Correlation(p-value)EffortRegulation (*)
TheoryofIntelligence
MasteryOrientation
PerformanceOrientation
PerformanceAvoidance
MathInterest
Self-Efficacy
25
(*)– Significantcorrelation (-)– Non-significantcorrelation
Convergent/Divergent Validity
SurveyMeasure Correlation(p-value)EffortRegulation (*)
TheoryofIntelligence (-)
MasteryOrientation (*)
PerformanceOrientation (-)
PerformanceAvoidance (-)
MathInterest (*)
Self-Efficacy (*)
26
(*)– Significantcorrelation (-)– Non-significantcorrelation
Convergent/Divergent Validity
SurveyMeasure Correlation(p-value)EffortRegulation 0.337(<.001)***
TheoryofIntelligence 0.05(.596)
MasteryOrientation .284(.003)***
PerformanceOrientation 0.189(.051)
PerformanceAvoidance .06(.52)
MathInterest 0.25(.01)**
Self-Efficacy .258(.007)**
27
Predictive Validity MethodResults
Testing Predictive Validity
29
Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse
PriorGrade
Q1Grade
Q2Grade
Q3Grade
FinalGrade
Q4Grade
Q1Q2Q3Q4
XTot XWR XPrior YDil~
Testing Predictive Validity
Zunits = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + εZfinal_grade = β0Ydiligence + β1Xint + β2Xdemo + β3Xprior + ε
Zunits – End of year # of units completedZfinal_grade – End of year gradeYdiligence – Learned diligence parameter from modelXint – Interest in mathXdemo – Other Demographic variables (Sex, Race, SES)Xprior - Prior Year Math Grade
30
Predictive Validity (Q1Q2Q3Q4)
31
ParameterFinalGradeβ(p-value)
UnitsCompletedβ(p-value)
Diligence 1.89(<0.001)*** 1.64(<0.001)***
Predictive Validity (Q1Q2Q3Q4)
32
ParameterFinalGradeβ(p-value)
UnitsCompletedβ(p-value)
Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)**MathInterest 0.12(.073)
Predictive Validity (Q1Q2Q3Q4)
33
ParameterFinalGradeβ(p-value)
UnitsCompletedβ(p-value)
Diligence 1.89(<0.001)*** 1.64(<0.001)***PriorGrade 0.20(<0.01)** 0.07(.217)MathInterest 0.12(.073) 0.15(.017)*
Testing Predictive Validity
34
Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse
PriorGrade
Q1Grade
Q2Grade
Q3Grade
FinalGrade
Q4Grade
Q2Q3
XTot XWR XPrior YDil~
Testing Predictive Validity
35
Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse
PriorGrade
Q1Grade
Q2Grade
Q3Grade
FinalGrade
Q4Grade
Q2Q3
XDemo Xint XPrior YDil~
ZFG
ZUnit
Testing Predictive Validity
36
Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse
PriorGrade
Q1Grade
Q2Grade
Q3Grade
FinalGrade
Q4Grade
Q1
XTot XWR XPrior YDil~
Testing Predictive Validity
37
Q1TutorUse Q2TutorUse Q3TutorUse Q4TutorUse
PriorGrade
Q1Grade
Q2Grade
Q3Grade
FinalGrade
Q4Grade
XDemo Xint XPrior YDil~
ZFG
ZUnit
Q1
Predictive ValidityUnitsCompleted
Samples β(p-value)
Q1Q2
Q3
Q4
Q1Q2
Q2Q3
Q3Q4
Q1Q2Q3
Q2Q3Q4
Q1Q2Q3Q4 1.64(<0.001)***
38
Predictive ValidityUnitsCompleted
Samples β(p-value)
Q1 2.23(<.001) ***Q2 1.72(<.001) ***Q3 1.16(<.001) ***Q4 2.09(<.001) ***
Q1Q2 2.17(<.001) ***Q2Q3 2.09(<.001) ***Q3Q4 1.18(<.001) ***
Q1Q2Q3 2.36(<.001) ***Q2Q3Q4 2.16(<.001)***
Q1Q2Q3Q4 1.64(<0.001)***
39
Predictive ValidityEnd-of-YearGrade
Samples β(p-value)
Q1 1.85(<.001)***Q2 1.69(<.001)***Q3 1.08(<.001) ***Q4 2.52(<.001) ***
Q1Q2 1.93(<.001) ***Q2Q3 1.95(<.001) ***Q3Q4 1.11(<.001) ***
Q1Q2Q3 2.11(<.001) ***Q2Q3Q4 2.04(<.001) ***
Q1Q2Q3Q4 1.89(<0.001)***
40
Conclusion
LimitationsHow much data is needed?– Frequency of observations– Length of sample time window
Task generalizability– Other math systems
• Alternate prior knowledge measures (eg: bkttraces)
– Other activities (eg: writing, scientific inquiry, programming)
42
Future WorkCharacterize interaction between task characteristics and diligenceInvestigating the study patterns of more/less diligent students
43
44
Matt Bernacki, Carnegie Learning, Steve Ritter, Steve Fanscali, Carnegie Mellon University, Julian Ramos Rojas, Queenie Kravitz, Scott Hudson, David Klahr, Audrey Russo, Judith Tucker, Learnlab
Datashop, Institute of Education Sciences, Program for Interdisciplinary Education Research
Acknowledgements
45
Questions?
Predictive ValidityEnd-of-YearGrade UnitsCompleted
Samples β(p-value) R2 β(p-value) R2
Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43
Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41
Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44
Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39
Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52
Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56
Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55
Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63
Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62
Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70
46
Generalizability
End-of-YearGrade
Samples β(p-value) R2
Q1 .345(.17) 0.621Q2 0.303(.43) 0.612Q3 0.450(.31) 0.615Q4 0.283(.18) 0.612
Q1Q2 0.259(.16) 0.618Q2Q3 -0.618(.54) 0.612Q3Q4 0.206(.46) 0.616
Q1Q2Q3 0.200(.26) 0.615Q1Q2Q3Q4 0.177(.30) 0.619
Repeated predictive analysis with geometry tutor
47
Predictive ValidityEnd-of-YearGrade UnitsCompleted
Samples β(p-value) R2 β(p-value) R2
Q1Q2Q3Q4 1.89(<0.001)*** 0.63 1.64(<0.001)*** 0.70Q1Q2Q3 2.11(<.001) *** 0.59 2.36(<.001) *** 0.63Q2Q3Q4 2.04(<.001) *** 0.62 2.16(<.001)*** 0.62Q1Q2 1.93(<.001) *** 0.50 2.17(<.001) *** 0.52Q2Q3 1.95(<.001) *** 0.56 2.09(<.001) *** 0.56Q3Q4 1.11(<.001) *** 0.56 1.18(<.001) *** 0.55Q1 1.85(<.001)*** 0.40 2.23(<.001) *** 0.43Q2 1.69(<.001)*** 0.45 1.72(<.001) *** 0.41Q3 1.08(<.001) *** 0.45 1.16(<.001) *** 0.44Q4 2.52(<.001) *** 0.51 2.09(<.001) *** 0.39
48