penetrating the black box of time-on-task estimation
TRANSCRIPT
Penetrating the Black Box of Time-on-task Estimation
http://bit.do/lak tot
Vitomir KovanovicSchool of Informatics,
University of Edinburgh,Edinburgh, United [email protected]
Dragan GasevicSchools of Education and
Informatics,University of Edinburgh,
Edinburgh, United [email protected]
Shane DawsonLearning and Teaching Unit,University of South Australia,
Adelaide, [email protected]
Srecko JoksimovicSchool of Interactive Arts
and Technology,Simon Fraser University,
Burnaby, [email protected]
Ryan S. BakerTeachers College,
Columbia University,New York, USA
Marek HatalaSchool of Interactive Arts
and Technology,Simon Fraser University,
Burnaby, [email protected]
March 19, 2015Marist College,
Poughkeepsie, NY, USA
Introduction
Time-on-task
“All learning, whether done in school orelsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
Introduction
Time-on-task
“All learning, whether done in school orelsewhere, requires time.” (Bloom, 1974, p. 682)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 1 / 28
Introduction
Time-on-task
However, there is a big difference between elapsed time,and time student actually spent on learning (Carroll,1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
Introduction
Time-on-task
However, there is a big difference between elapsed time,and time student actually spent on learning (Carroll,1963).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 2 / 28
Introduction
Origins of time-on-task in educational research
• Seminal work by J. Carroll: “A model of school learning” in 1963 put adirect link between time and learning outcomes.
• Carroll (1963) differentiaded between elapsed time and time spent onlearning.
• How time is spent is what ultimately matters!
• Increase of time-on-task was one of the key principles of effectiveeducation (Chickering and Gamson, 1989).
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 3 / 28
Introduction
Challenges with time-on-task measures
• Many different operationalizations and methods of measurement• Days in school,• Number of lectures attended,• Observing students’ behavior every X minutes.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 4 / 28
Introduction
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 5 / 28
Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
Introduction
Time-on-task in distance education / online leanring
• LMS produce a large amount of data that is used for learning analytics
• Typically data is stored as a list of events that occured during system use
• Many learning analytics studies use time-on-task measures
• Time-on-task typically calculated as time difference between recorded events
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 6 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?
• Limit all actions to 10 minutes?• Limit all actions to 30 minutes?• Discard those actions completely?• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation• Typycally simple heuristics are used (limit action duration to X minutes)• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?• Limit all actions to 10 minutes?• Limit all actions to 30 minutes?• Discard those actions completely?• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation• Typycally simple heuristics are used (limit action duration to X minutes)• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?• Limit all actions to 10 minutes?• Limit all actions to 30 minutes?• Discard those actions completely?• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation• Typycally simple heuristics are used (limit action duration to X minutes)• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Time-on-task estimation from LMS trace data
• What if action duration is too large?• Limit all actions to 10 minutes?• Limit all actions to 30 minutes?• Discard those actions completely?• Estimate their duration based on other available data points?
• There are many choices in time-on-task estimation
• Only few studies describe the process of time-on-task estimation• Typycally simple heuristics are used (limit action duration to X minutes)• Problems for replications
• How those choices affect the final study results?
• Can we trust findings based on time-on-task estimates from trace data?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 7 / 28
Introduction
Idea for the study
• Research study that involved clustering using time-on-task measures,
• Adopted one of the heuristics,
• Still, it didn’t feel totally right.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 8 / 28
Introduction Research Questions: Effects of time-on-task measuring on analytics results
Research Questions
Drawing on the Karweit and Slavin (1982) research, we conducted a study toanswer the following questions:
• What effects do different methods for estimation of time on-task-measuresfrom LMS data have on the results of analytical models?
• Are there differences in their statistical significance and overall conclusionsthat can be drawn from them?
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 9 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin
0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1
T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2
T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1
T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1
T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0s
Two types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Time-on-task estimation procedures
Time User Action Duration
T0 Walter UserLogin 0s
T1 Walter Start Viewing Discussion D1 T2 - T1
T2 Walter Start Viewing Discussion D2 T3 - T2
... ... very long time period
T3 Walter Start Viewing Assignment TMA1 T4 - T3
T4 Walter Start Viewing Resource R1 T5 - T4
... ... moderately long time period
T5 Walter User Login 0sTwo types of problems
• Outlier estimation
• Last action estimation
Typical solutions
• Ignore problematic actions
• Put certain upper limit on action duration
• Estimate based on other data points
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 10 / 28
Methods Dataset
Dataset: Trace data
• 6 offers of 13-week research-intensive fully online masters course at Canadianpublic university.
• Moodle used as LMS platform: total of 81 students and 167,261 log records.
Students Actions Messages
Winter 2008 15 33,976 212Fall 2008 22 49,928 633Summer 2009 10 21,059 243Fall 2009 7 11,346 63Winter 2010 14 31,169 359Winter 2011 13 19,783 237
Average (SD) 13.5 (5.1) 27,877 (13,561) 291.2 (192.4)Total 81 167,261 1747
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 11 / 28
Methods Dataset
Time-on-task measures / count measures
• Only used activities that were planned by the course design• Viewing assignments• Viewing resources• Viewing discussions• Posting to discussions• Updating discussion messages
• Extracted both count and time-on-task measures• Count measures used as baseline• Time-on-task measures extracted in 15 different ways
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 12 / 28
Methods Dataset
Outcome measures
• Grade structure:• TMA1 (15%): present a research paper• TMA2 (25%): write literature review paper• TMA3 (15%): write answers to 6 essay questions• TMA4 (30%): work in a team on a software project• Participation (15%): participate productively in course discussions.
• Final grade
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Outcome measures
• Grade structure:• TMA1 (15%): present a research paper• TMA2 (25%): write literature review paper• TMA3 (15%): write answers to 6 essay questions• TMA4 (30%): work in a team on a software project• Participation (15%): participate productively in course discussions.
• Final grade
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Outcome measures
• Grade structure:• TMA1 (15%): present a research paper• TMA2 (25%): write literature review paper• TMA3 (15%): write answers to 6 essay questions• TMA4 (30%): work in a team on a software project• Participation (15%): participate productively in course discussions.
• Final grade
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 13 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747messages using cognitive presencecoding scheme
• Excellent agreement, Cohen’s κ = 0.97(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%1 Triggering Event 308 17.63%2 Exploration 684 39.17%3 Integration 508 29.08%4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747messages using cognitive presencecoding scheme
• Excellent agreement, Cohen’s κ = 0.97(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%1 Triggering Event 308 17.63%2 Exploration 684 39.17%3 Integration 508 29.08%4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747messages using cognitive presencecoding scheme
• Excellent agreement, Cohen’s κ = 0.97(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%1 Triggering Event 308 17.63%2 Exploration 684 39.17%3 Integration 508 29.08%4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Dataset: discussion messages
• Coded discussion messages in accordance with Community of Inquiry (CoI)model.
• CoI model: important dimensions of distance education experience:1 Social presence: climate within course2 Teaching presence: role of instructor before and during the course3 Cognitive presence: development of critical thinking skills
• Focus on cognitive presence
• Two coders coded each of 1,747messages using cognitive presencecoding scheme
• Excellent agreement, Cohen’s κ = 0.97(only 32 disagreements)
ID Phase Messages (%)
0 Other 140 8.01%1 Triggering Event 308 17.63%2 Exploration 684 39.17%3 Integration 508 29.08%4 Resolution 107 6.12%
All phases 1747 100%
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 14 / 28
Methods Dataset
Outcome measures
1 TMA2 grade
2 TMA3 grade
3 Participation grade
4 Final grade
5 CoIHigh: number of integration and resolution messages
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 15 / 28
Methods Dataset
Measures summary
# Name Description
Count measures
1 AsignmentViewCount Number of assignment views.2 ResourceViewCount Number of resources views.3 DiscussionViewCount Number of course discussion views.4 AddPostCount Number of posted messages.5 UpdatePostCount Number of post updates.
Time-on-task measures
6 AsignmentViewTime Time spent on course assignments.7 ResourceViewTime Time spent reading course resources.8 DiscussionViewTime Time spent viewing course discussions.9 AddPostTime Time spent posting discussion messages.
10 UpdatePostTime Time spent updating discussion messages.
Performance measures
11 TMA2Grade Grade for literature review paper.12 TMA3Grade Grade for journal papers readings.13 ParticipationGrade Grade for participation in course discussions.14 FinalGrade Final grade in the course.15 CoIHigh Integration and resolution message count.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 16 / 28
Methods Experimental procedure
Time-on-task estimation strategies
# Name Description
Group 1: No outliers processing, different processing of last actions
1 x:x No outliers and last action processing.2 x:ev No outliers processing, estimation of last action duration.3 x:rm No outliers processing, removal of last action.4 x:l60 No outliers processing, 60 min last action duration limit.5 x:l30 No outliers processing, 30 min last action duration limit.6 x:l10 No outliers processing, 10 min last action duration limit.
Group 2: Thresholding outliers and last actions
7 l60 60 min duration limit.8 l30 30 min duration limit.9 l10 10 min duration limit.
Group 3: Thresholding outliers and estimating last actions
10 l60:ev 60 min duration limit, last actions estimated.11 l30:ev 30 min duration limit, last actions estimated.12 l10:ev 30 min duration limit, last actions estimated.
Group 4: Estimating outliers and last actions
13 +60ev Estimate last actions and actions longer than 60 min.14 +30ev Estimate last actions and actions longer than 30 min.15 +10ev Estimate last actions and actions longer than 10 min.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 17 / 28
Methods Experimental procedure
Statistical Analysis
• For each of the five outome measures, we constructed 16 multiple regressionmodels:
• 1 model using count measures• 15 using diferently extracted time-on-task measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 18 / 28
Results R2 variations of the overall models
Results
R2
Performance Measure Min Max Range Mean SD
TMA2Grade 0.08 0.26 0.18 0.14 0.04TMA3Grade 0.04 0.17 0.12 0.09 0.04ParticipationGrade 0.23 0.37 0.13 0.3 0.04FinalGrade 0.06 0.28 0.23 0.16 0.05CoIHigh 0.21 0.28 0.07 0.26 0.02
Variation in R2 values across five outcome measures
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 19 / 28
Results R2 variations of the overall models
Time-on-task extraction configuration
R2
0.1
50
.25
0.3
5
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Higher levels of cognitve presence (Integration + Resolution)
0.0
50
.15
0.2
5
Final percentage grade
0.2
00
.30
0.4
0
Course participation grade
0.0
00
.10
0.2
0
TMA3 grade: journal readings
Group 1:No outlier processing
Group 2:Duration limit
Group 3:Duration limit + estimation
Group 4:Estimation above limit
0.0
50
.15
0.2
5
TMA2 grade: literature review
CountsTime-on-task
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 20 / 28
Results Significance of individual predictors
Group 1 Group 2 Group 3 Group 4DV IV x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
TMA2Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
TMA3Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
Part.Grade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
FinalGrade Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
CoIHigh Assign.ViewTime
β coefficients Res.ViewTime
Disc.ViewTime
AddPostTime
UpdatePostTime
x:x x:ev x:rm x:l60 x:l30 x:l10 l60 l30 l10 l60:ev l30:ev l10:ev +60ev +30ev +10ev
Significant model at p < .05
Significant β > 0
Significant β < 0
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 21 / 28
Discussion
Discussion
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 22 / 28
Discussion
History repeats itself
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 23 / 28
Discussion
Slipery road ahead
• We need more caution when using time-on-task measures for buildinglearning analytics models.
• We need to provide details of how time-on-task has been estimated.• Supplemetary materials are great for this!• Source code repositories with time-on-task estimation code.• Develop plugins for time-on-task extraction from popular platforms.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 24 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:• Providing test statistic values and degrees of freedom,• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),• Reporting effect sizes (e.g., R2, η2, Hedges’ g, Cramer’s V),• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:• Providing test statistic values and degrees of freedom,• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),• Reporting effect sizes (e.g., R2, η2, Hedges’ g, Cramer’s V),• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:• Providing test statistic values and degrees of freedom,• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),• Reporting effect sizes (e.g., R2, η2, Hedges’ g, Cramer’s V),• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:• Providing test statistic values and degrees of freedom,• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),• Reporting effect sizes (e.g., R2, η2, Hedges’ g, Cramer’s V),• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
Implications for learning anlaytics research
• Implications on accepted standard of research in learning analytics,
Good research practices:• Providing test statistic values and degrees of freedom,• Reporting interrater agreement (e.g., Cohen’s κ, Krippendorff’s α),• Reporting effect sizes (e.g., R2, η2, Hedges’ g, Cramer’s V),• Family-wise significance corrections (e.g., Bonferroni, Holm-Bonferroni)
• Details of time-on-task estimation should be reported.
• Implications on validity of learning analytics findings,• too much emphasis on p-values, what goes into the model counts!
• Implications on replication potential,• Potential practical impact of learning analytics research.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 25 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and theirimplications.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and theirimplications.
Especially for big senstationalistic claims that conflictexisting literature!
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and theirimplications.
“Extraordinary claims require extraordinaryproof”(Truzzi, 1978, p. 11)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Implications for the Learning Analytics Community
More general problem
• Be more aware of all important methodological decissions and theirimplications.
• Validate results by adopting several different methods• Results in loss of test power,• Too much focus on small effects dependent on particular method being
adopted.
• Conduct replication studies.
• Avoid p-hacking and HARKing (Hypothesizing After the Results are Known)
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 26 / 28
Discussion Limitations
Limitations
• Ca not provide a definitive recommendation for practice,
• One statistical model,
• Despite 160,000 log records, it is still one dataset, and
• There are many more time-on-task estimation strategies.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 27 / 28
Discussion Future work
Future work
• Use DALMOOC data from Prosolo learning platform:• First six weeks 15 min innactivity logout.• Second six weeks 60 min innactivity logout.• How many of students returned (false positive), and how many did not (true
positive)?
• Looking upon ITS research, provide a gold-standard data.• LMS plugin that through javascript keeps a track of user activity.
Vitomir Kovanovic et al. Penetrating the Black Box of Time-on-task Estimation March 19, 2015, Marist College, USA 28 / 28
References I
Bloom, Benjamin S. (1974). “Time and learning”. In: American Psychologist 29.9, pp. 682–688.
Carroll, Jb (1963). “A Model of School Learning”. English. In: Teachers College Record 64.8.
WOS:A1963CAJ4400010, pp. 723–733.
Chickering, Arthur W and Zelda F Gamson (1989). “Seven principles for good practice in
undergraduate education”. In: Biochemical Education 17.3, pp. 140–141.
Karweit, Nancy and Robert E. Slavin (1982). “Time-on-task: Issues of timing, sampling, and
definition”. In: Journal of Educational Psychology 74.6, pp. 844–851.
Truzzi, Marcello (1978). “On the Extraordinary: An Attempt at Clarification”. In: Zetetic Scholar 1.1.