Bayesian Hierarchical Models of Individual Differences in Skill Acquisition

Dr Jeromy AnglimDeakin University

22nd May 2015

Functional form of the learning curve• Researchers have long been interested in functional

form of the learning curve – Power law of practice (Newell and Rosenbloom, 1981;

Snoddy 1926)– Evidence for exponential function at individual level

(Heathcote, Brown, & Mewhort, 2001)

Early example: 1024 choice-reaction time taskData from Seibel 1963; shown in Delaney et al 1998

Task Results

Relating subtask to overall task learning

• Issue of how to integrate basic findings from cognitive psychology with learning on more complex tasks

• Lee and Anderson (2001) proposed reducibility hypothesis suggesting that learning a complex task could be understood as the culmination of learning many component subtasks

• They also proposed that subtask learning will be consistent across subtasks and follow the power law of practice

Lee & Anderson (2001)

Overall Task Performance

KA Air-Traffic Controller TaskTask Analysis

Subtask Performance

Source: Lee, F. J., & Anderson, J. R. (2001). Does learning a complex task have to be complex?: A study in learning decomposition. Cognitive Psychology, 42(3), 267-316.

Gaps / Issues

Gaps• Reliance on group-level analysis• Need to refine definitions and tests of subtask

learning consistency• Lack of incorporation of trial level strategy use dataApproach• Need for task that facilitates measurement of

strategy use and subtask performance• A Bayesian hierarchical approach offers benefits over

piece-wise individual-level analysis.

Wynton-Anglim Booking (WAB) Task

1. Information Gathering (I)

2. Filtering (F)3. Timetabling (T)

Bayesian Hierarchical Models

• Increased interest in application of Bayesian Methods in psychology

• Benefits of Bayesian Approach– Clear and direct inference– Flexible model specification– Range of sophisticated model comparison tools

(e.g., DIC, Posterior predictive checks)– Well-suited to modelling repeated measures

psychological data (i.e., observations nested within people)

Models of Overall Performance

Models of Subtask Performance

Aims

1. Assess support for power and exponential functions on overall and subtask performance

2. Assess degree of consistency in subtask learning

3. Estimate effect of strategy use on subtask performance

4. Assess degree to which strategy use could explain inconsistency

Method

• Participants– 25 adults (68% female)

• Procedure– Read WAB Task instructions– Complete as many trials as possible in 50 minutes

• Processing– Extract strategy use, subtask performance and overall

task performance– Trial performance was aggregated into average block

performance (15 blocks with approximately equal numbers of trials)

Data analytic approach

• Bayesian hierarchical models were estimated using MCMC methods using JAGS with supporting analyses performed in R

• Model comparison– Graphs overlaying model fits and data– Deviance Information Criterion (DIC)– Posterior predictive checks

1. Overall performance

Does a power or exponential model provide a better model of the effect of practice on overall task performance?

Overall performance (group-level)

Overall task completion time by block (individual-level)

Overall performance: Parameter estimates and model comparison (DIC)

Interpretation• Power has larger deviance but

smaller penalty and smaller DIC• Differences are small

DIC = Mean Deviance + PenaltyRules of thumb for DIC difference:10+: rule out model with larger DIC5-10: model with smaller DIC is better

2. Subtask performance

Does a power or exponential model provide a better model of the effect of practice on subtask

performance and what is the effect of constraining subtask learning curve parameters?

Subtask performance (group-level)

Subtask performance (individual-level)

Subtask performance: Parameter estimates

Subtask Abbreviations:I = Information GatheringF = FilteringT = Timetabling

Parameters1: Amount of learning2: Rate of learning3: Asymptotic performance

Subtask performance: Model comparison (DIC)

• Power has lower DIC (3862 vs 3885); but larger mean deviance• Constraints substantially damage fit

Subtask performance: Model comparison (posterior predictive checks)

Interpretation:• When data is

simulated from a model and statistics are calculated on simulated data, good models generate statistics similar to actual data

• Bolding reflects discrepancies

3. Strategy Use on Subtask Performance

What is the effect of strategy use on subtask performance?

Strategy use (group-level)

Strategy use on performance: Parameter estimates

Note: • Parameter estimates (i.e., exp (lambda)) for

strategy covariates on subtask performance• exp(lambda): expected multiple to task

completion time resulting from strategy use• exp(lambda) greater than 1: strategy use

increases task completion time• exp(lambda) less than 1: strategy use

decreases task completion time

4. Strategy Use and Subtask Learning Consistency

To what extent does strategy use explain subtask learning

inconsistency?

Strategy use explaining subtask inconsistency (group-level)

Strategy use explaining subtask inconsistency (individual-level)

Subtask performance with strategies: Model comparison (DIC)

• Strategies improve fit (e.g., 3885 – 3506 = 379)

• Damage to DIC fit of constraints is less with strategies (e.g., 3794 – 3506 = 288) than without strategies (e.g., 4497 – 3885 = 612)

Subtask performance: Model Comparison (Posterior predictive checks)

Concluding Thoughts

Concluding thoughts

• Differences between power and exponential are fairly subtle

• Task learning may be decomposed into subtask learning but functional form of subtask learning can vary

• Strategy use both expresses learning and learning to trade-off time on subtasks is a strategy itself

• More generally, the study provides a case study of Bayesian hierarchical methods

Future Work

• Further Bayesian skill acquisition research– Formal models of strategy acquisition– Models of discontinuities in the learning curve– Integrating traits (ability and personality) into

dynamic models of performance• Extending Bayesian Hierarchical methods to a

range of domains– personality faking, longitudinal life satisfaction

data, diary employee well-being data

Notes

• Code and data– https://github.com/jeromyanglim/anglim-wynton-2014-subtasks

• Publication– Based on work with Sarah Wynton– Anglim, J., & Wynton, S. K. (2015). Hierarchical Bayesian

Models of Subtask Learning. Journal of Experimental Psychology. Learning, Memory, and Cognition. Online First. http://dx.doi.org/10.1037/xlm0000103

• My Contact details– jeromy.anglim@deakin.edu.au– http://jeromyanglim.blogspot.com

Thank you

Questions?