Plateaus, Dips, & Leaps to Expertise

Wayne D. GrayA Presentation for the In Vivo Studies of Solo and Team PerformanceSymposium at the 2019 Conference of the Cognitive Science Society

Cognitive Science Department, Rensselaer Polytechnic Institute

1

Thanks to the Office of Naval Research!

2

CogWorks Lab – 2019: Grad Students, Post-Doc*, & Faculty

Top Row L-R: Ropa Denga, “Sasha” Lutsevich, Rousell Rahman, Chris Joanis;Mid Row L-R: Matt Sangster, Jacquelyn Berry*, Catherine Sibert, SounakBanerjee 3

Flash from the Past

What Newell Told Us

ON METHODS: First Injunction of PsychologicalExperimentation:

• Know the method your subject is using to perform theexperimental task!

• To predict a subject you must know: (1) their goals; (2) thestructure of the task environment; and (3) the invariantstructure of human processing mechanisms

4Newell, A. (1973). You can’t play 20 questions with nature and win: Projective comments on the papers of thissymposium. In W. G. Chase (Ed.), Visual information processing (pp. 283–308). New York: Academic Press

Flash from the Past – update

What We are Doing About It!

Rahman, R., & Gray, W. D. (2019). Spotlight on dynamics ofindividual learning. In Proceedings of the InternationalConference on Cognitive Modeling, ICCM

Winner of the Allen Newell Best Student-Led Paper Award for2019

5

Why & How Should We Study Human Expertise? Advice fromThorndike

“The best illustrations of mental functions at their limit ofe�ciency are to be found among those occupations of workor play [in which excellence] is sought with great zeal andintelligence” (p. 178, Edward L. Thorndike, 1913).

6Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of Learning. NYC: Teachers College, ColumbiaUniversity

Allen Newell: “Accept a single complex task and do all of it.”

60 years after Thorndike, Newell dissed, “The current experimentalstyle” of his day as trying to “design speci�c small experiments toattempt to settle speci�c small questions.”

“An alternative is to focus a series of experimental and theoreticalstudies around a single complex task, the aim being to demonstratethat one has a su�cient theory of a genuine slab of humanbehavior.”

“Unfortunately, I know of no single example which successfullyshows this scheme at work. I attribute this not to its di�culty but toits not really having been tried.” (p. 303).

Comment from Gray, Sibert, Lindstedt, and colleagues . . . We havereally tried. Newell was right, this is a very interesting andproductive approach. But Newell was also an optimist! Thisapproach is very very very di�cult with no clear ending! But it hasbeen very rewarding!!!! 7

Tetris at Home

Outline

1. Tetris at Home

1.1 Constituents of Skill in Tetris

1.2 Can Your Robot Do This?

8

Changes in Individual Performance with Expertise

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

● ●

●

●

●

●

●

−2

0

2

4

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

game difficulty level

Com

p.1

(dis

arra

y)

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

● ●

●

−2

−1

0

1

2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

game difficulty level

Com

p.2

(4−

line

plan

ning

)

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

● ● ●

●

●

−1

0

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

game difficulty level

Com

p.3

(dec

ide−

mov

e−pl

aced

)

● ● ●highest (N = 27) lowest (N = 27) mid−range (N = 185)

3 components found in the Principal Component Analysis (PCA)explain just under 50% of the variance among all observedepisode-level behavioral measures of our Tetris data

9Lindstedt, J. K., & Gray, W. D. (2019). Distinguishing experts from novices by the mind’s hand and mind’s eye.Cognitive Psychology, 109, 1–25

The Disarray Component

10

Disarray is a general assessment of thedisorder in the pile, as sort of snapshotof the state of a building constructionsite.

The 4-Line Planning Component

11

4-line Planning seems less focused onimmediate behavior and more on thenature of the board structures beingbuild. This component provides anelement of higher-order cognitionwhich guides the “on board” structureas it is being built, maintained, andrepaired.

The Decide-Move-Placed Component

12

Decide-Move-Placed – our candidatemeasure of predictive processing. Thiscomponent (a) captures elements ofboth fast and accurate decision-making,(b) reveals a player’s skill across the fullspectrum of expertise represented inthis data set, and (c) across the fullbreadth of the levels of increasing timepressure.

Clark, A. (2015). Radical Predictive Processing. SouthernJournal of Philosophy, 53(1), 3–27.

Friston, K. (2018). Does predictive coding have a future?Nature Neuroscience, 21(8), 1019–1021.

The Challenge of Speed Up by Level – student players

Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7

10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 13

The Challenge of Speed Up by Level – student players

Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7

10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 14

Kill-o�at level 5

The Challenge of Speed Up by Level – student players

Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7

10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100 15

Kill-o�at level 10/12

Outline

1. Tetris at Home

1.1 Constituents of Skill in Tetris

1.2 Can Your Robot Do This?

16

Can Your Robot Do Science?

• Sibert, C., Speich, J., & Gray, W. D. (2019). Less is more:Additional information leads to lower performance intetris models. In Proceedings of the InternationalConference on Cognitive Modeling, ICCM

• Sibert, C., & Gray, W. D. (2018). The Tortoise and the Hare:Understanding the in�uence of sequence length andvariability on decision making in skilled performance.Computational Brain & Behavior, 1(3-4), 215–227

• Sibert, C., Gray, W. D., & Lindstedt, J. K. (2017).Interrogating feature learning models to discover insightsinto the development of human expertise in a real-time,dynamic decision-making task. Topics in CognitiveScience, 9(2), 374–394

17

Can Your Robot Mimic Human Experts and/or Novices?

18

Can Your Robot Do Feature Learning? (CERL)

19

Tetris in the Wild

Tetris in the Wild: CTWC

20The 2018 Classic Tetris World Championships. Photography by Anthony Hornof,Professor of Computer Science and Photojournalist Extraordinaire!

Changes in Group Expertise

21

Accessing 6 years of Tetris Championship Data

Mean Qualifying Round Score by Year (95% CI)400,000

500,000

600,000

700,000

800,000

2012 2013 2014 2015 2016 2017Year

Me

an

Qu

alif

yin

g S

co

re

Changes in Individual Expertise

22

Across 6 years of Tetris Championship Data

Qualifying Round Score by Year by SID for SID >= 6 year

Chris

Matt

Vince

Trey

Harry

Alex

Robin

Jonas

Rigel

Bo

Mike

Maximum High Score -- 1,000,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

2012 2013 2014 2015 2016 2017Year

Me

an

Qu

alif

yin

g S

co

re

If Only We Had a Time Machine!!!

Ah, but we do!!

23

If Only We Had a Time Machine!!!

Ah, but we do!!

23

Post-hoc Longitudinal Studies

24

Data out of 6 years of Tetris Championship Videos

Natural Experiments: What we are learning from the Game In-ventions of the Tetris Masters – NO NEXT BOX TETRIS!!

25

now is the time

Di�culty Seconds Speed Players by LevelLevel to Fall Up % Count % Lost0 16.00 2391 14.33 10.4 236 1.32 12.67 11.6 225 4.73 11.00 13.2 211 6.24 9.33 15.2 189 10.45 7.67 17.8 158 16.46 6.00 21.8 139 12.07 4.33 27.8 106 23.78 2.67 38.3 67 36.89 2.00 25.1 27 59.7

10-12 1.67 16.5 8 70.413-15 1.33 20.4 3 62.516-18 1.00 24.8 1 66.719-28 0.67 33.0 0 10029+ 0.33 50.7 0 100

26

Our championsbeginat level 18!

Was Newell Right?

Is the right way to advance cognitive science to take acomplex task and do all of it??

Was Newell naive!! Maybe.

We thought we had learned a lot about Tetris expertise fromour machine modeling and from the 240 undergraduates whoplayed Tetris in our lab. We had but . . . the Tetris masters hada lot more we could learn. Our champions begin at level 18 –far beyond our best ugrad.

27

28

Thank You!!Two excellent students on the job market! One istied to the New York City/Rutgers/Princeton area.The other is not. Please send me emailgrayw@rpi.edu for more info.