standing on the shoulders of chess masters: using rcts to produce evidence about teaching chess in...
TRANSCRIPT
Standing on the Shoulders of Chess Masters:
Using RCTs to Produce Evidence aboutTeaching Chess in Italian Elementary Schools
Gianluca Argentin Gianluca Argentin
Alberto MartiniAlberto Martini
Barbara RomanoBarbara [email protected]
Why should chess be taught in school?
According to the European Parliament for
several reasons
The EU Parliament claims that
“Chess can improve children’s concentration, patience and persistence
and can develop sense of creativity, intuition, memory, analytic and decision-
making skills. Chess also teaches determination, motivation and
sportsmanship”
The evidence on the effects of learning chess on cognitive outcomes is literally scattered all over the world: from chess association websites to academic journals in a variety of disciplines.
However
Rarely this evidence is based on rigorous evaluation designsuch as randomized controlled trials
We will present:
1. the results of a simple RCT to measure the effects of learning chess on math achievement (SAM)
2. the design of a more complex RCT to answer several questions that arise when incorporating chess in the regular curriculum (SC3)
Our true long run objective is promoting the use of random assignment in evaluation and
education research in Italy.
PROGETTO SAM
Scacchi e Apprendimento
della Matematica
We recruited 33 schools that had been involved in chess teaching before
(a non probabilistic sample, yet nicely spread all over Italy)
Each school agreed to exclude at least one 3° grade class, chosen
randomly
All the other 3° grade classes were taught 30 hours of chess by
professional instructors during 2009-10 school year
A total of 113 classes - about 2000 pupils - participated either
as treatment or control
All pupils were pre-tested and post-tested using modified versions of INVALSI tests
High level of compliance to the design
0 20 40 60 80 100
Gruppo di controlloGruppo sperimentale
0 20 40 60 80 100
Gruppo di controlloGruppo sperimentale
Clearly something is happening to the experimentals
ImpactPre-test Score distribution Post-test Score distribution
Estimated effect on math scores(in standard deviations)
ITT
Mod. 0: nessun controllo, tutti i casi
0.28
p value .01***
Effect size 0.34
P-value .01***
A third of a standard deviation is an exceptionally good result
This makes people (including us) doubt it
Is it because you included questions too favorable for chess players?
a. Draw the snail path
b. Which shape corresponds to the snail path?
We found an effect on almost each item on the test
Differential probability of correct response for each item
IT MUST BE THE EFFECT OF CHEATING!
-10
12
Med
ia
.4 .6 .8 1 1.2 1.4Dev. St.
Mandati Controlli
CHEATING SHOULD SHOW UP HERE, BUT IT’S EMPTY
We need to prove it with an RCT based on a probabilistic
sample of schools
Chess does work!
Sc3
SPERIMENTAZIONE CONTROLLATA DI
SCACCHI A SCUOLA
(Springer = Knight in German)
Does learning chess at school (at some point between 3rd and 4th grade) affects achievement in mathematics and reading measured in 5th grade?
How persistent are the effects of learning chess? Does any beneficial effect remain one or two years after training? Which fraction of pupils continue to play chess and possibly benefit from it, after it is no longer taught in class?
Does chess taught by professional instructors obtain better or similar results than when based on web teaching tools with the support of regular teachers? Which solution is more cost-effective and sustainable at scale?
First question: does learning chess improve achievement in math and reading, as measured by INVALSI standardized tests
80 control schools
160 treated schools
Fifth grade
INVALSINational
test
Fifth grade
INVALSINational
test
Second grade
INVALSINational
test
Second grade
INVALSINational
test
240 schools selected with probability prop. to sizeout of 18,000 primary
schools in Italy
RARA
The second question:External instructor or regular teacher+web?
80 schools“teaching by external instructor”
Fifth gradeINVALSINational
test
Fifth gradeINVALSINational
test
Second grade
INVALSINational
test
Second grade
INVALSINational
test80 schools
“web-based with support by regular teachers”
RARA
The third question:Do school-based activities increase persistency of the
effect of learning chess?
80 schools promote chess in 5° grade
Test of knowledge
of chess and
frequencyof use (late In
fifth grade)
Test of knowledge
of chess and
frequencyof use (late In
fifth grade)
80 schoolsdo not promote chess in 5° grade
Test of knowledge
of chess and
frequencyof use
(early in third
grade)
Test of knowledge
of chess and
frequencyof use
(early in third
grade)
RARA
The two dimensions crossed represent a factorial design
We are looking for sponsor/supporter
And for other countries willing to “play (it again) SAM”
Anyone interested?