gennari vittorini-terence-ai*ia-2014

Improving*Children’s*Reading*Comprehension

Arti6icial*Intelligence

for

Rosella*Gennari

http://www.inf.unibz.it/~gennari

Pierpaolo*Vittorini*http://vittorini.univaq.it

www.terenceproject.eu

http://www.inf.unibz.it/~gennari

http://vittorini.univaq.it

http://www.terenceproject.eu

ONCE%UPON%A%TIME…

[…]%After%two%days%of%hot%sun%and%of%swimming%in%the%lake,%today%it%was%the%big%match%for%Alessio,%the%last%game%of%the%season%in%his%home%town,%l’Aquila.%There%was%only%a%minute%of%the%game%left.%The%crowd%watched%in%silence%as%Alessio%took%the%penalty%shot…%%And%the%crowd%of%l’Aquila%cheered%and%roared!

ONCE%UPON%A%TIME…

What%does%Alessio%play?

[…]%After%two%days%of%hot%sun%and%of%swimming%in%the%lake,%today%it%was%the%big%match%for%Alessio,%the%last%game%of%the%season%in%his%home%town,%l’Aquila.%There%was%only%a%minute%of%the%game%left.%The%crowd%watched%in%silence%as%Alessio%took%the%penalty%shot…%%And%the%crowd%of%l’Aquila%cheered%and%roared!

When%does%the%game%take%place?%

Who%wins?

ONCE%UPON%A%TIME…

[…]%After%two%days%of%hot*sun*and*of*swimming*in*the*lake,%today%it%was%the%big%match%for%Alessio,%the%last%game%of%the%season%in%his%home%town,%l’Aquila.%There%was%only%a%minute%of%the%game%left.%The%crowd%watched%in%silence%as%Alessio%took%the%penalty%shot…%%And%the%crowd%of%l’Aquila%cheered%and%roared!



Who%scores?

ONCE%UPON%A%TIME…

[…]%After%two%days%of%hot%sun%and%of%swimming%in%the%lake,%today%it%was%the%big%match%for%Alessio,*the*last*game*of*the*season*in*his*home*town,*l’Aquila.%There%was%only%a%minute%of%the%game%left.%The*crowd*watched*in*silence*as*Alessio*took*the*penalty*shot…**And*the*crowd*of*l’Aquila*cheered*and*roared!



Who%scores?

ONCE%UPON%A%TIME…

[…]%After%two%days%of%hot%sun%and%of%swimming%in%the%lake,%today%it%was%the%big%match%for%Alessio,%the*last*game*of*the*season*in*his*home*town,*l’Aquila.*There*was*only*a*minute*of*the*game*left.*The*crowd*watched*in*silence*as*Alessio*took*the*penalty*shot…**And*the*crowd*of*l’Aquila*cheered*and*roared!

archivio.riparteilfuturo.it



Who%scores?

THE%PROBLEM

‣More%than%10%%of%primary%school%children,%older%than%8,%are%diagnosed%with%deep%text%comprehension%problems,%e.g.,%they%fail%answering%

when%does%the%game%take%place?%%

‣They%are%referred%to%as%poor*comprehenders

?

THE%PROBLEM

‣More%than%10%%of%primary%school%children,%older%than%8,%are%diagnosed%with%deep%text%comprehension%problems,%e.g.,%they%fail%answering%

when%does%the%game%take%place?%%

‣They%are%referred%to%as%poor*comprehenders

‣Can%technology%help%them?

?

OUTLINE

1. Technology%enhanced%learning%

2. The%TERENCE%case%study%

3. Game%over%

AI**TEL

TECHNOLOGY%ENHANCED%LEARNING%(TEL)="technology2for2the2growth"of"a"learning"experience

AI**TEL


innaffiatoi

o

AI**TEL


innaffiatoi

o

ARTIFICIAL INTELLIGENCE (AI)

AI**TEL


innaffiatoi

o

How%can%we%%design%and%evaluate%%AI%products%that%support%%the%growth%of%their%users’%learning*experience?%

ARTIFICIAL INTELLIGENCE (AI)

TEL%4%LEARNING%EXPERIENCE

Montessori:2adequate"tasks2that2come2in2a2prepared"environment2designed2on2top2of2the2learner2characteristics"can2effectively2support2learning

innaffiatoi

o

ousability%of%%learning%products



innaffiatoi

o

ousability%of%%learning%products

opedagogical%effectiveness%%of%products%for%learning*growth



innaffiatoi

o

OUTLINE1. Student%centred%learning%and%technology%

enhanced%learning%

2. The%TERENCE%case%study%3. Game%over

THE%TERENCE%ADAPTIVE%SOLUTION

w"h"o""a"r"e""y"o"u"?





a"d"e"q"u"a"t"e"b"o"o"k""o"f"""s"t"o"r"i"e"s






a"d"e"q"u"a"t"e"s"m"a"r"t"g"a"m"e"s






r"e"w"a"r"da"d"e"q"u"a"t"e"s"m"a"r"t"g"a"m"e"s





s"i"g"n""i"n

a"d"e"q"u"a"t"e"s"m"a"r"t"g"a"m"e"s

r"e"w"a"r"d



Requirements Prototypes Analytic + small ev. Int. prod. Large ev. Fin. prod.

HOW%DID%WE%GET%TO%THE%TERENCE%SOLUTION



problem: 256 stories,

each with ~12 games

Smart*game*design



how can we automatise the

development of smart games via AI (and, hopefully, be efficient)?

Smart*game*design



story

ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF

VisualisationManagerillustrations

NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule

SemiRautomated*generation



story

annotations

ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule




enriched annotations

story

annotations

ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule





story

annotations


ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule




story

texttext text

text

annotations


ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule



text text text

textimage

image image image


story

annotations




template visual

text text text

textimage

image image image


story

annotations


ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule



games

template visual

text text text

textimage

image image image


story

annotations


ALS LayerGUI Layer

Learner

EducatorExpert

Learner GUI

Expert GUI

Persistence Layer

OpenRDF

UserManager

OpenRDF

StoryManager

OpenRDF

GameManager

OpenRDF


NPL

Reasoner

AdaptiveEngine

Visualisation

��

Reasoning Module

AnnotationModule

VisualisationModule



text

story

text + visual

games

AUTOM. ➤MANUAL ➤ AUTOM.




Requirements Prototypes Analytic + small ev. Int. prod. Large ev. Fin. prod.

HOW%DID%WE%GET%TO%THE%TERENCE%SOLUTION



APPROACH WITH%WHOM EXAMPLE%METHODS WHEN

analytical HMI%experts%or%domain%experts

heuristic%evaluationformative,%summativeexpert%evaluation

cognitive%walk`through

small`scale learnersobservations

formativethink%aloud

large`scale learners aield%studies summative

EVALUATION%IN%TERENCE



APPROACH WITH%WHOM EXAMPLE%METHODS WHEN







EVALUATION%IN%TERENCE

From D7.4

LARGE`SCALE%EVALUATION%DESIGN

Common*design*of*the*intervention*with*TERENCE:%

‣how:*pretest/posttest%design,%with%experimental%and%control%groups%‣hypothesis:%TERENCE%improves%reading%comprehension%measured%with%standardised%text%comprehension%tests

ControlExperimental

From D7.4

LARGE`SCALE%EVALUATION%DESIGN

Common*design*of*the*intervention*with*TERENCE:%

‣how:*pretest/posttest%design,%with%experimental%and%control%groups%‣hypothesis:%TERENCE%improves%reading%comprehension%measured%with%standardised%text%comprehension%tests

ControlExperimental

Control*vs*experimental%

‣ analysis:%repeated%measure%ANOVA%%‣ results:*the%hypothesis%is%conairmed%(F=25.65,%p<0.0001)%

From D7.4

Pre`post%performances%for%text%comprehension%(dependent%variable)%were%as%follows:%‣Pescina:%` pre:%14%poor%comprehenders%(20.59%)%%` post:%6%poor%comprehenders%(8.82%)%

‣Avezzano:%` pre:%15%poor%comprehenders%(5.95%)%%` post:%2%poor%comprehenders%(0.79%)

EXPERIMENTAL%GROUP

PrePescina Avezzano

5,95%

20,59%

From D7.4



EXPERIMENTAL%GROUP

PostPescina Avezzano

0,79%

8,82%

From D7.4



EXPERIMENTAL%GROUP

PostPescina Avezzano

0,79%

8,82%

‣Wilcoxon%signed`rank%test%supports%that%differences%are%statistically*signi6icant*` Pescina:%z=`4.904,%p<0.0001%` Avezzano:%z=`2.266,%p=0.0234From D7.4

EXAMPLE%METHODS%IN%TERENCE

APPROACH WITH%WHOM EXAMPLE%METHODS HOW







From D4.2 and D4.3 technical annex


overall%generation

text

story

text$+$visual

games

revision%of%Natural%Language%Processing%(NLP)%generation%of%texts%for%labelling%%%%%%%%%

text

text text text

Evaluation*tasks

revision%of%Automated%Reasoning%selection%of%central%events%and%solutions

Automated*reasoning*selection*of*central*events*for*games:%%>Results:%only%in%15%out%of%250%cases%(6%),%it%was%necessary%to%select% a% different% central% event% than% the% automatically%generated%one


Selection*of*central*events

Automated*reasoning*selection*of*central*events*for*games:%%>Results:%only%in%15%out%of%250%cases%(6%),%it%was%necessary%to%select% a% different% central% event% than% the% automatically%generated%one


>Implications2 for2 AR:%none%picked%up

Selection*of*central*events

Automated*reasoning*selection*of*plausible*solutions:%%>Results:%out%of%140%changes%of%selection%of%solutions,%the%majority%was%for%wrong%solutions

Selection*of*solutions*and*distractors


Automated*reasoning*selection*of*plausible*solutions:%%>Results:%out%of%140%changes%of%selection%of%solutions,%the%majority%was%for%wrong%solutions>Implications2 for2WP4:*new%heuristics% for%wrong*plausible*solutions%in%the%last%part%of%Y3,



Automated*reasoning*selection*of*plausible*solutions:%%>Results:%out%of%140%changes%of%selection%of%solutions,%the%majority%was%for%wrong%solutions

` generate% a% wrong% solution% from%correct% one%by% changing%participants,%e.g.,%<correct_sentence%id="2">The%man%ran%and%fell%on%the%ground.%%%%%</correct_sentence>%<wrong_sentence%id="2wh1">Peter%ran%and%fell%on%the%ground.%</wrong_sentence>%

>Implications2 for2WP4:*new%heuristics% for%wrong*plausible*solutions%in%the%last%part%of%Y3,



Overall*generation:%development%times:%%>Results2for2revision2time:22` 12’6”%per%game%instance:%↑%12’8”%for%time%games%↓%10’6”%for%who%games%>Results2for2creation2time:22` avg.%23”%per%game%instance

text

story

text$+$visual

games


Overall*generation

Overall*generation:%development%times:%%>Results2for2revision2time:22` 12’6”%per%game%instance:%↑%12’8”%for%time%games%↓%10’6”%for%who%games%>Results2for2creation2time:22` avg.%23”%per%game%instance

text

story

text$+$visual

games


>Implications2for2the2generation2process:%the%AI`based%semi`automated%development%process%seems%to%be%promising%for%optimising%development%times

Overall*generation

OUTLINE

1. Student%centred%learning%and%technology%enhanced%learning%

2. The%TERENCE%case%study%

3. Game%over%

AND%THE%QUIZ%SOLUTION%IS…

MAIN%REFERENCES1. R.%Gennari,%S.%Tonelli,%and%P.%Vittorini.%An*AIRbased*Process*for*Generating*Games*from*Flat*Stories.%In%Proc.%of%the%33rd%SGAI%Conference%(AI`2013),%Cambridge,%UK,%December%2013.%

2. F.%de%la%Prieta,%T.%Di%Mascio,%R.%Gennari,%I.%Marenzi,%P.%Vittorini.%User*Centred*and*Evidence*Based*Design:*an*Experience*Report.%In%IJTEL%Journal,%Inderscience%Enterprise%Ltd.,%forthcoming%in%2014.%

3. D.%Di%Giacomo,%R.%Cecilia,%V.%Cofini,%T.%Di%Mascio,%R.%Gennari,%P.%Vittorini.%Silent*Reading*Supported*by*Adaptive*Learning*Technologies:*

Influence*In*the*Children*Outcomes.%In%Computers%in%Human%Behavior%Journal,%Elsevier,%forthcoming%in%2015.%

4. TERENCE%deliverables%for%technical*annex*to*D4.3.1,*D4.3.2,*D4.3.3%and%D7.4:%http://www.terenceproject.eu/web/guest/public`deliverables%%

Merry Seasonal Festivitieswith

TERENCE booksfor remembering

its best stories

THE TERENCE BOOK 2THE TERENCE BOOK 1

Dedicated to:TERENCE team, reviewers, teachers, children, and all those that contributed to the TERENCE stories. Our sincere “grazie”.

gennari vittorini-terence-ai*ia-2014

Technology

intelligence technology