terence automated reasoning and natural language processing for generating educational games from...

Rosella GennariKRDB - CS Faculty - UniBZ

http://www.terenceproject.euMain contributors: FBK-irst, LUH, UnivAQ, USAL

TERENCE WP4 for reasoning about annotations

and generating textual smart games

http://www.terenceproject.eu

http://www.terenceproject.eu

© R. Gennari, LUB, <[email protected]> TERENCE 3rd review meeting, 27th Nov 2013, l’Aquila

TERENCE WP4and its deliverables

STEP 3:query annotations and rank events

STEP 2:reason about

annotations and deduce further relations

STEP 5:generate textual information

for events

STEP 6:generate textual

games

STEP 4:select top-

ranked events

ARNL

GNL

P

STEP 1:annotate stories with events, entities and

relations

STEP 7:assemble textual

games and graphical

components

WP3 WP4 WP6

From D4.2 and D4.3 technical annex

Deliverable 1st release 2nd release 3rd release

D4.1 Sep. 2011D4.2 May 2012 November 2012 August 2013

D4.3 June 2012 December 2012 September 2013

mailto:[email protected]



year 1 year 2








year 1 year 2 year 3








{recap

{focus

year 1 year 2 year 3








Deliverable 1st release

2nd release 3rd release

D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013Automated Reasoning (AR)

about stories: -semantics, -consistency checking -deduction of non-‐local TLINKs

TERENCE WP4release 1






D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012



AR generation of textual games: -data structure from game frameworks

!"#$%&"'"()*+,-& ."*/&%0)%-(

!"#$"%&'()*'+,+'- ./%01*'+,+'-*#10*'+/02+0-34*"()&13*%5/0/%(+0*6/7+-84*%()"&'"2%3)"3*("7+*6/7+-84*)-03*%/&-/'"()*6/7+-9*

:%5+;&'"26*1#*0+/;"26*/2;*<'/)"26

=-(*-"'+2(*&"32%(48* >2;* <'/)"26* *'3&)* 6/7+-8*?0;*<'/)"26*&"536%(4*6/7+-

@12-(0/"2(-*12*/%("12-

A+/02+0-* -51&';* 6+(* #/-(+0B* 5+2%+* /* 6/7+* 5/-* /* 7/C"7/'*&"*-5$)%-(+)%'"

D0160+--*/2;*#++;E/%F

7-(%)-&* /2;* 6",+* '+/02+0-* G=H* ";+/* 1#* 0&-4&"**B* G>H*"6053(3)-&1*#++;E/%FB*G?H*0+%/''*(5+"0*/((+2("12*/2;**-5%/%)+(5+7*(1*6",+*/*0+-1'&("12*G"2*("7+H

I+<0+-+2(/("12 D01;&%("12*%/2*E+* "7</"0+;* 5+2%+*<0171(+* 0+-1'&("12*,"/*8%*$35*&"0&"*"()3)%-(+3(2+&"3*-(%(4

!"#$%&'()"# !"#$%&'$!"#$%&'$ (&%)*%&'*+(&%)*%&'*+(&%)*%&'*+(&%)*%&'*+ ,'-#.*/%&','-#.*/%&'

*+),'-# 01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#. 3526#73#)#'-3526#73#)#'-3526#73#)#'-

.)/&()"# 01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$

0--123'4 ,'-#.*/%&' 0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;> @6A+*'*-&.?@6A+*'*-&.?@6A+*'*-&.? B&+"%&'

.53%$67),"$# C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+C.&A&.%&'*+3-&3-1#3+#*.'#.’$3*92+2-?32'3-1#3<*8#3+#)#+

8-/39,":67),"$# 0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-

;<3$3% D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$D*AA?:$*73$-*-#$

=,5- $&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'- 2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-

8&/-# !"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%!"#"$%3&43-1#3$?$-#8E3#&'()%3&43-1#3+#*.'#.E3&()%"*#+)"%







D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012



AR generation of textual games: -data structure from game frameworks

!"#$%&"'"()*+,-& ."*/&%0)%-(

!"#$"%&'()*'+,+'- ./%01*'+,+'-*#10*'+/02+0-34*"()&13*%5/0/%(+0*6/7+-84*%()"&'"2%3)"3*("7+*6/7+-84*)-03*%/&-/'"()*6/7+-9*

:%5+;&'"26*1#*0+/;"26*/2;*<'/)"26

=-(*-"'+2(*&"32%(48* >2;* <'/)"26* *'3&)* 6/7+-8*?0;*<'/)"26*&"536%(4*6/7+-

@12-(0/"2(-*12*/%("12-

A+/02+0-* -51&';* 6+(* #/-(+0B* 5+2%+* /* 6/7+* 5/-* /* 7/C"7/'*&"*-5$)%-(+)%'"

D0160+--*/2;*#++;E/%F

7-(%)-&* /2;* 6",+* '+/02+0-* G=H* ";+/* 1#* 0&-4&"**B* G>H*"6053(3)-&1*#++;E/%FB*G?H*0+%/''*(5+"0*/((+2("12*/2;**-5%/%)+(5+7*(1*6",+*/*0+-1'&("12*G"2*("7+H

I+<0+-+2(/("12 D01;&%("12*%/2*E+* "7</"0+;* 5+2%+*<0171(+* 0+-1'&("12*,"/*8%*$35*&"0&"*"()3)%-(+3(2+&"3*-(%(4

!"#$%&'()"# !"#$%&'$!"#$%&'$ (&%)*%&'*+(&%)*%&'*+(&%)*%&'*+(&%)*%&'*+ ,'-#.*/%&','-#.*/%&'

*+),'-# 01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#. 3526#73#)#'-3526#73#)#'-3526#73#)#'-

.)/&()"# 01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$

0--123'4 ,'-#.*/%&' 0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;> @6A+*'*-&.?@6A+*'*-&.?@6A+*'*-&.? B&+"%&'


8-/39,":67),"$# 0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-


=,5- $&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'- 2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-


-population with generate-‐and-‐test





-AR generation: from generate-and-test to generate-with-constraints




D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




-AR generation: from generate-and-test to generate-with-constraints

-Natural language generation components for EN and IT




D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




Action 1: code optimisation for the generation workflow

Action 2: analysis of impact on WP4 of WP7 & re-design

Section 2.1: The TERENCE

Annotation Language

Chapter 5: Generation of

Game Events and Who Questions

Chapter 6: The Importance of

Being Plausible

Chapter 4: Generation of

Textual Games

Chapter 3: Enrichment of

Annotated Stories

Section 2.2: The Allen Interval

Algebra and Tractable

Subalgebras

Section 2.2: The TERENCE Smart Game

Design

Action 3: documenting in technical annex to D4.2 and D4.3





D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




Action 1: code optimisation for the generation workflow, e.g.,

-for reasoning about stories,

‣hard consistency checking

‣relaxed consistency checking

-for natural language generation components





D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




Action 2: (a) analysis of WP7 expert-based evaluation results



!"#$%&'(")&*'+",-'.$&'*.&#*

He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers.

revise selection of solutions

revise selection of central event



D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




Action 2: (a) analysis of WP7 expert-based evaluation results, (b) re-design generation components, e.g., for





D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013







revision of AR selection of central events and solutions



D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013








revision of NLG text

text

text text text



D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013







overall assessment of generation

text

story

text + visual

games


revision of NLG text

text

text text text



D4.2 May 2012

November 2012

August 2013

D4.3 June 2012

December 2012

September 2013




Selection of central events for games:

>Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one

TERENCE smart gamesrelease 3: action 2





Selection of central events for games:

>Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one



>Implications for WP4: none picked up




Selection of plausible solutions:

>Results: out of 140 changes of selection of solutions, the majority was for wrong solutions










>Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3,






-‐ generate a wrong solution from correct one by changing participants, e.g.,

<correct_sentence id="2">The man ran and fell on the ground.

</correct_sentence>

<wrong_sentence id="2wh1">Peter ran and fell on the ground.

</wrong_sentence>



>Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3,




Development times:

>Results for revision time:

-‐ 12’6” per game instance:

↑ 12’8” for time games

↓ 10’6” for who games

>Results for creation time:

-‐ avg. 23” per game instance

text

story

text + visual

games






Development times:

>Results for revision time:

-‐ 12’6” per game instance:

↑ 12’8” for time games

↓ 10’6” for who games

>Results for creation time:

-‐ avg. 23” per game instance

text

story

text + visual

games



>Implications for WP4: the semi-automated development process seems to be promising for optimising development times




TERENCE WP4 conclusions


D4.1 Sep. 2011

D4.2 May 2012 November 2012 August 2013


!"#$%&'(")&*'+",-'.$&'*.&#*




!"#$%&"'"()*+,-& ."*/&%0)%-(

!"#$"%&'()*'+,+'- ./%01*'+,+'-*#10*'+/02+0-34*"()&13*%5/0/%(+0*6/7+-84*%()"&'"2%3)"3*("7+*6/7+-84*)-03*%/&-/'"()*6/7+-9*

:%5+;&'"26*1#*0+/;"26*/2;*<'/)"26

=-(*-"'+2(*&"32%(48* >2;* <'/)"26* *'3&)* 6/7+-8*?0;*<'/)"26*&"536%(4*6/7+-

@12-(0/"2(-*12*/%("12-

A+/02+0-* -51&';* 6+(* #/-(+0B* 5+2%+* /* 6/7+* 5/-* /* 7/C"7/'*&"*-5$)%-(+)%'"

D0160+--*/2;*#++;E/%F

7-(%)-&* /2;* 6",+* '+/02+0-* G=H* ";+/* 1#* 0&-4&"**B* G>H*"6053(3)-&1*#++;E/%FB*G?H*0+%/''*(5+"0*/((+2("12*/2;**-5%/%)+(5+7*(1*6",+*/*0+-1'&("12*G"2*("7+H

I+<0+-+2(/("12 D01;&%("12*%/2*E+* "7</"0+;* 5+2%+*<0171(+* 0+-1'&("12*,"/*8%*$35*&"0&"*"()3)%-(+3(2+&"3*-(%(4

!"#$%&'()"# !"#$%&'$!"#$%&'$ (&%)*%&'*+(&%)*%&'*+(&%)*%&'*+(&%)*%&'*+ ,'-#.*/%&','-#.*/%&'

*+),'-# 01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#. 3526#73#)#'-3526#73#)#'-3526#73#)#'-

.)/&()"# 01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$

0--123'4 ,'-#.*/%&' 0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;> @6A+*'*-&.?@6A+*'*-&.?@6A+*'*-&.? B&+"%&'


8-/39,":67),"$# 0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-


=,5- $&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'- 2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-


data structures

AR1 for stories

AR1 for txt games

framework(D2.3)

AR2 + NLG1 for txt games

! AR2 for stories AR3 + NLG2 for txt games!

!~~

~




TERENCE WP4 conclusionsgame over ?


D4.1 Sep. 2011

D4.2 May 2012 November 2012 August 2013


!"#$%&'(")&*'+",-'.$&'*.&#*




!"#$%&"'"()*+,-& ."*/&%0)%-(

!"#$"%&'()*'+,+'- ./%01*'+,+'-*#10*'+/02+0-34*"()&13*%5/0/%(+0*6/7+-84*%()"&'"2%3)"3*("7+*6/7+-84*)-03*%/&-/'"()*6/7+-9*

:%5+;&'"26*1#*0+/;"26*/2;*<'/)"26

=-(*-"'+2(*&"32%(48* >2;* <'/)"26* *'3&)* 6/7+-8*?0;*<'/)"26*&"536%(4*6/7+-

@12-(0/"2(-*12*/%("12-

A+/02+0-* -51&';* 6+(* #/-(+0B* 5+2%+* /* 6/7+* 5/-* /* 7/C"7/'*&"*-5$)%-(+)%'"

D0160+--*/2;*#++;E/%F

7-(%)-&* /2;* 6",+* '+/02+0-* G=H* ";+/* 1#* 0&-4&"**B* G>H*"6053(3)-&1*#++;E/%FB*G?H*0+%/''*(5+"0*/((+2("12*/2;**-5%/%)+(5+7*(1*6",+*/*0+-1'&("12*G"2*("7+H

I+<0+-+2(/("12 D01;&%("12*%/2*E+* "7</"0+;* 5+2%+*<0171(+* 0+-1'&("12*,"/*8%*$35*&"0&"*"()3)%-(+3(2+&"3*-(%(4

!"#$%&'()"# !"#$%&'$!"#$%&'$ (&%)*%&'*+(&%)*%&'*+(&%)*%&'*+(&%)*%&'*+ ,'-#.*/%&','-#.*/%&'

*+),'-# 01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#.01&2/#$34&.3+#*.'#. 3526#73#)#'-3526#73#)#'-3526#73#)#'-

.)/&()"# 01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$01&2/#$3&.3-1#2.3/&892'*%&'$3-1*-3*.#3/&..#/-:;.&'<3=/:;>3$&+"%&'$

0--123'4 ,'-#.*/%&' 0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;>0&'$2$-#'/?3=/:;> @6A+*'*-&.?@6A+*'*-&.?@6A+*'*-&.? B&+"%&'


8-/39,":67),"$# 0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-0&'$-*'-


=,5- $&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'-$&+"%&'3/&'$-*'- 2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-2'-#.*/%&'3/&'$-*'-


data structures

AR1 for stories

AR1 for txt games

framework(D2.3)

AR2 + NLG1 for txt games

! AR2 for stories AR3 + NLG2 for txt games!

!~~

~



terence automated reasoning and natural language processing for generating educational games from...

Technology