co-funded by the european union the qall-me benchmark: a multilingual resource of annotated spoken...

Co-funded by the European Union

The QALL-ME Benchmark:The QALL-ME Benchmark:a Multilingual Resource of a Multilingual Resource of

Annotated Spoken Annotated Spoken Requests for Question Requests for Question

AnsweringAnsweringE. Cabrio, M. Kouylekov, B. Magnini, M. Negri (FBK-Irst)

L. Hasler, C. Orasan, (University of Wolverhampton)

D. Tomas, J.L. Vicedo (University of Alicante)

G. Neumann, C. Weber (DFKI)

Outline:Outline:o Motivations and goalso QALL-ME Projecto QALL-ME Benchmark

oData collectiono Translation into Englisho Speech Acts AnnotationoQuestion Answering Annotationo Annotation of relations

o Conclusion and Future Work

LREC - 28-30 May 2008 - Marrakech (Morocco)

Context: the Qall-me Context: the Qall-me projectproject

QALL-ME (Question Answering Learning QALL-ME (Question Answering Learning technologies in a multiLingual and multiModal technologies in a multiLingual and multiModal Environment): Environment): an EU-funded project aiming at the realization of a shared and distributed infrastructure for Question Answering systems on mobile devices (e.g. mobile phones).


SMSSMS

INPUT OUTPUT

SMSSMS

MMSMMS

VOICE

TEXT

TEXT

VOICE

VIDEO

DIGITAL

ASSISTANT

QALL-ME detailsQALL-ME detailso Reference: FP6 IST-033860o Contract Type: STREP o Start date: October 1st, 2006o Duration: 36 monthso Project Funding: 2.82 M euros

http://qallme.fbk.eu


FBK- Irst, Italy Comdata S,p.A., Italy

DFKI, Germany Ubiest S.p.A., Italy

University of Alicante, Spain Waycom S.r.l., Italy

University of Wolverhampton, UK

MotivationsMotivations

o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)


Motivation: beyond Motivation: beyond factoid…factoid…

has Venezia hotel a restaurant is there a toll free number for the INAIL office in via Gazzoletti in Trento

VERIFICATIONVERIFICATION

where is the INAIL office and how can I get there

how can I get to the pharmacy De Gerloni of Trento

PROCEDURALPROCEDURAL




o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).


ChallengesChallengeso Context aware QA

o What can I see tonight at cinema o Where is the nearest pharmacy

o Persistent vs dynamic information

o Multiple sources (database, newspaper, web)

LREC - 28-30 May 2008 - Marrakech (Morocco) 8

Challenges related to Challenges related to eventsevents

o Context aware QA

o What can I see tonight at cinema (in Trento)

o Where is the nearest pharmacy (to piazza Duomo)

o Persistent vs dynamic information

o Multiple sources (database, newspaper, web)

LREC - 28-30 May 2008 - Marrakech (Morocco) 9




o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).


QA annotationQA annotation

may I know where the ice stadium of Trento is located and at what time it opens


LOCATION

DATE

Expected Answer Type :




o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).

o Investigating of the portability of semantic annotation portability of semantic annotation through languages.


Portability of annotationsPortability of annotations

may I know where the ice stadium of Trento is located


Expected Answer Type: LOCATION

potrei sapere dov’è lo stadiodel ghiaccio di Trento

puedo saber donde esta el estadio de hielo de Trento

ich möchte wissen wo das Eisstadium von Trento ist

Data collectionData collection

o 14645 questions in four different languages:

ITALIAN, ENGLISH, GERMAN, SPANISHITALIAN, ENGLISH, GERMAN, SPANISHo Domain: cultural events in a town

Acquisition:Acquisition:Every speaker performs 30 questions, based on 15 scenarios :o Using a graphical interface, for each scenario is first generated a

spontaneous request and then a written one (previously predefined)o A telephone was used to acquire questions.


Data collectionData collection# words # utterances avg. len

(words)

ITALIANread utterances 25715 2290 11.2

spontaneous utterances 33492 2374 14.1

total utterances 59207 4664 12.7

SPANISHread utterances 25919 2250 11.52



ENGLISHread utterances 26626 2215 12



GERMANread utterances 10990 903 12.17




Data acquisition featuresData acquisition features

# speakers males females non-native

tot. speech duration

avg. utt. dur

IT 161 68 93 12 9h20’ 7”

SP 150 109 41 8 16h4’ 5.14”

EN 113 46 63 21 7h35’ 6.1”

GER 9 4 5 2 1h21’ 4.9”


TranscriptionTranscription

All the audio files acquired from a speaker were joined together and orthographically transcribed using the tool Transcriber. (http://trans.souceforge.net)

Being domain-restricted, our scenarios led sometimes to the same utterance (matching word sequence). However, the number of repetitions is actually small.


Translation into EnglishTranslation into EnglishTranslation made by simulating the real situation of an

English speaker visiting a foreign city.

E.g.

o what is the address of museo dell'aeronautica Gianni Caproni

Future work: Future work: all data collected for one language translated into the other three languages


Annotation of speech actsAnnotation of speech acts

o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.

hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks


from the QALL-ME benchmark



to greet hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks





to contextualisehallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks






to ask






to thank




REQUESTS

DIRECT:

• wh-questions

• Introduced by:

Could you tell me…

May I know…

• pronounced with ascendant intonation

INDIRECT:

•requests formulated in indirect or implicit ways

NON REQUESTSAll the utterances used by the speaker to introduce himself, to contextualize himself or his request in time and space, to thank, to greet.

ASSERT

GREETINGS

THANKS

OTHER

For our purposes, we used CLaRK, an XML Based System for Corpora Development (http://www.bultreebank.org/clark/index.html).


UTTERANCE

Agreement (speech acts)Agreement (speech acts)

Inter-annotator agreement (calculated on 1000 randomly picked sentences) for ITALIAN:

Dice coefficient = 2C/(A+B)C=number of common annotations

A , B =number of annotations provided by the first and the second annotator

Overall agreement 96.1%

ASSERT 85.5%

DIRECT 97.88%

INDIRECT 97.33%

OTHER 76.47%

THANKS 98.51%

GREETINGS 99.49%


Expected Answer Type Expected Answer Type

For EAT annotation we propose the following scheme:

EAT

PROCEDURAL VERIFICATION FACTOID DEFINITION/DESCRIPTION

• DOMAIN-INDEPENDENT (SEKINE’S ENE HIERARCHY)• DOMAIN-SPECIFIC (QALL-ME ONTOLOGY)


Extracted from Graesser’s (1988) taxonomy

Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.

LREC - Marrakech (Morocco ) , 28-30 May 2008 27

what is the restaurant in via Brennero in Trento

EAT

Sekine’s ENE hierarchy

Qall-me ontology

Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.

LREC - Marrakech (Morocco ) , 28-30 May 2008 28

can you give me the name of the pharmacy in piazza Pasi 20 in Trento

EAT

Sekine’s ENE hierarchy

Qall-me ontology

Annotation of RelationsAnnotation of Relationso Relations among entities: convey and complete the context

in which a specific request has to be interpreted


At what time is the movie il grande capo beginning tomorrow afternoon at Vittoria cinema

o Rel1 (MOVIE, DATE)o Rel2 (MOVIE, STARTINGHOUR)o Rel3 (MOVIE, CINEMA)

o 10% of the Italian questions (referring to Cinema/Movie domain) have been annotated with the 12 relations holding in such domain (Qall-me ontology).

Status of the benchmarkStatus of the benchmark

audio transcr. translat. speech acts

EAT Sekine

EAT ontology

ITALIAN X X X X X X

SPANISH X X X X X in progress

ENGLISH X X --- in progress in progress in progress

GERMAN in progress in progress in progress in progress in progress in progress


Present situation and tentative scheduling:

The QALL-ME benchmark is being made incrementally available at the project website (http://qallme.fbk.eu)

Future workFuture work

Additional annotation layers will be Additional annotation layers will be considered:considered:

o Focus of the questiono Multiwordso Named Entitieso Normalized Temporal Expressionso …


ConclusionsConclusions

o QALL-ME benchmarkQALL-ME benchmark: multilingual resource (for Italian, Spanish, English and German) of annotated spoken requests in the tourism domain.

o Beyond factoido Context aware QA and dynamic changeso QA annotation on spoken requestso Portability of semantic annotation

o Reference resource, useful to train and test ML based QA systems


Thank youThank you{cabrio, kouylekov, magnini, negri}@fbk.eu

{L.Hasler, c.orasan}@wlv.ac.uk

{tomas, vicedo}@disi.ua.es

{neumann, cowe01}@dfki.de

Project website: http://qallme.fbk.eu


Acquisition scenariosAcquisition scenarios

34

SubDomainDesiredOutput

MandatoryItems

OptionalItems

Example from the corpus Example from the corpus

<question id="3118">

<text>buongiorno chiamo da Trento avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</text>

<analysis>

<greetings>buongiorno</greetings>

<assert>chiamo da Trento</assert>

<indirect>avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</indirect></analysis>

<reference>

<ref>

<speaker>spk075_27mar07comd_it_sid023</speaker>

<turn>6</turn>

<originalString>

buongiorno chiamo da Trento ho [mmm] avrei bisogno dell'indirizzo del teatro Auditorium per un [eh] concerto di Salvatore Accardo del 17 gennaio 2007 [b]

</originalString>

</ref>

</reference>

<translation>good morning I am calling from Trento I would like to know the address of Auditorium theatre for Salvatore Accardo's concert on 17th January 2007</translation>

</question>

35

Expected Answer Type (1)Expected Answer Type (1)

The semantic category associated to the desired answer, chosen out of a predefined set of labels (e.g. PERSON, LOCATION, DATE).

o How many colors are in the Italian flag

QUANTITY

o Where is the Uffizi museum

LOCATION

Most QA systems described in literature heavily rely on EAT information, at least in the Answer Extraction phase, to narrow the potential answer candidate search space.

36

Example from the corpusExample from the corpus

37

What are the address and the telephone number of Venezia hotel in Trento

<eats><EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“PostalAddress” eaq= “one”/>

<EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“Contact” eaq= “one”/></eats>

Expected Answer Expected Answer QuantifierQuantifier

Attribute of the EAT that specifies the number of expected items in the answer.

o I would like to know the three colors of the Italian flag

o which movies are on tonight at Multisala Modena

all

The possible values are: one, at least one, all, n.

38

co-funded by the european union the qall-me benchmark: a multilingual resource of annotated spoken...

Documents

project domain

marrakech moroccoqall

marrakech moroccocontext

dataset of requests

marrakech moroccofbk

spoken requests speech

factoid questions

cultural events