co-funded by the european union the qall-me benchmark: a multilingual resource of annotated spoken...
Post on 17-Jan-2016
217 Views
Preview:
TRANSCRIPT
Co-funded by the European Union
The QALL-ME Benchmark:The QALL-ME Benchmark:a Multilingual Resource of a Multilingual Resource of
Annotated Spoken Annotated Spoken Requests for Question Requests for Question
AnsweringAnsweringE. Cabrio, M. Kouylekov, B. Magnini, M. Negri (FBK-Irst)
L. Hasler, C. Orasan, (University of Wolverhampton)
D. Tomas, J.L. Vicedo (University of Alicante)
G. Neumann, C. Weber (DFKI)
Outline:Outline:o Motivations and goalso QALL-ME Projecto QALL-ME Benchmark
oData collectiono Translation into Englisho Speech Acts AnnotationoQuestion Answering Annotationo Annotation of relations
o Conclusion and Future Work
LREC - 28-30 May 2008 - Marrakech (Morocco)
Context: the Qall-me Context: the Qall-me projectproject
QALL-ME (Question Answering Learning QALL-ME (Question Answering Learning technologies in a multiLingual and multiModal technologies in a multiLingual and multiModal Environment): Environment): an EU-funded project aiming at the realization of a shared and distributed infrastructure for Question Answering systems on mobile devices (e.g. mobile phones).
LREC - 28-30 May 2008 - Marrakech (Morocco)
SMSSMS
INPUT OUTPUT
SMSSMS
MMSMMS
VOICE
TEXT
TEXT
VOICE
VIDEO
DIGITAL
ASSISTANT
QALL-ME detailsQALL-ME detailso Reference: FP6 IST-033860o Contract Type: STREP o Start date: October 1st, 2006o Duration: 36 monthso Project Funding: 2.82 M euros
http://qallme.fbk.eu
LREC - 28-30 May 2008 - Marrakech (Morocco)
FBK- Irst, Italy Comdata S,p.A., Italy
DFKI, Germany Ubiest S.p.A., Italy
University of Alicante, Spain Waycom S.r.l., Italy
University of Wolverhampton, UK
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
LREC - 28-30 May 2008 - Marrakech (Morocco)
Motivation: beyond Motivation: beyond factoid…factoid…
has Venezia hotel a restaurant is there a toll free number for the INAIL office in via Gazzoletti in Trento
VERIFICATIONVERIFICATION
where is the INAIL office and how can I get there
how can I get to the pharmacy De Gerloni of Trento
PROCEDURALPROCEDURAL
LREC - 28-30 May 2008 - Marrakech (Morocco)
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
LREC - 28-30 May 2008 - Marrakech (Morocco)
ChallengesChallengeso Context aware QA
o What can I see tonight at cinema o Where is the nearest pharmacy
o Persistent vs dynamic information
o Multiple sources (database, newspaper, web)
LREC - 28-30 May 2008 - Marrakech (Morocco) 8
Challenges related to Challenges related to eventsevents
o Context aware QA
o What can I see tonight at cinema (in Trento)
o Where is the nearest pharmacy (to piazza Duomo)
o Persistent vs dynamic information
o Multiple sources (database, newspaper, web)
LREC - 28-30 May 2008 - Marrakech (Morocco) 9
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).
LREC - 28-30 May 2008 - Marrakech (Morocco)
QA annotationQA annotation
may I know where the ice stadium of Trento is located and at what time it opens
LREC - 28-30 May 2008 - Marrakech (Morocco)
LOCATION
DATE
Expected Answer Type :
MotivationsMotivations
o Providing a dataset of requests dataset of requests beyond factoid questions (e.g. verification, procedural)
o Investigating domain dependent domain dependent vs domain independent annotation schema (Qall-me project domain: cultural events in a town).
o Experimenting the impact of QA annotations QA annotations (e.g. EAT) on spoken requests (speech vs QAQA).
o Investigating of the portability of semantic annotation portability of semantic annotation through languages.
LREC - 28-30 May 2008 - Marrakech (Morocco)
Portability of annotationsPortability of annotations
may I know where the ice stadium of Trento is located
LREC - 28-30 May 2008 - Marrakech (Morocco)
Expected Answer Type: LOCATION
potrei sapere dov’è lo stadiodel ghiaccio di Trento
puedo saber donde esta el estadio de hielo de Trento
ich möchte wissen wo das Eisstadium von Trento ist
Data collectionData collection
o 14645 questions in four different languages:
ITALIAN, ENGLISH, GERMAN, SPANISHITALIAN, ENGLISH, GERMAN, SPANISHo Domain: cultural events in a town
Acquisition:Acquisition:Every speaker performs 30 questions, based on 15 scenarios :o Using a graphical interface, for each scenario is first generated a
spontaneous request and then a written one (previously predefined)o A telephone was used to acquire questions.
LREC - 28-30 May 2008 - Marrakech (Morocco)
Data collectionData collection# words # utterances avg. len
(words)
ITALIANread utterances 25715 2290 11.2
spontaneous utterances 33492 2374 14.1
total utterances 59207 4664 12.7
SPANISHread utterances 25919 2250 11.52
spontaneous utterances 26327 2250 11.70
total utterances 52246 4500 11.61
ENGLISHread utterances 26626 2215 12
spontaneous utterances 36000 2286 15.8
total utterances 62626 4501 13.9
GERMANread utterances 10990 903 12.17
spontaneous utterances 985 77 12.79
total utterances 11975 980 12.22
LREC - 28-30 May 2008 - Marrakech (Morocco)
Data acquisition featuresData acquisition features
# speakers males females non-native
tot. speech duration
avg. utt. dur
IT 161 68 93 12 9h20’ 7”
SP 150 109 41 8 16h4’ 5.14”
EN 113 46 63 21 7h35’ 6.1”
GER 9 4 5 2 1h21’ 4.9”
LREC - 28-30 May 2008 - Marrakech (Morocco)
TranscriptionTranscription
All the audio files acquired from a speaker were joined together and orthographically transcribed using the tool Transcriber. (http://trans.souceforge.net)
Being domain-restricted, our scenarios led sometimes to the same utterance (matching word sequence). However, the number of repetitions is actually small.
LREC - 28-30 May 2008 - Marrakech (Morocco)
Translation into EnglishTranslation into EnglishTranslation made by simulating the real situation of an
English speaker visiting a foreign city.
E.g.
o what is the address of museo dell'aeronautica Gianni Caproni
Future work: Future work: all data collected for one language translated into the other three languages
LREC - 28-30 May 2008 - Marrakech (Morocco)
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
to greet hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
to contextualisehallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
to ask
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
Annotation of speech actsAnnotation of speech acts
o As a starting point for further analyses, it is important to separate within an utterance (each speaker’s turn) what has to be interpreted as the actual request from what does not need an answer.
hallo I am in Trento and I would like to visit a church in the centre of the town I would like to know the name and the location of one of these churches thanks
to thank
LREC - 28-30 May 2008 - Marrakech (Morocco)
from the QALL-ME benchmark
Annotation of speech actsAnnotation of speech acts
REQUESTS
DIRECT:
• wh-questions
• Introduced by:
Could you tell me…
May I know…
• pronounced with ascendant intonation
INDIRECT:
•requests formulated in indirect or implicit ways
NON REQUESTSAll the utterances used by the speaker to introduce himself, to contextualize himself or his request in time and space, to thank, to greet.
ASSERT
GREETINGS
THANKS
OTHER
For our purposes, we used CLaRK, an XML Based System for Corpora Development (http://www.bultreebank.org/clark/index.html).
LREC - 28-30 May 2008 - Marrakech (Morocco)
UTTERANCE
Agreement (speech acts)Agreement (speech acts)
Inter-annotator agreement (calculated on 1000 randomly picked sentences) for ITALIAN:
Dice coefficient = 2C/(A+B)C=number of common annotations
A , B =number of annotations provided by the first and the second annotator
Overall agreement 96.1%
ASSERT 85.5%
DIRECT 97.88%
INDIRECT 97.33%
OTHER 76.47%
THANKS 98.51%
GREETINGS 99.49%
LREC - 28-30 May 2008 - Marrakech (Morocco)
Expected Answer Type Expected Answer Type
For EAT annotation we propose the following scheme:
EAT
PROCEDURAL VERIFICATION FACTOID DEFINITION/DESCRIPTION
• DOMAIN-INDEPENDENT (SEKINE’S ENE HIERARCHY)• DOMAIN-SPECIFIC (QALL-ME ONTOLOGY)
LREC - 28-30 May 2008 - Marrakech (Morocco)
Extracted from Graesser’s (1988) taxonomy
Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.
LREC - Marrakech (Morocco ) , 28-30 May 2008 27
what is the restaurant in via Brennero in Trento
EAT
Sekine’s ENE hierarchy
Qall-me ontology
Sekine’s ENE vs Qall-me Sekine’s ENE vs Qall-me ont.ont.
LREC - Marrakech (Morocco ) , 28-30 May 2008 28
can you give me the name of the pharmacy in piazza Pasi 20 in Trento
EAT
Sekine’s ENE hierarchy
Qall-me ontology
Annotation of RelationsAnnotation of Relationso Relations among entities: convey and complete the context
in which a specific request has to be interpreted
LREC - 28-30 May 2008 - Marrakech (Morocco)
At what time is the movie il grande capo beginning tomorrow afternoon at Vittoria cinema
o Rel1 (MOVIE, DATE)o Rel2 (MOVIE, STARTINGHOUR)o Rel3 (MOVIE, CINEMA)
o 10% of the Italian questions (referring to Cinema/Movie domain) have been annotated with the 12 relations holding in such domain (Qall-me ontology).
Status of the benchmarkStatus of the benchmark
audio transcr. translat. speech acts
EAT Sekine
EAT ontology
ITALIAN X X X X X X
SPANISH X X X X X in progress
ENGLISH X X --- in progress in progress in progress
GERMAN in progress in progress in progress in progress in progress in progress
LREC - 28-30 May 2008 - Marrakech (Morocco)
Present situation and tentative scheduling:
The QALL-ME benchmark is being made incrementally available at the project website (http://qallme.fbk.eu)
Future workFuture work
Additional annotation layers will be Additional annotation layers will be considered:considered:
o Focus of the questiono Multiwordso Named Entitieso Normalized Temporal Expressionso …
LREC - 28-30 May 2008 - Marrakech (Morocco)
ConclusionsConclusions
o QALL-ME benchmarkQALL-ME benchmark: multilingual resource (for Italian, Spanish, English and German) of annotated spoken requests in the tourism domain.
o Beyond factoido Context aware QA and dynamic changeso QA annotation on spoken requestso Portability of semantic annotation
o Reference resource, useful to train and test ML based QA systems
LREC - 28-30 May 2008 - Marrakech (Morocco)
Thank youThank you{cabrio, kouylekov, magnini, negri}@fbk.eu
{L.Hasler, c.orasan}@wlv.ac.uk
{tomas, vicedo}@disi.ua.es
{neumann, cowe01}@dfki.de
Project website: http://qallme.fbk.eu
LREC - 28-30 May 2008 - Marrakech (Morocco)
Acquisition scenariosAcquisition scenarios
34
SubDomainDesiredOutput
MandatoryItems
OptionalItems
Example from the corpus Example from the corpus
<question id="3118">
<text>buongiorno chiamo da Trento avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</text>
<analysis>
<greetings>buongiorno</greetings>
<assert>chiamo da Trento</assert>
<indirect>avrei bisogno dell'indirizzo del teatro Auditorium per un concerto di Salvatore Accardo del 17 gennaio 2007</indirect></analysis>
<reference>
<ref>
<speaker>spk075_27mar07comd_it_sid023</speaker>
<turn>6</turn>
<originalString>
buongiorno chiamo da Trento ho [mmm] avrei bisogno dell'indirizzo del teatro Auditorium per un [eh] concerto di Salvatore Accardo del 17 gennaio 2007 [b]
</originalString>
</ref>
</reference>
<translation>good morning I am calling from Trento I would like to know the address of Auditorium theatre for Salvatore Accardo's concert on 17th January 2007</translation>
</question>
35
Expected Answer Type (1)Expected Answer Type (1)
The semantic category associated to the desired answer, chosen out of a predefined set of labels (e.g. PERSON, LOCATION, DATE).
o How many colors are in the Italian flag
QUANTITY
o Where is the Uffizi museum
LOCATION
Most QA systems described in literature heavily rely on EAT information, at least in the Answer Extraction phase, to narrow the potential answer candidate search space.
36
Example from the corpusExample from the corpus
37
What are the address and the telephone number of Venezia hotel in Trento
<eats><EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“PostalAddress” eaq= “one”/>
<EAT type= “FACTOID” sekine=“ADDRESS_OTHER” qallme=“Contact” eaq= “one”/></eats>
Expected Answer Expected Answer QuantifierQuantifier
Attribute of the EAT that specifies the number of expected items in the answer.
o I would like to know the three colors of the Italian flag
o which movies are on tonight at Multisala Modena
all
The possible values are: one, at least one, all, n.
38
top related